Choosing the right LLM for your use case and Model Routing

Choosing the right LLM for your use case and Model Routing

Table of Contents

Choosing the right Large Language Model (LLM) is crucial for ensuring the effectiveness and efficiency of your application. LLMs are powerful tools that can be used for a wide range of tasks, from content moderation to query generation and dialogue systems.

When selecting a Large Language Model (LLM), it is crucial to evaluate the model's capabilities and limitations to ensure it meets the specific needs of your project. Choosing the wrong LLM can lead to a range of negative outcomes, from ** inefficient use of resources to compromised results **.

  1. Inefficient Resource Utilization Choosing an LLM that is not optimized for your task can result in inefficient use of computational resources. This can lead to longer processing times, increased costs, and a higher risk of errors. For instance, if you are working on a project that requires processing large volumes of text data, selecting an LLM that is not designed for this task can result in slow performance and increased computational costs.
  2. Compromised Results The primary purpose of an LLM is to generate high-quality text based on the input provided. If the chosen LLM is not capable of producing the desired output, the results can be compromised. This can lead to a range of issues, including: Inaccurate Information: If the LLM is not trained on a diverse range of texts, it may not be able to provide accurate information on specific topics. Poor Language Understanding: An LLM that lacks understanding of language nuances can produce text that is unclear, ambiguous, or even nonsensical. Lack of Contextual Understanding: If the LLM is not designed to understand context, it may not be able to generate text that is relevant to the topic or situation.
  3. Difficulty in Integration Choosing an LLM that is not compatible with your existing infrastructure can make integration challenging. This can lead to additional costs and delays in the development process. For example, if you are working on a project that requires integrating the LLM with a specific software or platform, selecting an LLM that is not compatible can result in significant difficulties.
  4. Limited Customization Options Some LLMs may not offer the level of customization required for your project. This can limit your ability to tailor the model to your specific needs, leading to suboptimal results. For instance, if you need to integrate the LLM with a specific database or API, selecting an LLM that does not support this integration can be problematic.
  5. Security Risks Choosing an LLM that is not secure can pose significant risks to your project. This can include: Data Breaches: If the LLM is not designed with security in mind, it can be vulnerable to data breaches, compromising sensitive information. Malicious Use: An LLM that is not secure can be exploited by malicious actors, leading to unauthorized access or manipulation of the model.

Challenges in evaluating Models

Large Language Model (LLM) application evaluation is currently an exhausting and time-consuming affair, which is precisely why we don't hear about it as much. In contrast, a well-defined set of measures, such as mean squared error (MSE), precision, and recall, are employed for assessing traditional machine learning models, like regression and classification.

Some reasons as to why evaluating large language models is challenging include: 1

  • Complex Tasks: LLMs are designed to perform complex tasks such as summarization, long-form question-answering, and code generation. These tasks require metrics that are more nuanced than traditional precision and recall metrics used for simpler classification tasks.
  • Probabilistic Nature: LLMs are inherently probabilistic, meaning that even small changes in the input can significantly impact the output. This makes it difficult to define a single "ground truth" for evaluation.
  • Time-Consuming Ground Truth Creation: Creating ground truth for LLM applications can be time-consuming and labor-intensive. This is especially true for tasks that require human evaluation, such as faithfulness and relevance.
  • Human evaluations: Abstractive summaries and other tasks requiring subjective judgement are less efficient when done by humans because they are costly, time-consuming, and possibly inconsistent.

Fundamentals of Evaluation metrics

How does one go about evaluating LLMs, if we do not have definite metrics to gauge the performance of each model?

Well, when we talk about the output of an LLM, it can be a response to a query, a task like summarization, translation, or a dialogue or even classification and categorization.

The following table provides a basic overview of the evaluation metrics that can be used to evaluate LLMs based on the task at hand.

TaskEvaluation Parameters
Content ModerationRecall and precision on toxicity and bias
Query GenerationCorrect output syntax and attributes, extracts the right information upon execution
Dialogue (chatbots, summarization, Q&A)Faithfulness, relevance
Table 1: Basics of Evaluating LLMs. Courtesy of: Apoorva Joshi

As the field of conversational AI continues to advance, developers are faced with the challenge of evaluating the performance of open-ended dialogue systems. Unlike more straightforward tasks like content moderation or query generation, where the expected answers are more definitive, open-ended dialogue presents a unique set of evaluation challenges. The key to effective evaluation in this domain is to focus on two critical aspects: factual consistency (faithfulness) and relevance of the system's responses to the user's questions. While this approach may still be subject to some of the challenges we face with large language models (LLMs) today, such as hallucinations and biases, it scales better than human evaluation alone.

When it comes to the evaluation process, it's important to assess each component of your system separately, as well as the overall performance. For example, in Retrieval-Augmented Generation (RAG) systems, you'll want to evaluate the retrieval and generation components individually to ensure that the right context is being retrieved and suitable answers are being generated. Similarly, in tool-calling agents, you'll need to validate the intermediate responses from each of the tools.

Significance of Evaluation Metrics

What do each metric mean? Where are they used to gauge the performance of an LLM?

Evaluation MetricRelevance to Use Case
Age AppropriatenessEssential for educational content aimed at children
Response RelevanceCrucial for consumer-facing applications like customer service bots and information retrieval systems
Question-Answering AccuracyKey in research, analytical tasks, and educational applications
ToxicityVital for all public-facing applications to avoid offensive or harmful content
BLEU ScoreMeasures similarity between machine-generated text and human reference, often used in translation tasks
ROUGE ScoreEvaluates automatic summarization and machine translations, focusing on recall of reference content
Table 2: Advanced Evaluation Techniques. Courtesy of: aisera.com
  • BLEU: Bilingual evaluation understudy (BLEU), often used for machine translation, calculates the overlap of n-grams(a contiguous sequence of n items from a given text sample) between the output of the model and a set of human-written reference translations. A higher BLEU score indicates better text generation, as the model's output is more similar to the reference. However, it's worth noting that BLEU has limitations, including its inability to evaluate semantic meaning or the relevance of the generated text. 2
  • ROUGE: Recall-oriented understudy for gisting evaluation (ROUGE) is another prominent evaluation metric useful for tasks such as text summarization. ROUGE includes several variants such as ROUGE-N, ROUGE-L, and ROUGE-S. 2

Here are the formulas for the different variants of the Rouge metric:

Rouge

The original Rouge metric measures the overlap between the generated text and the reference text using the following formula:

ROUGE=∑n=1Nmax(0,min(n,len(R))−n+1)len(R)\text{ROUGE} = \frac{\sum_{n=1}^{N} \text{max}(0, \text{min}(n, \text{len}(R)) - n + 1)}{\text{len}(R)}

where:

  • len(R)\text{len}(R) is the length of the reference text.
  • nn is the number of n-grams (sequences of n words) considered.
  • NN is the maximum number of n-grams considered.

Rouge-N

Rouge-N is a variant of Rouge that considers n-grams of a specific length nn. The formula is:

ROUGE-N=∑i=1ncount(i)len(R)\text{ROUGE-N} = \frac{\sum_{i=1}^{n} \text{count}(i)}{\text{len}(R)}

where:

  • count(i)\text{count}(i) is the number of ii-grams (sequences of ii words) that are common to both the generated text and the reference text.
  • nn is the length of the nn-grams considered.

Rouge-L

Rouge-L is another variant that focuses on the longest common subsequences between the generated text and the reference text. The formula is:

ROUGE-L=∑i=1nmax(0,min(i,len(R))−i+1)len(R)\text{ROUGE-L} = \frac{\sum_{i=1}^{n} \text{max}(0, \text{min}(i, \text{len}(R)) - i + 1)}{\text{len}(R)}

where:

  • len(R)\text{len}(R) is the length of the reference text.
  • nn is the maximum number of n-grams considered.

Rouge-S

Rouge-S is a variant that uses skip-bigram statistics to measure the similarity between the generated text and the reference text. The formula is:

ROUGE-S=∑i=1ncount(i)len(R)\text{ROUGE-S} = \frac{\sum_{i=1}^{n} \text{count}(i)}{\text{len}(R)}

where:

  • count(i)\text{count}(i) is the number of ii-grams (sequences of ii words) that are common to both the generated text and the reference text.
  • nn is the length of the nn-grams considered.

These formulas provide a way to quantify the similarity between the generated text and the reference text, allowing for the evaluation of text generation models using the Rouge metric.

  • ** MoverScore: ** MoverScore is a more recent evaluation metric designed to measure the semantic similarity between two pieces of text. MoverScore uses Word Mover's Distance, a method that calculates the minimum distance that words in one text need to “travel” to reach the exact distribution of words in another text. It then adjusts this distance based on the importance of different words to the overall meaning of the text. MoverScore offers a more nuanced evaluation of semantic similarity than some older metrics, but it’s computationally intensive and may not always align with human judgment.
  • ** Perplexity: ** Perplexity quantifies how well a model predicts a sample—in this case, a piece of text. A lower perplexity score means the model is better at sample prediction. In the context of LLMs, it measures the model's uncertainty in predicting the next word in a sequence. While perplexity can provide a useful quantitative measure of a model's performance, it doesn’t account for qualitative aspects of the generated text, such as its coherence or relevance. Therefore, it's often used alongside other evaluation metrics for a more robust assessment.
  • ** Exact match: ** Exact match is a widely used evaluation metric for question-answering and machine translation. It measures the percentage of predictions that exactly match the reference answers. While an exact match can be a useful indicator of a model's accuracy, it doesn't consider near misses or partially correct answers. It also doesn't account for the semantic similarity between the generated and reference texts. Therefore, it's often used in conjunction with other, more nuanced evaluation metrics.
  • ** Precision: ** Precision measures the proportion of predicted positive observations that are correct. In LLMs, precision would be the fraction of correctly predicted words or phrases over the total number of words or phrases predicted by the model. A high precision score indicates that when the model predicts a word or phrase, it's likely to be correct. However, precision doesn't consider the relevant words or phrases the model might have missed (false negatives), and hence it’s used along with recall for a more balanced evaluation.
  • ** Recall: ** Recall, also known as sensitivity or true positive rate, measures the proportion of actual positives correctly identified. Recall is a fraction of the correctly predicted words or phrases over the total number of correct words or phrases in the reference text. A high recall score indicates the model's efficiency at detecting relevant words or phrases. However, the recall doesn't consider the number of irrelevant words or phrases that the model might have incorrectly predicted (false positives), and hence it is often paired with precision for a more comprehensive evaluation.
  • ** F1 score: ** The F1 score is a popular evaluation metric that provides a balanced measure of a model's performance by considering both precision and recall. It's the harmonic mean of precision and recall, giving equal weight to both these metrics. A high F1 score indicates that the model has a good balance of precision (it can correctly predict words or phrases) and recall (it can correctly identify relevant words or phrases from the reference text). The F1 score ranges between 0 and 1, where 1 indicates perfect precision and recall. It's particularly useful in scenarios where both false positives and false negatives are equally important.

Comparative Analysis

Here's a little table to help you choose the appropriate performance indicators for your application so that the solution is well-aligned to the requirements.

Performance IndicatorMetricApplication in LLM Evaluation
AccuracyTask Success RateMeasuring the model’s ability to produce correct responses to prompts
FluencyPerplexityAssessing the natural flow and readability of text generated by the LLM
RelevanceROUGE ScoresEvaluating content relevance and alignment with user input
BiasDisparity AnalysisIdentifying and mitigating biases within model responses
CoherenceCoh-MetrixAnalyzing logical consistency and clarity over longer stretches of text
Table 3: Performance Indicators and their applications. Courtesy of: aisera.com

Fundamentals of Evaluation Benchmarks

GLUE, MMLU, and AlpacaEval are three significant benchmarks used to evaluate the performance of Large Language Models (LLMs). Each benchmark assesses different aspects of LLM capabilities, providing a comprehensive understanding of their strengths and weaknesses. 34

GLUE (General Language Understanding)

GLUE is a benchmark designed to evaluate LLMs on natural language understanding tasks. It includes several sub-benchmarks, such as HellaSWAG and MRPC, which test the models' ability to perform tasks like question answering, text classification, and document summarization. GLUE is considered a less challenging benchmark compared to SuperGLUE, but it remains an important tool for evaluating LLMs on natural language inference tasks.

MMLU (Massive Multitask Language Understanding)

MMLU is a benchmark that measures the ability of LLMs to multitask by evaluating their performance on a variety of tasks. These tasks include question answering, text classification, document summarization, and more. MMLU is designed to assess how well LLMs can handle complex, real-world scenarios and their ability to multitask effectively.

AlpacaEval

AlpacaEval is an automated benchmarking tool that evaluates the performance of LLMs in following instructions. It uses the AlpacaFarm dataset to measure models' ability to generate responses that align with human expectations. AlpacaEval is a rapid and cost-effective way to assess the capabilities of LLMs, making it a valuable tool for model development and evaluation. These benchmarks, along with others like HELM and SuperGLUE, provide a comprehensive framework for evaluating LLMs. Each benchmark is designed to test specific aspects of LLM capabilities, ensuring that model developers can assess their models' strengths and weaknesses accurately. By using a combination of these benchmarks, developers can gain a deeper understanding of their models' performance and make informed decisions about their development and deployment.

GEval

Leaderboard 5

To address the complex challenge of evaluating the performance of LLMs, the LMSys organization has launched the Chatbot Arena Leaderboard, a groundbreaking platform that ranks LLMs based on their conversational abilities.

Conversational AI systems are designed to interact with humans in a natural and intuitive manner. However, evaluating their performance is not a straightforward task. Traditional benchmarks, such as GLUE and SuperGLUE, focus on specific tasks like question answering and text classification. These benchmarks provide valuable insights but do not fully capture the complexities of conversational AI.

The LMSys Chatbot Arena Leaderboard (Chiang et al. 2024) addresses this limitation by providing a comprehensive evaluation platform that assesses LLMs in various conversational tasks. The platform uses an ELO rating system to rank models based on anonymous voting data collected from users interacting with the models. This approach ensures that the leaderboard is not biased towards any particular model or dataset.

The LMSys Chatbot Arena Leaderboard is not just a static ranking system. The organizers are actively working on expanding the platform to better capture the long-tail capabilities of LLMs. This includes incorporating expert-designed prompts and judges to evaluate complex reasoning and other advanced conversational skills. The leaderboard is also designed to be collaborative, with the goal of advancing the evaluation of conversational AI systems in a transparent and community-driven manner. This approach encourages open communication and knowledge sharing among developers, researchers, and users, ultimately leading to more capable and effective conversational AI systems.

The columns in the LMSys Chatbot Arena Leaderboard provide detailed information about each model participating in the evaluation. Here is a breakdown of each column:

1. Arena Elo

  • Description: The Arena Elo rating is a measure of a model's performance in the Chatbot Arena. It is calculated using the Elo rating system, which is a method for calculating the relative skill levels of players in competitive games and sports. The Elo rating system works well for evaluating models in the Chatbot Arena because it takes into account the pairwise battles between models and provides a comprehensive ranking. The Arena Elo ratings on the LMSys Chatbot Arena Leaderboard are derived using the Elo rating system. Here is a step-by-step explanation of how the ratings are calculated:

  1. Pairwise Comparisons: The Elo ratings are computed based on the results of pairwise comparisons between the LLMs in the Chatbot Arena. Users can interact with the models and vote on which one performs better in a given conversation.

  2. Elo Rating System: The Elo rating system is a method for calculating the relative skill levels of players in competitive games and sports. It has been widely adopted for evaluating the performance of large language models (LLMs) in conversational tasks.

  3. Notebook Calculations: The specific calculations for deriving the Arena Elo ratings are performed in a Colab notebook, as referenced in the search results. This notebook uses a Bradley-Terry model, a statistical method for modeling pairwise comparisons, to compute the Elo ratings.

  4. Leaderboard Presentation: The Arena Elo ratings, along with other metrics like the 95% confidence interval, number of votes, and model information, are then displayed on the LMSys Chatbot Arena Leaderboard.

  5. Continuous Updates: The leaderboard is updated continuously as more users interact with the models and provide their votes, allowing for a dynamic and up-to-date evaluation of the LLMs' conversational abilities.

  • Example Comparison For example, if a user interacts with both GPT-4 and Llama 3 in a conversation and votes that GPT-4 performed better, the Elo rating for GPT-4 would increase. Conversely, the Elo rating for Llama 3 would decrease. This process is repeated for each pairwise comparison, and the Elo ratings are updated accordingly.

2. 95% CI

  • Description: The 95% Confidence Interval (CI) is a statistical measure that provides a range of values within which the true value of a model's performance is likely to lie. In this context, it represents the uncertainty associated with the model's Elo rating. The CI is calculated based on the number of votes and the variability in the voting data.

3. Votes

  • Description: The number of votes received by each model in the Chatbot Arena. This indicates the level of engagement and the number of users who have interacted with the model.

4. Organization

  • Description: The organization or entity responsible for developing and maintaining the model. This includes both open-source and closed-source models.

5. License

  • Description: The license under which the model is distributed. This can be open-source, closed-source, or other types of licenses.

6. Knowledge Cutoff

  • Description: The knowledge cutoff refers to the point in time when the model's training data was last updated. This is important because it indicates the model's ability to handle new information and adapt to changing knowledge bases.

Evaluation of a RAG (Retrieval Augmented Generation) Application

🚧👷🏽‍♂️ This section is a work in progress!

References:

Footnotes

  1. Joshi, Apoorva. “How to Evaluate Your LLM Application | MongoDB.” Mongodb.com, 17 June 2024, www.mongodb.com/developer/products/atlas/evaluate-llm-applications-rag/?utm_campaign=devrel&utm_source=youtube&utm_medium=organic_social&utm_content=AH10-Ec0qG8&utm_term=apoorva.joshi#how-to-evaluate-a-rag-application. Accessed 22 June 2024. ↩

  2. “A Comprehensive Guide to LLM Evaluation and Benchmarking.” www.turing.com, 22 Jan. 2024, www.turing.com/resources/understanding-llm-evaluation-and-benchmarks#b.-limitations-of-existing-benchmarks. Accessed 22 June 2024. ↩ ↩2

  3. Nucci, Antonio. “LLM Evaluation: Large Language Model Performance Metrics.” Aisera: Best Generative AI Platform for Enterprise, 27 Dec. 2023, www.aisera.com/blog/llm-evaluation/. Accessed 22 June 2024. ↩

  4. “Chatbot Arena Leaderboard - a Hugging Face Space by Lmsys.” Huggingface.co, www.huggingface.co/spaces/lmsys/chatbot-arena-leaderboard. Accessed 22 June 2024. ↩

  5. Chiang, Wei-Lin, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. "Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference." arXiv, 2024, eprint 2403.04132, cs.AI. ↩