How AI models are evaluated: metrics, benchmarks and human review
AI model evaluation uses quantitative metrics for classification and regression, standardized benchmarks for language models, and human oversight to catch subtle failures. Metrics like accuracy, precision, and recall measure prediction quality, while benchmarks such as MMLU and BBQ assess capabilities and biases. Human review and risk management frameworks address limits of automated tests.
Core evaluation metrics
Model evaluation measures how well a machine learning model performs on making predictions about the real world. Classification models label data into categories, while regression models output a continuous number, and each type requires different metrics. For classification, common metrics include accuracy, precision, and recall, which are derived from a confusion matrix that tracks true positives, false positives, true negatives, and false negatives, as IBM's model evaluation overview explains.
Accuracy alone can mislead when classes are imbalanced. For example, a cancer detector that always predicts the majority class can appear highly accurate while missing rare positive cases. Precision and recall offer a more detailed view: precision is the share of positive predictions that were correct, while recall is the share of actual positives that the model caught. Tuning a model often involves trading precision for recall or vice versa. For regression models, metrics such as squared error and absolute error measure the distance between predictions and true values, and the choice of a scoring function should align with the intended use, as scikit-learn's model evaluation documentation describes.
Dataset splitting and validation
To evaluate a model honestly, it must be tested on examples it has not seen during training. Google's Machine Learning Crash Course recommends splitting the original dataset into three subsets: a training set, a validation set, and a test set. The validation set is used during training to tune hyperparameters, while the test set is reserved for a final check.
Repeated use of the same validation or test set can cause the model to implicitly fit those sets, so the sets 'wear out' over time. Fresh data should be collected to refresh them. Also, duplicates between the training and test sets can inflate performance, so they should be removed before testing.
Benchmarks for language models
Language models are often evaluated with standardized benchmarks such as MMLU and BBQ. Anthropic's discussion of evaluation challenges points out that MMLU measures accuracy on 57 tasks ranging from mathematics to law, while BBQ tests social biases along nine dimensions. These benchmarks provide simple scores, but they have limitations, including contamination from training data and sensitivity to formatting changes.
A broader approach is Holistic Evaluation of Language Models (HELM), presented in an arXiv paper by Stanford CRFM. HELM measures seven metrics, including accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency, across 16 core scenarios. It aims to improve transparency by releasing raw prompts and completions for further analysis. Human review and red teaming also play a role in uncovering issues that automated benchmarks miss, as the same paper and Anthropic's experience suggest.
Risk management frameworks
Automated metrics and benchmarks cannot capture all risks. NIST's AI Risk Management Framework emphasizes the need to incorporate trustworthiness considerations into the design, development, and evaluation of AI systems. The framework is voluntary and provides guidance for managing risks to individuals, organizations, and society.
The framework offers a structured approach to identify, assess, and manage AI risks, and it aligns with other governance efforts. By integrating evaluation into broader risk management, organizations can address concerns that simple performance scores overlook.
| Metric | Definition | When it matters |
|---|---|---|
| Accuracy | The share of correct predictions among all predictions. | Works well when classes are balanced. |
| Precision | Of all positive predictions, how many are correct. | When false positives are costly. |
| Recall | Of all actual positives, how many are caught. | When false negatives are costly. |
Frequently asked questions
What is the difference between accuracy and precision?
Accuracy is the share of all predictions that are correct, while precision is the share of positive predictions that are correct. Accuracy can be misleading when classes are imbalanced, whereas precision focuses on the cost of false positives.
Why use a separate test set?
A separate test set, not used during training, provides a stronger measure of how the model will perform on unseen data. Repeated use of the same test set can lead to overfitting to that set.
What are MMLU and BBQ?
MMLU is a multiple-choice benchmark measuring accuracy on 57 tasks, while BBQ tests social biases across nine dimensions. Both have limitations, including sensitivity to formatting and potential contamination.
How does human review complement automated evaluation?
Automated metrics miss subtle issues such as bias or safety risks. Human review, including red teaming and domain expert evaluation, helps uncover failures that quantitative scores overlook.
What is the NIST AI Risk Management Framework?
NIST's framework is a voluntary set of guidelines for incorporating trustworthiness into AI design, development, and evaluation, focusing on managing risks to individuals, organizations, and society.