Statistical significance in AI evals: error bars and sample size
AI evaluations are experiments that need statistical analysis. Reporting standard errors and confidence intervals helps avoid misinterpreting differences. Clustering and cross-validation are important for reliable conclusions.
Why significance testing matters
Statistical significance testing helps researchers decide whether an observed difference between two models is likely due to chance. A survey of recent NLP papers found that significance testing is often ignored or misused, even though experimental results are highly valued the ACL Anthology paper.
The choice of evaluation metric can also affect conclusions. Researchers at Stanford show that nonlinear or discontinuous metrics can create apparent emergent abilities, while continuous metrics yield smooth performance gains the arXiv paper on emergent abilities.
Error bars and confidence intervals
An AI evaluation score is an average over questions, but the observed average is only an estimate. Anthropic recommends using the Central Limit Theorem to compute the standard error of the mean (SEM), and then a 95% confidence interval as the mean plus or minus 1.96 times the SEM Anthropic's statistical approach.
Reporting the SEM alongside each score lets other researchers assess the precision of the result and avoid overinterpreting small differences.
Clustered data and question design
Many evals contain groups of related questions, such as reading comprehension items on the same passage. Ignoring this clustering leads to underestimated standard errors. The paper 'Adding Error Bars to Evals' proposes clustering standard errors on the unit of randomization the arXiv paper on error bars.
Clustered standard errors can be several times larger than naive ones, so proper clustering is necessary to avoid false positives.
Sample size and cross-validation
Choosing the number of evaluation questions affects the width of error bars. Cross-validation is a standard technique to estimate generalization performance when tuning model parameters. The scikit-learn documentation states that evaluating a model on the same data used for training is a methodological mistake and recommends k-fold cross-validation scikit-learn's cross-validation guide.
In k-fold cross-validation, the training set is split into k parts, and the model is trained on k-1 parts and validated on the remaining part, with the average score reported.
| Recommendation | Key rationale | Typical implementation |
|---|---|---|
| Report standard error of the mean | Quantifies sampling variability | Compute from Central Limit Theorem |
| Use 95% confidence intervals | Indicates range of plausible values | Mean ± 1.96 × SEM |
| Cluster standard errors for grouped questions | Avoids underestimation of uncertainty | Cluster on unit of randomization |
| Use cross-validation for model tuning | Prevents overfitting to test set | k-fold split |
| Choose continuous or linear metrics | Nonlinear metrics can create artifacts | Prefer linear scoring over discontinuous |
| Test for statistical significance | Ensures differences are not chance | Select appropriate test per protocol |
Frequently asked questions
What is a standard error of the mean in AI evals?
It is the standard deviation of the sampling distribution of the mean eval score, estimated using the Central Limit Theorem. It indicates how much the observed average would vary across random samples of questions.
When should clustered standard errors be used?
When eval questions come in groups, such as reading comprehension passages with multiple questions, clustering on the unit of randomization prevents underestimating the standard error.
How does sample size affect error bars?
A larger sample of questions generally reduces the standard error and narrows confidence intervals, provided the questions are representative.
Why is cross-validation important in model evaluation?
Cross-validation prevents overfitting by partitioning data into multiple train/test splits, giving a more reliable estimate of generalization performance than a single split.
How can metric choice change evaluation conclusions?
Nonlinear or discontinuous metrics can make model performance appear to jump sharply, creating the illusion of emergent abilities, while linear metrics often show gradual improvement.