AI leaderboards and their limits: Goodhart's law and missing context
AI leaderboards rank models by benchmark scores, but the scores are sensitive to small details and omit practical costs. When a measure becomes a target, models are optimized for the benchmark rather than for real-world utility. This article explains these limits and suggests how to read leaderboards responsibly.
The trap of Goodhart's law in AI leaderboards
AI leaderboards, such as those based on the GLUE benchmark, have driven rapid improvements in natural language processing by rewarding models that achieve higher scores. However, as Goodhart's law warns, when a measure becomes a target, it ceases to be a good measure. The critique of NLP leaderboards by Ethayarajh and Jurafsky argues that this focus on a single accuracy number ignores qualities the community values, including compactness, fairness, and energy efficiency.
The authors frame leaderboards and practitioners as consumers with different utility functions. A model that is inefficient in time or energy may rank high on the leaderboard but provide little utility in practice, since the cost is borne only by the user. They advocate for reporting statistics of practical concern, such as model size, energy efficiency, and inference latency.
Sensitivity to benchmark choices
The performance of an algorithm on a benchmark can be altered significantly by which tasks are included. The benchmark lottery paper by Dehghani and colleagues shows that many factors other than algorithmic superiority can make a method appear better, and this fragility can lead to biased progress in the community.
For multiple-choice benchmarks, even small perturbations can shift rankings. The study on LLM leaderboard sensitivity found that changing the order of choices or the method of answer selection can change ranks by up to 8 positions, explaining that this happens across three categories of perturbation and recommending a hybrid scoring method.
Elo ratings and human preference evaluations
Elo ratings, originally designed for chess, are now used to evaluate large language models through paired comparisons. The Elo evaluation study by Boubdir and colleagues examines whether this system satisfies reliability and transitivity, and finds that individual Elo computations exhibit volatility and that the axioms are not always satisfied, raising questions about current comparative evaluations.
An alternative is crowdsourced human preference. The Chatbot Arena platform described by Chiang and colleagues uses pairwise comparisons from a diverse user base, collecting over 240K votes, and confirms that crowdsourced votes agree well with expert raters, making it a widely referenced leaderboard.
Dynamic benchmarks and practical guidance
Static benchmarks eventually saturate as models become too good at them. The Dynabench proposal by Kiela and colleagues introduces a platform where annotators create examples that a target model misclassifies but other humans can answer, making dataset creation and model assessment dynamic and more reliable.
To interpret leaderboards correctly, practitioners should look beyond the headline number. The sources suggest reporting model size, energy, and latency, as well as using hybrid scoring for multiple-choice tasks and validating human evaluations against experts. The table below summarizes the evaluation options and their limitations.
| Method | Mechanism | Key limitation | Suggested improvement |
|---|---|---|---|
| Static benchmark (e.g., GLUE, MMLU) | Fixed dataset with a single accuracy score | Rankings are sensitive to small changes; omits model size and energy | Report model size and latency; use hybrid scoring |
| Dynamic benchmark (Dynabench) | Human-and-model-in-the-loop dataset creation | May be costly and slow | Use as a complement to static benchmarks |
| Elo rating for LLMs | Paired comparisons with an Elo algorithm | Volatile and not always transitive | Adjust hyperparameters carefully |
| Crowdsourced pairwise (Chatbot Arena) | Users vote on model outputs | Needs a large number of votes | Validate against expert ratings |
Frequently asked questions
What is Goodhart's law in AI leaderboards?
When a specific benchmark score becomes the target for optimization, the score loses its validity as a measure of real-world performance. Models may overfit to the test set, and other qualities like efficiency and fairness are ignored.
How can small changes in a benchmark affect rankings?
Changing the order of answer choices or the method of selecting answers can shift rankings by up to 8 positions, as shown for multiple-choice benchmarks.
What context do leaderboards usually omit?
Model size, energy consumption, and inference latency are often missing, though these affect practicality.
Are human preference evaluations more reliable?
Crowdsourced pairwise comparisons can agree with expert raters, but Elo-based rankings can be volatile and not always transitive.