Knowledge and language understanding benchmarks: from GLUE to MMLU
Language understanding benchmarks have evolved from GLUE's diverse NLU tasks to MMLU's massive multitask coverage. Each benchmark addresses different limitations, from task diversity to truthfulness and holistic evaluation. This article explains the purpose and design of these benchmarks and how to interpret them.
.jpg){kind=link}
Why general benchmarks matter
Language understanding technology must be general, not tailored to a single task. The GLUE benchmark was introduced to evaluate models across a diverse range of existing natural language understanding tasks. It is model-agnostic and encourages sharing knowledge across tasks because some tasks have very limited training data, as described in the GLUE paper.
GLUE includes a hand-crafted diagnostic test suite that enables detailed linguistic analysis of models. This allows researchers to probe specific aspects of natural language understanding beyond aggregate scores.
Baseline evaluations using multi-task and transfer learning did not immediately give substantial improvements over training a separate model per task, indicating room for developing more general and capable systems.
SuperGLUE raises the difficulty
Within a year, performance on GLUE surpassed the level of non-expert humans, leaving limited headroom for further research. To continue progress, SuperGLUE was introduced with a new set of more difficult language understanding tasks, as noted in the SuperGLUE paper.
SuperGLUE is styled after GLUE and includes a software toolkit and a public leaderboard to track model performance on these harder tasks.
MMLU and TruthfulQA address knowledge and truthfulness
MMLU (Massive Multitask Language Understanding) is a test covering 57 tasks including elementary mathematics, US history, computer science, law, and more. Models must possess extensive world knowledge and problem solving ability to achieve high accuracy, according to the MMLU paper.
Most recent models performed near random chance, but the largest GPT-3 model did better by about 20 percentage points on average. Even so, across all 57 tasks, the best models still fell far short of expert-level accuracy, and their scores were uneven.
TruthfulQA measures whether a language model generates truthful answers. It comprises 817 questions spanning 38 categories, including health, law, finance, and politics, crafted to elicit false answers from human misconceptions, as described in the TruthfulQA paper.
Broad evaluation and how to check a benchmark
BIG-bench quantifies and extrapolates the capabilities of language models through a large collaborative effort, probing model abilities across many tasks, as outlined in the BIG-bench paper.
HELM (Holistic Evaluation of Language Models) taxonomizes scenarios and metrics, adopting a multi-metric approach that measures accuracy, calibration, reliability, fairness, bias, toxicity, and efficiency across 16 core scenarios. It found that prior to HELM, models were evaluated on an average of only 17.9% of those scenarios, according to the HELM paper.
When checking any benchmark, consider the diversity of tasks, whether metrics go beyond accuracy, and the coverage of scenarios. A good benchmark should expose trade-offs and avoid narrowly tailored tests.
| Benchmark | Focus | Number of tasks | Key feature |
|---|---|---|---|
| GLUE | General NLU evaluation | Not stated | Diagnostic suite |
| SuperGLUE | Harder tasks | Not stated | Public leaderboard |
| MMLU | Massive multitask knowledge | 57 tasks | World knowledge and reasoning |
| TruthfulQA | Truthfulness | 817 questions across 38 categories | Measures falsehood mimicry |
| HELM | Holistic evaluation | 16 core scenarios | Multi-metric approach |
Frequently asked questions
What is GLUE?
GLUE is a benchmark that evaluates models across diverse natural language understanding tasks, with a diagnostic suite for linguistic analysis, as described in the GLUE paper.
How does MMLU differ from GLUE?
MMLU covers 57 tasks across many domains, requiring extensive world knowledge, while GLUE focuses on a diverse set of existing NLU tasks without specifying a count.
Why was TruthfulQA created?
TruthfulQA was designed to measure whether language models mimic human falsehoods, using 817 questions across 38 categories that elicit common misconceptions.
What does HELM add to benchmark evaluation?
HELM measures seven metrics including accuracy, calibration, reliability, fairness, bias, toxicity, and efficiency across 16 core scenarios, exposing trade-offs beyond simple accuracy.