Reasoning benchmarks: math problems, science questions and puzzles
Reasoning benchmarks evaluate how well AI systems solve problems that require multi-step thinking. They include datasets of math word problems, science questions and abstract puzzles. These benchmarks expose limitations in current models and guide research on better reasoning.
Math word problems: GSM8K and MATH
GSM8K is a dataset of 8.5K grade-school math word problems introduced to diagnose failures in multi-step reasoning. Even large transformer models struggle with this conceptually simple distribution, according to the GSM8K paper. The authors propose training verifiers to judge candidate solutions and select the highest-ranked one.
MATH is a more challenging set of 12,500 competition mathematics problems, each with a full step-by-step solution. As described in the MATH dataset paper, even enormous Transformers have low accuracy, and scaling alone is unlikely to achieve strong mathematical reasoning. The dataset includes an auxiliary pretraining corpus for fundamentals.
Science questions in grade school and graduate school
The AI2 Reasoning Challenge (ARC) contains 7,787 natural grade-school science questions derived from human tests. The ARC paper splits them into a Challenge Set and Easy Set, where the Challenge Set contains only questions that both a retrieval-based and a word co-occurrence algorithm answer incorrectly. Leading neural models from other tasks fail to beat a random baseline on the Challenge Set.
GPQA is a graduate-level question benchmark with 448 multiple-choice questions written by experts in biology, physics, and chemistry. The GPQA paper shows that PhD-level experts reach 65% accuracy, while skilled non-experts with web access only reach 34% after 30 minutes, demonstrating the questions are "Google-proof". A strong GPT-4 based baseline achieves 39% accuracy.
Puzzles and abstraction: the Abstraction and Reasoning Corpus
François Chollet argues that measuring skill at a single task does not capture intelligence because prior knowledge and training data can mask generalization power. He proposes a formal definition of intelligence as skill-acquisition efficiency and introduces the Abstraction and Reasoning Corpus (ARC) as a benchmark built on priors close to innate human priors, as explained in On the Measure of Intelligence.
ARC consists of abstract puzzle-like tasks requiring fluid reasoning rather than memorized knowledge. The benchmark is designed to enable fair comparisons between AI systems and humans on general, human-like intelligence.
Human-centric exam benchmarks: AGIEval
AGIEval is a benchmark built from standardized human exams such as college entrance tests, law school admission tests, math competitions, and lawyer qualification tests. The AGIEval paper evaluates foundation models and finds that GPT-4 surpasses average human performance on SAT, LSAT, and math competitions, with 95% accuracy on the SAT Math test and 92.5% on the English test of the Chinese national college entrance exam.
Despite these strong results, the same paper finds that models are less proficient on tasks requiring complex reasoning or specific domain knowledge. The benchmark analyzes capabilities such as understanding, knowledge, reasoning, and calculation to reveal strengths and limitations.
| Benchmark | Domain | Format | Item count |
|---|---|---|---|
| GSM8K | Grade-school math | Word problems | 8.5K |
| MATH | Competition math | Multi-step problems with solutions | 12,500 |
| AI2 ARC | Grade-school science | Multiple-choice questions | 7,787 |
| GPQA | Graduate-level science | Multiple-choice questions | 448 |
Frequently asked questions
What is GSM8K?
GSM8K is a dataset of 8.5K grade-school math word problems designed to test multi-step reasoning. It was introduced to diagnose why language models fail on simple math tasks.
How hard is the MATH dataset?
MATH contains 12,500 competition mathematics problems with step-by-step solutions. Even large transformer models achieve low accuracy, and scaling alone is not expected to solve it.
What does the ARC benchmark measure?
The AI2 Reasoning Challenge (ARC) consists of 7,787 natural grade-school science questions. The Challenge Set requires reasoning that retrieval and word co-occurrence algorithms cannot handle, and neural baselines perform no better than random.
What is GPQA?
GPQA is a graduate-level multiple-choice benchmark with 448 expert-written questions in biology, physics, and chemistry. It is designed to be "Google-proof" and is difficult for both non-experts and AI systems.
Why are reasoning benchmarks important?
They expose limits in AI models beyond memorization, guiding research toward stronger reasoning and verification. They also enable comparisons with human performance on tasks that require multi-step thinking.