How coding benchmarks work: unit tests, pass@k and HumanEval tasks

Updated October 21, 2026

Coding benchmarks measure how well AI models generate code by checking if outputs pass unit tests. The pass@k metric estimates the probability that at least one of k sampled solutions is correct. HumanEval and MBPP are common datasets, while newer benchmarks like LiveCodeBench aim to reduce contamination.

The role of unit tests in coding benchmarks

Coding benchmarks assess a model's ability to write programs that meet a specification. Instead of comparing text, they run the generated code against hidden unit tests. A solution is considered correct only if it passes all tests, which checks functional behavior rather than style. This approach is used in the HumanEval dataset, where each problem includes a docstring and a set of tests, as described in the OpenAI Codex paper.

Unit tests provide an objective signal, but they also have limits. A small set of tests may not cover all edge cases, so a program that passes could still be wrong for inputs not tested. Benchmarks like MBPP, introduced in the Google Research program synthesis paper, include multiple tests per problem to reduce this risk. The design of test cases is therefore a key part of benchmark quality.

The pass@k metric

A single generated program may fail even if the model knows the solution, because sampling is stochastic. The pass@k metric addresses this by generating k candidate solutions and checking if at least one passes the tests. The OpenAI Codex paper reports that with 100 samples per problem, the model solved 70.2% of HumanEval problems, compared to 28.8% with a single sample. This shows how repeated sampling improves measured performance.

Pass@k is an estimate, not a direct measurement. The paper explains that the unbiased estimator for pass@k is calculated from the number of unique correct solutions among the k samples. This avoids overestimating when the model produces many identical correct answers. Researchers report pass@k for various k values, such as 1, 10, and 100, to show how performance scales with sampling effort.

HumanEval and MBPP datasets

HumanEval consists of hand-written Python problems that test docstring-to-code synthesis. The OpenAI paper introduced this dataset to measure functional correctness, and it has become a standard benchmark. Each problem includes a function signature, a docstring, and several unit tests. The dataset is designed to avoid overlap with training data, though contamination remains a concern for later models.

MBPP, or Mostly Basic Programming Problems, contains 974 tasks intended for entry-level programmers. The Google Research paper describes how these tasks are solvable by novices and are used to evaluate few-shot and fine-tuned models. The paper reports that the largest models, without fine-tuning, solved 59.6% of MBPP problems using few-shot learning. Both datasets focus on short Python programs, but they differ in difficulty and style.

Contamination and newer benchmarks

A major issue with static benchmarks is contamination: if a model's training data includes the benchmark problems, the scores are inflated. The LiveCodeBench paper addresses this by continuously collecting new problems from programming contests on LeetCode, AtCoder, and CodeForces. The benchmark includes problems published over a one-year period, and it evaluates not only code generation but also self-repair and test output prediction.

LiveCodeBench also examines overfitting to existing benchmarks. The authors argue that as models improve, static benchmarks like HumanEval become saturated and less useful for distinguishing capabilities. By using fresh problems, LiveCodeBench aims to provide a more reliable measure of a model's coding ability. This highlights the need for benchmarks that evolve over time.

Frameworks for running benchmarks

Several open-source frameworks help researchers run coding benchmarks. The OpenAI Evals repository provides a framework for evaluating LLMs and systems built with them. It includes a registry of existing evals and allows users to write custom evals for their own use cases. The repository supports both basic and model-graded evals, and it can log results to a database.

The EleutherAI language model evaluation tool is another widely used option. It offers over 60 standard benchmarks, including coding tasks, and supports many model backends such as HuggingFace transformers and vLLM. The tool is designed for reproducible few-shot evaluation and is the backend for the Hugging Face Open LLM Leaderboard. These frameworks standardize how benchmarks are run, making results comparable across studies.

Frequently asked questions

What is pass@k in coding benchmarks?

Pass@k measures the probability that at least one of k generated code samples passes all unit tests. It is estimated from the number of unique correct solutions among the samples, as described in the OpenAI Codex paper.

How does HumanEval test code generation?

HumanEval provides a set of Python problems, each with a docstring and unit tests. A model's solution is considered correct if it passes all the tests for that problem.

Why are unit tests used instead of text comparison?

Unit tests check functional correctness, meaning the code actually does what it should. Text comparison would only check if the output looks similar, which is less reliable for code.

What is benchmark contamination?

Contamination happens when a model's training data includes the benchmark problems, inflating its scores. LiveCodeBench addresses this by using new problems from recent contests.

Which frameworks can run coding benchmarks?

OpenAI Evals and the EleutherAI language model evaluation tool are two open-source frameworks. They provide standard benchmarks and support for various models.