Data contamination: when benchmark questions leak into training data
Data contamination occurs when benchmark evaluation questions are unintentionally included in a model's training data, leading to inflated performance scores that do not reflect real capabilities. Simple string-matching filters fail against paraphrased questions, so researchers are developing stronger detection methods such as exchangeability tests and continuously updated benchmarks. Understanding contamination is critical for interpreting AI evaluation results.
What is data contamination?
Data contamination in AI evaluation happens when benchmark test questions appear in the data used to train a model. The worst form occurs when the test split of a benchmark is directly included in training, causing an overestimation of the model's true performance and leading to incorrect scientific conclusions, as argued in the position paper on NLP evaluation.
Even large web corpora can contain evaluation examples unintentionally. The documentation of the C4 corpus found that its text includes examples from other benchmark NLP datasets, as reported in the C4 corpus documentation study. This shows that contamination can arise simply from scraping the internet.
Why simple filters fail
String matching, such as n-gram overlap, is a common decontamination method, but paraphrased or translated versions of benchmark questions can easily bypass it. A study demonstrated that a 13B model can overfit a test benchmark when such variations are present, achieving scores comparable to GPT-4, according to the rephrased samples study.
The same study found that 8-18% of the HumanEval benchmark overlaps in pretraining sets like RedPajama and StarCoder, even after standard filtering. This indicates that simple methods miss many instances, so the authors propose an LLM-based decontamination approach and urge the community to adopt stronger decontamination measures.
Proven detection without pretraining data
Because proprietary models hide their pretraining data, contamination is hard to prove. An exchangeability-based test provides provable guarantees without needing access to training data or model weights, as described in the exchangeability contamination test paper. The test relies on the principle that if no contamination exists, all orderings of a benchmark are equally likely, while a contaminated model shows a higher likelihood for canonical orderings.
The test is sensitive enough to detect contamination in models as small as 1.4 billion parameters and test sets of only 1000 examples. Auditing five publicly accessible models, the authors found little evidence for pervasive contamination.
Building contamination-free benchmarks
One solution is to continuously collect problems from dynamic sources. LiveCodeBench gathers coding problems from contests on LeetCode, AtCoder, and CodeForces, and currently hosts four hundred high-quality coding problems, as presented in the LiveCodeBench paper. It also evaluates broader capabilities such as self-repair, code execution, and test output prediction.
The popularity of benchmarks like MMLU increases contamination risk. Anthropic notes that because MMLU is widely used, models are more likely to encounter its questions during training, which is comparable to students seeing the questions before the test, according to Anthropic's evaluation challenges article.
Best practices for researchers
The position paper on NLP evaluation calls for measuring contamination per benchmark and flagging papers with compromised conclusions. It defines different levels of contamination and suggests developing automatic and semi-automatic measures to detect when benchmark data was exposed to a model.
The C4 corpus documentation recommends careful documentation of web-scale datasets to understand their content, including potential contamination, and to assess the impact of filtering. Anthropic also highlights that formatting changes to multiple-choice options can shift accuracy by about 5%, and that developers implement benchmarks inconsistently, making score comparisons across labs difficult.
| Method | Strengths | Weaknesses |
|---|---|---|
| String matching | Fast and easy to apply | Misses paraphrased or translated questions |
| LLM-based decontamination | Catches variations that string matching misses | Requires careful implementation and may have false positives |
| Exchangeability test | Provides provable guarantees without data access | Applies only to exchangeable benchmarks |
| Live benchmark | Continuously adds problems to prevent future contamination | Limited to specific domains such as coding contests |
Frequently asked questions
What is data contamination?
Data contamination occurs when benchmark test questions appear in a model's training data, inflating its evaluation scores and masking true capabilities.
How do researchers detect contamination?
Researchers use string matching, LLM-based decontamination, and exchangeability tests that compare likelihoods of canonical versus shuffled orderings.
Why don't n-gram filters catch all contamination?
Paraphrasing and translation can bypass exact n-gram matches, so simple filters miss many variations of test questions.
What is LiveCodeBench?
LiveCodeBench is a benchmark that continuously collects coding problems from contests to avoid contamination and assess a range of code-related abilities.
Why is MMLU more prone to contamination?
MMLU is widely used, so its questions are more likely to appear in training data scraped from the internet, increasing the chance of contamination.