Reproducibility in AI evaluation: pinned versions, seeds and logs
Reproducible AI evaluation requires fixing the exact versions of code and data, controlling randomness with seeds, and recording complete logs. These practices help researchers verify results and compare models reliably. Standards like the NeurIPS checklist formalize these expectations.
The reproducibility problem in AI evaluation
Machine learning research often fails to reproduce published results. The NeurIPS reproducibility program introduced a code submission policy and a checklist to address this issue, as described in the NeurIPS reproducibility report. That program showed that many papers lacked sufficient detail to rerun experiments.
Evaluation results depend on many incidental choices: the order of multiple choice options, the decoding method, the prompt text, and the random seed. The paper When Benchmarks are Targets shows that minor perturbations can shift leaderboard rankings by up to 8 positions. Such sensitivity makes reproducibility essential.
Without reproducible evaluations, it is impossible to tell whether a new result is a genuine improvement or an artifact of implementation details.
Pinning versions of code, data, and dependencies
A reproducible evaluation starts with fixing the exact versions of the evaluation framework, the model code, the dataset, and all libraries. The EleutherAI LM evaluation framework provides unified evaluation code with public prompts and supports multiple model backends. Pinning the framework version and the model checkpoint ensures the same task is evaluated in the same way.
The NeurIPS paper checklist includes questions about code, data, and the computing environment, as shown in the NeurIPS checklist guidelines. Authors are expected to answer yes, no, or n/a for each item and point to the relevant sections.
When sharing evaluations, authors should specify the container or environment used, including operating system and hardware. The EleutherAI evaluation toolkit is one example of a tool that makes version pinning easier by allowing users to install specific optional components for different model backends.
Seeds and randomness control
Randomness affects training, sampling, and scoring. Setting a fixed random seed is a first step, but not sufficient if the evaluation code uses non-deterministic operations. The paper AI Agents That Matter notes a lack of standardization in agent evaluations, and recommends reporting metrics such as cost alongside accuracy.
A fixed seed should be recorded and reported, and multiple seeds should be used to measure variance. The choice of decoding method, such as greedy sampling or temperature sampling, also contributes to randomness.
The answer selection method can also change results. For instance, using the maximum predicted probability instead of an exact string match is a common choice. A reproducible protocol should specify both the seed and the scoring algorithm.
Logging and recording evaluation metadata
Logs capture the exact commands, model outputs, scores, and any errors that occurred. The Inspect framework includes a web-based viewer and logs that can be inspected, as mentioned in the Inspect documentation. This allows reviewers to trace how a score was obtained.
A good log should include the version control commit hash, the random seed, the timestamp, and the hardware details. It should also record any intermediate outputs and the final metrics.
Authors should provide logs as supplementary material to support their answers, along with multiple runs to show variance.
| Practice | What to record | Why it matters |
|---|---|---|
| Pin framework version | Commit hash or release tag | Ensures the same evaluation logic |
| Pin model checkpoint | Model ID, version | Model weights affect output |
| Pin dataset version | Dataset name, split, file hash | Data changes can shift scores |
| Set fixed random seed | Seed value for training and sampling | Controls randomness in generation |
| Specify decoding parameters | Temperature, top-p, max tokens | Affects sampling |
| Record environment details | OS, Python version, GPU type | Hardware can affect numerical results |
| Log all outputs | Command, metrics, logs | Allows verification |
Frequently asked questions
What does pinning a version mean?
Pinning a version means recording the exact commit hash or release tag of the evaluation framework, model, and dataset, so that they can be retrieved later.
Why is a random seed not enough?
A seed controls randomness, but non-deterministic operations in hardware or libraries can still produce different results. Recording the seed is necessary but not sufficient; also log the environment and decoding settings.
What should a log include?
A log should include the command line, version control hashes, seed values, hardware details, model outputs, and computed metrics.
How does the NeurIPS checklist help?
The NeurIPS checklist asks authors to answer questions about code, data, and computing environment, which encourages them to document reproducibility details.