Evaluation harnesses and frameworks: how eval suites are built and run
Evaluation frameworks are software that run language model benchmarks. They combine a dataset, a solver that produces outputs, and a scorer that grades them. Many examples exist, such as EleutherAI's framework, Inspect, OpenAI Evals, and Hugging Face's Evaluate, each with different design choices.
Purpose of evaluation frameworks
Evaluation suites provide a standardized way to test language models across many benchmarks. The EleutherAI evaluation framework supports over 60 academic benchmarks with hundreds of subtasks and variants. It supports multiple model backends, including HuggingFace transformers, vLLM, and API models.
The UK AI Security Institute's Inspect framework is designed for frontier AI evaluations and can measure coding, agentic tasks, reasoning, knowledge, behavior, and multimodal understanding. It provides over 200 pre-built evaluations and supports agent-based evaluation with tools and sandboxing.
OpenAI's Evals framework offers an existing registry of evals to test different dimensions of models and the ability to write custom evals. The repository states that evals can be built from private data to represent workflow patterns without exposing that data publicly.
Components of an evaluation task
Most frameworks follow a similar architecture: a dataset supplies input and target samples, a solver generates model answers, and a scorer grades the outputs. Inspect defines these three elements explicitly. The dataset can be loaded from Hugging Face, CSV, or JSON; the solver can be a simple generation call or a full agent; the scorer can use text comparisons or model-based grading.
The EleutherAI evaluation suite supports YAML configuration files for tasks, allowing users to define custom prompts, few-shot settings, and post-processing steps. It also supports output extraction and multiple generations per document, as noted in its documentation.
OpenAI Evals allows users to create evals without writing code by following an existing template; they provide JSON data and specify parameters in YAML, as explained in the Evals repository.
Creating evaluations
Traditional evaluation creation relied on crowdwork or existing data sources. A research paper by Perez et al. introduced using language models to generate their own evaluation sets. The model-written evaluations produced 154 datasets, some of which revealed inverse scaling behaviors such as sycophancy.
Hugging Face's Evaluate library offers a single line of code to access dozens of evaluation methods across domains. It also hosts community leaderboards that rank models on tasks like embedding quality and agentic performance.
OpenAI's Evals also encourages contributions via pull requests, though it currently does not accept custom code evals. It does accept model-graded evals with custom YAML files, as stated in the repository.
Running and reproducibility
Running an evaluation requires access to a model and appropriate API keys. Inspect supports over 20 model providers and local inference with HuggingFace, vLLM, or SGLang, and uses the `inspect eval` command. OpenAI's API documentation includes a guide for working with evals, covering similar setup steps.
The EleutherAI evaluation suite can be installed via pip, but model backends like transformers or vLLM need to be installed separately. This modular design allows users to choose their inference engine.
Reproducibility is a key concern. The EleutherAI evaluation suite uses publicly available prompts to ensure comparability between papers. OpenAI's Evals registry uses Git-LFS for data storage, and users can fetch specific eval data.
Choosing a framework
The right evaluation framework depends on the use case. EleutherAI's evaluation suite powers the Open LLM Leaderboard and is used in many research papers. Inspect offers advanced agent support and sandboxing for untrusted code. OpenAI's Evals is integrated with the OpenAI ecosystem.
Hugging Face's Evaluate library is lightweight and focuses on metrics, and it connects to the Hub for collaborative evaluation. It can be used with any model or dataset.
| Framework | Organization | Key features | Typical use |
|---|---|---|---|
| EleutherAI LM Evaluation Suite | EleutherAI | 60+ benchmarks, YAML config, multiple backends | Academic research, leaderboards |
| Inspect | UK AI Security Institute | 200+ evals, agent support, sandboxing | Frontier AI, agent tasks |
| OpenAI Evals | OpenAI | Registry, custom evals, YAML templates | OpenAI models, private evals |
| Hugging Face Evaluate | Hugging Face | Single-line metrics, Hub integration | Model evaluation, leaderboards |
Frequently asked questions
What is an evaluation framework?
An evaluation framework is a software tool that runs model benchmarks by managing datasets, solvers, and scorers in a reproducible way.
What are the common parts of an evaluation task?
A typical evaluation task has a dataset with input-target pairs, a solver that produces model answers, and a scorer that grades those answers against the targets.
Can evaluations be written without programming?
Yes, some frameworks like OpenAI Evals and EleutherAI's suite allow users to define evaluations using YAML configuration files and JSON data without writing code.
How do these frameworks ensure reproducibility?
They use public prompts, configurable few-shot settings, versioned data registries, and modular model backends, allowing results to be compared across studies.
How are model-written evaluations generated?
Models are instructed to write questions, sometimes with multiple stages of generation and filtering, as demonstrated in the Anthropic research paper on model-written evaluations.