Repository-level coding evaluation: SWE-bench style tasks
SWE-bench is an evaluation framework that tests language models on real GitHub issues and pull requests from popular Python repositories. Models receive a codebase and an issue description, then must edit the code to resolve it. Early results showed low success rates, and related work explores cost, reproducibility, and time horizons for software tasks.
What SWE-bench measures
The SWE-bench paper introduces an evaluation framework with 2,294 software engineering problems derived from actual GitHub issues and their associated pull requests across 12 popular Python repositories. Each task gives a language model a codebase and a description of an issue, and the model must edit the codebase to address the issue.
Resolving these issues often requires understanding and coordinating changes across multiple functions, classes, and even files at the same time. The tasks demand that models interact with execution environments, process very long contexts, and perform reasoning that goes beyond traditional code generation.
The original evaluation found that both proprietary models and a fine-tuned model called SWE-Llama could resolve only the simplest issues. The best model at the time, Claude 2, solved 1.96% of the issues, showing the difficulty of repository-level coding.
The SWE-bench leaderboard and variants
The SWE-bench project site hosts leaderboards for several variants, including Verified, Multimodal, Multilingual, Lite, and Full. These leaderboards let researchers compare results from different agents and models, with details such as per-instance resolution status and cost.
The site also provides logs and submission data for each agent run. For example, the Multilingual leaderboard lists entries with model names, organizations, and per-instance details that show which issues were resolved and how many API calls were used.
Researchers can use these leaderboards to track progress over time and to understand how different approaches perform on repository-level tasks. The site is a central resource for the SWE-bench community.
Related benchmarks and metrics
Before SWE-bench, the OpenAI Codex paper introduced HumanEval, a set of programming problems that measure functional correctness from docstrings. Codex solved 28.8% of the problems, while GPT-3 solved 0% and GPT-J solved 11.4%. Repeated sampling improved results to 70.2% with 100 samples per problem.
HumanEval focuses on single-function synthesis, whereas SWE-bench requires changes across an entire repository. This shift reflects a move toward more realistic software engineering tasks.
Another related effort is the METR paper on long software tasks, which proposes a metric called the 50%-task-completion time horizon. This metric represents the typical human time needed for tasks that AI models can complete half the time. On a combination of RE-Bench, HCAST, and 66 novel shorter tasks, current frontier models had a 50% time horizon of around 50 minutes.
Evaluation practices and pitfalls
The AI Agents That Matter paper analyzes current agent benchmarks and identifies several shortcomings. It notes a narrow focus on accuracy without attention to other metrics, leading to agents that are needlessly complex and costly.
The paper also points out that the benchmarking needs of model developers and downstream developers are conflated, making it hard to choose an agent for a specific application. Many benchmarks have inadequate holdout sets, which leads to agents that overfit and take shortcuts.
Finally, the paper calls for standardization in evaluation practices to improve reproducibility. These issues are relevant to SWE-bench style tasks, where cost and robustness matter as much as raw accuracy.
| Benchmark | Task type | Repository-level | Example metric |
|---|---|---|---|
| HumanEval | Function synthesis from docstring | No | Pass@k |
| SWE-bench | Resolve GitHub issue in codebase | Yes | Resolved rate |
| RE-Bench | Long software tasks with human timing | Yes | 50% time horizon |
| HCAST | Long software tasks | Yes | 50% time horizon |
Frequently asked questions
What is SWE-bench?
SWE-bench is an evaluation framework with 2,294 software engineering problems from real GitHub issues and pull requests across 12 Python repositories, as described in the arXiv paper.
How does SWE-bench differ from HumanEval?
HumanEval tests single-function synthesis from docstrings, while SWE-bench requires editing an entire codebase to resolve a GitHub issue, which involves multiple files and complex reasoning.
What is the 50%-task-completion time horizon?
It is the typical duration a human expert needs to finish tasks that an AI model can complete with a 50% success rate, as proposed in the METR paper.
Why is cost important in agent evaluation?
The AI Agents That Matter paper argues that focusing only on accuracy leads to costly agents, and suggests jointly optimizing cost and accuracy for real-world usefulness.
Where can SWE-bench results be viewed?
The SWE-bench project site hosts leaderboards for variants like Verified, Multimodal, Multilingual, Lite, and Full, with per-instance details and logs.