Evaluating AI agents on real tasks: environments and success checks
Evaluating AI agents requires realistic environments that mimic real-world tasks. Benchmarks such as WebArena, GAIA, and SWE-bench provide such environments and define success checks like functional correctness or issue resolution. Current AI agents still perform far below human levels on these tasks.
Realistic environments for agent evaluation
AI agents are increasingly evaluated in environments that simulate real-world tasks rather than simplified synthetic settings. The WebArena paper provides fully functional websites in four domains: e-commerce, social forum discussions, collaborative software development, and content management. It also includes tools like a map and external knowledge bases to encourage human-like task solving.
The AgentBench paper offers a multi-dimensional benchmark with eight distinct environments to assess reasoning and decision-making abilities. It tests both API-based and open-source language models as agents.
The GAIA paper focuses on general AI assistants by proposing real-world questions that require reasoning, multi-modality handling, web browsing, and tool use. The questions are conceptually simple for humans but challenging for advanced AI systems.
The SWE-bench paper targets software engineering by providing 2,294 real GitHub issues and corresponding pull requests across 12 popular Python repositories. Language models must edit the codebase to resolve the issue, which often requires understanding and coordinating changes across multiple files.
Measuring success: from functional correctness to time horizons
Success checks vary across benchmarks. WebArena evaluates the functional correctness of task completions, meaning the agent must achieve the correct end state of a web task. In their experiments, the best GPT-4-based agent achieved a success rate of only 14.41%, while humans reached 78.24%.
GAIA measures success by whether the agent provides the correct answer to a question. Human respondents achieved 92% accuracy, whereas GPT-4 equipped with plugins managed only 15%.
SWE-bench checks if the model's code changes correctly resolve the GitHub issue. The best-performing model, Claude 2, solved only 1.96% of the issues.
The METR study introduces a different metric: the 50%-task-completion time horizon, which indicates how long a human expert typically needs for a task that an AI can finish half the time. For Claude 3.7 Sonnet, this time horizon is around 50 minutes.
Challenges in agent evaluation
Current evaluation practices have several shortcomings. The paper AI Agents That Matter points out that benchmarks often focus narrowly on accuracy without considering cost, leading to needlessly complex and expensive agents. The authors advocate for jointly optimizing accuracy and cost.
Another issue is the conflation of benchmarking needs for model developers and downstream application developers, making it hard to choose the best agent for a specific use case. Many benchmarks also lack adequate holdout sets, causing agents to overfit and take shortcuts.
Lack of standardization in evaluation practices leads to poor reproducibility. The paper prescribes a principled framework to avoid overfitting and improve the real-world usefulness of agent evaluations.
| Environment | Domain | Task Type | Success Check |
|---|---|---|---|
| WebArena | Web | Perform web tasks (e.g., shopping, forum) | Functional correctness of task completion |
| AgentBench | Multi-domain | Reasoning and decision-making in 8 environments | Task success rate |
| GAIA | General | Answer real-world questions | Correct answer |
| SWE-bench | Software engineering | Resolve GitHub issues | Issue resolution (patch matches) |
| METR | Software | Complete long software tasks | 50% task-completion time horizon |
Frequently asked questions
What makes an environment realistic for AI agent evaluation?
Realistic environments simulate real-world tasks with fully functional tools and interfaces, such as websites or codebases, rather than simplified synthetic settings. Examples include WebArena's web environments and SWE-bench's GitHub issues.
How is success measured in agent benchmarks?
Success checks vary by benchmark. Common measures include functional correctness of task completion (WebArena), correct answer to a question (GAIA), resolution of a GitHub issue (SWE-bench), or the time horizon at which an agent achieves 50% success (METR).
Why do AI agents perform poorly on real tasks compared to humans?
Current AI agents struggle with long-term reasoning, decision-making, instruction following, and adapting to mistakes. For example, GPT-4 achieved only 14.41% success on WebArena tasks versus 78.24% for humans, and Claude 2 solved only 1.96% of SWE-bench issues.
What is the 50% time horizon metric?
The 50%-task-completion time horizon, introduced by METR, is the amount of time humans typically need to complete tasks that an AI model can complete with a 50% success rate. For Claude 3.7 Sonnet, this is about 50 minutes.
How can agent evaluation practices be improved?
Improvements include considering both accuracy and cost, using adequate holdout sets to prevent overfitting, standardizing evaluation practices for reproducibility, and separating the needs of model developers from downstream application developers.