Cost and latency as evaluation metrics: tokens, time and cost per task
Evaluating AI systems should include cost and latency alongside accuracy. The HELM framework measures efficiency as a core metric, while MLPerf benchmarks focus on inference speed and throughput. Anthropic's guidance recommends defining measurable success criteria for response time and budget.
Why cost and latency matter
Accuracy alone does not determine whether an AI system is useful in practice. The AI Agents That Matter paper notes that a narrow focus on accuracy leads to agents that are needlessly complex and costly, and it motivates jointly optimizing accuracy and cost.
The Holistic Evaluation of Language Models (HELM) project includes efficiency as one of seven core metrics, alongside accuracy, calibration, robustness, fairness, bias, and toxicity. This multi-metric approach exposes trade-offs between quality and resource use.
Latency and cost are operational metrics that directly affect user experience and deployment budgets. Anthropic's define success criteria and build evaluations guide lists latency and price as common success criteria, with examples such as a 95% response time under 200 milliseconds.
Measuring latency and throughput
Latency is the time a system takes to produce a result for a single request. Throughput is the number of requests processed per unit of time. The MLPerf Inference: Datacenter benchmark measures how fast systems process inputs and produce results using a trained model, with scenarios that apply different request patterns.
MLPerf defines four scenarios: Server, Offline, Single stream, and Multiple stream. Each scenario uses a standard load generator that sends inference requests in a particular pattern and measures a specific metric, such as latency for single-stream or throughput for offline.
The MLCommons benchmark overview explains that each benchmark suite is defined by a working group that sets the AI model, the dataset, and the rules for what changes are allowed. This ensures fair comparison of system performance.
For applications with real-time requirements, latency is critical. Anthropic's guidance suggests defining an acceptable response time based on the application's needs, and using quantitative methods like A/B testing to compare performance against a baseline.
Measuring cost per task
Cost per task can be measured in terms of tokens consumed, API calls, or compute resources. The AI Agents That Matter paper emphasizes that cost should be reported alongside accuracy, and it demonstrates an optimization that reduces cost while maintaining accuracy.
Anthropic's define success criteria guide advises considering the budget for running the model, including the cost per API call, the size of the model, and the frequency of usage. These factors determine the total cost of deploying an AI system.
To measure cost per task, one can track the number of tokens generated per request and multiply by the price per token, or use the total compute time and hardware cost. The HELM framework provides standardized conditions for evaluating models, which helps compare efficiency across different models.
Standardized benchmarks and reproducibility
Standardized benchmarks are essential for comparing cost and latency across systems. The MLCommons benchmark overview lists five high-level goals: fair comparison, accelerating progress, reproducibility, serving commercial and research communities, and keeping benchmarking affordable.
MLPerf Inference: Datacenter has two divisions: Closed and Open. The Closed division requires using the same model as the reference implementation for apples-to-apples comparison, while the Open division allows different models or retraining to encourage innovation.
Results are categorized by availability: Available, Preview, and Research, Development, or Internal (RDI). This helps users understand whether a system can be purchased or rented today.
The HELM project releases all raw model prompts and completions publicly for further analysis, and it provides a modular toolkit. This transparency supports reproducibility and allows others to verify efficiency measurements.
Defining success criteria for cost and latency
When setting success criteria, be specific and measurable. Anthropic's define success criteria guide recommends using quantitative metrics like response time in milliseconds and budget per API call, and aligning them with the application's purpose.
For example, a sentiment analysis model might have a criterion that 95% of responses complete in under 200 milliseconds, and that the cost per 1,000 calls stays within a set budget. These criteria should be achievable based on industry benchmarks and relevant to user needs.
Most use cases need multidimensional evaluation along several success criteria. Anthropic suggests combining task fidelity, consistency, relevance, tone, privacy, context utilization, latency, and price into a single evaluation framework.
By including cost and latency in evaluations, developers can make informed trade-offs and avoid deploying systems that are accurate but too slow or too expensive for real-world use.
| Framework | Metrics | Scenarios | Divisions |
|---|---|---|---|
| HELM | Accuracy, calibration, robustness, fairness, bias, toxicity, efficiency | 16 core scenarios, 26 targeted scenarios | Not specified |
| MLPerf Inference: Datacenter | Latency, throughput, power | Server, Offline, Single stream, Multiple stream | Closed, Open |
| Anthropic evaluation guide | Task fidelity, consistency, relevance, tone, privacy, context utilization, latency, price | Task-specific evals | Not specified |
Frequently asked questions
What is latency in AI evaluation?
Latency is the time a system takes to produce a result for a single request. It is typically measured in milliseconds and is critical for real-time applications.
How is cost per task measured?
Cost per task can be measured by tracking tokens consumed, API calls, or compute time, and multiplying by the applicable price. It should be reported alongside accuracy.
Why are cost and latency important metrics?
They affect user experience and deployment budgets. A system that is accurate but slow or expensive may not be practical for real-world use.
What are the MLPerf inference scenarios?
MLPerf defines four scenarios: Server, Offline, Single stream, and Multiple stream. Each uses a different request pattern and measures a specific metric like latency or throughput.
How can success criteria for cost and latency be defined?
Set specific, measurable targets such as a 95% response time under 200 milliseconds and a budget per API call. Align them with the application's needs and industry benchmarks.