MLPerf and AI hardware benchmarks: how systems are compared fairly

Updated October 21, 2026

MLPerf is a set of benchmarks developed by MLCommons to evaluate AI hardware performance fairly. It includes suites for training, inference (datacenter, edge, mobile), client PCs, and training algorithms. Systems are compared using standardized models, datasets, quality targets, and strict rules for reproducibility.

Overview of MLPerf benchmarks

MLPerf is a suite of benchmarks managed by MLCommons, an open engineering consortium. The MLPerf Training benchmark measures how fast systems can train models to a target quality metric, as described in the MLCommons MLPerf Training benchmark.

The inference benchmarks cover server and edge platforms. The MLPerf Inference: Datacenter suite measures how fast systems process inputs using a trained model, as stated on the MLCommons MLPerf Inference: Datacenter page. The MLPerf Inference: Edge suite similarly applies to edge devices, with details on the MLCommons MLPerf Inference: Edge page.

The MLPerf Inference: Mobile suite targets smartphones and tablets, as explained on the MLCommons MLPerf Inference: Mobile page. For personal computers, the MLCommons MLPerf Client benchmark evaluates large language models, image generation, and agentic AI workloads.

Separately, the MLCommons AlgoPerf training algorithms benchmark measures how much faster training algorithms can reach a target performance by changing the optimizer or hyperparameters, using fixed hardware to isolate algorithmic improvements.

Divisions and categories for fair comparison

To ensure apples-to-apples comparisons, MLPerf defines two divisions. The Closed division requires submitters to use the same model as the reference implementation, so differences in hardware or software frameworks are highlighted. The Open division allows using a different model or retraining, encouraging innovation. Both the training and inference suites follow this structure, as noted on the MLPerf Inference: Datacenter page already cited.

Results are also categorized by availability. Available systems contain only components that are for purchase or rent in the cloud. Preview systems must be submittable as Available in the next round. Research, Development, or Internal (RDI) systems include experimental or internal-use hardware. These categories help buyers understand whether a result reflects a shipping product or a prototype.

For each submission, the submitter must provide details such as the software framework, hardware platform, processor and accelerator counts, and links to code. This transparency allows independent verification and reproducibility.

Benchmark suites and their focus

The MLPerf Training benchmark measures wall-clock time to train models to a target quality. Results are obtained by running the benchmark multiple times, discarding the highest and lowest, and averaging the remainder. The training page lists benchmarks for language, vision, recommendation, and other domains, each with a specific dataset and quality target.

Inference benchmarks use four scenarios: Single stream, Multiple stream, Server, and Offline. Each scenario has a different query pattern and metric. For example, the Offline scenario sends all queries at once and measures throughput, while the Server scenario uses a Poisson distribution of queries and measures the maximum sustainable throughput under a latency constraint. The MLPerf Inference: Edge page explains these scenarios.

The MLPerf Client benchmark focuses on personal computers. It includes LLM tasks with varying prompt and response lengths, image generation using Flux 2 Klein 4B, and agentic AI tasks like software engineering and data analysis. The client page lists minimum system requirements, such as 16 GB of RAM for NPU configurations and specific GPU models.

AlgoPerf uses eight fixed workloads covering tasks like clickthrough rate prediction, image classification, and speech recognition. Submissions are scored based on time-to-result across all workloads, and the benchmark uses standardized hardware (8x V100 GPUs) to ensure fair comparison of training algorithms.

How results are measured and reported

Results are published in interactive tables on the MLCommons website. Each row represents a set of results from one submitter using the same software stack and hardware platform. Columns include submitter, software, system description, benchmark results, processor and accelerator counts, and links to metadata and code. The inference suites also include power measurements: system power for Server and Offline scenarios, and energy per stream for Single and Multiple stream scenarios, as described on the MLPerf Inference: Datacenter page.

Power is measured at the wall as average AC power during the performance measurement. The measured power is only valid for the accompanying benchmark, and MLCommons does not validate other power references like TDP. The inference datacenter page details these power metrics.

Published results may be modified or invalidated, and a change log tracks any updates. Users can filter results by division, category, or benchmark to find relevant comparisons.

AlgoPerf: benchmarking training algorithms

AlgoPerf focuses on the training algorithm itself, not hardware. Submissions must automate workload-specific tuning. Two rulesets exist: external tuning, which simulates parallel hyperparameter search, and self-tuning, which performs all tuning within a single run. Both have separate leaderboards. The AlgoPerf page explains that submissions cannot modify the workloads and must train all eight without additional settings.

The final benchmark score is the integrated performance profile, the area under the curve of a submission's performance relative to the fastest submission for each workload. Scores range from zero to one, with higher being better. All results are computed by the competition organizers on the same hardware to isolate algorithmic improvements.

Frequently asked questions

How does MLPerf ensure fair comparison between different hardware?

MLPerf uses the Closed division, which requires all submitters to use the same model and quality target. This isolates hardware and software differences. Additionally, results are categorized by availability (Available, Preview, RDI) to distinguish shipping products from prototypes.

What is the difference between the Closed and Open divisions?

The Closed division requires using the same model as the reference implementation, enabling apples-to-apples hardware comparison. The Open division allows using a different model or retraining, encouraging algorithmic innovation.

Can results from different MLPerf suites be compared?

Each suite targets a different use case (training, inference datacenter, edge, mobile, client). Results are only comparable within the same suite and division because the models, datasets, and scenarios differ.

How are power measurements taken in MLPerf inference benchmarks?

Power is measured at the wall as average AC power during the performance measurement. For Server and Offline scenarios, system power is reported; for Single and Multiple stream scenarios, energy per stream is reported. The measurement is only valid for the specific benchmark.

What is AlgoPerf and how is it different from other MLPerf benchmarks?

AlgoPerf measures the efficiency of training algorithms, not hardware. It uses fixed hardware (8x V100 GPUs) and eight fixed workloads. Submissions compete under external tuning or self-tuning rulesets, and the score is the area under the performance profile curve.