FLOPS and how AI compute is measured, from chips to training runs
FLOPS, floating-point operations per second, is the standard unit for AI compute. Peak FLOPS describes a chip's theoretical limit, while achieved FLOPS depends on memory and workload. Training run compute is the total FLOPs needed to train a model, which has grown rapidly and is tracked in public databases.
What FLOPS stands for and how it is counted
FLOPS is an acronym for floating-point operations per second, the standard unit for compute performance. In neural networks, most compute is in general matrix multiplications (GEMMs). A GEMM of matrices with dimensions M, N, and K requires 2 times M times N times K FLOPs because each multiply-add counts as two operations. This definition is given in NVIDIA's matrix multiplication guide.
The guide also introduces arithmetic intensity, the ratio of FLOPs to memory bytes accessed. It uses an example of an NVIDIA V100 with FP16 Tensor Core operations, where the ops per byte ratio is 138.9 when loading from memory. This ratio determines whether a workload is memory limited or math limited.
Tensor Cores accelerate matrix multiplies, but matrix dimensions should be aligned to multiples of 16 bytes for best efficiency. For FP16 data, this means dimensions that are multiples of 8 elements, or 64 on an A100 GPU.
Peak FLOPS versus achieved FLOPS on a GPU
Peak FLOPS is a theoretical figure calculated from the hardware design. NVIDIA's GPU performance background explains that each streaming multiprocessor (SM) has a certain multiply-add throughput per clock. For an A100 with 108 SMs at 1.41 GHz, the peak dense throughput is 156 TFLOPS for TF32 and 312 TFLOPS for FP16.
Achieved FLOPS are usually much lower. The guide notes that operations are executed by thousands of threads, and the number of thread blocks must be several times the number of SMs to minimize the tail effect, where only a few blocks remain at the end. Memory bandwidth and latency also constrain performance.
A practical check is to compare a workload's arithmetic intensity with the GPU's ops per byte ratio. If the intensity is lower, the operation is memory bound.
Benchmarking systems: the LINPACK test
To compare full systems, the TOP500 project uses the LINPACK benchmark, which solves a dense system of linear equations. The LINPACK benchmark measures the best achieved performance, called Rmax, for a given problem size, and also reports the theoretical peak, Rpeak.
The benchmark enforces a specific algorithm, LU factorization with partial pivoting, and a fixed operation count of 2/3 n^3 plus lower-order terms in double precision. This excludes faster matrix multiplication methods or low-precision iterative refinement.
Scaling laws and the growth of training compute
Training compute is the total FLOPs consumed to train a model. The scaling laws paper from OpenAI shows that language model loss follows a power-law with the amount of compute, model size, and dataset size. It also finds that larger models are more sample-efficient, so optimal compute allocation often trains large models on relatively modest data.
Epoch AI's trends page reports that training compute for frontier language models has grown at a rate of 5 times per year, with a doubling time of about 5.2 months. The stock of AI chips is growing at 3.4 times per year, and chip performance per dollar improves by 49% per year.
Tracking training runs in public databases
Researchers and regulators rely on databases that catalog training compute. Epoch AI's model data contains over 3600 models with estimates for training compute, parameter count, and dataset size, gathered from publications and other sources.
The database labels models by significance: notable models include those that advanced benchmarks, have over 1000 citations, or reached a million monthly users. Frontier models are the top 5 by training compute at release, while large-scale models exceed 10^23 FLOP. Confidence levels mark estimates as confident, likely, or speculative.
| Measure | What it indicates | Example from sources |
|---|---|---|
| Peak FLOPS | Theoretical maximum operations per second | A100 FP16: 312 TFLOPS (NVIDIA GPU background) |
| Achieved FLOPS | Actual rate on a workload | Depends on memory and launch configuration |
| LINPACK Rmax | Best rate on a dense linear solve | Reported by TOP500 for each system |
| Training compute | Total FLOP for a training run | 10^23 FLOP for large-scale models (Epoch AI model data) |
Frequently asked questions
What is a FLOPS?
FLOPS stands for floating-point operations per second, a unit that measures the rate of arithmetic operations, especially in scientific and AI computing.
How is training compute measured?
Training compute is the total number of FLOPs used to train a model, often estimated from the hardware, training duration, and model architecture, with confidence levels assigned in public databases like Epoch AI's.
What is the difference between peak and achieved FLOPS?
Peak FLOPS is a theoretical limit based on the hardware design, while achieved FLOPS is the actual rate measured for a specific workload, which is typically lower due to memory bandwidth, latency, and software inefficiencies.
Why do scaling laws matter for AI compute?
Scaling laws describe how model loss decreases with compute, model size, and data, helping researchers allocate resources efficiently. The OpenAI paper shows a power-law relationship that guides training decisions.
How reliable are training compute estimates?
Epoch AI marks records with confidence levels: confident records are accurate within a factor of 3, likely within a factor of 10, and speculative within a factor of 30, reflecting the uncertainty of public information.