Precision formats explained: FP32, FP16, BF16, FP8 and INT8 in AI
Precision formats trade range and accuracy for memory and speed. FP16 and BF16 enable mixed precision training, while FP8 and integer quantization further reduce compute cost. This article explains the technical details and how to apply these formats.
The role of precision in AI
AI models process tensors of numbers. The choice of numerical format affects memory footprint, bandwidth, and arithmetic speed. The IEEE 754 standard specifies interchange and arithmetic formats for binary and decimal floating-point arithmetic in computer systems (IEEE 754 standard).
Recent accelerator hardware supports reduced precision to improve throughput. Lower-precision formats require less memory and bandwidth, and half-precision arithmetic can run many times faster on GPUs with Tensor Cores, as described in NVIDIA's documentation on mixed precision training (NVIDIA's mixed precision guide). This motivates the use of FP16, BF16, FP8, and integer formats.
FP16 and mixed precision training
FP16 uses 16 bits with 1 sign bit, 5 exponent bits, and 10 mantissa bits. Its dynamic range is limited, so training with FP16 alone can lose small gradient values. The technique of mixed precision keeps a single-precision copy of weights and scales the loss to preserve gradients, as described in a foundational paper on mixed precision training (Mixed Precision Training paper).
The paper recommends maintaining a single-precision copy of the weights that accumulates gradients after each optimizer step, and scaling the loss to avoid underflow. This approach reduces memory consumption by nearly 2x and works for large models with more than 100 million parameters.
BF16 and FP8 formats
BF16 is a 16-bit format that keeps the same exponent range as FP32, making conversion straightforward. A study from Intel and Facebook shows that BF16 training achieves the same results as FP32 without hyperparameter changes (BFLOAT16 study on arXiv). This avoids the loss scaling needed with FP16.
FP8 is an 8-bit floating-point format proposed for further acceleration. The paper defines two encodings: E4M3 with 4 exponent bits and 3 mantissa bits, and E5M2 with 5 exponent bits and 2 mantissa bits. Experiments show that FP8 can match 16-bit training quality across CNNs, RNNs, and Transformers, including language models with up to 175 billion parameters (FP8 formats paper on arXiv).
INT8 and integer-only inference
Integer-only quantization enables inference on hardware without floating-point units. A quantized model uses integer arithmetic, which is more efficient on mobile CPUs. The paper by Jacob et al. proposes a quantization scheme that allows inference with integer-only operations, improving the accuracy-latency tradeoff (Quantization paper on arXiv).
The approach trains a model with quantization-aware techniques so that post-quantization accuracy is preserved. It does not rely on a fixed 8-bit integer format; rather, it describes a general method where weights and activations are represented as integers and all computations use integer operations.
| Format | Bit width | Key property |
|---|---|---|
| FP32 | 32 bits | Common single precision baseline |
| FP16 | 16 bits | 5 exponent bits, 10 mantissa bits; requires loss scaling for training |
| BF16 | Half precision | Same range as FP32; simple conversion |
| FP8 | 8 bits | Two encodings: E4M3 and E5M2 |
| INT8 | Not specified | Integer-only arithmetic for efficient inference |
Frequently asked questions
Why does FP16 training require loss scaling?
FP16 has a limited exponent range, so small gradient values can underflow to zero. Loss scaling multiplies the loss before backpropagation to bring gradients into a representable range.
What is the main advantage of BF16 over FP16?
BF16 keeps the same numeric range as FP32, so no hyperparameter tuning is needed for convergence, and conversion between BF16 and FP32 is simple.
How do FP8 and INT8 differ?
FP8 is a floating-point format with 4 or 5 exponent bits and 3 or 2 mantissa bits, while INT8 uses integer arithmetic only, typically for inference.
Can integer-only quantization be used for training?
The described quantization scheme is designed for inference; it aims to improve latency on integer-only hardware while preserving accuracy.