Quantization: lower-precision weights for faster, smaller models

Updated October 21, 2026

Quantization converts model weights from high-precision formats like FP32 to lower-precision integers such as INT8 or INT4. This reduces memory requirements and accelerates inference, often with minimal accuracy loss. Techniques like GPTQ, LLM.int8(), and QLoRA enable efficient deployment on consumer hardware.

What is quantization?

Quantization is the process of reducing the precision of a digital signal, commonly applied to large language models (LLMs) to convert weights and activation values from 32-bit floating point (FP32) or 16-bit floating point (FP16) to lower-precision formats like 8-bit integer (INT8) or 4-bit integer (INT4). According to IBM's explanation of quantization, this compression reduces computational demands and increases inference speed, though it introduces quantization error that can slightly affect accuracy.

The Hugging Face quantization overview notes that weights are typically stored in full-precision FP32, but half-precision (FP16 or bf16) and integer representations are increasingly used due to the large size of modern models. Quantization lowers memory requirements while trying to preserve as much accuracy as possible.

How quantization works

Two basic algorithms are absolute max quantization and affine quantization. Absolute max divides each value by the tensor's absolute maximum and scales to the target range, while affine uses a scaling factor and zero-point to map floating-point values to integers. IBM's guide provides examples of both methods.

Advanced post-training methods achieve higher compression with minimal accuracy loss. The GPTQ paper introduces a one-shot weight quantization method based on approximate second-order information that can quantize a 175 billion parameter model to 3 or 4 bits per weight in about four GPU hours, with negligible degradation relative to the uncompressed baseline.

The LLM.int8() paper describes a procedure for 8-bit matrix multiplication in transformers that cuts memory needed for inference by half while retaining full precision performance. It uses vector-wise quantization and a mixed-precision decomposition to handle outlier features.

For finetuning, the QLoRA paper introduces 4-bit NormalFloat (NF4), double quantization, and paged optimizers. These innovations allow finetuning a 65 billion parameter model using only a single 48 GB GPU while achieving the same task performance as full 16-bit finetuning.

Benefits and trade-offs

Quantization reduces memory footprint, allowing larger models to run on consumer hardware. For example, LLM.int8() halves memory usage for inference, as stated in the LLM.int8() paper. Lower precision also speeds up matrix multiplication because fewer bits need processing per operation.

Inference speed gains are significant. The GPTQ paper reports end-to-end speedups over FP16 of around 3.25x on high-end GPUs (NVIDIA A100) and 4.5x on more cost-effective ones (NVIDIA A6000). These improvements make real-time applications feasible on mobile CPUs and consumer GPUs.

The main trade-off is quantization error, which can degrade model accuracy. However, methods like GPTQ and LLM.int8() demonstrate that accuracy can be maintained within a negligible margin. IBM notes that 8-bit quantization is generally the goal, but 4-bit and lower have been successfully achieved.

Quantization in practice

Many frameworks support quantization. PyTorch provides quantization APIs, though development has moved to the torchao library as noted in the PyTorch quantization documentation. The llama.cpp repository implements integer quantization from 1.5 to 8 bits for CPU and GPU inference, supporting a wide range of hardware including Apple Silicon and NVIDIA GPUs.

Hugging Face Transformers integrates multiple quantization methods, allowing users to select based on model and hardware. The Hugging Face overview provides a compatibility table and a quantization picker to help choose the right method.

For those selecting hardware, a graphics card guide can help choose a GPU that supports quantized models efficiently, as consumer GPUs often benefit from reduced memory requirements.

Frequently asked questions

Does quantization always reduce accuracy?

Not always. Methods like GPTQ and LLM.int8() are designed to preserve accuracy within a negligible margin, though some quantization error is inherent.

Can any model be quantized?

Most transformer-based models can be quantized, but the degree of accuracy loss depends on the model size, quantization method, and bitwidth.

What is the difference between INT8 and FP16?

INT8 uses 8 bits per weight, while FP16 uses 16 bits. INT8 reduces memory and speeds up computation but may be less precise than FP16.

How can a user choose a quantization method?

Consider the hardware (CPU, GPU), model size, and accuracy requirements. Tools like the Hugging Face quantization picker can help match methods to the user's setup.

Is quantization only for inference?

No. QLoRA demonstrates that quantization can also be used during finetuning, reducing memory requirements while preserving task performance.