VRAM for local models: estimating the memory a model needs to run

Updated October 21, 2026

The weight memory rule states that each billion parameters needs roughly 2 GB in half precision and 4 GB in full precision. Quantization can reduce that further, but context length and batch size add extra memory. File formats like GGUF support efficient loading and quantized weights.

The weight memory rule

Large language models typically contain billions of parameters, each stored as a decimal number. The amount of VRAM required to load the model weights into memory can be estimated from the parameter count. According to the Hugging Face's optimization guide, loading a model with X billion parameters takes roughly 4 * X GB in float32 precision, and roughly 2 * X GB in bfloat16 or float16 precision. The same guide lists that Llama-2-70b requires about 140 GB, Falcon-40b about 80 GB, and bigcode/starcoder about 31 GB in bfloat16.

For shorter text inputs, such as under 1024 tokens, the memory for inference is largely dominated by the weights. Many large models need more VRAM than any single GPU provides, so tensor parallelism, which splits the model across multiple GPUs, becomes necessary.

Quantization and lower precision

Quantization reduces memory requirements by storing weights in a lower precision. The Hugging Face's quantization overview explains that weights are typically stored in fp32, but half precision (fp16 or bf16) is becoming common, and some methods go further to integers like int8 or int4.

One specific approach, described in the LLM.int8() paper, cuts inference memory by half while retaining full precision performance. The method uses vector-wise quantization and a mixed-precision decomposition to handle outlier features, allowing models with up to 175 billion parameters to run on a single server with consumer GPUs.

Different quantization methods have trade-offs. Some require calibration for extreme compression, while others work out of the box. The overview page includes a compatibility table listing methods and their supported precision levels.

Beyond weights: context and activations

Weights are not the only memory consumer in GPU memory. The Hugging Face model memory anatomy page describes that training uses multiple tensor types, including model weights, optimizer states, gradients, and activations. For inference, the key additional memory is the key-value cache and activations that grow with the input sequence length.

The same page notes that activations vary with batch size, sequence length, model depth, and hidden size. Long sequences or large batches can exhaust GPU memory even if the model weights themselves fit. In training, mixed precision requires 6 bytes per parameter for two weight copies, but for inference, the main concern is the forward pass.

Therefore, when estimating VRAM for local inference, one should add memory for the context length and batch size, not just the weight size.

File formats and practical tools

The GGUF format is a binary format optimized for quick loading and saving of models, making it efficient for inference. According to the Hugging Face GGUF documentation, GGUF is designed for use with GGML and other executors, and models originally in PyTorch can be converted to GGUF.

GGUF files include both tensors and standardized metadata, and they support many quantization types, such as Q4_K or Q2_K, with different bits per weight. Users can inspect metadata and tensor info using a viewer on the Hub.

Choosing a graphics card with sufficient VRAM for a given model requires comparing the memory needed with the card's capacity. For guidance on selecting a suitable card, readers can refer to a graphics card buying guide.

Frequently asked questions

How much VRAM does a 7B parameter model need?

A 7B model requires roughly 14 GB in bfloat16 or float16, based on the 2 GB per billion parameter rule stated in the Hugging Face guide. The exact amount also depends on context length and batch size.

What is quantization?

Quantization stores model weights in lower precision formats, such as 8-bit or 4-bit integers, to reduce memory usage while aiming to preserve accuracy. The Hugging Face quantization overview explains the trade-offs.

Does the context length affect VRAM?

Yes, the key-value cache and activations grow with the sequence length, so longer contexts require additional memory beyond the model weights.

What is the GGUF format?

GGUF is a binary file format optimized for quick loading and saving of models, commonly used with llama.cpp and other executors. It supports various quantization levels and includes metadata.