Batching for inference: static, dynamic and continuous batching
Batching combines multiple inference requests to improve GPU utilization and throughput. Static batching groups fixed-size batches, dynamic batching forms batches on the fly, and continuous batching (also called in-flight batching) schedules requests at the token level. Each approach trades latency for throughput, with continuous batching offering the most flexibility.
Why batching matters for LLM inference
Large language models generate text in two phases: prefill and decode. During prefill, the model processes input tokens in parallel, which saturates GPU compute. During decode, it generates output tokens one at a time, and the speed is limited by memory transfer rather than computation, as described in the NVIDIA inference optimization blog.
Batching multiple requests together improves GPU utilization because the model weights are loaded once and shared across requests. The memory cost of the weights is amortized, and the compute units are kept busy. Without batching, each request would underutilize the GPU, especially during the memory-bound decode phase.
The key-value cache (KV cache) stores intermediate attention states for each request. Its memory footprint grows with batch size and sequence length, which limits how many requests can be batched. Efficient memory management is essential for high throughput, as noted in the PagedAttention paper.
Static batching
Static batching groups a fixed number of requests into a batch before inference begins. The batch size is predetermined, and all requests in the batch are processed together from start to finish. This approach is simple and predictable, but it can waste resources if requests arrive at different times or have different lengths.
Static batching is often used in offline processing or when request arrival rates are steady. It does not adapt to varying load, so the server may wait to fill a batch, increasing latency, or process a partially filled batch, reducing throughput.
The NVIDIA Triton batcher documentation explains that dynamic batching is preferred for stateless models because it creates batches dynamically, unlike static batching which uses fixed sizes.
Dynamic batching
Dynamic batching combines requests that arrive at the server into a batch on the fly. The server waits a short time to collect more requests, then sends the batch for inference. This increases throughput compared to processing each request individually, as stated in the NVIDIA Triton batcher guide.
The dynamic batcher can be configured with a maximum batch size and a maximum delay. If a preferred batch size is set, it tries to form batches of that size. The delay option allows requests to wait for others to join, trading increased latency for higher throughput.
Dynamic batching works well for stateless models where requests are independent. However, it still processes each batch to completion before starting the next, so a long-running request in a batch can delay others.
Continuous batching and in-flight batching
Continuous batching, also called in-flight batching, schedules requests at the token level rather than the batch level. When one request finishes generating, a new request can join the batch immediately, without waiting for the entire batch to complete. This approach improves GPU utilization and throughput, as explained in the Red Hat vLLM overview.
vLLM uses PagedAttention to manage the KV cache in fixed-size blocks, which reduces memory fragmentation and allows more requests to be batched. The PagedAttention paper reports that vLLM improves throughput by 2 to 4 times compared to earlier systems like FasterTransformer and Orca.
Continuous batching is particularly beneficial for variable-length requests and long contexts. It requires careful memory management because the KV cache grows and shrinks dynamically as requests progress.
Choosing a batching strategy
The choice of batching strategy depends on the workload and latency requirements. Static batching is simplest but least efficient for variable traffic. Dynamic batching is a good middle ground for many online services. Continuous batching offers the highest throughput but requires more sophisticated scheduling and memory management.
The Google scaling transformer inference paper discusses partitioning and low-level optimizations to improve inference efficiency, including multi-query attention to reduce KV cache memory. Such techniques complement batching by enabling larger batch sizes.
When evaluating a serving system, check how it handles batching. Look for documentation on batch size limits, delay settings, and whether it supports continuous or in-flight batching. The NVIDIA Triton batcher guide provides configuration details for dynamic batching, including preferred batch sizes and queue policies.
| Strategy | Batch formation | Latency impact | Throughput impact | Memory management |
|---|---|---|---|---|
| Static | Fixed size, predetermined | May wait to fill batch | Moderate | Simple, but may waste memory |
| Dynamic | On the fly, with delay | Adds delay up to max | Higher than static | Moderate, grows with batch |
| Continuous | Token-level scheduling | Low, no batch wait | Highest | Requires paging or similar |
Frequently asked questions
What is the difference between dynamic and continuous batching?
Dynamic batching forms a batch of requests and processes it to completion before starting the next batch. Continuous batching schedules at the token level, allowing new requests to join as soon as others finish, which improves GPU utilization.
Why is the KV cache important for batching?
The KV cache stores intermediate attention states for each request. Its memory grows with batch size and sequence length, so efficient management is needed to fit more requests in memory and increase throughput.
Does batching increase latency?
Batching can increase latency because requests may wait for others to join a batch. Dynamic batching has a configurable delay, and continuous batching minimizes wait time by scheduling at the token level.
What is PagedAttention?
PagedAttention is an attention algorithm that manages the KV cache in fixed-size blocks, similar to virtual memory paging. It reduces memory fragmentation and enables flexible sharing, allowing larger batch sizes.
How does one choose a batching strategy?
Consider the workload and latency targets. Static batching suits steady, offline loads. Dynamic batching works for online services with moderate variability. Continuous batching is best for high-throughput, variable-length requests.