Memory bandwidth and HBM: why moving data limits AI speed
AI accelerators depend on moving large amounts of data between memory and compute. High-bandwidth memory (HBM) stacks DRAM dies to provide wider interfaces and higher throughput. Many deep learning operations are limited by memory bandwidth rather than compute capacity.
The memory bottleneck in AI
GPU operations are not always limited by arithmetic throughput. According to the NVIDIA GPU performance background user's guide, performance is limited by one of three factors: memory bandwidth, math bandwidth, or latency. Many deep learning layers, such as element-wise additions or scaling, involve far more data movement than computation.
Transformers and large language models amplify this problem because they process sequences that require access to many parameters and intermediate states. The Hugging Face optimization guide notes that loading a model with X billion parameters requires roughly 2 * X GB of VRAM in bfloat16 or float16 precision. For a model like BLOOM with 176 billion parameters, that means over 350 GB of data must be in memory.
The FlashAttention paper argues that attention algorithms should be IO-aware, accounting for reads and writes between GPU memory levels. Reducing memory accesses between high-bandwidth memory (HBM) and on-chip SRAM can yield significant speedups for transformers.
How high-bandwidth memory works
High-bandwidth memory (HBM) is a type of DRAM that stacks multiple dies vertically to achieve high data rates while reducing the distance and energy needed for data transfer. The Semiconductor Engineering HBM knowledge center explains that individual DRAM dies are connected using through-silicon vias (TSVs), and the stack is placed alongside GPUs or accelerators through advanced packaging.
The broad interface and short distance to compute make HBM a good fit for AI and high-performance computing, where moving large volumes of data is a major performance and power constraint. The same knowledge center notes that as generations increase bandwidth, capacity, and stack height, packaging and test challenges become more difficult, with yield losses as more dies are added.
HBM generations and bandwidth gains
HBM has evolved through several generations. The Micron high-bandwidth memory page states that Micron's HBM4 is in high-volume production, delivering over 11 Gb/s pin speeds and bandwidth greater than 2.8 TB/s, a 2.3 times increase over HBM3E and greater than 20% power efficiency improvement.
The NVIDIA Hopper architecture blog describes the H100 SXM5 as the first GPU to use HBM3 memory, providing more than 3 TB/sec of memory bandwidth, with a 50 MB L2 cache to reduce trips to DRAM. These bandwidth levels are essential for feeding large models and long sequences.
Software techniques to reduce memory traffic
Beyond hardware, algorithms can reduce how often data moves. The FlashAttention paper introduces an IO-aware attention algorithm that uses tiling to cut the number of memory reads and writes between GPU HBM and on-chip SRAM. It reports a 15% end-to-end speedup on BERT-large, a 3x speedup on GPT-2, and a 2.4x speedup on long-range tasks.
The Hugging Face optimization guide discusses lower precision (8-bit and 4-bit) to reduce memory footprint, as well as architectural innovations like multi-query attention and grouped-query attention. These techniques lower the bandwidth demand by shrinking the data that must be moved.
| Model | Parameters (billions) | Approximate VRAM (GB) |
|---|---|---|
| GPT-3 | 175 | 350 |
| BLOOM | 176 | 352 |
| Llama-2-70b | 70 | 140 |
| Falcon-40b | 40 | 80 |
| MPT-30b | 30 | 60 |
| bigcode/starcoder | 15.5 | 31 |
Frequently asked questions
What is memory bandwidth?
Memory bandwidth is the rate at which data can be read from or written to memory. In GPUs, memory bandwidth from high-bandwidth DRAM is often the limiting factor for operations that move large amounts of data, such as element-wise updates or attention calculations.
Why is HBM faster than standard DRAM?
HBM stacks multiple DRAM dies vertically and connects them with through-silicon vias, creating a wide interface. This allows many bits to move in parallel, and placement near the processor reduces distance and energy per transfer compared with conventional memory mounted farther away.
Does HBM replace all other memory in a system?
HBM is used primarily for accelerators and high-performance processors. It sits close to the compute die, while systems still contain conventional DRAM or storage for capacity. HBM offers high bandwidth but limited capacity per stack.
How can a developer reduce memory bandwidth pressure?
Techniques include lowering numeric precision to reduce the bytes per parameter, using IO-aware algorithms like FlashAttention that minimize HBM access, and designing attention mechanisms such as multi-query attention to cut memory footprint. These methods help keep data movement within the available bandwidth.