AI interconnects: NVLink, InfiniBand and Ethernet fabrics
AI workloads combine scale-up interconnects like NVLink for GPU-to-GPU traffic within a rack and scale-out networks like InfiniBand or Ethernet for server-to-server communication. The Ultra Ethernet Consortium aims to make standard Ethernet competitive with specialized interconnects for AI and HPC.
Scale-up versus scale-out interconnects
Large AI models are often too big for a single GPU, so they are split across multiple GPUs. A technique called intra-layer model parallelism, described in the Megatron-LM paper, distributes the transformer layers across GPUs, which requires frequent communication between them.
Interconnects fall into two categories. Scale-up networking links GPUs in close proximity, such as within a single server or rack, providing very high bandwidth and low latency. Scale-out networking connects separate servers to form a cluster, enabling larger deployments across many nodes.
The choice of interconnect depends on the workload size and topology, but both types are often used together in a data center. Scale-up handles communication inside a node, while scale-out handles traffic between nodes.
NVLink: a dedicated GPU scale-up fabric
NVLink is a direct GPU-to-GPU interconnect designed by NVIDIA. According to NVIDIA's NVLink page, the sixth generation provides 3 TB/s of bandwidth per GPU and 216 TB/s aggregate bandwidth for a rack-scale system connecting 72 GPUs. NVLink Switch enables all-to-all communication across a rack, effectively forming a single large GPU.
The NVLink Switch also includes engines for NVIDIA Scalable Hierarchical Aggregation and Reduction Protocol (SHARP), which performs in-network reductions and multicast acceleration. This feature offloads collective operations from the GPUs, improving system throughput.
NVLink is positioned as a scale-up fabric, connecting GPUs within a rack to support model parallelism and other all-to-all communication patterns common in AI training and inference.
InfiniBand: a specialized scale-out network
InfiniBand is an industry-standard I/O architecture for interconnecting servers, storage, and communications equipment, as defined by the InfiniBand Trade Association. It leverages switched, point-to-point channels and provides reliable messaging and Remote Direct Memory Access (RDMA), which reduces CPU overhead and latency.
The InfiniBand Trade Association page notes that InfiniBand offers ultra-low latency, with end-to-end delays measured at 600 nanoseconds, and that NDR 400Gb/s products are shipping with a roadmap to 1.6Tb/s. InfiniBand is used in a majority of the world's fastest supercomputers, connecting many of the top systems.
NVIDIA's InfiniBand platform, described on NVIDIA's InfiniBand page, includes adapters, switches, and software. It features SHARP for in-network computing, self-healing network capabilities, and support for various topologies such as Fat Tree and Dragonfly+. These features make InfiniBand a dedicated scale-out fabric for AI and HPC clusters.
Ethernet and the Ultra Ethernet Consortium
Ethernet is a widely deployed networking technology, but traditional Ethernet lacks features critical for AI workloads, such as multi-pathing, fast congestion reaction, and low tail latency. The Ultra Ethernet Consortium is an organization that aims to deliver an Ethernet-based open, interoperable, high-performance full-communications stack specifically for AI and HPC.
According to the consortium, it focuses on improving bandwidth, latency, tail latency, and scale while maintaining Ethernet interoperability. UEC plans to engage with standards development organizations to integrate its specifications into the broader Ethernet family of technologies.
UEC's goal is to combine the ubiquity and cost-effectiveness of Ethernet with the performance of specialized interconnects. The consortium is organized under the Linux Foundation and includes a wide range of members from the networking and computing industry.
| Interconnect | Fabric role | Key capability | Example bandwidth |
|---|---|---|---|
| NVLink | Scale-up (GPU-to-GPU) | All-to-all with SHARP | 3 TB/s per GPU |
| InfiniBand | Scale-out (server-to-server) | RDMA, low latency, SHARP | 400 Gb/s (NDR) |
| Ethernet (UEC) | Scale-out (general) | Open, interoperable, multi-pathing | No single figure; UEC aims for high performance |
Frequently asked questions
What is the difference between scale-up and scale-out interconnects?
Scale-up interconnects connect GPUs within a single rack or node for high bandwidth and low latency, while scale-out interconnects connect servers across a cluster to handle inter-node traffic.
How does NVLink enable GPU-to-GPU communication?
NVLink provides a direct high-bandwidth link between GPUs, and NVLink Switch extends this to connect many GPUs in an all-to-all topology, facilitating model parallelism.
Why is low latency important for AI networks?
AI training involves frequent all-to-all communications and collective operations, where tail latency can significantly impact overall performance, making low and consistent latency crucial.
What is the Ultra Ethernet Consortium?
It is an organization developing open Ethernet-based specifications to deliver high performance for AI and HPC networking, while maintaining interoperability with existing Ethernet standards.
Can Ethernet replace InfiniBand for AI workloads?
UEC aims to bring Ethernet to comparable performance levels, but InfiniBand remains a mature, dedicated solution with features like SHARP and self-healing, widely used in supercomputing.