Mixture of experts: sparse networks that route inputs to experts
Mixture of experts (MoE) is a neural network architecture that replaces dense feed-forward layers with multiple expert sub-networks and a learned router. The router selects a small subset of experts for each input token, so the model can have billions of parameters while only activating a fraction per step. This design increases capacity and speeds training and inference compared to dense models of similar quality.
Core idea and components
Mixture of experts is a machine learning approach that splits a model into several specialized sub-networks, each handling a portion of the input space, and a gating network decides which sub-network to use for each input, as explained in IBM's mixture of experts overview. This selective activation contrasts with dense models that run every parameter for every input.
In transformer models, MoE layers replace the dense feed-forward network (FFN) layers. Each MoE layer contains a number of experts, typically feed-forward networks, and a gate network that decides which tokens go to which expert, according to Hugging Face's mixture of experts explained. The router has learned parameters and is trained together with the rest of the network.
The key benefit is sparsity: only a small subset of the expert pool fires for each token, as noted in NVIDIA's mixture of experts glossary. This selective activation allows the model to grow its total parameter count without a proportional increase in computation, because not all parameters are used at any given time.
Early work and scaling
The roots of MoE go back to an early paper on adaptive mixtures of local experts, which proposed training separate networks each handling a different subset of training cases, with a gating network determining the weights for each expert, as explained in Hugging Face's history of MoEs. That early work showed faster training compared to a conventional model.
A major milestone was the Sparsely-Gated Mixture-of-Experts layer introduced by Google Brain, which scaled to up to 137 billion parameters for language modeling and machine translation, achieving large capacity gains with modest computational cost, as reported in the arXiv paper on outrageously large neural networks. The layer used a trainable gating network to select a sparse combination of experts for each example.
Later, GShard extended MoE to multilingual neural machine translation, scaling a Transformer model beyond 600 billion parameters using automatic sharding, and trained it efficiently on a large cluster of TPU v3 accelerators, according to the GShard paper. This demonstrated that MoE could be applied at very large scale with parallel hardware.
Simplifications and modern designs
The Switch Transformer simplified the MoE routing algorithm by sending each token to only one expert, which reduced communication and computational costs, and enabled training with lower precision bfloat16 formats, as described in the Switch Transformers paper. The authors reported up to 7x increases in pre-training speed with the same computational resources compared to T5 models.
Mixtral 8x7B is a sparse mixture of experts language model where each layer has 8 feedforward blocks as experts, and a router selects two experts per token at each layer, as stated in the Mixtral of experts paper. Although each token only sees two experts, the model has 47 billion total parameters, and only 13 billion are active during inference, allowing it to outperform or match larger dense models on several benchmarks.
Modern MoE models still face challenges such as load balancing, where the router must distribute tokens fairly across experts to avoid overfitting a small subset, and high memory requirements because all experts must be loaded in memory even though only some are used, as noted in Hugging Face's tradeoffs discussion.
Training and inference considerations
During training, experts become specialized in the areas where they perform best, and the gating network learns optimal routing, but careful load balancing is required to ensure all experts receive comparable training exposure, according to NVIDIA's explanation of expert specialization. Without balancing, popular experts can be overused and underperform.
For inference, MoE models can spread experts across many GPUs, with each device handling a small subset, which boosts throughput and latency when combined with high-speed networking, as described in NVIDIA's discussion of compute balancing. Effective load balancing prevents hotspots and ensures efficient use of compute, memory, and network resources.
The router must select experts for every token, and the outputs from those experts must be aggregated before moving to the next layer, which places strain on memory bandwidth when experts are distributed, as noted in NVIDIA's section on synchronizing experts. High-performance networking is critical to maintain throughput and minimize latency.
| Model or system | Number of experts per layer | Experts selected per token | Total parameters | Active parameters per token |
|---|---|---|---|---|
| Sparsely-Gated MoE | Up to thousands | Sparse combination | Up to 137 billion | Not stated |
| GShard | Not stated | Not stated | Beyond 600 billion | Not stated |
| Switch Transformer | Not stated | One | Up to trillion | Not stated |
| Mixtral 8x7B | 8 | 2 | 47 billion | 13 billion |
Frequently asked questions
What is a mixture of experts model?
A mixture of experts model divides a neural network into separate sub-networks, or experts, and uses a gating network to activate only the experts best suited for each input, as described by IBM.
How does sparsity work in MoE?
Sparsity means that only a small subset of the expert pool is activated for each token, so the model has many parameters but only uses a fraction per step, reducing computation.
What is the role of the router in MoE?
The router, or gating network, is a learned component that determines which experts should process each input token, and it is trained together with the rest of the network.
Why are MoE models memory intensive?
Even though only some experts are used per token, all experts must be loaded in memory during inference, so a model like Mixtral 8x7B requires enough VRAM for a dense 47B parameter model.
What are the main challenges of MoE?
Challenges include load balancing to avoid overusing a few experts, training instability, high communication costs when experts are distributed, and difficulties in fine-tuning that can lead to overfitting.