Neural scaling laws: how model size, data and compute relate
Neural scaling laws describe how language model performance improves as a power law with model size, dataset size, and training compute. Larger models are more sample-efficient, but compute-optimal training requires balancing model size and data. Recent work shows that many large models are undertrained and that model size and training tokens should scale equally.
Power-law scaling of loss
Empirical studies of language models show that the cross-entropy loss follows a power-law relationship with model size, dataset size, and the amount of compute used for training. The OpenAI scaling laws paper reports trends spanning more than seven orders of magnitude, with architectural details such as width or depth having minimal effects within a wide range.
These power laws allow researchers to predict performance before training and to allocate compute budgets optimally. The relationships also govern how overfitting depends on model and dataset size, and how training speed depends on model size.
The scaling behavior is not limited to language models. The Dive into Deep Learning chapter on large-scale pretraining notes that Transformer-based models in general, including vision Transformers, show superior scaling with increased model size and training compute.
Sample efficiency and model size
A key finding from the OpenAI scaling laws paper is that larger models are significantly more sample-efficient. This means that a larger model can achieve the same loss with fewer training examples than a smaller model.
Consequently, the best use of a fixed compute budget is to train a very large model on a relatively small dataset and stop well before convergence. This counterintuitive result has shaped subsequent model development.
The GPT-3 paper demonstrates that scaling up language models greatly improves task-agnostic few-shot performance. GPT-3, an autoregressive model with 175 billion parameters, was trained without fine-tuning and achieved strong results on translation, question-answering, and cloze tasks.
Compute-optimal training
The DeepMind compute-optimal training paper investigates the optimal model size and number of training tokens for a given compute budget. The authors trained hundreds of language models with parameter counts from 70 million to over 16 billion and token counts from 5 to 500 billion, and found that current large language models are significantly undertrained.
For compute-optimal training, the model size and the number of training tokens should increase at the same rate: when the model size doubles, the number of training tokens should also double. This finding led to the training of Chinchilla, a 70-billion-parameter model that outperformed larger models like Gopher (280B) and GPT-3 (175B) on many downstream tasks.
The NeurIPS version of the GPT-3 paper also notes that scaling up improves few-shot performance, sometimes becoming competitive with fine-tuning approaches, but identifies datasets where few-shot learning still struggles.
Implications for foundation models
The Stanford foundation models report discusses the paradigm shift toward models trained on broad data at scale, such as BERT, DALL-E, and GPT-3. These foundation models are adaptable to a wide range of downstream tasks, but their scale results in new emergent capabilities and homogenization.
Homogenization means that defects in the foundation model are inherited by all adapted models downstream. The report emphasizes that despite widespread deployment, there is a lack of clear understanding of how these models work, when they fail, and what they are capable of due to emergent properties.
The scaling laws provide a framework for understanding the trade-offs between model size, data, and compute, but they also highlight the need for careful consideration of the societal and technical risks associated with very large models.
| Study | Model sizes studied | Key finding | Optimal scaling |
|---|---|---|---|
| OpenAI scaling laws | Up to 1.5B parameters | Loss follows power-law with size, data, compute | Larger models are more sample-efficient |
| DeepMind compute-optimal | 70M to 16B parameters | Current models are undertrained | Model size and tokens scale equally |
| GPT-3 few-shot | 175B parameters | Few-shot performance improves with scale | No fine-tuning needed for many tasks |
Frequently asked questions
What are neural scaling laws?
Neural scaling laws are empirical relationships showing that language model loss decreases as a power-law with model size, dataset size, and training compute. They help predict performance and allocate resources.
Why are larger models more sample-efficient?
Larger models can achieve the same loss with fewer training examples, as shown in the OpenAI scaling laws paper. This means that compute-optimal training often uses very large models on relatively modest data.
What is compute-optimal training?
Compute-optimal training finds the model size and number of training tokens that minimize loss for a given compute budget. DeepMind's Chinchilla paper found that model size and tokens should scale equally.
How do scaling laws affect foundation models?
Scaling laws guide the development of foundation models by indicating that larger models trained on more data improve performance, but also introduce risks like homogenization and inherited defects, as noted in the Stanford report.
Do scaling laws apply to all architectures?
The scaling laws were observed for Transformer-based language models. The Dive into Deep Learning chapter notes that Transformers show superior scaling behavior, but the exact power-law exponents may vary.