Pretraining: how large networks learn general patterns first
Pretraining is a technique where a deep network is first trained on massive amounts of unlabeled data to learn general language or image representations. This pretrained model is then fine-tuned on a smaller, task-specific dataset, which substantially improves accuracy and reduces the need for labeled examples. The approach is central to modern deep learning systems like BERT and other transformer models.
The role of pretraining
Deep learning models benefit from large training sets, but many tasks have only a few thousand labeled examples. To close this gap, researchers pretrain a model on enormous unlabeled text from the web, then fine-tune it for a specific task. This method is described in the Google Research blog post as a way to address the shortage of task-specific data.
Pretraining allows a model to learn general language structure and semantics from raw text. After that, the model can be adapted to tasks like question answering and sentiment analysis with a small labeled dataset, yielding substantial accuracy improvements compared to training from scratch.
From context-free to context-aware representations
Traditional word embeddings such as word2vec and GloVe are context-free, meaning they assign the same vector to a word regardless of its meaning in a sentence. The Dive into Deep Learning textbook explains that these representations cannot capture polysemy, as with the word "crane" in different contexts.
Contextual representations were introduced with models like ELMo, which uses a bidirectional language model to assign each word a representation based on its surrounding text. The ELMo paper shows that adding these learned vectors improves performance across six NLP problems, including question answering and sentiment analysis.
GPT uses an autoregressive left-to-right language model, but its representations only consider the words before the target. This limits its ability to capture full sentence context compared to bidirectional approaches.
Masked language modeling and BERT
BERT pretrains a Transformer encoder using masked language modeling, where random tokens are replaced with a mask and the model predicts them from all other tokens. The Dive into Deep Learning chapter describes how the encoder has no attention constraints, so all tokens attend to each other.
Because the prediction depends on both left and right context, BERT is a deeply bidirectional encoder. This allows it to represent words with full sentence context, unlike unidirectional models. BERT also learns relationships between sentences through a next-sentence prediction task.
Fine-tuning for downstream tasks
After pretraining, the model is fine-tuned on a specific task by adding an output layer and updating all parameters. The Hugging Face course explains that pretrained models are trained in a self-supervised way, then fine-tuned with human-annotated labels.
During fine-tuning, the additional layer is trained from scratch while the pretrained weights are updated to fit the task data. This transfer learning approach reduces the compute and data needed for new tasks, making it possible to build powerful systems with limited resources.
Scaling and impact
Large pretrained models exhibit scaling behavior, where performance improves steadily with more parameters, tokens, and compute. The T5 paper introduces a unified text-to-text framework for studying transfer learning across dozens of language understanding tasks.
BERT achieved strong results on eleven NLP tasks, including a GLUE score of 80.5% and an SQuAD v1.1 F1 of 93.2, as reported in the BERT paper. These results demonstrate the effectiveness of pretraining a large model on unlabeled text before fine-tuning it on specific benchmarks.
| Model | Context direction | Architecture | Pretraining objective | Fine-tuning approach |
|---|---|---|---|---|
| ELMo | Bidirectional (shallow) | Bidirectional LSTM | Language modeling (forward and backward) | Added as features, frozen weights |
| GPT | Left-to-right (forward) | Transformer decoder | Autoregressive language modeling | Fine-tune all parameters |
| BERT | Bidirectional (deep) | Transformer encoder | Masked language modeling and next sentence prediction | Fine-tune all parameters plus new output layer |
Frequently asked questions
What is pretraining in deep learning?
Pretraining is the process of training a large neural network on a huge general dataset to learn broad patterns, then fine-tuning it for a specific task. This reduces the need for large labeled datasets.
Why is pretraining useful?
Pretraining allows models to learn general language or image representations from unlabeled data, which can then be adapted to specific tasks with small labeled datasets, improving accuracy and reducing training cost.
What is masked language modeling?
Masked language modeling randomly hides tokens in input text and trains the model to predict them using surrounding context. This is a self-supervised objective used by BERT to learn bidirectional representations.
How does fine-tuning work after pretraining?
Fine-tuning adds a task-specific output layer and updates all parameters of the pretrained model using labeled data for the target task. This adapts the general knowledge to the specific requirement.
What is the difference between pretraining and transfer learning?
Pretraining is the initial training on large unlabeled data, while transfer learning is the broader concept of applying knowledge from one task to another. Fine-tuning on a downstream task is a form of transfer learning.