Transfer learning: reusing a trained network for a new task
Transfer learning reuses knowledge from a model trained on one task to improve performance on a related task. It reduces the need for large datasets and computational resources. Common approaches include fine-tuning the entire network or using the pretrained model as a fixed feature extractor.
What is transfer learning?
Transfer learning is a machine learning technique where knowledge gained from one task or dataset is used to improve model performance on a related task, as explained in IBM's transfer learning overview. Traditional machine learning builds a new model for each task from scratch, but transfer learning starts from an already trained model.
Transfer learning reduces computational costs and the need for large datasets. According to AWS's explanation of transfer learning, it allows models to adapt to new tasks with smaller datasets and fewer resources. It also improves performance and robustness by exposing the model to varied initial training data.
How transfer learning works
The process of transfer learning can be broken into steps. Dive into Deep Learning's fine-tuning chapter describes pretraining a source model on a large dataset, creating a target model that copies all layers except the output layer, adding a new output layer with the right number of classes, and then training the target model on the target dataset.
Another common approach is to use the pretrained network as a fixed feature extractor. The PyTorch transfer learning tutorial freezes all network weights except the final fully connected layer, which is replaced and trained on the new data. In an example, it classifies ants and bees from about 120 training images per class.
Fine-tuning versus feature extraction
Fine-tuning and feature extraction differ in which layers are updated. The arXiv paper on feature transferability notes that early layer features are general, like Gabor filters and color blobs, while later layers become more specific. In fine-tuning, backpropagation continues through the network; in feature extraction, only a new classifier is trained on the activations.
Transfer learning is not limited to computer vision. The arXiv paper on ULMFiT applies inductive transfer learning to natural language processing by fine-tuning a language model for text classification. The method outperformed existing approaches on six tasks and matched training from scratch on 100 times more data when given only 100 labeled examples.
When to apply transfer learning
The decision between fine-tuning and feature extraction depends on the size of the new dataset and its similarity to the original dataset. According to Stanford CS231n notes, a small similar dataset suggests training a linear classifier on CNN features, while a large similar dataset supports fine-tuning the full network. For a small but different dataset, using features from an earlier layer often helps; for a large different dataset, fine-tuning the entire network with a pretrained initialization is common.
In practice, researchers must also watch for overfitting. Transfer learning reduces overfitting when the source and target tasks are related, but it can hurt performance when they are not. No standard metric yet exists to measure task similarity, so practitioners rely on empirical testing and domain knowledge.
| Aspect | Fine-tuning | Feature extraction |
|---|---|---|
| Training of pretrained layers | All layers are updated with a small learning rate | Pretrained layers are frozen; no weight updates |
| Training of new classifier | New output layer is trained from scratch | New classifier (e.g., linear SVM) is trained on extracted features |
| Dataset size requirement | Works well with medium to large datasets | Suitable for small datasets to avoid overfitting |
| Similarity requirement | Best when target dataset is similar to source | Can work even when datasets are somewhat different |
| Example use case | Fine-tuning a ResNet on a hot dog dataset | Using ImageNet features to classify ants and bees |
Frequently asked questions
What is negative transfer?
Negative transfer occurs when the source and target tasks are too dissimilar, causing the pretrained model to harm performance on the new task. IBM notes that transfer learning works best when both tasks are similar and data distributions do not vary greatly.
Can transfer learning be used for natural language processing?
Yes. The ULMFiT method, introduced by Howard and Ruder, applies transfer learning to text classification by fine-tuning a language model. It achieved strong results on six text classification tasks and matched the performance of training from scratch on 100 times more data using only 100 labeled examples.
How does transfer learning reduce overfitting?
By starting with a model that has already learned general features from a large dataset, transfer learning provides a strong prior. The IBM article explains that the retrained model contains knowledge from multiple datasets, which can improve generalization and inhibit overfitting on a small target dataset.
What is a pretrained model?
A pretrained model is a neural network that has already been trained on a large dataset, such as ImageNet. The Stanford CS231n notes state that these models are often shared publicly so others can use them for fine-tuning or feature extraction on related tasks.
Is transfer learning only for computer vision?
No. Transfer learning is used across many domains, including natural language processing and speech recognition. The AWS article describes its application in NLP, where models pretrained on large text corpora are fine-tuned for specific tasks like sentiment analysis.