Self-supervised learning: models that make their own training signal
Self-supervised learning is a machine learning technique that generates implicit labels from unstructured data, avoiding the need for manual annotation. It uses pretext tasks to learn representations, which are then applied to downstream tasks. This approach is central to models like BERT and SimCLR.
Definition and purpose
Self-supervised learning is a technique where models create their own supervisory signals from unstructured data, rather than relying on manually labeled datasets. This approach is especially valuable in computer vision and natural language processing, where obtaining labeled data is costly and time-consuming, as explained in IBM's self-supervised learning overview.
The term self-supervised learning was introduced by Yann LeCun to set these methods apart from traditional unsupervised learning, according to IBM's deep learning article. Unlike unsupervised learning, which does not compare results to any ground truth, self-supervised learning uses a ground truth that is implicitly derived from the data itself.
Self-supervised learning is a subset of unsupervised learning, but it differs in that it uses a loss function and gradient descent during training, similar to supervised learning. This allows it to be used for classification and regression tasks, whereas traditional unsupervised learning is used for clustering, anomaly detection, and dimensionality reduction.
Pretext and downstream tasks
In self-supervised learning, tasks are divided into two categories: pretext tasks and downstream tasks. A pretext task is used to train the model to learn meaningful representations of unstructured data, as described in IBM's self-supervised learning overview. These learned representations are then used as input to a downstream task, such as a supervised learning or reinforcement learning task.
The reuse of a pre-trained model on a new task is called transfer learning. This approach is common in large language models and computer vision models, where a model is first pre-trained on a large corpus of unlabeled data and then fine-tuned on a smaller labeled dataset for a specific task.
For example, BERT (Bidirectional Encoder Representations from Transformers) is pre-trained on unlabeled text by processing each token while considering both the words to its left and right in every layer, as stated in the BERT paper on arXiv. This pre-training allows BERT to be fine-tuned with just one additional output layer to achieve strong results on tasks like question answering and language inference.
Contrastive learning and SimCLR
Contrastive learning is a specific approach within self-supervised learning that learns representations by contrasting positive and negative pairs of data. The SimCLR framework, presented in the SimCLR paper on arXiv, simplifies contrastive self-supervised learning without requiring specialized architectures or a memory bank.
The SimCLR paper systematically studies the components of the framework and finds that the composition of data augmentations plays a critical role in defining effective predictive tasks. It also shows that adding a learnable nonlinear transformation between the representation and the contrastive loss greatly enhances the quality of the learned representations.
Contrastive learning performs better with larger batch sizes and more training steps than supervised learning. In the SimCLR paper, a linear classifier trained on self-supervised representations achieves 76.5% top-1 accuracy on ImageNet, matching the performance of a supervised ResNet-50.
Relationship to semi-supervised learning
Semi-supervised learning combines supervised and unsupervised learning by using both labeled and unlabeled data to train models for classification and regression tasks, as explained in IBM's semi-supervised learning article. It is distinguished from self-supervised learning, which uses no labeled data at all.
An increasingly common semi-supervised approach is to pre-train models via self-supervised tasks on unlabeled data, then fine-tune them on a small amount of labeled data. This is described in the same IBM article, which notes that when such tasks involve a ground truth and loss function without manual annotation, they are called self-supervised learning.
Both self-supervised and semi-supervised learning address the challenge of limited labeled data, but they do so differently. Self-supervised learning creates its own supervisory signal from the data, while semi-supervised learning uses a small amount of labeled data to guide learning from a larger unlabeled dataset.
| Paradigm | Uses labels | Ground truth | Typical tasks |
|---|---|---|---|
| Supervised | Yes | External labels | Classification, regression |
| Unsupervised | No | None | Clustering, anomaly detection |
| Self-supervised | No | Implicit from data | Classification, regression |
| Semi-supervised | Some | External labels for labeled subset | Classification, regression |
Frequently asked questions
What is self-supervised learning?
Self-supervised learning is a machine learning technique that generates implicit labels from unstructured data, allowing models to be trained without manual annotation. It is a subset of unsupervised learning but uses a ground truth derived from the data itself.
How does self-supervised learning differ from unsupervised learning?
Unlike traditional unsupervised learning, which does not measure results against any ground truth, self-supervised learning optimizes against a ground truth implicitly derived from unlabeled data, using a loss function and gradient descent.
What are pretext and downstream tasks?
Pretext tasks are used to train a model to learn meaningful representations of data, while downstream tasks are the actual tasks the model is applied to, such as classification or question answering, often after fine-tuning.
What is contrastive learning?
Contrastive learning is a self-supervised approach that learns representations by contrasting positive and negative pairs of data. SimCLR is a notable framework that simplifies this approach without specialized architectures.
How is self-supervised learning related to semi-supervised learning?
Semi-supervised learning uses both labeled and unlabeled data, while self-supervised learning uses only unlabeled data. However, self-supervised pre-training is often used in semi-supervised approaches to leverage unlabeled data before fine-tuning on labeled data.