Dropout: silencing random neurons in training to reduce overfitting
Dropout is a regularization technique that randomly removes neurons and their connections during training, preventing co-adaptation and overfitting. At test time, a single unthinned network with scaled weights approximates the average of many thinned networks. Modern frameworks implement dropout with a probability parameter and scaling factor, and it can also be used to estimate model uncertainty.
The problem of overfitting
Deep neural networks with many parameters are powerful but prone to overfitting, especially when trained on small datasets. Overfitting means the model performs well on training data but poorly on unseen test data. The JMLR dropout paper explains that large networks are also slow to use, making it impractical to combine predictions from many different networks at test time.
The arXiv paper on preventing co-adaptation notes that when a large feedforward network is trained on a small set, it typically performs poorly on held-out data. This poor generalization is substantially reduced by randomly turning off half of the feature detectors for each training example. The method prevents complex co-adaptations where a feature detector is only useful in the context of several other specific detectors.
How dropout works
Dropout injects noise into the internal layers of a network during forward propagation. As described in Dive into Deep Learning's dropout chapter, on each training iteration, a fraction of nodes in each layer are zeroed out before calculating the subsequent layer. This is equivalent to training a thinned network that contains only a subset of the original neurons.
The PyTorch Dropout documentation states that during training, elements of the input tensor are randomly zeroed with probability p, chosen independently for each forward call from a Bernoulli distribution. The outputs are scaled by a factor of 1/(1-p) during training, so that during evaluation the module computes an identity function.
The Keras Dropout layer documentation explains that inputs not set to 0 are scaled up by 1/(1-rate) so that the sum over all inputs is unchanged. The layer only applies dropout when training is set to True, so no values are dropped during inference.
Practical considerations
Typically, dropout is disabled at test time. Given a trained model and a new example, no nodes are dropped, so no normalization is needed. However, some researchers use dropout at test time as a heuristic for estimating prediction uncertainty, as noted in Dive into Deep Learning. If predictions agree across many different dropout outputs, the network is considered more confident.
The Stanford CS231n course notes discuss dropout as one of several regularization methods, alongside L2/L1 regularization and maxnorm. They describe dropout as a technique that randomly sets a fraction of activations to zero during forward pass, which helps prevent overfitting by making the network more resilient to noise.
When implementing dropout, the probability p (or rate) is a hyperparameter that controls the fraction of units dropped. A common default is 0.5, as seen in the PyTorch documentation. The noise_shape argument in Keras allows specifying a binary mask shape, useful for applying the same mask across certain dimensions, such as timesteps.
Dropout as Bayesian approximation
Beyond regularization, dropout can be interpreted as approximate Bayesian inference. The arXiv paper on dropout as a Bayesian approximation develops a theoretical framework that casts dropout training in deep neural networks as approximate inference in deep Gaussian processes. This provides tools to model uncertainty with dropout networks, extracting information that was previously discarded.
This Bayesian view mitigates the problem of representing uncertainty in deep learning without sacrificing computational complexity or test accuracy. The paper demonstrates considerable improvement in predictive log-likelihood and root mean square error compared to existing methods, and applies dropout's uncertainty to deep reinforcement learning.
| Framework | Parameter | Default value | Scaling during training | Training mode flag |
|---|---|---|---|---|
| PyTorch | p | 0.5 | Yes, by 1/(1-p) | Yes, module.train() |
| Keras | rate | None (required) | Yes, by 1/(1-rate) | Yes, training argument in call() |
| Dive into Deep Learning (custom) | p | 0.5 (example) | Yes, by 1/(1-p) | Yes, manual implementation |
Frequently asked questions
What is dropout in neural networks?
Dropout is a regularization technique that randomly drops units (neurons) along with their connections during training. This prevents units from co-adapting too much and reduces overfitting, as described in the JMLR paper.
How does dropout reduce overfitting?
By randomly omitting neurons during training, dropout forces each neuron to learn features that are generally helpful, rather than relying on specific combinations of other neurons. This breaks co-adaptation and improves generalization to unseen data.
Is dropout used at test time?
Typically, dropout is disabled at test time. The network uses all neurons without scaling. However, some researchers use dropout at test time as a heuristic to estimate prediction uncertainty, as mentioned in Dive into Deep Learning.
What is the scaling factor in dropout?
During training, the outputs are scaled by a factor of 1/(1-p) to keep the expected value of activations unchanged. This ensures that the sum over all inputs remains the same, as explained in the Keras documentation.
Can dropout be used for uncertainty estimation?
Yes, dropout can be interpreted as approximate Bayesian inference, as shown in the arXiv paper by Gal and Ghahramani. By applying dropout at test time and running multiple forward passes, one can estimate model uncertainty.