Vanishing and exploding gradients: why deep nets were hard to train
Deep networks face vanishing or exploding gradients because the gradient is a product of many layer Jacobians. Small eigenvalues shrink the product, while large eigenvalues inflate it. Proper initialization, bounded activations, gradient clipping, and residual connections mitigate these problems.
The gradient as a product of Jacobians
In a deep feedforward network, the gradient of the loss with respect to a parameter in an early layer is obtained by the chain rule. It becomes a product of Jacobian matrices for each layer between that parameter and the output. Dive into Deep Learning's numerical stability section explains that this gradient is the product of L-l matrices and a gradient vector. The eigenvalues of those matrices may be small or large, and their product can be very large or very small.
The multiplication is analogous to multiplying many probabilities, which can cause numerical underflow or overflow. Because the eigenvalues vary widely, the gradient magnitude is unpredictable. If the product is too large, parameter updates destroy the model; if too small, learning barely progresses.
Vanishing gradients and saturated activations
The logistic sigmoid activation is unsuited for deep networks with random initialization. Glorot and Bengio's study finds that its mean value can drive the top hidden layer into saturation. Saturated units move out of saturation slowly, which explains the plateaus seen during training.
The derivative of the sigmoid is near zero when its input is far from the center. Multiplying many such small derivatives across layers makes the gradient product decay exponentially, which is a primary cause of vanishing gradients.
Exploding gradients in recurrent networks
Recurrent neural networks reuse the same weights across time steps, so the gradient involves a product of Jacobians for each step. Pascanu, Mikolov, and Bengio's paper analyzes the vanishing and exploding gradient problems from analytical, geometric, and dynamical systems perspectives. It proposes gradient norm clipping for exploding gradients and a soft constraint for vanishing gradients.
Gradient norm clipping caps the size of the gradient before updating parameters. This prevents updates from becoming too large and destabilizing training. The paper validates the strategy empirically.
Remedies: initialization, activation, and residual connections
Careful weight initialization is a key remedy. He et al. derive an initialization that accounts for rectifier nonlinearities and enables training very deep rectified models directly from scratch. He's rectifier paper presents this stable method.
Residual connections provide another solution. He's residual learning paper reformulates layers to learn residual functions with reference to their inputs, making deeper networks easier to optimize and allowing accuracy gains from increased depth.
Beyond initialization, batch normalization and careful data preprocessing help stabilize training. The CS231n notes discuss weight initialization and batch normalization as tools to maintain stable gradients.
| Problem | Cause | Symptom | Mitigation |
|---|---|---|---|
| Vanishing | Product of Jacobians with eigenvalues below 1, often due to sigmoid saturation | Gradient becomes exponentially small; learning slows or stops | Glorot initialization, ReLU activation, residual connections |
| Exploding | Product of Jacobians with eigenvalues above 1 | Gradient becomes very large; parameter updates overshoot | Gradient norm clipping, careful initialization |
Frequently asked questions
What causes vanishing gradients?
Multiplication of many small derivatives, such as the sigmoid's near-zero gradient, makes the gradient shrink exponentially, so early layers learn slowly.
How does gradient clipping help?
It caps the gradient norm before updating parameters, preventing exploding gradients that would otherwise overshoot the optimum.
Why does initialization matter?
Proper initialization keeps the variance of activations and gradients consistent, avoiding initial products that are too large or too small.
What do residual connections do?
They offer a direct path for gradients to flow backward, reducing the number of Jacobian multiplications and mitigating vanishing gradients.