Backpropagation: how neural networks compute gradients layer by layer
Backpropagation is the standard method for computing gradients in neural networks. It applies the chain rule from calculus, traversing the network from output to input to find how each parameter affects the loss. This enables gradient descent for multi-layer networks.
What backpropagation is
Backpropagation, short for backward propagation of error, is a machine learning technique that calculates how changes to any weight or bias affect the accuracy of a model's predictions. It enables the use of gradient descent to update network parameters and is essential for training deep learning models. The formal algorithm was published by David E. Rumelhart, Geoffrey Hinton, and Ronald J. Williams in their 1986 paper 'Learning representations by back-propagating errors,' as described in IBM's overview of backpropagation.
The logic is based on the idea that the layers of neurons are a series of nested mathematical functions. During training, these are nested into a loss function that measures the difference between the desired output and the network's actual output. The chain rule, a calculus principle, is then used to compute the rate at which each neuron contributes to the overall loss.
In practice, many machine learning libraries handle backpropagation automatically, so the underlying calculations are performed by the software. Nevertheless, understanding the process helps in debugging and designing networks.
Forward propagation and computational graphs
Forward propagation, also called the forward pass, refers to the calculation and storage of intermediate variables in a neural network from the input layer to the output layer. According to Dive into Deep Learning's chapter on backpropagation, this includes the weighted sums, activation outputs, and the final loss. For a network with one hidden layer, the forward pass computes the hidden activations and then the output.
The dependencies between variables can be visualized with a computational graph, where squares denote variables and circles denote operators. The graph shows how data flows from input to output. Backpropagation then traverses this graph in reverse order, from output to input, applying the chain rule to calculate gradients with respect to the parameters.
The algorithm stores intermediate partial derivatives that are needed for the gradient calculations. This allows the gradient of the objective function to be computed efficiently.
Chain rule and gradient interpretation
The chain rule is the mathematical basis for backpropagation. For functions Y = f(X) and Z = g(Y), the derivative of Z with respect to X is the product of the derivative of Z with respect to Y and the derivative of Y with respect to X. In backpropagation, this is applied backward through the network, as explained in Stanford's CS231n notes on backpropagation.
Each derivative tells the sensitivity of the overall expression to a particular variable. For example, in a multiplication f(x,y) = xy, the derivative with respect to x is y, so a small increase in x changes the output by y times that increase. In an addition, the derivative is one for both inputs. The max operation passes the gradient only to the input that achieved the maximum.
The gradient is a vector of partial derivatives, and backpropagation can be seen as a backward flow of these gradients through the real-valued circuits of the network. This intuitive view helps in debugging and in designing new architectures.
Common training problems
Backpropagation can suffer from several failure modes. According to Google's Machine Learning Crash Course on backpropagation, vanishing gradients occur when the gradients for lower layers become very small because the product of many small terms. This can be mitigated by using the ReLU activation function.
Exploding gradients happen when large weights lead to gradients that are too large to converge. Batch normalization and lowering the learning rate can help. Dead ReLU units occur when a ReLU unit's output is stuck at zero, cutting off gradient flow. LeakyReLU and lower learning rates are common fixes.
Dropout regularization is a technique that randomly drops unit activations during a single gradient step, with the dropout rate controlling the strength of regularization.
Automatic differentiation in frameworks
Most deep learning frameworks provide automatic differentiation, which implements backpropagation behind the scenes. The PyTorch autograd documentation explains that tensors with the requires_grad=True flag are tracked, and calling backward() computes the gradients with respect to the leaf variables. The framework stores these gradients in the .grad attribute of each parameter.
A gentle introduction to torch.autograd describes the training loop: forward propagation makes a prediction, the loss is calculated, and backward propagation collects the derivatives of the error with respect to the parameters. An optimizer then uses these gradients to update the weights via gradient descent.
| Problem | Description | Common mitigation |
|---|---|---|
| Vanishing gradients | Gradients in lower layers become very small | Use ReLU activation |
| Exploding gradients | Gradients become too large to converge | Batch normalization or lower learning rate |
| Dead ReLU units | ReLU unit outputs zero, cutting gradient flow | Lower learning rate or use LeakyReLU |
Frequently asked questions
What is backpropagation?
Backpropagation is an algorithm that computes the gradient of a loss function with respect to all weights and biases in a neural network by applying the chain rule in reverse order.
How does backpropagation use the chain rule?
It multiplies partial derivatives of each layer's output with respect to its inputs, working backward from the output to the input, to find how each parameter affects the loss.
Why do neural networks need backpropagation?
It makes gradient descent practical for multi-layer networks by efficiently computing the gradients needed to update weights and minimize loss.
What is the role of automatic differentiation?
Frameworks like PyTorch implement backpropagation automatically, so users only need to set requires_grad and call backward() to get gradients.