Optimizers in deep learning: SGD, momentum, Adam and AdamW explained
Deep learning optimizers adjust model parameters to minimize a loss function. Stochastic gradient descent (SGD) samples one example per update, momentum adds a velocity term, and Adam combines momentum with per-parameter scaling. AdamW decouples weight decay from the gradient update, which improves generalization.
Gradient descent and stochastic gradient descent
Gradient descent minimizes a cost function by moving parameters along the negative gradient, as described in IBM's gradient descent overview. The learning rate controls step size, and the process repeats until the error is near zero. Batch gradient descent evaluates the entire dataset before updating, which can be slow for large data.
Stochastic gradient descent (SGD) instead samples one training example uniformly at random at each iteration and updates parameters based on that example's gradient. The Dive into Deep Learning chapter on SGD notes that this reduces per-iteration cost from O(n) to O(1) and that the stochastic gradient is an unbiased estimate of the full gradient. The noisy updates can help escape local minima but require careful learning rate tuning.
Momentum
Momentum modifies SGD by adding a short-term memory of past gradients. The update uses a velocity term that accumulates gradients with a decay factor beta, as explained in Distill's article on momentum. When beta is zero, the algorithm reduces to plain gradient descent; typical values are 0.9 or 0.99. Momentum dampens oscillations and can provide a quadratic speedup on many functions.
In practice, frameworks like PyTorch implement momentum as an option of the SGD optimizer. The PyTorch torch.optim documentation shows constructing an SGD optimizer with `momentum=0.9` and a learning rate, and it also allows per-parameter groups with different learning rates or weight decay.
Adam: adaptive moment estimation
Adam combines momentum with per-parameter scaling based on the second moment of gradients. The algorithm maintains exponentially weighted moving averages of both the gradient and its square, with default decay factors beta1=0.9 and beta2=0.999, as stated in the Dive into Deep Learning Adam chapter. Bias correction is applied to the initial estimates, and an epsilon term improves numerical stability.
The original Adam paper on arXiv describes the method as computationally efficient, invariant to diagonal rescaling of gradients, and suitable for large data or parameter spaces. It also discusses connections to other algorithms and provides convergence analysis.
Adam is often stable and requires little tuning, but it can diverge in some situations due to variance control issues, as noted in the Dive into Deep Learning chapter. Despite this, it remains a popular choice for training neural networks.
AdamW: decoupled weight decay
Weight decay is a regularization technique that penalizes large parameters. For standard SGD, L2 regularization and weight decay are equivalent when rescaled by the learning rate, but this is not true for adaptive optimizers like Adam. The decoupled weight decay paper on arXiv demonstrates that implementing L2 regularization within Adam's update is not equivalent to true weight decay, and proposes a modification that decouples the weight decay step from the gradient-based update.
This modification, called AdamW, improves generalization performance and allows the optimal weight decay factor to be chosen independently of the learning rate. The paper shows that AdamW can compete with SGD with momentum on image classification tasks. PyTorch includes AdamW as a separate optimizer class, and its usage follows the same pattern as other optimizers in the torch.optim package.
| Optimizer | Momentum term | Per-parameter scaling | Weight decay handling |
|---|---|---|---|
| SGD | No (or optional momentum) | No | L2 regularization integrated with gradient |
| SGD with momentum | Yes (velocity term) | No | L2 regularization integrated with gradient |
| Adam | Yes (first moment) | Yes (second moment) | L2 regularization in update (inequivalent to weight decay) |
| AdamW | Yes (first moment) | Yes (second moment) | Decoupled, applied separately |
Frequently asked questions
What is the difference between SGD and gradient descent?
Gradient descent computes the gradient over the entire dataset per update, while SGD uses one randomly sampled example. This reduces per-iteration cost and introduces noise that can help escape local minima.
How does momentum help optimization?
Momentum maintains a velocity that accumulates past gradients, which smooths updates and accelerates convergence, especially in regions with pathological curvature.
Why does Adam use bias correction?
The moving averages are initialized at zero, which introduces a bias toward small values early in training. Bias correction divides by a factor that accounts for the decay, yielding unbiased estimates.
What problem does AdamW solve?
In Adam, L2 regularization applied inside the adaptive update is not equivalent to true weight decay. AdamW decouples the weight decay from the gradient step, improving generalization and decoupling the choice of weight decay from learning rate.
Which optimizer should one choose for a new model?
The choice depends on the problem. SGD with momentum is simple and often generalizes well, while Adam and AdamW are stable and work well for many architectures. AdamW is often preferred when weight decay is needed.