Residual connections: skip paths that made deep networks trainable
Residual connections add a shortcut that lets a layer learn a residual mapping instead of the full mapping, making very deep networks trainable. Introduced with ResNet, they allow gradients to flow directly and are used in many modern architectures.
The training problem with deep networks
Deep neural networks become harder to train as they gain depth. The residual learning paper states that deeper networks are more difficult to train and that the proposed residual framework makes them easier to optimize, allowing accuracy gains from increased depth (the residual learning paper).
The paper demonstrates this by reporting that an ensemble of residual networks achieved a 3.57% error on the ImageNet test set and won the ILSVRC classification task. Residual networks could also be 8 times deeper than VGG nets while having lower complexity.
How a residual block works
A residual block uses the input x and learns a residual mapping defined as f(x) minus x, so the block output is the sum g(x) + x. This makes it easier to learn the identity mapping, because then g(x) is zero (the Dive into Deep Learning text).
The shortcut that adds x is called a skip connection. Research on identity mappings in deep residual networks shows that when the skip connection is an identity and activation comes after the addition, forward and backward signals can propagate directly from one block to any other block, which improves training (the identity mappings paper).
The earlier idea of highway networks
Highway networks, introduced by Srivastava et al., used gating units to regulate the flow of information across layers, describing these paths as information highways. They showed that networks with hundreds of layers could be trained directly using stochastic gradient descent (the highway networks abstract).
Highway networks share the goal of easing deep training but rely on learned gates rather than a fixed identity shortcut. The residual approach instead uses a constant identity path, which avoids additional parameters.
Later architectures and why skip connections matter
The idea of skip connections spread beyond image recognition. The Transformer architecture, introduced in the Attention is all you need paper, relies solely on attention mechanisms and dispenses with recurrence and convolutions. It achieves a BLEU score of 28.4 on the WMT English-to-German translation task (the Attention is all you need paper).
In biomedical image segmentation, the U-Net architecture uses a contracting path to capture context and a symmetric expanding path for precise localization. It can be trained end-to-end from very few images, and it outperformed the prior best method on the ISBI challenge for neuronal structure segmentation (the U-Net paper). This shows that skip-like connections, though different in form, are useful across domains.
| Property | Explanation |
|---|---|
| Residual mapping | Learn the residual function f(x) minus x; the output is g(x) plus x |
| Identity shortcut | The added path is the input itself, not a learned gate |
| Direct signal propagation | Forward and backward signals can flow directly when identity and post-addition activation are used |
| Ease of identity learning | If f(x) = x is desired, g(x) = 0 is easy to learn |
Frequently asked questions
What is a residual connection?
A residual connection adds a block's input to its output, so the block learns a residual function rather than the full mapping.
Why do residual connections make deep networks trainable?
With an identity shortcut and after-addition activation, signals can propagate directly forward and backward, easing optimization.
How do residual connections differ from highway networks?
Highway networks use learned gating units, while residual connections use a fixed identity shortcut.
Which well-known architectures use residual connections?
Residual blocks appear in later models such as Transformers, and U-Net uses skip connections of a different form for segmentation.