Residual Connections: Why Very Deep Networks Became Trainable

In 2015, researchers at Microsoft noticed something paradoxical. A 56-layer plain convolutional network had higher training error than a 20-layer one. This was not overfitting — the deeper network was worse even on the training data. A deeper network should be able to do at least as well, since the extra layers could simply learn the identity function. Yet optimisation failed to find that solution. Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun's solution — residual learning — enabled networks with more than 150 layers, won the 2015 ImageNet competition, and became one of the most important ideas in deep learning.

The degradation problem#

As depth increases in plain networks (even with BatchNorm and good initialisation), accuracy saturates and then degrades. Normalisation had largely addressed vanishing gradients in magnitude, so the issue is subtler: deep compositions of non-linear layers are hard to optimise, and learning even an identity mapping through a stack of non-linear layers is surprisingly difficult for SGD.

Residual learning#

Instead of asking a block of layers to learn a desired mapping $H(\mathbf{x})$ directly, let it learn the residual $F(\mathbf{x}) = H(\mathbf{x}) - \mathbf{x}$, and add the input back through a skip (shortcut) connection:

If the optimal mapping is close to the identity, the block only needs to push $F$ towards zero — which is easy (e.g. small weights). Each block learns a correction to its input rather than a whole new representation.

When dimensions differ (e.g. after downsampling), the shortcut uses a projection: $\mathbf{y} = F(\mathbf{x}) + \mathbf{W}_s\mathbf{x}$, typically a $1 \times 1$ convolution with stride.

The gradient highway#

For a stack of residual blocks, $\mathbf{x}_{L} = \mathbf{x}_l + \sum_{i=l}^{L-1}F(\mathbf{x}_i)$. The gradient is

The identity term means the gradient from the loss reaches every earlier layer directly, without passing through a product of many weight matrices. It cannot vanish merely because of depth (He et al., "Identity Mappings in Deep Residual Networks", 2016).

Other perspectives#

• Ensembles of paths: Veit et al. (2016) showed a ResNet behaves like an ensemble of exponentially many shallower paths; deleting a single block from a trained ResNet barely hurts, unlike a plain network.
• Iterative refinement: each block refines the representation incrementally.
• Dynamical systems: $\mathbf{x}_{l+1} = \mathbf{x}_l + F(\mathbf{x}_l)$ is an Euler step of an ordinary differential equation — the insight behind Neural ODEs.
• Smoother loss landscapes: visualisations show skip connections turn chaotic loss surfaces into much smoother, more convex-looking ones.

Residual blocks in code#

import torch
import torch.nn as nn

class BasicBlock(nn.Module):
    def __init__(self, cin, cout, stride=1):
        super().__init__()
        self.f = nn.Sequential(
            nn.Conv2d(cin, cout, 3, stride, 1, bias=False), nn.BatchNorm2d(cout), nn.ReLU(inplace=True),
            nn.Conv2d(cout, cout, 3, 1, 1, bias=False), nn.BatchNorm2d(cout))
        self.shortcut = nn.Identity() if stride == 1 and cin == cout else nn.Sequential(
            nn.Conv2d(cin, cout, 1, stride, bias=False), nn.BatchNorm2d(cout))
        nn.init.zeros_(self.f[-1].weight)          # zero-init: each block starts as identity
        self.act = nn.ReLU(inplace=True)
    def forward(self, x):
        return self.act(self.f(x) + self.shortcut(x))

class PreNormResidual(nn.Module):              # the transformer-style residual
    def __init__(self, dim, sublayer):
        super().__init__()
        self.norm, self.sublayer = nn.LayerNorm(dim), sublayer
    def forward(self, x):
        return x + self.sublayer(self.norm(x))

x = torch.randn(2, 64, 32, 32)
print(BasicBlock(64, 128, stride=2)(x).shape)       # (2, 128, 16, 16)

Zero-initialising the last normalisation weight in each residual branch makes every block start as an identity function — training begins as a shallow network and gradually "grows" depth. Goyal et al. found this improves large-batch training.

Variants and descendants#