Regularisation in Deep Learning: Weight Decay, Early Stopping, Augmentation and More

Zhang et al. (2017) showed that standard image-classification networks can reach 100% training accuracy on data with completely random labels. These networks have more than enough capacity to memorise. So why do they generalise on real data — and how do we help them? The answer is a combination of the implicit biases of training and a toolkit of explicit regularisers. This lecture surveys that toolkit.

1. Weight decay (L2 regularisation)#

Shrink weights towards zero at every step:

Small weights mean smoother functions that are less sensitive to input perturbations. For SGD this equals an L2 penalty; for Adam, use decoupled weight decay (AdamW). Typical values: $10^{-4}$–$5 \times 10^{-4}$ for SGD on CNNs; 0.01–0.1 for AdamW on transformers. Usually exclude biases and normalisation parameters.

2. Early stopping#

Monitor validation loss and stop when it stops improving, keeping the best checkpoint.

best, patience, bad_epochs = float("inf"), 10, 0
for epoch in range(max_epochs):
    train_one_epoch(model)
    val = evaluate(model)
    if val < best - 1e-4:
        best, bad_epochs = val, 0
        torch.save(model.state_dict(), "best.pt")
    else:
        bad_epochs += 1
        if bad_epochs >= patience:
            break
model.load_state_dict(torch.load("best.pt"))

For gradient descent on a quadratic loss, early stopping is provably related to L2 regularisation: the number of steps plays the role of the inverse penalty strength. It is cheap and nearly universal.

3. Data augmentation#

The most powerful regulariser in vision and speech: create new training examples by applying label-preserving transformations. It encodes invariances we know the task should have.

• Images: random crops, flips, rotations, colour jitter, blur, random erasing; learned policies like RandAugment and TrivialAugment.
• Audio: time shifting, speed perturbation, adding background noise, SpecAugment (masking time and frequency bands).
• Text: back-translation, synonym replacement, random deletion (use carefully — meaning changes easily).
• Tabular: harder; noise injection, or synthetic data generation, with caution.

4. Mixup and CutMix#

Mixup (Zhang et al., 2018) trains on convex combinations of pairs of examples and their labels:

It encourages linear behaviour between training examples, improving generalisation and calibration. CutMix pastes a rectangular patch from one image into another, mixing labels in proportion to the patch area — it keeps images locally realistic.

import torch
import numpy as np

def mixup(x, y_onehot, alpha=0.2):
    lam = np.random.beta(alpha, alpha)
    idx = torch.randperm(x.size(0))
    return lam * x + (1 - lam) * x[idx], lam * y_onehot + (1 - lam) * y_onehot[idx]

5. Label smoothing#

Replace hard one-hot targets with soft ones (e.g. 0.9 for the true class, the rest spread over others). It prevents logits from growing without bound and typically improves calibration and accuracy.

6. Dropout and stochastic depth#

Covered in the previous lecture — noise injected into activations or whole blocks.

7. Other techniques#

• Noise injection into inputs, weights or gradients.
• Parameter sharing — convolution shares weights across positions, a powerful built-in regulariser.
• Transfer learning — starting from pretrained weights acts as a strong prior.
• Ensembling and weight averaging — averaging weights along the training trajectory (Stochastic Weight Averaging, exponential moving averages of weights) finds flatter solutions and often generalises better.
• Sharpness-Aware Minimisation (SAM) — explicitly seeks parameters whose whole neighbourhood has low loss.

Implicit regularisation#

Even without explicit regularisers, SGD tends to find solutions that generalise: from small random initialisations, gradient descent on over-parameterised models favours low-norm, "simple" solutions, and SGD's noise biases it towards flat minima. This implicit bias is a major reason over-parameterised networks generalise despite their capacity — and why the double descent curve appears.

Putting it together#