Double Descent and the Generalisation Mystery of Deep Learning

Classical statistics teaches a U-shaped curve: as model complexity grows, test error first falls (less bias) and then rises (more variance). Past the point where a model perfectly fits the training data, the prediction is disaster. Yet modern deep networks, with far more parameters than training examples, interpolate their training data and generalise well. This lecture examines one of the most interesting open questions in machine learning.

The puzzle#

Zhang et al. (2017), "Understanding deep learning requires rethinking generalization", showed that:

Standard CNNs reach 100% training accuracy on CIFAR-10 with completely random labels — they can memorise.
The same architectures, on real labels, generalise well.
Explicit regularisers (weight decay, dropout, augmentation) help but are not necessary for good generalisation.

So capacity-based explanations (VC dimension, Rademacher complexity) cannot explain why these networks generalise: the same model class that generalises on real data can fit noise perfectly.

Double descent#

Belkin, Hsu, Ma and Mandal (2019) unified classical and modern regimes in the double descent curve:

In the under-parameterised regime, test error follows the classical U-shape.
At the interpolation threshold — when the model has just enough capacity to fit the training data exactly — test error peaks.
In the over-parameterised regime, test error decreases again, often below the classical minimum.

Nakkiran et al. (2019) showed double descent occurs in deep networks along several axes:

Model-wise: increasing width;
Epoch-wise: training longer can make test error rise then fall again;
Sample-wise: in a critical regime, adding more data can temporarily hurt.

The peak is most pronounced with label noise and little regularisation.

Why the peak at the threshold?#

Near the interpolation threshold there is essentially one way to fit the training data exactly, and that solution must contort itself to pass through every noisy point — huge norm, wild behaviour between points. Past the threshold there are infinitely many interpolating solutions, and the training algorithm can choose a well-behaved one.

Minimum-norm interpolation#

For linear regression with more features than examples ($d > n$), infinitely many $\mathbf{w}$ satisfy $\mathbf{X}\mathbf{w} = \mathbf{y}$. Gradient descent initialised at zero converges to the minimum-norm solution:

\hat{\mathbf{w}} = \mathbf{X}^\top(\mathbf{X}\mathbf{X}^\top)^{-1}\mathbf{y} = \mathbf{X}^+\mathbf{y}

As $d$ grows beyond $n$, the minimum norm solution can become smoother and test error can fall — double descent in its simplest form.

python

import numpy as np

rng = np.random.default_rng(0)
n_train, n_test, D = 40, 1000, 400
w_true = rng.normal(size=D) / np.sqrt(D)
X_all = rng.normal(size=(n_train + n_test, D))
y_all = X_all @ w_true + 0.3 * rng.normal(size=n_train + n_test)
Xtr, ytr, Xte, yte = X_all[:n_train], y_all[:n_train], X_all[n_train:], y_all[n_train:]

for p in [5, 10, 20, 30, 38, 40, 42, 50, 80, 150, 400]:
    w = np.linalg.pinv(Xtr[:, :p]) @ ytr          # least squares / minimum-norm solution
    mse = np.mean((Xte[:, :p] @ w - yte) ** 2)
    print(f"features={p:>3}  test MSE={mse:9.3f}")

Test error spikes as the number of features approaches the number of training examples (40), then falls again as features increase further.

Implicit regularisation#

The key idea: the training algorithm itself prefers certain solutions.

Gradient descent on linear models finds the minimum-norm interpolant.
On separable classification with logistic loss, gradient descent converges in direction to the maximum-margin solution (Soudry et al., 2018) — like an SVM, without explicit regularisation.
SGD noise biases training towards flat minima — wide basins where the loss changes little under parameter perturbations — which are associated with better generalisation (though "flatness" depends on parameterisation and is debated).
Small initialisation and architecture choices induce simplicity biases: networks tend to learn simple functions (low-frequency, low-complexity) first, a phenomenon called spectral bias.

Benign overfitting#

Bartlett et al. (2020) characterised benign overfitting in linear regression: interpolating noisy data can still give near-optimal test error when the data has many "unimportant" directions that can absorb the noise harmlessly, while the important directions are learned accurately. The noise is fitted, but in a way that barely affects predictions on new data.

Grokking#

Power et al. (2022) observed that small transformers trained on algorithmic tasks (e.g. modular arithmetic) can first memorise the training set, and then — long after training accuracy reaches 100% and with weight decay — suddenly generalise. Mechanistic analyses found the networks eventually form structured algorithms (e.g. Fourier-based representations of modular addition). Grokking illustrates how optimisation dynamics and regularisation interact over long training.

What this means in practice#

Theory is catching up — neural tangent kernels, mean-field analyses, PAC-Bayes and compression bounds, and studies of feature learning — but a complete explanation of deep learning's generalisation remains an active research frontier, and an excellent area for a thesis.

#generalization#double descent#overparameterization#implicit regularization#theory

Written by

Janin A Apurba

B.Sc. in CSE, AUST · Advanced ICT Officer, CNRS-UNHCR. Teaching AI, ML and Deep Learning to the next generation of engineers and researchers.