Andrej Karpathy has observed that neural network training "fails silently". A bug in ordinary software usually crashes the program; a bug in a training pipeline often just produces a model that is somewhat worse than it should be. You may never know. This lecture presents a disciplined, step-by-step process for building and debugging deep-learning systems — a process I ask all my project students to follow.
Principle: start simple, add complexity gradually#
Begin with the simplest pipeline that could work, verify each component, and add one piece at a time. When something breaks, you know exactly which change caused it.
Step 1: become one with the data#
Before writing a model, look at the data:
- Visualise dozens of random examples with their labels. Are labels correct? Are images rotated? Are texts truncated?
- Check class balance, duplicates, missing values, value ranges and units.
- Inspect the data after preprocessing and augmentation — the exact tensors the model sees. Many bugs live here: images normalised twice, channels swapped (RGB vs BGR), augmentations destroying labels, tokenisation errors.
import matplotlib.pyplot as plt
xb, yb = next(iter(train_loader))
print(xb.shape, xb.dtype, xb.min().item(), xb.max().item(), xb.mean().item())
print("label counts in batch:", yb.bincount())
fig, axes = plt.subplots(2, 8, figsize=(14, 4))
for ax, img, lab in zip(axes.flat, xb[:16], yb[:16]):
ax.imshow(img.permute(1, 2, 0).squeeze(), cmap="gray"); ax.set_title(int(lab)); ax.axis("off")
plt.show()Step 2: establish baselines and sanity checks#
- Check the initial loss. For $K$ balanced classes with a well-initialised network, the initial cross-entropy should be about $\ln K$ (2.30 for 10 classes). If it is much higher, the output layer is initialised poorly or the loss is misconfigured.
- Initialise the output bias sensibly: for imbalanced binary problems, set it to $\log(p/(1-p))$ of the base rate.
- Human or simple baseline: a logistic regression or the majority class. Your network must beat it.
- Input-independent baseline: train with inputs set to zero. If the real model does not do better, it is not using the inputs.
Step 3: overfit a single batch#
Take one small batch (e.g. 2–32 examples) and train on it repeatedly. The loss should go to nearly zero. If it cannot, there is a bug — in the model, loss, labels, optimiser or data pipeline. This is the single most valuable debugging test.
xb, yb = next(iter(train_loader))
model.train()
for step in range(300):
loss = F.cross_entropy(model(xb), yb)
opt.zero_grad(); loss.backward(); opt.step()
if step % 50 == 0:
print(step, round(loss.item(), 4)) # should approach 0Step 4: verify the plumbing#
- Gradient check custom layers with finite differences.
- Check that every parameter receives a gradient: after
backward(), any parameter withgrad is Noneor all-zero gradients is disconnected. - Batch independence test: set the loss to depend only on example $i$ and check that gradients with respect to other examples' inputs are zero. Non-zero gradients reveal accidental mixing across the batch dimension — e.g. a wrong
viewor reduction over the wrong axis.
for name, p in model.named_parameters():
if p.grad is None or p.grad.abs().sum() == 0:
print("NO GRADIENT:", name)Step 5: monitor the right signals#
Log, per step or epoch:
- training and validation loss and metrics;
- learning rate;
- gradient norms (global and per layer) — spikes precede divergence;
- update-to-weight ratio $\|\Delta\mathbf{w}\|/\|\mathbf{w}\|$ — around $10^{-3}$ per step is a common healthy range; much larger suggests too high a learning rate;
- activation statistics — fraction of dead ReLUs, saturated sigmoids, activation means and standard deviations per layer;
- predictions on a fixed set of examples over time.
Tools: TensorBoard, Weights & Biases, MLflow.
Step 6: read the curves#
| Symptom | Likely cause | Try |
|---|---|---|
| Loss is NaN/inf | LR too high, log(0), bad data, fp16 overflow | Lower LR, logits-based losses, clip gradients, check data |
| Loss flat from the start | LR too low, dead units, bug, frozen params | LR range test, overfit-one-batch, check gradients |
| Loss decreases then explodes | LR too high late, no clipping | Schedule/decay, clipping, warm-up |
| Train ↓, val ↑ early | Overfitting | Regularise, augment, more data, early stop |
| Train and val both high | Underfitting | Bigger model, train longer, better features |
| Val better than train | Dropout/augmentation in train only, or leakage | Check splits for leakage |
| Great val, poor production | Distribution shift, preprocessing mismatch | Compare pipelines, monitor inputs |
Step 7: error analysis#
Examine the examples the model gets wrong, and the ones it gets wrong with high confidence. Group them into categories (blurry images, rare classes, label errors, ambiguous cases). Fix the largest category first. You will often discover label noise — confident "mistakes" that are actually mislabelled data.
Step 8: then tune and scale#
Only after the pipeline is correct: tune the learning rate, regularisation and architecture; scale data and model; ensemble. Change one thing at a time, fix random seeds, and keep an experiment log.