Convolutional Neural Networks: The Core Ideas

An image of $224 \times 224$ pixels with three colour channels has 150,528 input values. A fully connected layer with just 1,000 hidden units would need 150 million weights — and it would treat a cat in the top-left corner as unrelated to the same cat in the bottom-right. Convolutional neural networks (CNNs) solve both problems by building knowledge about images directly into the architecture. They were the engine of the deep learning revolution in computer vision.

Three structural priors#

1. Local connectivity — each unit looks only at a small neighbourhood (its receptive field), because nearby pixels are strongly related and meaningful patterns (edges, textures) are local.
2. Weight sharing — the same small filter is applied at every position, because a pattern useful in one location is useful everywhere.
3. Translation equivariance — shifting the input shifts the feature map by the same amount. Combined with pooling, this gives approximate translation invariance.

These priors massively reduce parameters and improve generalisation.

The convolution operation#

For a 2-D input $X$ and a $k \times k$ kernel $K$, the output feature map is

(Strictly speaking, this is cross-correlation; deep learning libraries call it convolution because the kernel is learned, so flipping is irrelevant.)

With $C_{\text{in}}$ input channels and $C_{\text{out}}$ output channels, each output channel has its own kernel spanning all input channels:

Parameter count of a conv layer: $C_{\text{out}} \times C_{\text{in}} \times k \times k + C_{\text{out}}$. A $3 \times 3$ conv from 64 to 128 channels has $128 \times 64 \times 9 + 128 = 73{,}856$ parameters — independent of image size.

What filters learn#

Hand-designed filters illustrate the idea. The kernel

(the Sobel operator) responds to vertical edges. A CNN learns its kernels by backpropagation. Visualisations of trained networks consistently show:

• first layer: oriented edges, colour blobs, Gabor-like patterns;
• middle layers: textures, corners, repeated motifs;
• deep layers: object parts — eyes, wheels, windows — and whole objects.

This hierarchy emerges because each layer's receptive field grows: stacking two $3 \times 3$ convs gives a $5 \times 5$ receptive field; three give $7 \times 7$.

A typical CNN#

Input image
→ [Conv → BatchNorm → ReLU] × 2 → Pool     (low-level features, high resolution)
→ [Conv → BatchNorm → ReLU] × 2 → Pool     (more channels, lower resolution)
→ [Conv → BatchNorm → ReLU] × 2 → Global average pool
→ Linear → class logits

As depth increases, spatial resolution decreases and the number of channels increases — trading "where" information for "what" information.

Building one in PyTorch#

import torch
import torch.nn as nn
import torch.nn.functional as F
from torchvision import datasets, transforms

class SmallCNN(nn.Module):
    def __init__(self, n_classes=10):
        super().__init__()
        def block(cin, cout):
            return nn.Sequential(nn.Conv2d(cin, cout, 3, padding=1, bias=False),
                                 nn.BatchNorm2d(cout), nn.ReLU(inplace=True))
        self.features = nn.Sequential(
            block(1, 32), block(32, 32), nn.MaxPool2d(2),      # 28 -> 14
            block(32, 64), block(64, 64), nn.MaxPool2d(2),     # 14 -> 7
            block(64, 128), nn.AdaptiveAvgPool2d(1))           # 7 -> 1
        self.head = nn.Linear(128, n_classes)
    def forward(self, x):
        return self.head(self.features(x).flatten(1))

model = SmallCNN()
print("parameters:", sum(p.numel() for p in model.parameters()))

tf = transforms.ToTensor()
train = datasets.FashionMNIST(".", train=True, download=True, transform=tf)
loader = torch.utils.data.DataLoader(train, batch_size=128, shuffle=True)
opt = torch.optim.AdamW(model.parameters(), lr=2e-3, weight_decay=1e-4)
model.train()
for epoch in range(2):
    for x, y in loader:
        opt.zero_grad(); loss = F.cross_entropy(model(x), y); loss.backward(); opt.step()
    print(f"epoch {epoch}: last batch loss {loss.item():.3f}")

With about 100,000 parameters, this network typically exceeds 90% accuracy on Fashion-MNIST after a couple of epochs — far better than an MLP of similar size.

Convolution variants you will meet#

• 1×1 convolutions — mix channels at each position without spatial extent; used to change channel counts cheaply (bottlenecks).
• Dilated (atrous) convolutions — spread kernel taps apart to enlarge the receptive field without more parameters; used in segmentation.
• Depthwise separable convolutions — a per-channel spatial conv followed by a 1×1 conv; dramatically cheaper (MobileNet).
• Transposed convolutions — learnable upsampling for segmentation and generative models.
• 1-D and 3-D convolutions — for audio/time series and video/volumetric medical scans.