Generative Adversarial Networks: The Generator–Discriminator Game

In 2014 Ian Goodfellow and colleagues proposed a strikingly original idea: train two networks against each other. A generator creates fake data; a discriminator tries to tell fake from real. As each improves, the fakes become more convincing. Generative Adversarial Networks (GANs) went on to produce photorealistic faces, art and image translations, and dominated image generation until diffusion models arrived around 2021.

The game#

• Generator $G$: maps random noise $\mathbf{z} \sim p(\mathbf{z})$ (e.g. Gaussian) to a sample $G(\mathbf{z})$.
• Discriminator $D$: outputs $D(\mathbf{x}) \in (0, 1)$, the probability that $\mathbf{x}$ is real.

They play a minimax game:

The discriminator maximises a binary cross-entropy objective (label real as 1, fake as 0); the generator minimises it — it wants $D(G(\mathbf{z}))$ close to 1.

The theory#

For a fixed generator with sample distribution $p_g$, the optimal discriminator is

Substituting it back gives

where JSD is the Jensen–Shannon divergence. So, with an optimal discriminator, the generator minimises the JS divergence between the data and generated distributions; the global optimum is $p_g = p_{\text{data}}$, where $D^* = 1/2$ everywhere. Notice that GANs never evaluate a likelihood — they are implicit generative models.

The non-saturating loss#

Early in training the discriminator easily rejects poor fakes, so $\log(1 - D(G(\mathbf{z})))$ saturates and gives the generator vanishing gradients. In practice the generator instead maximises $\log D(G(\mathbf{z}))$ — the same fixed point, much stronger gradients.

Training loop#

Alternate: one (or a few) discriminator steps on a batch of real and fake data, then one generator step.

import torch
import torch.nn as nn

z_dim = 100
G = nn.Sequential(                                    # DCGAN-style generator for 28x28 images
    nn.Linear(z_dim, 128 * 7 * 7), nn.BatchNorm1d(128 * 7 * 7), nn.ReLU(),
    nn.Unflatten(1, (128, 7, 7)),
    nn.ConvTranspose2d(128, 64, 4, 2, 1), nn.BatchNorm2d(64), nn.ReLU(),   # 14x14
    nn.ConvTranspose2d(64, 1, 4, 2, 1), nn.Tanh())                        # 28x28, values in [-1, 1]
D = nn.Sequential(
    nn.Conv2d(1, 64, 4, 2, 1), nn.LeakyReLU(0.2),
    nn.Conv2d(64, 128, 4, 2, 1), nn.BatchNorm2d(128), nn.LeakyReLU(0.2),
    nn.Flatten(), nn.Linear(128 * 7 * 7, 1))                              # logit

bce = nn.BCEWithLogitsLoss()
optG = torch.optim.Adam(G.parameters(), lr=2e-4, betas=(0.5, 0.999))
optD = torch.optim.Adam(D.parameters(), lr=2e-4, betas=(0.5, 0.999))

def train_step(real):                                 # real: (B, 1, 28, 28) scaled to [-1, 1]
    B = real.size(0)
    fake = G(torch.randn(B, z_dim))
    # Discriminator: real -> 1, fake -> 0
    lossD = bce(D(real), torch.ones(B, 1)) + bce(D(fake.detach()), torch.zeros(B, 1))
    optD.zero_grad(); lossD.backward(); optD.step()
    # Generator: non-saturating loss, wants D(fake) -> 1
    lossG = bce(D(fake), torch.ones(B, 1))
    optG.zero_grad(); lossG.backward(); optG.step()
    return lossD.item(), lossG.item()

print(train_step(torch.rand(16, 1, 28, 28) * 2 - 1))

The DCGAN guidelines (Radford et al., 2016) made convolutional GANs train reliably: strided convolutions instead of pooling, batch normalisation, ReLU in the generator and LeakyReLU in the discriminator, tanh output, and Adam with $\beta_1 = 0.5$.

Why GANs are hard to train#

• Non-convergence: gradient descent on a two-player game can oscillate rather than converge to an equilibrium.
• Mode collapse: the generator produces only a few kinds of outputs that fool the discriminator (e.g. only one digit), ignoring the diversity of the data.
• Vanishing gradients when the discriminator becomes too strong — related to the JS divergence being constant when distributions do not overlap.
• Sensitivity to architecture and hyperparameters; loss values are not a reliable progress indicator — inspect samples and FID.

Stabilisation techniques#

• Wasserstein GAN (Arjovsky et al., 2017): replace JS with the Earth-Mover (Wasserstein-1) distance, which gives meaningful gradients even when distributions do not overlap; the "critic" must be 1-Lipschitz, enforced by weight clipping or, better, a gradient penalty (WGAN-GP).
• Spectral normalisation of discriminator weights (Miyato et al., 2018).
• Two time-scale update rule (different learning rates for G and D).
• Label smoothing, minibatch discrimination, feature matching, R1 regularisation, exponential moving averages of generator weights.

Strengths and legacy#

GANs produce sharp, high-fidelity samples and generate in a single forward pass — much faster than early diffusion models. They lost the lead in general image generation to diffusion models, which train more stably and cover modes better, but adversarial losses remain widely used: as a perceptual component in image super-resolution, in the autoencoders of latent diffusion, in neural vocoders for speech (HiFi-GAN), and to distil diffusion models into fast few-step generators.