Deep Q-Networks (DQN): Human-Level Atari from Pixels

In 2015 a paper in Nature by Mnih et al. at DeepMind reported an agent that learned to play 49 Atari 2600 games directly from screen pixels and the score, using the same architecture and hyperparameters for every game, and reached a level comparable to a professional human games tester on many of them. The Deep Q-Network (DQN) launched the field of deep reinforcement learning. Its key ideas were not a new learning rule — it is Q-learning — but the engineering that made Q-learning stable with deep neural networks.

The Q-network#

A convolutional network takes a state (a stack of the last 4 preprocessed $84 \times 84$ grayscale frames, to capture motion) and outputs a Q-value for each of the possible actions (up to 18 joystick actions). Loss for a transition $(s, a, r, s')$:

Stabiliser 1: experience replay#

Consecutive transitions are highly correlated, and the data distribution shifts as the policy changes — both bad for stochastic gradient descent. DQN stores transitions in a large replay buffer (1 million transitions) and trains on random mini-batches from it. Benefits:

• breaks temporal correlations (closer to i.i.d. data);
• reuses each transition many times (data efficiency);
• smooths the training distribution over many past policies.

This is possible because Q-learning is off-policy.

Stabiliser 2: target network#

The target depends on the same network being updated — a moving target that can chase itself into divergence. DQN computes targets with a separate target network $Q(\cdot; \theta^-)$, a copy of the online network updated only every $C$ steps (e.g. 10,000). Many implementations instead use soft updates $\theta^- \leftarrow \tau\theta + (1 - \tau)\theta^-$ with small $\tau$.

Other important details#

• Reward clipping to $[-1, 1]$ across games (so one learning rate works everywhere).
• Frame skipping: repeat each action for 4 frames.
• ε-greedy exploration annealed from 1.0 to 0.1 over the first million frames.
• Huber loss (or gradient clipping) for robustness to large TD errors.
• RMSProp optimiser (Adam in later work).

DQN for CartPole#

import random, collections
import numpy as np
import torch, torch.nn as nn, torch.nn.functional as F
import gymnasium as gym

env = gym.make("CartPole-v1")
obs_dim, n_actions = env.observation_space.shape[0], env.action_space.n
q = nn.Sequential(nn.Linear(obs_dim, 128), nn.ReLU(), nn.Linear(128, 128), nn.ReLU(), nn.Linear(128, n_actions))
q_target = nn.Sequential(nn.Linear(obs_dim, 128), nn.ReLU(), nn.Linear(128, 128), nn.ReLU(), nn.Linear(128, n_actions))
q_target.load_state_dict(q.state_dict())
opt = torch.optim.Adam(q.parameters(), lr=5e-4)
buffer = collections.deque(maxlen=50_000)
gamma, batch, steps = 0.99, 64, 0

def act(s, eps):
    if random.random() < eps:
        return env.action_space.sample()
    with torch.no_grad():
        return int(q(torch.as_tensor(s, dtype=torch.float32)).argmax())

for episode in range(400):
    s, _ = env.reset(seed=episode); done, ret = False, 0.0
    while not done:
        eps = max(0.05, 1.0 - steps / 10_000)             # annealed exploration
        a = act(s, eps)
        s2, r, term, trunc, _ = env.step(a); done = term or trunc
        buffer.append((s, a, r, s2, float(term))); s = s2; ret += r; steps += 1
        if len(buffer) >= 1_000:
            S, A, R, S2, D = map(np.array, zip(*random.sample(buffer, batch)))
            S, S2 = torch.tensor(S, dtype=torch.float32), torch.tensor(S2, dtype=torch.float32)
            A, R, D = torch.tensor(A), torch.tensor(R, dtype=torch.float32), torch.tensor(D)
            with torch.no_grad():
                target = R + gamma * (1 - D) * q_target(S2).max(1).values     # bootstrap only if not terminal
            pred = q(S).gather(1, A.unsqueeze(1)).squeeze(1)
            loss = F.smooth_l1_loss(pred, target)
            opt.zero_grad(); loss.backward(); nn.utils.clip_grad_norm_(q.parameters(), 10); opt.step()
        if steps % 500 == 0:
            q_target.load_state_dict(q.state_dict())       # periodic target update
    if episode % 50 == 0:
        print(f"episode {episode:3d} return {ret:5.0f} eps {eps:.2f}")

Note that we bootstrap only when the episode did not terminate: at a true terminal state the future value is zero, while at a time-limit truncation we still bootstrap.

What DQN learned — and could not#

DQN excelled at reactive games (Breakout — where it famously learned to tunnel through the wall — Pong, Space Invaders) but failed on games requiring long-horizon exploration with sparse rewards, notably Montezuma's Revenge, where random exploration almost never finds the first reward. That failure motivated research into intrinsic motivation and curiosity-driven exploration.