PPO and TRPO: Stable Policy Optimisation with Trust Regions

Policy gradient methods have a fragile step size. Move too little and learning crawls; move too far and the new policy may collapse, after which it collects bad data and cannot recover. Trust region methods limit how much the policy changes per update. TRPO did this with rigorous theory; PPO achieved similar stability with a first-order trick that is easy to implement. PPO became the default deep RL algorithm for many problems — from robotics simulation and game-playing agents to fine-tuning language models with human feedback.

The surrogate objective#

Suppose we collected data with an old policy $\pi_{\theta_{\text{old}}}$. Using importance sampling, the expected advantage of a new policy can be estimated from old data with the surrogate objective:

where $r_t(\theta)$ is the probability ratio. At $\theta = \theta_{\text{old}}$, its gradient equals the policy gradient. But the approximation is accurate only when the new policy stays close to the old one — maximising it blindly produces destructive updates.

TRPO: a KL-constrained step#

Schulman et al. (2015) — Trust Region Policy Optimisation — maximise the surrogate subject to a limit on the average KL divergence:

Theory guarantees (approximately) monotonic improvement. TRPO solves the constrained problem with a second-order approximation: the natural gradient direction $\mathbf{F}^{-1}\mathbf{g}$ (where $\mathbf{F}$ is the Fisher information matrix) computed with conjugate gradients, followed by a line search. Effective but complex, and awkward with architectures that share parameters between policy and value networks.

PPO: clipping the ratio#

Schulman et al. (2017) — Proximal Policy Optimisation — replaced the hard constraint with a clipped objective:

with $\epsilon \approx 0.2$. How it works:

• If an action was good ($\hat{A}_t > 0$), increasing its probability helps — but only until the ratio reaches $1 + \epsilon$; beyond that, the objective is flat, so there is no incentive to move further.
• If an action was bad ($\hat{A}_t < 0$), decreasing its probability helps — but only until the ratio reaches $1 - \epsilon$.
• Taking the minimum makes the objective a pessimistic lower bound: changes that would make things look better than the clipped estimate are ignored.

Because the objective limits incentive for large changes, PPO can safely run several epochs of mini-batch gradient steps on the same batch of experience — much more sample-efficient than vanilla policy gradients.

The PPO algorithm#

1. Collect $T$ steps from $N$ parallel environments with $\pi_{\theta_{\text{old}}}$.
2. Compute advantages with GAE and value targets.
3. For $K$ epochs (e.g. 4–10), shuffle the batch into mini-batches and maximise

1. Set $\theta_{\text{old}} \leftarrow \theta$ and repeat.

import torch
import torch.nn.functional as F

def ppo_loss(new_logp, old_logp, advantages, values, returns, entropy,
             clip_eps=0.2, vf_coef=0.5, ent_coef=0.01):
    ratio = torch.exp(new_logp - old_logp)
    adv = (advantages - advantages.mean()) / (advantages.std() + 1e-8)   # advantage normalisation
    surr1 = ratio * adv
    surr2 = torch.clamp(ratio, 1 - clip_eps, 1 + clip_eps) * adv
    policy_loss = -torch.min(surr1, surr2).mean()
    value_loss = F.mse_loss(values, returns)
    approx_kl = (old_logp - new_logp).mean()                           # useful for monitoring/early stop
    clip_frac = ((ratio - 1).abs() > clip_eps).float().mean()
    loss = policy_loss + vf_coef * value_loss - ent_coef * entropy.mean()
    return loss, {"approx_kl": approx_kl.item(), "clip_frac": clip_frac.item()}

n = 64
loss, stats = ppo_loss(torch.randn(n) * 0.1 - 1, torch.full((n,), -1.0), torch.randn(n),
                       torch.randn(n), torch.randn(n), torch.full((n,), 0.6))
print(loss.item(), stats)

Practical details that matter#

Engstrom et al. (2020) and Huang et al. ("The 37 Implementation Details of PPO") showed that PPO's performance depends heavily on implementation details: advantage normalisation, value-function clipping, observation and reward normalisation, orthogonal initialisation, learning-rate annealing, gradient clipping, GAE, and early stopping when the approximate KL exceeds a target. Use well-tested implementations (Stable-Baselines3, CleanRL) and log approx KL and clip fraction to diagnose training.

# Using a battle-tested implementation
from stable_baselines3 import PPO
model = PPO("MlpPolicy", "CartPole-v1", n_steps=1024, batch_size=64, n_epochs=10,
            gae_lambda=0.95, clip_range=0.2, ent_coef=0.0, verbose=0, seed=0)
model.learn(total_timesteps=100_000)

PPO in the wider world#

• Games and robotics: OpenAI Five (Dota 2) trained with a large-scale PPO variant; many robot locomotion and manipulation policies are trained with PPO in simulation.
• Language models: InstructGPT-style RLHF used PPO with a KL penalty to a reference model; GRPO (used for recent reasoning models) is a PPO variant that replaces the critic with advantages computed relative to a group of sampled responses.