Why were one-stage detectors less accurate than two-stage detectors for years? A 2017 paper from Facebook AI Research gave a precise answer — class imbalance — and a simple, elegant fix: the focal loss. Along the way we meet SSD, which introduced the multi-scale design that most one-stage detectors still follow.
SSD: Single Shot MultiBox Detector#
Liu et al. (2016) proposed SSD:
- A backbone (originally VGG-16) followed by extra convolutional layers of decreasing resolution.
- Predictions from multiple feature maps: high-resolution maps detect small objects, low-resolution maps large ones.
- At each location of each map, a set of default boxes (anchors) with several aspect ratios; small convolutional filters predict class scores and box offsets for each.
- Hard negative mining: because most default boxes are background, SSD keeps only the hardest negatives, at a negative:positive ratio of at most 3:1.
SSD was fast and more accurate than YOLOv1, but still behind Faster R-CNN on small objects.
The imbalance problem#
A one-stage detector evaluates on the order of $10^4$–$10^5$ candidate locations per image, while an image contains only a handful of objects. The vast majority are easy negatives — plain background, correctly classified with high confidence. Individually, each contributes a small loss; collectively, they dominate the total loss and gradient, swamping the few informative examples.
Two-stage detectors sidestep this: the RPN filters out most background, and the second stage samples a balanced mix of foreground and background. One-stage detectors used heuristics like hard negative mining, which discard data and require tuning.
The focal loss#
Lin, Goyal, Girshick, He and Dollár (2017) reshaped the cross-entropy loss. Let $p_t$ be the model's probability for the true class:
The focal loss adds a modulating factor:
- For easy examples ($p_t \to 1$), $(1 - p_t)^\gamma \to 0$ — their loss is strongly down-weighted.
- For hard examples ($p_t$ small), the factor is near 1 — the loss is almost unchanged.
- $\gamma$ controls the strength of focusing; $\gamma = 2$ worked best. With $\gamma = 2$, an example with $p_t = 0.9$ has its loss reduced 100×, and with $p_t \approx 0.968$ about 1000×.
- $\alpha_t$ is a class-balancing weight (0.25 for the positive class worked well with $\gamma = 2$).
import torch
import torch.nn.functional as F
def sigmoid_focal_loss(logits, targets, alpha=0.25, gamma=2.0):
p = torch.sigmoid(logits)
ce = F.binary_cross_entropy_with_logits(logits, targets, reduction="none")
p_t = p * targets + (1 - p) * (1 - targets)
alpha_t = alpha * targets + (1 - alpha) * (1 - targets)
return (alpha_t * (1 - p_t) ** gamma * ce).sum()
logits = torch.tensor([4.0, -4.0, 0.0, -1.0]); targets = torch.tensor([1.0, 0.0, 1.0, 1.0])
print("CE per example: ", F.binary_cross_entropy_with_logits(logits, targets, reduction="none"))
print("focal (summed): ", sigmoid_focal_loss(logits, targets))(torchvision.ops.sigmoid_focal_loss provides an optimised version.)
RetinaNet#
To test the focal loss, the authors built RetinaNet: a ResNet + FPN backbone with two small subnetworks applied at every pyramid level — one for classification (with sigmoid outputs per class) and one for box regression — using 9 anchors per location. Two details matter:
- Normalisation: the total loss is divided by the number of anchors assigned to ground-truth boxes, not the total number of anchors.
- Prior initialisation: the classification layer's bias is initialised so every anchor starts with a foreground probability of about $\pi = 0.01$:
Without this, the huge number of background anchors produces a destabilising loss at the start of training — the same trick we discussed for imbalanced problems generally.
RetinaNet matched or surpassed two-stage detectors' accuracy on COCO while keeping one-stage simplicity — a striking demonstration that the loss function, not the architecture, had been the bottleneck.
Legacy#
- Focal loss (and variants such as quality focal loss and varifocal loss) is used throughout detection, segmentation and other imbalanced classification problems — medical lesion detection, rare-event prediction.
- The "backbone + FPN + shared dense heads" design became the template for one-stage detectors (FCOS, ATSS, modern YOLOs).
- The idea of reweighting examples by difficulty connects to boosting, hard-example mining and curriculum learning.