Learning from Imbalanced Data

Fraud is rare. Serious disease is rare. Equipment failure is rare. Yet these rare events are usually exactly what we want to detect. In imbalanced datasets, a standard classifier minimising average error can achieve excellent accuracy by ignoring the minority class entirely. Today we study how to learn and evaluate properly when classes are imbalanced.

Step 1: fix the evaluation first#

Before changing the model, change the metric. Accuracy is meaningless when 99% of examples are negative. Use:

• Precision, recall and F1 for the minority class;
• the precision–recall curve and average precision (baseline = prevalence);
• balanced accuracy or MCC;
• cost-based metrics when error costs are known;
• stratified splits so every fold contains minority examples.

Often, a model trained normally on imbalanced data already ranks examples well (good PR-AUC) and only needs a better threshold. Check that before resampling anything.

Step 2: threshold moving#

A classifier outputs $P(y = 1 \mid \mathbf{x})$. The default threshold 0.5 is appropriate only for equal error costs. With false-negative cost $c_{FN}$ and false-positive cost $c_{FP}$, a well-calibrated model should predict positive when

Alternatively, choose the threshold on validation data to hit a required recall or a review capacity. Threshold moving is simple, preserves calibration and is often all you need.

Step 3: cost-sensitive learning (class weights)#

Weight the loss so that minority errors count more:

A common choice is $w_c \propto 1/n_c$ (class_weight="balanced" in scikit-learn; scale_pos_weight in XGBoost/LightGBM). Weighting changes the decision boundary without discarding or duplicating data.

Step 4: resampling#

• Random under-sampling of the majority class — fast and effective with huge datasets, but discards information.
• Random over-sampling of the minority class — duplicates examples; risks overfitting to those exact points.
• SMOTE (Synthetic Minority Over-sampling Technique, Chawla et al., 2002) — creates synthetic minority examples by interpolating between a minority point and one of its $k$ minority nearest neighbours:

Variants such as Borderline-SMOTE and ADASYN focus on hard examples near the boundary; Tomek links and ENN clean overlapping majority examples.

Step 5: algorithmic approaches#

• Ensembles with balanced sampling — e.g. Balanced Random Forest, EasyEnsemble (bagging on balanced subsamples).
• Focal loss — down-weights easy examples so training focuses on hard ones; popular in object detection where background dominates.
• Anomaly detection — when positives are extremely rare or diverse, model the normal class only.

Putting it together#

import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import StratifiedKFold, cross_val_predict
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import average_precision_score, f1_score, recall_score, precision_score
from imblearn.pipeline import make_pipeline as imb_pipeline
from imblearn.over_sampling import SMOTE
from sklearn.preprocessing import StandardScaler

X, y = make_classification(n_samples=20000, n_features=15, n_informative=6,
                           weights=[0.99, 0.01], class_sep=1.0, random_state=0)
cv = StratifiedKFold(5, shuffle=True, random_state=0)

setups = {
    "plain":        imb_pipeline(StandardScaler(), LogisticRegression(max_iter=2000)),
    "class weight": imb_pipeline(StandardScaler(), LogisticRegression(max_iter=2000, class_weight="balanced")),
    "SMOTE":        imb_pipeline(StandardScaler(), SMOTE(random_state=0), LogisticRegression(max_iter=2000)),
}
for name, pipe in setups.items():
    p = cross_val_predict(pipe, X, y, cv=cv, method="predict_proba")[:, 1]
    pred = (p >= 0.5).astype(int)
    print(f"{name:<13} AP={average_precision_score(y, p):.3f}  "
          f"P={precision_score(y, pred):.3f}  R={recall_score(y, pred):.3f}  F1={f1_score(y, pred):.3f}")

# Threshold moving on the plain model: pick the threshold maximising F1 (on out-of-fold predictions)
p = cross_val_predict(setups["plain"], X, y, cv=cv, method="predict_proba")[:, 1]
ths = np.linspace(0.02, 0.9, 89)
best = max(ths, key=lambda t: f1_score(y, (p >= t).astype(int)))
print(f"plain + threshold {best:.2f}: F1={f1_score(y, (p >= best).astype(int)):.3f}")

Typically all approaches reach similar average precision (ranking quality), while their F1 at threshold 0.5 differs greatly — confirming that much of the "imbalance problem" is really a threshold problem. (In a real project, choose the threshold on a validation set separate from the final test set.)

Practical recommendations#

1. Use appropriate metrics and stratified splits.
2. Start with a strong model (e.g. gradient boosting) and tune the threshold.
3. Try class weights next; they are cheap and preserve all data.
4. Try resampling (SMOTE, under-sampling) if the above is insufficient — inside pipelines only.
5. Recalibrate probabilities if they will be used as risks.
6. Above all, collect more minority examples if at all possible — nothing beats real data.