Semi-Supervised Learning: Learning from Few Labels and Many Unlabelled Examples

Labelling 100,000 medical images might require months of specialists' time; collecting the images themselves is easy. Semi-supervised learning (SSL) uses a small labelled set together with a large unlabelled set. When it works, it can approach fully supervised performance with a fraction of the labels. But it does not always work, and understanding why is the key.

When can unlabelled data help?#

Unlabelled data tells us about $p(\mathbf{x})$, the distribution of inputs. This helps predict $p(y \mid \mathbf{x})$ only if the two are linked. Three standard assumptions express such links:

1. Smoothness assumption — points close in a high-density region should share labels.
2. Cluster assumption — data forms clusters, and points in the same cluster tend to share a class. Equivalently, the decision boundary should pass through low-density regions.
3. Manifold assumption — data lies near a low-dimensional manifold, and labels vary smoothly along it.

Self-training (pseudo-labelling)#

The simplest method:

1. Train a model on labelled data.
2. Predict the unlabelled data; take the most confident predictions as pseudo-labels.
3. Add them to the training set and retrain. Repeat.

Risk: confirmation bias — early mistakes get reinforced. Mitigate with high confidence thresholds, class-balanced selection and re-evaluation of pseudo-labels each round.

Graph-based methods: label propagation#

Build a similarity graph over all points (labelled and unlabelled). Labels spread along edges: each unlabelled node iteratively takes a weighted average of its neighbours' label distributions, while labelled nodes stay fixed. This directly implements the smoothness and cluster assumptions. It works well for moderate dataset sizes with a meaningful distance.

import numpy as np
from sklearn.datasets import make_moons
from sklearn.semi_supervised import LabelSpreading, SelfTrainingClassifier
from sklearn.svm import SVC

X, y = make_moons(n_samples=1000, noise=0.08, random_state=0)
rng = np.random.default_rng(0)
y_partial = np.full_like(y, -1)                         # -1 means unlabelled
labelled = np.r_[rng.choice(np.where(y == 0)[0], 5, replace=False),
                 rng.choice(np.where(y == 1)[0], 5, replace=False)]
y_partial[labelled] = y[labelled]                       # only 10 labels!

sup = SVC(probability=True, gamma=2).fit(X[labelled], y[labelled])
print("supervised (10 labels):", round((sup.predict(X) == y).mean(), 3))

ls = LabelSpreading(kernel="knn", n_neighbors=10).fit(X, y_partial)
print("label spreading:       ", round((ls.transduction_ == y).mean(), 3))

st = SelfTrainingClassifier(SVC(probability=True, gamma=2), threshold=0.9).fit(X, y_partial)
print("self-training:         ", round((st.predict(X) == y).mean(), 3))

With only ten labels, label spreading follows the two crescent shapes and labels nearly everything correctly — the cluster assumption holds perfectly here.

Consistency regularisation#

Modern deep SSL relies on a powerful idea: a model's prediction should not change under realistic perturbations of the input. For an unlabelled example $\mathbf{x}$ and augmentations $\alpha(\cdot)$, add a loss

This pushes the decision boundary away from dense regions (where small perturbations would flip predictions). Methods include the Π-model, Mean Teacher (the target comes from an exponential moving average of the model's weights) and Virtual Adversarial Training.

FixMatch: a simple, strong recipe#

FixMatch (Sohn et al., 2020) combines pseudo-labelling and consistency:

1. For an unlabelled image, predict on a weakly augmented version (flip, crop).
2. If the maximum probability exceeds a threshold (e.g. 0.95), use its argmax as a pseudo-label.
3. Train the model to predict that pseudo-label on a strongly augmented version (colour distortion, cutout).

The loss is

On CIFAR-10, FixMatch reported strong accuracy using only a few labels per class — a dramatic demonstration of SSL's potential.

Self-supervised pretraining: the other route#

Today, a very common way to exploit unlabelled data is self-supervised pretraining (contrastive learning, masked modelling) followed by fine-tuning on the few labels. SSL and self-supervision are complementary: pretrain on everything, then apply semi-supervised fine-tuning.

Practical checklist#

• Ensure labelled and unlabelled data come from the same distribution; class-distribution mismatch or out-of-distribution unlabelled data harms SSL.
• Keep a labelled validation set — small but trustworthy — to verify SSL actually helps.
• Start simple: pseudo-labelling with a strong base model, or pretrained embeddings + label propagation.