Recommender Systems II: Matrix Factorisation and Latent Factors

The Netflix Prize (2006–2009) offered one million dollars to anyone who could improve Netflix's rating predictions by 10%. The decisive technique was matrix factorisation: represent every user and every item as a short vector of latent factors so that their dot product predicts the rating. The idea is simple, scales to huge data and underlies the embedding-based retrieval used by today's recommendation systems.

The model#

Choose a latent dimension $k$ (e.g. 20–200). Learn user vectors $\mathbf{p}_u \in \mathbb{R}^k$ and item vectors $\mathbf{q}_i \in \mathbb{R}^k$ such that

where $\mu$ is the global mean rating, $b_u$ a user bias (some people rate everything high) and $b_i$ an item bias (some films are universally liked). Stacking vectors, $\mathbf{R} \approx \mathbf{P}\mathbf{Q}^\top$ — a low-rank approximation.

The latent dimensions often become interpretable after training: one might separate serious dramas from light comedies; another, films for children from films for adults. Nobody labels these dimensions; they emerge from the data.

Why not just use the SVD?#

Classical SVD requires a complete matrix. Filling missing entries with zeros or means would treat "unseen" as "disliked" and swamp the model with fake data. Instead we fit only the observed entries $\mathcal{K}$:

Regularisation is essential: with many parameters and sparse data, the model would otherwise memorise the observed ratings.

Training method 1: stochastic gradient descent#

For each observed rating, compute the error $e_{ui} = r_{ui} - \hat{r}_{ui}$ and update:

(and similarly for biases). This is Simon Funk's famous approach from the Netflix Prize.

import numpy as np

def train_mf(ratings, n_users, n_items, k=20, lr=0.01, reg=0.05, epochs=30, seed=0):
    rng = np.random.default_rng(seed)
    P = 0.1 * rng.normal(size=(n_users, k)); Q = 0.1 * rng.normal(size=(n_items, k))
    bu, bi = np.zeros(n_users), np.zeros(n_items)
    mu = np.mean([r for _, _, r in ratings])
    for epoch in range(epochs):
        rng.shuffle(ratings)
        for u, i, r in ratings:
            e = r - (mu + bu[u] + bi[i] + P[u] @ Q[i])
            bu[u] += lr * (e - reg * bu[u]); bi[i] += lr * (e - reg * bi[i])
            P[u], Q[i] = P[u] + lr * (e * Q[i] - reg * P[u]), Q[i] + lr * (e * P[u] - reg * Q[i])
    return mu, bu, bi, P, Q

# Synthetic data with a true rank-3 structure
rng = np.random.default_rng(1)
U, I, K = 300, 200, 3
true = rng.normal(size=(U, K)) @ rng.normal(size=(I, K)).T
obs = [(u, i, float(np.clip(3 + true[u, i] + rng.normal(0, .3), 1, 5)))
       for u in range(U) for i in range(I) if rng.random() < 0.05]
split = int(0.8 * len(obs)); train, test = obs[:split], obs[split:]
mu, bu, bi, P, Q = train_mf(list(train), U, I)
rmse = np.sqrt(np.mean([(r - (mu + bu[u] + bi[i] + P[u] @ Q[i])) ** 2 for u, i, r in test]))
base = np.sqrt(np.mean([(r - mu) ** 2 for _, _, r in test]))
print(f"test RMSE: MF={rmse:.3f}  global-mean baseline={base:.3f}")

Training method 2: alternating least squares (ALS)#

The objective is not jointly convex in $\mathbf{P}$ and $\mathbf{Q}$, but it is convex in each with the other fixed. ALS alternates: fix $\mathbf{Q}$ and solve a ridge regression for every user vector in closed form,

then fix $\mathbf{P}$ and solve for every item. Each half-step is embarrassingly parallel, which is why ALS is the standard in distributed systems such as Spark MLlib.

Implicit feedback#

Hu, Koren and Volinsky (2008) adapted ALS to implicit data: treat every user–item pair as a preference $p_{ui} \in \{0, 1\}$ (interacted or not) with a confidence $c_{ui} = 1 + \alpha\,(\text{interaction count})$. All pairs enter the loss, weighted by confidence, and clever algebra keeps ALS efficient despite the dense matrix. Alternatives train with ranking losses such as Bayesian Personalised Ranking (BPR), which learns that observed items should score higher than sampled unobserved ones.

From factorisation to deep recommenders#

• Factorisation machines generalise MF to arbitrary features (user attributes, context) by modelling pairwise feature interactions with latent vectors.
• Two-tower models use neural networks to map user features and item features into the same embedding space — MF with learned encoders. They handle cold start (new items have features) and power large-scale candidate retrieval via approximate nearest-neighbour search.
• Sequential recommenders (e.g. transformer-based) model the order of a user's interactions.