PCA finds the directions of maximum variance, ignoring labels. But if our goal is classification, we want directions that separate classes. In 1936 Ronald Fisher, studying the famous Iris flowers, derived the linear projection that best discriminates between groups. Linear Discriminant Analysis (LDA) is both a classifier and a supervised dimensionality-reduction method — fast, closed-form and surprisingly competitive.
(Not to be confused with Latent Dirichlet Allocation, the topic model, which shares the acronym.)
Fisher's criterion#
For two classes, project data onto a direction $\mathbf{w}$. We want:
- the projected class means to be far apart;
- each class's projected points to be tightly clustered.
Define the between-class scatter $\mathbf{S}_B = (\boldsymbol{\mu}_1 - \boldsymbol{\mu}_2)(\boldsymbol{\mu}_1 - \boldsymbol{\mu}_2)^\top$ and the within-class scatter $\mathbf{S}_W = \sum_{c}\sum_{i \in c}(\mathbf{x}_i - \boldsymbol{\mu}_c)(\mathbf{x}_i - \boldsymbol{\mu}_c)^\top$. Fisher's criterion is the ratio
Maximising it (a generalised Rayleigh quotient) gives the closed form
Notice the role of $\mathbf{S}_W^{-1}$: the best direction is not simply the line joining the means. It is tilted to account for the shape of the classes — directions in which the classes are spread out are down-weighted.
Multiclass LDA#
For $K$ classes, define $\mathbf{S}_B = \sum_c n_c(\boldsymbol{\mu}_c - \boldsymbol{\mu})(\boldsymbol{\mu}_c - \boldsymbol{\mu})^\top$. The optimal projection directions are the top eigenvectors of $\mathbf{S}_W^{-1}\mathbf{S}_B$. Because $\mathbf{S}_B$ has rank at most $K - 1$, LDA yields at most $K - 1$ discriminant components. For 10 digit classes, LDA gives at most 9 dimensions — often an excellent compressed representation for classification.
The generative (probabilistic) view#
LDA can also be derived as a generative classifier. Assume each class is Gaussian with its own mean but a shared covariance $\boldsymbol{\Sigma}$:
Applying Bayes' theorem, the log-posterior ratio between two classes is
The quadratic terms cancel because the covariance is shared, so the decision boundary is linear — and its normal vector is exactly Fisher's direction. Two derivations, one answer.
If each class has its own covariance, the quadratic terms no longer cancel and we get Quadratic Discriminant Analysis (QDA) with curved boundaries — more flexible, but with many more parameters to estimate ($K \cdot d(d+1)/2$ covariance entries).
LDA vs PCA#
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import load_wine
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis, QuadraticDiscriminantAnalysis
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import cross_val_score
from sklearn.linear_model import LogisticRegression
X, y = load_wine(return_X_y=True)
Xs = StandardScaler().fit_transform(X)
Z_pca = PCA(2).fit_transform(Xs)
Z_lda = LinearDiscriminantAnalysis(n_components=2).fit_transform(Xs, y)
fig, axes = plt.subplots(1, 2, figsize=(11, 4))
for ax, Z, t in [(axes[0], Z_pca, "PCA (unsupervised)"), (axes[1], Z_lda, "LDA (supervised)")]:
ax.scatter(Z[:, 0], Z[:, 1], c=y, cmap="viridis", s=18); ax.set_title(t)
plt.tight_layout(); plt.show()
for name, m in [("LDA", LinearDiscriminantAnalysis()), ("QDA", QuadraticDiscriminantAnalysis()),
("Logistic", LogisticRegression(max_iter=2000))]:
print(name, round(cross_val_score(m, Xs, y, cv=5).mean(), 3))| PCA | LDA | |
|---|---|---|
| Uses labels | No | Yes |
| Objective | Maximise variance | Maximise class separation |
| Max components | $\min(n, d)$ | $K - 1$ |
| Assumptions | None (linear) | Gaussian classes, shared covariance (for optimality) |
Regularisation and high dimensions#
When $d$ is large relative to $n$, $\mathbf{S}_W$ is singular or poorly estimated. Shrinkage LDA replaces it with $(1 - \alpha)\hat{\mathbf{S}}_W + \alpha\,\nu\mathbf{I}$, with $\alpha$ chosen analytically (Ledoit–Wolf). In scikit-learn: LinearDiscriminantAnalysis(solver="lsqr", shrinkage="auto"). Shrinkage LDA is a standard, strong method in brain–computer interfaces, where EEG data is high-dimensional and noisy with few trials.
LDA vs logistic regression#
Both produce linear boundaries. LDA is generative and assumes Gaussian classes with shared covariance; logistic regression is discriminative and assumes only the log-odds are linear.
- When LDA's assumptions hold, it is more statistically efficient (needs less data).
- When they fail (heavy tails, outliers, categorical features), logistic regression is more robust.