Eigenvalues and Eigenvectors: The Natural Axes of a Transformation

Most vectors change direction when a matrix acts on them. But some special vectors only get stretched or shrunk — they keep their direction. These are eigenvectors, and the stretching factors are eigenvalues. They reveal the "natural axes" of a transformation and appear throughout machine learning: PCA, spectral clustering, PageRank, the convergence of gradient descent and the stability of recurrent networks.

Definition#

For a square matrix $\mathbf{A} \in \mathbb{R}^{n \times n}$, a non-zero vector $\mathbf{v}$ is an eigenvector with eigenvalue $\lambda$ if

Rearranging, $(\mathbf{A} - \lambda\mathbf{I})\mathbf{v} = \mathbf{0}$ must have a non-zero solution, so $\mathbf{A} - \lambda\mathbf{I}$ must be singular:

This characteristic equation is a polynomial of degree $n$ in $\lambda$, so there are $n$ eigenvalues (counted with multiplicity, possibly complex).

Two useful facts: the trace equals the sum of eigenvalues, and the determinant equals their product.

Diagonalisation#

If $\mathbf{A}$ has $n$ linearly independent eigenvectors, collect them as columns of $\mathbf{V}$ and the eigenvalues in diagonal $\boldsymbol{\Lambda}$:

Read right to left: change to the eigenvector coordinate system, scale each axis independently, change back. Powers become trivial:

So the long-term behaviour of repeatedly applying $\mathbf{A}$ is governed by the largest eigenvalue in magnitude. If $|\lambda_{\max}| > 1$, repeated application explodes; if all $|\lambda| < 1$, it decays to zero.

The spectral theorem#

For symmetric real matrices ($\mathbf{A} = \mathbf{A}^\top$), things are especially nice:

1. All eigenvalues are real.
2. Eigenvectors for distinct eigenvalues are orthogonal.
3. $\mathbf{A}$ can be diagonalised by an orthogonal matrix:

Covariance matrices, kernel (Gram) matrices, graph Laplacians and Hessians are all symmetric, so the spectral theorem is everywhere in ML. A symmetric matrix is positive definite iff all eigenvalues are positive; positive semi-definite iff all are non-negative.

Applications in ML#

Principal Component Analysis#

The eigenvectors of the data covariance matrix $\mathbf{\Sigma}$ are the principal directions; the eigenvalues are the variances along them. Keeping the top $k$ eigenvectors gives the best $k$-dimensional linear approximation of the data.

Optimisation and curvature#

Near a minimum, a loss looks like a quadratic with Hessian $\mathbf{H}$. The eigenvalues of $\mathbf{H}$ are curvatures along principal directions. Gradient descent converges only if the learning rate satisfies $\eta < 2/\lambda_{\max}(\mathbf{H})$, and it is slow when the condition number $\kappa = \lambda_{\max}/\lambda_{\min}$ is large — long narrow valleys. Momentum, Adam and normalisation layers all help with ill-conditioning.

PageRank and Markov chains#

The stationary distribution of a Markov chain is the eigenvector of the transition matrix with eigenvalue 1. Google's PageRank is exactly this for the web graph.

Spectral clustering#

Eigenvectors of the graph Laplacian with the smallest eigenvalues reveal cluster structure, even for non-convex clusters.

Computing eigenvectors: power iteration#

The simplest algorithm repeatedly multiplies a random vector by $\mathbf{A}$ and normalises. It converges to the dominant eigenvector at a rate governed by $|\lambda_2/\lambda_1|$.

import numpy as np

def power_iteration(A, iters=100):
    v = np.random.default_rng(0).normal(size=A.shape[0])
    for _ in range(iters):
        v = A @ v
        v /= np.linalg.norm(v)
    return v @ A @ v, v             # Rayleigh quotient gives the eigenvalue

A = np.array([[2., 1.], [1., 2.]])
lam, v = power_iteration(A)
print(lam, v)                        # ~3.0, ~[0.707, 0.707]

w, Q = np.linalg.eigh(A)             # eigh: for symmetric matrices (sorted ascending)
print(w, Q)