Information Theory for ML: Entropy, Cross-Entropy and KL Divergence

In 1948 Claude Shannon asked how to measure information, and founded a field that now sits at the core of machine learning. The loss you minimise when training a classifier or a language model is called cross-entropy for a reason. The regulariser in a VAE is a KL divergence. Decision trees split on information gain. Today we build these concepts from first principles.

Information content#

How surprising is an event? An event with probability 1 carries no information; a rare event carries a lot. Shannon defined the information content (surprisal) of an outcome $x$ as

I(x) = -\log_2 p(x) \quad \text{bits}

(Using natural logs gives nats.) A fair coin flip yields 1 bit. An event with probability 1/1024 yields 10 bits. Information from independent events adds, because probabilities multiply and logs turn products into sums.

Entropy#

Entropy is the expected surprisal — the average uncertainty of a distribution:

H(p) = -\sum_x p(x)\log p(x)

Maximum for the uniform distribution: $H = \log K$ for $K$ outcomes.
Zero for a deterministic distribution.
For a Bernoulli($q$): $H = -q\log q - (1-q)\log(1-q)$, maximised at $q = 0.5$.

Shannon's source coding theorem gives entropy an operational meaning: it is the minimum average number of bits per symbol needed to encode messages from $p$ losslessly. English text has a much lower entropy per character than $\log_2 26 \approx 4.7$ bits, because letters are predictable — which is why text compresses well and why language models can predict it.

Cross-entropy#

Suppose data comes from a true distribution $p$ but we encode it with a code optimised for a model $q$. The average code length is the cross-entropy:

H(p, q) = -\sum_x p(x)\log q(x)

Always $H(p, q) \ge H(p)$: using the wrong model costs extra bits.

Cross-entropy as a loss#

In classification, the "true distribution" for one example is the one-hot label $\mathbf{y}$, and the model outputs $\hat{\mathbf{p}}$:

H(\mathbf{y}, \hat{\mathbf{p}}) = -\sum_k y_k \log\hat{p}_k = -\log\hat{p}_{\text{true class}}

Averaged over the dataset, this is exactly the negative log-likelihood — so minimising cross-entropy is maximum likelihood.

KL divergence#

The Kullback–Leibler divergence measures the extra cost of using $q$ instead of $p$:

D_{\text{KL}}(p \,\|\, q) = \sum_x p(x)\log\frac{p(x)}{q(x)} = H(p, q) - H(p)

Properties:

$D_{\text{KL}}(p\|q) \ge 0$, with equality iff $p = q$ (Gibbs' inequality, from Jensen's inequality).
Not symmetric: $D_{\text{KL}}(p\|q) \ne D_{\text{KL}}(q\|p)$ in general, so it is not a true distance.

Since $H(p)$ is fixed by the data, minimising cross-entropy equals minimising KL divergence from the data distribution to the model.

Forward vs reverse KL#

Forward KL $D_{\text{KL}}(p\|q)$ (used in MLE) heavily penalises $q(x) \approx 0$ where $p(x) > 0$. The model must cover all data modes — mode-covering, producing blurry averages.
Reverse KL $D_{\text{KL}}(q\|p)$ (used in variational inference) penalises $q$ putting mass where $p$ has none — mode-seeking, locking onto one mode.

This distinction explains qualitative behaviour of generative models and variational approximations.

KL between Gaussians#

A closed form used in every VAE:

D_{\text{KL}}\big(\mathcal{N}(\mu, \sigma^2)\,\|\,\mathcal{N}(0, 1)\big) = \frac{1}{2}\left(\mu^2 + \sigma^2 - \ln\sigma^2 - 1\right)

Mutual information#

Mutual information measures how much knowing one variable reduces uncertainty about another:

I(X; Y) = H(X) - H(X \mid Y) = D_{\text{KL}}\big(p(x, y)\,\|\,p(x)p(y)\big)

It is zero iff $X$ and $Y$ are independent and, unlike correlation, captures non-linear dependence. Uses include feature selection, the information gain criterion for decision-tree splits, contrastive representation learning (InfoNCE bounds mutual information), and the information bottleneck theory of deep learning.

Computing it#

python

import numpy as np

def entropy(p):
    p = np.asarray(p, float); p = p[p > 0]
    return -(p * np.log2(p)).sum()

def cross_entropy(p, q):
    p, q = np.asarray(p, float), np.asarray(q, float)
    return -(p * np.log2(q + 1e-12)).sum()

def kl(p, q):
    return cross_entropy(p, q) - entropy(p)

p = [0.7, 0.2, 0.1]
q = [0.5, 0.3, 0.2]
print("H(p) =", round(entropy(p), 3), "bits")
print("H(p,q) =", round(cross_entropy(p, q), 3), " KL(p||q) =", round(kl(p, q), 3), " KL(q||p) =", round(kl(q, p), 3))

# Information gain of a split: parent labels vs children
parent = [0.5, 0.5]
left, right = [0.9, 0.1], [0.2, 0.8]          # each child holds half the data
gain = entropy(parent) - 0.5 * entropy(left) - 0.5 * entropy(right)
print("information gain:", round(gain, 3), "bits")

#information theory#entropy#cross-entropy#kl divergence#mutual information

Written by

Janin A Apurba

B.Sc. in CSE, AUST · Advanced ICT Officer, CNRS-UNHCR. Teaching AI, ML and Deep Learning to the next generation of engineers and researchers.