Entropy, Cross-Entropy and KL Divergence

Cross-entropy measures the surprise a model assigns to observed outcomes. Variational inference goes further: it bounds a difficult log-evidence calculation using an expectation and a KL divergence. Both begin with the same logarithm of probability.

Surprise and average surprise

An event of probability p carries surprise $-\log p$. Independent event probabilities multiply, so their surprises add. Base-two logs measure bits; natural logs measure nats.

For a discrete distribution P,

$H(P)=-\sum_xp(x)\log p(x).$

Use the limit convention $0\log0=0$. A certain outcome has zero entropy. A uniform distribution on K outcomes has entropy log K, the maximum on that support. A fair coin has one bit; four equally likely outcomes have two bits.

Cross-entropy and KL

If data follows P but a model predicts Q, its expected surprise is $H(P,Q)=-\sum_xp(x)\log q(x)$. Subtracting entropy gives

$D_{KL}(P\|Q)=\sum_xp(x)\log\frac{p(x)}{q(x)}=H(P,Q)-H(P).$

When p(x)>0 but q(x)=0, both KL and cross-entropy are infinite. Zero p terms contribute zero. KL is generally asymmetric and is not a metric: it does not obey all distance axioms.

To prove nonnegativity where the ratios are defined, use $\log u\le u-1$ with $u=q/p$. Then $-\sum p\log(q/p)\ge\sum(p-q)$ over the support of P, which is nonnegative. Equality requires the distributions to agree.

Minimising cross-entropy over Q therefore minimises KL from the fixed data distribution. Entropy of P is constant during that optimisation.

Perplexity and shared information

Perplexity is the exponential of mean negative log probability: $\exp(H)$ for nats or $2^H$ for bits. A uniform model over K symbols has perplexity K. Compare language-model perplexities only when tokenisation, dataset and averaging conventions match.

Mutual information is $I(X;Y)=D_{KL}(P_{XY}\|P_XP_Y)$: the divergence between the joint distribution and the distribution that would hold under independence. It is nonnegative and vanishes exactly at independence. It can detect dependence even when covariance is zero.

Differential entropy for continuous variables uses a density inside an integral. Unlike discrete entropy, it can be negative and depends on coordinate scale. Do not transfer every discrete intuition to continuous entropy.

Derive the evidence lower bound

Introduce a latent variable z and an approximate posterior q with appropriate support. Using Jensen’s inequality for concave log,

$\log p_\theta(x)=\log\mathbb E_q\left[\frac{p_\theta(x,z)}{q(z)}\right]\ge\mathbb E_q[\log p_\theta(x,z)-\log q(z)].$

Split the joint into likelihood times prior to recover the opening ELBO. Equivalently,

$\log p_\theta(x)=\mathcal L(q)+D_{KL}(q(z)\|p_\theta(z\mid x)).$

The rightmost divergence is the gap. Optimising the bound changes both the generative model and its approximate inference distribution; it is not generally the same as evaluating an exact log-likelihood.

Read beyond

Read the equation in context

The paper optimises a lower bound on log evidence with an approximate posterior. The first term rewards explaining observations, and the second measures divergence from the prior. The bound equals the log evidence only when the approximate posterior matches the model’s exact posterior.

Your turn

Tune a binary distribution to three entropy targets. Then compare KL in both directions for two distinct distributions.

Coding problems