The Multivariate Gaussian

Diffusion models gradually add Gaussian noise to data. A useful fact makes the whole forward chain easy to sample: independent Gaussian contributions remain Gaussian, with covariances that add after scaling.

A covariance matrix is a geometry

For a vector X with mean μ, define $\Sigma=\mathbb E[(X-\mu)(X-\mu)^T]$. Its diagonal entries are variances; its off-diagonal entries are covariances. It is symmetric and positive semidefinite because $v^T\Sigma v=\operatorname{Var}(v^TX)\ge0$.

A nonsingular d-dimensional normal has density

$p(x)=\frac{\exp[-\tfrac12(x-\mu)^T\Sigma^{-1}(x-\mu)]}{(2\pi)^{d/2}\sqrt{\det\Sigma}}.$

Positive definiteness is needed for this ordinary full-dimensional density. A singular covariance can still define a Gaussian supported on a lower-dimensional plane, but the inverse-and-determinant formula above does not apply.

Distance measured in standard deviations

The squared Mahalanobis distance is $(x-\mu)^T\Sigma^{-1}(x-\mu)$. When $\Sigma=I$, it becomes squared Euclidean distance. For diagonal covariance, it divides each squared coordinate deviation by its variance.

Covariance eigenvectors give ellipse directions, and square roots of eigenvalues give their relative axis lengths. The determinant controls the volume scale. Correlation rotates and stretches the contours; it cannot be chosen independently of a valid positive semidefinite covariance.

For a bivariate Gaussian, zero covariance does imply independence. This is a special property of joint Gaussian distributions, not a general consequence of uncorrelatedness.

Affine transformations and sampling

If X is Gaussian, then $Y=AX+b$ is Gaussian with mean $A\mu+b$ and covariance $A\Sigma A^T$. For sampling, find a Cholesky factor $\Sigma=LL^T$, draw independent standard normal coordinates ε, and set $X=\mu+L\epsilon$.

For $\Sigma=[[4,2],[2,2]]$, a factor is $L=[[2,0],[1,1]]$. It produces $X_1=2\epsilon_1$ and $X_2=\epsilon_1+\epsilon_2$; the shared first noise term creates covariance 2.

For a bivariate Gaussian with correlation ρ and positive marginal standard deviations, conditioning on $X_1=a$ gives

$\mathbb E[X_2\mid X_1=a]=\mu_2+\rho\frac{\sigma_2}{\sigma_1}(a-\mu_1),\qquad \operatorname{Var}(X_2\mid X_1=a)=\sigma_2^2(1-\rho^2).$

The mean shifts according to the observed coordinate and the uncertainty contracts. This formula assumes a nonsingular joint Gaussian; degenerate cases need separate treatment.

Collapse a forward diffusion chain

Write $x_t=\sqrt{\alpha_t}x_{t-1}+\sqrt{1-\alpha_t}\epsilon_t$ with independent standard Gaussian noise. Substituting earlier steps repeatedly gives a clean-input coefficient $\sqrt{\bar\alpha_t}$, where $\bar\alpha_t=\prod_{s=1}^t\alpha_s$.

Conditional noise covariance follows $v_t=\alpha_t v_{t-1}+1-\alpha_t$ with $v_0=0$. Induction gives $v_t=1-\bar\alpha_t$. Thus a single independent Gaussian draw samples any forward time directly. This algebra is why the opening density holds.

Read beyond

Read the equation in context

DDPM gives a closed-form distribution for a noisy state conditional on a clean input. The input need not itself be Gaussian. Conditioning on x0 makes the mean fixed; if x0 is drawn from a complex data distribution, the marginal distribution of xt can remain a mixture.

Your turn

Fit the mean, spreads and correlation of three point clouds. Then inspect conditional slices and watch the forward diffusion process reduce the signal while adding noise.

Coding problems