Expectation, Variance and Initialisation

Before a network learns anything, its initial weights decide whether signals stay measurable through its layers. A variance calculation supplies a principled scale, including the factor of two used for ReLU networks.

Expectation adds, even with dependence

For discrete X, $\mathbb E[f(X)]=\sum_x f(x)P(X=x)$; for a density, replace the sum with an integral. Linearity gives $\mathbb E[aX+bY]=a\mathbb EX+b\mathbb EY$ whenever these expectations exist. Independence is not required.

Variance centres the variable: $\operatorname{Var}(X)=\mathbb E[(X-\mu)^2]=\mathbb E[X^2]-\mu^2$. It obeys $\operatorname{Var}(aX+b)=a^2\operatorname{Var}(X)$.

Covariance records cross terms

Define $\operatorname{Cov}(X,Y)=\mathbb E[(X-\mu_X)(Y-\mu_Y)]$. Expanding a square gives

$\operatorname{Var}\Big(\sum_iX_i\Big)=\sum_i\operatorname{Var}(X_i)+2\sum_{i<j}\operatorname{Cov}(X_i,X_j).$

Independent variables with finite variance have zero covariance. The converse is false: for symmetric X, X and X² can be uncorrelated while clearly dependent. Correlation divides covariance by the two standard deviations, when both are positive, and lies between -1 and 1 by Cauchy–Schwarz.

For independent X and Y, $\mathbb E[XY]=\mathbb EX\mathbb EY$ and

$\operatorname{Var}(XY)=\operatorname{Var}(X)\operatorname{Var}(Y)+\operatorname{Var}(X)\mu_Y^2+\operatorname{Var}(Y)\mu_X^2.$

Independence matters here because we factor expectations of products.

Bounds that need little information

If X≥0, then $X\ge a\mathbf1_{X\ge a}$ for a>0. Taking expectations gives Markov’s inequality, $P(X\ge a)\le\mathbb EX/a$.

Apply it to $(X-\mu)^2$ to obtain Chebyshev’s inequality:

$P(|X-\mu|\ge t)\le\frac{\operatorname{Var}(X)}{t^2}.$

These bounds can be loose, but they do not require Gaussian data. A bound above one conveys no additional information; probabilities already cannot exceed one.

Derive an initialisation scale

Let $z=\sum_{i=1}^n w_i a_i$, with independent zero-mean weights, independent of the inputs. Cross terms vanish under these assumptions, giving $\operatorname{Var}(z)=n\operatorname{Var}(w)\mathbb E[a^2]$ when the input second moments agree.

For small tanh inputs, $\tanh z\approx z$, so a fan-in scale $\operatorname{Var}(w)=1/n$ approximately preserves forward variance. Xavier initialisation balances forward and backward requirements with $2/(n_{in}+n_{out})$; those requirements coincide when widths agree.

For symmetric zero-mean z, ReLU has $\mathbb E[\operatorname{ReLU}(z)^2]=\tfrac12\mathbb E[z^2]$. It has a positive mean, so its variance is smaller than this second moment. Compensating the second-moment loss in the next pre-activation gives Kaiming initialisation: $\operatorname{Var}(w)=2/n$.

For uniform weights on [-a,a], variance is $a^2/3$, giving $a=\sqrt{6/n}$ for ReLU. These arguments describe an initial approximation, not a promise that every trained layer or correlated dataset preserves variance.

Read beyond

Read the equation in context

He and colleagues derive a weight scale by tracking forward signal second moments and backward gradients under independence and symmetry assumptions. ReLU halves the second moment of a symmetric pre-activation; it does not generally halve the centred variance, because its output has a positive mean.

Your turn

Adjust the weight scale and watch the signal through thirty layers. Compare a small-signal tanh approximation with the ReLU second-moment calculation, then check both against the stated assumptions.

Coding problems