The Chain Rule: Deep Compositions

A deep network is a composition. Each layer changes how much an earlier input perturbation matters. Residual networks add a path that passes the input directly to the output of a block; differentiating that path explains part of their behaviour.

Multiply local sensitivities

If $y=f(u)$ and $u=g(x)$, then

$\frac{dy}{dx}=f'(g(x))g'(x).$

For $y=(3x+1)^2$, the outer slope is $2(3x+1)$ and the inner slope is 3, giving $6(3x+1)$. Evaluate each derivative at the input its function actually receives.

A proof that handles zero slopes

Differentiability of $f$ at $u=g(a)$ means

$f(u+k)=f(u)+f'(u)k+r(k)k,\qquad r(k)\to0.$

Set $k=g(a+h)-g(a)$. Differentiability of $g$ makes $k\to0$ and $k/h\to g'(a)$. Divide by $h$ to get a limit of $[f'(u)+r(k)]k/h=f'(u)g'(a)$. When $k=0$, define the remainder contribution as zero. This avoids dividing by $k$, which could be zero even for nonzero $h$.

Inverses and logarithms

If $f$ has a differentiable local inverse and $f'(x)\ne0$, differentiating $f^{-1}(f(x))=x$ gives $(f^{-1})'(f(x))=1/f'(x)$. Thus the derivative of $\ln x$, the inverse of $e^x$, is $1/x$ for $x>0$.

For a positive differentiable function $p_\theta$, the log-derivative identity is

$\nabla_\theta p_\theta=p_\theta\nabla_\theta\log p_\theta.$

For a finite fixed outcome space and rewards $R(x)$ independent of $\theta$, this yields

$\nabla_\theta\sum_xp_\theta(x)R(x)=\mathbb E_{p_\theta}[R(X)\nabla_\theta\log p_\theta(X)].$

This is the score-function route to policy gradients. Continuous or parameter-dependent supports require care about differentiating under the integral and boundary terms.

Twenty multiplications later

For $h_k=\sigma(w_kh_{k-1}+b_k)$,

$\frac{dh_L}{dh_0}=\prod_{k=1}^L w_k\sigma'(w_kh_{k-1}+b_k).$

Twenty factors of $1/2$ give about one millionth; twenty factors of 2 give over a million. Sigmoid's derivative is at most $1/4$, but the weights also matter. It is incorrect to conclude that every sigmoid network's gradient must shrink solely from its activation bound.

A residual scalar layer $h_{k+1}=h_k+F_k(h_k)$ contributes $1+F_k'(h_k)$. Small residual derivatives can keep products near one. If $F_k'=-1$, however, the total is zero. Residual connections help preserve paths for information; they do not abolish the chain rule.

For branched computations, multiply along each path and add across paths. In $y=x^2+x$, the two contributions are $2x$ and 1.

Read beyond

Read the equation in context

The residual block adds an identity shortcut to a learned transformation. In the scalar analogue, its derivative is $1+F'(x)$. This gives a direct gradient contribution, but does not guarantee a nonzero total: contributions can cancel, and later activations still matter.

Your turn

Build twenty identical scalar layers. Track both their activations and the product of their local derivatives, then tune the gain to preserve a useful gradient.

Coding problems