Backpropagation: The Chain Rule at Scale

The credit assignment problem

A network with a million weights makes a bad prediction. Which weights are to blame, and how much should each one change? Gradient descent needs $\partial L / \partial w$ for every weight, at every step.

The naive approach is to nudge one weight, rerun the network, see how the loss changes, and repeat for the next weight. That costs one forward pass per weight: a million passes for a single update. Hopeless.

Backpropagation computes every one of those derivatives in a single backward sweep, at a cost comparable to one extra forward pass. It is the reason deep learning is possible at all, and it rests entirely on one idea from calculus.

The chain rule

If $y$ depends on $x$, and $L$ depends on $y$, then

Rates of change multiply along a chain. The mathematician George Simmons put it memorably: if a car travels twice as fast as a bicycle, and the bicycle four times as fast as a walking man, then the car travels $2 \times 4 = 8$ times as fast as the man.

A quick example: $L = (2x - 1)^2$ at $x = 2$. Let $y = 2x - 1 = 3$. Then $\frac{dL}{dy} = 2y = 6$ and $\frac{dy}{dx} = 2$, so $\frac{dL}{dx} = 6 \times 2 = 12$.

Computational graphs

To apply the chain rule to a whole network, break the computation into a graph of primitive operations. Then run it twice:

1. 1

   Forward pass

   Compute every node's value from left to right, and remember them: the backward pass needs them.

2. 2

   Backward pass

   Start at the loss with $\frac{\partial L}{\partial L} = 1$ and walk right to left. At every node, apply the chain rule locally:

   $$\underbrace{\frac{\partial L}{\partial\, \text{input}}}_{\text{downstream}} = \underbrace{\frac{\partial L}{\partial\, \text{output}}}_{\text{upstream}} \times \underbrace{\frac{\partial\, \text{output}}{\partial\, \text{input}}}_{\text{local}}$$

Each node only ever needs its own local derivative. A handful of patterns covers most of what you'll meet:

Backprop by hand

Enough reading: it's your turn to be the backward pass. The lab gives you one neuron, $L = \big(\mathrm{ReLU}(wx + b) - y\big)^2$, with real numbers. Compute every gradient yourself. Get them all right on the first try and something shiny happens.

Backprop for a whole network

The same bookkeeping scales to entire networks, where it's written with vectors and matrices. Take a two-layer network on one example:

That elementwise multiply by $\phi'$ explains the vanishing gradient problem you were warned about. Each sigmoid layer multiplies the signal by at most $0.25$, so after ten layers it can shrink by a factor of $0.25^{10} \approx 10^{-6}$. That's one reason ReLU (slope 1), careful initialisation, normalisation layers and residual connections became standard.

Trust, but verify: gradient checking

Hand-derived gradients are easy to get slightly wrong. Worse, a buggy gradient often still reduces the loss, just badly. Careful researchers check their gradients numerically with the centred difference:

import numpy as np

def grad_check(f, grad_f, theta, eps=1e-5):
    """Relative error between an analytic gradient and centred finite differences."""
    analytic = grad_f(theta)
    numeric = np.zeros_like(theta)
    for i in range(theta.size):
        step = np.zeros_like(theta)
        step.flat[i] = eps
        numeric.flat[i] = (f(theta + step) - f(theta - step)) / (2 * eps)
    diff = np.linalg.norm(analytic - numeric)
    return diff / (np.linalg.norm(analytic) + np.linalg.norm(numeric))

# Around 1e-7 or smaller: great. Around 1e-2 or larger: almost certainly a bug.

Autograd: let the machine do it

In practice nobody writes backward passes by hand any more. Frameworks such as PyTorch and JAX record the computational graph as your code runs and apply the chain rule for you. Here is the lab's neuron in PyTorch. It prints exactly the gradients you just computed:

import torch

x, y = torch.tensor(3.0), torch.tensor(4.0)
w = torch.tensor(2.0, requires_grad=True)
b = torch.tensor(-1.0, requires_grad=True)

a = torch.relu(w * x + b)   # forward pass: a = 5
loss = (a - y) ** 2         # loss = 1

loss.backward()             # backward pass
print(w.grad, b.grad)       # tensor(6.) tensor(2.)

Understanding what loss.backward() does, and when it can silently go wrong, is what separates someone who uses deep learning from someone who can research it.

Watch: backpropagation, intuitively

Want to go deeper? Build a tiny autograd engine yourself, line by line, with Andrej Karpathy: