Gradient Descent: Rolling Downhill

The hiker in the fog

Imagine you're on a mountainside in thick fog and you want to reach the valley. You can't see more than a metre, but you can feel the ground tilt under your boots. A reasonable strategy:

1. Feel which direction slopes down most steeply.
2. Take a step that way.
3. Repeat until the ground feels flat.

That is the whole algorithm. In machine learning, the mountain is the loss landscape from Chamber 1, your position is the parameter vector $\theta$, and your altitude is the loss $\mathcal{L}(\theta)$. All we need is a precise way to "feel the slope".

Slopes, derivatives and gradients

The derivative measures how much the loss changes when you nudge a parameter a tiny amount:

Its sign tells you which way is uphill. If the derivative is positive, increasing $\theta$ increases the loss, so we should decrease $\theta$. If it's negative, we should increase it. Either way, the rule is the same: move against the derivative.

Take the simplest possible loss, $\mathcal{L}(\theta) = \theta^2$. Its derivative is $\mathcal{L}'(\theta) = 2\theta$. At $\theta = 3$ the slope is $6$: steeply uphill to the right, so we step left, towards the minimum at $0$.

Real models have many parameters, so we collect one derivative per parameter into a vector, the gradient:

The update rule

Now the algorithm fits on one line:

• $\theta_t$ is where we are after $t$ steps.
• The learning rate $\eta$ (Greek eta) sets the step size.
• The minus sign sends us downhill.

We repeat until the gradient is close to zero. At the bottom of a valley the ground is flat, so the steps shrink to nothing on their own.

A tiny proof: when does it converge?

Let's do something researchers do all the time: analyse an algorithm on a problem simple enough to solve exactly. For $\mathcal{L}(\theta) = \theta^2$ the update becomes

Everything depends on the factor $1 - 2\eta$. If its size is below $1$, then $\theta_t$ shrinks to $0$ and we converge. If it's above $1$, $\theta_t$ explodes.

So on this loss, gradient descent converges exactly when $0 < \eta < 1$. You just proved a convergence theorem.

Play with it

Things to try:

1. On the bowl, compare $\eta = 0.01$ (crawls), $0.3$ (smooth), $0.9$ (zig-zags), $1$ (bounces forever) and $1.05$ (explodes). Check each against the table above.
2. Find the single learning rate that reaches the bottom in exactly one step. (There's a hidden badge in it.)
3. Switch to the bumpy landscape. From $\theta_0 = 3.5$, small learning rates get trapped in the shallow valley: a local minimum. Large ones bounce chaotically. Plain gradient descent cannot escape… so read on and turn on momentum.

Momentum: rolling, not walking

Plain gradient descent is a hiker who stops dead after every step. Momentum turns the hiker into a heavy ball that remembers how fast it was moving:

The velocity $v$ accumulates past gradients, and $\beta$ (typically $0.9$) sets how much of it survives each step. Momentum smooths out zig-zags in narrow ravines and can carry the ball over small bumps. That's exactly what you need to escape the trap in the lab.

Modern optimisers build on this idea. Adam combines momentum with a separate, adaptive step size for every parameter, and it's the default choice across much of deep learning.

Stochastic gradient descent

There's a practical catch. The loss averages over the entire dataset, so computing its exact gradient means touching every example, perhaps millions of them, for every single step.

Stochastic gradient descent (SGD) estimates the gradient from a small random mini-batch $B$ instead:

• The estimate is noisy but unbiased: on average it points the right way.
• Each step is far cheaper, so you can take many more of them.
• The noise can even help, jostling the parameters out of sharp or flat regions. It's also why real training curves look jagged.

Some vocabulary you'll see in every paper: the batch size is $|B|$, one iteration is one update, and one epoch is one full pass through the training data.

Gradient descent for our line

Back to the coffee-and-code data from Chamber 1. Differentiating the mean squared error with respect to each parameter gives:

And now the machine can do what you did by hand in the lab:

import numpy as np

x = np.array([0.5, 1.5, 2.0, 3.0, 3.5, 4.5, 5.5, 6.5])
y = np.array([4.4, 5.2, 9.6, 10.5, 13.7, 13.8, 19.2, 21.1])

w, b = 1.0, 9.0        # the same poor start as the Chamber 1 lab
lr = 0.02

for step in range(2000):
    err = w * x + b - y
    grad_w = 2 * np.mean(err * x)
    grad_b = 2 * np.mean(err)
    w -= lr * grad_w
    b -= lr * grad_b

print(round(w, 3), round(b, 3))   # 2.874 2.486, the least-squares line

Two research-flavoured observations. First, after 200 steps it's still at $(2.64, 3.51)$: this little problem has a condition number of about 68, so one direction converges slowly. Second, try lr = 0.07 and it explodes. The largest curvature here is about $31.7$, so the safe limit is $2/31.7 \approx 0.063$. The theory from the table above predicts real code.

Watch: gradient descent, visualised