Gradients: Steepest Ascent in Many Dimensions

A loss surface may depend on millions of parameters. A two-dimensional slice lets us inspect a few directions through that space. The gradient tells us how the loss changes along any chosen direction at the point where we stand.

Hold everything else fixed

For $f(x,y)=x^2+3y^2$, a partial derivative changes one coordinate while holding the other fixed:

$\frac{\partial f}{\partial x}=2x,\qquad \frac{\partial f}{\partial y}=6y,\qquad \nabla f=(2x,6y)^T.$

At $(1,1)$ the gradient is $(2,6)^T$. A small displacement $h=(h_1,h_2)^T$ changes the function by approximately $2h_1+6h_2$. We use column gradients throughout this course.

Move in any direction

For a unit vector $u$, follow the path $x(t)=x+tu$. The chain rule gives the directional derivative

$D_uf(x)=\left.\frac{d}{dt}f(x+tu)\right|_{t=0}=\nabla f(x)^Tu.$

For $u=(3/5,4/5)$ and gradient $(2,6)$, the slope is $6/5+24/5=6$. A direction must be normalised when comparing slopes per unit distance; doubling an unnormalised vector doubles the parameterised rate of change.

Partial derivatives alone do not always imply differentiability. The dot-product formula assumes a genuine local linear approximation. Continuous partial derivatives in a neighbourhood are a useful sufficient condition.

Why the gradient points uphill

Cauchy–Schwarz gives $\nabla f^Tu\le\|\nabla f\|\|u\|=\|\nabla f\|$. When the gradient is nonzero, equality occurs at $u=\nabla f/\|\nabla f\|$. Negate it for steepest descent.

This statement uses Euclidean distance. Rescaling parameters changes what counts as a unit step and can change the optimisation path. Preconditioners exploit this by changing how directions are scaled.

At a stationary point, $\nabla f=0$, every first-order directional derivative is zero. Curvature, which we study later, is needed to distinguish a minimum from a saddle.

Gradients meet contours at right angles

Suppose a differentiable path $c(t)$ stays on a level set $f(c(t))=k$. Differentiating gives $\nabla f(c(t))^Tc'(t)=0$. Thus a nonzero gradient is perpendicular to the tangent of a smooth contour.

For $x^2+y^2=k$, the gradient $(2x,2y)$ points radially outward. A tangent $(-y,x)$ has zero dot product with it.

Taking a finite step

Gradient descent uses $x_{t+1}=x_t-\eta\nabla f(x_t)$. Its first-order change is $-\eta\|\nabla f\|^2$, negative at a nonstationary point. The neglected error means the step must still be small enough.

For $f(x)=x^2$, the update is $x_{t+1}=(1-2\eta)x_t$. It converges to zero for $0<\eta<1$, oscillates without shrinking at $\eta=1$, and diverges for larger positive steps. A direction can be downhill locally while a long step raises the loss.

Read beyond

Read the equation in context

The paper plots losses along selected directions and studies how normalising those directions changes the view. A flat-looking slice need not imply flatness in every direction. The plotted surface depends on the chosen plane and the scale of its axes.

Your turn

Choose a point, then aim your direction arrow down the loss surface. Compare the directional derivative with the best possible descent slope, and find a direction tangent to a contour.

Coding problems