Taylor Series, the Hessian and Curvature

Two weights with the same magnitude can have very different effects when removed. A quadratic approximation of the loss estimates the damage using curvature. That idea connects Taylor expansions with neural-network pruning.

Approximate more than the slope

A Taylor polynomial matches derivatives at a point:

$f(a+h)=f(a)+f'(a)h+\frac12f''(a)h^2+\cdots+\frac{f^{(n)}(a)}{n!}h^n+R_n.$

If $|f^{(n+1)}|\le M$ throughout the interval and the usual Taylor theorem hypotheses hold, then $|R_n|\le M|h|^{n+1}/(n+1)!$. This gives an error guarantee for a finite approximation. Smoothness alone does not guarantee that an infinite Taylor series equals the function.

For $e^h$ around zero, the first three terms are $1+h+h^2/2$. At $h=0.1$, they give 1.105. A cubic remainder bound is $e^{0.1}(0.1)^3/6<0.000185$.

The Hessian measures directional curvature

For a scalar function with continuous second partial derivatives, the Hessian is symmetric:

$H_{ij}=\frac{\partial^2f}{\partial x_i\partial x_j},\qquad f(x+h)\approx f(x)+g^Th+\frac12h^THh.$

For $f(x,y)=x^2+xy+2y^2$, $H=[[2,1],[1,4]]$. The mixed partial records how the slope in one coordinate changes as another moves. Along a unit direction $u$, the second derivative is $u^THu$.

Classify a stationary point

At $g=0$, Hessian eigenvalues reveal the quadratic behaviour:

The zero Hessian at zero cannot distinguish $x^4$ from $-x^4$. A positive semidefinite Hessian at one point is insufficient to prove a local minimum.

Newton follows the quadratic model

Set the gradient of the quadratic approximation to zero: $g+Hh=0$. Solve this linear system for $h$ and update $x\leftarrow x+h$. Writing $h=-H^{-1}g$ describes the formula; implementations usually solve the system without forming an inverse.

For $f(x)=(x-3)^2$, Newton reaches 3 from any initial point in one step. For a nonquadratic function, full steps can overshoot. An indefinite Hessian can even send Newton toward a maximum or saddle. Damping, line search, or replacing $H$ by a positive definite approximation can help.

A badly conditioned positive Hessian makes gradient descent zigzag: one step size must accommodate both steep and shallow directions. Newton rescales by curvature, but computing and storing a dense Hessian is expensive in a large model.

Predict the cost of deleting a weight

Deleting $w_i$ means setting $h_i=-w_i$ and other displacements to zero. The Taylor prediction is $-g_iw_i+\tfrac12H_{ii}w_i^2$. Near a stationary point the first term is small, leaving the saliency in the opening equation. A small weight on a very curved direction can matter more than a larger weight on a flat direction.

Read beyond

Read the equation in context

Optimal Brain Damage estimates deletion cost using second derivatives. The displayed saliency assumes training is near a stationary point and uses a diagonal quadratic approximation. Correlations between simultaneous deletions and higher-order effects can make the approximation inaccurate.

Your turn

Explore Himmelblau’s function. Locate stationary points and classify them from the two Hessian eigenvalues. Positive, negative and mixed curvature give different local geometry.

Coding problems