Derivatives: The Best Local Line

An activation function changes both the values moving forward through a network and the slopes moving backward. GELU makes a useful example: its formula is short, but understanding its slope takes the product rule.

The best local line

A derivative is the limit

$f'(a)=\lim_{h\to0}\frac{f(a+h)-f(a)}h.$

Equivalently, $f(a+h)=f(a)+f'(a)h+o(h)$: after subtracting the proposed line, the remaining error divided by $|h|$ tends to zero. This is why the tangent predicts the effect of a small parameter change.

For $f(x)=x^2$, the difference quotient at $a$ is $2a+h$, so $f'(a)=2a$. Near $a=3$, the prediction for $f(3.01)$ is $9+6(0.01)=9.06$. The exact value is $9.0601$; the error is the omitted $h^2$.

Rules that save work

Linearity gives $(af+bg)'=af'+bg'$. The product rule is $(fg)'=f'g+fg'$. To prove it, add and subtract $f(a+h)g(a)$ in the numerator:

Differentiability implies continuity, so taking the limit gives the rule. A product's derivative is generally not the product of the derivatives. The quotient rule, when $g\ne0$, is $(f/g)'=(f'g-fg')/g^2$.

Useful building blocks are $(x^n)'=nx^{n-1}$, $(e^x)'=e^x$, $(\ln x)'=1/x$ for $x>0$, and $(\sin x)'=\cos x$. Differentiating a composition will be the next chamber's main task.

The activation toolbox

Let $s=\sigma(x)=1/(1+e^{-x})$. Differentiating the quotient gives $\sigma'(x)=s(1-s)$. Since $s(1-s)=1/4-(s-1/2)^2$, its maximum is $1/4$ at $x=0$.

ReLU implementations choose a backward value at the kink, often zero. That convention does not turn the corner into a differentiable point. GELU's derivative can exceed one and can be negative: smoothness does not imply monotonicity.

The lab also includes $\arctan x$, a bounded smooth function with derivative $1/(1+x^2)$. Its maximum slope is 1 at zero.

Check a derivative numerically

The central difference is

$D_hf(x)=\frac{f(x+h)-f(x-h)}{2h}.$

For a sufficiently smooth function its truncation error is $O(h^2)$. Forward differences usually have $O(h)$ error. Floating-point subtraction eventually dominates when $h$ becomes very small; compare several step sizes. At a kink, central differences can average two incompatible one-sided slopes. For $|x|$ at zero they always give zero, although the derivative does not exist.

Read beyond

Read the equation in context

The paper defines GELU by weighting an input by a Gaussian cumulative probability. Differentiate this exact formula before considering its common approximations. The derivative we obtain below follows from the product rule and the fact that the derivative of a CDF is its density.

Your turn

Move along each activation and compare its output with its derivative. Find a point where the derivative reaches its maximum, then submit the slope.

Coding problems