Limits: Getting Arbitrarily Close

A learning rate can shrink to zero while its total travel remains unbounded. That distinction lets a noisy optimiser keep correcting its estimate without giving every new sample the same influence. To understand it, we need two different limits: the limit of a sequence and the limit of its partial sums.

A limit describes nearby values

The statement $\lim_{x\to a}f(x)=L$ concerns inputs near $a$, with $x\ne a$. The value at $a$ may be different or undefined. For example,

Cancelling is valid away from the missing point, which is exactly where the limit looks. A two-sided limit exists only when the left and right limits agree.

Make “close” precise

For every desired output error $\varepsilon>0$, there must be an input radius $\delta>0$ such that

The order matters: the challenger chooses $\varepsilon$, then you choose $\delta$. Your choice must work for all eligible $x$, not just the points drawn on a screen.

For $f(x)=3x+1$ at $a=2$, the target is $L=7$. Since $|f(x)-7|=3|x-2|$, choose $\delta=\varepsilon/3$.

For $f(x)=x^2$ at $a=2$, factor the error: $|x^2-4|=|x-2||x+2|$. First require $\delta\le1$, so $|x+2|<5$. Then $\delta=\min(1,\varepsilon/5)$ works. A valid radius need not be the largest one.

Continuity and a broken learning signal

A function is continuous at $a$ if its limit there equals $f(a)$. Differentiability will be a stronger condition: $|x|$ is continuous at zero but has a corner.

A hard threshold $s(x)=0$ for $x<0$ and $s(x)=1$ for $x\ge0$ jumps at zero. It has derivative zero away from zero and no derivative at zero. Ordinary gradient descent therefore gets no useful local signal through this activation. Smooth or piecewise linear activations give us more useful slopes.

A vanishing term is not a convergent sum

A sequence $a_n$ has a limit. A series $\sum a_n$ means the limit of partial sums $S_N=\sum_{n=1}^N a_n$. Those are different questions.

For $|r|<1$, subtract $rS_N$ from $S_N$ to obtain

The remaining tail after $N$ terms is $r^N/(1-r)$ for $0<r<1$. This gives an actual stopping rule for code.

The harmonic series diverges even though $1/n\to0$: group terms from $2^{k-1}+1$ to $2^k$. Each group contributes at least $1/2$, forever. By comparison with the integral of $x^{-p}$, the positive series $\sum n^{-p}$ converges exactly when $p>1$.

Consequently $a_n=n^{-p}$ satisfies both step-size conditions when $1/2<p\le1$. Squaring doubles the exponent. A geometric schedule has a finite total sum, so it fails the first condition even though it shrinks smoothly.

Read beyond

Read the equation in context

Robbins and Monro study finding a root from noisy observations. These conditions balance continued movement against accumulated noise. They are part of a convergence theorem with assumptions on the response and its noise; choosing these steps alone does not guarantee that an arbitrary neural network converges.

Your turn

Choose an input radius that guarantees the requested output accuracy. A graph can suggest a choice; an inequality certifies every point in the interval.

Coding problems