Functions: Machines That Map

Discover: the most cited line of the century

In December 2015, four researchers at Microsoft Research posted a paper about training very deep networks. A 2025 analysis by Nature found it to be the most cited paper of the twenty-first century. Its central idea fits on one line:

Read it aloud with what you already know: “bold y equals calligraphic F of bold x and the set of W sub i, plus bold x.” The bold letters are vectors (Chamber 1). The braces make a set (Chamber 2): $\{W_i\}$ is the collection of weight matrices inside the block. And $\mathcal{F}$ is a function: a machine that takes the input $\mathbf{x}$ and the weights, and returns a vector. The block's output is its input plus whatever $\mathcal{F}$ computes.

Why did one “$+\,\mathbf{x}$” matter so much? Before this paper, simply stacking more layers could make a network worse: the paper's Figure 1 shows a 56-layer network with higher training error than a 20-layer one. With the shortcut, the authors trained networks 152 layers deep and won the 2015 ImageNet competition.

This chamber gives you the language to see why. A network is a chain of machines, and the notation for machines, chains and undoing machines is the notation of functions. Here's a first taste. Suppose the block's learned function gives up and outputs zero, whatever it's fed.

Learn: a function is a machine with a contract

Papers announce a function with a line like

Read it as “$f$ maps the real numbers to the real numbers, sending $x$ to $x$ squared.” The first half is the contract: $X$, before the arrow, is the domain (the inputs $f$ accepts) and $Y$, after it, is the codomain (where outputs are promised to land). The second half is the rule. Notice the two arrows. The plain arrow $\to$ connects whole sets. The arrow with a little tail, $\mapsto$ (“maps to”), connects a single input to its output.

The codomain is a promise, not a description. Squares are never negative, so this $f$ only ever produces numbers in $[0, \infty)$. The set of outputs a function actually produces is its image, written $f(X) = \{f(x) : x \in X\}$ with the set-builder notation of Chamber 2. Here the codomain is $\mathbb{R}$ and the image is $[0, \infty)$.

A rule can break the contract in two ways. It can leave an input without an output: $\log x$ has no value at $x = 0$, so we shrink the domain and write $\log : (0, \infty) \to \mathbb{R}$. Or it can give an input two outputs: “the number whose square is 4” could be 2 or $-2$, so it isn't a function until you choose one. That's why $\sqrt{x}$ always means the non-negative root.

In machine learning, everything is a function. A digit classifier is $f : \mathbb{R}^{784} \to \{0, 1, \ldots, 9\}$: a flattened $28 \times 28$ image goes in, a label comes out. A loss is $\mathcal{L} : \mathbb{R}^{d} \to \mathbb{R}$: a setting of $d$ parameters goes in, one number comes out. Reading the contract first tells you what kind of object you're dealing with before you look at a single formula.

Learn: inputs, parameters and the semicolon

A model is a function with knobs. The input is what you feed it at prediction time; the parameters are the knobs that training turns. Papers separate the two in a few standard ways:

• $f(\mathbf{x}; \theta)$, “f of x, semicolon theta”. The semicolon splits the inputs (before it) from the parameters (after it).
• $f_\theta(\mathbf{x})$, “f sub theta of x”. The same idea: a whole family of functions, one for each $\theta$.
• $f(\mathbf{x}, \theta)$, with a plain comma. ResNet's $\mathcal{F}(\mathbf{x}, \{W_i\})$ is written this way, and you rely on the text to know which argument is which.

The distinction matters because the two kinds of input change at different times. When the model predicts, $\theta$ is fixed and $\mathbf{x}$ varies. When it trains, the data is fixed and $\theta$ varies. That's why the loss is usually written $\mathcal{L}(\theta)$, a function of the parameters alone, even though it depends on data too.

Functions can take several inputs. $g : \mathbb{R} \times \mathbb{R} \to \mathbb{R}$, $(x, y) \mapsto x^2 + y^2$ takes a pair; its domain is the Cartesian product from Chamber 2. You'll meet a famous two-input function later in this chamber, a formula that predicts a language model's loss from its size and its amount of data.

Learn: composition, machines in a chain

Feed the output of one machine into another and you get a new machine. That's composition:

Say $f \circ g$ as “f composed with g”, or better, “f after g”. The small circle is not multiplication, and the order is the whole point. In $f(g(x))$, the function nearest to $x$ acts first. So the notation reads right to left, even though you read the page left to right.

Take $g(x) = x + 1$ and $f(x) = x^2$. Then

At $x = 1$ the first gives 4 and the second gives 2. Same two machines, different order, different function: composition is not commutative. The lab below turns this into a game. Build each target by chaining machines, and keep an eye on the pipeline, which runs left to right, and the notation, which runs right to left.

Composition is how deep networks are built. A network with $L$ layers is

where each layer is itself a small composition, such as $f_\ell(\mathbf{h}) = \sigma(W_\ell \mathbf{h} + \mathbf{b}_\ell)$: multiply by a matrix, add a vector, apply an activation. The input enters on the right and the prediction leaves on the left.

One machine deserves a name of its own: the identity $\mathrm{id}(x) = x$, which changes nothing. Composing with it changes nothing either: $f \circ \mathrm{id} = \mathrm{id} \circ f = f$. ResNet's block is the identity plus a correction, $\mathbf{x} \mapsto \mathbf{x} + \mathcal{F}(\mathbf{x})$. If $\mathcal{F}$ learns nothing useful it can fall back to zero and the block becomes $\mathrm{id}$, so adding blocks shouldn't make a network worse. As the paper puts it, “if an identity mapping were optimal, it would be easier to push the residual to zero than to fit an identity mapping by a stack of nonlinear layers.”

A proof: brackets don't matter, order does

Why can we write $f \circ g \circ h$ with no brackets at all? Because composition is associative, and here's the complete proof.

Claim. Composition is associative. For any functions $h : W \to X$, $g : X \to Y$ and $f : Y \to Z$,

$$(f \circ g) \circ h = f \circ (g \circ h).$$

Two functions are equal when they have the same domain and give the same output for every input. Both sides here take inputs from $W$, so it's enough to compare outputs.

Proof. Let $x$ be any element of $W$. Apply the definition of $\circ$ twice to the left-hand side, first with $f \circ g$ as the outer function:

Now do the same on the right-hand side, this time with $g \circ h$ as the inner function:

Both sides equal $f(g(h(x)))$, and $x$ was arbitrary, so the two functions are equal. $\blacksquare$

The proof is short because it only unpacks a definition, which is what many good proofs do. Its message for networks: you may group layers any way you like (into “blocks”, say), but you may not reorder them.

Learn: inverses, undoing a machine

The inverse of $f$, written $f^{-1}$ and read “f inverse”, is the machine that undoes it:

Not every function can be undone. If two inputs land on the same output, no machine can tell which one you started from. A function that never does this is injective (one-to-one): $f(a) = f(b)$ implies $a = b$. A function whose image is its whole codomain is surjective (onto). A function that is both is bijective, and those are exactly the functions with an inverse. Three examples you'll meet again:

• $x \mapsto x^2$ on $\mathbb{R}$ isn't injective, since $(-2)^2 = 2^2$. Restrict it to $[0, \infty) \to [0, \infty)$ and it becomes bijective, with inverse $\sqrt{y}$.
• $\mathrm{ReLU}$ sends every negative number to 0. Once a ReLU outputs 0, the input is gone for good.
• $\exp : \mathbb{R} \to (0, \infty)$ is bijective. Its inverse is $\log : (0, \infty) \to \mathbb{R}$, which is how the logarithm is defined.

A derivation: the inverse of the sigmoid

The sigmoid $\sigma(x) = \frac{1}{1 + e^{-x}}$ squashes every real number into $(0, 1)$, and it's strictly increasing, so it has an inverse. To find it, solve $y = \sigma(x)$ for $x$, where $0 < y < 1$:

So $\sigma^{-1}(y) = \log\frac{y}{1 - y}$, defined for $y$ in $(0, 1)$. Check it: $\sigma(0) = \frac{1}{2}$, and $\log\frac{1/2}{1/2} = \log 1 = 0$. This function is the logit.

Undoing a chain works like taking off socks and shoes: whatever went on last comes off first, so $(f \circ g)^{-1} = g^{-1} \circ f^{-1}$. You'll assemble the proof in the last part of this chamber.

Learn: piecewise functions and the toolkit

Some functions follow different rules on different parts of the domain. Papers write them with a brace:

Read it as “ReLU of x is x if x is positive, and 0 otherwise.” Each line gives a value and a condition, and exactly one condition applies to any input. A handful of small piecewise functions do a lot of work in ML papers:

A few of them in action:

• $\mathrm{ReLU}(x) = \max(0, x)$, and $|x| = \max(x, -x)$.
• A convolution with input width $n$, kernel size $k$, padding $p$ and stride $s$ produces $\left\lfloor \frac{n + 2p - k}{s} \right\rfloor + 1$ outputs. For $n = 28$, $k = 5$, $p = 0$, $s = 2$: $\lfloor 23/2 \rfloor + 1 = 12$.
• Accuracy counts correct predictions with an indicator: example $i$ contributes $\mathbf{1}[\hat{y}_i = y_i]$, which is 1 if the prediction is right and 0 if not. (Adding them up is Chamber 5's job.)
• A one-hot label for class $c$ has entries $y_k = \delta_{kc}$: 1 in position $c$, 0 everywhere else. The identity matrix of Chamber 6 has entries $\delta_{ij}$ too.

Learn: exp and log

The number $e = 2.71828\ldots$ (Euler's number) is the natural base for growth and decay. The exponential function is $x \mapsto e^{x}$, also written $\exp(x)$ when the exponent is long: $\exp(-x^2/2\sigma^2)$ is easier to read than $e^{-x^2/2\sigma^2}$, with its tiny superscript. The logarithm is its inverse: $\log(e^{x}) = x$ for every $x$, and $e^{\log y} = y$ for every $y > 0$.

Which logarithm? In ML papers, $\log$ without a base almost always means the natural log, base $e$, which mathematicians also write $\ln$. Information theory uses $\log_2$, measured in bits. Python agrees with the papers: math.log and np.log are natural logs, and base 2 needs math.log2.

Every rule for logs is a rule for exponentials seen through the inverse:

The log rules need $a, b > 0$. To change base, divide: $\log_2 x = \frac{\log x}{\log 2}$.

A derivation: why logs turn products into sums

Let $a$ and $b$ be positive, and name their logs: $u = \log a$ and $v = \log b$. Because exp undoes log, $a = e^{u}$ and $b = e^{v}$. Multiply, using the exponential rule:

Now take the log of both sides. Log undoes exp, so $\log(ab) = u + v = \log a + \log b$. The product rule for logs is the sum rule for exponentials, read backwards through the inverse.

That one rule explains why logs are everywhere in machine learning. Probabilities get multiplied, and a product of thousands of small numbers is too small for a computer to store. Its log is a harmless sum (Chamber 5). Logs also straighten power laws. If $L = A / N^{\alpha}$, then $\log L = \log A - \alpha \log N$: a straight line when you plot $\log L$ against $\log N$. That's why scaling-law papers draw log-log plots. Here is the most famous one's formula:

“L hat of N and D is defined as E, plus A over N to the alpha, plus B over D to the beta.” It's a function of two inputs, the model's parameter count $N$ and its number of training tokens $D$, and the hat marks a prediction (Chamber 1). Its five constants $E$, $A$, $B$, $\alpha$ and $\beta$ were fitted to over 400 training runs. You'll take it apart in the paper decoder and use it to make predictions in the coding problems.

Learn: activation functions are functions

Every activation function is a function $\mathbb{R} \to \mathbb{R}$, applied to each entry of a vector. Read each one's contract and you know what it can output.

They're related by composition. The hyperbolic tangent is a sigmoid, stretched and shifted: $\tanh z = 2\sigma(2z) - 1$, which is four machines in a row (double, $\sigma$, double, subtract 1). Softplus is $\log \circ (1 + \cdot) \circ \exp$. And the sigmoid's image, $(0, 1)$, is why its output can be read as a probability.

Research keeps inventing new ones, and the definitions are always short sentences in function notation. In 2017 a team at Google Brain used automated search to look for new activation functions, and highlighted this one:

The paper calls it Swish, and adds that “$\beta$ is either a constant or a trainable parameter”. So strictly it's $f(x; \beta)$, a family of functions. Look closely at the second formula: there the $-1$ is an ordinary power of a number, so $(1 + \exp(-z))^{-1}$ means $\frac{1}{1 + \exp(-z)}$. Compare $\sigma^{-1}$, the inverse function you derived above. Same superscript, different meaning, and only the thing it's attached to tells you which.

A close cousin is GELU, defined by Hendrycks and Gimpel as $x\Phi(x)$, where $\Phi(x)$ is the probability that a standard normal random variable (Chamber 8) comes out at most $x$. They suggest the approximation $x\sigma(1.702x)$, which is Swish with $\beta = 1.702$. Finally, a preview: $\mathrm{softmax}$ maps a whole vector of scores to a vector of probabilities, and its formula needs a sum over all entries. That's the notation of Chamber 5, and Chamber 9 decodes softmax in full.

Read beyond the course

Papers and lectures

You can read both of this chamber's papers today, as long as you read selectively.

For ResNet, read Section 3.1 and Section 3.2. That's about one page. Section 3.1 is almost pure function notation: the authors call the mapping they want $\mathcal{H}(\mathbf{x})$ and define the residual as $\mathcal{F}(\mathbf{x}) := \mathcal{H}(\mathbf{x}) - \mathbf{x}$. That's a definition with the $:=$ of Chamber 1, so the original mapping becomes $\mathcal{F}(\mathbf{x}) + \mathbf{x}$. Then look at Figure 2, a picture of the block. Skip the architecture details and experiments, apart from Figure 1, which shows the problem the paper solves.

For Chinchilla, read the abstract, then Section 3.3, which introduces Eq. (2) in half a page. The fitted numbers are in Appendix D.2, Eq. (10). Skip the fitting procedure (Huber loss and L-BFGS are for later) and enjoy Figure 4, whose contours are level sets of the function $\hat{L}$.

Watch

A short, playful argument that our notation for powers, roots and logarithms hides how closely they're related. Watch for how $2^3 = 8$, $\sqrt[3]{8} = 2$ and $\log_2 8 = 3$ describe one fact, and for the inverse pairs hiding in the triangle.

A walk through the ResNet paper. Watch for how he explains the residual block of Figure 2, and pause each time a formula appears to read it aloud before he does.

Your turn

Time to use it all. Translate between symbols, words and code, assemble one proof and write another, then solve three problems. Two of them use a paper's formula to make real predictions.

Proofs

The first proof puts associativity to work: it's why socks come off before shoes. The second uses the log rules.

Code it up

The first problem undoes a composition. The other two take the Chinchilla formula at its word.