Notation Glossary · Ariadne Technology

Mathematics for Machine Learning · Chamber 1

Reading Mathematics Aloud

$\alpha$ “alpha”
A step size or learning rate (Adam calls it $\alpha$ ), or a coefficient in a weighted mix.
$\alpha = 0.001$
$\beta$ “beta”
Decay rates, like Adam's $\beta_1$ and $\beta_2$ ; also regression coefficients.
$\beta_1 = 0.9$
$\gamma$ “gamma”
The discount factor in reinforcement learning; the learned scale in batch normalisation.
$\gamma = 0.99$
$\delta$ “delta”
A small change, or the error signal flowing backwards through a network.
$\delta_j$
$\Delta$ “capital delta”
A difference or change: $\Delta\theta$ is “the change in $\theta$ ”.
$\Delta\theta = \theta_{t+1} - \theta_t$
$\epsilon$ “epsilon”
A tiny positive number, like $10^{-8}$ , added to avoid dividing by zero. In proofs, an arbitrarily small tolerance.
$\epsilon = 10^{-8}$
$\varepsilon$ “curly epsilon”
The same letter in another style, often used for random noise.
$y = f(x) + \varepsilon$
$\eta$ “eta”
The learning rate in gradient descent and its relatives.
$\theta \leftarrow \theta - \eta\, g$
$\theta$ “theta”
The parameters of a model: every weight and bias, bundled together.
$f_\theta(x)$
$\lambda$ “lambda”
The strength of a penalty (regularisation); later, an eigenvalue.
$\mathcal{L} + \lambda R$
$\mu$ “mu”
A mean (average), especially of a distribution.
$\mu = 0$
$\sigma$ “sigma”
A standard deviation (how spread out values are), or the sigmoid function $\sigma(z)$ .
$\sigma(z)$
$\Sigma$ “capital sigma”
“Add them all up” (Chamber 5), or a covariance matrix.
$\sum_{i=1}^{n} x_i$
$\pi$ “pi”
The constant $3.14159\ldots$ ; in reinforcement learning, a policy.
$\pi \approx 3.14159$
$\Pi$ “capital pi”
“Multiply them all together” (Chamber 5).
$\prod_{i=1}^{n} p_i$
$\phi$ “phi”
A feature map $\phi(x)$ , or the parameters of a second network.
$\phi(x)$
$\omega$ “omega”
A weight or a frequency. Easy to mistake for $w$ .
$\omega_1$
$\tau$ “tau”
A temperature (how soft a softmax is) or a time constant.
$\tau = 0.5$
$\rho$ “rho”
A correlation or a density. Easy to mistake for $p$ .
$\rho = 0.8$
$\kappa$ “kappa”
A condition number: how badly scaled an optimisation problem is.
$\kappa = 100$
$\nu$ “nu”
Degrees of freedom, or another rate. Looks almost exactly like $v$ .
$\nu = 3$
$\xi$ “xi”
Often a random variable or noise. Said “ksee” or “zai”, and famously hard to handwrite.
$\xi$
$\chi$ “chi”
Said “kai”. Best known from the $\chi^2$ (chi-squared) test.
$\chi^2$
$x$ “x”
Plain italic lower case: a scalar, one single number.
$x = 3.2$
$\mathbf{x}$ “bold x”
Bold lower case: a vector, an ordered list of numbers.
$\mathbf{x} = (x_1, x_2, x_3)$
$\mathbf{X}$ “bold capital X”
Bold (or plain) upper case: a matrix, a grid of numbers. Often a whole dataset, one example per row.
$\mathcal{L}$ “calligraphic L”
Curly capitals name special objects: $\mathcal{L}$ a loss, $\mathcal{D}$ a dataset, $\mathcal{N}$ the normal distribution.
$\mathcal{L}(\theta)$
$\mathbb{R}$ “blackboard bold R”
Double-struck capitals name number systems. $\mathbb{R}$ is the real numbers (Chamber 2).
$x \in \mathbb{R}$
$\mathrm{softmax}$ “upright softmax”
Upright letters spell the names of functions ( $\log$ , $\exp$ , $\mathrm{softmax}$ ), never a product of variables.
$\log x$
$x_i$ “x sub i”
The $i$ -th entry of a list (or the $i$ -th example). A subscript is an index: a position.
$x_1, x_2, \ldots, x_n$
$A_{ij}$ “A sub i j”
The entry in row $i$ and column $j$ of a matrix. Two indices, two directions.
$A_{23}$
$x^2$ “x squared”
A superscript number is usually a power: $x \cdot x$ .
$x^3 = x \cdot x \cdot x$
$x^{(i)}$ “x superscript i”
A superscript in brackets is an index, not a power: the $i$ -th training example.
$x^{(1)}, \ldots, x^{(n)}$
$\theta_t$ “theta at step t”
A subscript $t$ often means “at time step $t$ ”: the value after $t$ updates.
$\theta_0, \theta_1, \theta_2$
$x'$ “x prime”
Another $x$ (a second point, a new value). On a function, $f'$ means its derivative (Chamber 7).
$f'(x)$
$\hat{y}$ “y hat”
A hat means estimated or predicted: $\hat{y}$ is the model's guess at $y$ .
$\hat{y} = f_\theta(x)$
$\bar{x}$ “x bar”
A bar means an average: $\bar{x}$ is the mean of the $x$ values.
$\bar{x} = \tfrac{1}{2}(x_1 + x_2)$
$\tilde{x}$ “x tilde”
A tilde means a modified version: noisy, approximate or transformed.
$\tilde{x} = x + \varepsilon$
$\theta^\star$ “theta star”
A star means optimal: the best possible value. Also written $\theta^*$ .
$\theta^\star$
$\vec{v}$ “v arrow”
An arrow marks a vector in school and physics. ML papers prefer bold, $\mathbf{v}$ .
$\vec{v}$
$=$ “equals”
Both sides are the same value. A claim you could check.
$2 + 2 = 4$
$:=$ “is defined as”
The left side is a new name for the right side. Not a claim to check: a definition.
$\bar{x} := \tfrac{1}{2}(x_1 + x_2)$
$\triangleq$ “is defined as (delta-equals)”
Another way to write a definition, common in engineering papers.
$f(x) \triangleq x^2$
$\equiv$ “is identical to”
Equal for every value of the variables (an identity). Some authors also use it for definitions.
$(a+b)^2 \equiv a^2 + 2ab + b^2$
$\approx$ “is approximately”
Close, but not exactly equal.
$\pi \approx 3.14$
$\propto$ “is proportional to”
Equal up to a constant factor: $y \propto x$ means $y = c\,x$ for some fixed $c$ .
$y \propto x^2$
$\ll$ “is much less than”
Smaller by a large factor.
$r \ll d$
$\leftarrow$ “gets”
Assignment in an algorithm: compute the right side, store it in the left. An update, not an equation.
$m \leftarrow m + 1$
$\blacksquare$ “end of proof”
The “tombstone”: the proof is finished. Older texts write Q.E.D.
$\blacksquare$

Symbol	Say it	Meaning
$\alpha$ $\alpha = 0.001$	“alpha”	A step size or learning rate (Adam calls it $\alpha$ ), or a coefficient in a weighted mix.
$\beta$ $\beta_1 = 0.9$	“beta”	Decay rates, like Adam's $\beta_1$ and $\beta_2$ ; also regression coefficients.
$\gamma$ $\gamma = 0.99$	“gamma”	The discount factor in reinforcement learning; the learned scale in batch normalisation.
$\delta$ $\delta_j$	“delta”	A small change, or the error signal flowing backwards through a network.
$\Delta$ $\Delta\theta = \theta_{t+1} - \theta_t$	“capital delta”	A difference or change: $\Delta\theta$ is “the change in $\theta$ ”.
$\epsilon$ $\epsilon = 10^{-8}$	“epsilon”	A tiny positive number, like $10^{-8}$ , added to avoid dividing by zero. In proofs, an arbitrarily small tolerance.
$\varepsilon$ $y = f(x) + \varepsilon$	“curly epsilon”	The same letter in another style, often used for random noise.
$\eta$ $\theta \leftarrow \theta - \eta\, g$	“eta”	The learning rate in gradient descent and its relatives.
$\theta$ $f_\theta(x)$	“theta”	The parameters of a model: every weight and bias, bundled together.
$\lambda$ $\mathcal{L} + \lambda R$	“lambda”	The strength of a penalty (regularisation); later, an eigenvalue.
$\mu$ $\mu = 0$	“mu”	A mean (average), especially of a distribution.
$\sigma$ $\sigma(z)$	“sigma”	A standard deviation (how spread out values are), or the sigmoid function $\sigma(z)$ .
$\Sigma$ $\sum_{i=1}^{n} x_i$	“capital sigma”	“Add them all up” (Chamber 5), or a covariance matrix.
$\pi$ $\pi \approx 3.14159$	“pi”	The constant $3.14159\ldots$ ; in reinforcement learning, a policy.
$\Pi$ $\prod_{i=1}^{n} p_i$	“capital pi”	“Multiply them all together” (Chamber 5).
$\phi$ $\phi(x)$	“phi”	A feature map $\phi(x)$ , or the parameters of a second network.
$\omega$ $\omega_1$	“omega”	A weight or a frequency. Easy to mistake for $w$ .
$\tau$ $\tau = 0.5$	“tau”	A temperature (how soft a softmax is) or a time constant.
$\rho$ $\rho = 0.8$	“rho”	A correlation or a density. Easy to mistake for $p$ .
$\kappa$ $\kappa = 100$	“kappa”	A condition number: how badly scaled an optimisation problem is.
$\nu$ $\nu = 3$	“nu”	Degrees of freedom, or another rate. Looks almost exactly like $v$ .
$\xi$ $\xi$	“xi”	Often a random variable or noise. Said “ksee” or “zai”, and famously hard to handwrite.
$\chi$ $\chi^2$	“chi”	Said “kai”. Best known from the $\chi^2$ (chi-squared) test.
$x$ $x = 3.2$	“x”	Plain italic lower case: a scalar, one single number.
$\mathbf{x}$ $\mathbf{x} = (x_1, x_2, x_3)$	“bold x”	Bold lower case: a vector, an ordered list of numbers.
$\mathbf{X}$	“bold capital X”	Bold (or plain) upper case: a matrix, a grid of numbers. Often a whole dataset, one example per row.
$\mathcal{L}$ $\mathcal{L}(\theta)$	“calligraphic L”	Curly capitals name special objects: $\mathcal{L}$ a loss, $\mathcal{D}$ a dataset, $\mathcal{N}$ the normal distribution.
$\mathbb{R}$ $x \in \mathbb{R}$	“blackboard bold R”	Double-struck capitals name number systems. $\mathbb{R}$ is the real numbers (Chamber 2).
$\mathrm{softmax}$ $\log x$	“upright softmax”	Upright letters spell the names of functions ( $\log$ , $\exp$ , $\mathrm{softmax}$ ), never a product of variables.
$x_i$ $x_1, x_2, \ldots, x_n$	“x sub i”	The $i$ -th entry of a list (or the $i$ -th example). A subscript is an index: a position.
$A_{ij}$ $A_{23}$	“A sub i j”	The entry in row $i$ and column $j$ of a matrix. Two indices, two directions.
$x^2$ $x^3 = x \cdot x \cdot x$	“x squared”	A superscript number is usually a power: $x \cdot x$ .
$x^{(i)}$ $x^{(1)}, \ldots, x^{(n)}$	“x superscript i”	A superscript in brackets is an index, not a power: the $i$ -th training example.
$\theta_t$ $\theta_0, \theta_1, \theta_2$	“theta at step t”	A subscript $t$ often means “at time step $t$ ”: the value after $t$ updates.
$x'$ $f'(x)$	“x prime”	Another $x$ (a second point, a new value). On a function, $f'$ means its derivative (Chamber 7).
$\hat{y}$ $\hat{y} = f_\theta(x)$	“y hat”	A hat means estimated or predicted: $\hat{y}$ is the model's guess at $y$ .
$\bar{x}$ $\bar{x} = \tfrac{1}{2}(x_1 + x_2)$	“x bar”	A bar means an average: $\bar{x}$ is the mean of the $x$ values.
$\tilde{x}$ $\tilde{x} = x + \varepsilon$	“x tilde”	A tilde means a modified version: noisy, approximate or transformed.
$\theta^\star$ $\theta^\star$	“theta star”	A star means optimal: the best possible value. Also written $\theta^*$ .
$\vec{v}$ $\vec{v}$	“v arrow”	An arrow marks a vector in school and physics. ML papers prefer bold, $\mathbf{v}$ .
$=$ $2 + 2 = 4$	“equals”	Both sides are the same value. A claim you could check.
$:=$ $\bar{x} := \tfrac{1}{2}(x_1 + x_2)$	“is defined as”	The left side is a new name for the right side. Not a claim to check: a definition.
$\triangleq$ $f(x) \triangleq x^2$	“is defined as (delta-equals)”	Another way to write a definition, common in engineering papers.
$\equiv$ $(a+b)^2 \equiv a^2 + 2ab + b^2$	“is identical to”	Equal for every value of the variables (an identity). Some authors also use it for definitions.
$\approx$ $\pi \approx 3.14$	“is approximately”	Close, but not exactly equal.
$\propto$ $y \propto x^2$	“is proportional to”	Equal up to a constant factor: $y \propto x$ means $y = c\,x$ for some fixed $c$ .
$\ll$ $r \ll d$	“is much less than”	Smaller by a large factor.
$\leftarrow$ $m \leftarrow m + 1$	“gets”	Assignment in an algorithm: compute the right side, store it in the left. An update, not an equation.
$\blacksquare$ $\blacksquare$	“end of proof”	The “tombstone”: the proof is finished. Older texts write Q.E.D.

Mathematics for Machine Learning · Chamber 2

Sets: The Nouns of Mathematics

$\in$ “is an element of”
Membership: $x \in A$ says the object $x$ belongs to the set $A$ . Also read “is in” or “belongs to”.
$3 \in \{1, 2, 3\}$
$\notin$ “is not an element of”
$x \notin A$ says $x$ does not belong to $A$ .
$4 \notin \{1, 2, 3\}$
$\{a, b, c\}$ “the set containing a, b and c”
Curly braces list a set's elements (roster notation). Order and repeats don't matter: $\{1, 2\} = \{2, 1, 1\}$ .
$\{\text{cat}, \text{dog}, \text{bird}\}$
$\varnothing$ “the empty set”
The set with no elements at all, $\{\}$ . Also printed $\emptyset$ .
$|\varnothing| = 0$
$|A|$ “the size of A”
The number of elements of a finite set $A$ , called its cardinality. Around a number the same bars mean absolute value, so check what's inside.
$|\{2, 4, 6\}| = 3$
$\subseteq$ “is a subset of”
$A \subseteq B$ : every element of $A$ is also an element of $B$ . $A$ is allowed to equal $B$ .
$\{1, 2\} \subseteq \{1, 2, 3\}$
$\subsetneq$ “is a proper subset of”
$A \subsetneq B$ : $A \subseteq B$ and $A \ne B$ , so $B$ has at least one element that $A$ lacks.
$\mathbb{N} \subsetneq \mathbb{Z}$
$\subset$ “is a subset of (strict or not, check)”
Ambiguous: some authors mean $\subseteq$ , others $\subsetneq$ . ML papers mostly use it loosely for $\subseteq$ . Check the paper's convention.
$S \subset \mathbb{R}$
$\{x \in S : P(x)\}$ “the set of x in S such that P of x”
Go through $S$ and keep each $x$ that passes the test $P$ . The colon reads “such that”.
$\{x \in \mathbb{Z} : x > 0\}$
$\{x \mid P(x)\}$ “the set of all x such that P of x”
The same idea with a vertical bar instead of a colon: $\{x \mid x^2 = 4\} = \{-2, 2\}$ . Where $x$ comes from should be clear from context.
$\{x \mid x^2 = 4\}$
$\{f(x) : x \in S\}$ “the set of f of x for x in S”
A formula on the left builds the elements: apply $f$ to every $x$ in $S$ and collect the results.
$\{2k : k \in \mathbb{Z}\}$
$\mathbb{N}$ “the natural numbers”
The counting numbers. Authors disagree about whether $0 \in \mathbb{N}$ , so check. $\mathbb{N}_0$ or $\mathbb{Z}_{\ge 0}$ removes the doubt.
$n \in \mathbb{N}$
$\mathbb{Z}$ “the integers”
Whole numbers, positive, negative and zero: $\{\ldots, -2, -1, 0, 1, 2, \ldots\}$ .
$-3 \in \mathbb{Z}$
$\mathbb{Q}$ “the rational numbers”
Fractions $p/q$ with $p, q \in \mathbb{Z}$ and $q \ne 0$ .
$\tfrac{3}{4} \in \mathbb{Q}$
$\mathbb{R}$ “the real numbers”
Every point on the number line, including $\sqrt{2}$ and $\pi$ . The home of weights, losses and learning rates.
$\eta \in \mathbb{R}$
$\mathbb{R}_{\ge 0}$ “the non-negative reals”
Real numbers that are $\ge 0$ : distances, variances, ReLU outputs.
$\sigma^2 \in \mathbb{R}_{\ge 0}$
$\mathbb{R}_{+}$ “R plus”
The positive reals, or the non-negative reals: authors differ, so check. Learning rates live here.
$\eta \in \mathbb{R}_{+}$
$[a, b]$ “the closed interval from a to b”
All reals $x$ with $a \le x \le b$ : both ends included.
$p \in [0, 1]$
$(a, b)$ “the open interval from a to b”
All reals $x$ with $a < x < b$ : both ends excluded. The same brackets also write an ordered pair, so read the context.
$\sigma(z) \in (0, 1)$
$[a, b)$ “the half-open interval from a to b”
All reals $x$ with $a \le x < b$ . A square bracket includes its end, a round one excludes it.
$u \in [0, 1)$
$\cup$ “union”
$A \cup B$ : everything in $A$ or in $B$ (or in both).
$\{1\} \cup \{2\} = \{1, 2\}$
$\cap$ “intersection”
$A \cap B$ : everything in both $A$ and $B$ .
$\{1, 2\} \cap \{2, 3\} = \{2\}$
$\setminus$ “set minus”
$A \setminus B$ : everything in $A$ that is not in $B$ . Some authors write $A - B$ .
$\{1, 2\} \setminus \{2, 3\} = \{1\}$
$A^{c}$ “the complement of A”
Everything in the universe $U$ that is not in $A$ : $A^c = U \setminus A$ .
$(A^{c})^{c} = A$
$\overline{A}$ “A bar, the complement”
Another way to write the complement. In statistics the same bar means an average ( $\bar{x}$ , Chamber 1), so read the context.
$\overline{A \cup B} = \overline{A} \cap \overline{B}$
$U$ “the universal set”
Everything under discussion. Complements are taken inside it. Also written $\Omega$ or $\mathcal{U}$ .
$A \subseteq U$
$A \cap B = \varnothing$ “A and B are disjoint”
The two sets share no elements, like a training set and a test set should.
$\mathcal{D}_{\text{train}} \cap \mathcal{D}_{\text{test}} = \varnothing$
$(a, b)$ “the ordered pair a, b”
Two objects in a fixed order: $(1, 2) \ne (2, 1)$ , unlike the set $\{1, 2\} = \{2, 1\}$ .
$(x_i, y_i)$
$(x_1, \ldots, x_n)$ “the tuple x one to x n”
An ordered list of $n$ entries, called an $n$ -tuple. Order and repeats both count.
$(3, 1, 3) \ne (1, 3)$
$A \times B$ “A cross B”
The Cartesian product: every ordered pair $(a, b)$ with $a \in A$ and $b \in B$ . It has $|A| \cdot |B|$ elements.
$\{0, 1\} \times \{a, b\}$
$\mathbb{R}^n$ “R n”
All $n$ -tuples of real numbers, $\mathbb{R} \times \cdots \times \mathbb{R}$ with $n$ factors. A vector $\mathbf{x} \in \mathbb{R}^n$ is one element.
$\mathbf{x} \in \mathbb{R}^{784}$
$\mathbb{R}^{m \times n}$ “R m by n”
All grids of real numbers with $m$ rows and $n$ columns: real matrices (Chamber 6).
$W \in \mathbb{R}^{512 \times 64}$
$\{1, \ldots, K\}$ “the set one to K”
The integers from 1 to $K$ : the usual label set for a classifier with $K$ classes. Some papers write $[K]$ .
$y \in \{1, \ldots, K\}$
$\{(x_i, y_i)\}_{i=1}^{n}$ “the set of pairs x i, y i for i from 1 to n”
A dataset: one (input, label) pair for each index $i$ from 1 to $n$ .
$\mathcal{D} = \{(\mathbf{x}_i, y_i)\}_{i=1}^{n}$

Symbol	Say it	Meaning
$\in$ $3 \in \{1, 2, 3\}$	“is an element of”	Membership: $x \in A$ says the object $x$ belongs to the set $A$ . Also read “is in” or “belongs to”.
$\notin$ $4 \notin \{1, 2, 3\}$	“is not an element of”	$x \notin A$ says $x$ does not belong to $A$ .
$\{a, b, c\}$ $\{\text{cat}, \text{dog}, \text{bird}\}$	“the set containing a, b and c”	Curly braces list a set's elements (roster notation). Order and repeats don't matter: $\{1, 2\} = \{2, 1, 1\}$ .
$\varnothing$ $\|\varnothing\| = 0$	“the empty set”	The set with no elements at all, $\{\}$ . Also printed $\emptyset$ .
$\|A\|$ $\|\{2, 4, 6\}\| = 3$	“the size of A”	The number of elements of a finite set $A$ , called its cardinality. Around a number the same bars mean absolute value, so check what's inside.
$\subseteq$ $\{1, 2\} \subseteq \{1, 2, 3\}$	“is a subset of”	$A \subseteq B$ : every element of $A$ is also an element of $B$ . $A$ is allowed to equal $B$ .
$\subsetneq$ $\mathbb{N} \subsetneq \mathbb{Z}$	“is a proper subset of”	$A \subsetneq B$ : $A \subseteq B$ and $A \ne B$ , so $B$ has at least one element that $A$ lacks.
$\subset$ $S \subset \mathbb{R}$	“is a subset of (strict or not, check)”	Ambiguous: some authors mean $\subseteq$ , others $\subsetneq$ . ML papers mostly use it loosely for $\subseteq$ . Check the paper's convention.
$\{x \in S : P(x)\}$ $\{x \in \mathbb{Z} : x > 0\}$	“the set of x in S such that P of x”	Go through $S$ and keep each $x$ that passes the test $P$ . The colon reads “such that”.
$\{x \mid P(x)\}$ $\{x \mid x^2 = 4\}$	“the set of all x such that P of x”	The same idea with a vertical bar instead of a colon: $\{x \mid x^2 = 4\} = \{-2, 2\}$ . Where $x$ comes from should be clear from context.
$\{f(x) : x \in S\}$ $\{2k : k \in \mathbb{Z}\}$	“the set of f of x for x in S”	A formula on the left builds the elements: apply $f$ to every $x$ in $S$ and collect the results.
$\mathbb{N}$ $n \in \mathbb{N}$	“the natural numbers”	The counting numbers. Authors disagree about whether $0 \in \mathbb{N}$ , so check. $\mathbb{N}_0$ or $\mathbb{Z}_{\ge 0}$ removes the doubt.
$\mathbb{Z}$ $-3 \in \mathbb{Z}$	“the integers”	Whole numbers, positive, negative and zero: $\{\ldots, -2, -1, 0, 1, 2, \ldots\}$ .
$\mathbb{Q}$ $\tfrac{3}{4} \in \mathbb{Q}$	“the rational numbers”	Fractions $p/q$ with $p, q \in \mathbb{Z}$ and $q \ne 0$ .
$\mathbb{R}$ $\eta \in \mathbb{R}$	“the real numbers”	Every point on the number line, including $\sqrt{2}$ and $\pi$ . The home of weights, losses and learning rates.
$\mathbb{R}_{\ge 0}$ $\sigma^2 \in \mathbb{R}_{\ge 0}$	“the non-negative reals”	Real numbers that are $\ge 0$ : distances, variances, ReLU outputs.
$\mathbb{R}_{+}$ $\eta \in \mathbb{R}_{+}$	“R plus”	The positive reals, or the non-negative reals: authors differ, so check. Learning rates live here.
$[a, b]$ $p \in [0, 1]$	“the closed interval from a to b”	All reals $x$ with $a \le x \le b$ : both ends included.
$(a, b)$ $\sigma(z) \in (0, 1)$	“the open interval from a to b”	All reals $x$ with $a < x < b$ : both ends excluded. The same brackets also write an ordered pair, so read the context.
$[a, b)$ $u \in [0, 1)$	“the half-open interval from a to b”	All reals $x$ with $a \le x < b$ . A square bracket includes its end, a round one excludes it.
$\cup$ $\{1\} \cup \{2\} = \{1, 2\}$	“union”	$A \cup B$ : everything in $A$ or in $B$ (or in both).
$\cap$ $\{1, 2\} \cap \{2, 3\} = \{2\}$	“intersection”	$A \cap B$ : everything in both $A$ and $B$ .
$\setminus$ $\{1, 2\} \setminus \{2, 3\} = \{1\}$	“set minus”	$A \setminus B$ : everything in $A$ that is not in $B$ . Some authors write $A - B$ .
$A^{c}$ $(A^{c})^{c} = A$	“the complement of A”	Everything in the universe $U$ that is not in $A$ : $A^c = U \setminus A$ .
$\overline{A}$ $\overline{A \cup B} = \overline{A} \cap \overline{B}$	“A bar, the complement”	Another way to write the complement. In statistics the same bar means an average ( $\bar{x}$ , Chamber 1), so read the context.
$U$ $A \subseteq U$	“the universal set”	Everything under discussion. Complements are taken inside it. Also written $\Omega$ or $\mathcal{U}$ .
$A \cap B = \varnothing$ $\mathcal{D}_{\text{train}} \cap \mathcal{D}_{\text{test}} = \varnothing$	“A and B are disjoint”	The two sets share no elements, like a training set and a test set should.
$(a, b)$ $(x_i, y_i)$	“the ordered pair a, b”	Two objects in a fixed order: $(1, 2) \ne (2, 1)$ , unlike the set $\{1, 2\} = \{2, 1\}$ .
$(x_1, \ldots, x_n)$ $(3, 1, 3) \ne (1, 3)$	“the tuple x one to x n”	An ordered list of $n$ entries, called an $n$ -tuple. Order and repeats both count.
$A \times B$ $\{0, 1\} \times \{a, b\}$	“A cross B”	The Cartesian product: every ordered pair $(a, b)$ with $a \in A$ and $b \in B$ . It has $\|A\| \cdot \|B\|$ elements.
$\mathbb{R}^n$ $\mathbf{x} \in \mathbb{R}^{784}$	“R n”	All $n$ -tuples of real numbers, $\mathbb{R} \times \cdots \times \mathbb{R}$ with $n$ factors. A vector $\mathbf{x} \in \mathbb{R}^n$ is one element.
$\mathbb{R}^{m \times n}$ $W \in \mathbb{R}^{512 \times 64}$	“R m by n”	All grids of real numbers with $m$ rows and $n$ columns: real matrices (Chamber 6).
$\{1, \ldots, K\}$ $y \in \{1, \ldots, K\}$	“the set one to K”	The integers from 1 to $K$ : the usual label set for a classifier with $K$ classes. Some papers write $[K]$ .
$\{(x_i, y_i)\}_{i=1}^{n}$ $\mathcal{D} = \{(\mathbf{x}_i, y_i)\}_{i=1}^{n}$	“the set of pairs x i, y i for i from 1 to n”	A dataset: one (input, label) pair for each index $i$ from 1 to $n$ .

Mathematics for Machine Learning · Chamber 3

Logic: For All and There Exists

$\neg$ “not”
Negation: $\neg P$ is true exactly when $P$ is false. Some authors write $\sim P$ .
$\neg (x > 0)$
$\land$ “and”
Conjunction: $P \land Q$ is true only when both $P$ and $Q$ are true.
$x > 0 \land x < 1$
$\lor$ “or”
Disjunction: $P \lor Q$ is true when at least one of them is. Mathematical “or” is inclusive: both is fine.
$x < 0 \lor x > 1$
$\Rightarrow$ “implies”
$P \Rightarrow Q$ , “if $P$ then $Q$ ”: false only when $P$ is true and $Q$ is false. Also written $\implies$ or $\to$ .
$x > 2 \Rightarrow x^2 > 4$
$\Leftarrow$ “is implied by”
$P \Leftarrow Q$ means $Q \Rightarrow P$ . Proofs of an “if and only if” often label their two halves $(\Rightarrow)$ and $(\Leftarrow)$ .
$x^2 > 4 \Leftarrow x > 2$
$\Leftrightarrow$ “if and only if”
$P \Leftrightarrow Q$ : each implies the other, so they are always true or false together. Also written $\iff$ .
$n \text{ even} \Leftrightarrow n^2 \text{ even}$
$\equiv$ “is logically equivalent to”
Two statements with identical truth tables, like an implication and its contrapositive. (Chamber 1 met $\equiv$ as “is identical to”: the same idea, for logic.)
$\neg\neg P \equiv P$
$\forall$ “for all”
The universal quantifier: what follows holds for every element. Also read “for every” or “for each”.
$\forall x \in \mathbb{R},\; x^2 \ge 0$
$\exists$ “there exists”
The existential quantifier: at least one element makes what follows true. One witness is enough.
$\exists x \in \mathbb{R},\; x^2 = 2$
$\exists!$ “there exists exactly one”
Existence and uniqueness: one element works, and no second one does.
$\exists!\, x \in \mathbb{R},\; 2x = 6$
$\nexists$ “there does not exist”
No element works: the same as $\neg\exists$ , or “for all, not”.
$\nexists x \in \mathbb{R},\; x^2 < 0$
$\text{s.t.}$ “such that”
Introduces the condition after a quantifier. Papers also use a colon, a vertical bar or just a comma.
$\exists \theta \text{ s.t. } \mathcal{L}(\theta) = 0$
$\text{iff}$ “if and only if”
The written-out form of $\Leftrightarrow$ , said exactly the same way. Two statements are claimed, one in each direction.
$n \text{ even iff } n^2 \text{ even}$
$\text{WLOG}$ “without loss of generality”
“The other cases work the same way, so we only treat this one.” Honest only when the cases really are symmetric.
$\text{WLOG, } a \le b$
$\text{i.e.}$ “that is”
Latin id est: a restatement of what was just said, in other words.
$x \in \mathbb{R}_{\ge 0}, \text{ i.e. } x \ge 0$
$\text{e.g.}$ “for example”
Latin exempli gratia: one example among several, not a restatement.
$\text{e.g. ReLU}$
$\therefore$ “therefore”
Marks a conclusion. Rare in papers, common on whiteboards.
$\therefore\; x = 3$
$\Rightarrow\!\Leftarrow$ “contradiction”
Marks the moment a proof by contradiction reaches something impossible. Also drawn as $\bot$ or a lightning bolt.
$p = 2k = 2m + 1 \;\Rightarrow\!\Leftarrow$

Symbol	Say it	Meaning
$\neg$ $\neg (x > 0)$	“not”	Negation: $\neg P$ is true exactly when $P$ is false. Some authors write $\sim P$ .
$\land$ $x > 0 \land x < 1$	“and”	Conjunction: $P \land Q$ is true only when both $P$ and $Q$ are true.
$\lor$ $x < 0 \lor x > 1$	“or”	Disjunction: $P \lor Q$ is true when at least one of them is. Mathematical “or” is inclusive: both is fine.
$\Rightarrow$ $x > 2 \Rightarrow x^2 > 4$	“implies”	$P \Rightarrow Q$ , “if $P$ then $Q$ ”: false only when $P$ is true and $Q$ is false. Also written $\implies$ or $\to$ .
$\Leftarrow$ $x^2 > 4 \Leftarrow x > 2$	“is implied by”	$P \Leftarrow Q$ means $Q \Rightarrow P$ . Proofs of an “if and only if” often label their two halves $(\Rightarrow)$ and $(\Leftarrow)$ .
$\Leftrightarrow$ $n \text{ even} \Leftrightarrow n^2 \text{ even}$	“if and only if”	$P \Leftrightarrow Q$ : each implies the other, so they are always true or false together. Also written $\iff$ .
$\equiv$ $\neg\neg P \equiv P$	“is logically equivalent to”	Two statements with identical truth tables, like an implication and its contrapositive. (Chamber 1 met $\equiv$ as “is identical to”: the same idea, for logic.)
$\forall$ $\forall x \in \mathbb{R},\; x^2 \ge 0$	“for all”	The universal quantifier: what follows holds for every element. Also read “for every” or “for each”.
$\exists$ $\exists x \in \mathbb{R},\; x^2 = 2$	“there exists”	The existential quantifier: at least one element makes what follows true. One witness is enough.
$\exists!$ $\exists!\, x \in \mathbb{R},\; 2x = 6$	“there exists exactly one”	Existence and uniqueness: one element works, and no second one does.
$\nexists$ $\nexists x \in \mathbb{R},\; x^2 < 0$	“there does not exist”	No element works: the same as $\neg\exists$ , or “for all, not”.
$\text{s.t.}$ $\exists \theta \text{ s.t. } \mathcal{L}(\theta) = 0$	“such that”	Introduces the condition after a quantifier. Papers also use a colon, a vertical bar or just a comma.
$\text{iff}$ $n \text{ even iff } n^2 \text{ even}$	“if and only if”	The written-out form of $\Leftrightarrow$ , said exactly the same way. Two statements are claimed, one in each direction.
$\text{WLOG}$ $\text{WLOG, } a \le b$	“without loss of generality”	“The other cases work the same way, so we only treat this one.” Honest only when the cases really are symmetric.
$\text{i.e.}$ $x \in \mathbb{R}_{\ge 0}, \text{ i.e. } x \ge 0$	“that is”	Latin id est: a restatement of what was just said, in other words.
$\text{e.g.}$ $\text{e.g. ReLU}$	“for example”	Latin exempli gratia: one example among several, not a restatement.
$\therefore$ $\therefore\; x = 3$	“therefore”	Marks a conclusion. Rare in papers, common on whiteboards.
$\Rightarrow\!\Leftarrow$ $p = 2k = 2m + 1 \;\Rightarrow\!\Leftarrow$	“contradiction”	Marks the moment a proof by contradiction reaches something impossible. Also drawn as $\bot$ or a lightning bolt.

Mathematics for Machine Learning · Chamber 4

Functions: Machines That Map

$f : X \to Y$ “f from X to Y”
A function named $f$ with domain $X$ (the allowed inputs) and codomain $Y$ (where outputs are promised to live). Every input gets exactly one output.
$f : \mathbb{R} \to \mathbb{R}$
$x \mapsto x^2$ “x maps to x squared”
The rule: what happens to a single input. This arrow has a little tail; the plain arrow $\to$ connects whole sets.
$f : x \mapsto x^2$
$f(x)$ “f of x”
The output of $f$ at the input $x$ . The brackets mean “of”, not multiplication.
$f(3) = 9$
$f(X)$ “the image of X under f”
The set of outputs $f$ actually produces, $\{f(x) : x \in X\}$ . It sits inside the codomain and can be smaller.
$f(\mathbb{R}) = [0, \infty)$
$f(x; \theta)$ “f of x semicolon theta”
A model: the input $x$ before the semicolon, the parameters $\theta$ after it. Training changes $\theta$ ; prediction changes $x$ .
$f(\mathbf{x}; \theta)$
$f_\theta(x)$ “f sub theta of x”
The same idea with the parameters as a subscript: one function for every choice of $\theta$ .
$f_\theta(\mathbf{x})$
$f(x, y)$ “f of x and y”
A function of two inputs. Its domain is a Cartesian product such as $\mathbb{R} \times \mathbb{R}$ (Chamber 2).
$\hat{L}(N, D)$
$f \circ g$ “f composed with g”
“ $f$ after $g$ ”: apply $g$ first, then $f$ . The notation reads right to left.
$(f \circ g)(x) = f(g(x))$
$f(g(x))$ “f of g of x”
Nested brackets: the innermost function acts first, exactly as in code.
$\sigma(W\mathbf{x} + \mathbf{b})$
$\mathrm{id}$ “the identity function”
The machine that changes nothing: $\mathrm{id}(x) = x$ . Composing with it leaves any function unchanged.
$f \circ \mathrm{id} = f$
$f^{-1}$ “f inverse”
The function that undoes $f$ : $f^{-1}(f(x)) = x$ . It is not $1/f$ . It exists only when $f$ is a bijection.
$f^{-1} \circ f = \mathrm{id}$
$f_L \circ \cdots \circ f_1$ “f L after dots after f 1”
A deep network: $L$ layers applied in turn, $f_1$ first and $f_L$ last.
$f = f_L \circ \cdots \circ f_2 \circ f_1$
$\begin{cases} a & \text{if } P \\ b & \text{otherwise} \end{cases}$ “a if P, otherwise b”
A piecewise definition. The brace lists the cases, and exactly one applies to each input.
$|x|$ “the absolute value of x”
The size of $x$ without its sign: $x$ if $x \ge 0$ , and $-x$ if $x < 0$ .
$|-3| = 3$
$\lfloor x \rfloor$ “the floor of x”
Round down to the nearest integer. Careful with negatives: $\lfloor -2.3 \rfloor = -3$ .
$\lfloor 2.7 \rfloor = 2$
$\lceil x \rceil$ “the ceiling of x”
Round up to the nearest integer.
$\lceil 2.2 \rceil = 3$
$\max(a, b)$ “the max of a and b”
The larger of the two numbers.
$\mathrm{ReLU}(x) = \max(0, x)$
$\min(a, b)$ “the min of a and b”
The smaller of the two numbers.
$\min(3, -1) = -1$
$\mathbf{1}[P]$ “the indicator of P”
1 if the statement $P$ is true, 0 if it's false. Also written $\mathbf{1}_{P}$ , $\mathbb{I}[P]$ or with a double-struck one.
$\mathbf{1}[\hat{y} = y]$
$\delta_{ij}$ “Kronecker delta i j”
1 if $i = j$ and 0 otherwise: an indicator for “same index”.
$\delta_{33} = 1,\ \delta_{34} = 0$
$e$ “e, Euler's number”
The constant $2.71828\ldots$ , the natural base for exponentials and logarithms.
$e \approx 2.718$
$e^{x}$ “e to the x”
The exponential function. Always positive, and it turns sums into products: $e^{a+b} = e^{a}e^{b}$ .
$e^{0} = 1$
$\exp(x)$ “exp of x”
The same function written as a name, easier to read when the exponent is long.
$\exp\left(-\tfrac{x^2}{2}\right)$
$\log x$ “log x”
The inverse of exp. In ML papers a log with no base almost always means the natural log, base $e$ .
$\log e^{x} = x$
$\ln x$ “natural log of x”
The natural log, written explicitly. The same function as ML's $\log$ , and as Python's math.log.
$\ln e = 1$
$\log_2 x$ “log base two of x”
How many times you must double 1 to reach $x$ . Information measured in bits uses base 2.
$\log_2 8 = 3$
$\log_{10} x$ “log base ten of x”
How many times you must multiply 1 by ten to reach $x$ : roughly, the number of digits.
$\log_{10} 1000 = 3$
$\sigma(z)$ “sigma of z”
The sigmoid $\frac{1}{1 + e^{-z}}$ . It squashes any real number into $(0, 1)$ , so its output can be read as a probability.
$\sigma(0) = \tfrac{1}{2}$
$\tanh(z)$ “tanh of z”
The hyperbolic tangent: squashes into $(-1, 1)$ . It is a sigmoid, stretched and shifted: $\tanh z = 2\sigma(2z) - 1$ .
$\tanh(0) = 0$
$\mathrm{ReLU}(z)$ “relu of z”
The rectified linear unit, $\max(0, z)$ : negative inputs become 0, positive ones pass through.
$\mathrm{ReLU}(-2) = 0$
$\mathrm{softplus}(z)$ “softplus of z”
$\log(1 + e^{z})$ : a smooth version of ReLU that is always positive. The Deep Learning book writes it $\zeta(z)$ .
$\mathrm{softplus}(0) = \log 2$
$\mathrm{logit}(p)$ “logit of p”
The inverse of the sigmoid, $\log\frac{p}{1-p}$ . The raw scores a classifier feeds to its sigmoid or softmax are called logits after it.
$\mathrm{logit}\left(\tfrac{1}{2}\right) = 0$
$\mathrm{softmax}(\mathbf{z})_i$ “softmax of z, entry i”
A preview: turns a vector of scores into a vector of probabilities. The $\sum$ in its formula is Chamber 5's, and Chamber 9 decodes it in full.
$\mathrm{softmax}(\mathbf{z})_i = \frac{e^{z_i}}{\sum_j e^{z_j}}$

Symbol	Say it	Meaning
$f : X \to Y$ $f : \mathbb{R} \to \mathbb{R}$	“f from X to Y”	A function named $f$ with domain $X$ (the allowed inputs) and codomain $Y$ (where outputs are promised to live). Every input gets exactly one output.
$x \mapsto x^2$ $f : x \mapsto x^2$	“x maps to x squared”	The rule: what happens to a single input. This arrow has a little tail; the plain arrow $\to$ connects whole sets.
$f(x)$ $f(3) = 9$	“f of x”	The output of $f$ at the input $x$ . The brackets mean “of”, not multiplication.
$f(X)$ $f(\mathbb{R}) = [0, \infty)$	“the image of X under f”	The set of outputs $f$ actually produces, $\{f(x) : x \in X\}$ . It sits inside the codomain and can be smaller.
$f(x; \theta)$ $f(\mathbf{x}; \theta)$	“f of x semicolon theta”	A model: the input $x$ before the semicolon, the parameters $\theta$ after it. Training changes $\theta$ ; prediction changes $x$ .
$f_\theta(x)$ $f_\theta(\mathbf{x})$	“f sub theta of x”	The same idea with the parameters as a subscript: one function for every choice of $\theta$ .
$f(x, y)$ $\hat{L}(N, D)$	“f of x and y”	A function of two inputs. Its domain is a Cartesian product such as $\mathbb{R} \times \mathbb{R}$ (Chamber 2).
$f \circ g$ $(f \circ g)(x) = f(g(x))$	“f composed with g”	“ $f$ after $g$ ”: apply $g$ first, then $f$ . The notation reads right to left.
$f(g(x))$ $\sigma(W\mathbf{x} + \mathbf{b})$	“f of g of x”	Nested brackets: the innermost function acts first, exactly as in code.
$\mathrm{id}$ $f \circ \mathrm{id} = f$	“the identity function”	The machine that changes nothing: $\mathrm{id}(x) = x$ . Composing with it leaves any function unchanged.
$f^{-1}$ $f^{-1} \circ f = \mathrm{id}$	“f inverse”	The function that undoes $f$ : $f^{-1}(f(x)) = x$ . It is not $1/f$ . It exists only when $f$ is a bijection.
$f_L \circ \cdots \circ f_1$ $f = f_L \circ \cdots \circ f_2 \circ f_1$	“f L after dots after f 1”	A deep network: $L$ layers applied in turn, $f_1$ first and $f_L$ last.
$\begin{cases} a & \text{if } P \\ b & \text{otherwise} \end{cases}$	“a if P, otherwise b”	A piecewise definition. The brace lists the cases, and exactly one applies to each input.
$\|x\|$ $\|-3\| = 3$	“the absolute value of x”	The size of $x$ without its sign: $x$ if $x \ge 0$ , and $-x$ if $x < 0$ .
$\lfloor x \rfloor$ $\lfloor 2.7 \rfloor = 2$	“the floor of x”	Round down to the nearest integer. Careful with negatives: $\lfloor -2.3 \rfloor = -3$ .
$\lceil x \rceil$ $\lceil 2.2 \rceil = 3$	“the ceiling of x”	Round up to the nearest integer.
$\max(a, b)$ $\mathrm{ReLU}(x) = \max(0, x)$	“the max of a and b”	The larger of the two numbers.
$\min(a, b)$ $\min(3, -1) = -1$	“the min of a and b”	The smaller of the two numbers.
$\mathbf{1}[P]$ $\mathbf{1}[\hat{y} = y]$	“the indicator of P”	1 if the statement $P$ is true, 0 if it's false. Also written $\mathbf{1}_{P}$ , $\mathbb{I}[P]$ or with a double-struck one.
$\delta_{ij}$ $\delta_{33} = 1,\ \delta_{34} = 0$	“Kronecker delta i j”	1 if $i = j$ and 0 otherwise: an indicator for “same index”.
$e$ $e \approx 2.718$	“e, Euler's number”	The constant $2.71828\ldots$ , the natural base for exponentials and logarithms.
$e^{x}$ $e^{0} = 1$	“e to the x”	The exponential function. Always positive, and it turns sums into products: $e^{a+b} = e^{a}e^{b}$ .
$\exp(x)$ $\exp\left(-\tfrac{x^2}{2}\right)$	“exp of x”	The same function written as a name, easier to read when the exponent is long.
$\log x$ $\log e^{x} = x$	“log x”	The inverse of exp. In ML papers a log with no base almost always means the natural log, base $e$ .
$\ln x$ $\ln e = 1$	“natural log of x”	The natural log, written explicitly. The same function as ML's $\log$ , and as Python's `math.log`.
$\log_2 x$ $\log_2 8 = 3$	“log base two of x”	How many times you must double 1 to reach $x$ . Information measured in bits uses base 2.
$\log_{10} x$ $\log_{10} 1000 = 3$	“log base ten of x”	How many times you must multiply 1 by ten to reach $x$ : roughly, the number of digits.
$\sigma(z)$ $\sigma(0) = \tfrac{1}{2}$	“sigma of z”	The sigmoid $\frac{1}{1 + e^{-z}}$ . It squashes any real number into $(0, 1)$ , so its output can be read as a probability.
$\tanh(z)$ $\tanh(0) = 0$	“tanh of z”	The hyperbolic tangent: squashes into $(-1, 1)$ . It is a sigmoid, stretched and shifted: $\tanh z = 2\sigma(2z) - 1$ .
$\mathrm{ReLU}(z)$ $\mathrm{ReLU}(-2) = 0$	“relu of z”	The rectified linear unit, $\max(0, z)$ : negative inputs become 0, positive ones pass through.
$\mathrm{softplus}(z)$ $\mathrm{softplus}(0) = \log 2$	“softplus of z”	$\log(1 + e^{z})$ : a smooth version of ReLU that is always positive. The Deep Learning book writes it $\zeta(z)$ .
$\mathrm{logit}(p)$ $\mathrm{logit}\left(\tfrac{1}{2}\right) = 0$	“logit of p”	The inverse of the sigmoid, $\log\frac{p}{1-p}$ . The raw scores a classifier feeds to its sigmoid or softmax are called logits after it.
$\mathrm{softmax}(\mathbf{z})_i$ $\mathrm{softmax}(\mathbf{z})_i = \frac{e^{z_i}}{\sum_j e^{z_j}}$	“softmax of z, entry i”	A preview: turns a vector of scores into a vector of probabilities. The $\sum$ in its formula is Chamber 5's, and Chamber 9 decodes it in full.

Mathematics for Machine Learning · Chamber 5

Σ and Π: Loops Written in Maths

$\sum_{i=1}^{n} x_i$ “the sum from i equals 1 to n of x sub i”
Add $x_i$ for $i = 1, 2, \ldots, n$ . Below the $\Sigma$ : the index and where it starts. Above: where it stops. Both ends are included.
$\sum_{i=1}^{3} i = 6$
$i$ “the index”
The dummy variable: a loop counter that exists only inside the sum. Renaming it changes nothing.
$\sum_{i=1}^{n} x_i = \sum_{k=1}^{n} x_k$
$\sum_{i} x_i$ “the sum over i of x sub i”
Bounds left out: add over every $i$ that makes sense in context, usually all the data.
$\sum_{i \in \mathcal{B}} x_i$ “the sum over i in B of x sub i”
Add over the elements of a set, such as a mini-batch $\mathcal{B}$ . There are $|\mathcal{B}|$ terms.
$\frac{1}{|\mathcal{B}|}\sum_{i \in \mathcal{B}} \ell_i$
$\sum_{j \ne i} x_j$ “the sum over j not equal to i”
A condition under the $\Sigma$ : add over every $j$ except $j = i$ .
$\sum_{j \ne i} x_j = \sum_j x_j - x_i$
$\sum_{i=1}^{m}\sum_{j=1}^{n} a_{ij}$ “the double sum over i and j of a sub i j”
A sum of sums: two nested loops over a grid of $mn$ terms. For finite sums the two sums can be swapped.
$x_1 + \cdots + x_n$ “x 1 plus dots plus x n”
An ellipsis: “and so on, following the pattern”. Centred dots $\cdots$ go between operations, low dots $\ldots$ in lists.
$1 + 2 + \cdots + n$
$\prod_{i=1}^{n} x_i$ “the product from i equals 1 to n of x sub i”
Multiply the terms instead of adding them. An empty product is 1.
$\prod_{i=1}^{3} x_i = x_1 x_2 x_3$
$n!$ “n factorial”
$1 \cdot 2 \cdots n = \prod_{k=1}^{n} k$ , the number of ways to put $n$ things in order. By the empty-product rule, $0! = 1$ .
$5! = 120$
$\log \prod_i p_i$ “the log of the product of p sub i”
Equals $\sum_i \log p_i$ : a product of many small probabilities becomes a sum that a computer can store.
$\log \prod_i p_i = \sum_i \log p_i$
$\mu$ “mu, the mean”
The average, $\frac{1}{n}\sum_i x_i$ . Also written $\bar{x}$ .
$\mu = \tfrac{1}{n}\sum_{i} x_i$
$\sigma^2$ “sigma squared, the variance”
The average squared distance from the mean: how spread out the values are.
$\sigma^2 = \tfrac{1}{n}\sum_{i} (x_i - \mu)^2$
$\sigma$ “sigma, the standard deviation”
$\sqrt{\sigma^2}$ : the spread in the same units as the data.
$\sigma = \sqrt{\sigma^2}$
$\mu_{\mathcal{B}}, \sigma^2_{\mathcal{B}}$ “mu B and sigma squared B”
The mean and variance of one mini-batch $\mathcal{B}$ , as in batch normalisation.
$\widehat{x}_i$ “x hat sub i, normalised”
In batch normalisation, $x_i$ minus the batch mean, divided by the batch standard deviation: how many standard deviations $x_i$ sits from the mean.

Symbol	Say it	Meaning
$\sum_{i=1}^{n} x_i$ $\sum_{i=1}^{3} i = 6$	“the sum from i equals 1 to n of x sub i”	Add $x_i$ for $i = 1, 2, \ldots, n$ . Below the $\Sigma$ : the index and where it starts. Above: where it stops. Both ends are included.
$i$ $\sum_{i=1}^{n} x_i = \sum_{k=1}^{n} x_k$	“the index”	The dummy variable: a loop counter that exists only inside the sum. Renaming it changes nothing.
$\sum_{i} x_i$	“the sum over i of x sub i”	Bounds left out: add over every $i$ that makes sense in context, usually all the data.
$\sum_{i \in \mathcal{B}} x_i$ $\frac{1}{\|\mathcal{B}\|}\sum_{i \in \mathcal{B}} \ell_i$	“the sum over i in B of x sub i”	Add over the elements of a set, such as a mini-batch $\mathcal{B}$ . There are $\|\mathcal{B}\|$ terms.
$\sum_{j \ne i} x_j$ $\sum_{j \ne i} x_j = \sum_j x_j - x_i$	“the sum over j not equal to i”	A condition under the $\Sigma$ : add over every $j$ except $j = i$ .
$\sum_{i=1}^{m}\sum_{j=1}^{n} a_{ij}$	“the double sum over i and j of a sub i j”	A sum of sums: two nested loops over a grid of $mn$ terms. For finite sums the two sums can be swapped.
$x_1 + \cdots + x_n$ $1 + 2 + \cdots + n$	“x 1 plus dots plus x n”	An ellipsis: “and so on, following the pattern”. Centred dots $\cdots$ go between operations, low dots $\ldots$ in lists.
$\prod_{i=1}^{n} x_i$ $\prod_{i=1}^{3} x_i = x_1 x_2 x_3$	“the product from i equals 1 to n of x sub i”	Multiply the terms instead of adding them. An empty product is 1.
$n!$ $5! = 120$	“n factorial”	$1 \cdot 2 \cdots n = \prod_{k=1}^{n} k$ , the number of ways to put $n$ things in order. By the empty-product rule, $0! = 1$ .
$\log \prod_i p_i$ $\log \prod_i p_i = \sum_i \log p_i$	“the log of the product of p sub i”	Equals $\sum_i \log p_i$ : a product of many small probabilities becomes a sum that a computer can store.
$\mu$ $\mu = \tfrac{1}{n}\sum_{i} x_i$	“mu, the mean”	The average, $\frac{1}{n}\sum_i x_i$ . Also written $\bar{x}$ .
$\sigma^2$ $\sigma^2 = \tfrac{1}{n}\sum_{i} (x_i - \mu)^2$	“sigma squared, the variance”	The average squared distance from the mean: how spread out the values are.
$\sigma$ $\sigma = \sqrt{\sigma^2}$	“sigma, the standard deviation”	$\sqrt{\sigma^2}$ : the spread in the same units as the data.
$\mu_{\mathcal{B}}, \sigma^2_{\mathcal{B}}$	“mu B and sigma squared B”	The mean and variance of one mini-batch $\mathcal{B}$ , as in batch normalisation.
$\widehat{x}_i$	“x hat sub i, normalised”	In batch normalisation, $x_i$ minus the batch mean, divided by the batch standard deviation: how many standard deviations $x_i$ sits from the mean.

Mathematics for Machine Learning · Chamber 6

Vectors and Matrices: Reading the Shapes

$\mathbf{x} \in \mathbb{R}^n$ “x in R n”
$\mathbf{x}$ is a list of $n$ real numbers: a vector with $n$ entries. Unless a paper says otherwise, it stands as a column, $n \times 1$ .
$\mathbf{x} \in \mathbb{R}^{3}$
$x_i$ “x sub i”
The $i$ -th entry of $\mathbf{x}$ . It's a single number, so it's printed plain, not bold.
$\mathbf{x} = (x_1, x_2, x_3)^\top$
$\mathbf{x}^\top$ “x transpose”
The same numbers laid out as a row, $1 \times n$ . Also printed $\mathbf{x}^T$ , $\mathbf{x}^{\mathsf{T}}$ or $\mathbf{x}'$ . It is never a power.
$\mathbf{x}^\top = (x_1, \ldots, x_n)$
$\mathbf{0}$ “the zero vector”
Every entry is $0$ . Its length comes from context, or a subscript: $\mathbf{0}_n$ .
$\mathbf{x} + \mathbf{0} = \mathbf{x}$
$\mathbf{1}$ “the ones vector”
Every entry is $1$ . Handy for sums: $\mathbf{1}^\top \mathbf{x} = x_1 + \cdots + x_n$ .
$\mathbf{1}^\top \mathbf{x} = \sum_{i} x_i$
$\mathbf{e}_i$ “e sub i”
The $i$ -th standard basis vector: zeros everywhere except a $1$ in position $i$ . In ML it's a one-hot vector.
$\mathbf{e}_3 = (0, 0, 1, 0)^\top \in \mathbb{R}^4$
$\|\mathbf{x}\|_2$ “the two-norm of x”
The ordinary length, $\sqrt{x_1^2 + \cdots + x_n^2}$ . Often written just $\|\mathbf{x}\|$ .
$\|(3, 4)\|_2 = 5$
$\|\mathbf{x}\|_1$ “the one-norm of x”
Add up the sizes of the entries: $|x_1| + \cdots + |x_n|$ .
$\|(3, -4)\|_1 = 7$
$\|\mathbf{x}\|_2^2$ “the squared two-norm of x”
$\sum_i x_i^2$ , with no square root. Squared errors and weight penalties are written this way.
$\|\mathbf{x}\|_2^2 = \mathbf{x}^\top \mathbf{x}$
$\langle \mathbf{x}, \mathbf{y} \rangle$ “the inner product of x and y, in angle brackets”
Multiply matching entries and add: $\sum_i x_i y_i$ . Both vectors need the same length.
$\langle \mathbf{x}, \mathbf{y} \rangle = \sum_{i=1}^{n} x_i y_i$
$\mathbf{x}^\top \mathbf{y}$ “x transpose y”
The same number as $\langle \mathbf{x}, \mathbf{y} \rangle$ , written as a matrix product: a $1 \times n$ row times an $n \times 1$ column.
$\mathbf{w}^\top \mathbf{x} + b$
$\mathbf{x} \cdot \mathbf{y}$ “x dot y”
A third name for the same number, the dot product. Physics and school books prefer this one.
$q \cdot k = \sum_{i=1}^{d_k} q_i k_i$
$A \in \mathbb{R}^{m \times n}$ “A is an m by n matrix”
A grid of real numbers with $m$ rows and $n$ columns: $mn$ numbers in all. Rows first, always.
$W \in \mathbb{R}^{512 \times 64}$
$A_{ij}$ “A sub i j”
The entry in row $i$ , column $j$ . Also written $A_{i,j}$ , $a_{ij}$ or $[A]_{ij}$ .
$A_{23}$
$A_{i,:}$ “row i of A”
The whole $i$ -th row, a $1 \times n$ row. The colon means “every index here”, just as in NumPy.
$A_{2,:}$
$A_{:,j}$ “column j of A”
The whole $j$ -th column, an $m \times 1$ column.
$A_{:,1}$
$A^\top$ “A transpose”
Flip across the diagonal: $(A^\top)_{ij} = A_{ji}$ . Rows become columns, so an $m \times n$ matrix becomes $n \times m$ .
$(A^\top)_{ij} = A_{ji}$
$I$ “the identity matrix”
Square, with ones on the diagonal and zeros elsewhere, so $I\mathbf{x} = \mathbf{x}$ . Written $I_n$ when the size matters.
$I_3$
$\mathrm{diag}(\mathbf{d})$ “diag of d”
The square matrix with the entries of $\mathbf{d}$ down its diagonal and zeros everywhere else.
$\mathrm{diag}(2, 5) = \begin{bmatrix} 2 & 0 \\ 0 & 5 \end{bmatrix}$
$X \in \mathbb{R}^{n \times d}$ “the data matrix X”
A dataset of $n$ examples with $d$ features each: one example per row, one feature per column.
$X_{i,:} = (\mathbf{x}^{(i)})^\top$
$A\mathbf{x}$ “A times x”
A new vector whose $i$ -th entry is row $i$ of $A$ dotted with $\mathbf{x}$ . Needs $A$ 's column count to equal $\mathbf{x}$ 's length.
$(A\mathbf{x})_i = \sum_{j=1}^{n} A_{ij} x_j$
$AB$ “A times B”
$(AB)_{ij} = \sum_k A_{ik} B_{kj}$ . Shapes: $(m \times n)(n \times p) = m \times p$ . The inner sizes must match; the outer ones survive.
$(AB)_{ij} = \sum_{k=1}^{n} A_{ik} B_{kj}$
$A \odot B$ “A Hadamard B”
Multiply entry by entry: $(A \odot B)_{ij} = A_{ij} B_{ij}$ . Both must have the same shape. Also called the element-wise product.
$(A \odot B)_{ij} = A_{ij} B_{ij}$
$A + B$ “A plus B”
Add entry by entry. As with $\odot$ , both must have exactly the same shape.
$(A + B)_{ij} = A_{ij} + B_{ij}$
$\mathbf{x}\mathbf{y}^\top$ “the outer product of x and y”
A column times a row makes a whole matrix: $(\mathbf{x}\mathbf{y}^\top)_{ij} = x_i y_j$ . For $\mathbf{x} \in \mathbb{R}^m$ and $\mathbf{y} \in \mathbb{R}^n$ it is $m \times n$ .
$(\mathbf{x}\mathbf{y}^\top)_{ij} = x_i y_j$
$(AB)^\top = B^\top A^\top$ “the transpose of A B is B transpose A transpose”
Transposing a product reverses the order of the factors. Proved in this chamber, entry by entry.
$X \in \mathbb{R}^{B \times n \times d}$ “X is B by n by d”
A tensor: a grid with three indices. Here, a batch of $B$ sequences, each of $n$ tokens, each token a $d$ -dimensional vector.
$X \in \mathbb{R}^{16 \times 64 \times 256}$
$X_{b,:,:}$ “X b colon colon”
Fix the first index and keep the rest: example $b$ of the batch, an $n \times d$ matrix.
$X_{3,:,:} \in \mathbb{R}^{64 \times 256}$
$A + \mathbf{b}$ “A plus b, broadcast”
Deep-learning shorthand (and NumPy behaviour): add the vector $\mathbf{b}$ to every row of $A$ . Strict matrix algebra would call it undefined.
$C_{ij} = A_{ij} + b_j$

Symbol	Say it	Meaning
$\mathbf{x} \in \mathbb{R}^n$ $\mathbf{x} \in \mathbb{R}^{3}$	“x in R n”	$\mathbf{x}$ is a list of $n$ real numbers: a vector with $n$ entries. Unless a paper says otherwise, it stands as a column, $n \times 1$ .
$x_i$ $\mathbf{x} = (x_1, x_2, x_3)^\top$	“x sub i”	The $i$ -th entry of $\mathbf{x}$ . It's a single number, so it's printed plain, not bold.
$\mathbf{x}^\top$ $\mathbf{x}^\top = (x_1, \ldots, x_n)$	“x transpose”	The same numbers laid out as a row, $1 \times n$ . Also printed $\mathbf{x}^T$ , $\mathbf{x}^{\mathsf{T}}$ or $\mathbf{x}'$ . It is never a power.
$\mathbf{0}$ $\mathbf{x} + \mathbf{0} = \mathbf{x}$	“the zero vector”	Every entry is $0$ . Its length comes from context, or a subscript: $\mathbf{0}_n$ .
$\mathbf{1}$ $\mathbf{1}^\top \mathbf{x} = \sum_{i} x_i$	“the ones vector”	Every entry is $1$ . Handy for sums: $\mathbf{1}^\top \mathbf{x} = x_1 + \cdots + x_n$ .
$\mathbf{e}_i$ $\mathbf{e}_3 = (0, 0, 1, 0)^\top \in \mathbb{R}^4$	“e sub i”	The $i$ -th standard basis vector: zeros everywhere except a $1$ in position $i$ . In ML it's a one-hot vector.
$\\|\mathbf{x}\\|_2$ $\\|(3, 4)\\|_2 = 5$	“the two-norm of x”	The ordinary length, $\sqrt{x_1^2 + \cdots + x_n^2}$ . Often written just $\\|\mathbf{x}\\|$ .
$\\|\mathbf{x}\\|_1$ $\\|(3, -4)\\|_1 = 7$	“the one-norm of x”	Add up the sizes of the entries: $\|x_1\| + \cdots + \|x_n\|$ .
$\\|\mathbf{x}\\|_2^2$ $\\|\mathbf{x}\\|_2^2 = \mathbf{x}^\top \mathbf{x}$	“the squared two-norm of x”	$\sum_i x_i^2$ , with no square root. Squared errors and weight penalties are written this way.
$\langle \mathbf{x}, \mathbf{y} \rangle$ $\langle \mathbf{x}, \mathbf{y} \rangle = \sum_{i=1}^{n} x_i y_i$	“the inner product of x and y, in angle brackets”	Multiply matching entries and add: $\sum_i x_i y_i$ . Both vectors need the same length.
$\mathbf{x}^\top \mathbf{y}$ $\mathbf{w}^\top \mathbf{x} + b$	“x transpose y”	The same number as $\langle \mathbf{x}, \mathbf{y} \rangle$ , written as a matrix product: a $1 \times n$ row times an $n \times 1$ column.
$\mathbf{x} \cdot \mathbf{y}$ $q \cdot k = \sum_{i=1}^{d_k} q_i k_i$	“x dot y”	A third name for the same number, the dot product. Physics and school books prefer this one.
$A \in \mathbb{R}^{m \times n}$ $W \in \mathbb{R}^{512 \times 64}$	“A is an m by n matrix”	A grid of real numbers with $m$ rows and $n$ columns: $mn$ numbers in all. Rows first, always.
$A_{ij}$ $A_{23}$	“A sub i j”	The entry in row $i$ , column $j$ . Also written $A_{i,j}$ , $a_{ij}$ or $[A]_{ij}$ .
$A_{i,:}$ $A_{2,:}$	“row i of A”	The whole $i$ -th row, a $1 \times n$ row. The colon means “every index here”, just as in NumPy.
$A_{:,j}$ $A_{:,1}$	“column j of A”	The whole $j$ -th column, an $m \times 1$ column.
$A^\top$ $(A^\top)_{ij} = A_{ji}$	“A transpose”	Flip across the diagonal: $(A^\top)_{ij} = A_{ji}$ . Rows become columns, so an $m \times n$ matrix becomes $n \times m$ .
$I$ $I_3$	“the identity matrix”	Square, with ones on the diagonal and zeros elsewhere, so $I\mathbf{x} = \mathbf{x}$ . Written $I_n$ when the size matters.
$\mathrm{diag}(\mathbf{d})$ $\mathrm{diag}(2, 5) = \begin{bmatrix} 2 & 0 \\ 0 & 5 \end{bmatrix}$	“diag of d”	The square matrix with the entries of $\mathbf{d}$ down its diagonal and zeros everywhere else.
$X \in \mathbb{R}^{n \times d}$ $X_{i,:} = (\mathbf{x}^{(i)})^\top$	“the data matrix X”	A dataset of $n$ examples with $d$ features each: one example per row, one feature per column.
$A\mathbf{x}$ $(A\mathbf{x})_i = \sum_{j=1}^{n} A_{ij} x_j$	“A times x”	A new vector whose $i$ -th entry is row $i$ of $A$ dotted with $\mathbf{x}$ . Needs $A$ 's column count to equal $\mathbf{x}$ 's length.
$AB$ $(AB)_{ij} = \sum_{k=1}^{n} A_{ik} B_{kj}$	“A times B”	$(AB)_{ij} = \sum_k A_{ik} B_{kj}$ . Shapes: $(m \times n)(n \times p) = m \times p$ . The inner sizes must match; the outer ones survive.
$A \odot B$ $(A \odot B)_{ij} = A_{ij} B_{ij}$	“A Hadamard B”	Multiply entry by entry: $(A \odot B)_{ij} = A_{ij} B_{ij}$ . Both must have the same shape. Also called the element-wise product.
$A + B$ $(A + B)_{ij} = A_{ij} + B_{ij}$	“A plus B”	Add entry by entry. As with $\odot$ , both must have exactly the same shape.
$\mathbf{x}\mathbf{y}^\top$ $(\mathbf{x}\mathbf{y}^\top)_{ij} = x_i y_j$	“the outer product of x and y”	A column times a row makes a whole matrix: $(\mathbf{x}\mathbf{y}^\top)_{ij} = x_i y_j$ . For $\mathbf{x} \in \mathbb{R}^m$ and $\mathbf{y} \in \mathbb{R}^n$ it is $m \times n$ .
$(AB)^\top = B^\top A^\top$	“the transpose of A B is B transpose A transpose”	Transposing a product reverses the order of the factors. Proved in this chamber, entry by entry.
$X \in \mathbb{R}^{B \times n \times d}$ $X \in \mathbb{R}^{16 \times 64 \times 256}$	“X is B by n by d”	A tensor: a grid with three indices. Here, a batch of $B$ sequences, each of $n$ tokens, each token a $d$ -dimensional vector.
$X_{b,:,:}$ $X_{3,:,:} \in \mathbb{R}^{64 \times 256}$	“X b colon colon”	Fix the first index and keep the rest: example $b$ of the batch, an $n \times d$ matrix.
$A + \mathbf{b}$ $C_{ij} = A_{ij} + b_j$	“A plus b, broadcast”	Deep-learning shorthand (and NumPy behaviour): add the vector $\mathbf{b}$ to every row of $A$ . Strict matrix algebra would call it undefined.

Mathematics for Machine Learning · Chamber 7

Change: The Notation of Calculus

$\Delta x$ “delta x”
A finite change in $x$ : new value minus old value. Capital delta means “difference” (Chamber 1).
$\Delta x = x_{\text{new}} - x_{\text{old}}$
$\frac{\Delta y}{\Delta x}$ “delta y over delta x”
Rise over run: the slope of the straight line (the secant) through two points of a curve.
$\frac{\Delta y}{\Delta x} = \frac{f(x_0 + h) - f(x_0)}{h}$
$\lim_{h \to 0}$ “the limit as h tends to zero”
The value that the expression after it approaches as $h$ gets closer and closer to $0$ , without ever being set to $0$ .
$\lim_{h \to 0} (6 + h) = 6$
$dx$ “d x”
What $\Delta x$ becomes in the limit: an infinitesimally small change in $x$ . Thompson's Calculus Made Easy calls it “a little bit of $x$ ”.
$\frac{dy}{dx}$
$\frac{dy}{dx}$ “d y by d x”
Leibniz's notation for the derivative of $y$ with respect to $x$ : the limit of $\frac{\Delta y}{\Delta x}$ . It often behaves like a fraction, but it is one symbol.
$y = x^2 \implies \frac{dy}{dx} = 2x$
$f'(x)$ “f prime of x”
Lagrange's notation: $f'$ is the derivative of $f$ , a new function, and $f'(x)$ is its value at $x$ .
$f(x) = x^3 \implies f'(x) = 3x^2$
$\frac{d}{dx}$ “d by d x”
An operator: “take the derivative, with respect to $x$ , of whatever follows”.
$\frac{d}{dx}\, e^x = e^x$
$f''(x)$ “f double prime of x”
The second derivative, the derivative of the derivative. It measures how the slope itself changes: curvature. Leibniz writes it $\frac{d^2 y}{dx^2}$ .
$f(x) = x^4 \implies f''(x) = 12x^2$
$\left.\frac{df}{dx}\right|_{x=a}$ “d f by d x, evaluated at a”
The derivative, evaluated at the point $x = a$ . The same number as $f'(a)$ .
$\left.\frac{d}{dx} x^2\right|_{x=3} = 6$
$\frac{dy}{dx} = \frac{dy}{du}\,\frac{du}{dx}$ “the chain rule”
Differentiating a function of a function: multiply the rates. If $y$ depends on $u$ and $u$ on $x$ , the rates chain together. Backpropagation is this rule, applied over and over.
$\frac{\partial f}{\partial x_i}$ “partial f by partial x i”
The partial derivative: how $f$ changes when only $x_i$ moves and every other input is held fixed. The curly $\partial$ warns you there are other inputs.
$f = x_1^2 x_2 \implies \frac{\partial f}{\partial x_1} = 2x_1 x_2$
$\nabla f$ “grad f”
The gradient: every partial derivative of $f$ stacked into a vector, the same shape as the input. $\nabla$ is called nabla.
$\nabla f = \left(\frac{\partial f}{\partial x_1}, \ldots, \frac{\partial f}{\partial x_n}\right)^\top$
$\nabla_\theta \mathcal{L}$ “grad theta of L”
The gradient of the loss with respect to the parameters $\theta$ only. The subscript names the variables being nudged; everything else is held fixed.
$\theta \leftarrow \theta - \eta\, \nabla_\theta \mathcal{L}(\theta)$
$J_{\mathbf{f}}$ “the Jacobian of f”
For $\mathbf{f} : \mathbb{R}^n \to \mathbb{R}^m$ , the $m \times n$ matrix of all first partials, $J_{ij} = \frac{\partial f_i}{\partial x_j}$ . Also written $\frac{\partial \mathbf{f}}{\partial \mathbf{x}}$ .
$J_{ij} = \frac{\partial f_i}{\partial x_j}$
$H_f$ “the Hessian of f”
For $f : \mathbb{R}^n \to \mathbb{R}$ , the $n \times n$ matrix of second partials, $H_{ij} = \frac{\partial^2 f}{\partial x_i\, \partial x_j}$ . Also written $\nabla^2 f$ .
$H_{ij} = \frac{\partial^2 f}{\partial x_i\, \partial x_j}$
$\int_a^b f(x)\,dx$ “the integral from a to b of f of x, d x”
A continuous sum: add up height $f(x)$ times a little bit of width $dx$ , for every $x$ from $a$ to $b$ . The area under the curve.
$\int_0^1 x\,dx = \tfrac{1}{2}$
$\sum_{i} f(x_i)\,\Delta x$ “the Riemann sum”
The finite version: cut $[a, b]$ into strips of width $\Delta x$ and add up $f(x_i)\,\Delta x$ over the strips. As $\Delta x \to 0$ it becomes the integral, and $\Sigma$ turns into $\int$ .
$\min_{x} f(x)$ “the minimum over x of f”
The smallest value that $f$ takes: a number on the output side.
$\min_{x} (x + 2)^2 + 5 = 5$
$\argmin_{x} f(x)$ “the arg min over x of f”
The input at which $f$ is smallest. Read it “the $x$ that minimises $f$ ”.
$\argmin_{x} (x + 2)^2 + 5 = -2$
$\argmax_{k} p_k$ “the arg max over k of p k”
The index of the largest entry. A classifier's predicted class is $\argmax_k$ of its predicted probabilities.
$\argmax_{k} (0.1, 0.7, 0.2) = 2$
$\theta^\star = \argmin_{\theta} \mathcal{L}(\theta)$ “theta star is the arg min of L”
The best parameters: the $\theta$ with the lowest loss. The star marks the optimum, as in Chamber 1.
$\sup$ “the supremum”
The least upper bound. Like $\max$ , but it exists even when no element reaches it.
$\sup\left\{1 - \tfrac{1}{n} : n = 1, 2, 3, \ldots\right\} = 1$
$f(x + h) \approx f(x) + f'(x)\,h$ “the first-order approximation”
Close to $x$ , a smooth function is almost a straight line through $(x, f(x))$ with slope $f'(x)$ .
$e^{0.1} \approx e^0 + e^0 \cdot 0.1 = 1.1$
$O(h)$ “big O of h”
Some quantity no bigger than a constant times $h$ , once $h$ is small enough. It hides the constant to show the rate: halve $h$ , roughly halve the error.
$\frac{f(x+h) - f(x)}{h} = f'(x) + O(h)$
$O(n^2)$ “big O of n squared”
A cost that grows at most like $n^2$ for large $n$ : double $n$ and the work roughly quadruples. Self-attention's cost in the sequence length $n$ is quoted this way.
$O(n^2 \cdot d)$

Symbol	Say it	Meaning
$\Delta x$ $\Delta x = x_{\text{new}} - x_{\text{old}}$	“delta x”	A finite change in $x$ : new value minus old value. Capital delta means “difference” (Chamber 1).
$\frac{\Delta y}{\Delta x}$ $\frac{\Delta y}{\Delta x} = \frac{f(x_0 + h) - f(x_0)}{h}$	“delta y over delta x”	Rise over run: the slope of the straight line (the secant) through two points of a curve.
$\lim_{h \to 0}$ $\lim_{h \to 0} (6 + h) = 6$	“the limit as h tends to zero”	The value that the expression after it approaches as $h$ gets closer and closer to $0$ , without ever being set to $0$ .
$dx$ $\frac{dy}{dx}$	“d x”	What $\Delta x$ becomes in the limit: an infinitesimally small change in $x$ . Thompson's Calculus Made Easy calls it “a little bit of $x$ ”.
$\frac{dy}{dx}$ $y = x^2 \implies \frac{dy}{dx} = 2x$	“d y by d x”	Leibniz's notation for the derivative of $y$ with respect to $x$ : the limit of $\frac{\Delta y}{\Delta x}$ . It often behaves like a fraction, but it is one symbol.
$f'(x)$ $f(x) = x^3 \implies f'(x) = 3x^2$	“f prime of x”	Lagrange's notation: $f'$ is the derivative of $f$ , a new function, and $f'(x)$ is its value at $x$ .
$\frac{d}{dx}$ $\frac{d}{dx}\, e^x = e^x$	“d by d x”	An operator: “take the derivative, with respect to $x$ , of whatever follows”.
$f''(x)$ $f(x) = x^4 \implies f''(x) = 12x^2$	“f double prime of x”	The second derivative, the derivative of the derivative. It measures how the slope itself changes: curvature. Leibniz writes it $\frac{d^2 y}{dx^2}$ .
$\left.\frac{df}{dx}\right\|_{x=a}$ $\left.\frac{d}{dx} x^2\right\|_{x=3} = 6$	“d f by d x, evaluated at a”	The derivative, evaluated at the point $x = a$ . The same number as $f'(a)$ .
$\frac{dy}{dx} = \frac{dy}{du}\,\frac{du}{dx}$	“the chain rule”	Differentiating a function of a function: multiply the rates. If $y$ depends on $u$ and $u$ on $x$ , the rates chain together. Backpropagation is this rule, applied over and over.
$\frac{\partial f}{\partial x_i}$ $f = x_1^2 x_2 \implies \frac{\partial f}{\partial x_1} = 2x_1 x_2$	“partial f by partial x i”	The partial derivative: how $f$ changes when only $x_i$ moves and every other input is held fixed. The curly $\partial$ warns you there are other inputs.
$\nabla f$ $\nabla f = \left(\frac{\partial f}{\partial x_1}, \ldots, \frac{\partial f}{\partial x_n}\right)^\top$	“grad f”	The gradient: every partial derivative of $f$ stacked into a vector, the same shape as the input. $\nabla$ is called nabla.
$\nabla_\theta \mathcal{L}$ $\theta \leftarrow \theta - \eta\, \nabla_\theta \mathcal{L}(\theta)$	“grad theta of L”	The gradient of the loss with respect to the parameters $\theta$ only. The subscript names the variables being nudged; everything else is held fixed.
$J_{\mathbf{f}}$ $J_{ij} = \frac{\partial f_i}{\partial x_j}$	“the Jacobian of f”	For $\mathbf{f} : \mathbb{R}^n \to \mathbb{R}^m$ , the $m \times n$ matrix of all first partials, $J_{ij} = \frac{\partial f_i}{\partial x_j}$ . Also written $\frac{\partial \mathbf{f}}{\partial \mathbf{x}}$ .
$H_f$ $H_{ij} = \frac{\partial^2 f}{\partial x_i\, \partial x_j}$	“the Hessian of f”	For $f : \mathbb{R}^n \to \mathbb{R}$ , the $n \times n$ matrix of second partials, $H_{ij} = \frac{\partial^2 f}{\partial x_i\, \partial x_j}$ . Also written $\nabla^2 f$ .
$\int_a^b f(x)\,dx$ $\int_0^1 x\,dx = \tfrac{1}{2}$	“the integral from a to b of f of x, d x”	A continuous sum: add up height $f(x)$ times a little bit of width $dx$ , for every $x$ from $a$ to $b$ . The area under the curve.
$\sum_{i} f(x_i)\,\Delta x$	“the Riemann sum”	The finite version: cut $[a, b]$ into strips of width $\Delta x$ and add up $f(x_i)\,\Delta x$ over the strips. As $\Delta x \to 0$ it becomes the integral, and $\Sigma$ turns into $\int$ .
$\min_{x} f(x)$ $\min_{x} (x + 2)^2 + 5 = 5$	“the minimum over x of f”	The smallest value that $f$ takes: a number on the output side.
$\argmin_{x} f(x)$ $\argmin_{x} (x + 2)^2 + 5 = -2$	“the arg min over x of f”	The input at which $f$ is smallest. Read it “the $x$ that minimises $f$ ”.
$\argmax_{k} p_k$ $\argmax_{k} (0.1, 0.7, 0.2) = 2$	“the arg max over k of p k”	The index of the largest entry. A classifier's predicted class is $\argmax_k$ of its predicted probabilities.
$\theta^\star = \argmin_{\theta} \mathcal{L}(\theta)$	“theta star is the arg min of L”	The best parameters: the $\theta$ with the lowest loss. The star marks the optimum, as in Chamber 1.
$\sup$ $\sup\left\{1 - \tfrac{1}{n} : n = 1, 2, 3, \ldots\right\} = 1$	“the supremum”	The least upper bound. Like $\max$ , but it exists even when no element reaches it.
$f(x + h) \approx f(x) + f'(x)\,h$ $e^{0.1} \approx e^0 + e^0 \cdot 0.1 = 1.1$	“the first-order approximation”	Close to $x$ , a smooth function is almost a straight line through $(x, f(x))$ with slope $f'(x)$ .
$O(h)$ $\frac{f(x+h) - f(x)}{h} = f'(x) + O(h)$	“big O of h”	Some quantity no bigger than a constant times $h$ , once $h$ is small enough. It hides the constant to show the rate: halve $h$ , roughly halve the error.
$O(n^2)$ $O(n^2 \cdot d)$	“big O of n squared”	A cost that grows at most like $n^2$ for large $n$ : double $n$ and the work roughly quadruples. Self-attention's cost in the sequence length $n$ is quoted this way.

Mathematics for Machine Learning · Chamber 8

Chance: The Notation of Probability

$\Omega$ “capital omega, the sample space”
The sample space: the set of every possible outcome of the experiment.
$\Omega = \{1, 2, 3, 4, 5, 6\}$
$A \subseteq \Omega$ “A, an event”
An event is a set of outcomes, so a subset of $\Omega$ . “The roll is even” is the event $\{2, 4, 6\}$ .
$A = \{2, 4, 6\}$
$P(A)$ “P of A”
The probability of event $A$ : a number from 0 (impossible) to 1 (certain).
$P(\text{even}) = \tfrac{1}{2}$
$P(A^c)$ “P of A complement”
The probability that $A$ does not happen. It always equals $1 - P(A)$ .
$P(A^c) = 1 - P(A)$
$P(A \cap B)$ “P of A and B”
The probability that both happen. Often written with a comma instead: $P(A, B)$ .
$P(A \cap B) = P(A, B)$
$P(A \cup B)$ “P of A or B”
The probability that at least one happens. Inclusion–exclusion from Chamber 2 gives the formula.
$P(A \cup B) = P(A) + P(B) - P(A \cap B)$
$P(A \mid B)$ “P of A given B”
Conditional probability: how likely $A$ is once you know $B$ happened. The bar is read “given”.
$P(A \mid B) = \frac{P(A \cap B)}{P(B)}$
$X$ “random variable X”
A capital letter is a random variable: a number whose value depends on chance, like the result of a roll.
$X = \text{the number rolled}$
$P(X = x)$ “P of big X equals little x”
The probability that the random variable $X$ takes the particular value $x$ . Capital: the random quantity. Lower case: one value it could take.
$P(X = 6) = \tfrac{1}{6}$
$p(x)$ “p of x”
The distribution of $X$ at the value $x$ . For discrete $X$ it's a pmf, $p(x) = P(X = x)$ . For continuous $X$ it's a density: areas under it are probabilities.
$p(6) = \tfrac{1}{6}$
$p(x, y)$ “p of x comma y”
The joint distribution: how likely $X = x$ and $Y = y$ are together.
$p(\text{rain}, \text{umbrella}) = 0.3$
$p(y \mid x)$ “p of y given x”
The conditional distribution of $Y$ once $X = x$ is known: $p(x, y) / p(x)$ .
$p(y \mid x) = \frac{p(x, y)}{p(x)}$
$p_\theta(y \mid x)$ “p theta of y given x”
A model: the probability that a network with parameters $\theta$ assigns to label $y$ for input $x$ .
$p_\theta(\text{cat} \mid \text{image}) = 0.92$
$\sum_{y} p(x, y)$ “sum over y of p of x comma y”
Marginalising: add up the joint over every value of $y$ and you're left with $p(x)$ . People say $y$ has been “summed out”.
$p(x) = \sum_{y} p(x, y)$
$\int p(x, y)\, dy$ “integral of p of x comma y, d y”
The same marginalisation when $y$ is continuous: the integral is a continuous sum (Chamber 7).
$p(x) = \int p(x, y)\, dy$
$X \perp\!\!\!\perp Y$ “X is independent of Y”
Independence: knowing one tells you nothing about the other, so the joint factorises: $p(x, y) = p(x)\,p(y)$ .
$p(x, y) = p(x)\,p(y)$
$\sim$ “is drawn from”
Between a variable and a distribution, $\sim$ says the variable is a sample from it (“is distributed as”). Not to be confused with $\approx$ .
$x \sim p$
$\overset{\text{i.i.d.}}{\sim}$ “independent and identically distributed as”
Every sample comes from the same distribution, and none depends on any other. The standing assumption about datasets.
$x_1, \ldots, x_n \overset{\text{i.i.d.}}{\sim} p$
$\mathrm{Bernoulli}(p)$ “Bernoulli p”
A biased coin: the value 1 with probability $p$ and 0 otherwise. Dropout's masks are Bernoulli.
$r \sim \mathrm{Bernoulli}(0.8)$
$\mathcal{N}(\mu, \sigma^2)$ “normal with mean mu and variance sigma squared”
The bell curve centred at $\mu$ . The second slot is the variance $\sigma^2$ , not the standard deviation.
$\varepsilon \sim \mathcal{N}(0, 1)$
$\mathcal{U}(a, b)$ “uniform from a to b”
Every value between $a$ and $b$ equally likely. A common way to initialise weights.
$u \sim \mathcal{U}(0, 1)$
$\mathbb{E}[X]$ “the expectation of X”
The expected value: the average of $X$ , weighting each value by its probability. The long-run mean of many samples.
$\mathbb{E}[X] = \sum_x x\, p(x)$
$\mathbb{E}_{x \sim p}[f(x)]$ “expectation of f of x, x drawn from p”
The average of $f(x)$ when $x$ is drawn from $p$ . The subscript names the random variable and its distribution.
$\mathbb{E}_{x \sim p}[f(x)] = \sum_x f(x)\, p(x)$
$\mathrm{Var}(X)$ “the variance of X”
How spread out $X$ is: the expected squared distance from its mean. Its square root is the standard deviation $\sigma$ .
$\mathrm{Var}(X) = \mathbb{E}[(X - \mathbb{E}[X])^2]$
$\mathrm{Cov}(X, Y)$ “the covariance of X and Y”
Whether two random variables move together (positive), oppositely (negative), or neither (zero).
$\mathrm{Cov}(X, Y) = \mathbb{E}[(X - \mathbb{E}[X])(Y - \mathbb{E}[Y])]$
$p(\mathcal{D} \mid \theta)$ “the likelihood, p of D given theta”
The likelihood: how probable the whole dataset is under parameters $\theta$ . For i.i.d. data it's a product.
$p(\mathcal{D} \mid \theta) = \prod_{i=1}^{n} p_\theta(y_i \mid x_i)$
$\log p(\mathcal{D} \mid \theta)$ “the log-likelihood”
The log turns the product into a sum (Chamber 5). Maximising it is maximum likelihood; minimising its negative is the usual cross-entropy loss.
$\sum_{i=1}^{n} \log p_\theta(y_i \mid x_i)$
$D_{\mathrm{KL}}(p \,\|\, q)$ “the K L divergence of p from q”
The Kullback–Leibler divergence: how different $q$ is from $p$ . Never negative, zero only when $p = q$ , and not symmetric. The $\|$ is a separator, not a norm.
$D_{\mathrm{KL}}(p \,\|\, q) = \mathbb{E}_{x \sim p}\left[\log \frac{p(x)}{q(x)}\right]$

Symbol	Say it	Meaning
$\Omega$ $\Omega = \{1, 2, 3, 4, 5, 6\}$	“capital omega, the sample space”	The sample space: the set of every possible outcome of the experiment.
$A \subseteq \Omega$ $A = \{2, 4, 6\}$	“A, an event”	An event is a set of outcomes, so a subset of $\Omega$ . “The roll is even” is the event $\{2, 4, 6\}$ .
$P(A)$ $P(\text{even}) = \tfrac{1}{2}$	“P of A”	The probability of event $A$ : a number from 0 (impossible) to 1 (certain).
$P(A^c)$ $P(A^c) = 1 - P(A)$	“P of A complement”	The probability that $A$ does not happen. It always equals $1 - P(A)$ .
$P(A \cap B)$ $P(A \cap B) = P(A, B)$	“P of A and B”	The probability that both happen. Often written with a comma instead: $P(A, B)$ .
$P(A \cup B)$ $P(A \cup B) = P(A) + P(B) - P(A \cap B)$	“P of A or B”	The probability that at least one happens. Inclusion–exclusion from Chamber 2 gives the formula.
$P(A \mid B)$ $P(A \mid B) = \frac{P(A \cap B)}{P(B)}$	“P of A given B”	Conditional probability: how likely $A$ is once you know $B$ happened. The bar is read “given”.
$X$ $X = \text{the number rolled}$	“random variable X”	A capital letter is a random variable: a number whose value depends on chance, like the result of a roll.
$P(X = x)$ $P(X = 6) = \tfrac{1}{6}$	“P of big X equals little x”	The probability that the random variable $X$ takes the particular value $x$ . Capital: the random quantity. Lower case: one value it could take.
$p(x)$ $p(6) = \tfrac{1}{6}$	“p of x”	The distribution of $X$ at the value $x$ . For discrete $X$ it's a pmf, $p(x) = P(X = x)$ . For continuous $X$ it's a density: areas under it are probabilities.
$p(x, y)$ $p(\text{rain}, \text{umbrella}) = 0.3$	“p of x comma y”	The joint distribution: how likely $X = x$ and $Y = y$ are together.
$p(y \mid x)$ $p(y \mid x) = \frac{p(x, y)}{p(x)}$	“p of y given x”	The conditional distribution of $Y$ once $X = x$ is known: $p(x, y) / p(x)$ .
$p_\theta(y \mid x)$ $p_\theta(\text{cat} \mid \text{image}) = 0.92$	“p theta of y given x”	A model: the probability that a network with parameters $\theta$ assigns to label $y$ for input $x$ .
$\sum_{y} p(x, y)$ $p(x) = \sum_{y} p(x, y)$	“sum over y of p of x comma y”	Marginalising: add up the joint over every value of $y$ and you're left with $p(x)$ . People say $y$ has been “summed out”.
$\int p(x, y)\, dy$ $p(x) = \int p(x, y)\, dy$	“integral of p of x comma y, d y”	The same marginalisation when $y$ is continuous: the integral is a continuous sum (Chamber 7).
$X \perp\!\!\!\perp Y$ $p(x, y) = p(x)\,p(y)$	“X is independent of Y”	Independence: knowing one tells you nothing about the other, so the joint factorises: $p(x, y) = p(x)\,p(y)$ .
$\sim$ $x \sim p$	“is drawn from”	Between a variable and a distribution, $\sim$ says the variable is a sample from it (“is distributed as”). Not to be confused with $\approx$ .
$\overset{\text{i.i.d.}}{\sim}$ $x_1, \ldots, x_n \overset{\text{i.i.d.}}{\sim} p$	“independent and identically distributed as”	Every sample comes from the same distribution, and none depends on any other. The standing assumption about datasets.
$\mathrm{Bernoulli}(p)$ $r \sim \mathrm{Bernoulli}(0.8)$	“Bernoulli p”	A biased coin: the value 1 with probability $p$ and 0 otherwise. Dropout's masks are Bernoulli.
$\mathcal{N}(\mu, \sigma^2)$ $\varepsilon \sim \mathcal{N}(0, 1)$	“normal with mean mu and variance sigma squared”	The bell curve centred at $\mu$ . The second slot is the variance $\sigma^2$ , not the standard deviation.
$\mathcal{U}(a, b)$ $u \sim \mathcal{U}(0, 1)$	“uniform from a to b”	Every value between $a$ and $b$ equally likely. A common way to initialise weights.
$\mathbb{E}[X]$ $\mathbb{E}[X] = \sum_x x\, p(x)$	“the expectation of X”	The expected value: the average of $X$ , weighting each value by its probability. The long-run mean of many samples.
$\mathbb{E}_{x \sim p}[f(x)]$ $\mathbb{E}_{x \sim p}[f(x)] = \sum_x f(x)\, p(x)$	“expectation of f of x, x drawn from p”	The average of $f(x)$ when $x$ is drawn from $p$ . The subscript names the random variable and its distribution.
$\mathrm{Var}(X)$ $\mathrm{Var}(X) = \mathbb{E}[(X - \mathbb{E}[X])^2]$	“the variance of X”	How spread out $X$ is: the expected squared distance from its mean. Its square root is the standard deviation $\sigma$ .
$\mathrm{Cov}(X, Y)$ $\mathrm{Cov}(X, Y) = \mathbb{E}[(X - \mathbb{E}[X])(Y - \mathbb{E}[Y])]$	“the covariance of X and Y”	Whether two random variables move together (positive), oppositely (negative), or neither (zero).
$p(\mathcal{D} \mid \theta)$ $p(\mathcal{D} \mid \theta) = \prod_{i=1}^{n} p_\theta(y_i \mid x_i)$	“the likelihood, p of D given theta”	The likelihood: how probable the whole dataset is under parameters $\theta$ . For i.i.d. data it's a product.
$\log p(\mathcal{D} \mid \theta)$ $\sum_{i=1}^{n} \log p_\theta(y_i \mid x_i)$	“the log-likelihood”	The log turns the product into a sum (Chamber 5). Maximising it is maximum likelihood; minimising its negative is the usual cross-entropy loss.
$D_{\mathrm{KL}}(p \,\\|\, q)$ $D_{\mathrm{KL}}(p \,\\|\, q) = \mathbb{E}_{x \sim p}\left[\log \frac{p(x)}{q(x)}\right]$	“the K L divergence of p from q”	The Kullback–Leibler divergence: how different $q$ is from $p$ . Never negative, zero only when $p = q$ , and not symmetric. The $\\|$ is a separator, not a norm.

Mathematics for Machine Learning · Chamber 9

Reading the Equations of a Paper

$\text{w.r.t.}$ “with respect to”
Names the variable a derivative or an optimisation is taken over. The Adam paper computes “gradients w.r.t. stochastic objective”.
$\nabla_\theta f \text{ is the gradient w.r.t. } \theta$
$\min_{x} f(x) \ \text{s.t.}\ x \ge 0$ “minimise f of x subject to x at least zero”
In an optimisation problem, “s.t.” means subject to: it introduces the constraints the answer must satisfy. (In a definition, the same letters mean “such that”, as in Chamber 3.)
$f(\cdot)$ “f of dot”
The dot is a placeholder for “whatever argument goes here”, so $f(\cdot)$ means the function itself, not a value of it. The same dot turns up in norms, $\|\cdot\|_2$ .
$\|\cdot\|_2$
$x_{1:T}$ “x one to T”
The whole sequence $x_1, x_2, \ldots, x_T$ . Colon ranges are everywhere in papers on sequences.
$p(x_{1:T})$
$x_{<t}$ “x before t”
Everything before position $t$ : $x_1, \ldots, x_{t-1}$ . The notation of language models, which predict each token from the ones before it.
$p(x_{1:T}) = \prod_{t=1}^{T} p(x_t \mid x_{<t})$
$\text{const}$ “plus a constant”
Terms that don't depend on the variable of interest, lumped together and ignored because they don't change the argmin.
$\log p(x) = -\tfrac{1}{2}x^2 + \text{const}$
$[z]_+$ “the positive part of z”
Shorthand for $\max(0, z)$ , the ReLU of Chamber 4 in different clothes. Common in hinge losses.
$[1 - y\,\hat{y}]_+$
$\simeq$ “sim-equals, is estimated by”
A cousin of $\approx$ , called “sim-equals” after its LaTeX name, \simeq. The VAE paper uses it for “this sample average stands in for that expectation”.
$\mathbb{E}[f(z)] \simeq \frac{1}{L}\sum_{l=1}^{L} f(z^{(l)})$
$Q$ “Q, the queries”
In attention, a matrix with one row per position that is asking: shape $n \times d_k$ .
$Q \in \mathbb{R}^{n \times d_k}$
$K$ “K, the keys”
One row per position that can be looked up, compared with every query: shape $m \times d_k$ .
$K \in \mathbb{R}^{m \times d_k}$
$V$ “V, the values”
The content that gets mixed, one row per key: shape $m \times d_v$ .
$V \in \mathbb{R}^{m \times d_v}$
$d_k$ “d k, the key dimension”
The length of each query and key vector. The Transformer uses $d_k = 64$ per head.
$\sqrt{d_k} = 8$
$g_t^2$ “g t squared, element by element”
In Adam, the square of every entry of the gradient vector ( $g_t \odot g_t$ ), not a dot product.
$v_t \leftarrow \beta_2 v_{t-1} + (1 - \beta_2)\, g_t^2$
$q_{\boldsymbol{\phi}}(\mathbf{z} \mid \mathbf{x})$ “q phi of z given x, the encoder”
In a VAE, the encoder: a distribution over codes $\mathbf{z}$ for the input $\mathbf{x}$ , with parameters $\boldsymbol{\phi}$ .
$p_{\boldsymbol{\theta}}(\mathbf{x} \mid \mathbf{z})$ “p theta of x given z, the decoder”
In a VAE, the decoder: how likely the input $\mathbf{x}$ is to be rebuilt from the code $\mathbf{z}$ .
$\mathcal{L}(\boldsymbol{\theta}, \boldsymbol{\phi}; \mathbf{x}^{(i)})$ “the ELBO, L of theta and phi at x i”
The evidence lower bound for data point $i$ : the number a VAE maximises. Parameters before the semicolon, data after (Chamber 4's $f(x; \theta)$ the other way round).

Symbol	Say it	Meaning
$\text{w.r.t.}$ $\nabla_\theta f \text{ is the gradient w.r.t. } \theta$	“with respect to”	Names the variable a derivative or an optimisation is taken over. The Adam paper computes “gradients w.r.t. stochastic objective”.
$\min_{x} f(x) \ \text{s.t.}\ x \ge 0$	“minimise f of x subject to x at least zero”	In an optimisation problem, “s.t.” means subject to: it introduces the constraints the answer must satisfy. (In a definition, the same letters mean “such that”, as in Chamber 3.)
$f(\cdot)$ $\\|\cdot\\|_2$	“f of dot”	The dot is a placeholder for “whatever argument goes here”, so $f(\cdot)$ means the function itself, not a value of it. The same dot turns up in norms, $\\|\cdot\\|_2$ .
$x_{1:T}$ $p(x_{1:T})$	“x one to T”	The whole sequence $x_1, x_2, \ldots, x_T$ . Colon ranges are everywhere in papers on sequences.
$x_{<t}$ $p(x_{1:T}) = \prod_{t=1}^{T} p(x_t \mid x_{<t})$	“x before t”	Everything before position $t$ : $x_1, \ldots, x_{t-1}$ . The notation of language models, which predict each token from the ones before it.
$\text{const}$ $\log p(x) = -\tfrac{1}{2}x^2 + \text{const}$	“plus a constant”	Terms that don't depend on the variable of interest, lumped together and ignored because they don't change the argmin.
$[z]_+$ $[1 - y\,\hat{y}]_+$	“the positive part of z”	Shorthand for $\max(0, z)$ , the ReLU of Chamber 4 in different clothes. Common in hinge losses.
$\simeq$ $\mathbb{E}[f(z)] \simeq \frac{1}{L}\sum_{l=1}^{L} f(z^{(l)})$	“sim-equals, is estimated by”	A cousin of $\approx$ , called “sim-equals” after its LaTeX name, `\simeq`. The VAE paper uses it for “this sample average stands in for that expectation”.
$Q$ $Q \in \mathbb{R}^{n \times d_k}$	“Q, the queries”	In attention, a matrix with one row per position that is asking: shape $n \times d_k$ .
$K$ $K \in \mathbb{R}^{m \times d_k}$	“K, the keys”	One row per position that can be looked up, compared with every query: shape $m \times d_k$ .
$V$ $V \in \mathbb{R}^{m \times d_v}$	“V, the values”	The content that gets mixed, one row per key: shape $m \times d_v$ .
$d_k$ $\sqrt{d_k} = 8$	“d k, the key dimension”	The length of each query and key vector. The Transformer uses $d_k = 64$ per head.
$g_t^2$ $v_t \leftarrow \beta_2 v_{t-1} + (1 - \beta_2)\, g_t^2$	“g t squared, element by element”	In Adam, the square of every entry of the gradient vector ( $g_t \odot g_t$ ), not a dot product.
$q_{\boldsymbol{\phi}}(\mathbf{z} \mid \mathbf{x})$	“q phi of z given x, the encoder”	In a VAE, the encoder: a distribution over codes $\mathbf{z}$ for the input $\mathbf{x}$ , with parameters $\boldsymbol{\phi}$ .
$p_{\boldsymbol{\theta}}(\mathbf{x} \mid \mathbf{z})$	“p theta of x given z, the decoder”	In a VAE, the decoder: how likely the input $\mathbf{x}$ is to be rebuilt from the code $\mathbf{z}$ .
$\mathcal{L}(\boldsymbol{\theta}, \boldsymbol{\phi}; \mathbf{x}^{(i)})$	“the ELBO, L of theta and phi at x i”	The evidence lower bound for data point $i$ : the number a VAE maximises. Parameters before the semicolon, data after (Chamber 4's $f(x; \theta)$ the other way round).

Linear Algebra for Machine Learning · Chamber 1

Vectors: Arrows, Lists and Data

$\mathbb{R}^n$ “R n”
Every list of $n$ real numbers: the space the vectors live in. $\mathbb{R}^2$ is the plane, $\mathbb{R}^3$ ordinary space.
$\mathbf{x} \in \mathbb{R}^{784}$
$\begin{bmatrix} v_1 \\ v_2 \end{bmatrix}$ “the column vector v one, v two”
A vector written as a column. Its entries are the arrow's coordinates. In running text it's written $(v_1, v_2)$ to save space.
$\mathbf{v} = \begin{bmatrix} 3 \\ 1 \end{bmatrix} = (3, 1)$
$\overrightarrow{PQ}$ “vector P Q”
The arrow from point $P$ to point $Q$ : “destination minus start”.
$\overrightarrow{PQ} = Q - P$
$\mathbf{u} + \mathbf{v}$ “u plus v”
Add entry by entry. As arrows: put the tail of $\mathbf{v}$ at the tip of $\mathbf{u}$ (tip to tail).
$(3, 1) + (1, 2) = (4, 3)$
$c\,\mathbf{v}$ “c times v”
Scalar multiplication: multiply every entry by the number $c$ . Stretches or shrinks the arrow by $|c|$ , and flips it if $c < 0$ .
$3\,(1, -2) = (3, -6)$
$-\mathbf{v}$ “minus v”
The negative of $\mathbf{v}$ : the same length, pointing the opposite way. It equals $(-1)\mathbf{v}$ .
$-(2, -5) = (-2, 5)$
$\mathbf{u} - \mathbf{v}$ “u minus v”
$\mathbf{u} + (-\mathbf{v})$ . Drawn from the origin, it's the arrow from the tip of $\mathbf{v}$ to the tip of $\mathbf{u}$ .
$(4, 3) - (1, 2) = (3, 1)$
$\mathbf{0}$ “the zero vector”
Every entry zero. Adding it changes nothing, and it has no direction.
$\mathbf{v} + \mathbf{0} = \mathbf{v}$
$\mathbf{e}_i$ “e i”
The $i$ -th standard basis vector: 1 in position $i$ , 0 everywhere else. In machine learning it's called a one-hot vector.
$\mathbf{e}_2 = (0, 1, 0) \in \mathbb{R}^3$
$\sum_{i=1}^{k} c_i \mathbf{v}_i$ “the sum of c i times v i”
A linear combination: scale each vector $\mathbf{v}_i$ by its coefficient $c_i$ , then add the results.
$c_1\mathbf{v}_1 + c_2\mathbf{v}_2$
$\mathbf{x}_{\text{king}}$ “x king”
A word embedding: the learned vector for one word. Here the subscript names the word, not a position.
$\mathbf{x}_{\text{king}} \in \mathbb{R}^{300}$

Symbol	Say it	Meaning
$\mathbb{R}^n$ $\mathbf{x} \in \mathbb{R}^{784}$	“R n”	Every list of $n$ real numbers: the space the vectors live in. $\mathbb{R}^2$ is the plane, $\mathbb{R}^3$ ordinary space.
$\begin{bmatrix} v_1 \\ v_2 \end{bmatrix}$ $\mathbf{v} = \begin{bmatrix} 3 \\ 1 \end{bmatrix} = (3, 1)$	“the column vector v one, v two”	A vector written as a column. Its entries are the arrow's coordinates. In running text it's written $(v_1, v_2)$ to save space.
$\overrightarrow{PQ}$ $\overrightarrow{PQ} = Q - P$	“vector P Q”	The arrow from point $P$ to point $Q$ : “destination minus start”.
$\mathbf{u} + \mathbf{v}$ $(3, 1) + (1, 2) = (4, 3)$	“u plus v”	Add entry by entry. As arrows: put the tail of $\mathbf{v}$ at the tip of $\mathbf{u}$ (tip to tail).
$c\,\mathbf{v}$ $3\,(1, -2) = (3, -6)$	“c times v”	Scalar multiplication: multiply every entry by the number $c$ . Stretches or shrinks the arrow by $\|c\|$ , and flips it if $c < 0$ .
$-\mathbf{v}$ $-(2, -5) = (-2, 5)$	“minus v”	The negative of $\mathbf{v}$ : the same length, pointing the opposite way. It equals $(-1)\mathbf{v}$ .
$\mathbf{u} - \mathbf{v}$ $(4, 3) - (1, 2) = (3, 1)$	“u minus v”	$\mathbf{u} + (-\mathbf{v})$ . Drawn from the origin, it's the arrow from the tip of $\mathbf{v}$ to the tip of $\mathbf{u}$ .
$\mathbf{0}$ $\mathbf{v} + \mathbf{0} = \mathbf{v}$	“the zero vector”	Every entry zero. Adding it changes nothing, and it has no direction.
$\mathbf{e}_i$ $\mathbf{e}_2 = (0, 1, 0) \in \mathbb{R}^3$	“e i”	The $i$ -th standard basis vector: 1 in position $i$ , 0 everywhere else. In machine learning it's called a one-hot vector.
$\sum_{i=1}^{k} c_i \mathbf{v}_i$ $c_1\mathbf{v}_1 + c_2\mathbf{v}_2$	“the sum of c i times v i”	A linear combination: scale each vector $\mathbf{v}_i$ by its coefficient $c_i$ , then add the results.
$\mathbf{x}_{\text{king}}$ $\mathbf{x}_{\text{king}} \in \mathbb{R}^{300}$	“x king”	A word embedding: the learned vector for one word. Here the subscript names the word, not a position.

Linear Algebra for Machine Learning · Chamber 2

Dot Products, Lengths and Angles

$\mathbf{u} \cdot \mathbf{v}$ “u dot v”
The dot product: multiply matching entries and add. The result is a single number.
$(1, 2) \cdot (3, 4) = 3 + 8 = 11$
$\mathbf{u}^\top \mathbf{v}$ “u transpose v”
The same dot product, written as a row times a column. The usual form in machine learning papers.
$\hat{r}_{ui} = \mathbf{q}_i^\top \mathbf{p}_u$
$\langle \mathbf{u}, \mathbf{v} \rangle$ “the inner product of u and v”
Angle-bracket notation for the dot product, common in maths and theory papers.
$\langle \mathbf{u}, \mathbf{v} \rangle = \mathbf{u}^\top \mathbf{v}$
$\lVert \mathbf{v} \rVert$ “the norm of v”
The length of $\mathbf{v}$ . With no subscript it nearly always means the Euclidean norm $\lVert \mathbf{v} \rVert_2$ .
$\lVert (3, 4) \rVert = 5$
$\lVert \mathbf{v} \rVert_2$ “the two-norm of v”
Euclidean length: $\sqrt{v_1^2 + \cdots + v_n^2} = \sqrt{\mathbf{v} \cdot \mathbf{v}}$ . Also called the $\ell_2$ norm.
$\lVert \mathbf{v} \rVert_2 = \sqrt{\mathbf{v}^\top \mathbf{v}}$
$\lVert \mathbf{v} \rVert_1$ “the one-norm of v”
The sum of the absolute values of the entries: distance along a grid of streets (the taxicab or Manhattan norm).
$\lVert (3, -4) \rVert_1 = 7$
$\lVert \mathbf{v} \rVert_\infty$ “the infinity-norm of v”
The largest absolute value of any entry (the max norm).
$\lVert (3, -4) \rVert_\infty = 4$
$\hat{\mathbf{v}}$ “v hat”
The unit vector in the direction of $\mathbf{v}$ . On a vector, a hat often means “normalised” rather than “estimated”.
$\hat{\mathbf{v}} = \mathbf{v} / \lVert \mathbf{v} \rVert$
$\lVert \mathbf{u} - \mathbf{v} \rVert$ “the distance between u and v”
The length of the arrow from the tip of $\mathbf{v}$ to the tip of $\mathbf{u}$ .
$\lVert (4, 6) - (1, 2) \rVert = 5$
$\cos\theta$ “cos theta”
The cosine of the angle between two vectors, known in machine learning as their cosine similarity. Always between $-1$ and $1$ .
$\cos\theta = \frac{\mathbf{u} \cdot \mathbf{v}}{\lVert \mathbf{u} \rVert \lVert \mathbf{v} \rVert}$
$\mathbf{u} \perp \mathbf{v}$ “u is perpendicular to v”
Orthogonal: at right angles, which means $\mathbf{u} \cdot \mathbf{v} = 0$ .
$(2, 1) \perp (-1, 2)$
$\operatorname{proj}_{\mathbf{v}} \mathbf{u}$ “the projection of u onto v”
The shadow of $\mathbf{u}$ on the line through $\mathbf{v}$ : the point of that line closest to the tip of $\mathbf{u}$ .
$\operatorname{proj}_{\mathbf{v}} \mathbf{u} = \frac{\mathbf{u} \cdot \mathbf{v}}{\mathbf{v} \cdot \mathbf{v}}\,\mathbf{v}$

Symbol	Say it	Meaning
$\mathbf{u} \cdot \mathbf{v}$ $(1, 2) \cdot (3, 4) = 3 + 8 = 11$	“u dot v”	The dot product: multiply matching entries and add. The result is a single number.
$\mathbf{u}^\top \mathbf{v}$ $\hat{r}_{ui} = \mathbf{q}_i^\top \mathbf{p}_u$	“u transpose v”	The same dot product, written as a row times a column. The usual form in machine learning papers.
$\langle \mathbf{u}, \mathbf{v} \rangle$ $\langle \mathbf{u}, \mathbf{v} \rangle = \mathbf{u}^\top \mathbf{v}$	“the inner product of u and v”	Angle-bracket notation for the dot product, common in maths and theory papers.
$\lVert \mathbf{v} \rVert$ $\lVert (3, 4) \rVert = 5$	“the norm of v”	The length of $\mathbf{v}$ . With no subscript it nearly always means the Euclidean norm $\lVert \mathbf{v} \rVert_2$ .
$\lVert \mathbf{v} \rVert_2$ $\lVert \mathbf{v} \rVert_2 = \sqrt{\mathbf{v}^\top \mathbf{v}}$	“the two-norm of v”	Euclidean length: $\sqrt{v_1^2 + \cdots + v_n^2} = \sqrt{\mathbf{v} \cdot \mathbf{v}}$ . Also called the $\ell_2$ norm.
$\lVert \mathbf{v} \rVert_1$ $\lVert (3, -4) \rVert_1 = 7$	“the one-norm of v”	The sum of the absolute values of the entries: distance along a grid of streets (the taxicab or Manhattan norm).
$\lVert \mathbf{v} \rVert_\infty$ $\lVert (3, -4) \rVert_\infty = 4$	“the infinity-norm of v”	The largest absolute value of any entry (the max norm).
$\hat{\mathbf{v}}$ $\hat{\mathbf{v}} = \mathbf{v} / \lVert \mathbf{v} \rVert$	“v hat”	The unit vector in the direction of $\mathbf{v}$ . On a vector, a hat often means “normalised” rather than “estimated”.
$\lVert \mathbf{u} - \mathbf{v} \rVert$ $\lVert (4, 6) - (1, 2) \rVert = 5$	“the distance between u and v”	The length of the arrow from the tip of $\mathbf{v}$ to the tip of $\mathbf{u}$ .
$\cos\theta$ $\cos\theta = \frac{\mathbf{u} \cdot \mathbf{v}}{\lVert \mathbf{u} \rVert \lVert \mathbf{v} \rVert}$	“cos theta”	The cosine of the angle between two vectors, known in machine learning as their cosine similarity. Always between $-1$ and $1$ .
$\mathbf{u} \perp \mathbf{v}$ $(2, 1) \perp (-1, 2)$	“u is perpendicular to v”	Orthogonal: at right angles, which means $\mathbf{u} \cdot \mathbf{v} = 0$ .
$\operatorname{proj}_{\mathbf{v}} \mathbf{u}$ $\operatorname{proj}_{\mathbf{v}} \mathbf{u} = \frac{\mathbf{u} \cdot \mathbf{v}}{\mathbf{v} \cdot \mathbf{v}}\,\mathbf{v}$	“the projection of u onto v”	The shadow of $\mathbf{u}$ on the line through $\mathbf{v}$ : the point of that line closest to the tip of $\mathbf{u}$ .

Linear Algebra for Machine Learning · Chamber 3

Span, Basis and Dimension

$\mathrm{span}\{\mathbf{v}_1, \ldots, \mathbf{v}_k\}$ “the span of v one to v k”
Every linear combination $c_1\mathbf{v}_1 + \cdots + c_k\mathbf{v}_k$ with real coefficients: all the points these vectors can reach.
$(3, 6) \in \mathrm{span}\{(1, 2)\}$
$\{\mathbf{0}\}$ “the zero subspace”
The set whose only element is the zero vector. The smallest subspace there is, and the span of $\mathbf{0}$ alone.
$\mathrm{span}\{\mathbf{0}\} = \{\mathbf{0}\}$
$c_1\mathbf{v}_1 + \cdots + c_k\mathbf{v}_k = \mathbf{0}$ “a dependency relation”
A combination that lands on $\mathbf{0}$ . If one exists with some $c_i \neq 0$ , the vectors are dependent; if only all-zero coefficients work, they are independent.
$2(1, 2) - (2, 4) = \mathbf{0}$
$\det[\mathbf{u}\ \mathbf{v}]$ “the determinant of u and v”
For two vectors in the plane, $u_1v_2 - u_2v_1$ : the signed area of the parallelogram they span. Zero exactly when they are dependent. Chamber 5 builds determinants in general.
$\det\begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix} = 3$
$\mathcal{B} = (\mathbf{b}_1, \ldots, \mathbf{b}_n)$ “the basis B”
An ordered list of vectors that is linearly independent and spans the space. Bases usually get calligraphic letters.
$\mathcal{B} = ((1, 1), (1, -1))$
$[\mathbf{x}]_{\mathcal{B}}$ “the coordinates of x in basis B”
The unique coefficients $(c_1, \ldots, c_n)$ with $\mathbf{x} = c_1\mathbf{b}_1 + \cdots + c_n\mathbf{b}_n$ . Same arrow, different ruler.
$[(5, 1)]_{\mathcal{B}} = (3, 2)$
$\dim U$ “the dimension of U”
The number of vectors in any basis of the subspace $U$ . All bases of $U$ have the same size.
$\dim \mathbb{R}^n = n$
$d_{\text{int}}$ “d int, the intrinsic dimension”
From Li et al. (2018): the number of directions that actually matter for solving a problem, $D - s$ , where $s$ is the dimension of the solution set.
$D = d_{\text{int}} + s$

Symbol	Say it	Meaning
$\mathrm{span}\{\mathbf{v}_1, \ldots, \mathbf{v}_k\}$ $(3, 6) \in \mathrm{span}\{(1, 2)\}$	“the span of v one to v k”	Every linear combination $c_1\mathbf{v}_1 + \cdots + c_k\mathbf{v}_k$ with real coefficients: all the points these vectors can reach.
$\{\mathbf{0}\}$ $\mathrm{span}\{\mathbf{0}\} = \{\mathbf{0}\}$	“the zero subspace”	The set whose only element is the zero vector. The smallest subspace there is, and the span of $\mathbf{0}$ alone.
$c_1\mathbf{v}_1 + \cdots + c_k\mathbf{v}_k = \mathbf{0}$ $2(1, 2) - (2, 4) = \mathbf{0}$	“a dependency relation”	A combination that lands on $\mathbf{0}$ . If one exists with some $c_i \neq 0$ , the vectors are dependent; if only all-zero coefficients work, they are independent.
$\det[\mathbf{u}\ \mathbf{v}]$ $\det\begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix} = 3$	“the determinant of u and v”	For two vectors in the plane, $u_1v_2 - u_2v_1$ : the signed area of the parallelogram they span. Zero exactly when they are dependent. Chamber 5 builds determinants in general.
$\mathcal{B} = (\mathbf{b}_1, \ldots, \mathbf{b}_n)$ $\mathcal{B} = ((1, 1), (1, -1))$	“the basis B”	An ordered list of vectors that is linearly independent and spans the space. Bases usually get calligraphic letters.
$[\mathbf{x}]_{\mathcal{B}}$ $[(5, 1)]_{\mathcal{B}} = (3, 2)$	“the coordinates of x in basis B”	The unique coefficients $(c_1, \ldots, c_n)$ with $\mathbf{x} = c_1\mathbf{b}_1 + \cdots + c_n\mathbf{b}_n$ . Same arrow, different ruler.
$\dim U$ $\dim \mathbb{R}^n = n$	“the dimension of U”	The number of vectors in any basis of the subspace $U$ . All bases of $U$ have the same size.
$d_{\text{int}}$ $D = d_{\text{int}} + s$	“d int, the intrinsic dimension”	From Li et al. (2018): the number of directions that actually matter for solving a problem, $D - s$ , where $s$ is the dimension of the solution set.

Linear Algebra for Machine Learning · Chamber 4

Matrices as Transformations

$T : \mathbb{R}^n \to \mathbb{R}^m$ “T maps R n to R m”
A function that takes vectors with $n$ entries to vectors with $m$ entries. It is linear if it respects sums and scalings.
$T(\mathbf{u} + \mathbf{v}) = T(\mathbf{u}) + T(\mathbf{v})$
$A\mathbf{x}$ “A times x”
Apply the matrix to the vector: the combination $x_1\mathbf{a}_1 + \cdots + x_n\mathbf{a}_n$ of $A$ 's columns, weighted by the entries of $\mathbf{x}$ .
$\begin{bmatrix} 2 & -1 \\ 1 & 1 \end{bmatrix}\begin{pmatrix} 3 \\ 2 \end{pmatrix} = \begin{pmatrix} 4 \\ 5 \end{pmatrix}$
$\mathbf{a}_j = A\mathbf{e}_j$ “a j, the j-th column of A”
Where the $j$ -th standard basis vector lands. The columns of a matrix are the images of the basis vectors.
$A\mathbf{e}_1 = \mathbf{a}_1$
$AB$ “A B, or A after B”
The composition: apply $B$ first, then $A$ . Products are read right to left, like $f \circ g$ .
$(AB)\mathbf{x} = A(B\mathbf{x})$
$R_\theta$ “R theta, rotation by theta”
The matrix that rotates the plane anticlockwise by the angle $\theta$ .
$R_\theta = \begin{bmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{bmatrix}$
$\det A$ “det A, the determinant of A”
The signed factor by which $A$ scales areas. Negative means orientation flips (like a mirror); zero means the plane is flattened.
$\det\begin{bmatrix} a & b \\ c & d \end{bmatrix} = ad - bc$
$\mathbf{a}\mathbf{b}^\top$ “the outer product of a and b”
A column times a row: the matrix whose $(i, j)$ entry is $a_ib_j$ . Every product $AB$ is a sum of these, one per column of $A$ .
$AB = \sum_k A_{:,k}\, B_{k,:}$
$\mathbf{x} \mapsto W\mathbf{x} + \mathbf{b}$ “x maps to W x plus b”
An affine map: a linear map followed by a shift. It is not linear unless $\mathbf{b} = \mathbf{0}$ , because it moves the origin to $\mathbf{b}$ .
$\sigma(W\mathbf{x} + \mathbf{b})$ “sigma of W x plus b”
A neural-network layer: an affine map, then a nonlinearity $\sigma$ (ReLU, tanh, …) applied to each entry separately.
$\mathbf{h} = \mathrm{ReLU}(W\mathbf{x} + \mathbf{b})$
$\begin{bmatrix} \mathbf{x} \\ 1 \end{bmatrix}$ “x in homogeneous coordinates”
The vector with a 1 appended. A matrix acting on it can shift as well as transform, because its last column gets added.
$(3, 5) \mapsto (3, 5, 1)$

Symbol	Say it	Meaning
$T : \mathbb{R}^n \to \mathbb{R}^m$ $T(\mathbf{u} + \mathbf{v}) = T(\mathbf{u}) + T(\mathbf{v})$	“T maps R n to R m”	A function that takes vectors with $n$ entries to vectors with $m$ entries. It is linear if it respects sums and scalings.
$A\mathbf{x}$ $\begin{bmatrix} 2 & -1 \\ 1 & 1 \end{bmatrix}\begin{pmatrix} 3 \\ 2 \end{pmatrix} = \begin{pmatrix} 4 \\ 5 \end{pmatrix}$	“A times x”	Apply the matrix to the vector: the combination $x_1\mathbf{a}_1 + \cdots + x_n\mathbf{a}_n$ of $A$ 's columns, weighted by the entries of $\mathbf{x}$ .
$\mathbf{a}_j = A\mathbf{e}_j$ $A\mathbf{e}_1 = \mathbf{a}_1$	“a j, the j-th column of A”	Where the $j$ -th standard basis vector lands. The columns of a matrix are the images of the basis vectors.
$AB$ $(AB)\mathbf{x} = A(B\mathbf{x})$	“A B, or A after B”	The composition: apply $B$ first, then $A$ . Products are read right to left, like $f \circ g$ .
$R_\theta$ $R_\theta = \begin{bmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{bmatrix}$	“R theta, rotation by theta”	The matrix that rotates the plane anticlockwise by the angle $\theta$ .
$\det A$ $\det\begin{bmatrix} a & b \\ c & d \end{bmatrix} = ad - bc$	“det A, the determinant of A”	The signed factor by which $A$ scales areas. Negative means orientation flips (like a mirror); zero means the plane is flattened.
$\mathbf{a}\mathbf{b}^\top$ $AB = \sum_k A_{:,k}\, B_{k,:}$	“the outer product of a and b”	A column times a row: the matrix whose $(i, j)$ entry is $a_ib_j$ . Every product $AB$ is a sum of these, one per column of $A$ .
$\mathbf{x} \mapsto W\mathbf{x} + \mathbf{b}$	“x maps to W x plus b”	An affine map: a linear map followed by a shift. It is not linear unless $\mathbf{b} = \mathbf{0}$ , because it moves the origin to $\mathbf{b}$ .
$\sigma(W\mathbf{x} + \mathbf{b})$ $\mathbf{h} = \mathrm{ReLU}(W\mathbf{x} + \mathbf{b})$	“sigma of W x plus b”	A neural-network layer: an affine map, then a nonlinearity $\sigma$ (ReLU, tanh, …) applied to each entry separately.
$\begin{bmatrix} \mathbf{x} \\ 1 \end{bmatrix}$ $(3, 5) \mapsto (3, 5, 1)$	“x in homogeneous coordinates”	The vector with a 1 appended. A matrix acting on it can shift as well as transform, because its last column gets added.

Linear Algebra for Machine Learning · Chamber 5

Solving Systems: Elimination and Inverses

$[A \mid \mathbf{b}]$ “A augmented with b”
The augmented matrix: the coefficients of $A$ with the right-hand side $\mathbf{b}$ attached as an extra column. The bar marks where the equals signs were.
$\left[\begin{array}{cc|c} 2 & -1 & 1 \\ 1 & 1 & 5 \end{array}\right]$
$R_i$ “row i”
The $i$ -th row of the matrix you're working on. Row operations are written in terms of these.
$R_2$
$R_i \leftrightarrow R_j$ “swap rows i and j”
Exchange two rows. You need it when a pivot position holds a zero.
$R_1 \leftrightarrow R_3$
$R_i \leftarrow c\,R_i$ “row i gets c times row i”
Scale a row by a non-zero number $c$ , usually to make a pivot equal to 1.
$R_3 \leftarrow \tfrac{1}{12} R_3$
$R_i \leftarrow R_i + c\,R_j$ “row i gets row i plus c times row j”
Add a multiple of another row. The workhorse of elimination: it creates the zeros below (and above) each pivot, and it never changes the determinant.
$R_2 \leftarrow R_2 - 2R_1$
$\operatorname{rref}(A)$ “r r e f of A”
The reduced row echelon form of $A$ : every pivot is 1, with zeros above and below it. Every matrix has exactly one.
$\operatorname{rref}(I) = I$
$A^{-1}$ “A inverse”
The matrix that undoes $A$ : $A^{-1}A = AA^{-1} = I$ . Only square matrices can have one, and not all do. It is not the matrix of reciprocals $1/a_{ij}$ .
$\mathbf{x} = A^{-1}\mathbf{b}$
$I_n$ “the n by n identity”
Ones on the diagonal, zeros elsewhere: the matrix that changes nothing, $I_n \mathbf{x} = \mathbf{x}$ . Written just $I$ when the size is clear.
$I_2 = \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix}$
$\det A$ “det A”
The determinant of a square matrix: the signed factor by which $A$ scales areas (in 2D) or volumes (in 3D and beyond). It is zero exactly when $A$ is singular.
$\det\begin{bmatrix} a & b \\ c & d \end{bmatrix} = ad - bc$
$|A|$ “the determinant of A, written with bars”
Another way to write $\det A$ . Careful: in $\lvert \det J \rvert$ the bars mean absolute value, applied to the number $\det J$ .
$\begin{vmatrix} a & b \\ c & d \end{vmatrix} = ad - bc$
$\frac{\partial f(x)}{\partial x^T}$ “the Jacobian of f at x”
The matrix of partial derivatives $\partial f_i / \partial x_j$ , in the notation of the Real NVP paper. Near $x$ , $f$ behaves like this matrix, so its determinant is $f$ 's local volume scale.
$\frac{\partial y}{\partial x^T}$

Symbol	Say it	Meaning
$[A \mid \mathbf{b}]$ $\left[\begin{array}{cc\|c} 2 & -1 & 1 \\ 1 & 1 & 5 \end{array}\right]$	“A augmented with b”	The augmented matrix: the coefficients of $A$ with the right-hand side $\mathbf{b}$ attached as an extra column. The bar marks where the equals signs were.
$R_i$ $R_2$	“row i”	The $i$ -th row of the matrix you're working on. Row operations are written in terms of these.
$R_i \leftrightarrow R_j$ $R_1 \leftrightarrow R_3$	“swap rows i and j”	Exchange two rows. You need it when a pivot position holds a zero.
$R_i \leftarrow c\,R_i$ $R_3 \leftarrow \tfrac{1}{12} R_3$	“row i gets c times row i”	Scale a row by a non-zero number $c$ , usually to make a pivot equal to 1.
$R_i \leftarrow R_i + c\,R_j$ $R_2 \leftarrow R_2 - 2R_1$	“row i gets row i plus c times row j”	Add a multiple of another row. The workhorse of elimination: it creates the zeros below (and above) each pivot, and it never changes the determinant.
$\operatorname{rref}(A)$ $\operatorname{rref}(I) = I$	“r r e f of A”	The reduced row echelon form of $A$ : every pivot is 1, with zeros above and below it. Every matrix has exactly one.
$A^{-1}$ $\mathbf{x} = A^{-1}\mathbf{b}$	“A inverse”	The matrix that undoes $A$ : $A^{-1}A = AA^{-1} = I$ . Only square matrices can have one, and not all do. It is not the matrix of reciprocals $1/a_{ij}$ .
$I_n$ $I_2 = \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix}$	“the n by n identity”	Ones on the diagonal, zeros elsewhere: the matrix that changes nothing, $I_n \mathbf{x} = \mathbf{x}$ . Written just $I$ when the size is clear.
$\det A$ $\det\begin{bmatrix} a & b \\ c & d \end{bmatrix} = ad - bc$	“det A”	The determinant of a square matrix: the signed factor by which $A$ scales areas (in 2D) or volumes (in 3D and beyond). It is zero exactly when $A$ is singular.
$\|A\|$ $\begin{vmatrix} a & b \\ c & d \end{vmatrix} = ad - bc$	“the determinant of A, written with bars”	Another way to write $\det A$ . Careful: in $\lvert \det J \rvert$ the bars mean absolute value, applied to the number $\det J$ .
$\frac{\partial f(x)}{\partial x^T}$ $\frac{\partial y}{\partial x^T}$	“the Jacobian of f at x”	The matrix of partial derivatives $\partial f_i / \partial x_j$ , in the notation of the Real NVP paper. Near $x$ , $f$ behaves like this matrix, so its determinant is $f$ 's local volume scale.

Linear Algebra for Machine Learning · Chamber 6

Rank and the Four Subspaces

$C(A)$ “the column space of A”
Every output $A\mathbf{x}$ , that is, every combination of the columns of $A$ . A subspace of $\mathbb{R}^m$ . Also written $\operatorname{col}(A)$ , $\operatorname{range}(A)$ or $\operatorname{im}(A)$ .
$\mathbf{b} \in C(A)$
$N(A)$ “the null space of A”
Every input $\mathbf{x}$ with $A\mathbf{x} = \mathbf{0}$ : what $A$ destroys. A subspace of $\mathbb{R}^n$ . Also called the kernel, $\ker A$ .
$N(A) = \{\mathbf{x} \in \mathbb{R}^n : A\mathbf{x} = \mathbf{0}\}$
$C(A^\top)$ “the row space of A”
Every combination of the rows of $A$ (the column space of $A^\top$ ). A subspace of $\mathbb{R}^n$ .
$C(A^\top) \subseteq \mathbb{R}^n$
$N(A^\top)$ “the left null space of A”
Every $\mathbf{y}$ with $A^\top \mathbf{y} = \mathbf{0}$ , equivalently $\mathbf{y}^\top A = \mathbf{0}^\top$ . A subspace of $\mathbb{R}^m$ .
$\mathbf{y}^\top A = \mathbf{0}^\top$
$\operatorname{rank}(A)$ “the rank of A”
The number of pivots, which is also the number of independent columns and the number of independent rows. Some books write $\operatorname{rk}(A)$ .
$\operatorname{rank}(A) \le \min(m, n)$
$\dim V$ “the dimension of V”
The number of vectors in any basis of the subspace $V$ (Chamber 3). The dimension of the null space is sometimes called the nullity.
$\dim N(A) = n - r$
$\mathbf{u}\mathbf{v}^\top$ “u v transpose”
The outer product of $\mathbf{u} \in \mathbb{R}^m$ and $\mathbf{v} \in \mathbb{R}^n$ : the $m \times n$ matrix with entries $u_i v_j$ . It has rank 1 when $\mathbf{u}$ and $\mathbf{v}$ are non-zero.
$(\mathbf{u}\mathbf{v}^\top)_{ij} = u_i v_j$
$r \ll \min(d, k)$ “r is much less than the smaller of d and k”
LoRA's condition on the rank of its update: far smaller than either side of the $d \times k$ weight matrix.
$r = 4 \ll 12288$

Symbol	Say it	Meaning
$C(A)$ $\mathbf{b} \in C(A)$	“the column space of A”	Every output $A\mathbf{x}$ , that is, every combination of the columns of $A$ . A subspace of $\mathbb{R}^m$ . Also written $\operatorname{col}(A)$ , $\operatorname{range}(A)$ or $\operatorname{im}(A)$ .
$N(A)$ $N(A) = \{\mathbf{x} \in \mathbb{R}^n : A\mathbf{x} = \mathbf{0}\}$	“the null space of A”	Every input $\mathbf{x}$ with $A\mathbf{x} = \mathbf{0}$ : what $A$ destroys. A subspace of $\mathbb{R}^n$ . Also called the kernel, $\ker A$ .
$C(A^\top)$ $C(A^\top) \subseteq \mathbb{R}^n$	“the row space of A”	Every combination of the rows of $A$ (the column space of $A^\top$ ). A subspace of $\mathbb{R}^n$ .
$N(A^\top)$ $\mathbf{y}^\top A = \mathbf{0}^\top$	“the left null space of A”	Every $\mathbf{y}$ with $A^\top \mathbf{y} = \mathbf{0}$ , equivalently $\mathbf{y}^\top A = \mathbf{0}^\top$ . A subspace of $\mathbb{R}^m$ .
$\operatorname{rank}(A)$ $\operatorname{rank}(A) \le \min(m, n)$	“the rank of A”	The number of pivots, which is also the number of independent columns and the number of independent rows. Some books write $\operatorname{rk}(A)$ .
$\dim V$ $\dim N(A) = n - r$	“the dimension of V”	The number of vectors in any basis of the subspace $V$ (Chamber 3). The dimension of the null space is sometimes called the nullity.
$\mathbf{u}\mathbf{v}^\top$ $(\mathbf{u}\mathbf{v}^\top)_{ij} = u_i v_j$	“u v transpose”	The outer product of $\mathbf{u} \in \mathbb{R}^m$ and $\mathbf{v} \in \mathbb{R}^n$ : the $m \times n$ matrix with entries $u_i v_j$ . It has rank 1 when $\mathbf{u}$ and $\mathbf{v}$ are non-zero.
$r \ll \min(d, k)$ $r = 4 \ll 12288$	“r is much less than the smaller of d and k”	LoRA's condition on the rank of its update: far smaller than either side of the $d \times k$ weight matrix.

Linear Algebra for Machine Learning · Chamber 7

Orthogonality and Least Squares

$\mathbf{u} \perp \mathbf{v}$ “u is perpendicular to v”
The vectors are orthogonal: their dot product is zero, $\mathbf{u}^\top\mathbf{v} = 0$ . Also read “u is orthogonal to v”.
$(1, 2) \perp (2, -1)$
$V^\perp$ “V perp”
The orthogonal complement of a subspace $V$ : every vector that is perpendicular to all of $V$ . It is a subspace too.
$N(A) = C(A^\top)^\perp$
$\mathbf{q}_i^\top\mathbf{q}_j = \delta_{ij}$ “q i transpose q j equals delta i j”
The vectors $\mathbf{q}_1, \ldots, \mathbf{q}_n$ are orthonormal: each has length 1 (when $i = j$ , $\delta_{ij} = 1$ ) and every pair is perpendicular (when $i \ne j$ , $\delta_{ij} = 0$ ). $\delta_{ij}$ is the Kronecker delta.
$Q^\top Q = I$ “Q transpose Q equals I”
The columns of $Q$ are orthonormal. When $Q$ is square it is called an orthogonal matrix, and $Q^{-1} = Q^\top$ .
$Q = \tfrac{1}{5}\begin{bmatrix} 3 & -4 \\ 4 & 3 \end{bmatrix}$
$\operatorname{proj}_{\mathbf{a}}(\mathbf{b})$ “the projection of b onto a”
The point on the line through $\mathbf{a}$ that is closest to $\mathbf{b}$ : the shadow of $\mathbf{b}$ on that line.
$\operatorname{proj}_{\mathbf{a}}(\mathbf{b}) = \frac{\mathbf{a}^\top\mathbf{b}}{\mathbf{a}^\top\mathbf{a}}\,\mathbf{a}$
$P = A(A^\top A)^{-1}A^\top$ “P equals A, times A transpose A inverse, times A transpose”
The projection matrix onto the column space of $A$ (whose columns are independent): $P\mathbf{b}$ is the closest point to $\mathbf{b}$ in $C(A)$ .
$P^2 = P$ “P squared equals P”
Projecting twice changes nothing. Matrices with this property are called idempotent.
$\hat{\mathbf{x}}$ “x hat”
The least-squares solution of $A\mathbf{x} \approx \mathbf{b}$ : the best compromise when no exact solution exists. As always, the hat marks an estimate.
$\hat{\mathbf{x}} = (A^\top A)^{-1}A^\top\mathbf{b}$
$\mathbf{e} = \mathbf{b} - A\hat{\mathbf{x}}$ “e equals b minus A x hat”
The error (or residual) vector: the part of $\mathbf{b}$ the best fit can't reach. Statistics usually calls it $\mathbf{r}$ .
$A^\top A\hat{\mathbf{x}} = A^\top\mathbf{b}$ “A transpose A x hat equals A transpose b”
The normal equations. They say the error is perpendicular (“normal”) to every column of $A$ .
$\min_{\mathbf{x}}\, \|\mathbf{b} - A\mathbf{x}\|^2$ “the minimum over x of the squared length of b minus A x”
The least-squares problem: choose $\mathbf{x}$ to make the sum of squared errors as small as possible.
$\mathbf{v}_k = \mathbf{a}_k - \sum_{j<k} (\mathbf{q}_j^\top\mathbf{a}_k)\,\mathbf{q}_j$ “v k equals a k minus the sum over j less than k of q j transpose a k, times q j”
One Gram–Schmidt step: strip from $\mathbf{a}_k$ its projections onto the earlier directions, then normalise, $\mathbf{q}_k = \mathbf{v}_k / \|\mathbf{v}_k\|$ .
$A = QR$ “A equals Q R”
The QR factorisation: $Q$ has orthonormal columns (from Gram–Schmidt) and $R = Q^\top A$ is upper triangular.

Symbol	Say it	Meaning
$\mathbf{u} \perp \mathbf{v}$ $(1, 2) \perp (2, -1)$	“u is perpendicular to v”	The vectors are orthogonal: their dot product is zero, $\mathbf{u}^\top\mathbf{v} = 0$ . Also read “u is orthogonal to v”.
$V^\perp$ $N(A) = C(A^\top)^\perp$	“V perp”	The orthogonal complement of a subspace $V$ : every vector that is perpendicular to all of $V$ . It is a subspace too.
$\mathbf{q}_i^\top\mathbf{q}_j = \delta_{ij}$	“q i transpose q j equals delta i j”	The vectors $\mathbf{q}_1, \ldots, \mathbf{q}_n$ are orthonormal: each has length 1 (when $i = j$ , $\delta_{ij} = 1$ ) and every pair is perpendicular (when $i \ne j$ , $\delta_{ij} = 0$ ). $\delta_{ij}$ is the Kronecker delta.
$Q^\top Q = I$ $Q = \tfrac{1}{5}\begin{bmatrix} 3 & -4 \\ 4 & 3 \end{bmatrix}$	“Q transpose Q equals I”	The columns of $Q$ are orthonormal. When $Q$ is square it is called an orthogonal matrix, and $Q^{-1} = Q^\top$ .
$\operatorname{proj}_{\mathbf{a}}(\mathbf{b})$ $\operatorname{proj}_{\mathbf{a}}(\mathbf{b}) = \frac{\mathbf{a}^\top\mathbf{b}}{\mathbf{a}^\top\mathbf{a}}\,\mathbf{a}$	“the projection of b onto a”	The point on the line through $\mathbf{a}$ that is closest to $\mathbf{b}$ : the shadow of $\mathbf{b}$ on that line.
$P = A(A^\top A)^{-1}A^\top$	“P equals A, times A transpose A inverse, times A transpose”	The projection matrix onto the column space of $A$ (whose columns are independent): $P\mathbf{b}$ is the closest point to $\mathbf{b}$ in $C(A)$ .
$P^2 = P$	“P squared equals P”	Projecting twice changes nothing. Matrices with this property are called idempotent.
$\hat{\mathbf{x}}$ $\hat{\mathbf{x}} = (A^\top A)^{-1}A^\top\mathbf{b}$	“x hat”	The least-squares solution of $A\mathbf{x} \approx \mathbf{b}$ : the best compromise when no exact solution exists. As always, the hat marks an estimate.
$\mathbf{e} = \mathbf{b} - A\hat{\mathbf{x}}$	“e equals b minus A x hat”	The error (or residual) vector: the part of $\mathbf{b}$ the best fit can't reach. Statistics usually calls it $\mathbf{r}$ .
$A^\top A\hat{\mathbf{x}} = A^\top\mathbf{b}$	“A transpose A x hat equals A transpose b”	The normal equations. They say the error is perpendicular (“normal”) to every column of $A$ .
$\min_{\mathbf{x}}\, \\|\mathbf{b} - A\mathbf{x}\\|^2$	“the minimum over x of the squared length of b minus A x”	The least-squares problem: choose $\mathbf{x}$ to make the sum of squared errors as small as possible.
$\mathbf{v}_k = \mathbf{a}_k - \sum_{j<k} (\mathbf{q}_j^\top\mathbf{a}_k)\,\mathbf{q}_j$	“v k equals a k minus the sum over j less than k of q j transpose a k, times q j”	One Gram–Schmidt step: strip from $\mathbf{a}_k$ its projections onto the earlier directions, then normalise, $\mathbf{q}_k = \mathbf{v}_k / \\|\mathbf{v}_k\\|$ .
$A = QR$	“A equals Q R”	The QR factorisation: $Q$ has orthonormal columns (from Gram–Schmidt) and $R = Q^\top A$ is upper triangular.

Linear Algebra for Machine Learning · Chamber 8

Eigenvectors and Eigenvalues

$A\mathbf{v} = \lambda\mathbf{v}$ “A v equals lambda v”
The eigenvalue equation: $A$ only stretches $\mathbf{v}$ , by the factor $\lambda$ , and never turns it off its line. A non-zero $\mathbf{v}$ that satisfies it is an eigenvector, and $\lambda$ is its eigenvalue.
$\lambda$ “lambda”
An eigenvalue: the stretch factor. $|\lambda| > 1$ stretches, $|\lambda| < 1$ shrinks, a negative $\lambda$ flips, and $\lambda = 0$ flattens the vector to $\mathbf{0}$ .
$\lambda_1 = 3,\ \lambda_2 = 1$
$\det(A - \lambda I) = 0$ “det of A minus lambda I equals zero”
The characteristic equation. Its solutions are exactly the eigenvalues of $A$ .
$p(\lambda)$ “p of lambda”
The characteristic polynomial $\det(A - \lambda I)$ , of degree $n$ for an $n \times n$ matrix. For $2 \times 2$ matrices it is $\lambda^2 - \operatorname{tr}(A)\,\lambda + \det A$ .
$p(\lambda) = \lambda^2 - 4\lambda + 3$
$\operatorname{tr}(A)$ “trace of A”
The trace: the sum of the diagonal entries. It always equals the sum of the eigenvalues.
$\operatorname{tr}\begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix} = 4$
$\frac{\mathbf{x}^\top A\mathbf{x}}{\mathbf{x}^\top\mathbf{x}}$ “the Rayleigh quotient of x”
Exactly $\lambda$ when $\mathbf{x}$ is an eigenvector, and a good estimate of it when $\mathbf{x}$ is close to one. Power iteration uses it to read off the eigenvalue.
$\Lambda$ “capital lambda”
The diagonal matrix of eigenvalues, $\Lambda = \operatorname{diag}(\lambda_1, \ldots, \lambda_n)$ .
$\Lambda = \begin{bmatrix} 3 & 0 \\ 0 & 1 \end{bmatrix}$
$A = V\Lambda V^{-1}$ “A equals V capital lambda V inverse”
Diagonalisation. The columns of $V$ are eigenvectors of $A$ . Reading right to left: change to eigen-coordinates ( $V^{-1}$ ), stretch each coordinate ( $\Lambda$ ), change back ( $V$ ).
$A^k = V\Lambda^k V^{-1}$ “A to the k equals V capital lambda to the k V inverse”
Powers of a diagonalisable matrix: the eigenvectors stay put and only the eigenvalues are raised to the $k$ -th power.
$\mathbf{x}_{k+1} = \frac{A\mathbf{x}_k}{\|A\mathbf{x}_k\|}$ “x k plus one equals A x k over the norm of A x k”
Power iteration: multiply by $A$ and rescale, over and over. The direction converges to the eigenvector whose eigenvalue is largest in absolute value.
$\lambda_1$ “lambda one”
Usually the dominant eigenvalue, the one largest in absolute value, when eigenvalues are listed by size. It controls the long-run growth of $A^k\mathbf{x}$ .
$\varphi$ “phi”
The golden ratio $\frac{1 + \sqrt{5}}{2} \approx 1.618$ : the dominant eigenvalue of the Fibonacci matrix.
$\varphi = \tfrac{1 + \sqrt{5}}{2}$
$P\boldsymbol{\pi} = \boldsymbol{\pi}$ “P pi equals pi”
$\boldsymbol{\pi}$ (bold pi) is a stationary distribution of a Markov chain with transition matrix $P$ : an eigenvector with eigenvalue 1 whose entries are probabilities.

Symbol	Say it	Meaning
$A\mathbf{v} = \lambda\mathbf{v}$	“A v equals lambda v”	The eigenvalue equation: $A$ only stretches $\mathbf{v}$ , by the factor $\lambda$ , and never turns it off its line. A non-zero $\mathbf{v}$ that satisfies it is an eigenvector, and $\lambda$ is its eigenvalue.
$\lambda$ $\lambda_1 = 3,\ \lambda_2 = 1$	“lambda”	An eigenvalue: the stretch factor. $\|\lambda\| > 1$ stretches, $\|\lambda\| < 1$ shrinks, a negative $\lambda$ flips, and $\lambda = 0$ flattens the vector to $\mathbf{0}$ .
$\det(A - \lambda I) = 0$	“det of A minus lambda I equals zero”	The characteristic equation. Its solutions are exactly the eigenvalues of $A$ .
$p(\lambda)$ $p(\lambda) = \lambda^2 - 4\lambda + 3$	“p of lambda”	The characteristic polynomial $\det(A - \lambda I)$ , of degree $n$ for an $n \times n$ matrix. For $2 \times 2$ matrices it is $\lambda^2 - \operatorname{tr}(A)\,\lambda + \det A$ .
$\operatorname{tr}(A)$ $\operatorname{tr}\begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix} = 4$	“trace of A”	The trace: the sum of the diagonal entries. It always equals the sum of the eigenvalues.
$\frac{\mathbf{x}^\top A\mathbf{x}}{\mathbf{x}^\top\mathbf{x}}$	“the Rayleigh quotient of x”	Exactly $\lambda$ when $\mathbf{x}$ is an eigenvector, and a good estimate of it when $\mathbf{x}$ is close to one. Power iteration uses it to read off the eigenvalue.
$\Lambda$ $\Lambda = \begin{bmatrix} 3 & 0 \\ 0 & 1 \end{bmatrix}$	“capital lambda”	The diagonal matrix of eigenvalues, $\Lambda = \operatorname{diag}(\lambda_1, \ldots, \lambda_n)$ .
$A = V\Lambda V^{-1}$	“A equals V capital lambda V inverse”	Diagonalisation. The columns of $V$ are eigenvectors of $A$ . Reading right to left: change to eigen-coordinates ( $V^{-1}$ ), stretch each coordinate ( $\Lambda$ ), change back ( $V$ ).
$A^k = V\Lambda^k V^{-1}$	“A to the k equals V capital lambda to the k V inverse”	Powers of a diagonalisable matrix: the eigenvectors stay put and only the eigenvalues are raised to the $k$ -th power.
$\mathbf{x}_{k+1} = \frac{A\mathbf{x}_k}{\\|A\mathbf{x}_k\\|}$	“x k plus one equals A x k over the norm of A x k”	Power iteration: multiply by $A$ and rescale, over and over. The direction converges to the eigenvector whose eigenvalue is largest in absolute value.
$\lambda_1$	“lambda one”	Usually the dominant eigenvalue, the one largest in absolute value, when eigenvalues are listed by size. It controls the long-run growth of $A^k\mathbf{x}$ .
$\varphi$ $\varphi = \tfrac{1 + \sqrt{5}}{2}$	“phi”	The golden ratio $\frac{1 + \sqrt{5}}{2} \approx 1.618$ : the dominant eigenvalue of the Fibonacci matrix.
$P\boldsymbol{\pi} = \boldsymbol{\pi}$	“P pi equals pi”	$\boldsymbol{\pi}$ (bold pi) is a stationary distribution of a Markov chain with transition matrix $P$ : an eigenvector with eigenvalue 1 whose entries are probabilities.

Linear Algebra for Machine Learning · Chamber 9

SVD and PCA: The Shape of Data

$A = U\Sigma V^\top$ “A equals U sigma V transpose”
The singular value decomposition: rotate with $V^\top$ , stretch with $\Sigma$ , rotate with $U$ . Every real matrix has one.
$\sigma_i$ “sigma i”
The $i$ -th singular value: how much $A$ stretches along its $i$ -th direction. Never negative, and sorted from largest to smallest.
$\sigma_1 \ge \sigma_2 \ge \cdots \ge 0$
$\Sigma$ “capital sigma, the matrix”
The $m \times n$ matrix with $\sigma_1, \sigma_2, \ldots$ on its diagonal and zeros everywhere else. In this chamber it is neither a sum nor a covariance.
$\Sigma = \begin{pmatrix} \sigma_1 & 0 \\ 0 & \sigma_2 \end{pmatrix}$
$\mathbf{u}_i$ “u i, a left singular vector”
Column $i$ of $U$ : a unit vector in the output space $\mathbb{R}^m$ , the direction where the $i$ -th stretch lands.
$A\mathbf{v}_i = \sigma_i \mathbf{u}_i$
$\mathbf{v}_i$ “v i, a right singular vector”
Column $i$ of $V$ : a unit vector in the input space $\mathbb{R}^n$ . It is an eigenvector of $A^\top A$ with eigenvalue $\sigma_i^2$ .
$A^\top A\mathbf{v}_i = \sigma_i^2 \mathbf{v}_i$
$A_k$ “A sub k”
The truncated SVD: keep the first $k$ layers and drop the rest. It has rank $k$ , and it is the best rank- $k$ approximation of $A$ .
$A_k = \sum_{i=1}^{k} \sigma_i \mathbf{u}_i \mathbf{v}_i^\top$
$\|A\|_F$ “the Frobenius norm of A”
The length of $A$ unrolled into one long vector: the square root of the sum of all squared entries. It equals $\sqrt{\sigma_1^2 + \cdots + \sigma_r^2}$ .
$\|A\|_F = \sqrt{\textstyle\sum_{i,j} A_{ij}^2}$
$\|A\|_2$ “the spectral norm of A”
The biggest stretch $A$ applies to any unit vector. It equals the largest singular value.
$\|A\|_2 = \max_{\|\mathbf{x}\| = 1} \|A\mathbf{x}\| = \sigma_1$
$\bar{\mathbf{x}}$ “x bar, the mean vector”
The average data point. PCA subtracts it from every point first.
$\bar{\mathbf{x}} = \frac{1}{n}\sum_{i=1}^{n} \mathbf{x}_i$
$X_c$ “X centred”
The $n \times d$ data matrix (one point per row) with the mean subtracted from every row.
$X_c = X - \mathbf{1}\bar{\mathbf{x}}^\top$
$C$ “C, the covariance matrix”
The $d \times d$ matrix of covariances between features, with the variances on its diagonal. Symmetric, and its eigenvalues are never negative.
$C = \frac{1}{n} X_c^\top X_c$
$\lambda_i$ “lambda i”
The $i$ -th eigenvalue of $C$ : the variance of the data along the $i$ -th principal direction.
$\lambda_i = \sigma_i^2 / n$
$\frac{\lambda_i}{\sum_j \lambda_j}$ “the explained variance ratio”
The fraction of the total variance that component $i$ captures.
$\mathbf{z}_i$ “z i, the scores”
The low-dimensional code for data point $i$ : its coordinates along the top $k$ principal directions.
$\mathbf{z}_i = V_k^\top(\mathbf{x}_i - \bar{\mathbf{x}})$
$S = Q\Lambda Q^\top$ “S equals Q lambda Q transpose”
The spectral theorem: a symmetric matrix has orthonormal eigenvectors (the columns of $Q$ ) and real eigenvalues (the diagonal of $\Lambda$ ).

Symbol	Say it	Meaning
$A = U\Sigma V^\top$	“A equals U sigma V transpose”	The singular value decomposition: rotate with $V^\top$ , stretch with $\Sigma$ , rotate with $U$ . Every real matrix has one.
$\sigma_i$ $\sigma_1 \ge \sigma_2 \ge \cdots \ge 0$	“sigma i”	The $i$ -th singular value: how much $A$ stretches along its $i$ -th direction. Never negative, and sorted from largest to smallest.
$\Sigma$ $\Sigma = \begin{pmatrix} \sigma_1 & 0 \\ 0 & \sigma_2 \end{pmatrix}$	“capital sigma, the matrix”	The $m \times n$ matrix with $\sigma_1, \sigma_2, \ldots$ on its diagonal and zeros everywhere else. In this chamber it is neither a sum nor a covariance.
$\mathbf{u}_i$ $A\mathbf{v}_i = \sigma_i \mathbf{u}_i$	“u i, a left singular vector”	Column $i$ of $U$ : a unit vector in the output space $\mathbb{R}^m$ , the direction where the $i$ -th stretch lands.
$\mathbf{v}_i$ $A^\top A\mathbf{v}_i = \sigma_i^2 \mathbf{v}_i$	“v i, a right singular vector”	Column $i$ of $V$ : a unit vector in the input space $\mathbb{R}^n$ . It is an eigenvector of $A^\top A$ with eigenvalue $\sigma_i^2$ .
$A_k$ $A_k = \sum_{i=1}^{k} \sigma_i \mathbf{u}_i \mathbf{v}_i^\top$	“A sub k”	The truncated SVD: keep the first $k$ layers and drop the rest. It has rank $k$ , and it is the best rank- $k$ approximation of $A$ .
$\\|A\\|_F$ $\\|A\\|_F = \sqrt{\textstyle\sum_{i,j} A_{ij}^2}$	“the Frobenius norm of A”	The length of $A$ unrolled into one long vector: the square root of the sum of all squared entries. It equals $\sqrt{\sigma_1^2 + \cdots + \sigma_r^2}$ .
$\\|A\\|_2$ $\\|A\\|_2 = \max_{\\|\mathbf{x}\\| = 1} \\|A\mathbf{x}\\| = \sigma_1$	“the spectral norm of A”	The biggest stretch $A$ applies to any unit vector. It equals the largest singular value.
$\bar{\mathbf{x}}$ $\bar{\mathbf{x}} = \frac{1}{n}\sum_{i=1}^{n} \mathbf{x}_i$	“x bar, the mean vector”	The average data point. PCA subtracts it from every point first.
$X_c$ $X_c = X - \mathbf{1}\bar{\mathbf{x}}^\top$	“X centred”	The $n \times d$ data matrix (one point per row) with the mean subtracted from every row.
$C$ $C = \frac{1}{n} X_c^\top X_c$	“C, the covariance matrix”	The $d \times d$ matrix of covariances between features, with the variances on its diagonal. Symmetric, and its eigenvalues are never negative.
$\lambda_i$ $\lambda_i = \sigma_i^2 / n$	“lambda i”	The $i$ -th eigenvalue of $C$ : the variance of the data along the $i$ -th principal direction.
$\frac{\lambda_i}{\sum_j \lambda_j}$	“the explained variance ratio”	The fraction of the total variance that component $i$ captures.
$\mathbf{z}_i$ $\mathbf{z}_i = V_k^\top(\mathbf{x}_i - \bar{\mathbf{x}})$	“z i, the scores”	The low-dimensional code for data point $i$ : its coordinates along the top $k$ principal directions.
$S = Q\Lambda Q^\top$	“S equals Q lambda Q transpose”	The spectral theorem: a symmetric matrix has orthonormal eigenvectors (the columns of $Q$ ) and real eigenvalues (the diagonal of $\Lambda$ ).

Calculus for Machine Learning · Chamber 1

Limits: Getting Arbitrarily Close

$\lim_{x\to a}f(x)$ “limit of f as x approaches a”
The value approached near $a$ .
$\varepsilon$ “epsilon”
Requested output accuracy.
$\delta$ “delta”
An input radius chosen for that accuracy.
$|x-a|$ “distance from x to a”
Absolute error in the input.
$S_N$ “S N”
The sum of the first $N$ terms.
$\sum_{n=1}^{\infty}a_n$ “sum of a n from one to infinity”
The limit of partial sums, if it exists.

Symbol	Say it	Meaning
$\lim_{x\to a}f(x)$	“limit of f as x approaches a”	The value approached near $a$ .
$\varepsilon$	“epsilon”	Requested output accuracy.
$\delta$	“delta”	An input radius chosen for that accuracy.
$\|x-a\|$	“distance from x to a”	Absolute error in the input.
$S_N$	“S N”	The sum of the first $N$ terms.
$\sum_{n=1}^{\infty}a_n$	“sum of a n from one to infinity”	The limit of partial sums, if it exists.

Calculus for Machine Learning · Chamber 2

Derivatives: The Best Local Line

$f'(a)$ “f prime at a”
Instantaneous slope at $a$ .
$\frac{df}{dx}$ “d f by d x”
The derivative of a scalar function.
$o(h)$ “little o of h”
An error negligible compared with $|h|$ as $h\to0$ .
$\sigma(x)$ “sigmoid of x”
The logistic activation.
$\phi(x)$ “phi of x”
Standard normal density.
$\Phi(x)$ “capital phi of x”
Standard normal cumulative distribution.

Symbol	Say it	Meaning
$f'(a)$	“f prime at a”	Instantaneous slope at $a$ .
$\frac{df}{dx}$	“d f by d x”	The derivative of a scalar function.
$o(h)$	“little o of h”	An error negligible compared with $\|h\|$ as $h\to0$ .
$\sigma(x)$	“sigmoid of x”	The logistic activation.
$\phi(x)$	“phi of x”	Standard normal density.
$\Phi(x)$	“capital phi of x”	Standard normal cumulative distribution.

Calculus for Machine Learning · Chamber 3

The Chain Rule: Deep Compositions

$f\circ g$ “f composed with g”
Apply $g$ first, then $f$ .
$f'(g(x))$ “f prime evaluated at g of x”
Outer derivative at the inner output.
$\frac{dh_L}{dh_0}$ “d h L by d h zero”
Sensitivity through the full chain.
$\prod_k a_k$ “product of a k”
Multiply the local sensitivities.
$f^{-1}$ “inverse function”
The function that undoes $f$ , where it exists.
$\nabla_\theta\log p_\theta$ “gradient of log p”
The score of a differentiable probability model.

Symbol	Say it	Meaning
$f\circ g$	“f composed with g”	Apply $g$ first, then $f$ .
$f'(g(x))$	“f prime evaluated at g of x”	Outer derivative at the inner output.
$\frac{dh_L}{dh_0}$	“d h L by d h zero”	Sensitivity through the full chain.
$\prod_k a_k$	“product of a k”	Multiply the local sensitivities.
$f^{-1}$	“inverse function”	The function that undoes $f$ , where it exists.
$\nabla_\theta\log p_\theta$	“gradient of log p”	The score of a differentiable probability model.

Calculus for Machine Learning · Chamber 4

Gradients: Steepest Ascent in Many Dimensions

$\partial f/\partial x_i$ “partial f by partial x i”
Rate of change with other coordinates fixed.
$\nabla f$ “gradient of f”
Column of partial derivatives.
$D_uf$ “directional derivative along u”
Slope per unit distance when $u$ is a unit vector.
$\|u\|_2=1$ “u has Euclidean norm one”
A fair constraint when comparing directions.
$\nabla f^Tu$ “gradient dot u”
First-order change along the direction.
$\{x:f(x)=c\}$ “level set of f at c”
Points having the same function value.

Symbol	Say it	Meaning
$\partial f/\partial x_i$	“partial f by partial x i”	Rate of change with other coordinates fixed.
$\nabla f$	“gradient of f”	Column of partial derivatives.
$D_uf$	“directional derivative along u”	Slope per unit distance when $u$ is a unit vector.
$\\|u\\|_2=1$	“u has Euclidean norm one”	A fair constraint when comparing directions.
$\nabla f^Tu$	“gradient dot u”	First-order change along the direction.
$\{x:f(x)=c\}$	“level set of f at c”	Points having the same function value.

Calculus for Machine Learning · Chamber 5

Jacobians and Automatic Differentiation

$J_F$ “Jacobian of F”
Output-by-input matrix of partial derivatives.
$Jv$ “J times v”
Forward directional derivative of a vector output.
$J^Tv$ “J transpose times v”
Backward propagation of an output sensitivity.
$\bar v$ “v bar”
Adjoint: derivative of the scalar loss with respect to v.
$\epsilon^2=0$ “epsilon squared equals zero”
The defining dual-number rule; epsilon is formal, not a tiny float.
$\det J$ “determinant of J”
Signed local volume scaling for a square Jacobian.

Symbol	Say it	Meaning
$J_F$	“Jacobian of F”	Output-by-input matrix of partial derivatives.
$Jv$	“J times v”	Forward directional derivative of a vector output.
$J^Tv$	“J transpose times v”	Backward propagation of an output sensitivity.
$\bar v$	“v bar”	Adjoint: derivative of the scalar loss with respect to v.
$\epsilon^2=0$	“epsilon squared equals zero”	The defining dual-number rule; epsilon is formal, not a tiny float.
$\det J$	“determinant of J”	Signed local volume scaling for a square Jacobian.

Calculus for Machine Learning · Chamber 6

Matrix Calculus: Gradients of Layers and Losses

$dL$ “d L”
First-order loss change.
$\nabla_WL$ “gradient with respect to W”
A matrix with the same shape as W.
$\operatorname{tr}(A)$ “trace of A”
Sum of diagonal entries.
$gx^T$ “outer product of g and x”
Each entry is $g_i x_j$ .
$\delta_{ij}$ “Kronecker delta”
One when the indices agree; otherwise zero.
$\operatorname{diag}(p)$ “diagonal matrix of p”
Matrix with p on its diagonal.

Symbol	Say it	Meaning
$dL$	“d L”	First-order loss change.
$\nabla_WL$	“gradient with respect to W”	A matrix with the same shape as W.
$\operatorname{tr}(A)$	“trace of A”	Sum of diagonal entries.
$gx^T$	“outer product of g and x”	Each entry is $g_i x_j$ .
$\delta_{ij}$	“Kronecker delta”	One when the indices agree; otherwise zero.
$\operatorname{diag}(p)$	“diagonal matrix of p”	Matrix with p on its diagonal.

Calculus for Machine Learning · Chamber 7

Taylor Series, the Hessian and Curvature

$R_n$ “Taylor remainder”
Error after a degree-n Taylor polynomial.
$H$ “Hessian”
Matrix of second partial derivatives.
$h^THh$ “h transpose H h”
Curvature contribution along a displacement.
$\lambda_i(H)$ “eigenvalue i of H”
Curvature along a Hessian eigenvector.
$Hh=-g$ “H h equals minus g”
Linear system for a Newton displacement.
$H\succ0$ “H is positive definite”
Every nonzero direction has positive quadratic curvature.

Symbol	Say it	Meaning
$R_n$	“Taylor remainder”	Error after a degree-n Taylor polynomial.
$H$	“Hessian”	Matrix of second partial derivatives.
$h^THh$	“h transpose H h”	Curvature contribution along a displacement.
$\lambda_i(H)$	“eigenvalue i of H”	Curvature along a Hessian eigenvector.
$Hh=-g$	“H h equals minus g”	Linear system for a Newton displacement.
$H\succ0$	“H is positive definite”	Every nonzero direction has positive quadratic curvature.

Calculus for Machine Learning · Chamber 8

Optimisation: Convexity and Constraints

$f((1-t)x+ty)$ “f on a line segment”
Used to compare the graph with its chord.
$H\succeq0$ “H is positive semidefinite”
Every quadratic directional curvature is nonnegative.
$\mathcal L$ “Lagrangian”
Objective plus multiplier-weighted constraints.
$\lambda$ “lambda”
A multiplier for an inequality or equality, with sign rules depending on the constraint.
$g_i(x)\le0$ “inequality constraint”
Our convention for writing feasible inequalities.
$\lambda_i g_i(x)=0$ “complementary slackness”
An inequality with positive multiplier must be tight.

Symbol	Say it	Meaning
$f((1-t)x+ty)$	“f on a line segment”	Used to compare the graph with its chord.
$H\succeq0$	“H is positive semidefinite”	Every quadratic directional curvature is nonnegative.
$\mathcal L$	“Lagrangian”	Objective plus multiplier-weighted constraints.
$\lambda$	“lambda”	A multiplier for an inequality or equality, with sign rules depending on the constraint.
$g_i(x)\le0$	“inequality constraint”	Our convention for writing feasible inequalities.
$\lambda_i g_i(x)=0$	“complementary slackness”	An inequality with positive multiplier must be tight.

Calculus for Machine Learning · Chamber 9

Integrals: From Riemann Sums to Neural ODEs

$\Delta x$ “delta x”
Width of one integration strip.
$\int_a^b f(x)dx$ “integral of f from a to b”
Signed accumulation over an interval.
$F'=f$ “F prime equals f”
An antiderivative relation.
$\left|\det J\right|$ “absolute Jacobian determinant”
Local volume factor in a change of variables.
$O(n^{-2})$ “order n to the minus two”
Error bounded by a constant times n to the minus two for large n.
$\dot z$ “z dot”
Time derivative of the state.

Symbol	Say it	Meaning
$\Delta x$	“delta x”	Width of one integration strip.
$\int_a^b f(x)dx$	“integral of f from a to b”	Signed accumulation over an interval.
$F'=f$	“F prime equals f”	An antiderivative relation.
$\left\|\det J\right\|$	“absolute Jacobian determinant”	Local volume factor in a change of variables.
$O(n^{-2})$	“order n to the minus two”	Error bounded by a constant times n to the minus two for large n.
$\dot z$	“z dot”	Time derivative of the state.

Probability & Statistics for Machine Learning · Chamber 1

Bayes' Rule at Work: Odds and Evidence

$P(H\mid E)$ “probability of H given E”
Posterior belief after observing E.
$P(E\mid H)$ “probability of E given H”
Likelihood of the evidence under H.
$P(E)$ “evidence probability”
Normalising probability across all hypotheses.
$O(H)$ “odds of H”
Probability divided by its complement.
$\log O(H)$ “log odds”
An additive scale for evidence.
$X\perp Y\mid C$ “X independent of Y given C”
The conditional factorisation used by naive Bayes.

Symbol	Say it	Meaning
$P(H\mid E)$	“probability of H given E”	Posterior belief after observing E.
$P(E\mid H)$	“probability of E given H”	Likelihood of the evidence under H.
$P(E)$	“evidence probability”	Normalising probability across all hypotheses.
$O(H)$	“odds of H”	Probability divided by its complement.
$\log O(H)$	“log odds”	An additive scale for evidence.
$X\perp Y\mid C$	“X independent of Y given C”	The conditional factorisation used by naive Bayes.

Probability & Statistics for Machine Learning · Chamber 2

Distributions and How to Sample Them

$p(k)$ “mass at k”
Probability of a discrete outcome.
$p(x)$ “density at x”
Probability per unit width for a continuous variable.
$F(x)$ “CDF at x”
Probability of being at most x.
$F^{-1}(u)$ “quantile at level u”
The smallest point with cumulative probability at least u.
$X\sim\mathcal N(\mu,\sigma^2)$ “X is normally distributed”
A Gaussian with specified mean and variance.
$\binom nk$ “n choose k”
Number of k-element subsets of n trials.

Symbol	Say it	Meaning
$p(k)$	“mass at k”	Probability of a discrete outcome.
$p(x)$	“density at x”	Probability per unit width for a continuous variable.
$F(x)$	“CDF at x”	Probability of being at most x.
$F^{-1}(u)$	“quantile at level u”	The smallest point with cumulative probability at least u.
$X\sim\mathcal N(\mu,\sigma^2)$	“X is normally distributed”	A Gaussian with specified mean and variance.
$\binom nk$	“n choose k”	Number of k-element subsets of n trials.

Probability & Statistics for Machine Learning · Chamber 3

Expectation, Variance and Initialisation

$\mathbb E[X]$ “expectation of X”
Probability-weighted average.
$\mathbb E[X^2]$ “second moment of X”
Average square without subtracting the mean.
$\operatorname{Var}(X)$ “variance of X”
Expected squared deviation from the mean.
$\operatorname{Cov}(X,Y)$ “covariance of X and Y”
Expected product of centred deviations.
$\rho_{XY}$ “correlation of X and Y”
Covariance divided by both positive standard deviations.
$n_{in}$ “fan in”
Number of inputs contributing to a unit.

Symbol	Say it	Meaning
$\mathbb E[X]$	“expectation of X”	Probability-weighted average.
$\mathbb E[X^2]$	“second moment of X”	Average square without subtracting the mean.
$\operatorname{Var}(X)$	“variance of X”	Expected squared deviation from the mean.
$\operatorname{Cov}(X,Y)$	“covariance of X and Y”	Expected product of centred deviations.
$\rho_{XY}$	“correlation of X and Y”	Covariance divided by both positive standard deviations.
$n_{in}$	“fan in”	Number of inputs contributing to a unit.

Probability & Statistics for Machine Learning · Chamber 4

The Multivariate Gaussian

$\mu$ “mean vector”
Coordinatewise expected value.
$\Sigma$ “covariance matrix”
Expected outer product of centred deviations.
$\Sigma=LL^T$ “Cholesky factorisation”
A lower-triangular factor for positive definite covariance.
$(x-\mu)^T\Sigma^{-1}(x-\mu)$ “squared Mahalanobis distance”
Distance adjusted for covariance geometry.
$\mathcal N(\mu,\Sigma)$ “multivariate normal”
A jointly Gaussian vector with the specified moments.
$\bar\alpha_t$ “alpha bar t”
Product of signal-retention factors through time t.

Symbol	Say it	Meaning
$\mu$	“mean vector”	Coordinatewise expected value.
$\Sigma$	“covariance matrix”	Expected outer product of centred deviations.
$\Sigma=LL^T$	“Cholesky factorisation”	A lower-triangular factor for positive definite covariance.
$(x-\mu)^T\Sigma^{-1}(x-\mu)$	“squared Mahalanobis distance”	Distance adjusted for covariance geometry.
$\mathcal N(\mu,\Sigma)$	“multivariate normal”	A jointly Gaussian vector with the specified moments.
$\bar\alpha_t$	“alpha bar t”	Product of signal-retention factors through time t.

Probability & Statistics for Machine Learning · Chamber 5

Large Numbers and the Central Limit Theorem

$\bar X_n$ “sample mean”
Arithmetic average of n observations.
$\sigma/\sqrt n$ “standard error of the mean”
Sampling spread of an iid average.
$\xrightarrow{P}$ “converges in probability”
Fixed-error probabilities tend to zero.
$\xrightarrow{d}$ “converges in distribution”
Distribution functions converge at continuity points of the limit.
$Z_n$ “standardised average”
Centred mean divided by its standard error.
$\Sigma/B$ “gradient covariance divided by batch size”
Covariance of an independent batch average.

Symbol	Say it	Meaning
$\bar X_n$	“sample mean”	Arithmetic average of n observations.
$\sigma/\sqrt n$	“standard error of the mean”	Sampling spread of an iid average.
$\xrightarrow{P}$	“converges in probability”	Fixed-error probabilities tend to zero.
$\xrightarrow{d}$	“converges in distribution”	Distribution functions converge at continuity points of the limit.
$Z_n$	“standardised average”	Centred mean divided by its standard error.
$\Sigma/B$	“gradient covariance divided by batch size”	Covariance of an independent batch average.

Probability & Statistics for Machine Learning · Chamber 6

Maximum Likelihood: Fitting Models to Data

$L(\theta;x)$ “likelihood of theta given fixed data”
The observation-model expression viewed as a parameter function.
$\ell(\theta)$ “log-likelihood”
Logarithm of the likelihood.
$\hat\theta$ “theta hat”
An estimator computed from data.
$\operatorname*{argmax}_\theta$ “parameter maximising”
Returns a parameter rather than the objective value.
$\operatorname{Bias}(\hat\theta)$ “bias of an estimator”
Expected estimate minus the true parameter.
$\operatorname{NLL}$ “negative log-likelihood”
A loss formed by negating log probability or density.

Symbol	Say it	Meaning
$L(\theta;x)$	“likelihood of theta given fixed data”	The observation-model expression viewed as a parameter function.
$\ell(\theta)$	“log-likelihood”	Logarithm of the likelihood.
$\hat\theta$	“theta hat”	An estimator computed from data.
$\operatorname*{argmax}_\theta$	“parameter maximising”	Returns a parameter rather than the objective value.
$\operatorname{Bias}(\hat\theta)$	“bias of an estimator”	Expected estimate minus the true parameter.
$\operatorname{NLL}$	“negative log-likelihood”	A loss formed by negating log probability or density.

Probability & Statistics for Machine Learning · Chamber 7

Entropy, Cross-Entropy and KL Divergence

$-\log p(x)$ “surprise of x”
Information associated with an outcome probability.
$H(P)$ “entropy of P”
Expected surprise under the same distribution.
$H(P,Q)$ “cross-entropy”
Expected model surprise when observations follow P.
$D_{KL}(P\|Q)$ “KL divergence from P to Q”
Expected log probability ratio under P.
$I(X;Y)$ “mutual information”
Dependence measured against the product of marginals.
$\mathcal L(q)$ “evidence lower bound”
A tractable lower bound on log model evidence.

Symbol	Say it	Meaning
$-\log p(x)$	“surprise of x”	Information associated with an outcome probability.
$H(P)$	“entropy of P”	Expected surprise under the same distribution.
$H(P,Q)$	“cross-entropy”	Expected model surprise when observations follow P.
$D_{KL}(P\\|Q)$	“KL divergence from P to Q”	Expected log probability ratio under P.
$I(X;Y)$	“mutual information”	Dependence measured against the product of marginals.
$\mathcal L(q)$	“evidence lower bound”	A tractable lower bound on log model evidence.

Probability & Statistics for Machine Learning · Chamber 8

Bayesian Inference: Priors, Posteriors and Bandits

$p(\theta)$ “parameter prior”
Belief before observing the current data.
$p(\theta\mid D)$ “parameter posterior”
Updated distribution conditional on data.
$\operatorname{Beta}(\alpha,\beta)$ “Beta distribution”
A density over Bernoulli success probabilities.
$\hat\theta_{MAP}$ “maximum a posteriori estimate”
A mode of the posterior density.
$p(x_{new}\mid D)$ “posterior predictive”
Future-data distribution averaged over the posterior.
$1/\sigma^2$ “precision”
Reciprocal variance for a scalar Gaussian.

Symbol	Say it	Meaning
$p(\theta)$	“parameter prior”	Belief before observing the current data.
$p(\theta\mid D)$	“parameter posterior”	Updated distribution conditional on data.
$\operatorname{Beta}(\alpha,\beta)$	“Beta distribution”	A density over Bernoulli success probabilities.
$\hat\theta_{MAP}$	“maximum a posteriori estimate”	A mode of the posterior density.
$p(x_{new}\mid D)$	“posterior predictive”	Future-data distribution averaged over the posterior.
$1/\sigma^2$	“precision”	Reciprocal variance for a scalar Gaussian.

Probability & Statistics for Machine Learning · Chamber 9

Intervals, Tests and Honest Evaluation

$\alpha$ “significance level”
Target Type I error probability for a valid test.
$H_0$ “null hypothesis”
Model under which the test’s reference distribution is computed.
$p\text{-value}$ “p-value”
Null probability of a result at least as extreme as observed.
$\bar X\pm z\,\mathrm{SE}$ “mean plus or minus critical value times standard error”
A normal-based interval formula under its assumptions.
$b,c$ “discordant counts”
Paired examples where exactly one classifier is correct.
$\alpha/m$ “Bonferroni threshold”
Per-test level controlling familywise error across m tests.

Symbol	Say it	Meaning
$\alpha$	“significance level”	Target Type I error probability for a valid test.
$H_0$	“null hypothesis”	Model under which the test’s reference distribution is computed.
$p\text{-value}$	“p-value”	Null probability of a result at least as extreme as observed.
$\bar X\pm z\,\mathrm{SE}$	“mean plus or minus critical value times standard error”	A normal-based interval formula under its assumptions.
$b,c$	“discordant counts”	Paired examples where exactly one classifier is correct.
$\alpha/m$	“Bonferroni threshold”	Per-test level controlling familywise error across m tests.

Large Language Models: From Tokens to Production · Chamber 1

Tokens and Embeddings: How Text Becomes Numbers

$V$ “V”
Vocabulary size: the number of distinct token ids.
$V=50{,}257$
$d$ “d”
Embedding dimension: the length of every token's vector.
$E\in\mathbb R^{V\times d}$ “E, a V by d real matrix”
The embedding matrix. Row $i$ is the learned vector for token id $i$ .
$\mathbf x_i$ “x sub i”
The one-hot row vector for token id $i$ : a 1 in position $i$ and zeros elsewhere.
$\mathbf x_iE=E_{i,:}$
$(a,b)\to ab$ “merge a and b”
A BPE merge rule: wherever $a$ is followed by $b$ , write the single new token $ab$ .
$\cos(\mathbf u,\mathbf v)$ “cosine similarity of u and v”
$\mathbf u\cdot\mathbf v/(\|\mathbf u\|\|\mathbf v\|)$ : the cosine of the angle between two vectors, from $-1$ to $1$ .
$n_{\text{ctx}}$ “n context”
The context window: the most tokens the model can attend over, prompt and output together.
$v_w,\ v'_w$ “v sub w and v prime sub w”
Word2vec's input and output vectors for word $w$ , kept in two separate tables.

Symbol	Say it	Meaning
$V$ $V=50{,}257$	“V”	Vocabulary size: the number of distinct token ids.
$d$	“d”	Embedding dimension: the length of every token's vector.
$E\in\mathbb R^{V\times d}$	“E, a V by d real matrix”	The embedding matrix. Row $i$ is the learned vector for token id $i$ .
$\mathbf x_i$ $\mathbf x_iE=E_{i,:}$	“x sub i”	The one-hot row vector for token id $i$ : a 1 in position $i$ and zeros elsewhere.
$(a,b)\to ab$	“merge a and b”	A BPE merge rule: wherever $a$ is followed by $b$ , write the single new token $ab$ .
$\cos(\mathbf u,\mathbf v)$	“cosine similarity of u and v”	$\mathbf u\cdot\mathbf v/(\\|\mathbf u\\|\\|\mathbf v\\|)$ : the cosine of the angle between two vectors, from $-1$ to $1$ .
$n_{\text{ctx}}$	“n context”	The context window: the most tokens the model can attend over, prompt and output together.
$v_w,\ v'_w$	“v sub w and v prime sub w”	Word2vec's input and output vectors for word $w$ , kept in two separate tables.

Large Language Models: From Tokens to Production · Chamber 2

Next-Token Prediction and the Transformer

$x_{<t}$ “x before t”
The prefix: every token before position $t$ .
$x_{<3}=(x_1,x_2)$
$p_\theta(x_t\mid x_{<t})$ “p theta of x t given the prefix”
The model's next-token distribution at position $t$ .
$z_t$ “z t, the logits”
One real score per vocabulary token at position $t$ .
$z_t\in\mathbb R^{|\mathcal V|}$
$\operatorname{softmax}(z)_i$ “softmax of z, entry i”
Exponentiate and normalise: $e^{z_i}/\sum_je^{z_j}$ .
$\mathcal L$ “script L, the loss”
Cross-entropy: the mean negative log-likelihood per token.
$\mathcal L=-\frac1T\sum_t\log p_\theta(x_t\mid x_{<t})$
$\operatorname{PPL}$ “perplexity”
$e^{\mathcal L}$ : the effective number of tokens the model is choosing between.
$q_i,\ k_j,\ v_j$ “query i, key j, value j”
Projections used to score position $j$ for position $i$ , and to average.
$q_i=W_Qh_i$
$d_k$ “d k”
Dimension of queries and keys; scores are divided by $\sqrt{d_k}$ .
$M_{ij}$ “the causal mask”
$0$ when $j\le i$ and $-\infty$ when $j>i$ , added to the scores before the softmax.
$|\mathcal V|$ “size of the vocabulary”
The number of classes in every next-token prediction.
$N$ “N, the parameter count”
A forward pass costs about $2N$ FLOPs per token.

Symbol	Say it	Meaning
$x_{<t}$ $x_{<3}=(x_1,x_2)$	“x before t”	The prefix: every token before position $t$ .
$p_\theta(x_t\mid x_{<t})$	“p theta of x t given the prefix”	The model's next-token distribution at position $t$ .
$z_t$ $z_t\in\mathbb R^{\|\mathcal V\|}$	“z t, the logits”	One real score per vocabulary token at position $t$ .
$\operatorname{softmax}(z)_i$	“softmax of z, entry i”	Exponentiate and normalise: $e^{z_i}/\sum_je^{z_j}$ .
$\mathcal L$ $\mathcal L=-\frac1T\sum_t\log p_\theta(x_t\mid x_{<t})$	“script L, the loss”	Cross-entropy: the mean negative log-likelihood per token.
$\operatorname{PPL}$	“perplexity”	$e^{\mathcal L}$ : the effective number of tokens the model is choosing between.
$q_i,\ k_j,\ v_j$ $q_i=W_Qh_i$	“query i, key j, value j”	Projections used to score position $j$ for position $i$ , and to average.
$d_k$	“d k”	Dimension of queries and keys; scores are divided by $\sqrt{d_k}$ .
$M_{ij}$	“the causal mask”	$0$ when $j\le i$ and $-\infty$ when $j>i$ , added to the scores before the softmax.
$\|\mathcal V\|$	“size of the vocabulary”	The number of classes in every next-token prediction.
$N$	“N, the parameter count”	A forward pass costs about $2N$ FLOPs per token.

Large Language Models: From Tokens to Production · Chamber 3

From Pretraining to Assistant

$N$ “N”
Number of parameters in the model.
$N=7\times10^{9}$
$D$ “D”
Number of training tokens.
$D=1.4\times10^{12}$
$C\approx6ND$ “C is about six N D”
Training compute in FLOPs: about $2N$ per token forward and $4N$ backward.
$\hat L(N,D)$ “L hat of N and D”
Predicted final pretraining loss, in nats per token.
$\hat L=E+A/N^{\alpha}+B/D^{\beta}$
$E,\ A,\ B$ “E, A, B”
Constants of the Chinchilla fit: irreducible loss $1.69$ and term scales $406.4$ and $410.7$ .
$\alpha,\ \beta$ “alpha, beta (scaling law)”
Exponents of the model and data terms, fitted as $0.34$ and $0.28$ .
$N_{\text{opt}}(C)$ “N opt of C”
The model size that minimises the predicted loss at budget $C$ .
$N_{\text{opt}}\propto C^{\beta/(\alpha+\beta)}$
$\pi_\theta(y\mid x)$ “pi theta of y given x”
The policy: the model being tuned, as a distribution over responses $y$ to prompt $x$ .
$\pi_{\text{ref}}$ “pi ref”
The frozen reference model, usually the supervised fine-tuned model.
$r_\phi(x,y)$ “r phi of x and y”
A learned reward model's scalar score for response $y$ to prompt $x$ .
$y_w,\ y_l$ “y w, y l”
The preferred (winning) and dispreferred (losing) responses in a comparison.
$\beta$ “beta (RLHF and DPO)”
Strength of the KL penalty to the reference model. Not the scaling-law exponent.
$\beta=0.1$
$\sigma(z)$ “sigma of z”
The logistic sigmoid $1/(1+e^{-z})$ .
$\Delta W=BA$ “delta W equals B A”
LoRA's low-rank update, with $B\in\mathbb R^{d\times r}$ and $A\in\mathbb R^{r\times k}$ (a matrix, not the Chinchilla constant).
$W'=W+BA$

Symbol	Say it	Meaning
$N$ $N=7\times10^{9}$	“N”	Number of parameters in the model.
$D$ $D=1.4\times10^{12}$	“D”	Number of training tokens.
$C\approx6ND$	“C is about six N D”	Training compute in FLOPs: about $2N$ per token forward and $4N$ backward.
$\hat L(N,D)$ $\hat L=E+A/N^{\alpha}+B/D^{\beta}$	“L hat of N and D”	Predicted final pretraining loss, in nats per token.
$E,\ A,\ B$	“E, A, B”	Constants of the Chinchilla fit: irreducible loss $1.69$ and term scales $406.4$ and $410.7$ .
$\alpha,\ \beta$	“alpha, beta (scaling law)”	Exponents of the model and data terms, fitted as $0.34$ and $0.28$ .
$N_{\text{opt}}(C)$ $N_{\text{opt}}\propto C^{\beta/(\alpha+\beta)}$	“N opt of C”	The model size that minimises the predicted loss at budget $C$ .
$\pi_\theta(y\mid x)$	“pi theta of y given x”	The policy: the model being tuned, as a distribution over responses $y$ to prompt $x$ .
$\pi_{\text{ref}}$	“pi ref”	The frozen reference model, usually the supervised fine-tuned model.
$r_\phi(x,y)$	“r phi of x and y”	A learned reward model's scalar score for response $y$ to prompt $x$ .
$y_w,\ y_l$	“y w, y l”	The preferred (winning) and dispreferred (losing) responses in a comparison.
$\beta$ $\beta=0.1$	“beta (RLHF and DPO)”	Strength of the KL penalty to the reference model. Not the scaling-law exponent.
$\sigma(z)$	“sigma of z”	The logistic sigmoid $1/(1+e^{-z})$ .
$\Delta W=BA$ $W'=W+BA$	“delta W equals B A”	LoRA's low-rank update, with $B\in\mathbb R^{d\times r}$ and $A\in\mathbb R^{r\times k}$ (a matrix, not the Chinchilla constant).

Large Language Models: From Tokens to Production · Chamber 4

Sampling: Temperature, Top-k and Top-p

$u_i$ “u sub i”
The logit of token $i$ : the model's raw, unnormalised score.
$u=(3.2,\,2.6,\,2.3)$
$T$ “temperature”
Divides every logit before the softmax; $T=1$ leaves the model's distribution unchanged.
$p_i(T)$ “p sub i at temperature T”
Probability of token $i$ after temperature scaling.
$p_i(T)=e^{u_i/T}/\sum_je^{u_j/T}$
$|V|$ “size of the vocabulary”
Number of tokens the model can emit at each step.
$V^{(k)}$ “top k vocabulary”
The $k$ most probable tokens at this step.
$V^{(p)}$ “top p vocabulary, the nucleus”
The smallest set of most probable tokens whose total probability is at least $p$ .
$p_{\text{base}}\,p_{\max}$ “p base times p max”
The min- $p$ threshold: tokens less probable than this fraction of the top token are removed.
$H(p)$ “entropy of p”
Spread of the next-token distribution in bits, between 0 and $\log_2|V|$ .
$B$ “beam width”
Number of partial sequences beam search keeps at each step.

Symbol	Say it	Meaning
$u_i$ $u=(3.2,\,2.6,\,2.3)$	“u sub i”	The logit of token $i$ : the model's raw, unnormalised score.
$T$	“temperature”	Divides every logit before the softmax; $T=1$ leaves the model's distribution unchanged.
$p_i(T)$ $p_i(T)=e^{u_i/T}/\sum_je^{u_j/T}$	“p sub i at temperature T”	Probability of token $i$ after temperature scaling.
$\|V\|$	“size of the vocabulary”	Number of tokens the model can emit at each step.
$V^{(k)}$	“top k vocabulary”	The $k$ most probable tokens at this step.
$V^{(p)}$	“top p vocabulary, the nucleus”	The smallest set of most probable tokens whose total probability is at least $p$ .
$p_{\text{base}}\,p_{\max}$	“p base times p max”	The min- $p$ threshold: tokens less probable than this fraction of the top token are removed.
$H(p)$	“entropy of p”	Spread of the next-token distribution in bits, between 0 and $\log_2\|V\|$ .
$B$	“beam width”	Number of partial sequences beam search keeps at each step.

Large Language Models: From Tokens to Production · Chamber 5

Non-Determinism: Taming and Using Randomness

$\mathrm{fl}(x)$ “f l of x”
The floating-point number nearest to the real number $x$ : round to nearest, ties to even.
$\mathrm{fl}(10^8+1)=10^8\ \text{in float32}$
$a\oplus b$ “a circle-plus b”
Computed floating-point addition, $\mathrm{fl}(a+b)$ . It is commutative but not associative.
$(1\oplus10^8)\oplus(-10^8)=0$
$u$ “unit roundoff”
The largest relative error of a single rounding: $2^{-24}$ for float32.
$\mathrm{fl}(x)=x(1+\delta),\ |\delta|\le u$
$\mathbb{1}(a_i=a)$ “indicator that a i equals a”
1 if the $i$ th sample's final answer is $a$ , and 0 otherwise.
$m$ “m”
The number of reasoning paths sampled for self-consistency (40 in Wang et al.).
$a_1,\dots,a_m$
$\text{pass@}k$ “pass at k”
The probability that at least one of $k$ samples passes the tests.
$\text{pass@}k=1-(1-p)^k$
$\binom{n}{k}$ “n choose k”
The number of ways to choose $k$ of $n$ samples.
$\binom{5}{2}=10$
$\hat p$ “p hat”
An observed pass rate: passes divided by trials.
$\hat p=x/n$

Symbol	Say it	Meaning
$\mathrm{fl}(x)$ $\mathrm{fl}(10^8+1)=10^8\ \text{in float32}$	“f l of x”	The floating-point number nearest to the real number $x$ : round to nearest, ties to even.
$a\oplus b$ $(1\oplus10^8)\oplus(-10^8)=0$	“a circle-plus b”	Computed floating-point addition, $\mathrm{fl}(a+b)$ . It is commutative but not associative.
$u$ $\mathrm{fl}(x)=x(1+\delta),\ \|\delta\|\le u$	“unit roundoff”	The largest relative error of a single rounding: $2^{-24}$ for float32.
$\mathbb{1}(a_i=a)$	“indicator that a i equals a”	1 if the $i$ th sample's final answer is $a$ , and 0 otherwise.
$m$ $a_1,\dots,a_m$	“m”	The number of reasoning paths sampled for self-consistency (40 in Wang et al.).
$\text{pass@}k$ $\text{pass@}k=1-(1-p)^k$	“pass at k”	The probability that at least one of $k$ samples passes the tests.
$\binom{n}{k}$ $\binom{5}{2}=10$	“n choose k”	The number of ways to choose $k$ of $n$ samples.
$\hat p$ $\hat p=x/n$	“p hat”	An observed pass rate: passes divided by trials.

Large Language Models: From Tokens to Production · Chamber 6

Prompting as Programming

$\mathcal V$ “script V”
The vocabulary: every token the model can emit, including the end token.
$p(v\mid x_{<t})$ “p of v given x before t”
The model's probability that the next token is $v$ , given the prompt and the tokens generated so far.
$\delta(q,v)$ “delta of q and v”
The automaton's transition: the state reached from state $q$ by reading token $v$ , if any.
$q_{t}=\delta(q_{t-1},x_t)$
$A(q)$ “A of q”
The allowed set: tokens that lead from state $q$ to a state from which the schema can still be completed.
$m(q)$ “the mask at q”
A 0/1 vector over the vocabulary with ones exactly on $A(q)$ .
$m(q)\in\{0,1\}^{|\mathcal V|}$
$Z_t$ “Z t, the kept mass”
The probability the model put on allowed tokens at step $t$ .
$Z_t=\sum_{v\in A(q_{t-1})}p(v\mid x_{<t})$
$1-Z_t$ “one minus Z t, the removed mass”
The probability the mask takes away at step $t$ .
$\tilde p(v\mid x_{<t})$ “p tilde of v given x before t”
The masked, renormalised distribution the constrained sampler actually draws from.
$\tilde p(v\mid x_{<t})=m_v\,p(v\mid x_{<t})/Z_t$

Symbol	Say it	Meaning
$\mathcal V$	“script V”	The vocabulary: every token the model can emit, including the end token.
$p(v\mid x_{<t})$	“p of v given x before t”	The model's probability that the next token is $v$ , given the prompt and the tokens generated so far.
$\delta(q,v)$ $q_{t}=\delta(q_{t-1},x_t)$	“delta of q and v”	The automaton's transition: the state reached from state $q$ by reading token $v$ , if any.
$A(q)$	“A of q”	The allowed set: tokens that lead from state $q$ to a state from which the schema can still be completed.
$m(q)$ $m(q)\in\{0,1\}^{\|\mathcal V\|}$	“the mask at q”	A 0/1 vector over the vocabulary with ones exactly on $A(q)$ .
$Z_t$ $Z_t=\sum_{v\in A(q_{t-1})}p(v\mid x_{<t})$	“Z t, the kept mass”	The probability the model put on allowed tokens at step $t$ .
$1-Z_t$	“one minus Z t, the removed mass”	The probability the mask takes away at step $t$ .
$\tilde p(v\mid x_{<t})$ $\tilde p(v\mid x_{<t})=m_v\,p(v\mid x_{<t})/Z_t$	“p tilde of v given x before t”	The masked, renormalised distribution the constrained sampler actually draws from.

Large Language Models: From Tokens to Production · Chamber 7

Prompt Version Control and Evals

$s_i$ “s sub i”
Score of item $i$ : 1 for a pass and 0 for a fail, or the fraction of $K$ sampled answers that pass.
$\hat p$ “p hat”
Observed pass rate, the mean item score over the $n$ items of the suite.
$\hat p=\tfrac1n\textstyle\sum_i s_i$
$\mathrm{SE}$ “standard error”
How much an estimate would vary if the eval items were drawn again.
$\mathrm{SE}=\sqrt{\hat p(1-\hat p)/n}$
$d_i$ “d sub i”
Paired difference on item $i$ : the new version's score minus the old version's score on the same item.
$d_i=s_{\text{new},i}-s_{\text{old},i}$
$\operatorname{Cov}(a,b)$ “covariance of a and b”
How two versions' item scores move together; positive when both find the same items hard.
$K$ “K”
Number of answers sampled per item and version, averaged into the item score.
$h$ “half-width”
Half the width of a confidence interval: $1.96\,\mathrm{SE}$ at 95%.
$b,\ c$ “b and c”
Discordant counts: items only the old version passes, and items only the new version passes.
$\text{MAJOR.MINOR.PATCH}$ “semantic version”
A version number whose parts signal a breaking change, new compatible behaviour, or a fix.
$1.4.0\to1.4.1$

Symbol	Say it	Meaning
$s_i$	“s sub i”	Score of item $i$ : 1 for a pass and 0 for a fail, or the fraction of $K$ sampled answers that pass.
$\hat p$ $\hat p=\tfrac1n\textstyle\sum_i s_i$	“p hat”	Observed pass rate, the mean item score over the $n$ items of the suite.
$\mathrm{SE}$ $\mathrm{SE}=\sqrt{\hat p(1-\hat p)/n}$	“standard error”	How much an estimate would vary if the eval items were drawn again.
$d_i$ $d_i=s_{\text{new},i}-s_{\text{old},i}$	“d sub i”	Paired difference on item $i$ : the new version's score minus the old version's score on the same item.
$\operatorname{Cov}(a,b)$	“covariance of a and b”	How two versions' item scores move together; positive when both find the same items hard.
$K$	“K”	Number of answers sampled per item and version, averaged into the item score.
$h$	“half-width”	Half the width of a confidence interval: $1.96\,\mathrm{SE}$ at 95%.
$b,\ c$	“b and c”	Discordant counts: items only the old version passes, and items only the new version passes.
$\text{MAJOR.MINOR.PATCH}$ $1.4.0\to1.4.1$	“semantic version”	A version number whose parts signal a breaking change, new compatible behaviour, or a fix.

Large Language Models: From Tokens to Production · Chamber 8

Retrieval, Tools and Agents

$z$ “z”
A retrieved passage. The RAG paper treats it as a latent variable and sums over it.
$p_\eta(z\mid x)$ “p eta of z given x”
The retriever's probability of passage $z$ for input $x$ , with retriever parameters $\eta$ .
$p_\theta(y\mid x,z)$ “p theta of y given x and z”
The generator's probability of output $y$ given the input and one passage.
$\mathbf q(x),\ \mathbf d(z)$ “q of x, d of z”
Query and document embeddings from two encoders (a bi-encoder).
$p_\eta(z\mid x)\propto\exp(\mathbf d(z)^\top\mathbf q(x))$
$\cos\theta$ “cosine theta”
Cosine similarity: the dot product divided by both lengths.
$\cos\theta=\frac{\mathbf q\cdot\mathbf d}{\|\mathbf q\|\|\mathbf d\|}$
$\operatorname{idf}(t)$ “inverse document frequency of t”
How rare term $t$ is across $N$ documents; rare terms weigh more.
$\operatorname{idf}(t)=\ln\frac{N}{\mathrm{df}_t}$
$\mathrm{R@}k$ “recall at k”
The fraction of a query's relevant passages that appear in the top $k$ .
$\mathrm{MRR}$ “mean reciprocal rank”
The average over queries of one over the rank of the first relevant result (zero if none).
$p^n$ “p to the n”
Success probability of $n$ independent steps that each succeed with probability $p$ .

Symbol	Say it	Meaning
$z$	“z”	A retrieved passage. The RAG paper treats it as a latent variable and sums over it.
$p_\eta(z\mid x)$	“p eta of z given x”	The retriever's probability of passage $z$ for input $x$ , with retriever parameters $\eta$ .
$p_\theta(y\mid x,z)$	“p theta of y given x and z”	The generator's probability of output $y$ given the input and one passage.
$\mathbf q(x),\ \mathbf d(z)$ $p_\eta(z\mid x)\propto\exp(\mathbf d(z)^\top\mathbf q(x))$	“q of x, d of z”	Query and document embeddings from two encoders (a bi-encoder).
$\cos\theta$ $\cos\theta=\frac{\mathbf q\cdot\mathbf d}{\\|\mathbf q\\|\\|\mathbf d\\|}$	“cosine theta”	Cosine similarity: the dot product divided by both lengths.
$\operatorname{idf}(t)$ $\operatorname{idf}(t)=\ln\frac{N}{\mathrm{df}_t}$	“inverse document frequency of t”	How rare term $t$ is across $N$ documents; rare terms weigh more.
$\mathrm{R@}k$	“recall at k”	The fraction of a query's relevant passages that appear in the top $k$ .
$\mathrm{MRR}$	“mean reciprocal rank”	The average over queries of one over the rank of the first relevant result (zero if none).
$p^n$	“p to the n”	Success probability of $n$ independent steps that each succeed with probability $p$ .

Large Language Models: From Tokens to Production · Chamber 9

Open Weights or API? Choosing and Running Models

$N$ “N”
Total parameters. Every one must be held in memory, including experts a token does not use.
$N_{\text{act}}$ “N active”
Parameters read per generated token: $N$ for a dense model, far fewer for a mixture of experts.
$b$ “b”
Bytes per stored parameter: 2 at 16-bit, 1 at 8-bit, about 0.56 at 4.5 bits per weight.
$M_W=Nb$
$M_{KV}$ “KV-cache memory”
Bytes holding a key and a value for every layer, KV head, token and sequence.
$M_{KV}=2Lh_{kv}d_hTBb_{kv}$
$\Delta$ “delta, the step”
Quantisation step: the gap between neighbouring representable values.
$\Delta=\|\mathbf x\|_\infty/127$
$\lfloor y\rceil$ “y rounded to the nearest integer”
Round to nearest; it moves a number by at most one half.
$\beta$ “beta”
Memory bandwidth, in bytes per second.
$F$ “F”
Fixed monthly cost of running locally: amortised hardware, hosting, upkeep and people's time.
$p,\ c$ “p and c”
API price per token, and the local marginal cost per token (mostly electricity).
$V^*$ “V star”
Break-even monthly token volume.
$V^*=F/(p-c)$

Symbol	Say it	Meaning
$N$	“N”	Total parameters. Every one must be held in memory, including experts a token does not use.
$N_{\text{act}}$	“N active”	Parameters read per generated token: $N$ for a dense model, far fewer for a mixture of experts.
$b$ $M_W=Nb$	“b”	Bytes per stored parameter: 2 at 16-bit, 1 at 8-bit, about 0.56 at 4.5 bits per weight.
$M_{KV}$ $M_{KV}=2Lh_{kv}d_hTBb_{kv}$	“KV-cache memory”	Bytes holding a key and a value for every layer, KV head, token and sequence.
$\Delta$ $\Delta=\\|\mathbf x\\|_\infty/127$	“delta, the step”	Quantisation step: the gap between neighbouring representable values.
$\lfloor y\rceil$	“y rounded to the nearest integer”	Round to nearest; it moves a number by at most one half.
$\beta$	“beta”	Memory bandwidth, in bytes per second.
$F$	“F”	Fixed monthly cost of running locally: amortised hardware, hosting, upkeep and people's time.
$p,\ c$	“p and c”	API price per token, and the local marginal cost per token (mostly electricity).
$V^$ $V^=F/(p-c)$	“V star”	Break-even monthly token volume.