Skip to content
AriadneTechnology
0·0 XP

Reference

The Notation Glossary

Every symbol the academy teaches, in one place: how to say it aloud, what it means and how to type it in LaTeX. Keep it open in a tab while you read papers.

348 symbols

Mathematics for Machine Learning · Chamber 1

Reading Mathematics Aloud

  • α\alpha“alpha”
    A step size or learning rate (Adam calls it α\alpha), or a coefficient in a weighted mix.
    α=0.001\alpha = 0.001
  • β\beta“beta”
    Decay rates, like Adam's β1\beta_1 and β2\beta_2; also regression coefficients.
    β1=0.9\beta_1 = 0.9
  • γ\gamma“gamma”
    The discount factor in reinforcement learning; the learned scale in batch normalisation.
    γ=0.99\gamma = 0.99
  • δ\delta“delta”
    A small change, or the error signal flowing backwards through a network.
    δj\delta_j
  • Δ\Delta“capital delta”
    A difference or change: Δθ\Delta\theta is “the change in θ\theta”.
    Δθ=θt+1−θt\Delta\theta = \theta_{t+1} - \theta_t
  • ϵ\epsilon“epsilon”
    A tiny positive number, like 10−810^{-8}, added to avoid dividing by zero. In proofs, an arbitrarily small tolerance.
    ϵ=10−8\epsilon = 10^{-8}
  • ε\varepsilon“curly epsilon”
    The same letter in another style, often used for random noise.
    y=f(x)+εy = f(x) + \varepsilon
  • η\eta“eta”
    The learning rate in gradient descent and its relatives.
    θ←θ−η g\theta \leftarrow \theta - \eta\, g
  • θ\theta“theta”
    The parameters of a model: every weight and bias, bundled together.
    fθ(x)f_\theta(x)
  • λ\lambda“lambda”
    The strength of a penalty (regularisation); later, an eigenvalue.
    L+λR\mathcal{L} + \lambda R
  • μ\mu“mu”
    A mean (average), especially of a distribution.
    μ=0\mu = 0
  • σ\sigma“sigma”
    A standard deviation (how spread out values are), or the sigmoid function σ(z)\sigma(z).
    σ(z)\sigma(z)
  • Σ\Sigma“capital sigma”
    “Add them all up” (Chamber 5), or a covariance matrix.
    ∑i=1nxi\sum_{i=1}^{n} x_i
  • π\pi“pi”
    The constant 3.14159…3.14159\ldots; in reinforcement learning, a policy.
    π≈3.14159\pi \approx 3.14159
  • Π\Pi“capital pi”
    “Multiply them all together” (Chamber 5).
    ∏i=1npi\prod_{i=1}^{n} p_i
  • ϕ\phi“phi”
    A feature map ϕ(x)\phi(x), or the parameters of a second network.
    ϕ(x)\phi(x)
  • ω\omega“omega”
    A weight or a frequency. Easy to mistake for ww.
    ω1\omega_1
  • τ\tau“tau”
    A temperature (how soft a softmax is) or a time constant.
    τ=0.5\tau = 0.5
  • ρ\rho“rho”
    A correlation or a density. Easy to mistake for pp.
    ρ=0.8\rho = 0.8
  • κ\kappa“kappa”
    A condition number: how badly scaled an optimisation problem is.
    κ=100\kappa = 100
  • ν\nu“nu”
    Degrees of freedom, or another rate. Looks almost exactly like vv.
    ν=3\nu = 3
  • ξ\xi“xi”
    Often a random variable or noise. Said “ksee” or “zai”, and famously hard to handwrite.
    ξ\xi
  • χ\chi“chi”
    Said “kai”. Best known from the χ2\chi^2 (chi-squared) test.
    χ2\chi^2
  • xx“x”
    Plain italic lower case: a scalar, one single number.
    x=3.2x = 3.2
  • x\mathbf{x}“bold x”
    Bold lower case: a vector, an ordered list of numbers.
    x=(x1,x2,x3)\mathbf{x} = (x_1, x_2, x_3)
  • X\mathbf{X}“bold capital X”
    Bold (or plain) upper case: a matrix, a grid of numbers. Often a whole dataset, one example per row.
  • L\mathcal{L}“calligraphic L”
    Curly capitals name special objects: L\mathcal{L} a loss, D\mathcal{D} a dataset, N\mathcal{N} the normal distribution.
    L(θ)\mathcal{L}(\theta)
  • R\mathbb{R}“blackboard bold R”
    Double-struck capitals name number systems. R\mathbb{R} is the real numbers (Chamber 2).
    x∈Rx \in \mathbb{R}
  • softmax\mathrm{softmax}“upright softmax”
    Upright letters spell the names of functions (log⁡\log, exp⁡\exp, softmax\mathrm{softmax}), never a product of variables.
    log⁡x\log x
  • xix_i“x sub i”
    The ii-th entry of a list (or the ii-th example). A subscript is an index: a position.
    x1,x2,…,xnx_1, x_2, \ldots, x_n
  • AijA_{ij}“A sub i j”
    The entry in row ii and column jj of a matrix. Two indices, two directions.
    A23A_{23}
  • x2x^2“x squared”
    A superscript number is usually a power: x⋅xx \cdot x.
    x3=x⋅x⋅xx^3 = x \cdot x \cdot x
  • x(i)x^{(i)}“x superscript i”
    A superscript in brackets is an index, not a power: the ii-th training example.
    x(1),…,x(n)x^{(1)}, \ldots, x^{(n)}
  • θt\theta_t“theta at step t”
    A subscript tt often means “at time step tt”: the value after tt updates.
    θ0,θ1,θ2\theta_0, \theta_1, \theta_2
  • x′x'“x prime”
    Another xx (a second point, a new value). On a function, f′f' means its derivative (Chamber 7).
    f′(x)f'(x)
  • y^\hat{y}“y hat”
    A hat means estimated or predicted: y^\hat{y} is the model's guess at yy.
    y^=fθ(x)\hat{y} = f_\theta(x)
  • xˉ\bar{x}“x bar”
    A bar means an average: xˉ\bar{x} is the mean of the xx values.
    xˉ=12(x1+x2)\bar{x} = \tfrac{1}{2}(x_1 + x_2)
  • x~\tilde{x}“x tilde”
    A tilde means a modified version: noisy, approximate or transformed.
    x~=x+ε\tilde{x} = x + \varepsilon
  • θ⋆\theta^\star“theta star”
    A star means optimal: the best possible value. Also written θ∗\theta^*.
    θ⋆\theta^\star
  • v⃗\vec{v}“v arrow”
    An arrow marks a vector in school and physics. ML papers prefer bold, v\mathbf{v}.
    v⃗\vec{v}
  • ==“equals”
    Both sides are the same value. A claim you could check.
    2+2=42 + 2 = 4
  • :=:=“is defined as”
    The left side is a new name for the right side. Not a claim to check: a definition.
    xˉ:=12(x1+x2)\bar{x} := \tfrac{1}{2}(x_1 + x_2)
  • ≜\triangleq“is defined as (delta-equals)”
    Another way to write a definition, common in engineering papers.
    f(x)≜x2f(x) \triangleq x^2
  • ≡\equiv“is identical to”
    Equal for every value of the variables (an identity). Some authors also use it for definitions.
    (a+b)2≡a2+2ab+b2(a+b)^2 \equiv a^2 + 2ab + b^2
  • ≈\approx“is approximately”
    Close, but not exactly equal.
    π≈3.14\pi \approx 3.14
  • ∝\propto“is proportional to”
    Equal up to a constant factor: y∝xy \propto x means y=c xy = c\,x for some fixed cc.
    y∝x2y \propto x^2
  • ≪\ll“is much less than”
    Smaller by a large factor.
    r≪dr \ll d
  • ←\leftarrow“gets”
    Assignment in an algorithm: compute the right side, store it in the left. An update, not an equation.
    m←m+1m \leftarrow m + 1
  • ■\blacksquare“end of proof”
    The “tombstone”: the proof is finished. Older texts write Q.E.D.
    ■\blacksquare

Mathematics for Machine Learning · Chamber 2

Sets: The Nouns of Mathematics

  • ∈\in“is an element of”
    Membership: x∈Ax \in A says the object xx belongs to the set AA. Also read “is in” or “belongs to”.
    3∈{1,2,3}3 \in \{1, 2, 3\}
  • ∉\notin“is not an element of”
    x∉Ax \notin A says xx does not belong to AA.
    4∉{1,2,3}4 \notin \{1, 2, 3\}
  • {a,b,c}\{a, b, c\}“the set containing a, b and c”
    Curly braces list a set's elements (roster notation). Order and repeats don't matter: {1,2}={2,1,1}\{1, 2\} = \{2, 1, 1\}.
    {cat,dog,bird}\{\text{cat}, \text{dog}, \text{bird}\}
  • ∅\varnothing“the empty set”
    The set with no elements at all, {}\{\}. Also printed ∅\emptyset.
    ∣∅∣=0|\varnothing| = 0
  • ∣A∣|A|“the size of A”
    The number of elements of a finite set AA, called its cardinality. Around a number the same bars mean absolute value, so check what's inside.
    ∣{2,4,6}∣=3|\{2, 4, 6\}| = 3
  • ⊆\subseteq“is a subset of”
    A⊆BA \subseteq B: every element of AA is also an element of BB. AA is allowed to equal BB.
    {1,2}⊆{1,2,3}\{1, 2\} \subseteq \{1, 2, 3\}
  • ⊊\subsetneq“is a proper subset of”
    A⊊BA \subsetneq B: A⊆BA \subseteq B and A≠BA \ne B, so BB has at least one element that AA lacks.
    N⊊Z\mathbb{N} \subsetneq \mathbb{Z}
  • ⊂\subset“is a subset of (strict or not, check)”
    Ambiguous: some authors mean ⊆\subseteq, others ⊊\subsetneq. ML papers mostly use it loosely for ⊆\subseteq. Check the paper's convention.
    S⊂RS \subset \mathbb{R}
  • {x∈S:P(x)}\{x \in S : P(x)\}“the set of x in S such that P of x”
    Go through SS and keep each xx that passes the test PP. The colon reads “such that”.
    {x∈Z:x>0}\{x \in \mathbb{Z} : x > 0\}
  • {x∣P(x)}\{x \mid P(x)\}“the set of all x such that P of x”
    The same idea with a vertical bar instead of a colon: {x∣x2=4}={−2,2}\{x \mid x^2 = 4\} = \{-2, 2\}. Where xx comes from should be clear from context.
    {x∣x2=4}\{x \mid x^2 = 4\}
  • {f(x):x∈S}\{f(x) : x \in S\}“the set of f of x for x in S”
    A formula on the left builds the elements: apply ff to every xx in SS and collect the results.
    {2k:k∈Z}\{2k : k \in \mathbb{Z}\}
  • N\mathbb{N}“the natural numbers”
    The counting numbers. Authors disagree about whether 0∈N0 \in \mathbb{N}, so check. N0\mathbb{N}_0 or Z≥0\mathbb{Z}_{\ge 0} removes the doubt.
    n∈Nn \in \mathbb{N}
  • Z\mathbb{Z}“the integers”
    Whole numbers, positive, negative and zero: {…,−2,−1,0,1,2,…}\{\ldots, -2, -1, 0, 1, 2, \ldots\}.
    −3∈Z-3 \in \mathbb{Z}
  • Q\mathbb{Q}“the rational numbers”
    Fractions p/qp/q with p,q∈Zp, q \in \mathbb{Z} and q≠0q \ne 0.
    34∈Q\tfrac{3}{4} \in \mathbb{Q}
  • R\mathbb{R}“the real numbers”
    Every point on the number line, including 2\sqrt{2} and π\pi. The home of weights, losses and learning rates.
    η∈R\eta \in \mathbb{R}
  • R≥0\mathbb{R}_{\ge 0}“the non-negative reals”
    Real numbers that are ≥0\ge 0: distances, variances, ReLU outputs.
    σ2∈R≥0\sigma^2 \in \mathbb{R}_{\ge 0}
  • R+\mathbb{R}_{+}“R plus”
    The positive reals, or the non-negative reals: authors differ, so check. Learning rates live here.
    η∈R+\eta \in \mathbb{R}_{+}
  • [a,b][a, b]“the closed interval from a to b”
    All reals xx with a≤x≤ba \le x \le b: both ends included.
    p∈[0,1]p \in [0, 1]
  • (a,b)(a, b)“the open interval from a to b”
    All reals xx with a<x<ba < x < b: both ends excluded. The same brackets also write an ordered pair, so read the context.
    σ(z)∈(0,1)\sigma(z) \in (0, 1)
  • [a,b)[a, b)“the half-open interval from a to b”
    All reals xx with a≤x<ba \le x < b. A square bracket includes its end, a round one excludes it.
    u∈[0,1)u \in [0, 1)
  • ∪\cup“union”
    A∪BA \cup B: everything in AA or in BB (or in both).
    {1}∪{2}={1,2}\{1\} \cup \{2\} = \{1, 2\}
  • ∩\cap“intersection”
    A∩BA \cap B: everything in both AA and BB.
    {1,2}∩{2,3}={2}\{1, 2\} \cap \{2, 3\} = \{2\}
  • ∖\setminus“set minus”
    A∖BA \setminus B: everything in AA that is not in BB. Some authors write A−BA - B.
    {1,2}∖{2,3}={1}\{1, 2\} \setminus \{2, 3\} = \{1\}
  • AcA^{c}“the complement of A”
    Everything in the universe UU that is not in AA: Ac=U∖AA^c = U \setminus A.
    (Ac)c=A(A^{c})^{c} = A
  • A‾\overline{A}“A bar, the complement”
    Another way to write the complement. In statistics the same bar means an average (xˉ\bar{x}, Chamber 1), so read the context.
    A∪B‾=A‾∩B‾\overline{A \cup B} = \overline{A} \cap \overline{B}
  • UU“the universal set”
    Everything under discussion. Complements are taken inside it. Also written Ω\Omega or U\mathcal{U}.
    A⊆UA \subseteq U
  • A∩B=∅A \cap B = \varnothing“A and B are disjoint”
    The two sets share no elements, like a training set and a test set should.
    Dtrain∩Dtest=∅\mathcal{D}_{\text{train}} \cap \mathcal{D}_{\text{test}} = \varnothing
  • (a,b)(a, b)“the ordered pair a, b”
    Two objects in a fixed order: (1,2)≠(2,1)(1, 2) \ne (2, 1), unlike the set {1,2}={2,1}\{1, 2\} = \{2, 1\}.
    (xi,yi)(x_i, y_i)
  • (x1,…,xn)(x_1, \ldots, x_n)“the tuple x one to x n”
    An ordered list of nn entries, called an nn-tuple. Order and repeats both count.
    (3,1,3)≠(1,3)(3, 1, 3) \ne (1, 3)
  • A×BA \times B“A cross B”
    The Cartesian product: every ordered pair (a,b)(a, b) with a∈Aa \in A and b∈Bb \in B. It has ∣A∣⋅∣B∣|A| \cdot |B| elements.
    {0,1}×{a,b}\{0, 1\} \times \{a, b\}
  • Rn\mathbb{R}^n“R n”
    All nn-tuples of real numbers, R×⋯×R\mathbb{R} \times \cdots \times \mathbb{R} with nn factors. A vector x∈Rn\mathbf{x} \in \mathbb{R}^n is one element.
    x∈R784\mathbf{x} \in \mathbb{R}^{784}
  • Rm×n\mathbb{R}^{m \times n}“R m by n”
    All grids of real numbers with mm rows and nn columns: real matrices (Chamber 6).
    W∈R512×64W \in \mathbb{R}^{512 \times 64}
  • {1,…,K}\{1, \ldots, K\}“the set one to K”
    The integers from 1 to KK: the usual label set for a classifier with KK classes. Some papers write [K][K].
    y∈{1,…,K}y \in \{1, \ldots, K\}
  • {(xi,yi)}i=1n\{(x_i, y_i)\}_{i=1}^{n}“the set of pairs x i, y i for i from 1 to n”
    A dataset: one (input, label) pair for each index ii from 1 to nn.
    D={(xi,yi)}i=1n\mathcal{D} = \{(\mathbf{x}_i, y_i)\}_{i=1}^{n}

Mathematics for Machine Learning · Chamber 3

Logic: For All and There Exists

  • ¬\neg“not”
    Negation: ¬P\neg P is true exactly when PP is false. Some authors write ∼P\sim P.
    ¬(x>0)\neg (x > 0)
  • ∧\land“and”
    Conjunction: P∧QP \land Q is true only when both PP and QQ are true.
    x>0∧x<1x > 0 \land x < 1
  • ∨\lor“or”
    Disjunction: P∨QP \lor Q is true when at least one of them is. Mathematical “or” is inclusive: both is fine.
    x<0∨x>1x < 0 \lor x > 1
  • ⇒\Rightarrow“implies”
    P⇒QP \Rightarrow Q, “if PP then QQ”: false only when PP is true and QQ is false. Also written   ⟹  \implies or →\to.
    x>2⇒x2>4x > 2 \Rightarrow x^2 > 4
  • ⇐\Leftarrow“is implied by”
    P⇐QP \Leftarrow Q means Q⇒PQ \Rightarrow P. Proofs of an “if and only if” often label their two halves (⇒)(\Rightarrow) and (⇐)(\Leftarrow).
    x2>4⇐x>2x^2 > 4 \Leftarrow x > 2
  • ⇔\Leftrightarrow“if and only if”
    P⇔QP \Leftrightarrow Q: each implies the other, so they are always true or false together. Also written   ⟺  \iff.
    n even⇔n2 evenn \text{ even} \Leftrightarrow n^2 \text{ even}
  • ≡\equiv“is logically equivalent to”
    Two statements with identical truth tables, like an implication and its contrapositive. (Chamber 1 met ≡\equiv as “is identical to”: the same idea, for logic.)
    ¬¬P≡P\neg\neg P \equiv P
  • ∀\forall“for all”
    The universal quantifier: what follows holds for every element. Also read “for every” or “for each”.
    ∀x∈R,  x2≥0\forall x \in \mathbb{R},\; x^2 \ge 0
  • ∃\exists“there exists”
    The existential quantifier: at least one element makes what follows true. One witness is enough.
    ∃x∈R,  x2=2\exists x \in \mathbb{R},\; x^2 = 2
  • ∃!\exists!“there exists exactly one”
    Existence and uniqueness: one element works, and no second one does.
    ∃! x∈R,  2x=6\exists!\, x \in \mathbb{R},\; 2x = 6
  • ∄\nexists“there does not exist”
    No element works: the same as ¬∃\neg\exists, or “for all, not”.
    ∄x∈R,  x2<0\nexists x \in \mathbb{R},\; x^2 < 0
  • s.t.\text{s.t.}“such that”
    Introduces the condition after a quantifier. Papers also use a colon, a vertical bar or just a comma.
    ∃θ s.t. L(θ)=0\exists \theta \text{ s.t. } \mathcal{L}(\theta) = 0
  • iff\text{iff}“if and only if”
    The written-out form of ⇔\Leftrightarrow, said exactly the same way. Two statements are claimed, one in each direction.
    n even iff n2 evenn \text{ even iff } n^2 \text{ even}
  • WLOG\text{WLOG}“without loss of generality”
    “The other cases work the same way, so we only treat this one.” Honest only when the cases really are symmetric.
    WLOG, a≤b\text{WLOG, } a \le b
  • i.e.\text{i.e.}“that is”
    Latin id est: a restatement of what was just said, in other words.
    x∈R≥0, i.e. x≥0x \in \mathbb{R}_{\ge 0}, \text{ i.e. } x \ge 0
  • e.g.\text{e.g.}“for example”
    Latin exempli gratia: one example among several, not a restatement.
    e.g. ReLU\text{e.g. ReLU}
  • ∴\therefore“therefore”
    Marks a conclusion. Rare in papers, common on whiteboards.
    ∴  x=3\therefore\; x = 3
  • ⇒ ⁣⇐\Rightarrow\!\Leftarrow“contradiction”
    Marks the moment a proof by contradiction reaches something impossible. Also drawn as ⊥\bot or a lightning bolt.
    p=2k=2m+1  ⇒ ⁣⇐p = 2k = 2m + 1 \;\Rightarrow\!\Leftarrow

Mathematics for Machine Learning · Chamber 4

Functions: Machines That Map

  • f:X→Yf : X \to Y“f from X to Y”
    A function named ff with domain XX (the allowed inputs) and codomain YY (where outputs are promised to live). Every input gets exactly one output.
    f:R→Rf : \mathbb{R} \to \mathbb{R}
  • x↦x2x \mapsto x^2“x maps to x squared”
    The rule: what happens to a single input. This arrow has a little tail; the plain arrow →\to connects whole sets.
    f:x↦x2f : x \mapsto x^2
  • f(x)f(x)“f of x”
    The output of ff at the input xx. The brackets mean “of”, not multiplication.
    f(3)=9f(3) = 9
  • f(X)f(X)“the image of X under f”
    The set of outputs ff actually produces, {f(x):x∈X}\{f(x) : x \in X\}. It sits inside the codomain and can be smaller.
    f(R)=[0,∞)f(\mathbb{R}) = [0, \infty)
  • f(x;θ)f(x; \theta)“f of x semicolon theta”
    A model: the input xx before the semicolon, the parameters θ\theta after it. Training changes θ\theta; prediction changes xx.
    f(x;θ)f(\mathbf{x}; \theta)
  • fθ(x)f_\theta(x)“f sub theta of x”
    The same idea with the parameters as a subscript: one function for every choice of θ\theta.
    fθ(x)f_\theta(\mathbf{x})
  • f(x,y)f(x, y)“f of x and y”
    A function of two inputs. Its domain is a Cartesian product such as R×R\mathbb{R} \times \mathbb{R} (Chamber 2).
    L^(N,D)\hat{L}(N, D)
  • f∘gf \circ g“f composed with g”
    “ff after gg”: apply gg first, then ff. The notation reads right to left.
    (f∘g)(x)=f(g(x))(f \circ g)(x) = f(g(x))
  • f(g(x))f(g(x))“f of g of x”
    Nested brackets: the innermost function acts first, exactly as in code.
    σ(Wx+b)\sigma(W\mathbf{x} + \mathbf{b})
  • id\mathrm{id}“the identity function”
    The machine that changes nothing: id(x)=x\mathrm{id}(x) = x. Composing with it leaves any function unchanged.
    f∘id=ff \circ \mathrm{id} = f
  • f−1f^{-1}“f inverse”
    The function that undoes ff: f−1(f(x))=xf^{-1}(f(x)) = x. It is not 1/f1/f. It exists only when ff is a bijection.
    f−1∘f=idf^{-1} \circ f = \mathrm{id}
  • fL∘⋯∘f1f_L \circ \cdots \circ f_1“f L after dots after f 1”
    A deep network: LL layers applied in turn, f1f_1 first and fLf_L last.
    f=fL∘⋯∘f2∘f1f = f_L \circ \cdots \circ f_2 \circ f_1
  • {aif Pbotherwise\begin{cases} a & \text{if } P \\ b & \text{otherwise} \end{cases}“a if P, otherwise b”
    A piecewise definition. The brace lists the cases, and exactly one applies to each input.
  • ∣x∣|x|“the absolute value of x”
    The size of xx without its sign: xx if x≥0x \ge 0, and −x-x if x<0x < 0.
    ∣−3∣=3|-3| = 3
  • ⌊x⌋\lfloor x \rfloor“the floor of x”
    Round down to the nearest integer. Careful with negatives: ⌊−2.3⌋=−3\lfloor -2.3 \rfloor = -3.
    ⌊2.7⌋=2\lfloor 2.7 \rfloor = 2
  • ⌈x⌉\lceil x \rceil“the ceiling of x”
    Round up to the nearest integer.
    ⌈2.2⌉=3\lceil 2.2 \rceil = 3
  • max⁡(a,b)\max(a, b)“the max of a and b”
    The larger of the two numbers.
    ReLU(x)=max⁡(0,x)\mathrm{ReLU}(x) = \max(0, x)
  • min⁡(a,b)\min(a, b)“the min of a and b”
    The smaller of the two numbers.
    min⁡(3,−1)=−1\min(3, -1) = -1
  • 1[P]\mathbf{1}[P]“the indicator of P”
    1 if the statement PP is true, 0 if it's false. Also written 1P\mathbf{1}_{P}, I[P]\mathbb{I}[P] or with a double-struck one.
    1[y^=y]\mathbf{1}[\hat{y} = y]
  • δij\delta_{ij}“Kronecker delta i j”
    1 if i=ji = j and 0 otherwise: an indicator for “same index”.
    δ33=1, δ34=0\delta_{33} = 1,\ \delta_{34} = 0
  • ee“e, Euler's number”
    The constant 2.71828…2.71828\ldots, the natural base for exponentials and logarithms.
    e≈2.718e \approx 2.718
  • exe^{x}“e to the x”
    The exponential function. Always positive, and it turns sums into products: ea+b=eaebe^{a+b} = e^{a}e^{b}.
    e0=1e^{0} = 1
  • exp⁡(x)\exp(x)“exp of x”
    The same function written as a name, easier to read when the exponent is long.
    exp⁡(−x22)\exp\left(-\tfrac{x^2}{2}\right)
  • log⁡x\log x“log x”
    The inverse of exp. In ML papers a log with no base almost always means the natural log, base ee.
    log⁡ex=x\log e^{x} = x
  • ln⁡x\ln x“natural log of x”
    The natural log, written explicitly. The same function as ML's log⁡\log, and as Python's math.log.
    ln⁡e=1\ln e = 1
  • log⁡2x\log_2 x“log base two of x”
    How many times you must double 1 to reach xx. Information measured in bits uses base 2.
    log⁡28=3\log_2 8 = 3
  • log⁡10x\log_{10} x“log base ten of x”
    How many times you must multiply 1 by ten to reach xx: roughly, the number of digits.
    log⁡101000=3\log_{10} 1000 = 3
  • σ(z)\sigma(z)“sigma of z”
    The sigmoid 11+e−z\frac{1}{1 + e^{-z}}. It squashes any real number into (0,1)(0, 1), so its output can be read as a probability.
    σ(0)=12\sigma(0) = \tfrac{1}{2}
  • tanh⁡(z)\tanh(z)“tanh of z”
    The hyperbolic tangent: squashes into (−1,1)(-1, 1). It is a sigmoid, stretched and shifted: tanh⁡z=2σ(2z)−1\tanh z = 2\sigma(2z) - 1.
    tanh⁡(0)=0\tanh(0) = 0
  • ReLU(z)\mathrm{ReLU}(z)“relu of z”
    The rectified linear unit, max⁡(0,z)\max(0, z): negative inputs become 0, positive ones pass through.
    ReLU(−2)=0\mathrm{ReLU}(-2) = 0
  • softplus(z)\mathrm{softplus}(z)“softplus of z”
    log⁡(1+ez)\log(1 + e^{z}): a smooth version of ReLU that is always positive. The Deep Learning book writes it ζ(z)\zeta(z).
    softplus(0)=log⁡2\mathrm{softplus}(0) = \log 2
  • logit(p)\mathrm{logit}(p)“logit of p”
    The inverse of the sigmoid, log⁡p1−p\log\frac{p}{1-p}. The raw scores a classifier feeds to its sigmoid or softmax are called logits after it.
    logit(12)=0\mathrm{logit}\left(\tfrac{1}{2}\right) = 0
  • softmax(z)i\mathrm{softmax}(\mathbf{z})_i“softmax of z, entry i”
    A preview: turns a vector of scores into a vector of probabilities. The ∑\sum in its formula is Chamber 5's, and Chamber 9 decodes it in full.
    softmax(z)i=ezi∑jezj\mathrm{softmax}(\mathbf{z})_i = \frac{e^{z_i}}{\sum_j e^{z_j}}

Mathematics for Machine Learning · Chamber 5

Σ and Π: Loops Written in Maths

  • ∑i=1nxi\sum_{i=1}^{n} x_i“the sum from i equals 1 to n of x sub i”
    Add xix_i for i=1,2,…,ni = 1, 2, \ldots, n. Below the Σ\Sigma: the index and where it starts. Above: where it stops. Both ends are included.
    ∑i=13i=6\sum_{i=1}^{3} i = 6
  • ii“the index”
    The dummy variable: a loop counter that exists only inside the sum. Renaming it changes nothing.
    ∑i=1nxi=∑k=1nxk\sum_{i=1}^{n} x_i = \sum_{k=1}^{n} x_k
  • ∑ixi\sum_{i} x_i“the sum over i of x sub i”
    Bounds left out: add over every ii that makes sense in context, usually all the data.
  • ∑i∈Bxi\sum_{i \in \mathcal{B}} x_i“the sum over i in B of x sub i”
    Add over the elements of a set, such as a mini-batch B\mathcal{B}. There are ∣B∣|\mathcal{B}| terms.
    1∣B∣∑i∈Bℓi\frac{1}{|\mathcal{B}|}\sum_{i \in \mathcal{B}} \ell_i
  • ∑j≠ixj\sum_{j \ne i} x_j“the sum over j not equal to i”
    A condition under the Σ\Sigma: add over every jj except j=ij = i.
    ∑j≠ixj=∑jxj−xi\sum_{j \ne i} x_j = \sum_j x_j - x_i
  • ∑i=1m∑j=1naij\sum_{i=1}^{m}\sum_{j=1}^{n} a_{ij}“the double sum over i and j of a sub i j”
    A sum of sums: two nested loops over a grid of mnmn terms. For finite sums the two sums can be swapped.
  • x1+⋯+xnx_1 + \cdots + x_n“x 1 plus dots plus x n”
    An ellipsis: “and so on, following the pattern”. Centred dots ⋯\cdots go between operations, low dots …\ldots in lists.
    1+2+⋯+n1 + 2 + \cdots + n
  • ∏i=1nxi\prod_{i=1}^{n} x_i“the product from i equals 1 to n of x sub i”
    Multiply the terms instead of adding them. An empty product is 1.
    ∏i=13xi=x1x2x3\prod_{i=1}^{3} x_i = x_1 x_2 x_3
  • n!n!“n factorial”
    1⋅2⋯n=∏k=1nk1 \cdot 2 \cdots n = \prod_{k=1}^{n} k, the number of ways to put nn things in order. By the empty-product rule, 0!=10! = 1.
    5!=1205! = 120
  • log⁡∏ipi\log \prod_i p_i“the log of the product of p sub i”
    Equals ∑ilog⁡pi\sum_i \log p_i: a product of many small probabilities becomes a sum that a computer can store.
    log⁡∏ipi=∑ilog⁡pi\log \prod_i p_i = \sum_i \log p_i
  • μ\mu“mu, the mean”
    The average, 1n∑ixi\frac{1}{n}\sum_i x_i. Also written xˉ\bar{x}.
    μ=1n∑ixi\mu = \tfrac{1}{n}\sum_{i} x_i
  • σ2\sigma^2“sigma squared, the variance”
    The average squared distance from the mean: how spread out the values are.
    σ2=1n∑i(xi−μ)2\sigma^2 = \tfrac{1}{n}\sum_{i} (x_i - \mu)^2
  • σ\sigma“sigma, the standard deviation”
    σ2\sqrt{\sigma^2}: the spread in the same units as the data.
    σ=σ2\sigma = \sqrt{\sigma^2}
  • μB,σB2\mu_{\mathcal{B}}, \sigma^2_{\mathcal{B}}“mu B and sigma squared B”
    The mean and variance of one mini-batch B\mathcal{B}, as in batch normalisation.
  • x^i\widehat{x}_i“x hat sub i, normalised”
    In batch normalisation, xix_i minus the batch mean, divided by the batch standard deviation: how many standard deviations xix_i sits from the mean.

Mathematics for Machine Learning · Chamber 6

Vectors and Matrices: Reading the Shapes

  • x∈Rn\mathbf{x} \in \mathbb{R}^n“x in R n”
    x\mathbf{x} is a list of nn real numbers: a vector with nn entries. Unless a paper says otherwise, it stands as a column, n×1n \times 1.
    x∈R3\mathbf{x} \in \mathbb{R}^{3}
  • xix_i“x sub i”
    The ii-th entry of x\mathbf{x}. It's a single number, so it's printed plain, not bold.
    x=(x1,x2,x3)⊤\mathbf{x} = (x_1, x_2, x_3)^\top
  • x⊤\mathbf{x}^\top“x transpose”
    The same numbers laid out as a row, 1×n1 \times n. Also printed xT\mathbf{x}^T, xT\mathbf{x}^{\mathsf{T}} or x′\mathbf{x}'. It is never a power.
    x⊤=(x1,…,xn)\mathbf{x}^\top = (x_1, \ldots, x_n)
  • 0\mathbf{0}“the zero vector”
    Every entry is 00. Its length comes from context, or a subscript: 0n\mathbf{0}_n.
    x+0=x\mathbf{x} + \mathbf{0} = \mathbf{x}
  • 1\mathbf{1}“the ones vector”
    Every entry is 11. Handy for sums: 1⊤x=x1+⋯+xn\mathbf{1}^\top \mathbf{x} = x_1 + \cdots + x_n.
    1⊤x=∑ixi\mathbf{1}^\top \mathbf{x} = \sum_{i} x_i
  • ei\mathbf{e}_i“e sub i”
    The ii-th standard basis vector: zeros everywhere except a 11 in position ii. In ML it's a one-hot vector.
    e3=(0,0,1,0)⊤∈R4\mathbf{e}_3 = (0, 0, 1, 0)^\top \in \mathbb{R}^4
  • ∥x∥2\|\mathbf{x}\|_2“the two-norm of x”
    The ordinary length, x12+⋯+xn2\sqrt{x_1^2 + \cdots + x_n^2}. Often written just ∥x∥\|\mathbf{x}\|.
    ∥(3,4)∥2=5\|(3, 4)\|_2 = 5
  • ∥x∥1\|\mathbf{x}\|_1“the one-norm of x”
    Add up the sizes of the entries: ∣x1∣+⋯+∣xn∣|x_1| + \cdots + |x_n|.
    ∥(3,−4)∥1=7\|(3, -4)\|_1 = 7
  • ∥x∥22\|\mathbf{x}\|_2^2“the squared two-norm of x”
    ∑ixi2\sum_i x_i^2, with no square root. Squared errors and weight penalties are written this way.
    ∥x∥22=x⊤x\|\mathbf{x}\|_2^2 = \mathbf{x}^\top \mathbf{x}
  • ⟨x,y⟩\langle \mathbf{x}, \mathbf{y} \rangle“the inner product of x and y, in angle brackets”
    Multiply matching entries and add: ∑ixiyi\sum_i x_i y_i. Both vectors need the same length.
    ⟨x,y⟩=∑i=1nxiyi\langle \mathbf{x}, \mathbf{y} \rangle = \sum_{i=1}^{n} x_i y_i
  • x⊤y\mathbf{x}^\top \mathbf{y}“x transpose y”
    The same number as ⟨x,y⟩\langle \mathbf{x}, \mathbf{y} \rangle, written as a matrix product: a 1×n1 \times n row times an n×1n \times 1 column.
    w⊤x+b\mathbf{w}^\top \mathbf{x} + b
  • x⋅y\mathbf{x} \cdot \mathbf{y}“x dot y”
    A third name for the same number, the dot product. Physics and school books prefer this one.
    q⋅k=∑i=1dkqikiq \cdot k = \sum_{i=1}^{d_k} q_i k_i
  • A∈Rm×nA \in \mathbb{R}^{m \times n}“A is an m by n matrix”
    A grid of real numbers with mm rows and nn columns: mnmn numbers in all. Rows first, always.
    W∈R512×64W \in \mathbb{R}^{512 \times 64}
  • AijA_{ij}“A sub i j”
    The entry in row ii, column jj. Also written Ai,jA_{i,j}, aija_{ij} or [A]ij[A]_{ij}.
    A23A_{23}
  • Ai,:A_{i,:}“row i of A”
    The whole ii-th row, a 1×n1 \times n row. The colon means “every index here”, just as in NumPy.
    A2,:A_{2,:}
  • A:,jA_{:,j}“column j of A”
    The whole jj-th column, an m×1m \times 1 column.
    A:,1A_{:,1}
  • A⊤A^\top“A transpose”
    Flip across the diagonal: (A⊤)ij=Aji(A^\top)_{ij} = A_{ji}. Rows become columns, so an m×nm \times n matrix becomes n×mn \times m.
    (A⊤)ij=Aji(A^\top)_{ij} = A_{ji}
  • II“the identity matrix”
    Square, with ones on the diagonal and zeros elsewhere, so Ix=xI\mathbf{x} = \mathbf{x}. Written InI_n when the size matters.
    I3I_3
  • diag(d)\mathrm{diag}(\mathbf{d})“diag of d”
    The square matrix with the entries of d\mathbf{d} down its diagonal and zeros everywhere else.
    diag(2,5)=[2005]\mathrm{diag}(2, 5) = \begin{bmatrix} 2 & 0 \\ 0 & 5 \end{bmatrix}
  • X∈Rn×dX \in \mathbb{R}^{n \times d}“the data matrix X”
    A dataset of nn examples with dd features each: one example per row, one feature per column.
    Xi,:=(x(i))⊤X_{i,:} = (\mathbf{x}^{(i)})^\top
  • AxA\mathbf{x}“A times x”
    A new vector whose ii-th entry is row ii of AA dotted with x\mathbf{x}. Needs AA's column count to equal x\mathbf{x}'s length.
    (Ax)i=∑j=1nAijxj(A\mathbf{x})_i = \sum_{j=1}^{n} A_{ij} x_j
  • ABAB“A times B”
    (AB)ij=∑kAikBkj(AB)_{ij} = \sum_k A_{ik} B_{kj}. Shapes: (m×n)(n×p)=m×p(m \times n)(n \times p) = m \times p. The inner sizes must match; the outer ones survive.
    (AB)ij=∑k=1nAikBkj(AB)_{ij} = \sum_{k=1}^{n} A_{ik} B_{kj}
  • A⊙BA \odot B“A Hadamard B”
    Multiply entry by entry: (A⊙B)ij=AijBij(A \odot B)_{ij} = A_{ij} B_{ij}. Both must have the same shape. Also called the element-wise product.
    (A⊙B)ij=AijBij(A \odot B)_{ij} = A_{ij} B_{ij}
  • A+BA + B“A plus B”
    Add entry by entry. As with ⊙\odot, both must have exactly the same shape.
    (A+B)ij=Aij+Bij(A + B)_{ij} = A_{ij} + B_{ij}
  • xy⊤\mathbf{x}\mathbf{y}^\top“the outer product of x and y”
    A column times a row makes a whole matrix: (xy⊤)ij=xiyj(\mathbf{x}\mathbf{y}^\top)_{ij} = x_i y_j. For x∈Rm\mathbf{x} \in \mathbb{R}^m and y∈Rn\mathbf{y} \in \mathbb{R}^n it is m×nm \times n.
    (xy⊤)ij=xiyj(\mathbf{x}\mathbf{y}^\top)_{ij} = x_i y_j
  • (AB)⊤=B⊤A⊤(AB)^\top = B^\top A^\top“the transpose of A B is B transpose A transpose”
    Transposing a product reverses the order of the factors. Proved in this chamber, entry by entry.
  • X∈RB×n×dX \in \mathbb{R}^{B \times n \times d}“X is B by n by d”
    A tensor: a grid with three indices. Here, a batch of BB sequences, each of nn tokens, each token a dd-dimensional vector.
    X∈R16×64×256X \in \mathbb{R}^{16 \times 64 \times 256}
  • Xb,:,:X_{b,:,:}“X b colon colon”
    Fix the first index and keep the rest: example bb of the batch, an n×dn \times d matrix.
    X3,:,:∈R64×256X_{3,:,:} \in \mathbb{R}^{64 \times 256}
  • A+bA + \mathbf{b}“A plus b, broadcast”
    Deep-learning shorthand (and NumPy behaviour): add the vector b\mathbf{b} to every row of AA. Strict matrix algebra would call it undefined.
    Cij=Aij+bjC_{ij} = A_{ij} + b_j

Mathematics for Machine Learning · Chamber 7

Change: The Notation of Calculus

  • Δx\Delta x“delta x”
    A finite change in xx: new value minus old value. Capital delta means “difference” (Chamber 1).
    Δx=xnew−xold\Delta x = x_{\text{new}} - x_{\text{old}}
  • ΔyΔx\frac{\Delta y}{\Delta x}“delta y over delta x”
    Rise over run: the slope of the straight line (the secant) through two points of a curve.
    ΔyΔx=f(x0+h)−f(x0)h\frac{\Delta y}{\Delta x} = \frac{f(x_0 + h) - f(x_0)}{h}
  • lim⁡h→0\lim_{h \to 0}“the limit as h tends to zero”
    The value that the expression after it approaches as hh gets closer and closer to 00, without ever being set to 00.
    lim⁡h→0(6+h)=6\lim_{h \to 0} (6 + h) = 6
  • dxdx“d x”
    What Δx\Delta x becomes in the limit: an infinitesimally small change in xx. Thompson's Calculus Made Easy calls it “a little bit of xx”.
    dydx\frac{dy}{dx}
  • dydx\frac{dy}{dx}“d y by d x”
    Leibniz's notation for the derivative of yy with respect to xx: the limit of ΔyΔx\frac{\Delta y}{\Delta x}. It often behaves like a fraction, but it is one symbol.
    y=x2  ⟹  dydx=2xy = x^2 \implies \frac{dy}{dx} = 2x
  • f′(x)f'(x)“f prime of x”
    Lagrange's notation: f′f' is the derivative of ff, a new function, and f′(x)f'(x) is its value at xx.
    f(x)=x3  ⟹  f′(x)=3x2f(x) = x^3 \implies f'(x) = 3x^2
  • ddx\frac{d}{dx}“d by d x”
    An operator: “take the derivative, with respect to xx, of whatever follows”.
    ddx ex=ex\frac{d}{dx}\, e^x = e^x
  • f′′(x)f''(x)“f double prime of x”
    The second derivative, the derivative of the derivative. It measures how the slope itself changes: curvature. Leibniz writes it d2ydx2\frac{d^2 y}{dx^2}.
    f(x)=x4  ⟹  f′′(x)=12x2f(x) = x^4 \implies f''(x) = 12x^2
  • dfdx∣x=a\left.\frac{df}{dx}\right|_{x=a}“d f by d x, evaluated at a”
    The derivative, evaluated at the point x=ax = a. The same number as f′(a)f'(a).
    ddxx2∣x=3=6\left.\frac{d}{dx} x^2\right|_{x=3} = 6
  • dydx=dydu dudx\frac{dy}{dx} = \frac{dy}{du}\,\frac{du}{dx}“the chain rule”
    Differentiating a function of a function: multiply the rates. If yy depends on uu and uu on xx, the rates chain together. Backpropagation is this rule, applied over and over.
  • ∂f∂xi\frac{\partial f}{\partial x_i}“partial f by partial x i”
    The partial derivative: how ff changes when only xix_i moves and every other input is held fixed. The curly ∂\partial warns you there are other inputs.
    f=x12x2  ⟹  ∂f∂x1=2x1x2f = x_1^2 x_2 \implies \frac{\partial f}{\partial x_1} = 2x_1 x_2
  • ∇f\nabla f“grad f”
    The gradient: every partial derivative of ff stacked into a vector, the same shape as the input. ∇\nabla is called nabla.
    ∇f=(∂f∂x1,…,∂f∂xn)⊤\nabla f = \left(\frac{\partial f}{\partial x_1}, \ldots, \frac{\partial f}{\partial x_n}\right)^\top
  • ∇θL\nabla_\theta \mathcal{L}“grad theta of L”
    The gradient of the loss with respect to the parameters θ\theta only. The subscript names the variables being nudged; everything else is held fixed.
    θ←θ−η ∇θL(θ)\theta \leftarrow \theta - \eta\, \nabla_\theta \mathcal{L}(\theta)
  • JfJ_{\mathbf{f}}“the Jacobian of f”
    For f:Rn→Rm\mathbf{f} : \mathbb{R}^n \to \mathbb{R}^m, the m×nm \times n matrix of all first partials, Jij=∂fi∂xjJ_{ij} = \frac{\partial f_i}{\partial x_j}. Also written ∂f∂x\frac{\partial \mathbf{f}}{\partial \mathbf{x}}.
    Jij=∂fi∂xjJ_{ij} = \frac{\partial f_i}{\partial x_j}
  • HfH_f“the Hessian of f”
    For f:Rn→Rf : \mathbb{R}^n \to \mathbb{R}, the n×nn \times n matrix of second partials, Hij=∂2f∂xi ∂xjH_{ij} = \frac{\partial^2 f}{\partial x_i\, \partial x_j}. Also written ∇2f\nabla^2 f.
    Hij=∂2f∂xi ∂xjH_{ij} = \frac{\partial^2 f}{\partial x_i\, \partial x_j}
  • ∫abf(x) dx\int_a^b f(x)\,dx“the integral from a to b of f of x, d x”
    A continuous sum: add up height f(x)f(x) times a little bit of width dxdx, for every xx from aa to bb. The area under the curve.
    ∫01x dx=12\int_0^1 x\,dx = \tfrac{1}{2}
  • ∑if(xi) Δx\sum_{i} f(x_i)\,\Delta x“the Riemann sum”
    The finite version: cut [a,b][a, b] into strips of width Δx\Delta x and add up f(xi) Δxf(x_i)\,\Delta x over the strips. As Δx→0\Delta x \to 0 it becomes the integral, and Σ\Sigma turns into ∫\int.
  • min⁡xf(x)\min_{x} f(x)“the minimum over x of f”
    The smallest value that ff takes: a number on the output side.
    min⁡x(x+2)2+5=5\min_{x} (x + 2)^2 + 5 = 5
  • arg min⁡xf(x)\argmin_{x} f(x)“the arg min over x of f”
    The input at which ff is smallest. Read it “the xx that minimises ff”.
    arg min⁡x(x+2)2+5=−2\argmin_{x} (x + 2)^2 + 5 = -2
  • arg max⁡kpk\argmax_{k} p_k“the arg max over k of p k”
    The index of the largest entry. A classifier's predicted class is arg max⁡k\argmax_k of its predicted probabilities.
    arg max⁡k(0.1,0.7,0.2)=2\argmax_{k} (0.1, 0.7, 0.2) = 2
  • θ⋆=arg min⁡θL(θ)\theta^\star = \argmin_{\theta} \mathcal{L}(\theta)“theta star is the arg min of L”
    The best parameters: the θ\theta with the lowest loss. The star marks the optimum, as in Chamber 1.
  • sup⁡\sup“the supremum”
    The least upper bound. Like max⁡\max, but it exists even when no element reaches it.
    sup⁡{1−1n:n=1,2,3,…}=1\sup\left\{1 - \tfrac{1}{n} : n = 1, 2, 3, \ldots\right\} = 1
  • f(x+h)≈f(x)+f′(x) hf(x + h) \approx f(x) + f'(x)\,h“the first-order approximation”
    Close to xx, a smooth function is almost a straight line through (x,f(x))(x, f(x)) with slope f′(x)f'(x).
    e0.1≈e0+e0⋅0.1=1.1e^{0.1} \approx e^0 + e^0 \cdot 0.1 = 1.1
  • O(h)O(h)“big O of h”
    Some quantity no bigger than a constant times hh, once hh is small enough. It hides the constant to show the rate: halve hh, roughly halve the error.
    f(x+h)−f(x)h=f′(x)+O(h)\frac{f(x+h) - f(x)}{h} = f'(x) + O(h)
  • O(n2)O(n^2)“big O of n squared”
    A cost that grows at most like n2n^2 for large nn: double nn and the work roughly quadruples. Self-attention's cost in the sequence length nn is quoted this way.
    O(n2⋅d)O(n^2 \cdot d)

Mathematics for Machine Learning · Chamber 8

Chance: The Notation of Probability

  • Ω\Omega“capital omega, the sample space”
    The sample space: the set of every possible outcome of the experiment.
    Ω={1,2,3,4,5,6}\Omega = \{1, 2, 3, 4, 5, 6\}
  • A⊆ΩA \subseteq \Omega“A, an event”
    An event is a set of outcomes, so a subset of Ω\Omega. “The roll is even” is the event {2,4,6}\{2, 4, 6\}.
    A={2,4,6}A = \{2, 4, 6\}
  • P(A)P(A)“P of A”
    The probability of event AA: a number from 0 (impossible) to 1 (certain).
    P(even)=12P(\text{even}) = \tfrac{1}{2}
  • P(Ac)P(A^c)“P of A complement”
    The probability that AA does not happen. It always equals 1−P(A)1 - P(A).
    P(Ac)=1−P(A)P(A^c) = 1 - P(A)
  • P(A∩B)P(A \cap B)“P of A and B”
    The probability that both happen. Often written with a comma instead: P(A,B)P(A, B).
    P(A∩B)=P(A,B)P(A \cap B) = P(A, B)
  • P(A∪B)P(A \cup B)“P of A or B”
    The probability that at least one happens. Inclusion–exclusion from Chamber 2 gives the formula.
    P(A∪B)=P(A)+P(B)−P(A∩B)P(A \cup B) = P(A) + P(B) - P(A \cap B)
  • P(A∣B)P(A \mid B)“P of A given B”
    Conditional probability: how likely AA is once you know BB happened. The bar is read “given”.
    P(A∣B)=P(A∩B)P(B)P(A \mid B) = \frac{P(A \cap B)}{P(B)}
  • XX“random variable X”
    A capital letter is a random variable: a number whose value depends on chance, like the result of a roll.
    X=the number rolledX = \text{the number rolled}
  • P(X=x)P(X = x)“P of big X equals little x”
    The probability that the random variable XX takes the particular value xx. Capital: the random quantity. Lower case: one value it could take.
    P(X=6)=16P(X = 6) = \tfrac{1}{6}
  • p(x)p(x)“p of x”
    The distribution of XX at the value xx. For discrete XX it's a pmf, p(x)=P(X=x)p(x) = P(X = x). For continuous XX it's a density: areas under it are probabilities.
    p(6)=16p(6) = \tfrac{1}{6}
  • p(x,y)p(x, y)“p of x comma y”
    The joint distribution: how likely X=xX = x and Y=yY = y are together.
    p(rain,umbrella)=0.3p(\text{rain}, \text{umbrella}) = 0.3
  • p(y∣x)p(y \mid x)“p of y given x”
    The conditional distribution of YY once X=xX = x is known: p(x,y)/p(x)p(x, y) / p(x).
    p(y∣x)=p(x,y)p(x)p(y \mid x) = \frac{p(x, y)}{p(x)}
  • pθ(y∣x)p_\theta(y \mid x)“p theta of y given x”
    A model: the probability that a network with parameters θ\theta assigns to label yy for input xx.
    pθ(cat∣image)=0.92p_\theta(\text{cat} \mid \text{image}) = 0.92
  • ∑yp(x,y)\sum_{y} p(x, y)“sum over y of p of x comma y”
    Marginalising: add up the joint over every value of yy and you're left with p(x)p(x). People say yy has been “summed out”.
    p(x)=∑yp(x,y)p(x) = \sum_{y} p(x, y)
  • ∫p(x,y) dy\int p(x, y)\, dy“integral of p of x comma y, d y”
    The same marginalisation when yy is continuous: the integral is a continuous sum (Chamber 7).
    p(x)=∫p(x,y) dyp(x) = \int p(x, y)\, dy
  • X⊥ ⁣ ⁣ ⁣⊥YX \perp\!\!\!\perp Y“X is independent of Y”
    Independence: knowing one tells you nothing about the other, so the joint factorises: p(x,y)=p(x) p(y)p(x, y) = p(x)\,p(y).
    p(x,y)=p(x) p(y)p(x, y) = p(x)\,p(y)
  • ∼\sim“is drawn from”
    Between a variable and a distribution, ∼\sim says the variable is a sample from it (“is distributed as”). Not to be confused with ≈\approx.
    x∼px \sim p
  • ∼i.i.d.\overset{\text{i.i.d.}}{\sim}“independent and identically distributed as”
    Every sample comes from the same distribution, and none depends on any other. The standing assumption about datasets.
    x1,…,xn∼i.i.d.px_1, \ldots, x_n \overset{\text{i.i.d.}}{\sim} p
  • Bernoulli(p)\mathrm{Bernoulli}(p)“Bernoulli p”
    A biased coin: the value 1 with probability pp and 0 otherwise. Dropout's masks are Bernoulli.
    r∼Bernoulli(0.8)r \sim \mathrm{Bernoulli}(0.8)
  • N(μ,σ2)\mathcal{N}(\mu, \sigma^2)“normal with mean mu and variance sigma squared”
    The bell curve centred at μ\mu. The second slot is the variance σ2\sigma^2, not the standard deviation.
    ε∼N(0,1)\varepsilon \sim \mathcal{N}(0, 1)
  • U(a,b)\mathcal{U}(a, b)“uniform from a to b”
    Every value between aa and bb equally likely. A common way to initialise weights.
    u∼U(0,1)u \sim \mathcal{U}(0, 1)
  • E[X]\mathbb{E}[X]“the expectation of X”
    The expected value: the average of XX, weighting each value by its probability. The long-run mean of many samples.
    E[X]=∑xx p(x)\mathbb{E}[X] = \sum_x x\, p(x)
  • Ex∼p[f(x)]\mathbb{E}_{x \sim p}[f(x)]“expectation of f of x, x drawn from p”
    The average of f(x)f(x) when xx is drawn from pp. The subscript names the random variable and its distribution.
    Ex∼p[f(x)]=∑xf(x) p(x)\mathbb{E}_{x \sim p}[f(x)] = \sum_x f(x)\, p(x)
  • Var(X)\mathrm{Var}(X)“the variance of X”
    How spread out XX is: the expected squared distance from its mean. Its square root is the standard deviation σ\sigma.
    Var(X)=E[(X−E[X])2]\mathrm{Var}(X) = \mathbb{E}[(X - \mathbb{E}[X])^2]
  • Cov(X,Y)\mathrm{Cov}(X, Y)“the covariance of X and Y”
    Whether two random variables move together (positive), oppositely (negative), or neither (zero).
    Cov(X,Y)=E[(X−E[X])(Y−E[Y])]\mathrm{Cov}(X, Y) = \mathbb{E}[(X - \mathbb{E}[X])(Y - \mathbb{E}[Y])]
  • p(D∣θ)p(\mathcal{D} \mid \theta)“the likelihood, p of D given theta”
    The likelihood: how probable the whole dataset is under parameters θ\theta. For i.i.d. data it's a product.
    p(D∣θ)=∏i=1npθ(yi∣xi)p(\mathcal{D} \mid \theta) = \prod_{i=1}^{n} p_\theta(y_i \mid x_i)
  • log⁡p(D∣θ)\log p(\mathcal{D} \mid \theta)“the log-likelihood”
    The log turns the product into a sum (Chamber 5). Maximising it is maximum likelihood; minimising its negative is the usual cross-entropy loss.
    ∑i=1nlog⁡pθ(yi∣xi)\sum_{i=1}^{n} \log p_\theta(y_i \mid x_i)
  • DKL(p ∥ q)D_{\mathrm{KL}}(p \,\|\, q)“the K L divergence of p from q”
    The Kullback–Leibler divergence: how different qq is from pp. Never negative, zero only when p=qp = q, and not symmetric. The ∥\| is a separator, not a norm.
    DKL(p ∥ q)=Ex∼p[log⁡p(x)q(x)]D_{\mathrm{KL}}(p \,\|\, q) = \mathbb{E}_{x \sim p}\left[\log \frac{p(x)}{q(x)}\right]

Mathematics for Machine Learning · Chamber 9

Reading the Equations of a Paper

  • w.r.t.\text{w.r.t.}“with respect to”
    Names the variable a derivative or an optimisation is taken over. The Adam paper computes “gradients w.r.t. stochastic objective”.
    ∇θf is the gradient w.r.t. θ\nabla_\theta f \text{ is the gradient w.r.t. } \theta
  • min⁡xf(x) s.t. x≥0\min_{x} f(x) \ \text{s.t.}\ x \ge 0“minimise f of x subject to x at least zero”
    In an optimisation problem, “s.t.” means subject to: it introduces the constraints the answer must satisfy. (In a definition, the same letters mean “such that”, as in Chamber 3.)
  • f(⋅)f(\cdot)“f of dot”
    The dot is a placeholder for “whatever argument goes here”, so f(⋅)f(\cdot) means the function itself, not a value of it. The same dot turns up in norms, ∥⋅∥2\|\cdot\|_2.
    ∥⋅∥2\|\cdot\|_2
  • x1:Tx_{1:T}“x one to T”
    The whole sequence x1,x2,…,xTx_1, x_2, \ldots, x_T. Colon ranges are everywhere in papers on sequences.
    p(x1:T)p(x_{1:T})
  • x<tx_{<t}“x before t”
    Everything before position tt: x1,…,xt−1x_1, \ldots, x_{t-1}. The notation of language models, which predict each token from the ones before it.
    p(x1:T)=∏t=1Tp(xt∣x<t)p(x_{1:T}) = \prod_{t=1}^{T} p(x_t \mid x_{<t})
  • const\text{const}“plus a constant”
    Terms that don't depend on the variable of interest, lumped together and ignored because they don't change the argmin.
    log⁡p(x)=−12x2+const\log p(x) = -\tfrac{1}{2}x^2 + \text{const}
  • [z]+[z]_+“the positive part of z”
    Shorthand for max⁡(0,z)\max(0, z), the ReLU of Chamber 4 in different clothes. Common in hinge losses.
    [1−y y^]+[1 - y\,\hat{y}]_+
  • ≃\simeq“sim-equals, is estimated by”
    A cousin of ≈\approx, called “sim-equals” after its LaTeX name, \simeq. The VAE paper uses it for “this sample average stands in for that expectation”.
    E[f(z)]≃1L∑l=1Lf(z(l))\mathbb{E}[f(z)] \simeq \frac{1}{L}\sum_{l=1}^{L} f(z^{(l)})
  • QQ“Q, the queries”
    In attention, a matrix with one row per position that is asking: shape n×dkn \times d_k.
    Q∈Rn×dkQ \in \mathbb{R}^{n \times d_k}
  • KK“K, the keys”
    One row per position that can be looked up, compared with every query: shape m×dkm \times d_k.
    K∈Rm×dkK \in \mathbb{R}^{m \times d_k}
  • VV“V, the values”
    The content that gets mixed, one row per key: shape m×dvm \times d_v.
    V∈Rm×dvV \in \mathbb{R}^{m \times d_v}
  • dkd_k“d k, the key dimension”
    The length of each query and key vector. The Transformer uses dk=64d_k = 64 per head.
    dk=8\sqrt{d_k} = 8
  • gt2g_t^2“g t squared, element by element”
    In Adam, the square of every entry of the gradient vector (gt⊙gtg_t \odot g_t), not a dot product.
    vt←β2vt−1+(1−β2) gt2v_t \leftarrow \beta_2 v_{t-1} + (1 - \beta_2)\, g_t^2
  • qϕ(z∣x)q_{\boldsymbol{\phi}}(\mathbf{z} \mid \mathbf{x})“q phi of z given x, the encoder”
    In a VAE, the encoder: a distribution over codes z\mathbf{z} for the input x\mathbf{x}, with parameters ϕ\boldsymbol{\phi}.
  • pθ(x∣z)p_{\boldsymbol{\theta}}(\mathbf{x} \mid \mathbf{z})“p theta of x given z, the decoder”
    In a VAE, the decoder: how likely the input x\mathbf{x} is to be rebuilt from the code z\mathbf{z}.
  • L(θ,ϕ;x(i))\mathcal{L}(\boldsymbol{\theta}, \boldsymbol{\phi}; \mathbf{x}^{(i)})“the ELBO, L of theta and phi at x i”
    The evidence lower bound for data point ii: the number a VAE maximises. Parameters before the semicolon, data after (Chamber 4's f(x;θ)f(x; \theta) the other way round).

Linear Algebra for Machine Learning · Chamber 1

Vectors: Arrows, Lists and Data

  • Rn\mathbb{R}^n“R n”
    Every list of nn real numbers: the space the vectors live in. R2\mathbb{R}^2 is the plane, R3\mathbb{R}^3 ordinary space.
    x∈R784\mathbf{x} \in \mathbb{R}^{784}
  • [v1v2]\begin{bmatrix} v_1 \\ v_2 \end{bmatrix}“the column vector v one, v two”
    A vector written as a column. Its entries are the arrow's coordinates. In running text it's written (v1,v2)(v_1, v_2) to save space.
    v=[31]=(3,1)\mathbf{v} = \begin{bmatrix} 3 \\ 1 \end{bmatrix} = (3, 1)
  • PQ→\overrightarrow{PQ}“vector P Q”
    The arrow from point PP to point QQ: “destination minus start”.
    PQ→=Q−P\overrightarrow{PQ} = Q - P
  • u+v\mathbf{u} + \mathbf{v}“u plus v”
    Add entry by entry. As arrows: put the tail of v\mathbf{v} at the tip of u\mathbf{u} (tip to tail).
    (3,1)+(1,2)=(4,3)(3, 1) + (1, 2) = (4, 3)
  • c vc\,\mathbf{v}“c times v”
    Scalar multiplication: multiply every entry by the number cc. Stretches or shrinks the arrow by ∣c∣|c|, and flips it if c<0c < 0.
    3 (1,−2)=(3,−6)3\,(1, -2) = (3, -6)
  • −v-\mathbf{v}“minus v”
    The negative of v\mathbf{v}: the same length, pointing the opposite way. It equals (−1)v(-1)\mathbf{v}.
    −(2,−5)=(−2,5)-(2, -5) = (-2, 5)
  • u−v\mathbf{u} - \mathbf{v}“u minus v”
    u+(−v)\mathbf{u} + (-\mathbf{v}). Drawn from the origin, it's the arrow from the tip of v\mathbf{v} to the tip of u\mathbf{u}.
    (4,3)−(1,2)=(3,1)(4, 3) - (1, 2) = (3, 1)
  • 0\mathbf{0}“the zero vector”
    Every entry zero. Adding it changes nothing, and it has no direction.
    v+0=v\mathbf{v} + \mathbf{0} = \mathbf{v}
  • ei\mathbf{e}_i“e i”
    The ii-th standard basis vector: 1 in position ii, 0 everywhere else. In machine learning it's called a one-hot vector.
    e2=(0,1,0)∈R3\mathbf{e}_2 = (0, 1, 0) \in \mathbb{R}^3
  • ∑i=1kcivi\sum_{i=1}^{k} c_i \mathbf{v}_i“the sum of c i times v i”
    A linear combination: scale each vector vi\mathbf{v}_i by its coefficient cic_i, then add the results.
    c1v1+c2v2c_1\mathbf{v}_1 + c_2\mathbf{v}_2
  • xking\mathbf{x}_{\text{king}}“x king”
    A word embedding: the learned vector for one word. Here the subscript names the word, not a position.
    xking∈R300\mathbf{x}_{\text{king}} \in \mathbb{R}^{300}

Linear Algebra for Machine Learning · Chamber 2

Dot Products, Lengths and Angles

  • u⋅v\mathbf{u} \cdot \mathbf{v}“u dot v”
    The dot product: multiply matching entries and add. The result is a single number.
    (1,2)⋅(3,4)=3+8=11(1, 2) \cdot (3, 4) = 3 + 8 = 11
  • u⊤v\mathbf{u}^\top \mathbf{v}“u transpose v”
    The same dot product, written as a row times a column. The usual form in machine learning papers.
    r^ui=qi⊤pu\hat{r}_{ui} = \mathbf{q}_i^\top \mathbf{p}_u
  • ⟨u,v⟩\langle \mathbf{u}, \mathbf{v} \rangle“the inner product of u and v”
    Angle-bracket notation for the dot product, common in maths and theory papers.
    ⟨u,v⟩=u⊤v\langle \mathbf{u}, \mathbf{v} \rangle = \mathbf{u}^\top \mathbf{v}
  • ∥v∥\lVert \mathbf{v} \rVert“the norm of v”
    The length of v\mathbf{v}. With no subscript it nearly always means the Euclidean norm ∥v∥2\lVert \mathbf{v} \rVert_2.
    ∥(3,4)∥=5\lVert (3, 4) \rVert = 5
  • ∥v∥2\lVert \mathbf{v} \rVert_2“the two-norm of v”
    Euclidean length: v12+⋯+vn2=v⋅v\sqrt{v_1^2 + \cdots + v_n^2} = \sqrt{\mathbf{v} \cdot \mathbf{v}}. Also called the ℓ2\ell_2 norm.
    ∥v∥2=v⊤v\lVert \mathbf{v} \rVert_2 = \sqrt{\mathbf{v}^\top \mathbf{v}}
  • ∥v∥1\lVert \mathbf{v} \rVert_1“the one-norm of v”
    The sum of the absolute values of the entries: distance along a grid of streets (the taxicab or Manhattan norm).
    ∥(3,−4)∥1=7\lVert (3, -4) \rVert_1 = 7
  • ∥v∥∞\lVert \mathbf{v} \rVert_\infty“the infinity-norm of v”
    The largest absolute value of any entry (the max norm).
    ∥(3,−4)∥∞=4\lVert (3, -4) \rVert_\infty = 4
  • v^\hat{\mathbf{v}}“v hat”
    The unit vector in the direction of v\mathbf{v}. On a vector, a hat often means “normalised” rather than “estimated”.
    v^=v/∥v∥\hat{\mathbf{v}} = \mathbf{v} / \lVert \mathbf{v} \rVert
  • ∥u−v∥\lVert \mathbf{u} - \mathbf{v} \rVert“the distance between u and v”
    The length of the arrow from the tip of v\mathbf{v} to the tip of u\mathbf{u}.
    ∥(4,6)−(1,2)∥=5\lVert (4, 6) - (1, 2) \rVert = 5
  • cos⁡θ\cos\theta“cos theta”
    The cosine of the angle between two vectors, known in machine learning as their cosine similarity. Always between −1-1 and 11.
    cos⁡θ=u⋅v∥u∥∥v∥\cos\theta = \frac{\mathbf{u} \cdot \mathbf{v}}{\lVert \mathbf{u} \rVert \lVert \mathbf{v} \rVert}
  • u⊥v\mathbf{u} \perp \mathbf{v}“u is perpendicular to v”
    Orthogonal: at right angles, which means u⋅v=0\mathbf{u} \cdot \mathbf{v} = 0.
    (2,1)⊥(−1,2)(2, 1) \perp (-1, 2)
  • proj⁡vu\operatorname{proj}_{\mathbf{v}} \mathbf{u}“the projection of u onto v”
    The shadow of u\mathbf{u} on the line through v\mathbf{v}: the point of that line closest to the tip of u\mathbf{u}.
    proj⁡vu=u⋅vv⋅v v\operatorname{proj}_{\mathbf{v}} \mathbf{u} = \frac{\mathbf{u} \cdot \mathbf{v}}{\mathbf{v} \cdot \mathbf{v}}\,\mathbf{v}

Linear Algebra for Machine Learning · Chamber 3

Span, Basis and Dimension

  • span{v1,…,vk}\mathrm{span}\{\mathbf{v}_1, \ldots, \mathbf{v}_k\}“the span of v one to v k”
    Every linear combination c1v1+⋯+ckvkc_1\mathbf{v}_1 + \cdots + c_k\mathbf{v}_k with real coefficients: all the points these vectors can reach.
    (3,6)∈span{(1,2)}(3, 6) \in \mathrm{span}\{(1, 2)\}
  • {0}\{\mathbf{0}\}“the zero subspace”
    The set whose only element is the zero vector. The smallest subspace there is, and the span of 0\mathbf{0} alone.
    span{0}={0}\mathrm{span}\{\mathbf{0}\} = \{\mathbf{0}\}
  • c1v1+⋯+ckvk=0c_1\mathbf{v}_1 + \cdots + c_k\mathbf{v}_k = \mathbf{0}“a dependency relation”
    A combination that lands on 0\mathbf{0}. If one exists with some ci≠0c_i \neq 0, the vectors are dependent; if only all-zero coefficients work, they are independent.
    2(1,2)−(2,4)=02(1, 2) - (2, 4) = \mathbf{0}
  • det⁡[u v]\det[\mathbf{u}\ \mathbf{v}]“the determinant of u and v”
    For two vectors in the plane, u1v2−u2v1u_1v_2 - u_2v_1: the signed area of the parallelogram they span. Zero exactly when they are dependent. Chamber 5 builds determinants in general.
    det⁡[2112]=3\det\begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix} = 3
  • B=(b1,…,bn)\mathcal{B} = (\mathbf{b}_1, \ldots, \mathbf{b}_n)“the basis B”
    An ordered list of vectors that is linearly independent and spans the space. Bases usually get calligraphic letters.
    B=((1,1),(1,−1))\mathcal{B} = ((1, 1), (1, -1))
  • [x]B[\mathbf{x}]_{\mathcal{B}}“the coordinates of x in basis B”
    The unique coefficients (c1,…,cn)(c_1, \ldots, c_n) with x=c1b1+⋯+cnbn\mathbf{x} = c_1\mathbf{b}_1 + \cdots + c_n\mathbf{b}_n. Same arrow, different ruler.
    [(5,1)]B=(3,2)[(5, 1)]_{\mathcal{B}} = (3, 2)
  • dim⁡U\dim U“the dimension of U”
    The number of vectors in any basis of the subspace UU. All bases of UU have the same size.
    dim⁡Rn=n\dim \mathbb{R}^n = n
  • dintd_{\text{int}}“d int, the intrinsic dimension”
    From Li et al. (2018): the number of directions that actually matter for solving a problem, D−sD - s, where ss is the dimension of the solution set.
    D=dint+sD = d_{\text{int}} + s

Linear Algebra for Machine Learning · Chamber 4

Matrices as Transformations

  • T:Rn→RmT : \mathbb{R}^n \to \mathbb{R}^m“T maps R n to R m”
    A function that takes vectors with nn entries to vectors with mm entries. It is linear if it respects sums and scalings.
    T(u+v)=T(u)+T(v)T(\mathbf{u} + \mathbf{v}) = T(\mathbf{u}) + T(\mathbf{v})
  • AxA\mathbf{x}“A times x”
    Apply the matrix to the vector: the combination x1a1+⋯+xnanx_1\mathbf{a}_1 + \cdots + x_n\mathbf{a}_n of AA's columns, weighted by the entries of x\mathbf{x}.
    [2−111](32)=(45)\begin{bmatrix} 2 & -1 \\ 1 & 1 \end{bmatrix}\begin{pmatrix} 3 \\ 2 \end{pmatrix} = \begin{pmatrix} 4 \\ 5 \end{pmatrix}
  • aj=Aej\mathbf{a}_j = A\mathbf{e}_j“a j, the j-th column of A”
    Where the jj-th standard basis vector lands. The columns of a matrix are the images of the basis vectors.
    Ae1=a1A\mathbf{e}_1 = \mathbf{a}_1
  • ABAB“A B, or A after B”
    The composition: apply BB first, then AA. Products are read right to left, like f∘gf \circ g.
    (AB)x=A(Bx)(AB)\mathbf{x} = A(B\mathbf{x})
  • RθR_\theta“R theta, rotation by theta”
    The matrix that rotates the plane anticlockwise by the angle θ\theta.
    Rθ=[cos⁡θ−sin⁡θsin⁡θcos⁡θ]R_\theta = \begin{bmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{bmatrix}
  • det⁡A\det A“det A, the determinant of A”
    The signed factor by which AA scales areas. Negative means orientation flips (like a mirror); zero means the plane is flattened.
    det⁡[abcd]=ad−bc\det\begin{bmatrix} a & b \\ c & d \end{bmatrix} = ad - bc
  • ab⊤\mathbf{a}\mathbf{b}^\top“the outer product of a and b”
    A column times a row: the matrix whose (i,j)(i, j) entry is aibja_ib_j. Every product ABAB is a sum of these, one per column of AA.
    AB=∑kA:,k Bk,:AB = \sum_k A_{:,k}\, B_{k,:}
  • x↦Wx+b\mathbf{x} \mapsto W\mathbf{x} + \mathbf{b}“x maps to W x plus b”
    An affine map: a linear map followed by a shift. It is not linear unless b=0\mathbf{b} = \mathbf{0}, because it moves the origin to b\mathbf{b}.
  • σ(Wx+b)\sigma(W\mathbf{x} + \mathbf{b})“sigma of W x plus b”
    A neural-network layer: an affine map, then a nonlinearity σ\sigma (ReLU, tanh, …) applied to each entry separately.
    h=ReLU(Wx+b)\mathbf{h} = \mathrm{ReLU}(W\mathbf{x} + \mathbf{b})
  • [x1]\begin{bmatrix} \mathbf{x} \\ 1 \end{bmatrix}“x in homogeneous coordinates”
    The vector with a 1 appended. A matrix acting on it can shift as well as transform, because its last column gets added.
    (3,5)↦(3,5,1)(3, 5) \mapsto (3, 5, 1)

Linear Algebra for Machine Learning · Chamber 5

Solving Systems: Elimination and Inverses

  • [A∣b][A \mid \mathbf{b}]“A augmented with b”
    The augmented matrix: the coefficients of AA with the right-hand side b\mathbf{b} attached as an extra column. The bar marks where the equals signs were.
    [2−11115]\left[\begin{array}{cc|c} 2 & -1 & 1 \\ 1 & 1 & 5 \end{array}\right]
  • RiR_i“row i”
    The ii-th row of the matrix you're working on. Row operations are written in terms of these.
    R2R_2
  • Ri↔RjR_i \leftrightarrow R_j“swap rows i and j”
    Exchange two rows. You need it when a pivot position holds a zero.
    R1↔R3R_1 \leftrightarrow R_3
  • Ri←c RiR_i \leftarrow c\,R_i“row i gets c times row i”
    Scale a row by a non-zero number cc, usually to make a pivot equal to 1.
    R3←112R3R_3 \leftarrow \tfrac{1}{12} R_3
  • Ri←Ri+c RjR_i \leftarrow R_i + c\,R_j“row i gets row i plus c times row j”
    Add a multiple of another row. The workhorse of elimination: it creates the zeros below (and above) each pivot, and it never changes the determinant.
    R2←R2−2R1R_2 \leftarrow R_2 - 2R_1
  • rref⁡(A)\operatorname{rref}(A)“r r e f of A”
    The reduced row echelon form of AA: every pivot is 1, with zeros above and below it. Every matrix has exactly one.
    rref⁡(I)=I\operatorname{rref}(I) = I
  • A−1A^{-1}“A inverse”
    The matrix that undoes AA: A−1A=AA−1=IA^{-1}A = AA^{-1} = I. Only square matrices can have one, and not all do. It is not the matrix of reciprocals 1/aij1/a_{ij}.
    x=A−1b\mathbf{x} = A^{-1}\mathbf{b}
  • InI_n“the n by n identity”
    Ones on the diagonal, zeros elsewhere: the matrix that changes nothing, Inx=xI_n \mathbf{x} = \mathbf{x}. Written just II when the size is clear.
    I2=[1001]I_2 = \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix}
  • det⁡A\det A“det A”
    The determinant of a square matrix: the signed factor by which AA scales areas (in 2D) or volumes (in 3D and beyond). It is zero exactly when AA is singular.
    det⁡[abcd]=ad−bc\det\begin{bmatrix} a & b \\ c & d \end{bmatrix} = ad - bc
  • ∣A∣|A|“the determinant of A, written with bars”
    Another way to write det⁡A\det A. Careful: in ∣det⁡J∣\lvert \det J \rvert the bars mean absolute value, applied to the number det⁡J\det J.
    ∣abcd∣=ad−bc\begin{vmatrix} a & b \\ c & d \end{vmatrix} = ad - bc
  • ∂f(x)∂xT\frac{\partial f(x)}{\partial x^T}“the Jacobian of f at x”
    The matrix of partial derivatives ∂fi/∂xj\partial f_i / \partial x_j, in the notation of the Real NVP paper. Near xx, ff behaves like this matrix, so its determinant is ff's local volume scale.
    ∂y∂xT\frac{\partial y}{\partial x^T}

Linear Algebra for Machine Learning · Chamber 6

Rank and the Four Subspaces

  • C(A)C(A)“the column space of A”
    Every output AxA\mathbf{x}, that is, every combination of the columns of AA. A subspace of Rm\mathbb{R}^m. Also written col⁡(A)\operatorname{col}(A), range⁡(A)\operatorname{range}(A) or im⁡(A)\operatorname{im}(A).
    b∈C(A)\mathbf{b} \in C(A)
  • N(A)N(A)“the null space of A”
    Every input x\mathbf{x} with Ax=0A\mathbf{x} = \mathbf{0}: what AA destroys. A subspace of Rn\mathbb{R}^n. Also called the kernel, ker⁡A\ker A.
    N(A)={x∈Rn:Ax=0}N(A) = \{\mathbf{x} \in \mathbb{R}^n : A\mathbf{x} = \mathbf{0}\}
  • C(A⊤)C(A^\top)“the row space of A”
    Every combination of the rows of AA (the column space of A⊤A^\top). A subspace of Rn\mathbb{R}^n.
    C(A⊤)⊆RnC(A^\top) \subseteq \mathbb{R}^n
  • N(A⊤)N(A^\top)“the left null space of A”
    Every y\mathbf{y} with A⊤y=0A^\top \mathbf{y} = \mathbf{0}, equivalently y⊤A=0⊤\mathbf{y}^\top A = \mathbf{0}^\top. A subspace of Rm\mathbb{R}^m.
    y⊤A=0⊤\mathbf{y}^\top A = \mathbf{0}^\top
  • rank⁡(A)\operatorname{rank}(A)“the rank of A”
    The number of pivots, which is also the number of independent columns and the number of independent rows. Some books write rk⁡(A)\operatorname{rk}(A).
    rank⁡(A)≤min⁡(m,n)\operatorname{rank}(A) \le \min(m, n)
  • dim⁡V\dim V“the dimension of V”
    The number of vectors in any basis of the subspace VV (Chamber 3). The dimension of the null space is sometimes called the nullity.
    dim⁡N(A)=n−r\dim N(A) = n - r
  • uv⊤\mathbf{u}\mathbf{v}^\top“u v transpose”
    The outer product of u∈Rm\mathbf{u} \in \mathbb{R}^m and v∈Rn\mathbf{v} \in \mathbb{R}^n: the m×nm \times n matrix with entries uivju_i v_j. It has rank 1 when u\mathbf{u} and v\mathbf{v} are non-zero.
    (uv⊤)ij=uivj(\mathbf{u}\mathbf{v}^\top)_{ij} = u_i v_j
  • r≪min⁡(d,k)r \ll \min(d, k)“r is much less than the smaller of d and k”
    LoRA's condition on the rank of its update: far smaller than either side of the d×kd \times k weight matrix.
    r=4≪12288r = 4 \ll 12288

Linear Algebra for Machine Learning · Chamber 7

Orthogonality and Least Squares

  • u⊥v\mathbf{u} \perp \mathbf{v}“u is perpendicular to v”
    The vectors are orthogonal: their dot product is zero, u⊤v=0\mathbf{u}^\top\mathbf{v} = 0. Also read “u is orthogonal to v”.
    (1,2)⊥(2,−1)(1, 2) \perp (2, -1)
  • V⊥V^\perp“V perp”
    The orthogonal complement of a subspace VV: every vector that is perpendicular to all of VV. It is a subspace too.
    N(A)=C(A⊤)⊥N(A) = C(A^\top)^\perp
  • qi⊤qj=δij\mathbf{q}_i^\top\mathbf{q}_j = \delta_{ij}“q i transpose q j equals delta i j”
    The vectors q1,…,qn\mathbf{q}_1, \ldots, \mathbf{q}_n are orthonormal: each has length 1 (when i=ji = j, δij=1\delta_{ij} = 1) and every pair is perpendicular (when i≠ji \ne j, δij=0\delta_{ij} = 0). δij\delta_{ij} is the Kronecker delta.
  • Q⊤Q=IQ^\top Q = I“Q transpose Q equals I”
    The columns of QQ are orthonormal. When QQ is square it is called an orthogonal matrix, and Q−1=Q⊤Q^{-1} = Q^\top.
    Q=15[3−443]Q = \tfrac{1}{5}\begin{bmatrix} 3 & -4 \\ 4 & 3 \end{bmatrix}
  • proj⁡a(b)\operatorname{proj}_{\mathbf{a}}(\mathbf{b})“the projection of b onto a”
    The point on the line through a\mathbf{a} that is closest to b\mathbf{b}: the shadow of b\mathbf{b} on that line.
    proj⁡a(b)=a⊤ba⊤a a\operatorname{proj}_{\mathbf{a}}(\mathbf{b}) = \frac{\mathbf{a}^\top\mathbf{b}}{\mathbf{a}^\top\mathbf{a}}\,\mathbf{a}
  • P=A(A⊤A)−1A⊤P = A(A^\top A)^{-1}A^\top“P equals A, times A transpose A inverse, times A transpose”
    The projection matrix onto the column space of AA (whose columns are independent): PbP\mathbf{b} is the closest point to b\mathbf{b} in C(A)C(A).
  • P2=PP^2 = P“P squared equals P”
    Projecting twice changes nothing. Matrices with this property are called idempotent.
  • x^\hat{\mathbf{x}}“x hat”
    The least-squares solution of Ax≈bA\mathbf{x} \approx \mathbf{b}: the best compromise when no exact solution exists. As always, the hat marks an estimate.
    x^=(A⊤A)−1A⊤b\hat{\mathbf{x}} = (A^\top A)^{-1}A^\top\mathbf{b}
  • e=b−Ax^\mathbf{e} = \mathbf{b} - A\hat{\mathbf{x}}“e equals b minus A x hat”
    The error (or residual) vector: the part of b\mathbf{b} the best fit can't reach. Statistics usually calls it r\mathbf{r}.
  • A⊤Ax^=A⊤bA^\top A\hat{\mathbf{x}} = A^\top\mathbf{b}“A transpose A x hat equals A transpose b”
    The normal equations. They say the error is perpendicular (“normal”) to every column of AA.
  • min⁡x ∥b−Ax∥2\min_{\mathbf{x}}\, \|\mathbf{b} - A\mathbf{x}\|^2“the minimum over x of the squared length of b minus A x”
    The least-squares problem: choose x\mathbf{x} to make the sum of squared errors as small as possible.
  • vk=ak−∑j<k(qj⊤ak) qj\mathbf{v}_k = \mathbf{a}_k - \sum_{j<k} (\mathbf{q}_j^\top\mathbf{a}_k)\,\mathbf{q}_j“v k equals a k minus the sum over j less than k of q j transpose a k, times q j”
    One Gram–Schmidt step: strip from ak\mathbf{a}_k its projections onto the earlier directions, then normalise, qk=vk/∥vk∥\mathbf{q}_k = \mathbf{v}_k / \|\mathbf{v}_k\|.
  • A=QRA = QR“A equals Q R”
    The QR factorisation: QQ has orthonormal columns (from Gram–Schmidt) and R=Q⊤AR = Q^\top A is upper triangular.

Linear Algebra for Machine Learning · Chamber 8

Eigenvectors and Eigenvalues

  • Av=λvA\mathbf{v} = \lambda\mathbf{v}“A v equals lambda v”
    The eigenvalue equation: AA only stretches v\mathbf{v}, by the factor λ\lambda, and never turns it off its line. A non-zero v\mathbf{v} that satisfies it is an eigenvector, and λ\lambda is its eigenvalue.
  • λ\lambda“lambda”
    An eigenvalue: the stretch factor. ∣λ∣>1|\lambda| > 1 stretches, ∣λ∣<1|\lambda| < 1 shrinks, a negative λ\lambda flips, and λ=0\lambda = 0 flattens the vector to 0\mathbf{0}.
    λ1=3, λ2=1\lambda_1 = 3,\ \lambda_2 = 1
  • det⁡(A−λI)=0\det(A - \lambda I) = 0“det of A minus lambda I equals zero”
    The characteristic equation. Its solutions are exactly the eigenvalues of AA.
  • p(λ)p(\lambda)“p of lambda”
    The characteristic polynomial det⁡(A−λI)\det(A - \lambda I), of degree nn for an n×nn \times n matrix. For 2×22 \times 2 matrices it is λ2−tr⁡(A) λ+det⁡A\lambda^2 - \operatorname{tr}(A)\,\lambda + \det A.
    p(λ)=λ2−4λ+3p(\lambda) = \lambda^2 - 4\lambda + 3
  • tr⁡(A)\operatorname{tr}(A)“trace of A”
    The trace: the sum of the diagonal entries. It always equals the sum of the eigenvalues.
    tr⁡[2112]=4\operatorname{tr}\begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix} = 4
  • x⊤Axx⊤x\frac{\mathbf{x}^\top A\mathbf{x}}{\mathbf{x}^\top\mathbf{x}}“the Rayleigh quotient of x”
    Exactly λ\lambda when x\mathbf{x} is an eigenvector, and a good estimate of it when x\mathbf{x} is close to one. Power iteration uses it to read off the eigenvalue.
  • Λ\Lambda“capital lambda”
    The diagonal matrix of eigenvalues, Λ=diag⁡(λ1,…,λn)\Lambda = \operatorname{diag}(\lambda_1, \ldots, \lambda_n).
    Λ=[3001]\Lambda = \begin{bmatrix} 3 & 0 \\ 0 & 1 \end{bmatrix}
  • A=VΛV−1A = V\Lambda V^{-1}“A equals V capital lambda V inverse”
    Diagonalisation. The columns of VV are eigenvectors of AA. Reading right to left: change to eigen-coordinates (V−1V^{-1}), stretch each coordinate (Λ\Lambda), change back (VV).
  • Ak=VΛkV−1A^k = V\Lambda^k V^{-1}“A to the k equals V capital lambda to the k V inverse”
    Powers of a diagonalisable matrix: the eigenvectors stay put and only the eigenvalues are raised to the kk-th power.
  • xk+1=Axk∥Axk∥\mathbf{x}_{k+1} = \frac{A\mathbf{x}_k}{\|A\mathbf{x}_k\|}“x k plus one equals A x k over the norm of A x k”
    Power iteration: multiply by AA and rescale, over and over. The direction converges to the eigenvector whose eigenvalue is largest in absolute value.
  • λ1\lambda_1“lambda one”
    Usually the dominant eigenvalue, the one largest in absolute value, when eigenvalues are listed by size. It controls the long-run growth of AkxA^k\mathbf{x}.
  • φ\varphi“phi”
    The golden ratio 1+52≈1.618\frac{1 + \sqrt{5}}{2} \approx 1.618: the dominant eigenvalue of the Fibonacci matrix.
    φ=1+52\varphi = \tfrac{1 + \sqrt{5}}{2}
  • Pπ=πP\boldsymbol{\pi} = \boldsymbol{\pi}“P pi equals pi”
    π\boldsymbol{\pi} (bold pi) is a stationary distribution of a Markov chain with transition matrix PP: an eigenvector with eigenvalue 1 whose entries are probabilities.

Linear Algebra for Machine Learning · Chamber 9

SVD and PCA: The Shape of Data

  • A=UΣV⊤A = U\Sigma V^\top“A equals U sigma V transpose”
    The singular value decomposition: rotate with V⊤V^\top, stretch with Σ\Sigma, rotate with UU. Every real matrix has one.
  • σi\sigma_i“sigma i”
    The ii-th singular value: how much AA stretches along its ii-th direction. Never negative, and sorted from largest to smallest.
    σ1≥σ2≥⋯≥0\sigma_1 \ge \sigma_2 \ge \cdots \ge 0
  • Σ\Sigma“capital sigma, the matrix”
    The m×nm \times n matrix with σ1,σ2,…\sigma_1, \sigma_2, \ldots on its diagonal and zeros everywhere else. In this chamber it is neither a sum nor a covariance.
    Σ=(σ100σ2)\Sigma = \begin{pmatrix} \sigma_1 & 0 \\ 0 & \sigma_2 \end{pmatrix}
  • ui\mathbf{u}_i“u i, a left singular vector”
    Column ii of UU: a unit vector in the output space Rm\mathbb{R}^m, the direction where the ii-th stretch lands.
    Avi=σiuiA\mathbf{v}_i = \sigma_i \mathbf{u}_i
  • vi\mathbf{v}_i“v i, a right singular vector”
    Column ii of VV: a unit vector in the input space Rn\mathbb{R}^n. It is an eigenvector of A⊤AA^\top A with eigenvalue σi2\sigma_i^2.
    A⊤Avi=σi2viA^\top A\mathbf{v}_i = \sigma_i^2 \mathbf{v}_i
  • AkA_k“A sub k”
    The truncated SVD: keep the first kk layers and drop the rest. It has rank kk, and it is the best rank-kk approximation of AA.
    Ak=∑i=1kσiuivi⊤A_k = \sum_{i=1}^{k} \sigma_i \mathbf{u}_i \mathbf{v}_i^\top
  • ∥A∥F\|A\|_F“the Frobenius norm of A”
    The length of AA unrolled into one long vector: the square root of the sum of all squared entries. It equals σ12+⋯+σr2\sqrt{\sigma_1^2 + \cdots + \sigma_r^2}.
    ∥A∥F=∑i,jAij2\|A\|_F = \sqrt{\textstyle\sum_{i,j} A_{ij}^2}
  • ∥A∥2\|A\|_2“the spectral norm of A”
    The biggest stretch AA applies to any unit vector. It equals the largest singular value.
    ∥A∥2=max⁡∥x∥=1∥Ax∥=σ1\|A\|_2 = \max_{\|\mathbf{x}\| = 1} \|A\mathbf{x}\| = \sigma_1
  • xˉ\bar{\mathbf{x}}“x bar, the mean vector”
    The average data point. PCA subtracts it from every point first.
    xˉ=1n∑i=1nxi\bar{\mathbf{x}} = \frac{1}{n}\sum_{i=1}^{n} \mathbf{x}_i
  • XcX_c“X centred”
    The n×dn \times d data matrix (one point per row) with the mean subtracted from every row.
    Xc=X−1xˉ⊤X_c = X - \mathbf{1}\bar{\mathbf{x}}^\top
  • CC“C, the covariance matrix”
    The d×dd \times d matrix of covariances between features, with the variances on its diagonal. Symmetric, and its eigenvalues are never negative.
    C=1nXc⊤XcC = \frac{1}{n} X_c^\top X_c
  • λi\lambda_i“lambda i”
    The ii-th eigenvalue of CC: the variance of the data along the ii-th principal direction.
    λi=σi2/n\lambda_i = \sigma_i^2 / n
  • λi∑jλj\frac{\lambda_i}{\sum_j \lambda_j}“the explained variance ratio”
    The fraction of the total variance that component ii captures.
  • zi\mathbf{z}_i“z i, the scores”
    The low-dimensional code for data point ii: its coordinates along the top kk principal directions.
    zi=Vk⊤(xi−xˉ)\mathbf{z}_i = V_k^\top(\mathbf{x}_i - \bar{\mathbf{x}})
  • S=QΛQ⊤S = Q\Lambda Q^\top“S equals Q lambda Q transpose”
    The spectral theorem: a symmetric matrix has orthonormal eigenvectors (the columns of QQ) and real eigenvalues (the diagonal of Λ\Lambda).