Matrices as Transformations

Discover: six numbers that move an image

A network that reads house numbers from street photographs has a problem: the digits turn up at every size, angle and position. In 2015 a team at Google DeepMind gave networks a module that learns to straighten its input before reading it, the spatial transformer. At its heart is one equation. For each pixel $i$ of the output image, it computes where in the input image to look:

The superscript $t$ marks a target point on the output's grid, and $s$ marks the source point in the input where its value comes from. A small network looks at the image and predicts the six numbers $\theta_{11}, \ldots, \theta_{23}$, and those six numbers are enough to rotate, zoom, shear, shift and crop. How can a 2-by-3 grid of numbers do all that? And what is that lonely $1$ doing at the bottom of the vector? By the end of this chamber you'll be able to read every piece.

Learn: what makes a map linear

A map (or transformation) $T : \mathbb{R}^n \to \mathbb{R}^m$ is a function that takes a vector with $n$ entries and returns one with $m$ entries. Read it as “T maps R n to R m”. Most maps are hopelessly complicated. Linear maps are the ones that respect the two operations from Chamber 1: adding and scaling.

In words: transforming a sum gives the sum of the transforms, and transforming a stretched vector gives the stretched transform. Geometrically, a linear map keeps grid lines straight, parallel and evenly spaced, and keeps the origin fixed. That last part is a theorem, with a two-line proof.

Claim. Every linear map sends $\mathbf{0}$ to $\mathbf{0}$.

Proof. The zero vector is $0 \cdot \mathbf{0}$, so the scaling rule with $c = 0$ gives $T(\mathbf{0}) = T(0 \cdot \mathbf{0}) = 0 \cdot T(\mathbf{0}) = \mathbf{0}$. $\blacksquare$

The claim doubles as a test. The map $T(x, y) = (x + 1, y)$, which slides everything one step right, sends $\mathbf{0}$ to $(1, 0)$, so it can't be linear. The map $T(x, y) = (x^2, y)$ does keep the origin, but fails the scaling rule: $T(2, 0) = (4, 0)$, while $2\,T(1, 0) = (2, 0)$. And $T(x, y) = (2x - y,\ x + y)$ passes both rules, as you can check by expanding.

Learn: a matrix is where the basis vectors land

Here is the fact that makes linear maps easy to work with. Every vector in the plane is a combination of the standard basis vectors (Chamber 1): $\mathbf{x} = x_1\mathbf{e}_1 + x_2\mathbf{e}_2$. Apply a linear map $T$ and use its two rules:

The first step uses the addition rule, the second the scaling rule. So once you know the two vectors $T(\mathbf{e}_1)$ and $T(\mathbf{e}_2)$, you know $T$ everywhere. Store them as the columns of a matrix:

That's the most important sentence in this chamber: the columns of a matrix are where the basis vectors land, and $A\mathbf{x}$, read “A times x”, is the combination of the columns weighted by the entries of $\mathbf{x}$. The same holds in any size. An $m \times n$ matrix has $n$ columns, one for each basis vector $\mathbf{e}_j$ of $\mathbb{R}^n$, and each column $\mathbf{a}_j = A\mathbf{e}_j$ is a vector in $\mathbb{R}^m$. (3Blue1Brown calls $\mathbf{e}_1$ and $\mathbf{e}_2$ “i-hat” and “j-hat”, written $\hat{\imath}$ and $\hat{\jmath}$. You'll see those names in the lab.)

A worked example. Let $T$ send $\mathbf{e}_1$ to $(2, 1)$ and $\mathbf{e}_2$ to $(-1, 1)$. Where does $\mathbf{x} = (3, 2)$ go?

You can also compute it row by row, as in Mathematics for ML, Chamber 6: each entry of $A\mathbf{x}$ is a row of $A$ dotted with $\mathbf{x}$. The first entry is $(2, -1) \cdot (3, 2) = 4$ and the second is $(1, 1) \cdot (3, 2) = 5$. Same answer, two views. The column view tells you what the map does; the row view is how you'd work it out by hand.

Learn: a gallery of transformations

Now you can build transformations to order: decide where $\mathbf{e}_1$ and $\mathbf{e}_2$ should go, and write those images down as columns.

Only the rotation needs any thought, so let's derive it. Rotating keeps lengths, so $\mathbf{e}_1$ lands on the unit circle at angle $\theta$ from the $x$-axis: by the definition of cosine and sine, at $(\cos\theta, \sin\theta)$. The vector $\mathbf{e}_2$ starts at angle $90°$ and lands at angle $90° + \theta$, which is the point $(\cos(90° + \theta), \sin(90° + \theta)) = (-\sin\theta, \cos\theta)$. Those two images are the columns of the rotation matrix, written $R_\theta$ and read “R theta”:

Area, orientation and the determinant

Watch what each map does to the unit square, with corners $\mathbf{0}$, $\mathbf{e}_1$, $\mathbf{e}_1 + \mathbf{e}_2$ and $\mathbf{e}_2$. It lands on the parallelogram built on the two columns, and in Chamber 3 you met the number that measures a parallelogram's signed area. For a matrix it's called the determinant:

Because the map is linear, every region's area gets multiplied by the same factor, $|\det A|$ (Chamber 5 makes this precise). The sign carries orientation. A negative determinant means the map flips the plane over, like a mirror: a clockwise loop comes out anticlockwise. And $\det A = 0$ means the square has been squashed flat: the columns are dependent, and the whole plane lands on a line or a single point. Rotations and shears have determinant $1$: they move area around without changing how much there is.

Time to sculpt. In the lab, type the four entries of a matrix and watch the grid, the unit square and a letter F move. The F has no mirror symmetry, so it shows you at a glance when a matrix flips the plane.

Learn: composition is multiplication

Do one transformation, then another. First $B$, then $A$: the vector $\mathbf{x}$ goes to $B\mathbf{x}$, and then to $A(B\mathbf{x})$. The combined map is again linear (you'll assemble the proof in Your turn), so by the previous section it has a matrix. Which one? Follow the basis vectors. The vector $\mathbf{e}_j$ goes first to $B\mathbf{e}_j = \mathbf{b}_j$, the $j$-th column of $B$, and then to $A\mathbf{b}_j$. So

Matrix multiplication is defined to be composition. That is the whole reason for its strange-looking rule. Read off entry $i$ of column $j$ and you get back the formula from Mathematics for ML, Chamber 6, $(AB)_{ij} = \sum_k A_{ik}B_{kj}$. And just like $f \circ g$, the product is read right to left: in $AB$, the matrix $B$ acts first.

Shapes follow the same logic. If $B$ is $n \times p$, it takes $p$-vectors to $n$-vectors; then $A$ must accept $n$-vectors, so it's $m \times n$, and the composition takes $p$-vectors to $m$-vectors:

Order matters

Let $R$ rotate by $90°$ and let $F$ reflect in the $x$-axis. Follow the basis vectors through each order, or multiply:

Reflecting first and then rotating gives the mirror in the line $y = x$. Rotating first and then reflecting gives the mirror in $y = -x$. Different maps, so $RF \neq FR$. Matrix multiplication is not commutative.

What does hold is associativity: $(AB)C = A(BC)$. Both sides mean “do $C$, then $B$, then $A$”, so they must be the same map, and no index-juggling is needed to see it. Associativity never changes the answer, but it can change the cost enormously, which Problem 3 below explores.

A proof: rotations add up

Rotating by $\beta$ and then by $\alpha$ should be the same as rotating by $\alpha + \beta$. Geometrically that's obvious. Algebraically, it hides the trigonometric addition formulas inside it.

Claim. For all angles $\alpha$ and $\beta$, $R_\alpha R_\beta = R_{\alpha + \beta}$.

Proof. Multiply, taking each row of $R_\alpha$ dotted with each column of $R_\beta$:

Now use the angle-addition identities

The two diagonal entries are both $\cos(\alpha + \beta)$, the bottom-left entry is $\sin(\alpha + \beta)$, and the top-right entry is $-\sin(\alpha + \beta)$. So

Two bonuses come free. Since $\alpha + \beta = \beta + \alpha$, rotations of the plane commute with each other, even though matrices in general don't. And you can run the proof backwards: if you ever forget the addition formulas, multiply two rotation matrices and read them off.

Learn: four ways to multiply

The same product can be computed in four ways, and each is the natural view somewhere in machine learning. Take

1. 1

   Dot products, entry by entry

   Entry $(i, j)$ of $AB$ is row $i$ of $A$ dotted with column $j$ of $B$. For example, the top-right entry is $(1, 2) \cdot (6, 8) = 6 + 16 = 22$. This is the view behind attention scores: each entry of $QK^\top$ is one query dotted with one key.

2. 2

   Columns

   Column $j$ of $AB$ is $A$ times column $j$ of $B$: the first column is $5(1, 3) + 7(2, 4) = (19, 43)$. This is the composition view, with $A$ transforming each column of $B$ in turn. It's also a layer applied to a whole batch of inputs stored as columns.

3. 3

   Rows

   Row $i$ of $AB$ is row $i$ of $A$ times $B$, a combination of the rows of $B$: the first row is $1(5, 6) + 2(7, 8) = (19, 22)$. This is the view when data sits in rows, as in most code: X @ W transforms each example, one row of X at a time.

4. 4

   Outer products, column times row

   $AB$ is a sum of outer products, column $k$ of $A$ times row $k$ of $B$: the column $(1, 3)$ times the row $(5, 6)$ gives $\begin{bmatrix} 5 & 6 \\ 15 & 18 \end{bmatrix}$, the column $(2, 4)$ times the row $(7, 8)$ gives $\begin{bmatrix} 14 & 16 \\ 28 & 32 \end{bmatrix}$, and their sum is $AB$. Each piece is a matrix of rank one, the building block of LoRA (Chamber 6).

All four describe the same $mnp$ multiplications, grouped differently. Which grouping you choose decides what you can see, and how fast your code runs.

Learn: a neural layer, and maps that move the origin

A fully connected layer of a neural network computes

read “h equals sigma of W x plus b”. Take it apart with this chamber's eyes:

• $W\mathbf{x}$ is a linear map. The matrix $W \in \mathbb{R}^{m \times n}$ takes $n$ inputs to $m$ outputs, and its columns say where each input direction lands.
• Adding $\mathbf{b}$ shifts the result. Together, $\mathbf{x} \mapsto W\mathbf{x} + \mathbf{b}$ is an affine map: a linear map followed by a translation. It isn't linear unless $\mathbf{b} = \mathbf{0}$, because it moves the origin to $\mathbf{b}$.
• $\sigma$ is a nonlinear function, such as ReLU or tanh, applied to each entry separately. It bends the grid, which no matrix can do.

Why does the nonlinearity matter so much? Stack two layers without it and expand:

That's a single affine map, with matrix $W_2W_1$ and shift $W_2\mathbf{b}_1 + \mathbf{b}_2$. A hundred linear layers collapse into one. Without $\sigma$ between them, depth buys nothing: the whole network would be one matrix multiplication plus a shift.

Homogeneous coordinates: the trick with the 1

Affine maps aren't linear, which is a nuisance: we'd like to compose them by multiplying matrices. The fix is to append one extra coordinate that is always $1$. For $\mathbf{x} \in \mathbb{R}^2$ and an affine map $\mathbf{x} \mapsto A\mathbf{x} + \mathbf{t}$,

The last column gets multiplied by the $1$, so it gets added: a shift disguised as a matrix product. Now every affine map of the plane is a $3 \times 3$ matrix, and composing affine maps is plain matrix multiplication. 3D graphics software uses the same trick, with $4 \times 4$ matrices, to move objects around a scene.

This is precisely the $1$ in the Spatial Transformer's Eq. (1). Its matrix $\mathtt{A}_\theta$ is the top two rows of the $3 \times 3$ matrix above. The left $2 \times 2$ block ($\theta_{11}, \theta_{12}, \theta_{21}, \theta_{22}$) is the linear part, which rotates, scales and shears. The last column ($\theta_{13}, \theta_{23}$) is the translation. For attention, the paper restricts it further to a zoom $s$ and a shift $(t_x, t_y)$:

Its left block is $s$ times the identity, a pure zoom. With $s < 1$ the output grid covers only a small window of the input, centred at $(t_x, t_y)$: the network crops and zooms in on whatever it has learned to look at.

Read beyond the course

Papers and lectures

Spatial Transformer Networks is clearly written and well illustrated. Read the abstract and look at Figure 1, where distorted digits come out straightened. Then read Section 3 up to the end of Section 3.2: the localisation network that predicts $\theta$, and the parameterised sampling grid with Eq. (1) and Eq. (2). Notice the direction of the arrow: the matrix takes each point of the output grid and says where to sample in the input, because every output pixel needs exactly one value. Section 3.3, on sampling differentiably, is worth a skim. The experiments in Section 4 can wait, apart from the pictures in Table 1.

Watch

The first video is this chamber's big idea in motion: keep your eye on where $\hat{\imath}$ and $\hat{\jmath}$ land, and read the matrix off their coordinates. The second shows composition as “apply one, then the other” and explains, with no index-juggling, why the order of a product matters and why associativity is automatic.

Your turn

You can now read a matrix as a transformation, compose transformations and take a neural layer apart. Match matrices to their effects and formulas to code, then assemble a proof, write one, and solve three problems in code.

Proofs

The first proof shows that composing linear maps gives a linear map, which is why a product $AB$ is again a matrix. The second you write yourself, with the norm from Chamber 2.

Code it up

The first problem rewards linearity, the second applies the Spatial Transformer's Eq. (1) to a whole sampling grid, and the third asks what associativity is worth.