Vectors and Matrices: Reading the Shapes

Discover: counting what you can't yet read

Here is a line from the Transformer paper, the one whose Equation 1 opened Chamber 1:

You don't know yet what a “query projection” is, and you don't need to. Every piece of this line is notation you have already met, and Chamber 2's paper decoder took it apart as set membership. $W_i^Q$ is a capital letter with a subscript and a superscript label (Chamber 1). The symbol $\in$ says “is an element of”, and $\mathbb{R}^{a \times b}$ is the set of grids of real numbers with $a$ rows and $b$ columns (Chamber 2). So the line is a list of shapes.

Shapes are more informative than they look. The paper sets $d_{\text{model}} = 512$ in Section 3.1, and a few lines after this one it uses $h = 8$ heads with $d_k = d_v = 64$. That's enough to answer a real question about the model:

This chamber teaches you to read vectors and matrices the way researchers do: as named objects with shapes, whose entries you can address one at a time. The next course, Linear Algebra for Machine Learning, explains what these objects do. Here you learn to read and write them.

Learn: vectors are columns of numbers

A vector is an ordered list of numbers. Chamber 1's typeface rule says it's printed in bold, and Chamber 2 gave you the set it lives in: $\mathbf{x} \in \mathbb{R}^n$, read “x in R n”, means $\mathbf{x}$ is a list of $n$ real numbers. By convention the list stands up, as a column:

The entries $x_1, \ldots, x_n$ are single numbers, so they're printed plain. Columns eat vertical space, so in running text authors lay the vector down and add a transpose sign: $\mathbf{x} = (x_1, \ldots, x_n)^\top$. The little $\top$, read “transpose”, turns a row into a column and back. It is never a power.

A few vectors are so common they have their own names:

The one-hot vector $\mathbf{e}_i$ is how a class label becomes something a network can use: in a problem with 5 classes, label 3 is $\mathbf{e}_3 = (0, 0, 1, 0, 0)^\top$.

Learn: lengths and dot products

Two vectors of the same length can be combined into a single number by multiplying matching entries and adding. This is the dot product, and it has three common spellings:

The first spelling is the one ML papers use most, because it's a matrix product in disguise: a $1 \times n$ row times an $n \times 1$ column is $1 \times 1$, a single number. The $\Sigma$ is Chamber 5's loop, with $i$ as the dummy index.

A vector also has sizes, called norms. The subscript says which one:

Notation earns its keep when it makes long things short. Here is a derivation you'll see in the opening pages of many papers: the sum of squared errors, rewritten as a norm.

Let $\hat{\mathbf{y}} \in \mathbb{R}^n$ hold a model's predictions and $\mathbf{y} \in \mathbb{R}^n$ the true values, and call the difference $\mathbf{r} = \hat{\mathbf{y}} - \mathbf{y}$, the residual, so $r_i = \hat{y}_i - y_i$. Then

Step (1) uses the definition of $\mathbf{r}$, and (2) says a square is a number times itself. Step (3) recognises the dot product of $\mathbf{r}$ with itself, (4) is the definition of the two-norm, squared, and (5) substitutes $\mathbf{r}$ back in.

So the mean squared error, $\frac{1}{n}\sum_i (\hat{y}_i - y_i)^2$, is written $\frac{1}{n}\|\hat{\mathbf{y}} - \mathbf{y}\|_2^2$: one symbol for a whole loop. In the same spirit, $\mathbf{1}^\top \mathbf{x}$ adds up the entries of $\mathbf{x}$, so the mean of $\mathbf{x}$ is $\frac{1}{n}\mathbf{1}^\top \mathbf{x}$.

Learn: matrices, entries, rows and columns

A matrix is a grid of numbers. $A \in \mathbb{R}^{m \times n}$, read “A is an m by n matrix”, means $m$ rows and $n$ columns, rows first, always. The entry in row $i$ and column $j$ is $A_{ij}$:

For example, if

Papers often need a whole row or a whole column at once, and have borrowed the colon from code for it: $A_{2,:} = (1, 7, 4)$ is row 2, and $A_{:,3} = (2, 4)^\top$ is column 3. The transpose $A^\top$ flips the grid across its diagonal, so the rows become columns:

The data matrix $X$ ties this chamber to Chamber 1's bracketed superscripts. With $n$ examples $\mathbf{x}^{(1)}, \ldots, \mathbf{x}^{(n)}$, each a vector of $d$ features, $X$ stacks them as rows. The entry $X_{ij}$ is feature $j$ of example $i$, the same number as $x^{(i)}_j$.

Learn: products are sums over indices

Here is the most important formula in the chamber. A matrix times a vector is a new vector, and for $A \in \mathbb{R}^{m \times n}$ and $\mathbf{x} \in \mathbb{R}^n$ its $i$-th entry is a sum over $j$:

Read it aloud: “entry $i$ of A x is the sum over $j$ of A i j times x j.” Fix a row $i$, walk along it with $j$, multiply each entry by the matching entry of $\mathbf{x}$, and add. In other words, entry $i$ is row $i$ of $A$ dotted with $\mathbf{x}$. For instance,

A matrix times a matrix works the same way, one entry at a time. For $A \in \mathbb{R}^{m \times n}$ and $B \in \mathbb{R}^{n \times p}$,

Look at where the indices sit. The free indices $i$ and $j$ pick the entry; the dummy index $k$ is summed away, running along row $i$ of $A$ and down column $j$ of $B$. For that to work, the row of $A$ and the column of $B$ must have the same length, $n$. That single fact is the shape rule:

The inner sizes must match and disappear; the outer sizes survive. It also shows why order matters. If $A$ is $2 \times 3$ and $B$ is $3 \times 7$, then $AB$ is $2 \times 7$ while $BA$ doesn't even exist.

Two more operations complete the toolkit, and both have a stricter rule. Addition $A + B$ and the Hadamard product $A \odot B$ work entry by entry, so the two sides must have exactly the same shape:

And watch where the transpose goes on a pair of vectors. $\mathbf{x}^\top\mathbf{y}$ is $(1 \times n)(n \times 1)$, a single number. $\mathbf{x}\mathbf{y}^\top$ is $(n \times 1)(1 \times n)$, a whole $n \times n$ matrix, the outer product. Same letters, same transpose, completely different objects.

Learn: the transpose of a product

Now a proof, entirely by indices. It's short, but it's a model for how most matrix facts are proved: check the shapes, then compare one arbitrary entry.

Claim. For any $A \in \mathbb{R}^{m \times n}$ and $B \in \mathbb{R}^{n \times p}$, $(AB)^\top = B^\top A^\top$.

Proof. First the shapes. $AB$ is $m \times p$, so $(AB)^\top$ is $p \times m$. On the other side, $B^\top$ is $p \times n$ and $A^\top$ is $n \times m$: the inner sizes match, so $B^\top A^\top$ exists and is $p \times m$ too. Two matrices of the same shape are equal when every entry agrees, so take any row $i$ and column $j$:

Step (1) is the definition of the transpose and (2) the definition of the product. Step (3) swaps two numbers, which is allowed because multiplication of numbers commutes. Step (4) uses the definition of the transpose twice, and (5) recognises the definition of the product. Since $i$ and $j$ were arbitrary, every entry agrees, and $(AB)^\top = B^\top A^\top$. $\blacksquare$

The step that does the work is (3). To recognise a product in step (5), we need the pattern “row index $i$, summed index $k$, column index $j$”, and that pattern only appears once $B$ comes first. Swapping the two numbers is what reverses the order of the matrices. It's the same as socks and shoes: you put on socks, then shoes, and to undo it you take off the shoes first.

Learn: become a shape detective

Every matrix expression in a paper can be checked in a few seconds, and experienced readers do it without thinking. The routine:

1. 1

   Write the shape under every symbol

   Vectors are $n \times 1$ columns unless the paper says otherwise. A transpose swaps the two numbers.

2. 2

   Multiply left to right with the shape rule

   Inner sizes must match and vanish; outer sizes survive. Products are associative, so for the shape it doesn't matter which pair you do first.

3. 3

   Sums and ⊙ need identical shapes

   If they differ, the expression is undefined in mathematics, even if NumPy would happily broadcast it (more on that below).

4. 4

   Read the result

   $1 \times 1$ is a number, $n \times 1$ a vector, anything else a matrix.

Try it on the heart of LoRA, the paper you'll read later in this chamber. With $B \in \mathbb{R}^{d \times r}$, $A \in \mathbb{R}^{r \times k}$ and $\mathbf{x} \in \mathbb{R}^k$:

Both junctions match ($r = r$, $k = k$), so the result is a $d$-dimensional vector. Swap the first two and $AB\mathbf{x}$ fails immediately: $(r \times k)(d \times r)$ needs $k = d$. Now it's your turn.

Learn: tensors, batches and code

Real models push notation one step further. A tensor is a grid with three or more indices. A batch of $B$ sentences, each $n$ tokens long, each token a $d$-dimensional vector, is a single tensor $X \in \mathbb{R}^{B \times n \times d}$, with entries $X_{b,i,j}$. Code almost always puts the batch dimension first, so that $X_{b,:,:}$ is one whole example.

This is also where papers and code part ways on rows and columns. PyTorch's nn.Linear documentation states its formula as $y = xA^\top + b$, with a weight of shape (out_features, in_features) and inputs whose last axis holds the features. That's the row convention, and the transpose is exactly the one from the insight box above.

The third difference is broadcasting. In strict matrix algebra, adding a vector to a matrix is undefined. Deep-learning writing relaxes this on purpose. Goodfellow, Bengio and Courville's Deep Learning allows $C = A + \mathbf{b}$, meaning $C_{ij} = A_{ij} + b_j$: add $\mathbf{b}$ to every row. NumPy does the same automatically. It compares shapes from the right, and two sizes are compatible when they are equal or one of them is 1, which is then stretched to fit.

Read beyond the course

These all use exactly the notation of this chamber. Read them with a pencil, writing the shape under each symbol as you go.

Papers and lectures

You can read a real paper's shapes today. Open the LoRA paper and look first at Figure 1 on page 1: the frozen weights are drawn as a big square, and the two small trainable matrices $A$ and $B$ as thin slabs. Then read the first paragraph of Section 4.1, which ends in Equation (3). It's a string of $\in \mathbb{R}^{\cdot \times \cdot}$ statements followed by one equation, and you now know everything in it. Skip the experiments (Sections 5 onwards) for now. Afterwards, open the Transformer paper at Section 3.2.2 and read the last paragraph, where the Discover line comes from.

Watch

The video starts at 15:18, where transposes come in. Watch how a neural network becomes one matrix equation, then how the same shapes explain PyTorch's nn.Linear documentation and its error messages. The first fifteen minutes (linear transformations) are good preparation for the Linear Algebra course.

A short recap of the same notation from a different teacher. Pause whenever a symbol appears and say it aloud before the narrator does.

Your turn

Translate between symbols, words and code, then prove two shape facts and compute your way through three problems, one of them straight from LoRA.