Tokens and Embeddings: How Text Becomes Numbers

A language model never sees letters. It sees a list of integers, one per token, and the first thing it does with each integer is fetch a row from a table of learned vectors. Everything downstream, from attention to sampling, works on those vectors. In 2013, Mikolov and colleagues showed that such vectors can capture meaning well enough that vec("Madrid") − vec("Spain") + vec("France") lands closer to vec("Paris") than to any other word vector. Their model turned two vectors into a probability with a dot product and a softmax over the whole vocabulary: the same shape every LLM's output layer still has.

This chamber builds both halves: how text is cut into tokens, and how each token becomes a vector.

Why not characters, or words?

A tokeniser maps text to ids $t_1,\dots,t_n$ with each $t_i\in\{0,\dots,V-1\}$. The two obvious units both fail. The figures below are rough, for English prose.

Characters make sequences long, and attention, which the next chamber builds, costs time that grows with the square of sequence length; each position also carries little meaning, so the model must spend layers reassembling words. Whole words make the vocabulary enormous, leave rare words with too few examples to learn from, treat run, runs and runner as unrelated, and still meet words they have never seen, which older systems replaced with an [UNK] token. Chinese, Japanese and Thai are written without spaces, so there is not even an obvious word to look up. Subwords sit between: frequent words become single tokens, and rare words split into reusable pieces.

Byte-pair encoding

Byte-pair encoding (BPE) was a 1994 compression trick before Sennrich, Haddow and Birch used it to build subword vocabularies. Training is a loop: split the corpus into words (pre-tokenisation) and each word into base symbols; count every adjacent pair, weighting each word by its frequency; merge the most frequent pair everywhere and record the rule $(a,b)\to ab$; repeat until the vocabulary, base symbols plus one token per merge, reaches its target size.

def merge(symbols, pair):                       # left to right, never overlapping
    out, i = [], 0
    while i < len(symbols):
        if tuple(symbols[i:i + 2]) == pair:
            out.append(symbols[i] + symbols[i + 1]); i += 2
        else:
            out.append(symbols[i]); i += 1
    return out

def train_bpe(words, num_merges):               # words: {("l", "o", "w"): 5, ...}
    merges = []
    for _ in range(num_merges):
        pairs = {}
        for w, c in words.items():
            for p in zip(w, w[1:]):
                pairs[p] = pairs.get(p, 0) + c
        best = max(pairs, key=pairs.get)
        merges.append(best)
        words = {tuple(merge(list(w), best)): c for w, c in words.items()}
    return merges

def encode(word, merges):                       # replay merges in learned order
    symbols = list(word)
    for pair in merges:
        symbols = merge(symbols, pair)
    return symbols

Encoding recounts nothing. It splits new text into base symbols and replays the merge list in order, so a word never seen in training still encodes, as a sequence of learned pieces.

Why the most frequent pair? Suppose a merge replaces $k$ occurrences in a sequence of $n$ symbols. Each replacement turns two symbols into one, and the left-to-right scan never reuses a position, so the result has $n-2k+k=n-k$ symbols. Summed over the corpus, a pair's non-overlapping count is exactly the number of tokens its merge saves, and greedy BPE takes the largest one-step saving. Overlaps are the only subtlety: in $aaa$ the pair $(a,a)$ occurs twice but only one replacement fits. Greedy is also myopic: the best merge for the corpus now need not be best later, and it is certainly not best for every text you will encode. The lab below makes you beat it.

Bytes, SentencePiece and vocabulary size

Byte-level BPE. GPT-2 ran BPE on UTF-8 bytes rather than Unicode characters. Its paper notes that character-level BPE would need a base vocabulary of over 130,000 symbols, while bytes need only 256, and every possible string is a byte sequence. It also stopped merges across character categories (except spaces), so dog, dog. and dog! would not each claim a vocabulary slot. Its vocabulary has 50,257 tokens.

SentencePiece (Kudo and Richardson, 2018) treats the input as a raw stream, with whitespace as an ordinary symbol written ▁, so it needs no language-specific splitting on spaces and decoding is exactly reversible. It implements BPE and the unigram language model, which starts from a large vocabulary and prunes it, and can fall back to bytes for unseen characters. Families such as T5 and Llama 2 used it.

A common rule of thumb for English prose is about 0.75 words, or 4 characters, per token. Code, numbers, unusual names and most other languages cost more. Treat these as rough averages that vary by tokeniser, and count with the real one whenever it matters.

The token tax and other quirks

Tokenisers are not neutral. Petrov et al. (2023) found that the same text translated into different languages can differ in tokenised length by up to 15 times, and even character-level and byte-level models show more than 4-fold differences for some language pairs. Part of the gap is UTF-8 itself, so byte-level models are not immune; subword tokenisers widen it by spending their merges on whatever was frequent in their training data.

UTF-8 spends one byte on each ASCII character, two on accented Latin, Greek, Cyrillic, Arabic or Hebrew letters, three on Devanagari, Thai, Chinese, Japanese or Korean characters, and four on most emoji. A user writing in a heavily taxed language pays more per message, waits through more decoding steps and fits less text into the same window. Other quirks to know:

• Numbers. LLaMA split every number into single digits; other tokenisers group up to three digits; early byte-level tokenisers kept whatever digit strings were frequent, so neighbouring numbers could split differently. Arithmetic is harder when tokens do not line up with place value.
• Leading spaces. In GPT-style tokenisers hello and hello are different tokens, because the space travels with the word. A prompt that ends in a stray space can push the model towards unusual continuations.
• Glitch tokens. Rumbelow and Watkins found tokens such as SolidGoldMagikarp that GPT-2 and GPT-3 models handled bizarrely, often unable even to repeat them. The likely cause: strings that were frequent in the tokeniser's training data but almost absent from the model's, so their embeddings were barely trained.
• Special tokens. Reserved ids never produced by merges mark the end of a text (GPT-2's <|endoftext|>), padding, or chat roles. A chat template flattens a conversation into one token sequence; one common style looks like this:

<|im_start|>system
You are a concise assistant.<|im_end|>
<|im_start|>user
How many tokens is this?<|im_end|>
<|im_start|>assistant

Each model family has its own template, and the wrong one quietly degrades output. Check how your tokeniser treats a literal <|im_end|> typed by a user, because libraries differ. Chamber 6, Prompting as Programming, goes deeper.

Embeddings: a lookup table that learns

Write token id $i$ as the one-hot row vector $\mathbf x_i\in\{0,1\}^V$ and let $E\in\mathbb R^{V\times d}$. Then

$(\mathbf x_iE)_j=\sum_{k=1}^{V}(\mathbf x_i)_kE_{kj}=E_{ij},$

because only $k=i$ contributes: multiplying by a one-hot vector selects row $i$. Frameworks skip the multiplication and index the row directly (an embedding layer is a gather), and for a sequence of $n$ tokens the transformer consumes $X=SE\in\mathbb R^{n\times d}$, where $S$ stacks the one-hot rows. The embedding says nothing about position; transformers add that separately.

$E$ starts random and is trained by backpropagation like any other weight. Since $X=SE$, the chain rule gives $\partial L/\partial E=S^\top\,\partial L/\partial X$: row $i$ of the gradient sums the gradients at the positions where token $i$ occurred, and every token absent from the batch gets exactly zero. Rare tokens are therefore rarely updated, which is how glitch tokens survive.

At the other end, the final hidden state $\mathbf h\in\mathbb R^d$ is scored against every token, $\mathbf z=\mathbf hW_{\text{out}}^\top$, and a softmax turns the logits into next-token probabilities: Mikolov's Equation (2) with the rows of $W_{\text{out}}$ as the output vectors $v'$. Tied embeddings set $W_{\text{out}}=E$ (Press and Wolf, 2017), saving $Vd$ parameters: with an illustrative $V=128{,}000$ and $d=4096$, that is about 524 million. GPT-2 ties them; many larger models keep two tables.

Comparing embeddings: cosine similarity

$\cos(\mathbf u,\mathbf v)=\frac{\mathbf u\cdot\mathbf v}{\|\mathbf u\|\,\|\mathbf v\|}.$

By Cauchy–Schwarz, $|\mathbf u\cdot\mathbf v|\le\|\mathbf u\|\|\mathbf v\|$, so the cosine lies in $[-1,1]$; you will prove the equality cases below. It compares directions and ignores lengths, whereas a raw dot product also rewards long vectors. For word2vec-style embeddings, nearest neighbours by cosine tend to be related words, and differences can encode relations, as in the Madrid example; in code, normalise every row of $E$ once and a single matrix product gives all the cosines. An LLM's input embedding rows are a weaker notion of meaning than its contextual hidden states; dedicated embedding models produce one vector per passage, and Chamber 8, Retrieval, Tools and Agents, searches with them.

Context windows and paying per token

The context window $n_{\text{ctx}}$ is the most tokens a model attends over in one call, prompt and output together: $n_{\text{prompt}}+n_{\text{output}}\le n_{\text{ctx}}$. GPT-2's held 1,024 tokens; current models accept far more, but every token still costs compute, time and money. APIs bill per token, usually quoted per million, with input and output priced separately and output typically dearer; many discount input that repeats across calls through caching. Self-hosting pays in GPU time per token instead, which Chamber 9 prices out.

• Count with the model's own tokeniser: tiktoken for OpenAI models, Hugging Face tokenizers for open-weight models, or the token-counting endpoints that APIs such as Anthropic's and Google's Gemini provide.
• Budget the window: system prompt, history, retrieved documents, tool definitions and the reserved output must all fit.
• Measure tokens per message for each language your users write in, not per English word.

Read beyond

Read the equation in context

Sennrich and colleagues wanted translation without a fixed word list: names, compounds and loanwords can be translated piece by piece. Their BPE learner is a few lines of Python, and segmenting rare words this way improved over a back-off dictionary baseline by up to 1.1 and 1.3 BLEU on English→German and English→Russian.

Skip-gram learns vectors by predicting the words around each word. Section 2 maximises the average log probability (Equation 1)

$\frac{1}{T}\sum_{t=1}^{T}\sum_{-c\le j\le c,j\ne0}\log p(w_{t+j}|w_t),$

where $c$ is the window size, and defines each probability by the softmax of the opening Glimpse, with separate "input" and "output" vectors per word. The authors call that softmax impractical because its gradient costs time proportional to $W$, often $10^5$–$10^7$ terms, and replace it with hierarchical softmax and negative sampling. LLM training does compute the full softmax over $V$ at every position: GPUs made the price worth paying.

Your turn

Train a tokeniser by hand. Greedy BPE merges whatever is most frequent in the corpus; your job is to choose merges that squeeze three particular phrases into their token budgets, the last built from words the corpus never contains. Watch your curve against greedy's as you go.

Coding problems