From Pretraining to Assistant

In 2021 DeepMind trained Gopher: 280 billion parameters on 300 billion tokens. A few months later the same team spent the same compute on Chinchilla, with a quarter of the parameters and more than four times the data, and Chinchilla beat Gopher across a wide range of tasks. The difference was arithmetic, not architecture: how to split a fixed budget between a bigger model and more text. Their answer rests on a five-constant formula fitted to hundreds of training runs.

This chamber follows a model from that budget to an assistant: pretraining and its compute, scaling laws, then instruction tuning, preference optimisation and cheap fine-tuning with LoRA.

Pretraining: one objective, trillions of tokens

Pretraining is the previous chamber's next-token prediction run at enormous scale: minimise $-\frac1T\sum_t\log p_\theta(x_t\mid x_{<t})$ over a corpus of trillions of tokens. Most of it starts as web crawl, joined by code, books and papers. Raw crawl is mostly unusable, so the data pipeline does as much work as the optimiser:

• Extraction and language identification: pull the text out of HTML and keep the languages you want.
• Quality filtering: heuristic rules (length, symbol ratios, repeated lines) plus classifiers that score pages for quality or educational value.
• Deduplication: drop exact and near-duplicate documents, commonly with MinHash, so repeated text is not memorised and over-weighted.
• Hygiene: remove personal data and toxic content, and decontaminate by deleting benchmark test items so later evals stay honest.

FineWeb, an open dataset built this way, keeps 15 trillion tokens from 96 Common Crawl snapshots. The result of pretraining is a base model: it continues documents. Ask it a question and it may answer, or it may write three more questions, because a page of questions is plausible text.

Counting compute: C ≈ 6ND

In the forward pass each parameter takes part in one multiply and one add per token: about $2N$ FLOPs. The backward pass computes gradients for both activations and weights, roughly twice that, $4N$. So training on $D$ tokens costs

$C\approx6ND\ \text{FLOPs}.$

Kaplan et al. (2020) note that attention adds a context-length term, small while the model width exceeds about a twelfth of the context. Check the rule on a public number: Llama 3's 405B model saw 15.6T tokens, and $6\times4.05\times10^{11}\times1.56\times10^{13}\approx3.8\times10^{25}$, the figure its report gives. Inference runs only the forward pass, about $2N$ FLOPs per generated token. Remember that; it returns below.

Scaling laws and the compute-optimal split

Kaplan et al. found that loss falls as a power law in $N$, $D$ and $C$, with some trends spanning more than seven orders of magnitude, and advised spending most extra compute on a larger model ($N_{\text{opt}}\propto C^{0.73}$). Hoffmann et al. trained over 400 models, matched each run's learning-rate schedule to its length, and found instead that parameters and tokens should grow roughly equally. Their third method fits the Glimpse formula with $E=1.69$, $A=406.4$, $B=410.7$, $\alpha=0.34$ and $\beta=0.28$. Here $E$ plays the entropy of text, and the other two terms are the penalties for a finite model and for finite data.

Fix the budget $C$. Each model size $N$ then buys $D=C/(6N)$ tokens, and these pairs trace an iso-compute curve (the paper's IsoFLOP profile). Find its lowest point by substitution. Put $D=C/(6N)$ into the loss, differentiate in $N$, set the result to zero and multiply by $N$:

$\alpha AN^{-\alpha}=\beta B\left(\tfrac C6\right)^{-\beta}N^{\beta}=\beta BD^{-\beta}.$

At the optimum, $\alpha$ times the model term equals $\beta$ times the data term. Solving for $N$ gives the paper's Eq. (4):

$N_{\text{opt}}=G\left(\frac C6\right)^{\frac{\beta}{\alpha+\beta}},\qquad G=\left(\frac{\alpha A}{\beta B}\right)^{\frac1{\alpha+\beta}}.$

With the fitted exponents, $N_{\text{opt}}\propto C^{0.45}$ and $D_{\text{opt}}\propto C^{0.55}$.

Training past the optimum

Chinchilla minimises training compute. A deployed model also pays about $2N$ FLOPs for every token it generates, for as long as it serves. If it will generate trillions of tokens, a smaller model trained on more data reaches the same loss with a lower lifetime bill, as Sardana et al. (2024) work out with the Chinchilla fit. That is why open-weight families such as Llama 3 and Qwen2.5 report pretraining sets of 15–18 trillion tokens even for models of a few billion parameters: hundreds to thousands of tokens per parameter. Loss keeps falling out there, only slowly. The last coding problem puts numbers on the trade.

From base model to assistant

Supervised fine-tuning (SFT), or instruction tuning, continues training on prompt–response pairs: human-written demonstrations, curated datasets or a stronger model's outputs. Each conversation is rendered in the model's chat template, special tokens that mark roles and turns. An illustrative one:

<|system|>You are a concise assistant.<|end|>
<|user|>Name a prime between 10 and 15.<|end|>
<|assistant|>11 (or 13).<|end|>

The loss is ordinary cross-entropy, counted only on the assistant's tokens. Templates differ between model families, so render with the model's own (for example apply_chat_template in Hugging Face Transformers, or the template your inference server loads with the weights). A wrong template is a silent bug: the model still answers, just worse. SFT mostly teaches format and behaviour: LIMA got a strong assistant from 1,000 well-chosen examples.

Learning from preferences: RLHF

Many qualities are easier to judge than to write. RLHF collects comparisons: for a prompt $x$, a labeller marks response $y_w$ as better than $y_l$. A reward model $r_\phi$ is fitted with the Bradley–Terry likelihood

$P(y_w\succ y_l\mid x)=\sigma\big(r_\phi(x,y_w)-r_\phi(x,y_l)\big),$

minimising $-\log\sigma(r_w-r_l)$: logistic regression on reward differences. Only differences enter, so adding a constant to every reward for a prompt changes nothing (you will prove it). The policy is then optimised, classically with PPO, to

$\max_{\pi_\theta}\ \mathbb E_{x,\,y\sim\pi_\theta}\big[r_\phi(x,y)\big]-\beta\,D_{\mathrm{KL}}\big(\pi_\theta(\cdot\mid x)\,\|\,\pi_{\text{ref}}(\cdot\mid x)\big).$

The KL penalty tethers the policy to the reference (the SFT model), where the reward model was trained and can be trusted. Without it the policy finds outputs the reward model overrates: reward hacking. In InstructGPT, labellers preferred a 1.3B model trained this way to the 175B GPT-3.

DPO: preferences without a reward model

The KL-regularised objective has a closed-form optimum, $\pi^*(y\mid x)=\pi_{\text{ref}}(y\mid x)\,e^{r(x,y)/\beta}/Z(x)$. Take logs and solve for the reward: $r=\beta\log\frac{\pi^*}{\pi_{\text{ref}}}+\beta\log Z(x)$. The partition function $Z(x)$ is intractable, but it depends only on the prompt, so it cancels in the Bradley–Terry difference. Fitting the policy straight to the preferences gives the DPO loss, a classifier on log-probability ratios:

import torch.nn.functional as F

def dpo_loss(pol_w, pol_l, ref_w, ref_l, beta=0.1):
    # Summed log-probabilities of the chosen (w) and rejected (l) responses.
    margin = beta * ((pol_w - ref_w) - (pol_l - ref_l))
    return -F.logsigmoid(margin).mean()

DPO still needs preference pairs and a frozen reference, but no reward model and no sampling during training. Online methods that sample from the current policy remain common too; which wins depends on the task and the data.

Verifiable rewards and reasoning models

When a program can check the answer, you need no learned reward model. RL with verifiable rewards scores sampled solutions with unit tests, exact final answers or format checks. DeepSeek-R1 showed that this kind of RL on a strong base model, with rule-based rewards and no supervised reasoning traces (its R1-Zero variant), produces long chains of reasoning with self-checking. Reasoning can then be distilled: fine-tune a smaller student on a teacher's outputs, or match its token distributions. That is how R1's reasoning reached smaller open models.

LoRA: fine-tuning a low-rank update

Full fine-tuning updates every weight and keeps optimiser state for each. LoRA (Hu et al., 2021) freezes a pretrained $W\in\mathbb R^{d\times k}$ and learns

$W'=W+BA,\qquad B\in\mathbb R^{d\times r},\ A\in\mathbb R^{r\times k},\ r\ll\min(d,k).$

The adapter has $r(d+k)$ parameters instead of $dk$: at $r=16$ on a $4096\times4096$ matrix, $131{,}072$ instead of $16{,}777{,}216$. Its rank is at most $r$, because every column of $BA$ is $B$ times a column of $A$ and so lies in the span of $B$'s $r$ columns. $B$ starts at zero, so training begins exactly at the pretrained model. Afterwards $BA$ can be merged into $W$ at no inference cost, or kept apart so that many small adapters share one base. QLoRA (Dettmers et al., 2023) keeps the frozen base in 4-bit NormalFloat and trains 16-bit adapters through it, enough to fine-tune a 65B model on one 48 GB GPU.

Fine-tuning on narrow data can erase skills the model had: catastrophic forgetting. Mix some general data into the set, use a small learning rate and few epochs, and evaluate broadly before and after. LoRA helps here too: Biderman et al. (2024) found that it learns less than full fine-tuning but forgets less.

Prompt, retrieve or fine-tune?

Go down the table in order of cost, and let an eval suite decide each step (Prompt Version Control and Evals).

Read beyond

Read the equation in context

Section 3.3 proposes the parametric loss as a risk decomposition, fits its five constants with a Huber loss over all runs, and gives the efficient frontier in closed form as Eq. (4). Compare its exponents with Table 2, and the 20-tokens rule with Table 3.

Section 4 derives DPO in four moves: the optimal KL-regularised policy (Eq. 4), the reward rewritten through it (Eq. 5), the partition function cancelling inside Bradley–Terry (Eq. 6), and the loss (Eq. 7). Decode that loss below.

Your turn

Pick a budget, slide the model size along its iso-compute curve and find the bottom of the bowl. Watch the ratio of the two loss terms as you go: the balance condition says it settles at $\beta/\alpha\approx0.82$ at the optimum.

Coding problems