Prompting as Programming

Ask a model to label a review and you may get Sure! Here is the JSON: {'sentiment': 'mixed'} Hope that helps!. Every word is reasonable, and your parser rejects all of it. This chamber has two complementary fixes: write the prompt precisely enough that the model usually does what you mean, then constrain decoding so that the output always parses. The second fits in one line of maths: hide the tokens the grammar forbids, then sample as usual.

Roles, templates and the system prompt

A chat model never receives "messages". It receives one token sequence, rendered by a chat template that marks each turn with special tokens (Chamber 1) and leaves the assistant turn open:

<|system|>You label product reviews. Reply with JSON only.<|end|>
<|user|>Review: "Arrived late, but works perfectly."<|end|>
<|assistant|>

Markers differ by model family (ChatML's <|im_start|>, Llama 3's header tokens), and tokeniser libraries render them for you, for example Hugging Face's apply_chat_template. The system turn holds the operator's standing instructions, the user turn the request, and the assistant turn what the model writes. Instruction tuning (Chamber 3) teaches the model to give the system turn priority, but it is still just conditioning: the reply is sampled from $p(y\mid s,u)$. If you serve open weights, a missing or doubled special token in a hand-built template is a classic silent bug.

Write the prompt like an interface

A function has a name, typed arguments, documented behaviour and a return type. A good prompt specifies the same things, for a reader who knows nothing you haven't written down:

Separate instructions from data with delimiters such as XML-style tags, so the model can tell where a pasted document ends and you can refer to it by name. In code, a prompt is a template with variables, rendered per request; Chamber 7 versions and tests templates like any other code.

TEMPLATE = """Classify the sentiment of the review in <review> tags.
Allowed labels: positive, negative, neutral. A mixed review gets its dominant label.
Reply with exactly one JSON object: {{"sentiment": "<label>"}}

<review>
{review}
</review>"""

prompt = TEMPLATE.format(review=user_text)

Few-shot examples

Brown et al. (2020) showed that a large enough model can pick up a task from a few demonstrations in its context, with no gradient step: in-context learning. Choose examples as you would a test set: cover the hard cases, vary length and wording so the model doesn't copy surface features, balance the labels, and mind the order, since models can favour the majority label and labels near the end of the prompt (Zhao et al., 2021). Min et al. (2022) found something surprising: replacing the gold labels in demonstrations with random ones barely hurt classification and multiple-choice accuracy. The demonstrations mainly taught the label space, the distribution of inputs and the format. That is no licence for wrong labels: larger models can follow deliberately flipped ones (Wei et al., 2023).

Reasoning, decomposition and prefill

Chain of thought asks for intermediate steps before the answer: Wei et al. (2022) used a few worked exemplars, and Kojima et al. (2022) a zero-shot cue, "Let's think step by step". Each generated token gets one forward pass (Chamber 2), so writing reasoning $z$ first buys computation the answer can condition on: $p(y\mid x)=\sum_z p(z\mid x)\,p(y\mid x,z)$. Wei et al. saw gains mainly in models of about 100B parameters or more, and most on multi-step problems; on simple lookups, reasoning only adds latency and tokens. Reasoning models are trained with reinforcement learning to write such chains themselves. For them, state goals and constraints plainly instead of prescribing steps, and test whether examples still help: DeepSeek-R1's authors report that few-shot prompting degraded its performance (DeepSeek-AI, 2025).

Decomposition splits a long task into a prompt chain (extract, then classify, then summarise), with code checking each intermediate result; each stage is easier to test and can use its own model and temperature. Prefilling writes the first tokens of the assistant turn yourself, such as {"sentiment": ", so the model starts inside the format. Raw templates always allow it; some hosted APIs do and others don't.

Structured output that always parses

Validation also catches what no grammar expresses, such as "the quoted span must appear in the source". If each attempt passes independently with probability $p$, the number of attempts $N$ is geometric, $P(N=k)=(1-p)^{k-1}p$, so

$E[N]=\sum_{k\ge1}P(N\ge k)=\sum_{k\ge1}(1-p)^{k-1}=\frac1p.$

At $p=0.95$ retries are cheap; at $p=0.1$ you pay ten calls on average, plus latency. Better to raise $p$, or make it 1 with a mask.

Masking is conditioning

Compile the schema into a finite automaton with states $q$ and transitions $\delta(q,v)$. Before step $t$, the allowed set $A(q_{t-1})$ holds the tokens leading to states from which the schema can still be completed. Zero the rest and renormalise:

$\tilde p(v\mid x_{<t})=\frac{m_v\,p(v\mid x_{<t})}{Z_t},\qquad Z_t=\sum_{u\in A(q_{t-1})}p(u\mid x_{<t}).$

For one step this is exact conditioning: if $X\sim p$, then $P(X=v\mid X\in A)=p(v)/p(A)$ on $A$ and zero elsewhere. On logits, set masked entries to $-\infty$ (not 0, since $e^0=1$) and let the softmax renormalise. If every reachable state allows some token and can still reach acceptance, the sampler never gets stuck and can only stop in an accepting state, so the output always parses; you will prove this below. One token can span several grammar symbols, which is why Willard and Louf precompute, for each automaton state, the vocabulary tokens it can read.

def masked_step(probs, allowed, rng):
    kept = sum(probs[v] for v in allowed)                        # Z_t
    weights = [probs[v] / kept if v in allowed else 0.0 for v in range(len(probs))]
    return rng.choices(range(len(probs)), weights)[0], 1 - kept  # token, removed mass

The kept masses measure how far the raw model was from compliance. Along a valid output, $p(x)=\prod_t Z_t\,\tilde p(x_t\mid x_{<t})$, so if the schema forces a single path, $P(\text{valid})=\prod_tZ_t$; with branches, $P(\text{valid})=\mathbb E_{\tilde p}\bigl[\prod_tZ_t\bigr]$. But conditioning each step is not conditioning the whole string. The masked sampler produces each valid $x$ with probability $p(x)/\prod_tZ_t$, whereas $p(x\mid\text{valid})=p(x)/P(\text{valid})$, and the two agree only when $\prod_tZ_t$ is the same for every valid $x$. Masking over-weights prefixes from which the model rarely finishes validly (Park et al., 2024). It guarantees syntax, not quality: describe the schema in the prompt too (some engines apply it only as a mask, so the model never sees it), keep schemas close to what the model would write anyway, and put any reasoning field before the answer field.

Long contexts and caching

With long inputs, put the documents first and the question and instructions last; several providers' guides recommend this, some also repeating key instructions at both ends. Liu et al. (2024) showed why position matters: in multi-document question answering, accuracy was highest with the relevant document at the start or end of the context and dropped when it sat in the middle. Wrap each document in tags with its source, ask for the relevant quotes before the answer, and retrieve less but better (Chamber 8).

Prompt caching reuses the KV cache (Chamber 2) for a prefix the server has recently seen, cutting cost and time to first token. Hits need an exact prefix match, so order the prompt from most stable to least: system instructions, tool definitions, examples and long documents, then the per-request input. A timestamp at the top invalidates everything after it.

When a prompt fails

Debug a prompt like code: read the fully rendered prompt (many bugs live in the template, not the wording), keep a small set of hard inputs, change one thing at a time, and study failures rather than averages. Lower the temperature for extraction (Chamber 4), and remember Chamber 5: one good sample proves little. Chamber 7 turns this loop into versioned prompts and evals with error bars.

A prompt is not a security boundary. Any text the model reads, such as a retrieved web page or an uploaded file, can carry instructions the model may follow: prompt injection. Delimiters make accidents less likely, not attacks impossible. Chamber 8 covers the defences: least privilege for tools, keeping trusted and untrusted text apart, and checking actions outside the model.

Read beyond

Read the equation in context

Willard and Louf treat guided generation as ordinary sampling with a Boolean mask that restricts the support of the next-token distribution. The mask depends on the whole prefix through the state of a finite automaton built from a regular expression, and they extend the idea to context-free grammars with an LALR(1) parser. Their contribution is speed: an index from automaton states to vocabulary tokens turns each step's mask into a lookup instead of a scan over the vocabulary. Their library, Outlines, implements it, and other open-source engines and grammar libraries apply the same masked step.

Read Figure 1 for standard against chain-of-thought exemplars, then Section 3.2: the gains appear only at scale and grow with problem difficulty. Then ask what a mask does to a chain of thought: if the schema puts the answer field first, the model must answer before it reasons.

The talk uses one provider's models, but its habits (structure, context, examples, prefilled output, iterating on real failures) carry over to any model.

Your turn

The toy model below wants to chat, use single quotes, invent a "mixed" label and add pleasantries. With the mask on, produce all three valid labels and find the step where the mask cuts the most probability; then run 200 samples each way and compare with the exact rate.

Coding problems