Sampling: Temperature, Top-k and Top-p

A language model never writes a sentence. At each step it hands over a probability for every token in its vocabulary, and something else, the decoder, has to choose one. Always take the most likely token and a strong model can get stuck repeating "I don't know." Sample from the whole distribution and it drifts into nonsense. Holtzman and colleagues proposed a middle way: sample only from the nucleus, the few tokens that hold most of the probability.

Logits in, one token out

At each step the model produces a logit $u_l$ for each of the $|V|$ tokens (the next-token prediction chamber built them), the softmax turns logits into probabilities, and a decoding strategy turns probabilities into one token. Append it, run the model again, and repeat until something says stop.

Greedy decoding takes $\arg\max_l u_l$ every time. It is cheap and, in principle, deterministic. But it maximises one step at a time, not the whole sequence: a token that wins now can lead to a context where every continuation is mediocre. In the third coding problem below, you will measure how far greedy falls behind the most probable sequence.

Beam search, and why it degenerates

Beam search keeps the $B$ highest-scoring partial sequences. At each step it extends every beam by every token, scores each candidate by its total log-probability, and keeps the best $B$. Width $B=1$ is greedy. It costs about $B$ times the compute and is still not exact.

Total log-probability favours short outputs, since each extra token adds a negative term, so practical beam search normalises by length and retires beams that emit the end-of-sequence token. Even so, more search is not always better: with an exact search, Stahlberg and Byrne (2019) found that a translation model's single best output was the empty translation for more than half of the test sentences.

Beam search suits tasks where the input pins the output down, such as translation, summarisation and speech recognition: there, the most probable answer is usually close to the right one. On open-ended text it degenerates. Holtzman et al. show that the probability of a repeated phrase rises with each repetition, a positive feedback loop that a likelihood-seeking search walks straight into. The text it finds is also bland, because human text is not the most probable text: its per-token probabilities swing between likely and surprising.

Temperature

Temperature $T>0$ divides every logit before the softmax. Let $m$ be the index of the largest logit and multiply above and below by $e^{-u_m/T}$:

$p_i(T)=\frac{e^{u_i/T}}{\sum_je^{u_j/T}}=\frac{e^{(u_i-u_m)/T}}{\sum_je^{(u_j-u_m)/T}}.$

The second form is also how stable code computes it. Now take limits.

• $T\to0^+$. If the maximum is unique, every term with $j\ne m$ has an exponent that tends to $-\infty$, so it vanishes, while the $j=m$ term is always $e^0=1$. Hence $p_m\to1$: a one-hot vector on the argmax, which is greedy decoding. Implementations special-case $T=0$ as greedy rather than divide by zero.
• $T\to\infty$. Every exponent tends to 0, every weight to 1, and $p_i\to1/|V|$: uniform.
• Order is preserved. For any $T>0$, $p_i/p_j=e^{(u_i-u_j)/T}$ exceeds 1 exactly when $u_i>u_j$. Temperature reshapes the distribution but never re-ranks it, so on its own it cannot remove the tail.

Entropy rises steadily in between. Writing $\beta=1/T$, one can show $dH/d\beta=-\beta\operatorname{Var}_{p}(u)$, so $dH/dT=\operatorname{Var}_{p}(u)/T^3\ge0$ in nats: from 0 bits as $T\to0$ up to $\log_2|V|$ bits as $T\to\infty$. At $T=1$ you sample the model's own distribution.

Truncation: top-k, top-p and min-p

A vocabulary of 100,000 tokens has a long tail. Each tail token is unlikely, but together they hold real mass, and one bad draw can derail everything after it. Holtzman et al. call this the unreliable tail. Truncation samplers keep a set $S$ and renormalise:

$P'(x)=\frac{P(x)}{\sum_{y\in S}P(y)}\ \text{ for }x\in S,\qquad P'(x)=0\ \text{ otherwise}.$

That is exactly conditioning, $P'(x)=P(x\mid X\in S)$, so the ratios between survivors are unchanged.

Fan et al. generated stories by sampling from the $k=10$ most likely words, and found it substantially more effective than beam search. A fixed $k$ is the wrong size most of the time, though: too many candidates when the model is sure, too few when hundreds of words fit. The nucleus grows and shrinks with the distribution; the proof below shows it is always a non-empty prefix of the sorted tokens. Min-$p$ scales its cut-off by the model's confidence instead. With probabilities $0.8, 0.07, 0.03, 0.02, 0.01$, top-$p$ at 0.9 keeps three tokens, while min-$p$ at 0.1 sets the threshold at 0.08 and keeps one. Its authors argue this lets you raise the temperature for creativity while staying coherent.

The sampler, step by step

A typical sampler edits the logits, applies temperature, truncates and renormalises, then draws:

def next_token(u, history, rng, T=0.8, top_k=50, top_p=0.95, min_p=0.0,
               bias=None, freq=0.0, pres=0.0):
    u = u.copy()
    for tok, b in (bias or {}).items():          # logit bias
        u[tok] += b
    c = np.bincount(history, minlength=len(u))   # counts so far
    u -= freq * c + pres * (c > 0)               # frequency and presence penalties
    if T == 0:
        return int(np.argmax(u))                 # greedy
    p = softmax(u / T)                           # temperature
    p = renormalise(p * top_k_mask(p, top_k))    # truncate, renormalise, repeat
    p = renormalise(p * top_p_mask(p, top_p))
    p = renormalise(p * (p >= min_p * p.max()))
    return int(rng.choice(len(p), p=p))

Around it, the loop stops at the end-of-sequence token, at a stop sequence (a string such as </answer> that ends generation when it appears and is usually left out of the output), or at max tokens, a hard cap that cuts the output off mid-sentence. Many APIs report which one fired as a finish reason: check it before you trust a truncated answer.

The order is common, not universal. At the time of writing, vLLM applies penalties, then temperature, then min-$p$, then top-$k$ and top-$p$, while llama.cpp's default chain truncates first, applies temperature last, and lets you reorder the chain with --samplers. The min-$p$ authors found that two widely used libraries applied temperature at different points, which changed their high-temperature results. Compare settings across tools only after checking the order.

Penalties fight repetition by editing the logits of tokens already generated. With $c_j$ the count of token $j$ so far, frequency and presence penalties give $u_j-\alpha_{\text{freq}}c_j-\alpha_{\text{pres}}\mathbb 1[c_j>0]$: the first grows with each use, the second is a one-off charge for appearing at all. The repetition penalty from CTRL (Keskar et al., 2019) divides the logits of earlier tokens by $\theta\approx1.2$; implementations multiply negative logits instead, so the penalty always pushes down. Penalties also hit legitimate repeats: a variable name, a JSON key, the subject of a story. Logit bias adds a fixed $b_j$ to chosen tokens: a large negative bias bans a token, a positive one encourages it. It acts on tokens, not words, so check how your words tokenise.

Log-probabilities as a confidence signal

Many APIs and open-weight servers can return each output token's log-probability, often with the top alternatives. Summed, they give the log-probability of the whole output. For a one-token label they give a ready-made confidence score, and a small gap between the top two labels flags a borderline case. Treat them as a signal, not as truth. They measure the model's belief about the next token, not whether a fact is correct, and depending on the tool they are computed before or after your sampling settings reshape the distribution. Post-training also bends them: the GPT-4 technical report shows a pre-trained model whose answer probabilities were well calibrated on a multiple-choice benchmark, and a post-trained model that was markedly less so. Before you threshold on log-probabilities, bin outputs by confidence, compare with accuracy on labelled data, and recalibrate if needed.

Speculative decoding

Decoding is slow because every token needs a forward pass of a large model, one after another. Speculative decoding (Leviathan, Kalman and Matias, ICML 2023) lets a small draft model $q$ propose $\gamma$ tokens, then runs the target model $p$ once over all of them in parallel. Each draft token $x$ is accepted with probability $\min(1,p(x)/q(x))$; at the first rejection, a replacement is drawn from the normalised $\max(0,p-q)$. This rejection scheme makes every token distributed exactly as a sample from $p$, so quality is untouched and only the speed depends on how often the draft agrees. The paper reports 2–3× faster decoding of T5-XXL with identical outputs.

Settings by task

These are illustrative starting points, not laws: tune them on your own evals. Two cautions. Some reasoning models fix their sampling settings or ignore them: depending on the provider, a custom temperature or top-$p$ is rejected, silently ignored, or restricted while extended reasoning is on, and open-weight reasoning models usually publish recommended settings in their model cards. Read the documentation for the exact model you call. And temperature 0 is not a reproducibility guarantee: why identical requests can still differ, and what to do about it, is the next chamber.

Read beyond

Read the equation in context

Section 3 sets out every decoder the paper compares. Eq. (2) defines the top-$p$ vocabulary as the smallest set reaching mass $p$. Eq. (3) divides by $p'=\sum_{x\in V^{(p)}}P(x\mid x_{1:i-1})$ inside it, the conditioning derived above, and Section 3.2 reuses Eq. (3) for top-$k$. Section 3.3 writes temperature as a rescaled softmax, Eq. (4), noting that $t\in[0,1)$ lowers the mass in the tail at a cost in diversity. Decode Eq. (4), then read Figure 2 (per-token probabilities of beam search and of human text) and Figure 4 (the repetition loop) in this chamber's terms.

Your turn

Ten candidate tokens, fixed logits, four knobs. Shape the distribution three ways, sample 20 continuations to hear what each setting sounds like, and see how far you can push the temperature.

Coding problems