Non-Determinism: Taming and Using Randomness

Send a served model the same prompt twice at temperature zero and you can get two different answers. In 2025, Horace He and colleagues at Thinking Machines Lab asked Qwen3-235B for 1,000 temperature-zero completions of "Tell me about Richard Feynman" and got 80 different texts. All of them matched for 102 tokens. At token 103, after "Feynman was born on May 11, 1918, in", 992 went on with "Queens, New York" and 8 with "New York City". Nobody rolled a die. This chamber finds where that variation comes from, how to remove it when you need a run you can repeat, and how to put randomness to work when you want it. The most useful trick of the second half fits in one line: sample many answers and keep the most common.

Seven sources of variation

The first row is the only one temperature controls. At temperature above zero a sampler can draw $r\sim\mathrm U(0,1)$ and return the first token whose cumulative probability exceeds $r$, so a pseudo-random generator with a fixed seed reproduces the draws. Engines that batch many users must give each request its own reproducible stream rather than one shared generator: SGLang's deterministic mode, for example, draws its sampling noise from a seeded hash function, so that "the same (inputs, seed) pair always yields the same sample". Hosted APIs that accept a seed describe it as best effort, some report a fingerprint of the serving configuration so you can see when the backend changed, and some offer no seed at all. And a seed fixes only $r$: if the arithmetic upstream moves a cumulative probability across it, the token changes anyway.

Floating-point addition is not associative

A float32 number carries a 24-bit significand, so each rounding has relative error at most $u=2^{-24}$. Between $2^{26}$ and $2^{27}$ consecutive float32 numbers are $2^{3}=8$ apart. Write $a\oplus b=\mathrm{fl}(a+b)$ for computed addition.

Claim. In float32, $(1\oplus10^8)\oplus(-10^8)=0$ but $1\oplus(10^8\oplus(-10^8))=1$.

Proof. $10^8=8\times12{,}500{,}000$ lies between $2^{26}\approx6.7\times10^7$ and $2^{27}\approx1.3\times10^8$, so it is representable. The exact sum $10^8+1$ is 1 from $10^8$ and 7 from the next float, $10^8+8$, so it rounds to $10^8$, and subtracting $10^8$ leaves exactly 0. On the right, $10^8\oplus(-10^8)=0$ exactly, and $1\oplus0=1$. The two groupings differ, so $\oplus$ is not associative.

A dot product of length $K$ is a long sum. Added left to right, the computed result obeys $|\hat s-s|\lesssim(K-1)\,u\sum_i|x_i|$; a tree of pairwise sums improves the factor to about $u\log_2K$. Every order gives a slightly different answer within these bounds, and usually nobody cares. Greedy decoding cares when the top two logits are closer than the rounding error: then the order of additions picks the token, and since every later token is conditioned on it, the texts part ways from there.

Why temperature zero still varies

The folk explanation runs: GPUs add in parallel, threads finish in unpredictable orders, floating-point addition is not associative, so results wobble. He and Thinking Machines Lab call this incomplete. Run the same matrix multiplication on the same data and you get bitwise identical results, run after run. An order truly decided by racing threads needs atomic adds, and a typical LLM forward pass usually contains none. Each forward pass is run-to-run deterministic.

What varies is the batch. A server packs concurrent requests into one forward pass, and a fast kernel chooses its strategy from the shapes it receives. With few rows a matrix multiplication has too few output tiles to keep every core busy, so it splits the long reduction across cores and adds the partial sums afterwards (split-K); with many rows it doesn't. A different split is a different order of additions, so the numbers computed for your row depend on how many strangers arrived with it. In the post's words, "from the perspective of an individual user, the other concurrent users are not an 'input' to the system but rather a nondeterministic property". CPU and TPU servers have the same problem.

Batch invariance. A kernel $f$ is batch-invariant if the row it computes for input $x$ is bit-for-bit the same in every batch that contains $x$. The post states the requirement plainly: "the reduction order for each element must be fixed regardless of the batch-size of the kernel."

Three operations in a transformer reduce over many numbers: RMSNorm, matrix multiplication and attention. The post makes each one batch-invariant. Normalisation and matrix multiplication use one kernel configuration for every shape, giving up split-K. Attention must also add a token's keys in the same order whether they arrive in one prefill or one at a time from the KV cache (as with chunked prefill and prefix caching), so the cache is laid out consistently before the kernel runs, and where the keys are split across cores, the size of each split is fixed rather than the number of splits. With these kernels all 1,000 Feynman completions were identical. The price was speed: on one GPU running Qwen3-8B, 1,000 sequences took 26 s with vLLM's default kernels, 55 s with an unoptimised deterministic version and 42 s with an improved attention kernel.

Mixture-of-experts models add a discrete choice: a router sends each token to its top-scoring experts, so a rounding-level change in router scores can redirect a token, and where each expert has a capacity limit, other requests' tokens compete for its slots. The Feynman model was itself a mixture of experts, and batch-invariant kernels were enough to make it repeat itself.

Making runs reproducible

Temperature zero is not a guarantee: it removes the sampler's dice, not the arithmetic or the inputs. To repeat a result, whether to debug, to audit or to compare two prompts fairly, control what you can and record the rest:

1. Pin the model. Request a dated snapshot, never an alias the provider can move to new weights.
2. Pin every parameter. Temperature, top-p, maximum tokens, stop sequences, system prompt, tool schemas and response format. Defaults change too.
3. Seed where supported, knowing that hosted APIs promise only best effort.
4. Log everything. The full request and response, the model version string, any system fingerprint, timestamps, and the retrieved context and tool results that went into the prompt.
5. Cache by a hash of the request, so identical requests return the stored response.
6. Record and replay in tests. Unit tests replay recorded responses; a separate, scheduled suite calls the live model.
7. Self-host for bitwise reproducibility. Some open-source inference engines, vLLM and SGLang among them, now offer a batch-invariant or deterministic mode, at a cost in throughput and with limits on supported hardware and models. Check the current documentation before you rely on it.

import hashlib, json

def request_key(request: dict) -> str:
    # Sorted keys and fixed separators make the JSON canonical: same request, same key.
    canonical = json.dumps(request, sort_keys=True, separators=(",", ":"), ensure_ascii=False)
    return hashlib.sha256(canonical.encode("utf-8")).hexdigest()

def cached_call(call, request: dict, store: dict) -> dict:
    key = request_key(request)        # model snapshot, every parameter, full messages, tools
    if key not in store:
        store[key] = call(**request)  # record on a miss
    return store[key]                 # replay on a hit

Variety on purpose

Sometimes the same answer twice is the failure. Brainstorming, creative writing, test-case generation and synthetic training data all want spread.

Measure what you get. Distinct-n is the number of unique $n$-grams across all samples divided by the total number of $n$-grams (some papers divide by the total number of tokens instead); mean pairwise cosine distance between embeddings catches paraphrases that $n$-grams miss. For synthetic data, check for near-duplicates before you train on it.

Many samples, one better answer

When the task has a checkable final answer, variety buys accuracy. Self-consistency (Wang et al., 2023) samples $m$ chain-of-thought reasoning paths, parses each final answer $a_i$, and returns the most frequent one: the opening equation. Its intuition is that correct reasoning paths, however different, tend to agree on the final answer more often than incorrect ones. With PaLM-540B and 40 sampled paths at temperature 0.7, it raised GSM8K accuracy from 56.5% with greedy chain of thought to 74.4%.

A clean model explains when voting helps. If each sample is right with probability $p$, independently, and answers are binary, the vote over an odd number $n$ of samples is right with probability

$P_{\text{vote}}(n)=\sum_{k>n/2}\binom nk p^k(1-p)^{n-k},$

which rises towards 1 when $p>\tfrac12$ and falls towards 0 when $p<\tfrac12$: voting amplifies whatever the model usually says. All $n$ samples agree with probability $p^n+(1-p)^n$, so low agreement flags a question worth routing to a stronger model or a person. Real samples share blind spots, so gains flatten sooner than the formula suggests; when wrong answers scatter over many values, plurality voting can help even with $p<\tfrac12$ (Problem 3).

Best-of-n with a verifier samples $n$ answers and keeps the one a verifier scores highest: unit tests, a schema check, a calculator or a reward model. It works where voting can't, as with code and prose. pass@k asks whether at least one of $k$ samples passes. Chen et al. (2021) estimate it from $n\ge k$ samples of which $c$ pass, as $1-\binom{n-c}{k}\big/\binom nk$; the plug-in $1-(1-c/n)^k$ is biased low, and the proof puzzle below shows why the first is unbiased.

Testing a system that never answers the same way twice

A test that compares the reply with a stored string fails on the first harmless rephrasing. Assert properties instead: the reply parses and validates against its schema; required fields or facts are present and forbidden ones absent; numbers, lengths and labels fall in their allowed ranges; for free text, similarity to a reference or a model grader's verdict clears a threshold.

import json

def acceptable(reply: str) -> bool:
    try:
        data = json.loads(reply)
    except json.JSONDecodeError:
        return False
    return data.get("label") in {"refund", "exchange", "other"} and 0 <= data.get("confidence", -1) <= 1

def trials(generate, prompt: str, n: int = 30) -> tuple[int, int]:
    return sum(acceptable(generate(prompt)) for _ in range(n)), n   # passes, trials

Then treat the pass rate as an estimate. With $x$ passes in $n$ trials and $\hat p=x/n$, the Wilson interval

$\frac{\hat p+\frac{z^2}{2n}}{1+\frac{z^2}{n}}\pm\frac{z}{1+\frac{z^2}{n}}\sqrt{\frac{\hat p(1-\hat p)}{n}+\frac{z^2}{4n^2}}$

stays honest near 0 and 1, where the textbook $\hat p\pm z\sqrt{\hat p(1-\hat p)/n}$ shrinks to a point. Eighteen passes in twenty give $\hat p=0.9$ but a 95% interval of about $[0.70,0.97]$. Set thresholds before you look: ship only if the lower end clears, say, 0.9; call a test flaky when it both passes and fails at fixed inputs; and quarantine flaky tests instead of rerunning until green, which measures only your patience. Chamber 7 builds this into full eval suites, with graders and paired comparisons between prompt versions.

Read beyond

Read the equation in context

Section 2 defines the vote. Each sampled output is a pair $(r_i,a_i)$: a reasoning path $r_i$ and the final answer $a_i$ parsed from it, and voting over the $a_i$ marginalises out the paths. The paper also weights each vote by the model's probability of the whole output: normalised by length, that scored about the same as the plain vote on PaLM-540B, while the unnormalised version did clearly worse (Table 1).

The Codex paper introduced HumanEval and made "try $k$ times" a metric. It draws $n=200$ samples per problem, counts the $c$ that pass the unit tests, and estimates pass@k for every $k\le100$ from the same samples.

Your turn

The lab builds the Feynman fork in miniature: two candidate tokens whose float32 logits differ by a tenth of a float32 step in exact arithmetic. Find a batch size that flips greedy decoding, then switch on the batch-invariant kernel and show the flip is gone. The second panel turns the voting formulas into curves.

Coding problems