Prompt Version Control and Evals

A teammate edits one sentence of the support bot's prompt, tries it on a single customer question, and the answer looks better. Two questions decide whether that edit should reach production: exactly what changed, and did it help? The first is version control. The second is an experiment, and Evan Miller's statistical guide to evals gives it an error bar: run both versions on the same items and take the standard error of the per-item differences.

Prompts are code

A one-word change to a prompt can shift behaviour as much as a code change, so give prompts the same discipline as code.

• Files, not literals. Keep each prompt in its own file in the repository, not as strings scattered through the app.
• Templates with typed variables. Render with Jinja or Python format strings, and validate the inputs first. A missing policy should raise an error (Jinja's StrictUndefined does this), not render as an empty string.
• Reviewed diffs. Changes go through pull requests, where reviewers read a line-by-line diff of the prompt text, not a blob of escaped JSON.
• Tested in CI. A change under prompts/ runs the eval suite, just as a code change runs the unit tests.

# prompts/support_answer.prompt
---
id: support-answer
version: 1.4.0
model: provider-model-2026-05-14        # a dated snapshot, never "latest"
params: {temperature: 0.2, top_p: 0.95, max_tokens: 400, seed: 7}
inputs: {question: str, policy: str}
tools: [tools/lookup_order.json]
retrieval: {index: policies-2026-09, embedding: embed-2026-03, k: 5}
output_schema: schemas/support_answer.v1.json
changelog: "1.4.0 (MINOR): quote the policy section relied on"
---
Answer {{ question }} using only the policy below.
Quote the section you relied on, e.g. (§4.2).
Policy: {{ policy }}

A release is everything that shapes the output

The text is only one part. A release pins every input that changes what the model sees or how it samples:

Fingerprint the release by hashing a canonical serialisation of all of it, and log the fingerprint with every call, next to the inputs and outputs. Any output in your logs can then be traced to exactly what produced it. Two calls with the same fingerprint can still differ, but only through sampling randomness (the Non-Determinism chamber), never through a silent configuration change. A fingerprint is only as good as its parts, though: the string latest hashes the same before and after the provider moves it, which is why the release names a dated snapshot.

import hashlib, json

def fingerprint(release: dict) -> str:
    # Sorted keys and fixed separators: equal releases always hash equally.
    canonical = json.dumps(release, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(canonical.encode("utf-8")).hexdigest()

release = {
    "template": open("prompts/support_answer.prompt").read(),
    "model": "provider-model-2026-05-14",
    "params": {"temperature": 0.2, "top_p": 0.95, "max_tokens": 400, "seed": 7},
    "tools": [json.load(open("tools/lookup_order.json"))],
    "retrieval": {"index": "policies-2026-09", "embedding": "embed-2026-03", "k": 5},
}
fp = fingerprint(release)   # log fp[:12] with every call

The version number is a name for people; the fingerprint is the content. Semantic Versioning's build metadata can carry both, as in 1.4.0+fp.3c9a71d2 (an illustrative hash), because build metadata is ignored when versions are ordered.

Versions, labels and rollout

Semantic Versioning fits prompts once you treat the output contract as the public API:

• Versions are immutable. Once released, 1.4.0 never changes; any edit is a new version. Precedence compares MAJOR, MINOR and PATCH numerically from the left, so 1.10.0 follows 1.9.3, and a pre-release such as 2.0.0-rc.1 comes before 2.0.0.
• Labels move. prod, staging and canary point at versions, like container image tags. If the app resolves its label at runtime, with a cached fallback, rolling back means pointing prod at the previous version: no redeploy.
• Every version gets a changelog entry: what changed, why, and the eval result that justified it.
• Roll out gradually. Point canary at the new version for a small slice of traffic, or run an A/B test, and watch online metrics before promoting.
• Snapshots retire. Providers deprecate model snapshots on published schedules. Moving to a successor is a release like any other: rerun the full suite before any label moves, and expect some prompts to need retuning.

A PATCH that "changes nothing" is a claim, and claims get tested. The lab's third release is one.

Git or a prompt registry?

Prompt-management tools such as Langfuse, MLflow Prompt Registry, PromptLayer and Braintrust (among others) store versioned prompts with movable labels or aliases, an editing interface and an API for fetching prompts at runtime.

Common hybrids keep Git as the source of truth and let CI publish to the registry, or keep the registry as the source and have CI run the evals before a label may move. Whichever you choose, the rule is the same: no text reaches prod without passing the same eval gate.

Eval suites

An eval suite is a versioned dataset, a set of graders and a decision rule.

The golden dataset comes from real traces: sample production logs, label them, add every bug you fix as a regression item, and cover the slices that matter (languages, long inputs, adversarial users). Version the dataset too, and record which version produced each score.

Graders, cheapest first:

An LLM judge works pointwise (grade one output against a rubric, best as a binary pass or fail per criterion) or pairwise (which of two outputs is better). Pairwise suits a head-to-head between two versions; pointwise gives an absolute pass rate that scales to many versions. Zheng et al. found that strong judges agreed with human preferences over 80% of the time, about as often as humans agreed with each other, but also measured position bias (favouring the first answer), verbosity bias (favouring longer answers) and possible self-enhancement (favouring answers from the judge's own model). Later work (Panickssery, Bowman and Feng, 2024) found that models do recognise and favour their own generations. Swap the order and count only consistent wins, use a judge from a different model family, and calibrate: label a sample by hand and measure the judge's agreement on passes and on failures separately. Shankar et al. observed criteria drift: grading outputs changes what people think the criteria should be, so calibration is never finished. The judge prompt is a prompt too, so version it.

Gate merges in CI: run the suite on every pull request that touches prompts, post the comparison table, and block the merge if the paired interval shows a regression on a critical slice. After release, watch online metrics (user feedback, escalations, task completion, cost and latency), because offline evals only cover the inputs you thought of.

Error bars on a pass rate

Treat the $n$ items as a sample from every input you care about. With pass/fail scores, the pass rate $\hat p$ has standard error $\sqrt{\hat p(1-\hat p)/n}$ (Miller's Equation 2) and a 95% interval $\hat p\pm1.96\,\mathrm{SE}$. Since $p(1-p)\le1/4$, the standard error is at most $1/(2\sqrt n)$, so the 95% half-width is at most $0.98/\sqrt n$: $\pm9.8$ points for 100 items, $\pm4.9$ for 400 and $\pm3.1$ for 1,000. Most prompt suites are small, so most differences you see on them are noise.

Non-determinism adds a second layer: one item can pass on one sample and fail on the next. Score each item as the mean of $K$ samples. If $x_i$ is item $i$'s true pass probability and $\sigma_i^2$ the variance of one sampled answer's score,

Resampling shrinks only the second term; the spread of item difficulty is a floor that only more items lower. Miller also warns against lowering the temperature to reduce variance: evaluate at the settings you ship.

Compare versions on the same items

Both versions answer the same items, and an item that is hard for one is usually hard for the other. Compare item by item, with $d_i=s_{\text{new},i}-s_{\text{old},i}$:

The unpaired standard error $\sqrt{\mathrm{SE}_{\text{new}}^2+\mathrm{SE}_{\text{old}}^2}$ ignores the covariance. When the versions agree on which items are hard, $\operatorname{Cov}>0$ and pairing gives a narrower interval at no extra cost; you only need to keep the item ids aligned.

For pass/fail scores, McNemar's test looks only at the discordant items: $b$ items only the old version passes, and $c$ items only the new one passes. If neither version is better, each discordant item is equally likely to fall either way, so compare $(b-c)^2/(b+c)$ with 3.84, the 95% point of $\chi^2_1$. When $b+c$ is small, use the exact binomial test instead.

Plan the size before you run: for an interval of half-width $h$ you need $n\approx(1.96\,\sigma_d/h)^2$ items, with $\sigma_d$ from a pilot run. To detect a difference $\delta$ with given power, Miller's sample-size formula adds a second quantile (the decoder below).

Don't overfit the eval set. If you try 40 variants and keep the best, its dev score is biased upward by the selection alone (Problem 3 computes how much). Keep a held-out set that is used only for the release decision, and refresh the golden set from new traces. And don't peek: stopping the first time the interval clears zero raises the false-positive rate well above 5%. In CI, fix $n$ in advance, or use a sequential test designed for repeated looks.

import numpy as np

def paired_ci(new, old, z=1.96):
    d = np.asarray(new, float) - np.asarray(old, float)   # same items, same order
    se = d.std(ddof=1) / np.sqrt(len(d))
    return d.mean() - z * se, d.mean() + z * se

def decide(lo, hi, margin=0.03):
    if lo > 0: return "ship"
    if hi < 0: return "roll back"
    if -margin < lo and hi < margin: return "hold"
    return "keep evaluating"

Read beyond

Read the equation in context

Miller treats eval questions as draws from an unseen super-population, so a score is an estimate of a skill and deserves a standard error. The paper derives the standard error of a mean score, clusters it when questions come in related groups (several questions about one passage, or one question translated into many languages; in a product suite, several items cut from one conversation), reduces variance by resampling answers, compares two models with paired differences (the opening equation, Equation 7), and closes with the sample-size formula decoded here.

The study that measured judge agreement with human preferences, catalogued position and verbosity bias, and proposed swapping positions as a fix.

A tool and user study for aligning LLM graders with human grades, and the origin of the term criteria drift.

The talk surveys prompt engineering, retrieval and fine-tuning. For each step, ask how you would version it and which eval would show that it helped.

Your turn

Three releases wait on the canary label. Run the golden set in batches, watch the paired interval tighten inside the unpaired one, and make each call only when the interval backs it. The typo fix is the hardest: a hold needs a narrow interval, and there is more than one way to narrow it.

  Answer {{ question }} using only the policy below.
+ Quote the section you relied on, e.g. (§4.2).
+ If the policy does not cover it, say so and offer a human.
  Policy: {{ policy }}

Coding problems