The Eval Gate — Regression Testing for LLM Prompts

Each number below comes from a real run on synthetic data: tickets written from invented product asks, and answers researched from a fictional document collection. Nothing here is a projection unless it says so.

1. A PR that changes a prompt, model or parameters runs both versions, the base branch’s and the PR’s, on every case, several times each. A PR that changes nothing the model sees skips the paid run.
2. Each case gets a pass rate per version, and the gate takes the paired difference across cases with a bootstrap 95% interval. Pairing cancels out how hard each case is.
3. PROMOTE if the whole interval sits above minus the tolerance (5 points); ROLLBACK if it sits below, if any single check regresses on its own, or if a critical check newly fails on every sample and reruns confirm it; HOLD when the run can’t tell.
4. The result lands as one comment per suite on the PR, edited in place on every push.

I planted regressions on purpose: prompts that stop splitting bundled requests, stop flagging gaps, drop citations, never abstain, or ignore the documents. Each one ran through the same gate as a real change.

The one miss is the instructive one. A prompt that ignores the documents looks fine in a fictional world, because the model has no memory of it to fall back on. It only shows once the documents contradict facts the model already knows. So I added known-facts cases, and on those the same regression is a clear rollback.

A gate that fires on noise trains people to ignore it. So I ran the same prompt against itself, 33 times across both suites, and looked at every difference it reported.

On the research suite the no-change difference has a spread of 1.2%, and 12 of 13 runs came back PROMOTE. My first rule for critical failures would have fired on most no-change runs in simulation, because with three samples a few cases always fail by chance. It now needs a failure on every sample and a confirming rerun, and across all 33 no-change runs it has confirmed 0.

The first run in CI put the same prompt on both arms and said HOLD: a −1.1% difference, but with three samples per case it could only detect a drop of 6.8%, more than the 5-point tolerance. Noise in a paired pass rate falls with the square root of the runs, so doubling the samples should get under it.

At six samples the next run measured 5.1%, close to the projection and just over the line. That doubled the cost of a prompt PR to about $2.05. I took it: a gate that can’t see a 5-point drop isn’t enforcing a 5-point tolerance.

Spec writing v3: strict Gherkin and shorter titles

I looked at why v2 failed before writing v3. Most broken Gherkin was a scenario with no When step, and a third was an uppercase AND, which a strict parser rejects. Titles ran over 80 characters because models count characters badly, so v3 aims for 40 to 65 instead of quoting the limit.

Research agent v3: never pick a side in a conflict

When two documents disagreed, v2 usually said so and then answered with one side anyway, mostly on comparison questions where the disputed figure decides the answer. v3 makes the rule cover facts needed along the way and checks its own output before answering.

Before both of these, v2 itself went through the gate against v1 at +28.9% on 30 cases. The interval was wide, because 30 cases is few, but the change was large enough to clear it anyway.

Keyword checks can tell a requirement is mentioned, not whether a ticket invented one. That needs an LLM judge, and a judge is a measuring instrument: it gates only once it agrees with a human on at least 50 labels with κ of at least 0.6. I labeled tickets blind to the judge’s verdicts, on my phone.

The first result was barely better than chance: the judge failed tickets for sensible elaborations like error messages and permission checks. Loosening the rule helped. Then my own rechecks left only four fails in fifty, too few to measure agreement at all, so I added tickets from a prompt built to invent requirements. That got to κ 0.37. A stricter actor rule then backfired, failing tickets for not naming a user the request never named. It’s narrowed now, and a held-out batch the prompt was never tuned on will decide whether the judge earns its place. Until then it reports and does not gate.

Nothing in the gate knows about tickets or research. It compares two versions of anything that takes an input and returns structured output: a classifier, an extractor, a support reply, an agent that calls tools. What changes from feature to feature is what you write down about it, in this order.

1. Say what a good output must do, as checks. The right category, a citation to something it actually read, under a length, never blaming the customer. Prefer checks with a known answer; they’re cheap, exact and never drift.
2. Build cases with known answers. A few dozen to a couple of hundred, spread over the ways real inputs vary: tone, format, missing information, requests bundled together. Label each with what varies, so a regression in one kind of input can’t hide in the average.
3. Version what the model sees. Model, parameters and prompt templates in one file, hashed into an id. Edit a word of the prompt and it’s a new version; change anything else and the paid run is skipped.
4. Prove the gate before you trust it. Plant a worse prompt and check it rolls back. Run the prompt against itself and check it stays quiet. Read the smallest drop it can detect, and set samples and tolerance from that.
5. Put it on every pull request. Both versions run, the comment shows the decision and the evidence, and the check fails only on a rollback. A prompt change costs about cases × samples × 2 model calls: here $0.79 for the research agent and $2.05 for the spec writer.
6. Make LLM judges earn their place. For what no check can see, add a judge with a yes/no rubric, label about 50 of its outputs blind, and let it gate only once it agrees with you.

A whole suite for a support-triage feature fits on one screen:

# suite.yaml
suite: support-triage
runner: {type: prompt, output_schema: schemas/output.schema.json}
baseline: versions/v1.yaml        # model, params and prompt templates
cases:
  - from: data/cases.jsonl        # {"input": {"message": ...}, "expected": {"category": "billing", ...}}
plugins: [plugins/checks.py]      # your own checks and judges
checks:
  - field_matches: {field: category, critical: true}
  - keywords_present: {field: reply}
  - max_length: {field: reply, max_chars: 400}
  - no_blame: {}                  # from plugins/checks.py
  - checklist: {rubric: rubrics/reply_quality.yaml}   # an LLM judge: reports until calibrated
samples: 6
gate: {tolerance: 0.05}

I’ve packaged this so it can drop into another repository: a command that scaffolds a working suite, plug-ins for your own checks and judges, an offline mode for testing the wiring for free, and a GitHub Action for the pull-request gate. It’s open source under the MIT licence at github.com/bwyndham/llm-evals. Three commands take you from nothing to a gated suite, and the first run needs no API key:

uv tool install git+https://github.com/bwyndham/llm-evals@v0.1.0
llm-evals init evals/my-feature --ci      # a starter suite and a pull-request workflow
llm-evals gate evals/my-feature --candidate versions/v2.yaml --provider mock --skip-judges

Trying it from the outside

To see what adopting it is actually like, I set up a separate repository the way someone else would: a small app that pulls package dimensions and weight from messy supplier listings, with llm-evals installed as a tool and its GitHub Action on every pull request. The eval points at the app’s own prompt file, so a PR that edits the prompt is exactly what gets gated. Its 120 listings are generated from known values, with the mess real supplier data has: mixed units and fractions, packed and assembled sizes side by side, prices and model numbers in the way, and values that are missing or contradict each other.

The first prompt only asked for centimetres and kilograms. The rewrite said what to do with each kind of mess, and to leave a value empty rather than guess. Guessing is the check I made critical: a made-up weight in a warehouse system is worse than a blank, so one confirmed guess rolls a PR back on its own.

Then I measured the noise. 10 runs of the new prompt against itself all came back PROMOTE, never more than 0.8 points apart, and the gate could see drops of 2.2 points or less. That left room under the 5% tolerance, so I tightened it to 3%: at a 98.6% baseline, a 4-point drop would mean about four times as many wrong values.

It also earned its keep on the tool itself. The first run crashed, because CI loaded the base branch’s copy of the suite’s plug-ins next to the PR’s, and the check passed anyway, because a crash and a HOLD shared an exit code. llm-evals’ own repository never hit either bug; the first outside user found both on its first run. Both are fixed, and the check now passes only on PROMOTE, HOLD or a skip. The demo repository itself is private; its results are all here.

Pick any run to see what the gate decided, every check on both versions, and the cases that flipped. Open a flipped case to read both outputs side by side. The outputs are synthetic and load only when you open one.

Precomputed from synthetic and public eval data. No live model calls. See also The Eval Stack, the design tool this pipeline puts into practice.

The design started from two papers and took its statistics from a third. I kept what held up and left what didn’t: single-sample labels, failure counts as the score, and thresholds with no baseline to compare against.

• STELLAR: A Search-Based Testing Framework for Large Language Model Applications (Sorokin et al., 2026): generating tests over a space of input features, and checking the judges themselves.
• Automated Self-Testing as a Quality Gate: Evidence-Driven Release Management for LLM Applications (Maiorano, 2026): deterministic checks first, PROMOTE / HOLD / ROLLBACK, and failures fed back into the cases.
• Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations (Miller, 2024): paired differences, uncertainty clustered by case, several samples per case, and power analysis.

Particular choices trace to particular findings:

• Ouyang et al., 2023: temperature 0 doesn’t make output repeatable, so every case runs several times.
• Chen, Zaharia & Zou, 2023: the same model name can change behaviour, so versions pin model ids.
• Zheng et al., 2023: judges favour the answer shown first, so pairwise judging runs both orders.
• Shankar et al., UIST 2024: labeling shifts the criteria, which is exactly what happened to “invented” here.
• Cook et al., 2024: a checklist of yes/no items for the rubric judge.
• Landis & Koch, 1977: κ above 0.6 as “substantial” agreement, the bar a judge must clear to gate.
• Longpre et al., EMNLP 2021: models lean on memory over the documents, hence the known-facts cases.
• Min et al., EMNLP 2023: checking a research answer claim by claim.
• Ribeiro et al., ACL 2020: metamorphic tests: reword the input, expect the same answer.
• Luo et al., FSE 2014: rerunning before a critical failure is allowed to roll a change back.

Every source, what it informed and where it lives in the code, including the ideas not built yet: docs/references.md.