Regression Testing for AI: Catching What a Change Quietly Broke
How to detect degradation in systems whose output legitimately varies — why aggregate scores hide regressions, what to compare, and the habit that makes a suite worth having.
Someone improves a prompt for one case, ships it, and silently degrades six others nobody re-checked. There is no red build. There is no failing assertion. The first signal is a complaint, weeks later, about behaviour that changed on a Tuesday.
Regression testing exists to make that impossible. For AI systems it needs a different design from conventional regression testing, because "the output changed" is not by itself a failure.
What counts as a regression here#
Four things, and only the first is obvious:
Correctness dropped. A case that passed now fails.
Consistency dropped. A case that passed 10 times in 10 now passes 7. Same nominal result, materially different system — and invisible if you only run once.
Cost rose at flat quality. The prompt grew 40%, the score did not move. That is a regression in cost, and prompts grow by accretion unless someone measures.
Refusal behaviour changed. It now answers something it used to decline. This is the most dangerous category and the least monitored.
Why aggregate scores hide regressions#
A suite reporting "84% overall, was 84%" looks stable. Underneath, three cases may have broken while three others improved.
Report per-case deltas, always. The aggregate is for the dashboard; the per-case diff is what tells you what happened. Any suite that only produces one number will eventually let a regression through while looking healthy.
The suite that earns its keep#
Golden set — cases that must always pass. Small, stable, high-consequence. If one breaks, the build stops.
Regression set — every production failure, permanently. This is the section that compounds: after a year it is precisely shaped to where your system actually breaks, and no generic benchmark comes close.
Exploratory set — newer cases, allowed to fail while you investigate. Promoted to the golden set once stable.
Keep them separate. Mixing them means either the build breaks constantly and gets ignored, or real failures hide among known ones.
What triggers a run#
| Trigger | Why |
|---|---|
| Prompt change | Obvious |
| Model version change | The most common cause of silent degradation. Providers update models |
| Retrieval or chunking change | Alters what the model sees |
| Tool definition change | Alters what it can do |
| Document corpus change | The system changed even though the code did not |
| Scheduled, regardless | Because of the row above |
That last row catches teams out. In a RAG system the content is part of the system, and it changes without any deployment.
Pinning#
You cannot detect regression against a moving baseline.
Pin the model version. "Latest" means your system changes without a deploy and you cannot attribute the change.
Pin the temperature and every sampling parameter for the test run.
Version the prompt alongside the code, in the repository.
Snapshot the corpus for the golden set, or you will chase content changes as though they were code changes.
Reading a result#
When the suite goes red:
- Which cases? Per-case diff, not the aggregate.
- Correctness or consistency? A case at 10/10 → 6/10 is a different problem from 10/10 → 0/10.
- What changed? Prompt, model, corpus, tools — check in that order.
- Is the old behaviour actually correct? Sometimes the "regression" is the fix, and the test case was wrong. Update it deliberately, with a note.
That fourth step matters. A suite nobody is allowed to update becomes a suite people bypass.
The habit that matters more than the tooling#
Every production failure becomes a permanent test case, the same day.
Not "when we get time". The same day, while the reproduction is understood. This single practice does more for long-run quality than any evaluator design, any framework, or any model upgrade — and it costs ten minutes.
Keeping it fast#
A regression suite that takes an hour gets skipped under deadline pressure, which is exactly when it is needed.
- Run the golden set on every change; the full suite nightly
- Parallelise — these are independent API calls
- Cache results for unchanged prompt+model+input combinations
- Prune ruthlessly: a case that has never failed and never will is costing you money on every run
FAQ#
How big should the golden set be?#
Small enough to run in minutes. Twenty to fifty high-consequence cases. The regression set can be much larger and run less often.
What if a case fails intermittently?#
That is information, not noise. Run it more times and record the pass rate. An intermittently failing case usually means the prompt is underspecified — the model is guessing on something you did not pin down.
Should we test at temperature 0?#
Test at both. Temperature 0 gives a stable signal for detecting change. Production temperature tells you what users actually experience. A system that passes only at 0 is not the system you shipped.
How do we regression-test something with no clear right answer?#
Score against a rubric and track the score over time. You are detecting movement, which does not require an absolute ground truth — only a consistent scorer.
Our suite is green but users complain. Why?#
Your cases do not resemble real usage. Build the regression set from actual failures rather than imagined ones; the gap between the two is exactly the gap between green builds and complaints.
What else is coming for Regression Testing
Theory Not yet
What it is and why it is hard.
Best Practices Not yet
What holds up in production.
Examples Ready
Worked, with real numbers.
Checklists Ready
Run before you ship.
Templates Not yet
Editable starting files.
Sample Reports Ready
What the output should look like.
Tools Not yet
What to use, and what to avoid.