# Behavioural Regression Report — Sample

**This is an illustrative example.** The system, versions and figures are invented. The
situation is not: a provider updates a model, nothing in your code changes, and the behaviour
does.

---

## Behavioural regression — model version change

| | |
|---|---|
| System | Document classification and extraction, 40,000 documents a month |
| Change | Provider model version updated. No change to our code, prompt or data. |
| Notice given | 30 days, with a deprecation date for the previous version |
| Suite | 600 held-out documents, plus 240 case-based behavioural tests |
| Run | Both versions, same day, same inputs |

## 1. Should we take the update?

**Yes, with two changes first, and a two-week overlap.**

The new version is better on the measures that matter and worse on two that break downstream
systems. Both are fixable in a day. The overlap exists because two behaviours changed in ways
the suite could not have anticipated, and there will be others it did not find.

## 2. Aggregate

| | Previous | New | Change |
|---|---|---|---|
| Document type correct | 94.3% | 95.8% | +1.5 |
| All 6 fields correct | 86.9% | 89.4% | +2.5 |
| Refused or empty | 0.8% | 0.5% | −0.3 |
| Median latency | 1.4 s | 1.1 s | −0.3 s |
| Cost per document | $0.049 | $0.041 | −16% |

Better on every aggregate measure. If this were the whole report, the decision would be
trivial — which is exactly why aggregates are not the whole report.

## 3. What moved, case by case

Of 600 documents, **71 changed outcome**. Aggregates hid most of this: 44 improved and 27
regressed, netting to +17.

| | Count |
|---|---|
| Was wrong, now correct | 44 |
| Was correct, now wrong | 27 |
| Both wrong, differently | 9 |

The 27 regressions are the report. Grouped:

| Regression | Count | Note |
|---|---|---|
| Handwritten annotations now ignored | 11 | Previously extracted, now skipped |
| Two document types merged | 9 | Similar layouts, distinguished before |
| Multi-page documents truncated at page 4 | 5 | Only affects documents over 4 pages |
| No pattern found | 2 | — |

**The handwritten case is the one to note.** Nobody knew the previous version handled
handwritten annotations, because nobody had tested for it. It was working by accident, a
downstream process had come to depend on it, and the dependency became visible only when it
stopped.

## 4. The improvements that broke things

Two behaviours improved and broke downstream systems in doing so. This is the category teams
do not anticipate.

**Dates are now normalised to ISO format.** Previously returned as written. The downstream
system parses the previous formats and fails on ISO. Strictly an improvement; entirely a
breaking change.

**Currency values now include the symbol.** Previously bare numbers. The receiving field is
numeric and rejects the string.

Both were found by the behavioural suite because it asserts on output *shape*, not only on
correctness. An accuracy-only suite would have scored both as improvements and passed them
straight into a production failure.

## 5. What stayed the same

Worth checking, and worth reporting.

- Refusal behaviour on documents outside the 14 types is unchanged
- No new failure mode on poor-quality scans
- Prompt injection attempts in document text still fail, all 40 cases
- Field-level confidence remains uncalibrated in both versions — high confidence does not
  predict correctness in either

The last one is a standing finding rather than a regression. It has been true for three
versions and it is why the empty-field routing exists.

## 6. Cost of the two changes

| | Effort |
|---|---|
| Accept ISO dates downstream | Half a day |
| Strip the currency symbol before the numeric field | Two hours |
| Add handwritten annotation cases to the suite | Two hours |
| **Total** | **Under a day** |

Against a 16% reduction in cost per document, about $330 a month at current volume, plus the
accuracy improvement.

## 7. Recommendation

1. **Make the two downstream changes** before switching.
2. **Add the 11 handwritten cases to the suite** so the behaviour is asserted rather than
   assumed, whichever version is running.
3. **Run both versions in parallel for two weeks**, comparing on live traffic, before
   retiring the old one. The suite found 71 differences; production will find more.
4. **Accept the truncation regression for now.** Documents over 4 pages are 1.2% of volume and
   they already route to a person.
5. **Diarise the deprecation date.** The overlap is only available until then, and it is the
   single most useful thing the provider gave us.

---

## Notes on using this format

**Aggregates hide the report.** A net +17 concealed 27 regressions and two breaking
improvements. Compare case by case or you are not testing, you are scoring.

**Assert on shape, not only on correctness.** Both breaking changes were improvements in
accuracy. A suite that only measures whether the answer is right would have passed them.

**Some behaviour works by accident.** The handwritten annotations were never specified, never
tested, and depended on. A version change is how you find these, and the only remedy is to
add each one to the suite as it is discovered.

**Use the overlap.** A parallel run on live traffic is the only test performed on the real
distribution, and it is available only until the deprecation date.