# Retrieval Regression Report — Sample

**This is an illustrative example.** The system, migration and figures are invented. The
scenario is common: the knowledge base changes, no code changes, and retrieval quietly moves.

---

## Retrieval regression — knowledge base migration

| | |
|---|---|
| System | Internal technical support assistant |
| Change | Content migrated to a new documentation platform. Same material, re-exported. |
| Our code | Unchanged. Same chunker, same embedding model, same query path. |
| Golden set | 220 queries, each with the document that must be retrieved, labelled by hand |
| Run | Before migration and after, same queries, same day |

## 1. Result

**Do not cut over.** Aggregate recall is flat. Twenty-six per cent of the golden set changed
outcome, and the losses are concentrated in the queries that matter most.

## 2. The aggregate that hid everything

| | Before | After |
|---|---|---|
| Correct document in top 10 | 88.2% | 87.7% |
| Correct document in top 3 | 79.5% | 78.6% |
| Correct document ranked first | 64.1% | 63.2% |

Half a point down across the board. On these three rows the migration is a non-event, and
this is what would have been reported had the golden set been scored only in aggregate.

## 3. Query by query

| | Queries |
|---|---|
| Improved | 28 |
| Regressed | 30 |
| Unchanged | 162 |

**Fifty-eight queries — 26% of the set — moved.** They net to −2, which is how a quarter of a
system changing behaviour presents as half a point.

The 30 regressions group into three causes, and none of them is subtle once seen.

| Cause | Queries | What the migration did |
|---|---|---|
| Heading hierarchy flattened | 14 | H2/H3 exported as bold paragraphs, so chunking lost structure |
| Tables exported as pipe-delimited text | 9 | Column headers no longer attached to values |
| Code blocks merged with prose | 5 | Chunk boundaries fall mid-example |
| No pattern found | 2 | — |

All three are export format, not content. The material is identical. **The structure our
chunker depends on was not preserved, and nothing in the migration plan mentioned structure
because nothing had made it visible as a dependency.**

## 4. Why the regressions matter more than the improvements

The 30 regressions include 11 of the 40 highest-frequency queries in the set. The 28
improvements include 2.

Weighted by how often each query is actually asked, the picture inverts:

| | Before | After |
|---|---|---|
| Unweighted recall at 3 | 79.5% | 78.6% |
| **Frequency-weighted recall at 3** | **83.1%** | **74.8%** |

**An 8.3-point fall on the queries people actually ask.** The unweighted number is the one
that looked flat.

The 14 heading-hierarchy failures are concentrated in the configuration and troubleshooting
sections, which are the most-read pages in the library. That is not coincidence — heavily
structured documents are both the most useful and the most damaged by a flattening export.

## 5. What did not change

- **Embedding model and query path** — deliberately held constant, which is what makes this a
  clean comparison
- **Latency** unchanged
- **Queries with no correct answer in the library** still correctly return nothing useful,
  25 of 25
- **Duplicate detection** unaffected

Holding everything else constant is the reason a cause could be identified at all. Had the
platform migration coincided with a chunker change, this report could have described the
effect and not the reason.

## 6. What is needed before cutover

1. **Export headings as headings.** The platform supports it; the migration used a default
   template that did not.
2. **Export tables as tables**, or as structured data the chunker can keep intact.
3. **Keep code blocks as single units** with the surrounding explanation.
4. **Re-run this golden set** and require frequency-weighted recall at 3 to be within one point
   of the current system.

Estimated: three days of export configuration. The alternative — accepting the migration and
recovering with chunker changes — was considered and rejected, because it makes the chunker
absorb a problem created upstream and leaves the next export equally fragile.

## 7. What this run added permanently

**Frequency weighting is now part of the golden set.** Each query carries how often it is
asked. The unweighted figure remains reported, because a rare query that fails is still a
failure, but the weighted figure is the gate.

**Structure preservation is now a migration requirement**, written down. It was an invisible
dependency of the retrieval system and it will be an invisible dependency of the next one.

---

## Notes on using this format

**Compare query by query.** Twenty-six per cent of the set moved and the aggregate showed half
a point. Aggregates measure whether the average changed, not whether the system did.

**Weight by frequency, and report both.** The unweighted number said the migration was
harmless. The weighted number said it removed 8 points from the queries people ask.

**Hold everything else constant.** One change per comparison is what turns a result into a
cause.

**Golden sets are cheap and permanent.** 220 hand-labelled queries took two days once, and
this migration is the third time they have prevented a regression reaching production.