RAG Testing: Test the Two Halves Separately

Most RAG systems are tested by asking questions and judging the answers. That tells you the system is wrong. It does not tell you which half is wrong — and the two halves fail differently, are fixed differently, and are owned by different people.

Test retrieval and generation separately. Everything useful follows from that.

The ceiling nobody notices#

If retrieval surfaces the correct passage 60% of the time, your system's accuracy ceiling is 60%. No prompt improves it. No model upgrade improves it. The generator cannot answer from a document it was never given.

Teams routinely spend weeks tuning prompts on systems whose retrieval is the binding constraint. The first diagnostic is always: for these twenty questions, was the answer in what came back?

If not, stop working on the prompt.

Testing retrieval#

You need a set of questions paired with the passage that should answer them. Twenty real questions from actual users beats two hundred invented ones.

Run these before touching the generator. They are cheap, deterministic, and they localise the fault.

The retrieval failures you will find#

Exact identifiers. Semantic search is bad at these — ask for invoice INV-2026-0043 and you get chunks about invoices generally, because all invoice text is semantically similar. The fix is hybrid search: keyword and semantic together. In our experience this resolves more retrieval complaints than any amount of embedding tuning.

Chunks too small to answer. A fragment matches the query and contains no usable information. Symptom: retrieval metrics look fine, answers are vague.

Chunks that lost their context. "It must be approved by two directors" — two what? Prepend the document title and heading path to every chunk. Cheap, and it improves both retrieval and answers.

Multi-hop questions. The answer requires combining two documents. Standard retrieval returns passages similar to the question, not passages that together answer it. These need different handling, and the first step is knowing which of your questions are multi-hop.

Stale documents retrieved perfectly. The system worked; the content was wrong. Not a retrieval bug — a content ownership problem, and the most common cause of "the AI gave a wrong answer".

Testing generation#

Now supply the correct passages deliberately and test only the answer:

That last row is the one to build first. See Hallucination Testing.

The test set that matters most#

Beyond ordinary questions, include:

• Unanswerable questions — the answer is genuinely not in the corpus. Correct behaviour is refusal.
• Ambiguous questions — two documents give different answers. Correct behaviour is to surface both.
• Outdated-document questions — where an old and a current version both exist.
• Permission-boundary questions — asked as a user who should not see the answer. This is a security test, not a quality test, and it should fail loudly.
• Exact-identifier questions — the known weakness.
• Multi-hop questions — requiring two sources.

Six categories, five questions each, and you have a suite that tells you far more than thirty ordinary questions would.

Measuring in production#

Offline suites go stale as documents change. Two production signals are worth capturing:

Retrieval confidence. When the top result's similarity score is low, the system is about to answer from weak context. Log it; correlate with complaints.

Source click-through. If users never open the cited sources, either they trust it completely or they have stopped reading. Both are worth knowing.

And add every reported bad answer to the offline suite permanently. That single habit does more for long-run quality than any metric.

FAQ#

How many test questions do we need?#

Twenty real ones to start, covering the six categories above. Real questions from actual usage teach you more than invented ones, because they contain the phrasing and assumptions your users actually have.

Should we use an AI to evaluate the answers?#

For narrow questions — "is this claim supported by this text?" — yes, it works well. For broad quality judgements it is unreliable. Use a different model from the generator, and calibrate it against human judgement on a sample before trusting it.

Our retrieval scores look good but answers are poor. Why?#

Usually chunks that match the query without containing an answer, or context so long the model is diluted. Check chunk size and how many you retrieve — more context is not always better, and beyond a point it actively hurts.

How do we test permissions?#

Run the same question as users with different entitlements and confirm each sees only what they should. Treat a failure as a security incident, not a quality bug — permission-filtered retrieval is the mechanism preventing your RAG system from becoming an efficient document-leaking machine.

How often should we re-run the suite?#

On every prompt change, every model version change, and on a schedule regardless — because the documents change even when the system does not. That last case is the one that catches teams out.

What else is coming for RAG Testing