Hallucination Testing: Detecting Confident Output That Isn't Grounded

A hallucination is not a lie. It is the model doing exactly what it was built to do — producing the most plausible continuation — in a situation where plausible and true have come apart.

That distinction matters for testing, because it tells you where to look. Fabrication is not random. It concentrates in predictable places, and those places are testable.

Stop asking "is it true?"#

The instinct is to check answers against reality. That does not scale: verifying every output requires someone who already knows the answer, which defeats the point of the system.

The tractable question is "is this grounded?" — does every factual claim in the output trace to something the model was actually given?

Grounding is checkable mechanically. Truth is not.

An answer can be grounded and wrong, if the source document was wrong. That is a content problem with a named owner, and it is a different — more fixable — problem than the model inventing things.

Where fabrication concentrates#

Six places, in rough order of frequency:

When retrieval returned nothing useful. The model was asked a question, given irrelevant passages, and produced an answer anyway. This is the single largest source in RAG systems, and it is entirely preventable with an explicit instruction to refuse.

Specific identifiers. Invoice numbers, dates, case references, citations, section numbers, prices. Anything with a precise form is easy to generate plausibly and hard to notice as wrong.

Numbers in prose. "Roughly 40% of customers" appearing in a summary of a document that contains no percentage.

Bridging gaps. Two retrieved passages, a missing link between them, and the model supplies the connective tissue as though it were sourced.

Attribution. "According to the policy…" where the policy says nothing of the kind.

Long outputs. Fabrication rate rises with output length. The first paragraph is usually grounded; paragraph five is where invention creeps in.

The tests that work#

1. Source-attribution check. Split the output into factual claims. For each, does supporting text appear in the retrieved context? Claims with no support are your fabrication rate.

This is the core test and it is automatable — an evaluator model can perform the matching, provided you give it the retrieved context and ask a narrow question ("is this claim supported by the text below: yes/no") rather than a broad one.

2. The no-answer test. Feed questions whose answer is genuinely absent from the corpus. The correct behaviour is to say so. Measure the refusal rate.

This is the highest-value single test in the set, and most teams never run it. A system that scores 95% on questions it can answer, and fabricates on 100% of questions it cannot, is dangerous in exactly the situation where users are most reliant on it.

3. Identifier verification. Extract every number, date, code and proper noun from the output and confirm each appears in the source. Cheap, deterministic, and catches the most damaging errors.

4. Contradiction check. Ask the same question several times. Inconsistent answers indicate the model is generating rather than retrieving. Consistency is not proof of correctness, but inconsistency is strong evidence of fabrication.

5. Poisoned-context test. Supply a document containing a deliberate falsehood. Does the system repeat it? It should — that is correct grounding behaviour — and the test confirms the model is using your documents rather than its training data.

6. Empty-context test. Supply no context at all. Anything that comes back is coming from training data, which tells you how much the system falls back on memory when retrieval fails.

What to measure#

Track over-refusal deliberately. It is easy to eliminate fabrication by making a system refuse everything, and teams that optimise only for groundedness drift there without noticing.

Reducing it#

Testing tells you the rate. These reduce it:

• Instruct refusal explicitly. "If the passages do not contain the answer, say so." One sentence, large effect.
• Show sources in the interface. This does not reduce fabrication, but it makes it catchable — which is what turns a dangerous system into a usable one.
• Fix retrieval before prompting. If the right passage never surfaces, no instruction helps.
• Constrain output length. Long free-form answers invite invention.
• Separate extraction from generation for anything precise. Pull identifiers programmatically; let the model write the prose around them.

FAQ#

Can hallucinations be eliminated?#

No, and treat any claim otherwise with suspicion. They can be reduced substantially through grounding and made visible through source attribution. Designing on the assumption that some output will be wrong — and that a human can catch it — is the honest architecture.

Does a bigger model hallucinate less?#

Somewhat, and not enough to rely on. Grounding the model in the right documents does far more than upgrading it. A weak model with correct context outperforms a strong one guessing.

How do we test this without a labelled dataset?#

Start with the no-answer test — you do not need labels, only questions you know are unanswerable from the corpus. Twenty of those will tell you more about your system's safety than a hundred ordinary questions.

Can a model check another model's grounding?#

Yes, and it works reasonably well for the narrow question "is this claim supported by this text". Use a different model from the one that generated the output, because a model grading itself shares its own blind spots.

What is an acceptable fabrication rate?#

Depends entirely on consequence. For internal search with visible sources, a few percent is tolerable because users catch it. For anything feeding a customer-facing decision, the correct design is not a lower rate — it is a human in the loop.

What else is coming for Hallucination Testing