# Groundedness Audit Report — Sample

**This is an illustrative example.** The system and every figure are invented to show the
shape of an audit that produces fixes rather than anxiety. Copy the method; measure your own
system.

The audit measures one thing precisely: **what proportion of the claims in an answer are
supported by the source material provided**. Not whether the answer "seems right" — whether
each individual statement can be traced.

---

## Groundedness audit — customer policy assistant

| | |
|---|---|
| System | Assistant answering customer questions from a policy library |
| Audited | 2026-07-20 to 2026-07-27 |
| Sample | 240 answers — 180 from production logs, 60 written to probe known weak areas |
| Method | Each answer split into atomic claims; each claim checked against the retrieved passages |
| Reviewers | Two independent, third resolving disagreement |

## 1. Headline

| | Answers | Share |
|---|---|---|
| Every claim supported | 171 | 71% |
| One or more unsupported claims | 52 | 22% |
| Correctly declined to answer | 17 | 7% |

Claim-level, which is the number that matters:

| | Claims | Share |
|---|---|---|
| Total claims across all answers | 1,486 | |
| Supported by a retrieved passage | 1,342 | 90.3% |
| Not supported | 144 | 9.7% |

**Ninety per cent of claims supported sounds strong and is not.** An answer averages six
claims, so a 9.7% claim-level failure rate produces an unsupported statement in roughly a
fifth of answers.

## 2. The unsupported claims, by type

| Type | Claims | What it looks like |
|---|---|---|
| Plausible elaboration | 61 | Detail that fits, that no passage contains |
| Merged from two passages | 33 | Facts from separate products combined into one answer |
| Outdated but retrieved | 24 | Supported by a passage that is no longer current |
| Arithmetic on retrieved values | 15 | Correct inputs, wrong calculation |
| Contradicts the passage | 11 | Directly wrong |

**Plausible elaboration is the largest category and the hardest to notice.** The answer is
mostly correct; one sentence adds a detail that was never in the source. A reader checking a
citation finds the surrounding claims supported and stops checking.

The 33 merged claims share a cause worth naming: two products with similar names, whose
passages arrive together and are not distinguishable once inside the prompt.

## 3. The cases that matter most

Cross-tabulating unsupported claims against how confidently they were stated:

| | Hedged | Stated plainly | Stated emphatically |
|---|---|---|---|
| Supported claims | 214 | 1,009 | 119 |
| Unsupported claims | 12 | 98 | **34** |

**Thirty-four unsupported claims were stated emphatically** — "always", "in every case",
"you must". These are the most damaging output the system produces, because confidence is
the only signal a reader has when they cannot check.

There is no relationship between how confidently the system states something and whether it
is supported. That is worth stating plainly to anyone inclined to trust the tone.

## 4. Where the audit disagreed with itself

Reviewers disagreed on 31 of 1,486 claims, about 2%. Nearly all were the same question:
**does a claim need to be stated in a passage, or is a reasonable inference from a passage
acceptable?**

The rule adopted, after the third reviewer resolved them: an inference is supported only if a
competent reader would reach the same conclusion from the passage alone. Anything requiring
outside knowledge is unsupported, even when true.

Recording the rule matters more than the 31 claims. Without it, the next audit measures
something slightly different and the trend is meaningless.

## 5. What was changed

Three changes, in order of what they closed.

**Passages now carry their product name and effective date into the prompt**, and the answer
format requires each claim to name its source. Addressed the merged claims and the outdated
ones — 57 of 144.

**Arithmetic moved out of the model.** Calculations on retrieved values are performed in code
and the result is passed in. Addressed all 15 arithmetic failures, which is the only category
that goes to zero rather than down.

**The answer format now separates what the policy says from what it does not address.**
Elaboration fell but did not disappear; it is the hardest category and no format change
eliminates it.

## 6. After

| | Before | After |
|---|---|---|
| Claim-level groundedness | 90.3% | 96.8% |
| Answers with an unsupported claim | 22% | 9% |
| Emphatic unsupported claims | 34 | 6 |
| Arithmetic errors | 15 | 0 |
| Correctly declined | 7% | 11% |

The rise in declines is intended. The system now declines where the library genuinely does not
answer the question, which was previously where elaboration came from.

## 7. What this audit cannot tell you

**It measures grounding, not correctness.** A claim supported by a passage that is itself wrong
scores as grounded. This audit assumes the library is right; auditing the library is separate
work.

**It does not measure usefulness.** A system that declined every question would score
perfectly. That is why the decline rate is reported alongside, and why a rise in it needs a
reason.

**The 60 probe questions bias the sample toward known weak areas.** The production-only figure
is 3 points better. Both are reported; neither alone is honest.

---

## Notes on using this format

**Split answers into claims and check each one.** Rating whole answers as "correct" or "not"
loses the fact that most bad answers are mostly right, which is precisely what makes them
persuasive.

**Cross-tabulate against confidence.** The emphatic unsupported claims are the most damaging
output and they are invisible in a single accuracy figure.

**Write down the inference rule.** Two reviewers will otherwise apply different standards, and
the difference between audits will exceed the difference this measures.

**Report the decline rate next to groundedness.** Otherwise the easiest way to improve the
number is to make the system less useful.