RAG System Specification Template
A fill-in specification for a retrieval-augmented system — corpus ownership, chunking, retrieval, refusal behaviour, permissions and the measurements that decide whether it ships. Copy it, answer it, keep it with the code.
Markdown. No sign-up, no email.
Copy this file into your repository next to the code. A specification that lives in a wiki nobody opens is a document; one that sits beside the thing it describes is a contract. Every blank below is a decision someone has to make — leaving it blank is also a decision, just an unrecorded one.
System name: _______________ Owner (person, not team): _______________ Date: _______ Version: _______
1. The question this system answers#
Write one sentence describing the question a user brings. If it takes a paragraph, the scope is too wide for a first release.
_We answer: _______________________________________________
Who asks it today, and what do they do instead? _______________
What does a good answer save them? (minutes, escalations, tickets) _______________
2. Corpus#
| Document sets in scope | |
| Document sets explicitly OUT of scope | |
| Named owner of each set | |
| How often each set changes | |
| Where the truth lives if documents disagree |
🔴 If any owner cell reads "nobody", stop. The most common failure of a retrieval system is a stale document retrieved perfectly. Retrieval quality cannot fix content nobody maintains.
Knowledge that is NOT written down anywhere: _______________ (If the answer to the main question lives in people's heads, this is a documentation project wearing a retrieval project's clothes.)
3. Chunking#
| Decision | Value | Why |
|---|---|---|
| Split on | structure / fixed size | |
| Target chunk size | ||
| Overlap | ||
| Title + heading path prepended? | yes / no | |
| Metadata stored per chunk | owner, date, access level, type |
4. Retrieval#
| Decision | Value |
|---|---|
| Keyword search | yes / no |
| Semantic search | yes / no |
| Hybrid | yes / no |
| Embedding model + version (pinned) | |
| Chunks retrieved per query (k) | |
| Re-ranking | |
| Metadata filters available |
Semantic-only retrieval fails on exact identifiers — part numbers, error codes, policy references. Those are exactly what people search for. Record the decision either way.
5. Generation and refusal#
| Decision | Value |
|---|---|
| Model + version (pinned) | |
| Answer only from supplied passages | yes / no |
| Refusal instruction present | yes / no |
| What the user sees when the answer is not in the corpus | |
| Output length limit | |
| Retrieved content delimited as data, not instructions | yes / no |
Write the exact refusal text here: _______________
6. Sources shown to the user#
- Chunks used are displayed: yes / no
- Sources are clickable to the real document: yes / no
- Document date shown: yes / no
This is the mechanism by which users catch our mistakes. A system without it does not fail loudly — it fails quietly, and people stop using it without telling you.
7. Permissions#
| Retrieval filtered by the requesting user's entitlements | yes / no |
| Filtering happens at | query time / after generation |
| Privileged service account used | yes / no |
| Tested with users of different entitlements | yes / no |
🔴 A permissions failure here is a data breach, not a quality bug. Treat and report it as one.
8. Measurements — fill these BEFORE launch#
| Metric | Target | Baseline measured | Date |
|---|---|---|---|
| Retrieval hit rate / recall@k | |||
| Groundedness | |||
| Refusal accuracy (unanswerable set) | |||
| Over-refusal rate | |||
| Cost per answer | |||
| p95 latency |
Test set: _____ questions, held where: _______________ Every reported bad answer is added to this set permanently. That is the only way the set stays representative of what users actually ask rather than what we imagined they would ask.
9. Operations#
| Re-index trigger | |
| Retrieval confidence logged | yes / no |
| Bad-answer reporting path | |
| Scheduled re-test cadence | |
| Who is paged, and for what |
The corpus changes even when the code does not. A system with no scheduled re-test degrades without a single deployment.
10. Sign-off#
| Name | Date | |
|---|---|---|
| Built by | ||
| Content owner | ||
| Security review | ||
| Approved for users |