RAG — Frequently Asked Questions
Straight answers on retrieval-augmented generation — when to use it instead of fine-tuning, why hybrid search matters, how to measure it, what permissions really require, and what it costs to run.
Choosing RAG at all#
Should we use RAG or fine-tune the model?#
Use RAG when the system needs facts. Fine-tune when it needs a behaviour — a format, a tone, a task shape it keeps getting wrong.
Facts belong in documents, not in weights. A fine-tuned model cannot tell you where an answer came from, cannot be corrected by editing a file, and needs retraining every time a policy changes. If your answer would change when someone edits a document, that answer belongs in retrieval.
Do we still need RAG now that models have very large context windows?#
Usually yes. A large window lets you paste more text; it does not decide which text to paste. That decision is retrieval, and it is the part that determines answer quality. Long contexts also cost more per call and dilute attention — accuracy on a specific fact tends to fall as irrelevant surrounding material grows.
The honest exception: a small, stable corpus that fits comfortably in context, where you are willing to pay for the tokens every time. Then skip retrieval and keep the simpler system.
What makes a bad RAG candidate?#
Knowledge that is not written down. Questions requiring calculation across many documents rather than lookup. Corpora nobody owns. Anything where being confidently wrong has a serious cost and there is no reviewer in the loop.
Building it#
How should we chunk documents?#
Split on structure — sections, headings, paragraphs — not on a fixed character count. A fixed window cuts sentences in half and separates a claim from its condition.
Prepend the document title and heading path to every chunk. A chunk that says "this is not permitted" is useless without the section it came from. Add a small overlap so a thought that spans a boundary survives in at least one chunk.
Is semantic search enough on its own?#
No. Semantic search is good at paraphrase and bad at exact strings — part numbers, error codes, policy references, surnames. Those are precisely what people type when they need a real answer.
Run keyword and semantic retrieval together and merge the results. Hybrid retrieval is the single highest-value change most teams can make to an underperforming system.
How many chunks should we retrieve?#
Enough to contain the answer, and no more. More chunks raise cost and latency, and past a point they lower accuracy by burying the relevant passage. Tune it against your test set rather than picking a number from a tutorial — it depends on your chunk size and how concentrated answers are in your corpus.
How do we stop it inventing answers?#
Three things together, none sufficient alone: instruct it to answer only from the supplied passages; give it an explicit, worded permission to refuse; and show the sources so a reader can check. Then measure refusal accuracy on questions you know the corpus cannot answer.
A system that never refuses is not confident — it is unmeasured.
Permissions and security#
How do permissions work in a RAG system?#
Retrieval must be filtered by the entitlements of the person asking, at query time, before ranking. Filtering after generation is too late: the model has already seen the content, and the answer can leak it even when the sources are hidden.
Can we just use one service account for indexing?#
Index with a privileged account if you must, but never retrieve with one. If every user's query runs with full access, your retrieval layer is a search engine over everything the company has, exposed through a chat box. Test with accounts at different entitlement levels and treat a leak as a security incident, not a bug.
What about prompt injection from the documents themselves?#
Real, and often overlooked. Retrieved text is untrusted input: a document can contain instructions aimed at your model. Delimit retrieved content clearly as data, do not let it change system behaviour, and never wire an agent's tools directly to text that arrives from retrieval without a check in between.
Measuring and running it#
What do we measure?#
Retrieval and generation separately — that is the whole discipline in one line.
If retrieval hit rate is 60%, the system's ceiling is 60% and no amount of prompt work raises it. Measure hit rate or recall@k first, then groundedness, refusal accuracy, and over-refusal. Record a baseline before any tuning, or later changes cannot be compared to anything.
How big should the test set be?#
Start with twenty real questions with known answers — real ones, collected from the people who will use the system, not invented. Twenty is enough to expose the obvious failures. Grow it by adding every bad answer anyone reports, permanently. Within a few months the set describes your actual traffic.
Why did quality degrade when we changed nothing?#
Because the corpus changed. Documents were edited, added, or archived; an embedding model was updated behind an API; a source system started exporting a different format. A retrieval system sits on top of content that moves. Schedule a re-test on a cadence and treat a drop as a production incident.
What does it cost to run?#
Dominated by tokens per answer, which is driven mostly by how many chunks you retrieve, then by embedding and re-indexing, then by storage. Measure cost per answer from day one and cache repeated questions — in most internal deployments a small number of questions make up a large share of traffic.
How long does a first version take?#
A working prototype over one team's documents is a short piece of work. Getting it to the point where people trust it — sources, permissions, refusal behaviour, a real test set, a re-index path — is the majority of the effort, and it is the part that decides whether the system is still in use six months later.