Evaluating RAG: Faithfulness, Groundedness, and Retrieval Quality
Evaluating RAG: Faithfulness, Groundedness, and Retrieval Quality
Retrieval-augmented generation was sold as the cure for hallucination. Ground the model in your documents, the argument went, and it will stop making things up.
It does not stop making things up. It changes how it makes things up, and the new failure mode is considerably harder to catch — because the output now arrives with a citation attached, and a citation is the most persuasive thing you can put next to a false sentence.
The reason RAG evaluations fail to catch this is that most of them measure one thing and call it three. "Is the answer good?" collapses a pipeline with at least four independent failure points into a single score. When that score drops you have no idea what to fix, and when it holds you have no idea what it is hiding.
Four places a RAG answer goes wrong
The four failures are independent, and — this is the part that gets missed — they can cancel each other out in a way that produces a correct-looking answer from a broken system.
A model with strong parametric knowledge will answer a question correctly even when retrieval returned nothing useful. The answer is right. The user is happy. The evaluation passes. And the system has just demonstrated that it will answer from memory rather than from your corpus, which means the next time the corpus contains the updated policy and the model's memory contains the old one, it will confidently give the old one. Your evaluation had no way to see this coming, because it only looked at the final answer.
This is why answer accuracy alone is an unsafe metric for RAG. It rewards exactly the behaviour RAG exists to prevent.
The three measurements, defined precisely
Retrieval quality is a property of the retriever alone. Given a query, did the system surface the chunks that contain the information needed to answer it? This is measured against a labelled set of query-to-relevant-chunk mappings, and it does not involve the generator at all.
Two sub-measurements matter, and they trade off:
- Recall — of the chunks that could support a correct answer, how many appeared in the top-k? Recall failures are unrecoverable. If the passage is not in the context, no amount of generator quality will produce a grounded answer.
- Precision — of the chunks in the top-k, how many are actually relevant? Precision failures are recoverable but expensive: they burn context, and long, noisy contexts measurably degrade a model's ability to locate the passage that matters.
Faithfulness is a property of the answer relative to the retrieved context, and it is deliberately indifferent to truth. The question is not "is this claim correct?" but "is every claim in this answer entailed by the context that was provided?" An answer can be perfectly faithful and factually wrong, if the retrieved document was wrong. That is not a generator defect; it is a corpus defect, and you want your metric to tell those apart.
Measure it by decomposing the answer into atomic claims and checking each against the context. The output is not a vibe but a ratio: supported claims over total claims, with the unsupported ones enumerated. An answer with one unsupported clause buried in five supported ones is the archetypal RAG failure, and only decomposition finds it.
Groundedness — or attribution — is a property of the citation. When the answer says "per the 2026 policy [3]", does chunk 3 actually say that? Systems routinely cite plausibly: the citation is topically related, from the right document, and does not support the specific claim it is attached to. Reviewers rarely click through, which is precisely why the failure survives.
Attribution has to be checked at claim granularity, not answer granularity. "The answer cites three documents and all three are relevant" is not an attribution check. "Claim 4 cites chunk 2, and chunk 2 does not contain claim 4" is.
The metrics that actually diagnose
| Measurement | Question it answers | Unit | What a failure tells you to fix |
|---|---|---|---|
| Retrieval recall @ k | Was the answer even available? | Query | Chunking, embedding model, k, hybrid search, query rewriting |
| Retrieval precision @ k | How much noise did we inject? | Query | Reranking, threshold, filter logic |
| Context utilisation | Did the generator use the good chunk it was given? | Answer | Prompt design, context ordering, model choice |
| Faithfulness | Is every claim entailed by the context? | Claim | Generator, decoding, prompt constraints |
| Attribution accuracy | Does each citation support its claim? | Citation | Citation mechanism, post-hoc verification |
| Answer correctness | Is the final answer right? | Answer | Nothing directly — it is the outcome, not a cause |
| Refusal appropriateness | Does it decline when the corpus cannot answer? | Query | The most neglected metric on this list |
That last row deserves its own paragraph. A RAG system must be evaluated on the questions it cannot answer. Build a slice of queries whose answers are genuinely absent from the corpus — retired policies, out-of-scope topics, questions about a customer who does not exist — and score whether the system says so. A system that scores 92% on answerable questions and confabulates on every unanswerable one is not a 92% system. It is a liability with a good average, and in a regulated setting the unanswerable slice is the one that generates the complaint.
Report the two slices separately. Never blend them.
Why automated RAG scoring is a screen, not a verdict
There is a mature toolkit for computing these metrics automatically, typically by using a language model to perform the claim decomposition and entailment checks. It is genuinely useful. It is also, structurally, a model grading a model — and it fails in correlated ways: it is lenient toward fluent, confident prose, it inherits the same blind spots on domain-specific terminology, and it will happily rate a subtly wrong clinical or legal claim as "supported" because the context contains superficially similar language.
Treat automated faithfulness scoring as a screen. It is excellent for regression detection across thousands of queries and for surfacing the twenty answers most likely to be wrong. It is not adequate as the basis for a claim that a system is safe for a consequential domain.
For that, the entailment judgement has to be made by a person who understands the domain, working against a rubric that defines what "supported" means for this corpus and these stakes — because in a clinical context, "the context implies it" is not support, and in a marketing context it may well be. Structured evaluation handles this by scoring each dimension against anchored criteria, injecting gold tasks with a known correct assessment to calibrate the Evaluators reviewing them, and adjudicating cases where qualified people disagree — because a query where two domain experts disagree about whether a claim was grounded is not a scoring problem to average away. It is a finding about the corpus, the rubric, or the answer, and it needs to be resolved rather than smoothed.
What to do tomorrow
Take one hundred production queries and, for each, record four things separately: the retrieved chunk IDs, whether a human judges any of them sufficient to answer, whether every claim in the answer is entailed by those chunks, and whether each citation supports the specific claim it is attached to.
Then compute the number nobody computes: how often did the system produce a correct answer when retrieval had returned nothing sufficient? That is your parametric-leakage rate, and it is the single best predictor of how your RAG system will behave the day your corpus and the model's training data disagree.
Add the unanswerable slice. Score refusal. Then, and only then, look at the answer-quality number — because until you know which of the four stages is carrying the system, that number is not telling you anything you can act on.