A RAG system that gives a wrong answer can fail at three different points, and most shallow evaluations collapse them into a single number.
Retrieval may have selected the wrong passages. The model may have misread correct passages. Or the model may have added a claim that isn’t in the retrieved passages at all. These are three different errors, with three different fixes, and they only become visible if you evaluate them separately.
Why one number isn’t enough
The architecture of a RAG has two distinct phases: retrieval and generation. An evaluation that only measures “is the final answer correct” treats these two phases as a black box. If the number is low, you don’t know whether the problem is in the index, in retrieval, or in the model — and without knowing that, the fix becomes a guess.
That’s why it’s worth splitting the evaluation into three layers, each with its own measure.
First layer: did retrieval find the right passages?
Before generation even happens, the system has to find the passages that are genuinely relevant to the question. This is measured by comparing, on a set of questions with known answers, which passages the system actually retrieved against the ones an expert would have marked as relevant.
Two quantities matter here. How many of the relevant passages were found — if they’re missing, generation can never be correct, no matter how good the model is. And how many of the retrieved passages are actually relevant — too much noise in the context degrades even a capable generation step, because the model has to discard on its own what it doesn’t need.
A retrieval problem gets fixed at the indexing stage: how documents are chunked, which metadata is attached, which numerical representation best captures meaning for your domain. It doesn’t get fixed by changing the model.
Second layer: does generation correctly use what it received?
Given the same retrieved context, does the model always produce the same correct answer? This is where you isolate synthesis and reasoning ability from the retrieval problem: you hold the context fixed and evaluate generation alone.
It’s the layer closest to a traditional accuracy evaluation, with one difference: the comparison is against what the given context actually allows you to answer, not against an absolute truth. If the retrieved context is incomplete, even flawless generation will produce a partial answer — and that’s not a generation error, it’s an error inherited from the first layer.
Third layer: is the answer faithful to what was retrieved?
This is the layer that separates a reliable RAG from one that merely looks reliable. Faithfulness — or groundedness — means every claim in the answer is actually supported by the retrieved passages, not added by the model to complete a plausible pattern.
It’s measured by checking, claim by claim, whether the retrieved text supports it. An answer can be factually correct and still unfaithful, if the correct information didn’t come from the provided context but from the model’s memory: in that case the system worked by chance, not by design, and the source citation would be misleading, because it would point to a passage that doesn’t actually contain what it claims to.
The hallucination rate is measured here: not as a generic property of the model, but as the frequency with which generation drifts from what retrieval actually supplied.
Why the three layers should be read together, not added up
A high end-to-end accuracy can hide a faithfulness problem if the test questions are easy to answer from the model’s memory alone. A high faithfulness rate with low retrieval coverage describes a system that abstains correctly more than one that answers well. None of the three numbers, on its own, says whether the system is production-ready — reading them requires all three together, for the same reason a single percentage on a RAG is almost always an oversimplification.
In closing
Separating retrieval, generation and faithfulness costs more time than an end-to-end evaluation. The payoff is that when a production system degrades, you know immediately where to look: whether the index changed, whether the model changed, or whether the document corpus contains contradictory versions that no evaluation layer can resolve on its own.
If you’re building or evaluating a RAG system, we’re available to talk through the test set and the metrics best suited to your architecture.
