# How to evaluate a RAG system

> A RAG system is evaluated on two separate layers: whether it finds the right passages (retrieval) and whether it writes an answer that is faithful to those passages and relevant to the question (generation). That takes a test set built from real questions, metrics suited to each layer and a feedback loop in production. A single overall accuracy score will not show you where the failure is.

URL: https://corrplus.net/blog/rag-evaluation/
Updated: 2026-10-26
Corrplus Team

## Why is RAG evaluation a two-layer problem?

RAG evaluation is the testing process that measures, separately, whether a system finds the right information and whether it then writes a correct answer from it. The Ragas paper splits evaluation along the same lines: whether retrieval finds relevant and focused passages, whether the model uses those passages faithfully, and the overall quality of the generated answer.

The split matters because the fix is different. If the wrong passages come back, the problem is in chunking, the [embedding](/glossary/embedding/) model, the [vector database](/glossary/vector-database/) query or reranking. If the right passages come back but the answer is wrong, the problem is in the prompt, the model or the answer format. A single right-or-wrong score cannot tell these two cases apart.

We explain RAG itself on our [RAG glossary page](/glossary/rag/) and its use on WordPress in [RAG for WordPress](/blog/rag-for-wordpress/). This article covers measurement only.

## How do you build a test set?

A good test set is a small mirror of the questions the system will face in production. Anthropic's evaluation guide suggests three principles: be specific to your task, grade automatically wherever you can, and prefer more questions over a few perfect ones.

In practice these steps work:

1. **Start with real questions.** Support tickets, search logs and internal messages are the best source. Use users' own wording rather than inventing questions.
2. **Attach the expected source to each question.** Mark which document or page holds the answer. For critical questions, write a short reference answer.
3. **Add hard cases.** Questions whose answer is not in the documents, two documents that contradict each other, procedures with an outdated version, ambiguous questions. The system's ability to say "I don't know" needs testing too.
4. **Represent each language.** If content is mixed, for example English and Turkish, or English and German, add questions in each language. Try cross-language cases such as a Turkish question answered from an English document. Keep spelling and phrasing variants as separate questions.

Do not write the set once and leave it. Every question answered wrongly in production is a candidate for the set.

## Which metric measures what?

The Ragas documentation lists metrics for RAG such as context precision, context recall, faithfulness, response relevancy and noise sensitivity. Whether a metric needs a reference answer directly affects which ones you can run.

| Metric | What it measures | Needs a reference? | Failure it catches | How it is checked |
|---|---|---|---|---|
| Context precision | Whether relevant passages rank near the top | Variants with and without | The right passage buried among irrelevant ones | Looks at the order of retrieved passages |
| Context recall | How much of the needed information was retrieved | Yes | Missing retrieval, poor chunking | Whether claims in the reference answer appear in retrieved passages |
| Faithfulness | Whether every claim in the answer is supported by retrieved passages | No | Statements not grounded in the sources | The answer is split into claims and each is checked against the passages |
| Response relevancy | Whether the answer addresses the question | No | Drifting, incomplete or padded answers | Semantic similarity between question and answer |
| Source accuracy (your own metric) | Whether the cited link is the right document | Needs the expected source | Pointing users to the wrong document | Compared against the expected source list |

Low faithfulness means the model is going beyond the retrieved text; it is the most direct measure of [AI hallucination](/glossary/ai-hallucination/) risk. Low context recall means swapping the model will not help, and you need to look at retrieval. Anthropic's Contextual Retrieval work measures its improvements on the retrieval side too, as the share of relevant passages not found in the top 20 results. That is a simple, readable metric you can use on your own system.

## When can you trust LLM-as-judge, and when do you need a person?

Most of the metrics above rely on a language model acting as the grader (LLM-as-judge). It makes high-volume testing possible, but it brings its own errors. Anthropic's guide recommends, where possible, grading with a different model from the one that produced the answer.

You can rely on an LLM grader when:

- The grading rules are explicit and detailed (for example, "every amount in the answer must appear verbatim in the source").
- Its decisions have been compared with a small human-labelled sample and found consistent.
- The question type is in a domain the grader itself handles well.

Keep human review when the answer underpins a legal, financial or medical decision, when the grader and humans disagree often, or when a new document type or language is added. In [finance and insurance](/industries/finance-insurance/), for instance, answers that quote a policy or contract clause should get regular human checks, even if only on a sample.

## How do you set up live monitoring and a feedback loop?

The test set is the starting point. In production, users ask questions that are not in the set, documents change and model versions are updated. Log the following:

- **Question, retrieved passages, answer and cited sources.** Without these you cannot tell later whether a failure came from retrieval or generation.
- **User feedback.** A simple helpful or not helpful button with an optional comment.
- **"I don't know" and hand-off rate.** A sudden rise usually means the index is stale or a new topic is not documented.
- **Regular sampling.** Each period, re-grade a random batch of answers for faithfulness and source accuracy.

When the model, embedding or chunking settings change, run the same test set before and after. This matters most in document-heavy uses such as [document analysis](/solutions/document-analysis/): the technical specification archives of [construction and building materials](/industries/construction-building-materials/) firms or the product documentation of [technology and software](/industries/technology-software/) teams change often.

## Where should you start?

If you already run a RAG assistant, pull a small test set from recent real questions and mark the expected source for each. Measure only context recall and faithfulness at first; together they usually show whether the problem sits in retrieval or generation.

At Corrplus we build the test set during design, with the client and from real questions, and measure the first release against it before handover. For the build itself see [RAG enterprise knowledge assistant](/services/rag-enterprise-knowledge-assistant/), for the model and API layer [LLM integration](/services/llm-integration/), and for regular measurement and improvement in production [managed services](/services/managed-services/).

## Frequently asked questions

### How many questions does a test set need?

There is no fixed number. Anthropic's evaluation guide says more automatically graded questions are usually more useful than a few hand-graded ones. Start from real user questions, cover every important topic and question type, and grow the set over time.

### Can we evaluate without writing reference answers?

Partly. Metrics such as faithfulness and response relevancy do not need a reference answer. Retrieval metrics such as context recall do, because they need to know what should have been found. Write the expected source and a short reference answer at least for critical questions.

### Can LLM-as-judge replace human review?

Not fully. It suits high-volume, repeated checks. Compare the judge against a small human-labelled sample before you trust it. Answers that inform legal, financial or medical decisions should still get human review.

### What should we do when the model or embedding changes?

Run the same test set before and after the change and compare. Ship the change only if retrieval and generation metrics do not regress. When the system is run as a managed service, this is a routine step.

## Sources

- [Es et al., Ragas: Automated Evaluation of Retrieval Augmented Generation (arXiv, 2023)](https://arxiv.org/abs/2309.15217)
- [Ragas documentation, Available metrics](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/)
- [Ragas documentation, Context Recall](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_recall/)
- [Anthropic, Contextual Retrieval](https://www.anthropic.com/research/contextual-retrieval)
- [Anthropic, Define success criteria and build evaluations](https://platform.claude.com/docs/en/test-and-evaluate/develop-tests)