Evaluating RAG systems

A Retrieval-Augmented Generation (RAG) system can output a beautifully formatted, completely confident answer that is fundamentally wrong. Even worse, it can fail in two distinct ways that look identical from the outside:

1. Retrieval Failure: The document chunk needed to answer the question was never fetched from the vector database.
2. Generation Failure: The correct document chunk was fetched and placed in the context window, but the language model ignored it, misinterpreted it, or hallucinated facts beyond it.

Unless you evaluate retrieval and generation separately, diagnosing and fixing poor performance is impossible.

The Triad of RAG Metrics

To reliably assess a RAG pipeline, the community generally evaluates across three foundational dimensions (often referred to as the RAG Triad):

       [ User Query ]
             |
             +-----------------------+
             |                       |
             v                       v
     [ Retrieved Context ] -----> [ Generated Answer ]

1. Context Relevance (Retrieval Quality): Are the retrieved chunks actually relevant to answering the user’s question, or is the context cluttered with irrelevant noise?
2. Groundedness / Faithfulness (Hallucination Control): Can every claim in the generated answer be directly inferred from the retrieved chunks?
3. Answer Relevance (Task Alignment): Does the generated response directly answer what the user asked without digressing or omitting critical constraints?

Part 1: Evaluating Retrieval Independently

Retrieval is a classical information retrieval (IR) challenge. If you have a curated test set containing user queries paired with the specific document IDs that contain the true answers (known as a “golden dataset”), you can calculate standard deterministic metrics.

Here is a clean Python implementation calculating Hit Rate and Mean Reciprocal Rank (MRR):

from typing import List, Sequence

def hit_rate(retrieved_ids: Sequence[str], expected_id: str) -> float:
    """Returns 1.0 if the expected document is present in the top-k results, else 0.0."""
    return 1.0 if expected_id in retrieved_ids else 0.0

def reciprocal_rank(retrieved_ids: Sequence[str], expected_id: str) -> float:
    """
    Computes reciprocal rank: 1 / rank of the first relevant document.
    Returns 0.0 if the relevant document is absent.
    """
    for index, doc_id in enumerate(retrieved_ids):
        if doc_id == expected_id:
            return 1.0 / (index + 1)
    return 0.0

def evaluate_retriever(dataset: List[dict], top_k: int = 5) -> dict:
    total_hit = 0.0
    total_mrr = 0.0
    count = len(dataset)

    for item in dataset:
        retrieved = item["retrieved_ids"][:top_k]
        target = item["expected_id"]

        total_hit += hit_rate(retrieved, target)
        total_mrr += reciprocal_rank(retrieved, target)

    return {
        f"hit_rate@{top_k}": round(total_hit / count, 4),
        f"mrr@{top_k}": round(total_mrr / count, 4),
    }

# Example benchmark test run
sample_eval_data = [
    {"expected_id": "doc_42", "retrieved_ids": ["doc_10", "doc_42", "doc_88"]},
    {"expected_id": "doc_99", "retrieved_ids": ["doc_99", "doc_01", "doc_02"]},
    {"expected_id": "doc_15", "retrieved_ids": ["doc_04", "doc_08", "doc_12"]},
]

scores = evaluate_retriever(sample_eval_data, top_k=3)
print(scores)
# Output: {'hit_rate@3': 0.6667, 'mrr@3': 0.5}

Retrieval Metric Comparison

The table below summarizes standard retrieval measurements, what they signal, and when to prioritize them during search optimization:

[!TIP] Adding a cross-encoder re-ranking stage (such as Cohere Rerank or BGE-Reranker) often improves MRR by 20–35% without changing your underlying embedding index.

Part 2: Evaluating Generation (Faithfulness and Groundedness)

Once you know your retriever reliably supplies relevant chunks, you must verify that the language model does not introduce hallucinations.

Because natural language answers can be phrased in infinitely many valid ways, deterministic exact-match string metrics (like BLEU or ROUGE) are notoriously brittle for RAG. Instead, modern pipelines employ an LLM-as-a-Judge scoring methodology.

Here is a structured verification prompt that decomposes an answer into discrete factual claims and validates each claim against the supplied context:

import json
from typing import List, Dict

FAITHFULNESS_JUDGE_PROMPT = """
You are an expert evaluator assessing factual consistency.
Analyze the following context passages and the generated answer.

Context:
{context}

Answer:
{answer}

Follow this procedure:
1. Break down the generated answer into discrete, individual factual claims.
2. For each claim, check whether it is directly supported by the context.
3. Calculate the faithfulness score as: (supported claims) / (total claims).

Respond STRICTLY in valid JSON with no extraneous commentary:
{{
  "claims": [
    {{"claim": "statement text", "supported": true, "rationale": "evidence from context"}}
  ],
  "faithfulness_score": 0.0
}}
"""

def build_judge_payload(context_snippets: List[str], answer: str) -> str:
    combined_context = "\n---\n".join(context_snippets)
    return FAITHFULNESS_JUDGE_PROMPT.format(
        context=combined_context,
        answer=answer
    )

Common Failure Modes and Where They Lie

When an evaluation metric drops, consult this diagnostic matrix:

Symptoms & Diagnostics:
│
├── Low Hit Rate @ 5
│   └── Problem: Retrieval failure
│   └── Solutions:
│       ├── Experiment with hybrid search (dense semantic + BM25 keyword)
│       ├── Adjust chunk boundaries (try 512 tokens with 50-token overlap)
│       └── Generate hypothetical document embeddings (HyDE)
│
├── High Hit Rate, but Low Faithfulness
│   └── Problem: Generation hallucination
│   └── Solutions:
│       ├── Tighten system prompt instruction ("Answer ONLY using provided text")
│       ├── Lower temperature (e.g. 0.0 or 0.1 for factual synthesis)
│       └── Strip boilerplate from chunks before context injection
│
└── High Faithfulness, but Low Answer Relevance
    └── Problem: Evasion or misalignment
    └── Solutions:
        ├── Add explicit few-shot response examples
        └── Ensure query rewriting captures user intent

Building a Golden Dataset

Automated evaluation is only as good as the benchmark questions you test against. Aim for at least 50–100 curated query-document pairs spanning:

• Direct fact lookup: Simple, unambiguous questions with single-sentence answers.
• Multi-chunk synthesis: Questions requiring aggregation across two or more separate documents.
• Negative cases: Questions where the knowledge base explicitly does not have the answer (verifying the system politely declines rather than hallucinating).

Conclusion

RAG systems cannot be evaluated as black boxes. By decoupling retrieval quality metrics (Hit Rate, MRR) from generation quality metrics (Faithfulness, Relevance), engineering teams can turn ambiguous user complaints into actionable, reproducible improvements in their pipeline.