RAG Evaluation Metrics: Measuring Retrieval and Generation Quality

You built a RAG pipeline. It retrieves chunks, feeds them to an LLM, and produces answers. But is it any good? Without metrics, you are guessing. A change to your chunking strategy might improve retrieval or destroy it — you will not know until you measure.

RAG evaluation splits into two halves: retrieval quality (did you fetch the right documents?) and generation quality (did the LLM produce a correct, grounded answer?). This article covers both, with Python code you can run today.

Part 1: Retrieval metrics

These metrics require a test set of queries with known relevant documents (ground truth). Even 50-100 labeled query-document pairs are enough to start.

Recall@k

The most important retrieval metric for RAG. It answers: of all the relevant documents, how many did we retrieve in the top k results?

Recall@k = |relevant docs in top k| / |total relevant docs|

def recall_at_k(retrieved: list[str], relevant: set[str], k: int) -> float:
    """Compute Recall@k.

    Args:
        retrieved: Ordered list of retrieved document IDs.
        relevant: Set of ground-truth relevant document IDs.
        k: Number of top results to consider.
    """
    if not relevant:
        return 0.0
    top_k = set(retrieved[:k])
    return len(top_k & relevant) / len(relevant)

# Example
retrieved = ["doc_3", "doc_7", "doc_1", "doc_5", "doc_2"]
relevant = {"doc_1", "doc_5", "doc_9"}

print(f"Recall@3: {recall_at_k(retrieved, relevant, 3):.2f}")  # 0.33
print(f"Recall@5: {recall_at_k(retrieved, relevant, 5):.2f}")  # 0.67

For RAG, Recall@10 or Recall@20 is the standard. You typically retrieve 10-20 chunks and pass them all to the LLM. If the relevant chunk is not in that set, the LLM cannot use it — no amount of prompt engineering will help.

Mean Reciprocal Rank (MRR)

MRR measures how high the first relevant result appears. It is the average of 1/rank of the first relevant document across all queries:

MRR = (1/N) * sum(1 / rank_of_first_relevant_doc)

def reciprocal_rank(retrieved: list[str], relevant: set[str]) -> float:
    """Compute reciprocal rank for a single query."""
    for rank, doc_id in enumerate(retrieved, start=1):
        if doc_id in relevant:
            return 1.0 / rank
    return 0.0

def mean_reciprocal_rank(queries: list[dict]) -> float:
    """Compute MRR across multiple queries.

    Each query dict has 'retrieved' (list) and 'relevant' (set).
    """
    rr_scores = [
        reciprocal_rank(q["retrieved"], q["relevant"])
        for q in queries
    ]
    return sum(rr_scores) / len(rr_scores)

# Example
queries = [
    {"retrieved": ["doc_3", "doc_1", "doc_5"], "relevant": {"doc_1"}},  # RR = 1/2
    {"retrieved": ["doc_7", "doc_2", "doc_4"], "relevant": {"doc_4"}},  # RR = 1/3
    {"retrieved": ["doc_1", "doc_3", "doc_5"], "relevant": {"doc_1"}},  # RR = 1/1
]

print(f"MRR: {mean_reciprocal_rank(queries):.3f}")  # 0.611

MRR is useful when you care about the ranking of results, not just whether they appear. An MRR of 0.5 means the first relevant result typically appears at position 2.

NDCG@k (Normalized Discounted Cumulative Gain)

NDCG handles graded relevance — some documents are more relevant than others (e.g., a perfect answer vs. a partially relevant passage). It discounts the contribution of each result by its position:

import numpy as np

def dcg_at_k(relevance_scores: list[float], k: int) -> float:
    """Compute DCG@k."""
    scores = np.array(relevance_scores[:k])
    positions = np.arange(1, len(scores) + 1)
    return np.sum(scores / np.log2(positions + 1))

def ndcg_at_k(retrieved_relevance: list[float], k: int) -> float:
    """Compute NDCG@k.

    Args:
        retrieved_relevance: Relevance scores in retrieval order.
            e.g., [3, 0, 2, 1] means the first result has relevance 3,
            second has 0, third has 2, fourth has 1.
        k: Number of top results to consider.
    """
    actual_dcg = dcg_at_k(retrieved_relevance, k)
    ideal_order = sorted(retrieved_relevance, reverse=True)
    ideal_dcg = dcg_at_k(ideal_order, k)

    if ideal_dcg == 0:
        return 0.0
    return actual_dcg / ideal_dcg

# Example: retrieved docs have relevance scores [3, 0, 2, 1, 0]
scores = [3, 0, 2, 1, 0]
print(f"NDCG@3: {ndcg_at_k(scores, 3):.3f}")  # 0.846
print(f"NDCG@5: {ndcg_at_k(scores, 5):.3f}")  # 0.876

An NDCG@k of 1.0 means the ranking is perfect. For most RAG systems, aim for NDCG@10 above 0.7.

Part 2: Generation metrics

Retrieval metrics tell you if the right context reached the LLM. Generation metrics tell you if the LLM used that context correctly.

Faithfulness

Faithfulness measures whether the generated answer is supported by the retrieved context. An answer is faithful if every claim it makes can be traced back to a retrieved chunk. This is the most important generation metric — a low faithfulness score means the LLM is hallucinating.

from openai import OpenAI

client = OpenAI()

def evaluate_faithfulness(answer: str, context: str) -> dict:
    """Use an LLM to judge faithfulness of an answer given context."""
    prompt = f"""You are an evaluation judge. Given the context and the answer,
determine if every claim in the answer is supported by the context.

Context:
{context}

Answer:
{answer}

Respond with a JSON object:
- "faithful": true/false
- "unsupported_claims": list of claims not supported by context
- "score": float from 0.0 to 1.0 (proportion of claims that are supported)
"""
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        response_format={"type": "json_object"},
    )
    import json
    return json.loads(response.choices[0].message.content)

# Example
context = "The auth service returns error E-4021 when JWT tokens expire after 24 hours."
answer = "Error E-4021 occurs when JWT tokens expire. The default expiry is 24 hours. You can extend it to 48 hours in the config."

result = evaluate_faithfulness(answer, context)
print(f"Faithfulness: {result['score']}")
# The claim about extending to 48 hours is unsupported

Answer relevance

Does the answer actually address the question? A faithful answer can still be irrelevant if it talks about the right topic but misses the specific question.

def evaluate_relevance(question: str, answer: str) -> dict:
    """Use an LLM to judge if the answer addresses the question."""
    prompt = f"""You are an evaluation judge. Rate how well the answer
addresses the question.

Question: {question}
Answer: {answer}

Respond with a JSON object:
- "relevant": true/false
- "score": float from 0.0 to 1.0
- "reason": brief explanation
"""
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        response_format={"type": "json_object"},
    )
    import json
    return json.loads(response.choices[0].message.content)

Hallucination rate

The percentage of generated answers that contain at least one claim not supported by the retrieved context. Compute it across your test set:

def hallucination_rate(eval_results: list[dict]) -> float:
    """Compute hallucination rate across evaluated answers."""
    hallucinated = sum(1 for r in eval_results if not r["faithful"])
    return hallucinated / len(eval_results)

# Example: 8 out of 100 answers had unsupported claims
results = [{"faithful": True}] * 92 + [{"faithful": False}] * 8
print(f"Hallucination rate: {hallucination_rate(results):.1%}")  # 8.0%

A hallucination rate under 5% is good. Under 2% is excellent. Above 10% means your retrieval or prompting needs work.

Automating evaluation with RAGAS

Writing custom evaluation prompts is fine for understanding, but for production you want a framework. RAGAS (Retrieval Augmented Generation Assessment) automates all the metrics above.

from ragas import evaluate
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_precision,
    context_recall,
)
from datasets import Dataset

# Prepare your evaluation data
eval_data = {
    "question": [
        "What causes error E-4021?",
        "How do I configure JWT expiry?",
    ],
    "answer": [
        "Error E-4021 is caused by expired JWT tokens in the auth service.",
        "Set the JWT_EXPIRY_HOURS environment variable to your desired value.",
    ],
    "contexts": [
        ["The auth service returns E-4021 when JWT tokens expire after 24 hours."],
        ["JWT configuration is managed via environment variables including JWT_EXPIRY_HOURS."],
    ],
    "ground_truth": [
        "E-4021 occurs when JWT tokens expire.",
        "Configure JWT expiry by setting JWT_EXPIRY_HOURS env var.",
    ],
}

dataset = Dataset.from_dict(eval_data)

results = evaluate(
    dataset,
    metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
)

print(results)
# {'faithfulness': 0.95, 'answer_relevancy': 0.92,
#  'context_precision': 0.88, 'context_recall': 0.90}

RAGAS uses an LLM (GPT-4 by default) to judge the outputs. It decomposes answers into individual claims and checks each one against the context — much more thorough than a single yes/no judgment.

Building your evaluation dataset

The hardest part of RAG evaluation is creating ground truth. Here is a practical approach:

1. Start with 50 queries. Pick real questions users have asked (from logs, support tickets, or stakeholder interviews).
2. Label relevant documents manually. For each query, mark which chunks in your corpus contain the answer.
3. Write reference answers. Short, factual answers you can compare against.
4. Grow incrementally. Add 10 queries per week as you discover failure cases.

# Evaluation dataset format
eval_set = [
    {
        "query": "What causes error E-4021?",
        "relevant_doc_ids": {"doc_42", "doc_108"},
        "reference_answer": "E-4021 is caused by expired JWT tokens in the auth service.",
    },
    # ... more entries
]

You do not need thousands of examples. Fifty well-chosen queries covering your main use cases will catch most regressions.

Putting it all together

Here is a complete evaluation workflow:

def evaluate_rag_pipeline(pipeline, eval_set: list[dict], k: int = 10) -> dict:
    """Run full RAG evaluation."""
    recall_scores = []
    mrr_scores = []
    faithfulness_scores = []

    for item in eval_set:
        # Retrieval evaluation
        retrieved = pipeline.retrieve(item["query"], top_k=k)
        retrieved_ids = [doc["id"] for doc in retrieved]
        relevant_ids = item["relevant_doc_ids"]

        recall_scores.append(recall_at_k(retrieved_ids, relevant_ids, k))
        mrr_scores.append(reciprocal_rank(retrieved_ids, relevant_ids))

        # Generation evaluation
        answer = pipeline.generate(item["query"], retrieved)
        context = " ".join([doc["text"] for doc in retrieved])
        faith = evaluate_faithfulness(answer, context)
        faithfulness_scores.append(faith["score"])

    return {
        f"recall@{k}": sum(recall_scores) / len(recall_scores),
        "mrr": sum(mrr_scores) / len(mrr_scores),
        "faithfulness": sum(faithfulness_scores) / len(faithfulness_scores),
    }

Run this after every pipeline change — new chunking strategy, new embedding model, new prompt template — to ensure you are actually improving.

Metric targets

Metric	Poor	Acceptable	Good
Recall@10	Below 0.60	0.60-0.80	Above 0.80
MRR	Below 0.30	0.30-0.50	Above 0.50
NDCG@10	Below 0.50	0.50-0.70	Above 0.70
Faithfulness	Below 0.80	0.80-0.90	Above 0.90
Hallucination rate	Above 10%	5-10%	Below 5%

These are general targets. Your specific use case may have stricter requirements — medical or legal RAG systems should aim for faithfulness above 0.95 and hallucination rate below 2%.

Key takeaways

1. Measure retrieval and generation separately. A bad answer might be a retrieval failure (wrong chunks) or a generation failure (LLM hallucinating despite good chunks). You need to know which.
2. Recall@k is your most important retrieval metric. If the relevant chunk is not retrieved, nothing else matters.
3. Faithfulness is your most important generation metric. Users lose trust when the system confidently states something not in the source documents.
4. Use RAGAS for automated evaluation — it saves time and gives consistent, reproducible results.
5. Start with 50 labeled queries. You can always add more, but 50 catches most problems.