Evaluating RAG Pipelines: Precision, Recall, and Building a Test Set

You Can’t Improve What You Don’t Measure

Most teams build a RAG pipeline, eyeball a handful of answers, and ship it. That approach can’t tell you whether a chunking change actually improved retrieval quality, or whether it just changed which mistakes you’re making. Systematic evaluation fixes this.

Building a Test Set

Start with 30–50 real question/answer pairs pulled from actual user queries or support tickets, each with the source document(s) that should be retrieved:

test_set = [
    {
        "question": "What's our refund policy for annual plans?",
        "expected_source_ids": ["policy_doc_12"],
        "expected_answer_contains": ["30 days", "prorated"],
    },
    # ... 30-50 more real examples
]

Retrieval Metrics: Precision and Recall

def evaluate_retrieval(test_set, retriever, k=3):
    total_precision = 0
    total_recall = 0

    for item in test_set:
        retrieved = retriever.query(item["question"], top_k=k)
        retrieved_ids = {doc.id for doc in retrieved}
        expected_ids = set(item["expected_source_ids"])

        true_positives = len(retrieved_ids & expected_ids)
        precision = true_positives / len(retrieved_ids) if retrieved_ids else 0
        recall = true_positives / len(expected_ids) if expected_ids else 0

        total_precision += precision
        total_recall += recall

    n = len(test_set)
    return {"precision@k": total_precision / n, "recall@k": total_recall / n}

Answer Quality: Using an LLM as a Judge

def evaluate_answer_quality(question, generated_answer, expected_facts):
    judge_prompt = f"""
    Question: {question}
    Generated answer: {generated_answer}
    Required facts: {expected_facts}

    Does the answer correctly include all required facts, with no
    contradictions? Respond with only YES or NO.
    """
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": judge_prompt}],
    )
    return response.choices[0].message.content.strip() == "YES"

Tracking Results Over Changes

results = {
    "baseline_chunking_500": {"precision@3": 0.71, "recall@3": 0.64, "answer_quality": 0.78},
    "semantic_chunking": {"precision@3": 0.83, "recall@3": 0.79, "answer_quality": 0.88},
}

Run your test set against every meaningful pipeline change — chunk size, embedding model, retrieval count — and keep a running log. This turns “does this feel better” into an actual measurable comparison.

Common Failure Patterns Evaluation Surfaces

• High precision, low recall — retrieval is accurate but missing relevant chunks; try increasing top_k or improving chunk overlap.
• Low precision, high recall — too much irrelevant context is diluting the answer; tighten chunking or add a reranking step.
• Good retrieval, poor answer quality — the problem is in generation, not retrieval; revisit your system prompt.

Adding a Reranking Step

from cohere import Client

co = Client(api_key="...")

def rerank_results(query, candidates, top_n=3):
    results = co.rerank(query=query, documents=candidates, top_n=top_n)
    return [candidates[r.index] for r in results.results]

Conclusion

A RAG pipeline without an evaluation set is optimized by vibes. Building even a modest test set of real questions turns pipeline improvements into measurable, comparable experiments — and reliably surfaces whether your bottleneck is retrieval or generation, which changes what you should fix next.