Testing

RAG Pipeline Testing: Chunk Quality, Retrieval Precision, and Hallucination Detection with RAGAS

HelpMeTest

28 Aug 2026 — 6 min read

Retrieval-Augmented Generation (RAG) pipelines are complex systems with multiple failure points: chunking can destroy context, retrieval can miss relevant documents, and the LLM can hallucinate facts not present in the retrieved chunks. Testing a RAG system requires evaluating each stage and the full pipeline together.

This guide covers how to test RAG pipelines systematically, with a focus on RAGAS — the leading evaluation framework purpose-built for RAG assessment.

RAG Pipeline Failure Modes

Before testing, understand what can go wrong:

Chunking failures:

Chunks too small → context is cut mid-sentence, retrieval misses relevant passages
Chunks too large → retrieve irrelevant content along with relevant content
No overlap between chunks → ideas that span chunk boundaries get lost

Retrieval failures:

Low precision → retrieved chunks are not relevant to the query
Low recall → relevant chunks are not retrieved
Embedding model mismatch → query and document embeddings in different semantic spaces

Generation failures:

Hallucination → LLM generates facts not present in retrieved chunks
Context ignoring → LLM uses training knowledge instead of retrieved context
Answer irrelevance → response doesn't address the original question

Setting Up RAGAS

RAGAS (Retrieval Augmented Generation Assessment) is an open-source framework that provides standardized metrics for RAG evaluation.

pip install ragas langchain openai

RAGAS evaluates four core metrics:

Faithfulness — is the answer supported by the retrieved context?
Answer Relevancy — does the answer address the question?
Context Precision — are the retrieved chunks relevant to the question?
Context Recall — does the retrieved context cover the ground truth answer?

Basic RAGAS Evaluation

from ragas import evaluate
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_precision,
    context_recall,
)
from datasets import Dataset

# Your RAG pipeline output
data = {
    "question": [
        "What is the refund policy?",
        "How do I cancel my subscription?",
        "What payment methods are accepted?"
    ],
    "answer": [
        "You can request a refund within 30 days of purchase.",
        "You can cancel anytime from your account settings.",
        "We accept Visa, Mastercard, and PayPal."
    ],
    "contexts": [
        [
            "All purchases come with a 30-day money-back guarantee. "
            "To request a refund, contact support@example.com."
        ],
        [
            "Subscriptions can be cancelled at any time. "
            "Log into your account, go to Settings > Subscription > Cancel."
        ],
        [
            "Accepted payment methods include Visa, Mastercard, American Express, "
            "PayPal, and Apple Pay."
        ]
    ],
    "ground_truth": [  # required for context_recall
        "Refund is available within 30 days of purchase.",
        "Cancel through account settings under Subscription.",
        "Visa, Mastercard, American Express, PayPal, and Apple Pay are accepted."
    ]
}

dataset = Dataset.from_dict(data)

results = evaluate(
    dataset,
    metrics=[
        faithfulness,
        answer_relevancy,
        context_precision,
        context_recall,
    ]
)

print(results)
# {'faithfulness': 0.97, 'answer_relevancy': 0.92, 
#  'context_precision': 0.89, 'context_recall': 0.78}

Running RAGAS in Tests

Wrap RAGAS evaluations in pytest assertions:

import pytest
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision
from datasets import Dataset
import your_rag_pipeline  # your actual RAG implementation

# Test questions with known good answers
TEST_CASES = [
    {
        "question": "What is the return policy?",
        "ground_truth": "Items can be returned within 30 days for a full refund."
    },
    {
        "question": "Is there a free trial?",
        "ground_truth": "Yes, a 14-day free trial is available with no credit card required."
    },
]

THRESHOLDS = {
    "faithfulness": 0.85,
    "answer_relevancy": 0.80,
    "context_precision": 0.75,
}

def run_rag(question: str) -> tuple[str, list[str]]:
    """Run your RAG pipeline, return (answer, retrieved_contexts)"""
    return your_rag_pipeline.query(question)

@pytest.mark.llm
def test_rag_quality_metrics():
    rows = {"question": [], "answer": [], "contexts": [], "ground_truth": []}

    for case in TEST_CASES:
        answer, contexts = run_rag(case["question"])
        rows["question"].append(case["question"])
        rows["answer"].append(answer)
        rows["contexts"].append(contexts)
        rows["ground_truth"].append(case["ground_truth"])

    dataset = Dataset.from_dict(rows)
    results = evaluate(dataset, metrics=[faithfulness, answer_relevancy, context_precision])

    failures = []
    for metric, threshold in THRESHOLDS.items():
        score = results[metric]
        if score < threshold:
            failures.append(f"{metric}: {score:.2f} < {threshold}")

    assert not failures, f"RAG quality below thresholds:\n" + "\n".join(failures)

Testing Faithfulness (Hallucination Detection)

Faithfulness is the most critical RAG metric. An unfaithful answer contains claims not supported by the retrieved context — in other words, hallucinations.

from ragas.metrics import faithfulness

@pytest.mark.llm
def test_no_hallucinations():
    """Answers must be grounded in retrieved context"""
    test_question = "What are the system requirements?"
    answer, contexts = run_rag(test_question)

    dataset = Dataset.from_dict({
        "question": [test_question],
        "answer": [answer],
        "contexts": [contexts],
        "ground_truth": [""]  # not needed for faithfulness
    })

    result = evaluate(dataset, metrics=[faithfulness])
    score = result["faithfulness"]

    assert score >= 0.90, (
        f"Hallucination detected! Faithfulness score: {score:.2f}\n"
        f"Answer: {answer}\n"
        f"Context: {contexts}"
    )

Manual Faithfulness Check

For debugging, you can build a simpler hallucination checker:

from openai import OpenAI

client = OpenAI()

def check_faithfulness(answer: str, context: str) -> dict:
    prompt = f"""Given this context and answer, identify any claims in the answer 
that are NOT supported by the context.

Context: {context}

Answer: {answer}

Return JSON: {{
    "is_faithful": true/false,
    "unsupported_claims": ["claim1", "claim2"],
    "explanation": "..."
}}"""

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": prompt}],
        response_format={"type": "json_object"}
    )
    return json.loads(response.choices[0].message.content)

def test_specific_answer_faithfulness():
    answer, contexts = run_rag("What is the pricing?")
    result = check_faithfulness(answer, "\n".join(contexts))

    assert result["is_faithful"], (
        f"Unfaithful answer detected.\n"
        f"Unsupported claims: {result['unsupported_claims']}\n"
        f"Explanation: {result['explanation']}"
    )

Testing Retrieval Quality

Poor retrieval is often the root cause of poor RAG answers. Test retrieval separately:

def test_retrieval_precision():
    """Retrieved chunks should be relevant to the query"""
    query = "cancellation policy"
    retrieved_chunks = your_rag_pipeline.retrieve(query, k=5)

    # Each chunk should mention cancellation-related content
    relevant_keywords = ["cancel", "cancellation", "terminate", "end subscription"]
    relevant_count = sum(
        1 for chunk in retrieved_chunks
        if any(kw in chunk.lower() for kw in relevant_keywords)
    )

    precision = relevant_count / len(retrieved_chunks)
    assert precision >= 0.6, f"Retrieval precision too low: {precision:.2f}"

def test_retrieval_recall():
    """All relevant documents should be retrieved"""
    # These chunks are known to be relevant (from your document set)
    KNOWN_RELEVANT_DOCS = [
        "doc_cancellation_policy",
        "doc_subscription_terms",
    ]

    query = "how to cancel"
    retrieved_chunks = your_rag_pipeline.retrieve(query, k=10)
    retrieved_ids = {chunk.metadata["doc_id"] for chunk in retrieved_chunks}

    recalled = sum(1 for doc in KNOWN_RELEVANT_DOCS if doc in retrieved_ids)
    recall = recalled / len(KNOWN_RELEVANT_DOCS)

    assert recall >= 0.8, f"Retrieval recall too low: {recall:.2f}"

Testing Chunk Quality

The chunking strategy dramatically affects retrieval. Test it explicitly:

from langchain.text_splitter import RecursiveCharacterTextSplitter

def test_chunk_size_distribution():
    """Chunks should be within acceptable size bounds"""
    splitter = RecursiveCharacterTextSplitter(
        chunk_size=512,
        chunk_overlap=50,
    )

    with open("docs/policy.txt") as f:
        text = f.read()

    chunks = splitter.split_text(text)
    chunk_lengths = [len(chunk.split()) for chunk in chunks]

    # No chunk should be under 50 words (too small = context lost)
    assert min(chunk_lengths) >= 50, f"Chunk too small: {min(chunk_lengths)} words"

    # No chunk should be over 600 words (too large = retrieval noise)
    assert max(chunk_lengths) <= 600, f"Chunk too large: {max(chunk_lengths)} words"

    # Most chunks should be near the target size
    avg_length = sum(chunk_lengths) / len(chunk_lengths)
    assert 100 <= avg_length <= 400, f"Average chunk size {avg_length} is outside acceptable range"

def test_no_mid_sentence_splits():
    """Chunks should not start or end mid-sentence"""
    splitter = RecursiveCharacterTextSplitter(chunk_size=512, chunk_overlap=50)
    chunks = splitter.split_text(open("docs/policy.txt").read())

    for i, chunk in enumerate(chunks):
        # Chunk should end with sentence-ending punctuation (or be the last chunk)
        if i < len(chunks) - 1:
            assert chunk.rstrip()[-1] in ".!?", (
                f"Chunk {i} ends mid-sentence: ...{chunk[-50:]}"
            )

End-to-End RAG Test Suite

Combine all the above into a comprehensive test suite:

# tests/test_rag_pipeline.py
import pytest
from datasets import Dataset
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision, context_recall

# Mark slow tests — skip in fast CI runs
pytestmark = pytest.mark.slow

class TestRAGPipeline:
    def setup_method(self):
        self.rag = RAGPipeline()

        # Representative test cases with ground truth
        self.test_cases = load_test_cases("tests/fixtures/rag_test_cases.json")

    def test_faithfulness_above_threshold(self):
        """No hallucinations — answers must be grounded in retrieved context"""
        results = self._evaluate_metrics([faithfulness])
        assert results["faithfulness"] >= 0.90

    def test_answer_relevancy_above_threshold(self):
        """Answers must address the question asked"""
        results = self._evaluate_metrics([answer_relevancy])
        assert results["answer_relevancy"] >= 0.85

    def test_context_precision_above_threshold(self):
        """Retrieved context must be relevant"""
        results = self._evaluate_metrics([context_precision])
        assert results["context_precision"] >= 0.75

    def test_context_recall_above_threshold(self):
        """Must retrieve enough relevant context"""
        results = self._evaluate_metrics([context_recall])
        assert results["context_recall"] >= 0.70

    def _evaluate_metrics(self, metrics):
        rows = {"question": [], "answer": [], "contexts": [], "ground_truth": []}
        for case in self.test_cases:
            answer, contexts = self.rag.query(case["question"])
            rows["question"].append(case["question"])
            rows["answer"].append(answer)
            rows["contexts"].append(contexts)
            rows["ground_truth"].append(case["ground_truth"])

        return evaluate(Dataset.from_dict(rows), metrics=metrics)

CI/CD Pipeline

name: RAG Tests
on:
  push:
    paths:
      - 'rag/**'
      - 'prompts/**'
      - 'docs/**'

jobs:
  rag-eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Install
        run: pip install ragas langchain openai datasets pytest

      - name: Run RAG tests
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: pytest tests/test_rag_pipeline.py -v -m "not slow"

      - name: Full evaluation (on merge to main)
        if: github.ref == 'refs/heads/main'
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: pytest tests/test_rag_pipeline.py -v

Baseline and Regression Tracking

Track your RAGAS scores over time to catch gradual degradation:

import json
from pathlib import Path
from datetime import date

def save_metrics_history(results: dict):
    history_file = Path("tests/metrics_history.json")
    history = json.loads(history_file.read_text()) if history_file.exists() else []
    history.append({
        "date": str(date.today()),
        **{k: round(v, 3) for k, v in results.items()}
    })
    history_file.write_text(json.dumps(history, indent=2))

def test_no_metric_regression():
    """Metrics should not drop more than 5% from last run"""
    history_file = Path("tests/metrics_history.json")
    if not history_file.exists():
        pytest.skip("No history yet")

    history = json.loads(history_file.read_text())
    last_run = history[-1]

    current = evaluate(...)  # run evaluation

    for metric in ["faithfulness", "answer_relevancy", "context_precision"]:
        drop = last_run[metric] - current[metric]
        assert drop <= 0.05, (
            f"{metric} dropped {drop:.2f} from {last_run[metric]:.2f} "
            f"to {current[metric]:.2f}"
        )

RAG testing is not optional — it's how you catch the hallucinations, retrieval failures, and chunk quality issues that silently degrade user experience. Start with RAGAS on a small test set, gate your CI pipeline on faithfulness and answer relevancy, and you'll catch 80% of RAG failures before they reach production.