RAG Evaluation Frameworks: RAGAS, TruLens, and Custom Scorers

The RAG evaluation tooling landscape has exploded. RAGAS, TruLens, DeepEval, UpTrain, Arize Phoenix — there are at least a dozen frameworks claiming to be the definitive way to measure RAG quality. Evaluating the evaluators is its own non-trivial problem.

Most teams pick the first framework they encounter, discover it does not quite fit their use case, then either bolt on a second tool or build custom scorers. This guide is a direct comparison to help you make the right choice the first time — and know when to combine approaches.

Note: if you want a detailed walkthrough of using RAGAS and TruLens specifically, see our post on evaluating RAG with RAGAS and TruLens. This post focuses on selection guidance and how the frameworks compare across key dimensions.

The Core Evaluation Problem

Before comparing tools, be clear on what you are evaluating. RAG evaluation has three distinct concerns:

Development-time evaluation: You are tuning your pipeline — changing chunking strategies, swapping embedding models, adjusting retrieval parameters. You need fast, repeatable measurements to compare configurations.

Pre-deployment regression testing: Before releasing a change, verify that quality metrics have not degraded from your baseline. Needs CI integration and pass/fail thresholds.

Production monitoring: Ongoing measurement of live traffic quality. Needs sampling strategies, async processing, dashboards, and alerting.

No single framework excels at all three. Here is how each one maps.

RAGAS

RAGAS (Retrieval Augmented Generation Assessment) is the most widely adopted open-source framework. It was designed specifically for RAG and introduced a set of metrics that have become the de facto standard vocabulary for RAG evaluation.

What It Measures

RAGAS provides four core metrics:

• Faithfulness: Are claims in the answer supported by the retrieved context? (0-1)
• Answer Relevancy: Is the answer relevant to the question, regardless of the context? (0-1)
• Context Precision: Are the retrieved chunks ranked so more relevant ones appear first? (0-1)
• Context Recall: Does the retrieved context contain all information needed to answer? (requires ground truth)

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision, context_recall
from datasets import Dataset

# Your RAG pipeline outputs
data = {
    "question": ["What is the refund policy?", "How do I cancel my subscription?"],
    "answer": ["You can get a refund within 30 days.", "Go to Settings > Billing > Cancel."],
    "contexts": [
        ["Our refund policy allows returns within 30 days of purchase..."],
        ["To cancel, navigate to account Settings, then Billing, and click Cancel Subscription..."],
    ],
    "ground_truth": [
        "Refunds are available within 30 days.",
        "Cancel via Settings > Billing > Cancel Subscription."
    ]
}

dataset = Dataset.from_dict(data)
result = evaluate(dataset, metrics=[faithfulness, answer_relevancy, context_precision, context_recall])
print(result)
# {'faithfulness': 0.89, 'answer_relevancy': 0.92, 'context_precision': 0.87, 'context_recall': 0.84}

When RAGAS Is the Right Choice

Use RAGAS when:

• You want standardized, comparable metrics aligned with what the broader community uses
• You need quick setup for development-time evaluation
• Your team is doing academic-style comparison between RAG configurations
• You want ground truth-free metrics (faithfulness and answer relevancy do not require labeled data)

Avoid RAGAS when:

• You need production monitoring with real-time scoring (RAGAS is batch-oriented)
• Your LLM costs are a concern (it makes many LLM calls per evaluation)
• You need custom domain-specific metrics — extending RAGAS requires understanding its internals

Known Limitations

RAGAS uses an LLM (typically GPT-4) to score responses, which means its metrics are only as reliable as that judge. It can be inconsistent on borderline cases, and faithfulness scores can vary by ±0.05 between runs on the same dataset due to LLM non-determinism. Always run evaluations 2-3 times and average if precision matters.

Context recall requires ground truth answers, which means you need a labeled dataset. Without ground truth, you can only compute context precision, faithfulness, and answer relevancy.

TruLens

TruLens (now TruLens-Eval, part of the TruEra ecosystem) takes a different philosophical approach. It frames RAG evaluation as a feedback loop integrated into your application code, with a focus on the "RAG Triad": Answer Relevance, Context Relevance, and Groundedness (their term for faithfulness).

What Sets TruLens Apart

TruLens wraps your RAG application directly and instruments every call:

from trulens_eval import TruChain, Feedback, OpenAI as fOpenAI
from trulens_eval.feedback.provider import OpenAI
import numpy as np

openai = OpenAI()

# Define feedback functions
f_qa_relevance = Feedback(openai.relevance_with_cot_reasons).on_input_output()
f_context_relevance = (
    Feedback(openai.context_relevance_with_cot_reasons)
    .on_input()
    .on(TruChain.select_context())
    .aggregate(np.mean)
)
f_groundedness = (
    Feedback(openai.groundedness_measure_with_cot_reasons)
    .on(TruChain.select_context())
    .on_output()
)

# Wrap your LangChain RAG app
tru_recorder = TruChain(
    your_rag_chain,
    app_id="rag-v1",
    feedbacks=[f_qa_relevance, f_context_relevance, f_groundedness]
)

# Every call through tru_recorder is automatically evaluated
with tru_recorder as recording:
    response = your_rag_chain.invoke("What is the refund policy?")

TruLens stores results in a local SQLite database and provides a dashboard (tru.run_dashboard()) for exploration.

When TruLens Is the Right Choice

Use TruLens when:

• You want integrated tracing — see exactly which retrieved chunks caused a bad answer
• Your team iterates rapidly on the pipeline and needs immediate visual feedback
• You want to compare multiple pipeline versions (TruLens tracks app_id versions)
• You are already using LangChain or LlamaIndex (native integrations exist)

Avoid TruLens when:

• You need clean CI integration — the dashboard-first design makes headless CI more complex
• You want metrics that exactly match the broader RAGAS community benchmarks
• Your pipeline is not Python-based or not LangChain/LlamaIndex

TruLens vs RAGAS: Key Differences

DeepEval

DeepEval is a newer entrant that explicitly targets CI/CD integration. It is built around pytest and provides a natural test-writing experience for RAG evaluation:

from deepeval import assert_test
from deepeval.metrics import (
    AnswerRelevancyMetric,
    FaithfulnessMetric,
    ContextualPrecisionMetric,
    ContextualRecallMetric,
    HallucinationMetric
)
from deepeval.test_case import LLMTestCase

def test_rag_faithfulness():
    test_case = LLMTestCase(
        input="What is the cancellation policy?",
        actual_output="You can cancel anytime with no fees.",
        expected_output="Cancel anytime, no cancellation fees apply.",
        retrieval_context=[
            "Our cancellation policy: customers may cancel their subscription "
            "at any time without incurring cancellation fees."
        ]
    )
    
    metric = FaithfulnessMetric(threshold=0.8, model="gpt-4")
    assert_test(test_case, [metric])

def test_rag_no_hallucination():
    test_case = LLMTestCase(
        input="What is the free trial length?",
        actual_output="The free trial is 14 days.",
        context=["We offer a 30-day free trial for all new accounts."]
    )
    
    metric = HallucinationMetric(threshold=0.5)
    assert_test(test_case, [metric])

Run it like a normal pytest suite: pytest test_rag.py -v. DeepEval integrates with CI natively and has a companion SaaS (Confident AI) for storing and comparing results over time.

When DeepEval Is the Right Choice

Use DeepEval when:

• CI integration is your primary requirement
• Your team already works with pytest and wants RAG evaluation to feel like unit testing
• You want to mix RAG evaluation tests with other test types in the same runner
• You need per-test pass/fail results rather than aggregate scores

Avoid DeepEval when:

• You need the richest production monitoring — DeepEval is primarily test-time
• Your evaluation dataset is large and batch processing efficiency matters (RAGAS is faster for bulk evaluation)

Custom LLM-Judge Scorers

All three frameworks above use LLMs as judges under the hood. Sometimes you need more control — domain-specific criteria, proprietary rubrics, cost optimization by using a cheaper judge model, or evaluation criteria that do not map to existing metrics.

Building a custom LLM judge scorer is straightforward:

from openai import OpenAI
import json
from typing import Optional

client = OpenAI()

FAITHFULNESS_PROMPT = """You are evaluating an AI assistant's response for faithfulness.

Faithfulness means: every factual claim in the response is directly supported by the provided context.
A response is NOT faithful if it introduces facts not present in the context, even if those facts are true.

Context:
{context}

Response:
{response}

Evaluate the response on a scale of 0 to 1:
- 1.0: Every claim is directly supported by the context
- 0.7-0.9: Most claims supported; minor unsupported detail present
- 0.4-0.6: Some claims supported; significant unsupported content
- 0.0-0.3: Response mostly or entirely unsupported by context

Return JSON: {{"score": float, "reasoning": "string", "unsupported_claims": ["list"]}}"""

def score_faithfulness(
    context: str,
    response: str,
    model: str = "gpt-4o-mini",  # Use cheaper model for bulk scoring
    temperature: float = 0.0
) -> dict:
    prompt = FAITHFULNESS_PROMPT.format(context=context, response=response)
    
    completion = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}],
        temperature=temperature,
        response_format={"type": "json_object"}
    )
    
    return json.loads(completion.choices[0].message.content)

# Domain-specific metric: technical accuracy for a developer tool
TECHNICAL_ACCURACY_PROMPT = """You are a senior software engineer reviewing an AI assistant's technical response.

Question: {question}
Response: {response}
Retrieved Documentation: {context}

Evaluate technical accuracy specifically:
1. Are code examples syntactically correct for the language/framework mentioned?
2. Are API endpoints, method names, and parameters accurate per the documentation?
3. Are version-specific details correct?
4. Would following this response cause errors or unexpected behavior?

Return JSON: {{
    "technical_accuracy_score": float (0-1),
    "code_correct": bool,
    "api_accurate": bool,
    "issues_found": ["list of specific technical errors"]
}}"""

def score_technical_accuracy(question: str, response: str, context: str) -> dict:
    prompt = TECHNICAL_ACCURACY_PROMPT.format(
        question=question, response=response, context=context
    )
    completion = client.chat.completions.create(
        model="gpt-4o",  # Use better model for technical review
        messages=[{"role": "user", "content": prompt}],
        temperature=0.0,
        response_format={"type": "json_object"}
    )
    return json.loads(completion.choices[0].message.content)

When Custom Scorers Are the Right Choice

Build custom scorers when:

• You need domain-specific evaluation criteria (technical accuracy, legal correctness, medical safety)
• You want cost control — using gpt-4o-mini for bulk scoring instead of gpt-4
• You need consistent rubric wording that you can audit and revise
• You need to evaluate properties no standard framework covers (e.g., "response maintains appropriate professional tone for the context")

The main risk: Custom scorers require validation. Before relying on a custom judge, test it against human-labeled examples to verify it aligns with human judgment.

Selection Framework: A Decision Guide

Use this decision tree:

Step 1: What is your primary use case?

• Development-time comparison of configurations → RAGAS
• Integrated app tracing with visual exploration → TruLens
• CI/CD with pytest integration → DeepEval
• Production monitoring at scale → Custom scorers + observability platform

Step 2: Do you have labeled ground truth data?

• Yes → All frameworks are available; RAGAS context recall becomes available
• No → RAGAS (faithfulness + answer relevancy), TruLens, DeepEval all work

Step 3: How important is CI integration?

• Critical → DeepEval first, RAGAS as secondary
• Moderate → Any framework works
• Not needed → TruLens for exploration

Step 4: Do you have domain-specific requirements?

• Yes, well-defined → Custom scorers, possibly layered on top of a framework
• Yes, but standard RAG quality is the priority → Start with a framework, add custom metrics later

Combining Frameworks

The best production setups often combine tools:

Development: RAGAS batch evaluation for configuration comparison
CI gates: DeepEval pytest tests with pass/fail thresholds
Production: Custom LLM judges on sampled traffic, results to your observability platform
Debugging: TruLens for investigating specific failure cases

This is not over-engineering — each tool genuinely serves a different workflow.

CI Integration Patterns

Regardless of which framework you choose, the CI pattern is the same:

# .github/workflows/rag-evaluation.yml
name: RAG Quality Gate

on:
  pull_request:
    paths:
      - 'rag/**'
      - 'knowledge_base/**'
      - 'config/chunking.yaml'

jobs:
  rag-quality-gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Run RAG evaluation
        run: |
          # With DeepEval:
          pytest tests/rag_quality/ --tb=short
          
          # Or with RAGAS:
          python scripts/ragas_evaluate.py --fail-below faithfulness=0.80 answer_relevancy=0.75
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
      
      - name: Upload evaluation results
        if: always()
        uses: actions/upload-artifact@v3
        with:
          name: rag-evaluation-${{ github.sha }}
          path: evaluation-results.json

Set minimum thresholds based on your production baselines. A PR that drops faithfulness from 0.88 to 0.76 should fail the quality gate, just like a code regression fails unit tests.

HelpMeTest can add an end-to-end validation layer on top of your evaluation framework — running actual user-flow tests against your deployed RAG application on every release, verifying that the user experience (not just internal metrics) meets quality standards.

Summary

The right evaluation framework depends on your stage and workflow:

• RAGAS for standardized metrics and configuration comparison
• TruLens for integrated tracing and rapid iteration with visual feedback
• DeepEval for CI/CD-first teams who want evaluation as pytest tests
• Custom scorers for domain-specific requirements or cost-optimized production monitoring

Start with one framework and add complexity only when you hit its limits. The most important thing is measuring consistently — a single imperfect metric tracked over time is more valuable than five perfect metrics measured once.