Testing

DeepEval and Promptfoo: Automated LLM Evaluation Frameworks and CI/CD Integration

HelpMeTest

HelpMeTest

— 6 min read
DeepEval and Promptfoo: Automated LLM Evaluation Frameworks and CI/CD Integration

Shipping LLM features without automated evaluation is like deploying code without running tests. DeepEval and Promptfoo are the two leading open-source frameworks for automating LLM evaluation — they let you define quality metrics, run them against your prompts, and fail CI when quality drops.

This guide covers both tools: what they do well, when to use each, and how to integrate them into your deployment pipeline.

Why Dedicated Evaluation Frameworks?

You could write ad-hoc evaluation scripts with a judge LLM and some assertions. The problem: metrics implemented from scratch are inconsistent, lack established benchmarks, and don't integrate cleanly with CI/CD. DeepEval and Promptfoo solve this with:

  • Standardized metrics (G-Eval, faithfulness, answer relevancy, etc.)
  • Test case management — define cases in YAML/JSON, version them in git
  • CI/CD integration — exit codes, JUnit XML, GitHub Actions support
  • Regression tracking — compare results across runs
  • Cost control — built-in caching and model selection

DeepEval

DeepEval is a Python-based evaluation framework. It provides a pytest-like interface for writing LLM evaluation tests with a rich set of built-in metrics.

Installation

pip install deepeval

Basic Usage

from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import (
    AnswerRelevancyMetric,
    FaithfulnessMetric,
    ContextualPrecisionMetric,
    HallucinationMetric,
)

def test_customer_support_response():
    test_case = LLMTestCase(
        input="What is your return policy?",
        actual_output="You can return items within 30 days for a full refund.",
        expected_output="Returns are accepted within 30 days.",
        retrieval_context=[
            "All purchases come with a 30-day money-back guarantee.",
            "Contact support@example.com to initiate a return."
        ]
    )

    assert_test(test_case, [
        AnswerRelevancyMetric(threshold=0.8),
        FaithfulnessMetric(threshold=0.9),
        ContextualPrecisionMetric(threshold=0.7),
    ])

Run it like pytest:

deepeval test run tests/test_customer_support.py

Running with pytest

DeepEval integrates directly with pytest:

import pytest
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import AnswerRelevancyMetric, GEval
from deepeval.test_case import LLMTestCaseParams

# Custom metric using G-Eval (GPT-based evaluation with custom criteria)
professionalism_metric = GEval(
    name="Professionalism",
    criteria="The response is professional, courteous, and appropriate for customer service.",
    evaluation_params=[LLMTestCaseParams.ACTUAL_OUTPUT],
    threshold=0.7
)

@pytest.mark.parametrize("test_case", [
    LLMTestCase(
        input="I'm angry about my order being late!",
        actual_output=get_your_llm_response("I'm angry about my order being late!"),
        expected_output="Apology and assistance offered."
    ),
    LLMTestCase(
        input="Cancel my subscription NOW",
        actual_output=get_your_llm_response("Cancel my subscription NOW"),
        expected_output="Subscription cancellation acknowledged."
    ),
])
def test_customer_responses_are_professional(test_case):
    assert_test(test_case, [professionalism_metric])

DeepEval Metrics

DeepEval ships with 14+ ready-to-use metrics:

Metric Use Case
AnswerRelevancyMetric Does the answer address the question?
FaithfulnessMetric Is the answer supported by context?
ContextualPrecisionMetric Are retrieved chunks relevant?
ContextualRecallMetric Does retrieved context cover the answer?
HallucinationMetric Does the answer contradict context?
GEval Custom criteria via LLM judge
ToxicityMetric Does output contain harmful content?
BiasMetric Does output contain biases?
SummarizationMetric Is the summary accurate and complete?

Writing Custom Metrics

from deepeval.metrics import BaseMetric
from deepeval.test_case import LLMTestCase

class ResponseLengthMetric(BaseMetric):
    def __init__(self, max_words: int = 100, threshold: float = 1.0):
        self.max_words = max_words
        self.threshold = threshold
        self.name = "Response Length"

    def measure(self, test_case: LLMTestCase) -> float:
        word_count = len(test_case.actual_output.split())
        if word_count <= self.max_words:
            self.score = 1.0
            self.success = True
        else:
            self.score = self.max_words / word_count
            self.success = self.score >= self.threshold
        return self.score

    def is_successful(self) -> bool:
        return self.success

    @property
    def __name__(self):
        return self.name

Dataset-Level Evaluation

Evaluate a batch of test cases together:

from deepeval import evaluate
from deepeval.test_case import LLMTestCase
from deepeval.metrics import AnswerRelevancyMetric

test_cases = [
    LLMTestCase(
        input=q["question"],
        actual_output=your_llm(q["question"]),
        expected_output=q["expected"]
    )
    for q in load_test_dataset("tests/fixtures/qa_dataset.json")
]

results = evaluate(test_cases, [AnswerRelevancyMetric(threshold=0.8)])

# Access aggregate results
print(f"Pass rate: {results.passing_cases / results.total_cases:.0%}")

Promptfoo

Promptfoo takes a YAML-first, config-driven approach. You define prompts, providers, and test cases in YAML, and Promptfoo runs them — no Python required.

Installation

npm install -g promptfoo
# or
npx promptfoo@latest

Basic Configuration

# promptfooconfig.yaml
description: Customer Support Bot Evaluation

prompts:
  - id: support-prompt
    raw: |
      You are a customer support agent for Acme Corp.
      Be helpful, professional, and concise.
      
      Customer message: {{message}}

providers:
  - id: openai:gpt-4o-mini
  - id: openai:gpt-4o  # compare both models

tests:
  - vars:
      message: "What is your return policy?"
    assert:
      - type: contains
        value: "30 days"
      - type: llm-rubric
        value: "The response is helpful and mentions the return window"
      - type: not-contains
        value: "I don't know"

  - vars:
      message: "I want to cancel my subscription"
    assert:
      - type: llm-rubric
        value: "The agent offers to help with cancellation and asks for account details"
      - type: javascript
        value: "output.length < 300"  # concise response

  - vars:
      message: "Your service is terrible!"
    assert:
      - type: llm-rubric
        value: "The agent responds empathetically and offers to help resolve the issue"
      - type: not-regex
        value: "terrible|awful|bad"  # don't mirror negative language

Run it:

promptfoo eval

# View results in browser
promptfoo view

Promptfoo Assertion Types

Type Example
contains value: "return policy"
not-contains value: "I don't know"
regex value: "\\d+ days"
llm-rubric value: "Response is professional"
javascript value: "output.includes('30') && output.length < 200"
similarity value: "Returns accepted in 30 days", threshold: 0.8
factuality Check factual accuracy vs expected
answer-relevance Relevance to the prompt
context-faithfulness Answer grounded in context

Provider Comparison

Promptfoo excels at comparing multiple providers and models:

providers:
  - id: openai:gpt-4o
  - id: openai:gpt-4o-mini
  - id: anthropic:claude-3-5-sonnet-20241022
  - id: ollama:llama3  # local model

tests:
  - vars:
      question: "Explain quantum entanglement simply"
    assert:
      - type: llm-rubric
        value: "Explanation is accurate and accessible to a non-physicist"
      - type: javascript
        value: "output.length > 100 && output.length < 500"

The output table shows pass/fail for each provider × test combination — great for model selection decisions.

Prompt Variants

Test different prompt versions:

prompts:
  - id: concise
    raw: "Answer in 2 sentences: {{question}}"
  - id: detailed
    raw: "Provide a thorough answer to: {{question}}"
  - id: structured
    raw: |
      Answer this question: {{question}}
      Format: bullet points, max 5 bullets.

Promptfoo shows which prompt performs best across your test suite.

Regression Detection

# Track against a baseline
commandLineOptions:
  cache: true  # cache API responses for reproducibility

sharing:
  enabled: true  # share results URL for team review

defaultTest:
  options:
    # Fail if score drops more than 10% from baseline
    rubricPrompt: "Score this response 1-5..."
# Compare current run to a saved baseline
promptfoo eval --compare baseline_output.json

CI/CD Integration

DeepEval in GitHub Actions

name: LLM Eval
on: [push, pull_request]

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - run: pip install deepeval

      - name: Run DeepEval tests
        run: deepeval test run tests/eval/
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

      - name: Upload results
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: deepeval-results
          path: deepeval-results.json

Promptfoo in GitHub Actions

name: Promptfoo Eval
on: [pull_request]

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install Promptfoo
        run: npm install -g promptfoo

      - name: Run Promptfoo evaluation
        run: |
          promptfoo eval --output results.json
          promptfoo eval --output results.csv
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

      - name: Check pass rate
        run: |
          PASS_RATE=$(jq '.results.stats.passRate' results.json)
          echo "Pass rate: $PASS_RATE"
          python3 -c "import sys; sys.exit(0 if float('$PASS_RATE') >= 0.85 else 1)"

      - name: Comment PR with results
        uses: actions/github-script@v7
        if: always()
        with:
          script: |
            const fs = require('fs');
            const results = JSON.parse(fs.readFileSync('results.json'));
            const stats = results.results.stats;
            const body = `## LLM Eval Results
            - Pass rate: ${(stats.passRate * 100).toFixed(1)}%
            - Tests passed: ${stats.successes}/${stats.total}`;
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body
            });

DeepEval vs Promptfoo: When to Use Each

Scenario Tool
Python-heavy team DeepEval
YAML-first, non-Python team Promptfoo
RAG evaluation DeepEval (better RAG metrics)
Multi-model comparison Promptfoo (built for this)
Custom Python metrics DeepEval
Prompt A/B testing Promptfoo
pytest integration DeepEval
No-code evaluation Promptfoo

Many teams use both: Promptfoo for prompt iteration and model comparison during development, DeepEval for rigorous quality gates in CI.

Practical Evaluation Strategy

  1. Start small — 10-20 high-quality test cases beat 500 bad ones
  2. Cover your failure modes — edge cases, adversarial inputs, and known regressions
  3. Set realistic thresholds — 0.95 faithfulness is not achievable; start with 0.80
  4. Cache API responses — both tools support caching; use it to reduce costs
  5. Track trends — a score of 0.82 today is fine; a drop from 0.91 last week is a regression
# DeepEval: cache results
deepeval test run --cache

# Promptfoo: enable caching in config
# cache: true (in promptfooconfig.yaml)

DeepEval and Promptfoo turn LLM quality from a vague concern into a measurable, CI-gated property. Pick the one that fits your team's workflow — or use both. The goal is the same: catch prompt regressions before your users do.

Read more

Start now free