Testing

Prompt Testing and LLM Output Evaluation: Consistency, Factuality, and Quality

HelpMeTest

19 Aug 2026 — 5 min read

You can't ship LLM-based features without testing them. Unlike traditional software where the same input always produces the same output, LLMs are probabilistic — the same prompt can return different results across runs, models, and versions. Without structured prompt testing, you're deploying blind.

This guide covers how to systematically test LLM prompts for consistency, factuality, and quality, and how to build a regression testing pipeline that catches prompt degradation before it reaches production.

Why Prompt Testing Is Different

Traditional unit tests are deterministic: given input X, assert output equals Y. That doesn't work for LLMs. Instead, you need to:

Assert properties of the output, not exact values
Test distributions across multiple runs
Evaluate quality using metrics and judges
Track regressions when prompts or models change

The mental model shift: instead of testing that output == expected, you test that output satisfies constraints (format, factuality, tone) and meets quality thresholds (relevance score > 0.8).

Core Evaluation Dimensions

Every LLM evaluation covers some combination of:

Dimension	Question	How to Measure
Correctness	Is the answer factually accurate?	Compare to ground truth, use judge LLM
Relevance	Does it answer what was asked?	Cosine similarity, judge LLM
Consistency	Same prompt → similar answers?	Run N times, measure variance
Format	Does it follow the required structure?	Regex, JSON schema validation
Completeness	Does it cover all required points?	Checklist evaluation
Harmlessness	Does it avoid toxic/dangerous content?	Classifier, judge LLM
Conciseness	Is it appropriately brief?	Token count, judge LLM

Deterministic Tests First

Before using LLM judges or embedding similarity, start with the deterministic checks you can always rely on:

Format Validation

import json
import pytest
from openai import OpenAI

client = OpenAI()

def get_product_info(product_name: str) -> dict:
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": "Return product info as JSON with keys: name, price, category, in_stock (boolean)."},
            {"role": "user", "content": f"Product: {product_name}"}
        ],
        response_format={"type": "json_object"}
    )
    return json.loads(response.choices[0].message.content)

def test_product_info_format():
    result = get_product_info("MacBook Pro 16")

    # Format assertions — deterministic
    assert isinstance(result, dict)
    assert "name" in result
    assert "price" in result
    assert "category" in result
    assert "in_stock" in result
    assert isinstance(result["in_stock"], bool)
    assert isinstance(result["price"], (int, float, str))

Constraint Checking

def test_summary_length():
    """Summary should be under 100 words"""
    text = "..." * 500  # long input text
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": "Summarize in under 100 words."},
            {"role": "user", "content": text}
        ]
    )
    summary = response.choices[0].message.content
    word_count = len(summary.split())
    assert word_count <= 100, f"Summary was {word_count} words, expected <= 100"

def test_no_pii_in_response():
    """Verify PII is redacted"""
    import re
    response_text = get_llm_response("Describe John Smith at john@example.com")
    email_pattern = r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b'
    assert not re.search(email_pattern, response_text), "Response contains email address"

Consistency Testing

LLMs are non-deterministic. For critical prompts, test consistency across multiple runs:

import numpy as np
from sentence_transformers import SentenceTransformer

def test_answer_consistency():
    """The same question should produce semantically similar answers"""
    model = SentenceTransformer('all-MiniLM-L6-v2')
    question = "What is the capital of France?"

    # Run the prompt 5 times
    answers = []
    for _ in range(5):
        response = client.chat.completions.create(
            model="gpt-4o",
            messages=[{"role": "user", "content": question}],
            temperature=0.7
        )
        answers.append(response.choices[0].message.content)

    # Encode all answers
    embeddings = model.encode(answers)

    # Compute pairwise cosine similarity
    from sklearn.metrics.pairwise import cosine_similarity
    similarity_matrix = cosine_similarity(embeddings)

    # Average similarity should be high for factual questions
    avg_similarity = np.mean(similarity_matrix[np.triu_indices_from(similarity_matrix, k=1)])
    assert avg_similarity > 0.85, f"Answer consistency too low: {avg_similarity:.2f}"

Set temperature=0 for deterministic outputs when consistency is critical.

LLM-as-Judge Evaluation

For subjective quality assessments, use a more capable LLM to evaluate outputs:

def evaluate_with_judge(question: str, answer: str, criteria: str) -> dict:
    """Use GPT-4 to evaluate answer quality"""
    judge_prompt = f"""Evaluate this answer on a scale of 1-5.

Question: {question}
Answer: {answer}
Evaluation criteria: {criteria}

Respond with JSON: {{"score": <1-5>, "reasoning": "<explanation>"}}"""

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": judge_prompt}],
        response_format={"type": "json_object"}
    )
    return json.loads(response.choices[0].message.content)

def test_answer_quality():
    question = "Explain how database indexing works"
    answer = get_llm_response(question)

    evaluation = evaluate_with_judge(
        question=question,
        answer=answer,
        criteria="Technical accuracy, clarity, and appropriate depth for a software engineer"
    )

    assert evaluation["score"] >= 4, (
        f"Answer quality score {evaluation['score']}/5 is below threshold. "
        f"Reasoning: {evaluation['reasoning']}"
    )

The LLM-as-judge pattern is powerful but has costs:

Double the API calls (one for generation, one for evaluation)
Judge can be wrong or biased
Use it for quality audits, not every-run assertions

Factuality Testing

For factual questions, test against ground truth:

from sentence_transformers import SentenceTransformer, util

GROUND_TRUTH = {
    "What year was Python created?": "Python was created in 1991 by Guido van Rossum",
    "What does HTTP stand for?": "Hypertext Transfer Protocol",
}

def test_factual_accuracy():
    model = SentenceTransformer('all-MiniLM-L6-v2')

    for question, expected in GROUND_TRUTH.items():
        answer = get_llm_response(question)

        # Semantic similarity to ground truth
        embeddings = model.encode([answer, expected])
        similarity = util.cos_sim(embeddings[0], embeddings[1]).item()

        assert similarity > 0.8, (
            f"Answer for '{question}' is factually divergent.\n"
            f"Expected: {expected}\n"
            f"Got: {answer}\n"
            f"Similarity: {similarity:.2f}"
        )

Regression Testing: Catching Prompt Degradation

When you update prompts or switch models, you need regression tests that catch quality drops:

# tests/test_prompt_regression.py
import json
from pathlib import Path

# Load baseline results captured from the approved prompt version
BASELINE_FILE = Path("tests/baselines/prompt_v1_results.json")

def test_no_regression_vs_baseline():
    if not BASELINE_FILE.exists():
        pytest.skip("No baseline file — generate with: pytest --generate-baseline")

    baseline = json.loads(BASELINE_FILE.read_text())
    model = SentenceTransformer('all-MiniLM-L6-v2')

    failures = []
    for test_case in baseline["cases"]:
        current_answer = get_llm_response(test_case["question"])
        similarity = compute_similarity(model, current_answer, test_case["answer"])

        if similarity < 0.75:
            failures.append({
                "question": test_case["question"],
                "baseline": test_case["answer"],
                "current": current_answer,
                "similarity": similarity
            })

    assert not failures, f"Regression detected in {len(failures)} cases:\n{json.dumps(failures, indent=2)}"

Generate baselines when the current output quality is approved:

# pytest --generate-baseline
@pytest.fixture(autouse=True)
def generate_baseline(request):
    if request.config.getoption("--generate-baseline"):
        results = {"cases": []}
        for question in TEST_QUESTIONS:
            results["cases"].append({
                "question": question,
                "answer": get_llm_response(question)
            })
        BASELINE_FILE.write_text(json.dumps(results, indent=2))
        pytest.skip("Baseline generated")

CI/CD Integration

Run prompt tests on every PR:

# .github/workflows/prompt-tests.yml
name: Prompt Tests
on: [pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Install dependencies
        run: pip install pytest openai sentence-transformers scikit-learn

      - name: Run deterministic tests
        run: pytest tests/test_format.py tests/test_constraints.py -v
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

      - name: Run quality tests (nightly only)
        if: github.event_name == 'schedule'
        run: pytest tests/test_quality.py -v
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

Run expensive LLM-as-judge tests nightly, not on every PR. Run format and constraint tests on every commit.

Cost Management

LLM evaluation calls money. Keep costs down:

Cache deterministic calls — same input, same expected output, cache the result
Use cheaper models for simple checks — gpt-4o-mini for format validation
Sample in CI — don't run all 500 test cases every PR, run a representative sample
Gate expensive tests — LLM judge tests only on merge to main

# Use a cheaper model for format checking
JUDGE_MODEL = os.getenv("JUDGE_MODEL", "gpt-4o-mini")  # override in CI for cost

def test_response_format():
    response = client.chat.completions.create(
        model="gpt-4o-mini",  # cheap for format-only checks
        ...
    )

Practical Test Suite Structure

tests/
  prompts/
    test_format.py          # JSON schema, regex — fast, cheap, always run
    test_constraints.py     # length, no PII, required fields — fast, cheap
    test_consistency.py     # multi-run variance — medium cost, run on PR
    test_factuality.py      # ground truth comparison — medium cost
    test_quality.py         # LLM judge — expensive, run nightly
    test_regression.py      # vs baseline — medium cost, run on merge
  baselines/
    prompt_v2_results.json  # approved baseline

Prompt testing isn't optional for production AI features. Start with deterministic format and constraint tests — they're free, fast, and catch the most common failures. Add consistency and factuality tests for your critical paths. Use LLM-as-judge sparingly for quality audits. Build a baseline once your prompts are approved, then protect it with regression tests.