AI Model Regression Testing: How to Catch Quality Drops Before Your Users Do
Model regression testing is the practice of verifying that an LLM upgrade, prompt change, or fine-tune didn't make things worse. It sounds obvious. Most teams don't do it.
The typical pattern: team upgrades from GPT-4 to GPT-4o to save costs. Everything seems fine in manual testing. They ship. Two weeks later, users report that the chatbot started giving shorter, less helpful answers. Nobody knows which change caused it because nobody measured before and after.
Regression testing for AI models is different from traditional regression testing. You're not checking that a function returns the exact same value — you're checking that quality didn't drop across a distribution of inputs. Here's how to build that.
What Triggers an AI Regression Test Run
Run regression tests before shipping any of these changes:
- Model version upgrades —
gpt-4-turbo→gpt-4o,claude-3-sonnet→claude-3-5-sonnet - Prompt changes — any modification to system prompts or few-shot examples
- Fine-tune updates — new fine-tuned model checkpoint
- Temperature or parameter changes — lowering temperature affects consistency, raising it affects creativity
- Provider switches — moving from OpenAI to Anthropic or vice versa
- Context window changes — different truncation behavior at token limits
Each of these can silently degrade quality in ways that aren't obvious from manual spot-checking.
Building a Regression Benchmark
A regression benchmark is a fixed dataset of inputs with expected quality characteristics. You measure the current model against this benchmark before and after changes.
The Benchmark Dataset
from dataclasses import dataclass
from typing import Callable
@dataclass
class BenchmarkCase:
id: str
category: str # Group cases for category-level analysis
input: str
# Instead of exact expected output, define what quality looks like
quality_checks: list[Callable[[str], bool]]
quality_descriptions: list[str] # Human-readable descriptions for reporting
BENCHMARK = [
BenchmarkCase(
id="sum-001",
category="summarization",
input="Summarize this quarterly report: [500-word financial text]",
quality_checks=[
lambda r: len(r.split()) >= 50, # Not too short
lambda r: len(r.split()) <= 200, # Not too long
lambda r: "revenue" in r.lower(), # Key term present
lambda r: any(c.isdigit() for c in r), # Has numbers
],
quality_descriptions=[
"Response has at least 50 words",
"Response has at most 200 words",
"Contains 'revenue'",
"Contains numeric data",
]
),
BenchmarkCase(
id="class-001",
category="classification",
input="Classify this support ticket as: billing, technical, account, or general\nTicket: My payment failed and my card was charged twice.",
quality_checks=[
lambda r: "billing" in r.lower(),
lambda r: len(r.split()) < 10, # Should be concise
],
quality_descriptions=[
"Correctly classifies as billing",
"Response is concise (under 10 words)",
]
),
]Running the Benchmark
import json
from datetime import datetime
def run_benchmark(llm_fn: Callable[[str], str], benchmark: list[BenchmarkCase]) -> dict:
"""
Run the benchmark and return a results dict.
llm_fn: callable that takes a prompt and returns a string response.
"""
results = {
"run_timestamp": datetime.now().isoformat(),
"cases": [],
"category_scores": {},
"overall_score": 0.0,
}
all_passed = []
category_passes = {}
for case in benchmark:
response = llm_fn(case.input)
checks = [fn(response) for fn in case.quality_checks]
case_score = sum(checks) / len(checks)
all_passed.extend(checks)
if case.category not in category_passes:
category_passes[case.category] = []
category_passes[case.category].extend(checks)
results["cases"].append({
"id": case.id,
"category": case.category,
"score": case_score,
"checks": [
{"description": desc, "passed": passed}
for desc, passed in zip(case.quality_descriptions, checks)
],
"response_length": len(response.split()),
})
results["overall_score"] = sum(all_passed) / len(all_passed)
results["category_scores"] = {
cat: sum(passes) / len(passes)
for cat, passes in category_passes.items()
}
return resultsComparing Baseline to Current
def compare_to_baseline(current_results: dict, baseline_path: str, threshold: float = 0.05) -> list[str]:
"""
Compare current run to baseline.
Returns list of regression descriptions. Empty list = no regressions.
threshold: allowed degradation (0.05 = 5% drop triggers regression)
"""
with open(baseline_path) as f:
baseline = json.load(f)
regressions = []
# Check overall score
score_delta = current_results["overall_score"] - baseline["overall_score"]
if score_delta < -threshold:
regressions.append(
f"Overall score regressed: {baseline['overall_score']:.3f} → "
f"{current_results['overall_score']:.3f} ({score_delta:+.3f})"
)
# Check per-category scores
for category, current_score in current_results["category_scores"].items():
baseline_score = baseline["category_scores"].get(category)
if baseline_score is None:
continue
delta = current_score - baseline_score
if delta < -threshold:
regressions.append(
f"Category '{category}' regressed: {baseline_score:.3f} → "
f"{current_score:.3f} ({delta:+.3f})"
)
# Check individual cases that newly fail
baseline_case_scores = {c["id"]: c["score"] for c in baseline["cases"]}
for case in current_results["cases"]:
baseline_case_score = baseline_case_scores.get(case["id"], 1.0)
delta = case["score"] - baseline_case_score
if delta < -0.25: # Individual case dropped by more than 25%
failed_checks = [
c["description"] for c in case["checks"] if not c["passed"]
]
regressions.append(
f"Case '{case['id']}' degraded significantly. "
f"Failing checks: {', '.join(failed_checks)}"
)
return regressions
def test_no_regression():
"""Pytest test — fails if regression detected."""
current = run_benchmark(my_llm, BENCHMARK)
regressions = compare_to_baseline(current, "benchmarks/baseline.json")
assert not regressions, "Model regressions detected:\n" + "\n".join(f"- {r}" for r in regressions)Semantic Similarity as a Regression Metric
For cases where exact quality checks are hard to define, use semantic similarity to detect drift from baseline outputs.
from sentence_transformers import SentenceTransformer
import numpy as np
model = SentenceTransformer('all-MiniLM-L6-v2')
def measure_semantic_drift(
inputs: list[str],
baseline_outputs: list[str],
current_outputs: list[str]
) -> float:
"""
Returns average cosine similarity between baseline and current outputs.
High similarity (>0.9) = outputs are consistent.
Low similarity (<0.7) = outputs have drifted significantly.
"""
similarities = []
for baseline, current in zip(baseline_outputs, current_outputs):
emb_baseline = model.encode(baseline)
emb_current = model.encode(current)
sim = np.dot(emb_baseline, emb_current) / (
np.linalg.norm(emb_baseline) * np.linalg.norm(emb_current)
)
similarities.append(float(sim))
return sum(similarities) / len(similarities)
def test_semantic_consistency():
"""Verify semantic meaning of outputs hasn't drifted."""
test_inputs = [case.input for case in BENCHMARK]
# Load baseline outputs
with open("benchmarks/baseline_outputs.json") as f:
baseline_outputs = json.load(f)
current_outputs = [my_llm(inp) for inp in test_inputs]
drift_score = measure_semantic_drift(test_inputs, baseline_outputs, current_outputs)
assert drift_score >= 0.80, \
f"Semantic drift detected. Similarity score: {drift_score:.3f} (threshold: 0.80)"Testing Specific Regression Categories
Different model changes cause different types of regressions. Test for the specific ones relevant to your change.
Format Regression (after prompt changes)
def test_output_format_regression():
"""Verify output format contract is preserved after prompt change."""
test_cases = [
("List 3 items", lambda r: len([l for l in r.split('\n') if l.strip()]) >= 3),
("Reply in JSON", lambda r: is_valid_json(r)),
("One sentence only", lambda r: r.count('.') <= 2),
]
format_failures = []
for prompt, check in test_cases:
response = my_llm(prompt)
if not check(response):
format_failures.append(f"Format violation for: '{prompt}'\nResponse: '{response[:100]}'")
assert not format_failures, "\n".join(format_failures)Latency Regression (after model upgrades)
import time
import statistics
def test_latency_regression():
"""New model shouldn't be significantly slower than baseline."""
test_prompts = [case.input for case in BENCHMARK[:10]] # Use subset for speed
latencies = []
for prompt in test_prompts:
start = time.time()
my_llm(prompt)
latencies.append(time.time() - start)
p95 = sorted(latencies)[int(0.95 * len(latencies))]
mean = statistics.mean(latencies)
# Load baseline latencies
with open("benchmarks/baseline_latency.json") as f:
baseline = json.load(f)
assert mean <= baseline["mean"] * 1.3, \
f"Latency regression: mean {mean:.2f}s vs baseline {baseline['mean']:.2f}s"
assert p95 <= baseline["p95"] * 1.5, \
f"P95 latency regression: {p95:.2f}s vs baseline {baseline['p95']:.2f}s"Cost Regression (after model upgrades)
def test_cost_regression():
"""Track token usage to catch unexpected cost increases."""
from openai import OpenAI
client = OpenAI()
total_tokens = 0
for case in BENCHMARK:
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": case.input}]
)
total_tokens += response.usage.total_tokens
with open("benchmarks/baseline_cost.json") as f:
baseline = json.load(f)
cost_increase = (total_tokens - baseline["total_tokens"]) / baseline["total_tokens"]
assert cost_increase <= 0.20, \
f"Token usage increased by {cost_increase:.1%} vs baseline. " \
f"Current: {total_tokens}, Baseline: {baseline['total_tokens']}"CI Pipeline for AI Regression Tests
# .github/workflows/ai-regression.yml
name: AI Model Regression Tests
on:
pull_request:
paths:
- 'prompts/**'
- 'src/llm/**'
- '.model-version'
jobs:
regression-test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Run fast regression tests
run: |
pytest tests/ai_regression/test_format.py -v
pytest tests/ai_regression/test_quality.py -v
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
- name: Run full benchmark comparison
run: python scripts/run_benchmark.py --compare-to benchmarks/baseline.json
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
- name: Upload new benchmark results
uses: actions/upload-artifact@v4
with:
name: benchmark-results
path: benchmarks/latest.jsonUpdating the Baseline
When you intentionally improve the model (better prompts, better fine-tune), update the baseline to reflect the new quality level.
def update_baseline(new_results: dict, baseline_path: str):
"""
Save new results as the baseline after intentional improvement.
Call this manually after confirming improvements are intentional.
"""
with open(baseline_path, 'w') as f:
json.dump(new_results, f, indent=2)
print(f"Baseline updated: {baseline_path}")
print(f"New overall score: {new_results['overall_score']:.3f}")
for cat, score in new_results['category_scores'].items():
print(f" {cat}: {score:.3f}")The discipline: never update the baseline automatically in CI. Only update it manually after reviewing the diff. This prevents "fixing" a regression by just moving the goalposts.
What a Regression Test Suite Tells You
After 3 months of running regression tests before every change:
- You know exactly which model version and prompt combination produces your current quality level
- Model upgrades have a quantified impact — "GPT-4o is 8% faster but 3% worse on summarization at our price point" is a real decision you can make
- Prompt changes are no longer guesswork — you see the score before and after
- New team members can change prompts confidently because tests catch unintended breakage
That's the actual value: turning "I think this is better" into "this is 4% better on summarization and 2% worse on classification, here's the tradeoff."
