Fine-Tuned Model Evaluation: Benchmarks, Accuracy Regression, and RLHF Testing

06 Oct 2026 — 7 min read

Fine-tuning a language model improves it on your specific task — but at a cost. Fine-tuned models can regress on capabilities the base model had, forget general knowledge, or overfit to training data in ways that hurt generalization. This guide covers how to evaluate fine-tuned models systematically: benchmark datasets, accuracy regression testing, RLHF evaluation, and setting up automated evaluation pipelines.

The Fine-Tuning Evaluation Problem

When you fine-tune a model, you're trading some general capability for domain-specific performance. The evaluation challenge is measuring both sides of this trade:

Things you want to improve:

Task-specific accuracy (your target domain)
Format compliance (following your output format)
Tone and style (matching your requirements)

Things you must not regress:

General reasoning ability
Instruction following on related tasks
Safety and refusal behaviors
Basic factual knowledge

Without systematic evaluation, you won't know what you broke until users complain.

Pre-Training Baselines

Before fine-tuning, record baselines on the base model. These become your regression targets:

# scripts/record_baseline.py
"""
Run this BEFORE fine-tuning. Records base model performance.
Save the output — you'll compare against it after fine-tuning.
"""
import json
import openai
from datetime import datetime

client = openai.OpenAI()

BASE_MODEL = "gpt-4o-mini"  # or your base model

def evaluate_on_benchmark(model: str, benchmark: list[dict]) -> dict:
    """Evaluate model on a benchmark dataset."""
    correct = 0
    results = []
    
    for case in benchmark:
        response = client.chat.completions.create(
            model=model,
            messages=[
                {"role": "system", "content": case.get("system", "")},
                {"role": "user", "content": case["input"]},
            ],
            temperature=0,
        )
        actual = response.choices[0].message.content.strip()
        expected = case["expected"]
        
        is_correct = evaluate_response(actual, expected, case.get("eval_type", "exact"))
        results.append({
            "id": case["id"],
            "input": case["input"],
            "expected": expected,
            "actual": actual,
            "correct": is_correct,
        })
        if is_correct:
            correct += 1
    
    return {
        "model": model,
        "timestamp": datetime.utcnow().isoformat(),
        "accuracy": correct / len(benchmark),
        "n": len(benchmark),
        "failures": [r for r in results if not r["correct"]],
    }

def evaluate_response(actual: str, expected: str, eval_type: str) -> bool:
    if eval_type == "exact":
        return actual.strip().lower() == expected.strip().lower()
    elif eval_type == "contains":
        return expected.lower() in actual.lower()
    elif eval_type == "json_match":
        try:
            actual_json = json.loads(actual)
            expected_json = json.loads(expected)
            return all(actual_json.get(k) == v for k, v in expected_json.items())
        except (json.JSONDecodeError, AttributeError):
            return False
    return False

Domain-Specific Benchmark Dataset

Create a benchmark that reflects your actual use case:

# evals/domain_benchmark.py
"""
Domain-specific benchmark for a customer support classifier fine-tune.
Target: classify support tickets into categories.
"""

DOMAIN_BENCHMARK = [
    # Happy path — common categories
    {
        "id": "cls-001",
        "input": "I can't log in to my account, the password reset email never arrived",
        "expected": "account_access",
        "eval_type": "exact",
        "category": "account",
    },
    {
        "id": "cls-002",
        "input": "I was charged twice this month, please refund the duplicate",
        "expected": "billing",
        "eval_type": "exact",
        "category": "billing",
    },
    {
        "id": "cls-003",
        "input": "The test runner hangs after 60 seconds and never completes",
        "expected": "bug_report",
        "eval_type": "exact",
        "category": "bugs",
    },
    # Edge cases — ambiguous inputs
    {
        "id": "cls-004",
        "input": "It's not working",
        "expected": "unclassified",  # Insufficient info
        "eval_type": "exact",
        "category": "edge_cases",
    },
    # Out-of-distribution — should refuse or classify as general
    {
        "id": "cls-005",
        "input": "How do I make pasta?",
        "expected": "out_of_scope",
        "eval_type": "exact",
        "category": "ood",
    },
]

# General capability benchmark — should NOT regress after fine-tuning
GENERAL_CAPABILITY_BENCHMARK = [
    {
        "id": "gen-001",
        "input": "What is 47 + 83?",
        "expected": "130",
        "eval_type": "contains",
        "category": "arithmetic",
    },
    {
        "id": "gen-002",
        "input": "Translate to French: The cat is on the mat",
        "expected": "Le chat est sur le tapis",
        "eval_type": "contains",
        "category": "translation",
    },
    {
        "id": "gen-003",
        "input": "What is the capital of Japan?",
        "expected": "Tokyo",
        "eval_type": "contains",
        "category": "factual",
    },
]

Detecting Catastrophic Forgetting

The biggest risk in fine-tuning is catastrophic forgetting — the model loses general capabilities. Test for it explicitly:

# tests/finetuning/test_catastrophic_forgetting.py
import pytest
from evals.domain_benchmark import GENERAL_CAPABILITY_BENCHMARK

REGRESSION_TOLERANCE = 0.05  # Allow at most 5% regression on general capabilities

def test_no_catastrophic_forgetting(base_model_scores: dict, finetuned_model_scores: dict):
    """
    Fine-tuned model must not regress more than 5% on general capabilities.
    Parametrized by model scores computed during evaluation.
    """
    
    base_accuracy = base_model_scores["general"]["accuracy"]
    ft_accuracy = finetuned_model_scores["general"]["accuracy"]
    
    regression = base_accuracy - ft_accuracy
    
    assert regression <= REGRESSION_TOLERANCE, (
        f"Catastrophic forgetting detected!\n"
        f"Base model accuracy: {base_accuracy:.1%}\n"
        f"Fine-tuned accuracy: {ft_accuracy:.1%}\n"
        f"Regression: {regression:.1%} (max allowed: {REGRESSION_TOLERANCE:.1%})\n"
        f"Failures: {finetuned_model_scores['general']['failures']}"
    )

def test_domain_accuracy_improved(base_model_scores: dict, finetuned_model_scores: dict):
    """Fine-tuning must improve domain task accuracy."""
    
    base_domain = base_model_scores["domain"]["accuracy"]
    ft_domain = finetuned_model_scores["domain"]["accuracy"]
    
    assert ft_domain > base_domain, (
        f"Fine-tuning didn't improve domain accuracy:\n"
        f"Base: {base_domain:.1%}, Fine-tuned: {ft_domain:.1%}"
    )
    
    improvement = ft_domain - base_domain
    print(f"Domain accuracy improvement: +{improvement:.1%}")

def test_edge_case_handling_maintained(base_model_scores: dict, finetuned_model_scores: dict):
    """Edge case handling must not regress — model should still identify ambiguous inputs."""
    
    base_edge = base_model_scores.get("edge_cases", {}).get("accuracy", 0)
    ft_edge = finetuned_model_scores.get("edge_cases", {}).get("accuracy", 0)
    
    assert ft_edge >= base_edge - REGRESSION_TOLERANCE, (
        f"Edge case handling regressed:\n"
        f"Base: {base_edge:.1%}, Fine-tuned: {ft_edge:.1%}"
    )

RLHF Evaluation

If you're using RLHF (Reinforcement Learning from Human Feedback) or DPO (Direct Preference Optimization), you need to evaluate preference alignment — does the model produce outputs humans prefer?

Building a Preference Evaluation Dataset

# evals/preference_benchmark.py
"""
Preference dataset for evaluating RLHF fine-tuning.
Each case has two responses; the model should prefer "chosen" over "rejected".
"""

PREFERENCE_BENCHMARK = [
    {
        "id": "pref-001",
        "prompt": "A user is frustrated that a test keeps failing",
        "chosen": "I understand your frustration. Let's diagnose this systematically — can you share the test output? A few common causes are timeout settings, flaky selectors, or environment differences.",
        "rejected": "Tests fail sometimes. Did you check the documentation?",
        "preference_reason": "chosen is empathetic, specific, and actionable",
    },
    {
        "id": "pref-002",
        "prompt": "Explain what HelpMeTest does in one sentence",
        "chosen": "HelpMeTest monitors your web application 24/7 by running plain-English tests continuously, alerting you before users notice issues.",
        "rejected": "HelpMeTest is a testing platform that helps you test your applications.",
        "preference_reason": "chosen is specific and value-focused; rejected is generic",
    },
]

def evaluate_preference_alignment(model: str, preference_dataset: list[dict]) -> dict:
    """
    Evaluate how often the model's output is more similar to 'chosen' than 'rejected'.
    Uses semantic similarity to compare model output to chosen/rejected.
    """
    from sentence_transformers import SentenceTransformer
    from sklearn.metrics.pairwise import cosine_similarity
    
    encoder = SentenceTransformer('all-MiniLM-L6-v2')
    correct = 0
    
    for case in preference_dataset:
        # Get model response
        response = get_model_response(model, case["prompt"])
        
        # Encode all three
        embeddings = encoder.encode([response, case["chosen"], case["rejected"]])
        
        sim_to_chosen = cosine_similarity([embeddings[0]], [embeddings[1]])[0][0]
        sim_to_rejected = cosine_similarity([embeddings[0]], [embeddings[2]])[0][0]
        
        if sim_to_chosen > sim_to_rejected:
            correct += 1
    
    return {
        "preference_alignment": correct / len(preference_dataset),
        "n": len(preference_dataset),
    }

Automated Win Rate Evaluation

Use an LLM judge to compare outputs at scale:

def compute_win_rate(
    base_model: str,
    finetuned_model: str,
    test_prompts: list[str],
    judge_model: str = "gpt-4o"
) -> dict:
    """
    Compare outputs from base model and fine-tuned model.
    Judge model decides which is better.
    """
    wins = {"base": 0, "finetuned": 0, "tie": 0}
    
    for prompt in test_prompts:
        base_response = get_model_response(base_model, prompt)
        ft_response = get_model_response(finetuned_model, prompt)
        
        judge_prompt = f"""You are comparing two AI responses to determine which is better.

PROMPT: {prompt}

RESPONSE A: {base_response}

RESPONSE B: {ft_response}

Which response is better? Consider: accuracy, helpfulness, clarity, tone.
Answer ONLY: "A", "B", or "TIE"."""
        
        verdict = get_model_response(judge_model, judge_prompt).strip()
        
        if verdict == "B":
            wins["finetuned"] += 1
        elif verdict == "A":
            wins["base"] += 1
        else:
            wins["tie"] += 1
    
    total = len(test_prompts)
    return {
        "win_rate_finetuned": wins["finetuned"] / total,
        "win_rate_base": wins["base"] / total,
        "tie_rate": wins["tie"] / total,
        "wins": wins,
        "total": total,
    }

def test_finetuned_has_positive_win_rate():
    """Fine-tuned model should win more often than base on domain prompts."""
    domain_prompts = [case["prompt"] for case in PREFERENCE_BENCHMARK]
    
    win_rates = compute_win_rate("gpt-4o-mini", "ft:gpt-4o-mini:org:model:id", domain_prompts)
    
    assert win_rates["win_rate_finetuned"] > win_rates["win_rate_base"], (
        f"Fine-tuned model doesn't outperform base:\n"
        f"Fine-tuned win rate: {win_rates['win_rate_finetuned']:.1%}\n"
        f"Base win rate: {win_rates['win_rate_base']:.1%}"
    )

Full Evaluation Pipeline

Tie it together in a runnable script:

# scripts/evaluate_finetuned_model.py
"""
Full evaluation pipeline for a fine-tuned model.
Run after every fine-tuning job before deploying.
"""
import argparse
import json
from pathlib import Path

from evals.domain_benchmark import DOMAIN_BENCHMARK, GENERAL_CAPABILITY_BENCHMARK
from evals.preference_benchmark import PREFERENCE_BENCHMARK, evaluate_preference_alignment, compute_win_rate

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--base-model", required=True)
    parser.add_argument("--finetuned-model", required=True)
    parser.add_argument("--output", default="eval_results.json")
    args = parser.parse_args()
    
    print(f"Evaluating: {args.finetuned_model}")
    print(f"vs baseline: {args.base_model}")
    print("="*60)
    
    # 1. Domain accuracy
    print("\n[1/4] Domain accuracy...")
    base_domain = evaluate_on_benchmark(args.base_model, DOMAIN_BENCHMARK)
    ft_domain = evaluate_on_benchmark(args.finetuned_model, DOMAIN_BENCHMARK)
    
    print(f"  Base:        {base_domain['accuracy']:.1%}")
    print(f"  Fine-tuned:  {ft_domain['accuracy']:.1%} ({ft_domain['accuracy'] - base_domain['accuracy']:+.1%})")
    
    # 2. General capability (catastrophic forgetting check)
    print("\n[2/4] General capability...")
    base_general = evaluate_on_benchmark(args.base_model, GENERAL_CAPABILITY_BENCHMARK)
    ft_general = evaluate_on_benchmark(args.finetuned_model, GENERAL_CAPABILITY_BENCHMARK)
    
    print(f"  Base:        {base_general['accuracy']:.1%}")
    print(f"  Fine-tuned:  {ft_general['accuracy']:.1%} ({ft_general['accuracy'] - base_general['accuracy']:+.1%})")
    
    # 3. Preference alignment
    print("\n[3/4] Preference alignment...")
    ft_preference = evaluate_preference_alignment(args.finetuned_model, PREFERENCE_BENCHMARK)
    print(f"  Preference alignment: {ft_preference['preference_alignment']:.1%}")
    
    # 4. Head-to-head win rate
    print("\n[4/4] Win rate (fine-tuned vs base)...")
    domain_prompts = [c["input"] for c in DOMAIN_BENCHMARK[:20]]  # Use subset for cost
    win_rates = compute_win_rate(args.base_model, args.finetuned_model, domain_prompts)
    print(f"  Fine-tuned wins: {win_rates['win_rate_finetuned']:.1%}")
    print(f"  Base wins:       {win_rates['win_rate_base']:.1%}")
    print(f"  Ties:            {win_rates['tie_rate']:.1%}")
    
    # Summary
    print("\n" + "="*60)
    print("EVALUATION SUMMARY")
    
    issues = []
    if ft_domain["accuracy"] <= base_domain["accuracy"]:
        issues.append(f"❌ Domain accuracy did not improve: {ft_domain['accuracy']:.1%}")
    
    regression = base_general["accuracy"] - ft_general["accuracy"]
    if regression > 0.05:
        issues.append(f"❌ Catastrophic forgetting: -{regression:.1%} on general tasks")
    
    if win_rates["win_rate_finetuned"] < 0.5:
        issues.append(f"❌ Negative win rate: fine-tuned wins only {win_rates['win_rate_finetuned']:.1%}")
    
    if issues:
        print("FAILED - Do not deploy")
        for issue in issues:
            print(f"  {issue}")
    else:
        print("✅ PASSED - Safe to deploy")
        print(f"  Domain accuracy: {base_domain['accuracy']:.1%} → {ft_domain['accuracy']:.1%}")
        print(f"  General regression: -{regression:.1%} (within 5% threshold)")
        print(f"  Win rate: {win_rates['win_rate_finetuned']:.1%}")

if __name__ == "__main__":
    main()

CI Integration

# .github/workflows/evaluate-finetuned.yml
name: Fine-Tuned Model Evaluation

on:
  workflow_dispatch:
    inputs:
      finetuned_model_id:
        description: 'OpenAI fine-tuned model ID'
        required: true
      base_model:
        description: 'Base model to compare against'
        default: 'gpt-4o-mini'

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v4
        with:
          python-version: '3.11'
      - run: pip install -r requirements-eval.txt
      
      - name: Run full evaluation
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: |
          python scripts/evaluate_finetuned_model.py \
            --base-model ${{ github.event.inputs.base_model }} \
            --finetuned-model ${{ github.event.inputs.finetuned_model_id }} \
            --output results/eval_${{ github.run_id }}.json
      
      - name: Upload evaluation results
        uses: actions/upload-artifact@v4
        with:
          name: eval-results-${{ github.run_id }}
          path: results/

Key Takeaways

Record baselines before fine-tuning — you can't detect regression without a reference point
Test general capabilities explicitly — catastrophic forgetting is real and common
Use domain benchmarks with edge cases — OOD handling and ambiguous inputs reveal overfitting
RLHF/DPO evaluation requires preference data — win rate and preference alignment metrics
Run automated win rate comparisons using LLM-as-judge at scale
Never deploy a fine-tuned model without checking that domain accuracy improved AND general capabilities didn't regress by more than your tolerance
Integrate evaluation into CI as a gate before deployment — not a post-hoc review

Fine-Tuned Model Evaluation: Benchmarks, Accuracy Regression, and RLHF Testing

The Fine-Tuning Evaluation Problem

Pre-Training Baselines

Domain-Specific Benchmark Dataset

Detecting Catastrophic Forgetting

RLHF Evaluation

Building a Preference Evaluation Dataset

Automated Win Rate Evaluation

Full Evaluation Pipeline

CI Integration

Key Takeaways

Read more

Fireworks AI Testing: Serverless Inference, Compound AI, and Function Calling

RAG Pipeline Testing: Retrieval Quality, Grounding, and Faithfulness Metrics

Prompt Engineering Testing: Building Regression Suites for Prompt Changes

Testing LLMs for Bias and Fairness: A Practical Engineering Guide