RLHF Reward Model Evaluation: Accuracy, Ranking, and Human Agreement
Reward models in RLHF pipelines turn human preferences into scalar scores. Evaluate them with preference accuracy, Kendall tau ranking correlation, and human-AI agreement rate. A reward model with 65% pairwise accuracy can still produce a misaligned policy — measure all three dimensions before trusting it.
Reinforcement Learning from Human Feedback (RLHF) has become the standard method for aligning large language models with human preferences. The central component is the reward model: a neural network trained to score responses the way humans would rank them. If your reward model is miscalibrated, your policy optimization will faithfully maximize the wrong objective — a phenomenon called reward hacking.
This guide covers how to measure reward model quality across three dimensions: classification accuracy, ranking correlation, and agreement with human raters.
What Reward Models Do
A reward model takes a prompt and a response and outputs a scalar score. During RLHF training, a policy model generates responses, the reward model scores them, and a reinforcement learning algorithm (typically PPO) updates the policy to produce higher-scoring responses.
The reward model is trained on pairwise preference data: for each prompt, human raters compare two responses and indicate which they prefer. The model learns to assign a higher score to the preferred response.
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch
# Load a reward model (e.g., OpenAssistant's reward model)
model_name = "OpenAssistant/reward-model-deberta-v3-large-v2"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)
model.eval()
def score_response(prompt: str, response: str) -> float:
"""Score a single prompt-response pair."""
text = f"{prompt}\n{response}"
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512)
with torch.no_grad():
outputs = model(**inputs)
return outputs.logits[0].item()
prompt = "Explain the difference between TCP and UDP."
response_a = "TCP guarantees delivery and order. UDP is faster but unreliable."
response_b = "They're both protocols."
score_a = score_response(prompt, response_a)
score_b = score_response(prompt, response_b)
print(f"Score A: {score_a:.3f}, Score B: {score_b:.3f}")
print(f"Preferred: {'A' if score_a > score_b else 'B'}")Measuring Reward Model Accuracy
Pairwise Preference Accuracy
The most direct metric: given a held-out set of human preference pairs, how often does the reward model agree with the human judgment?
from datasets import load_dataset
from tqdm import tqdm
import numpy as np
def evaluate_pairwise_accuracy(model, tokenizer, dataset, max_examples=500):
"""
Evaluate reward model on pairwise preference data.
Dataset must have: prompt, chosen (preferred response), rejected (dispreferred).
"""
correct = 0
total = 0
for example in tqdm(dataset.select(range(min(max_examples, len(dataset))))):
prompt = example["prompt"]
chosen = example["chosen"]
rejected = example["rejected"]
score_chosen = score_response_batch(model, tokenizer, prompt, chosen)
score_rejected = score_response_batch(model, tokenizer, prompt, rejected)
if score_chosen > score_rejected:
correct += 1
total += 1
return correct / total
def score_response_batch(model, tokenizer, prompt: str, response: str) -> float:
text = f"Human: {prompt}\nAssistant: {response}"
inputs = tokenizer(
text,
return_tensors="pt",
truncation=True,
max_length=512,
padding=True
)
with torch.no_grad():
outputs = model(**inputs)
return outputs.logits[0].item()
# Load a standard preference dataset
dataset = load_dataset("Anthropic/hh-rlhf", split="test")
accuracy = evaluate_pairwise_accuracy(model, tokenizer, dataset)
print(f"Pairwise accuracy: {accuracy:.3f}")Confidence-Calibrated Accuracy
A reward model that is barely correct is almost as dangerous as one that is wrong. Measure accuracy stratified by the model's confidence (score margin between chosen and rejected):
def evaluate_accuracy_by_margin(model, tokenizer, dataset, max_examples=500):
margins = []
labels = [] # 1 = correct, 0 = wrong
for example in tqdm(dataset.select(range(min(max_examples, len(dataset))))):
score_chosen = score_response_batch(model, tokenizer, example["prompt"], example["chosen"])
score_rejected = score_response_batch(model, tokenizer, example["prompt"], example["rejected"])
margin = abs(score_chosen - score_rejected)
correct = int(score_chosen > score_rejected)
margins.append(margin)
labels.append(correct)
margins = np.array(margins)
labels = np.array(labels)
# Bin by margin quartile
quartiles = np.percentile(margins, [25, 50, 75])
bins = [0] + list(quartiles) + [float("inf")]
bin_labels = ["Q1 (low confidence)", "Q2", "Q3", "Q4 (high confidence)"]
print("Accuracy by confidence margin:")
for i in range(len(bins) - 1):
mask = (margins >= bins[i]) & (margins < bins[i+1])
if mask.sum() > 0:
acc = labels[mask].mean()
print(f" {bin_labels[i]}: {acc:.3f} (n={mask.sum()})")Ranking Correlation Metrics
Pairwise accuracy only tells you about binary comparisons. For ranking multiple responses to the same prompt, use correlation metrics.
Kendall Tau and Spearman Rho
from scipy import stats
def evaluate_ranking_correlation(model, tokenizer, multi_response_dataset):
"""
Evaluate on dataset where each prompt has multiple responses with human rankings.
multi_response_dataset: list of dicts with 'prompt', 'responses', 'human_ranks'
"""
all_tau = []
all_spearman = []
for example in multi_response_dataset:
prompt = example["prompt"]
responses = example["responses"]
human_ranks = example["human_ranks"] # lower = better (rank 1 = best)
# Get model scores (higher = better)
model_scores = [
score_response_batch(model, tokenizer, prompt, r)
for r in responses
]
# Convert scores to ranks (higher score → lower rank number)
model_ranks = stats.rankdata([-s for s in model_scores])
tau, p_tau = stats.kendalltau(human_ranks, model_ranks)
rho, p_rho = stats.spearmanr(human_ranks, model_ranks)
all_tau.append(tau)
all_spearman.append(rho)
return {
"kendall_tau_mean": np.mean(all_tau),
"kendall_tau_std": np.std(all_tau),
"spearman_rho_mean": np.mean(all_spearman),
"spearman_rho_std": np.std(all_spearman),
}
# Example usage with synthetic data
multi_response_data = [
{
"prompt": "Write a haiku about autumn.",
"responses": [
"Leaves fall gently down / Crisp air signals change ahead / Nature says goodbye",
"autumn is great fun / trees lose all of their leaves now / cold weather approaches",
"Red and gold and brown",
],
"human_ranks": [1, 2, 3], # first is best
}
]
metrics = evaluate_ranking_correlation(model, tokenizer, multi_response_data)
for k, v in metrics.items():
print(f"{k}: {v:.3f}")Interpreting these metrics:
- Kendall tau ranges from -1 to 1. Values above 0.4 indicate useful ranking signal.
- Spearman rho is more sensitive to large rank inversions. A value above 0.6 is good for a reward model.
- Both metrics near 0 mean the reward model ranks responses essentially at random relative to human judgment.
Human-AI Agreement Rate Measurement
Pairwise accuracy measures against a fixed test set. Human-AI agreement rate measures live agreement when humans and the model evaluate the same new examples.
import json
from dataclasses import dataclass
from typing import List, Optional
@dataclass
class EvaluationExample:
prompt: str
response_a: str
response_b: str
human_preference: str # "A", "B", or "tie"
model_score_a: Optional[float] = None
model_score_b: Optional[float] = None
model_preference: Optional[str] = None
def run_model_evaluation(model, tokenizer, examples: List[EvaluationExample]):
"""Score all examples and compute model preference."""
for ex in examples:
ex.model_score_a = score_response_batch(model, tokenizer, ex.prompt, ex.response_a)
ex.model_score_b = score_response_batch(model, tokenizer, ex.prompt, ex.response_b)
margin = abs(ex.model_score_a - ex.model_score_b)
if margin < 0.1: # threshold for "effectively tied"
ex.model_preference = "tie"
elif ex.model_score_a > ex.model_score_b:
ex.model_preference = "A"
else:
ex.model_preference = "B"
return examples
def compute_agreement_metrics(examples: List[EvaluationExample]):
total = len(examples)
agree = sum(1 for ex in examples if ex.human_preference == ex.model_preference)
# Agreement excluding ties
non_tie = [ex for ex in examples if ex.human_preference != "tie"]
non_tie_agree = sum(1 for ex in non_tie if ex.human_preference == ex.model_preference)
# Cohen's kappa
from sklearn.metrics import cohen_kappa_score
human_labels = [ex.human_preference for ex in examples]
model_labels = [ex.model_preference for ex in examples]
kappa = cohen_kappa_score(human_labels, model_labels)
return {
"overall_agreement": agree / total,
"non_tie_agreement": non_tie_agree / len(non_tie) if non_tie else None,
"cohen_kappa": kappa,
"n_examples": total,
"n_ties_human": sum(1 for ex in examples if ex.human_preference == "tie"),
"n_ties_model": sum(1 for ex in examples if ex.model_preference == "tie"),
}
# Simulate a human annotation batch
examples = [
EvaluationExample(
prompt="What is gradient descent?",
response_a="Gradient descent minimizes a function by iteratively moving in the direction of steepest descent.",
response_b="It's an optimization algorithm used in machine learning.",
human_preference="A",
),
# ... more examples from your annotation pipeline
]
scored = run_model_evaluation(model, tokenizer, examples)
metrics = compute_agreement_metrics(scored)
for k, v in metrics.items():
print(f"{k}: {v}")Interpreting Cohen's kappa:
- Below 0.2: slight agreement (model is not useful)
- 0.2–0.4: fair agreement (marginal utility)
- 0.4–0.6: moderate agreement (acceptable for many applications)
- 0.6–0.8: substantial agreement (good reward model)
- Above 0.8: almost perfect agreement (excellent)
Detecting Reward Hacking Signals
A reward model can score well on all three metrics above and still produce hacking when used for policy optimization. Watch for these patterns:
def analyze_score_distribution(model, tokenizer, prompts, policy_responses_over_time):
"""
Track reward score distribution across training steps.
Rising mean with falling variance often signals reward hacking.
"""
results = []
for step, responses in enumerate(policy_responses_over_time):
scores = []
for prompt, response in zip(prompts, responses):
score = score_response_batch(model, tokenizer, prompt, response)
scores.append(score)
results.append({
"step": step,
"mean_score": np.mean(scores),
"std_score": np.std(scores),
"max_score": np.max(scores),
"p95_score": np.percentile(scores, 95),
})
# Flag potential hacking: mean growing but std collapsing
if len(results) >= 2:
mean_growth = results[-1]["mean_score"] - results[0]["mean_score"]
std_change = results[-1]["std_score"] - results[0]["std_score"]
if mean_growth > 1.0 and std_change < -0.5:
print("WARNING: Possible reward hacking detected.")
print(f" Mean score grew by {mean_growth:.2f}")
print(f" Std dev collapsed by {abs(std_change):.2f}")
return resultsPutting It Together: A Reward Model Report Card
def reward_model_report(model, tokenizer, pairwise_dataset, multi_rank_dataset, annotation_examples):
print("=== Reward Model Evaluation Report ===\n")
# 1. Pairwise accuracy
acc = evaluate_pairwise_accuracy(model, tokenizer, pairwise_dataset, max_examples=1000)
print(f"Pairwise Accuracy: {acc:.3f}")
# 2. Ranking correlation
rank_metrics = evaluate_ranking_correlation(model, tokenizer, multi_rank_dataset)
print(f"Kendall Tau: {rank_metrics['kendall_tau_mean']:.3f} ± {rank_metrics['kendall_tau_std']:.3f}")
print(f"Spearman Rho: {rank_metrics['spearman_rho_mean']:.3f} ± {rank_metrics['spearman_rho_std']:.3f}")
# 3. Human agreement
scored = run_model_evaluation(model, tokenizer, annotation_examples)
agreement = compute_agreement_metrics(scored)
print(f"Human Agreement: {agreement['overall_agreement']:.3f}")
print(f"Cohen's Kappa: {agreement['cohen_kappa']:.3f}")
print("\nInterpretation:")
if acc < 0.60:
print(" [FAIL] Pairwise accuracy below 0.60 — model is not reliable for RLHF.")
elif acc < 0.70:
print(" [WARN] Pairwise accuracy below 0.70 — marginal, monitor closely.")
else:
print(" [OK] Pairwise accuracy acceptable.")
if agreement["cohen_kappa"] < 0.40:
print(" [FAIL] Cohen's kappa below 0.40 — poor human alignment.")
else:
print(" [OK] Human-AI agreement acceptable.")Key Takeaways
A reward model's job is to faithfully proxy human preferences so that policy optimization moves in the right direction. Measuring only pairwise accuracy misses two failure modes: bad ranking of multiple candidates and systematic disagreement with human raters on live data.
The evaluation stack to run before trusting any reward model in production:
- Pairwise accuracy on a held-out preference test set (target: >0.70)
- Kendall tau on multi-response ranking sets (target: >0.40)
- Human-AI agreement rate with Cohen's kappa on fresh annotations (target: >0.40)
- Score distribution monitoring during policy training to detect reward hacking early
No single metric is sufficient. Run all three, and treat any model that fails one with suspicion even if it passes the others.