ai-testing

AI Fairness Metrics: Measuring Demographic Parity, Equalized Odds, and Calibration

HelpMeTest

HelpMeTest

— 9 min read
AI Fairness Metrics: Measuring Demographic Parity, Equalized Odds, and Calibration

AI fairness is not a single number. Demographic parity, equalized odds, and calibration are distinct mathematical properties that are provably incompatible under realistic conditions. This guide explains what each metric actually measures, when to use which, how to compute them with Fairlearn and custom code, and what EU AI Act and US EEOC compliance requires in practice. Every definition includes a working Python implementation.

Key Takeaways

You cannot satisfy all fairness metrics simultaneously. Chouldechova's impossibility theorem shows demographic parity and calibration are incompatible when base rates differ across groups. Choose your metric based on the deployment context. The 80% rule is a legal threshold, not a fairness goal. EEOC's disparate impact rule (4/5ths rule) sets a floor at 0.80 disparate impact ratio. Aim for 0.90+. Individual fairness requires a similarity metric. You cannot test individual fairness without defining what "similar individuals" means for your domain — that definition is itself a policy choice.

In 2018, Amazon scrapped an AI hiring tool that had been in development for four years. The model had learned to downgrade resumes that included the word "women's" (as in "women's chess club") and penalized graduates of all-female colleges. The model was never given gender as a feature — it had learned it from proxies in 10 years of historical hiring data. The model was perfectly accurate at predicting who Amazon had historically hired. That was the problem.

The case illustrates why fairness metrics exist: accuracy measures how well a model predicts the past. Fairness metrics measure whether the model treats different groups equitably — a property that accuracy cannot capture.

Fairness Definitions and Their Trade-offs

Group fairness vs. individual fairness

The fundamental split in fairness definitions:

Group fairness (statistical fairness): Statistical properties of outcomes are equal across demographic groups.

Individual fairness (Dwork et al., 2012): Similar individuals receive similar predictions. Formally: if d(x₁, x₂) ≤ ε then |f(x₁) − f(x₂)| ≤ δ, where d is a domain-appropriate similarity metric.

Group and individual fairness can conflict. A model satisfying demographic parity may treat two superficially similar individuals differently if they belong to groups with different base rates.

The Impossibility Theorems

Three fairness criteria that practitioners commonly want simultaneously are mathematically incompatible when base rates differ across groups:

  1. Calibration (accuracy is equal across groups)
  2. Demographic parity (equal positive prediction rates)
  3. Equalized odds (equal TPR and FPR across groups)

Chouldechova (2017) proved: if base rates differ between groups, you cannot have both calibration and demographic parity. Kleinberg et al. (2017) proved related results for calibration and equalized odds.

The practical implication: you must choose which fairness property to optimize for, and document that choice and its trade-offs.

Scenario Recommended primary metric Rationale
Hiring Equalized odds False negatives (missed hires) and false positives (bad hires) both matter
Criminal recidivism risk Calibration Different base rates are legally defensible; uncalibrated scores are not
Credit scoring Demographic parity ratio (≥0.80) ECOA requires no disparate impact in credit decisions
Medical triage Equalized false negative rate Missing a high-risk patient in minority group is the critical error
Content moderation Equalized false positive rate Over-flagging minority voices is the critical failure mode

Demographic Parity

Demographic parity (also called statistical parity) requires that the positive prediction rate is equal across groups:

P(Ŷ=1 | A=0) = P(Ŷ=1 | A=1)

where A is the protected attribute and Ŷ is the predicted outcome.

import numpy as np
import pandas as pd
from scipy.stats import chi2_contingency, fisher_exact

def compute_demographic_parity(y_pred: np.ndarray, 
                                sensitive: np.ndarray) -> dict:
    """
    Compute demographic parity metrics for a binary classifier.
    
    Returns:
        selection_rates: per-group positive prediction rates
        demographic_parity_difference: max - min selection rate
        demographic_parity_ratio: min / max selection rate (80% rule)
        p_value: statistical significance of difference
    """
    groups = np.unique(sensitive)
    selection_rates = {}
    
    for group in groups:
        mask = sensitive == group
        selection_rates[group] = y_pred[mask].mean()
    
    rates = list(selection_rates.values())
    min_rate = min(rates)
    max_rate = max(rates)
    
    dpd = max_rate - min_rate
    dpr = min_rate / max_rate if max_rate > 0 else 1.0
    
    # Chi-square test for statistical significance
    # Contingency table: groups x (positive, negative)
    counts = []
    for group in groups:
        mask = sensitive == group
        n_pos = y_pred[mask].sum()
        n_neg = mask.sum() - n_pos
        counts.append([n_pos, n_neg])
    
    chi2, p_value, _, _ = chi2_contingency(counts)
    
    return {
        "selection_rates": selection_rates,
        "demographic_parity_difference": dpd,
        "demographic_parity_ratio": dpr,
        "chi2": chi2,
        "p_value": p_value,
        "statistically_significant": p_value < 0.05,
        "passes_80_percent_rule": dpr >= 0.80,
    }

def test_demographic_parity_hiring_model(y_pred, sensitive_race):
    result = compute_demographic_parity(y_pred, sensitive_race)
    
    print(f"Selection rates by race: {result['selection_rates']}")
    print(f"Disparate impact ratio: {result['demographic_parity_ratio']:.3f}")
    print(f"Statistical significance: p={result['p_value']:.4f}")
    
    assert result["passes_80_percent_rule"], \
        f"Disparate impact violation: DPR={result['demographic_parity_ratio']:.3f} < 0.80. " \
        f"Selection rates: {result['selection_rates']}"

When demographic parity is the wrong metric

Demographic parity is wrong when groups have legitimately different base rates for the predicted outcome. Example: a fraud detection model should not have equal fraud prediction rates across geographic regions if actual fraud rates differ. Forcing demographic parity would either over-flag low-fraud regions or under-flag high-fraud regions.

Equalized Odds

Equalized odds (Hardt et al., 2016) requires both the true positive rate (TPR) and false positive rate (FPR) to be equal across groups:

P(Ŷ=1 | Y=1, A=0) = P(Ŷ=1 | Y=1, A=1) [equal TPR] P(Ŷ=1 | Y=0, A=0) = P(Ŷ=1 | Y=0, A=1) [equal FPR]

Equalized opportunity is the weaker constraint requiring only equal TPR.

from sklearn.metrics import confusion_matrix

def compute_equalized_odds(y_true: np.ndarray, y_pred: np.ndarray,
                            sensitive: np.ndarray) -> dict:
    """
    Compute TPR and FPR per group. Returns equalized odds difference.
    """
    groups = np.unique(sensitive)
    group_metrics = {}
    
    for group in groups:
        mask = sensitive == group
        tn, fp, fn, tp = confusion_matrix(
            y_true[mask], y_pred[mask], labels=[0, 1]
        ).ravel()
        
        tpr = tp / (tp + fn) if (tp + fn) > 0 else 0.0
        fpr = fp / (fp + tn) if (fp + tn) > 0 else 0.0
        fnr = fn / (fn + tp) if (fn + tp) > 0 else 0.0
        
        group_metrics[group] = {"tpr": tpr, "fpr": fpr, "fnr": fnr}
    
    tpr_values = [m["tpr"] for m in group_metrics.values()]
    fpr_values = [m["fpr"] for m in group_metrics.values()]
    fnr_values = [m["fnr"] for m in group_metrics.values()]
    
    eod = max(
        max(tpr_values) - min(tpr_values),
        max(fpr_values) - min(fpr_values),
    )
    
    print("Per-group TPR/FPR:")
    for group, m in group_metrics.items():
        print(f"  {group}: TPR={m['tpr']:.3f}, FPR={m['fpr']:.3f}, FNR={m['fnr']:.3f}")
    
    return {
        "group_metrics": group_metrics,
        "equalized_odds_difference": eod,
        "tpr_difference": max(tpr_values) - min(tpr_values),
        "fpr_difference": max(fpr_values) - min(fpr_values),
        "fnr_difference": max(fnr_values) - min(fnr_values),
    }

def test_equalized_odds_recidivism_model(y_true, y_pred, sensitive_race):
    result = compute_equalized_odds(y_true, y_pred, sensitive_race)
    
    # COMPAS-style: FPR disparity is the key concern
    assert result["fpr_difference"] <= 0.10, \
        f"False positive rate disparity {result['fpr_difference']:.3f} exceeds 0.10. " \
        f"Model over-flags one demographic group."
    
    assert result["fnr_difference"] <= 0.10, \
        f"False negative rate disparity {result['fnr_difference']:.3f} exceeds 0.10."

Calibration Across Groups

A model is calibrated if its predicted probabilities match empirical frequencies. A model that predicts 70% probability of default should be correct about 70% of the time across all population groups.

from sklearn.calibration import calibration_curve
import numpy as np

def compute_group_calibration(y_true: np.ndarray, y_prob: np.ndarray,
                               sensitive: np.ndarray,
                               n_bins: int = 10) -> dict:
    """
    Compute Expected Calibration Error (ECE) per demographic group.
    High ECE disparity means model is better calibrated for some groups than others.
    """
    groups = np.unique(sensitive)
    group_ece = {}
    
    for group in groups:
        mask = sensitive == group
        if mask.sum() < 50:  # Skip small groups
            continue
        
        group_y_true = y_true[mask]
        group_y_prob = y_prob[mask]
        
        # Bin by predicted probability
        bin_boundaries = np.linspace(0, 1, n_bins + 1)
        ece = 0.0
        n_group = mask.sum()
        
        for i in range(n_bins):
            low, high = bin_boundaries[i], bin_boundaries[i + 1]
            bin_mask = (group_y_prob >= low) & (group_y_prob < high)
            
            if bin_mask.sum() == 0:
                continue
            
            bin_accuracy = group_y_true[bin_mask].mean()
            bin_confidence = group_y_prob[bin_mask].mean()
            bin_weight = bin_mask.sum() / n_group
            
            ece += bin_weight * abs(bin_accuracy - bin_confidence)
        
        group_ece[group] = ece
    
    ece_values = list(group_ece.values())
    calibration_gap = max(ece_values) - min(ece_values) if len(ece_values) >= 2 else 0.0
    
    print("ECE per group (lower is better):")
    for group, ece in sorted(group_ece.items()):
        print(f"  {group}: {ece:.4f}")
    print(f"Calibration gap: {calibration_gap:.4f}")
    
    return {
        "group_ece": group_ece,
        "calibration_gap": calibration_gap,
        "max_ece": max(ece_values) if ece_values else 0.0,
    }

def test_calibration_parity(y_true, y_prob, sensitive):
    result = compute_group_calibration(y_true, y_prob, sensitive)
    
    assert result["max_ece"] <= 0.10, \
        f"Worst-group ECE {result['max_ece']:.4f} exceeds 0.10"
    
    assert result["calibration_gap"] <= 0.05, \
        f"Calibration gap across groups {result['calibration_gap']:.4f} exceeds 0.05"

Individual Fairness

Individual fairness requires that similar individuals receive similar predictions. The key challenge is defining the similarity metric d(x₁, x₂) — this must be domain-specific and validated by domain experts.

from sklearn.metrics.pairwise import euclidean_distances
import numpy as np

def test_individual_fairness(model_fn, X: np.ndarray,
                               similarity_threshold: float = 0.1,
                               prediction_threshold: float = 0.05,
                               sample_size: int = 1000) -> dict:
    """
    Test individual fairness: if two individuals are similar by
    feature distance, their predictions should be close.
    
    similarity_threshold: max normalized feature distance to consider "similar"
    prediction_threshold: max allowed prediction difference for similar individuals
    """
    if len(X) > sample_size:
        idx = np.random.choice(len(X), sample_size, replace=False)
        X_sample = X[idx]
    else:
        X_sample = X
    
    # Normalize features
    from sklearn.preprocessing import StandardScaler
    scaler = StandardScaler()
    X_norm = scaler.fit_transform(X_sample)
    
    # Get predictions
    predictions = model_fn(X_sample)
    
    # Find similar pairs
    distances = euclidean_distances(X_norm)
    n = len(X_sample)
    n_features = X_sample.shape[1]
    
    # Normalize distance by sqrt(n_features) for comparability
    distances_norm = distances / np.sqrt(n_features)
    
    violations = []
    similar_pairs_count = 0
    
    for i in range(n):
        for j in range(i + 1, n):
            if distances_norm[i, j] <= similarity_threshold:
                similar_pairs_count += 1
                pred_diff = abs(predictions[i] - predictions[j])
                
                if pred_diff > prediction_threshold:
                    violations.append({
                        "i": i, "j": j,
                        "feature_distance": distances_norm[i, j],
                        "prediction_difference": pred_diff,
                    })
    
    violation_rate = len(violations) / similar_pairs_count if similar_pairs_count > 0 else 0.0
    
    print(f"Similar pairs found: {similar_pairs_count}")
    print(f"Individual fairness violations: {len(violations)} ({violation_rate:.2%})")
    
    assert violation_rate <= 0.05, \
        f"Individual fairness violation rate {violation_rate:.2%} exceeds 5%"
    
    return {
        "similar_pairs": similar_pairs_count,
        "violations": len(violations),
        "violation_rate": violation_rate,
    }

Fairlearn Metrics API

Fairlearn provides a production-ready implementation of all the above:

from fairlearn.metrics import (
    MetricFrame,
    demographic_parity_difference,
    demographic_parity_ratio,
    equalized_odds_difference,
    equalized_odds_ratio,
    true_positive_rate,
    false_positive_rate,
    false_negative_rate,
    selection_rate,
)
from sklearn.metrics import accuracy_score, roc_auc_score
import pandas as pd

def build_fairness_dashboard(y_true: np.ndarray, y_pred: np.ndarray,
                              y_prob: np.ndarray,
                              sensitive_features: pd.DataFrame) -> pd.DataFrame:
    """
    Build comprehensive fairness dashboard with all standard metrics.
    sensitive_features: DataFrame with one column per protected attribute
    """
    rows = []
    
    for col in sensitive_features.columns:
        sf = sensitive_features[col]
        
        mf = MetricFrame(
            metrics={
                "accuracy": accuracy_score,
                "selection_rate": selection_rate,
                "tpr": true_positive_rate,
                "fpr": false_positive_rate,
                "fnr": false_negative_rate,
            },
            y_true=y_true,
            y_pred=y_pred,
            sensitive_features=sf,
        )
        
        row = {
            "protected_attribute": col,
            "demographic_parity_difference": demographic_parity_difference(
                y_true, y_pred, sensitive_features=sf
            ),
            "demographic_parity_ratio": demographic_parity_ratio(
                y_true, y_pred, sensitive_features=sf
            ),
            "equalized_odds_difference": equalized_odds_difference(
                y_true, y_pred, sensitive_features=sf
            ),
            "accuracy_range": mf.difference(method="between_groups")["accuracy"],
            "worst_group_accuracy": mf.by_group["accuracy"].min(),
            "best_group_accuracy": mf.by_group["accuracy"].max(),
        }
        
        row["passes_eeoc"] = row["demographic_parity_ratio"] >= 0.80
        row["passes_strict"] = (
            row["demographic_parity_difference"] <= 0.10 and
            row["equalized_odds_difference"] <= 0.10
        )
        
        rows.append(row)
    
    dashboard = pd.DataFrame(rows)
    print(dashboard.to_string(index=False))
    
    return dashboard

Regulatory Context

EU AI Act

Under the EU AI Act (effective August 2024, enforcement 2026), high-risk AI systems must:

  • Document bias testing in the technical documentation (Annex IV)
  • Implement ongoing human oversight for high-risk decisions
  • Maintain logs for post-hoc auditing
  • Demonstrate conformity assessment before deployment

High-risk categories include AI in employment, credit, education, law enforcement, and biometric identification. For these systems, your fairness evaluation records are part of the legally required technical file.

import json
from datetime import datetime
from pathlib import Path

def generate_fairness_compliance_record(model_id: str,
                                         model_version: str,
                                         fairness_results: dict,
                                         regulatory_context: str = "EU_AI_ACT") -> dict:
    """
    Generate a compliance record for EU AI Act Annex IV documentation.
    """
    record = {
        "record_type": "fairness_evaluation",
        "regulatory_context": regulatory_context,
        "model_id": model_id,
        "model_version": model_version,
        "evaluation_date": datetime.utcnow().isoformat() + "Z",
        "evaluator": "automated-ci",
        
        "fairness_metrics": fairness_results,
        
        "eu_ai_act_compliance": {
            "high_risk_category": "employment_decisions",
            "annex_iv_section": "3.3",
            "conformity_assessment_required": True,
            "ongoing_monitoring": True,
            "human_oversight_implemented": True,
        },
        
        "thresholds_applied": {
            "demographic_parity_ratio": 0.80,
            "equalized_odds_difference": 0.10,
            "calibration_gap": 0.05,
        },
        
        "overall_assessment": (
            "PASS"
            if all([
                fairness_results.get("demographic_parity_ratio", 0) >= 0.80,
                fairness_results.get("equalized_odds_difference", 1) <= 0.10,
            ])
            else "FAIL"
        ),
    }
    
    # Save to append-only compliance store
    output_dir = Path(f"compliance/{regulatory_context.lower()}")
    output_dir.mkdir(parents=True, exist_ok=True)
    
    filename = f"fairness-{model_id}-{model_version}-{datetime.utcnow().strftime('%Y%m%d-%H%M%S')}.json"
    (output_dir / filename).write_text(json.dumps(record, indent=2))
    
    return record

US EEOC — 4/5ths rule

The EEOC's Uniform Guidelines on Employee Selection Procedures define adverse impact as a selection rate for a protected group that is less than 4/5ths (80%) of the rate for the highest-selected group.

def test_eeoc_adverse_impact(y_pred: np.ndarray, 
                               sensitive_race: np.ndarray,
                               sensitive_gender: np.ndarray) -> dict:
    """
    Test for adverse impact under EEOC 4/5ths rule.
    Must pass for both race and gender.
    """
    results = {}
    
    for attr_name, sensitive in [("race", sensitive_race), ("gender", sensitive_gender)]:
        result = compute_demographic_parity(y_pred, sensitive)
        
        eeoc_result = {
            "selection_rates": result["selection_rates"],
            "disparate_impact_ratio": result["demographic_parity_ratio"],
            "adverse_impact_detected": not result["passes_80_percent_rule"],
        }
        
        results[attr_name] = eeoc_result
        
        if eeoc_result["adverse_impact_detected"]:
            print(f"EEOC VIOLATION on {attr_name}: "
                  f"DI ratio={eeoc_result['disparate_impact_ratio']:.3f} < 0.80")
    
    violations = [attr for attr, r in results.items() if r["adverse_impact_detected"]]
    assert len(violations) == 0, \
        f"EEOC adverse impact violations: {violations}"
    
    return results

Fairness Metric Quick Reference

Metric Definition Ideal EEOC / legal threshold
Demographic Parity Ratio min(P(Ŷ=1|A)) / max(P(Ŷ=1|A)) 1.0 ≥ 0.80 (EEOC)
Demographic Parity Difference max - min positive prediction rate 0.0 ≤ 0.10 (guidance)
Equalized Odds Difference max(TPR gap, FPR gap) 0.0 ≤ 0.10 (guidance)
Calibration Gap max - min ECE across groups 0.0 ≤ 0.05 (guidance)
Accuracy Parity max - min accuracy across groups 0.0 Context-dependent
Individual Fairness Violation rate for similar pairs 0.0 ≤ 0.05

Fairness metrics alone do not make a system fair. They are measurement tools. What makes a system fair is a policy decision — which trade-offs are acceptable given the deployment context, the consequences of errors, and the applicable regulations.

Try HelpMeTest

Fairness metrics drift as your user population changes, as models are updated, and as deployment contexts evolve. HelpMeTest lets you run your fairness evaluation suite on a continuous schedule against production model endpoints, with alerting when demographic parity ratios or equalized odds metrics cross compliance thresholds. Visit https://helpmetest.com — usage-based pricing at $0.003/run, no base fee.

Read more

Start now free