ML Testing

AI Bias Testing: How to Detect and Measure Fairness Issues in ML Models

HelpMeTest

HelpMeTest

— 7 min read
AI Bias Testing: How to Detect and Measure Fairness Issues in ML Models

ML models trained on historical data inherit historical biases. A hiring model trained on past decisions might discriminate by gender. A credit scoring model might perpetuate racial bias from historical lending practices. A medical risk model might underperform for underrepresented demographic groups.

Testing for bias isn't optional — it's a basic quality gate. This guide shows you how to measure fairness in ML models and build automated tests that catch disparities before deployment.

What Is ML Bias?

Bias in ML means the model performs systematically differently across demographic groups. This can happen because:

  • Training data bias — historical data reflects past discrimination
  • Measurement bias — proxy features correlate with protected attributes
  • Selection bias — training data doesn't represent all subgroups equally
  • Label bias — ground truth labels themselves contain human bias

There's no universally "correct" definition of fairness — different metrics capture different aspects and can conflict with each other. The right metrics depend on your domain and the harms you're trying to prevent.

Core Fairness Metrics

Demographic Parity

The positive prediction rate should be similar across groups:

import numpy as np
import pandas as pd
from typing import Dict, List

def demographic_parity(y_pred: np.ndarray, protected_attr: np.ndarray) -> Dict:
    """
    Measure demographic parity: P(ŷ=1|A=a) should be similar for all groups a.
    
    Args:
        y_pred: Binary predictions (0 or 1)
        protected_attr: Group membership (categorical)
    
    Returns:
        Dict with positive prediction rates per group and disparity measures
    """
    results = {}
    groups = np.unique(protected_attr)
    
    positive_rates = {}
    for group in groups:
        mask = protected_attr == group
        positive_rates[group] = y_pred[mask].mean()
    
    max_rate = max(positive_rates.values())
    min_rate = min(positive_rates.values())
    
    return {
        "positive_rates": positive_rates,
        "max_disparity": max_rate - min_rate,
        "disparity_ratio": min_rate / max_rate if max_rate > 0 else 0,
        # "80% rule" / "four-fifths rule": ratio >= 0.8 is acceptable
        "passes_four_fifths_rule": (min_rate / max_rate) >= 0.8 if max_rate > 0 else True,
    }

Equal Opportunity

True positive rates (recall) should be equal across groups — important for high-stakes decisions:

def equal_opportunity(y_true: np.ndarray, y_pred: np.ndarray, 
                       protected_attr: np.ndarray) -> Dict:
    """
    Equal opportunity: P(ŷ=1|Y=1, A=a) should be similar for all groups.
    TPR (recall) parity — most important when false negatives cause harm.
    """
    groups = np.unique(protected_attr)
    tprs = {}
    
    for group in groups:
        mask = protected_attr == group
        group_true = y_true[mask]
        group_pred = y_pred[mask]
        
        positive_mask = group_true == 1
        if positive_mask.sum() == 0:
            tprs[group] = None  # Can't compute TPR without positives
            continue
        
        tprs[group] = group_pred[positive_mask].mean()
    
    valid_tprs = {k: v for k, v in tprs.items() if v is not None}
    
    if len(valid_tprs) < 2:
        return {"tprs": tprs, "max_disparity": None, "note": "Insufficient data"}
    
    max_tpr = max(valid_tprs.values())
    min_tpr = min(valid_tprs.values())
    
    return {
        "tprs": tprs,
        "max_disparity": max_tpr - min_tpr,
        "disparity_ratio": min_tpr / max_tpr if max_tpr > 0 else 0,
    }

Equalized Odds

Both TPR and FPR should be equal across groups:

def equalized_odds(y_true: np.ndarray, y_pred: np.ndarray,
                    protected_attr: np.ndarray) -> Dict:
    """
    Equalized odds: both TPR and FPR should be equal across groups.
    """
    from sklearn.metrics import confusion_matrix
    
    groups = np.unique(protected_attr)
    metrics = {}
    
    for group in groups:
        mask = protected_attr == group
        if mask.sum() < 10:  # Skip tiny groups
            continue
        
        tn, fp, fn, tp = confusion_matrix(
            y_true[mask], y_pred[mask], labels=[0, 1]
        ).ravel()
        
        metrics[group] = {
            "tpr": tp / (tp + fn) if (tp + fn) > 0 else 0,
            "fpr": fp / (fp + tn) if (fp + tn) > 0 else 0,
            "n": mask.sum(),
        }
    
    tprs = [m["tpr"] for m in metrics.values()]
    fprs = [m["fpr"] for m in metrics.values()]
    
    return {
        "group_metrics": metrics,
        "tpr_disparity": max(tprs) - min(tprs),
        "fpr_disparity": max(fprs) - min(fprs),
    }

Automated Bias Tests

# tests/fairness/test_model_fairness.py
import pytest
import pandas as pd
import numpy as np

@pytest.fixture
def predictions_with_demographics(trained_model, test_data):
    """Get model predictions alongside demographic attributes."""
    features = test_data.drop(["label", "gender", "race", "age_group"], axis=1)
    predictions = trained_model.predict(features)
    
    return pd.DataFrame({
        "y_true": test_data["label"],
        "y_pred": predictions,
        "gender": test_data["gender"],
        "race": test_data["race"],
        "age_group": test_data["age_group"],
    })

def test_demographic_parity_gender(predictions_with_demographics):
    """Positive prediction rate should not strongly differ by gender."""
    df = predictions_with_demographics
    
    result = demographic_parity(
        y_pred=df["y_pred"].values,
        protected_attr=df["gender"].values,
    )
    
    assert result["passes_four_fifths_rule"], \
        f"Gender demographic parity fails four-fifths rule:\n" \
        f"Positive rates: {result['positive_rates']}\n" \
        f"Disparity ratio: {result['disparity_ratio']:.3f}"

def test_equal_opportunity_race(predictions_with_demographics):
    """True positive rate should not strongly differ by race."""
    df = predictions_with_demographics
    
    result = equal_opportunity(
        y_true=df["y_true"].values,
        y_pred=df["y_pred"].values,
        protected_attr=df["race"].values,
    )
    
    if result.get("max_disparity") is None:
        pytest.skip("Insufficient data for one or more groups")
    
    assert result["max_disparity"] <= 0.10, \
        f"TPR disparity by race exceeds 10%:\n" \
        f"TPRs by group: {result['tprs']}\n" \
        f"Max disparity: {result['max_disparity']:.3f}"

def test_performance_parity_by_age_group(trained_model, test_data):
    """Model accuracy should not dramatically differ by age group."""
    from sklearn.metrics import roc_auc_score
    
    features = test_data.drop(["label", "age_group"], axis=1)
    probas = trained_model.predict_proba(features)[:, 1]
    
    aucs = {}
    for age_group in test_data["age_group"].unique():
        mask = test_data["age_group"] == age_group
        if mask.sum() < 50:
            continue
        
        aucs[age_group] = roc_auc_score(
            test_data["label"][mask], probas[mask]
        )
    
    min_auc = min(aucs.values())
    max_auc = max(aucs.values())
    
    assert max_auc - min_auc <= 0.10, \
        f"AUC varies by more than 10% across age groups:\n" \
        f"{aucs}\n" \
        f"Range: {min_auc:.3f} - {max_auc:.3f}"

Using Fairlearn

Fairlearn is the standard library for fairness-constrained ML in Python:

from fairlearn.metrics import MetricFrame, demographic_parity_difference
from fairlearn.metrics import equalized_odds_difference, selection_rate
from sklearn.metrics import accuracy_score, precision_score, recall_score
import pandas as pd

def create_fairness_report(y_true, y_pred, sensitive_features) -> pd.DataFrame:
    """Create a comprehensive fairness report using Fairlearn."""
    
    metrics = {
        "accuracy": accuracy_score,
        "precision": precision_score,
        "recall": recall_score,
        "selection_rate": selection_rate,
    }
    
    mf = MetricFrame(
        metrics=metrics,
        y_true=y_true,
        y_pred=y_pred,
        sensitive_features=sensitive_features,
    )
    
    return mf.by_group

# Aggregate fairness measures
def test_fairlearn_demographic_parity(y_true, y_pred, sensitive_features):
    """Use Fairlearn's built-in parity metrics."""
    dp_diff = demographic_parity_difference(
        y_true, y_pred, sensitive_features=sensitive_features
    )
    
    # Demographic parity difference: max - min across groups
    # 0 = perfect parity, closer to 0 is better
    assert dp_diff <= 0.10, \
        f"Demographic parity difference {dp_diff:.3f} exceeds threshold 0.10"
    
    eo_diff = equalized_odds_difference(
        y_true, y_pred, sensitive_features=sensitive_features
    )
    
    assert eo_diff <= 0.10, \
        f"Equalized odds difference {eo_diff:.3f} exceeds threshold 0.10"

Intersectional Fairness

Don't just test single protected attributes — test intersections:

def test_intersectional_fairness(predictions_with_demographics):
    """Test fairness at intersections of protected attributes."""
    df = predictions_with_demographics
    
    # Create intersectional groups
    df["gender_race"] = df["gender"] + "_" + df["race"]
    
    # Groups with at least 50 samples
    group_counts = df["gender_race"].value_counts()
    valid_groups = group_counts[group_counts >= 50].index
    
    df_filtered = df[df["gender_race"].isin(valid_groups)]
    
    result = demographic_parity(
        y_pred=df_filtered["y_pred"].values,
        protected_attr=df_filtered["gender_race"].values,
    )
    
    assert result["passes_four_fifths_rule"], \
        f"Intersectional fairness (gender × race) fails:\n" \
        f"Positive rates: {result['positive_rates']}"

Data Audit: Testing for Representation

Before model training, audit your training data for representation issues:

# tests/data/test_demographic_representation.py

def test_protected_group_representation(training_data):
    """Each protected group must have sufficient representation."""
    MINIMUM_GROUP_FRACTION = 0.05  # At least 5% of training data
    
    for attr in ["gender", "race", "age_group"]:
        group_fractions = training_data[attr].value_counts(normalize=True)
        
        for group, fraction in group_fractions.items():
            assert fraction >= MINIMUM_GROUP_FRACTION, \
                f"Group '{group}' in '{attr}' has only {fraction:.1%} representation — " \
                f"model may have poor performance and high variance for this group"

def test_label_rate_by_group(training_data):
    """Label rates should be examined for historical bias."""
    for attr in ["gender", "race"]:
        label_rates = training_data.groupby(attr)["label"].mean()
        
        max_rate = label_rates.max()
        min_rate = label_rates.min()
        
        # This is a warning, not a hard failure — disparate outcomes may be
        # the result of discrimination, but they may also be legitimate
        if max_rate / min_rate > 3:
            import warnings
            warnings.warn(
                f"Large label rate disparity in '{attr}': {min_rate:.1%} to {max_rate:.1%}. "
                f"Consider whether this reflects discrimination in historical data."
            )

Counterfactual Testing

Test what happens when you change only the protected attribute:

def test_counterfactual_fairness(model, sample_applicant):
    """
    Changing protected attributes should not change prediction
    when all other factors are identical.
    """
    features_male = {**sample_applicant, "gender": "male"}
    features_female = {**sample_applicant, "gender": "female"}
    
    pred_male = model.predict_proba([features_male])[0][1]
    pred_female = model.predict_proba([features_female])[0][1]
    
    # Predictions should be similar regardless of gender
    assert abs(pred_male - pred_female) <= 0.05, \
        f"Counterfactual gender disparity: " \
        f"male={pred_male:.3f}, female={pred_female:.3f}"

@pytest.mark.parametrize("race", ["white", "black", "hispanic", "asian"])
def test_counterfactual_race_fairness(model, sample_applicant, race):
    """Race change should not substantially affect prediction."""
    features = {**sample_applicant, "race": race}
    pred = model.predict_proba([features])[0][1]
    
    # Baseline (original applicant)
    baseline_pred = model.predict_proba([sample_applicant])[0][1]
    
    assert abs(pred - baseline_pred) <= 0.05, \
        f"Race '{race}' counterfactual shows disparity: " \
        f"baseline={baseline_pred:.3f}, with_race={pred:.3f}"

Bias Mitigation Testing

After applying bias mitigation, verify it worked:

from fairlearn.reductions import ExponentiatedGradient, DemographicParity
from sklearn.linear_model import LogisticRegression

def test_mitigated_model_passes_fairness(X_train, y_train, X_test, y_test, sensitive_train, sensitive_test):
    """Fairness-constrained model should pass fairness tests."""
    
    # Unmitigated baseline
    baseline = LogisticRegression()
    baseline.fit(X_train, y_train)
    
    # Fairness-constrained model
    constraint = DemographicParity()
    mitigator = ExponentiatedGradient(LogisticRegression(), constraint)
    mitigator.fit(X_train, y_train, sensitive_features=sensitive_train)
    
    mitigated_preds = mitigator.predict(X_test)
    
    dp_diff = demographic_parity_difference(
        y_test, mitigated_preds, sensitive_features=sensitive_test
    )
    
    # Mitigated model should have much lower parity difference
    assert dp_diff <= 0.05, \
        f"Mitigated model still has demographic parity difference {dp_diff:.3f}"

CI Integration for Fairness

# .github/workflows/fairness-tests.yml
name: ML Fairness Tests

on:
  push:
    paths: ['models/**', 'src/**', 'tests/fairness/**']

jobs:
  fairness:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Run data representation tests
        run: pytest tests/data/test_demographic_representation.py -v
      
      - name: Run fairness metric tests
        run: pytest tests/fairness/ -v --tb=short
      
      - name: Generate fairness report
        run: python scripts/generate_fairness_report.py
      
      - name: Upload fairness report
        uses: actions/upload-artifact@v4
        with:
          name: fairness-report
          path: reports/fairness/

Choosing Your Fairness Thresholds

There are no universal thresholds — they depend on domain and stakes:

Use Case Recommended Metrics Suggested Threshold
Credit scoring Demographic parity, Equal opportunity Four-fifths rule (0.8 ratio)
Hiring Demographic parity Four-fifths rule (EEOC standard)
Medical risk scoring Equal opportunity TPR disparity < 5%
Content recommendation Demographic parity Disparity < 10%
Fraud detection Equalized odds FPR disparity < 5%

Document your fairness thresholds and the reasoning behind them before training. Post-hoc threshold selection biases the evaluation.

Summary

AI bias testing requires systematic measurement, not good intentions:

  1. Choose metrics that match your harm model — TPR parity for false negatives; demographic parity for allocation fairness
  2. Test at data collection time — representation issues compound through training
  3. Test protected groups individually and intersectionally — intersectional bias is often worse
  4. Counterfactual testing — directly measure the effect of changing protected attributes
  5. Automate in CI — fairness can regress just like accuracy; test it continuously
  6. Document your thresholds — define what "acceptable" means before you measure

A model that passes accuracy tests but fails fairness tests is not production-ready. Use HelpMeTest to run fairness tests on every model update and get alerted when demographic disparities shift.

Read more

Start now free