Testing

Testing Facial Recognition Accuracy: A Practical Guide for ML/AI QA Engineers

03 Oct 2026 — 7 min read

Facial recognition systems fail in ways that are hard to catch without a disciplined test strategy. A model that hits 99% accuracy on a clean benchmark can still misidentify faces across demographic groups, degrade silently when lighting conditions shift, or drift over model updates. This guide walks through how to test facial recognition systems rigorously — from benchmark datasets and accuracy metrics to bias audits and regression harnesses against cloud APIs.

Benchmark Datasets

Two datasets dominate academic and production evaluation:

Labeled Faces in the Wild (LFW) contains 13,233 face images of 5,749 people collected from the web. It is a 1:1 verification benchmark — given two images, does this pair show the same person? LFW is easy to saturate (state-of-the-art systems exceed 99.8%), but it is still useful for sanity checks and establishing a baseline.

IJB-C (IARPA Janus Benchmark C) is a harder, more realistic dataset with 3,531 subjects, mixed media (still images + video frames), and intentional pose/illumination variation. It uses a 1:N identification protocol as well as verification, making it appropriate for production-grade evaluation.

For demographic bias testing, the Diversity in Faces (DiF) dataset from IBM and the BUPT-Balanced dataset provide annotations across age, gender, and skin tone, enabling disaggregated metric computation.

When testing a commercial API or fine-tuned model, you should evaluate against a held-out dataset that matches your production distribution, not just a public benchmark. Scraping a sample from your own users (with consent) and labeling pairs manually is expensive but necessary for high-stakes deployments.

Accuracy Metrics That Matter

TAR@FAR

The core metric for verification tasks is True Accept Rate at a given False Accept Rate (TAR@FAR). You set a FAR threshold (e.g., 1e-3 or 1e-4) and measure what fraction of genuine pairs are correctly accepted at that operating point.

import numpy as np
from sklearn.metrics import roc_curve

def tar_at_far(y_true, scores, target_far=1e-3):
    """
    Compute TAR at a specific FAR operating point.

    y_true: array of 1 (genuine pair) / 0 (impostor pair)
    scores: similarity scores from the model (higher = more similar)
    target_far: desired false accept rate
    """
    fpr, tpr, thresholds = roc_curve(y_true, scores)
    # fpr = FAR for a verification system
    idx = np.searchsorted(fpr, target_far)
    if idx >= len(tpr):
        return 0.0, thresholds[-1]
    return tpr[idx], thresholds[idx]

# Example usage
y_true = np.array([1, 1, 0, 0, 1, 0])
scores = np.array([0.91, 0.87, 0.43, 0.21, 0.78, 0.55])

tar, threshold = tar_at_far(y_true, scores, target_far=1e-2)
print(f"TAR@FAR=1e-2: {tar:.4f}, decision threshold: {threshold:.4f}")

For security applications (building access, border control), you typically target FAR=1e-4 or lower. For consumer applications, FAR=1e-2 is often acceptable.

AUC and ROC Curves

Area Under the Curve (AUC) gives a threshold-independent summary of discrimination ability. An AUC of 1.0 is perfect; 0.5 is random. Use AUC to compare models without committing to a threshold.

from sklearn.metrics import roc_auc_score, roc_curve
import matplotlib.pyplot as plt

def plot_roc(y_true, scores, label="Model"):
    fpr, tpr, _ = roc_curve(y_true, scores)
    auc = roc_auc_score(y_true, scores)
    plt.plot(fpr, tpr, label=f"{label} (AUC={auc:.4f})")
    plt.xlabel("False Accept Rate")
    plt.ylabel("True Accept Rate")
    plt.xscale("log")
    plt.grid(True, which="both", linestyle="--", alpha=0.5)
    plt.legend()
    return auc

Log-scale FAR (from 1e-5 to 1.0) is the standard x-axis for biometric ROC plots — the linear scale compresses the high-FAR region where security systems never operate.

Equal Error Rate (EER)

EER is the point where FAR equals FRR (False Reject Rate). It is a single-number summary commonly used in academic papers. Lower is better.

def equal_error_rate(y_true, scores):
    fpr, tpr, thresholds = roc_curve(y_true, scores)
    fnr = 1 - tpr
    idx = np.argmin(np.abs(fpr - fnr))
    return (fpr[idx] + fnr[idx]) / 2, thresholds[idx]

Demographic Bias Testing

A system with 99% TAR overall can still have 85% TAR for darker-skinned women and 99.9% TAR for lighter-skinned men. This is not a hypothetical — it has been documented repeatedly in commercial systems by the NIST FRVT program.

The test procedure:

Partition your evaluation set by demographic attributes (age band, perceived gender, Fitzpatrick skin tone scale 1–6).
Compute TAR@FAR independently for each group.
Report the worst-group TAR and the max disparity ratio (best group TAR / worst group TAR).

import pandas as pd

def disaggregated_tar_at_far(df, score_col, label_col, group_col, target_far=1e-3):
    """
    df: DataFrame with columns [score_col, label_col, group_col]
    Returns per-group TAR@FAR and overall disparity ratio.
    """
    results = {}
    for group, subset in df.groupby(group_col):
        y_true = subset[label_col].values
        scores = subset[score_col].values
        if y_true.sum() < 10:
            # Not enough genuine pairs for reliable estimate
            results[group] = None
            continue
        tar, _ = tar_at_far(y_true, scores, target_far)
        results[group] = tar

    valid = {k: v for k, v in results.items() if v is not None}
    if len(valid) < 2:
        return results, None

    disparity = max(valid.values()) / min(valid.values())
    return results, disparity

# Example
data = pd.DataFrame({
    "score": [0.9, 0.6, 0.85, 0.4, 0.92, 0.3],
    "is_genuine": [1, 0, 1, 0, 1, 0],
    "skin_tone_group": ["light", "light", "medium", "medium", "dark", "dark"],
})

per_group, disparity = disaggregated_tar_at_far(
    data, "score", "is_genuine", "skin_tone_group"
)
print(per_group)
print(f"Disparity ratio: {disparity:.2f}x")

A disparity ratio above 1.5x at your operating FAR is a red flag. Above 2x warrants blocking the release until addressed.

Threshold Tuning

Every facial recognition system exposes a similarity score. The decision threshold converts that score into a binary match/no-match decision. Choosing the wrong threshold is a common deployment mistake.

Threshold tuning procedure:

Use a calibration set (disjoint from both training and evaluation sets).
Plot TAR and FAR as a function of threshold.
Select the threshold that meets your FAR budget while maximizing TAR.
Document the threshold and the calibration set it was derived from.

def tune_threshold(y_true, scores, target_far=1e-3):
    fpr, tpr, thresholds = roc_curve(y_true, scores)
    # Find the lowest threshold where FAR <= target_far
    valid = thresholds[fpr <= target_far]
    if len(valid) == 0:
        raise ValueError("Cannot achieve target FAR — try a stricter threshold")
    # Return the lowest threshold in the valid range (most permissive while meeting FAR budget)
    return valid[-1]

threshold = tune_threshold(y_true_calibration, scores_calibration, target_far=1e-3)
print(f"Deploy threshold: {threshold:.4f}")

Thresholds are not portable across model versions. Re-tune after every model update.

Testing Cloud APIs

AWS Rekognition

import boto3

rekognition = boto3.client("rekognition", region_name="us-east-1")

def rekognition_similarity(image_path_a, image_path_b):
    with open(image_path_a, "rb") as f:
        bytes_a = f.read()
    with open(image_path_b, "rb") as f:
        bytes_b = f.read()

    response = rekognition.compare_faces(
        SourceImage={"Bytes": bytes_a},
        TargetImage={"Bytes": bytes_b},
        SimilarityThreshold=0,  # Return all matches regardless of score
    )

    if not response["FaceMatches"]:
        return 0.0
    return response["FaceMatches"][0]["Similarity"] / 100.0  # Normalize to [0, 1]

Rekognition returns a Similarity score from 0–100. Set SimilarityThreshold=0 when building an evaluation harness so you get back the raw score rather than a filtered list.

Azure Face API

import requests

AZURE_FACE_ENDPOINT = "https://<your-resource>.cognitiveservices.azure.com"
AZURE_FACE_KEY = "<your-key>"

def azure_detect(image_path):
    with open(image_path, "rb") as f:
        image_bytes = f.read()
    response = requests.post(
        f"{AZURE_FACE_ENDPOINT}/face/v1.0/detect",
        headers={
            "Ocp-Apim-Subscription-Key": AZURE_FACE_KEY,
            "Content-Type": "application/octet-stream",
        },
        params={"returnFaceId": "true"},
        data=image_bytes,
    )
    response.raise_for_status()
    faces = response.json()
    if not faces:
        return None
    return faces[0]["faceId"]

def azure_verify(face_id_a, face_id_b):
    response = requests.post(
        f"{AZURE_FACE_ENDPOINT}/face/v1.0/verify",
        headers={
            "Ocp-Apim-Subscription-Key": AZURE_FACE_KEY,
            "Content-Type": "application/json",
        },
        json={"faceId1": face_id_a, "faceId2": face_id_b},
    )
    response.raise_for_status()
    result = response.json()
    return result["confidence"]  # Already in [0, 1]

Azure face IDs expire after 24 hours by default. For large evaluations, cache IDs and re-detect on the next day's run.

Google Vision API

Google Vision does not expose a native face verification endpoint. You can use it for face detection quality checks (blur, occlusion, landmark visibility) as a pre-filter before sending pairs to a verification model.

from google.cloud import vision

client = vision.ImageAnnotatorClient()

def google_face_quality(image_path):
    with open(image_path, "rb") as f:
        content = f.read()
    image = vision.Image(content=content)
    response = client.face_detection(image=image)
    if not response.face_annotations:
        return None
    face = response.face_annotations[0]
    return {
        "detection_confidence": face.detection_confidence,
        "blur": face.blur_likelihood.name,      # UNKNOWN, VERY_UNLIKELY, ..., VERY_LIKELY
        "occlusion": face.under_exposed_likelihood.name,
        "headwear": face.headwear_likelihood.name,
    }

Use blur_likelihood >= LIKELY as an exclusion criterion when building your evaluation set. Blurry images inflate FAR and distort your metrics.

Building a Regression Test Harness

Model updates, API version changes, and threshold drift are the three most common causes of silent accuracy regression. A regression harness runs a fixed evaluation set on every release and alerts when metrics fall outside acceptable bounds.

import json
import datetime
from pathlib import Path

class FaceRecognitionRegressionHarness:
    def __init__(self, eval_pairs, score_fn, baseline_path="baseline_metrics.json"):
        """
        eval_pairs: list of {"path_a": str, "path_b": str, "is_genuine": int}
        score_fn: callable(path_a, path_b) -> float similarity score
        """
        self.eval_pairs = eval_pairs
        self.score_fn = score_fn
        self.baseline_path = Path(baseline_path)

    def run(self):
        scores, labels = [], []
        errors = 0
        for pair in self.eval_pairs:
            try:
                s = self.score_fn(pair["path_a"], pair["path_b"])
                scores.append(s)
                labels.append(pair["is_genuine"])
            except Exception as e:
                print(f"Error scoring pair {pair}: {e}")
                errors += 1

        y_true = np.array(labels)
        scores_arr = np.array(scores)

        auc = roc_auc_score(y_true, scores_arr)
        tar_1e3, threshold = tar_at_far(y_true, scores_arr, target_far=1e-3)
        eer, _ = equal_error_rate(y_true, scores_arr)

        return {
            "timestamp": datetime.datetime.utcnow().isoformat(),
            "n_pairs": len(labels),
            "n_errors": errors,
            "auc": round(auc, 6),
            "tar_at_far_1e3": round(tar_1e3, 6),
            "eer": round(eer, 6),
            "threshold_at_far_1e3": round(threshold, 6),
        }

    def assert_no_regression(self, current_metrics, tolerance=0.005):
        if not self.baseline_path.exists():
            print("No baseline found — saving current metrics as baseline.")
            self._save_baseline(current_metrics)
            return True

        with open(self.baseline_path) as f:
            baseline = json.load(f)

        regressions = []
        for metric in ["auc", "tar_at_far_1e3"]:
            delta = current_metrics[metric] - baseline[metric]
            if delta < -tolerance:
                regressions.append(
                    f"{metric}: {baseline[metric]:.4f} → {current_metrics[metric]:.4f} "
                    f"(dropped {abs(delta):.4f}, tolerance {tolerance})"
                )

        if regressions:
            raise AssertionError("Accuracy regression detected:\n" + "\n".join(regressions))

        print("No regression. Updating baseline.")
        self._save_baseline(current_metrics)
        return True

    def _save_baseline(self, metrics):
        with open(self.baseline_path, "w") as f:
            json.dump(metrics, f, indent=2)

Integrate this into your CI pipeline:

# ci_eval.py
from harness import FaceRecognitionRegressionHarness
from api_clients import rekognition_similarity
import json

pairs = json.load(open("eval/lfw_pairs_1000.json"))

harness = FaceRecognitionRegressionHarness(
    eval_pairs=pairs,
    score_fn=rekognition_similarity,
    baseline_path="eval/baseline_rekognition.json",
)

metrics = harness.run()
print(json.dumps(metrics, indent=2))
harness.assert_no_regression(metrics, tolerance=0.005)

Set the tolerance based on the measurement variance of your evaluation set. For 1,000 pairs, 0.005 AUC is a reasonable threshold — smaller sets have higher variance and require a wider tolerance to avoid false alarms.

Practical Checklist

Before shipping a facial recognition feature or upgrading a vendor model:

Evaluate on a held-out set that matches your production image distribution (not just LFW)
Compute TAR@FAR at your actual operating point, not just AUC
Run disaggregated metrics across age, gender, and skin tone groups
Re-tune the decision threshold on the new model — do not carry over the old one
Run the regression harness and confirm AUC and TAR@FAR are within tolerance
Check for increased API error rates (detection failures, timeouts) that can mask accuracy drops
Log all API responses in your test run for post-hoc debugging

Facial recognition is one of the few ML systems where accuracy failures have direct human consequences. The test infrastructure required to ship responsibly is not optional — it is the minimum bar. TAR@FAR, disaggregated metrics, and automated regression checks are not research concepts; they are production hygiene.