Testing

Testing Speech Recognition Systems: WER, Noise Robustness, and Regression Harnesses

HelpMeTest

HelpMeTest

— 9 min read
Testing Speech Recognition Systems: WER, Noise Robustness, and Regression Harnesses

Speech recognition has graduated from novelty to critical infrastructure. Voice assistants, call center automation, medical transcription, and accessibility tooling all depend on ASR systems that must perform reliably across accents, noise environments, and edge cases. Yet testing ASR is fundamentally different from testing a REST API or a UI component — the outputs are probabilistic, the failure modes are linguistic, and "close enough" can mean everything or nothing depending on context.

This guide covers the full testing stack for ASR systems: metric definitions, corpus construction, acoustic simulation, multi-provider testing, and a pytest regression harness you can drop into CI today.

Understanding ASR Quality Metrics

Before writing a single test, you need to agree on what "correct" means for your system.

Word Error Rate (WER)

WER is the canonical ASR metric. It measures the edit distance between the reference transcript and the hypothesis (ASR output) at the word level:

WER = (S + D + I) / N

Where:

  • S = substitutions (wrong word)
  • D = deletions (word missed)
  • I = insertions (extra word added)
  • N = total words in the reference

A WER of 0.05 means 5% of words were wrong. Industry benchmarks vary: 5–8% is typical for clean speech on major providers, but real-world deployments often see 15–30% in noisy conditions.

Character Error Rate (CER)

CER applies the same formula at the character level. It's more useful for languages without clear word boundaries (Mandarin, Japanese) and for catching partial-word errors that WER misses entirely. A substitution of "their" for "there" is one word error but zero character errors — CER surfaces that difference.

Match Error Rate and Word Information Lost

WER penalizes insertions and deletions equally, which isn't always the right tradeoff. Match Error Rate (MER) and Word Information Lost (WIL) give you additional signal when the error distribution matters more than the total count.

For most teams, WER + CER at the utterance level plus aggregate statistics across your test corpus is sufficient.

Building a Test Audio Corpus

The quality of your test corpus determines the validity of every metric you compute. A corpus built from studio recordings of native speakers will tell you nothing about how your system performs in the wild.

Corpus Dimensions

A production-grade corpus should cover:

  • Acoustic environments: quiet office, open-plan office, car, outdoors, phone (8kHz), video call (16kHz)
  • Speaker demographics: age groups, gender, native language, regional accents
  • Content domains: your specific vocabulary (product names, technical terms, proper nouns)
  • Utterance lengths: short commands (1–3 words), sentences (5–15 words), paragraphs (30+ words)
  • Speaking styles: read speech, spontaneous speech, disfluencies (um, uh, false starts)

Corpus Format

Store each test case as a pair: an audio file and a reference transcript. Use a manifest file for metadata:

# corpus/manifest.json
[
  {
    "id": "env_office_001",
    "audio_path": "audio/office/speaker_f_us_01.wav",
    "reference": "please schedule a meeting for tomorrow at three pm",
    "tags": ["environment:office", "accent:us", "domain:calendar"],
    "snr_db": 25
  },
  {
    "id": "env_car_002",
    "audio_path": "audio/car/speaker_m_uk_03.wav",
    "reference": "navigate to the nearest petrol station",
    "tags": ["environment:car", "accent:uk", "domain:navigation"],
    "snr_db": 12
  }
]

Synthetic Noise Augmentation

You rarely have recordings from every environment you care about. Use pydub and librosa to augment clean recordings with noise profiles:

import numpy as np
import soundfile as sf
from pydub import AudioSegment
from pydub.generators import WhiteNoise

def add_noise_at_snr(clean_audio_path: str, output_path: str, snr_db: float, noise_type: str = "white"):
    """Add noise to a clean recording at a specified SNR."""
    clean = AudioSegment.from_wav(clean_audio_path)
    clean_rms = clean.rms

    if noise_type == "white":
        noise = WhiteNoise().to_audio_segment(duration=len(clean))
    elif noise_type == "babble":
        # Load a pre-recorded babble noise file
        noise = AudioSegment.from_wav("noise_profiles/babble.wav")
        # Loop noise to match duration
        noise = (noise * (len(clean) // len(noise) + 1))[:len(clean)]
    else:
        raise ValueError(f"Unknown noise type: {noise_type}")

    # Scale noise to achieve target SNR
    noise_rms = noise.rms
    target_noise_rms = clean_rms / (10 ** (snr_db / 20))
    scaling_factor = target_noise_rms / noise_rms if noise_rms > 0 else 0
    scaled_noise = noise - (20 * np.log10(1 / scaling_factor)) if scaling_factor > 0 else noise

    mixed = clean.overlay(scaled_noise)
    mixed.export(output_path, format="wav")
    return output_path

Generate variants of your clean corpus at 5 dB, 10 dB, 15 dB, and 25 dB SNR. This gives you a noise robustness curve rather than a single data point.

Testing with Major ASR Providers

AWS Transcribe

import boto3
import time
import json
import requests
from urllib.parse import urlparse

def transcribe_with_aws(audio_s3_uri: str, job_name: str, language_code: str = "en-US") -> str:
    """Submit an audio file to AWS Transcribe and return the transcript."""
    client = boto3.client("transcribe", region_name="us-east-1")

    client.start_transcription_job(
        TranscriptionJobName=job_name,
        Media={"MediaFileUri": audio_s3_uri},
        MediaFormat="wav",
        LanguageCode=language_code,
        Settings={
            "ShowSpeakerLabels": False,
            "ChannelIdentification": False,
        }
    )

    # Poll for completion
    while True:
        response = client.get_transcription_job(TranscriptionJobName=job_name)
        status = response["TranscriptionJob"]["TranscriptionJobStatus"]
        if status == "COMPLETED":
            transcript_uri = response["TranscriptionJob"]["Transcript"]["TranscriptFileUri"]
            transcript_data = requests.get(transcript_uri).json()
            return transcript_data["results"]["transcripts"][0]["transcript"]
        elif status == "FAILED":
            raise RuntimeError(f"Transcription job failed: {response['TranscriptionJob'].get('FailureReason')}")
        time.sleep(2)

Google Cloud Speech-to-Text

from google.cloud import speech
import io

def transcribe_with_google(audio_path: str, language_code: str = "en-US", sample_rate: int = 16000) -> str:
    """Transcribe a local audio file using Google Cloud Speech-to-Text."""
    client = speech.SpeechClient()

    with io.open(audio_path, "rb") as f:
        audio_content = f.read()

    audio = speech.RecognitionAudio(content=audio_content)
    config = speech.RecognitionConfig(
        encoding=speech.RecognitionConfig.AudioEncoding.LINEAR16,
        sample_rate_hertz=sample_rate,
        language_code=language_code,
        enable_automatic_punctuation=True,
        model="latest_long",
    )

    response = client.recognize(config=config, audio=audio)

    transcript_parts = [result.alternatives[0].transcript for result in response.results]
    return " ".join(transcript_parts)

Azure Cognitive Services Speech

import requests
import time

def transcribe_with_azure(audio_path: str, subscription_key: str, region: str, language: str = "en-US") -> str:
    """Transcribe audio using Azure Speech REST API (batch transcription)."""
    base_url = f"https://{region}.api.cognitive.microsoft.com/speechtotext/v3.1"
    headers = {"Ocp-Apim-Subscription-Key": subscription_key, "Content-Type": "application/json"}

    # For short utterances, use the simple REST endpoint
    endpoint = f"https://{region}.stt.speech.microsoft.com/speech/recognition/conversation/cognitiveservices/v1"
    params = {"language": language, "format": "detailed"}
    audio_headers = {
        "Ocp-Apim-Subscription-Key": subscription_key,
        "Content-Type": "audio/wav; codecs=audio/pcm; samplerate=16000",
        "Accept": "application/json",
    }

    with open(audio_path, "rb") as f:
        audio_data = f.read()

    response = requests.post(endpoint, headers=audio_headers, params=params, data=audio_data)
    response.raise_for_status()

    result = response.json()
    if result.get("RecognitionStatus") == "Success":
        return result["DisplayText"]
    else:
        raise RuntimeError(f"Azure recognition failed: {result.get('RecognitionStatus')}")

Computing WER and CER in Python

from jiwer import wer, cer, transforms
import re

def normalize_text(text: str) -> str:
    """Normalize transcript for fair comparison."""
    text = text.lower()
    # Remove punctuation
    text = re.sub(r"[^\w\s]", "", text)
    # Collapse whitespace
    text = re.sub(r"\s+", " ", text).strip()
    return text

def compute_metrics(reference: str, hypothesis: str) -> dict:
    """Compute WER, CER, and word-level diff for a single utterance."""
    ref_norm = normalize_text(reference)
    hyp_norm = normalize_text(hypothesis)

    word_error_rate = wer(ref_norm, hyp_norm)
    char_error_rate = cer(ref_norm, hyp_norm)

    ref_words = ref_norm.split()
    hyp_words = hyp_norm.split()

    return {
        "wer": round(word_error_rate, 4),
        "cer": round(char_error_rate, 4),
        "reference_word_count": len(ref_words),
        "hypothesis_word_count": len(hyp_words),
        "reference": ref_norm,
        "hypothesis": hyp_norm,
    }

Install with pip install jiwer.

Accent and Dialect Testing

Accent testing is where most ASR evaluations fall short. A system that achieves 5% WER on General American English may reach 25% WER on Indian English or Southern American English — and that disparity is a fairness issue, not just a performance issue.

Structure your corpus to tag accent and dialect explicitly, then compute per-group metrics:

import json
from collections import defaultdict

def evaluate_corpus_by_group(manifest_path: str, transcribe_fn, group_tag_prefix: str = "accent:") -> dict:
    """Evaluate WER broken down by a tag group (e.g., accent)."""
    with open(manifest_path) as f:
        manifest = json.load(f)

    group_results = defaultdict(list)

    for item in manifest:
        hypothesis = transcribe_fn(item["audio_path"])
        metrics = compute_metrics(item["reference"], hypothesis)

        # Find the relevant tag
        group = "unknown"
        for tag in item.get("tags", []):
            if tag.startswith(group_tag_prefix):
                group = tag[len(group_tag_prefix):]
                break

        group_results[group].append(metrics["wer"])

    # Aggregate
    summary = {}
    for group, wers in group_results.items():
        summary[group] = {
            "mean_wer": round(sum(wers) / len(wers), 4),
            "max_wer": round(max(wers), 4),
            "sample_count": len(wers),
        }

    return summary

Set explicit thresholds per accent group in your test suite. If your SLA is WER < 0.10 for US English, you should have a separate (perhaps more lenient, documented) threshold for accents where your training data is sparse — and a roadmap to close the gap.

Noise Robustness Testing

A noise robustness test measures how WER degrades as SNR decreases. The goal is to find your system's operating envelope — the minimum SNR at which WER stays below your SLA.

import os
import pytest

SNR_LEVELS = [25, 15, 10, 5]  # dB
WER_THRESHOLDS = {25: 0.08, 15: 0.12, 10: 0.20, 5: 0.35}  # acceptable degradation curve

@pytest.mark.parametrize("snr_db", SNR_LEVELS)
def test_noise_robustness(snr_db, clean_audio_fixture, transcribe_fn):
    """WER must stay within threshold at each noise level."""
    noisy_path = add_noise_at_snr(
        clean_audio_fixture["path"],
        output_path=f"/tmp/noisy_{snr_db}db.wav",
        snr_db=snr_db,
        noise_type="babble",
    )

    hypothesis = transcribe_fn(noisy_path)
    metrics = compute_metrics(clean_audio_fixture["reference"], hypothesis)

    threshold = WER_THRESHOLDS[snr_db]
    assert metrics["wer"] <= threshold, (
        f"WER {metrics['wer']:.3f} exceeded threshold {threshold} at {snr_db}dB SNR. "
        f"Reference: '{metrics['reference']}' | Hypothesis: '{metrics['hypothesis']}'"
    )

Track this curve across releases. A new model version might improve clean-speech WER while regressing noise robustness — a regression harness catches this.

Pytest Regression Harness

Here is a complete regression harness that runs against all three providers and computes per-environment, per-accent WER with configurable thresholds:

# tests/test_asr_regression.py
import json
import os
import pytest
from pathlib import Path

CORPUS_MANIFEST = Path(__file__).parent / "corpus" / "manifest.json"
WER_SLA = float(os.getenv("ASR_WER_SLA", "0.10"))
PROVIDER = os.getenv("ASR_PROVIDER", "google")  # aws | google | azure


def get_transcribe_fn():
    if PROVIDER == "aws":
        import boto3
        def fn(audio_path):
            # Upload to S3 first, then call transcribe_with_aws
            s3 = boto3.client("s3")
            bucket = os.environ["ASR_TEST_BUCKET"]
            key = f"asr-test/{Path(audio_path).name}"
            s3.upload_file(audio_path, bucket, key)
            return transcribe_with_aws(f"s3://{bucket}/{key}", f"job-{Path(audio_path).stem}")
        return fn
    elif PROVIDER == "google":
        return lambda p: transcribe_with_google(p)
    elif PROVIDER == "azure":
        return lambda p: transcribe_with_azure(
            p,
            subscription_key=os.environ["AZURE_SPEECH_KEY"],
            region=os.environ["AZURE_SPEECH_REGION"],
        )
    else:
        raise ValueError(f"Unknown provider: {PROVIDER}")


@pytest.fixture(scope="session")
def corpus():
    with open(CORPUS_MANIFEST) as f:
        return json.load(f)


@pytest.fixture(scope="session")
def transcribe():
    return get_transcribe_fn()


def pytest_generate_tests(metafunc):
    if "corpus_item" in metafunc.fixturenames:
        with open(CORPUS_MANIFEST) as f:
            items = json.load(f)
        metafunc.parametrize("corpus_item", items, ids=[i["id"] for i in items])


def test_utterance_wer(corpus_item, transcribe):
    """Each utterance must meet the per-item WER SLA."""
    hypothesis = transcribe(corpus_item["audio_path"])
    metrics = compute_metrics(corpus_item["reference"], hypothesis)

    assert metrics["wer"] <= WER_SLA, (
        f"[{corpus_item['id']}] WER {metrics['wer']:.3f} > SLA {WER_SLA}\n"
        f"  REF: {metrics['reference']}\n"
        f"  HYP: {metrics['hypothesis']}\n"
        f"  Tags: {corpus_item.get('tags', [])}"
    )


def test_domain_vocabulary(corpus, transcribe):
    """Product-specific terms must not be systematically missed."""
    domain_items = [i for i in corpus if any("domain:product" in t for t in i.get("tags", []))]
    if not domain_items:
        pytest.skip("No domain:product items in corpus")

    wers = []
    for item in domain_items:
        hyp = transcribe(item["audio_path"])
        metrics = compute_metrics(item["reference"], hyp)
        wers.append(metrics["wer"])

    mean_wer = sum(wers) / len(wers)
    assert mean_wer <= WER_SLA * 1.5, (
        f"Domain vocabulary mean WER {mean_wer:.3f} exceeds 1.5x SLA ({WER_SLA * 1.5:.3f})"
    )


def test_no_accent_disparity(corpus, transcribe):
    """WER gap between best and worst accent group must not exceed 0.15."""
    accent_wers = defaultdict(list)

    for item in corpus:
        hyp = transcribe(item["audio_path"])
        metrics = compute_metrics(item["reference"], hyp)
        for tag in item.get("tags", []):
            if tag.startswith("accent:"):
                accent_wers[tag[7:]].append(metrics["wer"])

    if len(accent_wers) < 2:
        pytest.skip("Need at least 2 accent groups to test disparity")

    group_means = {g: sum(v) / len(v) for g, v in accent_wers.items()}
    best = min(group_means.values())
    worst = max(group_means.values())
    disparity = worst - best

    assert disparity <= 0.15, (
        f"Accent WER disparity {disparity:.3f} exceeds 0.15.\n"
        f"Per-group means: {json.dumps(group_means, indent=2)}"
    )

Run with:

ASR_PROVIDER=google ASR_WER_SLA=0.08 pytest tests/test_asr_regression.py -v --tb=short

In CI, set ASR_PROVIDER and credentials via environment secrets. Gate merges on this suite passing.

Testing Latency and Streaming

For real-time applications, latency matters as much as accuracy. Measure time-to-first-word and end-to-end latency separately:

import time

def measure_latency(audio_path: str, transcribe_fn) -> dict:
    """Measure transcription latency for a single audio file."""
    audio_duration = get_audio_duration(audio_path)  # seconds

    start = time.perf_counter()
    transcript = transcribe_fn(audio_path)
    elapsed = time.perf_counter() - start

    return {
        "audio_duration_s": round(audio_duration, 2),
        "transcription_latency_s": round(elapsed, 3),
        "real_time_factor": round(elapsed / audio_duration, 2),  # < 1.0 = faster than real-time
        "transcript": transcript,
    }

A real-time factor below 1.0 means your system can keep up with live audio. Flag any test case where RTF exceeds your SLA (typically 0.5–2.0 depending on use case).

Continuous Integration Strategy

Structure your ASR test suite in tiers:

  1. Smoke tier (every PR, ~2 minutes): 20 utterances, clean audio, single provider, fast feedback
  2. Regression tier (nightly, ~30 minutes): full corpus, all noise levels, single primary provider
  3. Multi-provider tier (weekly or pre-release, ~2 hours): full corpus against all providers with disparity analysis

Store WER results as CI artifacts and track them over time. A 0.5% WER regression is invisible in a single run but clear on a trend chart. Tools like MLflow or simple JSON files committed to the repo both work — what matters is the trend, not the tool.

What to Test That Most Teams Skip

  • Silence and non-speech audio: Does your system return empty string, an error, or garbage?
  • Very long utterances: Many providers have duration limits or degrade on paragraphs.
  • Numbers and dates: "February 14th" vs "February fourteenth" — decide which normalization you require.
  • Proper nouns and brand names: Your product name will be mangled. Test it. Configure custom vocabulary/phrase hints if the provider supports it.
  • Code-switching: Users switch languages mid-sentence. Does your system degrade gracefully?
  • Channel conditions: 8kHz telephone audio vs 16kHz vs 44.1kHz broadband all behave differently.

ASR testing is a discipline, not a checkbox. The teams that treat it as one — building corpora, tracking metrics over time, and catching regressions before users do — are the ones who can actually commit to accuracy SLAs in production.

Read more

Start now free