Testing Text-to-Speech Quality: MOS, PESQ, STOI, and Automated Regression Pipelines

Testing Text-to-Speech Quality: MOS, PESQ, STOI, and Automated Regression Pipelines

Text-to-speech systems have reached a level of quality where differences between engines are subtle — a slightly unnatural pause, a mispronounced proper noun, or a prosodic pattern that sounds robotic in long-form narration. Catching these regressions before they reach production requires a systematic testing approach that combines subjective evaluation methodology, objective signal metrics, and automated pipelines.

This guide covers the full stack of TTS quality evaluation: from the industry-standard MOS methodology to neural MOS estimators, objective metrics like PESQ and STOI, pronunciation accuracy testing, prosody evaluation, and how to tie it all together in a regression test suite that runs against AWS Polly, Google Cloud TTS, and ElevenLabs.

Mean Opinion Score (MOS): The Ground Truth

MOS is the gold standard for speech quality evaluation. Human listeners rate audio samples on a 1–5 scale:

Score Label Description
5 Excellent Imperceptible impairment
4 Good Perceptible but not annoying
3 Fair Slightly annoying
2 Poor Annoying
1 Bad Very annoying

For TTS evaluation specifically, ITU-T P.85 defines the framework. Listeners evaluate naturalness, intelligibility, and overall impression. A production TTS system should target MOS ≥ 4.0 for general use cases; interactive voice response (IVR) systems often accept MOS ≥ 3.5.

Running a MOS Study

A proper MOS study requires 20–30 listeners, balanced presentation order, and anchor conditions (a known-good reference at ~4.3 MOS and a degraded anchor at ~2.5 MOS). The anchors let you detect inattentive raters and normalize scores across listener panels.

For continuous regression testing, full human MOS studies are too slow. This is where automated MOS estimation becomes essential.

UTMOS: Neural MOS Estimation

UTMOS (Unified Text-to-Speech Mean Opinion Score) is a neural predictor trained on the VoiceMOS Challenge dataset. It predicts MOS scores with a Pearson correlation of ~0.94 with human judgments — good enough to catch regressions, though not a replacement for human evaluation when launching a new voice.

import torch
import torchaudio
import numpy as np

# pip install utmos  (or clone from https://github.com/sarulab-speech/UTMOS22)
from utmos import UTMOSScore

def estimate_mos(audio_path: str) -> float:
    """Estimate MOS score for a WAV file using UTMOS."""
    predictor = UTMOSScore(device="cpu")
    wav, sr = torchaudio.load(audio_path)
    
    # UTMOS expects 16kHz mono
    if sr != 16000:
        resampler = torchaudio.transforms.Resample(sr, 16000)
        wav = resampler(wav)
    if wav.shape[0] > 1:
        wav = wav.mean(dim=0, keepdim=True)
    
    score = predictor.score(wav)
    return float(score)

# Example usage
mos = estimate_mos("output_polly.wav")
print(f"Estimated MOS: {mos:.3f}")
# Expected range for production TTS: 3.8–4.4

For batch evaluation across multiple samples and engines:

import os
import json
from pathlib import Path
from dataclasses import dataclass, asdict
from typing import List

@dataclass
class MOSResult:
    engine: str
    text_id: str
    audio_path: str
    estimated_mos: float

def batch_evaluate_mos(samples_dir: str, engines: List[str]) -> List[MOSResult]:
    predictor = UTMOSScore(device="cuda" if torch.cuda.is_available() else "cpu")
    results = []
    
    for engine in engines:
        engine_dir = Path(samples_dir) / engine
        for wav_file in sorted(engine_dir.glob("*.wav")):
            wav, sr = torchaudio.load(wav_file)
            if sr != 16000:
                wav = torchaudio.transforms.Resample(sr, 16000)(wav)
            if wav.shape[0] > 1:
                wav = wav.mean(dim=0, keepdim=True)
            
            score = float(predictor.score(wav))
            results.append(MOSResult(
                engine=engine,
                text_id=wav_file.stem,
                audio_path=str(wav_file),
                estimated_mos=score
            ))
    
    return results

PESQ and STOI: Objective Signal Metrics

PESQ (Perceptual Evaluation of Speech Quality, ITU-T P.862) and STOI (Short-Time Objective Intelligibility) are signal-level metrics that compare a test signal against a reference. For TTS testing, the "reference" is typically a high-quality recording of the same text by a human voice talent, or a known-good version of the same TTS engine.

PESQ scores range from -0.5 to 4.5 (wideband mode). Scores above 3.5 indicate good quality for voice communication. PESQ is sensitive to codec artifacts, background noise, and distortion — useful for catching audio processing regressions.

STOI ranges from 0 to 1. Values above 0.85 indicate high intelligibility. STOI is better than PESQ for measuring whether speech is actually understandable.

import numpy as np
import soundfile as sf
from pesq import pesq
from pystoi import stoi

def load_audio_16k(path: str) -> tuple[np.ndarray, int]:
    """Load audio and resample to 16kHz."""
    import librosa
    audio, sr = librosa.load(path, sr=16000, mono=True)
    return audio, 16000

def compute_pesq_stoi(reference_path: str, degraded_path: str) -> dict:
    """
    Compute PESQ and STOI between reference and degraded audio.
    Both files should be same-length recordings of the same text.
    """
    ref, sr_ref = load_audio_16k(reference_path)
    deg, sr_deg = load_audio_16k(degraded_path)
    
    # Align lengths (trim to shorter)
    min_len = min(len(ref), len(deg))
    ref = ref[:min_len]
    deg = deg[:min_len]
    
    pesq_score = pesq(16000, ref, deg, "wb")  # wideband
    stoi_score = stoi(ref, deg, 16000, extended=False)
    
    return {
        "pesq": round(pesq_score, 4),
        "stoi": round(stoi_score, 4),
        "pesq_ok": pesq_score >= 3.5,
        "stoi_ok": stoi_score >= 0.85,
    }

# Example: compare two engine versions
metrics = compute_pesq_stoi(
    reference_path="baseline_polly_v1.wav",
    degraded_path="candidate_polly_v2.wav"
)
print(metrics)
# {'pesq': 3.71, 'stoi': 0.92, 'pesq_ok': True, 'stoi_ok': True}

One important caveat: PESQ and STOI are designed for narrowband/wideband telephony. For neural TTS systems producing 22kHz or 44kHz audio, these metrics become less meaningful — they don't capture the naturalness dimension that UTMOS targets. Use them primarily for detecting processing artifacts (clipping, resampling errors, codec degradation) rather than as absolute quality measures.

Pronunciation Accuracy Testing

Pronunciation errors are a common regression vector when TTS providers update their models. A word that was correctly stressed in version N may be mispronounced in version N+1. Testing this requires a pronunciation test corpus and forced alignment.

import boto3
import io
import wave
from pathlib import Path

# pip install boto3 stable-ts (Whisper-based forced alignment)
import stable_whisper

def synthesize_polly(text: str, voice_id: str = "Joanna", 
                      output_path: str = "output.wav") -> str:
    """Synthesize text with AWS Polly and save to WAV."""
    client = boto3.client("polly", region_name="us-east-1")
    
    response = client.synthesize_speech(
        Text=text,
        OutputFormat="pcm",
        VoiceId=voice_id,
        SampleRate="16000",
        Engine="neural"
    )
    
    pcm_data = response["AudioStream"].read()
    
    # PCM → WAV
    with wave.open(output_path, "wb") as wav_file:
        wav_file.setnchannels(1)
        wav_file.setsampwidth(2)  # 16-bit
        wav_file.setframerate(16000)
        wav_file.writeframes(pcm_data)
    
    return output_path

def check_pronunciation(text: str, expected_words: list[str], 
                         audio_path: str) -> dict:
    """
    Use Whisper transcription to verify pronunciation.
    Returns per-word confidence and flags low-confidence words.
    """
    model = stable_whisper.load_model("base")
    result = model.transcribe(audio_path, word_timestamps=True)
    
    transcribed_words = [
        {"word": seg["word"].strip().lower(), "confidence": seg.get("probability", 1.0)}
        for seg in result["segments"]
        for seg in seg["words"]
    ]
    
    issues = []
    for expected in expected_words:
        match = next(
            (w for w in transcribed_words if w["word"] == expected.lower()), 
            None
        )
        if match is None:
            issues.append({"word": expected, "issue": "not_found"})
        elif match["confidence"] < 0.85:
            issues.append({
                "word": expected, 
                "issue": "low_confidence", 
                "confidence": match["confidence"]
            })
    
    return {
        "text": text,
        "transcription": " ".join(w["word"] for w in transcribed_words),
        "issues": issues,
        "passed": len(issues) == 0
    }

Build a pronunciation test corpus covering your domain vocabulary — technical terms, product names, abbreviations, and numerals. Run this corpus on every TTS model update.

PRONUNCIATION_CORPUS = [
    {"text": "The API returns a JSON response.", 
     "critical_words": ["API", "JSON"]},
    {"text": "Please contact our CEO at headquarters.", 
     "critical_words": ["CEO", "headquarters"]},
    {"text": "The dosage is 500 milligrams twice daily.", 
     "critical_words": ["milligrams", "dosage"]},
    {"text": "Connect to WiFi network 802.11ax.", 
     "critical_words": ["WiFi", "802.11ax"]},
]

def run_pronunciation_suite(engine_synthesize_fn, corpus=PRONUNCIATION_CORPUS):
    results = []
    for item in corpus:
        audio_path = engine_synthesize_fn(item["text"])
        result = check_pronunciation(item["text"], item["critical_words"], audio_path)
        results.append(result)
    
    passed = sum(1 for r in results if r["passed"])
    print(f"Pronunciation tests: {passed}/{len(results)} passed")
    return results

Prosody Evaluation

Prosody — the rhythm, stress, and intonation of speech — is harder to measure objectively. The main signal-level features to track are:

  • Pitch (F0) trajectory: Monotone pitch is a common TTS regression. Extract F0 and measure variance.
  • Speaking rate (syllables/second): Too fast or too slow relative to a baseline signals a regression.
  • Pause duration distribution: Unnatural pauses (too long or missing) affect perceived naturalness.
import parselmouth  # pip install praat-parselmouth
import numpy as np

def extract_prosody_features(audio_path: str) -> dict:
    """Extract pitch, speaking rate, and pause features using Praat."""
    snd = parselmouth.Sound(audio_path)
    
    # Pitch extraction
    pitch = snd.to_pitch(time_step=0.01, pitch_floor=75, pitch_ceiling=500)
    f0_values = pitch.selected_array["frequency"]
    f0_voiced = f0_values[f0_values > 0]
    
    # Intensity for pause detection
    intensity = snd.to_intensity(time_step=0.01)
    intensity_values = intensity.values.T.flatten()
    
    # Silence threshold: frames below 40 dB are pauses
    pause_frames = intensity_values < 40
    pause_ratio = pause_frames.mean()
    
    # Detect pause durations
    in_pause = False
    pause_durations = []
    current_pause = 0
    frame_duration = 0.01
    
    for is_pause in pause_frames:
        if is_pause:
            current_pause += frame_duration
            in_pause = True
        elif in_pause:
            pause_durations.append(current_pause)
            current_pause = 0
            in_pause = False
    
    return {
        "f0_mean": float(np.mean(f0_voiced)) if len(f0_voiced) > 0 else 0,
        "f0_std": float(np.std(f0_voiced)) if len(f0_voiced) > 0 else 0,
        "voiced_fraction": len(f0_voiced) / max(len(f0_values), 1),
        "pause_ratio": float(pause_ratio),
        "mean_pause_duration_ms": float(np.mean(pause_durations) * 1000) if pause_durations else 0,
        "max_pause_duration_ms": float(np.max(pause_durations) * 1000) if pause_durations else 0,
        "audio_duration_s": snd.duration,
    }

def compare_prosody(baseline_path: str, candidate_path: str, 
                     thresholds: dict = None) -> dict:
    """Compare prosody features between baseline and candidate audio."""
    if thresholds is None:
        thresholds = {
            "f0_std_min": 10.0,       # Hz — below this is monotone
            "max_pause_ms": 2000,      # ms — above this is unnatural
            "voiced_fraction_min": 0.5,
        }
    
    baseline = extract_prosody_features(baseline_path)
    candidate = extract_prosody_features(candidate_path)
    
    issues = []
    if candidate["f0_std"] < thresholds["f0_std_min"]:
        issues.append(f"Monotone pitch: f0_std={candidate['f0_std']:.1f} Hz "
                      f"(baseline={baseline['f0_std']:.1f})")
    
    if candidate["max_pause_duration_ms"] > thresholds["max_pause_ms"]:
        issues.append(f"Abnormally long pause: {candidate['max_pause_duration_ms']:.0f}ms")
    
    if candidate["voiced_fraction"] < thresholds["voiced_fraction_min"]:
        issues.append(f"Low voiced fraction: {candidate['voiced_fraction']:.2f}")
    
    return {
        "baseline": baseline,
        "candidate": candidate,
        "issues": issues,
        "passed": len(issues) == 0,
    }

Regression Testing Across TTS APIs

The final piece is a regression harness that runs all the above checks against multiple TTS providers on every release or weekly schedule.

import requests
import json
import hashlib
from datetime import datetime

# Google Cloud TTS synthesis
def synthesize_google(text: str, language_code: str = "en-US",
                       voice_name: str = "en-US-Neural2-F",
                       output_path: str = "google_output.wav") -> str:
    """Synthesize with Google Cloud TTS."""
    import google.cloud.texttospeech as tts
    
    client = tts.TextToSpeechClient()
    synthesis_input = tts.SynthesisInput(text=text)
    voice = tts.VoiceSelectionParams(
        language_code=language_code,
        name=voice_name,
    )
    audio_config = tts.AudioConfig(
        audio_encoding=tts.AudioEncoding.LINEAR16,
        sample_rate_hertz=16000,
    )
    
    response = client.synthesize_speech(
        input=synthesis_input, voice=voice, audio_config=audio_config
    )
    
    with open(output_path, "wb") as f:
        f.write(response.audio_content)
    return output_path

# ElevenLabs synthesis
def synthesize_elevenlabs(text: str, voice_id: str,
                            api_key: str,
                            output_path: str = "elevenlabs_output.mp3") -> str:
    """Synthesize with ElevenLabs API."""
    url = f"https://api.elevenlabs.io/v1/text-to-speech/{voice_id}"
    headers = {
        "xi-api-key": api_key,
        "Content-Type": "application/json",
    }
    payload = {
        "text": text,
        "model_id": "eleven_multilingual_v2",
        "voice_settings": {"stability": 0.5, "similarity_boost": 0.75},
    }
    
    response = requests.post(url, json=payload, headers=headers)
    response.raise_for_status()
    
    with open(output_path, "wb") as f:
        f.write(response.content)
    return output_path

# Full regression runner
def run_tts_regression_suite(
    test_texts: list[str],
    engines: dict,  # {"engine_name": synthesize_fn}
    baseline_dir: str,
    output_dir: str,
    mos_threshold: float = 3.8,
    pesq_threshold: float = 3.5,
) -> dict:
    """
    Run full regression suite: MOS, PESQ, STOI, prosody.
    Compares candidate outputs against stored baselines.
    """
    from pathlib import Path
    import time
    
    results = {"timestamp": datetime.utcnow().isoformat(), "engines": {}}
    Path(output_dir).mkdir(parents=True, exist_ok=True)
    
    predictor = UTMOSScore(device="cpu")
    
    for engine_name, synthesize_fn in engines.items():
        engine_results = []
        
        for i, text in enumerate(test_texts):
            text_id = hashlib.md5(text.encode()).hexdigest()[:8]
            candidate_path = f"{output_dir}/{engine_name}_{text_id}.wav"
            baseline_path = f"{baseline_dir}/{engine_name}_{text_id}.wav"
            
            # Synthesize candidate
            synthesize_fn(text, output_path=candidate_path)
            time.sleep(0.2)  # rate limiting
            
            # MOS
            wav, sr = torchaudio.load(candidate_path)
            if sr != 16000:
                wav = torchaudio.transforms.Resample(sr, 16000)(wav)
            if wav.shape[0] > 1:
                wav = wav.mean(0, keepdim=True)
            mos = float(predictor.score(wav))
            
            result = {
                "text_id": text_id,
                "text_preview": text[:60],
                "mos": round(mos, 3),
                "mos_ok": mos >= mos_threshold,
            }
            
            # PESQ/STOI only if baseline exists
            if Path(baseline_path).exists():
                metrics = compute_pesq_stoi(baseline_path, candidate_path)
                result.update(metrics)
                
                prosody = compare_prosody(baseline_path, candidate_path)
                result["prosody_issues"] = prosody["issues"]
                result["prosody_ok"] = prosody["passed"]
            else:
                result["note"] = "No baseline — saving as new baseline"
                import shutil
                shutil.copy(candidate_path, baseline_path)
            
            engine_results.append(result)
        
        passed = sum(1 for r in engine_results if r["mos_ok"] 
                     and r.get("pesq_ok", True) 
                     and r.get("prosody_ok", True))
        results["engines"][engine_name] = {
            "passed": passed,
            "total": len(engine_results),
            "samples": engine_results,
        }
    
    # Write report
    report_path = f"{output_dir}/regression_report.json"
    with open(report_path, "w") as f:
        json.dump(results, f, indent=2)
    
    return results

CI Integration and Alerting

Integrate the regression suite into your CI pipeline to catch regressions automatically:

# .github/workflows/tts-regression.yml
name: TTS Quality Regression

on:
  schedule:
    - cron: "0 6 * * 1"  # Weekly Monday 6am
  push:
    paths:
      - "tts/**"
      - "tests/tts_regression.py"

jobs:
  regression:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Install dependencies
        run: pip install utmos pesq pystoi parselmouth stable-ts boto3 requests
      - name: Run TTS regression suite
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          ELEVENLABS_API_KEY: ${{ secrets.ELEVENLABS_API_KEY }}
        run: python tests/tts_regression.py
      - name: Upload regression report
        uses: actions/upload-artifact@v4
        with:
          name: tts-regression-report
          path: outputs/regression_report.json
      - name: Fail on quality regression
        run: python tests/assert_no_regression.py outputs/regression_report.json

The assert_no_regression.py script should fail the build if any engine shows MOS below threshold or PESQ regression greater than 0.3 points from baseline:

import json, sys

with open(sys.argv[1]) as f:
    report = json.load(f)

failures = []
for engine, data in report["engines"].items():
    for sample in data["samples"]:
        if not sample.get("mos_ok", True):
            failures.append(f"{engine}/{sample['text_id']}: MOS={sample['mos']}")
        if sample.get("pesq", 99) < 3.0:
            failures.append(f"{engine}/{sample['text_id']}: PESQ={sample['pesq']}")
        for issue in sample.get("prosody_issues", []):
            failures.append(f"{engine}/{sample['text_id']}: {issue}")

if failures:
    print("TTS REGRESSION FAILURES:")
    for f in failures:
        print(f"  - {f}")
    sys.exit(1)

print(f"All TTS quality checks passed.")

Putting It Together

A mature TTS quality testing program combines these layers:

  1. UTMOS automated scoring on every synthesis — catches naturalness regressions without human review
  2. PESQ/STOI against baseline — catches audio processing artifacts (resampling, encoding changes)
  3. Pronunciation test corpus — domain vocabulary coverage on every model update
  4. Prosody feature tracking — monotone detection, pause distribution over time
  5. Weekly human MOS studies on a 50-sentence benchmark corpus — ground truth that neural estimators might miss
  6. CI regression gate — fails the build if any metric drops below threshold

The test corpus matters as much as the metrics. Cover diverse sentence types: short commands, long narrative sentences, questions, technical terminology, numerals, abbreviations, and edge cases like single-word utterances and sentences with heavy punctuation. A corpus of 200–500 sentences balanced across these categories will catch most regression patterns.

TTS quality is rarely binary — it degrades gradually until it becomes noticeable to end users. Systematic measurement is the only way to catch it early enough to matter.

Read more

Start now free