Testing Text-to-Speech Quality: MOS, PESQ, STOI, and Automated Regression Pipelines
Text-to-speech systems have reached a level of quality where differences between engines are subtle — a slightly unnatural pause, a mispronounced proper noun, or a prosodic pattern that sounds robotic in long-form narration. Catching these regressions before they reach production requires a systematic testing approach that combines subjective evaluation methodology, objective signal metrics, and automated pipelines.
This guide covers the full stack of TTS quality evaluation: from the industry-standard MOS methodology to neural MOS estimators, objective metrics like PESQ and STOI, pronunciation accuracy testing, prosody evaluation, and how to tie it all together in a regression test suite that runs against AWS Polly, Google Cloud TTS, and ElevenLabs.
Mean Opinion Score (MOS): The Ground Truth
MOS is the gold standard for speech quality evaluation. Human listeners rate audio samples on a 1–5 scale:
| Score | Label | Description |
|---|---|---|
| 5 | Excellent | Imperceptible impairment |
| 4 | Good | Perceptible but not annoying |
| 3 | Fair | Slightly annoying |
| 2 | Poor | Annoying |
| 1 | Bad | Very annoying |
For TTS evaluation specifically, ITU-T P.85 defines the framework. Listeners evaluate naturalness, intelligibility, and overall impression. A production TTS system should target MOS ≥ 4.0 for general use cases; interactive voice response (IVR) systems often accept MOS ≥ 3.5.
Running a MOS Study
A proper MOS study requires 20–30 listeners, balanced presentation order, and anchor conditions (a known-good reference at ~4.3 MOS and a degraded anchor at ~2.5 MOS). The anchors let you detect inattentive raters and normalize scores across listener panels.
For continuous regression testing, full human MOS studies are too slow. This is where automated MOS estimation becomes essential.
UTMOS: Neural MOS Estimation
UTMOS (Unified Text-to-Speech Mean Opinion Score) is a neural predictor trained on the VoiceMOS Challenge dataset. It predicts MOS scores with a Pearson correlation of ~0.94 with human judgments — good enough to catch regressions, though not a replacement for human evaluation when launching a new voice.
import torch
import torchaudio
import numpy as np
# pip install utmos (or clone from https://github.com/sarulab-speech/UTMOS22)
from utmos import UTMOSScore
def estimate_mos(audio_path: str) -> float:
"""Estimate MOS score for a WAV file using UTMOS."""
predictor = UTMOSScore(device="cpu")
wav, sr = torchaudio.load(audio_path)
# UTMOS expects 16kHz mono
if sr != 16000:
resampler = torchaudio.transforms.Resample(sr, 16000)
wav = resampler(wav)
if wav.shape[0] > 1:
wav = wav.mean(dim=0, keepdim=True)
score = predictor.score(wav)
return float(score)
# Example usage
mos = estimate_mos("output_polly.wav")
print(f"Estimated MOS: {mos:.3f}")
# Expected range for production TTS: 3.8–4.4For batch evaluation across multiple samples and engines:
import os
import json
from pathlib import Path
from dataclasses import dataclass, asdict
from typing import List
@dataclass
class MOSResult:
engine: str
text_id: str
audio_path: str
estimated_mos: float
def batch_evaluate_mos(samples_dir: str, engines: List[str]) -> List[MOSResult]:
predictor = UTMOSScore(device="cuda" if torch.cuda.is_available() else "cpu")
results = []
for engine in engines:
engine_dir = Path(samples_dir) / engine
for wav_file in sorted(engine_dir.glob("*.wav")):
wav, sr = torchaudio.load(wav_file)
if sr != 16000:
wav = torchaudio.transforms.Resample(sr, 16000)(wav)
if wav.shape[0] > 1:
wav = wav.mean(dim=0, keepdim=True)
score = float(predictor.score(wav))
results.append(MOSResult(
engine=engine,
text_id=wav_file.stem,
audio_path=str(wav_file),
estimated_mos=score
))
return resultsPESQ and STOI: Objective Signal Metrics
PESQ (Perceptual Evaluation of Speech Quality, ITU-T P.862) and STOI (Short-Time Objective Intelligibility) are signal-level metrics that compare a test signal against a reference. For TTS testing, the "reference" is typically a high-quality recording of the same text by a human voice talent, or a known-good version of the same TTS engine.
PESQ scores range from -0.5 to 4.5 (wideband mode). Scores above 3.5 indicate good quality for voice communication. PESQ is sensitive to codec artifacts, background noise, and distortion — useful for catching audio processing regressions.
STOI ranges from 0 to 1. Values above 0.85 indicate high intelligibility. STOI is better than PESQ for measuring whether speech is actually understandable.
import numpy as np
import soundfile as sf
from pesq import pesq
from pystoi import stoi
def load_audio_16k(path: str) -> tuple[np.ndarray, int]:
"""Load audio and resample to 16kHz."""
import librosa
audio, sr = librosa.load(path, sr=16000, mono=True)
return audio, 16000
def compute_pesq_stoi(reference_path: str, degraded_path: str) -> dict:
"""
Compute PESQ and STOI between reference and degraded audio.
Both files should be same-length recordings of the same text.
"""
ref, sr_ref = load_audio_16k(reference_path)
deg, sr_deg = load_audio_16k(degraded_path)
# Align lengths (trim to shorter)
min_len = min(len(ref), len(deg))
ref = ref[:min_len]
deg = deg[:min_len]
pesq_score = pesq(16000, ref, deg, "wb") # wideband
stoi_score = stoi(ref, deg, 16000, extended=False)
return {
"pesq": round(pesq_score, 4),
"stoi": round(stoi_score, 4),
"pesq_ok": pesq_score >= 3.5,
"stoi_ok": stoi_score >= 0.85,
}
# Example: compare two engine versions
metrics = compute_pesq_stoi(
reference_path="baseline_polly_v1.wav",
degraded_path="candidate_polly_v2.wav"
)
print(metrics)
# {'pesq': 3.71, 'stoi': 0.92, 'pesq_ok': True, 'stoi_ok': True}One important caveat: PESQ and STOI are designed for narrowband/wideband telephony. For neural TTS systems producing 22kHz or 44kHz audio, these metrics become less meaningful — they don't capture the naturalness dimension that UTMOS targets. Use them primarily for detecting processing artifacts (clipping, resampling errors, codec degradation) rather than as absolute quality measures.
Pronunciation Accuracy Testing
Pronunciation errors are a common regression vector when TTS providers update their models. A word that was correctly stressed in version N may be mispronounced in version N+1. Testing this requires a pronunciation test corpus and forced alignment.
import boto3
import io
import wave
from pathlib import Path
# pip install boto3 stable-ts (Whisper-based forced alignment)
import stable_whisper
def synthesize_polly(text: str, voice_id: str = "Joanna",
output_path: str = "output.wav") -> str:
"""Synthesize text with AWS Polly and save to WAV."""
client = boto3.client("polly", region_name="us-east-1")
response = client.synthesize_speech(
Text=text,
OutputFormat="pcm",
VoiceId=voice_id,
SampleRate="16000",
Engine="neural"
)
pcm_data = response["AudioStream"].read()
# PCM → WAV
with wave.open(output_path, "wb") as wav_file:
wav_file.setnchannels(1)
wav_file.setsampwidth(2) # 16-bit
wav_file.setframerate(16000)
wav_file.writeframes(pcm_data)
return output_path
def check_pronunciation(text: str, expected_words: list[str],
audio_path: str) -> dict:
"""
Use Whisper transcription to verify pronunciation.
Returns per-word confidence and flags low-confidence words.
"""
model = stable_whisper.load_model("base")
result = model.transcribe(audio_path, word_timestamps=True)
transcribed_words = [
{"word": seg["word"].strip().lower(), "confidence": seg.get("probability", 1.0)}
for seg in result["segments"]
for seg in seg["words"]
]
issues = []
for expected in expected_words:
match = next(
(w for w in transcribed_words if w["word"] == expected.lower()),
None
)
if match is None:
issues.append({"word": expected, "issue": "not_found"})
elif match["confidence"] < 0.85:
issues.append({
"word": expected,
"issue": "low_confidence",
"confidence": match["confidence"]
})
return {
"text": text,
"transcription": " ".join(w["word"] for w in transcribed_words),
"issues": issues,
"passed": len(issues) == 0
}Build a pronunciation test corpus covering your domain vocabulary — technical terms, product names, abbreviations, and numerals. Run this corpus on every TTS model update.
PRONUNCIATION_CORPUS = [
{"text": "The API returns a JSON response.",
"critical_words": ["API", "JSON"]},
{"text": "Please contact our CEO at headquarters.",
"critical_words": ["CEO", "headquarters"]},
{"text": "The dosage is 500 milligrams twice daily.",
"critical_words": ["milligrams", "dosage"]},
{"text": "Connect to WiFi network 802.11ax.",
"critical_words": ["WiFi", "802.11ax"]},
]
def run_pronunciation_suite(engine_synthesize_fn, corpus=PRONUNCIATION_CORPUS):
results = []
for item in corpus:
audio_path = engine_synthesize_fn(item["text"])
result = check_pronunciation(item["text"], item["critical_words"], audio_path)
results.append(result)
passed = sum(1 for r in results if r["passed"])
print(f"Pronunciation tests: {passed}/{len(results)} passed")
return resultsProsody Evaluation
Prosody — the rhythm, stress, and intonation of speech — is harder to measure objectively. The main signal-level features to track are:
- Pitch (F0) trajectory: Monotone pitch is a common TTS regression. Extract F0 and measure variance.
- Speaking rate (syllables/second): Too fast or too slow relative to a baseline signals a regression.
- Pause duration distribution: Unnatural pauses (too long or missing) affect perceived naturalness.
import parselmouth # pip install praat-parselmouth
import numpy as np
def extract_prosody_features(audio_path: str) -> dict:
"""Extract pitch, speaking rate, and pause features using Praat."""
snd = parselmouth.Sound(audio_path)
# Pitch extraction
pitch = snd.to_pitch(time_step=0.01, pitch_floor=75, pitch_ceiling=500)
f0_values = pitch.selected_array["frequency"]
f0_voiced = f0_values[f0_values > 0]
# Intensity for pause detection
intensity = snd.to_intensity(time_step=0.01)
intensity_values = intensity.values.T.flatten()
# Silence threshold: frames below 40 dB are pauses
pause_frames = intensity_values < 40
pause_ratio = pause_frames.mean()
# Detect pause durations
in_pause = False
pause_durations = []
current_pause = 0
frame_duration = 0.01
for is_pause in pause_frames:
if is_pause:
current_pause += frame_duration
in_pause = True
elif in_pause:
pause_durations.append(current_pause)
current_pause = 0
in_pause = False
return {
"f0_mean": float(np.mean(f0_voiced)) if len(f0_voiced) > 0 else 0,
"f0_std": float(np.std(f0_voiced)) if len(f0_voiced) > 0 else 0,
"voiced_fraction": len(f0_voiced) / max(len(f0_values), 1),
"pause_ratio": float(pause_ratio),
"mean_pause_duration_ms": float(np.mean(pause_durations) * 1000) if pause_durations else 0,
"max_pause_duration_ms": float(np.max(pause_durations) * 1000) if pause_durations else 0,
"audio_duration_s": snd.duration,
}
def compare_prosody(baseline_path: str, candidate_path: str,
thresholds: dict = None) -> dict:
"""Compare prosody features between baseline and candidate audio."""
if thresholds is None:
thresholds = {
"f0_std_min": 10.0, # Hz — below this is monotone
"max_pause_ms": 2000, # ms — above this is unnatural
"voiced_fraction_min": 0.5,
}
baseline = extract_prosody_features(baseline_path)
candidate = extract_prosody_features(candidate_path)
issues = []
if candidate["f0_std"] < thresholds["f0_std_min"]:
issues.append(f"Monotone pitch: f0_std={candidate['f0_std']:.1f} Hz "
f"(baseline={baseline['f0_std']:.1f})")
if candidate["max_pause_duration_ms"] > thresholds["max_pause_ms"]:
issues.append(f"Abnormally long pause: {candidate['max_pause_duration_ms']:.0f}ms")
if candidate["voiced_fraction"] < thresholds["voiced_fraction_min"]:
issues.append(f"Low voiced fraction: {candidate['voiced_fraction']:.2f}")
return {
"baseline": baseline,
"candidate": candidate,
"issues": issues,
"passed": len(issues) == 0,
}Regression Testing Across TTS APIs
The final piece is a regression harness that runs all the above checks against multiple TTS providers on every release or weekly schedule.
import requests
import json
import hashlib
from datetime import datetime
# Google Cloud TTS synthesis
def synthesize_google(text: str, language_code: str = "en-US",
voice_name: str = "en-US-Neural2-F",
output_path: str = "google_output.wav") -> str:
"""Synthesize with Google Cloud TTS."""
import google.cloud.texttospeech as tts
client = tts.TextToSpeechClient()
synthesis_input = tts.SynthesisInput(text=text)
voice = tts.VoiceSelectionParams(
language_code=language_code,
name=voice_name,
)
audio_config = tts.AudioConfig(
audio_encoding=tts.AudioEncoding.LINEAR16,
sample_rate_hertz=16000,
)
response = client.synthesize_speech(
input=synthesis_input, voice=voice, audio_config=audio_config
)
with open(output_path, "wb") as f:
f.write(response.audio_content)
return output_path
# ElevenLabs synthesis
def synthesize_elevenlabs(text: str, voice_id: str,
api_key: str,
output_path: str = "elevenlabs_output.mp3") -> str:
"""Synthesize with ElevenLabs API."""
url = f"https://api.elevenlabs.io/v1/text-to-speech/{voice_id}"
headers = {
"xi-api-key": api_key,
"Content-Type": "application/json",
}
payload = {
"text": text,
"model_id": "eleven_multilingual_v2",
"voice_settings": {"stability": 0.5, "similarity_boost": 0.75},
}
response = requests.post(url, json=payload, headers=headers)
response.raise_for_status()
with open(output_path, "wb") as f:
f.write(response.content)
return output_path
# Full regression runner
def run_tts_regression_suite(
test_texts: list[str],
engines: dict, # {"engine_name": synthesize_fn}
baseline_dir: str,
output_dir: str,
mos_threshold: float = 3.8,
pesq_threshold: float = 3.5,
) -> dict:
"""
Run full regression suite: MOS, PESQ, STOI, prosody.
Compares candidate outputs against stored baselines.
"""
from pathlib import Path
import time
results = {"timestamp": datetime.utcnow().isoformat(), "engines": {}}
Path(output_dir).mkdir(parents=True, exist_ok=True)
predictor = UTMOSScore(device="cpu")
for engine_name, synthesize_fn in engines.items():
engine_results = []
for i, text in enumerate(test_texts):
text_id = hashlib.md5(text.encode()).hexdigest()[:8]
candidate_path = f"{output_dir}/{engine_name}_{text_id}.wav"
baseline_path = f"{baseline_dir}/{engine_name}_{text_id}.wav"
# Synthesize candidate
synthesize_fn(text, output_path=candidate_path)
time.sleep(0.2) # rate limiting
# MOS
wav, sr = torchaudio.load(candidate_path)
if sr != 16000:
wav = torchaudio.transforms.Resample(sr, 16000)(wav)
if wav.shape[0] > 1:
wav = wav.mean(0, keepdim=True)
mos = float(predictor.score(wav))
result = {
"text_id": text_id,
"text_preview": text[:60],
"mos": round(mos, 3),
"mos_ok": mos >= mos_threshold,
}
# PESQ/STOI only if baseline exists
if Path(baseline_path).exists():
metrics = compute_pesq_stoi(baseline_path, candidate_path)
result.update(metrics)
prosody = compare_prosody(baseline_path, candidate_path)
result["prosody_issues"] = prosody["issues"]
result["prosody_ok"] = prosody["passed"]
else:
result["note"] = "No baseline — saving as new baseline"
import shutil
shutil.copy(candidate_path, baseline_path)
engine_results.append(result)
passed = sum(1 for r in engine_results if r["mos_ok"]
and r.get("pesq_ok", True)
and r.get("prosody_ok", True))
results["engines"][engine_name] = {
"passed": passed,
"total": len(engine_results),
"samples": engine_results,
}
# Write report
report_path = f"{output_dir}/regression_report.json"
with open(report_path, "w") as f:
json.dump(results, f, indent=2)
return resultsCI Integration and Alerting
Integrate the regression suite into your CI pipeline to catch regressions automatically:
# .github/workflows/tts-regression.yml
name: TTS Quality Regression
on:
schedule:
- cron: "0 6 * * 1" # Weekly Monday 6am
push:
paths:
- "tts/**"
- "tests/tts_regression.py"
jobs:
regression:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Install dependencies
run: pip install utmos pesq pystoi parselmouth stable-ts boto3 requests
- name: Run TTS regression suite
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
ELEVENLABS_API_KEY: ${{ secrets.ELEVENLABS_API_KEY }}
run: python tests/tts_regression.py
- name: Upload regression report
uses: actions/upload-artifact@v4
with:
name: tts-regression-report
path: outputs/regression_report.json
- name: Fail on quality regression
run: python tests/assert_no_regression.py outputs/regression_report.jsonThe assert_no_regression.py script should fail the build if any engine shows MOS below threshold or PESQ regression greater than 0.3 points from baseline:
import json, sys
with open(sys.argv[1]) as f:
report = json.load(f)
failures = []
for engine, data in report["engines"].items():
for sample in data["samples"]:
if not sample.get("mos_ok", True):
failures.append(f"{engine}/{sample['text_id']}: MOS={sample['mos']}")
if sample.get("pesq", 99) < 3.0:
failures.append(f"{engine}/{sample['text_id']}: PESQ={sample['pesq']}")
for issue in sample.get("prosody_issues", []):
failures.append(f"{engine}/{sample['text_id']}: {issue}")
if failures:
print("TTS REGRESSION FAILURES:")
for f in failures:
print(f" - {f}")
sys.exit(1)
print(f"All TTS quality checks passed.")Putting It Together
A mature TTS quality testing program combines these layers:
- UTMOS automated scoring on every synthesis — catches naturalness regressions without human review
- PESQ/STOI against baseline — catches audio processing artifacts (resampling, encoding changes)
- Pronunciation test corpus — domain vocabulary coverage on every model update
- Prosody feature tracking — monotone detection, pause distribution over time
- Weekly human MOS studies on a 50-sentence benchmark corpus — ground truth that neural estimators might miss
- CI regression gate — fails the build if any metric drops below threshold
The test corpus matters as much as the metrics. Cover diverse sentence types: short commands, long narrative sentences, questions, technical terminology, numerals, abbreviations, and edge cases like single-word utterances and sentences with heavy punctuation. A corpus of 200–500 sentences balanced across these categories will catch most regression patterns.
TTS quality is rarely binary — it degrades gradually until it becomes noticeable to end users. Systematic measurement is the only way to catch it early enough to matter.