Testing Metrics That Matter: Defect Escape Rate, MTTR, and Flakiness
Most QA teams track the wrong things. They count bugs opened and closed, celebrate high test case numbers, and report coverage percentages that don't actually measure coverage. These metrics are easy to collect and look good in a dashboard, but they don't tell you whether your testing is working.
The metrics that matter are harder to collect, less flattering, and more honest. This post covers three of them — defect escape rate, mean time to recovery, and flakiness rate — along with how to calculate them, benchmark them, and improve them.
Why Vanity Metrics Are Dangerous
Before the useful metrics, it's worth naming the harmful ones.
Total tests written is a vanity metric. A test suite with 1,000 tests that are all checking whether buttons are visible is worse than a suite with 100 tests that verify actual user workflows end-to-end. More tests isn't better. Better tests are better.
Bug count is a vanity metric. Opening 50 bugs in a sprint can mean your team is thorough, or it can mean the codebase is a disaster, or it can mean QA has low standards for what constitutes a reportable issue. The number alone tells you nothing.
Test pass rate (in isolation) is a vanity metric. 95% pass rate sounds good. If 80% of those tests are skipped or trivially simple, it means nothing. Pass rate only matters in context of what the tests are actually asserting.
The metrics below are different. They measure outcomes, not activity.
Defect Escape Rate
What It Is
Defect escape rate (DER) measures the percentage of bugs that make it to production despite going through your QA process.
Formula:
DER = (Production Bugs / Total Bugs) × 100
Total Bugs = Production Bugs + Pre-Production BugsIf you catch 90 bugs before release and 10 bugs make it to production, your DER is:
10 / (90 + 10) × 100 = 10%What's a Good DER?
There's no universal benchmark — it varies by industry, product type, and release frequency. But as a rough guide:
- Under 5%: Excellent. Your testing is catching the vast majority of issues.
- 5–15%: Acceptable for most product teams.
- 15–30%: Your testing is missing a lot. Either coverage is too shallow or the areas you're testing don't match where bugs are actually occurring.
- Over 30%: This is a process problem, not just a testing problem.
If you're at a startup with no prior QA infrastructure, your DER on day one might be 50%+. That's expected. The goal is to establish a baseline and improve it over time, not to benchmark against companies with mature QA programs.
How to Track It
You need two data sources: your bug tracker (for pre-production bugs) and your incident log or production monitoring (for production bugs).
The hard part is attribution. A bug reported by a customer in production may have been present for weeks before anyone noticed. You need to decide how to count it — when it was introduced, when it was found, or when it was fixed? Pick a consistent definition and stick with it. The exact definition matters less than consistency.
Tag all bugs in your tracker with "found in: testing" or "found in: production." That's the data collection. The metric calculation follows automatically.
How to Improve It
A high DER usually means one of three things:
- Wrong test coverage: You're testing the wrong areas. Compare where production bugs are found to where your test coverage is concentrated. If most production bugs are in the checkout flow and most of your tests are in the settings pages, that's the problem.
- Shallow test depth: Tests are passing but not asserting meaningful outcomes. A test that checks "login page loads" is not the same as a test that checks "user can log in with valid credentials and is redirected to the correct dashboard."
- Missing regression testing: You have good coverage of features as they were built, but not of interactions between features. Regression is where most production bugs originate.
Mean Time to Recovery (MTTR)
What It Is
MTTR measures how long it takes to restore service after a production incident. It's usually defined as the time from when an incident is detected to when it's resolved.
Formula:
MTTR = Total Downtime / Number of IncidentsIf you had 4 production incidents in a month with downtimes of 15 min, 45 min, 2 hours, and 30 min:
MTTR = (15 + 45 + 120 + 30) / 4 = 52.5 minutesWhy QA Cares About MTTR
MTTR is traditionally an SRE metric, not a QA metric. But QA teams should track it for two reasons:
First, your test coverage directly affects how quickly developers can diagnose and fix production issues. When a production bug is found, the first question is: "why didn't our tests catch this?" A test suite with good coverage usually means the bug is in a known, well-mapped area of the code. A test suite with poor coverage means you're debugging in the dark.
Second, monitoring is part of the QA function at many organizations. If you own the automated monitoring that detects production issues, your MTTR starts the clock on your watch. Improving monitoring sensitivity reduces detection time, which reduces MTTR.
What's a Good MTTR?
- Under 15 minutes: Excellent. Requires good monitoring, fast CI/CD rollback capability, and clear incident runbooks.
- 15–60 minutes: Good for most B2B SaaS products.
- 1–4 hours: Common for teams without mature incident management processes.
- Over 4 hours: Needs attention. Either monitoring is poor (late detection), rollback is manual and slow, or the root cause analysis process is chaotic.
How to Improve MTTR
The fastest wins are in detection speed. If incidents are caught by user complaints rather than automated monitoring, detection time alone could be adding hours to MTTR.
Set up automated health checks on your critical paths. A monitor that checks your login flow, core API endpoints, and payment flow every 5 minutes will catch most incidents within 5 minutes of occurrence — instead of whenever someone reports it. Tools like HelpMeTest run these kinds of checks on a schedule (every 15–30 minutes by default, configurable) and alert immediately on failure.
The second lever is rollback speed. If every production fix requires a full deploy cycle, MTTR is constrained by your deploy pipeline. Push for feature flags, blue-green deployments, or one-click rollback capability. That's not strictly a QA concern, but advocating for it is.
Flakiness Rate
What It Is
A flaky test is one that produces different results on different runs without any code change. It passes 90% of the time and fails 10% of the time for reasons unrelated to actual product behavior — timing issues, external dependencies, race conditions, order-dependent state.
Formula:
Flakiness Rate = (Flaky Test Failures / Total Test Runs) × 100If you run a suite of 100 tests 50 times in a month (5,000 total test executions), and 300 of those executions fail due to flakiness:
Flakiness Rate = 300 / 5,000 × 100 = 6%Why Flakiness Is Catastrophic
Flakiness doesn't just waste time re-running tests. It destroys the trust signal that your test suite provides.
When a test fails, the expected response is "something is broken, we need to investigate." When tests are chronically flaky, the actual response becomes "probably just a flake, re-run it." And once that culture is established, you can't tell the difference between a real failure and a flake without manual investigation — which means real failures get missed.
A test suite with high flakiness is worse than no test suite in one critical way: it gives the team false confidence that they have a safety net when they don't.
Google's engineering research found that flaky tests were one of the most productivity-destroying problems in their CI pipelines, leading teams to lose trust in test results entirely.
What's an Acceptable Flakiness Rate?
- Under 1%: Good. Some flakiness is nearly unavoidable in large suites with external dependencies.
- 1–3%: Tolerable, but worth actively tracking and reducing.
- 3–10%: This is a problem that's degrading team trust in the test suite.
- Over 10%: The test suite is effectively unreliable. Developers are ignoring failures. This needs to be treated as a crisis.
How to Find Flaky Tests
Most CI systems will show you test runs over time. Look for tests that have both pass and fail results in recent history without corresponding code changes. That's the flakiness signature.
Some CI platforms (like GitHub Actions with third-party integrations) will automatically flag tests that have inconsistent results. If you're running on a simpler setup, a weekly review of CI results to identify tests that fail intermittently is a reasonable manual process.
You can also instrument this:
# Run the test suite 10 times and check for inconsistent results
for i in {1..10}; do
npx playwright test --reporter=json >> test-results.json
done
# Then parse test-results.json to find tests that pass some runs and fail othersHow to Fix Flaky Tests
Explicit waits, not implicit waits or arbitrary sleeps. The most common cause of UI test flakiness is timing. Tests that use sleep(2000) to wait for an element to appear will fail on slow CI machines and pass on fast developer machines. Replace sleeps with explicit waits for specific conditions:
// Bad
await page.click('#submit');
await page.waitForTimeout(2000);
await expect(page.locator('#success')).toBeVisible();
// Good
await page.click('#submit');
await expect(page.locator('#success')).toBeVisible({ timeout: 10000 });Isolate test state. Tests that depend on database state from previous tests are inherently flaky when run in different orders. Each test should set up its own state and clean up after itself.
Mock external dependencies. Tests that call real external APIs (email, payment, third-party services) will fail when those APIs have issues. Mock them in tests, or have dedicated test accounts with predictable behavior.
Quarantine before fixing. When you identify a flaky test, quarantine it (skip it with a tag, move it to a separate suite) while you fix it. Don't let it continue to pollute the main suite.
Putting It Together: A Simple Metrics Dashboard
You don't need fancy tooling to track these metrics. A spreadsheet updated monthly works. What matters is that you're tracking them at all and that the team can see the trend.
Suggested fields to track monthly:
| Metric | This Month | Last Month | 3-Month Avg | Target |
|---|---|---|---|---|
| Defect Escape Rate | 12% | 15% | 14% | <10% |
| MTTR (avg) | 35 min | 52 min | 45 min | <30 min |
| Flakiness Rate | 4% | 7% | 6% | <2% |
Trends matter more than absolute numbers. A DER that went from 25% to 12% in three months tells a better story than a static 10% with no improvement trajectory.
Share this with engineering leadership monthly. Connect the metrics to business outcomes: "Our DER dropped by 8% this quarter, which means fewer customer-reported bugs and less emergency engineering time on production fixes." That's a language leadership understands.
The Underlying Principle
The goal of QA metrics is not to look good. It's to find problems in your testing process before they show up as customer-facing failures. Defect escape rate, MTTR, and flakiness rate each point to a different kind of failure mode. Track all three, improve them consistently, and you'll have a QA program that actually delivers on its promise — fewer production incidents, faster recovery, and a test suite the team trusts.
That's measurably better quality. Everything else is noise.