QA Metrics That Matter: What to Track in Test Management Tools

QA teams measure the wrong things. They count test cases written (volume, not quality). They count bugs found (output, not outcomes). They report pass rates on test runs that only include the easy cases. The result is a dashboard full of green metrics and a production environment full of bugs.

Good QA metrics answer one question: how likely is a bug to reach production and how quickly do we find it when it does? Everything else is noise. This guide covers the metrics that actually answer that question, how to calculate them, and how to set up dashboards in TestRail, Jira/Xray, and Qase.

Test Coverage: What Percentage of Requirements Are Tested

Coverage measures whether your test cases actually exercise the product's behavior. There are two meaningful forms:

Requirement coverage: percentage of user stories, acceptance criteria, or requirements that have at least one test case linked to them.

Requirement Coverage = (Requirements with ≥1 linked test) / (Total requirements) × 100

Execution coverage: of the requirements that have tests, what percentage have been executed and passed in the current sprint or release.

Execution Coverage = (Requirements with all linked tests passing) / (Requirements with tests) × 100

The gap between these two numbers tells you something important. High requirement coverage + low execution coverage means you have the tests but aren't running them (stale automation, skipped manual runs). Low requirement coverage means you're shipping features without testing them.

How to track in TestRail:

TestRail doesn't natively link tests to requirements, but you can approximate with milestones. Create a custom field on test cases for "User Story ID". Then use the Query function to count cases by story ID and compare to your story count from Jira.

The better approach: use Xray in Jira, which calculates requirement coverage natively in the Coverage Report.

How to track in Xray:

Xray's Requirement Coverage Report shows every story in the current sprint with its coverage status (OK, NOK, Not Run). Filter by sprint, version, or label. The dashboard gadget shows a simple percentage.

Target: 100% requirement coverage for new features before they ship. Not 90%, not "most". 100%. If you don't have a test for it, you don't know it works.

Defect Escape Rate: Bugs That Reach Production

This is the single most important QA metric. It measures your team's effectiveness at finding bugs before users do.

Defect Escape Rate = (Bugs found in production) / (Total bugs found) × 100

Calculate it per sprint or per release. "Total bugs found" includes bugs found in QA plus bugs found in production.

A 5% escape rate means 95% of bugs were caught before production. A 20% escape rate means 1 in 5 bugs makes it to users. Most mature QA teams target below 10%.

Track this separately for bug severity:

• Critical defect escape rate (should be 0%)
• High defect escape rate (target <5%)
• Medium defect escape rate (target <15%)

How to calculate:

In Jira, create a filter for issues with type = Bug, and add a custom field "Found in" with values "QA" and "Production". Run the filter weekly and export to a spreadsheet.

A simpler proxy: use the Jira issue label "production-bug" on any bug reported by users or monitoring. Compare count of "production-bug" issues to total bugs created in the same time period.

Tracking trends matters more than absolute numbers. A team improving from 25% to 15% escape rate over three months is doing better work than a team stuck at 10% for a year. Plot it on a run chart, not just a single number.

Test Execution Pass Rate: Signal vs. Noise

Pass rate is the most commonly tracked QA metric and the most commonly misread.

Pass Rate = (Tests Passed) / (Total Tests Executed) × 100

A 95% pass rate sounds great. But it's meaningless without context:

• Did you only run smoke tests? (Easy subset)
• Did you mark flaky tests as blocked? (Ignoring failures)
• Did you rerun failing tests until they passed? (Washing away real failures)
• Is the 5% failure rate in critical flows or low-priority edge cases?

Track pass rate at multiple granularities:

By test type: Critical test pass rate, regression pass rate, new feature pass rate. A 100% critical pass rate and 70% regression pass rate tells you something different than 85% across the board.

By feature area: Authentication pass rate, payment flow pass rate, etc. Dropping pass rate in a specific section points to a problem area.

By sprint: Pass rate trend over time. Improving or declining?

How to set up in TestRail:

Create test plans with separate runs per area. The summary report on a test plan shows pass rates per run. Schedule the "Summary" report to email to the team weekly.

How to set up in Qase:

Qase's Analytics dashboard shows pass rate over time automatically. Configure it to show the last 10 sprints for trend visibility. The "Test Run Report" breaks down pass rate by suite section.

Mean Time to Detect (MTTD): How Fast Do You Find Bugs

MTTD measures the time between when a bug is introduced (usually when code is merged) and when it's detected (when a test fails or QA reports it).

MTTD = (Time bug detected) - (Time bug was introduced)

A bug introduced Monday that's caught in QA Thursday has an MTTD of 3 days. A bug introduced Monday that's caught by a user the following Thursday has an MTTD of 10 days.

Reducing MTTD requires:

• Faster CI runs (automated tests run within minutes of merge)
• More frequent manual testing (daily exploratory sessions vs. end-of-sprint marathon)
• Better monitoring (alerts that fire when error rates increase in production)

How to approximate MTTD in Jira:

Filter bugs to the current sprint. For each bug, calculate: (created date) - (related feature PR merge date). This is an approximation — the PR merge date as "introduced" date works for most software bugs.

Create a custom Jira dashboard gadget using a filter that calculates average time-in-sprint for bugs before they were created. The Jira Built-in "Average Age" calculation across your bug filter gives you a rough MTTD proxy.

Target: MTTD under 24 hours for critical code paths. Under 1 week for full regression coverage. Anything longer suggests your test suite doesn't cover the area that broke.

Test Suite Health: Flakiness Rate

Flakiness is the silent killer of test suite reliability. A flaky test passes sometimes and fails sometimes without any change in the code. It trains your team to ignore failures ("it's probably just that flaky test"), which means real failures get ignored too.

Flakiness Rate = (Tests that produce inconsistent results) / (Total automated tests) × 100

Identifying flaky tests: run your full automated suite 5 times with no code changes. Any test that changes result between runs is flaky.

Categories of flakiness:

Timing-dependent: Test passes when the system is fast, fails when it's slow. Usually an async operation without proper await handling.

State-dependent: Test passes when run alone, fails when run with other tests. Usually shared state not being reset between tests.

Environment-dependent: Test passes locally, fails in CI. Usually missing environment setup, different file paths, or timezone issues.

Data-dependent: Test passes with one set of test data, fails with another. Usually insufficient test data isolation.

How to track in CI:

Add flakiness tracking to your CI pipeline. For each test run, record results in a database. Calculate pass rate per test over the last 100 runs. Any test with a pass rate between 10% and 90% is flaky.

# Simple bash script to identify flaky tests from JUnit XML across multiple runs
python scripts/analyze_flakiness.py \
  --run-count 20 \
  --threshold 0.90 \
  --output flaky-tests-report.csv

Target: Under 2% flakiness rate. Zero flaky tests in your smoke suite (flaky smoke tests are the most damaging because they block CI and erode trust).

Setting Up Dashboards

TestRail Dashboard

TestRail's dashboard system uses pre-built reports and custom views:

1. Test Plan Summary (built-in): Go to a milestone, click "Reports > Summary". Shows pass/fail counts, completion percentage. Schedule it to email stakeholders weekly.
2. Activity Report (built-in): Shows testing volume over time by tester. Use it in sprint retrospectives to see test throughput.
3. Custom Report for Coverage: Create a saved query filtering to "Failed" tests with Priority = Critical. Run this before every release. If the result is non-zero, you don't ship.
4. Defect Link Export: TestRail can export all test cases with their linked defect IDs to CSV. Import this to a spreadsheet to calculate escape rate manually.

Jira/Xray Dashboard

Build a Jira dashboard with these gadgets:

1. Xray Test Execution Coverage (Xray gadget): Coverage % for current sprint
2. Two-Dimensional Filter Statistics: Bugs by status × sprint (shows production bugs per sprint)
3. Pie Chart by Issue Status: Filter = bugs this sprint; shows open vs. closed
4. Created vs. Resolved Chart: Bug velocity — are you creating bugs faster than you're resolving them?

Configure the dashboard to refresh daily and share it with the whole team. When developers can see test coverage and bug rates on their sprint board, they pay more attention to QA outcomes.

Qase Analytics Dashboard

Qase's built-in analytics tracks:

• Test runs per week (volume trend)
• Pass rate trend (quality trend)
• Defects created from test runs (by severity)
• Test case coverage (cases executed vs. not run)

The Analytics tab requires no configuration — it builds from your actual run data. After three sprints, you'll have meaningful trends.

For custom metrics not covered natively, use Qase's API to export run data and build a spreadsheet or Grafana dashboard.

Using Metrics in Sprint Retrospectives

Metrics are only useful if they change behavior. The format that works:

Before the sprint: "Last sprint, our defect escape rate was 12%. Our critical test pass rate was 94%. Our flakiness rate is 8%. This sprint, we're targeting <10% escape rate and eliminating the 4 flaky tests in the authentication suite."

During the sprint: Check pass rate after every major deployment. If critical test pass rate drops below 90%, stop and investigate before adding features.

After the sprint: Present the same metrics: did escape rate improve? Did flakiness decrease? Celebrate wins and investigate misses.

Metrics without targets are trivia. Targets without tracking are wishes. The combination — a tracked metric against a stated goal, reviewed regularly — is a quality improvement program.

The teams that ship reliable software aren't the ones with the most comprehensive test plans. They're the ones that look at three or four metrics obsessively and actually change their process when the numbers move in the wrong direction.