Building a Flaky Test Dashboard: What to Track and How

You can't manage flakiness you can't measure. Teams that successfully eliminate flaky tests share one common trait: they track the data. Not ad hoc — systematically, run after run, test by test.

A flaky test dashboard gives you visibility into which tests are unreliable, how much they cost you, and whether your fixes are working. Here's what to track and how to build it.

What to Track

1. Flakiness Rate Per Test

The core metric: what percentage of runs does each test fail?

flakiness_rate = failed_runs / total_runs

Calculate this over a rolling window — the last 30 days or last 500 runs, whichever is less. A test that failed 3 times last month but is now fixed shouldn't still appear at the top of your list.

Group by:

• Individual test name
• Test file/module
• Test suite

This lets you spot both individual flaky tests and entire modules that are systematically unreliable.

2. Flakiness Trend

Is flakiness getting better or worse over time? A static count doesn't tell you this. Track flakiness rate over time so you can see:

• Whether your fixes are working
• Whether new code is introducing new flakiness
• Seasonal patterns (does CI degrade under certain load conditions?)

A simple chart with flakiness rate on Y and time on X, broken down by test file, shows trends immediately.

3. Engineering Time Lost

Raw flakiness rate doesn't capture business impact. A test that fails 20% of the time in a 5-minute suite costs less than a test that fails 5% of the time in a 60-minute suite.

Track estimated time lost:

time_lost_per_month = (flakiness_rate × runs_per_month × avg_rerun_time)

This converts technical debt into a number your manager can understand. "We have 14 flaky tests" lands differently than "flaky tests cost us 40 engineer-hours per month."

4. Build Impact Rate

How often does a flaky test cause a build to fail that would have otherwise passed?

build_impact_rate = builds_with_flaky_failure / total_builds

If 1 in 5 builds fails because of flaky tests, that means 20% of your team's CI runs are false negatives. This metric captures how much friction flakiness is actually creating.

5. Time-to-Detection

How long after a test becomes flaky does your team notice? If tests run on a schedule and you track results, you can measure detection lag.

Low detection lag (hours) means your monitoring is working. High detection lag (days or weeks) means flakiness is accumulating invisibly.

6. Quarantine Status

Track which tests are currently quarantined, how long they've been quarantined, and who owns the fix. This prevents quarantine from becoming a permanent state.

How to Collect the Data

Option 1: CI metadata + database

Most CI platforms provide test result metadata via API or artifacts. Parse the test results (JUnit XML, TAP, or JSON) and store them in a time-series database.

Collecting from GitHub Actions:

# .github/workflows/test.yml
- name: Run tests
  run: npm test -- --reporters=json > test-results.json

- name: Upload results
  uses: actions/upload-artifact@v3
  with:
    name: test-results
    path: test-results.json

Then a separate job or service pulls results from completed workflow runs and stores them:

// collect-results.js
const { Octokit } = require('@octokit/rest');
const db = require('./db');

async function collectResults(workflowRunId) {
  const artifacts = await octokit.actions.listWorkflowRunArtifacts({
    owner, repo, run_id: workflowRunId
  });
  
  const resultsArtifact = artifacts.data.artifacts.find(a => a.name === 'test-results');
  // download, parse, store in db
}

Schema for flakiness tracking:

CREATE TABLE test_runs (
  id          SERIAL PRIMARY KEY,
  run_date    TIMESTAMP,
  commit_sha  VARCHAR(40),
  test_name   TEXT,
  test_file   TEXT,
  status      VARCHAR(10), -- 'pass' or 'fail'
  duration_ms INTEGER
);

-- Flakiness view: tests with both pass and fail in same period
CREATE VIEW flaky_tests AS
SELECT 
  test_name,
  test_file,
  COUNT(*) FILTER (WHERE status = 'fail') as fail_count,
  COUNT(*) as total_count,
  ROUND(COUNT(*) FILTER (WHERE status = 'fail')::numeric / COUNT(*) * 100, 1) as flakiness_pct
FROM test_runs
WHERE run_date > NOW() - INTERVAL '30 days'
GROUP BY test_name, test_file
HAVING 
  COUNT(*) FILTER (WHERE status = 'fail') > 0 AND
  COUNT(*) FILTER (WHERE status = 'pass') > 0
ORDER BY flakiness_pct DESC;

Option 2: Use your test runner's built-in analytics

Several platforms have flakiness detection built in:

• GitHub Actions: test result dashboards with flakiness detection (in beta)
• CircleCI: Insights dashboard tracks test performance over time
• Buildkite: Test Analytics tracks flakiness natively
• Datadog CI Visibility: tracks test metrics including flakiness rate

If your CI platform has this capability, use it — the integration cost is much lower than building custom tooling.

Option 3: Continuous test monitoring

Tools like HelpMeTest run your tests on a schedule and track pass rates across runs. This catches flakiness that only surfaces in certain environments or at certain times — not just during your regular CI runs.

Continuous monitoring is particularly useful for catching flakiness that's environment-dependent (works on developer laptops, fails on CI) or time-dependent (only fails late at night when shared test databases have accumulated state).

Building the Dashboard

You don't need a complex BI tool. A simple dashboard with four views is enough:

View 1: Current Flakiness Leaderboard

A sorted table of your flakiest tests right now:

Filter by status (active vs. quarantined), sort by flakiness rate or estimated time lost.

View 2: Trend Chart

Line chart showing overall flakiness rate over the last 90 days. Annotate significant events: major releases, infrastructure changes, fixes applied.

Flakiness Rate (%)
15% |     *
    |    * *
10% |   *   *
    |  *     * *
 5% | *         * *_____
    +---------------------> time

A downward trend tells you fixes are working. A sudden spike tells you something changed.

View 3: Impact Heatmap

Which test files have the most flakiness? A heatmap by module shows where to focus cleanup efforts:

auth/           ████████░░ 8 flaky tests
checkout/       █████░░░░░ 5 flaky tests
user-profile/   ██░░░░░░░░ 2 flaky tests
notifications/  █░░░░░░░░░ 1 flaky test

View 4: Quarantine Tracker

Pending fixes with age and owner:

Highlight tests approaching or exceeding your quarantine time limit.

Simple Implementation with Grafana

If you're already using Grafana for infrastructure monitoring, you can add test metrics to the same stack:

# push-test-metrics.py — run after each CI run
import requests
import json

def push_to_influxdb(test_name, status, duration_ms):
    line = f'test_result,test_name={test_name} status={"1" if status == "pass" else "0"},duration={duration_ms}'
    requests.post(
        'http://your-influxdb:8086/write?db=tests',
        data=line,
        headers={'Authorization': 'Token your-token'}
    )

Then build Grafana panels on top:

• Stat panel: current flakiness rate (last 7 days)
• Table panel: top 10 flakiest tests
• Time series panel: flakiness rate trend
• Bar chart: flakiness by test file

The whole setup takes a few hours and gives you persistent visibility into test reliability.

Using the Dashboard

Data without action is just noise. Build habits around the dashboard:

Weekly review: spend 15 minutes reviewing the leaderboard. Are the same tests still flaky? Are quarantined tests getting fixed? Is the overall rate trending down?

Prioritization: fix flaky tests by estimated time impact, not just flakiness rate. A test that fails 30% of the time in a fast suite costs less than one that fails 10% of the time in a slow suite.

Celebrate progress: when flakiness rate drops, acknowledge it. Teams that feel like they're making progress keep making progress.

Set a target: define what "acceptable" looks like. Many teams target <1% flakiness rate for their required CI gate. Having a concrete target makes progress measurable.

What Good Looks Like

A team with a healthy test reliability posture has:

• Flakiness rate below 1% for tests in the required CI gate
• No test quarantined longer than 14 days without an explicit decision (fix or delete)
• Detection lag under 24 hours (you know about flakiness the day it starts)
• Flakiness trend line that's flat or decreasing over 90 days

Getting there takes months of consistent effort. The dashboard doesn't fix the tests — it tells you where to look and whether the fixes are working.