Testing

Flakiness Metrics and Dashboards for CI Pipelines

HelpMeTest

HelpMeTest

— 6 min read
Flakiness Metrics and Dashboards for CI Pipelines

You can't manage what you can't measure. This is as true for flaky tests as it is for application performance. Teams that "know they have flaky tests" but don't measure them will always have too many flaky tests—because there's no data to prioritize fixes, no way to know if improvements are working, and no visibility into which tests are getting worse.

This guide covers the metrics that matter, how to collect them, and how to build dashboards that make flakiness visible and actionable.

The Metrics That Matter

Not all test data is equally useful. Focus on these five metrics:

1. Flakiness Rate Per Test

The percentage of runs where a test produced an inconsistent result (failed on first attempt but passed on retry, or passed sometimes and failed others across identical commits).

flakiness_rate = (runs_with_inconsistent_result / total_runs) * 100

A test with a 5% flakiness rate fails about once in every 20 runs. On a team running CI 20 times per day, that's a failure every day.

2. Mean Time to Detect (MTTD)

How long does it take for a flaky test to be noticed and addressed? If tests are flaky for weeks before anyone acts, your measurement system isn't working.

3. Quarantine Inventory

How many tests are currently quarantined? What's the age distribution? Are quarantined tests getting fixed or accumulating?

4. False Positive Rate

What percentage of CI failures are caused by flaky tests rather than real bugs? This is the number that tells you how much your team trusts CI. If developers are regularly ignoring red builds because "it's probably just a flaky test," this number is too high.

5. Flakiness Trend

Is the situation getting better or worse over the last 30 days? A rising trend needs immediate attention; a falling trend indicates your fixes are working.

Collecting Test Result Data

The foundation of any metrics system is collecting structured test results from every CI run.

Step 1: Emit Structured Results

Most test frameworks can output JSON:

# Jest
npx jest --json --outputFile=test-results.json

# pytest
pytest --json-report --json-report-file=test-results.json

# Go
go test ./... -json > test-results.json

# Playwright
npx playwright test --reporter=json > test-results.json

Step 2: Enrich with Context

Raw test results aren't enough. Enrich them with:

import json
import os
import subprocess
from datetime import datetime

def enrich_test_results(raw_results_file, output_file):
    with open(raw_results_file) as f:
        results = json.load(f)

    # Add context
    results['metadata'] = {
        'run_id': os.environ.get('GITHUB_RUN_ID', 'local'),
        'commit_sha': subprocess.check_output(['git', 'rev-parse', 'HEAD']).decode().strip(),
        'branch': os.environ.get('GITHUB_REF_NAME', 'local'),
        'runner': os.environ.get('RUNNER_NAME', 'local'),
        'timestamp': datetime.utcnow().isoformat(),
        'retry_count': int(os.environ.get('TEST_RETRY_COUNT', '0')),
    }

    with open(output_file, 'w') as f:
        json.dump(results, f, indent=2)

Step 3: Store Results

Options by complexity:

Simple: GitHub Actions artifacts + Python analysis

- name: Upload test results
  uses: actions/upload-artifact@v4
  with:
    name: test-results-${{ github.run_id }}
    path: test-results.json
    retention-days: 30

Better: Time-series database

import psycopg2
import json

def store_results(results_file, db_url):
    with open(results_file) as f:
        results = json.load(f)

    conn = psycopg2.connect(db_url)
    cur = conn.cursor()

    for test in results['testResults']:
        for assertion in test['assertionResults']:
            cur.execute("""
                INSERT INTO test_runs
                    (test_name, test_file, status, duration_ms, run_id, commit_sha, timestamp)
                VALUES (%s, %s, %s, %s, %s, %s, %s)
            """, (
                assertion['fullName'],
                test['testFilePath'],
                assertion['status'],
                assertion.get('duration'),
                results['metadata']['run_id'],
                results['metadata']['commit_sha'],
                results['metadata']['timestamp'],
            ))

    conn.commit()

Production-grade: BigQuery, ClickHouse, or DataDog

Most large engineering teams use a columnar database for test analytics because of query performance at scale. ClickHouse is common for self-hosted; BigQuery for Google Cloud shops.

Identifying Flaky Tests from Historical Data

Once you have a history of test results, flaky tests emerge clearly:

-- Tests with inconsistent results on the same commit
SELECT
    test_name,
    commit_sha,
    COUNT(DISTINCT status) as distinct_statuses,
    COUNT(*) as total_runs,
    SUM(CASE WHEN status = 'failed' THEN 1 ELSE 0 END) as failures
FROM test_runs
WHERE timestamp > NOW() - INTERVAL '7 days'
GROUP BY test_name, commit_sha
HAVING COUNT(DISTINCT status) > 1  -- both passed and failed on same commit
ORDER BY failures DESC;
-- Overall flakiness rate per test (last 30 days)
WITH test_stats AS (
    SELECT
        test_name,
        COUNT(*) as total_runs,
        SUM(CASE WHEN status = 'failed' THEN 1 ELSE 0 END) as failures,
        -- A test is "flaky" on a commit if it produced both pass and fail
        COUNT(DISTINCT CASE
            WHEN EXISTS (
                SELECT 1 FROM test_runs t2
                WHERE t2.test_name = t1.test_name
                  AND t2.commit_sha = t1.commit_sha
                  AND t2.status != t1.status
            ) THEN commit_sha
        END) as flaky_commits
    FROM test_runs t1
    WHERE timestamp > NOW() - INTERVAL '30 days'
    GROUP BY test_name
)
SELECT
    test_name,
    total_runs,
    failures,
    ROUND(100.0 * flaky_commits / total_runs, 2) as flakiness_rate
FROM test_stats
WHERE total_runs >= 10  -- only tests with enough data
ORDER BY flakiness_rate DESC
LIMIT 20;

Building the Dashboard

Grafana with PostgreSQL/ClickHouse

Grafana connects to most SQL databases and works well for CI dashboards.

Key panels:

1. Flakiness Rate Over Time (30-day trend)

SELECT
    date_trunc('day', timestamp) as time,
    AVG(CASE WHEN is_flaky THEN 100.0 ELSE 0 END) as flakiness_rate
FROM daily_test_runs
WHERE timestamp > NOW() - INTERVAL '30 days'
GROUP BY 1
ORDER BY 1;

2. Top 10 Flakiest Tests (this week)

SELECT test_name, flakiness_rate, total_runs
FROM test_flakiness_stats
WHERE period = 'week'
ORDER BY flakiness_rate DESC
LIMIT 10;

3. Quarantine Inventory

SELECT
    test_name,
    quarantined_at,
    expected_fix_date,
    assignee,
    CASE WHEN expected_fix_date < NOW() THEN 'overdue' ELSE 'on_track' END as status
FROM quarantine_registry
WHERE resolved_at IS NULL
ORDER BY quarantined_at ASC;

4. CI False Positive Rate

SELECT
    date_trunc('week', timestamp) as week,
    COUNT(CASE WHEN failure_type = 'flaky' THEN 1 END) as flaky_failures,
    COUNT(CASE WHEN failure_type = 'real' THEN 1 END) as real_failures,
    ROUND(100.0 * COUNT(CASE WHEN failure_type = 'flaky' THEN 1 END) / COUNT(*), 1) as false_positive_rate
FROM ci_failures
WHERE timestamp > NOW() - INTERVAL '90 days'
GROUP BY 1
ORDER BY 1;

DataDog

If you're already using DataDog, use custom metrics:

from datadog import initialize, statsd

initialize(api_key=os.environ['DD_API_KEY'])

def report_test_results(results):
    for test in results['tests']:
        tags = [
            f"test:{test['name']}",
            f"file:{test['file']}",
            f"branch:{os.environ.get('BRANCH', 'unknown')}",
        ]

        if test['status'] == 'passed':
            statsd.increment('ci.test.passed', tags=tags)
        elif test['status'] == 'failed':
            statsd.increment('ci.test.failed', tags=tags)

        if test.get('retries_needed', 0) > 0:
            statsd.increment('ci.test.retried', tags=tags)
            statsd.gauge(
                'ci.test.retries_needed',
                test['retries_needed'],
                tags=tags
            )

This gives you DataDog monitors that can alert when:

  • Any test's flakiness rate exceeds 10% over 7 days
  • Quarantine inventory exceeds 15 tests
  • CI false positive rate exceeds 20%

Lightweight Option: GitHub Actions Summary

If you don't want a dedicated metrics system yet, GitHub Actions step summaries provide immediate visibility:

# generate-summary.py
import json
import os

with open('test-results.json') as f:
    results = json.load(f)

retried = [t for t in results['tests'] if t.get('retries_needed', 0) > 0]
failed = [t for t in results['tests'] if t['status'] == 'failed']

summary = f"""## Test Results

| Metric | Value |
|--------|-------|
| Total Tests | {len(results['tests'])} |
| Passed | {len([t for t in results['tests'] if t['status'] == 'passed'])} |
| Failed | {len(failed)} |
| Required Retries | {len(retried)} |

"""

if retried:
    summary += "### Tests Requiring Retries\n\n"
    summary += "| Test | Retries Needed |\n|------|----------------|\n"
    for t in retried:
        summary += f"| `{t['name']}` | {t['retries_needed']} |\n"

# Write to GitHub Actions step summary
with open(os.environ['GITHUB_STEP_SUMMARY'], 'a') as f:
    f.write(summary)

Alerting on Flakiness

Dashboards are passive—you have to look at them. Alerts are active.

Alert: New Flaky Test Detected

# Run after each CI build
def check_for_new_flaky_tests(results_file, history_db):
    with open(results_file) as f:
        results = json.load(f)

    commit = results['metadata']['commit_sha']
    new_flaky = []

    for test in results['tests']:
        if test.get('retries_needed', 0) > 0:
            # Was this test flaky before this commit?
            historical_flakiness = history_db.get_flakiness_rate(
                test['name'],
                before_commit=commit,
                window_days=14
            )
            if historical_flakiness < 0.05:  # was stable before
                new_flaky.append(test['name'])

    if new_flaky:
        notify_slack(
            channel='#engineering',
            message=f"🟡 New flaky tests detected in {commit[:7]}:\n" +
                    "\n".join(f"  - `{t}`" for t in new_flaky)
        )

Alert: Flakiness Threshold Exceeded

# DataDog monitor
monitors:
  - name: "CI Flakiness Rate Too High"
    type: metric alert
    query: "avg(last_7d):avg:ci.test.flakiness_rate{*} > 15"
    message: |
      CI flakiness rate has exceeded 15% over the last 7 days.
      Current rate: {{value}}%
      
      Check the flakiness dashboard: https://grafana.example.com/d/flakiness
      
      @engineering-oncall

Targets and SLOs

What's an acceptable flakiness rate? Industry benchmarks:

Maturity Level Flakiness Rate False Positive Rate
Excellent < 1% < 2%
Good 1–5% 2–10%
Acceptable 5–10% 10–20%
Needs work > 10% > 20%

Set explicit targets and track against them:

FLAKINESS_SLO = {
    'max_rate': 0.05,          # 5% overall
    'max_per_test': 0.15,      # no single test above 15%
    'max_quarantine': 10,      # max 10 quarantined tests
    'max_false_positives': 0.10  # 10% of CI failures are flaky
}

Connecting Metrics to Action

Metrics are only useful if they drive decisions:

  1. Weekly review: top 5 flakiest tests get owners assigned
  2. Quarantine audit: every 2 weeks, review quarantine list; expired items get escalated
  3. Trend gate: if flakiness rate is rising 3 weeks in a row, freeze new features until it stabilizes
  4. Hire/tooling signal: if flakiness persists above SLO despite fixing, the root cause may be infrastructure

HelpMeTest Integration

HelpMeTest's monitoring continuously runs your test suite against your live application and tracks pass/fail rates over time. This data feeds directly into flakiness detection: when a test passes 9 out of 10 runs, HelpMeTest surfaces it as a flakiness candidate with the full trace from the failing run.

This continuous external view complements CI metrics by separating infrastructure-related flakiness (which appears only in CI) from application-related flakiness (which appears in both CI and monitoring).

Summary

The five metrics that matter:

  1. Flakiness rate per test — which tests are unreliable
  2. Mean time to detect — how quickly flakiness is noticed
  3. Quarantine inventory — backlog of known flaky tests
  4. False positive rate — how much CI trust is being eroded
  5. Flakiness trend — is the situation improving

The minimum viable measurement setup:

  • JSON test output on every CI run
  • Store results with commit SHA, run ID, and timestamp
  • Weekly SQL query to find tests failing intermittently
  • Slack alert when the flakiness rate crosses your SLO

Once you can see the problem clearly, you can fix it systematically.

Read more

Start now free