Flaky Test Management: Detection, Quarantine, and Prevention

08 Oct 2026 — 6 min read

A flaky test is a test that sometimes passes and sometimes fails for the same code. It's not a test that occasionally fails — it's a test that randomly fails even when nothing changed.

Flaky tests are expensive. They trigger false CI failures, require manual re-runs, create alert fatigue, and erode trust in the test suite. When developers stop trusting test failures, real bugs slip through.

This guide covers detecting, quarantining, and eliminating flaky tests.

Why Flaky Tests Matter

They mask real failures. When CI fails, developers check whether it's a flaky test. If the failure is real, they often dismiss it anyway. A test suite with 5% flakiness makes it impossible to distinguish random noise from a real regression.

They waste CI time. Re-running tests to confirm real vs. flaky failures consumes compute time. A pipeline that requires 3 re-runs to confirm stability is 3× more expensive than a stable pipeline.

They erode trust slowly. One flaky test is annoying. Ten is a crisis. Teams with high flakiness disable test suites entirely — "the tests are unreliable anyway."

Industry data: Google's internal research found that 1.5% of their test runs contained at least one flaky test. At scale, that's millions of wasted CI runs per year.

Root Causes of Flaky Tests

Timing and Asynchrony

The most common cause. Tests that wait a fixed duration (sleep 1) instead of waiting for a condition.

// Flaky — waits 1 second regardless of whether element is ready
cy.wait(1000)
cy.get('[data-testid="result"]').click()

// Stable — waits until element is interactable
cy.get('[data-testid="result"]').should('be.visible').click()

Test Order Dependencies

Tests that pass in isolation but fail when run in a specific order. Test A modifies shared state that Test B expects to be clean.

# Flaky — modifies global state
def test_create_user():
    db.users.insert({'id': 1, 'name': 'Alice'})
    assert db.users.count() == 1

def test_empty_users():
    # Fails if test_create_user ran first
    assert db.users.count() == 0

External Dependencies

Tests that call real APIs, databases, or services that are occasionally unavailable or slow.

Concurrency

Race conditions in the application code triggered by parallel test execution.

Resource Contention

Tests competing for the same port, file, or temporary directory.

Date/Time Sensitivity

Tests that behave differently based on the current time, day, or year.

// Flaky — breaks on January 1st
const year = new Date().getFullYear()
expect(footer).toContain(`© ${year}`)
// If test runs at 23:59 on Dec 31, the page already shows the new year but getFullYear() returns old year

Detecting Flaky Tests

Automatic Flake Detection in CI

Most CI platforms support detecting flaky tests via repeated runs:

GitHub Actions with test retry:

- name: Run tests
  run: |
    npx jest --ci --testRetryAttempts=3 --json --outputFile=results.json
  continue-on-error: true

- name: Check for flakes
  run: |
    node scripts/check-flakes.js results.json

A test that fails then passes on retry is likely flaky, not broken.

Buildkite flaky test detection:

steps:
  - label: "Test"
    command: "bundle exec rspec"
    retry:
      automatic:
        - exit_status: "*"
          limit: 2
    plugins:
      - test-collector#v1.3.0:
          files: "spec/reports/*.xml"

Buildkite's test analytics tracks failure rates over time and surfaces tests with high variance.

Jest with flake detection:

// jest.config.js
module.exports = {
  testRetryAttempts: 2,
  reporters: [
    'default',
    ['jest-flaky-reporter', { outputFile: 'flaky-report.json' }]
  ]
}

Manual Flake Detection

Run the test in isolation 10 times:

# Run a single test 10 times
for i in {1..10}; do
  npx jest tests/auth.test.js --verbose
done

If it fails even once, it's flaky.

Tracking Flake Rate Over Time

Build a flake database. Every time a test fails and a re-run passes, record it:

// Post-CI flake tracking script
const results = loadTestResults()
const flakes = []

for (const test of results.tests) {
  if (test.attempts > 1 && test.finalStatus === 'passed') {
    flakes.push({
      name: test.name,
      file: test.file,
      failureCount: test.attempts - 1,
      date: new Date().toISOString()
    })
  }
}

// Store in database or post to Slack
postToSlack(`Flaky tests detected: ${flakes.map(f => f.name).join(', ')}`)

After two weeks, you'll have a ranked list of your most flaky tests.

Quarantine Strategy

Quarantine lets you remove flaky tests from the critical path without deleting them. The goal: prevent flaky tests from blocking CI while the root cause is being fixed.

Quarantine Tags

Tag tests as quarantined so they run in a separate job:

Jest:

// Add to test description
describe.skip('QUARANTINE: UserAuth', () => {
  it('logs in successfully', () => { /* ... */ })
})

// Or use a custom marker
it.skip('[FLAKY] submits form after network timeout', () => { /* ... */ })

pytest:

import pytest

@pytest.mark.quarantine
def test_payment_retry_on_timeout():
    # known flaky
    pass

Run quarantined tests separately:

pytest -m quarantine --ignore-glob="tests/stable/**"

Quarantine Job in CI

# .github/workflows/test.yml

jobs:
  # Main job — blocks PR merge
  tests:
    runs-on: ubuntu-latest
    steps:
      - run: pytest -m "not quarantine"

  # Quarantine job — non-blocking
  quarantine-tests:
    runs-on: ubuntu-latest
    continue-on-error: true
    steps:
      - run: pytest -m quarantine
      - name: Report flake status
        run: python scripts/report-quarantine.py

Quarantined tests still run — they just don't block deployment. The separate job keeps them visible.

Quarantine SLA

A quarantined test without a fix deadline gets forgotten. Set an SLA:

Tests quarantined for >2 weeks without a fix get deleted
Before deleting, open a ticket to re-implement the test correctly
Track quarantine queue in your test management tool

This prevents quarantine from becoming a graveyard.

Fixing Common Flaky Test Patterns

Fix: Timing Issues

Replace fixed waits with condition-based waits:

// Before — flaky
await page.waitForTimeout(2000)
await page.click('#submit')

// After — stable
await page.waitForSelector('#submit', { state: 'visible' })
await page.click('#submit')
await page.waitForSelector('.success-message')

For React/Vue apps, wait for loading states to complete:

// Wait for spinner to disappear
await page.waitForSelector('.loading-spinner', { state: 'hidden' })
// Then interact
await page.click('.result-item')

Fix: Test Isolation

Use fresh state for every test:

# pytest fixture for database isolation
@pytest.fixture(autouse=True)
def clean_database():
    yield
    db.rollback_transaction()

# Or use transaction rollback
@pytest.fixture
def db_session():
    with db.begin_nested():
        yield db
        db.rollback()

For integration tests, use unique identifiers per test run:

// Generate unique test data to avoid collisions
const testId = `test-${Date.now()}-${Math.random().toString(36).slice(2)}`
const user = await createUser({ email: `${testId}@example.com` })

Fix: Randomize Test Order

If tests must be isolated, randomize order to surface hidden dependencies:

# pytest-randomly
pip install pytest-randomly
pytest --randomly-seed=random

# Jest
npx jest --randomize

If tests fail in random order but pass in original order, you have order dependencies to fix.

Fix: Freeze Time

For time-sensitive tests:

# pytest with freezegun
from freezegun import freeze_time

@freeze_time("2024-06-15 12:00:00")
def test_quarterly_report():
    report = generate_report()
    assert report.period == "Q2 2024"

// Jest with fake timers
beforeEach(() => {
  jest.useFakeTimers()
  jest.setSystemTime(new Date('2024-06-15'))
})

afterEach(() => {
  jest.useRealTimers()
})

Fix: External Dependencies

Mock external APIs in unit/integration tests:

// Jest — mock HTTP client
jest.mock('axios')
axios.get.mockResolvedValue({ data: { status: 'ok' } })

For end-to-end tests, use a test environment with stable dependencies or HTTP replay:

// Playwright — intercept and mock API
await page.route('**/api/weather', route => {
  route.fulfill({
    status: 200,
    body: JSON.stringify({ temperature: 72, conditions: 'sunny' })
  })
})

Prioritizing Flake Fixes

Not all flakes are equal. Prioritize by:

Impact on CI: Tests that block PRs daily come first
Failure frequency: Tests that fail 20% of the time vs 1% of the time
Fix complexity: Easy wins first to build momentum

Track flake rate per test. Fix the 20% of tests responsible for 80% of CI failures.

Preventing Flakiness

In test reviews: Look for fixed waits, shared state, external calls, and time dependencies. Flag them during PR review.

In test templates: Provide pre-built fixtures that enforce isolation:

// test-utils.js — enforces isolation
export function createIsolatedBrowser() {
  return {
    before: () => browser.newContext(),
    after: (ctx) => ctx.close()
  }
}

// All E2E tests use this instead of direct browser access
const { before, after } = createIsolatedBrowser()
beforeEach(before)
afterEach(after)

In CI configuration: Fail fast on new flakes. A test that flips from stable to flaky is a signal that the recent code change introduced a race condition.

# Treat retry-pass as a failure
- run: npx jest --ci --bail
  env:
    FAIL_ON_FLAKE: "true"

Summary

Detect: Track retry-pass events in CI. Run tests repeatedly to measure flake rate. Surface the top 10 most flaky tests weekly.

Quarantine: Move flaky tests to a non-blocking job immediately. Set a 2-week SLA to fix or delete.

Fix: Address root causes — timing, isolation, external dependencies, time sensitivity. Don't patch symptoms.

Prevent: Review tests for flakiness patterns before merge. Provide isolation utilities that make the right behavior easy.

A test suite with zero flakiness is achievable. It requires consistent discipline in test design and a zero-tolerance policy for known flakes that don't have fix deadlines.