How to Detect, Quarantine, and Fix Flaky Tests

Flaky tests don't fix themselves. Left unaddressed, they multiply — developers copy patterns from flaky tests, shared infrastructure accumulates timing assumptions, and eventually the CI pipeline becomes a game of "rerun until green."

This is a practical playbook for detecting flakiness, quarantining failing tests without abandoning coverage, and fixing root causes systematically.

Step 1: Detection — Find Your Flaky Tests

You can't fix what you can't see. Most teams discover flaky tests reactively — a developer complains that a test keeps failing on CI but passes locally. Reactive discovery means you're always behind.

Track test pass rates over time

The most reliable detection method is tracking test pass rates across runs. A test that fails 1 in 10 runs has a 10% flakiness rate. You want to see this before your team notices.

Set up a simple database or use your CI platform's built-in analytics to record pass/fail for every test on every run. After a week, sort by failure rate — any test with intermittent failures is a flakiness candidate.

Run tests multiple times on the same commit

For suspected flaky tests, run the full suite multiple times against the same commit. If the code hasn't changed but results vary, you have confirmed flakiness.

Some CI platforms support this natively. Others require scripting:

for i in {1..10}; do
  npm test -- --testPathPattern="SuspectedFlaky" 2>&1 | tail -1
done

Ten runs is usually enough to surface flakiness rates above 10%. For subtler flakiness, run more.

Monitor CI failure patterns

Look for tests that fail more often on CI than locally. This points to environment-specific flakiness — timing differences, resource contention, or missing environment variables.

Also look for tests that fail in certain run orders but not others. If your CI randomizes test order (it should), failures that appear inconsistently by position are likely order-dependent.

Use continuous test monitoring

Tools like HelpMeTest run your tests on a schedule and track pass rates across runs. This surfaces flakiness that only appears occasionally — you'd never catch it by looking at a single CI run.

Step 2: Triage — Classify Flakiness by Type

Before quarantining or fixing, understand what type of flakiness you're dealing with. The fix depends on the cause.

Timing flakiness

Symptom: test fails with timeout errors or assertion failures on conditions that should be true but haven't occurred yet.

Common causes:

• sleep(500) assumptions about async operations
• Polling for a condition that takes variable time
• Race conditions between concurrent operations

Distinguishing test: add logging around the failing assertion. If you see the assertion firing before the expected state is reached, it's timing.

State contamination

Symptom: test fails when run in a certain order but passes in isolation.

Common causes:

• Previous test writes to a shared database without cleanup
• Global variables modified by one test affecting another
• Singleton objects retaining state between tests

Distinguishing test: run the suspected failing test in complete isolation. If it passes alone but fails after test X, check what test X writes to shared storage.

External dependency flakiness

Symptom: test fails with network errors, timeout errors, or unexpected API responses.

Common causes:

• Tests hitting real external APIs that have rate limits or availability issues
• Tests depending on third-party services that occasionally degrade
• DNS resolution failures in CI environments

Distinguishing test: add logging around network calls. If the same HTTP request sometimes returns 200 and sometimes times out or returns 503, the external service is the problem.

Resource contention

Symptom: tests fail with port-in-use errors, file lock errors, or database connection errors.

Common causes:

• Parallel tests trying to bind to the same port
• Test teardown not releasing resources before the next test starts
• Database connection pool exhaustion under parallel load

Distinguishing test: run tests sequentially. If flakiness disappears with --runInBand (Jest) or equivalent, it's resource contention under parallelism.

Step 3: Quarantine — Remove From Required Pass List

Quarantine is not giving up. It's separating known flaky tests from your required CI gate so you can investigate them without blocking every deploy.

The goal: keep the flaky tests running and tracked, but don't let them block merges while you fix them.

Create a quarantine test group

Most test runners support tagging or grouping:

Jest:

// Mark the test as quarantined
describe.skip('QUARANTINED: flaky suite', () => {
  it('should do something', () => { ... });
});

Better: use a custom runner that runs quarantined tests separately and reports results without failing the build:

// package.json
{
  "scripts": {
    "test": "jest --testPathIgnorePatterns=quarantine",
    "test:quarantine": "jest --testPathPattern=quarantine"
  }
}

Pytest:

@pytest.mark.flaky(reruns=3)
def test_something():
    ...

Or use a custom mark:

@pytest.mark.quarantine
def test_something():
    ...

Then exclude from required runs: pytest -m "not quarantine" and run separately with pytest -m quarantine.

Track quarantined tests as technical debt

Every quarantined test should have a tracking ticket. The ticket should include:

• Which test is quarantined
• What failure mode was observed
• What the hypothesis is for the root cause
• Who owns the fix

Without tracking, quarantine becomes a graveyard. Tests go in but never come out.

Set a quarantine time limit

Establish a policy: quarantined tests get fixed or deleted within N days (14 days is reasonable). A test in quarantine longer than that is either impossible to fix or covering dead code. In either case, make an explicit decision rather than letting it rot.

Step 4: Fix — Address Root Causes

Fix timing flakiness

Replace time-based waits with condition-based waits:

Before (fragile):

await page.click('#submit');
await page.waitForTimeout(2000); // hope it loads in time
expect(await page.textContent('#result')).toBe('Success');

After (deterministic):

await page.click('#submit');
await page.waitForSelector('#result:has-text("Success")', { timeout: 10000 });
expect(await page.textContent('#result')).toBe('Success');

For backend tests, wait for observable state changes rather than sleeping:

// Instead of: await sleep(1000)
await waitFor(() => db.query('SELECT count(*) FROM events WHERE type = ?', ['completed']), {
  until: count => count > 0,
  timeout: 5000
});

Fix state contamination

Each test must own its setup and teardown completely:

describe('UserService', () => {
  let db;
  
  beforeEach(async () => {
    db = await createTestDatabase(); // fresh DB per test
    await db.migrate();
  });
  
  afterEach(async () => {
    await db.destroy(); // clean up completely
  });
  
  it('creates a user', async () => {
    const service = new UserService(db);
    const user = await service.create({ email: 'test@example.com' });
    expect(user.id).toBeDefined();
  });
});

If creating a fresh database per test is too slow, use transactions that roll back:

beforeEach(async () => {
  await db.beginTransaction();
});

afterEach(async () => {
  await db.rollback(); // undo everything the test did
});

Fix external dependency flakiness

Mock external APIs in tests that don't need to test the integration:

jest.mock('./httpClient', () => ({
  get: jest.fn().mockResolvedValue({ status: 200, data: { id: 1 } })
}));

For integration tests that do need real external services, use dedicated test instances or service sandboxes. Never depend on production third-party services in automated tests.

Fix resource contention

Assign unique resources per test worker:

// Use unique port per test process
const port = 3000 + parseInt(process.env.JEST_WORKER_ID || '0');
const server = app.listen(port);

Or use port 0 to get an OS-assigned free port:

const server = app.listen(0); // OS assigns free port
const port = server.address().port;

Step 5: Verify the Fix

A fix for flakiness needs more verification than a fix for a deterministic bug.

Run the previously flaky test many times:

for i in {1..50}; do
  npm test -- --testPathPattern="PreviouslyFlaky" --forceExit 2>&1 | grep -E "(PASS|FAIL)"
done

If it passes 50 times in a row, you've likely fixed it. Ten times isn't enough — if the original flakiness rate was 5%, you'd expect to see it only once in 20 runs.

Also verify the fix didn't introduce new flakiness in other tests. Run the full suite a few times and check for any new intermittent failures.

Building a Sustainable Process

Fixing existing flaky tests is only half the battle. You also need to prevent new ones:

1. Require timing annotations in code review: any sleep() or hardcoded timeout should get flagged
2. Run tests in random order in CI: catches order-dependent failures automatically
3. Track new test failures for the first week: new tests sometimes fail intermittently and it's better to catch this immediately
4. Make quarantine easy: the lower the friction to quarantine, the more likely developers will do it instead of just clicking "rerun"

The teams with the lowest flakiness rates treat it as a metric they actively manage, not a nuisance they occasionally address.