Root Causes of Flaky Tests: How to Identify and Fix Them

Flaky tests are the silent killers of CI/CD confidence. A test that passes sometimes and fails other times isn't just annoying—it erodes trust in your entire test suite. When developers start ignoring red builds because "it's probably just a flaky test," you've lost your safety net.

This guide breaks down the root causes of test flakiness and gives you concrete methods to identify which category each failure falls into.

What Makes a Test Flaky?

A flaky test is non-deterministic: given the same code, it produces different results on different runs. This non-determinism always has a cause. The goal is to find it.

There are roughly eight categories:

1. Race conditions and timing dependencies
2. Shared mutable state
3. External service dependencies
4. Order-dependent tests
5. Resource contention
6. Date/time sensitivity
7. Random data
8. Infrastructure variance

Race Conditions and Timing Issues

This is the most common cause, especially in end-to-end and integration tests.

// Flaky: assumes element appears within 1 second
await page.click('#submit');
await page.waitForTimeout(1000);
expect(await page.isVisible('#success-message')).toBe(true);

The test passes when the server responds fast. It fails when the server is under load. Fix it by waiting for the condition, not for time:

// Stable: waits for the actual outcome
await page.click('#submit');
await page.waitForSelector('#success-message', { state: 'visible', timeout: 10000 });

How to Identify Timing Issues

Look for:

• sleep(), waitForTimeout(), or Thread.sleep() in test code
• Assertions that run immediately after async operations
• Tests that fail more frequently on CI (slower) than locally

Run the test 10 times in a loop to observe the failure rate:

for i in {1..10}; do
  npx jest --testPathPattern="my-test" 2>&1 | tail -1
done

If the failure rate is 20-40%, you likely have a timing issue.

Shared Mutable State

Tests that share state through global variables, databases, or file systems will interfere with each other.

# Flaky: test_b assumes test_a hasn't modified the user
def test_a():
    user = get_user(1)
    user.email = "changed@example.com"
    save(user)
    assert user.email == "changed@example.com"

def test_b():
    user = get_user(1)
    assert user.email == "original@example.com"  # fails if test_a ran first

Fix this by isolating state:

• Use transactions that roll back after each test
• Use unique identifiers per test run
• Create test fixtures that clean up after themselves

@pytest.fixture(autouse=True)
def fresh_user(db):
    user = User.create(email="test@example.com")
    yield user
    user.delete()

How to Identify Shared State Issues

Run your test suite in different orders:

# pytest: randomize order
pytest --randomly-seed=random

# Jest: run in band (sequential, but randomized)
jest --runInBand --testSequencer=./random-sequencer.js

If different orderings produce different failures, you have state leakage.

External Service Dependencies

Tests that call real external services—APIs, databases on other machines, cloud services—will fail when those services are slow or unavailable.

// Flaky: depends on external API response time
it('fetches user data', async () => {
  const user = await externalApi.getUser(123);
  expect(user.name).toBe('Alice');
});

Solutions:

• Mock external dependencies in unit tests
• Use contract testing (see Pact) for integration boundaries
• Use VCR-style recording/playback (nock, ruby-vcr)
• Separate tests that require external services and run them less frequently

// Stable: mocked external dependency
jest.mock('../api/external');
externalApi.getUser.mockResolvedValue({ name: 'Alice' });

it('fetches user data', async () => {
  const user = await externalApi.getUser(123);
  expect(user.name).toBe('Alice');
});

Order-Dependent Tests

Some tests only pass when run after a specific other test. This usually happens when:

• One test creates data another test reads
• One test sets global configuration another test depends on
• Setup logic is embedded in test bodies instead of fixtures

describe('shopping cart', () => {
  it('adds item to cart', () => {
    cart.add(item);
    // saves item to cart
  });

  it('removes item from cart', () => {
    cart.remove(item); // fails if run independently—item was never added
  });
});

Fix by making each test independent:

describe('shopping cart', () => {
  beforeEach(() => {
    cart.clear();
  });

  it('adds item to cart', () => {
    cart.add(item);
    expect(cart.items).toContain(item);
  });

  it('removes item from cart', () => {
    cart.add(item);  // sets up its own precondition
    cart.remove(item);
    expect(cart.items).not.toContain(item);
  });
});

Date and Time Sensitivity

Tests that depend on the current date will fail as time passes.

# Flaky: fails after 2024
def test_user_is_not_expired():
    user = User(expires_at=datetime(2024, 12, 31))
    assert not user.is_expired()  # fails in 2025

Always control time in tests:

from freezegun import freeze_time

@freeze_time("2024-06-01")
def test_user_is_not_expired():
    user = User(expires_at=datetime(2024, 12, 31))
    assert not user.is_expired()

In JavaScript, use jest.useFakeTimers() or libraries like sinon.

Random Data

Tests that generate random input will produce different results:

// Flaky: random input may not satisfy the condition
const price = Math.random() * 100;
expect(formatPrice(price)).toMatch(/^\$\d+\.\d{2}$/);
// Fails when price rounds to something like "$5.1" (only 1 decimal place)

Fix: use deterministic test data, or seed your random number generator:

// Stable: fixed inputs test the actual cases
test.each([
  [10.5, '$10.50'],
  [100, '$100.00'],
  [0.1 + 0.2, '$0.30'],  // test floating point edge case
])('formatPrice(%f) === %s', (input, expected) => {
  expect(formatPrice(input)).toBe(expected);
});

Infrastructure Variance

Tests fail on CI but pass locally because the CI environment differs:

• Different OS (Windows vs Linux path separators)
• Different locale/timezone settings
• Different available memory
• Port conflicts from parallel test runs

For parallel test runs, ensure tests use unique ports:

// Use dynamic port allocation
const server = app.listen(0); // OS assigns available port
const port = server.address().port;

For timezone issues:

# In CI, always set timezone explicitly
TZ=UTC npx jest

A Systematic Approach to Identification

When you encounter a flaky test, work through this checklist:

1. Run it 20 times locally — determine the failure rate
2. Run it in isolation — remove test suite interference
3. Run it with CI constraints — lower CPU, memory limits
4. Check for async operations — any place that doesn't await
5. Check for shared state — databases, files, globals, singletons
6. Check for hardcoded values — dates, ports, file paths
7. Check for external calls — network, filesystem, environment variables

Tools that help:

• jest-circus — better async error reporting
• pytest-rerunfailures — re-run and log flaky tests
• Playwright's retry — built-in retry with trace recording
• Test Impact Analysis — identify which tests change frequently

Using HelpMeTest to Detect Flakiness

HelpMeTest's continuous monitoring runs your tests repeatedly against your live application. When a test fails intermittently, HelpMeTest tracks the failure rate over time and surfaces patterns—so you're not manually running tests 20 times to diagnose flakiness.

The self-healing test feature also detects when a test is failing due to UI changes rather than application bugs, preventing false flakiness reports when your UI is intentionally updated.

# Run tests with monitoring enabled
helpmetest run --monitor --retry 3

When flakiness is detected, HelpMeTest logs the full browser trace and network traffic for each failure, giving you the evidence you need to find the root cause.

Summary

Flaky tests are a symptom of non-deterministic design. Fix the design, and the flakiness goes away.