Handling Flaky E2E Tests: A Systematic Approach to Elimination

Handling Flaky E2E Tests: A Systematic Approach to Elimination

Flaky E2E tests are the tax every team pays for not taking reliability seriously early enough. A test that passes 9 out of 10 times isn't a test—it's a coin flip with extra steps. Teams that tolerate flakiness eventually find themselves re-running CI three times to get a green build, ignoring failures "because they're probably just flaky," and then missing real bugs because they've been trained to dismiss failures.

This guide is about systematic flakiness elimination: how to categorize flaky tests by their actual cause, apply targeted fixes to each category, and build the monitoring needed to catch new flaky tests before they multiply.

A Taxonomy of E2E Flakiness

Most flakiness articles list generic causes. This one gets specific.

Category 1: Timing Assumptions

The most common source of flakiness. The test assumes an operation completes within a certain time, and the assumption occasionally fails.

Symptoms: Test fails with "element not found," "expected X but got empty string," or timeout errors. Failures are often time-correlated (slower in CI, during resource-intensive operations).

Sub-types:

  • Hard-coded sleep()/waitForTimeout() calls that aren't long enough
  • Assertions that run before async operations complete
  • Missing waits for animations, transitions, or loading states
  • Race conditions between navigation and assertion

Fixes:

// BAD: Fixed timeout that's sometimes too short
await page.waitForTimeout(2000);
await expect(page.getByText('Saved')).toBeVisible();

// GOOD: Wait for the specific condition
await expect(page.getByText('Saved')).toBeVisible({ timeout: 10_000 });

// BAD: Asserting immediately after action
await page.getByRole('button', { name: 'Submit' }).click();
expect(await page.getByTestId('success-message').textContent()).toBe('Done!');

// GOOD: Wait for the expected state
await page.getByRole('button', { name: 'Submit' }).click();
await expect(page.getByTestId('success-message')).toHaveText('Done!');

For network-dependent operations, wait for both the network response and the UI update:

// Wait for both the API call to complete AND the UI to update
await Promise.all([
  page.waitForResponse(resp =>
    resp.url().includes('/api/orders') && resp.status() === 201
  ),
  page.getByRole('button', { name: 'Place Order' }).click(),
]);

// Now it's safe to assert on the result
await expect(page.getByTestId('confirmation-number')).toBeVisible();

Category 2: Test Ordering and State Leakage

Tests that depend on state created by other tests, or that leave state behind.

Symptoms: Tests pass individually but fail when run in a specific order. Tests fail on the first run in CI but pass on retry. Tests in parallel fail but pass serially.

Fixes:

Identify state leakage by running tests in a shuffled order:

# Playwright: force random order
npx playwright test --retries=0 2>&1 | head -20

# Check if specific test files affect others
npx playwright test tests/orders.spec.ts tests/inventory.spec.ts

Once identified, fix the root cause:

// BAD: Test depends on globally-created user
test('user sees welcome message', async ({ page }) => {
  // Assumes 'john@example.com' was created by a previous test
  await loginAs(page, 'john@example.com');
  // ...
});

// GOOD: Each test owns its data
test('user sees welcome message', async ({ page, isolatedUser }) => {
  // isolatedUser fixture creates a fresh user for this test
  await loginAs(page, isolatedUser.email);
  // ...
});

Category 3: External Service Instability

Third-party services (payment processors, email providers, SMS gateways) that occasionally fail, timeout, or return unexpected responses.

Symptoms: Failures correlate with third-party service incidents. Errors mention external domains. Tests pass when you mock the external service.

Fixes:

Mock external services in your test environment:

// Mock Stripe for all tests
test.beforeEach(async ({ page }) => {
  await page.route('https://api.stripe.com/**', route => {
    const url = route.request().url();

    if (url.includes('payment_intents')) {
      route.fulfill({
        status: 200,
        body: JSON.stringify({
          id: 'pi_mock_success',
          status: 'succeeded',
        }),
      });
    } else {
      route.continue(); // Let other Stripe calls through
    }
  });
});

For server-side external calls, use environment-specific configuration:

# .env.test
STRIPE_API_URL=http://localhost:12111  # stripe-mock
SENDGRID_API_URL=http://localhost:1025  # mailhog

Category 4: Browser/Environment Inconsistency

Differences between browsers, viewport sizes, operating systems, or font rendering cause assertions to fail intermittently.

Symptoms: Tests pass on developer machines but fail in CI. Visual snapshot tests are especially prone. Failures are consistent across multiple runs on the same machine.

Fixes:

Use explicit viewport settings instead of relying on defaults:

// playwright.config.ts
use: {
  viewport: { width: 1280, height: 720 },  // Explicit, consistent
}

For visual snapshot tests, ensure consistent rendering:

// Use a stable font and disable animations for snapshots
test.beforeEach(async ({ page }) => {
  await page.addStyleTag({
    content: `
      *, *::before, *::after {
        animation-duration: 0s !important;
        transition-duration: 0s !important;
      }
    `
  });
});

Category 5: Infrastructure Issues

Slow CI machines, network congestion, or resource limits cause intermittent failures.

Symptoms: Failures correlate with CI machine load. Increasing timeouts reduces (but doesn't eliminate) failures. Tests that do many API calls fail more often.

Fixes:

Increase timeouts for CI specifically (as a band-aid while you investigate root causes):

// playwright.config.ts
timeout: process.env.CI ? 60_000 : 30_000,

Use --workers to prevent overloading CI machines:

- name: Run E2E tests
  run: npx playwright test --workers=2  # Limit parallelism on constrained CI

Detecting Flaky Tests

You can't fix what you can't measure.

Automated Flakiness Detection

Run each test multiple times to identify unstable ones:

# Run tests 3 times with no retries—identify genuine flakes
npx playwright test --repeat-each=3 --retries=0 2>&1 | grep -E "FAILED|flaky"

CI-Level Flakiness Tracking

Log test outcomes with timing to a database or service. A test that fails on 1 in 10 CI runs is flaky, even if it always passes locally. Frameworks like Playwright provide JSON reporters that make this analysis straightforward:

// playwright.config.ts
reporter: [
  ['json', { outputFile: 'results.json' }],
];

Parse results.json after each CI run and track failure rates per test over time.

The Quarantine Approach

When you identify a flaky test, quarantine it immediately to stop it from blocking CI:

// Mark the test as flaky/quarantined until fixed
test.skip('checkout with discount code @flaky', async ({ page }) => {
  // This test is known flaky—fix tracked in GH-4521
  // ...
});

Or use a soft-retry approach for tests that are harder to fix immediately:

// Only for known flaky tests while investigating root cause
test.retries(2);
test('payment integration', async ({ page }) => { ... });

Fixing Flakiness Systematically

Reproduce Reliably First

Never fix a flaky test you can't reproduce. Use --repeat-each to reproduce locally:

# Run the specific test 20 times to reproduce flakiness
npx playwright test tests/checkout.spec.ts --repeat-each=20 --retries=0 --workers=1

Add Playwright's trace recording to capture the failure:

// playwright.config.ts
use: {
  trace: 'retain-on-failure',
  video: 'retain-on-failure',
  screenshot: 'only-on-failure',
}

Using the Playwright Trace Viewer

When a test fails, open the trace to see exactly what happened:

npx playwright show-trace test-results/checkout-failed/trace.zip

The trace shows a timeline of every action, network request, and screenshot. Most timing-related flakiness is immediately obvious when you see the trace.

Incremental Tightening

After fixing the root cause, verify with a high repeat count:

# Confidence test: run 50 times, zero failures expected
npx playwright test tests/checkout.spec.ts --repeat-each=50 --retries=0

Preventing New Flakiness

Code Review Checklist for E2E Tests

Every E2E test PR should check for:

  • No waitForTimeout() or sleep() calls
  • All async operations awaited before assertions
  • No hardcoded test data (emails, IDs, timestamps)
  • Test cleans up its own state (or uses isolated fixtures)
  • Assertions on final state, not intermediate state
  • Network calls either mocked or awaited explicitly

Lint Rules

Enforce some of these checks automatically:

// .eslintrc.js
module.exports = {
  rules: {
    // Flag waitForTimeout in test files
    'no-restricted-syntax': [
      'error',
      {
        selector: 'CallExpression[callee.property.name="waitForTimeout"]',
        message: 'Use explicit waits instead of waitForTimeout. It is a code smell.',
      },
    ],
  },
};

Monitoring Flakiness Rate

Set a flakiness budget and alert when you exceed it:

# scripts/check-flakiness.py
import json

with open('results.json') as f:
    results = json.load(f)

total = len(results['suites'])
flaky = sum(1 for s in results['suites'] if s.get('retries', 0) > 0)
rate = flaky / total

print(f"Flakiness rate: {rate:.1%} ({flaky}/{total})")

if rate > 0.05:  # Alert if >5% of tests are flaky
    print("ALERT: Flakiness rate exceeds 5% threshold")
    exit(1)

The Organizational Side

Flakiness isn't just a technical problem—it's a team discipline problem. Technical fixes fail unless you also:

Make flakiness visible: Display flakiness rates in your CI dashboard. "Green but flaky" should look different from "green and reliable."

Set an elimination schedule: When a test is quarantined, set a due date for the fix. Quarantine without a fix date means permanent quarantine.

Assign ownership: Each test file or area should have an owner responsible for keeping its flakiness rate below threshold.

Celebrate eliminations: Track the number of flaky tests eliminated per sprint. Making it a positive metric changes behavior.

Summary

Flaky E2E tests are eliminable, not just manageable:

  1. Classify first: Timing, state leakage, external services, browser differences, and infrastructure each need different fixes
  2. Reproduce before fixing: If you can't reproduce it reliably, you can't verify you've fixed it
  3. Use traces: Playwright's trace viewer shows exactly why a test failed
  4. Quarantine immediately: Don't let flaky tests pollute your CI signal while you investigate
  5. Monitor the rate: Track flakiness over time; if it's growing, you're not investing enough in reliability
  6. Enforce standards: Lint rules and code review checklists prevent new flakiness from entering

A team that systematically eliminates flakiness builds confidence in their test suite. When CI is red, it means something is broken—not "probably flaky, let me re-run it."

Read more

Start now free