Handling Flaky E2E Tests: A Systematic Approach to Elimination
Flaky E2E tests are the tax every team pays for not taking reliability seriously early enough. A test that passes 9 out of 10 times isn't a test—it's a coin flip with extra steps. Teams that tolerate flakiness eventually find themselves re-running CI three times to get a green build, ignoring failures "because they're probably just flaky," and then missing real bugs because they've been trained to dismiss failures.
This guide is about systematic flakiness elimination: how to categorize flaky tests by their actual cause, apply targeted fixes to each category, and build the monitoring needed to catch new flaky tests before they multiply.
A Taxonomy of E2E Flakiness
Most flakiness articles list generic causes. This one gets specific.
Category 1: Timing Assumptions
The most common source of flakiness. The test assumes an operation completes within a certain time, and the assumption occasionally fails.
Symptoms: Test fails with "element not found," "expected X but got empty string," or timeout errors. Failures are often time-correlated (slower in CI, during resource-intensive operations).
Sub-types:
- Hard-coded
sleep()/waitForTimeout()calls that aren't long enough - Assertions that run before async operations complete
- Missing waits for animations, transitions, or loading states
- Race conditions between navigation and assertion
Fixes:
// BAD: Fixed timeout that's sometimes too short
await page.waitForTimeout(2000);
await expect(page.getByText('Saved')).toBeVisible();
// GOOD: Wait for the specific condition
await expect(page.getByText('Saved')).toBeVisible({ timeout: 10_000 });
// BAD: Asserting immediately after action
await page.getByRole('button', { name: 'Submit' }).click();
expect(await page.getByTestId('success-message').textContent()).toBe('Done!');
// GOOD: Wait for the expected state
await page.getByRole('button', { name: 'Submit' }).click();
await expect(page.getByTestId('success-message')).toHaveText('Done!');For network-dependent operations, wait for both the network response and the UI update:
// Wait for both the API call to complete AND the UI to update
await Promise.all([
page.waitForResponse(resp =>
resp.url().includes('/api/orders') && resp.status() === 201
),
page.getByRole('button', { name: 'Place Order' }).click(),
]);
// Now it's safe to assert on the result
await expect(page.getByTestId('confirmation-number')).toBeVisible();Category 2: Test Ordering and State Leakage
Tests that depend on state created by other tests, or that leave state behind.
Symptoms: Tests pass individually but fail when run in a specific order. Tests fail on the first run in CI but pass on retry. Tests in parallel fail but pass serially.
Fixes:
Identify state leakage by running tests in a shuffled order:
# Playwright: force random order
npx playwright test --retries=0 2>&1 | head -20
# Check if specific test files affect others
npx playwright test tests/orders.spec.ts tests/inventory.spec.tsOnce identified, fix the root cause:
// BAD: Test depends on globally-created user
test('user sees welcome message', async ({ page }) => {
// Assumes 'john@example.com' was created by a previous test
await loginAs(page, 'john@example.com');
// ...
});
// GOOD: Each test owns its data
test('user sees welcome message', async ({ page, isolatedUser }) => {
// isolatedUser fixture creates a fresh user for this test
await loginAs(page, isolatedUser.email);
// ...
});Category 3: External Service Instability
Third-party services (payment processors, email providers, SMS gateways) that occasionally fail, timeout, or return unexpected responses.
Symptoms: Failures correlate with third-party service incidents. Errors mention external domains. Tests pass when you mock the external service.
Fixes:
Mock external services in your test environment:
// Mock Stripe for all tests
test.beforeEach(async ({ page }) => {
await page.route('https://api.stripe.com/**', route => {
const url = route.request().url();
if (url.includes('payment_intents')) {
route.fulfill({
status: 200,
body: JSON.stringify({
id: 'pi_mock_success',
status: 'succeeded',
}),
});
} else {
route.continue(); // Let other Stripe calls through
}
});
});For server-side external calls, use environment-specific configuration:
# .env.test
STRIPE_API_URL=http://localhost:12111 # stripe-mock
SENDGRID_API_URL=http://localhost:1025 # mailhogCategory 4: Browser/Environment Inconsistency
Differences between browsers, viewport sizes, operating systems, or font rendering cause assertions to fail intermittently.
Symptoms: Tests pass on developer machines but fail in CI. Visual snapshot tests are especially prone. Failures are consistent across multiple runs on the same machine.
Fixes:
Use explicit viewport settings instead of relying on defaults:
// playwright.config.ts
use: {
viewport: { width: 1280, height: 720 }, // Explicit, consistent
}For visual snapshot tests, ensure consistent rendering:
// Use a stable font and disable animations for snapshots
test.beforeEach(async ({ page }) => {
await page.addStyleTag({
content: `
*, *::before, *::after {
animation-duration: 0s !important;
transition-duration: 0s !important;
}
`
});
});Category 5: Infrastructure Issues
Slow CI machines, network congestion, or resource limits cause intermittent failures.
Symptoms: Failures correlate with CI machine load. Increasing timeouts reduces (but doesn't eliminate) failures. Tests that do many API calls fail more often.
Fixes:
Increase timeouts for CI specifically (as a band-aid while you investigate root causes):
// playwright.config.ts
timeout: process.env.CI ? 60_000 : 30_000,Use --workers to prevent overloading CI machines:
- name: Run E2E tests
run: npx playwright test --workers=2 # Limit parallelism on constrained CIDetecting Flaky Tests
You can't fix what you can't measure.
Automated Flakiness Detection
Run each test multiple times to identify unstable ones:
# Run tests 3 times with no retries—identify genuine flakes
npx playwright test --repeat-each=3 --retries=0 2>&1 | grep -E "FAILED|flaky"CI-Level Flakiness Tracking
Log test outcomes with timing to a database or service. A test that fails on 1 in 10 CI runs is flaky, even if it always passes locally. Frameworks like Playwright provide JSON reporters that make this analysis straightforward:
// playwright.config.ts
reporter: [
['json', { outputFile: 'results.json' }],
];Parse results.json after each CI run and track failure rates per test over time.
The Quarantine Approach
When you identify a flaky test, quarantine it immediately to stop it from blocking CI:
// Mark the test as flaky/quarantined until fixed
test.skip('checkout with discount code @flaky', async ({ page }) => {
// This test is known flaky—fix tracked in GH-4521
// ...
});Or use a soft-retry approach for tests that are harder to fix immediately:
// Only for known flaky tests while investigating root cause
test.retries(2);
test('payment integration', async ({ page }) => { ... });Fixing Flakiness Systematically
Reproduce Reliably First
Never fix a flaky test you can't reproduce. Use --repeat-each to reproduce locally:
# Run the specific test 20 times to reproduce flakiness
npx playwright test tests/checkout.spec.ts --repeat-each=20 --retries=0 --workers=1Add Playwright's trace recording to capture the failure:
// playwright.config.ts
use: {
trace: 'retain-on-failure',
video: 'retain-on-failure',
screenshot: 'only-on-failure',
}Using the Playwright Trace Viewer
When a test fails, open the trace to see exactly what happened:
npx playwright show-trace test-results/checkout-failed/trace.zipThe trace shows a timeline of every action, network request, and screenshot. Most timing-related flakiness is immediately obvious when you see the trace.
Incremental Tightening
After fixing the root cause, verify with a high repeat count:
# Confidence test: run 50 times, zero failures expected
npx playwright test tests/checkout.spec.ts --repeat-each=50 --retries=0Preventing New Flakiness
Code Review Checklist for E2E Tests
Every E2E test PR should check for:
- No
waitForTimeout()orsleep()calls - All async operations awaited before assertions
- No hardcoded test data (emails, IDs, timestamps)
- Test cleans up its own state (or uses isolated fixtures)
- Assertions on final state, not intermediate state
- Network calls either mocked or awaited explicitly
Lint Rules
Enforce some of these checks automatically:
// .eslintrc.js
module.exports = {
rules: {
// Flag waitForTimeout in test files
'no-restricted-syntax': [
'error',
{
selector: 'CallExpression[callee.property.name="waitForTimeout"]',
message: 'Use explicit waits instead of waitForTimeout. It is a code smell.',
},
],
},
};Monitoring Flakiness Rate
Set a flakiness budget and alert when you exceed it:
# scripts/check-flakiness.py
import json
with open('results.json') as f:
results = json.load(f)
total = len(results['suites'])
flaky = sum(1 for s in results['suites'] if s.get('retries', 0) > 0)
rate = flaky / total
print(f"Flakiness rate: {rate:.1%} ({flaky}/{total})")
if rate > 0.05: # Alert if >5% of tests are flaky
print("ALERT: Flakiness rate exceeds 5% threshold")
exit(1)The Organizational Side
Flakiness isn't just a technical problem—it's a team discipline problem. Technical fixes fail unless you also:
Make flakiness visible: Display flakiness rates in your CI dashboard. "Green but flaky" should look different from "green and reliable."
Set an elimination schedule: When a test is quarantined, set a due date for the fix. Quarantine without a fix date means permanent quarantine.
Assign ownership: Each test file or area should have an owner responsible for keeping its flakiness rate below threshold.
Celebrate eliminations: Track the number of flaky tests eliminated per sprint. Making it a positive metric changes behavior.
Summary
Flaky E2E tests are eliminable, not just manageable:
- Classify first: Timing, state leakage, external services, browser differences, and infrastructure each need different fixes
- Reproduce before fixing: If you can't reproduce it reliably, you can't verify you've fixed it
- Use traces: Playwright's trace viewer shows exactly why a test failed
- Quarantine immediately: Don't let flaky tests pollute your CI signal while you investigate
- Monitor the rate: Track flakiness over time; if it's growing, you're not investing enough in reliability
- Enforce standards: Lint rules and code review checklists prevent new flakiness from entering
A team that systematically eliminates flakiness builds confidence in their test suite. When CI is red, it means something is broken—not "probably flaky, let me re-run it."