Test Reliability
Flaky tests don't fix themselves. Left unaddressed, they multiply — developers copy patterns from flaky tests, shared infrastructure accumulates timing assumptions, and eventually the CI pipeline becomes a game of "rerun until green." This is a practical playbook for detecting flakiness, quarantining failing tests without abandoning coverage,
Test Reliability
Retry logic in CI pipelines feels like a practical solution to flaky tests. Test fails? Run it again. If it passes on the second try, merge. Engineers can keep working, pipelines stay green, velocity is maintained. The problem: you're not fixing the flakiness. You're hiding it.
Testing
Retrying failing tests is a controversial topic. Critics say it hides flakiness and trains developers to ignore failures.
Testing
You've identified a flaky test. You can't fix it right now — it's intermittent, the root cause isn't clear, and you have a release to ship. What do you do?
Testing
A test suite developers trust is worth more than a large test suite developers ignore.
Testing
"It works on my machine" is one of the oldest complaints in software development.
Testing
Flaky tests are tests that pass sometimes and fail sometimes without any code changes.
Test Reliability
Flaky tests cost CI time and erode confidence. Playwright gives you several tools to manage flakiness: automatic retries that reduce false negatives, --repeat-each for detecting flakiness before it hits CI, and test.fail() for acknowledging known issues without deleting coverage. This post covers all three strategies — when to use
Test Reliability
When a flaky test blocks your CI pipeline, you have three bad options: delete it (lose the coverage), skip it (test.skip, forget it forever), or fix it immediately (takes time you don't have). The quarantine pattern is the fourth option: isolate it, run it separately, and track
Test Reliability
A flaky test is a test that passes sometimes and fails sometimes without any change to the code. It fails in CI, you re-run it, it passes. You ship.
Test Reliability
Production services have SLOs — "99.9% of requests succeed within 500ms." Test suites have... hope.
Test Reliability
Flaky tests are hard to detect manually — a test that fails 5% of the time looks like noise until someone tracks it across 20 CI runs.