Testing

Retry Strategies for Flaky Tests: When They Help vs When They Hide Problems

08 Oct 2026 — 5 min read

Retrying failed tests is the fastest way to make flaky tests disappear from your CI dashboard. It's also the fastest way to miss a real bug. Understanding the difference between these two outcomes determines whether your retry strategy helps or hurts.

The Seductive Simplicity of Retries

Adding retries is a one-line change in most frameworks:

// jest.config.js
module.exports = {
  retry: 2  // retry failed tests up to 2 times
}

# pytest
pytest --reruns 3

Suddenly, your green build rate goes from 70% to 98%. Problem solved, right?

Not quite. What you've actually done is increased the probability that a flaky test produces a passing result. You haven't fixed anything—you've just increased your false negative rate. Real failures now need to fail consistently to be detected.

When Retries Are Legitimate

Retries are a valid tool in specific circumstances:

1. Known Infrastructure Flakiness

When your test infrastructure has documented instability—a flaky test container host, a staging database that occasionally times out, a CDN with variable response times—retries buy time while you fix the infrastructure.

The key: the flakiness source is external to your application code. Your tests are correct; your infrastructure is the problem.

# GitHub Actions: retry on runner-level failures
- name: Run tests
  uses: nick-fields/retry@v2
  with:
    timeout_minutes: 10
    max_attempts: 3
    command: npx jest
    retry_on: error  # only retry on errors, not test failures

2. Network-Dependent Integration Tests

Tests that hit real external APIs will occasionally fail due to network conditions. If these tests are explicitly labeled as integration tests and the failure is clearly network-related (timeout, 503, connection reset), a retry is reasonable.

import pytest
from pytest_httpretty import httpretty_fixture

@pytest.mark.flaky(reruns=3, reruns_delay=2)  # 2 second delay between retries
def test_external_api_integration():
    response = requests.get('https://api.example.com/data', timeout=5)
    assert response.status_code == 200

3. Browser Tests with Known Rendering Variance

Playwright and Cypress tests can fail due to rendering timing across browsers. When you've verified the test logic is correct but browser variance causes intermittent failures, retries with re-navigation are legitimate:

// playwright.config.js
module.exports = {
  retries: process.env.CI ? 2 : 0,  // only retry in CI
  use: {
    // capture traces on retry for diagnosis
    trace: 'on-first-retry',
    screenshot: 'only-on-failure',
    video: 'retain-on-failure',
  }
};

The critical detail: retries: 0 locally means developers see real failures immediately. retries: 2 in CI handles the genuine infrastructure variance.

When Retries Hide Real Problems

1. Application Logic Bugs

A race condition in your application code will produce inconsistent test results. Retrying the test doesn't fix the race condition—it just reduces the probability of catching it.

// This test is flaky because the application has a race condition
it('saves user preferences immediately', async () => {
  await userService.updatePreference('theme', 'dark');
  const prefs = await userService.getPreferences();
  expect(prefs.theme).toBe('dark');  // sometimes fails
});

Adding retry: 3 means the race condition still exists in production. Users will hit it. The test just stopped catching it.

2. Data Isolation Failures

If your test is flaky because it's reading data written by another test, retrying makes it worse—it's now retrying against potentially different data states.

# This will be non-deterministically flaky depending on test ordering
def test_user_count():
    count = db.users.count()
    assert count == 5  # assumes exactly 5 users exist

The fix is data isolation, not retries.

3. Memory Leaks and Resource Exhaustion

A test that fails after running a long suite might indicate memory leaks or resource exhaustion. Retrying it—after a brief pause—might pass because the GC had time to run. You've hidden a memory leak.

# Dangerous: this test is telling you something important
[FAIL] test-database-connection: ETIMEDOUT
# But with retries:
[RETRY 1] test-database-connection: ETIMEDOUT
[RETRY 2] test-database-connection: PASS  # connection pool recovered
# The log shows: "passed on retry 2"
# What it means: connection pool is exhausted by earlier tests

Configuring Retries Correctly

If you're going to use retries, configure them to reveal rather than hide information.

Always Log Retry Attempts

// Custom Jest reporter that logs retry details
class RetryReporter {
  onTestResult(test, testResult) {
    testResult.testResults.forEach(result => {
      if (result.invocations > 1) {
        console.warn(
          `[FLAKY] "${result.fullName}" ` +
          `passed on attempt ${result.invocations}/${result.invocations} ` +
          `(${result.invocations - 1} retries needed)`
        );
      }
    });
  }
}

Track Retry Rates Over Time

A test that consistently needs 2 retries to pass is a flaky test in disguise. Track this:

# Post-test hook to record retry data
def pytest_runtest_logreport(report):
    if report.when == 'call':
        rerun_count = getattr(report, 'rerun', 0)
        if rerun_count > 0:
            metrics.increment(
                'test.retried',
                tags=[f'test:{report.nodeid}', f'reruns:{rerun_count}']
            )

If a specific test is being retried in 30% of runs, it needs to be fixed.

Use Exponential Backoff for Network Tests

For tests that fail due to network conditions, exponential backoff improves reliability without masking fast failures:

@pytest.mark.flaky(
    reruns=4,
    reruns_delay=2,  # 2s, 4s, 8s, 16s
    condition=lambda e: "ConnectionError" in str(e)  # only retry on network errors
)
def test_api_endpoint():
    response = requests.get(url, timeout=10)
    assert response.status_code == 200

The condition parameter is crucial: only retry on the specific error types you expect from infrastructure flakiness. Let assertion failures propagate immediately.

Fail Loudly When Retries Are Used

Add CI step summaries that highlight retried tests:

# After test run
RETRY_COUNT=$(cat test-results.json | jq '[.testResults[].testResults[] | select(.invocations > 1)] | length')

if [ "$RETRY_COUNT" -gt 0 ]; then
  echo "::warning::$RETRY_COUNT test(s) required retries to pass. These may be flaky."
  cat test-results.json | jq -r '
    .testResults[].testResults[]
    | select(.invocations > 1)
    | "  - \(.fullName) (needed \(.invocations - 1) retries)"
  '
fi

This doesn't fail the build—the tests ultimately passed—but makes the retry usage visible.

The Retry Budget

A useful mental model: treat retries as a budget. Each test suite is allowed N retried test runs per day. When the budget is exhausted, new retried tests start blocking CI.

# quarantine_gate.py
import json
from datetime import date

DAILY_RETRY_BUDGET = 50

def check_retry_budget(today_results_file):
    with open(today_results_file) as f:
        results = json.load(f)

    retried = sum(
        1 for test in results['tests']
        if test.get('retries_needed', 0) > 0
    )

    if retried > DAILY_RETRY_BUDGET:
        print(f"ERROR: {retried} retried tests today (budget: {DAILY_RETRY_BUDGET})")
        print("Too many flaky tests. Fix them before adding retries.")
        return False
    return True

This forces the team to address flakiness rather than just increasing the retry count.

Framework-Specific Retry Configuration

Jest

// jest.config.js
module.exports = {
  // Global retry - use sparingly
  // retry: 2,

  // Better: use jest-circus with custom retry logic
  testRunner: 'jest-circus/runner',
};

// test-utils/retry.js - manual retry for specific tests
async function withRetry(fn, attempts = 3) {
  for (let i = 0; i < attempts; i++) {
    try {
      return await fn();
    } catch (err) {
      if (i === attempts - 1) throw err;
      console.warn(`Retry ${i + 1}/${attempts - 1}: ${err.message}`);
      await new Promise(r => setTimeout(r, 1000 * Math.pow(2, i)));
    }
  }
}

Playwright

// playwright.config.js
module.exports = defineConfig({
  retries: 2,
  use: {
    // Full traces on retry helps debug
    trace: 'on-first-retry',
    // Screenshots on failure
    screenshot: 'only-on-failure',
    // Video only when needed
    video: 'retain-on-failure',
  },
  reporter: [
    ['html'],
    ['json', { outputFile: 'results.json' }],
  ],
});

pytest

# pytest.ini
[pytest]
addopts = --reruns 2 --reruns-delay 1 --only-rerun "TimeoutError|ConnectionError"

The --only-rerun flag is critical: only retry on specific exception types, not all failures.

Cypress

// cypress.config.js
module.exports = defineConfig({
  retries: {
    runMode: 2,   // in CI
    openMode: 0,  // locally: fail fast
  }
});

Decision Flowchart

When a test is flaky, decide on retries:

Is the failure cause external to application code?
  YES → Is it documented/known infrastructure issue?
    YES → Retry with logging + fix deadline
    NO  → Investigate infrastructure before adding retries
  NO  → Fix the test/application. Do not add retries.

Does the test pass on retry consistently (>90% pass on retry 1)?
  YES → Might be acceptable for external dependencies
  NO  → Retries won't help; fix the root cause

Is there a fix deadline and owner?
  YES → Retries as temporary measure
  NO  → No retries until there's accountability

Summary

Retries are a bandage, not a fix. They have legitimate uses—infrastructure flakiness, network-dependent integration tests, browser rendering variance—but only when configured to be transparent about when they're being used.

The cardinal rule: never configure retries to hide information. Every retry should be logged, tracked, and counted against a budget. When retries become common, that's a signal the flakiness problem has grown beyond acceptable limits.

Fix the root cause. Use retries as a temporary measure with an expiration date.