Retry Logic in CI/CD — When It Helps and When It Hides Bugs

Retry logic in CI pipelines feels like a practical solution to flaky tests. Test fails? Run it again. If it passes on the second try, merge. Engineers can keep working, pipelines stay green, velocity is maintained.

The problem: you're not fixing the flakiness. You're hiding it.

This doesn't mean retries are always wrong. There are legitimate uses for retry logic. But using them indiscriminately is how teams end up with CI pipelines that are technically passing but not actually reliable.

When Retry Logic Is Legitimate

Genuinely transient infrastructure failures

Some failures are not flakiness in your tests — they're transient failures in CI infrastructure itself:

• Network requests to external services timing out due to CI provider connectivity issues
• Docker image pulls failing due to registry rate limits
• Spot instance preemption mid-run (common with cheap CI configurations)
• Filesystem operations failing due to temporary disk pressure

These failures are random, infrastructure-level, and not caused by anything in your code or tests. A retry is appropriate here because the test didn't actually run — the environment failed to start.

The distinguishing characteristic: the failure occurs in setup/teardown, not in the assertion itself.

External integration tests with documented SLA limitations

If you're testing against a third-party sandbox API that has explicit SLA limitations (like "99% uptime, with occasional 503s"), and you're intentionally testing against it rather than mocking, retries acknowledge the documented limitations of the external service.

Even here, the retry should be narrow: retry on specific HTTP error codes (503, 429), not on any assertion failure.

Rate-limited external services in test environments

Some external services (Stripe sandbox, Twilio test mode, etc.) have rate limits lower than production. If your test suite runs these in parallel, you may legitimately hit rate limits that a sequential or throttled retry would avoid.

The right fix is parallelism control or mock substitution. But a retry with exponential backoff on 429 responses is acceptable while you implement the proper solution.

When Retry Logic Hides Real Problems

Timing-dependent test logic

If a test is failing because it checks a condition before it's true — a database write that hasn't committed, a UI element that hasn't rendered, an event that hasn't propagated — a retry will often catch it on the second attempt when enough time has passed.

This is not a transient infrastructure failure. This is a race condition in your test or your code. A retry papers over it.

The test was telling you something: "this operation sometimes takes longer than your code assumes." That's important information. If you retry instead of fixing it, you'll eventually see the race condition in production, where you have no retry.

How to spot it: the failure mode is "condition not yet true at assertion time" — timeouts waiting for expected state, or assertions on intermediate states.

The fix: replace time-based assumptions with deterministic waits. Wait for specific conditions, not for a duration.

Nondeterministic test logic

Some tests use random data without proper seed control. Or they depend on hash ordering, UUID generation, or other sources of randomness. These tests produce different results on different runs.

A retry will usually pass because the random values will be different. But you haven't fixed anything — the next run might produce the problematic values again.

How to spot it: failures are associated with specific data values, not timing. Logs show different input values on passing vs. failing runs.

The fix: use deterministic test data. If you need variety, use deterministic parameterization rather than random generation.

Order-dependent test failures

If tests leak state and test order matters, some orderings will cause failures. A retry changes the test isolation context enough that the state might be different on the second run — so it passes.

This is particularly insidious because retrying often "works" while masking a serious test design problem. Your tests are not independent.

How to spot it: the failure only occurs after specific other tests have run. Running the test in isolation passes every time.

The fix: proper setup/teardown. Each test owns its state.

Real regressions in recently changed code

The most dangerous case: a developer merges code that introduces a real bug. The bug doesn't always manifest — maybe it's a race condition in production code, or a specific edge case that affects 20% of runs.

With retry logic, the pipeline sees a failure, retries, gets a pass, and merges. The bug ships.

This is the fundamental problem with aggressive retry policies: you've redefined "passing" as "passing at least once," which is a much weaker guarantee than "passing reliably."

How to spot it: the failure is associated with a recent code change, not historical flakiness in that test.

The prevention: track which tests are historically flaky. If a test that has never been flaky starts failing intermittently after a commit, treat it as a potential regression before attributing it to flakiness.

The Retry Trap: How It Gets Worse Over Time

Retry logic tends to expand. You add retries for tests in module A because of legitimate infrastructure issues. Developers notice that retries clear up failures in module B too. Soon retries are everywhere, and the codebase has accumulated technical debt that nobody sees because the pipeline is "green."

The insidious part: teams with aggressive retry policies often have higher actual defect rates in production, not lower. Retries mask the signal that test flakiness provides about code quality and test design problems.

Over time, the retry count also tends to grow. What started as "retry once" becomes "retry three times" as flakiness increases. Each increase buys a little more false stability while the underlying problem gets worse.

A Principled Approach to Retries

Retry policy 1: Infrastructure-level, narrow scope

# Acceptable: retry only on infrastructure failures
retry:
  max_attempts: 2
  when:
    - runner_system_failure
    - stuck_or_timeout_failure

This is GitHub Actions syntax. The key is runner_system_failure — this catches actual infrastructure problems without masking test failures.

Do not use on_failure, which retries on any failure including legitimate test failures.

Retry policy 2: Specific error codes, external services only

In your test code, retry logic should be scoped to specific, expected transient errors from external services:

async function callExternalAPI(params, attempt = 0) {
  try {
    return await apiClient.call(params);
  } catch (error) {
    if (error.status === 429 && attempt < 2) {
      await sleep(1000 * Math.pow(2, attempt)); // exponential backoff
      return callExternalAPI(params, attempt + 1);
    }
    throw error; // don't retry other errors
  }
}

This is explicit about what's being retried and why. It's not "retry on any failure."

Track retries as flakiness signals

If you have any retry logic, track every retry that occurs. A test that requires retrying is a flakiness signal even if it eventually passes.

afterEach(() => {
  if (currentTest.retryCount > 0) {
    metrics.increment('test.retried', {
      test: currentTest.name,
      retries: currentTest.retryCount
    });
  }
});

Review this data weekly. Tests that consistently require retries should move to your flakiness fix queue.

Set a sunset policy for retries

Any retry added to fix a specific problem should have a sunset date or condition. "We added retry logic for the Stripe sandbox rate limit issue; this should be removed once we add proper mock substitution in issue #456."

Without sunset policies, retries accumulate indefinitely.

The Alternative to Retries: Quarantine + Fix

The disciplined alternative to retries:

1. Detect the flaky test (fails sometimes, passes sometimes)
2. Quarantine it — remove from the required pass gate while you investigate
3. Investigate — reproduce, understand root cause
4. Fix — address the actual problem
5. Verify — confirm the fix works by running many times

This takes more time upfront than adding retry: 3. But it results in a test suite that actually reflects software quality, rather than one that appears to work because failures are retried away.

Tools like HelpMeTest can help with step 1 and 3 by running your tests continuously and tracking which tests produce inconsistent results across runs. Catching flakiness before it accumulates is much cheaper than fixing an entrenched retry-everything culture later.

Summary

Retries are not a test reliability strategy. They're a coping mechanism. Used narrowly for genuine infrastructure issues, they're fine. Used broadly as a substitute for fixing flakiness, they undermine everything your test suite is supposed to do.