Test Quarantine Strategies for Stable CI

Test Quarantine Strategies for Stable CI

A flaky test creates a dilemma: keep it in the required CI gate (and deal with false failures blocking your team) or remove it (and lose coverage you need).

Test quarantine resolves this dilemma. You keep the test — it still runs, results are still tracked — but it no longer blocks your pipeline. You can investigate and fix it without the pressure of constant CI failures.

Done poorly, quarantine becomes a graveyard where flaky tests go to be forgotten. Done well, it's a triage system that keeps your pipeline stable while you fix the underlying problems.

What Quarantine Is (and Isn't)

Quarantine is:

  • Temporary separation of unreliable tests from your required CI gate
  • A structured holding state with tracking and time limits
  • A signal that a test needs attention

Quarantine is not:

  • A way to permanently hide flaky tests
  • Equivalent to deleting tests
  • An excuse to stop caring about the failing test

The critical difference from skip or xit: quarantined tests still run. They generate results. Someone owns fixing them. There's a deadline.

Tag-based quarantine uses test runner labels to separate flaky tests from the required pass gate, without removing them from the codebase.

Jest

Use a custom marker in the test description and filter by it in CI:

// Mark flaky tests with [QUARANTINE] prefix
describe('[QUARANTINE] UserCheckout', () => {
  it('completes payment with saved card', async () => {
    // test body unchanged
  });
});

In CI, run two separate jobs:

# .github/workflows/test.yml
jobs:
  test-required:
    name: Required Tests
    runs-on: ubuntu-latest
    steps:
      - run: |
          jest --testPathIgnorePatterns="\[QUARANTINE\]" \
               --testNamePattern="^(?!.*\[QUARANTINE\])"
    # This job is required for merge

  test-quarantine:
    name: Quarantined Tests (Non-blocking)
    runs-on: ubuntu-latest
    continue-on-error: true  # does NOT block merge
    steps:
      - run: |
          jest --testNamePattern="\[QUARANTINE\]" || true

The continue-on-error: true means the quarantine job runs and its results are visible, but failure doesn't block the PR.

pytest

Use a custom mark:

# conftest.py
import pytest

def pytest_configure(config):
    config.addinivalue_line(
        "markers", "quarantine: mark test as quarantined (flaky, under investigation)"
    )
# test_payment.py
@pytest.mark.quarantine
def test_payment_with_saved_card():
    # test body unchanged

CI configuration:

- name: Required tests
  run: pytest -m "not quarantine"  # required, fails build

- name: Quarantined tests  
  run: pytest -m "quarantine" || true  # informational only
  continue-on-error: true

Go

Use build tags or test name conventions:

func TestPaymentFlaky(t *testing.T) {
    if os.Getenv("RUN_QUARANTINE") == "" {
        t.Skip("quarantined: flaky test, see issue #456")
    }
    // test body
}

In CI:

# Required tests (skip quarantined)
go test ./...

# Quarantine job (run quarantined)
RUN_QUARANTINE=1 go test ./... || true

Pattern 2: Separate Test File Quarantine

Instead of marking individual tests, move flaky tests to a separate directory:

tests/
  unit/           # required in CI
  integration/    # required in CI
  quarantine/     # informational only
    checkout.flaky.test.js
    payment.flaky.test.js

CI configuration:

- name: Required tests
  run: jest --testPathIgnorePatterns="quarantine/"

- name: Quarantine tests
  run: jest --testPathPattern="quarantine/" || true
  continue-on-error: true

Pros: very clear separation, easy to see what's quarantined at a glance Cons: moving files changes git history, harder to see quarantine context alongside the original test suite

Use this when you have many tests to quarantine at once or want the strongest possible separation.

Pattern 3: Quarantine via CI Retry with Tracking

Some teams don't want to mark tests but still need to manage flakiness. An alternative: allow retries but track every retry as a quarantine signal.

// jest.config.js
module.exports = {
  reporters: [
    'default',
    ['./reporters/flakiness-reporter.js', { threshold: 1 }]
  ]
};

// reporters/flakiness-reporter.js
class FlakinessReporter {
  onTestResult(test, testResult) {
    testResult.testResults.forEach(result => {
      if (result.numPassingAsserts > 0 && result.failureMessages.length === 0 
          && result.invocations > 1) {
        // test passed but required retries — log it
        console.warn(`FLAKINESS DETECTED: ${result.fullName} (required ${result.invocations} attempts)`);
        // send to your metrics system
      }
    });
  }
}

Pros: no manual marking required, works automatically Cons: requires custom tooling, retry cost still paid

Quarantine Hygiene: Making It Work Long-Term

Quarantine is only as good as its governance. Without discipline, tests enter quarantine and never leave.

Rule 1: Every quarantine needs a ticket

Before marking a test as quarantined, create a tracking issue:

// [QUARANTINE] UserCheckout — see issue #789
describe('[QUARANTINE] UserCheckout', () => {

The issue should include:

  • What failure mode was observed
  • Hypothesis for root cause
  • Who owns the investigation
  • Target resolution date

No ticket = the quarantine gets forgotten.

Rule 2: Time limits

Establish a policy: quarantine lasts at most 14 days. After 14 days, the test gets either fixed or deleted. Make an explicit decision either way.

Some teams enforce this with CI checks:

// check-quarantine-age.js
const { execSync } = require('child_process');
const maxAgedays = 14;

const quarantinedTests = findQuarantinedTests('./tests');
quarantinedTests.forEach(test => {
  const blame = execSync(`git log -1 --format="%ar" -- ${test.file}`).toString();
  const ageMatch = blame.match(/(\d+) days? ago/);
  if (ageMatch && parseInt(ageMatch[1]) > maxAgeDays) {
    console.error(`QUARANTINE EXPIRED: ${test.name} has been quarantined for ${ageMatch[1]} days`);
    process.exit(1);
  }
});

Rule 3: Regular review

Add "quarantine review" to your team's weekly engineering sync. Walk through:

  • Which tests are currently quarantined?
  • What's the status of each fix?
  • Are any approaching or past the deadline?

Five minutes per week prevents quarantine from accumulating silently.

Rule 4: Quarantine is reversible

When you fix a flaky test, remove the quarantine marker immediately and verify it passes reliably. Don't leave the marker "just in case."

If a fixed test becomes flaky again, re-quarantine it — with a new ticket that references the original and the fix that didn't hold.

What to Do During Quarantine

Quarantine creates space to investigate without blocking your team. Use that space:

Reproduce systematically

# Run the quarantined test many times to characterize flakiness
for i in {1..50}; do
  jest --testNamePattern="\[QUARANTINE\] UserCheckout" 2>&1 | tail -1
done | sort | uniq -c

If it fails 5 out of 50 times, you have 10% flakiness. Now you know what you're dealing with.

Check environmental factors

Does it fail more on CI than locally? If so, investigate environment differences:

  • Parallelism (CI often runs tests concurrently)
  • Resource constraints (CI has less memory/CPU)
  • Network latency (CI network is different from local)
  • Missing environment variables

Bisect the failure

If order-dependent, bisect to find which test causes the failure:

# Pytest: run test in isolation
pytest tests/test_payment.py::test_payment_with_saved_card -v

# Run with specific test ordering
pytest --randomly-seed=12345 tests/

Add logging without changing behavior

Add temporary logging to understand what state the test is in when it fails:

it('completes payment', async () => {
  console.log('Starting test, cart state:', await getCartState());
  // ... test body
  console.log('Assertion state:', await getPaymentState());
  expect(payment.status).toBe('completed');
});

Logging the state at the point of failure often reveals the root cause immediately.

Measuring Quarantine Effectiveness

Track:

  • Quarantine intake rate: how many tests enter quarantine per week? Rising intake signals growing flakiness in new code.
  • Resolution rate: how many quarantined tests get fixed per week? Should exceed intake rate.
  • Average time-to-fix: how long does a test stay in quarantine before being fixed? Should be well under your limit.
  • Recidivism: how often do "fixed" tests return to quarantine? High recidivism means fixes are addressing symptoms, not causes.

A healthy quarantine system has:

  • ≤5 tests quarantined at any given time
  • Average resolution time <7 days
  • Intake rate declining over time

If you have 40 tests in quarantine with some there for months, the system isn't working — the underlying flakiness is accumulating faster than fixes.

Integrating With HelpMeTest

If you're using HelpMeTest for continuous test monitoring, you can see which tests are failing intermittently across runs — even without a formal quarantine system. This helps you identify candidates for quarantine proactively, before a developer reports "this test keeps failing."

The workflow:

  1. HelpMeTest flags tests with intermittent failures over the past week
  2. You review the list and quarantine the confirmed flaky ones
  3. HelpMeTest continues monitoring quarantined tests so you can verify fixes

Proactive identification means you're quarantining tests before they've blocked multiple developers, not after.

Summary: The Quarantine Checklist

When a test needs quarantine:

  • Add quarantine marker ([QUARANTINE], @pytest.mark.quarantine, etc.)
  • Create tracking ticket with failure description and root cause hypothesis
  • Set resolution deadline in ticket
  • Assign owner
  • Verify quarantined test still runs in CI (informational job)
  • Start investigation

When resolving a quarantine:

  • Remove quarantine marker
  • Run test 50+ times to confirm reliability
  • Close tracking ticket
  • Document what the root cause was and what fixed it

Quarantine isn't the goal — it's the bridge between discovering a flaky test and fixing it. The goal is a test suite where every test in the required gate is genuinely reliable.

Read more

Start now free