Testing

Quarantine Strategies for Flaky Tests: Isolate Without Ignoring

HelpMeTest

HelpMeTest

— 5 min read
Quarantine Strategies for Flaky Tests: Isolate Without Ignoring

Flaky tests put engineering teams in an impossible position. You can't ignore them—a flaky test might be covering a real bug. But if flaky tests block every CI build, developers start merging anyway, defeating the purpose of CI entirely.

Quarantine is the middle path: isolate flaky tests from your main pipeline while ensuring they still run and get fixed.

The Problem with "Just Delete It"

When a test becomes flaky, the tempting solution is deletion. The test is causing pain; removing it removes the pain.

But that test probably existed for a reason. Maybe it caught a real race condition in your payment flow. Maybe it tests an edge case that's been bitten before. Deleting it means that protection is gone, and you won't know until something fails in production.

The goal of quarantine is to protect your pipeline while preserving the test until it can be fixed.

Strategy 1: Tag-Based Quarantine

The simplest approach: mark flaky tests with a tag, exclude them from the main run, and run them separately.

Jest:

// Mark test as quarantined
it.skip('processes payment with slow network', async () => {
  // ...
});

// Better: use a custom tag without skipping entirely
it('[FLAKY] processes payment with slow network', async () => {
  // ...
});

Then in your CI, run quarantined tests separately:

# .github/workflows/ci.yml
jobs:
  test:
    steps:
      - name: Run stable tests
        run: jest --testNamePattern="^(?!\[FLAKY\])"

  quarantine:
    steps:
      - name: Run quarantined tests
        continue-on-error: true  # don't fail build
        run: jest --testNamePattern="\[FLAKY\]"

pytest:

import pytest

@pytest.mark.flaky
def test_payment_with_slow_network():
    pass
# pytest.ini
markers =
    flaky: marks tests as known flaky (deselect with '-m "not flaky"')
# Main pipeline
pytest -m "not flaky"

# Quarantine pipeline (non-blocking)
pytest -m "flaky" || true

Strategy 2: Dedicated Quarantine Pipeline

Create a separate CI pipeline that:

  1. Runs quarantined tests on every commit (but doesn't block merges)
  2. Tracks failure rates over time
  3. Alerts when a quarantined test starts failing more than a threshold
# .github/workflows/quarantine.yml
name: Quarantine Tests
on:
  push:
    branches: [main, develop]
  schedule:
    - cron: '0 */4 * * *'  # also run every 4 hours

jobs:
  quarantine:
    runs-on: ubuntu-latest
    continue-on-error: true
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - name: Run quarantined tests with retry data
        run: |
          npx jest --testPathPattern="__quarantine__" \
            --json \
            --outputFile=quarantine-results.json || true
      - name: Upload results
        uses: actions/upload-artifact@v4
        with:
          name: quarantine-results
          path: quarantine-results.json

The key difference from the main pipeline: continue-on-error: true at the job level ensures failures don't block PRs, but the results are still captured and visible.

Strategy 3: File-Based Quarantine Directory

Keep quarantined tests in a separate directory to make the separation explicit:

tests/
  unit/           # stable, blocks CI
  integration/    # stable, blocks CI
  e2e/            # stable, blocks CI
  quarantine/     # flaky, runs separately
    payment-slow-network.test.js
    user-session-timeout.test.js

This has the advantage of making quarantine status visible in code review—moving a test to quarantine/ shows up in the diff, making it a deliberate team decision rather than a quiet skip.

Strategy 4: Database-Backed Quarantine

For larger teams, track quarantine status in a database or config file instead of code annotations:

// quarantine.json
{
  "flaky": [
    {
      "test": "processes payment with slow network",
      "file": "tests/payment.test.js",
      "quarantined_at": "2026-01-15",
      "quarantined_by": "alice",
      "issue": "PROJ-1234",
      "expected_fix_by": "2026-02-01"
    }
  ]
}

Your test runner reads this file and skips or re-routes those tests accordingly. This makes quarantine status auditable and queryable—you can report "we have 7 tests in quarantine, 3 are past their expected fix date."

Example runner integration:

// jest.config.js
const quarantine = require('./quarantine.json');
const quarantinedTests = quarantine.flaky.map(t => t.test);

module.exports = {
  testResultsProcessor: './quarantine-processor.js',
  globals: {
    quarantinedTests,
  }
};
// quarantine-processor.js
module.exports = (results) => {
  const quarantined = global.quarantinedTests;
  results.testResults.forEach(suite => {
    suite.testResults.forEach(test => {
      if (quarantined.includes(test.fullName)) {
        // Override failure to warning
        if (test.status === 'failed') {
          test.status = 'pending';
          test.pendingMessage = 'Quarantined - see quarantine.json';
        }
      }
    });
  });
  return results;
};

Strategy 5: Auto-Quarantine Based on Failure Rate

The most sophisticated approach: automatically quarantine tests that exceed a failure rate threshold, without manual intervention.

# quarantine_manager.py
import json
from pathlib import Path

FAILURE_RATE_THRESHOLD = 0.15  # 15% failure rate triggers quarantine
SAMPLE_SIZE = 20               # needs 20 runs to make a decision

class QuarantineManager:
    def __init__(self, history_file="test_history.json"):
        self.history_file = Path(history_file)
        self.history = self._load()

    def record_result(self, test_name: str, passed: bool):
        if test_name not in self.history:
            self.history[test_name] = []
        self.history[test_name].append(passed)
        # Keep only last N results
        self.history[test_name] = self.history[test_name][-SAMPLE_SIZE:]
        self._save()

    def should_quarantine(self, test_name: str) -> bool:
        results = self.history.get(test_name, [])
        if len(results) < SAMPLE_SIZE:
            return False
        failure_rate = results.count(False) / len(results)
        return failure_rate >= FAILURE_RATE_THRESHOLD

    def _load(self):
        if self.history_file.exists():
            return json.loads(self.history_file.read_text())
        return {}

    def _save(self):
        self.history_file.write_text(json.dumps(self.history, indent=2))

This approach requires a persistent store for test history—a database, artifact storage, or a committed file works. The key insight is that you're making quarantine decisions based on evidence rather than human judgment.

Preventing Quarantine Accumulation

Quarantine only works if tests actually get fixed. Without discipline, tests pile up in quarantine and stay there forever.

Enforce these policies:

Time limits:

# In your CI, check for expired quarantine
import json
from datetime import datetime, timedelta

with open('quarantine.json') as f:
    quarantine = json.load(f)

now = datetime.now()
expired = [
    t for t in quarantine['flaky']
    if datetime.fromisoformat(t['expected_fix_by']) < now
]

if expired:
    print("ERROR: These quarantined tests are past their fix date:")
    for t in expired:
        print(f"  - {t['test']} (due: {t['expected_fix_by']}, issue: {t['issue']})")
    exit(1)  # fail the build

Caps on quarantine size:

# In CI
QUARANTINE_COUNT=$(jq '.flaky | length' quarantine.json)
MAX_QUARANTINE=10

if [ "$QUARANTINE_COUNT" -gt "$MAX_QUARANTINE" ]; then
  echo "ERROR: $QUARANTINE_COUNT tests in quarantine (max: $MAX_QUARANTINE)"
  echo "Fix flaky tests before adding more to quarantine"
  exit 1
fi

Regular quarantine reviews: Add a weekly agenda item to review the quarantine list. Each test should have an owner, an issue, and a fix deadline.

Quarantine in Practice: A Decision Tree

When a test starts failing intermittently:

  1. Can you fix it immediately? Fix it. Don't quarantine.
  2. Is it a known infrastructure issue? Quarantine with a linked incident.
  3. Is the fix complex and non-urgent? Quarantine with an issue and deadline.
  4. Is the test coverage still valuable? Keep it quarantined and fix it.
  5. Is the test redundant? Delete it (document why).

The worst outcome is a test that's effectively disabled but not officially quarantined—a skip or xit with no issue, no deadline, no owner. That test will be in your codebase forever.

HelpMeTest and Flaky Test Management

HelpMeTest tracks test results over time and surfaces tests with inconsistent pass rates. Rather than manually identifying which tests to quarantine, you get data: this test has failed 3 of the last 20 runs, here are the failure traces.

The continuous monitoring also helps distinguish between tests that are flaky in CI (infrastructure issues) versus tests that are genuinely non-deterministic (design issues). This distinction matters for how you prioritize fixes.

Summary

Strategy Best For Tooling Required
Tag-based Small teams, simple setup CI config only
Dedicated pipeline Teams with separate test ownership CI config + separate workflow
File-based directory Teams that want visible quarantine Directory convention
Database-backed Large teams needing auditability Config file + runner integration
Auto-quarantine Scale, reducing manual toil History store + manager script

Quarantine is a temporary holding area, not a cemetery. Every test in quarantine should have an owner, an issue, and a deadline.

Read more

Start now free