Test Quarantine Strategies for Stable CI
A flaky test creates a dilemma: keep it in the required CI gate (and deal with false failures blocking your team) or remove it (and lose coverage you need).
Test quarantine resolves this dilemma. You keep the test — it still runs, results are still tracked — but it no longer blocks your pipeline. You can investigate and fix it without the pressure of constant CI failures.
Done poorly, quarantine becomes a graveyard where flaky tests go to be forgotten. Done well, it's a triage system that keeps your pipeline stable while you fix the underlying problems.
What Quarantine Is (and Isn't)
Quarantine is:
- Temporary separation of unreliable tests from your required CI gate
- A structured holding state with tracking and time limits
- A signal that a test needs attention
Quarantine is not:
- A way to permanently hide flaky tests
- Equivalent to deleting tests
- An excuse to stop caring about the failing test
The critical difference from skip or xit: quarantined tests still run. They generate results. Someone owns fixing them. There's a deadline.
Pattern 1: Tag-Based Quarantine (Recommended)
Tag-based quarantine uses test runner labels to separate flaky tests from the required pass gate, without removing them from the codebase.
Jest
Use a custom marker in the test description and filter by it in CI:
// Mark flaky tests with [QUARANTINE] prefix
describe('[QUARANTINE] UserCheckout', () => {
it('completes payment with saved card', async () => {
// test body unchanged
});
});In CI, run two separate jobs:
# .github/workflows/test.yml
jobs:
test-required:
name: Required Tests
runs-on: ubuntu-latest
steps:
- run: |
jest --testPathIgnorePatterns="\[QUARANTINE\]" \
--testNamePattern="^(?!.*\[QUARANTINE\])"
# This job is required for merge
test-quarantine:
name: Quarantined Tests (Non-blocking)
runs-on: ubuntu-latest
continue-on-error: true # does NOT block merge
steps:
- run: |
jest --testNamePattern="\[QUARANTINE\]" || trueThe continue-on-error: true means the quarantine job runs and its results are visible, but failure doesn't block the PR.
pytest
Use a custom mark:
# conftest.py
import pytest
def pytest_configure(config):
config.addinivalue_line(
"markers", "quarantine: mark test as quarantined (flaky, under investigation)"
)# test_payment.py
@pytest.mark.quarantine
def test_payment_with_saved_card():
# test body unchangedCI configuration:
- name: Required tests
run: pytest -m "not quarantine" # required, fails build
- name: Quarantined tests
run: pytest -m "quarantine" || true # informational only
continue-on-error: trueGo
Use build tags or test name conventions:
func TestPaymentFlaky(t *testing.T) {
if os.Getenv("RUN_QUARANTINE") == "" {
t.Skip("quarantined: flaky test, see issue #456")
}
// test body
}In CI:
# Required tests (skip quarantined)
go test ./...
# Quarantine job (run quarantined)
RUN_QUARANTINE=1 go test ./... || truePattern 2: Separate Test File Quarantine
Instead of marking individual tests, move flaky tests to a separate directory:
tests/
unit/ # required in CI
integration/ # required in CI
quarantine/ # informational only
checkout.flaky.test.js
payment.flaky.test.jsCI configuration:
- name: Required tests
run: jest --testPathIgnorePatterns="quarantine/"
- name: Quarantine tests
run: jest --testPathPattern="quarantine/" || true
continue-on-error: truePros: very clear separation, easy to see what's quarantined at a glance Cons: moving files changes git history, harder to see quarantine context alongside the original test suite
Use this when you have many tests to quarantine at once or want the strongest possible separation.
Pattern 3: Quarantine via CI Retry with Tracking
Some teams don't want to mark tests but still need to manage flakiness. An alternative: allow retries but track every retry as a quarantine signal.
// jest.config.js
module.exports = {
reporters: [
'default',
['./reporters/flakiness-reporter.js', { threshold: 1 }]
]
};
// reporters/flakiness-reporter.js
class FlakinessReporter {
onTestResult(test, testResult) {
testResult.testResults.forEach(result => {
if (result.numPassingAsserts > 0 && result.failureMessages.length === 0
&& result.invocations > 1) {
// test passed but required retries — log it
console.warn(`FLAKINESS DETECTED: ${result.fullName} (required ${result.invocations} attempts)`);
// send to your metrics system
}
});
}
}Pros: no manual marking required, works automatically Cons: requires custom tooling, retry cost still paid
Quarantine Hygiene: Making It Work Long-Term
Quarantine is only as good as its governance. Without discipline, tests enter quarantine and never leave.
Rule 1: Every quarantine needs a ticket
Before marking a test as quarantined, create a tracking issue:
// [QUARANTINE] UserCheckout — see issue #789
describe('[QUARANTINE] UserCheckout', () => {The issue should include:
- What failure mode was observed
- Hypothesis for root cause
- Who owns the investigation
- Target resolution date
No ticket = the quarantine gets forgotten.
Rule 2: Time limits
Establish a policy: quarantine lasts at most 14 days. After 14 days, the test gets either fixed or deleted. Make an explicit decision either way.
Some teams enforce this with CI checks:
// check-quarantine-age.js
const { execSync } = require('child_process');
const maxAgedays = 14;
const quarantinedTests = findQuarantinedTests('./tests');
quarantinedTests.forEach(test => {
const blame = execSync(`git log -1 --format="%ar" -- ${test.file}`).toString();
const ageMatch = blame.match(/(\d+) days? ago/);
if (ageMatch && parseInt(ageMatch[1]) > maxAgeDays) {
console.error(`QUARANTINE EXPIRED: ${test.name} has been quarantined for ${ageMatch[1]} days`);
process.exit(1);
}
});Rule 3: Regular review
Add "quarantine review" to your team's weekly engineering sync. Walk through:
- Which tests are currently quarantined?
- What's the status of each fix?
- Are any approaching or past the deadline?
Five minutes per week prevents quarantine from accumulating silently.
Rule 4: Quarantine is reversible
When you fix a flaky test, remove the quarantine marker immediately and verify it passes reliably. Don't leave the marker "just in case."
If a fixed test becomes flaky again, re-quarantine it — with a new ticket that references the original and the fix that didn't hold.
What to Do During Quarantine
Quarantine creates space to investigate without blocking your team. Use that space:
Reproduce systematically
# Run the quarantined test many times to characterize flakiness
for i in {1..50}; do
jest --testNamePattern="\[QUARANTINE\] UserCheckout" 2>&1 | tail -1
done | sort | uniq -cIf it fails 5 out of 50 times, you have 10% flakiness. Now you know what you're dealing with.
Check environmental factors
Does it fail more on CI than locally? If so, investigate environment differences:
- Parallelism (CI often runs tests concurrently)
- Resource constraints (CI has less memory/CPU)
- Network latency (CI network is different from local)
- Missing environment variables
Bisect the failure
If order-dependent, bisect to find which test causes the failure:
# Pytest: run test in isolation
pytest tests/test_payment.py::test_payment_with_saved_card -v
# Run with specific test ordering
pytest --randomly-seed=12345 tests/Add logging without changing behavior
Add temporary logging to understand what state the test is in when it fails:
it('completes payment', async () => {
console.log('Starting test, cart state:', await getCartState());
// ... test body
console.log('Assertion state:', await getPaymentState());
expect(payment.status).toBe('completed');
});Logging the state at the point of failure often reveals the root cause immediately.
Measuring Quarantine Effectiveness
Track:
- Quarantine intake rate: how many tests enter quarantine per week? Rising intake signals growing flakiness in new code.
- Resolution rate: how many quarantined tests get fixed per week? Should exceed intake rate.
- Average time-to-fix: how long does a test stay in quarantine before being fixed? Should be well under your limit.
- Recidivism: how often do "fixed" tests return to quarantine? High recidivism means fixes are addressing symptoms, not causes.
A healthy quarantine system has:
- ≤5 tests quarantined at any given time
- Average resolution time <7 days
- Intake rate declining over time
If you have 40 tests in quarantine with some there for months, the system isn't working — the underlying flakiness is accumulating faster than fixes.
Integrating With HelpMeTest
If you're using HelpMeTest for continuous test monitoring, you can see which tests are failing intermittently across runs — even without a formal quarantine system. This helps you identify candidates for quarantine proactively, before a developer reports "this test keeps failing."
The workflow:
- HelpMeTest flags tests with intermittent failures over the past week
- You review the list and quarantine the confirmed flaky ones
- HelpMeTest continues monitoring quarantined tests so you can verify fixes
Proactive identification means you're quarantining tests before they've blocked multiple developers, not after.
Summary: The Quarantine Checklist
When a test needs quarantine:
- Add quarantine marker (
[QUARANTINE],@pytest.mark.quarantine, etc.) - Create tracking ticket with failure description and root cause hypothesis
- Set resolution deadline in ticket
- Assign owner
- Verify quarantined test still runs in CI (informational job)
- Start investigation
When resolving a quarantine:
- Remove quarantine marker
- Run test 50+ times to confirm reliability
- Close tracking ticket
- Document what the root cause was and what fixed it
Quarantine isn't the goal — it's the bridge between discovering a flaky test and fixing it. The goal is a test suite where every test in the required gate is genuinely reliable.