Chaos Engineering for SRE: GameDays and Failure Injection That Actually Work
Chaos engineering is the practice of intentionally breaking production systems to find weaknesses before they find you. Done wrong, it's just causing outages on purpose. Done right, it's a systematic method for building confidence in your system's resilience.
The difference between chaos that helps and chaos that hurts is structure: hypothesis, blast radius control, measurement, and learning.
The Hypothesis-First Approach
Every chaos experiment starts with a hypothesis, not an action.
Bad: "Let's kill some pods and see what happens." Good: "We believe that when the database primary fails, our application will detect it within 30 seconds and promote a replica with less than 5 seconds of user-visible impact."
The hypothesis has four parts:
- Steady state: What does normal look like? (success rate 99.9%, p99 < 200ms)
- Hypothesis: What do we believe will happen during failure?
- Experiment: What failure will we inject?
- Result: Does the hypothesis hold? What did we learn?
Document every experiment this way. If you can't write the hypothesis, you're not ready to run the experiment.
GameDay Structure
A GameDay is a structured event where the team intentionally causes failures and validates their responses. It's not ad-hoc breaking things — it's a rehearsal with defined success criteria.
Pre-GameDay preparation (1 week before):
- Define 3-5 failure scenarios with hypotheses
- Set blast radius limits (max customers affected, rollback triggers)
- Brief the on-call team on what's coming
- Prepare rollback procedures for each scenario
- Confirm monitoring and alerting is working
Day-of structure:
09:00 - Briefing: review scenarios, assign roles
09:30 - Establish baseline: verify steady state metrics
10:00 - Scenario 1: inject failure, observe, validate hypothesis
10:30 - Debrief Scenario 1: what happened vs. hypothesis?
11:00 - Scenario 2...
14:00 - Full debrief: learnings, action items, SLO impact assessmentRoles:
- Chaos lead: Runs the experiments, controls blast radius
- Observer: Watches metrics and records what actually happens
- Responder: Responds as if this is a real incident (not "just a test")
- Rollback owner: Authorized to stop the experiment immediately
Common Failure Injection Patterns
Pod failures (Kubernetes):
# Kill random pods in a deployment
kubectl delete pod -l app=api --force --grace-period=0
# Kill the leader of a stateful set
kubectl delete pod api-statefulset-0
# Drain a node (simulates node failure)
kubectl drain node-1 --ignore-daemonsets --delete-emptydir-dataNetwork failures:
# Add latency to network traffic
tc qdisc add dev eth0 root netem delay 200ms 50ms
# Drop 10% of packets
tc qdisc add dev eth0 root netem loss 10%
# Partition network (block traffic to/from specific IP)
iptables -A OUTPUT -d 10.0.0.5 -j DROPResource exhaustion:
# CPU saturation
stress-ng --cpu $(nproc) --timeout 60
# Memory pressure
stress-ng --vm 1 --vm-bytes 80% --timeout 60
# Disk I/O saturation
stress-ng --io 4 --timeout 60Application-level failures (Chaos Monkey patterns):
class ChaosMiddleware:
"""Inject failures at the application layer."""
def __init__(self, error_rate: float = 0.0, latency_ms: float = 0.0):
self.error_rate = error_rate
self.latency_ms = latency_ms
def process_request(self, request):
# Inject latency
if self.latency_ms > 0:
time.sleep(self.latency_ms / 1000)
# Inject errors
if random.random() < self.error_rate:
raise InjectedException("Chaos: simulated failure")
return self.next_handler(request)Chaos Experiment Automation
Manual GameDays are valuable but rare. Automated chaos experiments run continuously.
# chaos_experiments.py
class ChaosExperiment:
def __init__(self, name: str, hypothesis: str, blast_radius: BlastRadius):
self.name = name
self.hypothesis = hypothesis
self.blast_radius = blast_radius
def run(self) -> ExperimentResult:
# Measure steady state
steady_state = self.measure_steady_state()
if not steady_state.is_healthy:
return ExperimentResult.skipped(
reason="System not in steady state, unsafe to inject failure"
)
# Inject failure within blast radius limits
try:
self.inject_failure()
# Observe for experiment duration
observations = self.observe(duration_seconds=60)
# Verify hypothesis
hypothesis_held = self.evaluate_hypothesis(observations)
return ExperimentResult(
hypothesis_held=hypothesis_held,
steady_state_before=steady_state,
observations=observations,
hypothesis=self.hypothesis
)
finally:
# Always clean up
self.restore_steady_state()
class DatabaseFailoverExperiment(ChaosExperiment):
def __init__(self):
super().__init__(
name="database_primary_failure",
hypothesis="Application detects primary failure within 30s and promotes replica "
"with <5s user-visible impact (error rate spike < 10%)",
blast_radius=BlastRadius(max_error_rate=0.10, max_duration_seconds=120)
)
def inject_failure(self):
kill_database_primary()
def evaluate_hypothesis(self, observations):
# Time to detect failover
failover_detected_at = observations.first_event("replica_promoted")
detection_time = failover_detected_at - observations.start_time
# User-visible impact window
error_spike_duration = observations.duration_above_error_rate(threshold=0.01)
max_error_rate = observations.max_error_rate()
return (
detection_time <= 30 and
error_spike_duration <= 5 and
max_error_rate <= 0.10
)Measuring Chaos ROI
Chaos engineering has real costs: time spent running experiments, risk of actual customer impact, and engineering effort to fix discovered weaknesses. Measure the return.
Track:
- Hypotheses tested per quarter
- % of hypotheses that failed (the interesting ones — these found real problems)
- Mean time to detect injected failures (measures monitoring quality)
- Fixes implemented after chaos discovery
- SLO violations prevented (estimated based on what would have happened without the fix)
def generate_chaos_roi_report(quarter: str) -> ChaosROIReport:
experiments = get_experiments(quarter=quarter)
failed_hypotheses = [e for e in experiments if not e.hypothesis_held]
fixes_implemented = get_fixes_from_chaos(quarter=quarter)
return ChaosROIReport(
experiments_run=len(experiments),
hypotheses_failed=len(failed_hypotheses),
failure_rate=len(failed_hypotheses) / len(experiments),
weaknesses_discovered=len(failed_hypotheses),
fixes_implemented=len(fixes_implemented),
fix_implementation_rate=len(fixes_implemented) / max(len(failed_hypotheses), 1),
estimated_incidents_prevented=estimate_prevented_incidents(fixes_implemented)
)A healthy chaos program finds real problems 20-30% of the time. If every experiment passes, you're not testing hard enough. If every experiment fails, your system is too fragile to chaos safely.
Chaos in CI/CD
Not all chaos needs humans present. Some experiments can run automatically in staging:
# .github/workflows/chaos-staging.yml
name: Chaos Engineering - Staging
on:
schedule:
- cron: '0 10 * * 2' # Every Tuesday at 10am
jobs:
chaos-experiments:
runs-on: ubuntu-latest
environment: staging
steps:
- name: Verify staging health before chaos
run: pytest tests/chaos/prereqs/ --timeout=60
- name: Run pod failure experiment
run: python chaos/experiments/pod_failure.py --environment staging
- name: Run network partition experiment
run: python chaos/experiments/network_partition.py --environment staging
- name: Run dependency latency experiment
run: python chaos/experiments/dependency_latency.py --environment staging
- name: Generate chaos report
if: always()
run: python chaos/reports/generate.py --output chaos-report.json
- name: Post results
if: always()
run: python chaos/reports/post-to-slack.py --file chaos-report.jsonStopping Chaos Safely
Define automatic abort conditions before every experiment. If these trigger, the experiment stops immediately — no human decision required.
class BlastRadiusGuard:
def __init__(self, max_error_rate: float, max_duration_seconds: int):
self.max_error_rate = max_error_rate
self.max_duration_seconds = max_duration_seconds
self.start_time = time.time()
def check(self) -> bool:
"""Returns True if experiment should abort."""
# Abort if error rate exceeds blast radius limit
current_error_rate = get_current_error_rate()
if current_error_rate > self.max_error_rate:
logger.warning(
f"Aborting: error rate {current_error_rate:.2%} > limit {self.max_error_rate:.2%}"
)
return True
# Abort if experiment has been running too long
elapsed = time.time() - self.start_time
if elapsed > self.max_duration_seconds:
logger.warning(f"Aborting: experiment exceeded {self.max_duration_seconds}s limit")
return True
return FalseHelpMeTest can continuously validate your system during chaos experiments — running health checks against your endpoints and measuring response times throughout the experiment. This gives you independent verification of what users experience, separate from your internal metrics.
Summary
Effective chaos engineering requires:
- Hypothesis-first design — define expected behavior before injecting failures
- Structured GameDays with assigned roles, defined success criteria, and debrief
- Automated experiments for continuous staging validation
- Measured ROI to justify the investment and direct future experiments
- Automatic abort conditions that stop experiments before they cause real harm
Chaos engineering done right doesn't cause outages — it prevents them.
