SRE Game Days: Running Incident Simulations to Test Your Runbooks
A runbook that hasn't been used in a real incident is a hypothesis. You've documented a procedure, but you haven't proven it works under the conditions that matter — when the database is actually failing, the on-call engineer has been paged at 2am, and the monitoring dashboard is showing five different alerts at once.
A game day tests the runbook before the real incident does. It's a scheduled, scripted simulation of an incident that lets you measure how well your team and systems respond — without the actual blast radius of a production outage.
Game Days vs. Chaos Engineering
These two practices are related but distinct.
Chaos engineering focuses on discovering unknown failure modes in the system. You inject faults and observe what happens. The system's behavior is the unknown.
Game days focus on testing the response capability — how quickly you detect the incident, how effectively the on-call follows the runbook, whether escalation paths work, whether communication channels are set up correctly. The team's behavior is the unknown.
You need both. Chaos engineering tells you if the system survives failures. Game days tell you if the team can handle them when they occur.
Planning a Game Day
A game day needs a scenario, a set of roles, success criteria, and an explicit abort condition. Document all of this before the day.
Scenario Selection
Good game day scenarios are:
- Based on real incidents that have happened (or near-misses)
- Specific enough to test a particular runbook
- Realistic in terms of what on-call would actually see
Classic scenario types:
| Scenario | Runbook Being Tested | Signals on-call Would See |
|---|---|---|
| Primary database failure | DB failover runbook | Availability alerts, connection errors in logs |
| Traffic spike (5x normal) | Scaling runbook | High latency, CPU alerts, rate limiting logs |
| Memory leak | Pod restart / OOM runbook | Memory alerts, OOMKilled events, pod restarts |
| Cascading failure (downstream dep) | Dependency failure runbook | Increased error rate, circuit breaker open logs |
| Certificate expiry | Cert renewal runbook | HTTPS errors, cert monitoring alerts |
| Data pipeline lag | Pipeline recovery runbook | Data freshness SLI alert, job failure events |
Roles
Every game day needs:
Scenario Controller — knows the full script, injects faults at the right time, controls when the simulation ends. Does not help the responders unless they're about to cause real damage.
Incident Commander — the on-call or team lead playing the role of IC. Runs the response as if it's real.
Responders — 1-3 engineers who follow the runbook to resolve the incident.
Observer — takes timestamped notes on what happens. This is the most valuable role for improving the runbook afterwards.
Safety Officer (optional but recommended for production game days) — has authority to stop the simulation if real systems are at risk.
Pre-Game Day Checklist
# Game Day Pre-Flight Checklist
## Scenario Preparation
- [ ] Scenario script documented (fault injection steps, timing)
- [ ] Fault injection commands tested in staging (verify they work)
- [ ] Rollback commands documented and tested
- [ ] Abort conditions defined (what would cause immediate stop)
## System Preparation
- [ ] Runbook under test is current and accessible
- [ ] Monitoring dashboards are working
- [ ] Alerting is configured (alerts will actually fire)
- [ ] Communication channels are set up (incident Slack channel, video bridge)
- [ ] Change freeze period confirmed (no other changes during game day)
## Team Preparation
- [ ] Roles assigned and acknowledged
- [ ] Observer has template for timestamped notes
- [ ] All participants know the abort signal
- [ ] Debrief scheduled for same day or next morning
## Metrics Baseline
- [ ] Pre-game day metrics recorded (availability, latency, error rate)
- [ ] Current error budget remaining noted
- [ ] All services showing healthy in monitoringExample Scenario: Database Failover
This is the most valuable game day for any team with a critical database.
Hypothesis: When the primary PostgreSQL instance fails, the system detects the failure within 5 minutes, engineers follow the failover runbook, and service is restored within 30 minutes.
Success criteria:
- MTTD (time from fault injection to alert firing): < 5 minutes
- MTTR (time from alert to service restored): < 30 minutes
- Runbook is followed without needing verbal guidance
- No data loss
Script:
## Database Failover Game Day Script
### T-15: Setup
- [ ] Controller: verify staging DB is healthy
- [ ] Controller: verify backup alerts are working
- [ ] Observer: start timer, recording sheet open
- [ ] All: join incident channel (do not interact until paged)
### T-0: Fault Injection
Controller runs:
```bash
# Simulate primary DB failure (staging environment only)
kubectl exec -n staging deployment/postgres-primary -- \
pg_ctl stop -m immediate
# Alternatively, use a network policy to block DB traffic:
kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: block-db-chaos
namespace: staging
spec:
podSelector:
matchLabels:
app: payments-api
egress:
- ports:
- port: 5432
to: [] # block all DB egress
EOFObserver: note exact time of fault injection.
T+0 to T+5: Detection Phase
- Does the availability alert fire within 5 minutes? (Note exact time)
- Does the error rate alert fire?
- Are the correct people paged?
Observer records:
- Time to first alert
- Which alert fired first
- Who was paged and how (PagerDuty, Slack, etc.)
T+5 to T+30: Response Phase
Responders receive the alert. Observer watches without intervening.
Expected runbook steps:
- Acknowledge the alert
- Open the DB failover runbook
- Verify primary is actually down (not a false positive)
- Check replica status
- Promote replica to primary
- Update connection strings (if not automatic)
- Verify service recovery
- Notify stakeholders
Observer records:
- Each runbook step started and completed (with timestamps)
- Any step where responders got stuck (> 2 min on one step)
- Any step where responders deviated from the runbook
- Any communication sent to stakeholders
T+X: Resolution
Once service is restored, controller calls "END GAME DAY". Observer records exact restoration time.
Abort Conditions
Stop immediately if:
- Real production traffic is affected
- Fault injection cannot be reversed
- Responders take action that could cause data loss
**Post-scenario metrics to capture**:
```python
# game_day_metrics.py
import datetime
game_day_results = {
"scenario": "DB Failover - Staging",
"date": "2026-05-15",
"fault_injection_time": datetime.datetime(2026, 5, 15, 14, 0, 0),
"first_alert_time": datetime.datetime(2026, 5, 15, 14, 3, 22),
"page_received_time": datetime.datetime(2026, 5, 15, 14, 3, 45),
"runbook_opened_time": datetime.datetime(2026, 5, 15, 14, 5, 10),
"service_restored_time": datetime.datetime(2026, 5, 15, 14, 22, 30),
"mttd_seconds": (datetime.datetime(2026, 5, 15, 14, 3, 22) -
datetime.datetime(2026, 5, 15, 14, 0, 0)).seconds,
"mttr_seconds": (datetime.datetime(2026, 5, 15, 14, 22, 30) -
datetime.datetime(2026, 5, 15, 14, 3, 22)).seconds,
"runbook_deviations": [
"Step 3: Responder verified primary status via kubectl instead of runbook command",
"Step 5: Responder promoted replica manually — automation script was not known",
],
"stuck_points": [
{"step": 5, "duration_seconds": 240, "reason": "Could not find the replica promotion command in runbook"},
],
"success_criteria_met": {
"mttd_under_5min": True, # 3m 22s
"mttr_under_30min": True, # 18m 45s
"no_verbal_guidance_needed": False, # Controller had to hint at replica promotion
"no_data_loss": True,
}
}
print(f"MTTD: {game_day_results['mttd_seconds'] // 60}m {game_day_results['mttd_seconds'] % 60}s")
print(f"MTTR: {game_day_results['mttr_seconds'] // 60}m {game_day_results['mttr_seconds'] % 60}s")
print(f"Success: {all(game_day_results['success_criteria_met'].values())}")Example Scenario: Traffic Spike
Hypothesis: When traffic increases to 5x normal, the service scales up via HPA and maintains p95 < 500ms without manual intervention.
Fault injection with k6:
// game-day/traffic-spike.js
import http from 'k6/http';
import { sleep } from 'k6';
export const options = {
stages: [
{ duration: '2m', target: 100 }, // normal load
{ duration: '1m', target: 500 }, // SPIKE: 5x normal
{ duration: '10m', target: 500 }, // sustained spike
{ duration: '2m', target: 100 }, // recovery
],
};
export default function () {
http.get(`${__ENV.BASE_URL}/api/products`);
sleep(0.1);
}# Controller runs this to inject the traffic spike
k6 run \
--env BASE_URL=https://staging.example.com \
game-day/traffic-spike.js
# Responders are watching:
# - HPA events: kubectl get hpa -w -n staging
# - Pod count: kubectl get pods -n staging -w
# - Latency dashboard: Grafana
# - Alerts: PagerDuty/SlackWhat to watch for:
- Did the latency alert fire when p95 exceeded threshold?
- Did the HPA scale up? How long did it take?
- Did the team know how to manually trigger scaling if HPA failed?
- Were stakeholders notified of the performance degradation?
Example Scenario: Cascading Failure
Hypothesis: When a downstream dependency fails, the circuit breaker opens and the service returns degraded responses rather than timing out or cascading.
# Controller: simulate downstream dependency failure
# Network policy that drops connections to downstream service
kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: chaos-downstream-block
namespace: staging
spec:
podSelector:
matchLabels:
app: payments-api
egress:
- to:
- podSelector:
matchLabels:
app: catalog-service
ports: [] # block all
EOFWhat to test: Does the payments API return 200 with degraded recommendations (circuit open) or 500 (circuit not working)? Does the circuit breaker alert fire? Does the runbook tell responders how to verify circuit breaker state and force reset?
Running the Debrief
The debrief is where the value is captured. Run it within 24 hours of the game day, while details are fresh.
Structure:
## Game Day Debrief: [scenario] — [date]
### What happened (facts, no blame)
Timeline of events from Observer's notes.
### What went well
- [Example: HPA scaled up within 2 minutes of spike]
- [Example: All responders knew where to find the runbook]
### What could be improved
- [Example: Replica promotion command was not in the runbook — 4-minute delay]
- [Example: Stakeholder communication template was missing from runbook]
### Runbook improvements
| Issue | Runbook Change | Owner | Due |
|---|---|---|---|
| Replica promotion command missing | Add explicit kubectl command to step 5 | DB team | 2026-05-22 |
| No stakeholder template | Add template to runbook appendix | Ops | 2026-05-22 |
### System improvements
| Issue | System Change | Owner | Due |
|---|---|---|---|
| HPA scale-up took 4 minutes | Reduce HPA cooldown from 5m to 2m | Platform | 2026-05-29 |
### MTTD / MTTR vs. target
| Metric | Target | Actual | Status |
|---|---|---|---|
| MTTD | < 5 min | 3m 22s | PASS |
| MTTR | < 30 min | 18m 45s | PASS |
### Next game day
- Date: [4-6 weeks from now]
- Scenario: [something that failed or was weak this time]Measuring Improvement Over Time
Track MTTD and MTTR across game days. If they're not improving, something is wrong with how you're incorporating learnings.
game_day_history = [
{"date": "2026-02-15", "scenario": "DB Failover", "mttd": 8.5, "mttr": 42},
{"date": "2026-03-20", "scenario": "DB Failover", "mttd": 5.1, "mttr": 28},
{"date": "2026-04-18", "scenario": "DB Failover", "mttd": 3.4, "mttr": 22},
{"date": "2026-05-15", "scenario": "DB Failover", "mttd": 3.4, "mttr": 19},
]MTTD improved from 8.5 to 3.4 minutes. MTTR improved from 42 to 19 minutes. That improvement happened because of runbook changes and system improvements found during game days. That's the point.
Your runbooks are only as good as the last time you tested them. Schedule game days quarterly. Test your most critical scenarios. Update your runbooks from what you find. The next real incident will be shorter because of it.
