SRE Incident Response Testing: Game Days, Tabletops, and Runbook Validation

Format: 90-minute meeting

Facilitator presents: "At 2:47 AM, PagerDuty fires. Error rate on checkout is 12%.
Revenue is down. On-call gets the alert."

Questions to work through:
1. Who is the incident commander?
2. What are your first three diagnostic steps?
3. When do you escalate? To whom?
4. How do you communicate status to stakeholders?
5. At what point do you consider rollback?

Output:
- List of runbook gaps found
- Missing alert definitions
- Unclear escalation paths
- Action items with owners

Game Day: Database Failover
Duration: 2 hours
Scope: Staging environment (or production with careful planning)
Participants: On-call engineer, SRE lead, database team, product manager

Scenario:
10:00 AM - Facilitator triggers database primary failure
10:00 AM - On-call receives alert, starts timer
10:00-10:05 - Detection and initial diagnosis
10:05-10:20 - Incident response per runbook
10:20 - Expected: traffic fails over to replica, error rate recovers
10:20-10:40 - Debrief: what worked, what didn't

Measured:
- Time to detect: target < 2 minutes
- Time to diagnose root cause: target < 5 minutes  
- Time to recover: target < 15 minutes
- Runbook gaps found

DiRT Exercise: Region Failover
Scope: Production region us-east-1 → us-west-2
Notice: 2 hours (to responsible parties), no notice to responders
Duration: 4 hours

Steps:
1. Traffic shifted 100% to us-east-1 normally
2. Simulate us-east-1 network partition (block traffic at load balancer)
3. Responders must:
   a. Detect the failure
   b. Update DNS to route to us-west-2
   c. Verify data consistency
   d. Restore normal routing
4. Measure MTTD, MTTR, data loss
5. Compare against RTO/RPO objectives

Success criteria:
- RTO < 30 minutes (recovery time objective)
- RPO < 5 minutes (recovery point objective — acceptable data loss)

## Game Day Plan: Payment Service Outage

### Objectives
1. Validate that the payment-service-degraded alert fires within 3 minutes
2. Verify the runbook leads to correct diagnosis in < 10 minutes
3. Test PagerDuty escalation chain
4. Measure impact on checkout conversion (expected: detectable but < 5%)

### Scenario
Payment processor returns 503 for 30% of requests.
No announcement to responders.

### Execution
- Time: Tuesday, 10:00 AM PST (low traffic)
- Environment: Production (acceptable risk; payment processor has sandbox)
- Trigger: Facilitator enables feature flag `payment_errors_30pct`
- Duration: Max 45 minutes regardless of resolution

### Roles
- Facilitator: [Name] — controls scenario, takes notes
- Safety: [Name] — authorized to abort if real impact detected
- Responders: On-call rotation (unaware of game day)
- Observer: Product manager

### Abort Criteria
- Real error rate > 5% (beyond simulated)
- Customer contacts support
- Revenue impact > $1,000

### Data Collection
- Alert fire time
- First action time
- Root cause identified at time
- Mitigation time
- Full recovery time
- Runbook steps followed vs. skipped
- Escalations made

Debrief format (blameless):

1. Timeline (5 min)
   - Walk through what happened in chronological order
   - Focus on events, not people

2. What went well (5 min)
   - Alert fired on time ✓
   - Incident commander identified quickly ✓
   - Stakeholder communication was clear ✓

3. What could improve (15 min)
   - Runbook step 3 was confusing — "check the logs" with no specific query
   - Database dashboard took 3 minutes to load
   - Escalation path unclear after primary on-call

4. Action items (5 min)
   - [Name]: Update runbook step 3 with specific log query by [date]
   - [Name]: Investigate dashboard performance by [date]
   - [Name]: Document escalation path in PagerDuty by [date]

## Runbook Validation: API Gateway Restart

Validator: [Name]
Date: 2024-01-15
Environment: Staging

### Step-by-step validation

Step 1: "Check pod status with `kubectl get pods -n production`"
- Executed: ✓
- Command worked: ✓
- Output matched description: ✓

Step 2: "If pods are in CrashLoopBackOff, check logs with `kubectl logs <pod> --previous`"
- Executed: ✓
- Command worked: ✓
- Output matched description: ✗ — Description says logs show memory error, but pod was OOM killed (different output format)
- Fix needed: Update log example in runbook

Step 3: "Scale down and up: `kubectl rollout restart deployment/api-gateway`"
- Executed: ✓
- Time to complete: 47 seconds
- Runbook says "30 seconds" — update estimate

### Issues Found
1. Log output example outdated (OOM kill format changed in k8s 1.27)
2. Restart time estimate too optimistic

### Runbook Updated: ✓

# tests/runbooks/test_api_gateway_restart.py

def test_restart_recovers_from_crash_loop(cluster):
    """
    Validates: api-gateway-crashloop runbook
    Scenario: API gateway is in CrashLoopBackOff
    Expected: Runbook steps restore service within 5 minutes
    """
    # Setup: force CrashLoopBackOff
    cluster.inject_config_error('api-gateway', bad_env_var='DB_URL=invalid')
    
    # Wait for crash loop
    pod_status = cluster.wait_for_status('api-gateway', 'CrashLoopBackOff', timeout=120)
    assert pod_status == 'CrashLoopBackOff', "Pod did not enter CrashLoopBackOff"
    
    # Execute runbook step 1: fix config
    cluster.restore_config('api-gateway')
    
    # Execute runbook step 2: restart
    cluster.restart_deployment('api-gateway')
    
    # Validate: service should recover
    pod_status = cluster.wait_for_status('api-gateway', 'Running', timeout=300)
    assert pod_status == 'Running', f"Pod did not recover: {pod_status}"
    
    # Validate: health check passes
    health = cluster.health_check('api-gateway')
    assert health['status'] == 'ok'
    
    # Validate: within SLO recovery time
    total_time = cluster.get_recovery_time()
    assert total_time < 300, f"Recovery took {total_time}s, SLO is 300s"

Q1 2024: MTTR = 47 minutes (2 game days, 1 real incident)
Q2 2024: MTTR = 31 minutes (3 game days, 0 real incidents)
Q3 2024: MTTR = 18 minutes (4 game days, 2 real incidents)

## Incident Communication Protocol

### Severity Levels
- SEV1: Full outage, all customers affected → notify VP Engineering immediately
- SEV2: Partial outage or severe degradation → notify on-call manager
- SEV3: Minor degradation, monitoring → team channel only

### Status Page Updates (SEV1/SEV2)
- First update within 10 minutes of declaration
- Updates every 30 minutes until resolved
- Template: "We are investigating [symptom]. [# of customers] affected. Next update in 30 minutes."

### Internal Updates
- Incident Slack channel: #incidents
- Status every 15 minutes: "Still investigating. Current theory: [X]. Next step: [Y]."
- Escalation: If no progress in 20 minutes, escalate to next tier

Month 1: First tabletop exercise
- Scenario: "Top 3 alert types"
- 90 minutes, no system changes
- Collect 5+ action items

Month 2: First simple game day
- Scenario: Non-critical service restart
- Staging only
- 2 hours total with debrief

Month 3: Quarterly rhythm established
- Monthly tabletops
- Quarterly game days in staging
- Annual DiRT in production

Month 6: Advanced game days
- Production game days (careful, limited scope)
- Multi-team scenarios
- Cross-region failover

## Incident Post-Mortem: Payment API Outage
Date: 2024-03-15
Duration: 47 minutes
Impact: 23% of checkout attempts failed

### What happened
[5-minute timeline]

### Root cause
Database connection pool exhausted due to connection leak in payment-service v2.3.1.

### What worked well
- Alert fired within 2 minutes
- On-call followed runbook correctly
- Customer communication was clear

### What could improve
- Runbook didn't cover connection pool exhaustion
- No automated rollback triggered despite error rate > 10%
- Recovery took 20 extra minutes because responder didn't know how to force connection pool reset

### Action items
| Action | Owner | Due |
|---|---|---|
| Add connection pool exhaustion to runbook | [Name] | 2024-03-22 |
| Implement auto-rollback at 10% error rate | [Name] | 2024-04-01 |
| Add connection pool metrics dashboard | [Name] | 2024-03-22 |

### Next game day scenario
Test: connection pool exhaustion → use as next game day scenario
Date: 2024-04-15