Game Day & Chaos Readiness Testing: Prepare for Production Failures

14 Oct 2026 — 6 min read

Chaos engineering tools inject failures into systems. Game days are the events where teams deliberately experience those failures together, practicing their response before a real incident forces the issue. The combination — technical failure injection and human readiness practice — is what separates organizations that handle outages well from organizations that scramble.

What Is a Game Day?

A game day (sometimes called a fire drill, chaos day, or reliability exercise) is a planned event where a team intentionally triggers a failure scenario and practices responding to it. The goal is to:

Discover weaknesses in systems before production incidents reveal them
Validate that runbooks and incident procedures actually work
Build team muscle memory for incident response
Surface communication and coordination gaps

Game days originated at Amazon and were popularized through Netflix's chaos engineering program. They're now common in any organization running distributed systems at scale.

Why Game Days, Not Just Chaos Tests?

Automated chaos tests verify that systems handle failures correctly. Game days verify that teams handle failures correctly.

A system can have perfect retry logic and circuit breakers and still cause a 4-hour production incident because:

Nobody knew which service to check first
The runbook pointed to a dashboard that no longer existed
The on-call engineer had never seen this failure mode before
Communication channels weren't established for the incident
Escalation paths weren't clear

Game days surface these human and process issues in a controlled environment where the cost of mistakes is a lesson, not a customer impact event.

Planning a Game Day

Select a Failure Scenario

Start with realistic failure scenarios that have happened before or are plausible given your architecture:

Database primary node failure
Cache layer (Redis) becomes unavailable
Third-party payment processor returns 503s
Upstream API latency spikes to 5x normal
Disk on a specific server fills up
Network partition between two data centers
Deployment goes wrong and new pods are crash-looping

Choose scenarios that test critical paths — not edge cases. The first game days should cover the most impactful failure modes.

Define the Hypothesis

Before the game day, write a hypothesis — what you believe will happen:

Hypothesis: When the primary database becomes unavailable, 
the application will promote the replica within 30 seconds,
serving read-only requests during promotion and resuming 
full functionality after promotion completes.

Success criteria:
- Users see a "read-only mode" banner during promotion
- No data is lost
- Full functionality restored within 2 minutes
- Alert fires within 60 seconds
- On-call engineer acknowledges within 5 minutes

The hypothesis documents assumptions. When reality differs, that's a learning.

Set Up Safety Constraints

Game days in production require blast radius controls:

Time window: Run during low-traffic periods unless specifically testing peak-load behavior
Rollback plan: Document how to stop the experiment immediately if needed
User impact monitoring: Dashboard showing error rates and user-visible impact in real time
Stop conditions: Define thresholds that trigger automatic or manual halt (e.g., "if error rate exceeds 5%, stop")
Communication: Notify customer support so they can explain any user-visible impact

Assemble the Team

A game day involves multiple roles:

Facilitator: Runs the day, manages time, documents observations
Chaos engineer: Executes the failure injection
System owner: Leads the response, makes decisions
Observer(s): Watch for unexpected behaviors, ask questions
Communication lead: Manages Slack/incident channel, writes status updates

For the first game days, include people who don't normally work with the system — their confusion reveals undocumented assumptions.

Running the Game Day

The Timeline

T-30min  Briefing — review hypothesis, assign roles, confirm rollback plan
T-0      Inject failure
T+0:30   First check — is the system behaving as expected?
T+2:00   Response review — is the team following the runbook?
T+5:00   Communication check — are stakeholders informed?
T+varies Recovery — system restored to normal
T+15min  Hot debrief — immediate observations while fresh
T+1day   Full retrospective — written document with findings

During the Failure

The facilitator takes notes in real time:

What the system did (metrics, logs, alerts)
What the team did (actions taken, time taken, decisions made)
What worked as expected
What was surprising
What tools were missing or inadequate

Document everything. Memory degrades quickly after a stressful event.

The Hot Debrief

Immediately after recovery, run a 30-minute verbal debrief while the experience is fresh:

What happened? (facts, not opinions)
What went well?
What was harder than expected?
What was missing (tools, runbooks, knowledge, access)?
What would we do differently?

Keep it blameless. The goal is learning, not accountability for individual mistakes. Systems and processes caused the difficulty, not people.

Chaos Readiness Assessment

Before running a full game day, assess your readiness with a structured checklist:

Observability Readiness

Dashboards exist for all critical services
Alerts are configured for all high-priority failure modes
Alert routing is correct (right people get the right alerts)
Logging is sufficient to diagnose failures in the affected area
Distributed tracing covers the critical path

Incident Response Readiness

On-call rotation is defined and staffed
Runbooks exist for common failure scenarios
Runbooks have been tested recently (not just written and filed)
Escalation paths are documented and known
Communication channels for incidents are established

Recovery Readiness

Rollback procedures are documented and tested
Database backups exist and restoration has been tested
Failover procedures are documented
Key contacts for third-party dependencies are known
Access controls don't block incident response (responders can access what they need)

Score each section. Areas with gaps need remediation before they're exposed in a game day.

Failure Injection Techniques

Database Failures

# MySQL primary failure simulation
# Stop the primary, let replication promote
systemctl stop mysql

# Or using chaos tools
gremlin attack-shutdown --length 300 --targets tags/role=db-primary

# Verify promotion
mysql -h db-replica -e "SHOW SLAVE STATUS\G" | grep "Seconds_Behind_Master"

Network Partition

# Block traffic between app servers and cache
iptables -A OUTPUT -d 10.0.1.0/24 -p tcp --dport 6379 -j DROP

# Undo
iptables -D OUTPUT -d 10.0.1.0/24 -p tcp --dport 6379 -j DROP

API Latency Injection with ToxiProxy

# Add 3-second latency to payment API calls
Toxiproxy[:payment_api].downstream(:latency, latency: 3000).apply do
  # Observe system behavior during game day
  sleep(300)  # 5 minutes
end

Pod Failure in Kubernetes

# Kill a specific pod
kubectl delete pod payment-service-6b7d9c-xk4f2 -n production

# Kill all pods for a deployment (test rolling restart)
kubectl rollout restart deployment/payment-service -n production

# Simulate node failure with pod disruption
kubectl drain node-03 --ignore-daemonsets --delete-emptydir-data

Measuring Game Day Outcomes

Time-Based Metrics

MTTD (Mean Time to Detect): Time from failure injection to alert firing
MTTR (Mean Time to Resolve): Time from detection to full recovery
MTTA (Mean Time to Acknowledge): Time from alert to on-call acknowledgment

Track these across game days to measure improvement.

Process Metrics

Runbook accuracy: Was the runbook sufficient? How many steps needed to be improvised?
Communication quality: Were stakeholders kept informed? Were updates timely and accurate?
Tooling gaps: What information was hard to find? What actions required manual investigation?

System Metrics

Did the system behave as hypothesized?
Were there unexpected failure modes?
Did monitoring catch everything it should have?

Building a Game Day Program

Cadence

Start with one game day per quarter. After the team builds confidence and the most critical scenarios are covered, consider monthly game days for different scenarios.

Don't run game days so frequently that they become disruptive. The goal is preparedness improvement, not production chaos.

Rotation

Rotate the chaos scenario each game day. Cover your highest-risk failure modes first, then expand to more complex or less likely scenarios.

Keep a log of scenarios tested and their results. This becomes your failure scenario library.

Continuous Improvement

After each game day, create specific action items:

Runbook X needs to be updated to include step Y
Alert Z needs its threshold adjusted
Engineer A needs access to system B for incident response
Monitoring gap C needs to be closed before the next game day

Track these action items to completion. The game day retrospective is only valuable if it produces changes.

Integrating HelpMeTest

HelpMeTest tests running during a game day serve as a second verification channel. While the ops team watches infrastructure metrics, HelpMeTest tests verify user-visible behavior:

Is the checkout flow completing successfully?
Are user logins working?
Is the API returning correct responses?

If infrastructure metrics say "healthy" but HelpMeTest tests are failing, there's a discrepancy worth investigating. This dual-layer validation catches failures that infrastructure monitoring misses — the server is up, but the feature is broken.

Setting up HelpMeTest health checks on your critical user flows before a game day means you enter the exercise with automated behavioral monitoring already in place. The game day tests your team's response; HelpMeTest tests your users' experience.

Starting Small

If game days feel daunting, start with a chaos readiness assessment (no live failure injection). Identify the top three gaps. Fix them. Then run a small game day — non-critical dependency failure, limited blast radius, short duration.

The first game day will be chaotic. That's the point. The second will be less chaotic. By the fifth, your team will handle failures with the confidence that comes from having done it before.