GameDay Exercises: How to Run Chaos Engineering Drills with Your Team

What Is a GameDay?

A GameDay is a scheduled exercise where you:

1. Agree on a failure scenario in advance (e.g., "database becomes unavailable for 5 minutes")
2. Prepare the team (everyone knows something is going to break, but maybe not exactly what or when)
3. Execute the failure
4. Observe how the system responds and how the team responds
5. Debrief and document findings

The key difference from automated chaos: a GameDay tests people and processes as much as it tests the system. Can the on-call engineer detect the incident? Do they follow the runbook? Can they escalate correctly? Does the runbook work?

When to Run GameDays

Before a major release — test resilience of the new architecture before it carries production traffic

After a significant incident — verify the fix actually addresses the root cause, not just the surface symptom

Quarterly for critical services — maintain resilience as systems evolve

When onboarding new engineers — hands-on incident response training in a safe environment

Before on-call rotations — make sure new on-call engineers have experienced failure conditions

Planning a GameDay

Choose a Scenario

Good GameDay scenarios are:

• Realistic — things that have happened or could plausibly happen
• Bounded — clear start/stop conditions, limited blast radius
• Instructive — reveal something about system or team resilience you don't already know

Example scenarios by complexity:

Beginner:

• Kill one application pod in a 3-pod deployment
• Simulate a 500ms latency spike on a database connection
• Fill disk on one application server to 90%

Intermediate:

• Kill all pods in one service and observe how dependent services behave
• Simulate a complete loss of one AWS availability zone
• Inject 30% packet loss between your API and database

Advanced:

• Simulate a cascading failure where one service's failure causes downstream timeouts that exhaust a connection pool, causing a second service to fail
• Simulate a "grey failure" where a service returns 200 OK but with corrupted data
• Test failover to a different region

Define the Hypothesis

Write it down before the exercise:

Hypothesis: If [failure condition], then [expected system behavior],
measured by [specific metrics].

Example:
If we terminate the primary database and force failover to the replica,
the application will return errors for no more than 30 seconds,
p99 latency will return to baseline within 60 seconds,
and zero orders will be lost.

Set Success and Failure Criteria

Assign Roles

• Chaos executor: the person who introduces the failure (usually known to organizers only)
• On-call engineer: responds as if it's a real incident (doesn't know when/what will happen)
• Observer(s): watch metrics, take notes, don't intervene
• Facilitator: manages the exercise, calls stop if needed, controls communications

For learning-focused exercises, tell the on-call what type of failure will be introduced (e.g., "a network partition will occur at some point") but not exactly when. For more advanced exercises, keep them completely in the dark.

Set Stop Conditions

Define before starting when you'll abort:

• Real production impact beyond the expected scope
• Data loss (unexpected)
• Cascading failures beyond the target zone
• Any safety-critical customer-facing SLA at risk

Document who has the authority to call stop and how to revert.

Running the GameDay

Pre-Exercise Checklist

□ All team members know their roles
□ Stop conditions documented and agreed
□ Communication channel established (Slack, call bridge)
□ Monitoring dashboards open and visible
□ Runbooks accessible
□ Rollback plan ready
□ Stakeholders notified (if customer-facing systems involved)
□ Start time agreed

The Timeline

T-30 minutes: Briefing

• Explain the scenario type and scope to all participants
• Confirm everyone has access to monitoring tools
• Review stop conditions
• Start recording the session (for debrief)

T-0: Failure injection

• The chaos executor introduces the failure
• Note exact time
• Start the timer on detection/response metrics

T+0 to T+N: Observation phase

• On-call engineers respond normally
• Observers record:
  • When failure was detected
  • What alerts fired (and when)
  • What the on-call did first, second, third
  • What the system did (graceful degradation? full failure?)
  • Any surprises

T+N: Recovery or stop

• Either the system recovers on its own, or the on-call resolves it
• Or the facilitator calls stop
• Note recovery time

Post-exercise: Restoration

• Verify all systems returned to steady state
• Check for data integrity issues
• Confirm no lingering effects

Communication Template for Stakeholders

If the exercise might cause visible degradation:

[GameDay Notice]
Date: [DATE]
Time: 2:00 PM - 3:00 PM EST
Scope: [SERVICE NAME] in staging environment
Expected impact: Possible degradation of [FEATURE] for up to 5 minutes
Point of contact: [NAME] at [CONTACT]

This is a planned resilience exercise. Any issues outside this window are real incidents.

The Debrief

The debrief is where the learning happens. Run it within 24 hours while the exercise is fresh.

Questions to Cover

Detection:

• How was the failure detected? Alert, user report, dashboard, or not at all?
• How long did detection take?
• Were the right people notified?

Diagnosis:

• How long to identify the root cause?
• Was the runbook helpful?
• What information was missing that would have helped?

Response:

• Was the response correct?
• Were any steps taken that made things worse?
• Was escalation handled correctly?

Recovery:

• How long did recovery take?
• Was recovery automated or manual?
• Could the recovery have been faster?

System behavior:

• Did the system degrade gracefully or fail catastrophically?
• Were circuit breakers, retries, and fallbacks effective?
• Were there any unexpected dependencies?

Blameless Post-Mortem Format

Use a blameless format — the goal is system improvement, not attribution:

## GameDay Post-Mortem: [SCENARIO NAME]
**Date:** [DATE]
**Duration:** [START] to [END]
**Participants:** [LIST]

## What We Did
Brief description of the failure scenario.

## Timeline
- 14:02:00 - Failure injected: [what was done]
- 14:03:45 - First alert fired: [which alert]
- 14:05:12 - On-call began investigation
- 14:07:30 - Root cause identified
- 14:09:00 - Fix applied
- 14:10:00 - System recovered to steady state

## What Worked
- Auto-scaling replaced the failed pod within 90 seconds
- Alert fired within 2 minutes as expected
- Runbook accurately described the recovery steps

## What Didn't Work
- Alert email went to old distribution list — no one received it
- Database failover took 4 minutes, not 30 seconds as expected
- The runbook step "restart the application" was ambiguous — which app?

## Action Items
| Action | Owner | Due Date |
|--------|-------|----------|
| Update alert email distribution list | @devops | 2026-06-07 |
| Investigate slow database failover | @dba-team | 2026-06-14 |
| Clarify runbook step 3 | @on-call | 2026-06-07 |

GameDay Progression

Run GameDays in increasing complexity:

Level 1: Component failure Single service, controlled environment, team knows exactly what will happen. Goal: verify monitoring and runbooks work.

Level 2: Dependency failure Fail a dependency (database, cache, external API) and observe how dependent services behave. Goal: verify circuit breakers and fallbacks.

Level 3: Multi-component failure Fail multiple components to simulate realistic cascading failures. Goal: test incident response coordination.

Level 4: Region/AZ failure Simulate loss of an entire availability zone. Goal: test failover procedures and data replication.

Level 5: Surprise scenario The team knows there will be a GameDay this week but not what or when. Goal: test readiness under realistic uncertainty.

Common Pitfalls

Not defining stop conditions — the exercise escalates beyond the intended scope because no one knows when to call it. Always define stop conditions in advance.

Skipping the debrief — the debrief is where GameDays produce value. An exercise without a structured debrief produces experience but not learning.

Running too infrequently — a GameDay once a year teaches you about the system as it was a year ago. Quarterly or monthly exercises track current resilience.

Punishing mistakes — if engineers who make mistakes during GameDays face blame or criticism, they'll stop participating honestly. GameDays require psychological safety.

Not following up on action items — post-mortem action items that go unaddressed mean the same failure will surface again in a real incident.

GameDay Template Scenarios

Database Failover

• Inject: Terminate primary database, force failover to replica
• Observe: How long does failover take? Do writes queue or fail? Are there duplicate records?
• Measure: Failover time, error rate, data integrity

Cache Eviction

• Inject: Flush all caches (Redis, Memcached)
• Observe: Does the application fall back to database gracefully? Does response time spike?
• Measure: Response time before/after, database query rate, error rate

Message Queue Overflow

• Inject: Stop consumers, let the queue fill for 5 minutes, restart consumers
• Observe: Do messages process in order? Are any lost? Do producers back off?
• Measure: Queue depth, processing latency, message loss rate

DNS Resolution Failure

• Inject: Point a service's DNS entry to an invalid IP for 60 seconds
• Observe: Do clients use cached DNS? How quickly do connection errors surface?
• Measure: Time to first error, error rate, recovery time after DNS fix