Game Day Exercises: How to Run a Team Chaos Experiment

What a Game Day Is

A game day (sometimes called a chaos day or fire drill) is a structured event where a team intentionally introduces failures into their system and observes how it behaves. The name comes from sports: just as teams practice under game-like conditions, engineering teams practice under failure-like conditions.

The format varies, but the structure is consistent:

• A facilitator designs experiments in advance
• The team gathers (in-person or remote) with dashboards and monitoring visible
• Experiments are executed one at a time with the whole team watching
• Results are observed, documented, and discussed
• A retrospective follows

Unlike continuous chaos experiments running on a schedule, a game day is an intentional team-level learning event.

Why Run Game Days

Practice incident response. Most engineers learn how to handle production incidents by being on-call and getting paged at 3am. Game days let teams practice incident response when they're rested, prepared, and have time to learn from the experience.

Surface hidden assumptions. System design documents say how a system should behave under failure. A game day reveals how it does behave. The gap between those is where the most valuable bugs live.

Build team confidence. Teams that have deliberately broken their system and successfully recovered from it are measurably more confident handling real incidents. The experience transfers.

Test runbooks. Runbooks written months ago may be outdated. A game day validates that runbooks work before they're needed in a real incident.

Create organizational alignment. When product managers and support leads watch engineers deliberately break the system and handle the fallout, it builds cross-functional understanding of reliability requirements.

Planning a Game Day: 3 Weeks Out

Week 1: Design and Scope

Define the objectives. What does the team want to learn? Options:

• "We want to verify our database failover works as documented."
• "We want to test how users experience a 2-second latency injection."
• "We want to practice our on-call runbook for service X going down."

Select experiments. 4–6 experiments per half-day game day. Each should:

• Have a specific hypothesis
• Be reversible
• Have a defined blast radius
• Have a clear abort condition

Identify the target system. Beginners should run game days in staging with synthetic traffic. Advanced teams run in production with real traffic, but this requires established resilience practices and fast rollback capability.

Assign roles:

• Facilitator: Runs the agenda, calls experiments, calls aborts. Doesn't also monitor dashboards.
• Chaos engineer: Executes the actual failure injections.
• Observers: Watch dashboards, document what they see.
• Scribe: Records results, anomalies, and observations in real time.

Week 2: Preparation

Verify observability. If you can't measure the impact of a failure, the experiment teaches nothing. Confirm that dashboards are up to date and cover the services involved. Check that alerting is configured.

Communicate with stakeholders. If the game day involves production traffic, notify customer support, customer success, and product management. Set expectations for possible service degradation.

Prepare rollback procedures. For each experiment, document exactly how to reverse the failure injection. Test rollback procedures in staging before game day.

Write the experiment runbook. For each experiment:

Experiment: <name>
Hypothesis: <what we expect to happen>
Failure to inject: <what we'll break>
Blast radius: <what's affected>
Duration: <how long the experiment runs>
Abort condition: <when to stop immediately>
Rollback: <exactly how to reverse it>
Steady state: <what metrics define normal>

Week 3: Rehearsal and Prep

Rehearse experiment execution. Have the chaos engineer practice each injection in staging. Verify that tooling works, rollback procedures work, and timing is understood.

Prepare dashboards. Set up monitoring dashboards with 15-minute windows and clear baselines. The whole team should be able to read them during the event.

Brief the team. Walk through the agenda, roles, and abort conditions the day before. Everyone should know the plan before executing it.

Running the Game Day

Opening (30 minutes)

• Review the objectives and agenda
• Confirm roles
• Verify dashboards are showing steady-state metrics
• Review abort conditions for all experiments
• Confirm rollback procedures are ready

Experiment Execution

For each experiment:

1. State the hypothesis. Facilitator reads it aloud. Everyone should understand what's expected before the failure is injected.
2. Confirm steady state. Look at dashboards. Are metrics nominal? If not, abort this experiment and investigate.
3. Inject the failure. Chaos engineer executes the injection. Note the exact time.
4. Observe. Watch dashboards for the duration. Scribe documents what happens, with timestamps.
5. Call it. At the end of the planned duration (or when the abort condition is triggered), facilitator calls the experiment done.
6. Rollback. Restore the system to normal state. Verify metrics return to baseline.
7. Debrief the experiment. 5 minutes: did the hypothesis hold? What surprised us? What do we want to investigate further?
8. Record results. Scribe documents hypothesis, outcome, and key observations.

Between Experiments

Allow 10–15 minutes between experiments for:

• Metrics to stabilize
• Brief discussion of observations
• Any investigation of anomalies discovered

Don't rush to the next experiment if the previous one raised questions.

Abort Protocol

The facilitator has authority to abort any experiment if:

• Abort conditions are triggered
• The team is uncertain about the impact
• Monitoring shows unexpected effects on systems outside the blast radius
• Anyone on the team calls for an abort

Abort is not failure. Abort means the experiment discovered something interesting that requires investigation.

Game Day Retrospective (2 hours after)

The retrospective is where learning is captured. Run it the same day while memory is fresh.

Structure

1. What did we expect? (15 min) Go through each experiment's hypothesis. What did we predict?

2. What actually happened? (20 min) Go through actual results. Where did behavior match the hypothesis? Where did it diverge?

3. What did we learn? (30 min) Key insights from the gap between expectation and reality. What does this mean for our system's resilience?

4. What do we fix? (20 min) For each weakness discovered, assign an owner and a timeline. Put it in the issue tracker immediately.

5. What should the next game day cover? (15 min) Based on what we learned, what should we explore next? The retrospective seeds the next game day's planning.

Outputs

The retrospective should produce:

• A written summary of each experiment (hypothesis, result, key finding)
• A list of remediation tasks with owners and timelines
• A list of proposed future experiments
• An updated runbook for any procedures that needed correction

Share the summary with stakeholders (product, support, leadership) within 48 hours.

Example Game Day Agenda

Duration: 4 hours

Common Mistakes

Too many experiments, too fast. 4–6 experiments is plenty for a half-day. Teams that pack in 10 experiments rush through each one and don't have time to observe and discuss.

No observability. If you can't see the metrics clearly during the experiment, you're flying blind. Spend time on dashboards before the game day.

Skipping the retrospective. The game day without a retrospective is a demo, not a learning event. Block retrospective time before the game day starts, not after.

Only involving engineers. Support leads and product managers gain valuable insight from game days. Their presence also creates organizational pressure to actually fix the weaknesses found.

Never repeating experiments. If an experiment revealed a weakness and you fixed it, re-run the same experiment 2 months later to confirm the fix holds. Game days aren't one-time events.