Chaos Engineering for Beginners: Getting Started with Netflix Chaos Monkey

Why Netflix Built Chaos Monkey

In 2011, Netflix moved from their own data centers to AWS. In data centers, hardware failures are handled by ops teams who can physically replace components. In the cloud, failures are abstracted — an EC2 instance can disappear with no warning, and your application needs to handle that automatically.

The engineering team realized they had a choice: either believe they'd built resilient software, or prove it. Belief doesn't hold up at 2am during a production incident. Proof does.

Chaos Monkey was their proof mechanism: a tool that randomly killed production instances during business hours, forcing the team to build systems that could survive instance failure. If an instance died and caused an outage, better to find that out on a Tuesday afternoon than during peak holiday traffic.

The broader discipline — chaos engineering — emerged from that practice.

The Core Principles

The Principles of Chaos Engineering (principlesofchaos.org) defines the discipline around four ideas:

1. Build a hypothesis around steady-state behavior. What does normal look like? Define it in measurable terms: 99.9% of requests succeed, p99 latency is under 200ms, error rate is below 0.1%.

2. Vary real-world events. The failures you inject should reflect failures that actually happen: server crashes, network timeouts, dependency outages, disk full, CPU saturation.

3. Run experiments in production. The most valuable experiments run where real users are affected — that's where the real risk is. (But start in staging.)

4. Minimize blast radius. Start small. One instance, one region, one service. Expand scope as confidence grows.

What Chaos Monkey Actually Does

Despite its reputation as a chaos engineering mascot, Netflix Chaos Monkey is narrow in scope. It does one thing: randomly terminate EC2 instances in an Auto Scaling Group.

The premise: if your application auto-recovers from a terminated instance (because it's distributed, stateless, and behind a load balancer), you've proven resilience to that specific failure mode.

Chaos Monkey doesn't:

• Inject network latency
• Corrupt data
• Fill disks
• Simulate dependency failures
• Kill databases

For those scenarios, Netflix built other tools (Chaos Kong, Latency Monkey, Doctor Monkey) — collectively the Simian Army.

Getting Started with Chaos Monkey

Prerequisites

Before running any chaos experiment:

1. You have load balancing. If all traffic goes to one instance, killing it means total outage. You need multiple instances behind a load balancer.
2. You have auto-scaling. Chaos Monkey kills instances. Your system should automatically replace them.
3. You have observability. Metrics, tracing, and dashboards to measure impact.
4. You have a runbook. What do operators do if a chaos experiment causes an actual outage? Write it before you need it.

Installation

Chaos Monkey is open-source. The modern version runs as a service:

# Clone the repo
git clone https://github.com/Netflix/chaosmonkey.git

# Build
cd chaosmonkey
go build ./...

Chaos Monkey requires Spinnaker (Netflix's deployment platform) to track application instances. For teams not using Spinnaker, alternatives like Chaos Toolkit or Gremlin are more practical starting points.

A Minimal Chaos Monkey Configuration

# chaosmonkey.toml
[chaosmonkey]
enabled = true
schedule_enabled = true

# Only run during business hours
[chaosmonkey.schedule]
weekdays = "Mon-Fri"
hours = "9-17"

# Maximum instances to kill per group per day
[chaosmonkey.groups]
max_kill = 1

Key settings:

• enabled = false for staging environments when you first start. Validate behavior before enabling in production.
• Time windows: Restrict chaos experiments to business hours so your team is available to respond.
• max_kill = 1: Never kill more than one instance per group per run when starting.

Your First Chaos Experiment

A chaos experiment follows a structured format:

Step 1: Define the Steady State

Before introducing any failure, observe and document the baseline:

Steady state definition:
- API success rate: ≥ 99.9% over 5 minutes
- p99 request latency: ≤ 200ms
- Error rate: ≤ 0.1%

Use your monitoring tool (Datadog, Prometheus, CloudWatch) to confirm these metrics are currently met.

Step 2: Formulate a Hypothesis

Hypothesis:
Terminating one application instance in the web tier will not
affect the steady-state API success rate or latency, because
requests will be automatically routed to the remaining instances
by the load balancer within 30 seconds.

Step 3: Define the Abort Condition

Abort if:
- Error rate exceeds 5% for more than 2 minutes
- Any downstream service reports elevated error rates
- On-call engineer requests stop

Write the abort condition before starting. During an experiment, pressure to continue is real. The abort condition removes the decision.

Step 4: Run the Experiment

Using Chaos Toolkit as a simpler starting point:

{
  "title": "Terminate one instance from web tier",
  "description": "Verify that the web tier survives single instance failure",
  "steady-states": {
    "before": {
      "title": "API is healthy",
      "probes": [
        {
          "type": "probe",
          "name": "api-success-rate",
          "tolerance": 0.999,
          "provider": {
            "type": "http",
            "url": "http://monitoring.internal/api/success-rate"
          }
        }
      ]
    }
  },
  "method": [
    {
      "type": "action",
      "name": "terminate-instance",
      "provider": {
        "type": "python",
        "module": "chaosaws.ec2.actions",
        "func": "terminate_instances",
        "arguments": {
          "filters": [{"Name": "tag:app", "Values": ["web-tier"]}],
          "az": "us-east-1a"
        }
      }
    }
  ]
}

Step 5: Observe and Learn

Watch your dashboards. Did the steady state hold? Did error rates spike? Did the load balancer route around the failure as expected?

Document results whether the experiment succeeds or reveals a gap:

• Hypothesis confirmed: System is resilient to this failure mode. Schedule next experiment at wider scope.
• Hypothesis refuted: Document the gap, create a remediation task, fix it, re-run.

Common First Experiments

Start with the simplest — terminate an instance — and work toward more sophisticated experiments as your team's confidence grows.

The Learning Loop

Chaos engineering is valuable when it's a loop, not a one-time event:

1. Observe a weakness in the system
2. Hypothesize how the system should behave under that failure
3. Design an experiment to test the hypothesis
4. Run it, observe results
5. Fix weaknesses discovered
6. Expand scope of the next experiment

Teams that run chaos experiments monthly find and fix weaknesses continuously. Teams that run a one-time "chaos day" find problems they don't have time to fix.

Start small, be structured, and treat every confirmed weakness as a gift: you found it before your users did.