Chaos Monkey: How Netflix's Resilience Tool Works and How to Use It

The Origin Story

In 2010, Netflix was migrating from their own datacenter to AWS. During the migration, they encountered a problem: their engineers were building services that assumed the infrastructure was reliable. When AWS had outages (which happened), Netflix would go down.

The solution they landed on: force the problem to become part of the normal operating environment. If production servers are randomly terminated during business hours, engineers have no choice but to build services that tolerate instance failure. A service that can't survive losing one instance isn't ready for production.

They called the tool Chaos Monkey — a metaphor for a monkey in your data center randomly breaking things. Released as open source in 2012, it became the foundational tool for chaos engineering.

What Chaos Monkey Does

Chaos Monkey randomly selects instances from your production Auto Scaling Groups (ASGs) and terminates them. That's it.

The effect:

• Engineers learn immediately if a service has a single point of failure
• Load balancers learn to route around failed instances (if configured correctly)
• Auto Scaling automatically replaces terminated instances
• Alert systems fire if recovery doesn't happen fast enough

What it does NOT do:

• Introduce network latency
• Fill disks
• Simulate AZ outages
• Test application-level failures

For those scenarios, you need the broader Simian Army tools (now largely replaced by Chaos Mesh, Gremlin, and LitmusChaos).

The Simian Army

Netflix expanded Chaos Monkey into a suite of tools called the Simian Army:

Most of the Simian Army tools have been replaced by modern equivalents or folded into Netflix's internal tooling. The open-source Chaos Monkey you can run yourself is specifically the instance-termination tool.

Installing Chaos Monkey (Open Source)

The open-source Chaos Monkey requires:

• AWS account with EC2 Auto Scaling Groups
• Spinnaker (Netflix's deployment platform) for scheduling
• MySQL or PostgreSQL for state storage

Prerequisites

# Install Go (Chaos Monkey is written in Go)
brew install go  # macOS

# Install MySQL
brew install mysql
mysql -u root -e "CREATE DATABASE chaos_monkey; CREATE USER 'monkey'@'localhost' IDENTIFIED BY 'password'; GRANT ALL ON chaos_monkey.* TO 'monkey'@'localhost';"

Clone and Build

git clone git@github.com:Netflix/chaosmonkey.git
cd chaosmonkey
go build

Configuration

Create chaosmonkey.toml:

[chaosmonkey]
enabled = true
schedule_enabled = true
leashed = false
accounts = ["my-aws-account"]

[database]
host = "localhost"
name = "chaos_monkey"
user = "monkey"
password = "password"
port = 3306
encrypt_encrypted = false

[spinnaker]
endpoint = "https://spinnaker.yourcompany.com"
x509certfile = "/path/to/cert.pem"
x509keyfile = "/path/to/key.pem"

[dynamic_property_provider]
endpoint = ""

[outage]
skip_days = ["Saturday", "Sunday"]
start_hour = 9
end_hour = 17

AWS IAM Requirements

Chaos Monkey needs permission to describe and terminate instances:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "autoscaling:DescribeAutoScalingGroups",
        "autoscaling:DescribeAutoScalingInstances",
        "ec2:TerminateInstances"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "ses:SendEmail"
      ],
      "Resource": "*"
    }
  ]
}

Running

# Start the HTTP server (for Spinnaker integration)
./chaosmonkey serve

# Manually trigger termination (for testing)
./chaosmonkey terminate --account my-account --region us-east-1 --stack my-stack

Configuring Chaos Monkey Behavior

Grouping — Chaos Monkey terminates one instance per "group" per day. By default, a group is a Spinnaker cluster (an ASG). You can configure:

[chaosmonkey.groups]
enabled = true
# Terminate at most 1 instance per group per day

Opt-out — Tag specific ASGs to be excluded:

Key: chaosmonkey.enabled
Value: false

Mean Time Between Terminations (MTBT) — How often each group gets hit:

[chaosmonkey]
mean_time_between_kills_in_working_hours = 5  # Average 5 days between kills

Leash mode — Run in "leashed" mode to log what would happen without actually terminating:

[chaosmonkey]
leashed = true

Use leashed mode for the first week to see what Chaos Monkey would do before enabling terminations.

Chaos Monkey Without Spinnaker

The official Chaos Monkey requires Spinnaker, which is a significant dependency. Alternatives for the same "kill random EC2 instance" behavior:

AWS-native approach with Lambda:

import boto3
import random

def lambda_handler(event, context):
    ec2 = boto3.client('ec2', region_name='us-east-1')
    autoscaling = boto3.client('autoscaling', region_name='us-east-1')
    
    # Get all ASGs
    asgs = autoscaling.describe_auto_scaling_groups(
        Filters=[{'Name': 'tag:chaos-monkey-enabled', 'Values': ['true']}]
    )
    
    if not asgs['AutoScalingGroups']:
        return {'message': 'No eligible ASGs found'}
    
    # Pick a random ASG
    asg = random.choice(asgs['AutoScalingGroups'])
    instances = [i['InstanceId'] for i in asg['Instances'] if i['LifecycleState'] == 'InService']
    
    if len(instances) < 2:
        return {'message': f'Skipping {asg["AutoScalingGroupName"]}: only {len(instances)} instance(s)'}
    
    # Terminate a random instance
    victim = random.choice(instances)
    ec2.terminate_instances(InstanceIds=[victim])
    
    return {
        'asg': asg['AutoScalingGroupName'],
        'terminated': victim
    }

Schedule with EventBridge:

aws events put-rule \
  --name "chaos-monkey" \
  --schedule-expression "cron(0 10-16 ? * MON-FRI *)"  # Weekdays 10am-4pm

aws events put-targets \
  --rule chaos-monkey \
  --targets "Id=chaos-lambda,Arn=arn:aws:lambda:us-east-1:123456789:function:chaos-monkey"

Chaos Monkey Equivalents for Kubernetes

Chaos Monkey is EC2-specific. For Kubernetes environments, use:

Chaos Mesh — CNCF project, pod/container termination:

apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
  name: pod-kill-example
spec:
  action: pod-kill
  mode: one
  selector:
    namespaces:
      - production
    labelSelectors:
      app: my-service
  scheduler:
    cron: "@every 5m"

LitmusChaos — comprehensive Kubernetes chaos (covered in its own post)

kube-monkey — Chaos Monkey clone for Kubernetes:

# kube-monkey config
apiVersion: v1
kind: ConfigMap
metadata:
  name: kube-monkey-config
data:
  config.toml: |
    [kubemonkey]
    dry_run = false
    run_hour = 8
    start_grace_period_sec = 100
    blacklisted_namespaces = ["kube-system"]
    whitelisted_namespaces = ["production"]

Opt-in pods with labels:

metadata:
  labels:
    kube-monkey/enabled: "enabled"
    kube-monkey/identifier: "my-service"
    kube-monkey/mtbf: "1"  # Mean time between failures (days)
    kube-monkey/kill-mode: "fixed"
    kube-monkey/kill-value: "1"

What Chaos Monkey Teaches You

After running Chaos Monkey for a few weeks, you'll discover:

Services with single instance deployments — a terminated instance takes the service down. The fix: run at least 2 instances, use health checks, configure your load balancer properly.

Services without health checks — the load balancer keeps routing to terminated instances. The fix: implement /health endpoints, configure LB health checks, set deregistration delays.

Services that don't recover automatically — Auto Scaling doesn't bring new instances up, or the new instance fails its health check. The fix: fix your launch configuration, validate your bootstrap scripts.

Alert gaps — an instance gets terminated, users experience errors for 2 minutes before an alert fires. The fix: tighten your alert thresholds or improve error detection.

Dependent services that don't handle upstream failures — Service B calls Service A, Service A's instance gets terminated, Service B starts returning 500s instead of gracefully degrading. The fix: circuit breakers, timeouts, fallback behavior.

The Cultural Impact of Chaos Monkey

The most important outcome of Chaos Monkey at Netflix wasn't technical — it was cultural. Engineers started building systems with the explicit assumption that any server could disappear at any time. This led to:

• Stateless service design (no local state that gets lost on termination)
• External state management (Redis, databases) instead of in-memory caching
• Graceful shutdown handling (drain connections before termination)
• Automated recovery as a baseline expectation, not an afterthought

Teams who run Chaos Monkey long enough stop thinking about instance termination as an exceptional event — it becomes a normal operating condition that their architecture handles automatically.