Chaos Engineering with Chaos Monkey and Litmus: A Practical Guide

19 Aug 2026 — 7 min read

Chaos engineering is the practice of deliberately introducing failure into a running system to build confidence that it can withstand turbulent conditions. Netflix coined the term and open-sourced the original tools. The ideas have since matured into a disciplined engineering practice with clear methodology, tooling ecosystems, and measurable outcomes.

This post covers the foundational concepts, how to use Chaos Monkey and Litmus in practice, and — critically — how to run chaos experiments as structured tests rather than random destruction.

The Core Idea: Break It Before It Breaks You

Every system has failure modes. The question is not whether they'll occur, but whether you'll discover them in a controlled experiment or in a 3am incident.

Chaos engineering gives you a methodology for discovering those failure modes proactively:

Define steady-state behavior (what does "healthy" look like, measurably?).
Hypothesize that the system will maintain steady-state despite the failure you're about to introduce.
Introduce the failure in a controlled way.
Look for deviations from steady-state.
Fix what you find.

This is scientific method applied to distributed systems. The hypothesis-driven structure is what separates chaos engineering from just running chaos for its own sake.

Chaos Monkey: Process-Level Chaos at Scale

Netflix's Chaos Monkey was the original chaos tool. It randomly terminates virtual machine instances in production. The goal is to ensure that the loss of any single instance doesn't cause a visible service degradation.

Modern Chaos Monkey (Spinnaker-integrated) targets EC2 instances and can be configured to:

Target specific Auto Scaling Groups
Apply a frequency (how often to kill instances)
Operate during a configurable schedule (business hours only, for example)
Exclude instances with specific tags

The key insight from Netflix's experience: if you run Chaos Monkey every weekday, engineers will quickly fix everything that breaks. The pain of a Chaos Monkey termination during work hours is much less than the pain of an unexpected termination at 2am, so teams are motivated to build resilience.

Running Chaos Monkey

Chaos Monkey runs as a service and integrates with Spinnaker for deployment pipeline awareness. Basic configuration:

# spinnaker-chaos-monkey.yml
simianarmy:
  chaos:
    enabled: true
    leashed: false
    asgs:
      enabled: true
    frequency: 1  # terminations per hour per ASG
    schedule:
      start: 9  # 9am
      end: 17   # 5pm
    exceptions:
      - account: production
        region: us-east-1
        stack: database  # don't kill database ASGs

For teams not on AWS or Spinnaker, the conceptual equivalent in Kubernetes is pod deletion — which is what Litmus covers.

LitmusChaos: Kubernetes-Native Chaos Engineering

Litmus (now a CNCF project) is the most mature chaos engineering framework for Kubernetes. It provides:

A library of pre-built chaos experiments (pod deletion, node drain, CPU hog, memory hog, network latency, disk fill, etc.)
CRDs for defining and scheduling experiments
A Chaos Center UI for managing and observing experiments
Integration hooks for observability platforms

Installing Litmus

kubectl apply -f https://litmuschaos.github.io/litmus/litmus-operator-v3.0.0.yaml

# Verify the operator is running
kubectl get pods -n litmus

# Install experiment CRDs
kubectl apply -f https://hub.litmuschaos.io/api/chaos/3.0.0?file=charts/generic/experiments.yaml

Your First Experiment: Pod Delete

The most fundamental chaos experiment is pod deletion. It tests whether your deployment is truly resilient to the loss of a pod instance.

Create a service account with the necessary RBAC:

# rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: pod-delete-sa
  namespace: default
  labels:
    name: pod-delete-sa
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-delete-sa
  namespace: default
  labels:
    name: pod-delete-sa
rules:
  - apiGroups: [""]
    resources: ["pods", "events"]
    verbs: ["create", "list", "get", "patch", "update", "delete", "deletecollection"]
  - apiGroups: [""]
    resources: ["pods/exec", "pods/log", "replicationcontrollers"]
    verbs: ["create", "list", "get"]
  - apiGroups: ["apps"]
    resources: ["deployments", "statefulsets", "replicasets", "daemonsets"]
    verbs: ["list", "get"]
  - apiGroups: ["batch"]
    resources: ["jobs"]
    verbs: ["create", "list", "get", "delete", "deletecollection"]
  - apiGroups: ["litmuschaos.io"]
    resources: ["chaosengines", "chaosexperiments", "chaosresults"]
    verbs: ["create", "list", "get", "patch", "update", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: pod-delete-sa
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: pod-delete-sa
subjects:
  - kind: ServiceAccount
    name: pod-delete-sa
    namespace: default

Define the ChaosEngine:

# chaos-engine-pod-delete.yaml
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: order-service-pod-delete
  namespace: default
spec:
  appinfo:
    appns: default
    applabel: "app=order-service"
    appkind: deployment
  chaosServiceAccount: pod-delete-sa
  monitoring: true
  jobCleanUpPolicy: retain
  experiments:
    - name: pod-delete
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: "60"         # Run chaos for 60 seconds
            - name: CHAOS_INTERVAL
              value: "10"         # Delete a pod every 10 seconds
            - name: FORCE
              value: "false"      # Graceful deletion
            - name: PODS_AFFECTED_PERC
              value: "50"         # Affect 50% of matching pods

Apply and watch:

kubectl apply -f rbac.yaml
kubectl apply -f chaos-engine-pod-delete.yaml

# Watch the experiment progress
kubectl describe chaosengine order-service-pod-delete

# Check results
kubectl describe chaosresult order-service-pod-delete-pod-delete

Defining Steady State with Probes

Running chaos without measuring impact is noise. Litmus probes let you define what "healthy" means and validate it before, during, and after the experiment:

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: order-service-pod-delete
  namespace: default
spec:
  appinfo:
    appns: default
    applabel: "app=order-service"
    appkind: deployment
  chaosServiceAccount: pod-delete-sa
  experiments:
    - name: pod-delete
      spec:
        probe:
          - name: check-order-api-availability
            type: httpProbe
            httpProbe/inputs:
              url: "http://order-service/health"
              insecureSkipVerify: false
              responseTimeout: 500  # ms
              method:
                get:
                  criteria: "=="
                  responseCode: "200"
            mode: Continuous
            runProperties:
              probeTimeout: 1s
              interval: 2s
              retry: 1
              probePollingInterval: 1s

          - name: check-error-rate
            type: promProbe
            promProbe/inputs:
              endpoint: "http://prometheus:9090"
              query: "sum(rate(http_requests_total{service='order-service',status=~'5..'}[1m]))"
              comparator:
                type: float
                criteria: "<="
                value: "0.01"  # Less than 1% error rate
            mode: Continuous
            runProperties:
              probeTimeout: 5s
              interval: 10s
              retry: 2

Now the experiment succeeds only if the order service health endpoint keeps returning 200 throughout the chaos, and the error rate stays below 1%. If pods are being deleted and the service degrades, the probes catch it and mark the experiment as failed.

Node-Level Chaos: Node Drain and Node CPU Hog

Pod deletion tests pod-level resilience. Node-level experiments test whether your workloads survive losing an entire node:

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: node-drain-test
spec:
  chaosServiceAccount: node-chaos-sa
  experiments:
    - name: node-drain
      spec:
        components:
          env:
            - name: TARGET_NODE
              value: "worker-node-02"
            - name: TOTAL_CHAOS_DURATION
              value: "120"
            - name: REINJECTION_COUNT
              value: "0"

CPU hog is useful for testing behavior under resource pressure:

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: cpu-hog-test
spec:
  appinfo:
    appns: default
    applabel: "app=payment-service"
  experiments:
    - name: pod-cpu-hog
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: "60"
            - name: CPU_CORES
              value: "1"        # Consume 1 CPU core
            - name: PODS_AFFECTED_PERC
              value: "100"      # All matching pods

Network Chaos: Latency and Packet Loss

Network experiments are often the most revealing. Real-world failures frequently involve degraded network conditions rather than complete outages:

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: network-latency-test
spec:
  appinfo:
    appns: default
    applabel: "app=payment-service"
  experiments:
    - name: pod-network-latency
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: "60"
            - name: NETWORK_INTERFACE
              value: "eth0"
            - name: LATENCY
              value: "200"      # 200ms added latency
            - name: JITTER
              value: "50"       # ±50ms jitter
            - name: DESTINATION_IPS
              value: "10.0.0.0/8"  # Target internal traffic

Automating Chaos Experiments in CI

Random manual chaos experiments are valuable for discovery. Automated chaos experiments in CI are valuable for regression prevention. Here's a Go test harness for running Litmus experiments programmatically:

package chaos_test

import (
    "context"
    "encoding/json"
    "fmt"
    "testing"
    "time"

    litmuschaos "github.com/litmuschaos/chaos-operator/pkg/client/clientset/versioned"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

func TestOrderServiceSurvivesPodDeletion(t *testing.T) {
    ctx := context.Background()
    client := buildLitmusClient(t)

    // Apply the ChaosEngine
    engine := buildPodDeleteEngine("order-service", "app=order-service", 60)
    _, err := client.LitmuschaosV1alpha1().ChaosEngines("default").Create(ctx, engine, metav1.CreateOptions{})
    if err != nil {
        t.Fatalf("failed to create chaos engine: %v", err)
    }
    t.Cleanup(func() {
        client.LitmuschaosV1alpha1().ChaosEngines("default").Delete(ctx, engine.Name, metav1.DeleteOptions{})
        client.LitmuschaosV1alpha1().ChaosResults("default").Delete(ctx, engine.Name+"-pod-delete", metav1.DeleteOptions{})
    })

    // Poll for completion
    deadline := time.Now().Add(3 * time.Minute)
    for time.Now().Before(deadline) {
        result, err := client.LitmuschaosV1alpha1().ChaosResults("default").Get(
            ctx, engine.Name+"-pod-delete", metav1.GetOptions{},
        )
        if err != nil {
            time.Sleep(5 * time.Second)
            continue
        }

        verdict := result.Status.ExperimentStatus.Verdict
        t.Logf("Chaos experiment verdict: %s", verdict)

        switch verdict {
        case "Pass":
            t.Log("Chaos experiment passed — service maintained steady state under pod deletion")
            return
        case "Fail":
            failDetails, _ := json.MarshalIndent(result.Status, "", "  ")
            t.Fatalf("Chaos experiment failed:\n%s", failDetails)
        case "Stopped":
            t.Fatal("Chaos experiment was stopped unexpectedly")
        }

        time.Sleep(5 * time.Second)
    }

    t.Fatal("chaos experiment did not complete within timeout")
}

Measuring Blast Radius Before Running Experiments

Before applying any chaos, understand what's affected. Litmus's Chaos Center provides a dependency map, but you can also query this programmatically:

# Check how many pods would be affected by a 50% deletion
kubectl get pods -l app=order-service --no-headers | wc -l
# If you only have 2 pods, deleting 50% means deleting 1 — is that acceptable?

# Check if PodDisruptionBudgets would prevent the experiment
kubectl get pdb
# A PDB with minAvailable:2 on a 2-replica deployment means pod deletion will be blocked

Running an experiment against a deployment that can't tolerate any disruption won't teach you about resilience — it'll just break things. The pre-experiment checklist matters.

Chaos Experiment Runbook Template

Good chaos experiments are documented. Here's a template:

# Experiment: Order Service Pod Deletion

## Hypothesis
The order-service will maintain >99% availability and <500ms P99 latency
even when 50% of its pods are deleted every 10 seconds.

## Scope
- Target: order-service deployment, default namespace
- Duration: 60 seconds
- Blast radius: ~50% of order-service pods

## Steady State
- HTTP 200 from /health endpoint with <100ms response
- Error rate <1% (from Prometheus)
- P99 latency <200ms

## Pre-Conditions
- Deployment has ≥4 replicas
- PodDisruptionBudget allows disruption
- HPA is not scaling down below 4 replicas during test window

## Expected Outcome
Pass — probes stay green throughout

## Rollback
Delete ChaosEngine resource if experiment hangs or environment degrades

## Date Last Run
2026-05-01 — Pass (4/4 probes green, error rate peaked at 0.3%)

Common Findings and What They Mean

Running chaos experiments surfaces predictable classes of bugs:

Insufficient replicas: Single-replica deployments fail immediately on pod deletion. The fix is replicas ≥ 2 and a PodDisruptionBudget.

Missing health checks: If a pod restarts and your load balancer doesn't wait for it to be healthy, traffic hits it before it's ready. The fix is properly configured readiness probes.

No circuit breakers: When a service is slow but not dead, callers pile up waiting, exhausting connection pools and cascading failures. The fix is circuit breakers with fallback behavior.

Hardcoded timeouts: Timeouts that are too short cause failures on normal latency spikes. Timeouts that are too long cause cascading failures when upstream services degrade. The fix is explicit timeout configuration and testing under injected latency.

Database connection pool exhaustion: During pod restarts, connection pools don't drain cleanly. The fix is proper connection pool lifecycle management and retry logic.

Wrapping Up

Chaos engineering with Chaos Monkey and Litmus is not about creating chaos for its own sake. It's about running structured experiments with clear hypotheses, measurable steady-state definitions, and actionable findings.

The teams that do this well treat chaos experiments the same way they treat test suites: they run them regularly, they track results over time, and they fix what they find before it becomes an incident. The teams that do this poorly run a chaos experiment once during a hackathon, call it "chaos engineering done," and go back to being surprised by failures in production.

The difference is structure. Define your steady state. State your hypothesis. Inject the failure. Measure the deviation. Fix the weakness. Repeat.