Chaos Engineering for Microservices with Chaos Monkey and Litmus

HelpMeTest

19 Aug 2026 — 6 min read

Your microservices work in staging. They fall over in production. Chaos engineering is the practice of intentionally injecting failures to find weaknesses before users do. This guide covers two tools — Netflix Chaos Monkey (for instance/pod termination) and LitmusChaos (for Kubernetes-native chaos) — with a focus on meaningful experiments, not random destruction.

What Chaos Engineering Is (and Isn't)

Chaos engineering is not: randomly breaking things and hoping nothing catches fire.

Chaos engineering is: hypothesizing that your system is resilient to a specific failure, then proving it with a controlled experiment.

The process:

Define steady state (what "normal" looks like in metrics)
Hypothesize: "The system remains in steady state if pod X is killed"
Inject the failure in a controlled environment
Observe: does the system stay in steady state?
Fix if it doesn't; document if it does

Without this discipline, chaos engineering is just a way to create incidents.

Chaos Monkey: Pod and Instance Termination

Netflix's Chaos Monkey was originally designed to randomly terminate EC2 instances. The modern Kubernetes equivalent terminates pods.

Installing Chaos Monkey for Kubernetes

The official kube-monkey is a Kubernetes adaptation:

# Install with Helm
helm repo add kube-monkey https://asobti.github.io/kube-monkey/charts/repo
helm install kube-monkey kube-monkey/kube-monkey \
  --namespace kube-monkey \
  --create-namespace \
  --set config.dryRun=true  # Start with dry run

Or use the Chaos Monkey for Spring Boot (for Java services):

<!-- pom.xml -->
<dependency>
  <groupId>de.codecentric</groupId>
  <artifactId>chaos-monkey-spring-boot</artifactId>
  <version>3.1.0</version>
</dependency>

# application-chaos.yml
chaos:
  monkey:
    enabled: true
    watcher:
      rest-controller: true
      service: true
    assaults:
      level: 5
      latency-active: true
      latency-range-start: 1000
      latency-range-end: 3000
      exceptions-active: true
      kill-application-active: false  # Enable carefully

Configuring kube-monkey Experiments

Enable kube-monkey for specific deployments with annotations:

# Your service deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-service
  namespace: production
  labels:
    kube-monkey/enabled: "enabled"
    kube-monkey/identifier: "order-service"
    kube-monkey/mtbf: "2"          # Mean time between failures: 2 hours
    kube-monkey/kill-value: "1"    # Kill 1 pod per attack
    kube-monkey/kill-mode: "fixed" # fixed | random-max-percent
spec:
  replicas: 3  # Must have replicas > 1 for chaos to be safe
  # ...

kube-monkey attacks pods that opt in via labels, leaving others untouched.

Validating Pod Termination Resilience

Before enabling kube-monkey, manually test pod termination:

#!/bin/bash
# test-pod-resilience.sh

SERVICE="order-service"
NAMESPACE="production"

echo "=== Pod Resilience Test: $SERVICE ==="

# Baseline: verify service is healthy
echo "Step 1: Verify baseline health"
INITIAL_STATUS=$(curl -s http://order-service.production.svc/health | jq -r '.status')
[ "$INITIAL_STATUS" = "healthy" ] || { echo "FAIL: Baseline unhealthy"; exit 1; }

# Record initial pod count
INITIAL_PODS=$(kubectl get pods -n $NAMESPACE -l app=$SERVICE --no-headers | wc -l)
echo "Initial pod count: $INITIAL_PODS"

# Kill one pod
echo "Step 2: Kill one pod"
VICTIM=$(kubectl get pods -n $NAMESPACE -l app=$SERVICE -o name | head -1)
kubectl delete $VICTIM -n $NAMESPACE
echo "Killed: $VICTIM"

# Monitor recovery
echo "Step 3: Monitor recovery"
RECOVERED=false
for i in $(seq 1 30); do
  sleep 2
  
  RUNNING=$(kubectl get pods -n $NAMESPACE -l app=$SERVICE --field-selector=status.phase=Running --no-headers | wc -l)
  echo "  t=${i}s: Running pods: $RUNNING"
  
  if [ "$RUNNING" -ge "$INITIAL_PODS" ]; then
    RECOVERED=true
    echo "  Recovered to $RUNNING pods after ${i}s"
    break
  fi
done

[ "$RECOVERED" = "true" ] || { echo "FAIL: Did not recover within 60s"; exit 1; }

# Verify service still works during and after recovery
echo "Step 4: Verify service availability during recovery"
ERRORS=0
for i in $(seq 1 20); do
  STATUS=$(curl -s -o /dev/null -w "%{http_code}" http://order-service.production.svc/health)
  [ "$STATUS" != "200" ] && ((ERRORS++))
  sleep 1
done

echo "Errors during recovery: $ERRORS/20"

if [ $ERRORS -gt 2 ]; then
  echo "FAIL: Service unavailable during pod recovery ($ERRORS/20 requests failed)"
  exit 1
fi

echo "PASS: Service resilient to pod termination"

LitmusChaos: Kubernetes-Native Chaos Engineering

LitmusChaos provides a rich library of chaos experiments as CRDs. It goes beyond pod termination to include:

Pod CPU/memory stress
Node drain and shutdown
Network packet loss and latency
Disk I/O throttling
Container kill
DNS errors

Installing LitmusChaos

# Install LitmusChaos CRDs and operator
kubectl apply -f https://litmuschaos.github.io/litmus/litmus-operator-v3.0.0.yaml

# Install chaos experiment library
kubectl apply -f https://hub.litmuschaos.io/api/chaos?file=charts/generic/experiments.yaml -n litmus

# Verify
kubectl get chaosexperiments -n litmus

Experiment 1: Pod Network Latency

Test that your service handles high latency on dependencies:

# network-latency-experiment.yaml
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: order-service-network-latency
  namespace: production
spec:
  appinfo:
    appns: production
    applabel: "app=payment-service"  # Inject latency into payment-service
    appkind: deployment
  
  chaosServiceAccount: litmus-admin
  
  experiments:
    - name: pod-network-latency
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: "60"     # seconds
            - name: NETWORK_LATENCY
              value: "2000"   # 2000ms latency
            - name: JITTER
              value: "500"    # ±500ms jitter
            - name: CONTAINER_RUNTIME
              value: "containerd"
        probe:
          - name: verify-order-service-degrades-gracefully
            type: httpProbe
            mode: Continuous
            httpProbe/inputs:
              url: http://order-service:8080/health
              insecureSkipVerify: false
              responseTimeout: 10000  # 10s timeout (should succeed despite 2s latency)
              method:
                get:
                  criteria: "=="
                  responseCode: "200"
            runProperties:
              probeTimeout: 15
              interval: 5
              attempt: 3
              stopOnFailure: false

Apply and monitor:

kubectl apply -f network-latency-experiment.yaml

# Watch experiment status
kubectl get chaosresult order-service-network-latency-pod-network-latency -n production -w

# Check probe results
kubectl describe chaosresult order-service-network-latency-pod-network-latency -n production

Experiment 2: Pod CPU Stress

Validate behavior under CPU exhaustion:

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: inventory-service-cpu-stress
  namespace: production
spec:
  appinfo:
    appns: production
    applabel: "app=inventory-service"
    appkind: deployment
  
  experiments:
    - name: pod-cpu-hog
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: "60"
            - name: CPU_CORES
              value: "2"       # Consume 2 CPU cores
            - name: CPU_LOAD
              value: "100"     # 100% load
        probe:
          - name: verify-inventory-api-still-responds
            type: httpProbe
            mode: Continuous
            httpProbe/inputs:
              url: http://inventory-service:8080/stock/prod-001
              responseTimeout: 5000
              method:
                get:
                  criteria: "=="
                  responseCode: "200"
            runProperties:
              probeTimeout: 10
              interval: 5
              attempt: 3
              stopOnFailure: false

Experiment 3: Node Drain

Validate pod rescheduling across node failures:

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: node-drain-experiment
  namespace: litmus
spec:
  jobCleanUpPolicy: "retain"
  
  experiments:
    - name: node-drain
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: "120"
            - name: TARGET_NODE
              value: "worker-node-2"  # The node to drain
            - name: NODE_LABEL
              value: ""  # Empty to use TARGET_NODE explicitly
        probe:
          - name: verify-services-on-other-nodes
            type: cmdProbe
            mode: Continuous
            cmdProbe/inputs:
              command: >
                kubectl get pods -n production --field-selector=status.phase=Running 
                -l app=order-service --no-headers | wc -l
              comparator:
                type: int
                criteria: ">="
                value: "2"  # At least 2 pods running
            runProperties:
              probeTimeout: 30
              interval: 10
              attempt: 3

Experiment 4: DNS Errors

Test behavior when DNS resolution fails (service discovery breaks):

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: payment-service-dns-error
  namespace: production
spec:
  appinfo:
    appns: production
    applabel: "app=order-service"
    appkind: deployment
  
  experiments:
    - name: pod-dns-error
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: "60"
            - name: TARGET_HOSTNAMES
              value: "payment-service"  # Break DNS for this hostname
        probe:
          - name: verify-order-service-handles-dns-failure
            type: httpProbe
            mode: SOT  # Start of Test
            httpProbe/inputs:
              url: http://order-service:8080/orders/health-check
              responseTimeout: 5000
              method:
                get:
                  criteria: "=="
                  responseCode: "503"  # Expect graceful degradation, not crash
            runProperties:
              probeTimeout: 10
              interval: 2
              attempt: 5

Designing Meaningful Chaos Experiments

The experiment design matters as much as the tooling. For each experiment:

1. Define Steady State Metrics

Before injecting chaos, capture your baseline:

# Record key metrics before experiment
echo "=== Pre-chaos metrics ==="
echo "Error rate: $(promtool query 'rate(http_requests_total{status=~"5.."}[1m])' | tail -1)"
echo "p99 latency: $(promtool query 'histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[1m]))' | tail -1)"
echo "Pod count: $(kubectl get pods -n production -l app=order-service --no-headers | grep Running | wc -l)"

2. Write Your Hypothesis

Document before running:

## Experiment: Pod Deletion - Order Service

**Steady state**: 
- Error rate < 0.1%
- p99 latency < 500ms
- All pods Running

**Hypothesis**: If one of three order-service pods is killed, 
Kubernetes will reschedule it within 60 seconds, and the error 
rate will not exceed 1% during recovery.

**Variables**:
- Independent: Killing one pod
- Dependent: Error rate, latency, recovery time

**Abort condition**: If error rate exceeds 5% during experiment

3. Set Abort Conditions

Always define when to stop the experiment:

# ChaosEngine with abort threshold
spec:
  components:
    statusCheckTimeouts:
      delay: 2
      timeout: 180
  # If probes fail consistently, experiment aborts automatically
  experiments:
    - name: pod-delete
      spec:
        probe:
          - name: error-rate-guard
            type: promProbe
            mode: Continuous
            promProbe/inputs:
              endpoint: http://prometheus:9090
              query: >
                rate(http_requests_total{app="order-service",status=~"5.."}[1m]) 
                / rate(http_requests_total{app="order-service"}[1m])
              comparator:
                type: float
                criteria: "<="
                value: "0.05"  # Abort if error rate exceeds 5%
            runProperties:
              probeTimeout: 10
              interval: 5
              attempt: 3
              stopOnFailure: true  # Abort chaos if this fails

Building a Chaos Engineering Maturity Model

Start small and expand:

Level 1 — Pod termination in staging:

Kill individual pods, verify auto-recovery
Validate that readiness probes work correctly

Level 2 — Dependency failures:

Inject latency and errors into service dependencies
Validate circuit breakers and fallbacks

Level 3 — Infrastructure failures:

Node drain and restart
Network partition between availability zones

Level 4 — Production chaos (with caution):

Start with low-traffic periods
Use feature flags to limit blast radius
Have an immediate rollback plan

Common Findings from Chaos Experiments

Teams running their first chaos experiments typically discover:

Readiness probes missing or too lenient: Pods receive traffic before they're ready
No circuit breakers: One slow service cascades into full system failure
Resource limits not set: CPU-starved pods crash under stress
Single replica deployments: One pod death = zero availability
Hardcoded timeouts too long: 30-second timeouts cause cascading delays
No graceful shutdown: Pods killed mid-request lose in-flight transactions

Each finding is a bug you'd rather find in a chaos experiment than in production at 2am.

Key Takeaways

Chaos engineering requires hypothesis + measurement, not random destruction
Start with kube-monkey or manual pod deletion before using more complex tools
Use LitmusChaos for Kubernetes-native experiments: network latency, CPU stress, node drain
Always define abort conditions — stop the experiment if things get worse than expected
Probes are mandatory — an experiment without assertions is just breaking things
Build maturity progressively: staging first, low-traffic production windows second, continuous chaos last