Testing

Chaos Engineering for SLOs: Testing Resilience Without Burning Error Budget

HelpMeTest

HelpMeTest

— 6 min read
Chaos Engineering for SLOs: Testing Resilience Without Burning Error Budget

Chaos engineering and SLO management seem to be in conflict. SLOs protect reliability commitments. Chaos engineering deliberately breaks things. How do you test resilience without burning your error budget and violating your SLAs?

The answer is planning — scoping experiments to stay within budget, baselining before you inject faults, and defining abort conditions before you start. Done right, chaos experiments give you proof that your system tolerates real failure modes. Done wrong, they become production incidents.

This post covers how to design and run chaos experiments that validate SLO resilience without causing the SLO violations you're trying to prevent.

Chaos Testing vs. Chaos Engineering

Two distinct practices often get conflated:

Chaos testing is scheduled, controlled fault injection in a test or staging environment. You know what you're injecting, you control the blast radius, and you observe recovery. Low risk. High learning.

Chaos engineering (in the Netflix/Principles of Chaos sense) is running experiments in production to discover unknown failure modes. Higher risk. Higher confidence because it's testing the real system.

Most teams should start with chaos testing in staging before anything touches production. The discipline of designing a well-scoped chaos experiment applies to both.

Pre-Chaos Baselining

Before injecting any fault, measure your current SLI state. This establishes the control condition: what does normal look like?

# Baseline script — run before any chaos experiment
#!/bin/bash
PROMETHEUS_URL="${PROMETHEUS_URL:-http://prometheus:9090}"
SERVICE="${1:-payments-api}"
WINDOW="${2:-5m}"

echo "=== Chaos Experiment Baseline ==="
echo "Service: $SERVICE"
echo "Window: $WINDOW"
echo "Timestamp: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
echo ""

# Availability SLI
availability=$(curl -s "$PROMETHEUS_URL/api/v1/query" \
  --data-urlencode "query=job:sli_availability:rate5m{job=\"$SERVICE\"}" \
  | jq -r '.data.result[0].value[1] // "N/A"')
echo "Availability SLI: $availability"

# Latency p95
latency_p95=$(curl -s "$PROMETHEUS_URL/api/v1/query" \
  --data-urlencode "query=histogram_quantile(0.95, rate(http_request_duration_seconds_bucket{job=\"$SERVICE\"}[$WINDOW]))" \
  | jq -r '.data.result[0].value[1] // "N/A"')
echo "Latency p95: ${latency_p95}s"

# Error rate
error_rate=$(curl -s "$PROMETHEUS_URL/api/v1/query" \
  --data-urlencode "query=sum(rate(http_requests_total{job=\"$SERVICE\",status=~\"5..\"}[$WINDOW])) / sum(rate(http_requests_total{job=\"$SERVICE\"}[$WINDOW]))" \
  | jq -r '.data.result[0].value[1] // "0"')
echo "Error rate: $error_rate"

# Error budget remaining
echo ""
echo "Error budget remaining (query Prometheus for full burn rate)..."

Save this baseline. After the experiment, you'll compare against it.

Scoping Experiments to SLO Impact

Define the maximum SLO degradation you're willing to accept during the experiment before you start:

# chaos-experiment-plan.yaml
experiment:
  name: "Payment API - DB Connection Pool Exhaustion"
  hypothesis: "When the database connection pool is exhausted, the service returns 503 with retry guidance rather than hanging or returning 500"
  
  scope:
    service: payments-api
    environment: staging
    traffic_percentage: 100  # 100% of staging traffic
    
  slo_budget_allocation:
    # We're willing to consume this much error budget during the experiment
    max_availability_degradation: 0.01  # allow up to 1% error rate
    max_latency_increase_multiplier: 3  # allow latency to triple
    experiment_duration_minutes: 15
    
  abort_conditions:
    # If ANY of these are breached, stop the experiment immediately
    - "availability drops below 90%"
    - "latency p95 exceeds 10 seconds"  
    - "experiment exceeds 15 minutes"
    - "any data corruption detected"
    
  steady_state_hypothesis:
    # Pre and post conditions that must be true
    before:
      - "availability > 99.9%"
      - "latency p95 < 200ms"
    after:
      - "availability > 99.9%"
      - "latency p95 < 200ms"
      - "no errors in application logs"

This plan exists before a single command is run. It forces you to think through what you expect to happen and what would cause you to abort.

Chaos Experiments with Litmus Chaos

Litmus Chaos is a CNCF project for Kubernetes-native chaos engineering. It provides pre-built fault types with configurable blast radius.

Install Litmus on your staging cluster:

kubectl apply -f https://litmuschaos.github.io/litmus/litmus-operator-v3.7.0.yaml
kubectl get pods -n litmus

A pod CPU stress experiment scoped to one deployment:

# experiments/pod-cpu-stress.yaml
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: payments-api-cpu-stress
  namespace: staging
spec:
  appinfo:
    appns: staging
    applabel: "app=payments-api"
    appkind: deployment
  
  engineState: active
  
  # Only target 1 pod at a time — limit blast radius
  jobCleanUpPolicy: retain
  
  experiments:
    - name: pod-cpu-hog
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: "300"  # 5 minutes
            - name: CPU_CORES
              value: "1"    # stress 1 CPU core
            - name: PODS_AFFECTED_PERC
              value: "50"   # affect 50% of pods (not all)
            
        # Probe to verify hypothesis during chaos
        probe:
          - name: "availability-probe"
            type: httpProbe
            mode: Continuous
            httpProbe/inputs:
              url: "http://payments-api.staging.svc:8080/health"
              method:
                get:
                  criteria: "=="
                  responseCode: "200"
            runProperties:
              probeTimeout: 5
              interval: 10
              attempt: 3
# Apply the experiment
kubectl apply -f experiments/pod-cpu-stress.yaml

# Monitor SLIs during experiment
watch -n 10 bash baseline.sh payments-api 1m

# Check experiment status
kubectl describe chaosengine payments-api-cpu-stress -n staging

A network latency experiment (simulates slow downstream dependency):

# experiments/network-latency.yaml
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: payments-api-network-latency
  namespace: staging
spec:
  appinfo:
    appns: staging
    applabel: "app=payments-api"
    appkind: deployment
  engineState: active
  experiments:
    - name: pod-network-latency
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: "300"
            - name: NETWORK_LATENCY
              value: "2000"         # 2000ms additional latency
            - name: JITTER
              value: "200"          # ±200ms jitter
            - name: PODS_AFFECTED_PERC
              value: "50"
            - name: DESTINATION_IPS
              value: "10.0.0.50"    # target only the DB service IP

Chaos Monkey for JVM Services

Netflix's Chaos Monkey (via the Simian Army) can be configured to terminate instances on a schedule. For Spring Boot services, the chaos-monkey-spring-boot library injects chaos directly in the application:

<!-- pom.xml -->
<dependency>
  <groupId>de.codecentric</groupId>
  <artifactId>chaos-monkey-spring-boot</artifactId>
  <version>3.1.0</version>
</dependency>
# application-chaos.yml
chaos:
  monkey:
    enabled: true
    watcher:
      service: true
      repository: true
    assaults:
      level: 3            # 1 in 3 calls get assaulted
      latencyActive: true
      latencyRangeStart: 1000
      latencyRangeEnd: 3000
      exceptionsActive: false  # only latency, not exceptions
// Integration test with chaos enabled
@SpringBootTest
@ActiveProfiles("chaos")
@TestPropertySource(properties = {
    "chaos.monkey.enabled=true",
    "chaos.monkey.assaults.level=5",
    "chaos.monkey.assaults.latencyActive=true"
})
class ResilienceUnderLatencyTest {

    @Autowired
    private PaymentService paymentService;

    @Test
    @Timeout(value = 5, unit = TimeUnit.SECONDS)
    void processPaymentCompletesWithinTimeoutEvenWithLatency() {
        // Circuit breaker should open and fail fast, not hang
        PaymentResult result = paymentService.processPayment(
            new PaymentRequest("user-1", 1999)
        );
        
        // Should either succeed or fail fast — not timeout
        assertThat(result.getStatus()).isIn(
            PaymentStatus.SUCCESS,
            PaymentStatus.CIRCUIT_OPEN
        );
    }
}

SLO Tracking During Experiments

During a chaos experiment, track SLIs in real time. Use a Grafana dashboard that shows:

  1. Current availability SLI (5m window)
  2. Error budget burn rate
  3. p95 latency
  4. Number of error budget minutes consumed

Here's a Prometheus query to track consumed error budget during an experiment:

# Minutes of error budget consumed in the last hour
(
  1 - job:sli_availability:rate5m{job="payments-api"}
) / (1 - 0.999) * 60

If this number is growing faster than 1 minute per minute, you're burning error budget at an unsustainable rate and should consider aborting.

Automated Abort Conditions

Don't rely on a human to watch the dashboard and decide to abort. Automate it:

# chaos_watchdog.py
import time
import sys
import requests
import subprocess

PROMETHEUS_URL = os.environ['PROMETHEUS_URL']
CHAOS_ENGINE_NAME = sys.argv[1]
NAMESPACE = sys.argv[2]

def get_availability():
    r = requests.get(f"{PROMETHEUS_URL}/api/v1/query", params={
        'query': 'job:sli_availability:rate5m{job="payments-api"}'
    })
    result = r.json()['data']['result']
    return float(result[0]['value'][1]) if result else 1.0

def abort_experiment(reason):
    print(f"ABORTING: {reason}")
    subprocess.run([
        'kubectl', 'patch', 'chaosengine', CHAOS_ENGINE_NAME,
        '-n', NAMESPACE,
        '--type=merge',
        '-p', '{"spec":{"engineState":"stop"}}'
    ])
    sys.exit(1)

start_time = time.time()
MAX_DURATION = 15 * 60  # 15 minutes

while True:
    elapsed = time.time() - start_time
    
    if elapsed > MAX_DURATION:
        abort_experiment("Maximum experiment duration exceeded")
    
    availability = get_availability()
    print(f"[{elapsed:.0f}s] Availability: {availability:.4f}")
    
    if availability < 0.90:
        abort_experiment(f"Availability dropped below 90%: {availability:.4f}")
    
    time.sleep(30)

Run this in a separate process during the experiment:

python chaos_watchdog.py payments-api-cpu-stress staging &
WATCHDOG_PID=$!

# Apply chaos experiment
kubectl apply -f experiments/pod-cpu-stress.yaml

# Wait for experiment to complete
kubectl wait chaosresult payments-api-cpu-stress-pod-cpu-hog \
  --for=condition=completed \
  --timeout=20m \
  -n staging

kill $WATCHDOG_PID

Post-Experiment Analysis

After the experiment completes, compare post-chaos state to baseline:

echo "=== Post-Experiment State ==="
bash baseline.sh payments-api 5m

echo ""
echo "=== Chaos Result ==="
kubectl get chaosresult payments-api-cpu-stress-pod-cpu-hog \
  -n staging \
  -o jsonpath='{.status.experimentStatus.verdict}'

Document findings:

## Experiment: Pod CPU Stress — 2026-05-15

**Result: PASS**

Pre-chaos baseline:
- Availability: 99.97%
- p95 latency: 145ms
- Error rate: 0.03%

During chaos (50% pods at 100% CPU):
- Availability: 99.81% (degraded but within SLO)
- p95 latency: 312ms (increased but under 500ms threshold)
- Error rate: 0.19% (within error budget)

Post-chaos recovery:
- Availability returned to 99.97% within 2 minutes of chaos end
- Latency returned to baseline within 90 seconds

**Hypothesis confirmed:** The service degrades gracefully under CPU pressure — 
it slows down but does not produce 5xx errors. Recovery is automatic and within 2 minutes.

**Findings:**
- Circuit breaker opened correctly at 30% error rate
- Kubernetes HPA scaled up from 3 to 5 pods during experiment — confirm HPA config is correct
- One pod failed liveness probe during high CPU — this is expected

**No error budget impact:** Total estimated budget consumed: 0.04% (well within the 1% allocation)

Starting Small

Don't start with production. Don't start with a full service outage. Start with:

  1. Staging environment — low risk, maximum learning
  2. Single fault type — latency first (less risky than pod kill)
  3. Low blast radius — 25% of pods, not 100%
  4. Short duration — 5 minutes, not 30
  5. With an abort watchdog running

Gradually expand scope as you build confidence in your abort mechanisms and your understanding of failure modes. When your staging chaos experiments consistently confirm your hypotheses, you've earned the right to consider production experiments — with even tighter scope and abort conditions.

The goal is not to cause incidents. The goal is to discover whether your system survives incident conditions. There's a big difference, and it all comes down to preparation.

Read more

Start now free