Chaos Engineering and Observability: Testing System Resilience

Why Systems Fail Unexpectedly Despite Testing

Unit tests and integration tests verify that code does what it is supposed to do under normal conditions. They do not verify what happens when a dependency is slow, a network partition isolates a service, a disk fills up, or a downstream API starts returning 503s.

Real systems operate under Murphy's Law: if something can fail, it will. The question is whether you discover failure modes during a controlled chaos experiment at 2pm on a Tuesday, or during peak traffic at 2am on a Friday.

Chaos engineering is the discipline of creating controlled failures to answer that question on your terms.

The Steady-State Hypothesis

The foundational concept in chaos engineering is the steady-state hypothesis — a measurable definition of normal system behavior. Without it, you cannot determine whether your system degraded during an experiment.

A good steady-state hypothesis is specific and metric-backed:

# chaos/hypotheses/order-service.yaml
steady_state:
  description: "Order service processes requests normally"
  probes:
    - name: success-rate
      type: prometheus
      query: |
        rate(http_requests_total{service="orders",status=~"2.."}[2m])
        /
        rate(http_requests_total{service="orders"}[2m])
      threshold:
        min: 0.99  # 99% success rate
      
    - name: p95-latency
      type: prometheus
      query: |
        histogram_quantile(0.95,
          rate(http_request_duration_seconds_bucket{service="orders"}[2m])
        )
      threshold:
        max: 0.2  # 200ms

    - name: error-logs
      type: loki
      query: |
        count_over_time({service="orders"} |= "ERROR"[2m])
      threshold:
        max: 10  # Fewer than 10 error logs per 2 minutes

Before injecting any fault, you verify the steady state is met. After the experiment, you verify it returns to steady state within your recovery time objective.

Chaos Tooling Overview

Several mature tools exist for chaos experiments at different layers:

Chaos Monkey (Netflix) — randomly terminates EC2 instances or containers. Simple, effective for testing service-level redundancy.

Litmus Chaos (CNCF) — Kubernetes-native chaos operator. Rich experiment library covering pod failures, network faults, node stress, and more.

Gremlin — commercial platform with a wide experiment catalog, blast radius controls, and native Kubernetes and cloud integrations.

Chaos Toolkit — open-source, declarative YAML-based framework that integrates with any system and any observability backend.

For most teams starting with chaos engineering, Litmus on Kubernetes or Chaos Toolkit is the right entry point.

A Structured Chaos Experiment: Pod Failure

Here is a complete Litmus ChaosEngine experiment for testing how the orders service handles pod termination:

# chaos/experiments/orders-pod-failure.yaml
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: orders-pod-failure
  namespace: production
spec:
  appinfo:
    appns: production
    applabel: "app=orders-service"
    appkind: deployment

  # Only affect 1 of N replicas (blast radius control)
  chaosServiceAccount: litmus-admin

  experiments:
    - name: pod-delete
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: "60"  # 60 seconds of chaos
            - name: CHAOS_INTERVAL
              value: "10"  # Kill a pod every 10 seconds
            - name: FORCE
              value: "false"  # Graceful termination
            - name: PODS_AFFECTED_PERC
              value: "50"  # Kill up to 50% of pods

Before running this experiment, you verify the steady state. During the 60-second window, you watch your dashboards. After it ends, you verify the steady state returned.

Observability During Chaos Experiments

The chaos experiment itself is not the deliverable — the observability data collected during it is. You need to know:

1. Did the steady-state metrics degrade during the fault?
2. If they degraded, by how much and for how long?
3. Did the system recover automatically, and how quickly?
4. Were there any unexpected failure modes (cascading failures, silent data corruption)?

Grafana annotations let you mark chaos experiment windows on your dashboards:

# Mark experiment start in Grafana
curl -X POST http://grafana:3000/api/annotations \
  -H "Content-Type: application/json" \
  -d '{
    "text": "Chaos: orders pod-delete (50% of pods, 60s)",
    "tags": ["chaos", "orders-service", "pod-failure"],
    "time": '$(date +%s%3N)'
  }'

# Run the chaos experiment
kubectl apply -f chaos/experiments/orders-pod-failure.yaml

sleep 60

# Mark experiment end
curl -X POST http://grafana:3000/api/annotations \
  -H "Content-Type: application/json" \
  -d '{
    "text": "Chaos end: orders pod-delete",
    "tags": ["chaos", "orders-service", "pod-failure", "end"],
    "time": '$(date +%s%3N)'
  }'

Now your Grafana dashboard shows exactly when the experiment ran, making the correlation between fault injection and metric behavior visually obvious.

Network Chaos: The More Realistic Failure Mode

Pod deaths are visible — Kubernetes reschedules and recovers quickly. Network degradation is subtler and more damaging. A service that dies returns errors immediately; a service that is slow degrades everything that calls it.

Litmus network chaos experiments:

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: orders-network-latency
spec:
  appinfo:
    applabel: "app=orders-service"
  experiments:
    - name: pod-network-latency
      spec:
        components:
          env:
            - name: NETWORK_LATENCY
              value: "200"  # Add 200ms to all network calls
            - name: JITTER
              value: "50"   # ±50ms jitter
            - name: TOTAL_CHAOS_DURATION
              value: "120"
            - name: DESTINATION_IPS
              value: ""  # Empty = all destinations
            - name: DESTINATION_HOSTS
              value: "payment-service"  # Target only payment calls

With 200ms added latency to payment service calls, you want to verify:

1. The orders service circuit breaker opens after 5 consecutive slow calls
2. Orders fail fast (not after 30 seconds) with a clear error
3. The orders service does not cascade — other endpoints stay healthy
4. Metrics show the failure isolated to the /checkout path

# Watch for circuit breaker opening during experiment
orders_circuit_breaker_state{downstream="payment-service"}

# Verify non-checkout endpoints stay healthy
rate(http_requests_total{service="orders",handler!="/checkout",status=~"5.."}[1m])

Automated Chaos in CI

Running chaos experiments manually in staging is valuable but insufficient. Resilience regressions — removing a retry mechanism, accidentally setting a timeout too high — can ship in any PR. Automated chaos in CI catches these before deployment.

Chaos Toolkit integrates cleanly into CI:

# chaos/experiments/api-resilience.json
{
  "title": "Order service handles payment service unavailability",
  "description": "When payment service is down, orders return 503 within 500ms",
  "steady-states": {
    "before": {
      "title": "System is healthy",
      "probes": [
        {
          "name": "orders-healthy",
          "type": "probe",
          "provider": {
            "type": "http",
            "url": "http://orders-service/health",
            "expected_status": 200,
            "timeout": 1
          }
        }
      ]
    }
  },
  "method": [
    {
      "type": "action",
      "name": "stop-payment-service",
      "provider": {
        "type": "process",
        "path": "kubectl",
        "arguments": "scale deployment payment-service --replicas=0 -n test"
      },
      "pauses": { "after": 5 }
    },
    {
      "type": "probe",
      "name": "orders-fail-fast",
      "provider": {
        "type": "http",
        "url": "http://orders-service/checkout",
        "method": "POST",
        "expected_status": 503,
        "timeout": 0.5
      }
    }
  ],
  "rollbacks": [
    {
      "type": "action",
      "name": "restart-payment-service",
      "provider": {
        "type": "process",
        "path": "kubectl",
        "arguments": "scale deployment payment-service --replicas=2 -n test"
      }
    }
  ]
}

# .github/workflows/chaos.yml
name: Resilience Tests
on:
  push:
    branches: [main]

jobs:
  chaos:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install Chaos Toolkit
        run: pip install chaostoolkit chaostoolkit-kubernetes

      - name: Run resilience experiments
        run: |
          chaos run chaos/experiments/api-resilience.json
          chaos run chaos/experiments/db-connection-exhaustion.json

The rollback section ensures the experiment always cleans up, even if the probe fails. CI chaos experiments need to be self-contained and leave no side effects.

GameDays: The Human Dimension of Chaos

Automated chaos tests what your system does. GameDays test what your team does.

A GameDay is a scheduled exercise where engineers deliberately inject failures while the whole team watches — on-call engineers, developers, product managers — to observe both system and human response. The goals are:

1. Verify your runbooks are accurate and complete
2. Find gaps in alert routing (who gets paged for what?)
3. Build muscle memory for incident response
4. Identify missing observability (failures that are hard to diagnose)

A simple GameDay agenda:

09:00 — Brief: what we're testing, steady-state definition, blast radius
09:15 — Verify steady state (all probes green)
09:20 — Inject fault: kill 2 of 3 order service replicas
09:20-09:35 — Observe: does the alert fire? Does Kubernetes recover? How long?
09:35 — Inject fault: add 500ms latency to payment service
09:35-09:50 — Observe: does circuit breaker open? Do orders queue or fail?
09:50 — Rollback, verify steady state returns
10:00 — Retrospective: what did we learn? What do we fix?

The retrospective is where the value lives. Missing an alert that should have fired, a runbook step that was wrong, a metric that was not visible — these go into tickets immediately.

HelpMeTest: Validating Recovery After Chaos

After a chaos experiment, you want confidence that normal behavior has been fully restored — not just that metrics look healthy, but that the full user journey works end to end. HelpMeTest runs your critical user journey scenarios against production continuously. After a chaos experiment, a full test run in HelpMeTest confirms that checkout, authentication, order creation, and other key flows are working correctly.

This is especially valuable after GameDays: at the end of the exercise, you trigger a full HelpMeTest run to verify the system is fully recovered before declaring the exercise closed.

Common Chaos Engineering Mistakes

Starting too big. Your first chaos experiment should not target your production database. Start with a single non-critical service in a staging environment with synthetic traffic.

No observability, no experiment. If you cannot measure steady state before and after, you cannot interpret results. Set up dashboards before running experiments.

Skipping rollbacks. Every chaos action must have a rollback. If the experiment fails mid-way, you need automated cleanup. Manual cleanup after a failed experiment in production is an incident.

Chaos without hypothesis. "Let's kill some pods and see what happens" is not an experiment — it is a production incident waiting to happen. Every experiment starts with a specific hypothesis: "When X fails, Y should happen within Z seconds."

Conclusion

Chaos engineering is how you find out whether your system is as resilient as your architecture diagrams suggest. Observability is how you interpret what happens during experiments. Together, they give you evidence-based confidence in your system's failure behavior — not theoretical confidence from design reviews.

Start small: pick one service, define a measurable steady state, inject one fault, and see what you learn. The first experiment is usually the most valuable.