Testing

Chaos Engineering on Kubernetes with Litmus

HelpMeTest

19 Aug 2026 — 7 min read

LitmusChaos is a CNCF project that runs controlled chaos experiments on Kubernetes — pod deletion, CPU stress, network latency, disk fill. It uses CRDs to define experiments declaratively and produces structured results you can evaluate in CI. This post walks through installation, your first experiments, and how to read the results.

Your Kubernetes application is deployed, health probes are configured, your Helm chart is unit tested, and your E2E suite passes. You're confident. But confidence is not the same as resilience.

What happens when a pod is killed unexpectedly? Does your application recover without user-visible impact? What happens when network latency between services spikes to 500ms? Do your timeouts fire correctly? What happens when a node runs out of memory? Does your scheduler place pods correctly?

Chaos engineering answers these questions by introducing controlled failures into your running system and measuring the impact. LitmusChaos is the leading open-source chaos engineering platform for Kubernetes. It is a CNCF incubating project, widely adopted, and designed from the ground up for Kubernetes-native workflows.

What Chaos Engineering Is (and Is Not)

Chaos engineering is not random destruction. It is a disciplined practice:

Define a steady state — a measurable baseline of normal behavior (e.g., p99 latency < 200ms, error rate < 0.1%)
Hypothesize — "introducing X failure will not affect the steady state because we have Y in place"
Run an experiment — inject the failure with a defined blast radius and duration
Measure — compare actual behavior to the steady state
Learn — if the hypothesis was wrong, fix the weakness; if right, document the resilience evidence

LitmusChaos operationalizes steps 2–4 on Kubernetes.

LitmusChaos Architecture

LitmusChaos uses Kubernetes Custom Resource Definitions (CRDs) to define and run experiments:

ChaosExperiment — defines a type of fault (pod-delete, cpu-hog, network-latency, etc.)
ChaosEngine — orchestrates the experiment against a target workload
ChaosResult — stores the outcome (Passed, Failed, Stopped, Awaited)

The Litmus operator watches for ChaosEngine resources and runs the corresponding experiment pods. This declarative approach means you can store chaos experiments in git alongside your application manifests and run them in CI.

Installing LitmusChaos

Litmus 3.x installs via Helm:

# Add the Litmus Helm repo
helm repo add litmuschaos https://litmuschaos.github.io/litmus-helm/
helm repo update

# Create namespace
kubectl create namespace litmus

# Install Litmus
helm install chaos litmuschaos/litmus \
  --namespace litmus \
  --set portal.frontend.service.type=NodePort

# Wait for Litmus components to be ready
kubectl wait deployment/chaos-litmus-frontend-service \
  --for=condition=Available \
  --timeout=120s \
  -n litmus

kubectl wait deployment/chaos-litmus-server-service \
  --for=condition=Available \
  --timeout=120s \
  -n litmus

Install the generic experiment library (includes pod-delete, node-drain, network faults, etc.):

kubectl apply -f https://hub.litmuschaos.io/api/chaos?file=charts/generic/experiments.yaml \
  -n litmus

Verify experiments are available:

kubectl get chaosexperiments -n litmus

You should see experiments like pod-delete, pod-cpu-hog, pod-network-latency, node-drain, disk-fill.

Setting Up RBAC for Experiments

LitmusChaos experiment pods need permissions to delete pods, inject faults, etc. Create a service account for your experiment namespace:

# rbac-chaos.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: pod-delete-sa
  namespace: default
  labels:
    name: pod-delete-sa
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-delete-sa
  namespace: default
  labels:
    name: pod-delete-sa
rules:
  - apiGroups: [""]
    resources: ["pods", "events"]
    verbs: ["create", "list", "get", "patch", "update", "delete", "deletecollection"]
  - apiGroups: [""]
    resources: ["pods/exec", "pods/log", "replicationcontrollers"]
    verbs: ["create", "list", "get"]
  - apiGroups: ["apps"]
    resources: ["deployments", "statefulsets", "replicasets", "daemonsets"]
    verbs: ["list", "get"]
  - apiGroups: ["litmuschaos.io"]
    resources: ["chaosengines", "chaosexperiments", "chaosresults"]
    verbs: ["create", "list", "get", "patch", "update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: pod-delete-sa
  namespace: default
  labels:
    name: pod-delete-sa
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: pod-delete-sa
subjects:
  - kind: ServiceAccount
    name: pod-delete-sa
    namespace: default

kubectl apply -f rbac-chaos.yaml

Experiment 1: Pod Delete

Pod deletion is the most fundamental chaos experiment. It simulates a pod crash and tests whether your deployment recovers quickly enough to maintain availability.

Steady state assumption: Your deployment maintains at least 1 ready replica at all times and recovers within 30 seconds of a pod deletion.

Define the experiment:

# chaos-pod-delete.yaml
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: pod-delete-chaos
  namespace: default
spec:
  appinfo:
    appns: default
    applabel: "app=myapp"
    appkind: deployment
  jobCleanUpPolicy: retain
  chaosServiceAccount: pod-delete-sa
  experiments:
    - name: pod-delete
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: "60"       # run for 60 seconds
            - name: CHAOS_INTERVAL
              value: "10"       # delete a pod every 10 seconds
            - name: FORCE
              value: "false"    # graceful deletion
            - name: PODS_AFFECTED_PERC
              value: "50"       # kill 50% of matching pods

Apply it:

kubectl apply -f chaos-pod-delete.yaml

Watch the experiment run:

kubectl get chaosengine pod-delete-chaos -w -n default

While it runs, observe your deployment:

watch kubectl get pods -l app=myapp -n default

You should see pods being deleted and new ones starting. If your deployment has proper health probes and RollingUpdate strategy, user-facing traffic should remain unaffected during this experiment.

Experiment 2: Network Latency

Network latency between services is one of the most common real-world failure modes. Services get slow, timeouts fire (or don't), and the cascade begins.

Set up a separate service account for network experiments (they need additional permissions for tc network manipulation):

# chaos-network-latency.yaml
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: network-latency-chaos
  namespace: default
spec:
  appinfo:
    appns: default
    applabel: "app=order-service"
    appkind: deployment
  chaosServiceAccount: pod-network-chaos-sa
  experiments:
    - name: pod-network-latency
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: "120"
            - name: NETWORK_LATENCY
              value: "500"      # 500ms added latency
            - name: JITTER
              value: "50"       # ±50ms jitter
            - name: CONTAINER_RUNTIME
              value: "containerd"
            - name: SOCKET_PATH
              value: "/run/containerd/containerd.sock"
            - name: PODS_AFFECTED_PERC
              value: "100"
            - name: DESTINATION_IPS
              value: ""         # empty = all destinations

This injects 500ms (±50ms) of latency into all network traffic from the order-service pods for 120 seconds.

What you're looking for during this experiment:

Does your API gateway time out correctly? (Not hang indefinitely)
Does it return a meaningful error to clients? (503 with a retry-after header, not a silent 500)
Does the latency affect only order-related endpoints, or does it cascade?
Do your metrics/dashboards show the latency spike?

Experiment 3: Pod CPU Hog

CPU starvation causes slow responses and eventually OOM kills if the container tries to compensate. Test that your resource limits protect other pods on the same node:

# chaos-cpu-hog.yaml
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: cpu-hog-chaos
  namespace: default
spec:
  appinfo:
    appns: default
    applabel: "app=worker-service"
    appkind: deployment
  chaosServiceAccount: pod-cpu-hog-sa
  experiments:
    - name: pod-cpu-hog
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: "60"
            - name: CPU_CORES
              value: "2"        # hog 2 CPU cores
            - name: PODS_AFFECTED_PERC
              value: "100"

Your CPU limits should prevent this from impacting other services. If they don't, the experiment reveals that your resource quotas are misconfigured.

Reading ChaosResult

When an experiment completes, check the result:

kubectl describe chaosresult pod-delete-chaos-pod-delete -n default

Output:

Name:         pod-delete-chaos-pod-delete
Namespace:    default
Status:
  Experimentstatus:
    Failstep:             N/A
    Phase:                Completed
    Probestatus:
      Name:    check-pod-running
      Status:
        Continuous:  Passed
      Type:    cmdProbe
    Verdict:  Pass
  History:
    Failed Runs:   0
    Passed Runs:   3
    Stopped Runs:  0

Verdict: Pass means the experiment completed and all probes passed. Verdict: Fail means the system did not recover as expected.

Get the raw result as JSON for CI evaluation:

VERDICT=$(kubectl get chaosresult pod-delete-chaos-pod-delete \
  -n default \
  -o jsonpath='{.status.experimentstatus.verdict}')

if [ "$VERDICT" = "Pass" ]; then
  echo "PASS: System resilient to pod deletion"
else
  echo "FAIL: System not resilient. Verdict: $VERDICT"
  exit 1
fi

Adding Probes for Steady-State Validation

The real power of LitmusChaos comes from probes — checks that run during the experiment to verify your steady state is maintained. Without probes, Litmus only checks whether the experiment itself ran. With probes, it checks whether your application stayed healthy.

Add an HTTP probe that pings your API during the chaos:

  experiments:
    - name: pod-delete
      spec:
        probe:
          - name: check-api-health
            type: httpProbe
            httpProbe/inputs:
              url: "http://api-gateway.default.svc.cluster.local/healthz"
              insecureSkipVerify: false
              method:
                get:
                  criteria: "=="
                  responseCode: "200"
            mode: Continuous
            runProperties:
              probeTimeout: 5
              interval: 3
              retry: 2
              probePollingInterval: 2
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: "60"
            - name: CHAOS_INTERVAL
              value: "10"

This probe hits /healthz every 3 seconds during the experiment. If it gets a non-200 response and retries fail, the experiment verdict becomes Fail — meaning the chaos revealed a real resilience gap.

Running Chaos in CI

Chaos experiments can run in CI against a kind cluster or a dedicated staging namespace. Here is a GitHub Actions step:

      - name: Run chaos experiment
        run: |
          kubectl apply -f chaos/rbac-chaos.yaml
          kubectl apply -f chaos/chaos-pod-delete.yaml

          # Wait for experiment to complete (max 3 minutes)
          TIMEOUT=180
          ELAPSED=0
          while true; do
            PHASE=$(kubectl get chaosengine pod-delete-chaos \
              -o jsonpath='{.status.engineStatus}' -n default)
            if [ "$PHASE" = "completed" ]; then break; fi
            if [ "$ELAPSED" -ge "$TIMEOUT" ]; then
              echo "Chaos experiment timed out"
              exit 1
            fi
            sleep 5
            ELAPSED=$((ELAPSED+5))
          done

          VERDICT=$(kubectl get chaosresult pod-delete-chaos-pod-delete \
            -o jsonpath='{.status.experimentstatus.verdict}' -n default)

          echo "Chaos verdict: $VERDICT"
          [ "$VERDICT" = "Pass" ] || exit 1

Start with pod-delete in CI — it is the most impactful and easiest to stabilize. Add network latency experiments once you have timeout configurations dialed in.

Interpreting Results and Building Resilience

A Fail verdict is not a test failure in the traditional sense — it is a finding. It tells you that your system does not handle a specific failure mode as expected. The response is not to delete the test; it is to fix the weakness.

Common findings and their fixes:

Finding	Fix
Pod deletion causes 30s+ downtime	Add PodDisruptionBudget; increase replica count; fix readiness probe
Network latency causes cascade failures	Add timeouts and circuit breakers; reduce default HTTP client timeout
CPU hog affects other services on node	Set CPU limits; add resource quotas to namespace
Recovery takes > 60s after node drain	Increase replica count; spread across zones with topologySpreadConstraints

Each experiment you run and pass is evidence of a resilience property. Document it. When an incident happens in production and someone asks "why didn't we know the system would fail like this?", you can point to the chaos run that validated the opposite scenario — and the chaos run you hadn't run yet that would have caught this one.

Chaos engineering is not about finding that everything is broken. It is about building confidence through controlled evidence. Start small, start with pod-delete, and grow your experiment library as your system matures.