Fault Injection Testing: Tools and Techniques (Chaos Mesh, Litmus, Gremlin)

What Fault Injection Testing Covers

Fault injection testing introduces specific, controlled failure conditions into a running system:

• Pod/process kills: Terminate a service instance and observe recovery
• Network faults: Add latency, packet loss, bandwidth limits, or simulate a full partition
• Resource stress: Exhaust CPU, memory, or disk to simulate overload
• Disk faults: Fill disks, slow I/O, corrupt writes
• Application-level faults: Return specific HTTP error codes, delay specific responses
• Time chaos: Jump the clock forward to test timeout and TTL handling

Each fault type tests a different assumption about system resilience. Pod kill tests recovery and load balancing. Network latency tests timeout handling and circuit breakers. Memory pressure tests garbage collection and OOM behavior.

Chaos Mesh

Chaos Mesh is a CNCF project (sandbox → incubating) built specifically for Kubernetes. It installs as a set of CRDs and controllers. Experiments are defined as Kubernetes resources — you apply a YAML file, the experiment runs.

Installation

# Install with Helm
helm repo add chaos-mesh https://charts.chaos-mesh.org
helm install chaos-mesh chaos-mesh/chaos-mesh \
  --namespace=chaos-mesh \
  --create-namespace

Fault Types

Chaos Mesh supports a wide range of fault types:

Example: Network Latency Injection

apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: network-delay-experiment
  namespace: production
spec:
  action: delay
  mode: one              # affects one pod at random
  selector:
    namespaces:
      - production
    labelSelectors:
      app: payment-service
  delay:
    latency: "500ms"
    correlation: "25"
    jitter: "100ms"
  duration: "5m"
  direction: to          # delay inbound traffic

Apply it: kubectl apply -f network-delay.yaml

Watch dashboards for 5 minutes, then delete the resource to stop the experiment:

kubectl delete networkchaos network-delay-experiment -n production

Strengths

• Wide fault coverage for Kubernetes workloads
• No agents required (uses Linux kernel features via daemonsets)
• Chaos Dashboard for visualization
• Supports scheduled and workflow experiments
• CNCF project — open source, community-maintained

Weaknesses

• Kubernetes-only
• YAML configuration is verbose
• Steeper learning curve than Gremlin
• Dashboard is functional but not polished

Litmus Chaos

Litmus is another CNCF project, also Kubernetes-native. Its distinguishing feature is the concept of Chaos Workflows — experiments organized into multi-step sequences with scheduling, observability hooks, and approval gates.

Installation

kubectl apply -f https://litmuschaos.github.io/litmus/litmus-operator-v3.0.0.yaml

Experiment Structure

Litmus organizes chaos experiments hierarchically:

• Chaos Experiments — individual fault types (pod kill, node CPU hog, etc.)
• Chaos Scenarios — sequences of experiments with scheduling
• Probes — steady-state checks that run before and after experiments

Example: Pod Delete Experiment

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: pod-delete-engine
  namespace: default
spec:
  appinfo:
    appns: "production"
    applabel: "app=web-service"
    appkind: "deployment"
  engineState: "active"
  chaosServiceAccount: litmus-admin
  experiments:
    - name: pod-delete
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: "60"
            - name: CHAOS_INTERVAL
              value: "10"
            - name: FORCE
              value: "false"
        probe:
          - name: "check-web-service-availability"
            type: "httpProbe"
            httpProbe/inputs:
              url: "http://web-service/health"
              responseTimeout: 2000
              method:
                get:
                  criteria: "=="
                  responseCode: "200"
            mode: "Continuous"
            runProperties:
              probeTimeout: 5
              interval: 5
              retry: 1

Strengths

• Workflow-based organization scales to many experiments
• Built-in probes for steady-state verification
• Governance features (who can run what experiments)
• Good Prometheus integration for metrics
• CNCF project, active community

Weaknesses

• More complex to set up than Chaos Mesh for simple experiments
• Kubernetes-only
• Workflow configuration has a learning curve

Gremlin

Gremlin is a commercial platform for chaos engineering. Unlike Chaos Mesh and Litmus, Gremlin uses an agent model — you install a lightweight agent on hosts, VMs, or containers, and manage experiments from the Gremlin web UI.

Key Difference: Agent-Based

Gremlin doesn't require Kubernetes. The agent runs on any Linux host (including bare metal, VMs, EC2 instances, GKE nodes). This makes it the most flexible option for mixed infrastructure.

Fault Types

Gremlin organizes faults into categories:

Using the API

Gremlin provides a REST API for programmatic experiment execution:

# Start a CPU attack
curl -X POST https://api.gremlin.com/v1/attacks/new \
  -H "Content-Type: application/json" \
  -H "Authorization: Key $GREMLIN_API_KEY" \
  -d '{
    "target": {
      "type": "Exact",
      "hosts": ["my-host-identifier"]
    },
    "command": {
      "type": "cpu",
      "args": ["-c", "2", "-l", "80"]
    }
  }'

Gremlin's Safety Features

Gremlin includes "Halt" — a kill switch that immediately stops all running attacks across all targets. This is particularly valuable in production: if an experiment goes wrong, one button stops everything.

Other safety features:

• Attack throttling (maximum concurrent attacks per environment)
• Team-based access controls
• Audit logs of all experiments

Strengths

• Works on any Linux infrastructure (not Kubernetes-dependent)
• Polished web UI for non-engineers to observe experiments
• Halt kill switch for emergency stops
• Commercial support
• Scenario and GameDay features for orchestrated experiments

Weaknesses

• Not free (pricing based on number of hosts)
• Agent installation on every host required
• Less Kubernetes-native than Chaos Mesh or Litmus

Tool Comparison

Choosing a Tool

Choose Chaos Mesh if:

• You're on Kubernetes
• You want broad fault type coverage
• You're comfortable with YAML/CRD workflows
• You want an open-source CNCF project

Choose Litmus if:

• You're on Kubernetes
• You want workflow orchestration and scheduling
• Your team needs governance controls over who can run experiments
• You're integrating chaos into CI/CD pipelines

Choose Gremlin if:

• You have VMs, bare metal, or mixed infrastructure
• You want a polished UI for cross-team visibility
• You need commercial support
• You want the strongest safety controls (Halt button)

Getting Started: A Tool-Agnostic First Experiment

Regardless of which tool you choose, start here:

1. Pick your simplest, most reversible fault: Pod/process kill is the safest starting point.
2. Run in staging first: Always validate the experiment works as expected in a non-production environment before touching production traffic.
3. Verify observability before injection: Open dashboards, confirm metrics are flowing, confirm you can see the steady-state baseline.
4. Inject the fault, watch for 60–120 seconds: Observe how your monitoring responds. Does the system recover automatically? How long does recovery take?
5. Document and repeat: Record what happened. Fix any gaps found. Move to a more complex fault type.

Start simple, build confidence, expand scope.