Chaos Engineering in Kubernetes: Resilience Testing for Cloud-Native Apps

Why Kubernetes Needs Its Own Chaos Approach

A Kubernetes cluster isn't a static environment. Under normal operation, Kubernetes constantly:

• Kills and reschedules pods when nodes are memory-pressured
• Drains nodes during upgrades and maintenance
• Evicts pods when resource limits are exceeded
• Fails health checks and routes traffic away from unhealthy pods
• Scales deployments up and down in response to load

These events happen in production whether you test for them or not. Chaos engineering in Kubernetes means making these events happen on purpose, in controlled circumstances, so you discover how your workloads respond before a production incident teaches you.

Failure Modes Specific to Kubernetes

Pod Scheduling Failures

If a pod's resource requests can't be satisfied, it stays in Pending state. Common causes: resource requests too high, no node has capacity, node affinity rules can't be satisfied.

Test: Create a pod with resource requests that can't be scheduled. Verify your monitoring alerts and that your deployment doesn't silently shed replicas.

Readiness Probe Failures

Readiness probes tell Kubernetes whether a pod is ready to receive traffic. A misconfigured probe (wrong path, too-short timeout, wrong port) causes Kubernetes to route traffic away from healthy pods or never route traffic to them.

Test: Temporarily fail the readiness probe endpoint. Verify that Kubernetes removes the pod from the load balancer but doesn't kill it. Verify that traffic is correctly distributed to remaining healthy pods.

Resource Exhaustion and Eviction

Pods running near their memory limits get evicted first during node memory pressure. If your limits are too low, a legitimate traffic spike evicts your pods. If limits are too high, you waste cluster capacity.

Test: Inject memory pressure into a pod and observe eviction behavior. Verify that your deployment recovers, that evicted pods are rescheduled, and that the cluster doesn't cascade.

Network Policy Misconfiguration

NetworkPolicy resources define which pods can talk to which. A missing or incorrect policy silently blocks traffic — no error, just connection refused. This is especially common when new services are added to a namespace that has default-deny policies.

Test: Apply a restrictive NetworkPolicy to a namespace, then verify that service-to-service calls succeed (or fail gracefully with clear error messages).

Setting Up Chaos Mesh for Kubernetes

Chaos Mesh is the most widely used Kubernetes-native chaos tool. Here's a working setup:

# Install Chaos Mesh via Helm
helm repo add chaos-mesh https://charts.chaos-mesh.org
helm repo update

helm install chaos-mesh chaos-mesh/chaos-mesh \
  --namespace chaos-testing \
  --create-namespace \
  --set chaosDaemon.runtime=containerd \
  --set chaosDaemon.socketPath=/run/containerd/containerd.sock

Verify installation:

kubectl get pods -n chaos-testing
# Should show controller-manager, chaos-daemon, chaos-dashboard pods

Core Kubernetes Chaos Experiments

1. Pod Kill

The simplest and safest starting point. Terminate a pod and verify that Kubernetes reschedules it and that traffic is redistributed.

apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
  name: pod-kill-experiment
  namespace: chaos-testing
spec:
  action: pod-kill
  mode: one
  selector:
    namespaces:
      - production
    labelSelectors:
      app: web-api
  duration: "1m"

What to observe:

• Pod termination appears in kubectl get events -n production
• New pod starts within your rollout window (typically 30–60 seconds for a healthy deployment)
• Load balancer stops routing to the killed pod immediately
• No user-visible errors (or if there are, quantify how many)

2. Pod Failure (without termination)

Puts the pod in a failure state without terminating it — simulates a process crash or health check failure.

apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
  name: pod-failure-experiment
  namespace: chaos-testing
spec:
  action: pod-failure
  mode: fixed-percent
  value: "25"
  selector:
    namespaces:
      - production
    labelSelectors:
      app: worker-service
  duration: "5m"

With mode: fixed-percent and value: "25", this affects 25% of matching pods — useful for testing graceful degradation at reduced capacity.

3. Network Delay

Injects latency into network calls from or to a specific pod. Tests timeout handling, circuit breakers, and user-visible latency degradation.

apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: api-latency-experiment
  namespace: chaos-testing
spec:
  action: delay
  mode: all
  selector:
    namespaces:
      - production
    labelSelectors:
      app: payment-service
  delay:
    latency: "300ms"
    jitter: "50ms"
    correlation: "25"
  duration: "10m"
  direction: to
  target:
    selector:
      namespaces:
        - production
      labelSelectors:
        app: database-proxy
    mode: all

This adds 300ms ± 50ms latency specifically on traffic from payment-service to database-proxy. Other traffic is unaffected.

What to observe:

• Does payment-service timeout gracefully or hang?
• Does latency propagate upstream (caller of payment-service also slows down)?
• Are timeout values configured correctly?

4. Network Partition

Simulates a complete network split — one service can't reach another. Tests circuit breaker behavior and graceful degradation.

apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: partition-experiment
  namespace: chaos-testing
spec:
  action: partition
  mode: all
  selector:
    namespaces:
      - production
    labelSelectors:
      app: feature-flag-service
  direction: both
  duration: "3m"

What to observe:

• Does your application fail open or fail closed when the feature flag service is unreachable?
• Do circuit breakers open and prevent cascading requests?
• Do error rates spike and then stabilize as circuit breakers engage?

5. Memory Stress

Allocates memory inside a pod to push it toward its memory limit, triggering Kubernetes eviction or OOM.

apiVersion: chaos-mesh.org/v1alpha1
kind: StressChaos
metadata:
  name: memory-stress-experiment
  namespace: chaos-testing
spec:
  mode: one
  selector:
    namespaces:
      - production
    labelSelectors:
      app: report-generator
  stressors:
    memory:
      workers: 2
      size: "1GB"
  duration: "5m"

What to observe:

• Does Kubernetes evict the pod or OOM-kill the container?
• Is the memory limit set correctly for expected peak usage?
• Does the deployment recover after eviction?

6. Node Drain

For simpler node experiments, use kubectl drain manually:

# Cordon node (prevent new pods from being scheduled)
kubectl cordon node-name

# Drain node (evict all non-daemonset pods)
kubectl drain node-name --ignore-daemonsets --delete-emptydir-data

# Restore
kubectl uncordon node-name

What to observe:

• Are all pods rescheduled to remaining nodes within your SLA?
• Does the cluster have enough capacity to absorb one node's worth of pods?
• Are any pods stuck in Pending after the drain?

Essential Prerequisites for Kubernetes Chaos

Pod Disruption Budgets

Before running any pod chaos, configure PDBs to limit how many pods can be disrupted simultaneously:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web-api-pdb
  namespace: production
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: web-api

With minAvailable: 2, Kubernetes will always maintain at least 2 healthy web-api pods, regardless of what chaos experiments or rolling updates try to do.

Namespace Scoping

Always scope chaos experiments to specific namespaces. Chaos Mesh selectors accept namespace filters — use them:

selector:
  namespaces:
    - production  # Only target production namespace
  labelSelectors:
    app: web-api

Never use namespaces: [] (all namespaces) in early experiments.

Observability Requirements

You need to see the impact of chaos experiments in real time. At minimum:

• Pod health: kubectl get pods -n production -w
• Resource metrics: Kubernetes Dashboard or Prometheus/Grafana showing CPU, memory, request rates
• Application metrics: Request success rate, error rate, latency by percentile
• Logs: Structured logs from the affected service, accessible in near-real-time

Chaos Experiment Schedule

A sustainable cadence for a production Kubernetes service:

Continuous automated experiments (Chaos Monkey-style pod killing in production) are appropriate once you've validated each experiment type in staging and have established recovery SLOs.