Observability

Chaos Engineering with Observability: Validating Experiments with Traces and Metrics

HelpMeTest

HelpMeTest

— 7 min read
Chaos Engineering with Observability: Validating Experiments with Traces and Metrics

Chaos engineering without observability is guesswork. You inject a fault, something breaks (or doesn't), and you're left reading error rates and hoping you measured the right thing. When you combine chaos experiments with trace-based and metric-based assertions, each experiment becomes a falsifiable test: "under this failure condition, these specific metrics stay within bounds and these trace patterns appear." This post shows how to build that.

The Steady-State Hypothesis with Telemetry

The foundational concept in chaos engineering is the steady-state hypothesis — a quantified statement about how your system behaves normally. Every chaos experiment is structured as:

  1. Verify steady state holds before the experiment.
  2. Inject the fault.
  3. Verify steady state still holds (or measure how it degrades).
  4. Remove the fault, verify steady state recovers.

The mistake most teams make is defining steady state as "the service is up." That's not measurable enough. Define it with actual telemetry:

# steady-state-hypothesis.yaml
hypothesis:
  title: "Order service maintains SLO under dependency failure"
  probes:
    - name: "error rate below 1%"
      type: prometheus
      query: |
        rate(orders_created_total{status="error"}[1m])
        /
        rate(orders_created_total[1m])
      threshold: 0.01
      
    - name: "p99 latency below 2s"
      type: prometheus
      query: |
        histogram_quantile(0.99, 
          rate(http_request_duration_seconds_bucket{handler="/api/orders"}[5m]))
      threshold: 2.0

    - name: "no orphaned spans in traces"
      type: jaeger
      service: order-service
      operation: POST /api/orders
      assertion: all_spans_have_parent

    - name: "payment circuit breaker telemetry is present"
      type: jaeger
      service: order-service
      assertion: span_attribute_exists
      span_name: charge_payment
      attribute: circuit_breaker.state

Setting Up Chaos Mesh with OpenTelemetry

Chaos Mesh is a Kubernetes-native chaos engineering platform. Pairing it with OTel gives you precise before/after telemetry for each experiment.

Installation

# Install Chaos Mesh via Helm
helm repo add chaos-mesh https://charts.chaos-mesh.org
helm install chaos-mesh chaos-mesh/chaos-mesh \
  --namespace=chaos-testing \
  --create-namespace \
  --version 2.6.3

# Verify
kubectl get pods -n chaos-testing

Annotating Namespaces for Chaos Injection

# namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: staging
  labels:
    chaos-mesh.org/inject: enabled

Experiment Definitions

Network delay injection:

# chaos/payment-service-latency.yaml
apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: payment-service-latency
  namespace: staging
spec:
  action: delay
  mode: all
  selector:
    namespaces:
      - staging
    labelSelectors:
      app: payment-service
  delay:
    latency: "500ms"
    correlation: "25"
    jitter: "100ms"
  direction: both
  duration: "2m"

Pod failure injection:

# chaos/inventory-pod-failure.yaml
apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
  name: inventory-pod-failure
  namespace: staging
spec:
  action: pod-failure
  mode: one
  selector:
    namespaces:
      - staging
    labelSelectors:
      app: inventory-service
  duration: "90s"
  gracePeriod: 0

DNS failure for external dependency:

# chaos/stripe-dns-failure.yaml
apiVersion: chaos-mesh.org/v1alpha1
kind: DNSChaos
metadata:
  name: stripe-dns-failure
  namespace: staging
spec:
  action: error
  mode: all
  selector:
    namespaces:
      - staging
    labelSelectors:
      app: payment-service
  patterns:
    - "api.stripe.com"
  duration: "60s"

Writing Chaos Tests with Telemetry Assertions

The test structure follows the steady-state → inject → assert → recover pattern, with telemetry checks at each phase:

# chaos_test_framework.py
import subprocess
import time
import requests
from dataclasses import dataclass
from typing import Callable, Optional
from prometheus_client_api import PrometheusClient
from jaeger_client import JaegerClient

prometheus = PrometheusClient("http://prometheus:9090")
jaeger = JaegerClient("http://jaeger:16686")

@dataclass
class SteadyStateProbe:
    name: str
    check: Callable[[], bool]
    description: str

class ChaosExperiment:
    def __init__(self, name: str, manifest_path: str):
        self.name = name
        self.manifest_path = manifest_path
        self.probes: list[SteadyStateProbe] = []

    def add_probe(self, probe: SteadyStateProbe):
        self.probes.append(probe)
        return self

    def verify_steady_state(self, phase: str):
        failures = []
        for probe in self.probes:
            if not probe.check():
                failures.append(f"{probe.name}: {probe.description}")
        if failures:
            raise AssertionError(
                f"Steady state FAILED at {phase}:\n" + "\n".join(failures)
            )

    def inject(self):
        subprocess.run(
            ["kubectl", "apply", "-f", self.manifest_path],
            check=True, capture_output=True
        )

    def remove(self):
        subprocess.run(
            ["kubectl", "delete", "-f", self.manifest_path, "--ignore-not-found"],
            check=True, capture_output=True
        )

    def run(self, load_generator: Callable, fault_duration: int = 120,
            recovery_timeout: int = 60):
        print(f"[{self.name}] Verifying steady state before fault...")
        self.verify_steady_state("pre-fault")

        print(f"[{self.name}] Injecting fault...")
        self.inject()

        print(f"[{self.name}] Running load during fault ({fault_duration}s)...")
        load_generator(duration=fault_duration)

        print(f"[{self.name}] Verifying steady state under fault...")
        self.verify_steady_state("during-fault")

        print(f"[{self.name}] Removing fault...")
        self.remove()

        print(f"[{self.name}] Waiting for recovery ({recovery_timeout}s)...")
        time.sleep(recovery_timeout)

        print(f"[{self.name}] Verifying steady state after recovery...")
        self.verify_steady_state("post-recovery")

        print(f"[{self.name}] PASSED")

Prometheus Probes for Steady-State

# prometheus_client_api.py
import requests
import time

class PrometheusClient:
    def __init__(self, base_url: str):
        self.base_url = base_url

    def query(self, promql: str) -> float:
        resp = requests.get(
            f"{self.base_url}/api/v1/query",
            params={"query": promql}
        )
        resp.raise_for_status()
        data = resp.json()
        results = data.get("data", {}).get("result", [])
        if not results:
            return 0.0
        return float(results[0]["value"][1])

    def error_rate(self, service: str, window: str = "2m") -> float:
        return self.query(f"""
            rate(http_requests_total{{service="{service}",status=~"5.."}}[{window}])
            /
            rate(http_requests_total{{service="{service}"}}[{window}])
        """)

    def p99_latency(self, service: str, handler: str, window: str = "2m") -> float:
        return self.query(f"""
            histogram_quantile(0.99, 
                rate(http_request_duration_seconds_bucket{{
                    service="{service}",
                    handler="{handler}"
                }}[{window}])
            )
        """)

    def success_rate(self, counter_name: str, service: str, window: str = "2m") -> float:
        return self.query(f"""
            rate({counter_name}{{service="{service}",status="success"}}[{window}])
            /
            rate({counter_name}{{service="{service}"}}[{window}])
        """)
# test_payment_service_chaos.py
import pytest
from chaos_test_framework import ChaosExperiment, SteadyStateProbe

def make_order_load(api_url: str):
    def load_generator(duration: int):
        import threading
        import time

        stop_event = threading.Event()
        results = {"success": 0, "error": 0}

        def worker():
            while not stop_event.is_set():
                try:
                    resp = requests.post(f"{api_url}/api/orders",
                                         json={"item": "widget", "qty": 1},
                                         timeout=5)
                    if resp.status_code == 201:
                        results["success"] += 1
                    else:
                        results["error"] += 1
                except Exception:
                    results["error"] += 1
                time.sleep(0.1)

        threads = [threading.Thread(target=worker) for _ in range(5)]
        for t in threads:
            t.start()
        time.sleep(duration)
        stop_event.set()
        for t in threads:
            t.join()
        return results

    return load_generator

def test_order_service_resilient_to_payment_latency():
    experiment = ChaosExperiment(
        name="payment-service-500ms-latency",
        manifest_path="chaos/payment-service-latency.yaml"
    )

    experiment.add_probe(SteadyStateProbe(
        name="order error rate < 5%",
        check=lambda: prometheus.error_rate("order-service") < 0.05,
        description="More than 5% of orders are failing"
    ))

    experiment.add_probe(SteadyStateProbe(
        name="order p99 latency < 3s",
        check=lambda: prometheus.p99_latency("order-service", "/api/orders") < 3.0,
        description="p99 order latency exceeds 3s SLO"
    ))

    experiment.add_probe(SteadyStateProbe(
        name="circuit breaker activates under load",
        check=lambda: prometheus.query(
            'circuit_breaker_state{service="order-service",dependency="payment"}'
        ) >= 0,  # metric exists
        description="Circuit breaker metric not being emitted"
    ))

    experiment.run(
        load_generator=make_order_load("http://order-service:8080"),
        fault_duration=120,
        recovery_timeout=30
    )

Trace-Based Assertions for Chaos Experiments

While metrics tell you aggregates (error rate, p99), traces tell you the specific failure path. Use Jaeger queries to validate that resilience patterns are behaving correctly during faults:

# test_chaos_trace_assertions.py
from jaeger_client import JaegerClient
import time

jaeger = JaegerClient("http://jaeger:16686")

def test_circuit_breaker_spans_appear_during_payment_fault(chaos_context):
    """During payment service failure, circuit breaker spans should appear."""
    with chaos_context("chaos/payment-pod-failure.yaml"):
        # Send a few requests to trip the circuit
        for _ in range(10):
            requests.post("http://order-service/api/orders",
                          json={"item": "widget"})
            time.sleep(0.2)

        # Give traces time to flush
        time.sleep(2)

        traces = jaeger.get_traces("order-service", "POST /api/orders",
                                   limit=20, lookback="5m")

        # At least some traces should show circuit breaker open
        cb_open_traces = [
            t for t in traces
            if any(
                s.tags.get("circuit_breaker.state") == "open"
                for s in t.spans
            )
        ]
        assert len(cb_open_traces) > 0, (
            "No traces show circuit breaker in open state during payment failure"
        )

        # Spans after circuit opens should NOT have payment child spans
        for trace in cb_open_traces:
            payment_spans = [s for s in trace.spans
                             if s.operation_name == "charge_payment"]
            assert len(payment_spans) == 0, (
                "Payment spans found in trace despite circuit breaker being open — "
                "circuit breaker is not preventing calls"
            )

def test_fallback_span_appears_when_inventory_down(chaos_context):
    """When inventory service is down, fallback logic should produce a span."""
    with chaos_context("chaos/inventory-pod-failure.yaml"):
        time.sleep(5)  # let the pod die

        requests.get("http://product-service/api/products/123")
        time.sleep(2)

        trace = jaeger.wait_for_trace("product-service", "GET /api/products/{id}",
                                      timeout=10)

        fallback_span = trace.span_by_operation("get_product.cache_fallback")
        assert fallback_span is not None, (
            "Fallback span not found — fallback logic may not be executing "
            "when inventory service is unavailable"
        )
        assert fallback_span.tags.get("fallback.reason") == "inventory_service_unavailable"

Steady-State Recovery Validation

Recovery is as important as resilience during the fault. After removing the fault, validate that metrics return to baseline and traces return to normal structure:

def test_metrics_recover_after_fault_removal():
    # Measure baseline
    baseline_error_rate = prometheus.error_rate("order-service", window="5m")
    baseline_p99 = prometheus.p99_latency("order-service", "/api/orders", window="5m")

    # Inject fault
    subprocess.run(["kubectl", "apply", "-f", "chaos/payment-service-latency.yaml"],
                   check=True)
    time.sleep(120)  # let fault run

    # Measure under fault
    fault_error_rate = prometheus.error_rate("order-service", window="2m")
    fault_p99 = prometheus.p99_latency("order-service", "/api/orders", window="2m")

    # Remove fault
    subprocess.run(["kubectl", "delete", "-f", "chaos/payment-service-latency.yaml"],
                   check=True)

    # Poll for recovery
    recovery_deadline = time.time() + 120
    recovered = False
    while time.time() < recovery_deadline:
        current_error_rate = prometheus.error_rate("order-service", window="2m")
        current_p99 = prometheus.p99_latency("order-service", "/api/orders", window="2m")

        if (current_error_rate <= baseline_error_rate * 1.1 and
                current_p99 <= baseline_p99 * 1.2):
            recovered = True
            break
        time.sleep(5)

    assert recovered, (
        f"Service did not recover within 120s. "
        f"Baseline error rate: {baseline_error_rate:.3f}, "
        f"Current: {prometheus.error_rate('order-service', window='2m'):.3f}"
    )

Chaos Mesh + OTel: Complete Workflow

# .github/workflows/chaos-tests.yml
name: Chaos Tests

on:
  schedule:
    - cron: '0 2 * * 1'  # weekly, Monday 2am
  workflow_dispatch:

jobs:
  chaos:
    runs-on: ubuntu-latest
    environment: staging
    steps:
      - uses: actions/checkout@v4

      - name: Configure kubectl
        uses: azure/k8s-set-context@v3
        with:
          kubeconfig: ${{ secrets.KUBECONFIG_STAGING }}

      - name: Verify Chaos Mesh is installed
        run: kubectl get pods -n chaos-testing

      - name: Run chaos experiments
        run: |
          pytest tests/chaos/ -v \
            --junit-xml=chaos-results.xml \
            -k "not slow"

      - name: Export traces on failure
        if: failure()
        run: |
          kubectl port-forward svc/jaeger-query 16686:16686 -n monitoring &
          sleep 3
          curl -s "http://localhost:16686/api/traces?service=order-service&limit=100&lookback=2h" \
            > chaos-traces.json

      - uses: actions/upload-artifact@v3
        if: always()
        with:
          name: chaos-evidence
          path: |
            chaos-results.xml
            chaos-traces.json

Chaos Dashboard: Correlating Faults with Telemetry

Add Chaos Mesh annotations to your Grafana dashboards so fault injection periods are visible alongside metric changes:

# grafana_annotations.py
import requests

def annotate_chaos_start(grafana_url: str, api_key: str, experiment_name: str):
    requests.post(
        f"{grafana_url}/api/annotations",
        headers={"Authorization": f"Bearer {api_key}"},
        json={
            "text": f"Chaos: {experiment_name} started",
            "tags": ["chaos", "fault-injection"],
            "time": int(time.time() * 1000)
        }
    )

def annotate_chaos_end(grafana_url: str, api_key: str, experiment_name: str):
    requests.post(
        f"{grafana_url}/api/annotations",
        headers={"Authorization": f"Bearer {api_key}"},
        json={
            "text": f"Chaos: {experiment_name} ended",
            "tags": ["chaos", "recovery"],
            "time": int(time.time() * 1000)
        }
    )

This gives you a visual correlation between "fault injected at T" and "error rate spiked at T+5s" in Grafana — the telemetry becomes the experiment record.

Conclusion

Chaos engineering earns its value when experiments are falsifiable — when you can say "under this fault condition, these specific metrics stayed within bounds, and these trace patterns confirmed the resilience mechanism fired." OpenTelemetry traces tell you the structural story (circuit breaker opened, fallback span appeared, payment calls stopped); Prometheus metrics tell you the aggregate story (error rate held below 5%, p99 recovered within 30 seconds). Together, they turn a chaos experiment from "we poked it and it seemed okay" into a reproducible, automatable test that runs on a schedule and produces evidence. Start with your single most important dependency, write a steady-state hypothesis with three metric probes, inject a pod failure, and see what breaks.

Read more

Start now free