Testing

Hypothesis-Driven Chaos Engineering: Writing Experiments That Actually Tell You Something

HelpMeTest

HelpMeTest

— 7 min read
Hypothesis-Driven Chaos Engineering: Writing Experiments That Actually Tell You Something

Most teams run chaos experiments the wrong way. They pick a failure mode — kill a pod, inject network latency — watch things break, declare the experiment "done," and learn nothing they couldn't have predicted. The output is a Slack message saying "yep, that broke things," which isn't useful.

Hypothesis-driven chaos engineering is different. You write down exactly what you expect to happen before you inject anything. You define what "working normally" means with measurable numbers. You specify when to abort. When the experiment finishes, you have a binary answer: your system behaved as predicted, or it didn't — and either result is valuable.

This is the scientific method applied to distributed systems. It works.

The Structure of a Hypothesis

Every chaos experiment needs a hypothesis in this form:

While [steady-state condition is met], if [we inject this failure], then [specific observable outcome] because [the mechanism we believe is in place].

The "because" clause is what separates a hypothesis from a guess. It forces you to articulate the resilience mechanism you're relying on: the circuit breaker, the retry policy, the fallback cache, the redundant instance. If you can't name the mechanism, you don't understand why you expect the system to survive.

Example of a bad hypothesis:

"The system will be fine if one database replica fails."

Example of a good hypothesis:

"While the order service maintains p99 latency below 200ms and error rate below 0.1%, if we terminate the primary database replica, then p99 latency will remain below 500ms and error rate will stay below 1% within 30 seconds of failover, because the application connection pool is configured to retry connections and the database cluster promotes a replica automatically within 15 seconds."

The good version is testable, falsifiable, and tells you exactly which part of your architecture you're betting on.

Defining Steady-State

Steady-state is the system's normal baseline behavior. You measure it before the experiment starts and use it as the comparison point during and after.

Steady-state is always expressed as measurable metrics with thresholds:

steady_state:
  metrics:
    - name: error_rate
      query: 'sum(rate(http_requests_total{status=~"5.."}[1m])) / sum(rate(http_requests_total[1m]))'
      threshold: "< 0.001"  # Less than 0.1%
      
    - name: p99_latency_ms
      query: 'histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[1m])) by (le)) * 1000'
      threshold: "< 200"
      
    - name: successful_order_rate
      query: 'sum(rate(orders_completed_total[1m]))'
      threshold: "> 95"  # At least 95 orders per minute
      
    - name: checkout_success_rate
      query: 'sum(rate(checkout_success_total[1m])) / sum(rate(checkout_attempts_total[1m]))'
      threshold: "> 0.99"

Steady-state must be verified before the experiment starts. If your system isn't in steady-state, you don't run the experiment — you're not measuring the effect of the chaos, you're measuring the superposition of chaos and an existing problem.

def verify_steady_state(prometheus_client, metrics, window_seconds=60):
    """Verify all steady-state conditions before starting experiment."""
    results = {}
    for metric in metrics:
        value = prometheus_client.query(metric['query'])
        condition = metric['threshold']
        passed = evaluate_threshold(value, condition)
        results[metric['name']] = {
            'value': value,
            'threshold': condition,
            'passed': passed
        }
    
    failed = [name for name, result in results.items() if not result['passed']]
    if failed:
        raise SteadyStateViolation(
            f"System not in steady state. Failed metrics: {failed}. "
            f"Aborting experiment."
        )
    return results

What steady-state is not: steady-state is not "all services healthy" or "no alerts firing." Those are binary pass/fail checks. Steady-state is a quantitative baseline against which you compare behavior during chaos.

Blast Radius

Blast radius is the scope of impact your experiment can have. You control it along three dimensions:

Scope — which instances, pods, nodes, or regions are affected. Start narrow: one instance, not all instances. One availability zone, not the region.

Magnitude — the severity of the failure. 10% packet loss, not 100%. One pod termination, not the entire deployment.

Duration — how long the fault persists. 60 seconds, not indefinite.

Document blast radius explicitly before running any experiment:

experiment:
  name: "order-service-dependency-failure"
  blast_radius:
    scope:
      service: "inventory-service"
      percentage_of_instances: 50  # Half the instances, not all
      environment: "production-canary"  # Not full production
    magnitude:
      failure_type: "latency_injection"
      latency_ms: 500
      jitter_ms: 100
    duration:
      experiment_duration: 120s
      observation_window_after: 300s
    
  estimated_impact:
    affected_users_percentage: 5  # Canary traffic only
    affected_transactions_per_minute: 50
    revenue_at_risk_per_minute: "$200"

The "estimated impact" section forces you to think about what happens if your hypothesis is wrong. If the experiment goes sideways, what's the actual business impact per minute? This is the number your incident commander needs to make the abort decision.

Progressive Blast Radius

Don't jump straight to production. Use a blast radius progression:

  1. Staging — full failure, all instances, no traffic limits
  2. Production canary — partial failure (50%), canary traffic only (1-5%)
  3. Production limited — partial failure, feature-flagged subset of users
  4. Production full — if all previous stages confirm the hypothesis
blast_radius_progression:
  stage_1:
    environment: staging
    scope: all_instances
    run_if: "staging_environment_healthy"
    
  stage_2:
    environment: production-canary
    scope: 50_percent_canary_instances
    run_if: "stage_1_hypothesis_confirmed"
    
  stage_3:
    environment: production
    scope: 25_percent_of_instances
    run_if: "stage_2_hypothesis_confirmed AND change_freeze_not_active"

Abort Conditions

Abort conditions are automatic or manual triggers that stop the experiment and roll back the fault injection immediately. They're not optional.

There are two categories: automatic abort (the system does it) and manual abort (a human does it).

Automatic Abort Conditions

abort_conditions:
  automatic:
    - condition: "error_rate > 0.05"  # 5% — 50x steady state threshold
      description: "Error rate exceeded 5x safety threshold"
      action: "immediate_rollback"
      
    - condition: "p99_latency_ms > 2000"
      description: "Latency exceeds 10x steady state — cascading failure likely"
      action: "immediate_rollback"
      
    - condition: "checkout_success_rate < 0.90"
      description: "10% checkout failure — revenue impact threshold"
      action: "immediate_rollback"
      
    - condition: "downstream_service_circuit_breaker_open == true AND service != 'inventory'"
      description: "Unexpected cascade to unrelated service"
      action: "immediate_rollback_and_alert"

The abort thresholds should be calibrated to be sensitive enough to catch real problems but not so tight that normal fluctuation triggers them. A good starting point: 5-10x your steady-state threshold.

Manual Abort Protocol

Define who can abort, how, and what they do:

abort_conditions:
  manual:
    authorized_roles:
      - on_call_engineer
      - chaos_experiment_owner
      - incident_commander
      
    abort_channels:
      - slack_command: "/chaos abort experiment-id"
      - pagerduty_incident_creation: "auto_pages_owner_for_manual_abort"
      - runbook_url: "https://runbooks.internal/chaos/abort"
      
    rollback_procedure:
      estimated_time: "< 30 seconds"
      automated: true
      verification: "steady_state_metrics_return_to_baseline_within_60s"

Testing Your Abort Conditions

Abort conditions are only useful if they actually work. Test them separately:

def test_abort_condition_triggers_rollback():
    """Verify that abort conditions automatically terminate experiments."""
    experiment = ChaosExperiment(
        fault=NetworkLatencyFault(target="inventory-service", latency_ms=5000),
        abort_conditions=[
            MetricThreshold("error_rate", ">", 0.05)
        ]
    )
    
    # Manually trigger the abort condition
    metrics_stub.set_metric("error_rate", 0.10)
    
    experiment.start()
    
    # Should abort within the check interval
    time.sleep(experiment.check_interval_seconds * 2)
    
    assert experiment.state == ExperimentState.ABORTED
    assert experiment.rollback_completed
    assert NetworkLatencyFault.is_active() == False

Writing the Full Experiment Document

A complete experiment document looks like this:

experiment:
  id: "EXP-042"
  title: "Order service handles inventory service degradation"
  owner: "platform-team"
  created: "2024-01-15"
  
  context:
    service: "order-service"
    dependency: "inventory-service"
    last_incident: "INC-2023-1102 — inventory latency caused order timeouts"
    motivation: "Verify circuit breaker added in v2.3.1 works under realistic conditions"
  
  hypothesis:
    statement: >
      While the order service maintains p99 latency < 200ms and error rate < 0.1%,
      if inventory-service responds with 2000ms latency for 50% of requests,
      then order-service p99 latency will remain < 800ms and error rate < 1%
      within 30 seconds of fault injection,
      because the circuit breaker will open after 5 failures and the fallback
      will return cached inventory data for non-critical checks.
    
    mechanism: "circuit-breaker + stale cache fallback"
    circuit_breaker_config:
      failure_threshold: 5
      open_duration: 30s
      fallback: "cached_inventory_response"
  
  steady_state:
    measurement_window: 5m
    metrics:
      - name: order_service_error_rate
        threshold: "< 0.001"
      - name: order_service_p99_latency_ms  
        threshold: "< 200"
      - name: order_completion_rate_per_minute
        threshold: "> 100"
  
  fault:
    type: network_latency
    target: inventory-service
    parameters:
      latency_ms: 2000
      jitter_ms: 200
      affected_percentage: 50
    duration: 120s
  
  blast_radius:
    environment: production-canary
    affected_instances: "2 of 4 inventory-service pods"
    affected_traffic_percentage: 5
    estimated_revenue_at_risk_per_minute: "$50"
  
  abort_conditions:
    - "order_service_error_rate > 0.05"
    - "order_service_p99_latency_ms > 2000"
    - "payment_service_error_rate > 0.01"  # Cascade detection
  
  rollback:
    automated: true
    estimated_time: "< 10 seconds"
    verification: "steady_state_restored_within_60s"
  
  expected_observations:
    timeline:
      - t: "0s"
        event: "Fault injection starts"
      - t: "5-15s"
        event: "Circuit breaker opens after threshold failures"
      - t: "15-30s"
        event: "Requests served from fallback, latency normalizes"
      - t: "120s"
        event: "Fault removed, circuit transitions to HALF_OPEN"
      - t: "150s"
        event: "Circuit CLOSED, normal operation resumes"
  
  success_criteria:
    - "error_rate stayed below 1% throughout experiment"
    - "p99 latency stayed below 800ms throughout experiment"
    - "circuit breaker observed to open within 30s of fault injection"
    - "system returned to steady state within 60s of fault removal"
  
  failure_criteria:
    - "error rate exceeded 1% — hypothesis falsified, investigate why circuit breaker didn't activate"
    - "cascade to payment service — blast radius exceeded, abort and investigate dependency graph"

Analyzing Results

After the experiment runs, you compare actual observations against the hypothesis. There are three outcomes:

Hypothesis confirmed. The system behaved exactly as predicted. This is evidence (not proof) that the resilience mechanism works under this specific failure condition. Document it, but don't stop — this failure mode at a higher magnitude or in a different environment might tell a different story.

Hypothesis falsified. The system didn't behave as predicted. Either the failure was worse than expected (the mechanism failed) or better (the mechanism is stronger than you thought). Both are valuable. The worse case means you have a reliability gap to fix. Document the finding, create a ticket, don't run more experiments until it's fixed.

Experiment aborted. An abort condition triggered. This is still valuable data — the system was more fragile than anticipated under this blast radius. Scale back and understand why before proceeding.

def analyze_experiment_results(experiment_id, hypothesis, observations):
    result = ExperimentResult(experiment_id=experiment_id)
    
    for criterion in hypothesis.success_criteria:
        observed_value = observations.get_metric(criterion.metric, 
                                                   window=criterion.window)
        passed = criterion.evaluate(observed_value)
        result.add_criterion_result(criterion.name, passed, observed_value)
    
    if all(r.passed for r in result.criteria):
        result.outcome = "HYPOTHESIS_CONFIRMED"
        result.recommendation = "Increase blast radius in next experiment"
    elif result.was_aborted:
        result.outcome = "EXPERIMENT_ABORTED"
        result.recommendation = "Reduce blast radius or fix identified fragility before retrying"
    else:
        result.outcome = "HYPOTHESIS_FALSIFIED"
        result.recommendation = "Create incident ticket, fix resilience gap, re-run to verify fix"
    
    return result

Common Mistakes

Running experiments during known bad periods. Don't run chaos experiments during deployments, database maintenance windows, high-traffic events, or when there are open incidents. You're measuring the effect of the fault injection, not a background noisy environment.

Not measuring the right things. If you hypothesize that a circuit breaker protects users, measure user-facing error rates, not internal service error rates. The circuit breaker might be working perfectly while users still see errors because the fallback is broken.

Skipping the "because" clause. If you don't know why you expect the system to survive, you don't understand your own architecture. That's fine — the experiment will teach you. But write down your guess, however uncertain, so you have something to compare against.

Running experiments to confirm, not to learn. If you only run experiments you're confident will pass, you're not doing chaos engineering — you're doing theater. The experiments that reveal unexpected failures are the valuable ones. Run experiments on the parts of your system you're least certain about.

Not automating abort conditions. Manual abort requires a human to notice, decide, and act. Automated abort is instant and doesn't require anyone to be awake. Both are necessary, but automated abort is the safety net that makes running experiments at reasonable blast radius possible.

Getting Started

If you've never run hypothesis-driven chaos experiments before:

  1. Pick one service with at least one resilience mechanism (circuit breaker, retry, bulkhead)
  2. Write the hypothesis: what does that mechanism protect against, and what's the measurable prediction?
  3. Define steady-state with three metrics
  4. Set up two abort conditions
  5. Run the experiment in staging first, full blast radius
  6. Compare results against hypothesis
  7. If confirmed in staging, repeat at 5% blast radius in production

The discipline of writing the hypothesis first is what transforms chaos engineering from "breaking things to see what happens" into a systematic method for building confidence in your system's resilience.

Read more

Start now free