Chaos Engineering Metrics: How to Measure Blast Radius and MTTR

Why Chaos Engineering Needs Its Own Metrics

Chaos engineering experiments generate observations. Without a framework for what to measure and how to interpret the measurements, those observations don't translate into improvements.

The four primary metrics for chaos engineering:

1. Blast radius — scope of impact during an experiment
2. MTTR (Mean Time to Recovery) — how long until the system returned to steady state
3. Steady-state deviation — how far key metrics drifted during the experiment
4. Detection time — how long until monitoring caught the failure

Each metric answers a different question about your system's resilience.

Blast Radius

Definition

Blast radius is the scope of impact of a failure — which services, users, or operations were affected and to what degree.

In chaos engineering, blast radius has two dimensions:

• Planned blast radius: The maximum scope of impact the experiment was designed to have
• Actual blast radius: What was actually affected

If your actual blast radius exceeds your planned blast radius, the experiment escaped its containment. That's important data — it means your isolation assumptions were wrong.

How to Measure

Service scope: Which services reported elevated error rates during the experiment window?

Blast radius (service) = number of services with >1% error rate increase / total services

User scope: What percentage of users experienced degraded service?

Blast radius (users) = users with degraded requests during window / total active users

Revenue scope: What was the estimated revenue impact during the experiment?

Blast radius (revenue) = degraded transaction rate × average order value × experiment duration

Setting Blast Radius Limits

Define blast radius limits in your experiment runbook before running:

Planned blast radius:
- Service scope: payment-service only (1 service)
- User scope: ≤5% of active users (traffic from affected AZ)
- Revenue scope: ≤$100 estimated impact
- Abort if: error rate exceeds 10% in any other service

If during the experiment you observe error rates rising in services outside the planned scope, abort.

MTTR (Mean Time to Recovery)

Definition

MTTR measures how long it takes the system to return to steady state after a fault is introduced. In chaos engineering:

MTTR = time when steady state is restored - time when fault was injected

Note: this is different from incident MTTR, which typically starts when the incident is detected (not when the fault occurred). Chaos engineering MTTR starts at injection — including the time before detection.

Components of MTTR

Breaking MTTR into components reveals where to optimize:

MTTR = detection lag + diagnosis time + remediation time + verification time

• Detection lag: Time from fault injection to first alert firing. If this is 5 minutes, your system was broken for 5 minutes before anyone knew.
• Diagnosis time: Time from alert to understanding root cause. High diagnosis time indicates poor observability.
• Remediation time: Time from diagnosis to fix applied (or automatic recovery).
• Verification time: Time from fix applied to confirming system is healthy.

For automated recovery (like Kubernetes pod rescheduling), MTTR should be seconds to minutes. For manually-remediated failures, it can be much longer.

Measuring MTTR from Prometheus

If you're using Prometheus and Grafana:

# Time series of error rate for a service
rate(http_requests_total{status=~"5..", job="web-api"}[1m])
  /
rate(http_requests_total{job="web-api"}[1m])

During a chaos experiment:

1. Record the timestamp when fault is injected (T0)
2. Find the timestamp when error rate returns to baseline ± noise threshold (T1)
3. MTTR = T1 - T0

If the error rate never returns to baseline within the experiment window, that's a failure mode worth investigating separately.

MTTR Targets by Experiment Type

If measured MTTR exceeds target, the gap is a remediation task — not just an observation.

Steady-State Deviation

Definition

Steady-state deviation measures how far key metrics moved from their pre-experiment baseline during the experiment window.

This is the primary quality metric for a chaos experiment. A good experiment has a clear hypothesis about expected deviation: "We expect error rate to spike to 2–5% for 30 seconds while Kubernetes reschedules the killed pod, then return to < 0.1%."

Metrics to Track

The specific metrics depend on your application, but common steady-state indicators:

How to Measure Steady-State Deviation

Before the experiment: Record the baseline value of each metric over a 5-minute window.

During the experiment: Record the peak deviation from baseline.

After the experiment: Record how long until each metric returned to baseline ± 10%.

Deviation score = (peak_value - baseline_value) / baseline_value × 100
Recovery time = time for metric to return to baseline ± 10%

Example:

• Baseline error rate: 0.05%
• Peak error rate during experiment: 3.2%
• Deviation score: (3.2 - 0.05) / 0.05 × 100 = 6,300%
• Recovery time: 47 seconds

The deviation score of 6,300% sounds alarming but is expected for a pod kill — the question is whether the recovery time (47 seconds) is within SLA.

Detection Time

Definition

Detection time is the gap between fault injection and the first observable signal in your monitoring system (alert, dashboard spike, log entry).

Detection time = time of first alert - time of fault injection

Why Detection Time Matters

A system with short MTTR but long detection time is still exposed. If a fault goes undetected for 5 minutes, 5 minutes of users experience degraded service before anyone starts investigating.

Chaos experiments are a direct test of your observability stack. If you inject a fault that should trigger an alert within 30 seconds, and the alert fires 8 minutes later, that's an observability gap — not just a resilience gap.

Measurement

During each chaos experiment, record:

1. Exact timestamp of fault injection
2. Timestamp when the first alert fired (from your alerting tool's notification log)
3. Timestamp when a human first noticed on dashboards (if different from alert)

Detection time = alert timestamp - injection timestamp

Detection Time Targets

If detection time exceeds targets, tune alert thresholds, reduce scrape intervals, or add more granular probes.

Tracking Metrics Over Time

Per-experiment metrics are data points. Trends tell you whether your resilience program is working.

Resilience Scorecard

Track these monthly:

If MTTR isn't improving over time, either the experiments aren't revealing real weaknesses, or discovered weaknesses aren't being remediated.

Linking Chaos Metrics to Business Metrics

The ultimate validation of a chaos engineering program: does it correlate with reduced production incidents and improved uptime?

Track side by side:

• Experiments run per quarter
• Production incidents per quarter (unplanned)
• MTTR for production incidents

A mature chaos engineering program should show a negative correlation between experiments run and production incidents. If both are rising, you're not fixing the weaknesses you're finding.

Tooling for Chaos Metrics

The simplest approach: a shared spreadsheet where each chaos experiment is a row, with columns for planned/actual blast radius, MTTR, peak deviation, and detection time. Populate it manually after each experiment. Review it monthly.

Measurement doesn't need to be automated to be useful — it needs to be consistent.