Testing

Testing Error Budget Burn Rate Alerts: Validation Patterns

28 Sep 2026 — 7 min read

Multi-window burn rate alerting is the approach Google recommends in the SRE Workbook for SLO-based alerting. It avoids the problems with simple threshold alerting (too many false positives, or alerts that only fire hours into an incident). But it's also complex enough that bugs in the rule configuration are common — and hard to catch without testing.

This post covers how to test multi-window burn rate alerts: validating the math, writing promtool unit tests for each window, building synthetic SLO violation scenarios, and running alert rule tests in CI.

Multi-Window Burn Rate Alerting Review

The core idea from the SRE Workbook: alert when you're burning through your error budget faster than expected, using multiple time windows to balance detection speed vs. noise.

For a 99.9% availability SLO (0.1% error budget):

If you burn the entire monthly budget in 1 hour, you're burning at 720x rate (720 hours in 30 days)
If you burn it in 6 hours, that's 120x rate
If you burn it in 3 days, that's 10x rate

The multi-window approach uses a fast window (short) to detect that something is happening NOW, and a slow window (longer) to confirm it's a sustained problem, not a spike. You alert when BOTH windows exceed the burn rate threshold.

Standard configuration from the SRE Workbook:

# rules/burn-rate-alerts.yaml
groups:
  - name: slo.burn-rate
    rules:
      # Critical: burning budget at 14.4x rate
      # Fast window: 1h, Slow window: 5m
      - alert: ErrorBudgetBurnCritical
        expr: |
          (
            job:error_budget_burn_rate:1h{job="$service"} > 14.4
            and
            job:error_budget_burn_rate:5m{job="$service"} > 14.4
          )
        for: 2m
        labels:
          severity: critical
          window: "1h"
        annotations:
          summary: "High error budget burn rate on {{ $labels.job }}"
          description: "Burning at {{ $value | humanize }}x the sustainable rate"

      # High: burning at 6x rate  
      # Fast window: 6h, Slow window: 30m
      - alert: ErrorBudgetBurnHigh
        expr: |
          (
            job:error_budget_burn_rate:6h{job="$service"} > 6
            and
            job:error_budget_burn_rate:30m{job="$service"} > 6
          )
        for: 15m
        labels:
          severity: warning
          window: "6h"

      # Medium: burning at 3x rate
      # Fast window: 1d, Slow window: 2h
      - alert: ErrorBudgetBurnMedium  
        expr: |
          (
            job:error_budget_burn_rate:1d{job="$service"} > 3
            and
            job:error_budget_burn_rate:2h{job="$service"} > 3
          )
        for: 1h
        labels:
          severity: warning
          window: "1d"

      # Low: burning at 1x rate (will exhaust budget in 30 days)
      # Fast window: 3d, Slow window: 6h
      - alert: ErrorBudgetBurnLow
        expr: |
          (
            job:error_budget_burn_rate:3d{job="$service"} > 1
            and
            job:error_budget_burn_rate:6h{job="$service"} > 1
          )
        for: 3h
        labels:
          severity: info
          window: "3d"

The recording rules that these alerts depend on:

# rules/burn-rate-recording.yaml
groups:
  - name: slo.burn-rate.recording
    interval: 30s
    rules:
      # Error ratio (1 - availability)
      - record: job:error_ratio:rate1h
        expr: |
          (
            sum by (job) (rate(http_requests_total{status=~"5.."}[1h]))
            /
            sum by (job) (rate(http_requests_total[1h]))
          )

      # Burn rate = error ratio / error budget (0.001 for 99.9% SLO)
      - record: job:error_budget_burn_rate:1h
        expr: job:error_ratio:rate1h / 0.001

      - record: job:error_ratio:rate5m
        expr: |
          sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))
          /
          sum by (job) (rate(http_requests_total[5m]))

      - record: job:error_budget_burn_rate:5m
        expr: job:error_ratio:rate5m / 0.001

      # ... similar for 6h/30m, 1d/2h, 3d/6h windows

Validating the Burn Rate Math

Before writing tests, verify the math manually. For a 99.9% SLO:

# burn_rate_math.py — verify before testing
SLO = 0.999
ERROR_BUDGET = 1 - SLO  # 0.001 (0.1%)
WINDOW_HOURS = 720      # 30 days in hours

def burn_rate(error_rate):
    return error_rate / ERROR_BUDGET

def hours_to_exhaustion(error_rate):
    return WINDOW_HOURS / burn_rate(error_rate)

# Verify critical threshold: 14.4x burn rate
# At 14.4x, budget exhausts in: 720/14.4 = 50 hours
print(f"14.4x burn rate: exhaustion in {hours_to_exhaustion(0.001 * 14.4):.1f} hours")
# → 50.0 hours ≈ 2 days

# What error rate produces 14.4x burn?
# error_rate = 14.4 * error_budget = 14.4 * 0.001 = 0.0144 = 1.44%
print(f"Error rate at 14.4x: {0.001 * 14.4 * 100:.2f}%")
# → 1.44%

# At 6x burn rate: exhaustion in 720/6 = 120 hours = 5 days
print(f"6x burn rate: exhaustion in {hours_to_exhaustion(0.001 * 6):.1f} hours")
# → 120.0 hours = 5 days

# Error rate at 6x: 0.001 * 6 = 0.006 = 0.6%
print(f"Error rate at 6x: {0.001 * 6 * 100:.2f}%")
# → 0.60%

Now you know what error rates to inject in your tests.

Promtool Unit Tests for Each Window

# tests/rules/burn-rate_test.yaml
rule_files:
  - ../../rules/burn-rate-recording.yaml
  - ../../rules/burn-rate-alerts.yaml

evaluation_interval: 1m

tests:
  # ── 1h window tests ──────────────────────────────────────────────
  
  - name: "burn rate is 1.0 at baseline (0.1% error rate)"
    interval: 1m
    input_series:
      - series: 'http_requests_total{job="api", status="200"}'
        values: "0+999x70"   # 999 successful per minute for 70 minutes
      - series: 'http_requests_total{job="api", status="500"}'
        values: "0+1x70"     # 1 error per minute (0.1% error rate = 1x burn rate)
    promql_expr_test:
      - expr: job:error_budget_burn_rate:1h{job="api"}
        eval_time: 65m
        exp_samples:
          - labels: '{job="api"}'
            value: 1.0

  - name: "burn rate is 14.4 at 1.44% error rate"
    interval: 1m
    input_series:
      - series: 'http_requests_total{job="api", status="200"}'
        values: "0+9856x70"  # 9856 successful
      - series: 'http_requests_total{job="api", status="500"}'
        values: "0+144x70"   # 144 errors = 1.44% error rate
    promql_expr_test:
      - expr: job:error_budget_burn_rate:1h{job="api"}
        eval_time: 65m
        exp_samples:
          - labels: '{job="api"}'
            value: 14.4

  # ── Alert firing tests ────────────────────────────────────────────

  - name: "ErrorBudgetBurnCritical fires when both windows exceed 14.4x"
    interval: 1m
    input_series:
      - series: 'http_requests_total{job="api", status="200"}'
        values: "0+9856x70"
      - series: 'http_requests_total{job="api", status="500"}'
        values: "0+144x70"
    alert_rule_test:
      - eval_time: 68m   # after for: 2m
        alertname: ErrorBudgetBurnCritical
        exp_alerts:
          - exp_labels:
              severity: critical
              job: api
              window: "1h"

  - name: "ErrorBudgetBurnCritical does NOT fire at 10x burn rate"
    interval: 1m
    input_series:
      - series: 'http_requests_total{job="api", status="200"}'
        values: "0+9900x70"
      - series: 'http_requests_total{job="api", status="500"}'
        values: "0+100x70"  # 1% error rate = 10x burn rate (below 14.4 threshold)
    alert_rule_test:
      - eval_time: 68m
        alertname: ErrorBudgetBurnCritical
        exp_alerts: []  # should NOT fire

  - name: "ErrorBudgetBurnCritical does NOT fire on brief spike (window mismatch)"
    # This tests the dual-window property:
    # The 1h window shows elevated burn, but the 5m window doesn't
    # so the alert should NOT fire
    interval: 1m
    input_series:
      # 1.44% error rate for the last 5 minutes only
      - series: 'http_requests_total{job="api", status="200"}'
        values: "0+1000x65 +9856x5"
      - series: 'http_requests_total{job="api", status="500"}'
        values: "0+1x65 +144x5"  # high errors only at the end
    promql_expr_test:
      - expr: job:error_budget_burn_rate:1h{job="api"}
        eval_time: 70m
        exp_samples:
          - labels: '{job="api"}'
            # 1h rate: mostly good data, slightly elevated
            # Exact value depends on the mix — the point is it's below 14.4
            value: 2.153  # approximate — brief spike averaged over 1h

  # ── 6h window tests ──────────────────────────────────────────────

  - name: "ErrorBudgetBurnHigh fires after sustained 6x burn (6h window)"
    interval: 5m
    input_series:
      - series: 'http_requests_total{job="api", status="200"}'
        values: "0+9940x100"   # 99.4% success
      - series: 'http_requests_total{job="api", status="500"}'
        values: "0+60x100"     # 0.6% error rate = 6x burn rate
    alert_rule_test:
      - eval_time: 450m   # 7.5 hours — for: 15m
        alertname: ErrorBudgetBurnHigh
        exp_alerts:
          - exp_labels:
              severity: warning
              window: "6h"

  # ── Zero traffic edge cases ──────────────────────────────────────

  - name: "No alerts fire when there is no traffic"
    interval: 1m
    input_series:
      - series: 'http_requests_total{job="idle-api", status="200"}'
        values: "0 0 0 0 0 0 0 0 0 0"
    alert_rule_test:
      - eval_time: 9m
        alertname: ErrorBudgetBurnCritical
        exp_alerts: []
      - eval_time: 9m
        alertname: ErrorBudgetBurnHigh
        exp_alerts: []

Run tests:

promtool test rules tests/rules/burn-rate_test.yaml

# Expected output:
# Unit Testing: tests/rules/burn-rate_test.yaml
#   SUCCESS

Synthetic SLO Violation Scenarios

Beyond promtool unit tests, test the end-to-end alerting pipeline by generating synthetic traffic that intentionally violates SLOs:

# tools/synthetic-slo-violation.py
"""
Generate synthetic metrics to test SLO alerting end-to-end.
Uses Prometheus pushgateway to inject metric values.
"""
import time
import argparse
from prometheus_client import CollectorRegistry, Counter, push_to_gateway

def generate_traffic(pushgateway_url: str, job: str, 
                     success_rate: float, rps: int, duration_seconds: int):
    """
    Push synthetic request metrics to simulate a specific error rate.
    
    success_rate: float between 0 and 1 (e.g., 0.986 for 1.4% error rate)
    rps: requests per second to simulate
    """
    registry = CollectorRegistry()
    success_counter = Counter(
        'http_requests_total', 'HTTP requests',
        ['job', 'status'],
        registry=registry
    )
    
    success_per_interval = int(rps * success_rate)
    errors_per_interval = rps - success_per_interval
    
    print(f"Generating {rps} req/s with {(1-success_rate)*100:.2f}% error rate")
    print(f"Success: {success_per_interval}/s, Errors: {errors_per_interval}/s")
    
    total_success = 0
    total_errors = 0
    
    for second in range(duration_seconds):
        total_success += success_per_interval
        total_errors += errors_per_interval
        
        success_counter.labels(job=job, status='200')._value.set(total_success)
        success_counter.labels(job=job, status='500')._value.set(total_errors)
        
        push_to_gateway(pushgateway_url, job=job, registry=registry)
        
        if second % 30 == 0:
            print(f"[{second}s] Total: {total_success + total_errors} requests, "
                  f"error rate: {total_errors/(total_success+total_errors)*100:.2f}%")
        
        time.sleep(1)

if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('--pushgateway', default='http://localhost:9091')
    parser.add_argument('--job', default='test-api')
    parser.add_argument('--success-rate', type=float, default=0.986)  # 1.4% errors = 14x burn
    parser.add_argument('--rps', type=int, default=100)
    parser.add_argument('--duration', type=int, default=300)  # 5 minutes
    args = parser.parse_args()
    
    generate_traffic(
        args.pushgateway, args.job, 
        args.success_rate, args.rps, args.duration
    )

# Run synthetic violation to test critical alert
python tools/synthetic-slo-violation.py \
  --job test-api \
  --success-rate 0.986 \  # 1.4% error rate = 14x burn
  --rps 100 \
  --duration 600 &        # run for 10 minutes

# Monitor for alert
VIOLATION_PID=$!
sleep 180  # wait 3 minutes for fast window to populate

# Check Alertmanager for firing alerts
curl -s http://localhost:9093/api/v2/alerts | \
  jq '.[] | select(.labels.alertname == "ErrorBudgetBurnCritical")'

wait $VIOLATION_PID

CI-Based Alerting Rule Testing

# .github/workflows/alert-rules-test.yml
name: Test Alerting Rules

on:
  pull_request:
    paths:
      - 'rules/**'
      - 'tests/rules/**'

jobs:
  promtool-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Download promtool
        run: |
          VERSION=2.48.0
          wget -q "https://github.com/prometheus/prometheus/releases/download/v${VERSION}/prometheus-${VERSION}.linux-amd64.tar.gz"
          tar xf "prometheus-${VERSION}.linux-amd64.tar.gz"
          sudo install "prometheus-${VERSION}.linux-amd64/promtool" /usr/local/bin/promtool

      - name: Validate rule syntax
        run: |
          find rules/ -name "*.yaml" -exec promtool check rules {} \;

      - name: Run unit tests
        run: |
          find tests/rules/ -name "*_test.yaml" | while read f; do
            echo "Testing: $f"
            promtool test rules "$f"
          done

      - name: Verify burn rate math
        run: python tools/verify-burn-rate-math.py

      - name: Test Alertmanager routing
        run: bash ci/test-alertmanager-routing.sh

Common Mistakes the Tests Catch

Wrong error budget divisor The burn rate formula is error_ratio / error_budget. Using 0.01 instead of 0.001 as the divisor for a 99.9% SLO would produce burn rate values that are 10x too low — your critical alert would never fire until the error rate was 14.4%, not 1.44%. The recording rule unit test catches this.

Incorrect for duration A critical alert with for: 30m means it takes 30 minutes to fire after a major outage starts. The alert timing test (checking that the alert fires at eval_time: 2m but not at eval_time: 0m) catches this.

Single-window instead of dual-window If you accidentally write a single-window alert instead of requiring both windows to be elevated, you'll get false positive alerts on brief spikes. The "brief spike" test case catches this.

Zero-traffic false positives A division-by-zero in the burn rate formula can produce NaN or Inf values that falsely trigger alerts during low-traffic periods. The zero-traffic test case catches this.

Each of these is a real class of bug that has caused missed incidents or alert fatigue in production. The test suite for your alerting rules is worth at least as much effort as the tests for your application code.