SRE

Error Budget Burn Rate Alerting: The Google SRE Approach to SLO Monitoring

28 Sep 2026 — 6 min read

Most SLO monitoring alerts on the wrong thing. "Alert when error rate exceeds 1%" fires when you're already failing — and misses slow degradation that doesn't cross a threshold for hours. Google's SRE book introduced a better approach: alert on burn rate, not raw metrics.

This guide explains error budget burn rate alerting, how to implement it, and how to test that your alerts fire correctly.

What Is Error Budget?

If your SLO is 99.9% availability over 30 days:

Total requests per month: 2,000,000
Allowed failures: 2,000,000 × 0.001 = 2,000 failures
That's your error budget: 2,000 bad requests

When you've used 2,000 bad requests, your error budget is exhausted and you're violating your SLO.

The Problem with Simple Threshold Alerts

Simple alert: "fire when error rate > 1% for 5 minutes"

Problems:

Fast burn, no alert: 100% error rate for 30 minutes exhausts your entire monthly budget — but only fires if it lasts > 5 minutes
Slow bleed, no alert: 0.5% error rate (well below 1% threshold) for 30 days exhausts your budget slowly — never triggers

Neither scenario gets caught until it's too late.

Burn Rate: The Right Abstraction

Burn rate = how fast you're consuming your error budget, relative to the budget replenishment rate.

Burn rate 1 = consuming budget at exactly the rate that exhausts it by month end
Burn rate 10 = consuming 10× faster — exhausts budget in 3 days instead of 30

At burn rate 36, you exhaust your monthly budget in 20 hours.

Formula:

burn_rate = current_error_rate / (1 - SLO_target)

For SLO = 99.9% (error budget = 0.1%):

Current error rate 0.5% → burn rate = 0.5 / 0.1 = 5
Current error rate 5% → burn rate = 5 / 0.1 = 50

Multi-Window Burn Rate Alerting

Google's approach uses two complementary windows for each alert:

Short window: detects current burn rate (high sensitivity, some false positives)
Long window: confirms the burn is sustained (reduces false positives)

Both windows must exceed the burn rate threshold to alert.

# prometheus/rules/error-budget.yaml
groups:
  - name: error-budget-burn-rate
    rules:
      # Record: error rate per service
      - record: job:error_rate:rate5m
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m])) by (job)
          /
          sum(rate(http_requests_total[5m])) by (job)
      
      - record: job:error_rate:rate1h
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[1h])) by (job)
          /
          sum(rate(http_requests_total[1h])) by (job)
      
      - record: job:error_rate:rate6h
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[6h])) by (job)
          /
          sum(rate(http_requests_total[6h])) by (job)
      
      # Page immediately: burn rate ≥ 14.4 (budget exhausted in 2h)
      # Requires BOTH windows to exceed threshold
      - alert: ErrorBudgetFastBurn
        expr: |
          (job:error_rate:rate5m{job="api"} > (14.4 * 0.001))
          and
          (job:error_rate:rate1h{job="api"} > (14.4 * 0.001))
        for: 2m
        labels:
          severity: page
          team: backend
        annotations:
          summary: "Fast budget burn: will exhaust monthly budget in ~2h"
          description: |
            Current 5m error rate: {{ $value | humanizePercentage }}
            Burn rate: {{ printf "%.1f" (divf $value 0.001) }}x
            Action: investigate immediately, consider rollback
      
      # Ticket: burn rate ≥ 6 (budget exhausted in 5 days)
      - alert: ErrorBudgetMediumBurn
        expr: |
          (job:error_rate:rate5m{job="api"} > (6 * 0.001))
          and
          (job:error_rate:rate1h{job="api"} > (6 * 0.001))
        for: 15m
        labels:
          severity: ticket
          team: backend
        annotations:
          summary: "Medium budget burn: will exhaust monthly budget in ~5 days"
          description: |
            Burn rate: {{ printf "%.1f" (divf $value 0.001) }}x
            Action: investigate this sprint
      
      # Warning: slow bleed — burn rate ≥ 1 sustained for 6h
      - alert: ErrorBudgetSlowBurn
        expr: |
          (job:error_rate:rate1h{job="api"} > (1.5 * 0.001))
          and
          (job:error_rate:rate6h{job="api"} > (1.5 * 0.001))
        for: 0m
        labels:
          severity: warning
          team: backend
        annotations:
          summary: "Slow budget burn: error rate above SLO for extended period"

Burn Rate Reference Table

Burn Rate	Budget Exhausted In	Alert Urgency
1	30 days	None (on track)
2	15 days	Warning
5	6 days	Ticket
14.4	50 hours	Page
36	20 hours	Page immediately
360	2 hours	All hands

Testing Burn Rate Alerts

The most critical thing to test: do your alerts actually fire when expected?

# tests/alerting/test_burn_rate_alerts.py
import pytest
import requests
import json

PROMETHEUS_URL = "http://prometheus:9090"
ALERTMANAGER_URL = "http://alertmanager:9093"

class PrometheusTestHarness:
    """Inject synthetic metrics and verify alert behavior."""
    
    def __init__(self, pushgateway_url: str):
        self.pushgateway = pushgateway_url
    
    def set_error_rate(self, job: str, error_rate: float):
        """Inject synthetic error rate via pushgateway."""
        # Total: 1000 req/s, errors proportional to error_rate
        total_rps = 1000
        error_rps = total_rps * error_rate
        
        metrics = f"""
# HELP http_requests_total Total HTTP requests
# TYPE http_requests_total counter
http_requests_total{{job="{job}",status="200"}} {total_rps - error_rps}
http_requests_total{{job="{job}",status="500"}} {error_rps}
"""
        requests.post(
            f"{self.pushgateway}/metrics/job/{job}",
            data=metrics,
            headers={"Content-Type": "text/plain"},
        )
    
    def get_active_alerts(self) -> list[dict]:
        """Get currently firing alerts from Alertmanager."""
        response = requests.get(f"{ALERTMANAGER_URL}/api/v2/alerts")
        return response.json()
    
    def alert_is_firing(self, alert_name: str) -> bool:
        """Check if a specific alert is currently active."""
        alerts = self.get_active_alerts()
        return any(
            a["labels"]["alertname"] == alert_name
            for a in alerts
            if a["status"]["state"] == "active"
        )

@pytest.fixture
def prometheus_harness():
    return PrometheusTestHarness("http://pushgateway:9091")

def test_fast_burn_alert_fires(prometheus_harness):
    """ErrorBudgetFastBurn should fire at burn rate 14.4+ (error rate 1.44%)."""
    import time
    
    # Set error rate that triggers fast burn (14.4 × 0.1% = 1.44%)
    prometheus_harness.set_error_rate("api", 0.015)  # 1.5% — above threshold
    
    # Wait for alert to fire (prometheus eval interval + for duration)
    # FastBurn has `for: 2m` — wait 4 minutes to be safe
    deadline = time.time() + 300  # 5 minute timeout
    while time.time() < deadline:
        if prometheus_harness.alert_is_firing("ErrorBudgetFastBurn"):
            break
        time.sleep(15)
    
    assert prometheus_harness.alert_is_firing("ErrorBudgetFastBurn"), \
        "ErrorBudgetFastBurn did not fire within 5 minutes at 1.5% error rate"

def test_fast_burn_alert_resolves(prometheus_harness):
    """ErrorBudgetFastBurn should stop firing when error rate returns to normal."""
    import time
    
    # Set normal error rate
    prometheus_harness.set_error_rate("api", 0.0001)  # 0.01% — well under SLO
    
    deadline = time.time() + 300
    while time.time() < deadline:
        if not prometheus_harness.alert_is_firing("ErrorBudgetFastBurn"):
            break
        time.sleep(15)
    
    assert not prometheus_harness.alert_is_firing("ErrorBudgetFastBurn"), \
        "ErrorBudgetFastBurn still firing after error rate returned to normal"

def test_slow_burn_not_triggered_by_brief_spike(prometheus_harness):
    """Brief error spike should not trigger ErrorBudgetSlowBurn."""
    import time
    
    # Set high error rate briefly
    prometheus_harness.set_error_rate("api", 0.05)
    time.sleep(30)  # Only 30 seconds
    
    # Return to normal
    prometheus_harness.set_error_rate("api", 0.0001)
    time.sleep(60)  # Wait a minute
    
    # Slow burn requires sustained burn over 6h — should NOT fire from brief spike
    assert not prometheus_harness.alert_is_firing("ErrorBudgetSlowBurn"), \
        "ErrorBudgetSlowBurn fired from brief spike — multi-window not working correctly"

Error Budget Tracking Dashboard

# scripts/error_budget_dashboard.py
from datetime import datetime, timedelta
import requests

def get_error_budget_status(
    prometheus_url: str,
    service: str,
    slo: float = 0.999,
    window_days: int = 30,
) -> dict:
    """Get current error budget consumption."""
    
    window_seconds = window_days * 24 * 3600
    
    # Query: fraction of requests that failed in the window
    query = f"""
    1 - (
      sum(increase(http_requests_total{{job="{service}",status=~"2.."}}[{window_seconds}s]))
      /
      sum(increase(http_requests_total{{job="{service}"}}[{window_seconds}s]))
    )
    """
    
    response = requests.get(
        f"{prometheus_url}/api/v1/query",
        params={"query": query}
    )
    
    data = response.json()
    
    if not data["data"]["result"]:
        return {"error": "No data"}
    
    current_error_rate = float(data["data"]["result"][0]["value"][1])
    
    # Error budget = allowed error rate = 1 - SLO
    error_budget = 1 - slo
    
    # Budget consumed
    budget_consumed = current_error_rate / error_budget
    budget_remaining = max(0, 1 - budget_consumed)
    
    # Burn rate
    burn_rate = current_error_rate / error_budget
    
    # Days until budget exhaustion at current rate
    if burn_rate > 0:
        days_remaining = window_days / burn_rate
    else:
        days_remaining = float('inf')
    
    return {
        "service": service,
        "slo_target": f"{slo:.3%}",
        "current_error_rate": f"{current_error_rate:.4%}",
        "error_budget": f"{error_budget:.3%}",
        "budget_consumed": f"{budget_consumed:.0%}",
        "budget_remaining": f"{budget_remaining:.0%}",
        "burn_rate": f"{burn_rate:.1f}x",
        "days_until_exhaustion": f"{days_remaining:.1f}" if days_remaining != float('inf') else "∞",
        "status": "⚠️  BURNING" if budget_consumed > 0.5 else "✓ OK",
    }

Alerting Runbook

Every burn rate alert should have a runbook:

# ErrorBudgetFastBurn Runbook

## What This Alert Means
Your service is consuming its error budget 14.4× faster than the sustainable rate.
At this burn rate, you will exhaust your monthly error budget in under 50 hours.

## Immediate Actions (< 5 minutes)
1. Check the error rate dashboard: [link]
2. Look at recent deployments: `kubectl rollout history deployment/myapp`
3. Check error logs: `kubectl logs -l app=myapp --since=10m | grep ERROR`

## Investigation
1. Is this a new deployment? → Consider rollback: `kubectl rollout undo deployment/myapp`
2. Is this a dependency issue? → Check database, cache, external API health
3. Is this traffic-based? → Check if traffic pattern is unusual

## Escalation
- After 15 min with no resolution: page on-call lead
- After 30 min: invoke incident response process

Summary

Error budget burn rate alerting is more effective than threshold alerting because:

It catches fast burns — 100% error rate for 30 minutes exhausts your monthly budget
It catches slow bleeds — 0.2% error rate sustained for 20 days also exhausts your budget
Multi-window prevents false positives — brief spikes don't page you at 3am
Urgency scales with impact — burn rate 36 → page immediately; burn rate 2 → ticket

Test your alerts the same way you test your application: inject synthetic failures, verify alerts fire within the expected window, verify they resolve when the issue clears.