Testing

SRE Testing Practices: Error Budgets, SLOs, and Reliability Testing

HelpMeTest

HelpMeTest

— 6 min read
SRE Testing Practices: Error Budgets, SLOs, and Reliability Testing

Site Reliability Engineering (SRE) reframes reliability as a software problem. Instead of treating testing and monitoring as separate concerns, SRE treats them as integrated practices built around measurable objectives. This guide covers the core SRE testing practices: SLOs, error budgets, reliability testing, and how they work together.

The SRE Testing Philosophy

Traditional QA finds bugs before release. SRE testing answers a different question: how does the system behave when things go wrong in production?

SRE teams test for:

  • Reliability under normal load
  • Behavior under degraded conditions
  • Recovery speed after failures
  • Correctness of monitoring and alerting
  • Toil reduction through automation

The key insight: 100% reliability is the wrong goal. A system with five nines (99.999%) availability burns engineering budget on reliability that users can't notice. Users on mobile networks with 99% uptime can't tell the difference between 99.9% and 99.999% availability. Error budgets formalize this tradeoff.

Service Level Objectives (SLOs)

An SLO defines the target reliability level for a service:

SLO: 99.9% of HTTP requests complete in < 200ms over a 30-day rolling window

SLOs have three components:

  1. Metric — what you're measuring (latency, availability, error rate, throughput)
  2. Target — the percentage of time the metric should be within acceptable range
  3. Window — the time period over which the SLO is evaluated

Defining Good SLOs

Bad SLO: "The service should be fast."

Good SLO: "99.5% of requests to /api/search return results within 500ms, measured over a 28-day rolling window."

Good SLOs are:

  • Measurable — can be calculated from real data
  • Actionable — violation triggers specific response
  • User-focused — reflect what users actually experience
  • Conservative — start lower, tighten over time

Common SLO types:

Type Example
Availability 99.9% of requests return non-5xx
Latency 95th percentile response < 300ms
Error rate < 0.1% of transactions fail
Throughput Handle 10,000 RPS without degradation
Freshness Data updated within 5 minutes of source change

SLIs: What You Measure

A Service Level Indicator (SLI) is the actual metric. The SLO is the target:

SLI: (successful requests / total requests) * 100
SLO: SLI >= 99.9%

For latency:

SLI: percentage of requests under 200ms
SLO: SLI >= 95% (p95 latency < 200ms)

Error Budgets

If your SLO is 99.9% availability over 30 days, you have 0.1% of requests allowed to fail — your error budget:

Monthly error budget = (1 - SLO) × total requests
                     = 0.001 × 10,000,000 requests
                     = 10,000 failed requests allowed

Or in time: 0.1% of 30 days = 43.2 minutes of downtime allowed per month.

Error Budget Policy

The error budget policy defines what happens when budget is consumed:

error_budget_policy:
  service: api-gateway
  slo: 99.9%
  window: 30d
  
  thresholds:
    - consumed: 25%
      action: "Review recent changes, increase test coverage"
    
    - consumed: 50%
      action: "Freeze non-essential releases"
    
    - consumed: 75%
      action: "All hands on reliability improvements"
    
    - consumed: 100%
      action: "Production freeze. Only reliability fixes ship."

Error budgets create a shared language between product and engineering: "We can ship fast OR we can be reliable. The error budget tells us which we currently have room for."

Reliability Testing Strategies

Load Testing to SLO

Don't load test to "find the breaking point." Test to your SLO:

// k6 test: verify SLO under expected load
import http from 'k6/http';
import { check } from 'k6';
import { Rate } from 'k6/metrics';

const errorRate = new Rate('errors');
const sloLatency = new Rate('slo_latency');

export const options = {
  vus: 1000,
  duration: '10m',
  thresholds: {
    // SLO: 99.9% availability
    errors: ['rate<0.001'],
    // SLO: p95 latency < 200ms
    http_req_duration: ['p(95)<200'],
    // SLO: p99 latency < 500ms
    'http_req_duration{expected_response:true}': ['p(99)<500'],
  },
};

export default function () {
  const res = http.get('https://api.example.com/health');
  
  const ok = check(res, {
    'status is 200': (r) => r.status === 200,
    'latency < 200ms': (r) => r.timings.duration < 200,
  });
  
  errorRate.add(!ok);
  sloLatency.add(res.timings.duration < 200);
}

Canary Testing

Deploy to a small percentage of traffic before full rollout:

Traffic routing:
  - version: v1.2.3 (current)
    weight: 90%
  - version: v1.3.0 (canary)
    weight: 10%

SLO monitoring:
  - Compare error rates between versions
  - Compare p99 latency between versions
  - Auto-rollback if canary exceeds SLO threshold

The canary's error budget consumption rate should not exceed the stable version's rate by more than a configurable multiplier (e.g., 2x).

Soak Testing

Run at production load for extended periods to find slow leaks:

# Artillery soak test: 12 hours at 70% capacity
artillery run \
  --target https://api.example.com \
  --duration 43200 \
  --arrival-rate 700 \
  soak-test.yaml

Soak tests catch:

  • Memory leaks (gradual increase over hours)
  • Connection pool exhaustion
  • Database connection leaks
  • Log file growth
  • Certificate rotation issues

Stress Testing to Find SLO Limits

Find where your SLO breaks, then build headroom above it:

# Ramp to breaking point
config:
  phases:
    - duration: 120
      arrivalRate: 100
      name: "Warm up"
    - duration: 120
      arrivalRate: 100
      rampTo: 500
      name: "Ramp up"
    - duration: 120
      arrivalRate: 500
      rampTo: 1000
      name: "Stress"
    - duration: 120
      arrivalRate: 1000
      rampTo: 2000
      name: "Breaking point"

If your SLO is 99.9% at 500 RPS, design for 1000 RPS. The SLO should be comfortable at 2× expected load.

Measuring Toil

Toil is repetitive, manual, automatable work with no lasting value. SRE teams track and limit it:

Toil categories:
1. Manual interventions (restart service, clear queue)
2. Interrupt-driven work (alert response, customer tickets)  
3. Undifferentiated work (on-call rotations for known issues)

Target: < 50% of SRE time spent on toil

Toil tracking feeds directly into reliability testing priorities: high-toil areas are candidates for automation and reliability improvements.

Runbooks as Executable Tests

Every alert should have a runbook. Runbooks should be testable:

# Service Unresponsive Runbook

## Detection
Alert: `api_gateway_error_rate > 1%` for 5 minutes

## Diagnosis
1. Check pod status: `kubectl get pods -n production`
2. Check recent deployments: `kubectl rollout history deployment/api-gateway`
3. Check error logs: `kubectl logs -l app=api-gateway --tail=100`

## Resolution
### If OOM:
- Increase memory limit: `kubectl set resources deployment/api-gateway --limits=memory=2Gi`
- File capacity planning ticket

### If bad deployment:
- Rollback: `kubectl rollout undo deployment/api-gateway`
- Verify: wait 2 minutes, check error rate drops below 0.1%

## Validation
- Error rate < 0.1% for 5 consecutive minutes
- No pod restarts in last 10 minutes

Test runbooks regularly with game days (planned chaos exercises):

Game day format:
1. Announce scenario: "API gateway is down"
2. Team executes runbook without coordinator help
3. Measure: time to detect, time to diagnose, time to resolve
4. Retrospective: where did the runbook fail? What was unclear?

SLO-Driven Alert Design

Alerts should fire based on SLO burn rate, not raw thresholds:

Bad alert: "Error rate > 1% for 5 minutes" — triggers too often on transient spikes.

Good alert: Burn rate alerting — "At this error rate, you'll exhaust your monthly error budget in 1 hour."

# Alert: Fast burn (1 hour budget consumption)
- alert: SLOBudgetBurnFast
  expr: |
    (
      sum(rate(http_requests_total{status=~"5.."}[5m]))
      /
      sum(rate(http_requests_total[5m]))
    ) > (14.4 * 0.001)  # 14.4x burn rate burns 1hr budget in 5 min
  for: 2m
  severity: critical

# Alert: Slow burn (6 hour budget consumption)
- alert: SLOBudgetBurnSlow
  expr: |
    (
      sum(rate(http_requests_total{status=~"5.."}[1h]))
      /
      sum(rate(http_requests_total[1h]))
    ) > (6 * 0.001)  # 6x burn rate burns 6hr budget in 1hr
  for: 15m
  severity: warning

Testing the Monitoring Stack

Your monitoring system is itself a system that needs testing:

# Test: alert fires within acceptable time
def test_error_rate_alert_fires_in_time():
    # Inject errors
    inject_error_rate(service='api', rate=0.05)  # 5% errors
    
    # Should alert within 5 minutes
    alert = wait_for_alert('HighErrorRate', timeout=300)
    
    assert alert is not None, "Alert did not fire in time"
    assert alert.severity == 'critical'
    
    # Clean up
    stop_error_injection()

# Test: dashboard shows correct data
def test_error_rate_dashboard_accuracy():
    inject_errors(count=1000, total_requests=100000)  # 1% error rate
    
    dashboard_value = read_dashboard_metric('error_rate_1h')
    assert abs(dashboard_value - 0.01) < 0.001  # within 0.1%

Production Readiness Reviews

Before a service goes to production, run a Production Readiness Review (PRR):

## PRR Checklist

### SLOs
- [ ] SLOs defined and documented
- [ ] SLIs instrumented and measurable
- [ ] Error budget policy defined
- [ ] SLO dashboard created

### Reliability Testing
- [ ] Load tested to 2× expected peak traffic
- [ ] Soak tested for 24+ hours
- [ ] Chaos experiments run for top 3 failure modes

### Observability
- [ ] Structured logging in place
- [ ] Metrics exposed (RED: Rate, Errors, Duration)
- [ ] Distributed tracing enabled
- [ ] Alerts cover top failure modes with runbooks

### Operations
- [ ] Runbooks written for all critical alerts
- [ ] Rollback procedure documented and tested
- [ ] Capacity planning done for 6-month growth
- [ ] On-call rotation and escalation path defined

Summary

SRE testing practices make reliability measurable and actionable:

  1. SLOs define what reliability means in concrete, measurable terms
  2. Error budgets create a shared language for reliability tradeoffs
  3. Load and soak testing verify the system meets SLOs under real conditions
  4. Burn rate alerts catch SLO violations early enough to prevent budget exhaustion
  5. PRRs ensure services are production-ready before they carry traffic

The goal isn't zero failures — it's controlled, budgeted failures that keep engineers shipping while maintaining user trust.

Read more

Start now free