SRE

SLO-Driven Testing: How to Define and Validate Service Level Objectives

HelpMeTest

HelpMeTest

— 5 min read
SLO-Driven Testing: How to Define and Validate Service Level Objectives

Most teams write tests to verify features work. SRE teams write tests to verify that reliability commitments are met. The difference isn't philosophical — it produces a fundamentally different kind of test suite that catches a different class of failures.

SLO-driven testing means your SLOs are the spec, and tests verify you're meeting them. If you don't have tests that can fail when your SLO is violated, your SLO is aspirational, not operational.

What an SLO Actually Is

A Service Level Objective is a target reliability measurement over a time window:

  • "99.9% of API requests complete successfully in each 30-day window"
  • "p99 latency is under 500ms for 99.5% of 30-day windows"
  • "Error rate is below 0.1% measured over any 24-hour period"

An SLO has three components:

  1. SLI (Service Level Indicator): The metric you measure (request success rate, latency, etc.)
  2. Target: The threshold (99.9%, 500ms)
  3. Window: The time period (30 days, 24 hours)

The SLO is only as good as your SLI measurement. If your SLI doesn't capture what users actually experience, you can meet the SLO while users are suffering.

Defining Testable SLIs

The first step in SLO-driven testing is defining SLIs that are:

  • Measurable from your test infrastructure
  • Reflective of real user experience
  • Unambiguous (no room for interpretation)

Availability SLI:

def measure_availability_sli(window_seconds: int) -> float:
    """Calculate availability SLI as successful_requests / total_requests."""
    metrics = query_prometheus(
        query='sum(rate(http_requests_total{status!~"5.."}[{window}s])) / '
              'sum(rate(http_requests_total[{window}s]))',
        window=window_seconds
    )
    return float(metrics[0]["value"][1])

def test_availability_sli_measurement():
    """SLI measurement code must accurately reflect actual availability."""
    # Send 1000 requests, 10 of which will fail
    send_requests(count=990, expected_status=200)
    send_requests(count=10, expected_status=500)
    
    # Wait for metrics to propagate
    time.sleep(15)
    
    # Measure SLI
    sli = measure_availability_sli(window_seconds=60)
    
    # Should be ~99% (990/1000)
    assert 0.985 <= sli <= 0.995, \
        f"SLI measurement inaccurate: got {sli:.3f}, expected ~0.990"

Latency SLI:

def measure_latency_sli(threshold_ms: float, percentile: float = 0.99) -> float:
    """Calculate fraction of requests completing under threshold at given percentile."""
    p_latency = query_prometheus_percentile(
        metric="http_request_duration_seconds",
        percentile=percentile
    ) * 1000  # Convert to ms
    
    return 1.0 if p_latency <= threshold_ms else 0.0

# More granular version:
def measure_latency_sli_granular(threshold_ms: float) -> float:
    """Calculate fraction of requests completing under threshold_ms."""
    under_threshold = query_prometheus(
        f'sum(rate(http_request_duration_seconds_bucket{{le="{threshold_ms/1000}"}}[5m]))'
    )
    total = query_prometheus(
        'sum(rate(http_request_duration_seconds_count[5m]))'
    )
    return float(under_threshold) / float(total)

Writing SLO Compliance Tests

Once you have SLIs, write tests that verify your system meets its SLOs under defined conditions.

class SLOComplianceTests:
    """Tests that verify SLO compliance under realistic conditions."""
    
    SLO_AVAILABILITY = 0.999  # 99.9%
    SLO_LATENCY_P99_MS = 500.0
    SLO_LATENCY_FRACTION = 0.995  # 99.5% of requests under 500ms
    MEASUREMENT_WINDOW_SECONDS = 300  # 5-minute test window
    
    def test_availability_slo_under_normal_load(self):
        """Availability must meet SLO under normal production load."""
        load_generator = LoadGenerator(
            rps=100,
            duration_seconds=self.MEASUREMENT_WINDOW_SECONDS
        )
        
        load_generator.start()
        time.sleep(self.MEASUREMENT_WINDOW_SECONDS)
        results = load_generator.stop()
        
        availability = results.successful / results.total
        assert availability >= self.SLO_AVAILABILITY, (
            f"Availability SLO violated: {availability:.4f} < {self.SLO_AVAILABILITY}\n"
            f"Failed requests: {results.failed}\n"
            f"Errors: {results.error_breakdown}"
        )
    
    def test_latency_slo_under_normal_load(self):
        """p99 latency must meet SLO under normal production load."""
        latencies = collect_latencies(
            rps=100,
            duration_seconds=self.MEASUREMENT_WINDOW_SECONDS
        )
        
        p99 = numpy.percentile(latencies, 99)
        assert p99 <= self.SLO_LATENCY_P99_MS, (
            f"Latency SLO violated: p99 = {p99:.0f}ms > {self.SLO_LATENCY_P99_MS}ms\n"
            f"p50: {numpy.percentile(latencies, 50):.0f}ms\n"
            f"p95: {numpy.percentile(latencies, 95):.0f}ms"
        )
    
    def test_availability_slo_under_dependency_degradation(self):
        """Availability SLO must hold even when dependencies are degraded."""
        # Simulate 10% error rate from downstream dependency
        with inject_dependency_errors(service="database", error_rate=0.1):
            results = run_load(rps=100, duration=60)
        
        availability = results.successful / results.total
        # SLO should hold via retries and fallbacks
        assert availability >= self.SLO_AVAILABILITY, \
            f"SLO broken by dependency degradation: availability={availability:.4f}"

SLO Burn Rate Tests

SLO burn rate tells you how fast you're consuming your error budget. A burn rate of 1.0 means you're exactly on track. A burn rate of 60 means you'll exhaust your monthly error budget in 12 hours.

Testing burn rate detection:

def test_burn_rate_alert_fires_correctly():
    """Burn rate alert must fire when error budget is consumed 60x faster than normal."""
    # 99.9% SLO = 0.1% error budget over 30 days
    # 60x burn rate = 0.1% * 60 = 6% error rate
    
    # Inject 6% error rate
    with inject_error_rate(0.06):
        time.sleep(60)  # Run for 1 minute
    
    # Alert should have fired
    alerts = get_fired_alerts(name="SLOBurnRateHigh")
    assert len(alerts) > 0, "Burn rate alert did not fire at 60x burn rate"
    
    latest_alert = alerts[-1]
    assert latest_alert["labels"]["severity"] == "critical"
    assert float(latest_alert["annotations"]["burn_rate"]) > 50

def calculate_burn_rate(error_rate: float, slo_target: float) -> float:
    """Calculate error budget burn rate."""
    error_budget = 1.0 - slo_target  # e.g., 0.001 for 99.9% SLO
    return error_rate / error_budget

Multi-Window SLO Testing

Google's SRE book recommends multi-window burn rate alerting. Test both your fast-burn and slow-burn alerts.

@pytest.mark.parametrize("scenario,error_rate,duration_minutes,expected_alert", [
    # Fast burn: 14.4x in 1h window (exhausts 30-day budget in 2d)
    ("fast_burn", 0.014, 65, "SLOBurnRateFast"),
    # Slow burn: 3x in 6h window (exhausts 30-day budget in 10d)
    ("slow_burn", 0.003, 370, "SLOBurnRateSlow"),
    # Normal: no alerts
    ("normal", 0.0005, 60, None),
])
def test_multi_window_burn_rate_alerts(scenario, error_rate, duration_minutes, expected_alert):
    """Burn rate alerts must fire at the correct thresholds."""
    with inject_error_rate(error_rate):
        time.sleep(duration_minutes * 60)
    
    alerts = get_fired_alerts()
    alert_names = {a["alertname"] for a in alerts}
    
    if expected_alert:
        assert expected_alert in alert_names, \
            f"Scenario '{scenario}': expected alert {expected_alert} did not fire. "
            f"Alerts fired: {alert_names}"
    else:
        slo_alerts = {a for a in alert_names if "SLO" in a or "BurnRate" in a}
        assert len(slo_alerts) == 0, \
            f"Scenario '{scenario}': unexpected SLO alerts fired: {slo_alerts}"

SLO as CI Gate

Make SLO compliance a CI gate for every production deployment. Before a release ships:

  1. Run a load test at production traffic levels
  2. Measure SLI values over the test window
  3. Verify SLOs are met
  4. Only proceed if all SLOs pass
# .github/workflows/slo-gate.yml
name: SLO Compliance Gate

on:
  push:
    branches: [main]

jobs:
  slo-gate:
    runs-on: ubuntu-latest
    steps:
      - name: Deploy to staging
        run: ./deploy.sh --environment staging

      - name: Run SLO load test
        run: |
          python -m pytest tests/slo/test_availability.py \
            tests/slo/test_latency.py \
            tests/slo/test_throughput.py \
            -v --timeout=600 \
            --junitxml=slo-results.xml

      - name: Check SLO compliance
        run: python scripts/check-slo-compliance.py --results slo-results.xml

      - name: Block deployment if SLO violated
        if: failure()
        run: |
          echo "SLO compliance gate FAILED"
          echo "Deployment to production blocked"
          exit 1

Monitoring SLO Health Continuously

Tests run at deployment time. Production SLOs must be monitored continuously.

HelpMeTest can run continuous health checks against your endpoints — verifying availability and measuring response latency every 5 minutes. When your SLI metrics drop below SLO thresholds, you get alerted before your error budget is consumed. Set up monitoring for your critical user journeys: the API endpoints your SLOs are built on, the health check URLs that confirm your system is serving correctly.

This gives you two layers: CI gates that prevent bad releases, and continuous monitoring that catches production degradation.

Summary

SLO-driven testing requires:

  1. Testable SLIs — precise measurements that reflect user experience
  2. SLO compliance tests under normal and degraded conditions
  3. Burn rate tests that verify your alerting fires at the right thresholds
  4. Multi-window burn rate testing for both fast-burn and slow-burn scenarios
  5. SLO as a CI gate that blocks deployments failing reliability requirements

When your SLOs are backed by tests that can fail, they stop being aspirational and start being real commitments.

Read more

Start now free