Testing

SLO-Driven Testing: Building Your Test Suite Around Error Budgets

30 Sep 2026 — 6 min read

Most test strategies are organized around features: we test what we build. SLO-driven testing flips this: you test what you've promised. If you've committed to 99.9% availability, your test suite should have an explicit answer to the question "how do we know we're meeting that?"

This post explains how to translate SLOs and error budgets into concrete test design decisions — which tests protect which SLIs, what pass rate targets mean in terms of error budget consumption, and how to build CI gates that respond to error budget burn rate.

SLOs Define What You Must Test

A Service Level Objective is a target for a Service Level Indicator over a time window. Classic SLIs and their corresponding SLO commitments:

SLI	SLO Example
Availability (% of requests that succeed)	99.9% over 30 days
Latency (p95 request duration)	< 200ms over 7 days
Error rate (% of requests that return 5xx)	< 0.1% over 30 days
Throughput (requests processed per second)	> 100 rps sustained
Freshness (age of most recent data)	< 5 minutes

Each SLO tells you something you need to test. If you've committed to p95 < 200ms, you need a test that verifies p95 latency under realistic load. If you've committed to 99.9% availability, you need tests that verify the service recovers from the failure modes that would cause downtime.

Mapping SLIs to test types:

SLI	Test Type
Availability	Integration tests + chaos tests
Latency	Load tests with percentile assertions
Error rate	Integration tests for error paths + smoke tests in production
Throughput	Load tests
Freshness	Pipeline tests + synthetic monitors

Error Budget = Acceptable Failure Rate = Test Pass Rate Target

Your error budget is the complement of your SLO. A 99.9% availability SLO gives you a 0.1% error budget — 0.1% of requests can fail over the measurement window.

This has a direct implication for your test suite: your tests must fail less often than your error budget allows in production. A test that's flaky 2% of the time is noisier than your production error budget. It teaches your team to ignore red CI, and eventually a real production regression slips through the noise.

Translating error budget to test reliability targets:

Error budget: 0.1% (SLO: 99.9% availability)
Test flakiness target: < 0.05% (half the error budget)

Error budget: 1% (SLO: 99% availability)  
Test flakiness target: < 0.5%

Error budget: 0.01% (SLO: 99.99% availability)
Test flakiness target: < 0.005% — extremely tight; near-zero test flakiness required

If your SLO is 99.99%, a 1% flaky test suite is a joke. Every flaky test run costs more than your monthly production error budget. Tight SLOs require tight tests.

Which Tests Directly Protect SLOs

Availability SLO → Integration tests + health check tests

Availability fails when the service crashes, restarts, or rejects requests. Tests that protect availability:

// health-check.test.ts
import request from 'supertest';
import { app } from '../src/app';

describe('Service availability', () => {
  it('returns 200 on health check', async () => {
    const response = await request(app).get('/health');
    expect(response.status).toBe(200);
    expect(response.body.status).toBe('ok');
  });

  it('returns 200 when database is connected', async () => {
    const response = await request(app).get('/health/deep');
    expect(response.status).toBe(200);
    expect(response.body.database).toBe('connected');
  });

  it('returns 503 when database is unreachable', async () => {
    // Simulate DB unavailability
    await pauseDatabase();
    const response = await request(app).get('/health/deep');
    expect(response.status).toBe(503);
    await resumeDatabase();
  });
});

Latency SLO → Load tests with percentile assertions

A unit test cannot verify latency under load. You need a load test:

// k6 load test for latency SLO
import http from 'k6/http';
import { check } from 'k6';

export const options = {
  scenarios: {
    slo_validation: {
      executor: 'constant-vus',
      vus: 100,           // simulates realistic concurrency
      duration: '5m',
    },
  },
  thresholds: {
    // These thresholds ARE your SLO test
    'http_req_duration{name:checkout}': ['p(95)<200'],   // p95 < 200ms
    'http_req_duration{name:checkout}': ['p(99)<500'],   // p99 < 500ms
    'http_req_failed{name:checkout}': ['rate<0.001'],    // error rate < 0.1%
  },
};

export default function () {
  const res = http.post(
    `${__ENV.BASE_URL}/api/checkout`,
    JSON.stringify({ cartId: `cart-${__VU}` }),
    {
      tags: { name: 'checkout' },
      headers: { 'Content-Type': 'application/json' },
    }
  );

  check(res, {
    'status 200': (r) => r.status === 200,
    'under 200ms': (r) => r.timings.duration < 200,
  });
}

The thresholds block is the SLO in code. When the k6 run exits with non-zero, your SLO test has failed.

Error rate SLO → Integration tests for all error paths

Error rate failures come from uncaught exceptions, unhandled rejection cases, and invalid state transitions. Integration tests that deliberately trigger error conditions and verify correct error handling:

# tests/integration/test_error_handling.py
import pytest

class TestErrorRateSLO:
    """Tests that protect the 0.1% error rate SLO"""

    def test_invalid_cart_id_returns_400_not_500(self, client):
        """A 400 is expected, counted as 'success' for error rate SLO"""
        response = client.post('/api/checkout', json={'cartId': 'invalid!'})
        assert response.status_code == 400  # client error, NOT server error
        assert 'error' in response.json()

    def test_expired_session_returns_401_not_500(self, client):
        response = client.post(
            '/api/checkout',
            json={'cartId': 'cart-1'},
            headers={'Authorization': 'Bearer expired-token'}
        )
        assert response.status_code == 401  # expected auth failure

    def test_database_timeout_returns_503_with_retry_after(self, client, db):
        """DB timeout should return 503 with retry guidance, not crash"""
        db.simulate_timeout()
        response = client.post('/api/checkout', json={'cartId': 'cart-1'})
        assert response.status_code == 503
        assert 'Retry-After' in response.headers
        db.restore()

    def test_concurrent_requests_dont_cause_500s(self, client):
        """Race conditions should not produce server errors"""
        import concurrent.futures
        results = []
        with concurrent.futures.ThreadPoolExecutor(max_workers=20) as executor:
            futures = [
                executor.submit(client.post, '/api/checkout', json={'cartId': f'cart-{i}'})
                for i in range(20)
            ]
            results = [f.result() for f in futures]

        server_errors = [r for r in results if r.status_code >= 500]
        assert len(server_errors) == 0

Building CI Gates That Respond to Error Budget

The most powerful application of SLO-driven testing is making CI gates aware of error budget burn rate. When your error budget is healthy, allow more risk. When your error budget is depleted, block all releases except fixes.

This requires reading burn rate from your monitoring system during CI:

# ci/check-error-budget.py
import sys
import requests

def get_error_budget_remaining(service: str, window: str = "30d") -> float:
    """Query Prometheus for error budget remaining percentage"""
    prometheus_url = os.environ['PROMETHEUS_URL']
    
    # Query burn rate over the window
    query = f'''
        1 - (
          sum(rate(http_requests_total{{service="{service}",status=~"5.."}}[{window}]))
          /
          sum(rate(http_requests_total{{service="{service}"}}[{window}]))
        )
    '''
    
    response = requests.get(
        f'{prometheus_url}/api/v1/query',
        params={'query': query}
    )
    result = response.json()['data']['result']
    
    if not result:
        return 1.0  # assume full budget if no data
    
    current_slo = float(result[0]['value'][1])
    target_slo = 0.999  # 99.9%
    
    # Budget consumed = how much we've used of the error budget
    total_error_budget = 1 - target_slo
    consumed = max(0, target_slo - current_slo)
    remaining = 1 - (consumed / total_error_budget)
    
    return remaining

def main():
    service = sys.argv[1]
    budget_remaining = get_error_budget_remaining(service)
    
    print(f"Error budget remaining: {budget_remaining:.1%}")
    
    if budget_remaining < 0.10:
        print("ERROR: Error budget below 10% — blocking deployment")
        print("Only bug fixes and reliability improvements may be deployed")
        sys.exit(1)
    elif budget_remaining < 0.25:
        print("WARNING: Error budget below 25% — proceed with caution")
        # Could automatically require a second approval
    else:
        print("OK: Error budget healthy")

if __name__ == '__main__':
    main()

Add this to your CI pipeline:

# .github/workflows/deploy.yml
jobs:
  check-error-budget:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Check error budget before deploy
        env:
          PROMETHEUS_URL: ${{ secrets.PROMETHEUS_URL }}
        run: python ci/check-error-budget.py payments-api

  deploy:
    needs: check-error-budget
    # only runs if check-error-budget succeeds

Now your deployment pipeline respects your reliability commitments. The team can't accidentally burn through an error budget by deploying risky changes when budget is low.

SLO-Aware Test Tagging

Tag your tests with the SLI they protect. This makes the relationship explicit and lets you filter during an incident:

# pytest markers for SLO coverage
# conftest.py
import pytest

def pytest_configure(config):
    config.addinivalue_line("markers", "slo_availability: tests protecting availability SLO")
    config.addinivalue_line("markers", "slo_latency: tests protecting latency SLO")
    config.addinivalue_line("markers", "slo_error_rate: tests protecting error rate SLO")

@pytest.mark.slo_availability
def test_service_recovers_after_crash(client):
    ...

@pytest.mark.slo_error_rate
def test_invalid_input_returns_400_not_500(client):
    ...

Run only SLO-protecting tests for a fast reliability check:

pytest -m "slo_availability or slo_error_rate" --tb=short

Connecting Tests to SLO Dashboards

The final piece: make the test results visible alongside your SLO dashboards. When a k6 load test runs in CI, publish the summary to your monitoring system:

// k6 with Prometheus remote write
export const options = {
  // ... thresholds as before
};

// k6 can push metrics to Prometheus via remote write
// Run with: k6 run --out experimental-prometheus-rw script.js

Or publish a custom metric after the test run:

# Push test result to Prometheus pushgateway
cat <<EOF | curl --data-binary @- http://prometheus-pushgateway:9091/metrics/job/slo_tests/service/payments-api
# HELP slo_test_passed Whether the SLO validation test passed (1=pass, 0=fail)
# TYPE slo_test_passed gauge
slo_test_passed{slo="availability"} 1
slo_test_passed{slo="latency_p95"} 1
slo_test_passed{slo="error_rate"} 0
EOF

Now your Grafana dashboard can show SLO test status alongside production SLO burn rate. When the test fails, you see it in the same dashboard where you watch production reliability.

The Core Principle

SLO-driven testing is about making your reliability commitments first-class requirements with explicit test coverage. You've made a promise to your users. Your test suite should have an answer for how that promise is verified.

Map each SLI to the tests that validate it. Set flakiness targets that reflect your error budgets. Build CI gates that block risky deploys when budget is low. Make test results visible alongside production metrics.

When the tests pass, you have evidence — not hope — that your SLOs will hold.