SLA Testing

SLA Testing: How to Define, Measure, and Verify Service Level Agreements

HelpMeTest

HelpMeTest

— 6 min read
SLA Testing: How to Define, Measure, and Verify Service Level Agreements

An SLA (Service Level Agreement) is only as valuable as your ability to verify it. Most engineering teams define SLAs, deploy to production, and then find out they're violating them from a customer complaint. That's not SLA management — that's reactive damage control.

This guide covers how to build SLA testing into your development and operations process: from defining measurable SLOs to automated compliance checks.

SLA, SLO, SLI — The Hierarchy

Before testing, get the terminology straight:

  • SLA — the external promise to customers ("99.9% uptime, or you get a refund")
  • SLO — the internal target you need to hit to meet the SLA ("99.95% uptime" — stricter than the SLA so you have buffer)
  • SLI — the actual measurement ("current uptime: 99.97%")

You test SLIs to verify SLOs to ensure SLA compliance.

Defining Testable SLOs

Vague SLOs can't be tested. "The system should be fast" is untestable. "99th percentile response time < 200ms for the /checkout endpoint, measured over any rolling 30-day window" is testable.

Good SLO format:

[Metric] for [service/endpoint] must be [threshold] 
over [window period], measured [how/where].

Examples:

Service SLI SLO
API Gateway Request success rate 99.9% of requests return 2xx over any 30-day window
Checkout p99 latency < 500ms for 99.9% of requests over any 7-day window
Search p50 latency < 100ms for 99% of search queries over any 24-hour window
Auth service Availability No more than 43 minutes downtime per month (99.9%)

SLO Implementation with Prometheus

# prometheus/rules/slo.yaml
groups:
  - name: slo.rules
    interval: 30s
    rules:
      # SLI: success rate
      - record: job:request_success_rate:rate5m
        expr: |
          sum(rate(http_requests_total{status=~"2.."}[5m])) by (job)
          /
          sum(rate(http_requests_total[5m])) by (job)
      
      # SLO: alert when success rate drops below 99.9%
      - alert: SLOViolationErrorRate
        expr: job:request_success_rate:rate5m{job="api-gateway"} < 0.999
        for: 2m
        labels:
          severity: critical
          slo: error-rate
        annotations:
          summary: "SLO violation: error rate exceeds 0.1%"
          description: "Current success rate: {{ $value | humanizePercentage }}"
      
      # SLI: p99 latency
      - record: job:http_request_duration_p99:rate5m
        expr: |
          histogram_quantile(0.99,
            sum(rate(http_request_duration_seconds_bucket[5m])) by (job, le)
          )
      
      # SLO: p99 latency
      - alert: SLOViolationLatency
        expr: job:http_request_duration_p99:rate5m{job="checkout"} > 0.5
        for: 5m
        labels:
          severity: warning
          slo: latency
        annotations:
          summary: "SLO violation: p99 latency exceeds 500ms"

SLA Testing with Load Tests

Load tests are how you verify SLAs before production. Define your SLOs as assertions:

# tests/sla/test_api_sla.py
import pytest
import asyncio
import aiohttp
import time
import statistics
from dataclasses import dataclass, field
from typing import List

@dataclass
class RequestResult:
    status: int
    latency_ms: float
    error: str = None

@dataclass  
class LoadTestResults:
    results: List[RequestResult] = field(default_factory=list)
    
    @property
    def success_rate(self) -> float:
        if not self.results:
            return 0
        successes = sum(1 for r in self.results if 200 <= r.status < 300)
        return successes / len(self.results)
    
    @property
    def p50_ms(self) -> float:
        latencies = [r.latency_ms for r in self.results if r.error is None]
        return statistics.median(latencies) if latencies else 0
    
    @property
    def p95_ms(self) -> float:
        latencies = sorted(r.latency_ms for r in self.results if r.error is None)
        if not latencies:
            return 0
        idx = int(len(latencies) * 0.95)
        return latencies[idx]
    
    @property
    def p99_ms(self) -> float:
        latencies = sorted(r.latency_ms for r in self.results if r.error is None)
        if not latencies:
            return 0
        idx = int(len(latencies) * 0.99)
        return latencies[idx]
    
    @property
    def error_count(self) -> int:
        return sum(1 for r in self.results if r.error or r.status >= 500)

async def run_load_test(
    url: str,
    concurrent_users: int,
    duration_seconds: int,
) -> LoadTestResults:
    """Run a load test and collect results."""
    results = LoadTestResults()
    end_time = time.time() + duration_seconds
    
    async def make_request(session):
        start = time.perf_counter()
        try:
            async with session.get(url, timeout=aiohttp.ClientTimeout(total=10)) as resp:
                await resp.read()
                latency_ms = (time.perf_counter() - start) * 1000
                results.results.append(RequestResult(
                    status=resp.status,
                    latency_ms=latency_ms,
                ))
        except Exception as e:
            latency_ms = (time.perf_counter() - start) * 1000
            results.results.append(RequestResult(
                status=0, latency_ms=latency_ms, error=str(e)
            ))
    
    async with aiohttp.ClientSession() as session:
        while time.time() < end_time:
            tasks = [make_request(session) for _ in range(concurrent_users)]
            await asyncio.gather(*tasks)
            await asyncio.sleep(0.1)
    
    return results

@pytest.mark.asyncio
@pytest.mark.sla
async def test_api_error_rate_sla():
    """API must achieve 99.9% success rate under load."""
    results = await run_load_test(
        url="https://staging.example.com/api/products",
        concurrent_users=50,
        duration_seconds=60,
    )
    
    assert len(results.results) >= 100, "Insufficient requests for statistical validity"
    
    assert results.success_rate >= 0.999, \
        f"SLA violation: success rate {results.success_rate:.4f} < 99.9%\n" \
        f"Errors: {results.error_count}/{len(results.results)}"

@pytest.mark.asyncio
@pytest.mark.sla
async def test_api_latency_sla():
    """p99 latency must be under 200ms under normal load."""
    results = await run_load_test(
        url="https://staging.example.com/api/products",
        concurrent_users=20,
        duration_seconds=60,
    )
    
    assert results.p99_ms <= 200, \
        f"SLA violation: p99 latency {results.p99_ms:.1f}ms exceeds 200ms\n" \
        f"p50={results.p50_ms:.1f}ms, p95={results.p95_ms:.1f}ms, p99={results.p99_ms:.1f}ms"

@pytest.mark.asyncio
@pytest.mark.sla
async def test_checkout_latency_sla():
    """Checkout endpoint p99 must be under 500ms."""
    results = await run_load_test(
        url="https://staging.example.com/api/checkout",
        concurrent_users=10,
        duration_seconds=120,
    )
    
    assert results.p99_ms <= 500, \
        f"Checkout SLA violation: p99={results.p99_ms:.1f}ms > 500ms"

k6 SLA Tests

k6 is excellent for SLA testing with threshold-based pass/fail:

// tests/sla/checkout-sla-test.js
import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  // Load profile
  scenarios: {
    normal_load: {
      executor: 'constant-vus',
      vus: 50,
      duration: '5m',
    },
    peak_load: {
      executor: 'ramping-vus',
      startTime: '5m',
      stages: [
        { duration: '2m', target: 200 },  // Ramp up
        { duration: '3m', target: 200 },  // Hold
        { duration: '1m', target: 0 },    // Ramp down
      ],
    },
  },
  
  // SLA thresholds — test fails if any are violated
  thresholds: {
    // Error rate: < 0.1%
    'http_req_failed': ['rate<0.001'],
    
    // Latency SLOs
    'http_req_duration': [
      'p(50)<100',   // p50 < 100ms
      'p(95)<200',   // p95 < 200ms
      'p(99)<500',   // p99 < 500ms
    ],
    
    // Checkout-specific SLO
    'http_req_duration{endpoint:checkout}': ['p(99)<500'],
    
    // Custom business metric: checkout success rate
    'checks{type:checkout_success}': ['rate>0.99'],
  },
};

const BASE_URL = __ENV.TARGET_URL || 'https://staging.example.com';

export default function () {
  // Browse products
  const productsRes = http.get(`${BASE_URL}/api/products`, {
    tags: { endpoint: 'products' },
  });
  
  check(productsRes, {
    'products 200': (r) => r.status === 200,
    'products has items': (r) => JSON.parse(r.body).items.length > 0,
  });
  
  sleep(1);
  
  // Checkout
  const checkoutRes = http.post(
    `${BASE_URL}/api/checkout`,
    JSON.stringify({ product_id: 'test-001', quantity: 1 }),
    {
      headers: { 'Content-Type': 'application/json' },
      tags: { endpoint: 'checkout' },
    }
  );
  
  check(checkoutRes, {
    'checkout success': (r) => r.status === 200 || r.status === 201,
  }, { type: 'checkout_success' });
  
  sleep(2);
}

Run and fail CI on SLA violation:

k6 run --env TARGET_URL=https://staging.example.com tests/sla/checkout-sla-test.js

# k6 exits with non-zero code if thresholds are violated
# This fails the CI pipeline

Synthetic SLA Monitoring

Load tests verify SLAs at deployment time. Synthetic monitoring verifies them continuously in production:

# monitoring/sla_monitor.py
import time
import requests
import statistics
from prometheus_client import Gauge, Counter, Histogram, start_http_server

# Prometheus metrics for SLA tracking
SLA_SUCCESS_RATE = Gauge('sla_success_rate', 'Rolling 1h success rate', ['service'])
SLA_P99_LATENCY = Gauge('sla_p99_latency_ms', 'Rolling 1h p99 latency', ['service'])
SLA_VIOLATION = Counter('sla_violations_total', 'SLA violations', ['service', 'metric'])

class SLAMonitor:
    def __init__(self, endpoint: str, service_name: str):
        self.endpoint = endpoint
        self.service_name = service_name
        self.history = []  # (timestamp, latency_ms, success)
    
    def probe(self):
        """Make one synthetic request and record result."""
        start = time.perf_counter()
        success = False
        latency_ms = 0
        
        try:
            response = requests.get(self.endpoint, timeout=10)
            latency_ms = (time.perf_counter() - start) * 1000
            success = 200 <= response.status_code < 300
        except Exception:
            latency_ms = (time.perf_counter() - start) * 1000
        
        self.history.append((time.time(), latency_ms, success))
        
        # Keep only last 1 hour
        cutoff = time.time() - 3600
        self.history = [(t, l, s) for t, l, s in self.history if t > cutoff]
        
        self._update_metrics()
    
    def _update_metrics(self):
        """Update Prometheus metrics from recent history."""
        if not self.history:
            return
        
        recent = self.history
        success_rate = sum(s for _, _, s in recent) / len(recent)
        latencies = sorted(l for _, l, s in recent if s)  # Only successful requests
        
        SLA_SUCCESS_RATE.labels(service=self.service_name).set(success_rate)
        
        if latencies:
            p99_idx = int(len(latencies) * 0.99)
            SLA_P99_LATENCY.labels(service=self.service_name).set(latencies[p99_idx])
        
        # Check SLA violations
        if success_rate < 0.999:
            SLA_VIOLATION.labels(
                service=self.service_name, metric="availability"
            ).inc()
        
        if latencies and latencies[int(len(latencies) * 0.99)] > 200:
            SLA_VIOLATION.labels(
                service=self.service_name, metric="latency"
            ).inc()

SLA Compliance Reporting

Generate SLA compliance reports from production data:

# scripts/sla_report.py
from datetime import datetime, timedelta
import requests

def generate_monthly_sla_report(
    prometheus_url: str,
    services: list[str],
    month: datetime,
) -> dict:
    """Generate SLA compliance report for a given month."""
    
    start = month.replace(day=1)
    if month.month == 12:
        end = month.replace(year=month.year + 1, month=1, day=1)
    else:
        end = month.replace(month=month.month + 1, day=1)
    
    duration_seconds = (end - start).total_seconds()
    
    report = {}
    
    for service in services:
        # Query success rate over the entire month
        query = f"""
        sum(increase(http_requests_total{{job="{service}",status=~"2.."}}[{int(duration_seconds)}s]))
        /
        sum(increase(http_requests_total{{job="{service}"}}[{int(duration_seconds)}s]))
        """
        
        response = requests.get(
            f"{prometheus_url}/api/v1/query",
            params={"query": query, "time": end.timestamp()}
        )
        
        data = response.json()
        
        if data["data"]["result"]:
            success_rate = float(data["data"]["result"][0]["value"][1])
        else:
            success_rate = None
        
        # Calculate SLA compliance
        sla_target = 0.999  # 99.9% SLA
        slo_target = 0.9995  # Internal SLO is stricter
        
        report[service] = {
            "success_rate": success_rate,
            "sla_target": sla_target,
            "sla_met": success_rate >= sla_target if success_rate else None,
            "slo_met": success_rate >= slo_target if success_rate else None,
            "downtime_minutes": (1 - success_rate) * duration_seconds / 60 if success_rate else None,
        }
    
    return report

Summary

SLA testing requires more than defining targets — it requires continuous verification:

  1. Define testable SLOs — specific, measurable, time-bounded targets
  2. Load test SLAs pre-production — catch violations before customers do
  3. Use threshold-based test failures — k6 and pytest assertions fail the pipeline on SLA violation
  4. Synthetic monitoring — continuously verify SLAs in production
  5. Generate compliance reports — track SLA adherence over time

The SLA test is the contract between your engineering team and your customers. Automate its verification.

Start now free