Testing

Testing SLI Implementation with Prometheus and Alertmanager

HelpMeTest

HelpMeTest

— 7 min read
Testing SLI Implementation with Prometheus and Alertmanager

Your SLI implementation is code. Your alerting rules are code. Your Prometheus recording rules are code. All of it can and should be tested — before it goes into production, before an incident shows you that your alert fires three hours too late or routes to the wrong Slack channel.

This post covers the full testing stack for SLI instrumentation: unit testing recording rules and alerting rules with promtool, validating alert thresholds with synthetic load tests, and verifying that PagerDuty and Slack routing works end-to-end.

What to Test in SLI/Alerting Code

The things that commonly go wrong with Prometheus-based SLIs:

  1. Recording rules have bugs — wrong PromQL expression produces incorrect aggregation
  2. Alert thresholds are wrong — alert fires too early (noise) or too late (missed incidents)
  3. Alert firing windows are wrong — for: 5m is too long for a critical outage
  4. Alertmanager routing is wrong — P1 alerts route to a general channel instead of PagerDuty
  5. Alert inhibition is misconfigured — a higher-severity alert should suppress lower-severity noise
  6. Label matchers are wrong — alert fires for all environments instead of just production

All of these are testable with promtool and integration tests against a live Alertmanager.

Writing Promtool Unit Tests for Recording Rules

Promtool's test rules subcommand lets you write unit tests for Prometheus rules. You define input series, run the rules against them at specific timestamps, and assert on output values.

First, the recording rule under test:

# rules/sli-availability.yaml
groups:
  - name: sli.availability
    interval: 30s
    rules:
      - record: job:http_requests:rate5m
        expr: |
          sum by (job, status_class) (
            rate(http_requests_total[5m])
          )

      - record: job:sli_availability:rate5m
        expr: |
          (
            sum by (job) (job:http_requests:rate5m{status_class="success"})
            /
            sum by (job) (job:http_requests:rate5m)
          )

Unit test file:

# tests/rules/sli-availability_test.yaml
rule_files:
  - ../../rules/sli-availability.yaml

evaluation_interval: 30s

tests:
  - name: "availability is 1.0 when all requests succeed"
    interval: 30s
    input_series:
      - series: 'http_requests_total{job="payments-api", status_class="success"}'
        values: "0 100 200 300 400 500"
      - series: 'http_requests_total{job="payments-api", status_class="error"}'
        values: "0 0 0 0 0 0"
    promql_expr_test:
      - expr: job:sli_availability:rate5m{job="payments-api"}
        eval_time: 2m30s
        exp_samples:
          - labels: '{job="payments-api"}'
            value: 1.0

  - name: "availability reflects error rate correctly"
    interval: 30s
    input_series:
      - series: 'http_requests_total{job="payments-api", status_class="success"}'
        values: "0 90 180 270 360 450"
      - series: 'http_requests_total{job="payments-api", status_class="error"}'
        values: "0 10 20 30 40 50"
    promql_expr_test:
      - expr: job:sli_availability:rate5m{job="payments-api"}
        eval_time: 2m30s
        exp_samples:
          - labels: '{job="payments-api"}'
            value: 0.9  # 90 success / 100 total

  - name: "recording rule handles zero total requests"
    interval: 30s
    input_series:
      - series: 'http_requests_total{job="idle-service", status_class="success"}'
        values: "0 0 0 0 0 0"
      - series: 'http_requests_total{job="idle-service", status_class="error"}'
        values: "0 0 0 0 0 0"
    promql_expr_test:
      - expr: job:sli_availability:rate5m{job="idle-service"}
        eval_time: 2m30s
        exp_samples: []  # no samples when rate is 0/0

Run the tests:

promtool test rules tests/rules/sli-availability_test.yaml

Output when passing:

Unit Testing:  tests/rules/sli-availability_test.yaml
  SUCCESS

When failing:

Unit Testing:  tests/rules/sli-availability_test.yaml
FAILED:
  - name "availability reflects error rate correctly" (line 23)
      expr "job:sli_availability:rate5m{job="payments-api"}", time: 2m30s, 
        expected: [{job="payments-api"} => 0.9]
        got:      [{job="payments-api"} => 0.8999999999999999]

Note: floating point comparisons in promtool are approximate. You may need to round or use the ~= operator for certain calculations.

Testing Alerting Rules with promtool

Alert unit tests work the same way but assert on alert firing state:

# rules/slo-alerts.yaml
groups:
  - name: slo.alerts
    rules:
      - alert: SLOAvailabilityBurning
        expr: |
          (
            job:sli_availability:rate5m < 0.99
          ) and on(job) (
            job:http_requests:rate5m > 1
          )
        for: 2m
        labels:
          severity: warning
          slo: availability
        annotations:
          summary: "{{ $labels.job }} availability below 99%"
          description: "Current availability: {{ $value | humanizePercentage }}"

      - alert: SLOAvailabilityCritical
        expr: |
          job:sli_availability:rate5m < 0.95
        for: 1m
        labels:
          severity: critical
          slo: availability
        annotations:
          summary: "{{ $labels.job }} availability critically low"

Test file:

# tests/rules/slo-alerts_test.yaml
rule_files:
  - ../../rules/sli-availability.yaml
  - ../../rules/slo-alerts.yaml

evaluation_interval: 1m

tests:
  - name: "SLOAvailabilityBurning fires when availability drops below 99%"
    interval: 1m
    input_series:
      - series: 'http_requests_total{job="payments-api", status_class="success"}'
        values: "0 98 196 294 392 490"
      - series: 'http_requests_total{job="payments-api", status_class="error"}'
        values: "0 2 4 6 8 10"
    alert_rule_test:
      - eval_time: 3m
        alertname: SLOAvailabilityBurning
        exp_alerts:
          - exp_labels:
              severity: warning
              slo: availability
              job: payments-api
            exp_annotations:
              summary: "payments-api availability below 99%"

  - name: "SLOAvailabilityBurning does not fire when traffic is zero"
    interval: 1m
    input_series:
      - series: 'http_requests_total{job="idle-service", status_class="success"}'
        values: "0 0 0 0 0 0"
      - series: 'http_requests_total{job="idle-service", status_class="error"}'
        values: "0 0 0 0 0 0"
    alert_rule_test:
      - eval_time: 3m
        alertname: SLOAvailabilityBurning
        exp_alerts: []  # no false positive for idle services

  - name: "SLOAvailabilityBurning respects 'for: 2m' — not pending before that"
    interval: 1m
    input_series:
      - series: 'http_requests_total{job="payments-api", status_class="success"}'
        values: "0 98 196"
      - series: 'http_requests_total{job="payments-api", status_class="error"}'
        values: "0 2 4"
    alert_rule_test:
      - eval_time: 1m   # after 1 minute — should still be pending, not firing
        alertname: SLOAvailabilityBurning
        exp_alerts: []
      - eval_time: 3m   # after 3 minutes — should fire
        alertname: SLOAvailabilityBurning
        exp_alerts:
          - exp_labels:
              severity: warning
              job: payments-api

This last test is critical. A wrong for duration on a critical alert can mean 30 minutes of downtime before anyone is paged.

Validating Alert Thresholds with Synthetic Load Tests

Unit tests verify rule logic. They can't verify that the threshold is the right value for your system. You need to know: when latency actually exceeds your p95 target under real load, does the alert fire within an acceptable time?

Set up a test environment with a real Prometheus and Alertmanager:

# docker-compose.test.yml
services:
  prometheus:
    image: prom/prometheus:v2.48.0
    volumes:
      - ./prometheus-test.yml:/etc/prometheus/prometheus.yml
      - ./rules:/etc/prometheus/rules
    ports:
      - "9090:9090"

  alertmanager:
    image: prom/alertmanager:v0.26.0
    volumes:
      - ./alertmanager-test.yml:/etc/alertmanager/alertmanager.yml
    ports:
      - "9093:9093"

  test-service:
    image: prom/prometheus  # or your own service
    ports:
      - "8080:8080"

Then write a Python integration test that:

  1. Starts the stack
  2. Injects metrics that should trigger the alert
  3. Polls Alertmanager until the alert appears (or times out)
# tests/integration/test_alert_thresholds.py
import time
import requests
import pytest
from prometheus_client import CollectorRegistry, Counter, push_to_gateway

PROMETHEUS_URL = "http://localhost:9090"
ALERTMANAGER_URL = "http://localhost:9093"
PUSHGATEWAY_URL = "http://localhost:9091"

def push_metric(name, labels, value):
    """Push a metric to Prometheus via pushgateway"""
    registry = CollectorRegistry()
    c = Counter(name, 'test metric', list(labels.keys()), registry=registry)
    c.labels(**labels).inc(value)
    push_to_gateway(PUSHGATEWAY_URL, job='test', registry=registry)

def get_firing_alerts():
    response = requests.get(f"{ALERTMANAGER_URL}/api/v2/alerts")
    return [a for a in response.json() if a['status']['state'] == 'active']

def wait_for_alert(alert_name, timeout_seconds=300):
    start = time.time()
    while time.time() - start < timeout_seconds:
        alerts = get_firing_alerts()
        matching = [a for a in alerts if a['labels'].get('alertname') == alert_name]
        if matching:
            return matching[0]
        time.sleep(10)
    return None

class TestAlertThresholds:
    def test_availability_alert_fires_on_high_error_rate(self):
        """Inject 5% error rate — SLOAvailabilityBurning should fire within 5 minutes"""
        job = "test-payments-api"
        
        # Inject error rate data for 5 minutes
        for _ in range(10):
            push_metric(
                'http_requests_total',
                {'job': job, 'status_class': 'success', 'instance': 'test'},
                95
            )
            push_metric(
                'http_requests_total', 
                {'job': job, 'status_class': 'error', 'instance': 'test'},
                5
            )
            time.sleep(30)

        alert = wait_for_alert('SLOAvailabilityBurning', timeout_seconds=300)
        
        assert alert is not None, "Alert did not fire within 5 minutes of high error rate"
        assert alert['labels']['job'] == job
        assert alert['labels']['severity'] == 'warning'

    def test_availability_alert_does_not_fire_on_brief_blip(self):
        """A 30-second spike should not fire (for: 2m must be respected)"""
        job = "test-stable-service"
        
        # Short burst of errors
        push_metric('http_requests_total', {'job': job, 'status_class': 'error'}, 100)
        push_metric('http_requests_total', {'job': job, 'status_class': 'success'}, 900)
        time.sleep(60)
        # Back to healthy
        push_metric('http_requests_total', {'job': job, 'status_class': 'error'}, 0)
        push_metric('http_requests_total', {'job': job, 'status_class': 'success'}, 1000)
        time.sleep(60)

        alerts = get_firing_alerts()
        matching = [a for a in alerts if a['labels'].get('alertname') == 'SLOAvailabilityBurning'
                    and a['labels'].get('job') == job]
        
        assert len(matching) == 0, "Alert fired on a brief blip — for: duration is wrong"

Verifying PagerDuty and Slack Routing

Alertmanager routing configuration has its own failure modes. Route trees are complex; a mismatch in label selectors means the right person never gets paged.

Test routing with the amtool CLI:

# Test that a critical alert routes to PagerDuty
amtool config routes test \
  --config.file=alertmanager.yml \
  --verify.receivers=pagerduty-critical \
  alertname=SLOAvailabilityCritical severity=critical environment=production

# Expected output:
# Receiver: pagerduty-critical
# Resolved: true

# Test that a warning alert routes to Slack (not PagerDuty)
amtool config routes test \
  --config.file=alertmanager.yml \
  --verify.receivers=slack-alerts \
  alertname=SLOAvailabilityBurning severity=warning environment=production

# Test that staging alerts do NOT page PagerDuty
amtool config routes test \
  --config.file=alertmanager.yml \
  alertname=SLOAvailabilityCritical severity=critical environment=staging
# Should route to slack-staging, not pagerduty

Wrap this in a CI test:

#!/bin/bash
# ci/test-alertmanager-routing.sh

set -euo pipefail

AMTOOL="docker run --rm -v $(pwd):/config prom/alertmanager amtool"

test_routing() {
  local test_name="$1"
  local expected_receiver="$2"
  shift 2
  local labels=("$@")

  result=$($AMTOOL config routes test \
    --config.file=/config/alertmanager.yml \
    "${labels[@]}" 2>&1)

  if echo "$result" | grep -q "Receiver: $expected_receiver"; then
    echo "PASS: $test_name"
  else
    echo "FAIL: $test_name"
    echo "  Expected receiver: $expected_receiver"
    echo "  Got: $result"
    exit 1
  fi
}

test_routing \
  "critical production alert goes to PagerDuty" \
  "pagerduty-critical" \
  alertname=SLOAvailabilityCritical severity=critical environment=production

test_routing \
  "warning alert goes to Slack" \
  "slack-sre-alerts" \
  alertname=SLOAvailabilityBurning severity=warning environment=production

test_routing \
  "staging critical alert stays in Slack" \
  "slack-staging" \
  alertname=SLOAvailabilityCritical severity=critical environment=staging

echo "All routing tests passed"

CI Integration

Put it all together in a GitHub Actions workflow:

# .github/workflows/prometheus-rules.yml
name: Test Prometheus Rules

on:
  pull_request:
    paths:
      - 'rules/**'
      - 'alertmanager.yml'
      - 'tests/rules/**'

jobs:
  test-rules:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Install promtool
        run: |
          wget -q https://github.com/prometheus/prometheus/releases/download/v2.48.0/prometheus-2.48.0.linux-amd64.tar.gz
          tar xf prometheus-2.48.0.linux-amd64.tar.gz
          sudo mv prometheus-2.48.0.linux-amd64/promtool /usr/local/bin/

      - name: Lint rules
        run: promtool check rules rules/**/*.yaml

      - name: Unit test recording rules
        run: promtool test rules tests/rules/*_test.yaml

      - name: Install amtool
        run: |
          wget -q https://github.com/prometheus/alertmanager/releases/download/v0.26.0/alertmanager-0.26.0.linux-amd64.tar.gz
          tar xf alertmanager-0.26.0.linux-amd64.tar.gz
          sudo mv alertmanager-0.26.0.linux-amd64/amtool /usr/local/bin/

      - name: Test Alertmanager routing
        run: bash ci/test-alertmanager-routing.sh

What This Gives You

Before this testing setup, alerting rule changes were deployed by hand and validated by waiting to see if an alert fired in production. After:

  • Rule syntax errors caught before merge
  • Recording rule PromQL bugs caught before they corrupt SLO calculations
  • Alert threshold issues caught before they cause missed incidents
  • Routing bugs caught before they cause silent failures on-call

Your alerting system is the last line of defense before a human is woken up. Test it like it matters — because it does.

Read more

Start now free