Toil Automation and Testing SRE Runbooks: Making Automation Trustworthy

Toil Automation and Testing SRE Runbooks: Making Automation Trustworthy

SRE teams automate toil to reduce manual work. But automation is code, and code has bugs. An automated runbook that breaks during an incident is worse than no automation — it adds confusion and delays while you're already under pressure.

Testing your automation and runbooks is not optional. It's the difference between automation that helps and automation that fails when you need it most.

What Is Toil and Why Does Its Automation Need Testing?

Toil is repetitive operational work that could be automated. Restarting a service when memory exceeds a threshold, adding capacity when queue depth grows, rotating credentials on a schedule — these are toil.

When you automate toil, you create:

  • Automation scripts that run actions
  • Runbooks that document both manual and automated procedures
  • Triggers that decide when automation should run

Each of these can be wrong. An automation script that restarts the wrong service. A runbook with outdated commands. A trigger that fires at the wrong threshold. Testing catches these before the production incident where they matter.

Testing Automation Scripts

Automation scripts are production code. Test them like production code.

# tests/automation/test_service_restart.py

class TestServiceRestartAutomation:
    
    def test_restarts_correct_service(self, mock_k8s):
        """Restart automation must target the correct deployment."""
        automation = ServiceRestartAutomation(
            service_name="api",
            namespace="production"
        )
        
        automation.execute()
        
        # Verify the correct deployment was restarted
        assert mock_k8s.rollout_restart.called
        call_args = mock_k8s.rollout_restart.call_args
        assert call_args.kwargs["deployment"] == "api"
        assert call_args.kwargs["namespace"] == "production"
    
    def test_does_not_restart_when_service_healthy(self, mock_k8s, mock_metrics):
        """Automation must not restart a healthy service."""
        mock_metrics.set_memory_usage("api", 60)  # 60% — under 80% threshold
        
        automation = MemoryBasedRestartAutomation(
            service="api",
            restart_threshold=80
        )
        
        automation.check_and_execute()
        
        assert not mock_k8s.rollout_restart.called, \
            "Service restarted unnecessarily when memory was healthy"
    
    def test_restarts_when_memory_exceeds_threshold(self, mock_k8s, mock_metrics):
        """Automation must restart service when memory exceeds threshold."""
        mock_metrics.set_memory_usage("api", 92)  # 92% — over 80% threshold
        
        automation = MemoryBasedRestartAutomation(
            service="api",
            restart_threshold=80
        )
        
        automation.check_and_execute()
        
        assert mock_k8s.rollout_restart.called, \
            "Service not restarted despite memory exceeding threshold"
    
    def test_sends_alert_on_automation_failure(self, mock_k8s, mock_alerting):
        """If automation fails, it must alert rather than silently fail."""
        mock_k8s.rollout_restart.side_effect = KubernetesAPIError("Permission denied")
        
        automation = ServiceRestartAutomation(service="api")
        
        with pytest.raises(AutomationFailedError):
            automation.execute()
        
        # Alert must be sent when automation fails
        assert mock_alerting.send.called
        alert_msg = mock_alerting.send.call_args.kwargs["message"]
        assert "automation" in alert_msg.lower()
        assert "failed" in alert_msg.lower()

Runbook Testing

Runbooks go stale. A command that worked 6 months ago may no longer exist. Test your runbooks regularly.

Runbook validation framework:

class RunbookValidator:
    """Validate that runbook commands work in a staging environment."""
    
    def validate_runbook(self, runbook_path: str) -> RunbookValidationResult:
        runbook = parse_runbook(runbook_path)
        results = []
        
        for step in runbook.steps:
            if step.type == "command":
                result = self._validate_command(step)
            elif step.type == "check":
                result = self._validate_check(step)
            elif step.type == "api_call":
                result = self._validate_api_call(step)
            else:
                result = StepResult.skipped(step, reason="Manual step")
            
            results.append(result)
        
        return RunbookValidationResult(steps=results)
    
    def _validate_command(self, step: RunbookStep) -> StepResult:
        """Validate that a command exists and runs without error."""
        # Run in dry-run mode if supported
        cmd = step.command
        if "--dry-run" in step.flags or step.supports_dry_run:
            cmd += " --dry-run"
        
        result = subprocess.run(
            cmd,
            shell=True,
            capture_output=True,
            timeout=30,
            env={**os.environ, "ENVIRONMENT": "staging"}
        )
        
        if result.returncode != 0:
            return StepResult.failed(
                step=step,
                error=result.stderr.decode(),
                recommendation="Update this command — it failed in staging"
            )
        
        return StepResult.passed(step)

Run runbook validation in CI weekly. A failing step in a runbook is a bug — fix it before the incident where it matters.

Example runbook test:

def test_database_recovery_runbook():
    """Database recovery runbook must be valid and executable in staging."""
    validator = RunbookValidator(environment="staging")
    result = validator.validate_runbook("runbooks/database-recovery.md")
    
    failed_steps = [s for s in result.steps if s.status == "failed"]
    
    assert len(failed_steps) == 0, (
        f"Runbook has {len(failed_steps)} failing steps:\n"
        + "\n".join(f"  Step {s.number}: {s.error}" for s in failed_steps)
    )

Testing Alert-to-Runbook Linkage

Every alert should link to a runbook. Every runbook should be tested. Test the linkage:

def test_all_alerts_have_runbook_links():
    """Every alert definition must include a runbook URL."""
    alert_rules = load_prometheus_alert_rules()
    
    alerts_without_runbooks = [
        rule["alert"] for rule in alert_rules
        if "runbook_url" not in rule.get("annotations", {})
    ]
    
    assert len(alerts_without_runbooks) == 0, (
        f"Alerts missing runbook URLs:\n"
        + "\n".join(f"  - {name}" for name in alerts_without_runbooks)
    )

def test_all_runbook_urls_are_reachable():
    """Runbook URLs in alerts must return 200 (not 404)."""
    alert_rules = load_prometheus_alert_rules()
    
    broken_links = []
    for rule in alert_rules:
        url = rule.get("annotations", {}).get("runbook_url")
        if url:
            response = requests.get(url, timeout=10)
            if response.status_code != 200:
                broken_links.append({
                    "alert": rule["alert"],
                    "url": url,
                    "status": response.status_code
                })
    
    assert len(broken_links) == 0, (
        f"Broken runbook links:\n"
        + "\n".join(f"  - {l['alert']}: {l['url']} → {l['status']}" for l in broken_links)
    )

Testing Automation Idempotency

Automation runs multiple times. It must be idempotent — running it twice should have the same result as running it once.

def test_credential_rotation_is_idempotent():
    """Rotating credentials twice must succeed and leave system in correct state."""
    rotator = CredentialRotator(service="api-gateway")
    
    # First rotation
    result1 = rotator.rotate()
    assert result1.success
    first_credential = get_current_credential("api-gateway")
    
    # Second rotation (immediately after)
    result2 = rotator.rotate()
    assert result2.success, f"Second rotation failed: {result2.error}"
    
    second_credential = get_current_credential("api-gateway")
    
    # Both rotations succeeded — second should have created a new credential
    assert second_credential != first_credential, \
        "Second rotation did not create a new credential"
    
    # System must be using the latest credential
    assert is_credential_active(second_credential), \
        "System is not using the most recent credential after second rotation"

def test_scaling_automation_is_idempotent():
    """Scaling to N replicas twice must result in exactly N replicas."""
    scaler = AutoScaler(service="worker", namespace="production")
    
    scaler.scale_to(10)
    scaler.scale_to(10)  # Run again
    
    current_replicas = get_replica_count("worker", "production")
    assert current_replicas == 10, \
        f"Expected 10 replicas after idempotent scale, got {current_replicas}"

Simulation-Based Runbook Testing

The most valuable runbook test is a full incident simulation: inject the failure the runbook is designed for, execute the runbook steps, and verify the system recovers.

def test_high_error_rate_runbook():
    """Simulate high error rate incident and verify runbook resolves it."""
    # Step 1: Inject the failure
    inject_fault(service="api", error_rate=0.5)
    
    # Wait for alert to fire
    time.sleep(60)
    alert = wait_for_alert("HighErrorRate", timeout=120)
    assert alert is not None, "HighErrorRate alert did not fire"
    
    # Step 2: Execute runbook
    runbook = HighErrorRateRunbook()
    
    # Step 2a: Identify error source
    error_source = runbook.identify_error_source()
    assert error_source is not None, "Runbook failed to identify error source"
    
    # Step 2b: Check recent deployments
    recent_deploy = runbook.check_recent_deployments()
    
    # Step 2c: Roll back if deployment is the cause
    if recent_deploy and recent_deploy.correlates_with_errors:
        rollback_result = runbook.rollback_deployment()
        assert rollback_result.success
    else:
        # Step 2d: Scale up if it's a capacity issue
        runbook.scale_up_service()
    
    # Remove injected fault
    remove_fault(service="api")
    
    # Step 3: Verify recovery
    time.sleep(30)
    current_error_rate = get_error_rate("api")
    assert current_error_rate < 0.01, \
        f"Error rate still elevated after runbook: {current_error_rate:.2%}"
    
    # Alert should resolve
    active_alerts = get_active_alerts("HighErrorRate")
    assert len(active_alerts) == 0, "Alert still firing after runbook resolved the issue"

Continuous Runbook Validation with HelpMeTest

HelpMeTest can continuously validate that your runbooks' target systems are reachable and responding — before you need them in an incident. Monitor the Kubernetes API endpoint your runbooks target, the monitoring dashboards they reference, and the health check URLs they verify.

When any of these are unreachable, you'll know before an incident forces you to discover it.

Summary

Toil automation and runbook testing requires:

  1. Unit tests for automation scripts covering both normal and error cases
  2. Runbook validation that checks every command still works in staging
  3. Alert-to-runbook linkage tests — every alert has a working runbook URL
  4. Idempotency tests — automation run twice produces the correct result
  5. Full incident simulations that run the runbook against an injected failure

Untested automation is a liability. Tested automation is your most reliable on-call engineer.

Read more

Start now free