Testing

Disaster Recovery Testing Guide: How to Validate RTO and RPO

HelpMeTest

HelpMeTest

— 9 min read
Disaster Recovery Testing Guide: How to Validate RTO and RPO

Most disaster recovery plans have never actually been tested. They exist as documents — carefully written, officially approved, and completely unvalidated. When an actual disaster hits, teams discover that the documented 4-hour RTO takes 14 hours, the backup restoration process is broken, and nobody has the credentials to access the failover environment.

Disaster recovery testing is the engineering practice that ensures your DR plan works before you need it. This guide covers how to design, execute, and automate DR tests that give you real confidence in your recovery capabilities.

Understanding RTO and RPO

Before testing, you need to understand what you're validating.

Recovery Time Objective (RTO): The maximum acceptable downtime after a disaster. If your RTO is 4 hours, your system must be restored and serving users within 4 hours of the failure event.

Recovery Point Objective (RPO): The maximum acceptable data loss, expressed as time. An RPO of 1 hour means you can lose at most 1 hour of data — your backups must be at most 1 hour old at any point.

These are business requirements, not technical defaults. They should be defined by stakeholders based on business impact:

System Business Impact of 1hr Down Typical RTO Typical RPO
Payment processing $50k+/hour lost revenue 15 min 0 (zero data loss)
E-commerce store $10k/hour lost sales 1 hour 5 min
Internal dashboard Low 4 hours 1 hour
Marketing site Very low 24 hours 24 hours

The gap problem: Most teams have defined RTO/RPO targets, but have never measured whether they can actually achieve them. DR testing closes this gap.

Types of DR Tests

1. Tabletop Exercise

A discussion-based walkthrough of your DR plan. No systems are touched.

How it works: Key stakeholders gather and walk through the DR plan step-by-step. Someone plays the role of a "facilitator" describing what's happening, and the team describes their response.

What it validates:

  • Everyone knows their role in the DR process
  • Communication paths are clear (who calls whom?)
  • The written plan doesn't have obvious gaps
  • Decision points are identified

Limitations: Does not validate technical steps, timing, or actual system behavior.

When to use: Annually for senior leadership, quarterly for operational teams. Use as a prerequisite for more complex testing.

Example agenda:

09:00 — Scenario briefing: Primary datacenter loses power
09:15 — Detection: How do we know? Who gets paged?
09:30 — Assessment: Is this a temporary outage or full recovery needed?
09:45 — Decision: When do we declare a disaster and activate DR plan?
10:00 — Execution: Walk through DR runbook step by step
11:00 — Communication: Customer notification, status page updates
11:30 — Recovery validation: How do we know we're back?
11:45 — Debrief: Gaps identified, action items

2. Backup Restoration Test

Validates that your backups are complete, accessible, and restorable.

What it tests:

  • Backups are being created on schedule
  • Backup files are not corrupted
  • Restoration process works correctly
  • Data is complete and consistent after restore
  • Restoration time meets RPO
#!/bin/bash
<span class="hljs-comment"># backup_restore_test.sh — Validate backup restoration

<span class="hljs-built_in">set -euo pipefail

BACKUP_BUCKET=<span class="hljs-string">"s3://company-backups"
TEST_DB=<span class="hljs-string">"postgres://test-db-host:5432/restore_test"
TIMESTAMP=$(<span class="hljs-built_in">date -u +%Y-%m-%dT%H:%M:%S)

<span class="hljs-built_in">echo <span class="hljs-string">"=== Backup Restoration Test — $TIMESTAMP ==="

<span class="hljs-comment"># 1. Find the most recent backup
<span class="hljs-built_in">echo <span class="hljs-string">"Finding latest backup..."
LATEST_BACKUP=$(aws s3 <span class="hljs-built_in">ls <span class="hljs-string">"$BACKUP_BUCKET/postgres/" \
  --recursive <span class="hljs-pipe">| <span class="hljs-built_in">sort -k1,2 <span class="hljs-pipe">| <span class="hljs-built_in">tail -1 <span class="hljs-pipe">| awk <span class="hljs-string">'{print $4}')
<span class="hljs-built_in">echo <span class="hljs-string">"Latest backup: $LATEST_BACKUP"

BACKUP_DATE=$(<span class="hljs-built_in">echo <span class="hljs-string">"$LATEST_BACKUP" <span class="hljs-pipe">| grep -oP <span class="hljs-string">'\d{4}-\d{2}-\d{2}')
BACKUP_AGE_HOURS=$(( ($(date +%s) - $(date -d "<span class="hljs-variable">$BACKUP_DATE" +%s)) / 3600 ))

<span class="hljs-comment"># 2. Validate backup age meets RPO
MAX_AGE_HOURS=25  <span class="hljs-comment"># RPO = 24 hours, with 1 hour tolerance
<span class="hljs-keyword">if [ <span class="hljs-string">"$BACKUP_AGE_HOURS" -gt <span class="hljs-string">"$MAX_AGE_HOURS" ]; <span class="hljs-keyword">then
  <span class="hljs-built_in">echo <span class="hljs-string">"FAIL: Latest backup is $BACKUP_AGE_HOURS hours old (RPO: 24 hours)"
  <span class="hljs-built_in">exit 1
<span class="hljs-keyword">fi
<span class="hljs-built_in">echo <span class="hljs-string">"OK: Backup age ${BACKUP_AGE_HOURS}h is within RPO"

<span class="hljs-comment"># 3. Download and validate backup integrity
<span class="hljs-built_in">echo <span class="hljs-string">"Downloading backup..."
START_TIME=$(<span class="hljs-built_in">date +%s)
aws s3 <span class="hljs-built_in">cp <span class="hljs-string">"s3://$BACKUP_BUCKET/<span class="hljs-variable">$LATEST_BACKUP" /tmp/latest_backup.sql.gz

<span class="hljs-comment"># Check integrity
gzip -t /tmp/latest_backup.sql.gz <span class="hljs-pipe">|| {
  <span class="hljs-built_in">echo <span class="hljs-string">"FAIL: Backup file is corrupted"
  <span class="hljs-built_in">exit 1
}
<span class="hljs-built_in">echo <span class="hljs-string">"OK: Backup integrity verified"

<span class="hljs-comment"># 4. Restore to test database
<span class="hljs-built_in">echo <span class="hljs-string">"Restoring to test database..."
zcat /tmp/latest_backup.sql.gz <span class="hljs-pipe">| psql <span class="hljs-string">"$TEST_DB"

END_TIME=$(<span class="hljs-built_in">date +%s)
RESTORE_DURATION=$(( END_TIME - START_TIME ))
<span class="hljs-built_in">echo <span class="hljs-string">"Restore completed in ${RESTORE_DURATION}s"

<span class="hljs-comment"># 5. Validate data completeness
<span class="hljs-built_in">echo <span class="hljs-string">"Validating restored data..."
ROW_COUNT=$(psql <span class="hljs-string">"$TEST_DB" -t -c <span class="hljs-string">"SELECT COUNT(*) FROM orders")
EXPECTED_MIN=100000  <span class="hljs-comment"># We should have at least 100k orders

<span class="hljs-keyword">if [ <span class="hljs-string">"$ROW_COUNT" -lt <span class="hljs-string">"$EXPECTED_MIN" ]; <span class="hljs-keyword">then
  <span class="hljs-built_in">echo <span class="hljs-string">"FAIL: Restored ${ROW_COUNT} rows, expected >= <span class="hljs-variable">${EXPECTED_MIN}"
  <span class="hljs-built_in">exit 1
<span class="hljs-keyword">fi
<span class="hljs-built_in">echo <span class="hljs-string">"OK: Restored $ROW_COUNT rows"

<span class="hljs-comment"># 6. Check against RPO/RTO requirements
MAX_RESTORE_SECONDS=3600  <span class="hljs-comment"># RTO requires restore in < 1 hour
<span class="hljs-keyword">if [ <span class="hljs-string">"$RESTORE_DURATION" -gt <span class="hljs-string">"$MAX_RESTORE_SECONDS" ]; <span class="hljs-keyword">then
  <span class="hljs-built_in">echo <span class="hljs-string">"WARN: Restore took ${RESTORE_DURATION}s — may not meet RTO"
<span class="hljs-keyword">fi

<span class="hljs-built_in">echo <span class="hljs-string">"=== BACKUP TEST PASSED ==="
<span class="hljs-built_in">echo <span class="hljs-string">"Backup age: ${BACKUP_AGE_HOURS}h <span class="hljs-pipe">| Restore time: <span class="hljs-variable">${RESTORE_DURATION}s"

<span class="hljs-comment"># Cleanup
<span class="hljs-built_in">rm -f /tmp/latest_backup.sql.gz
psql <span class="hljs-string">"$TEST_DB" -c <span class="hljs-string">"DROP SCHEMA public CASCADE; CREATE SCHEMA public;"

3. Component Failover Test

Test failover for individual system components without touching production.

Scope: Single service, database, or infrastructure component.

Examples:

  • Fail over the primary database to the standby replica
  • Restart a Kubernetes deployment and verify pod recovery
  • Simulate an availability zone failure for a specific microservice
  • Test load balancer health check behavior when backends fail
# failover_test.py — Automated component failover testing

import time
import requests
import subprocess
from dataclasses import dataclass

@dataclass
class FailoverTestResult:
    component: str
    failover_detected_at: float   # seconds from failure to detection
    service_restored_at: float    # seconds from failure to restoration
    rto_met: bool
    requests_failed: int
    requests_total: int

def test_api_failover(
    primary_url: str,
    failover_trigger_cmd: str,
    rto_seconds: int = 30,
    test_duration: int = 120
) -> FailoverTestResult:
    """
    Test that an API survives a failover event within RTO.
    """
    start_time = time.time()
    failover_time = None
    recovery_time = None
    requests_failed = 0
    requests_total = 0
    was_down = False
    
    # Start sending requests in background
    def send_requests():
        nonlocal failover_time, recovery_time, requests_failed, requests_total, was_down
        
        while time.time() - start_time < test_duration:
            try:
                resp = requests.get(f"{primary_url}/health", timeout=2)
                if resp.status_code == 200:
                    if was_down:
                        recovery_time = time.time()
                        was_down = False
                else:
                    requests_failed += 1
                    if not was_down:
                        failover_time = time.time()
                        was_down = True
            except requests.RequestException:
                requests_failed += 1
                if not was_down:
                    failover_time = time.time()
                    was_down = True
            
            requests_total += 1
            time.sleep(0.5)
    
    import threading
    t = threading.Thread(target=send_requests)
    t.start()
    
    # Wait a bit, then trigger the failover
    time.sleep(10)
    print(f"Triggering failover: {failover_trigger_cmd}")
    subprocess.run(failover_trigger_cmd, shell=True, check=True)
    
    t.join(timeout=test_duration)
    
    failover_detected = failover_time - start_time if failover_time else None
    service_restored = recovery_time - failover_time if recovery_time and failover_time else None
    
    return FailoverTestResult(
        component=primary_url,
        failover_detected_at=failover_detected,
        service_restored_at=service_restored,
        rto_met=service_restored <= rto_seconds if service_restored else False,
        requests_failed=requests_failed,
        requests_total=requests_total
    )

# Example usage
result = test_api_failover(
    primary_url="http://api.internal",
    failover_trigger_cmd="kubectl delete pod -l app=api -n production --grace-period=0",
    rto_seconds=30
)

print(f"Failover detected: {result.failover_detected_at:.1f}s")
print(f"Service restored: {result.service_restored_at:.1f}s")
print(f"RTO met: {result.rto_met}")
print(f"Error rate during failover: {result.requests_failed}/{result.requests_total}")

4. Full DR Test (Simulation)

The most comprehensive test: simulate a complete disaster and execute the full DR plan in a staging/test environment.

What it tests:

  • Complete recovery time from scratch
  • Team coordination and communication
  • Runbook accuracy
  • All system dependencies
  • Data consistency after full restore

Environment: Should use production-equivalent infrastructure. Ideally, restore into an environment that mirrors production exactly but is completely isolated.

Frequency: Annually for most organizations; quarterly for critical systems.

5. Live Failover (Production)

Actually fail over production to the DR environment. The highest-confidence test — and the highest risk.

When to do this:

  • Planned maintenance windows
  • DR environment upgrades that require production traffic
  • Regulatory compliance requirements
  • After a major infrastructure change

Always have a rollback plan documented before executing a live failover.

Measuring and Tracking RTO/RPO

RTO Measurement

Track each step of your recovery process with timestamps:

from datetime import datetime, timezone
from dataclasses import dataclass, field

@dataclass
class RecoveryTimeline:
    incident_start: datetime | None = None
    incident_detected: datetime | None = None
    dr_invoked: datetime | None = None
    environment_provisioned: datetime | None = None
    data_restored: datetime | None = None
    services_started: datetime | None = None
    health_checks_passed: datetime | None = None
    traffic_shifted: datetime | None = None
    
    def time_to_recover(self) -> float | None:
        """Total RTO: incident start to traffic restored (seconds)."""
        if self.incident_start and self.traffic_shifted:
            return (self.traffic_shifted - self.incident_start).total_seconds()
        return None
    
    def time_to_detect(self) -> float | None:
        if self.incident_start and self.incident_detected:
            return (self.incident_detected - self.incident_start).total_seconds()
        return None
    
    def report(self) -> dict:
        return {
            'total_rto_minutes': self.time_to_recover() / 60 if self.time_to_recover() else None,
            'detection_time_minutes': self.time_to_detect() / 60 if self.time_to_detect() else None,
            'phases': {
                'detection': self.time_to_detect(),
                'dr_decision': (self.dr_invoked - self.incident_detected).total_seconds() if self.dr_invoked and self.incident_detected else None,
                'provisioning': (self.environment_provisioned - self.dr_invoked).total_seconds() if self.environment_provisioned and self.dr_invoked else None,
                'data_restore': (self.data_restored - self.environment_provisioned).total_seconds() if self.data_restored and self.environment_provisioned else None,
                'service_start': (self.health_checks_passed - self.services_started).total_seconds() if self.health_checks_passed and self.services_started else None,
                'traffic_shift': (self.traffic_shifted - self.health_checks_passed).total_seconds() if self.traffic_shifted and self.health_checks_passed else None,
            }
        }

RPO Measurement

Verify data completeness after recovery:

def validate_rpo(
    source_db_url: str,
    restored_db_url: str,
    max_age_minutes: int,
    key_table: str = "events"
) -> dict:
    """
    Validate that restored database meets RPO requirements.
    """
    import psycopg2
    
    with psycopg2.connect(source_db_url) as src_conn, \
         psycopg2.connect(restored_db_url) as rst_conn:
        
        src_cur = src_conn.cursor()
        rst_cur = rst_conn.cursor()
        
        # Get latest event timestamp in source
        src_cur.execute(f"SELECT MAX(created_at) FROM {key_table}")
        source_latest = src_cur.fetchone()[0]
        
        # Get latest event timestamp in restore
        rst_cur.execute(f"SELECT MAX(created_at) FROM {key_table}")
        restore_latest = rst_cur.fetchone()[0]
        
        # Calculate data loss
        data_loss_minutes = (source_latest - restore_latest).total_seconds() / 60
        
        # Count missing records
        src_cur.execute(f"SELECT COUNT(*) FROM {key_table} WHERE created_at > %s", (restore_latest,))
        missing_records = src_cur.fetchone()[0]
        
        return {
            'source_latest': source_latest.isoformat(),
            'restore_latest': restore_latest.isoformat(),
            'data_loss_minutes': data_loss_minutes,
            'missing_records': missing_records,
            'rpo_met': data_loss_minutes <= max_age_minutes,
            'rpo_target_minutes': max_age_minutes
        }

Automating DR Tests in CI

Schedule regular automated DR tests:

# .github/workflows/dr-test.yml
name: DR Test — Weekly

on:
  schedule:
    - cron: '0 2 * * 0'  # Sunday 2am UTC
  workflow_dispatch:      # Allow manual trigger

jobs:
  backup-restore-test:
    runs-on: ubuntu-latest
    environment: staging
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ secrets.DR_TEST_ROLE_ARN }}
          aws-region: us-east-1
      
      - name: Run backup restoration test
        run: |
          ./dr/backup_restore_test.sh
        env:
          TEST_DB: ${{ secrets.TEST_DB_URL }}
      
      - name: Run component failover tests
        run: |
          python dr/failover_tests.py \
            --component api \
            --rto-seconds 30 \
            --environment staging
      
      - name: Publish DR test report
        if: always()
        run: |
          python dr/publish_report.py \
            --results dr_results.json \
            --slack-webhook ${{ secrets.SLACK_DR_WEBHOOK }}

Common Failures Found During DR Testing

  1. Credentials expired: AWS access keys, database passwords, or SSL certificates used in the DR runbook have rotated since the plan was written
  2. Missing dependencies: A service has been added since the last DR test, and nobody updated the recovery order
  3. Backup lag: Backups were configured to run daily but actually haven't run in 3 days due to a silent failure
  4. DNS propagation delays: The failover completes in 30 minutes, but DNS TTL is set to 3600 — actual RTO is 90 minutes
  5. Capacity limits: The DR environment was sized for development, not production load
  6. Wrong version: The restore procedure uses an old script that doesn't handle the current database schema

Each of these failures, when discovered during a test, is a success. When discovered during an actual disaster, it's catastrophic.

Summary

DR testing follows a maturity ladder:

  1. Tabletop — Know the plan (do this first)
  2. Backup restoration — Prove backups work (do this monthly)
  3. Component failover — Prove individual systems recover (do this quarterly)
  4. Full DR simulation — Prove the whole plan works (do this annually)
  5. Live failover — Prove it with real traffic (do when confident in stages 1-4)

Start where you are. If you've never tested backup restoration, start there. One validated step is infinitely better than an untested plan — because when disaster strikes, the only thing that matters is whether the recovery actually works.

Read more