Resilience Testing Strategies: How to Verify Your System Handles Failures

HelpMeTest

06 Sep 2026 — 6 min read

Resilience testing is the practice of verifying that your system handles failures gracefully. This covers more than chaos engineering — it includes unit-level tests for circuit breakers and retry logic, integration tests that simulate dependency failures, and chaos experiments that inject real failures into running systems. This guide covers the full spectrum with concrete code examples.

The Resilience Testing Pyramid

Resilience testing exists at multiple levels:

                    ┌─────────────────┐
                    │  Chaos / GameDay │  ← Production or staging
                    │  (hours, rare)   │
                 ┌──┴─────────────────┴──┐
                 │   Integration Tests    │  ← Test environment
                 │  (minutes, per-PR)     │
              ┌──┴────────────────────────┴──┐
              │      Unit Tests               │  ← Every build
              │  (seconds, every commit)      │
              └──────────────────────────────┘

Unit tests: verify that circuit breakers, retry logic, and timeout handling work correctly in isolation
Integration tests: simulate dependency failures in a test environment with real network calls
Chaos experiments: inject real failures into a running system to find unexpected failure modes

Teams often jump straight to chaos tools without building the unit and integration layers first. This is backwards — chaos engineering finds failure modes; unit and integration tests prevent known failure modes from regressing.

Circuit Breaker Testing

A circuit breaker protects your service from cascading failures by stopping calls to a failing dependency after a threshold of failures is reached.

Testing the closed → open transition:

import { CircuitBreaker } from 'opossum';
import { describe, test, expect, vi } from 'vitest';

describe('Circuit Breaker', () => {
  test('opens after consecutive failures', async () => {
    const failingFn = vi.fn().mockRejectedValue(new Error('Service unavailable'));
    
    const breaker = new CircuitBreaker(failingFn, {
      errorThresholdPercentage: 50,
      timeout: 3000,
      resetTimeout: 30000,
      volumeThreshold: 5,
    });

    // Trigger enough failures to open the circuit
    const failures = Array(5).fill(null).map(() =>
      breaker.fire().catch(() => {})
    );
    await Promise.all(failures);

    // Circuit should now be open
    expect(breaker.opened).toBe(true);

    // Next call should fail fast (not call the underlying service)
    await expect(breaker.fire()).rejects.toThrow('Breaker is open');
    
    // The underlying function should only have been called 5 times, not 6
    expect(failingFn).toHaveBeenCalledTimes(5);
  });

  test('transitions to half-open after resetTimeout', async () => {
    vi.useFakeTimers();
    
    const fn = vi.fn()
      .mockRejectedValueOnce(new Error('fail'))
      .mockResolvedValue('success');
    
    const breaker = new CircuitBreaker(fn, {
      errorThresholdPercentage: 100,
      timeout: 1000,
      resetTimeout: 10000,
      volumeThreshold: 1,
    });

    await breaker.fire().catch(() => {});
    expect(breaker.opened).toBe(true);

    // Advance past reset timeout
    vi.advanceTimersByTime(10001);
    
    // Should now be in half-open state — allows one call through
    const result = await breaker.fire();
    expect(result).toBe('success');
    expect(breaker.closed).toBe(true);  // Closed after successful probe

    vi.useRealTimers();
  });
});

Testing with Resilience4j (Java):

import io.github.resilience4j.circuitbreaker.CircuitBreaker;
import io.github.resilience4j.circuitbreaker.CircuitBreakerConfig;
import org.junit.jupiter.api.Test;

import java.time.Duration;
import java.util.function.Supplier;

import static org.junit.jupiter.api.Assertions.*;

class CircuitBreakerTest {
    
    @Test
    void opens_after_failure_threshold() {
        CircuitBreakerConfig config = CircuitBreakerConfig.custom()
            .slidingWindowType(CircuitBreakerConfig.SlidingWindowType.COUNT_BASED)
            .slidingWindowSize(5)
            .failureRateThreshold(50.0f)
            .waitDurationInOpenState(Duration.ofSeconds(10))
            .build();
        
        CircuitBreaker cb = CircuitBreaker.of("test", config);
        
        Supplier<String> failingSupplier = CircuitBreaker
            .decorateSupplier(cb, () -> { throw new RuntimeException("fail"); });
        
        // Trigger 3 failures out of 5 window (60% > 50% threshold)
        for (int i = 0; i < 5; i++) {
            try { failingSupplier.get(); } catch (Exception ignored) {}
        }
        
        assertEquals(CircuitBreaker.State.OPEN, cb.getState());
    }
}

Retry Logic Testing

Retry logic should back off exponentially and respect a maximum attempt count.

import { retry } from 'ts-retry-promise';

describe('Retry logic', () => {
  test('retries with exponential backoff', async () => {
    vi.useFakeTimers();
    
    const callTimes: number[] = [];
    const fn = vi.fn().mockImplementation(async () => {
      callTimes.push(Date.now());
      throw new Error('Temporary failure');
    });

    const retryFn = () => retry(fn, {
      retries: 3,
      delay: 100,
      backOff: 'EXPONENTIAL',
      timeout: 10000,
    });

    const promise = retryFn().catch(() => {});
    
    // Initial call
    await vi.runAllTimersAsync();
    
    expect(fn).toHaveBeenCalledTimes(4); // 1 initial + 3 retries
    
    // Verify exponential backoff: 100ms, 200ms, 400ms
    const gaps = callTimes.slice(1).map((t, i) => t - callTimes[i]);
    expect(gaps[0]).toBeGreaterThanOrEqual(100);
    expect(gaps[1]).toBeGreaterThanOrEqual(200);
    expect(gaps[2]).toBeGreaterThanOrEqual(400);
    
    vi.useRealTimers();
  });

  test('does not retry on non-retryable errors', async () => {
    const fn = vi.fn()
      .mockRejectedValue(Object.assign(new Error('Not Found'), { status: 404 }));

    await expect(
      retry(fn, {
        retries: 3,
        retryIf: (error: any) => error.status >= 500,
      })
    ).rejects.toThrow('Not Found');

    expect(fn).toHaveBeenCalledTimes(1); // No retries for 404
  });
});

Timeout Testing

Services must have explicit timeouts — a hanging dependency should not cause your service to hang.

describe('Timeout handling', () => {
  test('requests timeout after configured duration', async () => {
    // Server that never responds
    const server = http.createServer(() => {
      // Intentionally never respond
    });
    server.listen(0);
    const { port } = server.address() as net.AddressInfo;

    const client = axios.create({ timeout: 500 });

    await expect(
      client.get(`http://localhost:${port}/data`)
    ).rejects.toMatchObject({
      code: 'ECONNABORTED',
    });

    server.close();
  });

  test('returns fallback value on timeout', async () => {
    const slowFetch = () => new Promise<string>((resolve) => {
      setTimeout(() => resolve('slow-result'), 5000);
    });

    const result = await Promise.race([
      slowFetch(),
      new Promise<string>((resolve) => setTimeout(() => resolve('fallback'), 100)),
    ]);

    expect(result).toBe('fallback');
  });
});

Bulkhead Pattern Testing

Bulkheads limit concurrent calls to a dependency, preventing one slow dependency from exhausting all threads/connections.

import Bottleneck from 'bottleneck';

describe('Bulkhead', () => {
  test('limits concurrent calls to dependency', async () => {
    const maxConcurrent = 3;
    const limiter = new Bottleneck({ maxConcurrent });
    
    let activeCalls = 0;
    let maxActiveSeen = 0;

    const slowFn = async () => {
      activeCalls++;
      maxActiveSeen = Math.max(maxActiveSeen, activeCalls);
      await new Promise(r => setTimeout(r, 50));
      activeCalls--;
      return 'done';
    };

    // Fire 10 concurrent requests
    const wrapped = () => limiter.schedule(slowFn);
    await Promise.all(Array(10).fill(null).map(() => wrapped()));

    expect(maxActiveSeen).toBeLessThanOrEqual(maxConcurrent);
  });
});

Integration Tests: Simulating Dependency Failures

Use test doubles or real infrastructure with injected failures to test dependency failure handling.

Mock Server with Failure Injection (Node.js)

import { setupServer } from 'msw/node';
import { http, HttpResponse } from 'msw';

const server = setupServer();

beforeAll(() => server.listen());
afterEach(() => server.resetHandlers());
afterAll(() => server.close());

describe('Payment service resilience', () => {
  test('returns 503 when payment provider times out', async () => {
    server.use(
      http.post('https://api.payment-provider.com/charge', async () => {
        await new Promise(r => setTimeout(r, 5000));  // Simulate timeout
        return HttpResponse.json({ success: true });
      })
    );

    const response = await request(app)
      .post('/api/orders')
      .send({ amount: 100, card: 'tok_visa' });

    expect(response.status).toBe(503);
    expect(response.body.message).toContain('Payment service unavailable');
  });

  test('retries on 503 and succeeds on second attempt', async () => {
    let attempts = 0;
    server.use(
      http.post('https://api.payment-provider.com/charge', () => {
        attempts++;
        if (attempts === 1) {
          return new HttpResponse(null, { status: 503 });
        }
        return HttpResponse.json({ transactionId: 'txn_123' });
      })
    );

    const response = await request(app)
      .post('/api/orders')
      .send({ amount: 100, card: 'tok_visa' });

    expect(response.status).toBe(200);
    expect(attempts).toBe(2);
  });
});

Database Failure Tests

# Python / pytest
import pytest
from unittest.mock import patch, MagicMock
from psycopg2 import OperationalError

class TestDatabaseResilience:
    def test_returns_cached_data_on_db_timeout(self, app, redis_client):
        # Pre-populate cache
        redis_client.set('user:123', json.dumps({'id': 123, 'name': 'Alice'}))
        
        # Simulate database timeout
        with patch('app.db.execute', side_effect=OperationalError('connection timeout')):
            response = app.get('/users/123')
        
        assert response.status_code == 200
        assert response.json['name'] == 'Alice'
        assert response.json['_source'] == 'cache'  # Verify fallback was used

    def test_returns_503_when_both_db_and_cache_fail(self, app):
        with patch('app.db.execute', side_effect=OperationalError('timeout')):
            with patch('app.cache.get', side_effect=ConnectionError('redis unavailable')):
                response = app.get('/users/123')
        
        assert response.status_code == 503
        assert 'Retry-After' in response.headers  # Proper retry guidance

Graceful Degradation Testing

Test that your service degrades gracefully when features become unavailable.

describe('Graceful degradation', () => {
  test('serves page without recommendations when recommendation service is down', async () => {
    server.use(
      http.get('http://recommendations-service/recommendations', () => {
        return new HttpResponse(null, { status: 503 });
      })
    );

    const response = await request(app).get('/products/123');

    expect(response.status).toBe(200);
    expect(response.body.product).toBeDefined();
    // Recommendations should be absent, not cause a 500
    expect(response.body.recommendations).toBeUndefined();
  });

  test('logs warning but does not fail when analytics service is down', async () => {
    const warnSpy = vi.spyOn(logger, 'warn');
    
    server.use(
      http.post('http://analytics-service/events', () => {
        return new HttpResponse(null, { status: 503 });
      })
    );

    const response = await request(app)
      .post('/orders')
      .send(validOrder);

    expect(response.status).toBe(201);  // Order still created
    expect(warnSpy).toHaveBeenCalledWith(
      expect.stringContaining('Analytics service unavailable')
    );
  });
});

Load Shedding Tests

Verify your service rejects requests when under extreme load rather than degrading for everyone.

describe('Load shedding', () => {
  test('returns 429 when request queue is full', async () => {
    // Fill the queue to capacity
    const backgroundRequests = Array(20).fill(null).map(() =>
      request(app).get('/slow-endpoint')
    );

    // This request should be shed
    const response = await request(app)
      .get('/products/123')
      .set('X-Request-Priority', 'low');

    expect(response.status).toBe(429);
    expect(response.headers['retry-after']).toBeDefined();

    // Clean up
    await Promise.allSettled(backgroundRequests);
  });
});

Building a Resilience Test Suite

Organize your resilience tests into three categories:

1. Fast unit tests (run on every commit):

Circuit breaker state transitions
Retry count and backoff timing
Timeout behavior
Fallback value returns

2. Integration tests (run on every PR):

External service returning 5xx responses
External service timing out
Database connectivity loss
Cache miss fallback to database

3. Chaos experiments (run weekly or before releases):

Pod termination
Network partition
Resource exhaustion
Zone failure

This layered approach catches different failure modes at the right cost level. Unit tests run in milliseconds; chaos experiments take hours. Use the right tool for each type of assertion.

Checklist: Is Your Service Resilient?

□ HTTP clients have explicit timeout settings
□ External API calls have retry logic with exponential backoff
□ Retry logic does NOT retry non-retryable errors (400s, 401s)
□ Circuit breakers protect calls to critical dependencies
□ Connection pools have size limits (not unbounded)
□ Services degrade gracefully when non-critical dependencies fail
□ Health endpoints exist and are accurate
□ Readiness probes prevent traffic before service is ready
□ Liveness probes restart unhealthy instances
□ All of the above are tested with automated tests

If you can check every box and prove it with tests, your service handles the most common failure modes. Add chaos engineering on top to find the failure modes you didn't anticipate.