ToxiProxy: Simulating Network Failures in Tests

ToxiProxy: Simulating Network Failures in Tests

Network failures are one of the hardest classes of bugs to reproduce. Production incidents caused by DNS resolution timeouts, database connection resets, and upstream API latency spikes are easy to diagnose in retrospect but nearly impossible to write regression tests for — unless you have a tool that lets you simulate them programmatically. ToxiProxy is that tool.

What Is ToxiProxy?

ToxiProxy is a TCP proxy built by Shopify that sits between your application and its dependencies. By routing traffic through ToxiProxy, you can inject network conditions — latency, bandwidth limits, packet loss, connection resets — under programmatic control during tests. When the test is over, you disable the toxics and traffic returns to normal.

The proxy itself is a small Go binary. Client libraries exist for Ruby, Python, Go, JavaScript, Java, PHP, and others. The workflow is:

  1. Start ToxiProxy server
  2. Create a proxy that routes to your real dependency
  3. Inject toxics via the API or client library
  4. Run your test
  5. Clean up

Installation

# macOS
brew install toxiproxy

# Linux
wget https://github.com/Shopify/toxiproxy/releases/download/v2.9.0/toxiproxy-server-linux-amd64
chmod +x toxiproxy-server-linux-amd64
mv toxiproxy-server-linux-amd64 /usr/local/bin/toxiproxy-server

# Docker
docker run -d --name toxiproxy \
  -p 8474:8474 \
  -p 5432:5432 \
  ghcr.io/shopify/toxiproxy

Start the server:

toxiproxy-server &

The server runs on port 8474 by default. This is the management API. Your proxies run on ports you configure.

Core Concepts

Proxies

A proxy is a named TCP tunnel. You create one for each upstream dependency you want to control:

# Create a proxy for PostgreSQL
toxiproxy-cli create postgres --listen 0.0.0.0:55432 --upstream localhost:5432

# Create a proxy for Redis
toxiproxy-cli create redis --listen 0.0.0.0:56379 --upstream localhost:6379

Your application then connects to the proxy port (55432) instead of the real PostgreSQL port (5432). ToxiProxy forwards traffic transparently unless toxics are active.

Toxics

Toxics are conditions injected into the proxy. Each toxic has a name, type, stream direction (upstream/downstream), toxicity (probability 0.0–1.0), and type-specific attributes.

Available toxic types:

Toxic What It Does
latency Adds fixed delay + optional jitter
bandwidth Throttles throughput (KB/s)
slow_close Delays TCP close by N ms
timeout Closes connection after N ms
slicer Slices data into smaller packets
limit_data Closes connection after N bytes
reset_peer Sends TCP RST (connection reset)

Ruby Integration

Shopify's Ruby client is the most mature:

gem 'toxiproxy'

In your test setup:

require 'toxiproxy'

Toxiproxy.host = 'http://127.0.0.1:8474'

# In RSpec before(:suite)
Toxiproxy.populate([
  {
    name: 'postgres',
    listen: '127.0.0.1:55432',
    upstream: '127.0.0.1:5432'
  },
  {
    name: 'redis',
    listen: '127.0.0.1:56379',
    upstream: '127.0.0.1:6379'
  }
])

Test with latency:

RSpec.describe 'slow database' do
  it 'times out after 5 seconds when postgres is slow' do
    Toxiproxy[:postgres].downstream(:latency, latency: 6000).apply do
      expect {
        User.find(1)  # ActiveRecord configured to use port 55432
      }.to raise_error(ActiveRecord::QueryTimeout)
    end
  end
end

Test with connection reset:

it 'retries on connection reset' do
  call_count = 0
  
  Toxiproxy[:redis].downstream(:reset_peer, timeout: 0).apply do
    # First call fails with connection reset
    # Retry logic should handle it
    result = with_retry { redis.get('key') }
    expect(result).not_to be_nil
  end
end

Go Integration

import "github.com/Shopify/toxiproxy/v2/api/client"

func TestDatabaseTimeout(t *testing.T) {
    proxy, err := client.NewClient("http://localhost:8474").CreateProxy(
        "postgres",
        "localhost:55432",
        "localhost:5432",
    )
    if err != nil {
        t.Fatal(err)
    }
    defer proxy.Delete()

    _, err = proxy.AddToxic("latency_downstream", "latency", "downstream", 1.0, client.Attributes{
        "latency": 3000,
        "jitter":  500,
    })
    if err != nil {
        t.Fatal(err)
    }

    // Your database call that should timeout
    ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
    defer cancel()

    _, err = db.QueryContext(ctx, "SELECT 1")
    if !errors.Is(err, context.DeadlineExceeded) {
        t.Errorf("expected timeout, got %v", err)
    }

    proxy.RemoveToxic("latency_downstream")
}

Python Integration

pip install toxiproxy-python
import pytest
from toxiproxy import Toxiproxy

@pytest.fixture(scope='session')
def toxiproxy_client():
    client = Toxiproxy()
    client.create(name='postgres', listen='127.0.0.1:55432', upstream='127.0.0.1:5432')
    yield client
    client.destroy_all()

def test_query_timeout(toxiproxy_client):
    proxy = toxiproxy_client.get_proxy('postgres')
    proxy.add_toxic('latency_down', type='latency', stream='downstream', 
                    attributes={'latency': 5000})
    
    try:
        with pytest.raises(Exception):  # your ORM's timeout exception
            db.execute('SELECT 1', timeout=2)
    finally:
        proxy.remove_toxic('latency_down')

Docker Compose Setup

For CI environments, run ToxiProxy alongside your services:

version: '3.8'
services:
  postgres:
    image: postgres:15
    environment:
      POSTGRES_PASSWORD: test
    ports:
      - "5432:5432"

  redis:
    image: redis:7
    ports:
      - "6379:6379"

  toxiproxy:
    image: ghcr.io/shopify/toxiproxy:2.9.0
    ports:
      - "8474:8474"  # management API
      - "55432:55432"  # postgres proxy
      - "56379:56379"  # redis proxy
    command: >
      -host 0.0.0.0
      -config /config/toxiproxy.json
    volumes:
      - ./toxiproxy.json:/config/toxiproxy.json

  app:
    build: .
    environment:
      DATABASE_URL: postgres://postgres:test@toxiproxy:55432/testdb
      REDIS_URL: redis://toxiproxy:56379
    depends_on:
      - postgres
      - redis
      - toxiproxy

Pre-configure proxies via JSON:

[
  {
    "name": "postgres",
    "listen": "0.0.0.0:55432",
    "upstream": "postgres:5432",
    "enabled": true
  },
  {
    "name": "redis",
    "listen": "0.0.0.0:56379",
    "upstream": "redis:6379",
    "enabled": true
  }
]

What to Test

Retry Logic

Most production systems have retry logic for transient failures. Verify it actually works:

it 'retries up to 3 times on connection timeout' do
  attempts = 0
  Toxiproxy[:postgres].downstream(:timeout, timeout: 100).apply do
    begin
      with_retry(max_attempts: 3) { 
        attempts += 1
        User.count 
      }
    rescue
    end
  end
  expect(attempts).to eq(3)
end

Circuit Breaker Behavior

If you're using a circuit breaker (Hystrix, Resilience4j, etc.), verify it opens when the dependency is consistently failing:

it 'opens circuit after 5 consecutive failures' do
  Toxiproxy[:payment_api].downstream(:reset_peer).apply do
    5.times { payment_service.charge(100) rescue nil }
    expect(payment_service.circuit_open?).to be true
  end
end

Degraded Performance Under Latency

Slow responses can exhaust thread pools. Test that your system degrades gracefully rather than locking up:

it 'returns cached response when upstream is slow' do
  cache.write('product:123', { name: 'Widget' })
  
  Toxiproxy[:product_api].downstream(:latency, latency: 10_000).apply do
    result = product_service.fetch(123)
    expect(result[:name]).to eq('Widget')  # served from cache
  end
end

Bandwidth Throttling

Test file upload/download behavior on slow connections:

def test_large_upload_completes_on_slow_connection(toxiproxy_client):
    proxy = toxiproxy_client.get_proxy('s3')
    proxy.add_toxic('bandwidth', type='bandwidth', 
                    attributes={'rate': 100})  # 100 KB/s
    
    try:
        result = upload_service.upload(large_file_fixture)
        assert result['success'] is True
    finally:
        proxy.remove_toxic('bandwidth')

Toxicity Parameter

The toxicity parameter (0.0–1.0) controls the probability that a given packet is affected. toxicity: 0.5 means 50% of packets get the toxic applied. This is useful for simulating intermittent failures:

# 30% packet loss
Toxiproxy[:redis].downstream(:timeout, timeout: 0, toxicity: 0.3).apply do
  # Test behavior under intermittent failures
  results = 100.times.map { redis.get('key') rescue :error }
  success_rate = results.count { |r| r != :error }.to_f / 100
  expect(success_rate).to be > 0.6  # at least 60% succeed
end

Integration with HelpMeTest

Network resilience tests are particularly valuable as continuous monitors. A unit test that verifies retry logic runs fast but doesn't catch regressions where a dependency change removed retry behavior in the deployed application.

HelpMeTest can run integration scenarios that include network condition simulation — verifying that the deployed version of your application handles slow dependencies correctly. Setting up a ToxiProxy alongside your test environment and running HelpMeTest's Robot Framework tests against it gives you ongoing confidence that your production retry and fallback logic works, not just in a local test run but continuously against the live system.

Common Pitfalls

Forgetting to clean up toxics. Always remove toxics after each test. Use try/finally blocks or the proxy's apply block pattern that cleans up automatically.

Not configuring application timeouts. ToxiProxy exposes the need for explicit timeouts. If your application doesn't have timeouts configured, it will hang indefinitely. That's a real finding — fix the timeout configuration, not the test.

Testing in the wrong direction. Some toxics can be applied upstream or downstream. Latency on the downstream side slows the response from the dependency back to your app. Latency on the upstream side slows your app's requests to the dependency. Know which direction you intend.

Not testing the happy path after chaos. After removing a toxic, verify that normal operations resume. Some bugs only surface when a system tries to recover from a degraded state.

ToxiProxy turns "I hope our retry logic works" into a test that proves it does — or exposes that it doesn't.

Read more

Start now free