ToxiProxy: Simulating Network Failures in Tests
Network failures are one of the hardest classes of bugs to reproduce. Production incidents caused by DNS resolution timeouts, database connection resets, and upstream API latency spikes are easy to diagnose in retrospect but nearly impossible to write regression tests for — unless you have a tool that lets you simulate them programmatically. ToxiProxy is that tool.
What Is ToxiProxy?
ToxiProxy is a TCP proxy built by Shopify that sits between your application and its dependencies. By routing traffic through ToxiProxy, you can inject network conditions — latency, bandwidth limits, packet loss, connection resets — under programmatic control during tests. When the test is over, you disable the toxics and traffic returns to normal.
The proxy itself is a small Go binary. Client libraries exist for Ruby, Python, Go, JavaScript, Java, PHP, and others. The workflow is:
- Start ToxiProxy server
- Create a proxy that routes to your real dependency
- Inject toxics via the API or client library
- Run your test
- Clean up
Installation
# macOS
brew install toxiproxy
# Linux
wget https://github.com/Shopify/toxiproxy/releases/download/v2.9.0/toxiproxy-server-linux-amd64
chmod +x toxiproxy-server-linux-amd64
mv toxiproxy-server-linux-amd64 /usr/local/bin/toxiproxy-server
# Docker
docker run -d --name toxiproxy \
-p 8474:8474 \
-p 5432:5432 \
ghcr.io/shopify/toxiproxyStart the server:
toxiproxy-server &The server runs on port 8474 by default. This is the management API. Your proxies run on ports you configure.
Core Concepts
Proxies
A proxy is a named TCP tunnel. You create one for each upstream dependency you want to control:
# Create a proxy for PostgreSQL
toxiproxy-cli create postgres --listen 0.0.0.0:55432 --upstream localhost:5432
# Create a proxy for Redis
toxiproxy-cli create redis --listen 0.0.0.0:56379 --upstream localhost:6379Your application then connects to the proxy port (55432) instead of the real PostgreSQL port (5432). ToxiProxy forwards traffic transparently unless toxics are active.
Toxics
Toxics are conditions injected into the proxy. Each toxic has a name, type, stream direction (upstream/downstream), toxicity (probability 0.0–1.0), and type-specific attributes.
Available toxic types:
| Toxic | What It Does |
|---|---|
latency |
Adds fixed delay + optional jitter |
bandwidth |
Throttles throughput (KB/s) |
slow_close |
Delays TCP close by N ms |
timeout |
Closes connection after N ms |
slicer |
Slices data into smaller packets |
limit_data |
Closes connection after N bytes |
reset_peer |
Sends TCP RST (connection reset) |
Ruby Integration
Shopify's Ruby client is the most mature:
gem 'toxiproxy'In your test setup:
require 'toxiproxy'
Toxiproxy.host = 'http://127.0.0.1:8474'
# In RSpec before(:suite)
Toxiproxy.populate([
{
name: 'postgres',
listen: '127.0.0.1:55432',
upstream: '127.0.0.1:5432'
},
{
name: 'redis',
listen: '127.0.0.1:56379',
upstream: '127.0.0.1:6379'
}
])Test with latency:
RSpec.describe 'slow database' do
it 'times out after 5 seconds when postgres is slow' do
Toxiproxy[:postgres].downstream(:latency, latency: 6000).apply do
expect {
User.find(1) # ActiveRecord configured to use port 55432
}.to raise_error(ActiveRecord::QueryTimeout)
end
end
endTest with connection reset:
it 'retries on connection reset' do
call_count = 0
Toxiproxy[:redis].downstream(:reset_peer, timeout: 0).apply do
# First call fails with connection reset
# Retry logic should handle it
result = with_retry { redis.get('key') }
expect(result).not_to be_nil
end
endGo Integration
import "github.com/Shopify/toxiproxy/v2/api/client"
func TestDatabaseTimeout(t *testing.T) {
proxy, err := client.NewClient("http://localhost:8474").CreateProxy(
"postgres",
"localhost:55432",
"localhost:5432",
)
if err != nil {
t.Fatal(err)
}
defer proxy.Delete()
_, err = proxy.AddToxic("latency_downstream", "latency", "downstream", 1.0, client.Attributes{
"latency": 3000,
"jitter": 500,
})
if err != nil {
t.Fatal(err)
}
// Your database call that should timeout
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
defer cancel()
_, err = db.QueryContext(ctx, "SELECT 1")
if !errors.Is(err, context.DeadlineExceeded) {
t.Errorf("expected timeout, got %v", err)
}
proxy.RemoveToxic("latency_downstream")
}Python Integration
pip install toxiproxy-pythonimport pytest
from toxiproxy import Toxiproxy
@pytest.fixture(scope='session')
def toxiproxy_client():
client = Toxiproxy()
client.create(name='postgres', listen='127.0.0.1:55432', upstream='127.0.0.1:5432')
yield client
client.destroy_all()
def test_query_timeout(toxiproxy_client):
proxy = toxiproxy_client.get_proxy('postgres')
proxy.add_toxic('latency_down', type='latency', stream='downstream',
attributes={'latency': 5000})
try:
with pytest.raises(Exception): # your ORM's timeout exception
db.execute('SELECT 1', timeout=2)
finally:
proxy.remove_toxic('latency_down')Docker Compose Setup
For CI environments, run ToxiProxy alongside your services:
version: '3.8'
services:
postgres:
image: postgres:15
environment:
POSTGRES_PASSWORD: test
ports:
- "5432:5432"
redis:
image: redis:7
ports:
- "6379:6379"
toxiproxy:
image: ghcr.io/shopify/toxiproxy:2.9.0
ports:
- "8474:8474" # management API
- "55432:55432" # postgres proxy
- "56379:56379" # redis proxy
command: >
-host 0.0.0.0
-config /config/toxiproxy.json
volumes:
- ./toxiproxy.json:/config/toxiproxy.json
app:
build: .
environment:
DATABASE_URL: postgres://postgres:test@toxiproxy:55432/testdb
REDIS_URL: redis://toxiproxy:56379
depends_on:
- postgres
- redis
- toxiproxyPre-configure proxies via JSON:
[
{
"name": "postgres",
"listen": "0.0.0.0:55432",
"upstream": "postgres:5432",
"enabled": true
},
{
"name": "redis",
"listen": "0.0.0.0:56379",
"upstream": "redis:6379",
"enabled": true
}
]What to Test
Retry Logic
Most production systems have retry logic for transient failures. Verify it actually works:
it 'retries up to 3 times on connection timeout' do
attempts = 0
Toxiproxy[:postgres].downstream(:timeout, timeout: 100).apply do
begin
with_retry(max_attempts: 3) {
attempts += 1
User.count
}
rescue
end
end
expect(attempts).to eq(3)
endCircuit Breaker Behavior
If you're using a circuit breaker (Hystrix, Resilience4j, etc.), verify it opens when the dependency is consistently failing:
it 'opens circuit after 5 consecutive failures' do
Toxiproxy[:payment_api].downstream(:reset_peer).apply do
5.times { payment_service.charge(100) rescue nil }
expect(payment_service.circuit_open?).to be true
end
endDegraded Performance Under Latency
Slow responses can exhaust thread pools. Test that your system degrades gracefully rather than locking up:
it 'returns cached response when upstream is slow' do
cache.write('product:123', { name: 'Widget' })
Toxiproxy[:product_api].downstream(:latency, latency: 10_000).apply do
result = product_service.fetch(123)
expect(result[:name]).to eq('Widget') # served from cache
end
endBandwidth Throttling
Test file upload/download behavior on slow connections:
def test_large_upload_completes_on_slow_connection(toxiproxy_client):
proxy = toxiproxy_client.get_proxy('s3')
proxy.add_toxic('bandwidth', type='bandwidth',
attributes={'rate': 100}) # 100 KB/s
try:
result = upload_service.upload(large_file_fixture)
assert result['success'] is True
finally:
proxy.remove_toxic('bandwidth')Toxicity Parameter
The toxicity parameter (0.0–1.0) controls the probability that a given packet is affected. toxicity: 0.5 means 50% of packets get the toxic applied. This is useful for simulating intermittent failures:
# 30% packet loss
Toxiproxy[:redis].downstream(:timeout, timeout: 0, toxicity: 0.3).apply do
# Test behavior under intermittent failures
results = 100.times.map { redis.get('key') rescue :error }
success_rate = results.count { |r| r != :error }.to_f / 100
expect(success_rate).to be > 0.6 # at least 60% succeed
endIntegration with HelpMeTest
Network resilience tests are particularly valuable as continuous monitors. A unit test that verifies retry logic runs fast but doesn't catch regressions where a dependency change removed retry behavior in the deployed application.
HelpMeTest can run integration scenarios that include network condition simulation — verifying that the deployed version of your application handles slow dependencies correctly. Setting up a ToxiProxy alongside your test environment and running HelpMeTest's Robot Framework tests against it gives you ongoing confidence that your production retry and fallback logic works, not just in a local test run but continuously against the live system.
Common Pitfalls
Forgetting to clean up toxics. Always remove toxics after each test. Use try/finally blocks or the proxy's apply block pattern that cleans up automatically.
Not configuring application timeouts. ToxiProxy exposes the need for explicit timeouts. If your application doesn't have timeouts configured, it will hang indefinitely. That's a real finding — fix the timeout configuration, not the test.
Testing in the wrong direction. Some toxics can be applied upstream or downstream. Latency on the downstream side slows the response from the dependency back to your app. Latency on the upstream side slows your app's requests to the dependency. Know which direction you intend.
Not testing the happy path after chaos. After removing a toxic, verify that normal operations resume. Some bugs only surface when a system tries to recover from a degraded state.
ToxiProxy turns "I hope our retry logic works" into a test that proves it does — or exposes that it doesn't.