Traffic Shadowing for API Testing: Test With Real Production Requests

Traffic Shadowing for API Testing: Test With Real Production Requests

No matter how good your test suite is, it doesn't cover what real users actually send. Traffic shadowing fixes this by duplicating live production requests to a staging environment in parallel — your users get served by production, and your new version gets tested with real traffic simultaneously, with no impact on users.

Traffic shadowing (also called traffic mirroring or dark launching) is one of the most reliable ways to validate API changes before full rollout, because it uses actual production request patterns rather than synthetic tests.

What Traffic Shadowing Is

In a normal request flow, a client sends a request to your service, your service processes it, and sends back a response. Traffic shadowing adds a step: a proxy or load balancer duplicates the incoming request and sends a copy to a shadow service running your new version.

Key properties:

  • The shadow service's responses are discarded — users only see production responses
  • Shadow traffic has zero latency impact on production requests
  • Shadow processing happens asynchronously or in parallel
  • Shadow failures don't affect users

This lets you answer: "Does the new version handle real traffic correctly?" before actually exposing users to it.

Nginx Traffic Mirroring

Nginx 1.13.4+ has native traffic mirroring via the mirror directive:

location /api/ {
    # Send to production
    proxy_pass http://production-backend;

    # Also send a copy to shadow environment
    mirror /shadow;
    mirror_request_body on;
}

location /shadow {
    internal;
    proxy_pass http://shadow-backend;
}

The mirror directive sends an identical copy of every request to /shadow, which proxies to your shadow backend. The mirror_request_body on directive ensures POST bodies are also copied. Responses from the shadow backend are silently discarded.

This configuration works at the proxy level with no changes to your application code.

Envoy Traffic Mirroring

Envoy Proxy (used in Istio and many service meshes) supports mirroring via route configuration:

route_config:
  virtual_hosts:
  - name: api-service
    domains: ["api.example.com"]
    routes:
    - match:
        prefix: "/api/"
      route:
        cluster: production-cluster
        request_mirror_policies:
        - cluster: shadow-cluster
          runtime_fraction:
            default_value:
              numerator: 100
              denominator: HUNDRED

The runtime_fraction field lets you mirror a percentage of traffic rather than all of it — useful for high-traffic services where shadowing 100% would overload the shadow environment. Start with 1-5% and scale up as you gain confidence.

For Istio, the equivalent VirtualService configuration:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: api-service
spec:
  hosts:
  - api.example.com
  http:
  - route:
    - destination:
        host: production-service
        port:
          number: 8080
      weight: 100
    mirror:
      host: shadow-service
      port:
        number: 8080
    mirrorPercentage:
      value: 10.0

What to Measure in the Shadow Environment

Shadowing is only useful if you're capturing and comparing data from the shadow service. The typical setup:

Log all shadow responses — compare status codes, response bodies, and latency:

# Shadow response comparison service
import json
from collections import defaultdict

class ShadowComparator:
    def __init__(self):
        self.results = defaultdict(list)
    
    def compare(self, endpoint, prod_response, shadow_response):
        match = {
            'status_match': prod_response.status == shadow_response.status,
            'body_match': self._compare_bodies(
                prod_response.json(), 
                shadow_response.json()
            ),
            'latency_ratio': shadow_response.latency / prod_response.latency,
            'endpoint': endpoint
        }
        self.results[endpoint].append(match)
        
        if not match['status_match'] or not match['body_match']:
            self.alert(endpoint, prod_response, shadow_response)
    
    def _compare_bodies(self, prod, shadow):
        # Ignore fields that legitimately differ (timestamps, IDs)
        IGNORED_FIELDS = {'requestId', 'timestamp', 'processingTime'}
        prod_filtered = {k: v for k, v in prod.items() if k not in IGNORED_FIELDS}
        shadow_filtered = {k: v for k, v in shadow.items() if k not in IGNORED_FIELDS}
        return prod_filtered == shadow_filtered

Track divergence rates by endpoint:

POST /api/orders         → 0.0% divergence ✅
GET  /api/users/{id}     → 0.3% divergence ✅ (within acceptable range)
PUT  /api/orders/{id}    → 4.2% divergence ⚠️  (investigate before rollout)

A divergence rate above your threshold (often 0.5-1%) indicates the new version behaves differently than production on real traffic — catch this before rollout, not after.

Traffic Shadowing with GoReplay

GoReplay is a dedicated tool for capturing and replaying HTTP traffic. It can mirror live traffic to multiple targets simultaneously:

# Mirror 100% of production traffic to shadow environment
goreplay --input-raw :8080 \
         --output-http http://production:8080 \
         --output-http http://shadow:8080

# Mirror 10% of traffic to shadow
goreplay --input-raw :8080 \
         --output-http http://production:8080 \
         --output-http "http://shadow:8080|10%"

GoReplay also handles the comparison — you can configure it to log responses from both targets and emit differences. This is useful when you don't have infrastructure-level mirroring available and want an application-layer solution.

Database Isolation for Shadow Testing

The most critical operational concern with traffic shadowing is write operations. If your shadow environment shares a production database, shadow traffic will write real data — creating duplicate records, triggering emails, charging payment methods.

Required isolation strategies:

Separate shadow database — copy production data to a shadow database and point the shadow service at it. Refresh periodically (daily snapshots work for most teams). Shadow writes go to the shadow database only.

Write interception — intercept write operations at the shadow service level and discard them. Return a synthetic success response without persisting. This works for testing read paths but doesn't validate write correctness.

Idempotency keys — if your API uses idempotency keys, shadow requests can use modified keys that your system recognizes as shadow traffic and handles separately.

For read-heavy APIs (search, recommendations, catalog browsing), database isolation is simpler — reads don't modify state, so sharing a read replica is often acceptable.

Comparing Shadow Results Automatically

Manual comparison of shadow vs. production responses doesn't scale. Automated comparison tools include:

Diffy (Twitter open-source) — runs three instances (two production, one candidate), compares responses and filters out known noise like timestamps. Designed specifically for shadow testing API versions.

Custom comparison services — for teams with specific comparison needs (ignoring certain fields, applying business-logic-aware equality), a small service that subscribes to a shadow response log and emits divergence metrics.

The goal is a dashboard showing divergence rates by endpoint over time. A new deployment should show 0% divergence initially, and divergence patterns reveal exactly which request types the new version handles differently.

Shadow Testing vs. Blue/Green Deployments

Blue/green deployments switch 100% of traffic to a new version at once, with a fast rollback path. Shadow testing validates the new version before any users touch it. They complement each other:

  1. Shadow test the new version against real traffic for 24-48 hours
  2. Confirm divergence rate is acceptable
  3. Switch to blue/green deployment at a comfortable pace

For high-stakes API changes (authentication flows, payment processing, data migrations), shadow testing before any user exposure is worth the operational overhead.

Continuous API Monitoring After Rollout

Traffic shadowing validates behavior before rollout. After rollout, you need continuous monitoring to catch regressions that appear under real conditions. HelpMeTest provides 24/7 API health checks that run against your live endpoints — detecting when behavior changes between deployments or when dependencies degrade unexpectedly. The combination of shadow testing (pre-rollout confidence) and continuous monitoring (post-rollout vigilance) gives you coverage across the full deployment lifecycle.

Read more

Start now free