Envoy

Envoy Proxy Testing: Filter Chains, Health Checks, and Rate Limit Validation

HelpMeTest

HelpMeTest

— 5 min read
Envoy Proxy Testing: Filter Chains, Health Checks, and Rate Limit Validation

Envoy is the data plane proxy behind Istio, AWS App Mesh, and Contour. It handles routing, load balancing, authentication, rate limiting, and observability for your services. When Envoy is misconfigured, the symptoms are subtle — requests fail under specific conditions, rate limits don't enforce correctly, or health checks pass when they should fail. Testing Envoy configurations before they reach production prevents these surprises.

Understanding What to Test

Envoy configuration has three main layers to test:

  1. Filter chains — how requests are processed (auth, rate limiting, header manipulation)
  2. Health checks — how Envoy determines upstream health
  3. Routing and load balancing — how traffic is distributed

Setting Up Envoy for Testing

For isolated Envoy testing, run it as a standalone container:

# docker-compose.yaml for Envoy test environment
services:
  envoy:
    image: envoyproxy/envoy:v1.28-latest
    ports:
      - "10000:10000"   # listener port
      - "9901:9901"     # admin port
    volumes:
      - ./envoy.yaml:/etc/envoy/envoy.yaml
    command: envoy -c /etc/envoy/envoy.yaml --log-level debug

  upstream:
    image: kennethreitz/httpbin
    ports:
      - "8080:80"

The admin port (9901) is critical for testing — it exposes configuration, stats, and health check status.

Testing Filter Chains

Header Manipulation Filter

Test that Envoy adds and strips headers correctly:

ENVOY_URL="http://localhost:10000"

# Verify Envoy adds X-Request-Id header
HEADERS=$(curl -s -D - "$ENVOY_URL/api/test" -o /dev/null)
echo "$HEADERS" | grep -i "x-request-id" || (echo "FAIL: X-Request-Id not added"; exit 1)
echo "PASS: X-Request-Id header present"

# Verify Envoy strips internal headers from client requests
RESPONSE=$(curl -s -H "X-Internal-Token: secret" "$ENVOY_URL/api/echo-headers")
echo "$RESPONSE" | grep -i "x-internal-token" && (echo "FAIL: internal header leaked to upstream"; exit 1)
echo "PASS: internal header stripped"

Auth Filter Testing

# Request without token should return 401
RESPONSE=$(curl -s -o /dev/null -w "%{http_code}" "$ENVOY_URL/api/users")
[ "$RESPONSE" == "401" ] || (echo "FAIL: expected 401 without token, got $RESPONSE"; exit 1)

# Valid token should pass through
VALID_TOKEN="eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9..."
RESPONSE=$(curl -s -o /dev/null -w "%{http_code}" \
  -H "Authorization: Bearer $VALID_TOKEN" \
  "$ENVOY_URL/api/users")
[ "$RESPONSE" == "200" ] || (echo "FAIL: valid token rejected with $RESPONSE"; exit 1)
echo "PASS: auth filter working"

Verifying Filter Configuration via Admin API

The admin API at port 9901 exposes the active configuration:

# Dump active filter chains
curl -s http://localhost:9901/config_dump | \
  jq '.configs[] | select(.["@type"] | contains("Listener")) | \
  .. | .http_filters? | select(. != null) | [.[].name]'

Use this to verify your configuration was actually loaded, not just that requests work.

Testing Health Checks

# Cluster config with health checking
clusters:
- name: upstream_service
  health_checks:
  - timeout: 1s
    interval: 5s
    unhealthy_threshold: 2
    healthy_threshold: 1
    http_health_check:
      path: /health

Test the health check behavior:

# Make upstream unhealthy
curl -X POST http://localhost:8080/admin/set-health?status=unhealthy

sleep 15  # wait for 2 consecutive failures (2 * 5s interval + buffer)

# Verify Envoy marks host unhealthy
UNHEALTHY=$(curl -s http://localhost:9901/clusters | grep "upstream_service.*unhealthy")
[ -n "$UNHEALTHY" ] || (echo "FAIL: upstream not marked unhealthy"; exit 1)
echo "PASS: unhealthy upstream detected"

# Restore and verify recovery
curl -X POST http://localhost:8080/admin/set-health?status=healthy
sleep 10
HEALTHY=$(curl -s http://localhost:9901/clusters | grep "upstream_service.*healthy")
[ -n "$HEALTHY" ] || (echo "FAIL: upstream not marked healthy after recovery"; exit 1)
echo "PASS: upstream health recovery detected"

Outlier Detection Testing

outlier_detection:
  consecutive_5xx: 3
  interval: 10s
  base_ejection_time: 30s
  max_ejection_percent: 50
# Trigger 3 consecutive 5xx errors
for i in 1 2 3; do
  curl -s -o /dev/null "$ENVOY_URL/force-500"
  sleep 1
done

sleep 5

# Verify the host was ejected
EJECTED=$(curl -s http://localhost:9901/clusters | grep "outlier_detection.*ejected")
[ -n "$EJECTED" ] || (echo "FAIL: outlier not ejected after 3 consecutive 5xx"; exit 1)
echo "PASS: outlier ejection worked"

# After ejection time, verify host rejoins
sleep 35
ACTIVE=$(curl -s http://localhost:9901/clusters | grep "upstream_service" | grep -v "ejected")
[ -n "$ACTIVE" ] || (echo "FAIL: host did not rejoin after ejection timeout"; exit 1)
echo "PASS: ejected host rejoined after 30s"

Rate Limit Validation

Rate limiting is one of the most common Envoy configurations and one of the most commonly misconfigured.

http_filters:
- name: envoy.filters.http.local_ratelimit
  typed_config:
    "@type": type.googleapis.com/envoy.extensions.filters.http.local_ratelimit.v3.LocalRateLimit
    stat_prefix: http_local_rate_limiter
    token_bucket:
      max_tokens: 10
      tokens_per_fill: 10
      fill_interval: 60s
    filter_enabled:
      default_value:
        numerator: 100
        denominator: HUNDRED
    filter_enforced:
      default_value:
        numerator: 100
        denominator: HUNDRED

Test rate limiting exhaustion:

RATE_LIMITED=0
SUCCESS=0

# Send 15 requests; first 10 should succeed, rest should get 429
for i in $(seq 1 15); do
  CODE=$(curl -s -o /dev/null -w "%{http_code}" "$ENVOY_URL/api/data")
  if [ "$CODE" == "429" ]; then
    ((RATE_LIMITED++))
  elif [ "$CODE" == "200" ]; then
    ((SUCCESS++))
  fi
done

[ $SUCCESS -eq 10 ] || (echo "FAIL: expected 10 successes, got $SUCCESS"; exit 1)
[ $RATE_LIMITED -eq 5 ] || (echo "FAIL: expected 5 rate limits, got $RATE_LIMITED"; exit 1)
echo "PASS: rate limiting enforced correctly (10 success, 5 rejected)"

Verify Rate Limit Headers

HEADERS=$(curl -s -D - -o /dev/null "$ENVOY_URL/api/data")
echo "$HEADERS" | grep -i "x-ratelimit-remaining" || (echo "FAIL: X-RateLimit-Remaining missing"; exit 1)
echo "$HEADERS" | grep -i "x-ratelimit-limit" || (echo "FAIL: X-RateLimit-Limit missing"; exit 1)

# Verify counter decrements across requests
FIRST=$(curl -s -D - -o /dev/null "$ENVOY_URL/api/data" | grep -i "x-ratelimit-remaining" | tr -d '\r' | awk '{print $2}')
SECOND=$(curl -s -D - -o /dev/null "$ENVOY_URL/api/data" | grep -i "x-ratelimit-remaining" | tr -d '\r' | awk '{print $2}')
[ "$SECOND" -lt "$FIRST" ] || (echo "FAIL: rate limit counter not decrementing"; exit 1)
echo "PASS: rate limit counter decrements correctly"

Stats-Based Assertions

Envoy's stats endpoint is a goldmine for testing. Every filter action increments counters:

# Check rate limit decisions
curl -s http://localhost:9901/stats | grep "ratelimit.over_limit"

# Check upstream health
curl -s http://localhost:9901/stats | grep "health_check.success"
curl -s http://localhost:9901/stats | grep "health_check.failure"

# Check HTTP filter execution
curl -s http://localhost:9901/stats | grep "http.ingress_http.downstream_rq_total"

Assert against these counters before and after test traffic to verify filters are executing correctly.

Static Config Validation

Always validate Envoy config before deployment:

docker run --rm -v $(pwd)/envoy.yaml:/envoy.yaml \
  envoyproxy/envoy:v1.28-latest \
  envoy --mode validate -c /envoy.yaml

Expected output: configuration '/envoy.yaml' OK

Integration Test in CI

# .github/workflows/envoy-tests.yaml
jobs:
  test-envoy:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
    - name: Start Envoy + upstream
      run: docker-compose up -d && sleep 5
    - name: Wait for admin port
      run: timeout 30 bash -c 'until curl -s http://localhost:9901/ready; do sleep 1; done'
    - name: Run filter chain tests
      run: bash tests/envoy/filter-chain.sh
    - name: Run rate limit tests
      run: bash tests/envoy/rate-limits.sh
    - name: Run health check tests
      run: bash tests/envoy/health-checks.sh
    - name: Collect stats on failure
      if: failure()
      run: curl -s http://localhost:9901/stats > envoy-stats.txt
    - name: Cleanup
      if: always()
      run: docker-compose down

Envoy configuration is powerful but unforgiving — a wrong value in a filter chain can silently mishandle every request. Testing each layer in isolation, then together, is the only way to trust that your proxy configuration does what you think it does.

Envoy testing covers your proxy layer. For end-to-end testing and 24/7 uptime monitoring of services behind Envoy, HelpMeTest provides behavioral coverage with usage-based pricing ($0.003/run).

Read more

Start now free