Testing with Istio: Traffic Management, Fault Injection, and mTLS Verification

Testing with Istio: Traffic Management, Fault Injection, and mTLS Verification

Istio adds a control plane on top of your Kubernetes services: traffic routing, mutual TLS, circuit breaking, and retries. But these configurations only work if you test them. A misconfigured VirtualService silently routes traffic wrong. An mTLS policy misconfiguration lets unencrypted traffic through. Fault injection reveals whether your services actually handle failures gracefully.

This guide covers the testing patterns that verify Istio configurations work as intended.

Setting Up a Test Environment

Before writing tests, you need visibility into Istio's behavior:

# Check Istio sidecar injection is enabled
kubectl get namespace default -o jsonpath='{.metadata.labels.istio-injection}'

# Verify all pods have sidecars
kubectl get pods -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[*].name}{"\n"}{end}'

# Check Istio configuration validation
istioctl analyze

# View current traffic configuration
istioctl proxy-config routes <pod-name>
istioctl proxy-config cluster <pod-name>

Always run istioctl analyze after applying new configurations. It catches common errors before traffic does.

Testing Traffic Management

Istio's VirtualService and DestinationRule control traffic routing. Testing them means verifying that traffic actually flows where the configuration says it should.

Canary Deployment Test

Given this configuration that routes 10% of traffic to v2:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: reviews
spec:
  hosts:
  - reviews
  http:
  - route:
    - destination:
        host: reviews
        subset: v1
      weight: 90
    - destination:
        host: reviews
        subset: v2
      weight: 10

Test it with repeated requests and count the distribution:

#!/bin/bash
V1_COUNT=0
V2_COUNT=0

for i in $(seq 1 100); do
  RESPONSE=$(kubectl exec -n test deploy/curl -- curl -s http://reviews/api/version)
  if echo "$RESPONSE" | grep -q '"version":"v1"'; then
    ((V1_COUNT++))
  elif echo "$RESPONSE" | grep -q '"version":"v2"'; then
    ((V2_COUNT++))
  fi
done

echo "v1: $V1_COUNT/100, v2: $V2_COUNT/100"

# Assert roughly 90/10 split (allow ±15% statistical variance)
if [ $V2_COUNT -lt 5 ] || [ $V2_COUNT -gt 25 ]; then
  echo "FAIL: v2 traffic outside expected range"
  exit 1
fi
echo "PASS: traffic distribution within expected range"

100 requests gives enough statistical resolution to verify a 10% weight without excessive flakiness.

Header-Based Routing

Test user-group routing:

# Route users with header X-Beta-User: true to v2
http:
- match:
  - headers:
      x-beta-user:
        exact: "true"
  route:
  - destination:
      host: reviews
      subset: v2
- route:
  - destination:
      host: reviews
      subset: v1
# Beta user should hit v2
BETA_RESPONSE=$(kubectl exec deploy/curl -- curl -s \
  -H "x-beta-user: true" \
  http://reviews/api/version)
echo "$BETA_RESPONSE" | jq -e '.version == "v2"' || (echo "FAIL: beta user not on v2"; exit 1)

# Regular user should hit v1
REGULAR_RESPONSE=$(kubectl exec deploy/curl -- curl -s \
  http://reviews/api/version)
echo "$REGULAR_RESPONSE" | jq -e '.version == "v1"' || (echo "FAIL: regular user not on v1"; exit 1)

echo "PASS: header-based routing works"

Circuit Breaker Verification

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: reviews
spec:
  host: reviews
  trafficPolicy:
    outlierDetection:
      consecutiveErrors: 3
      interval: 10s
      baseEjectionTime: 30s

Test that the circuit breaker ejects failing pods:

# Make the pod return errors
kubectl exec deploy/reviews-v1 -- curl -s -X POST http://localhost:9080/admin/fail

# Send requests and verify circuit opens after 3 consecutive errors
FAILURES=0
for i in $(seq 1 20); do
  CODE=$(kubectl exec deploy/curl -- curl -s -o /dev/null -w "%{http_code}" http://reviews/api/reviews/1)
  if [ "$CODE" != "200" ]; then
    ((FAILURES++))
  fi
done

# After circuit opens, most requests should succeed (routed to healthy pods)
SUCCESS=$((20 - FAILURES))
if [ $SUCCESS -lt 15 ]; then
  echo "FAIL: circuit breaker not functioning, only $SUCCESS/20 requests succeeded"
  exit 1
fi
echo "PASS: circuit breaker ejected failing pod, $SUCCESS/20 requests succeeded"

Fault Injection Testing

Istio's fault injection deliberately introduces failures to test service resilience. This is the correct way to test whether your retry logic, timeouts, and fallbacks actually work.

Delay Injection

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: ratings
spec:
  hosts:
  - ratings
  http:
  - fault:
      delay:
        percentage:
          value: 100
        fixedDelay: 7s
    route:
    - destination:
        host: ratings

Test that the calling service handles the delay correctly (e.g., has a 5-second timeout):

START=$(date +%s%N)
RESPONSE=$(kubectl exec deploy/curl -- curl -s -w "\n%{http_code}" \
  --max-time 10 \
  http://productpage/api/v1/products/1)
END=$(date +%s%N)

HTTP_CODE=$(echo "$RESPONSE" | tail -1)
ELAPSED=$(( (END - START) / 1000000 ))

# Service should timeout and return fallback within 5 seconds
if [ $ELAPSED -gt 6000 ]; then
  echo "FAIL: request took ${ELAPSED}ms, timeout not working"
  exit 1
fi

# Should return degraded response, not 500
if [ "$HTTP_CODE" == "500" ]; then
  echo "FAIL: service crashed on upstream delay instead of degrading gracefully"
  exit 1
fi

echo "PASS: service handled 7s upstream delay within ${ELAPSED}ms with status $HTTP_CODE"

Abort Injection

fault:
  abort:
    percentage:
      value: 50
    httpStatus: 503

Test that 50% upstream failures don't cascade:

ERRORS=0
for i in $(seq 1 50); do
  CODE=$(kubectl exec deploy/curl -- curl -s -o /dev/null -w "%{http_code}" \
    http://productpage/api/v1/products/1)
  if [ "$CODE" == "500" ]; then
    ((ERRORS++))
  fi
done

if [ $ERRORS -gt 5 ]; then
  echo "FAIL: $ERRORS/50 requests returned 500 — upstream faults cascaded"
  exit 1
fi
echo "PASS: service handled 50% upstream failures gracefully ($ERRORS/50 500s)"

The product page should degrade (show product without ratings) rather than returning 500 when the ratings service fails intermittently.

mTLS Verification

mTLS ensures service-to-service traffic is encrypted and mutually authenticated. Testing it means confirming that plaintext traffic is rejected.

Verify mTLS Policy is Active

# Check PeerAuthentication policy
kubectl get peerauthentication -A

# Verify Istio reports mTLS as active for the service
istioctl authn tls-check <pod-name> reviews.default.svc.cluster.local

Expected output for STRICT mode:

HOST:PORT                                      STATUS     SERVER     CLIENT     AUTHN POLICY  DESTINATION RULE
reviews.default.svc.cluster.local:9080         OK         mTLS       mTLS       reviews/default  reviews/default

Test That Plaintext is Rejected

In STRICT mode, unencrypted requests should fail:

# Try to connect directly without Istio sidecar (bypasses mTLS)
kubectl run --rm -it --restart=Never \
  --image=curlimages/curl \
  --overrides='{"spec":{"automountServiceAccountToken":false}}' \
  nocert-test -- curl -v http://reviews.default:9080/api/reviews/1 2>&1

# Expected: connection refused or TLS handshake failure
# If it succeeds, mTLS is NOT enforced

For a programmatic check:

RESPONSE=$(kubectl run --rm -it --restart=Never \
  --image=curlimages/curl nocert-test \
  --overrides='{"spec":{"automountServiceAccountToken":false}}' \
  -- curl -s -o /dev/null -w "%{http_code}" \
  http://reviews.default:9080/ 2>/dev/null)

if [ "$RESPONSE" == "200" ]; then
  echo "FAIL: plaintext request succeeded — mTLS not enforced"
  exit 1
fi
echo "PASS: plaintext request rejected (status: $RESPONSE)"

Verify Certificate Rotation

Check that certificates are being rotated on schedule:

# Get the certificate expiry from the sidecar
kubectl exec <pod-name> -c istio-proxy -- \
  openssl s_client -connect reviews.default:9080 </dev/null 2>/dev/null | \
  openssl x509 -noout -dates

# Verify SPIFFE identity in cert
kubectl exec <pod-name> -c istio-proxy -- \
  openssl s_client -connect reviews.default:9080 </dev/null 2>/dev/null | \
  openssl x509 -noout -text | grep URI
# Should show: URI:spiffe://cluster.local/ns/default/sa/reviews

Retry Policy Testing

http:
- route:
  - destination:
      host: reviews
  retries:
    attempts: 3
    perTryTimeout: 2s
    retryOn: 5xx,reset,connect-failure

Test that retries happen and that they're bounded:

# Make service fail twice, then succeed
# (requires test endpoint on the service)
kubectl exec deploy/reviews -- curl -X POST http://localhost:9080/admin/fail-next/2

START=$(date +%s%N)
RESPONSE=$(kubectl exec deploy/curl -- curl -s -w "\n%{http_code}" http://reviews/api/reviews/1)
END=$(date +%s%N)

HTTP_CODE=$(echo "$RESPONSE" | tail -1)
ELAPSED=$(( (END - START) / 1000000 ))

# Should succeed (retried after 2 failures)
[ "$HTTP_CODE" == "200" ] || (echo "FAIL: expected 200, got $HTTP_CODE"; exit 1)

# Should take between 1 and 6 seconds (2 retries × 2s timeout)
[ $ELAPSED -gt 1000 ] && [ $ELAPSED -lt 7000 ] || \
  (echo "FAIL: timing $ELAPSED ms doesn't match 2-retry pattern"; exit 1)

echo "PASS: retries worked, request succeeded in ${ELAPSED}ms"

Using Kiali for Visual Verification

Kiali (Istio's dashboard) shows traffic flow visually. Automate verification via its API:

# Get traffic graph via Kiali API
curl -s "http://kiali.istio-system:20001/kiali/api/namespaces/default/graph?duration=60s" \
  | jq '.elements.edges[] | select(.data.traffic.rates.http > "0") | {source: .data.source, target: .data.target, rate: .data.traffic.rates.http}'

Use this in CI to verify that expected service-to-service traffic is flowing and unexpected routes aren't active.

Integration with CI

Run Istio tests as part of your deployment pipeline:

# .github/workflows/istio-tests.yaml
- name: Deploy to test namespace
  run: kubectl apply -f k8s/ -n test

- name: Wait for rollout
  run: kubectl rollout status deployment -n test --timeout=120s

- name: Run Istio validation
  run: |
    istioctl analyze -n test
    bash tests/istio/traffic-routing.sh
    bash tests/istio/mtls-verification.sh
    bash tests/istio/fault-injection.sh

- name: Cleanup
  if: always()
  run: kubectl delete namespace test

The key principle: Istio configuration is code. It needs the same test coverage as application code. Traffic weights, fault tolerance, and security policies only work if someone verified them against a real mesh.


Istio tests verify your mesh behavior. For application-level monitoring, end-to-end testing, and 24/7 uptime checks of your Kubernetes services, HelpMeTest provides behavioral testing without requiring service mesh expertise.

Start now free