gRPC in Kubernetes: Testing Services with Istio, Envoy, and Service Mesh

gRPC in Kubernetes: Testing Services with Istio, Envoy, and Service Mesh

gRPC services in Kubernetes behave differently than in local tests. Load balancing works at the connection level, not the request level. Istio's mTLS intercepts and re-encrypts traffic. Retries and circuit breakers in the service mesh can mask service failures. And health check probes need to speak gRPC, not HTTP/1.1.

This guide covers how to test gRPC services in Kubernetes deployments, including service mesh integration.

The Kubernetes gRPC Stack

In a typical Kubernetes + Istio setup:

Client Pod
    ↓ (gRPC/HTTP2)
Envoy Sidecar (injected by Istio)
    ↓ (mTLS)
Envoy Sidecar (in server pod)
    ↓ (plaintext to localhost)
Server Container

Your application never sees mTLS certificates directly — Istio handles it. But this means your tests need to account for Istio's retry, timeout, and circuit-breaker policies.

gRPC Health Checks in Kubernetes

Kubernetes liveness and readiness probes default to HTTP/1.1. For gRPC services, use grpc-health-probe or Kubernetes' native gRPC probe (K8s 1.24+):

Native Kubernetes gRPC Probe (K8s 1.24+)

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: user-service
spec:
  template:
    spec:
      containers:
        - name: user-service
          image: myorg/user-service:latest
          ports:
            - containerPort: 50051
          livenessProbe:
            grpc:
              port: 50051
              service: ""  # empty = default health check
            initialDelaySeconds: 10
            periodSeconds: 10
          readinessProbe:
            grpc:
              port: 50051
              service: "user.v1.UserService"  # specific service
            initialDelaySeconds: 5
            periodSeconds: 5

Implementing the Health Service

Your gRPC server must implement the grpc.health.v1 protocol:

// main.go
import (
    "google.golang.org/grpc/health"
    healthpb "google.golang.org/grpc/health/grpc_health_v1"
)

func main() {
    server := grpc.NewServer()
    
    // Register your service
    pb.RegisterUserServiceServer(server, &UserServiceImpl{})
    
    // Register health service
    healthServer := health.NewServer()
    healthpb.RegisterHealthServer(server, healthServer)
    
    // Set service health
    healthServer.SetServingStatus("user.v1.UserService", healthpb.HealthCheckResponse_SERVING)
    
    server.Serve(lis)
}

Test the health endpoint:

# From within the cluster
kubectl run grpcurl --rm -it --image=fullstorydev/grpcurl -- \
  grpcurl -plaintext user-service:50051 grpc.health.v1.Health/Check

Testing Health Checks with grpc-health-probe

# CI test: verify health check works before deploying
- name: Test gRPC health check
  run: |
    kubectl run test-health --rm --image=grpc-ecosystem/grpc-health-probe \
      -- /bin/grpc_health_probe -addr=user-service:50051 -service=user.v1.UserService

Load Balancing with gRPC

gRPC uses HTTP/2, which multiplexes requests over a single TCP connection. Standard Kubernetes Services (ClusterIP) do TCP-level load balancing — all requests go to the same pod.

Fix this with:

  1. Headless service + client-side load balancing (for same-namespace services)
  2. Istio VirtualService (recommended for service mesh deployments)

Testing Load Distribution

# Deploy test: check that requests actually spread across pods
kubectl run load-test --rm -it --image=fullstorydev/grpcurl -- bash

# From inside the test pod:
for i in $(seq 1 20); do
  grpcurl -plaintext user-service.default.svc.cluster.local:50051 \
    user.v1.UserService/GetUser <<< '{"user_id": "user-1"}' | \
    grep hostname  # Your service should return its pod hostname
done | sort | uniq -c

With a headless service and client-side load balancing, you should see roughly equal distribution. With a ClusterIP service, all 20 go to the same pod.

Testing Istio Traffic Policies

Istio can inject delays, return specific error codes, and abort requests — useful for chaos testing and resilience testing.

Injecting Delays to Test Timeout Behavior

# istio-fault-delay.yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: user-service
spec:
  hosts:
    - user-service
  http:
    - fault:
        delay:
          percentage:
            value: 100.0
          fixedDelay: 5s
      route:
        - destination:
            host: user-service

Apply and test:

kubectl apply -f istio-fault-delay.yaml

# Your client should timeout (if configured correctly)
grpcurl -max-time 3 -plaintext user-service:50051 user.v1.UserService/GetUser <<< '{}'
# Should fail with: Error: context deadline exceeded

Injecting Errors for Resilience Testing

# istio-fault-abort.yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: user-service
spec:
  hosts:
    - user-service
  http:
    - fault:
        abort:
          percentage:
            value: 50.0
          grpcStatus: "UNAVAILABLE"
      route:
        - destination:
            host: user-service

This makes 50% of requests fail with UNAVAILABLE. Your client should retry and eventually succeed (or gracefully degrade).

Testing Retries with Istio

Configure retries in the VirtualService and verify they work:

# istio-retry.yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: user-service
spec:
  hosts:
    - user-service
  http:
    - timeout: 10s
      retries:
        attempts: 3
        perTryTimeout: 3s
        retryOn: "reset,connect-failure,retriable-4xx,grpc-status-codes"
        retryRemoteStatuses: "503"
      route:
        - destination:
            host: user-service

Test retry behavior:

# Apply fault injection that fails 30% of requests
# With 3 retries, the overall success rate should be ~97%+
kubectl apply -f istio-fault-abort-30pct.yaml
kubectl apply -f istio-retry.yaml

# Run 100 requests, count failures
for i in $(seq 1 100); do
  grpcurl -plaintext user-service:50051 user.v1.UserService/GetUser <<< '{}' \
    && echo "OK" || echo "FAIL"
done | grep -c "FAIL"
# Should be close to 0 with retries; close to 30 without

Integration Testing in Kubernetes with Ephemeral Test Pods

Run integration tests from within the cluster to test real gRPC endpoints:

# k8s-test-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: grpc-integration-tests
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: test-runner
          image: myorg/grpc-tests:${IMAGE_TAG}
          command: ["go", "test", "./integration/...", "-v", "-timeout=5m"]
          env:
            - name: USER_SERVICE_ADDR
              value: "user-service.default.svc.cluster.local:50051"
            - name: ORDER_SERVICE_ADDR
              value: "order-service.default.svc.cluster.local:50051"
# Deploy and wait for completion
kubectl apply -f k8s-test-job.yaml
kubectl wait --for=condition=complete job/grpc-integration-tests --timeout=300s

# Get test output
kubectl logs job/grpc-integration-tests

Testing mTLS Is Working

Verify that Istio's mTLS is actually enforced (plaintext traffic is rejected):

# Try to connect with plaintext from outside the mesh — should fail
kubectl run plaintext-test --rm -it --image=fullstorydev/grpcurl \
  -- grpcurl -plaintext user-service:50051 grpc.health.v1.Health/Check
# Expected: connection refused or TLS required error

# Test from inside the mesh (Istio injects mTLS) — should succeed
kubectl run mesh-test --rm -it --image=fullstorydev/grpcurl \
  -- grpcurl -plaintext user-service:50051 grpc.health.v1.Health/Check
# Expected: success (Istio handles TLS transparently)

CI/CD Pipeline for gRPC in Kubernetes

# .github/workflows/grpc-k8s-test.yml
name: gRPC Kubernetes Tests

on:
  push:
    branches: [main]

jobs:
  integration-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Setup kind cluster
        uses: helm/kind-action@v1
        with: { cluster_name: test-cluster }
      
      - name: Install Istio
        run: |
          curl -L https://istio.io/downloadIstio | sh -
          istio-*/bin/istioctl install --set profile=minimal -y
          kubectl label namespace default istio-injection=enabled
      
      - name: Deploy services
        run: |
          kubectl apply -f k8s/
          kubectl rollout status deployment/user-service --timeout=120s
      
      - name: Run gRPC health checks
        run: |
          kubectl run health-check --rm --restart=Never \
            --image=grpc-ecosystem/grpc-health-probe \
            -- /bin/grpc_health_probe -addr=user-service:50051
      
      - name: Run integration tests
        run: |
          kubectl apply -f k8s/test-job.yaml
          kubectl wait --for=condition=complete job/grpc-integration-tests --timeout=300s
          kubectl logs job/grpc-integration-tests

Conclusion

Testing gRPC in Kubernetes requires thinking at multiple layers: the service implementation, the health check protocol, load balancing behavior, and the service mesh policies. Unit tests cover service logic. Kubernetes health checks verify availability. Istio fault injection tests resilience. And ephemeral test jobs running inside the cluster give you true end-to-end coverage that no amount of local integration testing can replicate.

Read more

Start now free