gRPC in Kubernetes: Testing Services with Istio, Envoy, and Service Mesh
gRPC services in Kubernetes behave differently than in local tests. Load balancing works at the connection level, not the request level. Istio's mTLS intercepts and re-encrypts traffic. Retries and circuit breakers in the service mesh can mask service failures. And health check probes need to speak gRPC, not HTTP/1.1.
This guide covers how to test gRPC services in Kubernetes deployments, including service mesh integration.
The Kubernetes gRPC Stack
In a typical Kubernetes + Istio setup:
Client Pod
↓ (gRPC/HTTP2)
Envoy Sidecar (injected by Istio)
↓ (mTLS)
Envoy Sidecar (in server pod)
↓ (plaintext to localhost)
Server ContainerYour application never sees mTLS certificates directly — Istio handles it. But this means your tests need to account for Istio's retry, timeout, and circuit-breaker policies.
gRPC Health Checks in Kubernetes
Kubernetes liveness and readiness probes default to HTTP/1.1. For gRPC services, use grpc-health-probe or Kubernetes' native gRPC probe (K8s 1.24+):
Native Kubernetes gRPC Probe (K8s 1.24+)
# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: user-service
spec:
template:
spec:
containers:
- name: user-service
image: myorg/user-service:latest
ports:
- containerPort: 50051
livenessProbe:
grpc:
port: 50051
service: "" # empty = default health check
initialDelaySeconds: 10
periodSeconds: 10
readinessProbe:
grpc:
port: 50051
service: "user.v1.UserService" # specific service
initialDelaySeconds: 5
periodSeconds: 5Implementing the Health Service
Your gRPC server must implement the grpc.health.v1 protocol:
// main.go
import (
"google.golang.org/grpc/health"
healthpb "google.golang.org/grpc/health/grpc_health_v1"
)
func main() {
server := grpc.NewServer()
// Register your service
pb.RegisterUserServiceServer(server, &UserServiceImpl{})
// Register health service
healthServer := health.NewServer()
healthpb.RegisterHealthServer(server, healthServer)
// Set service health
healthServer.SetServingStatus("user.v1.UserService", healthpb.HealthCheckResponse_SERVING)
server.Serve(lis)
}Test the health endpoint:
# From within the cluster
kubectl run grpcurl --rm -it --image=fullstorydev/grpcurl -- \
grpcurl -plaintext user-service:50051 grpc.health.v1.Health/CheckTesting Health Checks with grpc-health-probe
# CI test: verify health check works before deploying
- name: Test gRPC health check
run: |
kubectl run test-health --rm --image=grpc-ecosystem/grpc-health-probe \
-- /bin/grpc_health_probe -addr=user-service:50051 -service=user.v1.UserServiceLoad Balancing with gRPC
gRPC uses HTTP/2, which multiplexes requests over a single TCP connection. Standard Kubernetes Services (ClusterIP) do TCP-level load balancing — all requests go to the same pod.
Fix this with:
- Headless service + client-side load balancing (for same-namespace services)
- Istio VirtualService (recommended for service mesh deployments)
Testing Load Distribution
# Deploy test: check that requests actually spread across pods
kubectl run load-test --rm -it --image=fullstorydev/grpcurl -- bash
# From inside the test pod:
for i in $(seq 1 20); do
grpcurl -plaintext user-service.default.svc.cluster.local:50051 \
user.v1.UserService/GetUser <<< '{"user_id": "user-1"}' | \
grep hostname # Your service should return its pod hostname
done | sort | uniq -cWith a headless service and client-side load balancing, you should see roughly equal distribution. With a ClusterIP service, all 20 go to the same pod.
Testing Istio Traffic Policies
Istio can inject delays, return specific error codes, and abort requests — useful for chaos testing and resilience testing.
Injecting Delays to Test Timeout Behavior
# istio-fault-delay.yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: user-service
spec:
hosts:
- user-service
http:
- fault:
delay:
percentage:
value: 100.0
fixedDelay: 5s
route:
- destination:
host: user-serviceApply and test:
kubectl apply -f istio-fault-delay.yaml
# Your client should timeout (if configured correctly)
grpcurl -max-time 3 -plaintext user-service:50051 user.v1.UserService/GetUser <<< '{}'
# Should fail with: Error: context deadline exceededInjecting Errors for Resilience Testing
# istio-fault-abort.yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: user-service
spec:
hosts:
- user-service
http:
- fault:
abort:
percentage:
value: 50.0
grpcStatus: "UNAVAILABLE"
route:
- destination:
host: user-serviceThis makes 50% of requests fail with UNAVAILABLE. Your client should retry and eventually succeed (or gracefully degrade).
Testing Retries with Istio
Configure retries in the VirtualService and verify they work:
# istio-retry.yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: user-service
spec:
hosts:
- user-service
http:
- timeout: 10s
retries:
attempts: 3
perTryTimeout: 3s
retryOn: "reset,connect-failure,retriable-4xx,grpc-status-codes"
retryRemoteStatuses: "503"
route:
- destination:
host: user-serviceTest retry behavior:
# Apply fault injection that fails 30% of requests
# With 3 retries, the overall success rate should be ~97%+
kubectl apply -f istio-fault-abort-30pct.yaml
kubectl apply -f istio-retry.yaml
# Run 100 requests, count failures
for i in $(seq 1 100); do
grpcurl -plaintext user-service:50051 user.v1.UserService/GetUser <<< '{}' \
&& echo "OK" || echo "FAIL"
done | grep -c "FAIL"
# Should be close to 0 with retries; close to 30 withoutIntegration Testing in Kubernetes with Ephemeral Test Pods
Run integration tests from within the cluster to test real gRPC endpoints:
# k8s-test-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
name: grpc-integration-tests
spec:
template:
spec:
restartPolicy: Never
containers:
- name: test-runner
image: myorg/grpc-tests:${IMAGE_TAG}
command: ["go", "test", "./integration/...", "-v", "-timeout=5m"]
env:
- name: USER_SERVICE_ADDR
value: "user-service.default.svc.cluster.local:50051"
- name: ORDER_SERVICE_ADDR
value: "order-service.default.svc.cluster.local:50051"# Deploy and wait for completion
kubectl apply -f k8s-test-job.yaml
kubectl wait --for=condition=complete job/grpc-integration-tests --timeout=300s
# Get test output
kubectl logs job/grpc-integration-testsTesting mTLS Is Working
Verify that Istio's mTLS is actually enforced (plaintext traffic is rejected):
# Try to connect with plaintext from outside the mesh — should fail
kubectl run plaintext-test --rm -it --image=fullstorydev/grpcurl \
-- grpcurl -plaintext user-service:50051 grpc.health.v1.Health/Check
# Expected: connection refused or TLS required error
# Test from inside the mesh (Istio injects mTLS) — should succeed
kubectl run mesh-test --rm -it --image=fullstorydev/grpcurl \
-- grpcurl -plaintext user-service:50051 grpc.health.v1.Health/Check
# Expected: success (Istio handles TLS transparently)CI/CD Pipeline for gRPC in Kubernetes
# .github/workflows/grpc-k8s-test.yml
name: gRPC Kubernetes Tests
on:
push:
branches: [main]
jobs:
integration-test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup kind cluster
uses: helm/kind-action@v1
with: { cluster_name: test-cluster }
- name: Install Istio
run: |
curl -L https://istio.io/downloadIstio | sh -
istio-*/bin/istioctl install --set profile=minimal -y
kubectl label namespace default istio-injection=enabled
- name: Deploy services
run: |
kubectl apply -f k8s/
kubectl rollout status deployment/user-service --timeout=120s
- name: Run gRPC health checks
run: |
kubectl run health-check --rm --restart=Never \
--image=grpc-ecosystem/grpc-health-probe \
-- /bin/grpc_health_probe -addr=user-service:50051
- name: Run integration tests
run: |
kubectl apply -f k8s/test-job.yaml
kubectl wait --for=condition=complete job/grpc-integration-tests --timeout=300s
kubectl logs job/grpc-integration-testsConclusion
Testing gRPC in Kubernetes requires thinking at multiple layers: the service implementation, the health check protocol, load balancing behavior, and the service mesh policies. Unit tests cover service logic. Kubernetes health checks verify availability. Istio fault injection tests resilience. And ephemeral test jobs running inside the cluster give you true end-to-end coverage that no amount of local integration testing can replicate.