gRPC Load Testing with ghz and k6
gRPC performs differently under load than REST. HTTP/2 multiplexing changes the concurrency model, binary serialization reduces per-request overhead, and streaming RPCs have unique load characteristics. This guide covers load testing gRPC with ghz and k6, what metrics to capture, and how to interpret them.
ghz: The gRPC Benchmarking Tool
ghz is purpose-built for gRPC load testing. It's the equivalent of wrk or hey for HTTP, but it understands protobuf serialization, streaming, and gRPC metadata.
# Install
brew install ghz # macOS
go install github.com/bojand/ghz/cmd/ghz@latest # go
# or download from https://github.com/bojand/ghz/releasesBasic Unary Load Test
ghz --insecure \
--proto ./proto/user.proto \
--call UserService.GetUser \
--data '{"id": "user-1"}' \
--concurrency 50 \
--total 10000 \
localhost:50051Output:
Summary:
Count: 10000
Total: 2.47 s
Slowest: 48.12 ms
Fastest: 0.23 ms
Average: 11.84 ms
Requests/sec: 4048.58
Response time histogram:
0.230 [1] |
4.991 [1520] |■■■■■■■■■■■■■■■
9.752 [3211] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
14.513 [2897] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■
19.274 [1544] |■■■■■■■■■■■■■■■
24.035 [497] |■■■■■
28.796 [193] |■■
33.557 [88] |■
38.318 [37] |
43.079 [9] |
47.840 [3] |
Latency distribution:
10 % in 4.47 ms
25 % in 6.95 ms
50 % in 11.26 ms
75 % in 16.42 ms
90 % in 21.10 ms
95 % in 24.83 ms
99 % in 34.52 ms
Status code distribution:
[OK] 10000 responsesThe key metrics: p50 (median), p95, p99. p50 tells you typical latency. p99 tells you what the slowest 1% of users experience. The gap between p50 and p99 tells you about variance — a high gap indicates inconsistent performance (often GC pauses, lock contention, or database query variance).
Testing with Dynamic Request Data
Hardcoding one request ID in load tests is unrealistic — you're likely hitting the same cached result. Use ghz's data file feature to randomize inputs:
# Create a data file with multiple request bodies (one JSON per line)
cat > testdata/users.jsonl << 'EOF'
{"id": "user-1"}
{"id": "user-2"}
{"id": "user-3"}
{"id": "user-4"}
{"id": "user-5"}
EOF
ghz --insecure \
--proto ./proto/user.proto \
--call UserService.GetUser \
--data-file testdata/users.jsonl \
--concurrency 50 \
--total 10000 \
localhost:50051For truly random data, use --data with template variables:
ghz --insecure \
--proto ./proto/user.proto \
--call UserService.GetUser \
--data '{"id": "{{randomString 8}}"}' \
--concurrency 50 \
--total 10000 \
localhost:50051Testing with Metadata and Auth
# Bearer token authentication
ghz --insecure \
--proto ./proto/user.proto \
--call UserService.GetUser \
--data '{"id": "user-1"}' \
--metadata '{"authorization": "Bearer eyJhbGci..."}' \
--concurrency 50 \
--total 5000 \
localhost:50051Ramp-Up Load Testing
Test how your service handles increasing load:
# Run ghz in a loop, increasing concurrency
for concurrency in 1 5 10 25 50 100 200; do
echo "=== Concurrency: $concurrency ==="
ghz --insecure \
--proto ./proto/user.proto \
--call UserService.GetUser \
--data '{"id": "user-1"}' \
--concurrency $concurrency \
--total 1000 \
--format summary \
localhost:50051
doneghz Output Formats
# JSON output for CI assertions
ghz --insecure \
--proto ./proto/user.proto \
--call UserService.GetUser \
--data '{"id": "user-1"}' \
--concurrency 50 \
--total 10000 \
--format json \
localhost:50051 | jq '{
rps: .rps,
p50: .latencyDistribution[] | select(.percentage == 50) | .latency,
p99: .latencyDistribution[] | select(.percentage == 99) | .latency
}'
# CSV output for tracking over time
ghz ... --format csv > results/$(date +%Y%m%d_%H%M%S).csv
# HTML report
ghz ... --format html -o report.htmlAsserting on Performance in CI
You don't want load tests to pass silently when p99 doubles. Add assertions:
#!/bin/bash
set -e
RESULT=$(ghz --insecure \
--proto ./proto/user.proto \
--call UserService.GetUser \
--data '{"id": "user-1"}' \
--concurrency 50 \
--total 5000 \
--format json \
localhost:50051)
P99=$(echo $RESULT | jq '.latencyDistribution[] | select(.percentage == 99) | .latency')
RPS=$(echo $RESULT | jq '.rps')
ERROR_RATE=$(echo $RESULT | jq '(.statusCodeDistribution["Unknown"] // 0 + .statusCodeDistribution["Internal"] // 0) / .count')
echo "p99: ${P99}ms, RPS: $RPS, error_rate: $ERROR_RATE"
# Assert p99 < 50ms
if (( $(echo "$P99 > 50" | bc -l) )); then
echo "FAIL: p99 ${P99}ms exceeds 50ms threshold"
exit 1
fi
# Assert error rate < 0.1%
if (( $(echo "$ERROR_RATE > 0.001" | bc -l) )); then
echo "FAIL: error rate $ERROR_RATE exceeds 0.1%"
exit 1
fi
echo "PASS"k6 gRPC Extension
k6 has native gRPC support via the k6/net/grpc module. It's more powerful than ghz for complex scenarios — conditional logic, mixed workloads, VU (virtual user) lifecycle management.
// grpc-load-test.js
import grpc from 'k6/net/grpc';
import { check, sleep } from 'k6';
import { Counter, Trend } from 'k6/metrics';
const client = new grpc.Client();
client.load(['./proto'], 'user.proto');
const grpcErrors = new Counter('grpc_errors');
const userFetchDuration = new Trend('user_fetch_duration');
export const options = {
stages: [
{ duration: '30s', target: 10 }, // ramp up
{ duration: '2m', target: 50 }, // steady state
{ duration: '30s', target: 100 }, // spike
{ duration: '30s', target: 0 }, // ramp down
],
thresholds: {
'grpc_req_duration': ['p(95)<100', 'p(99)<200'],
'grpc_errors': ['count<10'],
},
};
export default function() {
client.connect('localhost:50051', { plaintext: true });
const response = client.invoke('UserService/GetUser', {
id: `user-${Math.floor(Math.random() * 1000)}`,
}, {
metadata: {
'authorization': 'Bearer test-token',
'x-request-id': `req-${__VU}-${__ITER}`,
},
timeout: '5s',
});
check(response, {
'status is OK': (r) => r && r.status === grpc.StatusOK,
'response has id': (r) => r && r.message && r.message.id !== '',
});
if (response.status !== grpc.StatusOK) {
grpcErrors.add(1);
console.error(`Error: ${JSON.stringify(response.error)}`);
}
userFetchDuration.add(response.trailers ? response.trailers['x-server-duration'] : 0);
client.close();
sleep(0.1);
}k6 run grpc-load-test.jsk6 for Mixed Workload Testing
Where k6 shines is testing realistic mixed workloads — a mix of read and write operations with different ratios:
import grpc from 'k6/net/grpc';
import { check } from 'k6';
const client = new grpc.Client();
client.load(['./proto'], 'user.proto');
export const options = {
scenarios: {
read_heavy: {
executor: 'constant-vus',
vus: 40,
duration: '2m',
exec: 'readUser',
},
write_light: {
executor: 'constant-vus',
vus: 10,
duration: '2m',
exec: 'createUser',
},
},
thresholds: {
'grpc_req_duration{scenario:read_heavy}': ['p(99)<50'],
'grpc_req_duration{scenario:write_light}': ['p(99)<200'],
},
};
export function readUser() {
client.connect('localhost:50051', { plaintext: true });
const response = client.invoke('UserService/GetUser', {
id: `user-${Math.floor(Math.random() * 10000)}`,
});
check(response, { 'status is OK': (r) => r.status === grpc.StatusOK });
client.close();
}
export function createUser() {
client.connect('localhost:50051', { plaintext: true });
const response = client.invoke('UserService/CreateUser', {
email: `user-${Date.now()}@example.com`,
name: 'Load Test User',
});
check(response, { 'status is OK or AlreadyExists': (r) =>
r.status === grpc.StatusOK || r.status === grpc.StatusAlreadyExists
});
client.close();
}Measuring Latency Percentiles Correctly
A common mistake in gRPC load testing is measuring only the average. Averages hide the long tail. Always capture:
- p50 — median, typical user experience
- p95 — 1 in 20 requests exceeds this
- p99 — 1 in 100 requests exceeds this; often the SLA target
- p999 — 1 in 1000; relevant for high-traffic services
Another common mistake: warming up the service. The first few hundred requests are slower due to JIT compilation (Java/Python), connection establishment, and cache warming. Always include a warmup phase:
# Warmup phase (not measured)
ghz --insecure --proto ./proto/user.proto --call UserService.GetUser \
--data '{"id": "user-1"}' --concurrency 10 --total 500 --format summary \
localhost:50051 > /dev/null
# Actual measurement
ghz --insecure --proto ./proto/user.proto --call UserService.GetUser \
--data '{"id": "user-1"}' --concurrency 50 --total 10000 \
localhost:50051Concurrent Connection Testing
gRPC over HTTP/2 multiplexes multiple RPCs over a single connection. But under very high concurrency, a single connection becomes a bottleneck. Test how your service handles multiple connections:
# ghz defaults to one connection — use --connections to test multi-connection behavior
ghz --insecure \
--proto ./proto/user.proto \
--call UserService.GetUser \
--data '{"id": "user-1"}' \
--concurrency 100 \
--connections 10 \
--total 50000 \
localhost:50051Compare single-connection vs multi-connection results. If multi-connection is significantly faster, your server may be bottlenecked on a per-connection resource (e.g., a connection-scoped mutex).
gRPC vs REST Performance Comparison
To compare gRPC and REST on the same service, run equivalent tests:
# gRPC
ghz --insecure --proto ./proto/user.proto --call UserService.GetUser \
--data '{"id": "user-1"}' --concurrency 50 --total 10000 \
--format json localhost:50051 > grpc-results.json
# REST (using hey)
hey -n 10000 -c 50 -H "Content-Type: application/json" \
-d '{"id": "user-1"}' \
http://localhost:8080/v1/users/user-1 > rest-results.txtTypical findings:
- Small payloads (<1KB): gRPC is 10-30% faster due to binary encoding overhead savings
- Large payloads (>10KB): gRPC advantage grows due to protobuf's more efficient encoding
- High concurrency: gRPC often scales better due to HTTP/2 multiplexing reducing connection overhead
- Streaming: gRPC wins decisively — HTTP/1.1 streaming is awkward and inefficient
The real performance advantage of gRPC is often not raw throughput but resource consumption — fewer connections, lower memory per RPC, and lower CPU for serialization at high request rates.
Load Testing Streaming RPCs
Streaming requires a different approach. ghz supports server streaming:
# Server streaming load test
ghz --insecure \
--proto ./proto/user.proto \
--call UserService.ListUsers \
--data '{"page_size": 10}' \
--concurrency 20 \
--total 1000 \
localhost:50051For bidirectional streaming, you need k6 or a custom load testing script. The k6 gRPC module supports streaming as of v0.45.
HelpMeTest can run performance regression tests on your gRPC endpoints after every deploy — measuring response time trends and alerting when latency degrades beyond your defined thresholds. If you're shipping gRPC services frequently and need to catch performance regressions before users do, that's the tool.