gRPC Load Testing with ghz and k6

gRPC Load Testing with ghz and k6

gRPC performs differently under load than REST. HTTP/2 multiplexing changes the concurrency model, binary serialization reduces per-request overhead, and streaming RPCs have unique load characteristics. This guide covers load testing gRPC with ghz and k6, what metrics to capture, and how to interpret them.

ghz: The gRPC Benchmarking Tool

ghz is purpose-built for gRPC load testing. It's the equivalent of wrk or hey for HTTP, but it understands protobuf serialization, streaming, and gRPC metadata.

# Install
brew install ghz          # macOS
go install github.com/bojand/ghz/cmd/ghz@latest   # go
# or download from https://github.com/bojand/ghz/releases

Basic Unary Load Test

ghz --insecure \
    --proto ./proto/user.proto \
    --call UserService.GetUser \
    --data '{"id": "user-1"}' \
    --concurrency 50 \
    --total 10000 \
    localhost:50051

Output:

Summary:
  Count:        10000
  Total:        2.47 s
  Slowest:      48.12 ms
  Fastest:      0.23 ms
  Average:      11.84 ms
  Requests/sec: 4048.58

Response time histogram:
  0.230 [1]      |
  4.991 [1520]   |■■■■■■■■■■■■■■■
  9.752 [3211]   |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  14.513 [2897]  |■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  19.274 [1544]  |■■■■■■■■■■■■■■■
  24.035 [497]   |■■■■■
  28.796 [193]   |■■
  33.557 [88]    |■
  38.318 [37]    |
  43.079 [9]     |
  47.840 [3]     |

Latency distribution:
  10 % in 4.47 ms
  25 % in 6.95 ms
  50 % in 11.26 ms
  75 % in 16.42 ms
  90 % in 21.10 ms
  95 % in 24.83 ms
  99 % in 34.52 ms

Status code distribution:
  [OK]   10000 responses

The key metrics: p50 (median), p95, p99. p50 tells you typical latency. p99 tells you what the slowest 1% of users experience. The gap between p50 and p99 tells you about variance — a high gap indicates inconsistent performance (often GC pauses, lock contention, or database query variance).

Testing with Dynamic Request Data

Hardcoding one request ID in load tests is unrealistic — you're likely hitting the same cached result. Use ghz's data file feature to randomize inputs:

# Create a data file with multiple request bodies (one JSON per line)
cat > testdata/users.jsonl << 'EOF'
{"id": "user-1"}
{"id": "user-2"}
{"id": "user-3"}
{"id": "user-4"}
{"id": "user-5"}
EOF

ghz --insecure \
    --proto ./proto/user.proto \
    --call UserService.GetUser \
    --data-file testdata/users.jsonl \
    --concurrency 50 \
    --total 10000 \
    localhost:50051

For truly random data, use --data with template variables:

ghz --insecure \
    --proto ./proto/user.proto \
    --call UserService.GetUser \
    --data '{"id": "{{randomString 8}}"}' \
    --concurrency 50 \
    --total 10000 \
    localhost:50051

Testing with Metadata and Auth

# Bearer token authentication
ghz --insecure \
    --proto ./proto/user.proto \
    --call UserService.GetUser \
    --data '{"id": "user-1"}' \
    --metadata '{"authorization": "Bearer eyJhbGci..."}' \
    --concurrency 50 \
    --total 5000 \
    localhost:50051

Ramp-Up Load Testing

Test how your service handles increasing load:

# Run ghz in a loop, increasing concurrency
for concurrency in 1 5 10 25 50 100 200; do
    echo "=== Concurrency: $concurrency ==="
    ghz --insecure \
        --proto ./proto/user.proto \
        --call UserService.GetUser \
        --data '{"id": "user-1"}' \
        --concurrency $concurrency \
        --total 1000 \
        --format summary \
        localhost:50051
done

ghz Output Formats

# JSON output for CI assertions
ghz --insecure \
    --proto ./proto/user.proto \
    --call UserService.GetUser \
    --data '{"id": "user-1"}' \
    --concurrency 50 \
    --total 10000 \
    --format json \
    localhost:50051 | jq '{
        rps: .rps,
        p50: .latencyDistribution[] | select(.percentage == 50) | .latency,
        p99: .latencyDistribution[] | select(.percentage == 99) | .latency
    }'

# CSV output for tracking over time
ghz ... --format csv > results/$(date +%Y%m%d_%H%M%S).csv

# HTML report
ghz ... --format html -o report.html

Asserting on Performance in CI

You don't want load tests to pass silently when p99 doubles. Add assertions:

#!/bin/bash
set -e

RESULT=$(ghz --insecure \
    --proto ./proto/user.proto \
    --call UserService.GetUser \
    --data '{"id": "user-1"}' \
    --concurrency 50 \
    --total 5000 \
    --format json \
    localhost:50051)

P99=$(echo $RESULT | jq '.latencyDistribution[] | select(.percentage == 99) | .latency')
RPS=$(echo $RESULT | jq '.rps')
ERROR_RATE=$(echo $RESULT | jq '(.statusCodeDistribution["Unknown"] // 0 + .statusCodeDistribution["Internal"] // 0) / .count')

echo "p99: ${P99}ms, RPS: $RPS, error_rate: $ERROR_RATE"

# Assert p99 < 50ms
if (( $(echo "$P99 > 50" | bc -l) )); then
    echo "FAIL: p99 ${P99}ms exceeds 50ms threshold"
    exit 1
fi

# Assert error rate < 0.1%
if (( $(echo "$ERROR_RATE > 0.001" | bc -l) )); then
    echo "FAIL: error rate $ERROR_RATE exceeds 0.1%"
    exit 1
fi

echo "PASS"

k6 gRPC Extension

k6 has native gRPC support via the k6/net/grpc module. It's more powerful than ghz for complex scenarios — conditional logic, mixed workloads, VU (virtual user) lifecycle management.

// grpc-load-test.js
import grpc from 'k6/net/grpc';
import { check, sleep } from 'k6';
import { Counter, Trend } from 'k6/metrics';

const client = new grpc.Client();
client.load(['./proto'], 'user.proto');

const grpcErrors = new Counter('grpc_errors');
const userFetchDuration = new Trend('user_fetch_duration');

export const options = {
    stages: [
        { duration: '30s', target: 10 },   // ramp up
        { duration: '2m', target: 50 },    // steady state
        { duration: '30s', target: 100 },  // spike
        { duration: '30s', target: 0 },    // ramp down
    ],
    thresholds: {
        'grpc_req_duration': ['p(95)<100', 'p(99)<200'],
        'grpc_errors': ['count<10'],
    },
};

export default function() {
    client.connect('localhost:50051', { plaintext: true });

    const response = client.invoke('UserService/GetUser', {
        id: `user-${Math.floor(Math.random() * 1000)}`,
    }, {
        metadata: {
            'authorization': 'Bearer test-token',
            'x-request-id': `req-${__VU}-${__ITER}`,
        },
        timeout: '5s',
    });

    check(response, {
        'status is OK': (r) => r && r.status === grpc.StatusOK,
        'response has id': (r) => r && r.message && r.message.id !== '',
    });

    if (response.status !== grpc.StatusOK) {
        grpcErrors.add(1);
        console.error(`Error: ${JSON.stringify(response.error)}`);
    }

    userFetchDuration.add(response.trailers ? response.trailers['x-server-duration'] : 0);

    client.close();
    sleep(0.1);
}
k6 run grpc-load-test.js

k6 for Mixed Workload Testing

Where k6 shines is testing realistic mixed workloads — a mix of read and write operations with different ratios:

import grpc from 'k6/net/grpc';
import { check } from 'k6';

const client = new grpc.Client();
client.load(['./proto'], 'user.proto');

export const options = {
    scenarios: {
        read_heavy: {
            executor: 'constant-vus',
            vus: 40,
            duration: '2m',
            exec: 'readUser',
        },
        write_light: {
            executor: 'constant-vus',
            vus: 10,
            duration: '2m',
            exec: 'createUser',
        },
    },
    thresholds: {
        'grpc_req_duration{scenario:read_heavy}': ['p(99)<50'],
        'grpc_req_duration{scenario:write_light}': ['p(99)<200'],
    },
};

export function readUser() {
    client.connect('localhost:50051', { plaintext: true });
    const response = client.invoke('UserService/GetUser', {
        id: `user-${Math.floor(Math.random() * 10000)}`,
    });
    check(response, { 'status is OK': (r) => r.status === grpc.StatusOK });
    client.close();
}

export function createUser() {
    client.connect('localhost:50051', { plaintext: true });
    const response = client.invoke('UserService/CreateUser', {
        email: `user-${Date.now()}@example.com`,
        name: 'Load Test User',
    });
    check(response, { 'status is OK or AlreadyExists': (r) =>
        r.status === grpc.StatusOK || r.status === grpc.StatusAlreadyExists
    });
    client.close();
}

Measuring Latency Percentiles Correctly

A common mistake in gRPC load testing is measuring only the average. Averages hide the long tail. Always capture:

  • p50 — median, typical user experience
  • p95 — 1 in 20 requests exceeds this
  • p99 — 1 in 100 requests exceeds this; often the SLA target
  • p999 — 1 in 1000; relevant for high-traffic services

Another common mistake: warming up the service. The first few hundred requests are slower due to JIT compilation (Java/Python), connection establishment, and cache warming. Always include a warmup phase:

# Warmup phase (not measured)
ghz --insecure --proto ./proto/user.proto --call UserService.GetUser \
    --data '{"id": "user-1"}' --concurrency 10 --total 500 --format summary \
    localhost:50051 > /dev/null

# Actual measurement
ghz --insecure --proto ./proto/user.proto --call UserService.GetUser \
    --data '{"id": "user-1"}' --concurrency 50 --total 10000 \
    localhost:50051

Concurrent Connection Testing

gRPC over HTTP/2 multiplexes multiple RPCs over a single connection. But under very high concurrency, a single connection becomes a bottleneck. Test how your service handles multiple connections:

# ghz defaults to one connection — use --connections to test multi-connection behavior
ghz --insecure \
    --proto ./proto/user.proto \
    --call UserService.GetUser \
    --data '{"id": "user-1"}' \
    --concurrency 100 \
    --connections 10 \
    --total 50000 \
    localhost:50051

Compare single-connection vs multi-connection results. If multi-connection is significantly faster, your server may be bottlenecked on a per-connection resource (e.g., a connection-scoped mutex).

gRPC vs REST Performance Comparison

To compare gRPC and REST on the same service, run equivalent tests:

# gRPC
ghz --insecure --proto ./proto/user.proto --call UserService.GetUser \
    --data '{"id": "user-1"}' --concurrency 50 --total 10000 \
    --format json localhost:50051 > grpc-results.json

# REST (using hey)
hey -n 10000 -c 50 -H "Content-Type: application/json" \
    -d '{"id": "user-1"}' \
    http://localhost:8080/v1/users/user-1 > rest-results.txt

Typical findings:

  • Small payloads (<1KB): gRPC is 10-30% faster due to binary encoding overhead savings
  • Large payloads (>10KB): gRPC advantage grows due to protobuf's more efficient encoding
  • High concurrency: gRPC often scales better due to HTTP/2 multiplexing reducing connection overhead
  • Streaming: gRPC wins decisively — HTTP/1.1 streaming is awkward and inefficient

The real performance advantage of gRPC is often not raw throughput but resource consumption — fewer connections, lower memory per RPC, and lower CPU for serialization at high request rates.

Load Testing Streaming RPCs

Streaming requires a different approach. ghz supports server streaming:

# Server streaming load test
ghz --insecure \
    --proto ./proto/user.proto \
    --call UserService.ListUsers \
    --data '{"page_size": 10}' \
    --concurrency 20 \
    --total 1000 \
    localhost:50051

For bidirectional streaming, you need k6 or a custom load testing script. The k6 gRPC module supports streaming as of v0.45.


HelpMeTest can run performance regression tests on your gRPC endpoints after every deploy — measuring response time trends and alerting when latency degrades beyond your defined thresholds. If you're shipping gRPC services frequently and need to catch performance regressions before users do, that's the tool.

Read more

Start now free