gRPC Load Testing with ghz and k6 xk6-grpc: Advanced Patterns

gRPC Load Testing with ghz and k6 xk6-grpc: Advanced Patterns

REST API load testing is well understood — tools like k6, Locust, and JMeter handle it easily. gRPC load testing is less documented, especially for streaming RPCs. This guide covers both ghz (purpose-built gRPC benchmarking) and k6 with xk6-grpc (scriptable, CI-friendly load testing) with real examples.

Why gRPC Load Testing Is Different

gRPC over HTTP/2 behaves differently under load than REST over HTTP/1.1:

  • Multiplexing: Multiple requests share a single TCP connection — connection count doesn't scale linearly with request count
  • Backpressure: gRPC has built-in flow control that REST lacks
  • Streaming: Long-lived streams consume connections differently than short unary calls
  • Protobuf overhead: Serialization/deserialization costs affect throughput

Your gRPC load test must measure things REST benchmarks ignore: stream concurrency, connection reuse efficiency, and serialization throughput.

ghz: Purpose-Built gRPC Benchmarking

ghz is the simplest way to benchmark a gRPC service. It's a single binary with no scripting required.

Install:

# macOS
brew install ghz

# Linux
wget https://github.com/bojand/ghz/releases/download/v0.120.0/ghz-linux-x86_64.tar.gz
tar -xzf ghz-linux-x86_64.tar.gz
chmod +x ghz
mv ghz /usr/local/bin/

Basic Benchmarking with ghz

Benchmark a unary RPC with server reflection:

ghz --insecure \
  --call payment.PaymentService/ProcessPayment \
  -d '{"order_id":"ord-123","amount":99.99,"currency":"USD","customer_id":"cust-456"}' \
  -n 1000 \
  -c 50 \
  localhost:50051

Parameters:

  • -n 1000 — total number of requests
  • -c 50 — concurrent workers (goroutines)

Example output:

Summary:
  Count:        1000
  Total:        2.34 s
  Slowest:      45.21 ms
  Fastest:      1.23 ms
  Average:      8.76 ms
  Requests/sec: 427.35

Response time histogram:
  1.23  [1]    |
  5.70  [412]  |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  10.17 [387]  |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  14.64 [142]  |∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  19.11 [38]   |∎∎∎∎
  23.58 [12]   |∎
  28.05 [5]    |
  32.52 [2]    |
  36.99 [1]    |

Status code distribution:
  [OK]   1000 responses

Without Server Reflection (Using Proto Files)

ghz --insecure \
  --proto ./proto/payment/v1/payment.proto \
  --call payment.v1.PaymentService/ProcessPayment \
  -d '{"order_id":"ord-123","amount":99.99,"currency":"USD","customer_id":"cust-456"}' \
  -n 1000 \
  -c 50 \
  localhost:50051

Load Testing with Request Data from File

For realistic load tests with varied data:

# Create a data file with one JSON per line
cat > requests.json << 'EOF'
{"order_id":"ord-001","amount":10.00,"currency":"USD","customer_id":"cust-001"}
{"order_id":"ord-002","amount":250.00,"currency":"EUR","customer_id":"cust-002"}
{"order_id":"ord-003","amount":1500.00,"currency":"GBP","customer_id":"cust-003"}
EOF

ghz --insecure \
  --call payment.PaymentService/ProcessPayment \
  --data-file requests.json \
  -n 3000 \
  -c 100 \
  localhost:50051

ghz cycles through the data file, distributing requests across all workers.

Setting Performance Thresholds

Run ghz and fail if performance degrades:

ghz --insecure \
  --call payment.PaymentService/ProcessPayment \
  -d '{"order_id":"ord-123","amount":99.99,"currency":"USD","customer_id":"cust-456"}' \
  -n 1000 \
  -c 50 \
  --threshold-average 15ms \
  --threshold-p99 50ms \
  --threshold-errors 1 \
  localhost:50051

# Exit code 1 if thresholds exceeded
echo "Exit code: $?"

Threshold options:

  • --threshold-average — average latency limit
  • --threshold-p95 / --threshold-p99 — percentile limits
  • --threshold-errors — max error percentage (0-100)

Rate-Limited Load Testing

Test behavior at a specific request rate (QPS):

# Test at exactly 200 RPS for 30 seconds
ghz --insecure \
  --call payment.PaymentService/ProcessPayment \
  -d '{"order_id":"ord-123","amount":99.99,"currency":"USD","customer_id":"cust-456"}' \
  --rps 200 \
  -z 30s \
  localhost:50051

Streaming Load Tests with ghz

# Test server-streaming RPC
ghz --insecure \
  --call payment.PaymentService/StreamTransactions \
  -d '{"customer_id":"cust-456","limit":10}' \
  -n 500 \
  -c 20 \
  --stream-recv-msg-interval 100ms \
  localhost:50051

JSON Output for CI Integration

ghz --insecure \
  --call payment.PaymentService/ProcessPayment \
  -d '{"order_id":"ord-123","amount":99.99,"currency":"USD","customer_id":"cust-456"}' \
  -n 1000 \
  -c 50 \
  -O json \
  -o results.json \
  localhost:50051

# Parse results with jq
cat results.json | jq '{
  rps: .rps,
  average_ms: (.average / 1000000),
  p99_ms: (.latencyDistribution[] | select(.percentage == 99) | .latency / 1000000),
  error_count: .errorCount
}'

k6 with xk6-grpc: Scriptable Load Testing

k6 with the xk6-grpc extension gives you the full power of k6 scripting — scenarios, ramp-up curves, thresholds, and metrics — applied to gRPC.

Installing k6 with xk6-grpc

xk6-grpc is now part of the official k6 build:

# macOS
brew install k6

# Or download from https://k6.io/docs/get-started/installation/

Basic gRPC Test Script

import grpc from 'k6/net/grpc';
import { check, sleep } from 'k6';

const client = new grpc.Client();

client.load(['proto/'], 'payment/v1/payment.proto');

export const options = {
  vus: 50,
  duration: '30s',
  thresholds: {
    'grpc_req_duration': ['p(99)<50', 'avg<15'],
    'grpc_req_failed': ['rate<0.01'],
  },
};

export default () => {
  client.connect('localhost:50051', { plaintext: true });

  const response = client.invoke('payment.v1.PaymentService/ProcessPayment', {
    order_id: `ord-${Math.random().toString(36).substr(2, 9)}`,
    amount: Math.random() * 1000,
    currency: 'USD',
    customer_id: `cust-${Math.floor(Math.random() * 1000)}`,
  });

  check(response, {
    'status is OK': (r) => r && r.status === grpc.StatusOK,
    'has order_id': (r) => r && r.message && r.message.order_id !== '',
  });

  client.close();
  sleep(0.1);
};

Run it:

k6 run grpc-load-test.js

Ramp-Up Scenarios

Test how your service behaves as load increases:

export const options = {
  scenarios: {
    ramp_up: {
      executor: 'ramping-vus',
      startVUs: 0,
      stages: [
        { duration: '30s', target: 50 },   // ramp up to 50 users
        { duration: '1m', target: 50 },    // stay at 50
        { duration: '30s', target: 100 },  // ramp to 100
        { duration: '1m', target: 100 },   // stay at 100
        { duration: '30s', target: 0 },    // ramp down
      ],
    },
  },
  thresholds: {
    'grpc_req_duration': ['p(95)<100', 'p(99)<200'],
    'grpc_req_failed': ['rate<0.05'],
  },
};

Spike Testing

Verify your service survives sudden traffic spikes:

export const options = {
  scenarios: {
    spike: {
      executor: 'ramping-arrival-rate',
      startRate: 10,
      timeUnit: '1s',
      preAllocatedVUs: 200,
      stages: [
        { duration: '10s', target: 10 },   // normal load
        { duration: '5s', target: 500 },   // sudden spike
        { duration: '10s', target: 500 },  // sustain spike
        { duration: '10s', target: 10 },   // recover
        { duration: '10s', target: 10 },   // verify recovery
      ],
    },
  },
};

Testing with Authentication

import grpc from 'k6/net/grpc';
import { check } from 'k6';

const client = new grpc.Client();
client.load(['proto/'], 'payment/v1/payment.proto');

const AUTH_TOKEN = __ENV.AUTH_TOKEN;

export default () => {
  client.connect('api.example.com:443', {
    tls: true,
  });

  const metadata = {
    'authorization': `Bearer ${AUTH_TOKEN}`,
    'x-request-id': `req-${Date.now()}`,
  };

  const response = client.invoke(
    'payment.v1.PaymentService/ProcessPayment',
    {
      order_id: 'ord-123',
      amount: 99.99,
      currency: 'USD',
      customer_id: 'cust-456',
    },
    { metadata }
  );

  check(response, {
    'status is OK': (r) => r && r.status === grpc.StatusOK,
  });

  client.close();
};

Multi-Service Load Test

Test multiple gRPC services in a single k6 script:

import grpc from 'k6/net/grpc';
import { check, group } from 'k6';

const paymentClient = new grpc.Client();
const inventoryClient = new grpc.Client();

paymentClient.load(['proto/'], 'payment/v1/payment.proto');
inventoryClient.load(['proto/'], 'inventory/v1/inventory.proto');

export const options = {
  vus: 20,
  duration: '30s',
};

export default () => {
  paymentClient.connect('payment-service:50051', { plaintext: true });
  inventoryClient.connect('inventory-service:50052', { plaintext: true });

  group('checkout flow', () => {
    // Check inventory first
    const stockResponse = inventoryClient.invoke(
      'inventory.v1.InventoryService/CheckStock',
      { product_id: 'prod-123', quantity: 1 }
    );
    
    check(stockResponse, {
      'stock available': (r) => r && r.message && r.message.available,
    });

    if (stockResponse.message && stockResponse.message.available) {
      // Process payment
      const paymentResponse = paymentClient.invoke(
        'payment.v1.PaymentService/ProcessPayment',
        { order_id: 'ord-123', amount: 99.99, currency: 'USD', customer_id: 'cust-456' }
      );
      
      check(paymentResponse, {
        'payment successful': (r) => r && r.status === grpc.StatusOK,
      });
    }
  });

  paymentClient.close();
  inventoryClient.close();
};

k6 Custom Metrics for gRPC

Track business-specific metrics:

import grpc from 'k6/net/grpc';
import { check } from 'k6';
import { Counter, Trend } from 'k6/metrics';

const paymentErrors = new Counter('payment_errors');
const paymentLatency = new Trend('payment_latency_ms');

const client = new grpc.Client();
client.load(['proto/'], 'payment/v1/payment.proto');

export default () => {
  client.connect('localhost:50051', { plaintext: true });

  const startTime = Date.now();
  
  const response = client.invoke('payment.v1.PaymentService/ProcessPayment', {
    order_id: 'ord-123',
    amount: 99.99,
    currency: 'USD',
    customer_id: 'cust-456',
  });

  paymentLatency.add(Date.now() - startTime);

  const success = check(response, {
    'status is OK': (r) => r && r.status === grpc.StatusOK,
  });

  if (!success) {
    paymentErrors.add(1);
  }

  client.close();
};

export const options = {
  thresholds: {
    'payment_latency_ms': ['p(99)<100'],
    'payment_errors': ['count<10'],
  },
};

CI/CD Integration

GitHub Actions with ghz

name: gRPC Performance Test

on:
  push:
    branches: [main]

jobs:
  grpc-load-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Start gRPC service
        run: docker-compose up -d grpc-service
      
      - name: Wait for service
        run: |
          for i in $(seq 1 30); do
            grpcurl -plaintext localhost:50051 list && break
            sleep 2
          done
      
      - name: Install ghz
        run: |
          wget -q https://github.com/bojand/ghz/releases/download/v0.120.0/ghz-linux-x86_64.tar.gz
          tar -xzf ghz-linux-x86_64.tar.gz
          sudo mv ghz /usr/local/bin/
      
      - name: Run load test
        run: |
          ghz --insecure \
            --call payment.PaymentService/ProcessPayment \
            -d '{"order_id":"ord-123","amount":99.99,"currency":"USD","customer_id":"cust-456"}' \
            -n 1000 \
            -c 50 \
            --threshold-average 20ms \
            --threshold-p99 100ms \
            --threshold-errors 1 \
            -O json \
            -o results.json \
            localhost:50051
      
      - name: Upload results
        uses: actions/upload-artifact@v4
        with:
          name: grpc-load-test-results
          path: results.json

Establishing and Tracking Performance Baselines

Load tests are useless without baselines. Establish them on day one:

  1. Run ghz at controlled concurrency (e.g., 50 VUs) and record:
    • Average latency
    • p95 and p99 latency
    • Maximum sustainable RPS
    • Error rate at peak load
  2. Store results in a time-series database or artifact storage — compare against baseline in every PR
  3. Set regression thresholds: fail CI if p99 increases by more than 20%
  4. Profile before optimizing: use pprof on the server while ghz runs to find hot paths
# Profile while load testing
go tool pprof http://localhost:6060/debug/pprof/profile?seconds=30 &
ghz --insecure --call payment.PaymentService/ProcessPayment -d '...' -n 5000 -c 50 localhost:50051

Key Takeaways

  • Use ghz for quick benchmarks and CI threshold checks — zero scripting required
  • Use k6 + xk6-grpc for scripted scenarios: ramp-ups, spikes, multi-service flows
  • Always test with realistic data — don't benchmark with the same request repeated 1000 times
  • Test streaming RPCs explicitly — they behave very differently from unary under load
  • Establish baselines before optimizing — you can't improve what you don't measure
  • Run load tests in CI on every merge to catch regressions early, not in production

Read more

Start now free