Load Testing Kafka with Realistic Throughput Scenarios
Kafka can handle millions of messages per second — in theory. Your specific configuration, message size, replication factor, network topology, and consumer group setup determine what it actually handles in practice. Load testing Kafka isn't about proving the broker can handle load; it's about finding where your specific pipeline breaks under realistic conditions.
This post covers the tools and techniques for load testing Kafka producers and consumers: built-in perf tools, k6 with the Kafka extension, JMeter, and what metrics to actually watch.
What You're Testing
Before running any tool, be precise about what question you're answering:
Producer throughput: How many messages per second can your producers send before they start backing off, dropping messages, or increasing latency?
Consumer lag: How quickly does your consumer group process messages relative to the rate they arrive? Lag under steady state should be near zero. Lag under burst indicates your consumers can't keep up.
End-to-end latency: Time from producer send to consumer processing complete. This is the number that matters for real-time systems.
Partition saturation: At what point does adding more partitions stop helping? This tells you whether you're CPU-bound, network-bound, or disk-bound.
Each of these needs a different test setup.
Built-in Tool: kafka-producer-perf-test
Kafka ships with kafka-producer-perf-test.sh. It's the fastest way to establish a producer baseline without writing code:
# Basic throughput test — 1M messages, 1KB each
kafka-producer-perf-test.sh \
--topic load-test-topic \
--num-records 1000000 \
--record-size 1024 \
--throughput -1 \
--producer-props bootstrap.servers=localhost:9092 \
acks=1 \
compression.type=lz4 \
batch.size=65536 \
linger.ms=5
# Throttled test — simulate realistic 10K msg/s rate
kafka-producer-perf-test.sh \
--topic load-test-topic \
--num-records 100000 \
--record-size 512 \
--throughput 10000 \
--producer-props bootstrap.servers=localhost:9092 acks=allSample output:
50000 records sent, 49975.0 records/sec (48.80 MB/sec), 2.1 ms avg latency, 48.0 ms max latency
100000 records sent, 99900.1 records/sec (97.56 MB/sec), 1.8 ms avg latency, 35.0 ms max latency
1000000 records sent, 99980.0 records/sec (97.64 MB/sec), 2.0 ms avg latency, 48.0 ms max latency.The numbers to record: avg latency, max latency, and MB/sec. Run this 3-5 times and take the median — cold JVM and OS page cache state affect single runs.
The companion consumer tool:
kafka-consumer-perf-test.sh \
--topic load-test-topic \
--messages 1000000 \
--bootstrap-server localhost:9092 \
--group perf-consumer-group \
--threads 1Measuring Consumer Lag
Consumer lag is the gap between the last committed offset and the current partition end offset. Monitor it while your producer test runs:
# Watch lag in real-time during a load test
watch -n 2 kafka-consumer-groups.sh \
--bootstrap-server localhost:9092 \
--describe \
--group my-consumer-groupOutput format:
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
orders 0 50234 50234 0
orders 1 49876 50100 224
orders 2 48999 50100 1101Partition 2 has 1101 messages of lag. Under normal load, lag should stay near 0. If it grows during your test, your consumer throughput is less than your producer throughput.
Automate lag measurement and plot it:
#!/usr/bin/env python3
import subprocess
import time
import json
def measure_consumer_lag(bootstrap_server, group):
result = subprocess.run([
'kafka-consumer-groups.sh',
'--bootstrap-server', bootstrap_server,
'--describe',
'--group', group,
'--timeout', '5000'
], capture_output=True, text=True)
total_lag = 0
for line in result.stdout.strip().split('\n')[2:]: # skip headers
parts = line.split()
if len(parts) >= 6 and parts[5].isdigit():
total_lag += int(parts[5])
return total_lag
# Sample every 5 seconds during test
start = time.time()
samples = []
while time.time() - start < 300: # 5-minute test window
lag = measure_consumer_lag('localhost:9092', 'orders-consumer-group')
samples.append({'time': time.time() - start, 'lag': lag})
print(f"t={time.time()-start:.0f}s lag={lag}")
time.sleep(5)Load Testing with k6 Kafka Extension
k6's xk6-kafka extension lets you drive Kafka loads from k6 scripts, which gives you the same scenario-based structure as HTTP load tests:
// install: xk6 build --with github.com/mostafa/xk6-kafka@latest
import { writer, reader, Connection } from "k6/x/kafka";
const brokers = ["localhost:9092"];
const topic = "load-test-orders";
// Define the producer
const kafkaWriter = writer(brokers, topic);
export const options = {
scenarios: {
ramp_up: {
executor: 'ramping-vus',
startVUs: 1,
stages: [
{ duration: '30s', target: 50 }, // ramp to 50 producers
{ duration: '2m', target: 50 }, // hold steady
{ duration: '30s', target: 200 }, // spike
{ duration: '1m', target: 200 }, // hold at spike
{ duration: '30s', target: 0 }, // scale down
],
},
},
thresholds: {
kafka_writer_error_count: ['count<10'],
kafka_writer_write_time: ['p(95)<100'], // 95th percentile under 100ms
},
};
export default function () {
const orderId = `order-${__VU}-${__ITER}`;
const messages = [
{
key: schemaRegistry.serialize({
data: orderId,
schemaType: STRING_SERIALIZER,
}),
value: schemaRegistry.serialize({
data: {
orderId: orderId,
userId: `user-${__VU}`,
amount: Math.random() * 1000,
createdAt: new Date().toISOString(),
},
schemaType: JSON_SERIALIZER,
}),
},
];
kafkaWriter.produce({ messages });
}
export function teardown() {
kafkaWriter.close();
}Run it:
k6 run kafka-load-test.js --out influxdb=http://localhost:8086/k6The thresholds fail the test if error count exceeds 10 or p95 write latency exceeds 100ms. Connect the InfluxDB output to Grafana for real-time visualization.
JMeter Kafka Load Testing
JMeter with the kafkameter plugin is useful if your team already uses JMeter for HTTP load testing:
<!-- Thread Group config -->
<ThreadGroup>
<numThreads>100</numThreads>
<rampTime>30</rampTime>
<duration>180</duration>
</ThreadGroup>
<!-- Kafka Producer Sampler -->
<kg.apc.jmeter.samplers.KafkaSampler>
<kafkaBrokers>localhost:9092</kafkaBrokers>
<kafkaTopic>load-test-orders</kafkaTopic>
<kafkaKey>${orderId}</kafkaKey>
<kafkaMessage>{"orderId":"${orderId}","userId":"user-${threadNum}","amount":${amount}}</kafkaMessage>
<kafkaProducerConfig>
acks=1
compression.type=lz4
batch.size=65536
linger.ms=10
buffer.memory=67108864
</kafkaProducerConfig>
</kg.apc.jmeter.samplers.KafkaSampler>JMeter's advantage here is its built-in assertions and response time graphs. Set throughput assertions to fail the test if avg latency exceeds your SLA.
Testing Partition Strategies Under Load
Partition count affects throughput. The rule of thumb is that a partition can handle about 10MB/s for producers and 50MB/s for consumers. But the right number depends on your broker hardware.
Test partition scaling with a controlled experiment:
#!/bin/bash
# Test throughput at different partition counts
for partitions in 1 3 6 12 24; do
# Create topic with N partitions
kafka-topics.sh --bootstrap-server localhost:9092 \
--create --topic test-partitions-$partitions \
--partitions $partitions --replication-factor 1
echo "=== Testing with $partitions partitions ==="
# Run producer perf test
kafka-producer-perf-test.sh \
--topic test-partitions-$partitions \
--num-records 500000 \
--record-size 1024 \
--throughput -1 \
--producer-props bootstrap.servers=localhost:9092 acks=1 \
2>&1 | tail -1
# Cleanup
kafka-topics.sh --bootstrap-server localhost:9092 \
--delete --topic test-partitions-$partitions
doneTypical output pattern:
=== Testing with 1 partitions ===
500000 records sent, 185000 records/sec (180.66 MB/sec)...
=== Testing with 6 partitions ===
500000 records sent, 510000 records/sec (498.05 MB/sec)...
=== Testing with 12 partitions ===
500000 records sent, 520000 records/sec (507.81 MB/sec)...
=== Testing with 24 partitions ===
500000 records sent, 518000 records/sec (505.86 MB/sec)...Throughput usually plateaus around 6-12 partitions per broker. Beyond that, you're adding coordination overhead without throughput gains. The right number for your setup is where the curve flattens.
Testing Consumer Scalability
Verify that adding consumer instances actually reduces lag:
@Test
void consumerScaling_lagReducesWithMoreInstances() throws Exception {
// Produce 100K messages with no consumer running
produceMessages(100_000, "scaling-test");
long initialLag = measureConsumerLag("scaling-test-group");
assertThat(initialLag).isGreaterThan(90_000);
// Start 1 consumer
startConsumerInstance("scaling-test-group", 1);
Thread.sleep(10_000);
long lagWith1 = measureConsumerLag("scaling-test-group");
// Start 2 more consumers
startConsumerInstance("scaling-test-group", 2);
startConsumerInstance("scaling-test-group", 3);
Thread.sleep(10_000);
long lagWith3 = measureConsumerLag("scaling-test-group");
assertThat(lagWith3).isLessThan(lagWith1 / 2); // rough 3x improvement
}Key Producer Configuration Parameters Under Load
These settings have the biggest impact on producer throughput under load:
Map<String, Object> producerProps = new HashMap<>();
producerProps.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
producerProps.put(ProducerConfig.ACKS_CONFIG, "1"); // "all" for durability, "1" for throughput
producerProps.put(ProducerConfig.BATCH_SIZE_CONFIG, 65536); // 64KB batches
producerProps.put(ProducerConfig.LINGER_MS_CONFIG, 10); // Wait up to 10ms to fill batches
producerProps.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "lz4"); // lz4 is best throughput/CPU tradeoff
producerProps.put(ProducerConfig.BUFFER_MEMORY_CONFIG, 67108864L); // 64MB buffer
producerProps.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, 5);linger.ms is the most impactful single setting. Default is 0 (send immediately). Setting it to 5-20ms allows batching, which dramatically improves throughput at the cost of slight latency increase. Under load, this tradeoff almost always makes sense.
End-to-End Latency Under Load
Producer throughput is only half the picture. Measure end-to-end latency:
@Test
void endToEndLatency_p99UnderTargetAtSteadyState() throws Exception {
List<Long> latencies = new ArrayList<>();
CountDownLatch processed = new CountDownLatch(1000);
// Consumer records timestamp when it processes a message
consumerService.setProcessingCallback((messageId, receivedAt) -> {
Long sentAt = sentTimestamps.get(messageId);
if (sentAt != null) {
latencies.add(receivedAt - sentAt);
processed.countDown();
}
});
// Produce 1000 messages at 500 msg/s
for (int i = 0; i < 1000; i++) {
String messageId = "latency-" + i;
sentTimestamps.put(messageId, System.currentTimeMillis());
kafkaTemplate.send("orders", messageId, new Order(messageId, "u1", BigDecimal.ONE));
Thread.sleep(2); // 2ms = 500 msg/s
}
assertThat(processed.await(60, TimeUnit.SECONDS)).isTrue();
// Calculate percentiles
List<Long> sorted = latencies.stream().sorted().collect(Collectors.toList());
long p50 = sorted.get((int)(sorted.size() * 0.50));
long p95 = sorted.get((int)(sorted.size() * 0.95));
long p99 = sorted.get((int)(sorted.size() * 0.99));
System.out.printf("p50=%dms p95=%dms p99=%dms%n", p50, p95, p99);
assertThat(p99).isLessThan(500); // SLA: p99 < 500ms
assertThat(p50).isLessThan(50); // Steady state: p50 < 50ms
}What to Do with the Numbers
After running load tests, you'll have numbers. Here's how to interpret them:
Producer throughput below expected: Check network bandwidth first. Then check producer batching config (batch.size, linger.ms). Then check broker disk throughput with iostat.
Consumer lag growing under load: Either your consumer is CPU-bound (add more consumer instances), your consumer is doing slow I/O (profile it), or your consumer group has fewer instances than topic partitions (max parallelism = partition count).
High p99 latency but low p50: Garbage collection pauses. Check JVM GC logs. Use G1GC or ZGC for Kafka clients with high throughput.
Throughput plateaus at N partitions: You've hit broker I/O limits. Distribute partitions across more brokers or add storage IOPS.
Run load tests in an environment that matches production as closely as possible — same broker count, same replication factor, same network topology. Load test results from a single-broker local setup don't transfer to a 3-broker production cluster.
HelpMeTest can monitor your deployed Kafka consumers in production — flagging when consumer lag spikes, when message processing rates drop, or when end-to-end latency exceeds your SLA. It runs continuously so you know about degradation before your users do.