Load Testing Kafka with Gatling and JMeter
Load testing Kafka reveals the configuration gaps that correctness tests miss — producer batch size that limits throughput, linger.ms settings that add latency, and consumer lag that accumulates under sustained load. This guide covers running meaningful load tests with Gatling's Kafka DSL and the JMeter Kafka plugin, interpreting the output, and tuning the parameters that matter most.
Key Takeaways
Producer throughput is dominated by batch.size and linger.ms. Small batches at zero linger maximizes latency but kills throughput. Tuning these two parameters can multiply throughput by 5–10x with no code changes.
Consumer lag is the load test metric that predicts production incidents. A consumer group falling behind under load will eventually fall behind in production. Measure lag, not just throughput, in every load test.
Gatling's Kafka DSL lets you express load shapes as code. Ramp-up, constant load, spike injection, and complex multi-step shapes are all first-class constructs. Results are automatically charted.
JMeter's Kafka plugin is more accessible for teams already running JMeter. Configuration is GUI-driven, integrates with existing JMeter infrastructure, and supports distributed load generation without writing code.
Isolate the broker from the test harness. Running producer load tests on the same machine as the broker contaminates results. Use separate machines or a Kubernetes namespace with resource limits.
A Kafka cluster that passes all correctness tests can still fail in production. The failure mode is not incorrect messages — it is throughput collapse, runaway consumer lag, or GC pauses on the broker under sustained write load. These failures are invisible in unit and integration tests because those tests never generate the volume that triggers them.
Load testing Kafka means answering three questions: How fast can this producer publish? At what point does consumer lag start to grow? And what configuration changes close the gap?
What to Measure
Before running any tool, define your metrics:
Producer side:
- Records per second (throughput)
- Bytes per second
- Average and p99 produce latency (time from
producer.send()to broker acknowledgment) - Record error rate
Consumer side:
- Consumer lag (records behind the latest offset)
- Records consumed per second
- End-to-end latency (time from produce to consume)
Broker side:
- Under-replicated partitions (indicates follower lag)
- Active controller count (should be exactly 1)
- Request queue time (queuing at the broker network layer)
- Log flush latency
JMX is the standard interface for broker metrics. Expose them via Prometheus with the JMX Exporter agent, then visualize in Grafana during load tests.
Gatling Kafka DSL
The gatling-kafka-plugin adds first-class Kafka support to Gatling's simulation DSL. It handles both synchronous (fire-and-forget) and request-reply patterns.
Add the dependency to your build.sbt or pom.xml:
// build.sbt
libraryDependencies += "io.github.amerousful" %% "gatling-kafka" % "2.1.0" % Test
libraryDependencies += "io.gatling.highcharts" % "gatling-charts-highcharts" % "3.10.5" % TestA basic producer load simulation:
import io.github.amerousful.kafka.Predef._
import io.gatling.core.Predef._
import scala.concurrent.duration._
class OrderProducerSimulation extends Simulation {
val kafkaConf = kafka
.broker(KafkaBroker("localhost", 9092))
.acks("1")
.producerIdenticalSerializer(classOf[StringSerializer])
val scn = scenario("Produce order events")
.exec(
kafka("publish order.created")
.send[String, String](
key = session => s"order-${session.userId}",
payload = session => s"""{"orderId":"${session.userId}","status":"CREATED","ts":${System.currentTimeMillis()}}"""
)
.topic("order-events")
)
setUp(
scn.inject(
rampUsersPerSec(10).to(500).during(60.seconds),
constantUsersPerSec(500).during(120.seconds),
rampUsersPerSec(500).to(0).during(30.seconds)
)
).protocols(kafkaConf)
.assertions(
global.successfulRequests.percent.gt(99),
global.responseTime.percentile3.lt(50) // p99 < 50ms
)
}The load shape here is: ramp from 10 to 500 virtual users over 60 seconds, hold at 500 for 2 minutes, then ramp down. Each virtual user produces one message per iteration.
Run the simulation:
mvn gatling:test -Dgatling.simulationClass=OrderProducerSimulation
# or with Gradle:
gradle gatlingRun-OrderProducerSimulationGatling generates an HTML report in target/gatling/. The report shows throughput over time, response time distributions, and assertion results.
Gatling: Request-Reply Pattern
For testing consumer throughput in a request-reply setup (e.g., RPC over Kafka), Gatling can correlate sent and received messages:
val kafkaConf = kafka
.broker(KafkaBroker("localhost", 9092))
.acks("all")
.producerIdenticalSerializer(classOf[StringSerializer])
.consumerIdenticalDeserializer(classOf[StringDeserializer])
.replyTimeout(10.seconds)
.matchByKey
val scn = scenario("Order processing RPC")
.exec(
kafka("send order and await reply")
.requestReply[String, String, String, String](
key = session => session.userId.toString,
payload = session => s"""{"orderId":"${session.userId}"}""",
requestTopic = "order-requests",
replyTopic = "order-replies"
)
)matchByKey tells Gatling to match a reply message to the request with the same key. The replyTimeout controls how long Gatling waits before marking the interaction as failed.
Tuning Producer Configuration for Load Tests
Run your load test twice: once with default configuration, once with optimized configuration. The difference is instructive.
Default producer configuration produces one record at a time:
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
// Defaults: batch.size=16384, linger.ms=0, compression.type=noneWith linger.ms=0, the producer sends every record immediately — no batching. This maximizes latency and minimizes throughput.
Optimized for throughput:
props.put(ProducerConfig.BATCH_SIZE_CONFIG, 131072); // 128KB batches
props.put(ProducerConfig.LINGER_MS_CONFIG, 20); // wait up to 20ms to fill a batch
props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "lz4"); // compress batches
props.put(ProducerConfig.BUFFER_MEMORY_CONFIG, 67108864); // 64MB send buffer
props.put(ProducerConfig.ACKS_CONFIG, "1"); // leader ack onlyTypical results on a single-node broker with a modern NVMe disk:
| Configuration | Throughput | p99 Latency |
|---|---|---|
| Default (linger.ms=0, batch.size=16KB) | ~40,000 records/sec | ~25ms |
| batch.size=128KB, linger.ms=20ms | ~350,000 records/sec | ~45ms |
| + lz4 compression | ~520,000 records/sec | ~50ms |
The trade-off: linger.ms increases latency slightly in exchange for batching. For throughput-sensitive pipelines, 20ms is usually acceptable. For latency-sensitive pipelines (real-time fraud detection), keep linger.ms low and accept lower throughput.
JMeter Kafka Plugin
JMeter's Kafka support comes via the community plugin pepper-box. Install it through the JMeter Plugin Manager or by dropping the JAR into lib/ext/:
wget https://github.com/rsvihladremio/jmeter-kafka/releases/download/v1.1.0/rsvihladremio-jmeter-kafka-1.1.0.jar \
-P $JMETER_HOME/lib/ext/For a GUI-free setup in CI, define your test plan as XML. Here is a minimal plan that ramps to 100 threads producing Kafka messages:
<?xml version="1.0" encoding="UTF-8"?>
<jmeterTestPlan version="1.2">
<hashTree>
<TestPlan testname="Kafka Load Test" enabled="true">
<hashTree>
<ThreadGroup testname="Producers" enabled="true">
<stringProp name="ThreadGroup.num_threads">100</stringProp>
<stringProp name="ThreadGroup.ramp_time">30</stringProp>
<intProp name="ThreadGroup.duration">120</intProp>
<boolProp name="ThreadGroup.scheduler">true</boolProp>
<hashTree>
<GenericSampler testname="Kafka Producer" enabled="true"
classname="com.rsvihladremio.jmeter.kafka.KafkaProducerSampler">
<stringProp name="kafka.broker">localhost:9092</stringProp>
<stringProp name="kafka.topic">order-events</stringProp>
<stringProp name="kafka.key">${__UUID()}</stringProp>
<stringProp name="kafka.message">{"orderId":"${__UUID()}","status":"CREATED"}</stringProp>
<stringProp name="kafka.producer.acks">1</stringProp>
<stringProp name="kafka.producer.batch.size">131072</stringProp>
<stringProp name="kafka.producer.linger.ms">20</stringProp>
</GenericSampler>
</hashTree>
</ThreadGroup>
</hashTree>
</TestPlan>
</hashTree>
</jmeterTestPlan>Run headless in CI:
jmeter -n \
-t kafka-load-test.jmx \
-l results.jtl \
-e -o report/ \
-Jkafka.broker=$KAFKA_BOOTSTRAP_SERVERSThe -e -o report/ flags generate an HTML dashboard. The results.jtl file can be archived as a CI artifact and compared across runs to detect regressions.
Measuring Consumer Lag
Producer throughput tests tell you how fast you can write. Consumer lag tests tell you how fast you can read — and whether you can keep up.
The Kafka consumer group offset API exposes lag directly:
public Map<TopicPartition, Long> measureConsumerLag(
String bootstrapServers,
String groupId,
String topic
) {
try (AdminClient admin = AdminClient.create(
Map.of(BOOTSTRAP_SERVERS_CONFIG, bootstrapServers))) {
// Get end offsets
Map<TopicPartition, OffsetSpec> request = new HashMap<>();
for (int i = 0; i < 12; i++) { // 12 partitions
request.put(new TopicPartition(topic, i), OffsetSpec.latest());
}
Map<TopicPartition, ListOffsetsResult.ListOffsetsResultInfo> endOffsets =
admin.listOffsets(request).all().get();
// Get committed offsets for the consumer group
Map<TopicPartition, OffsetAndMetadata> committedOffsets =
admin.listConsumerGroupOffsets(groupId)
.partitionsToOffsetAndMetadata()
.get();
// Lag = end offset - committed offset
Map<TopicPartition, Long> lag = new HashMap<>();
for (Map.Entry<TopicPartition, ListOffsetsResult.ListOffsetsResultInfo> e
: endOffsets.entrySet()) {
OffsetAndMetadata committed = committedOffsets.get(e.getKey());
long committedOffset = committed != null ? committed.offset() : 0;
lag.put(e.getKey(), e.getValue().offset() - committedOffset);
}
return lag;
}
}Call this during your load test on a 5-second interval to track lag growth:
ScheduledExecutorService lagMonitor = Executors.newSingleThreadScheduledExecutor();
lagMonitor.scheduleAtFixedRate(() -> {
Map<TopicPartition, Long> lag = measureConsumerLag(
bootstrapServers, "order-processors", "order-events");
long totalLag = lag.values().stream().mapToLong(Long::longValue).sum();
System.out.printf("[%s] Total consumer lag: %d%n",
Instant.now(), totalLag);
lagMetrics.record(totalLag);
}, 0, 5, TimeUnit.SECONDS);If total lag stays near zero throughout the test, your consumers can keep up. If lag grows linearly, your consumer is slower than your producer — you need more consumer instances or faster processing logic. If lag grows then stabilizes, you've found the saturation point.
Kafka Consumer Tuning
When load tests reveal consumer lag, the first tuning lever is fetch.min.bytes and fetch.max.wait.ms:
// Consumer optimized for throughput
Properties consumerProps = new Properties();
consumerProps.put(ConsumerConfig.FETCH_MIN_BYTES_CONFIG, 50000); // wait for 50KB
consumerProps.put(ConsumerConfig.FETCH_MAX_WAIT_MS_CONFIG, 500); // or 500ms
consumerProps.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, 500); // 500 records per poll
consumerProps.put(ConsumerConfig.MAX_PARTITION_FETCH_BYTES_CONFIG, 1048576); // 1MB per partitionIf the processing logic is the bottleneck (not the consumer I/O), add consumer instances in the same consumer group. Kafka distributes partitions across consumers in a group — up to one consumer per partition. A topic with 12 partitions can be consumed by up to 12 consumers in parallel.
Running Load Tests in CI
Load tests belong in CI, but not on every PR. A typical setup:
- Every PR: integration correctness tests (Testcontainers, fast)
- Nightly: load tests against a staging cluster, compare to baseline
- Pre-release: extended load tests (30+ minutes) with consumer lag monitoring
GitHub Actions example for a nightly load test:
name: Kafka Load Tests
on:
schedule:
- cron: "0 2 * * *" # 2 AM nightly
workflow_dispatch: # allow manual trigger
jobs:
load-test:
runs-on: ubuntu-latest
services:
zookeeper:
image: confluentinc/cp-zookeeper:7.6.1
env:
ZOOKEEPER_CLIENT_PORT: 2181
kafka:
image: confluentinc/cp-kafka:7.6.1
env:
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
ports:
- 9092:9092
steps:
- uses: actions/checkout@v4
- uses: actions/setup-java@v4
with:
java-version: '21'
distribution: 'temurin'
- name: Run Gatling simulation
run: mvn gatling:test -Dgatling.simulationClass=OrderProducerSimulation
env:
KAFKA_BOOTSTRAP_SERVERS: localhost:9092
- name: Upload Gatling report
uses: actions/upload-artifact@v4
if: always()
with:
name: gatling-report
path: target/gatling/
- name: Check throughput regression
run: |
THROUGHPUT=$(grep "Requests.*total" target/gatling/*/index.html | \
grep -oP '\d+(?= req/s)')
echo "Throughput: $THROUGHPUT req/s"
if [ "$THROUGHPUT" -lt "200000" ]; then
echo "FAIL: Throughput below 200k req/s baseline"
exit 1
fiThe regression check compares throughput against a known baseline. When a producer code change drops throughput by 30%, the CI run fails and the developer knows immediately — before it reaches production.
Interpreting Load Test Results
A Gatling report shows three key charts: requests per second over time, response time over time, and response time distribution. For Kafka producer tests, interpret them as:
- Flat requests/sec with rising response time: the broker is saturating. Add partitions, upgrade the broker, or reduce replication factor.
- Requests/sec drops mid-test: producer buffer is full (
buffer.memoryexhausted). Increasebuffer.memoryor reducelinger.ms. - High p99, low p50: outliers from GC pauses. Tune JVM GC settings on the broker (G1GC with
-XX:MaxGCPauseMillis=20is a starting point). - Error rate spikes with
TimeoutException:request.timeout.msis too short for the current load. Increase it, or reduce load.
Load testing without establishing a baseline first is pattern-matching on noise. Run the same simulation three times before making tuning changes, average the results, and track the trend over time. A one-time result is a data point; a trend is knowledge.