Load Testing Kafka with Gatling and JMeter

Load Testing Kafka with Gatling and JMeter

Load testing Kafka reveals the configuration gaps that correctness tests miss — producer batch size that limits throughput, linger.ms settings that add latency, and consumer lag that accumulates under sustained load. This guide covers running meaningful load tests with Gatling's Kafka DSL and the JMeter Kafka plugin, interpreting the output, and tuning the parameters that matter most.

Key Takeaways

Producer throughput is dominated by batch.size and linger.ms. Small batches at zero linger maximizes latency but kills throughput. Tuning these two parameters can multiply throughput by 5–10x with no code changes.

Consumer lag is the load test metric that predicts production incidents. A consumer group falling behind under load will eventually fall behind in production. Measure lag, not just throughput, in every load test.

Gatling's Kafka DSL lets you express load shapes as code. Ramp-up, constant load, spike injection, and complex multi-step shapes are all first-class constructs. Results are automatically charted.

JMeter's Kafka plugin is more accessible for teams already running JMeter. Configuration is GUI-driven, integrates with existing JMeter infrastructure, and supports distributed load generation without writing code.

Isolate the broker from the test harness. Running producer load tests on the same machine as the broker contaminates results. Use separate machines or a Kubernetes namespace with resource limits.

A Kafka cluster that passes all correctness tests can still fail in production. The failure mode is not incorrect messages — it is throughput collapse, runaway consumer lag, or GC pauses on the broker under sustained write load. These failures are invisible in unit and integration tests because those tests never generate the volume that triggers them.

Load testing Kafka means answering three questions: How fast can this producer publish? At what point does consumer lag start to grow? And what configuration changes close the gap?

What to Measure

Before running any tool, define your metrics:

Producer side:

  • Records per second (throughput)
  • Bytes per second
  • Average and p99 produce latency (time from producer.send() to broker acknowledgment)
  • Record error rate

Consumer side:

  • Consumer lag (records behind the latest offset)
  • Records consumed per second
  • End-to-end latency (time from produce to consume)

Broker side:

  • Under-replicated partitions (indicates follower lag)
  • Active controller count (should be exactly 1)
  • Request queue time (queuing at the broker network layer)
  • Log flush latency

JMX is the standard interface for broker metrics. Expose them via Prometheus with the JMX Exporter agent, then visualize in Grafana during load tests.

Gatling Kafka DSL

The gatling-kafka-plugin adds first-class Kafka support to Gatling's simulation DSL. It handles both synchronous (fire-and-forget) and request-reply patterns.

Add the dependency to your build.sbt or pom.xml:

// build.sbt
libraryDependencies += "io.github.amerousful" %% "gatling-kafka" % "2.1.0" % Test
libraryDependencies += "io.gatling.highcharts" % "gatling-charts-highcharts" % "3.10.5" % Test

A basic producer load simulation:

import io.github.amerousful.kafka.Predef._
import io.gatling.core.Predef._
import scala.concurrent.duration._

class OrderProducerSimulation extends Simulation {

  val kafkaConf = kafka
    .broker(KafkaBroker("localhost", 9092))
    .acks("1")
    .producerIdenticalSerializer(classOf[StringSerializer])

  val scn = scenario("Produce order events")
    .exec(
      kafka("publish order.created")
        .send[String, String](
          key = session => s"order-${session.userId}",
          payload = session => s"""{"orderId":"${session.userId}","status":"CREATED","ts":${System.currentTimeMillis()}}"""
        )
        .topic("order-events")
    )

  setUp(
    scn.inject(
      rampUsersPerSec(10).to(500).during(60.seconds),
      constantUsersPerSec(500).during(120.seconds),
      rampUsersPerSec(500).to(0).during(30.seconds)
    )
  ).protocols(kafkaConf)
    .assertions(
      global.successfulRequests.percent.gt(99),
      global.responseTime.percentile3.lt(50) // p99 < 50ms
    )
}

The load shape here is: ramp from 10 to 500 virtual users over 60 seconds, hold at 500 for 2 minutes, then ramp down. Each virtual user produces one message per iteration.

Run the simulation:

mvn gatling:test -Dgatling.simulationClass=OrderProducerSimulation
# or with Gradle:
gradle gatlingRun-OrderProducerSimulation

Gatling generates an HTML report in target/gatling/. The report shows throughput over time, response time distributions, and assertion results.

Gatling: Request-Reply Pattern

For testing consumer throughput in a request-reply setup (e.g., RPC over Kafka), Gatling can correlate sent and received messages:

val kafkaConf = kafka
  .broker(KafkaBroker("localhost", 9092))
  .acks("all")
  .producerIdenticalSerializer(classOf[StringSerializer])
  .consumerIdenticalDeserializer(classOf[StringDeserializer])
  .replyTimeout(10.seconds)
  .matchByKey

val scn = scenario("Order processing RPC")
  .exec(
    kafka("send order and await reply")
      .requestReply[String, String, String, String](
        key = session => session.userId.toString,
        payload = session => s"""{"orderId":"${session.userId}"}""",
        requestTopic = "order-requests",
        replyTopic = "order-replies"
      )
  )

matchByKey tells Gatling to match a reply message to the request with the same key. The replyTimeout controls how long Gatling waits before marking the interaction as failed.

Tuning Producer Configuration for Load Tests

Run your load test twice: once with default configuration, once with optimized configuration. The difference is instructive.

Default producer configuration produces one record at a time:

Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
// Defaults: batch.size=16384, linger.ms=0, compression.type=none

With linger.ms=0, the producer sends every record immediately — no batching. This maximizes latency and minimizes throughput.

Optimized for throughput:

props.put(ProducerConfig.BATCH_SIZE_CONFIG, 131072);      // 128KB batches
props.put(ProducerConfig.LINGER_MS_CONFIG, 20);           // wait up to 20ms to fill a batch
props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "lz4"); // compress batches
props.put(ProducerConfig.BUFFER_MEMORY_CONFIG, 67108864); // 64MB send buffer
props.put(ProducerConfig.ACKS_CONFIG, "1");               // leader ack only

Typical results on a single-node broker with a modern NVMe disk:

Configuration Throughput p99 Latency
Default (linger.ms=0, batch.size=16KB) ~40,000 records/sec ~25ms
batch.size=128KB, linger.ms=20ms ~350,000 records/sec ~45ms
+ lz4 compression ~520,000 records/sec ~50ms

The trade-off: linger.ms increases latency slightly in exchange for batching. For throughput-sensitive pipelines, 20ms is usually acceptable. For latency-sensitive pipelines (real-time fraud detection), keep linger.ms low and accept lower throughput.

JMeter Kafka Plugin

JMeter's Kafka support comes via the community plugin pepper-box. Install it through the JMeter Plugin Manager or by dropping the JAR into lib/ext/:

wget https://github.com/rsvihladremio/jmeter-kafka/releases/download/v1.1.0/rsvihladremio-jmeter-kafka-1.1.0.jar \
  -P $JMETER_HOME/lib/ext/

For a GUI-free setup in CI, define your test plan as XML. Here is a minimal plan that ramps to 100 threads producing Kafka messages:

<?xml version="1.0" encoding="UTF-8"?>
<jmeterTestPlan version="1.2">
  <hashTree>
    <TestPlan testname="Kafka Load Test" enabled="true">
      <hashTree>
        <ThreadGroup testname="Producers" enabled="true">
          <stringProp name="ThreadGroup.num_threads">100</stringProp>
          <stringProp name="ThreadGroup.ramp_time">30</stringProp>
          <intProp name="ThreadGroup.duration">120</intProp>
          <boolProp name="ThreadGroup.scheduler">true</boolProp>
          <hashTree>
            <GenericSampler testname="Kafka Producer" enabled="true"
              classname="com.rsvihladremio.jmeter.kafka.KafkaProducerSampler">
              <stringProp name="kafka.broker">localhost:9092</stringProp>
              <stringProp name="kafka.topic">order-events</stringProp>
              <stringProp name="kafka.key">${__UUID()}</stringProp>
              <stringProp name="kafka.message">{"orderId":"${__UUID()}","status":"CREATED"}</stringProp>
              <stringProp name="kafka.producer.acks">1</stringProp>
              <stringProp name="kafka.producer.batch.size">131072</stringProp>
              <stringProp name="kafka.producer.linger.ms">20</stringProp>
            </GenericSampler>
          </hashTree>
        </ThreadGroup>
      </hashTree>
    </TestPlan>
  </hashTree>
</jmeterTestPlan>

Run headless in CI:

jmeter -n \
  -t kafka-load-test.jmx \
  -l results.jtl \
  -e -o report/ \
  -Jkafka.broker=$KAFKA_BOOTSTRAP_SERVERS

The -e -o report/ flags generate an HTML dashboard. The results.jtl file can be archived as a CI artifact and compared across runs to detect regressions.

Measuring Consumer Lag

Producer throughput tests tell you how fast you can write. Consumer lag tests tell you how fast you can read — and whether you can keep up.

The Kafka consumer group offset API exposes lag directly:

public Map<TopicPartition, Long> measureConsumerLag(
    String bootstrapServers,
    String groupId,
    String topic
) {
    try (AdminClient admin = AdminClient.create(
            Map.of(BOOTSTRAP_SERVERS_CONFIG, bootstrapServers))) {

        // Get end offsets
        Map<TopicPartition, OffsetSpec> request = new HashMap<>();
        for (int i = 0; i < 12; i++) { // 12 partitions
            request.put(new TopicPartition(topic, i), OffsetSpec.latest());
        }
        Map<TopicPartition, ListOffsetsResult.ListOffsetsResultInfo> endOffsets =
            admin.listOffsets(request).all().get();

        // Get committed offsets for the consumer group
        Map<TopicPartition, OffsetAndMetadata> committedOffsets =
            admin.listConsumerGroupOffsets(groupId)
                .partitionsToOffsetAndMetadata()
                .get();

        // Lag = end offset - committed offset
        Map<TopicPartition, Long> lag = new HashMap<>();
        for (Map.Entry<TopicPartition, ListOffsetsResult.ListOffsetsResultInfo> e
                : endOffsets.entrySet()) {
            OffsetAndMetadata committed = committedOffsets.get(e.getKey());
            long committedOffset = committed != null ? committed.offset() : 0;
            lag.put(e.getKey(), e.getValue().offset() - committedOffset);
        }
        return lag;
    }
}

Call this during your load test on a 5-second interval to track lag growth:

ScheduledExecutorService lagMonitor = Executors.newSingleThreadScheduledExecutor();
lagMonitor.scheduleAtFixedRate(() -> {
    Map<TopicPartition, Long> lag = measureConsumerLag(
        bootstrapServers, "order-processors", "order-events");
    long totalLag = lag.values().stream().mapToLong(Long::longValue).sum();
    System.out.printf("[%s] Total consumer lag: %d%n",
        Instant.now(), totalLag);
    lagMetrics.record(totalLag);
}, 0, 5, TimeUnit.SECONDS);

If total lag stays near zero throughout the test, your consumers can keep up. If lag grows linearly, your consumer is slower than your producer — you need more consumer instances or faster processing logic. If lag grows then stabilizes, you've found the saturation point.

Kafka Consumer Tuning

When load tests reveal consumer lag, the first tuning lever is fetch.min.bytes and fetch.max.wait.ms:

// Consumer optimized for throughput
Properties consumerProps = new Properties();
consumerProps.put(ConsumerConfig.FETCH_MIN_BYTES_CONFIG, 50000);    // wait for 50KB
consumerProps.put(ConsumerConfig.FETCH_MAX_WAIT_MS_CONFIG, 500);    // or 500ms
consumerProps.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, 500);     // 500 records per poll
consumerProps.put(ConsumerConfig.MAX_PARTITION_FETCH_BYTES_CONFIG, 1048576); // 1MB per partition

If the processing logic is the bottleneck (not the consumer I/O), add consumer instances in the same consumer group. Kafka distributes partitions across consumers in a group — up to one consumer per partition. A topic with 12 partitions can be consumed by up to 12 consumers in parallel.

Running Load Tests in CI

Load tests belong in CI, but not on every PR. A typical setup:

  • Every PR: integration correctness tests (Testcontainers, fast)
  • Nightly: load tests against a staging cluster, compare to baseline
  • Pre-release: extended load tests (30+ minutes) with consumer lag monitoring

GitHub Actions example for a nightly load test:

name: Kafka Load Tests
on:
  schedule:
    - cron: "0 2 * * *"  # 2 AM nightly
  workflow_dispatch:      # allow manual trigger

jobs:
  load-test:
    runs-on: ubuntu-latest
    services:
      zookeeper:
        image: confluentinc/cp-zookeeper:7.6.1
        env:
          ZOOKEEPER_CLIENT_PORT: 2181
      kafka:
        image: confluentinc/cp-kafka:7.6.1
        env:
          KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
          KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
          KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
          KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
        ports:
          - 9092:9092
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-java@v4
        with:
          java-version: '21'
          distribution: 'temurin'
      - name: Run Gatling simulation
        run: mvn gatling:test -Dgatling.simulationClass=OrderProducerSimulation
        env:
          KAFKA_BOOTSTRAP_SERVERS: localhost:9092
      - name: Upload Gatling report
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: gatling-report
          path: target/gatling/
      - name: Check throughput regression
        run: |
          THROUGHPUT=$(grep "Requests.*total" target/gatling/*/index.html | \
            grep -oP '\d+(?= req/s)')
          echo "Throughput: $THROUGHPUT req/s"
          if [ "$THROUGHPUT" -lt "200000" ]; then
            echo "FAIL: Throughput below 200k req/s baseline"
            exit 1
          fi

The regression check compares throughput against a known baseline. When a producer code change drops throughput by 30%, the CI run fails and the developer knows immediately — before it reaches production.

Interpreting Load Test Results

A Gatling report shows three key charts: requests per second over time, response time over time, and response time distribution. For Kafka producer tests, interpret them as:

  • Flat requests/sec with rising response time: the broker is saturating. Add partitions, upgrade the broker, or reduce replication factor.
  • Requests/sec drops mid-test: producer buffer is full (buffer.memory exhausted). Increase buffer.memory or reduce linger.ms.
  • High p99, low p50: outliers from GC pauses. Tune JVM GC settings on the broker (G1GC with -XX:MaxGCPauseMillis=20 is a starting point).
  • Error rate spikes with TimeoutException: request.timeout.ms is too short for the current load. Increase it, or reduce load.

Load testing without establishing a baseline first is pattern-matching on noise. Run the same simulation three times before making tuning changes, average the results, and track the trend over time. A one-time result is a data point; a trend is knowledge.

Read more

Start now free