Consumer Group Lag Monitoring in Kafka Tests

Consumer Group Lag Monitoring in Kafka Tests

Consumer group lag is the gap between the last produced message offset and the last committed consumer offset. When lag grows unbounded, your consumers are falling behind producers — a condition that leads to stale data, delayed processing, and eventually out-of-memory crashes as your consumer tries to catch up. Monitoring lag is standard practice in production, but testing that your consumers stay within acceptable lag thresholds is something most teams skip entirely.

This guide covers how to write integration tests that validate consumer lag behavior: how to measure lag programmatically, how to assert maximum acceptable lag in tests, and how to verify that your consumer group recovers after being paused or restarted.

Why Test Lag?

Consumer lag issues are subtle. A consumer can be healthy (connected, polling) but still accumulate lag if:

  • Processing time per message exceeds the poll interval
  • The consumer is blocked on an external call (database, HTTP) that is slower than expected
  • Thread pool saturation prevents the listener from picking up new messages
  • The consumer has been assigned too many partitions for its throughput capacity

These issues do not cause consumer crashes or connection errors — the consumer group stays alive and looks healthy from the outside. Only by measuring the actual offset gap do you detect the problem.

Writing lag assertions in integration tests gives you a regression check: if a code change causes your consumer to process 10x slower, the lag test fails before the change reaches production.

Setup: AdminClient for Lag Measurement

Kafka's AdminClient API provides everything needed to compute lag:

public class KafkaLagMeasurer implements AutoCloseable {

    private final AdminClient adminClient;

    public KafkaLagMeasurer(String bootstrapServers) {
        this.adminClient = AdminClient.create(Map.of(
            AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers,
            AdminClientConfig.REQUEST_TIMEOUT_MS_CONFIG, "5000"
        ));
    }

    /**
     * Returns total lag across all partitions for a given consumer group and topic.
     */
    public long getTotalLag(String groupId, String topic) throws Exception {
        // Step 1: Get committed offsets for the consumer group
        Map<TopicPartition, OffsetAndMetadata> committed = adminClient
            .listConsumerGroupOffsets(groupId)
            .partitionsToOffsetAndMetadata()
            .get(10, TimeUnit.SECONDS);

        // Step 2: Get end offsets (log end offset) for those partitions
        Set<TopicPartition> partitions = committed.keySet().stream()
            .filter(tp -> tp.topic().equals(topic))
            .collect(Collectors.toSet());

        if (partitions.isEmpty()) {
            return 0L; // No committed offsets yet
        }

        Map<TopicPartition, ListOffsetsResult.ListOffsetsResultInfo> endOffsets = adminClient
            .listOffsets(partitions.stream()
                .collect(Collectors.toMap(tp -> tp, tp -> OffsetSpec.latest())))
            .all()
            .get(10, TimeUnit.SECONDS);

        // Step 3: Compute lag per partition and sum
        long totalLag = 0;
        for (TopicPartition partition : partitions) {
            long committedOffset = committed.get(partition).offset();
            long endOffset = endOffsets.get(partition).offset();
            totalLag += Math.max(0, endOffset - committedOffset);
        }

        return totalLag;
    }

    /**
     * Returns lag per partition for detailed inspection.
     */
    public Map<TopicPartition, Long> getLagPerPartition(String groupId, String topic) 
            throws Exception {
        Map<TopicPartition, OffsetAndMetadata> committed = adminClient
            .listConsumerGroupOffsets(groupId)
            .partitionsToOffsetAndMetadata()
            .get(10, TimeUnit.SECONDS);

        Set<TopicPartition> partitions = committed.keySet().stream()
            .filter(tp -> tp.topic().equals(topic))
            .collect(Collectors.toSet());

        Map<TopicPartition, ListOffsetsResult.ListOffsetsResultInfo> endOffsets = adminClient
            .listOffsets(partitions.stream()
                .collect(Collectors.toMap(tp -> tp, tp -> OffsetSpec.latest())))
            .all()
            .get(10, TimeUnit.SECONDS);

        return partitions.stream()
            .collect(Collectors.toMap(
                tp -> tp,
                tp -> Math.max(0,
                    endOffsets.get(tp).offset() - committed.get(tp).offset())
            ));
    }

    @Override
    public void close() {
        adminClient.close();
    }
}

Base Test Class

@SpringBootTest
@Testcontainers
public abstract class LagTestBase {

    @Container
    static final KafkaContainer kafka = new KafkaContainer(
            DockerImageName.parse("confluentinc/cp-kafka:7.6.0"))
        .withReuse(true);

    protected KafkaLagMeasurer lagMeasurer;

    @DynamicPropertySource
    static void kafkaProperties(DynamicPropertyRegistry registry) {
        registry.add("spring.kafka.bootstrap-servers", kafka::getBootstrapServers);
        registry.add("spring.kafka.consumer.auto-offset-reset", () -> "earliest");
        registry.add("spring.kafka.consumer.enable-auto-commit", () -> "false");
    }

    @BeforeEach
    void setUpLagMeasurer() {
        lagMeasurer = new KafkaLagMeasurer(kafka.getBootstrapServers());
    }

    @AfterEach
    void tearDownLagMeasurer() {
        lagMeasurer.close();
    }
}

Test: Consumer Processes All Messages (Zero Lag)

The most basic lag test: produce N messages, wait for them to be consumed, verify lag is zero:

@Test
void shouldConsumeAllMessagesWithZeroLag() throws Exception {
    String topic = "orders";
    String groupId = "order-processor";
    int messageCount = 100;

    // Produce messages
    for (int i = 0; i < messageCount; i++) {
        kafkaTemplate.send(topic, "KEY-" + i, "ORDER-" + i).get();
    }

    // Wait for consumer to process all messages
    Awaitility.await()
        .atMost(Duration.ofSeconds(30))
        .pollInterval(Duration.ofMillis(500))
        .untilAsserted(() -> {
            long lag = lagMeasurer.getTotalLag(groupId, topic);
            assertThat(lag).isZero();
        });
}

Using Awaitility here is important — you are polling an async condition. Sleeping for a fixed duration is fragile; Awaitility retries until the assertion passes or times out.

Test: Lag Stays Below Maximum Threshold Under Load

This test validates throughput: if you publish 1000 messages in a burst, the consumer should catch up within an acceptable time window without the lag ever exceeding your SLA:

@Test
void shouldNotExceedMaxLagUnderBurstLoad() throws Exception {
    String topic = "events";
    String groupId = "event-processor";
    int burst = 1000;
    long maxAllowedLag = 100; // Maximum acceptable outstanding messages

    // Produce burst
    List<CompletableFuture<?>> futures = new ArrayList<>();
    for (int i = 0; i < burst; i++) {
        futures.add(kafkaTemplate.send(topic, "KEY-" + i, "EVENT-" + i)
            .toCompletableFuture());
    }
    CompletableFuture.allOf(futures.toArray(new CompletableFuture[0])).get(30, TimeUnit.SECONDS);

    // Poll lag and assert it never exceeds threshold after a brief warmup
    Thread.sleep(2000); // Allow consumer to start processing

    Awaitility.await()
        .atMost(Duration.ofSeconds(60))
        .pollInterval(Duration.ofSeconds(1))
        .untilAsserted(() -> {
            long lag = lagMeasurer.getTotalLag(groupId, topic);
            assertThat(lag).isZero();
        });

    // Additionally verify the peak lag during processing was captured and logged
    // (implement with a separate polling thread if strict SLA enforcement is needed)
}

Test: Consumer Recovers After Restart

This test simulates the common scenario where a consumer restarts (pod restart, deployment) and verifies it resumes from where it left off — not from the beginning, and not missing messages:

@Test
void shouldResumeFromLastCommittedOffsetAfterRestart() throws Exception {
    String topic = "resumable-events";
    String groupId = "resumable-group";

    // Step 1: Produce 50 messages and consume 30 (partial consumption)
    produceMessages(topic, 50);
    
    // Let consumer process exactly 30 messages
    CountDownLatch partialLatch = new CountDownLatch(30);
    startCountingConsumer(topic, groupId, partialLatch);
    assertThat(partialLatch.await(20, TimeUnit.SECONDS)).isTrue();

    // Verify 20 messages of lag remain
    long lagBeforeRestart = lagMeasurer.getTotalLag(groupId, topic);
    assertThat(lagBeforeRestart).isEqualTo(20);

    // Step 2: Stop the consumer (simulate restart)
    stopConsumer(groupId);

    // Step 3: Restart consumer and verify it picks up remaining 20 messages
    CountDownLatch resumeLatch = new CountDownLatch(20);
    startCountingConsumer(topic, groupId, resumeLatch);
    assertThat(resumeLatch.await(20, TimeUnit.SECONDS)).isTrue();

    // Step 4: Verify lag is now zero — consumer resumed from committed offset
    Awaitility.await()
        .atMost(Duration.ofSeconds(10))
        .untilAsserted(() -> 
            assertThat(lagMeasurer.getTotalLag(groupId, topic)).isZero()
        );
}

Test: Lag Per Partition Is Balanced

For topics with multiple partitions, you want to verify that the consumer group's partitions are balanced — no single partition accumulating disproportionate lag:

@Test
void shouldHaveBalancedLagAcrossPartitions() throws Exception {
    String topic = "partitioned-orders";
    String groupId = "balanced-consumer";

    // Create topic with 6 partitions
    createTopicWithPartitions(topic, 6);

    // Produce messages evenly across partitions
    produceMessages(topic, 600);

    // Wait for initial consumption
    Thread.sleep(3000);

    // Check per-partition lag
    Map<TopicPartition, Long> lagPerPartition = 
        lagMeasurer.getLagPerPartition(groupId, topic);

    if (!lagPerPartition.isEmpty()) {
        long maxPartitionLag = lagPerPartition.values().stream()
            .mapToLong(Long::longValue).max().orElse(0);
        long minPartitionLag = lagPerPartition.values().stream()
            .mapToLong(Long::longValue).min().orElse(0);
        
        // Max lag on any single partition should not be more than 2x the minimum
        // This catches partition assignment imbalance
        if (minPartitionLag > 0) {
            assertThat((double) maxPartitionLag / minPartitionLag)
                .isLessThanOrEqualTo(2.0);
        }
    }
}

Monitoring Lag in Tests with a Background Thread

For tests that need to capture peak lag during processing, use a background sampling thread:

public class LagMonitor {
    
    private final KafkaLagMeasurer lagMeasurer;
    private final String groupId;
    private final String topic;
    private final List<Long> samples = new CopyOnWriteArrayList<>();
    private volatile boolean running = true;

    public LagMonitor(KafkaLagMeasurer lagMeasurer, String groupId, String topic) {
        this.lagMeasurer = lagMeasurer;
        this.groupId = groupId;
        this.topic = topic;
    }

    public void start() {
        Thread sampler = new Thread(() -> {
            while (running) {
                try {
                    samples.add(lagMeasurer.getTotalLag(groupId, topic));
                    Thread.sleep(500);
                } catch (InterruptedException e) {
                    Thread.currentThread().interrupt();
                    break;
                } catch (Exception e) {
                    // Ignore transient failures
                }
            }
        });
        sampler.setDaemon(true);
        sampler.start();
    }

    public void stop() {
        running = false;
    }

    public long getPeakLag() {
        return samples.stream().mapToLong(Long::longValue).max().orElse(0);
    }

    public double getAverageLag() {
        return samples.stream().mapToLong(Long::longValue).average().orElse(0);
    }
}

Using it in a test:

@Test
void shouldKeepPeakLagUnder200DuringBurstOf500Messages() throws Exception {
    LagMonitor monitor = new LagMonitor(lagMeasurer, "burst-group", "burst-topic");
    monitor.start();

    produceMessages("burst-topic", 500);
    
    Awaitility.await()
        .atMost(Duration.ofSeconds(60))
        .until(() -> lagMeasurer.getTotalLag("burst-group", "burst-topic") == 0);

    monitor.stop();

    assertThat(monitor.getPeakLag()).isLessThanOrEqualTo(200);
    System.out.printf("Peak lag: %d, Average lag: %.1f%n",
        monitor.getPeakLag(), monitor.getAverageLag());
}

Using Micrometer Metrics for Lag in Tests

If your application exposes Micrometer metrics, you can assert on kafka.consumer.fetch-latency-avg and kafka.consumer.records-lag-max in tests:

@Autowired
private MeterRegistry meterRegistry;

@Test
void shouldExposeLagMetric() throws Exception {
    produceAndConsumeMessages("metrics-topic", "metrics-group", 10);

    // Verify lag metric is registered and has a value
    Gauge lagGauge = meterRegistry.find("kafka.consumer.records-lag-max")
        .tag("topic", "metrics-topic")
        .gauge();

    assertThat(lagGauge).isNotNull();
    // After full consumption, max lag should be 0
    assertThat(lagGauge.value()).isLessThanOrEqualTo(0.0);
}

Key Takeaways

Consumer group lag testing is an under-appreciated area of Kafka integration testing. The patterns covered here give you:

  • Zero-lag assertions — the most common test, verifying all messages are eventually consumed
  • Threshold assertions — ensuring lag stays within acceptable bounds under load
  • Restart recovery tests — confirming offset commits are correct and consumers resume properly
  • Partition balance checks — catching skewed assignment before it causes uneven throughput

The AdminClient API is the right tool for all of these. It gives you committed offsets and log-end offsets with a few API calls, and the computation is straightforward subtraction. Combined with Awaitility for async polling, you have everything needed to add lag assertions to your existing Kafka integration test suite today.

Read more

Start now free