Testing Kafka Consumers and Producers with Testcontainers

Testing Kafka Consumers and Producers with Testcontainers

Testcontainers lets you run a real Kafka broker inside your test suite — no mocks, no stubs, no "it works on my machine" surprises. This guide walks through spinning up Kafka in Java and Python tests, writing producer and consumer assertions, controlling offsets, and wiring everything into CI.

Key Takeaways

Use real Kafka, not mocks. EmbeddedKafka and mock producers hide serialization bugs, rebalance behavior, and partition assignment issues that only show up with a real broker.

Share one Kafka container per test class. Starting a new container per test takes 10–15 seconds each. A @Container with @TestContainers on the class level starts once and saves minutes on large suites.

Always set auto.offset.reset=earliest in tests. Without it, consumers miss messages produced before they subscribed — a common source of flaky tests.

Assert on consumed records, not on side effects. Poll in a loop with a deadline rather than sleeping a fixed duration. Tests become faster and deterministic.

Pin the Kafka image version in CI. Floating latest tags break pipelines when Confluent ships a new image with changed defaults.

Testing event-driven systems is hard. The problem is not writing the code — it is trusting it. A producer that "looks like it sends" and a consumer that "looks like it reads" can still fail silently in production because the serialization format drifted, the topic was misconfigured, or an offset reset swallowed half the messages. The only way to catch those failures before production is to test against a real Kafka broker.

Testcontainers makes this practical. It manages Docker containers as JUnit or pytest fixtures, starting them before your test and stopping them after. You get a real broker, real partitions, real consumer groups — and the test still runs on a laptop or in CI without any pre-installed infrastructure.

Why Not EmbeddedKafka?

Spring Kafka ships EmbeddedKafkaBroker. The Apache Kafka project has KafkaEmbedded. These are tempting because they start fast and require no Docker. The trade-off is fidelity.

Embedded brokers run in the same JVM as your test. They share the same classloader, skip certain network-layer checks, and sometimes paper over errors that a real broker would surface. Serialization bugs, schema registry interactions, and consumer group rebalancing all behave differently. A test that passes against an embedded broker has failed before in production for exactly this reason.

Testcontainers adds 5–15 seconds of startup time. In exchange, you get a broker that behaves identically to production.

Java Setup

Add the dependencies to pom.xml:

<dependency>
  <groupId>org.testcontainers</groupId>
  <artifactId>kafka</artifactId>
  <version>1.19.7</version>
  <scope>test</scope>
</dependency>
<dependency>
  <groupId>org.testcontainers</groupId>
  <artifactId>junit-jupiter</artifactId>
  <version>1.19.7</version>
  <scope>test</scope>
</dependency>
<dependency>
  <groupId>org.apache.kafka</groupId>
  <artifactId>kafka-clients</artifactId>
  <version>3.7.0</version>
</dependency>

A minimal test class that starts one Kafka container for all tests in the class:

@Testcontainers
class OrderEventProducerTest {

    @Container
    static final KafkaContainer kafka =
        new KafkaContainer(DockerImageName.parse("confluentinc/cp-kafka:7.6.1"));

    private KafkaProducer<String, String> producer;
    private KafkaConsumer<String, String> consumer;

    @BeforeEach
    void setUp() {
        producer = new KafkaProducer<>(Map.of(
            ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, kafka.getBootstrapServers(),
            ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName(),
            ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName()
        ));

        consumer = new KafkaConsumer<>(Map.of(
            ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, kafka.getBootstrapServers(),
            ConsumerConfig.GROUP_ID_CONFIG, "test-group-" + UUID.randomUUID(),
            ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName(),
            ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName(),
            ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest"
        ));
    }

    @AfterEach
    void tearDown() {
        producer.close();
        consumer.close();
    }
}

Two things to note here. The container is static — it starts once when the class loads and is reused across all tests. Using a random UUID for GROUP_ID_CONFIG ensures each test gets a fresh consumer group with no offset history.

Writing a Producer Test

Testing a producer means verifying that a message arrives on the correct topic with the expected key and value. The simplest approach: produce a message, consume it from the same broker, assert on what came back.

@Test
void producesOrderCreatedEvent() throws Exception {
    String topic = "orders";
    String orderId = "order-123";
    String payload = "{\"orderId\":\"order-123\",\"status\":\"CREATED\"}";

    // Produce
    producer.send(new ProducerRecord<>(topic, orderId, payload)).get();

    // Consume
    consumer.subscribe(List.of(topic));
    ConsumerRecords<String, String> records = pollWithTimeout(consumer, Duration.ofSeconds(10));

    assertThat(records.count()).isEqualTo(1);
    ConsumerRecord<String, String> record = records.iterator().next();
    assertThat(record.key()).isEqualTo(orderId);
    assertThat(record.value()).isEqualTo(payload);
}

private ConsumerRecords<String, String> pollWithTimeout(
    KafkaConsumer<String, String> consumer,
    Duration timeout
) {
    Instant deadline = Instant.now().plus(timeout);
    ConsumerRecords<String, String> records = ConsumerRecords.empty();
    while (records.isEmpty() && Instant.now().isBefore(deadline)) {
        records = consumer.poll(Duration.ofMillis(500));
    }
    return records;
}

The pollWithTimeout helper is important. Never use Thread.sleep before polling. Sleep-based tests are fragile — they fail when the machine is slow and waste time when delivery is fast. The loop polls every 500ms and returns as soon as records arrive.

Writing a Consumer Test

Testing a consumer means verifying that your application logic handles an incoming message correctly. The typical pattern: produce a message onto the topic, let your consumer application read it, assert on the observable side effect.

Suppose you have a OrderConsumer that persists orders to a database:

@Test
void consumerPersistsOrderOnMessage() throws Exception {
    String topic = "orders";
    String payload = "{\"orderId\":\"order-456\",\"amount\":99.99}";

    // Seed the topic
    producer.send(new ProducerRecord<>(topic, "order-456", payload)).get();

    // Start the consumer under test
    OrderConsumer orderConsumer = new OrderConsumer(kafka.getBootstrapServers(), orderRepository);
    orderConsumer.start();

    // Assert side effect
    await().atMost(10, SECONDS).untilAsserted(() ->
        assertThat(orderRepository.findById("order-456")).isPresent()
    );

    orderConsumer.stop();
}

This uses Awaitility (await().atMost(...)) for the assertion loop. Awaitility polls the assertion on a configurable schedule until it passes or times out — cleaner than writing the loop yourself.

Testing Offset Behavior

Offset handling is where consumer bugs hide. Common failure modes: the consumer starts from the wrong offset after a restart, or processes messages twice due to a missing commit.

Test the restart scenario explicitly:

@Test
void consumerResumesFromLastCommittedOffset() throws Exception {
    String topic = "payments";
    String groupId = "payment-processor";

    // Produce 3 messages
    for (int i = 0; i < 3; i++) {
        producer.send(new ProducerRecord<>(topic, "payment-" + i,
            "{\"id\":" + i + "}")).get();
    }

    // First consumer run — processes and commits after first 2 messages
    try (KafkaConsumer<String, String> c1 = buildConsumer(groupId)) {
        c1.subscribe(List.of(topic));
        List<ConsumerRecord<String, String>> seen = new ArrayList<>();
        Instant deadline = Instant.now().plusSeconds(10);
        while (seen.size() < 2 && Instant.now().isBefore(deadline)) {
            c1.poll(Duration.ofMillis(500)).forEach(seen::add);
        }
        // Commit after processing 2
        c1.commitSync();
    }

    // Second consumer run — should pick up only the 3rd message
    List<String> resumedKeys = new ArrayList<>();
    try (KafkaConsumer<String, String> c2 = buildConsumer(groupId)) {
        c2.subscribe(List.of(topic));
        Instant deadline = Instant.now().plusSeconds(5);
        while (Instant.now().isBefore(deadline)) {
            c2.poll(Duration.ofMillis(500)).forEach(r -> resumedKeys.add(r.key()));
        }
    }

    assertThat(resumedKeys).containsExactly("payment-2");
}

This test catches bugs where enable.auto.commit=true silently commits before the message is processed, or where the consumer resets to earliest on every restart.

Python Setup

For Python services using confluent-kafka, Testcontainers Python provides the same container management:

# requirements-test.txt
testcontainers[kafka]==3.7.6
confluent-kafka==2.3.0
pytest==8.1.0
import pytest
import json
from testcontainers.kafka import KafkaContainer
from confluent_kafka import Producer, Consumer, KafkaError

@pytest.fixture(scope="module")
def kafka():
    with KafkaContainer("confluentinc/cp-kafka:7.6.1") as container:
        yield container

def make_producer(bootstrap_servers: str) -> Producer:
    return Producer({"bootstrap.servers": bootstrap_servers})

def make_consumer(bootstrap_servers: str, group_id: str) -> Consumer:
    return Consumer({
        "bootstrap.servers": bootstrap_servers,
        "group.id": group_id,
        "auto.offset.reset": "earliest",
        "enable.auto.commit": False,
    })

The scope="module" on the fixture means one Kafka container per test module — the same startup-cost optimization as static in Java.

A producer test in Python:

def test_produces_user_event(kafka):
    bootstrap = kafka.get_bootstrap_server()
    topic = "user-events"
    event = {"userId": "u-1", "action": "login"}

    producer = make_producer(bootstrap)
    producer.produce(topic, key="u-1", value=json.dumps(event))
    producer.flush()

    consumer = make_consumer(bootstrap, group_id="test-consumer")
    consumer.subscribe([topic])

    received = None
    from datetime import datetime, timedelta
    deadline = datetime.utcnow() + timedelta(seconds=10)
    while datetime.utcnow() < deadline:
        msg = consumer.poll(0.5)
        if msg is None:
            continue
        if msg.error():
            if msg.error().code() == KafkaError._PARTITION_EOF:
                continue
            raise Exception(msg.error())
        received = json.loads(msg.value())
        break

    consumer.close()

    assert received is not None
    assert received["userId"] == "u-1"
    assert received["action"] == "login"

CI Configuration

The main concern in CI is Docker availability. GitHub Actions, GitLab CI, and most hosted runners support Docker out of the box.

A GitHub Actions workflow:

name: Kafka Tests
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-java@v4
        with:
          java-version: '21'
          distribution: 'temurin'
      - name: Cache Maven
        uses: actions/cache@v4
        with:
          path: ~/.m2
          key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
      - name: Pull Kafka image
        run: docker pull confluentinc/cp-kafka:7.6.1
      - name: Run tests
        run: mvn test
        env:
          TESTCONTAINERS_RYUK_DISABLED: false

The docker pull step before running tests warms the image cache. Without it, the first test run in a fresh CI environment pays the full image download cost inside the test timeout. With the pull step, the download happens outside your test timer and is cached across runs.

Set TESTCONTAINERS_RYUK_DISABLED=false (the default) unless your CI explicitly forbids privileged containers. Ryuk is the Testcontainers cleanup daemon — it ensures containers are removed even if the JVM crashes mid-test.

For GitLab CI, add the Docker-in-Docker service:

kafka-tests:
  image: maven:3.9-eclipse-temurin-21
  services:
    - docker:dind
  variables:
    DOCKER_HOST: tcp://docker:2375
    DOCKER_TLS_CERTDIR: ""
    TESTCONTAINERS_HOST_OVERRIDE: docker
  script:
    - docker pull confluentinc/cp-kafka:7.6.1
    - mvn test

The TESTCONTAINERS_HOST_OVERRIDE variable tells Testcontainers to connect to the DinD service instead of the local Docker socket.

Common Pitfalls

Topic auto-creation. By default, Kafka auto-creates topics when a producer writes to them. This is fine for tests, but if your production setup has auto-creation disabled, your tests are testing a different configuration. Create topics explicitly in @BeforeEach:

try (AdminClient admin = AdminClient.create(Map.of(
    AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, kafka.getBootstrapServers()
))) {
    admin.createTopics(List.of(
        new NewTopic("orders", 3, (short) 1)
    )).all().get();
}

Partition assignment lag. After consumer.subscribe(), Kafka needs to assign partitions before the consumer can poll messages. If you produce a message before assignment completes, and auto.offset.reset=latest, the consumer misses it. Setting auto.offset.reset=earliest avoids this in tests.

Multiple tests, same topic name. If two tests use the same topic and run concurrently, they contaminate each other's messages. Either use unique topic names per test (e.g., append a UUID) or configure test parallelism to run Kafka tests sequentially.

Slow cleanup. The @Container lifecycle calls container.stop() after tests. On some CI environments, container stop is slow and delays pipeline feedback. Use withStartupTimeout and withShutdownTimeout to tune these:

static final KafkaContainer kafka = new KafkaContainer(
    DockerImageName.parse("confluentinc/cp-kafka:7.6.1"))
    .withStartupTimeout(Duration.ofSeconds(90));

Summary

Testcontainers gives you a real Kafka broker in tests with minimal setup. The key patterns: one container per class or module for startup efficiency, auto.offset.reset=earliest to avoid timing races, poll loops with deadlines instead of sleep, and explicit topic creation when partition count matters. With this foundation you can test serialization, offset commits, rebalancing, and schema evolution with confidence that what passes locally will also pass in production.

Read more

Start now free