Testing Kafka Consumers and Producers with Testcontainers

Testing Kafka Consumers and Producers with Testcontainers

Kafka is notoriously hard to test well. A shared dev Kafka cluster means tests interfere with each other. Mocking the Kafka client means you never test your serialisation logic, topic configuration, or consumer group behaviour. Testcontainers' KafkaContainer solves this: a real Kafka broker, per test run, with no shared state.

Why Not Mock Kafka?

When you mock KafkaProducer and KafkaConsumer, you are testing your code's interaction with an interface — not with Kafka itself. The things that break in production are rarely "did I call producer.send()?". They are:

  • Wrong serialiser producing malformed bytes
  • Consumer group offsets not committing correctly
  • Topic compaction eating messages your consumer expected
  • Schema Registry rejecting a message your producer assumed was valid
  • Partition assignment behaving differently than you expected

A real broker catches all of these. A mock catches none of them.

Java: KafkaContainer with the Confluent Image

Add the Testcontainers Kafka module to your pom.xml:

<dependency>
    <groupId>org.testcontainers</groupId>
    <artifactId>kafka</artifactId>
    <version>1.19.8</version>
    <scope>test</scope>
</dependency>

Start a broker in your test:

import org.testcontainers.containers.KafkaContainer;
import org.testcontainers.utility.DockerImageName;

@Testcontainers
class OrderEventConsumerTest {

    @Container
    static KafkaContainer kafka = new KafkaContainer(
        DockerImageName.parse("confluentinc/cp-kafka:7.6.1")
    );

    private KafkaProducer<String, String> producer;
    private KafkaConsumer<String, String> consumer;

    @BeforeEach
    void setUp() {
        Map<String, Object> producerConfig = Map.of(
            ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, kafka.getBootstrapServers(),
            ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class,
            ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class
        );
        producer = new KafkaProducer<>(producerConfig);

        Map<String, Object> consumerConfig = Map.of(
            ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, kafka.getBootstrapServers(),
            ConsumerConfig.GROUP_ID_CONFIG, "test-group",
            ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest",
            ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class,
            ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class
        );
        consumer = new KafkaConsumer<>(consumerConfig);
    }

    @AfterEach
    void tearDown() {
        producer.close();
        consumer.close();
    }

Verifying Message Processing End-to-End

The real test is not "did a message appear on the topic?" but "did my consumer process it correctly?":

    @Test
    void consumerProcessesOrderPlacedEvent() throws Exception {
        String topic = "order-placed";
        consumer.subscribe(List.of(topic));

        // Publish an event
        var record = new ProducerRecord<>(topic, "order-1",
            """
            {"orderId":"order-1","customerId":"cust-42","total":99.99}
            """.strip()
        );
        producer.send(record).get(); // get() ensures the message is committed

        // Poll until we receive it or timeout
        ConsumerRecords<String, String> records = ConsumerRecords.empty();
        long deadline = System.currentTimeMillis() + 10_000;

        while (records.isEmpty() && System.currentTimeMillis() < deadline) {
            records = consumer.poll(Duration.ofMillis(500));
        }

        assertFalse(records.isEmpty(), "Expected at least one record");

        var received = records.iterator().next();
        assertEquals("order-1", received.key());

        var payload = new ObjectMapper().readTree(received.value());
        assertEquals("cust-42", payload.get("customerId").asText());
    }

For testing your own consumer service (not just raw Kafka consumers), wire the getBootstrapServers() value into your service under test before starting it.

Spring Boot: Using @SpringBootTest with KafkaContainer

Spring Boot's Kafka auto-configuration picks up spring.kafka.bootstrap-servers. Override it with a dynamic property:

@SpringBootTest
@Testcontainers
class OrderServiceIntegrationTest {

    @Container
    static KafkaContainer kafka = new KafkaContainer(
        DockerImageName.parse("confluentinc/cp-kafka:7.6.1")
    );

    @DynamicPropertySource
    static void kafkaProperties(DynamicPropertyRegistry registry) {
        registry.add("spring.kafka.bootstrap-servers", kafka::getBootstrapServers);
    }

    @Autowired
    private OrderService orderService;

    @Autowired
    private KafkaConsumer<String, String> testConsumer;

    @Test
    void placeOrder_PublishesEventToKafka() throws Exception {
        orderService.placeOrder(new Order("cust-1", List.of("item-A")));

        // Poll topic and verify
        testConsumer.subscribe(List.of("order-placed"));
        var records = testConsumer.poll(Duration.ofSeconds(5));
        assertFalse(records.isEmpty());
    }
}

The @DynamicPropertySource injection happens before the application context starts, so KafkaTemplate and @KafkaListener beans are all configured against the container.

Python: Testing with kafka-python

import json
import time
import pytest
from kafka import KafkaProducer, KafkaConsumer
from testcontainers.kafka import KafkaContainer

@pytest.fixture(scope="module")
def kafka_container():
    with KafkaContainer("confluentinc/cp-kafka:7.6.1") as kafka:
        yield kafka

def test_message_round_trip(kafka_container):
    bootstrap = kafka_container.get_bootstrap_server()
    topic = "payments"

    producer = KafkaProducer(
        bootstrap_servers=bootstrap,
        value_serializer=lambda v: json.dumps(v).encode("utf-8"),
    )
    consumer = KafkaConsumer(
        topic,
        bootstrap_servers=bootstrap,
        auto_offset_reset="earliest",
        group_id="test-group",
        value_deserializer=lambda m: json.loads(m.decode("utf-8")),
        consumer_timeout_ms=10000,
    )

    event = {"paymentId": "pay-1", "amount": 49.95, "currency": "USD"}
    producer.send(topic, value=event)
    producer.flush()

    received = []
    for msg in consumer:
        received.append(msg.value)
        break  # stop after first message

    assert len(received) == 1
    assert received[0]["paymentId"] == "pay-1"
    assert received[0]["amount"] == pytest.approx(49.95)

Node.js: KafkaJS with Testcontainers

const { KafkaContainer } = require("@testcontainers/kafka");
const { Kafka } = require("kafkajs");

describe("notification consumer", () => {
  let container;
  let kafka;

  beforeAll(async () => {
    container = await new KafkaContainer("confluentinc/cp-kafka:7.6.1").start();

    kafka = new Kafka({
      brokers: [`${container.getHost()}:${container.getMappedPort(9093)}`],
    });
  }, 60_000);

  afterAll(async () => {
    await container.stop();
  });

  it("processes notification events", async () => {
    const topic = "notifications";
    const admin = kafka.admin();
    await admin.connect();
    await admin.createTopics({ topics: [{ topic, numPartitions: 1 }] });
    await admin.disconnect();

    const producer = kafka.producer();
    await producer.connect();
    await producer.send({
      topic,
      messages: [{ key: "user-1", value: JSON.stringify({ type: "welcome", userId: "user-1" }) }],
    });
    await producer.disconnect();

    const received = [];
    const consumer = kafka.consumer({ groupId: "test-group" });
    await consumer.connect();
    await consumer.subscribe({ topic, fromBeginning: true });

    await new Promise((resolve) => {
      consumer.run({
        eachMessage: async ({ message }) => {
          received.push(JSON.parse(message.value.toString()));
          resolve();
        },
      });
    });

    await consumer.disconnect();

    expect(received).toHaveLength(1);
    expect(received[0].type).toBe("welcome");
  }, 30_000);
});

Tips for Reliable Kafka Tests

Use earliest offset reset. Your producer publishes before the consumer subscribes. Without auto.offset.reset=earliest (or fromBeginning: true in KafkaJS), the consumer will miss messages published before it joined.

Await producer flush. Call producer.flush() or use the Java .get() on the Future to ensure the broker has acknowledged the message before you poll.

Use unique topic names. When tests run in parallel, prefix topics with a random UUID or the test name to avoid cross-test interference within the same container.

Increase timeouts in CI. Kafka startup on a cold CI runner takes longer than on your laptop. Add 20–30 seconds of buffer to your wait timeouts.

Closing Thoughts

KafkaContainer makes Kafka tests reproducible and honest. Your serialisers, consumer groups, and offset logic are all exercised against a real broker — not a mock that pretends to be one.

To keep your Kafka-powered services healthy in production, pair your test suite with HelpMeTest. HelpMeTest monitors your services 24/7, so you know the moment a consumer stops processing — before your users do.

Read more

Start now free