Testing Kafka Producers and Consumers with Testcontainers

Testing Kafka Producers and Consumers with Testcontainers

Kafka is the backbone of event-driven architectures, but testing it has historically been painful. Embedded Kafka brokers drift from real broker behavior, shared test environments create flaky tests, and mocking away the broker entirely means you are not testing anything meaningful. Testcontainers solves this by spinning up a real Kafka broker inside Docker for every test run, giving you production-fidelity without the shared-environment headaches.

This guide walks through everything you need to write fast, reliable integration tests for Kafka producers and consumers using Testcontainers with Spring Boot.

Why Testcontainers Over Embedded Kafka

Before diving into code, it is worth understanding what you gain by switching from @EmbeddedKafka to Testcontainers.

Embedded Kafka runs an in-process broker. It is fast to start, but it uses an older version of Kafka and diverges from real broker behavior in subtle ways — topic auto-creation semantics, log compaction, replication, and ACL enforcement all differ. When your production cluster runs Kafka 3.6 and your tests use embedded Kafka 2.x, you are building false confidence.

Testcontainers runs the exact same Docker image your production cluster uses. You can pin to confluentinc/cp-kafka:7.6.0 and know your tests exercise the actual broker behavior.

Project Setup

Add the following dependencies to your build.gradle:

dependencies {
    implementation 'org.springframework.boot:spring-boot-starter'
    implementation 'org.springframework.kafka:spring-kafka'
    
    testImplementation 'org.springframework.boot:spring-boot-starter-test'
    testImplementation 'org.springframework.kafka:spring-kafka-test'
    testImplementation 'org.testcontainers:kafka:1.19.7'
    testImplementation 'org.testcontainers:junit-jupiter:1.19.7'
}

Or with Maven:

<dependency>
    <groupId>org.testcontainers</groupId>
    <artifactId>kafka</artifactId>
    <version>1.19.7</version>
    <scope>test</scope>
</dependency>

Creating a Reusable Kafka Test Base

The most important decision is how to manage the container lifecycle. Starting a new Kafka container for every test class is slow. The reuse feature in Testcontainers lets you share a single container across the entire test suite.

@SpringBootTest
@Testcontainers
public abstract class KafkaIntegrationTestBase {

    @Container
    static final KafkaContainer kafka = new KafkaContainer(
            DockerImageName.parse("confluentinc/cp-kafka:7.6.0"))
        .withReuse(true);

    @DynamicPropertySource
    static void kafkaProperties(DynamicPropertyRegistry registry) {
        registry.add("spring.kafka.bootstrap-servers", kafka::getBootstrapServers);
        registry.add("spring.kafka.consumer.auto-offset-reset", () -> "earliest");
        registry.add("spring.kafka.consumer.group-id", () -> "test-group");
    }
}

The @DynamicPropertySource hook injects the dynamically assigned bootstrap server address into your Spring context before it starts. This is the critical piece — without it, your application context will try to connect to localhost:9092 which will not exist.

Writing a Producer

Here is a simple order event producer:

@Service
public class OrderEventProducer {

    private static final String TOPIC = "orders";
    private final KafkaTemplate<String, OrderEvent> kafkaTemplate;

    public OrderEventProducer(KafkaTemplate<String, OrderEvent> kafkaTemplate) {
        this.kafkaTemplate = kafkaTemplate;
    }

    public CompletableFuture<SendResult<String, OrderEvent>> sendOrder(OrderEvent event) {
        return kafkaTemplate.send(TOPIC, event.orderId(), event)
            .toCompletableFuture();
    }
}
public record OrderEvent(String orderId, String customerId, BigDecimal amount, String status) {}

Writing a Consumer

@Service
public class OrderEventConsumer {

    private final List<OrderEvent> receivedEvents = new CopyOnWriteArrayList<>();
    private final CountDownLatch latch = new CountDownLatch(1);

    @KafkaListener(topics = "orders", groupId = "order-processor")
    public void consume(OrderEvent event) {
        receivedEvents.add(event);
        latch.countDown();
    }

    public List<OrderEvent> getReceivedEvents() {
        return Collections.unmodifiableList(receivedEvents);
    }

    public CountDownLatch getLatch() {
        return latch;
    }
}

Testing the Producer

class OrderEventProducerTest extends KafkaIntegrationTestBase {

    @Autowired
    private OrderEventProducer producer;

    @Test
    void shouldSendOrderEventToKafka() throws Exception {
        // Arrange
        OrderEvent event = new OrderEvent("ORD-001", "CUST-123", 
            new BigDecimal("99.99"), "CREATED");

        // Act
        SendResult<String, OrderEvent> result = producer.sendOrder(event)
            .get(10, TimeUnit.SECONDS);

        // Assert
        assertThat(result.getRecordMetadata().topic()).isEqualTo("orders");
        assertThat(result.getRecordMetadata().partition()).isGreaterThanOrEqualTo(0);
        assertThat(result.getProducerRecord().key()).isEqualTo("ORD-001");
    }

    @Test
    void shouldSendToCorrectPartitionForOrderKey() throws Exception {
        // Send multiple events with same key — should land on same partition
        OrderEvent event1 = new OrderEvent("ORD-100", "CUST-1", 
            new BigDecimal("10.00"), "CREATED");
        OrderEvent event2 = new OrderEvent("ORD-100", "CUST-1", 
            new BigDecimal("10.00"), "UPDATED");

        SendResult<?, ?> r1 = producer.sendOrder(event1).get(10, TimeUnit.SECONDS);
        SendResult<?, ?> r2 = producer.sendOrder(event2).get(10, TimeUnit.SECONDS);

        assertThat(r1.getRecordMetadata().partition())
            .isEqualTo(r2.getRecordMetadata().partition());
    }
}

Testing the Consumer End-to-End

For consumer tests, you want to produce a message and verify the consumer processes it correctly. The tricky part is synchronization — Kafka consumption is asynchronous.

class OrderEventConsumerTest extends KafkaIntegrationTestBase {

    @Autowired
    private KafkaTemplate<String, OrderEvent> kafkaTemplate;

    @Autowired
    private OrderEventConsumer consumer;

    @Test
    void shouldConsumeOrderEventFromKafka() throws Exception {
        // Arrange
        OrderEvent event = new OrderEvent("ORD-002", "CUST-456",
            new BigDecimal("149.99"), "SHIPPED");

        // Act
        kafkaTemplate.send("orders", event.orderId(), event).get();

        // Wait for consumer to process
        boolean consumed = consumer.getLatch().await(15, TimeUnit.SECONDS);

        // Assert
        assertThat(consumed).isTrue();
        assertThat(consumer.getReceivedEvents()).hasSize(1);
        assertThat(consumer.getReceivedEvents().get(0).orderId()).isEqualTo("ORD-002");
        assertThat(consumer.getReceivedEvents().get(0).status()).isEqualTo("SHIPPED");
    }
}

Testing Consumer Error Handling

Production consumers need robust error handling. Here is how to test dead letter topic behavior:

@Test
void shouldSendToDeadLetterTopicOnProcessingFailure() throws Exception {
    // Arrange — send a malformed event that triggers processing error
    String badPayload = "{\"invalid\": true}";
    ProducerRecord<String, String> record = 
        new ProducerRecord<>("orders", "BAD-KEY", badPayload);

    // Use raw string template to bypass serialization
    KafkaTemplate<String, String> rawTemplate = buildRawStringTemplate();
    rawTemplate.send(record).get();

    // Assert the DLT receives the message
    ConsumerRecord<String, String> dlqRecord = 
        KafkaTestUtils.getSingleRecord(dltConsumer, "orders.DLT", 
            Duration.ofSeconds(15));

    assertThat(dlqRecord).isNotNull();
    assertThat(dlqRecord.key()).isEqualTo("BAD-KEY");
}

Using KafkaTestUtils for Verification

Spring Kafka ships KafkaTestUtils, which provides convenient methods for asserting broker state:

@Test
void shouldPublishExactlyOneRecord() throws Exception {
    // Create a consumer to read from the topic
    Map<String, Object> consumerProps = KafkaTestUtils.consumerProps(
        kafka.getBootstrapServers(), "verify-group", "true");
    
    Consumer<String, String> consumer = new DefaultKafkaConsumerFactory<String, String>(consumerProps)
        .createConsumer();
    consumer.subscribe(Collections.singletonList("orders"));

    // Produce
    producer.sendOrder(new OrderEvent("ORD-003", "CUST-789",
        new BigDecimal("299.99"), "CREATED")).get();

    // Verify
    ConsumerRecords<String, String> records = 
        KafkaTestUtils.getRecords(consumer, Duration.ofSeconds(10));

    assertThat(records.count()).isEqualTo(1);
    consumer.close();
}

Testing Topic Configuration

Often you need to verify topics are created with the right configuration — correct replication factor, compaction policy, or retention settings:

@Test
void shouldCreateOrdersTopicWithCorrectConfig() throws Exception {
    AdminClient admin = AdminClient.create(Map.of(
        AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, kafka.getBootstrapServers()
    ));

    DescribeTopicsResult result = admin.describeTopics(List.of("orders"));
    TopicDescription description = result.allTopicNames().get()
        .get("orders");

    assertThat(description.partitions()).hasSize(3);
    assertThat(description.partitions().get(0).replicas()).hasSize(1);
    admin.close();
}

Handling Slow CI Environments

On CI, Docker startup can be slower than locally. A few patterns help:

Set longer timeouts in test configuration:

# application-test.yaml
spring:
  kafka:
    consumer:
      auto-offset-reset: earliest
    listener:
      ack-mode: record
      
# Increase poll timeout for slow CI
kafka:
  consumer:
    poll-timeout: 5000

Use @BeforeAll to pre-warm topics:

@BeforeAll
static void createTopics() throws Exception {
    AdminClient admin = AdminClient.create(Map.of(
        AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, kafka.getBootstrapServers()
    ));
    
    NewTopic ordersTopic = new NewTopic("orders", 3, (short) 1);
    admin.createTopics(List.of(ordersTopic)).all().get(30, TimeUnit.SECONDS);
    admin.close();
}

Running Tests in Parallel

When running multiple test classes in parallel, you need to isolate consumers by using unique group IDs per test:

@DynamicPropertySource
static void kafkaProperties(DynamicPropertyRegistry registry) {
    registry.add("spring.kafka.consumer.group-id",
        () -> "test-group-" + UUID.randomUUID());
}

This prevents one test class's consumer from stealing messages intended for another.

Key Takeaways

Testcontainers-based Kafka testing gives you real broker fidelity at the cost of Docker being available in your CI environment. The main patterns to remember:

  • Use @DynamicPropertySource to inject the runtime bootstrap address into Spring context
  • Use withReuse(true) to share the container across test classes and speed up the suite
  • Synchronize async consumer assertions with CountDownLatch or Awaitility
  • Pre-create topics in @BeforeAll to avoid auto-creation races
  • Use unique consumer group IDs when running tests in parallel

With these patterns in place, your Kafka integration tests will be both fast and trustworthy, catching real broker-level issues before they reach production.

Read more

Start now free