Testing Kafka Producers and Consumers with Testcontainers
Kafka is the backbone of event-driven architectures, but testing it has historically been painful. Embedded Kafka brokers drift from real broker behavior, shared test environments create flaky tests, and mocking away the broker entirely means you are not testing anything meaningful. Testcontainers solves this by spinning up a real Kafka broker inside Docker for every test run, giving you production-fidelity without the shared-environment headaches.
This guide walks through everything you need to write fast, reliable integration tests for Kafka producers and consumers using Testcontainers with Spring Boot.
Why Testcontainers Over Embedded Kafka
Before diving into code, it is worth understanding what you gain by switching from @EmbeddedKafka to Testcontainers.
Embedded Kafka runs an in-process broker. It is fast to start, but it uses an older version of Kafka and diverges from real broker behavior in subtle ways — topic auto-creation semantics, log compaction, replication, and ACL enforcement all differ. When your production cluster runs Kafka 3.6 and your tests use embedded Kafka 2.x, you are building false confidence.
Testcontainers runs the exact same Docker image your production cluster uses. You can pin to confluentinc/cp-kafka:7.6.0 and know your tests exercise the actual broker behavior.
Project Setup
Add the following dependencies to your build.gradle:
dependencies {
implementation 'org.springframework.boot:spring-boot-starter'
implementation 'org.springframework.kafka:spring-kafka'
testImplementation 'org.springframework.boot:spring-boot-starter-test'
testImplementation 'org.springframework.kafka:spring-kafka-test'
testImplementation 'org.testcontainers:kafka:1.19.7'
testImplementation 'org.testcontainers:junit-jupiter:1.19.7'
}Or with Maven:
<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>kafka</artifactId>
<version>1.19.7</version>
<scope>test</scope>
</dependency>Creating a Reusable Kafka Test Base
The most important decision is how to manage the container lifecycle. Starting a new Kafka container for every test class is slow. The reuse feature in Testcontainers lets you share a single container across the entire test suite.
@SpringBootTest
@Testcontainers
public abstract class KafkaIntegrationTestBase {
@Container
static final KafkaContainer kafka = new KafkaContainer(
DockerImageName.parse("confluentinc/cp-kafka:7.6.0"))
.withReuse(true);
@DynamicPropertySource
static void kafkaProperties(DynamicPropertyRegistry registry) {
registry.add("spring.kafka.bootstrap-servers", kafka::getBootstrapServers);
registry.add("spring.kafka.consumer.auto-offset-reset", () -> "earliest");
registry.add("spring.kafka.consumer.group-id", () -> "test-group");
}
}The @DynamicPropertySource hook injects the dynamically assigned bootstrap server address into your Spring context before it starts. This is the critical piece — without it, your application context will try to connect to localhost:9092 which will not exist.
Writing a Producer
Here is a simple order event producer:
@Service
public class OrderEventProducer {
private static final String TOPIC = "orders";
private final KafkaTemplate<String, OrderEvent> kafkaTemplate;
public OrderEventProducer(KafkaTemplate<String, OrderEvent> kafkaTemplate) {
this.kafkaTemplate = kafkaTemplate;
}
public CompletableFuture<SendResult<String, OrderEvent>> sendOrder(OrderEvent event) {
return kafkaTemplate.send(TOPIC, event.orderId(), event)
.toCompletableFuture();
}
}public record OrderEvent(String orderId, String customerId, BigDecimal amount, String status) {}Writing a Consumer
@Service
public class OrderEventConsumer {
private final List<OrderEvent> receivedEvents = new CopyOnWriteArrayList<>();
private final CountDownLatch latch = new CountDownLatch(1);
@KafkaListener(topics = "orders", groupId = "order-processor")
public void consume(OrderEvent event) {
receivedEvents.add(event);
latch.countDown();
}
public List<OrderEvent> getReceivedEvents() {
return Collections.unmodifiableList(receivedEvents);
}
public CountDownLatch getLatch() {
return latch;
}
}Testing the Producer
class OrderEventProducerTest extends KafkaIntegrationTestBase {
@Autowired
private OrderEventProducer producer;
@Test
void shouldSendOrderEventToKafka() throws Exception {
// Arrange
OrderEvent event = new OrderEvent("ORD-001", "CUST-123",
new BigDecimal("99.99"), "CREATED");
// Act
SendResult<String, OrderEvent> result = producer.sendOrder(event)
.get(10, TimeUnit.SECONDS);
// Assert
assertThat(result.getRecordMetadata().topic()).isEqualTo("orders");
assertThat(result.getRecordMetadata().partition()).isGreaterThanOrEqualTo(0);
assertThat(result.getProducerRecord().key()).isEqualTo("ORD-001");
}
@Test
void shouldSendToCorrectPartitionForOrderKey() throws Exception {
// Send multiple events with same key — should land on same partition
OrderEvent event1 = new OrderEvent("ORD-100", "CUST-1",
new BigDecimal("10.00"), "CREATED");
OrderEvent event2 = new OrderEvent("ORD-100", "CUST-1",
new BigDecimal("10.00"), "UPDATED");
SendResult<?, ?> r1 = producer.sendOrder(event1).get(10, TimeUnit.SECONDS);
SendResult<?, ?> r2 = producer.sendOrder(event2).get(10, TimeUnit.SECONDS);
assertThat(r1.getRecordMetadata().partition())
.isEqualTo(r2.getRecordMetadata().partition());
}
}Testing the Consumer End-to-End
For consumer tests, you want to produce a message and verify the consumer processes it correctly. The tricky part is synchronization — Kafka consumption is asynchronous.
class OrderEventConsumerTest extends KafkaIntegrationTestBase {
@Autowired
private KafkaTemplate<String, OrderEvent> kafkaTemplate;
@Autowired
private OrderEventConsumer consumer;
@Test
void shouldConsumeOrderEventFromKafka() throws Exception {
// Arrange
OrderEvent event = new OrderEvent("ORD-002", "CUST-456",
new BigDecimal("149.99"), "SHIPPED");
// Act
kafkaTemplate.send("orders", event.orderId(), event).get();
// Wait for consumer to process
boolean consumed = consumer.getLatch().await(15, TimeUnit.SECONDS);
// Assert
assertThat(consumed).isTrue();
assertThat(consumer.getReceivedEvents()).hasSize(1);
assertThat(consumer.getReceivedEvents().get(0).orderId()).isEqualTo("ORD-002");
assertThat(consumer.getReceivedEvents().get(0).status()).isEqualTo("SHIPPED");
}
}Testing Consumer Error Handling
Production consumers need robust error handling. Here is how to test dead letter topic behavior:
@Test
void shouldSendToDeadLetterTopicOnProcessingFailure() throws Exception {
// Arrange — send a malformed event that triggers processing error
String badPayload = "{\"invalid\": true}";
ProducerRecord<String, String> record =
new ProducerRecord<>("orders", "BAD-KEY", badPayload);
// Use raw string template to bypass serialization
KafkaTemplate<String, String> rawTemplate = buildRawStringTemplate();
rawTemplate.send(record).get();
// Assert the DLT receives the message
ConsumerRecord<String, String> dlqRecord =
KafkaTestUtils.getSingleRecord(dltConsumer, "orders.DLT",
Duration.ofSeconds(15));
assertThat(dlqRecord).isNotNull();
assertThat(dlqRecord.key()).isEqualTo("BAD-KEY");
}Using KafkaTestUtils for Verification
Spring Kafka ships KafkaTestUtils, which provides convenient methods for asserting broker state:
@Test
void shouldPublishExactlyOneRecord() throws Exception {
// Create a consumer to read from the topic
Map<String, Object> consumerProps = KafkaTestUtils.consumerProps(
kafka.getBootstrapServers(), "verify-group", "true");
Consumer<String, String> consumer = new DefaultKafkaConsumerFactory<String, String>(consumerProps)
.createConsumer();
consumer.subscribe(Collections.singletonList("orders"));
// Produce
producer.sendOrder(new OrderEvent("ORD-003", "CUST-789",
new BigDecimal("299.99"), "CREATED")).get();
// Verify
ConsumerRecords<String, String> records =
KafkaTestUtils.getRecords(consumer, Duration.ofSeconds(10));
assertThat(records.count()).isEqualTo(1);
consumer.close();
}Testing Topic Configuration
Often you need to verify topics are created with the right configuration — correct replication factor, compaction policy, or retention settings:
@Test
void shouldCreateOrdersTopicWithCorrectConfig() throws Exception {
AdminClient admin = AdminClient.create(Map.of(
AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, kafka.getBootstrapServers()
));
DescribeTopicsResult result = admin.describeTopics(List.of("orders"));
TopicDescription description = result.allTopicNames().get()
.get("orders");
assertThat(description.partitions()).hasSize(3);
assertThat(description.partitions().get(0).replicas()).hasSize(1);
admin.close();
}Handling Slow CI Environments
On CI, Docker startup can be slower than locally. A few patterns help:
Set longer timeouts in test configuration:
# application-test.yaml
spring:
kafka:
consumer:
auto-offset-reset: earliest
listener:
ack-mode: record
# Increase poll timeout for slow CI
kafka:
consumer:
poll-timeout: 5000Use @BeforeAll to pre-warm topics:
@BeforeAll
static void createTopics() throws Exception {
AdminClient admin = AdminClient.create(Map.of(
AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, kafka.getBootstrapServers()
));
NewTopic ordersTopic = new NewTopic("orders", 3, (short) 1);
admin.createTopics(List.of(ordersTopic)).all().get(30, TimeUnit.SECONDS);
admin.close();
}Running Tests in Parallel
When running multiple test classes in parallel, you need to isolate consumers by using unique group IDs per test:
@DynamicPropertySource
static void kafkaProperties(DynamicPropertyRegistry registry) {
registry.add("spring.kafka.consumer.group-id",
() -> "test-group-" + UUID.randomUUID());
}This prevents one test class's consumer from stealing messages intended for another.
Key Takeaways
Testcontainers-based Kafka testing gives you real broker fidelity at the cost of Docker being available in your CI environment. The main patterns to remember:
- Use
@DynamicPropertySourceto inject the runtime bootstrap address into Spring context - Use
withReuse(true)to share the container across test classes and speed up the suite - Synchronize async consumer assertions with
CountDownLatchorAwaitility - Pre-create topics in
@BeforeAllto avoid auto-creation races - Use unique consumer group IDs when running tests in parallel
With these patterns in place, your Kafka integration tests will be both fast and trustworthy, catching real broker-level issues before they reach production.