Testing Kafka Producers and Consumers with Embedded Kafka
Testing Kafka in production-like conditions without standing up a full cluster used to be painful. Today you have two solid options: the embedded Kafka broker that ships with spring-kafka-test, and the Testcontainers Kafka module that spins up a real broker in Docker. Both are fast enough for CI. The right choice depends on what you're actually testing.
This post covers both approaches in depth: producer and consumer test patterns, schema validation, and error-path testing that most tutorials skip.
Why Embedded Kafka Over Mocking
You could mock KafkaTemplate and @KafkaListener. The problem is that mocking bypasses the serializer, the partitioner, the consumer group offset logic, and the rebalance behavior — all of which are frequent sources of production bugs. Embedded Kafka keeps all that machinery intact while staying in-process.
The tradeoff: embedded Kafka uses more memory than a mock, and it doesn't cover Kafka version-specific behaviors. For version fidelity, use Testcontainers.
Setting Up Embedded Kafka
Add the dependency:
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka-test</artifactId>
<scope>test</scope>
</dependency>Annotate your test class:
@SpringBootTest
@EmbeddedKafka(
partitions = 3,
topics = {"orders", "order-events"},
brokerProperties = {
"auto.create.topics.enable=false",
"log.retention.ms=60000"
}
)
@TestPropertySource(properties = {
"spring.kafka.bootstrap-servers=${spring.embedded.kafka.brokers}",
"spring.kafka.consumer.auto-offset-reset=earliest"
})
class OrderKafkaTest {
@Autowired
private EmbeddedKafkaBroker embeddedKafka;
@Autowired
private KafkaTemplate<String, Order> kafkaTemplate;
}${spring.embedded.kafka.brokers} is a Spring placeholder that resolves to the random port the embedded broker binds on. Don't hardcode a port.
Testing a Producer
The cleanest producer test creates a real consumer on the test side and asserts what it receives:
@Test
void producer_sendsOrderWithCorrectKeyAndPartition() throws Exception {
// Create a direct consumer — no @KafkaListener overhead
Map<String, Object> consumerProps = KafkaTestUtils.consumerProps(
"test-group", "true", embeddedKafka
);
consumerProps.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
consumerProps.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, JsonDeserializer.class);
consumerProps.put(JsonDeserializer.TRUSTED_PACKAGES, "*");
DefaultKafkaConsumerFactory<String, Order> consumerFactory =
new DefaultKafkaConsumerFactory<>(consumerProps);
try (Consumer<String, Order> consumer = consumerFactory.createConsumer()) {
embeddedKafka.consumeFromAnEmbeddedTopic(consumer, "orders");
Order order = new Order("order-123", "user-456", BigDecimal.valueOf(99.99));
kafkaTemplate.send("orders", order.getId(), order).get(5, TimeUnit.SECONDS);
ConsumerRecords<String, Order> records =
KafkaTestUtils.getRecords(consumer, Duration.ofSeconds(10));
assertThat(records.count()).isEqualTo(1);
ConsumerRecord<String, Order> record = records.iterator().next();
assertThat(record.key()).isEqualTo("order-123");
assertThat(record.value().getUserId()).isEqualTo("user-456");
assertThat(record.value().getAmount()).isEqualByComparingTo("99.99");
}
}Notice .get(5, TimeUnit.SECONDS) on the send future — this makes the test fail fast if the broker rejects the message rather than swallowing the error silently.
Testing a Consumer
For consumer testing, flip it: produce from the test side, let your application consumer handle it, then assert against your downstream (database, another topic, etc.):
@Autowired
private OrderRepository orderRepository;
@Test
void consumer_persistsOrderOnReceive() throws Exception {
Map<String, Object> producerProps = KafkaTestUtils.producerProps(embeddedKafka);
producerProps.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
producerProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, JsonSerializer.class);
DefaultKafkaProducerFactory<String, Order> producerFactory =
new DefaultKafkaProducerFactory<>(producerProps);
KafkaTemplate<String, Order> testProducer = new KafkaTemplate<>(producerFactory);
Order order = new Order("order-789", "user-111", BigDecimal.valueOf(49.00));
testProducer.send("orders", order.getId(), order).get(5, TimeUnit.SECONDS);
// Poll until the listener processes it — don't sleep
await().atMost(Duration.ofSeconds(15))
.pollInterval(Duration.ofMillis(200))
.untilAsserted(() ->
assertThat(orderRepository.findById("order-789")).isPresent()
);
Order saved = orderRepository.findById("order-789").orElseThrow();
assertThat(saved.getAmount()).isEqualByComparingTo("49.00");
assertThat(saved.getStatus()).isEqualTo(OrderStatus.RECEIVED);
}Use awaitility rather than Thread.sleep. The consumer runs asynchronously; a fixed sleep will either be too short on a loaded CI runner or too long on a developer's machine.
Testing with Testcontainers Kafka
When you need a specific Kafka version or want to test broker-level behavior (ACLs, compaction, quotas), use Testcontainers:
@Testcontainers
@SpringBootTest
class OrderKafkaContainerTest {
@Container
static KafkaContainer kafka = new KafkaContainer(
DockerImageName.parse("confluentinc/cp-kafka:7.5.0")
).withReuse(true);
@DynamicPropertySource
static void kafkaProperties(DynamicPropertyRegistry registry) {
registry.add("spring.kafka.bootstrap-servers", kafka::getBootstrapServers);
}
}.withReuse(true) keeps the container alive between test class runs, which cuts startup overhead from 10s to ~0 after the first run. Enable reuse in `/.testcontainers.properties`:
testcontainers.reuse.enable=trueSchema Validation Testing
If you're using Avro or JSON Schema, test that invalid messages are rejected at the producer level before they reach consumers:
@Test
void producer_throwsOnInvalidSchema() {
// Order with null required field
Order invalidOrder = new Order(null, "user-456", BigDecimal.valueOf(10.00));
assertThatThrownBy(() ->
kafkaTemplate.send("orders", invalidOrder.getId(), invalidOrder).get(5, TimeUnit.SECONDS)
).hasCauseInstanceOf(SchemaRegistryException.class)
.hasMessageContaining("orderId: must not be null");
}For Schema Registry integration tests, Testcontainers has a dedicated module:
@Container
static SchemaRegistryContainer schemaRegistry = new SchemaRegistryContainer("7.5.0")
.withKafka(kafka);Testing Deserialization Error Handling
This is where most applications have silent failures. When a message can't be deserialized, the default Spring Kafka behavior is to log and skip it, which means poison pills accumulate in your topics indefinitely.
Test that your error handler routes bad messages to a dead letter topic:
@Test
void consumer_routesMalformedMessageToDlt() throws Exception {
// Send raw bytes that are not valid JSON
Map<String, Object> producerProps = KafkaTestUtils.producerProps(embeddedKafka);
producerProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, ByteArraySerializer.class);
DefaultKafkaProducerFactory<String, byte[]> producerFactory =
new DefaultKafkaProducerFactory<>(producerProps);
KafkaTemplate<String, byte[]> rawProducer = new KafkaTemplate<>(producerFactory);
rawProducer.send("orders", "bad-key", "not-valid-json{{{".getBytes()).get(5, TimeUnit.SECONDS);
// Consume from DLT and verify the message landed there
Map<String, Object> consumerProps = KafkaTestUtils.consumerProps("dlt-test-group", "true", embeddedKafka);
try (Consumer<String, byte[]> dltConsumer = new DefaultKafkaConsumerFactory<String, byte[]>(consumerProps).createConsumer()) {
embeddedKafka.consumeFromAnEmbeddedTopic(dltConsumer, "orders.DLT");
ConsumerRecords<String, byte[]> dltRecords =
KafkaTestUtils.getRecords(dltConsumer, Duration.ofSeconds(10));
assertThat(dltRecords.count()).isEqualTo(1);
ConsumerRecord<String, byte[]> dltRecord = dltRecords.iterator().next();
// Spring Kafka adds exception info as headers
Header exceptionHeader = dltRecord.headers().lastHeader(KafkaHeaders.DLT_EXCEPTION_MESSAGE);
assertThat(new String(exceptionHeader.value())).contains("JsonParseException");
}
}Testing Message Ordering Within a Partition
Kafka guarantees ordering within a partition, not across partitions. If your application depends on ordering, test that messages with the same key always go to the same partition:
@Test
void producer_sameKeyAlwaysSamePartition() throws Exception {
List<Integer> partitions = new ArrayList<>();
for (int i = 0; i < 10; i++) {
SendResult<String, Order> result = kafkaTemplate
.send("orders", "user-456", new Order("order-" + i, "user-456", BigDecimal.ONE))
.get(5, TimeUnit.SECONDS);
partitions.add(result.getRecordMetadata().partition());
}
// All messages with same key must land in same partition
assertThat(partitions).containsOnly(partitions.get(0));
}Resetting State Between Tests
Embedded Kafka doesn't reset offsets or delete messages between tests by default. Use unique group IDs per test to avoid consumers picking up messages from previous tests:
private String uniqueGroupId() {
return "test-group-" + UUID.randomUUID();
}Alternatively, use @DirtiesContext to restart the application context — but that's expensive. Unique group IDs are cheaper.
What to Test vs. What to Skip
Test with embedded/Testcontainers Kafka:
- Serializer/deserializer correctness
- Consumer group behavior and offset commits
- Retry and error handler logic
- DLT routing
- Schema validation
Don't bother testing:
- Kafka cluster failover (test with Chaos Monkey or Toxiproxy in integration environments)
- Exactly-once semantics under network partition (you need a real cluster for this)
- Rebalance behavior at scale
End-to-End Flow Test
Combine producer and consumer tests into a single end-to-end scenario to verify the full message lifecycle:
@Test
void orderFlow_producedOrderIsProcessedAndAcknowledged() throws Exception {
Order order = new Order("order-e2e-1", "user-999", BigDecimal.valueOf(250.00));
// Produce
kafkaTemplate.send("orders", order.getId(), order).get(5, TimeUnit.SECONDS);
// Wait for processing
await().atMost(Duration.ofSeconds(20))
.untilAsserted(() -> {
assertThat(orderRepository.findById("order-e2e-1")).isPresent();
assertThat(orderRepository.findById("order-e2e-1").get().getStatus())
.isEqualTo(OrderStatus.PROCESSED);
});
// Verify acknowledgment event was published
// ... consume from order-events topic and verify
}Putting It Together
The pattern is always the same: produce real messages, consume real messages, assert on real side effects. The embedded broker and Testcontainers give you a real Kafka runtime without the ops overhead. The key discipline is using awaitility instead of sleeps, unique consumer group IDs per test, and always asserting on DLT behavior for unhappy paths.
If you want to push these tests further — testing consumer behavior against a live UI, verifying that failed messages trigger alerts, or running multi-topic saga flows — HelpMeTest can run those end-to-end flows as automated scenarios against your deployed service. It speaks plain English, so you can describe the message flow and let it generate and run the test without writing framework code.