Testing Kafka Producers and Consumers with Embedded Kafka

Testing Kafka Producers and Consumers with Embedded Kafka

Testing Kafka in production-like conditions without standing up a full cluster used to be painful. Today you have two solid options: the embedded Kafka broker that ships with spring-kafka-test, and the Testcontainers Kafka module that spins up a real broker in Docker. Both are fast enough for CI. The right choice depends on what you're actually testing.

This post covers both approaches in depth: producer and consumer test patterns, schema validation, and error-path testing that most tutorials skip.

Why Embedded Kafka Over Mocking

You could mock KafkaTemplate and @KafkaListener. The problem is that mocking bypasses the serializer, the partitioner, the consumer group offset logic, and the rebalance behavior — all of which are frequent sources of production bugs. Embedded Kafka keeps all that machinery intact while staying in-process.

The tradeoff: embedded Kafka uses more memory than a mock, and it doesn't cover Kafka version-specific behaviors. For version fidelity, use Testcontainers.

Setting Up Embedded Kafka

Add the dependency:

<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka-test</artifactId>
    <scope>test</scope>
</dependency>

Annotate your test class:

@SpringBootTest
@EmbeddedKafka(
    partitions = 3,
    topics = {"orders", "order-events"},
    brokerProperties = {
        "auto.create.topics.enable=false",
        "log.retention.ms=60000"
    }
)
@TestPropertySource(properties = {
    "spring.kafka.bootstrap-servers=${spring.embedded.kafka.brokers}",
    "spring.kafka.consumer.auto-offset-reset=earliest"
})
class OrderKafkaTest {

    @Autowired
    private EmbeddedKafkaBroker embeddedKafka;

    @Autowired
    private KafkaTemplate<String, Order> kafkaTemplate;
}

${spring.embedded.kafka.brokers} is a Spring placeholder that resolves to the random port the embedded broker binds on. Don't hardcode a port.

Testing a Producer

The cleanest producer test creates a real consumer on the test side and asserts what it receives:

@Test
void producer_sendsOrderWithCorrectKeyAndPartition() throws Exception {
    // Create a direct consumer — no @KafkaListener overhead
    Map<String, Object> consumerProps = KafkaTestUtils.consumerProps(
        "test-group", "true", embeddedKafka
    );
    consumerProps.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
    consumerProps.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, JsonDeserializer.class);
    consumerProps.put(JsonDeserializer.TRUSTED_PACKAGES, "*");

    DefaultKafkaConsumerFactory<String, Order> consumerFactory =
        new DefaultKafkaConsumerFactory<>(consumerProps);

    try (Consumer<String, Order> consumer = consumerFactory.createConsumer()) {
        embeddedKafka.consumeFromAnEmbeddedTopic(consumer, "orders");

        Order order = new Order("order-123", "user-456", BigDecimal.valueOf(99.99));
        kafkaTemplate.send("orders", order.getId(), order).get(5, TimeUnit.SECONDS);

        ConsumerRecords<String, Order> records =
            KafkaTestUtils.getRecords(consumer, Duration.ofSeconds(10));

        assertThat(records.count()).isEqualTo(1);
        ConsumerRecord<String, Order> record = records.iterator().next();
        assertThat(record.key()).isEqualTo("order-123");
        assertThat(record.value().getUserId()).isEqualTo("user-456");
        assertThat(record.value().getAmount()).isEqualByComparingTo("99.99");
    }
}

Notice .get(5, TimeUnit.SECONDS) on the send future — this makes the test fail fast if the broker rejects the message rather than swallowing the error silently.

Testing a Consumer

For consumer testing, flip it: produce from the test side, let your application consumer handle it, then assert against your downstream (database, another topic, etc.):

@Autowired
private OrderRepository orderRepository;

@Test
void consumer_persistsOrderOnReceive() throws Exception {
    Map<String, Object> producerProps = KafkaTestUtils.producerProps(embeddedKafka);
    producerProps.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
    producerProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, JsonSerializer.class);

    DefaultKafkaProducerFactory<String, Order> producerFactory =
        new DefaultKafkaProducerFactory<>(producerProps);
    KafkaTemplate<String, Order> testProducer = new KafkaTemplate<>(producerFactory);

    Order order = new Order("order-789", "user-111", BigDecimal.valueOf(49.00));
    testProducer.send("orders", order.getId(), order).get(5, TimeUnit.SECONDS);

    // Poll until the listener processes it — don't sleep
    await().atMost(Duration.ofSeconds(15))
           .pollInterval(Duration.ofMillis(200))
           .untilAsserted(() ->
               assertThat(orderRepository.findById("order-789")).isPresent()
           );

    Order saved = orderRepository.findById("order-789").orElseThrow();
    assertThat(saved.getAmount()).isEqualByComparingTo("49.00");
    assertThat(saved.getStatus()).isEqualTo(OrderStatus.RECEIVED);
}

Use awaitility rather than Thread.sleep. The consumer runs asynchronously; a fixed sleep will either be too short on a loaded CI runner or too long on a developer's machine.

Testing with Testcontainers Kafka

When you need a specific Kafka version or want to test broker-level behavior (ACLs, compaction, quotas), use Testcontainers:

@Testcontainers
@SpringBootTest
class OrderKafkaContainerTest {

    @Container
    static KafkaContainer kafka = new KafkaContainer(
        DockerImageName.parse("confluentinc/cp-kafka:7.5.0")
    ).withReuse(true);

    @DynamicPropertySource
    static void kafkaProperties(DynamicPropertyRegistry registry) {
        registry.add("spring.kafka.bootstrap-servers", kafka::getBootstrapServers);
    }
}

.withReuse(true) keeps the container alive between test class runs, which cuts startup overhead from 10s to ~0 after the first run. Enable reuse in `/.testcontainers.properties`:

testcontainers.reuse.enable=true

Schema Validation Testing

If you're using Avro or JSON Schema, test that invalid messages are rejected at the producer level before they reach consumers:

@Test
void producer_throwsOnInvalidSchema() {
    // Order with null required field
    Order invalidOrder = new Order(null, "user-456", BigDecimal.valueOf(10.00));

    assertThatThrownBy(() ->
        kafkaTemplate.send("orders", invalidOrder.getId(), invalidOrder).get(5, TimeUnit.SECONDS)
    ).hasCauseInstanceOf(SchemaRegistryException.class)
     .hasMessageContaining("orderId: must not be null");
}

For Schema Registry integration tests, Testcontainers has a dedicated module:

@Container
static SchemaRegistryContainer schemaRegistry = new SchemaRegistryContainer("7.5.0")
    .withKafka(kafka);

Testing Deserialization Error Handling

This is where most applications have silent failures. When a message can't be deserialized, the default Spring Kafka behavior is to log and skip it, which means poison pills accumulate in your topics indefinitely.

Test that your error handler routes bad messages to a dead letter topic:

@Test
void consumer_routesMalformedMessageToDlt() throws Exception {
    // Send raw bytes that are not valid JSON
    Map<String, Object> producerProps = KafkaTestUtils.producerProps(embeddedKafka);
    producerProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, ByteArraySerializer.class);

    DefaultKafkaProducerFactory<String, byte[]> producerFactory =
        new DefaultKafkaProducerFactory<>(producerProps);
    KafkaTemplate<String, byte[]> rawProducer = new KafkaTemplate<>(producerFactory);

    rawProducer.send("orders", "bad-key", "not-valid-json{{{".getBytes()).get(5, TimeUnit.SECONDS);

    // Consume from DLT and verify the message landed there
    Map<String, Object> consumerProps = KafkaTestUtils.consumerProps("dlt-test-group", "true", embeddedKafka);
    try (Consumer<String, byte[]> dltConsumer = new DefaultKafkaConsumerFactory<String, byte[]>(consumerProps).createConsumer()) {
        embeddedKafka.consumeFromAnEmbeddedTopic(dltConsumer, "orders.DLT");

        ConsumerRecords<String, byte[]> dltRecords =
            KafkaTestUtils.getRecords(dltConsumer, Duration.ofSeconds(10));

        assertThat(dltRecords.count()).isEqualTo(1);
        ConsumerRecord<String, byte[]> dltRecord = dltRecords.iterator().next();

        // Spring Kafka adds exception info as headers
        Header exceptionHeader = dltRecord.headers().lastHeader(KafkaHeaders.DLT_EXCEPTION_MESSAGE);
        assertThat(new String(exceptionHeader.value())).contains("JsonParseException");
    }
}

Testing Message Ordering Within a Partition

Kafka guarantees ordering within a partition, not across partitions. If your application depends on ordering, test that messages with the same key always go to the same partition:

@Test
void producer_sameKeyAlwaysSamePartition() throws Exception {
    List<Integer> partitions = new ArrayList<>();

    for (int i = 0; i < 10; i++) {
        SendResult<String, Order> result = kafkaTemplate
            .send("orders", "user-456", new Order("order-" + i, "user-456", BigDecimal.ONE))
            .get(5, TimeUnit.SECONDS);
        partitions.add(result.getRecordMetadata().partition());
    }

    // All messages with same key must land in same partition
    assertThat(partitions).containsOnly(partitions.get(0));
}

Resetting State Between Tests

Embedded Kafka doesn't reset offsets or delete messages between tests by default. Use unique group IDs per test to avoid consumers picking up messages from previous tests:

private String uniqueGroupId() {
    return "test-group-" + UUID.randomUUID();
}

Alternatively, use @DirtiesContext to restart the application context — but that's expensive. Unique group IDs are cheaper.

What to Test vs. What to Skip

Test with embedded/Testcontainers Kafka:

  • Serializer/deserializer correctness
  • Consumer group behavior and offset commits
  • Retry and error handler logic
  • DLT routing
  • Schema validation

Don't bother testing:

  • Kafka cluster failover (test with Chaos Monkey or Toxiproxy in integration environments)
  • Exactly-once semantics under network partition (you need a real cluster for this)
  • Rebalance behavior at scale

End-to-End Flow Test

Combine producer and consumer tests into a single end-to-end scenario to verify the full message lifecycle:

@Test
void orderFlow_producedOrderIsProcessedAndAcknowledged() throws Exception {
    Order order = new Order("order-e2e-1", "user-999", BigDecimal.valueOf(250.00));

    // Produce
    kafkaTemplate.send("orders", order.getId(), order).get(5, TimeUnit.SECONDS);

    // Wait for processing
    await().atMost(Duration.ofSeconds(20))
           .untilAsserted(() -> {
               assertThat(orderRepository.findById("order-e2e-1")).isPresent();
               assertThat(orderRepository.findById("order-e2e-1").get().getStatus())
                   .isEqualTo(OrderStatus.PROCESSED);
           });

    // Verify acknowledgment event was published
    // ... consume from order-events topic and verify
}

Putting It Together

The pattern is always the same: produce real messages, consume real messages, assert on real side effects. The embedded broker and Testcontainers give you a real Kafka runtime without the ops overhead. The key discipline is using awaitility instead of sleeps, unique consumer group IDs per test, and always asserting on DLT behavior for unhappy paths.

If you want to push these tests further — testing consumer behavior against a live UI, verifying that failed messages trigger alerts, or running multi-topic saga flows — HelpMeTest can run those end-to-end flows as automated scenarios against your deployed service. It speaks plain English, so you can describe the message flow and let it generate and run the test without writing framework code.

Read more

Start now free