Testing Kafka Dead Letter Queues: Retry Topics, DLT Patterns, and Spring Kafka
```tldr Kafka doesn't have native dead letter queues — but Spring Kafka, kafka-streams error handlers, and custom retry patterns implement DLQ behavior via retry topics and dead letter topics (DLT). Testing these paths means deliberately triggering consumer failures and asserting that messages route to the right retry or DLT topic. ```
```takeaways **Spring Kafka's @RetryableTopic auto-creates retry and DLT topics.** Test that failed messages appear in topic-retry-0, topic-retry-1, and finally topic-dlt after exhausting retries.
Use EmbeddedKafka for fast DLT unit tests. It runs Kafka in-process — no Docker required for testing the retry/DLT routing.
Inject a failing consumer to trigger retries. Use a counter or flag to fail the first N deliveries, then succeed — test that the message eventually processes.
DLT consumers need their own test assertions. After a message lands in the DLT, test that your DLT consumer logs it, alerts, or stores it correctly.
Test backoff timing with a @MockBean clock. Real exponential backoff takes minutes — override the clock or use @SpyBean to speed up tests. ```
Kafka DLQ Architecture
Kafka has no native DLQ. The common pattern adds retry topics with increasing delays:
main-topic
└─► main-topic-retry-0 (immediate retry)
└─► main-topic-retry-1 (delayed retry, e.g., 5s)
└─► main-topic-retry-2 (delayed retry, e.g., 30s)
└─► main-topic-dlt (dead letter topic — give up)Spring Kafka's @RetryableTopic implements this automatically.
Setup: Spring Kafka + EmbeddedKafka
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka-test</artifactId>
<scope>test</scope>
</dependency>@SpringBootTest
@EmbeddedKafka(
partitions = 1,
topics = {"orders", "orders-retry-0", "orders-retry-1", "orders-dlt"},
brokerProperties = {
"auto.create.topics.enable=false",
"offsets.topic.replication.factor=1"
}
)
class OrderConsumerDltTest {
@Autowired
private KafkaTemplate<String, String> kafkaTemplate;
@Autowired
private EmbeddedKafkaBroker embeddedKafka;
}Testing @RetryableTopic
@Component
class OrderConsumer {
private final AtomicInteger failCount = new AtomicInteger(0);
@RetryableTopic(
attempts = "3",
backoff = @Backoff(delay = 100, multiplier = 2),
dltTopicSuffix = "-dlt",
retryTopicSuffix = "-retry"
)
@KafkaListener(topics = "orders", groupId = "order-processor")
public void consume(String message) {
if (failCount.incrementAndGet() < 3) {
throw new RuntimeException("Processing failed (attempt " + failCount.get() + ")");
}
// Success on 3rd attempt
processOrder(message);
}
@DltHandler
public void handleDlt(String message) {
// Log or alert — message gave up
dltMessages.add(message);
}
public List<String> dltMessages = new ArrayList<>();
public void resetFailCount() { failCount.set(0); }
}
@Test
void retryableTopic_succeedsAfterRetries() throws Exception {
orderConsumer.resetFailCount();
CountDownLatch processedLatch = new CountDownLatch(1);
// assume processOrder calls latch.countDown()
kafkaTemplate.send("orders", "order-123").get();
assertTrue(processedLatch.await(30, TimeUnit.SECONDS), "Order should be processed");
assertTrue(orderConsumer.dltMessages.isEmpty(), "Nothing should land in DLT");
}
@Test
void retryableTopic_landsInDltAfterMaxAttempts() throws Exception {
// Fail every time
doThrow(new RuntimeException("always fails"))
.when(orderService).process(any());
CountDownLatch dltLatch = new CountDownLatch(1);
// assume DLT handler calls latch.countDown()
kafkaTemplate.send("orders", "bad-order").get();
assertTrue(dltLatch.await(60, TimeUnit.SECONDS));
assertFalse(orderConsumer.dltMessages.isEmpty());
assertEquals("bad-order", orderConsumer.dltMessages.get(0));
}Manual DLT Routing Test
For custom error handlers without Spring's @RetryableTopic:
@Bean
ConsumerFactory<String, String> consumerFactory(EmbeddedKafkaBroker broker) {
Map<String, Object> props = new HashMap<>();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, broker.getBrokersAsString());
props.put(ConsumerConfig.GROUP_ID_CONFIG, "test-group");
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
return new DefaultKafkaConsumerFactory<>(props);
}
@Bean
ConcurrentKafkaListenerContainerFactory<String, String> kafkaListenerContainerFactory(
ConsumerFactory<String, String> consumerFactory,
KafkaTemplate<String, String> dltTemplate
) {
ConcurrentKafkaListenerContainerFactory<String, String> factory =
new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactory);
factory.setCommonErrorHandler(
new DefaultErrorHandler(
(record, exception) -> {
// Route to DLT on exhausted retries
dltTemplate.send("orders-dlt", record.key(), (String) record.value());
},
new FixedBackOff(100L, 2L) // 2 retries, 100ms apart
)
);
return factory;
}
@Test
void customErrorHandler_routesFailedMessageToDlt() throws Exception {
// Listen for messages on DLT
CountDownLatch dltLatch = new CountDownLatch(1);
AtomicReference<String> dltMessage = new AtomicReference<>();
Consumer<String, String> dltConsumer = createTestConsumer("orders-dlt");
kafkaTemplate.send("orders", "corrupt-data").get();
// Poll DLT
await().atMost(30, TimeUnit.SECONDS).until(() -> {
ConsumerRecords<String, String> records = dltConsumer.poll(Duration.ofMillis(500));
records.forEach(r -> {
dltMessage.set(r.value());
dltLatch.countDown();
});
return dltLatch.getCount() == 0;
});
assertEquals("corrupt-data", dltMessage.get());
}Testing DLT Consumer Logic
The DLT consumer (the code that processes failed messages) needs its own tests:
@Component
class DltConsumer {
private final AlertService alertService;
private final FailedMessageRepository repository;
@KafkaListener(topics = "orders-dlt", groupId = "dlt-processor")
public void handleDeadLetter(
ConsumerRecord<String, String> record,
@Header(KafkaHeaders.EXCEPTION_MESSAGE) String exceptionMessage
) {
log.error("Dead letter received: key={}, error={}", record.key(), exceptionMessage);
alertService.sendAlert("DLT message: " + record.key());
repository.save(FailedMessage.from(record, exceptionMessage));
}
}
// Unit test — no Kafka needed
class DltConsumerTest {
@Mock AlertService alertService;
@Mock FailedMessageRepository repository;
DltConsumer consumer;
@BeforeEach
void setup() {
consumer = new DltConsumer(alertService, repository);
}
@Test
void handleDeadLetter_alertsAndPersists() {
ConsumerRecord<String, String> record = new ConsumerRecord<>(
"orders-dlt", 0, 100L, "order-key", "invalid-payload"
);
String errorMsg = "JSON parse error";
consumer.handleDeadLetter(record, errorMsg);
verify(alertService).sendAlert(contains("order-key"));
verify(repository).save(argThat(fm ->
fm.getKey().equals("order-key") &&
fm.getError().equals("JSON parse error")
));
}
}Testing With Testcontainers
For production-accurate tests:
@Testcontainers
class KafkaDltIntegrationTest {
@Container
static KafkaContainer kafka = new KafkaContainer(
DockerImageName.parse("confluentinc/cp-kafka:7.6.0")
);
@DynamicPropertySource
static void kafkaProperties(DynamicPropertyRegistry registry) {
registry.add("spring.kafka.bootstrap-servers", kafka::getBootstrapServers);
}
@Test
void dltFlowWithRealKafka() throws Exception {
// Same test logic but with a real Kafka broker
kafkaTemplate.send("orders", "bad-payload").get();
await().atMost(Duration.ofSeconds(60)).untilAsserted(() ->
assertFalse(dltRepository.findAll().isEmpty())
);
}
}Testing Poison Pill Handling
A poison pill is a malformed message that crashes the consumer every time:
@Test
void poisonPill_doesNotBlockSubsequentMessages() throws Exception {
CountDownLatch goodMessageLatch = new CountDownLatch(1);
// Send poison pill first
kafkaTemplate.send("orders", "not-valid-json").get();
// Send valid message second
kafkaTemplate.send("orders", "{\"orderId\": \"999\"}").get();
// Valid message should still process (poison pill goes to DLT)
assertTrue(goodMessageLatch.await(30, TimeUnit.SECONDS),
"Valid message should process despite preceding poison pill");
}Verifying Headers on DLT Messages
Spring Kafka adds exception headers to DLT messages:
@Test
void dltMessageContainsExceptionHeaders() throws Exception {
kafkaTemplate.send("orders", "bad-order").get();
// Wait for DLT delivery
ConsumerRecord<String, String> dltRecord = awaitDltMessage("orders-dlt", 30);
// Spring Kafka adds these headers automatically
assertNotNull(dltRecord.headers().lastHeader(
KafkaHeaders.EXCEPTION_FQCN
));
assertNotNull(dltRecord.headers().lastHeader(
KafkaHeaders.EXCEPTION_MESSAGE
));
assertNotNull(dltRecord.headers().lastHeader(
KafkaHeaders.ORIGINAL_TOPIC
));
assertEquals("orders",
new String(dltRecord.headers().lastHeader(KafkaHeaders.ORIGINAL_TOPIC).value())
);
}Common DLT Testing Mistakes
Not waiting long enough for retries — exponential backoff with real delays takes minutes. Either reduce delays in test config or use await() with a generous timeout.
Testing only the happy path — most bugs in DLT flows are in the error path. Test what happens when the DLT consumer itself fails (double-fault).
Asserting too early — retries happen asynchronously. A synchronous assertEquals immediately after kafkaTemplate.send() will fail before retries complete.
Forgetting auto.create.topics.enable=false — without this, Kafka auto-creates topics before your consumer can set up @RetryableTopic config, bypassing retry logic.