Testing Kafka Dead Letter Queues: Retry Topics, DLT Patterns, and Spring Kafka

Testing Kafka Dead Letter Queues: Retry Topics, DLT Patterns, and Spring Kafka

```tldr Kafka doesn't have native dead letter queues — but Spring Kafka, kafka-streams error handlers, and custom retry patterns implement DLQ behavior via retry topics and dead letter topics (DLT). Testing these paths means deliberately triggering consumer failures and asserting that messages route to the right retry or DLT topic. ```

```takeaways **Spring Kafka's @RetryableTopic auto-creates retry and DLT topics.** Test that failed messages appear in topic-retry-0, topic-retry-1, and finally topic-dlt after exhausting retries.

Use EmbeddedKafka for fast DLT unit tests. It runs Kafka in-process — no Docker required for testing the retry/DLT routing.

Inject a failing consumer to trigger retries. Use a counter or flag to fail the first N deliveries, then succeed — test that the message eventually processes.

DLT consumers need their own test assertions. After a message lands in the DLT, test that your DLT consumer logs it, alerts, or stores it correctly.

Test backoff timing with a @MockBean clock. Real exponential backoff takes minutes — override the clock or use @SpyBean to speed up tests. ```

Kafka DLQ Architecture

Kafka has no native DLQ. The common pattern adds retry topics with increasing delays:

main-topic
  └─► main-topic-retry-0      (immediate retry)
  └─► main-topic-retry-1      (delayed retry, e.g., 5s)
  └─► main-topic-retry-2      (delayed retry, e.g., 30s)
  └─► main-topic-dlt          (dead letter topic — give up)

Spring Kafka's @RetryableTopic implements this automatically.

Setup: Spring Kafka + EmbeddedKafka

<dependency>
  <groupId>org.springframework.kafka</groupId>
  <artifactId>spring-kafka</artifactId>
</dependency>
<dependency>
  <groupId>org.springframework.kafka</groupId>
  <artifactId>spring-kafka-test</artifactId>
  <scope>test</scope>
</dependency>
@SpringBootTest
@EmbeddedKafka(
    partitions = 1,
    topics = {"orders", "orders-retry-0", "orders-retry-1", "orders-dlt"},
    brokerProperties = {
        "auto.create.topics.enable=false",
        "offsets.topic.replication.factor=1"
    }
)
class OrderConsumerDltTest {
    @Autowired
    private KafkaTemplate<String, String> kafkaTemplate;

    @Autowired
    private EmbeddedKafkaBroker embeddedKafka;
}

Testing @RetryableTopic

@Component
class OrderConsumer {
    private final AtomicInteger failCount = new AtomicInteger(0);

    @RetryableTopic(
        attempts = "3",
        backoff = @Backoff(delay = 100, multiplier = 2),
        dltTopicSuffix = "-dlt",
        retryTopicSuffix = "-retry"
    )
    @KafkaListener(topics = "orders", groupId = "order-processor")
    public void consume(String message) {
        if (failCount.incrementAndGet() < 3) {
            throw new RuntimeException("Processing failed (attempt " + failCount.get() + ")");
        }
        // Success on 3rd attempt
        processOrder(message);
    }

    @DltHandler
    public void handleDlt(String message) {
        // Log or alert — message gave up
        dltMessages.add(message);
    }

    public List<String> dltMessages = new ArrayList<>();
    public void resetFailCount() { failCount.set(0); }
}

@Test
void retryableTopic_succeedsAfterRetries() throws Exception {
    orderConsumer.resetFailCount();
    CountDownLatch processedLatch = new CountDownLatch(1);
    // assume processOrder calls latch.countDown()

    kafkaTemplate.send("orders", "order-123").get();

    assertTrue(processedLatch.await(30, TimeUnit.SECONDS), "Order should be processed");
    assertTrue(orderConsumer.dltMessages.isEmpty(), "Nothing should land in DLT");
}

@Test
void retryableTopic_landsInDltAfterMaxAttempts() throws Exception {
    // Fail every time
    doThrow(new RuntimeException("always fails"))
        .when(orderService).process(any());

    CountDownLatch dltLatch = new CountDownLatch(1);
    // assume DLT handler calls latch.countDown()

    kafkaTemplate.send("orders", "bad-order").get();

    assertTrue(dltLatch.await(60, TimeUnit.SECONDS));
    assertFalse(orderConsumer.dltMessages.isEmpty());
    assertEquals("bad-order", orderConsumer.dltMessages.get(0));
}

Manual DLT Routing Test

For custom error handlers without Spring's @RetryableTopic:

@Bean
ConsumerFactory<String, String> consumerFactory(EmbeddedKafkaBroker broker) {
    Map<String, Object> props = new HashMap<>();
    props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, broker.getBrokersAsString());
    props.put(ConsumerConfig.GROUP_ID_CONFIG, "test-group");
    props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
    props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
    props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
    return new DefaultKafkaConsumerFactory<>(props);
}

@Bean
ConcurrentKafkaListenerContainerFactory<String, String> kafkaListenerContainerFactory(
    ConsumerFactory<String, String> consumerFactory,
    KafkaTemplate<String, String> dltTemplate
) {
    ConcurrentKafkaListenerContainerFactory<String, String> factory =
        new ConcurrentKafkaListenerContainerFactory<>();
    factory.setConsumerFactory(consumerFactory);
    factory.setCommonErrorHandler(
        new DefaultErrorHandler(
            (record, exception) -> {
                // Route to DLT on exhausted retries
                dltTemplate.send("orders-dlt", record.key(), (String) record.value());
            },
            new FixedBackOff(100L, 2L)  // 2 retries, 100ms apart
        )
    );
    return factory;
}

@Test
void customErrorHandler_routesFailedMessageToDlt() throws Exception {
    // Listen for messages on DLT
    CountDownLatch dltLatch = new CountDownLatch(1);
    AtomicReference<String> dltMessage = new AtomicReference<>();

    Consumer<String, String> dltConsumer = createTestConsumer("orders-dlt");

    kafkaTemplate.send("orders", "corrupt-data").get();

    // Poll DLT
    await().atMost(30, TimeUnit.SECONDS).until(() -> {
        ConsumerRecords<String, String> records = dltConsumer.poll(Duration.ofMillis(500));
        records.forEach(r -> {
            dltMessage.set(r.value());
            dltLatch.countDown();
        });
        return dltLatch.getCount() == 0;
    });

    assertEquals("corrupt-data", dltMessage.get());
}

Testing DLT Consumer Logic

The DLT consumer (the code that processes failed messages) needs its own tests:

@Component
class DltConsumer {
    private final AlertService alertService;
    private final FailedMessageRepository repository;

    @KafkaListener(topics = "orders-dlt", groupId = "dlt-processor")
    public void handleDeadLetter(
        ConsumerRecord<String, String> record,
        @Header(KafkaHeaders.EXCEPTION_MESSAGE) String exceptionMessage
    ) {
        log.error("Dead letter received: key={}, error={}", record.key(), exceptionMessage);
        alertService.sendAlert("DLT message: " + record.key());
        repository.save(FailedMessage.from(record, exceptionMessage));
    }
}

// Unit test — no Kafka needed
class DltConsumerTest {
    @Mock AlertService alertService;
    @Mock FailedMessageRepository repository;

    DltConsumer consumer;

    @BeforeEach
    void setup() {
        consumer = new DltConsumer(alertService, repository);
    }

    @Test
    void handleDeadLetter_alertsAndPersists() {
        ConsumerRecord<String, String> record = new ConsumerRecord<>(
            "orders-dlt", 0, 100L, "order-key", "invalid-payload"
        );
        String errorMsg = "JSON parse error";

        consumer.handleDeadLetter(record, errorMsg);

        verify(alertService).sendAlert(contains("order-key"));
        verify(repository).save(argThat(fm ->
            fm.getKey().equals("order-key") &&
            fm.getError().equals("JSON parse error")
        ));
    }
}

Testing With Testcontainers

For production-accurate tests:

@Testcontainers
class KafkaDltIntegrationTest {
    @Container
    static KafkaContainer kafka = new KafkaContainer(
        DockerImageName.parse("confluentinc/cp-kafka:7.6.0")
    );

    @DynamicPropertySource
    static void kafkaProperties(DynamicPropertyRegistry registry) {
        registry.add("spring.kafka.bootstrap-servers", kafka::getBootstrapServers);
    }

    @Test
    void dltFlowWithRealKafka() throws Exception {
        // Same test logic but with a real Kafka broker
        kafkaTemplate.send("orders", "bad-payload").get();

        await().atMost(Duration.ofSeconds(60)).untilAsserted(() ->
            assertFalse(dltRepository.findAll().isEmpty())
        );
    }
}

Testing Poison Pill Handling

A poison pill is a malformed message that crashes the consumer every time:

@Test
void poisonPill_doesNotBlockSubsequentMessages() throws Exception {
    CountDownLatch goodMessageLatch = new CountDownLatch(1);

    // Send poison pill first
    kafkaTemplate.send("orders", "not-valid-json").get();

    // Send valid message second
    kafkaTemplate.send("orders", "{\"orderId\": \"999\"}").get();

    // Valid message should still process (poison pill goes to DLT)
    assertTrue(goodMessageLatch.await(30, TimeUnit.SECONDS),
        "Valid message should process despite preceding poison pill");
}

Verifying Headers on DLT Messages

Spring Kafka adds exception headers to DLT messages:

@Test
void dltMessageContainsExceptionHeaders() throws Exception {
    kafkaTemplate.send("orders", "bad-order").get();

    // Wait for DLT delivery
    ConsumerRecord<String, String> dltRecord = awaitDltMessage("orders-dlt", 30);

    // Spring Kafka adds these headers automatically
    assertNotNull(dltRecord.headers().lastHeader(
        KafkaHeaders.EXCEPTION_FQCN
    ));
    assertNotNull(dltRecord.headers().lastHeader(
        KafkaHeaders.EXCEPTION_MESSAGE
    ));
    assertNotNull(dltRecord.headers().lastHeader(
        KafkaHeaders.ORIGINAL_TOPIC
    ));
    assertEquals("orders",
        new String(dltRecord.headers().lastHeader(KafkaHeaders.ORIGINAL_TOPIC).value())
    );
}

Common DLT Testing Mistakes

Not waiting long enough for retries — exponential backoff with real delays takes minutes. Either reduce delays in test config or use await() with a generous timeout.

Testing only the happy path — most bugs in DLT flows are in the error path. Test what happens when the DLT consumer itself fails (double-fault).

Asserting too early — retries happen asynchronously. A synchronous assertEquals immediately after kafkaTemplate.send() will fail before retries complete.

Forgetting auto.create.topics.enable=false — without this, Kafka auto-creates topics before your consumer can set up @RetryableTopic config, bypassing retry logic.

Read more

Start now free