Testing Exactly-Once Semantics in Kafka: A Practical Guide

Testing Exactly-Once Semantics in Kafka: A Practical Guide

Exactly-once semantics (EOS) in Kafka is one of the most misunderstood guarantees in distributed systems. Developers often assume that enabling enable.idempotence=true on the producer is sufficient, but EOS requires a coordinated setup across producers, brokers, and consumers. More importantly, it needs to be tested — the configuration is subtle, and silent misconfiguration leads to duplicate processing in production that only surfaces under specific failure conditions.

This guide covers how to write tests that validate exactly-once behavior end-to-end, including what to test, what tools to use, and how to simulate the failure scenarios that actually trigger duplicates.

What Exactly-Once Actually Means

Kafka's exactly-once guarantee has three layers:

  1. Idempotent producer: A single producer session will not duplicate messages on broker retries. Enabled by enable.idempotence=true.
  2. Transactional producer: Messages sent across multiple topics/partitions are committed atomically. Enabled by setting transactional.id.
  3. Read-committed consumers: Consumers only see committed transactional messages. Enabled by isolation.level=read_committed.

Testing EOS means verifying all three layers interact correctly, especially under retry and failure scenarios.

Test Setup

You need a real Kafka broker to test transactions — embedded brokers have limited transaction support.

@SpringBootTest
@Testcontainers
public abstract class ExactlyOnceTestBase {

    @Container
    static final KafkaContainer kafka = new KafkaContainer(
            DockerImageName.parse("confluentinc/cp-kafka:7.6.0"))
        .withReuse(true);

    @DynamicPropertySource
    static void kafkaProperties(DynamicPropertyRegistry registry) {
        registry.add("spring.kafka.bootstrap-servers", kafka::getBootstrapServers);
        // Consumer must use read_committed to see only committed messages
        registry.add("spring.kafka.consumer.isolation-level", () -> "read_committed");
        registry.add("spring.kafka.consumer.auto-offset-reset", () -> "earliest");
        registry.add("spring.kafka.consumer.enable-auto-commit", () -> "false");
    }
}

Configuring a Transactional Producer

@Configuration
public class KafkaTransactionalConfig {

    @Bean
    public ProducerFactory<String, String> transactionalProducerFactory(
            KafkaProperties kafkaProperties) {
        Map<String, Object> props = new HashMap<>(kafkaProperties.buildProducerProperties());
        props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);
        props.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, "order-tx-");
        props.put(ProducerConfig.ACKS_CONFIG, "all");
        props.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, 5);
        return new DefaultKafkaProducerFactory<>(props);
    }

    @Bean
    public KafkaTemplate<String, String> transactionalKafkaTemplate(
            ProducerFactory<String, String> factory) {
        KafkaTemplate<String, String> template = new KafkaTemplate<>(factory);
        template.setTransactionIdPrefix("order-tx-");
        return template;
    }
}

A Transactional Service Under Test

@Service
public class OrderProcessingService {

    private final KafkaTemplate<String, String> kafkaTemplate;
    private final ObjectMapper objectMapper;

    public OrderProcessingService(KafkaTemplate<String, String> kafkaTemplate,
                                   ObjectMapper objectMapper) {
        this.kafkaTemplate = kafkaTemplate;
        this.objectMapper = objectMapper;
    }

    /**
     * Atomically sends to two topics. Both succeed or neither does.
     */
    public void processOrder(String orderId, String customerId, BigDecimal amount) {
        kafkaTemplate.executeInTransaction(ops -> {
            ops.send("orders.created", orderId, 
                toJson(Map.of("orderId", orderId, "amount", amount)));
            ops.send("inventory.reserve", orderId,
                toJson(Map.of("orderId", orderId, "customerId", customerId)));
            return null;
        });
    }

    private String toJson(Object obj) {
        try {
            return objectMapper.writeValueAsString(obj);
        } catch (JsonProcessingException e) {
            throw new RuntimeException(e);
        }
    }
}

Test: Atomic Commit Across Topics

The core EOS test verifies that when a transaction commits, ALL messages appear in ALL topics, and a read-committed consumer sees them atomically:

@Test
void shouldAtomicallyPublishToMultipleTopics() throws Exception {
    // Arrange
    List<String> orderEvents = new CopyOnWriteArrayList<>();
    List<String> inventoryEvents = new CopyOnWriteArrayList<>();
    CountDownLatch latch = new CountDownLatch(2);

    // Subscribe read-committed consumer to both topics
    startReadCommittedConsumer("orders.created", orderEvents, latch);
    startReadCommittedConsumer("inventory.reserve", inventoryEvents, latch);

    // Act
    orderProcessingService.processOrder("ORD-001", "CUST-1", new BigDecimal("99.99"));

    // Assert — both messages must appear
    assertThat(latch.await(15, TimeUnit.SECONDS)).isTrue();
    assertThat(orderEvents).hasSize(1);
    assertThat(inventoryEvents).hasSize(1);

    // Verify message content
    Map<?, ?> orderMsg = objectMapper.readValue(orderEvents.get(0), Map.class);
    assertThat(orderMsg).containsEntry("orderId", "ORD-001");
}

Test: Aborted Transaction Is Not Visible

This is the most important EOS test. If a transaction aborts (due to an exception), read-committed consumers must not see the partial messages:

@Service
public class FaultyOrderService {

    private final KafkaTemplate<String, String> kafkaTemplate;

    public void processOrderWithFailure(String orderId) {
        kafkaTemplate.executeInTransaction(ops -> {
            ops.send("orders.created", orderId, "{\"orderId\":\"" + orderId + "\"}");
            // Simulate failure halfway through
            throw new RuntimeException("Simulated processing failure");
        });
    }
}
@Test
void shouldNotPublishMessagesWhenTransactionAborts() throws Exception {
    // Arrange
    List<String> receivedMessages = new CopyOnWriteArrayList<>();
    AtomicBoolean messageReceived = new AtomicBoolean(false);
    startReadCommittedConsumer("orders.created", receivedMessages, new CountDownLatch(1));

    // Act — this will throw because the transaction aborts
    assertThatThrownBy(() -> faultyOrderService.processOrderWithFailure("ORD-FAIL"))
        .isInstanceOf(RuntimeException.class)
        .hasMessageContaining("Simulated processing failure");

    // Wait to confirm no message arrives (read-committed consumer sees nothing)
    Thread.sleep(3000); // Brief wait — we're asserting absence

    // Assert — aborted transaction messages must not be visible
    assertThat(receivedMessages).isEmpty();
}

Test: Idempotent Producer Does Not Duplicate on Retry

This test simulates a network glitch where the broker acknowledges a message but the producer does not receive the ack, triggering a retry:

@Test
void shouldNotDuplicateMessageOnProducerRetry() throws Exception {
    // Build a producer factory with aggressive retry settings
    Map<String, Object> props = new HashMap<>();
    props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, kafka.getBootstrapServers());
    props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);
    props.put(ProducerConfig.RETRIES_CONFIG, Integer.MAX_VALUE);
    props.put(ProducerConfig.ACKS_CONFIG, "all");
    props.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, 5);
    props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
    props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);

    KafkaProducer<String, String> producer = new KafkaProducer<>(props);

    // Send same message twice — idempotent producer should dedup at broker level
    // In practice, the producer tracks sequence numbers to prevent this
    ProducerRecord<String, String> record = 
        new ProducerRecord<>("idempotent-test", "KEY-1", "VALUE-1");
    
    producer.send(record).get(10, TimeUnit.SECONDS);

    // Verify broker received exactly one message
    Consumer<String, String> verifyConsumer = createReadCommittedConsumer("verify-group");
    verifyConsumer.subscribe(List.of("idempotent-test"));
    
    ConsumerRecords<String, String> records = 
        verifyConsumer.poll(Duration.ofSeconds(5));
    
    assertThat(records.count()).isEqualTo(1);
    producer.close();
    verifyConsumer.close();
}

Test: Consume-Transform-Produce Pattern

The most common real-world EOS use case is reading from one topic, transforming, and writing to another — atomically. Spring Kafka's KafkaTransactionManager handles this:

@Service
public class TransformationService {

    private final KafkaTemplate<String, String> kafkaTemplate;

    @Transactional
    @KafkaListener(topics = "raw-orders", groupId = "transformer-group",
                   containerFactory = "transactionalListenerFactory")
    public void transformAndForward(ConsumerRecord<String, String> record) {
        String transformed = record.value().toUpperCase(); // Simplified transform
        kafkaTemplate.send("processed-orders", record.key(), transformed);
        // Both the consumer offset commit AND the producer send happen in one transaction
    }
}

Testing this pattern:

@Test
void shouldAtomicallyConsumeTransformAndProduce() throws Exception {
    // Produce a raw message
    rawTemplate.send("raw-orders", "KEY-1", "create order for customer 123").get();

    // Wait for transformation
    List<String> processedMessages = new CopyOnWriteArrayList<>();
    CountDownLatch latch = new CountDownLatch(1);
    startReadCommittedConsumer("processed-orders", processedMessages, latch);

    assertThat(latch.await(20, TimeUnit.SECONDS)).isTrue();
    assertThat(processedMessages).hasSize(1);
    assertThat(processedMessages.get(0))
        .isEqualTo("CREATE ORDER FOR CUSTOMER 123");

    // Verify the consumer offset was committed (message won't be reprocessed on restart)
    long committedOffset = getCommittedOffset("transformer-group", "raw-orders", 0);
    assertThat(committedOffset).isEqualTo(1); // After first message
}

Detecting Silent EOS Misconfiguration

A common trap is having enable.idempotence=true on the producer but isolation.level=read_uncommitted on the consumer. You think you have EOS, but the consumer sees uncommitted messages. Write a test for this:

@Test
void shouldNotSeeMessagesFromAbortedTransactionWithReadCommittedIsolation() {
    // This test FAILS if isolation.level is misconfigured to read_uncommitted
    Consumer<String, String> readUncommittedConsumer = createConsumer(
        "read_uncommitted", "test-group-uc");
    Consumer<String, String> readCommittedConsumer = createConsumer(
        "read_committed", "test-group-rc");

    // Start a transaction but DO NOT commit
    KafkaProducer<String, String> txProducer = createTransactionalProducer();
    txProducer.beginTransaction();
    txProducer.send(new ProducerRecord<>("eos-test", "KEY", "SHOULD-NOT-BE-SEEN"));
    // Don't commit — simulate crash before commit

    // read_uncommitted consumer sees it (this is expected behavior)
    readUncommittedConsumer.subscribe(List.of("eos-test"));
    ConsumerRecords<?, ?> uncommitted = readUncommittedConsumer.poll(Duration.ofSeconds(3));

    // read_committed consumer does NOT see it
    readCommittedConsumer.subscribe(List.of("eos-test"));
    ConsumerRecords<?, ?> committed = readCommittedConsumer.poll(Duration.ofSeconds(3));

    // This assertion proves your consumer has the correct isolation level
    assertThat(committed.count()).isEqualTo(0);

    txProducer.abortTransaction();
    txProducer.close();
}

Key Properties Checklist

Before calling your EOS configuration complete, verify all of these in your test environment:

Property Required Value Why
enable.idempotence true Prevents producer-level duplicates
acks all Required for idempotence
max.in.flight.requests.per.connection ≤ 5 Required for idempotence
transactional.id Set Enables transactional API
isolation.level (consumer) read_committed Hides uncommitted messages
enable.auto.commit (consumer) false Manual offset management for EOS

Key Takeaways

Testing exactly-once semantics requires more than verifying that messages arrive. You need tests that:

  • Confirm aborted transactions produce zero visible messages to read-committed consumers
  • Verify atomic cross-topic commits appear together or not at all
  • Validate that consume-transform-produce pipelines commit offsets and output messages in the same transaction
  • Detect misconfiguration by asserting isolation level behavior explicitly

The failure mode for misconfigured EOS is silent — your system appears to work, but under network partitions or broker restarts, duplicates appear. Tests written against a real Kafka broker via Testcontainers are the only reliable way to catch this class of bug before it reaches production.

Read more

Start now free