Testing Exactly-Once Semantics in Kafka: A Practical Guide
Exactly-once semantics (EOS) in Kafka is one of the most misunderstood guarantees in distributed systems. Developers often assume that enabling enable.idempotence=true on the producer is sufficient, but EOS requires a coordinated setup across producers, brokers, and consumers. More importantly, it needs to be tested — the configuration is subtle, and silent misconfiguration leads to duplicate processing in production that only surfaces under specific failure conditions.
This guide covers how to write tests that validate exactly-once behavior end-to-end, including what to test, what tools to use, and how to simulate the failure scenarios that actually trigger duplicates.
What Exactly-Once Actually Means
Kafka's exactly-once guarantee has three layers:
- Idempotent producer: A single producer session will not duplicate messages on broker retries. Enabled by
enable.idempotence=true. - Transactional producer: Messages sent across multiple topics/partitions are committed atomically. Enabled by setting
transactional.id. - Read-committed consumers: Consumers only see committed transactional messages. Enabled by
isolation.level=read_committed.
Testing EOS means verifying all three layers interact correctly, especially under retry and failure scenarios.
Test Setup
You need a real Kafka broker to test transactions — embedded brokers have limited transaction support.
@SpringBootTest
@Testcontainers
public abstract class ExactlyOnceTestBase {
@Container
static final KafkaContainer kafka = new KafkaContainer(
DockerImageName.parse("confluentinc/cp-kafka:7.6.0"))
.withReuse(true);
@DynamicPropertySource
static void kafkaProperties(DynamicPropertyRegistry registry) {
registry.add("spring.kafka.bootstrap-servers", kafka::getBootstrapServers);
// Consumer must use read_committed to see only committed messages
registry.add("spring.kafka.consumer.isolation-level", () -> "read_committed");
registry.add("spring.kafka.consumer.auto-offset-reset", () -> "earliest");
registry.add("spring.kafka.consumer.enable-auto-commit", () -> "false");
}
}Configuring a Transactional Producer
@Configuration
public class KafkaTransactionalConfig {
@Bean
public ProducerFactory<String, String> transactionalProducerFactory(
KafkaProperties kafkaProperties) {
Map<String, Object> props = new HashMap<>(kafkaProperties.buildProducerProperties());
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);
props.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, "order-tx-");
props.put(ProducerConfig.ACKS_CONFIG, "all");
props.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, 5);
return new DefaultKafkaProducerFactory<>(props);
}
@Bean
public KafkaTemplate<String, String> transactionalKafkaTemplate(
ProducerFactory<String, String> factory) {
KafkaTemplate<String, String> template = new KafkaTemplate<>(factory);
template.setTransactionIdPrefix("order-tx-");
return template;
}
}A Transactional Service Under Test
@Service
public class OrderProcessingService {
private final KafkaTemplate<String, String> kafkaTemplate;
private final ObjectMapper objectMapper;
public OrderProcessingService(KafkaTemplate<String, String> kafkaTemplate,
ObjectMapper objectMapper) {
this.kafkaTemplate = kafkaTemplate;
this.objectMapper = objectMapper;
}
/**
* Atomically sends to two topics. Both succeed or neither does.
*/
public void processOrder(String orderId, String customerId, BigDecimal amount) {
kafkaTemplate.executeInTransaction(ops -> {
ops.send("orders.created", orderId,
toJson(Map.of("orderId", orderId, "amount", amount)));
ops.send("inventory.reserve", orderId,
toJson(Map.of("orderId", orderId, "customerId", customerId)));
return null;
});
}
private String toJson(Object obj) {
try {
return objectMapper.writeValueAsString(obj);
} catch (JsonProcessingException e) {
throw new RuntimeException(e);
}
}
}Test: Atomic Commit Across Topics
The core EOS test verifies that when a transaction commits, ALL messages appear in ALL topics, and a read-committed consumer sees them atomically:
@Test
void shouldAtomicallyPublishToMultipleTopics() throws Exception {
// Arrange
List<String> orderEvents = new CopyOnWriteArrayList<>();
List<String> inventoryEvents = new CopyOnWriteArrayList<>();
CountDownLatch latch = new CountDownLatch(2);
// Subscribe read-committed consumer to both topics
startReadCommittedConsumer("orders.created", orderEvents, latch);
startReadCommittedConsumer("inventory.reserve", inventoryEvents, latch);
// Act
orderProcessingService.processOrder("ORD-001", "CUST-1", new BigDecimal("99.99"));
// Assert — both messages must appear
assertThat(latch.await(15, TimeUnit.SECONDS)).isTrue();
assertThat(orderEvents).hasSize(1);
assertThat(inventoryEvents).hasSize(1);
// Verify message content
Map<?, ?> orderMsg = objectMapper.readValue(orderEvents.get(0), Map.class);
assertThat(orderMsg).containsEntry("orderId", "ORD-001");
}Test: Aborted Transaction Is Not Visible
This is the most important EOS test. If a transaction aborts (due to an exception), read-committed consumers must not see the partial messages:
@Service
public class FaultyOrderService {
private final KafkaTemplate<String, String> kafkaTemplate;
public void processOrderWithFailure(String orderId) {
kafkaTemplate.executeInTransaction(ops -> {
ops.send("orders.created", orderId, "{\"orderId\":\"" + orderId + "\"}");
// Simulate failure halfway through
throw new RuntimeException("Simulated processing failure");
});
}
}@Test
void shouldNotPublishMessagesWhenTransactionAborts() throws Exception {
// Arrange
List<String> receivedMessages = new CopyOnWriteArrayList<>();
AtomicBoolean messageReceived = new AtomicBoolean(false);
startReadCommittedConsumer("orders.created", receivedMessages, new CountDownLatch(1));
// Act — this will throw because the transaction aborts
assertThatThrownBy(() -> faultyOrderService.processOrderWithFailure("ORD-FAIL"))
.isInstanceOf(RuntimeException.class)
.hasMessageContaining("Simulated processing failure");
// Wait to confirm no message arrives (read-committed consumer sees nothing)
Thread.sleep(3000); // Brief wait — we're asserting absence
// Assert — aborted transaction messages must not be visible
assertThat(receivedMessages).isEmpty();
}Test: Idempotent Producer Does Not Duplicate on Retry
This test simulates a network glitch where the broker acknowledges a message but the producer does not receive the ack, triggering a retry:
@Test
void shouldNotDuplicateMessageOnProducerRetry() throws Exception {
// Build a producer factory with aggressive retry settings
Map<String, Object> props = new HashMap<>();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, kafka.getBootstrapServers());
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);
props.put(ProducerConfig.RETRIES_CONFIG, Integer.MAX_VALUE);
props.put(ProducerConfig.ACKS_CONFIG, "all");
props.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, 5);
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
KafkaProducer<String, String> producer = new KafkaProducer<>(props);
// Send same message twice — idempotent producer should dedup at broker level
// In practice, the producer tracks sequence numbers to prevent this
ProducerRecord<String, String> record =
new ProducerRecord<>("idempotent-test", "KEY-1", "VALUE-1");
producer.send(record).get(10, TimeUnit.SECONDS);
// Verify broker received exactly one message
Consumer<String, String> verifyConsumer = createReadCommittedConsumer("verify-group");
verifyConsumer.subscribe(List.of("idempotent-test"));
ConsumerRecords<String, String> records =
verifyConsumer.poll(Duration.ofSeconds(5));
assertThat(records.count()).isEqualTo(1);
producer.close();
verifyConsumer.close();
}Test: Consume-Transform-Produce Pattern
The most common real-world EOS use case is reading from one topic, transforming, and writing to another — atomically. Spring Kafka's KafkaTransactionManager handles this:
@Service
public class TransformationService {
private final KafkaTemplate<String, String> kafkaTemplate;
@Transactional
@KafkaListener(topics = "raw-orders", groupId = "transformer-group",
containerFactory = "transactionalListenerFactory")
public void transformAndForward(ConsumerRecord<String, String> record) {
String transformed = record.value().toUpperCase(); // Simplified transform
kafkaTemplate.send("processed-orders", record.key(), transformed);
// Both the consumer offset commit AND the producer send happen in one transaction
}
}Testing this pattern:
@Test
void shouldAtomicallyConsumeTransformAndProduce() throws Exception {
// Produce a raw message
rawTemplate.send("raw-orders", "KEY-1", "create order for customer 123").get();
// Wait for transformation
List<String> processedMessages = new CopyOnWriteArrayList<>();
CountDownLatch latch = new CountDownLatch(1);
startReadCommittedConsumer("processed-orders", processedMessages, latch);
assertThat(latch.await(20, TimeUnit.SECONDS)).isTrue();
assertThat(processedMessages).hasSize(1);
assertThat(processedMessages.get(0))
.isEqualTo("CREATE ORDER FOR CUSTOMER 123");
// Verify the consumer offset was committed (message won't be reprocessed on restart)
long committedOffset = getCommittedOffset("transformer-group", "raw-orders", 0);
assertThat(committedOffset).isEqualTo(1); // After first message
}Detecting Silent EOS Misconfiguration
A common trap is having enable.idempotence=true on the producer but isolation.level=read_uncommitted on the consumer. You think you have EOS, but the consumer sees uncommitted messages. Write a test for this:
@Test
void shouldNotSeeMessagesFromAbortedTransactionWithReadCommittedIsolation() {
// This test FAILS if isolation.level is misconfigured to read_uncommitted
Consumer<String, String> readUncommittedConsumer = createConsumer(
"read_uncommitted", "test-group-uc");
Consumer<String, String> readCommittedConsumer = createConsumer(
"read_committed", "test-group-rc");
// Start a transaction but DO NOT commit
KafkaProducer<String, String> txProducer = createTransactionalProducer();
txProducer.beginTransaction();
txProducer.send(new ProducerRecord<>("eos-test", "KEY", "SHOULD-NOT-BE-SEEN"));
// Don't commit — simulate crash before commit
// read_uncommitted consumer sees it (this is expected behavior)
readUncommittedConsumer.subscribe(List.of("eos-test"));
ConsumerRecords<?, ?> uncommitted = readUncommittedConsumer.poll(Duration.ofSeconds(3));
// read_committed consumer does NOT see it
readCommittedConsumer.subscribe(List.of("eos-test"));
ConsumerRecords<?, ?> committed = readCommittedConsumer.poll(Duration.ofSeconds(3));
// This assertion proves your consumer has the correct isolation level
assertThat(committed.count()).isEqualTo(0);
txProducer.abortTransaction();
txProducer.close();
}Key Properties Checklist
Before calling your EOS configuration complete, verify all of these in your test environment:
| Property | Required Value | Why |
|---|---|---|
enable.idempotence |
true |
Prevents producer-level duplicates |
acks |
all |
Required for idempotence |
max.in.flight.requests.per.connection |
≤ 5 |
Required for idempotence |
transactional.id |
Set | Enables transactional API |
isolation.level (consumer) |
read_committed |
Hides uncommitted messages |
enable.auto.commit (consumer) |
false |
Manual offset management for EOS |
Key Takeaways
Testing exactly-once semantics requires more than verifying that messages arrive. You need tests that:
- Confirm aborted transactions produce zero visible messages to read-committed consumers
- Verify atomic cross-topic commits appear together or not at all
- Validate that consume-transform-produce pipelines commit offsets and output messages in the same transaction
- Detect misconfiguration by asserting isolation level behavior explicitly
The failure mode for misconfigured EOS is silent — your system appears to work, but under network partitions or broker restarts, duplicates appear. Tests written against a real Kafka broker via Testcontainers are the only reliable way to catch this class of bug before it reaches production.