Kafka Testing Guide: From Unit Tests to End-to-End Verification
Kafka is hard to test because it's inherently asynchronous, distributed, and stateful. A message produced now might be consumed milliseconds or hours later. Consumer group offsets persist across restarts. Schema evolution affects producers and consumers independently. "It worked in dev" means nothing in a system where timing is everything.
This guide covers practical Kafka testing strategies for Java, Python, and Node.js teams.
What to Test in Kafka Systems
Testing a Kafka-based system means testing several distinct concerns:
- Producer correctness: Are messages produced to the right topic with the right key and value?
- Consumer correctness: Does the consumer process each message type correctly?
- Consumer offset handling: Does the consumer commit offsets only after successful processing?
- Error handling: What happens when a message can't be processed?
- Schema evolution: Do consumers handle messages from older or newer schema versions?
- Ordering guarantees: Are messages processed in the correct partition order?
- Exactly-once semantics: Are messages processed exactly once under failure scenarios?
Unit Testing Producers and Consumers
The fastest tests mock the Kafka client and test just the message logic:
Java Producer Unit Test
// OrderEventProducer.java
@Service
public class OrderEventProducer {
private final KafkaTemplate<String, OrderEvent> kafkaTemplate;
public OrderEventProducer(KafkaTemplate<String, OrderEvent> kafkaTemplate) {
this.kafkaTemplate = kafkaTemplate;
}
public CompletableFuture<SendResult<String, OrderEvent>> publishOrderCreated(Order order) {
OrderEvent event = OrderEvent.builder()
.eventType("ORDER_CREATED")
.orderId(order.getId())
.customerId(order.getCustomerId())
.total(order.getTotal())
.timestamp(Instant.now())
.build();
return kafkaTemplate.send("orders", order.getId().toString(), event);
}
}
// OrderEventProducerTest.java
@ExtendWith(MockitoExtension.class)
class OrderEventProducerTest {
@Mock
private KafkaTemplate<String, OrderEvent> kafkaTemplate;
@InjectMocks
private OrderEventProducer producer;
@Test
void publishOrderCreated_sendsToCorrectTopic() {
Order order = Order.builder()
.id(123L)
.customerId(456L)
.total(new BigDecimal("99.99"))
.build();
when(kafkaTemplate.send(any(), any(), any()))
.thenReturn(CompletableFuture.completedFuture(mock(SendResult.class)));
producer.publishOrderCreated(order);
ArgumentCaptor<String> topicCaptor = ArgumentCaptor.forClass(String.class);
ArgumentCaptor<String> keyCaptor = ArgumentCaptor.forClass(String.class);
ArgumentCaptor<OrderEvent> valueCaptor = ArgumentCaptor.forClass(OrderEvent.class);
verify(kafkaTemplate).send(
topicCaptor.capture(),
keyCaptor.capture(),
valueCaptor.capture()
);
assertThat(topicCaptor.getValue()).isEqualTo("orders");
assertThat(keyCaptor.getValue()).isEqualTo("123");
assertThat(valueCaptor.getValue().getEventType()).isEqualTo("ORDER_CREATED");
assertThat(valueCaptor.getValue().getOrderId()).isEqualTo(123L);
assertThat(valueCaptor.getValue().getTotal()).isEqualTo(new BigDecimal("99.99"));
}
}Python Consumer Unit Test
from unittest.mock import MagicMock, patch
import pytest
from your_app.consumers import OrderConsumer
class TestOrderConsumer:
def test_processes_order_created_event(self):
"""Consumer correctly handles ORDER_CREATED event."""
mock_order_service = MagicMock()
consumer = OrderConsumer(order_service=mock_order_service)
message = MagicMock()
message.key.return_value = b"123"
message.value.return_value = {
"event_type": "ORDER_CREATED",
"order_id": 123,
"customer_id": 456,
"total": "99.99"
}
consumer.process(message)
mock_order_service.create_order.assert_called_once_with(
order_id=123,
customer_id=456,
total="99.99"
)
def test_ignores_unknown_event_types(self):
"""Unknown event types are ignored without error."""
mock_order_service = MagicMock()
consumer = OrderConsumer(order_service=mock_order_service)
message = MagicMock()
message.value.return_value = {"event_type": "UNKNOWN_EVENT"}
# Should not raise
consumer.process(message)
mock_order_service.create_order.assert_not_called()
def test_does_not_commit_on_processing_failure(self):
"""Offset is not committed if message processing fails."""
mock_order_service = MagicMock()
mock_order_service.create_order.side_effect = RuntimeError("DB unavailable")
consumer = OrderConsumer(order_service=mock_order_service)
message = MagicMock()
message.value.return_value = {"event_type": "ORDER_CREATED", "order_id": 1}
with pytest.raises(RuntimeError):
consumer.process(message)
# Offset should NOT be committed on failure
message.offset.assert_not_called()Integration Testing with EmbeddedKafka
For integration tests, use an in-process Kafka:
Spring Boot with EmbeddedKafka
@SpringBootTest
@EmbeddedKafka(
partitions = 1,
topics = {"orders", "order-events"},
brokerProperties = {
"listeners=PLAINTEXT://localhost:9099",
"port=9099"
}
)
@TestPropertySource(properties = {
"spring.kafka.bootstrap-servers=${spring.embedded.kafka.brokers}",
"spring.kafka.consumer.auto-offset-reset=earliest"
})
class OrderEventIntegrationTest {
@Autowired
private OrderEventProducer producer;
@Autowired
private KafkaConsumer<String, String> testConsumer;
@Test
void orderCreated_eventIsPublishedAndConsumed() throws Exception {
Order order = Order.builder()
.id(123L)
.customerId(456L)
.total(new BigDecimal("99.99"))
.build();
producer.publishOrderCreated(order);
// Wait for consumer to receive message
ConsumerRecords<String, String> records = KafkaTestUtils.getRecords(
testConsumer,
Duration.ofSeconds(5)
);
assertThat(records.count()).isEqualTo(1);
ConsumerRecord<String, String> record = records.iterator().next();
assertThat(record.key()).isEqualTo("123");
assertThat(record.value()).contains("ORDER_CREATED");
}
}Python with pytest-kafka or Testcontainers
# conftest.py
import pytest
from testcontainers.kafka import KafkaContainer
from kafka import KafkaProducer, KafkaConsumer
import json
@pytest.fixture(scope="session")
def kafka_container():
with KafkaContainer("confluentinc/cp-kafka:7.5.0") as kafka:
yield kafka
@pytest.fixture
def kafka_bootstrap(kafka_container):
return kafka_container.get_bootstrap_server()
@pytest.fixture
def producer(kafka_bootstrap):
p = KafkaProducer(
bootstrap_servers=kafka_bootstrap,
value_serializer=lambda v: json.dumps(v).encode('utf-8'),
key_serializer=lambda k: k.encode('utf-8') if k else None
)
yield p
p.close()
def make_consumer(bootstrap, topic, group_id="test-group"):
return KafkaConsumer(
topic,
bootstrap_servers=bootstrap,
value_deserializer=lambda v: json.loads(v.decode('utf-8')),
auto_offset_reset='earliest',
group_id=group_id,
consumer_timeout_ms=5000
)def test_order_event_published_to_correct_topic(producer, kafka_bootstrap):
"""ORDER_CREATED event is published with correct key and value."""
consumer = make_consumer(kafka_bootstrap, "orders")
producer.send("orders", key="123", value={
"event_type": "ORDER_CREATED",
"order_id": 123,
"customer_id": 456
})
producer.flush()
messages = list(consumer)
consumer.close()
assert len(messages) == 1
assert messages[0].key == b"123"
assert messages[0].value["event_type"] == "ORDER_CREATED"
assert messages[0].value["order_id"] == 123
def test_consumer_processes_messages_in_order(producer, kafka_bootstrap):
"""Messages in a single partition are consumed in order."""
consumer = make_consumer(kafka_bootstrap, "ordered-events", group_id="order-test")
# Send 5 messages to same partition (same key)
for i in range(5):
producer.send("ordered-events", key="user:1", value={"seq": i})
producer.flush()
messages = list(consumer)
consumer.close()
sequences = [m.value["seq"] for m in messages]
assert sequences == [0, 1, 2, 3, 4], f"Messages out of order: {sequences}"Testing Dead Letter Queues
DLQ handling is critical — test that failed messages land in the DLQ correctly:
def test_failed_messages_go_to_dlq(producer, kafka_bootstrap, mock_db_that_fails):
"""Messages that fail processing are routed to the DLQ topic."""
dlq_consumer = make_consumer(kafka_bootstrap, "orders.DLT")
# Send a message that will fail processing
producer.send("orders", key="bad-order", value={
"event_type": "ORDER_CREATED",
"order_id": None, # Invalid — will cause processing failure
})
producer.flush()
# Start consumer (will fail and route to DLQ)
order_consumer = OrderConsumer(bootstrap_servers=kafka_bootstrap)
order_consumer.start_once() # Process one batch
# Verify message appeared in DLQ
dlq_messages = list(dlq_consumer)
dlq_consumer.close()
assert len(dlq_messages) == 1
assert dlq_messages[0].headers # DLQ messages should have error headers
error_header = dict(dlq_messages[0].headers).get(b"kafka_dlt-exception-message")
assert error_header is not NoneSchema Registry Testing
If you use Confluent Schema Registry, test schema compatibility:
from confluent_kafka.schema_registry import SchemaRegistryClient
from confluent_kafka.schema_registry.avro import AvroSerializer
def test_schema_backward_compatible(schema_registry_url):
"""New schema version is backward compatible with consumers on old version."""
client = SchemaRegistryClient({"url": schema_registry_url})
# Register new schema version
new_schema = """
{
"type": "record",
"name": "OrderEvent",
"fields": [
{"name": "order_id", "type": "long"},
{"name": "customer_id", "type": "long"},
{"name": "total", "type": "string"},
{"name": "currency", "type": "string", "default": "USD"} // new field with default
]
}
"""
# This should succeed (backward compatible — new field has default)
schema_id = client.register_schema(
"orders-value",
Schema(new_schema, "AVRO")
)
assert schema_id > 0
def test_schema_breaking_change_is_rejected(schema_registry_url):
"""Breaking schema change is rejected by registry."""
client = SchemaRegistryClient({"url": schema_registry_url})
breaking_schema = """
{
"type": "record",
"name": "OrderEvent",
"fields": [
{"name": "order_id", "type": "long"},
{"name": "customer_id", "type": "string"} // type changed from long to string — BREAKING
]
}
"""
with pytest.raises(SchemaRegistryError, match="INCOMPATIBLE_SCHEMA"):
client.register_schema("orders-value", Schema(breaking_schema, "AVRO"))CI/CD Integration
# .github/workflows/kafka-tests.yml
name: Kafka Integration Tests
on: [push, pull_request]
jobs:
kafka-tests:
runs-on: ubuntu-latest
services:
zookeeper:
image: confluentinc/cp-zookeeper:7.5.0
env:
ZOOKEEPER_CLIENT_PORT: 2181
ports:
- 2181:2181
kafka:
image: confluentinc/cp-kafka:7.5.0
env:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
ports:
- 9092:9092
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.12'
- run: pip install pytest kafka-python confluent-kafka testcontainers
- name: Wait for Kafka
run: |
timeout 60 bash -c 'until echo "" | nc -z localhost 9092; do sleep 1; done'
- name: Run Kafka tests
run: pytest tests/kafka/ -v
env:
KAFKA_BOOTSTRAP_SERVERS: localhost:9092Common Kafka Testing Mistakes
No timeout on consumer reads: A consumer that never receives a message will hang forever. Always set consumer_timeout_ms in tests.
Not testing offset commits: The most common Kafka bug is committing offsets before processing succeeds. Test that your consumer only commits after the business logic succeeds.
Sharing consumer groups between tests: Two tests with the same group_id will split messages between them. Use unique group IDs per test or per test run.
Not testing DLQ routing: DLQ logic is often untested until a production incident. Always test failure scenarios.
Testing only happy paths: Kafka consumers must handle malformed messages, schema mismatches, and transient failures gracefully.
Kafka testing requires testing at multiple levels: unit tests for message logic, integration tests with real Kafka for consumer offset and ordering behavior, and schema compatibility tests if you use a schema registry. The patterns here give you confidence that your event-driven system behaves correctly under both normal and failure conditions.