Kafka Testing Guide: From Unit Tests to End-to-End Verification

Kafka Testing Guide: From Unit Tests to End-to-End Verification

Kafka is hard to test because it's inherently asynchronous, distributed, and stateful. A message produced now might be consumed milliseconds or hours later. Consumer group offsets persist across restarts. Schema evolution affects producers and consumers independently. "It worked in dev" means nothing in a system where timing is everything.

This guide covers practical Kafka testing strategies for Java, Python, and Node.js teams.

What to Test in Kafka Systems

Testing a Kafka-based system means testing several distinct concerns:

  • Producer correctness: Are messages produced to the right topic with the right key and value?
  • Consumer correctness: Does the consumer process each message type correctly?
  • Consumer offset handling: Does the consumer commit offsets only after successful processing?
  • Error handling: What happens when a message can't be processed?
  • Schema evolution: Do consumers handle messages from older or newer schema versions?
  • Ordering guarantees: Are messages processed in the correct partition order?
  • Exactly-once semantics: Are messages processed exactly once under failure scenarios?

Unit Testing Producers and Consumers

The fastest tests mock the Kafka client and test just the message logic:

Java Producer Unit Test

// OrderEventProducer.java
@Service
public class OrderEventProducer {
    private final KafkaTemplate<String, OrderEvent> kafkaTemplate;
    
    public OrderEventProducer(KafkaTemplate<String, OrderEvent> kafkaTemplate) {
        this.kafkaTemplate = kafkaTemplate;
    }
    
    public CompletableFuture<SendResult<String, OrderEvent>> publishOrderCreated(Order order) {
        OrderEvent event = OrderEvent.builder()
            .eventType("ORDER_CREATED")
            .orderId(order.getId())
            .customerId(order.getCustomerId())
            .total(order.getTotal())
            .timestamp(Instant.now())
            .build();
        
        return kafkaTemplate.send("orders", order.getId().toString(), event);
    }
}

// OrderEventProducerTest.java
@ExtendWith(MockitoExtension.class)
class OrderEventProducerTest {
    
    @Mock
    private KafkaTemplate<String, OrderEvent> kafkaTemplate;
    
    @InjectMocks
    private OrderEventProducer producer;
    
    @Test
    void publishOrderCreated_sendsToCorrectTopic() {
        Order order = Order.builder()
            .id(123L)
            .customerId(456L)
            .total(new BigDecimal("99.99"))
            .build();
        
        when(kafkaTemplate.send(any(), any(), any()))
            .thenReturn(CompletableFuture.completedFuture(mock(SendResult.class)));
        
        producer.publishOrderCreated(order);
        
        ArgumentCaptor<String> topicCaptor = ArgumentCaptor.forClass(String.class);
        ArgumentCaptor<String> keyCaptor = ArgumentCaptor.forClass(String.class);
        ArgumentCaptor<OrderEvent> valueCaptor = ArgumentCaptor.forClass(OrderEvent.class);
        
        verify(kafkaTemplate).send(
            topicCaptor.capture(),
            keyCaptor.capture(),
            valueCaptor.capture()
        );
        
        assertThat(topicCaptor.getValue()).isEqualTo("orders");
        assertThat(keyCaptor.getValue()).isEqualTo("123");
        assertThat(valueCaptor.getValue().getEventType()).isEqualTo("ORDER_CREATED");
        assertThat(valueCaptor.getValue().getOrderId()).isEqualTo(123L);
        assertThat(valueCaptor.getValue().getTotal()).isEqualTo(new BigDecimal("99.99"));
    }
}

Python Consumer Unit Test

from unittest.mock import MagicMock, patch
import pytest
from your_app.consumers import OrderConsumer

class TestOrderConsumer:
    def test_processes_order_created_event(self):
        """Consumer correctly handles ORDER_CREATED event."""
        mock_order_service = MagicMock()
        consumer = OrderConsumer(order_service=mock_order_service)
        
        message = MagicMock()
        message.key.return_value = b"123"
        message.value.return_value = {
            "event_type": "ORDER_CREATED",
            "order_id": 123,
            "customer_id": 456,
            "total": "99.99"
        }
        
        consumer.process(message)
        
        mock_order_service.create_order.assert_called_once_with(
            order_id=123,
            customer_id=456,
            total="99.99"
        )
    
    def test_ignores_unknown_event_types(self):
        """Unknown event types are ignored without error."""
        mock_order_service = MagicMock()
        consumer = OrderConsumer(order_service=mock_order_service)
        
        message = MagicMock()
        message.value.return_value = {"event_type": "UNKNOWN_EVENT"}
        
        # Should not raise
        consumer.process(message)
        mock_order_service.create_order.assert_not_called()
    
    def test_does_not_commit_on_processing_failure(self):
        """Offset is not committed if message processing fails."""
        mock_order_service = MagicMock()
        mock_order_service.create_order.side_effect = RuntimeError("DB unavailable")
        consumer = OrderConsumer(order_service=mock_order_service)
        
        message = MagicMock()
        message.value.return_value = {"event_type": "ORDER_CREATED", "order_id": 1}
        
        with pytest.raises(RuntimeError):
            consumer.process(message)
        
        # Offset should NOT be committed on failure
        message.offset.assert_not_called()

Integration Testing with EmbeddedKafka

For integration tests, use an in-process Kafka:

Spring Boot with EmbeddedKafka

@SpringBootTest
@EmbeddedKafka(
    partitions = 1,
    topics = {"orders", "order-events"},
    brokerProperties = {
        "listeners=PLAINTEXT://localhost:9099",
        "port=9099"
    }
)
@TestPropertySource(properties = {
    "spring.kafka.bootstrap-servers=${spring.embedded.kafka.brokers}",
    "spring.kafka.consumer.auto-offset-reset=earliest"
})
class OrderEventIntegrationTest {
    
    @Autowired
    private OrderEventProducer producer;
    
    @Autowired
    private KafkaConsumer<String, String> testConsumer;
    
    @Test
    void orderCreated_eventIsPublishedAndConsumed() throws Exception {
        Order order = Order.builder()
            .id(123L)
            .customerId(456L)
            .total(new BigDecimal("99.99"))
            .build();
        
        producer.publishOrderCreated(order);
        
        // Wait for consumer to receive message
        ConsumerRecords<String, String> records = KafkaTestUtils.getRecords(
            testConsumer,
            Duration.ofSeconds(5)
        );
        
        assertThat(records.count()).isEqualTo(1);
        ConsumerRecord<String, String> record = records.iterator().next();
        assertThat(record.key()).isEqualTo("123");
        assertThat(record.value()).contains("ORDER_CREATED");
    }
}

Python with pytest-kafka or Testcontainers

# conftest.py
import pytest
from testcontainers.kafka import KafkaContainer
from kafka import KafkaProducer, KafkaConsumer
import json

@pytest.fixture(scope="session")
def kafka_container():
    with KafkaContainer("confluentinc/cp-kafka:7.5.0") as kafka:
        yield kafka

@pytest.fixture
def kafka_bootstrap(kafka_container):
    return kafka_container.get_bootstrap_server()

@pytest.fixture
def producer(kafka_bootstrap):
    p = KafkaProducer(
        bootstrap_servers=kafka_bootstrap,
        value_serializer=lambda v: json.dumps(v).encode('utf-8'),
        key_serializer=lambda k: k.encode('utf-8') if k else None
    )
    yield p
    p.close()

def make_consumer(bootstrap, topic, group_id="test-group"):
    return KafkaConsumer(
        topic,
        bootstrap_servers=bootstrap,
        value_deserializer=lambda v: json.loads(v.decode('utf-8')),
        auto_offset_reset='earliest',
        group_id=group_id,
        consumer_timeout_ms=5000
    )
def test_order_event_published_to_correct_topic(producer, kafka_bootstrap):
    """ORDER_CREATED event is published with correct key and value."""
    consumer = make_consumer(kafka_bootstrap, "orders")
    
    producer.send("orders", key="123", value={
        "event_type": "ORDER_CREATED",
        "order_id": 123,
        "customer_id": 456
    })
    producer.flush()
    
    messages = list(consumer)
    consumer.close()
    
    assert len(messages) == 1
    assert messages[0].key == b"123"
    assert messages[0].value["event_type"] == "ORDER_CREATED"
    assert messages[0].value["order_id"] == 123

def test_consumer_processes_messages_in_order(producer, kafka_bootstrap):
    """Messages in a single partition are consumed in order."""
    consumer = make_consumer(kafka_bootstrap, "ordered-events", group_id="order-test")
    
    # Send 5 messages to same partition (same key)
    for i in range(5):
        producer.send("ordered-events", key="user:1", value={"seq": i})
    producer.flush()
    
    messages = list(consumer)
    consumer.close()
    
    sequences = [m.value["seq"] for m in messages]
    assert sequences == [0, 1, 2, 3, 4], f"Messages out of order: {sequences}"

Testing Dead Letter Queues

DLQ handling is critical — test that failed messages land in the DLQ correctly:

def test_failed_messages_go_to_dlq(producer, kafka_bootstrap, mock_db_that_fails):
    """Messages that fail processing are routed to the DLQ topic."""
    dlq_consumer = make_consumer(kafka_bootstrap, "orders.DLT")
    
    # Send a message that will fail processing
    producer.send("orders", key="bad-order", value={
        "event_type": "ORDER_CREATED",
        "order_id": None,  # Invalid — will cause processing failure
    })
    producer.flush()
    
    # Start consumer (will fail and route to DLQ)
    order_consumer = OrderConsumer(bootstrap_servers=kafka_bootstrap)
    order_consumer.start_once()  # Process one batch
    
    # Verify message appeared in DLQ
    dlq_messages = list(dlq_consumer)
    dlq_consumer.close()
    
    assert len(dlq_messages) == 1
    assert dlq_messages[0].headers  # DLQ messages should have error headers
    error_header = dict(dlq_messages[0].headers).get(b"kafka_dlt-exception-message")
    assert error_header is not None

Schema Registry Testing

If you use Confluent Schema Registry, test schema compatibility:

from confluent_kafka.schema_registry import SchemaRegistryClient
from confluent_kafka.schema_registry.avro import AvroSerializer

def test_schema_backward_compatible(schema_registry_url):
    """New schema version is backward compatible with consumers on old version."""
    client = SchemaRegistryClient({"url": schema_registry_url})
    
    # Register new schema version
    new_schema = """
    {
      "type": "record",
      "name": "OrderEvent",
      "fields": [
        {"name": "order_id", "type": "long"},
        {"name": "customer_id", "type": "long"},
        {"name": "total", "type": "string"},
        {"name": "currency", "type": "string", "default": "USD"}  // new field with default
      ]
    }
    """
    
    # This should succeed (backward compatible — new field has default)
    schema_id = client.register_schema(
        "orders-value",
        Schema(new_schema, "AVRO")
    )
    assert schema_id > 0

def test_schema_breaking_change_is_rejected(schema_registry_url):
    """Breaking schema change is rejected by registry."""
    client = SchemaRegistryClient({"url": schema_registry_url})
    
    breaking_schema = """
    {
      "type": "record",
      "name": "OrderEvent",
      "fields": [
        {"name": "order_id", "type": "long"},
        {"name": "customer_id", "type": "string"}  // type changed from long to string — BREAKING
      ]
    }
    """
    
    with pytest.raises(SchemaRegistryError, match="INCOMPATIBLE_SCHEMA"):
        client.register_schema("orders-value", Schema(breaking_schema, "AVRO"))

CI/CD Integration

# .github/workflows/kafka-tests.yml
name: Kafka Integration Tests

on: [push, pull_request]

jobs:
  kafka-tests:
    runs-on: ubuntu-latest

    services:
      zookeeper:
        image: confluentinc/cp-zookeeper:7.5.0
        env:
          ZOOKEEPER_CLIENT_PORT: 2181
        ports:
          - 2181:2181
      
      kafka:
        image: confluentinc/cp-kafka:7.5.0
        env:
          KAFKA_BROKER_ID: 1
          KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
          KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
          KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
        ports:
          - 9092:9092

    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'

      - run: pip install pytest kafka-python confluent-kafka testcontainers

      - name: Wait for Kafka
        run: |
          timeout 60 bash -c 'until echo "" | nc -z localhost 9092; do sleep 1; done'

      - name: Run Kafka tests
        run: pytest tests/kafka/ -v
        env:
          KAFKA_BOOTSTRAP_SERVERS: localhost:9092

Common Kafka Testing Mistakes

No timeout on consumer reads: A consumer that never receives a message will hang forever. Always set consumer_timeout_ms in tests.

Not testing offset commits: The most common Kafka bug is committing offsets before processing succeeds. Test that your consumer only commits after the business logic succeeds.

Sharing consumer groups between tests: Two tests with the same group_id will split messages between them. Use unique group IDs per test or per test run.

Not testing DLQ routing: DLQ logic is often untested until a production incident. Always test failure scenarios.

Testing only happy paths: Kafka consumers must handle malformed messages, schema mismatches, and transient failures gracefully.


Kafka testing requires testing at multiple levels: unit tests for message logic, integration tests with real Kafka for consumer offset and ordering behavior, and schema compatibility tests if you use a schema registry. The patterns here give you confidence that your event-driven system behaves correctly under both normal and failure conditions.

Read more

Start now free