Testing Kafka Consumers and Producers with Testcontainers
Testcontainers lets you run a real Kafka broker inside your test suite — no mocks, no stubs, no "it works on my machine" surprises. This guide walks through spinning up Kafka in Java and Python tests, writing producer and consumer assertions, controlling offsets, and wiring everything into CI.
Key Takeaways
Use real Kafka, not mocks. EmbeddedKafka and mock producers hide serialization bugs, rebalance behavior, and partition assignment issues that only show up with a real broker.
Share one Kafka container per test class. Starting a new container per test takes 10–15 seconds each. A @Container with @TestContainers on the class level starts once and saves minutes on large suites.
Always set auto.offset.reset=earliest in tests. Without it, consumers miss messages produced before they subscribed — a common source of flaky tests.
Assert on consumed records, not on side effects. Poll in a loop with a deadline rather than sleeping a fixed duration. Tests become faster and deterministic.
Pin the Kafka image version in CI. Floating latest tags break pipelines when Confluent ships a new image with changed defaults.
Testing event-driven systems is hard. The problem is not writing the code — it is trusting it. A producer that "looks like it sends" and a consumer that "looks like it reads" can still fail silently in production because the serialization format drifted, the topic was misconfigured, or an offset reset swallowed half the messages. The only way to catch those failures before production is to test against a real Kafka broker.
Testcontainers makes this practical. It manages Docker containers as JUnit or pytest fixtures, starting them before your test and stopping them after. You get a real broker, real partitions, real consumer groups — and the test still runs on a laptop or in CI without any pre-installed infrastructure.
Why Not EmbeddedKafka?
Spring Kafka ships EmbeddedKafkaBroker. The Apache Kafka project has KafkaEmbedded. These are tempting because they start fast and require no Docker. The trade-off is fidelity.
Embedded brokers run in the same JVM as your test. They share the same classloader, skip certain network-layer checks, and sometimes paper over errors that a real broker would surface. Serialization bugs, schema registry interactions, and consumer group rebalancing all behave differently. A test that passes against an embedded broker has failed before in production for exactly this reason.
Testcontainers adds 5–15 seconds of startup time. In exchange, you get a broker that behaves identically to production.
Java Setup
Add the dependencies to pom.xml:
<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>kafka</artifactId>
<version>1.19.7</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>junit-jupiter</artifactId>
<version>1.19.7</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>3.7.0</version>
</dependency>A minimal test class that starts one Kafka container for all tests in the class:
@Testcontainers
class OrderEventProducerTest {
@Container
static final KafkaContainer kafka =
new KafkaContainer(DockerImageName.parse("confluentinc/cp-kafka:7.6.1"));
private KafkaProducer<String, String> producer;
private KafkaConsumer<String, String> consumer;
@BeforeEach
void setUp() {
producer = new KafkaProducer<>(Map.of(
ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, kafka.getBootstrapServers(),
ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName(),
ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName()
));
consumer = new KafkaConsumer<>(Map.of(
ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, kafka.getBootstrapServers(),
ConsumerConfig.GROUP_ID_CONFIG, "test-group-" + UUID.randomUUID(),
ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName(),
ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName(),
ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest"
));
}
@AfterEach
void tearDown() {
producer.close();
consumer.close();
}
}Two things to note here. The container is static — it starts once when the class loads and is reused across all tests. Using a random UUID for GROUP_ID_CONFIG ensures each test gets a fresh consumer group with no offset history.
Writing a Producer Test
Testing a producer means verifying that a message arrives on the correct topic with the expected key and value. The simplest approach: produce a message, consume it from the same broker, assert on what came back.
@Test
void producesOrderCreatedEvent() throws Exception {
String topic = "orders";
String orderId = "order-123";
String payload = "{\"orderId\":\"order-123\",\"status\":\"CREATED\"}";
// Produce
producer.send(new ProducerRecord<>(topic, orderId, payload)).get();
// Consume
consumer.subscribe(List.of(topic));
ConsumerRecords<String, String> records = pollWithTimeout(consumer, Duration.ofSeconds(10));
assertThat(records.count()).isEqualTo(1);
ConsumerRecord<String, String> record = records.iterator().next();
assertThat(record.key()).isEqualTo(orderId);
assertThat(record.value()).isEqualTo(payload);
}
private ConsumerRecords<String, String> pollWithTimeout(
KafkaConsumer<String, String> consumer,
Duration timeout
) {
Instant deadline = Instant.now().plus(timeout);
ConsumerRecords<String, String> records = ConsumerRecords.empty();
while (records.isEmpty() && Instant.now().isBefore(deadline)) {
records = consumer.poll(Duration.ofMillis(500));
}
return records;
}The pollWithTimeout helper is important. Never use Thread.sleep before polling. Sleep-based tests are fragile — they fail when the machine is slow and waste time when delivery is fast. The loop polls every 500ms and returns as soon as records arrive.
Writing a Consumer Test
Testing a consumer means verifying that your application logic handles an incoming message correctly. The typical pattern: produce a message onto the topic, let your consumer application read it, assert on the observable side effect.
Suppose you have a OrderConsumer that persists orders to a database:
@Test
void consumerPersistsOrderOnMessage() throws Exception {
String topic = "orders";
String payload = "{\"orderId\":\"order-456\",\"amount\":99.99}";
// Seed the topic
producer.send(new ProducerRecord<>(topic, "order-456", payload)).get();
// Start the consumer under test
OrderConsumer orderConsumer = new OrderConsumer(kafka.getBootstrapServers(), orderRepository);
orderConsumer.start();
// Assert side effect
await().atMost(10, SECONDS).untilAsserted(() ->
assertThat(orderRepository.findById("order-456")).isPresent()
);
orderConsumer.stop();
}This uses Awaitility (await().atMost(...)) for the assertion loop. Awaitility polls the assertion on a configurable schedule until it passes or times out — cleaner than writing the loop yourself.
Testing Offset Behavior
Offset handling is where consumer bugs hide. Common failure modes: the consumer starts from the wrong offset after a restart, or processes messages twice due to a missing commit.
Test the restart scenario explicitly:
@Test
void consumerResumesFromLastCommittedOffset() throws Exception {
String topic = "payments";
String groupId = "payment-processor";
// Produce 3 messages
for (int i = 0; i < 3; i++) {
producer.send(new ProducerRecord<>(topic, "payment-" + i,
"{\"id\":" + i + "}")).get();
}
// First consumer run — processes and commits after first 2 messages
try (KafkaConsumer<String, String> c1 = buildConsumer(groupId)) {
c1.subscribe(List.of(topic));
List<ConsumerRecord<String, String>> seen = new ArrayList<>();
Instant deadline = Instant.now().plusSeconds(10);
while (seen.size() < 2 && Instant.now().isBefore(deadline)) {
c1.poll(Duration.ofMillis(500)).forEach(seen::add);
}
// Commit after processing 2
c1.commitSync();
}
// Second consumer run — should pick up only the 3rd message
List<String> resumedKeys = new ArrayList<>();
try (KafkaConsumer<String, String> c2 = buildConsumer(groupId)) {
c2.subscribe(List.of(topic));
Instant deadline = Instant.now().plusSeconds(5);
while (Instant.now().isBefore(deadline)) {
c2.poll(Duration.ofMillis(500)).forEach(r -> resumedKeys.add(r.key()));
}
}
assertThat(resumedKeys).containsExactly("payment-2");
}This test catches bugs where enable.auto.commit=true silently commits before the message is processed, or where the consumer resets to earliest on every restart.
Python Setup
For Python services using confluent-kafka, Testcontainers Python provides the same container management:
# requirements-test.txt
testcontainers[kafka]==3.7.6
confluent-kafka==2.3.0
pytest==8.1.0import pytest
import json
from testcontainers.kafka import KafkaContainer
from confluent_kafka import Producer, Consumer, KafkaError
@pytest.fixture(scope="module")
def kafka():
with KafkaContainer("confluentinc/cp-kafka:7.6.1") as container:
yield container
def make_producer(bootstrap_servers: str) -> Producer:
return Producer({"bootstrap.servers": bootstrap_servers})
def make_consumer(bootstrap_servers: str, group_id: str) -> Consumer:
return Consumer({
"bootstrap.servers": bootstrap_servers,
"group.id": group_id,
"auto.offset.reset": "earliest",
"enable.auto.commit": False,
})The scope="module" on the fixture means one Kafka container per test module — the same startup-cost optimization as static in Java.
A producer test in Python:
def test_produces_user_event(kafka):
bootstrap = kafka.get_bootstrap_server()
topic = "user-events"
event = {"userId": "u-1", "action": "login"}
producer = make_producer(bootstrap)
producer.produce(topic, key="u-1", value=json.dumps(event))
producer.flush()
consumer = make_consumer(bootstrap, group_id="test-consumer")
consumer.subscribe([topic])
received = None
from datetime import datetime, timedelta
deadline = datetime.utcnow() + timedelta(seconds=10)
while datetime.utcnow() < deadline:
msg = consumer.poll(0.5)
if msg is None:
continue
if msg.error():
if msg.error().code() == KafkaError._PARTITION_EOF:
continue
raise Exception(msg.error())
received = json.loads(msg.value())
break
consumer.close()
assert received is not None
assert received["userId"] == "u-1"
assert received["action"] == "login"CI Configuration
The main concern in CI is Docker availability. GitHub Actions, GitLab CI, and most hosted runners support Docker out of the box.
A GitHub Actions workflow:
name: Kafka Tests
on: [push, pull_request]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-java@v4
with:
java-version: '21'
distribution: 'temurin'
- name: Cache Maven
uses: actions/cache@v4
with:
path: ~/.m2
key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
- name: Pull Kafka image
run: docker pull confluentinc/cp-kafka:7.6.1
- name: Run tests
run: mvn test
env:
TESTCONTAINERS_RYUK_DISABLED: falseThe docker pull step before running tests warms the image cache. Without it, the first test run in a fresh CI environment pays the full image download cost inside the test timeout. With the pull step, the download happens outside your test timer and is cached across runs.
Set TESTCONTAINERS_RYUK_DISABLED=false (the default) unless your CI explicitly forbids privileged containers. Ryuk is the Testcontainers cleanup daemon — it ensures containers are removed even if the JVM crashes mid-test.
For GitLab CI, add the Docker-in-Docker service:
kafka-tests:
image: maven:3.9-eclipse-temurin-21
services:
- docker:dind
variables:
DOCKER_HOST: tcp://docker:2375
DOCKER_TLS_CERTDIR: ""
TESTCONTAINERS_HOST_OVERRIDE: docker
script:
- docker pull confluentinc/cp-kafka:7.6.1
- mvn testThe TESTCONTAINERS_HOST_OVERRIDE variable tells Testcontainers to connect to the DinD service instead of the local Docker socket.
Common Pitfalls
Topic auto-creation. By default, Kafka auto-creates topics when a producer writes to them. This is fine for tests, but if your production setup has auto-creation disabled, your tests are testing a different configuration. Create topics explicitly in @BeforeEach:
try (AdminClient admin = AdminClient.create(Map.of(
AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, kafka.getBootstrapServers()
))) {
admin.createTopics(List.of(
new NewTopic("orders", 3, (short) 1)
)).all().get();
}Partition assignment lag. After consumer.subscribe(), Kafka needs to assign partitions before the consumer can poll messages. If you produce a message before assignment completes, and auto.offset.reset=latest, the consumer misses it. Setting auto.offset.reset=earliest avoids this in tests.
Multiple tests, same topic name. If two tests use the same topic and run concurrently, they contaminate each other's messages. Either use unique topic names per test (e.g., append a UUID) or configure test parallelism to run Kafka tests sequentially.
Slow cleanup. The @Container lifecycle calls container.stop() after tests. On some CI environments, container stop is slow and delays pipeline feedback. Use withStartupTimeout and withShutdownTimeout to tune these:
static final KafkaContainer kafka = new KafkaContainer(
DockerImageName.parse("confluentinc/cp-kafka:7.6.1"))
.withStartupTimeout(Duration.ofSeconds(90));Summary
Testcontainers gives you a real Kafka broker in tests with minimal setup. The key patterns: one container per class or module for startup efficiency, auto.offset.reset=earliest to avoid timing races, poll loops with deadlines instead of sleep, and explicit topic creation when partition count matters. With this foundation you can test serialization, offset commits, rebalancing, and schema evolution with confidence that what passes locally will also pass in production.