Testing Kafka with Schema Registry: Avro, Protobuf, and JSON Schema
Kafka without Schema Registry passes raw bytes. With Schema Registry, your producers and consumers enforce serialization contracts — Avro, Protobuf, or JSON Schema. This adds testability (schemas are explicit) but also complexity (your tests need a Schema Registry).
What You're Testing
Schema Registry testing covers three areas:
- Producer tests: does the producer serialize the correct schema version?
- Consumer tests: does the consumer deserialize correctly and handle schema evolution?
- Schema compatibility tests: does a schema change break compatibility with the registry's configured strategy?
Most bugs live in (3) — a schema change that passes unit tests but breaks compatibility with existing consumers in production.
Unit Testing with MockSchemaRegistryClient
Confluent's Java client ships with a MockSchemaRegistryClient for unit tests — no real registry needed.
// pom.xml
<dependency>
<groupId>io.confluent</groupId>
<artifactId>kafka-avro-serializer</artifactId>
<version>7.6.0</version>
<scope>test</scope>
</dependency>import io.confluent.kafka.schemaregistry.client.MockSchemaRegistryClient;
import io.confluent.kafka.serializers.KafkaAvroSerializer;
import io.confluent.kafka.serializers.KafkaAvroDeserializer;
@Test
void producerSerializesUserEventCorrectly() throws Exception {
MockSchemaRegistryClient mockRegistry = new MockSchemaRegistryClient();
// Register schema upfront (mimics what Schema Registry does)
Schema userSchema = new Schema.Parser().parse(
getClass().getResourceAsStream("/schemas/user-event-v1.avsc")
);
mockRegistry.register("user-events-value", userSchema);
Map<String, Object> serializerConfig = Map.of(
"schema.registry.url", "mock://",
AbstractKafkaSchemaSerDeConfig.AUTO_REGISTER_SCHEMAS, false
);
KafkaAvroSerializer serializer = new KafkaAvroSerializer(mockRegistry);
serializer.configure(serializerConfig, false);
// Build an Avro GenericRecord
GenericRecord userEvent = new GenericData.Record(userSchema);
userEvent.put("user_id", "u1");
userEvent.put("email", "alice@example.com");
userEvent.put("event_type", "USER_CREATED");
byte[] serialized = serializer.serialize("user-events", userEvent);
// Deserialize and verify round-trip
KafkaAvroDeserializer deserializer = new KafkaAvroDeserializer(mockRegistry);
deserializer.configure(serializerConfig, false);
GenericRecord deserialized = (GenericRecord) deserializer.deserialize("user-events", serialized);
assertEquals("u1", deserialized.get("user_id").toString());
assertEquals("alice@example.com", deserialized.get("email").toString());
}Testing Schema Evolution
The most dangerous Kafka bugs: schema changes that break running consumers. Test backward and forward compatibility explicitly.
@Test
void newOptionalFieldIsBackwardCompatible() throws Exception {
MockSchemaRegistryClient mockRegistry = new MockSchemaRegistryClient();
// V1 schema — consumer is running this
String v1SchemaJson = """
{
"type": "record",
"name": "UserEvent",
"fields": [
{"name": "user_id", "type": "string"},
{"name": "email", "type": "string"}
]
}
""";
Schema v1Schema = new Schema.Parser().parse(v1SchemaJson);
// V2 schema — producer wants to add a new field
String v2SchemaJson = """
{
"type": "record",
"name": "UserEvent",
"fields": [
{"name": "user_id", "type": "string"},
{"name": "email", "type": "string"},
{"name": "name", "type": ["null", "string"], "default": null}
]
}
""";
Schema v2Schema = new Schema.Parser().parse(v2SchemaJson);
// V1 consumer can read V2 messages — this is backward compatible
boolean isCompatible = SchemaCompatibility
.checkReaderWriterCompatibility(v1Schema, v2Schema)
.getType() == SchemaCompatibility.SchemaCompatibilityType.COMPATIBLE;
assertTrue(isCompatible, "V2 should be backward compatible with V1 reader");
}
@Test
void removingRequiredFieldBreaksCompatibility() {
String v1SchemaJson = """
{
"type": "record",
"name": "UserEvent",
"fields": [
{"name": "user_id", "type": "string"},
{"name": "email", "type": "string"}
]
}
""";
// V2 removes email (required in V1)
String v2SchemaJson = """
{
"type": "record",
"name": "UserEvent",
"fields": [
{"name": "user_id", "type": "string"}
]
}
""";
Schema v1Schema = new Schema.Parser().parse(v1SchemaJson);
Schema v2Schema = new Schema.Parser().parse(v2SchemaJson);
boolean isCompatible = SchemaCompatibility
.checkReaderWriterCompatibility(v1Schema, v2Schema)
.getType() == SchemaCompatibility.SchemaCompatibilityType.COMPATIBLE;
assertFalse(isCompatible, "Removing a required field should be incompatible");
}Integration Testing with Testcontainers
For integration tests, use Testcontainers to spin up Kafka + Schema Registry together.
@Testcontainers
class SchemaRegistryIntegrationTest {
@Container
static final Network network = Network.newNetwork();
@Container
static final KafkaContainer kafka = new KafkaContainer(
DockerImageName.parse("confluentinc/cp-kafka:7.6.0")
).withNetwork(network).withNetworkAliases("kafka");
@Container
static final GenericContainer<?> schemaRegistry = new GenericContainer<>(
DockerImageName.parse("confluentinc/cp-schema-registry:7.6.0")
)
.withNetwork(network)
.withExposedPorts(8081)
.withEnv("SCHEMA_REGISTRY_HOST_NAME", "schema-registry")
.withEnv("SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS",
"PLAINTEXT://kafka:9092")
.dependsOn(kafka)
.waitingFor(Wait.forHttp("/subjects").forStatusCode(200));
static String schemaRegistryUrl;
@BeforeAll
static void setup() {
schemaRegistryUrl = "http://localhost:" + schemaRegistry.getMappedPort(8081);
}
@Test
void producerRegistersSchemaAndConsumerReceivesMessage() throws Exception {
Properties producerProps = new Properties();
producerProps.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, kafka.getBootstrapServers());
producerProps.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
producerProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, KafkaAvroSerializer.class);
producerProps.put("schema.registry.url", schemaRegistryUrl);
producerProps.put(AbstractKafkaSchemaSerDeConfig.AUTO_REGISTER_SCHEMAS, true);
Schema schema = new Schema.Parser().parse(
getClass().getResourceAsStream("/schemas/user-event-v1.avsc")
);
GenericRecord event = new GenericData.Record(schema);
event.put("user_id", "u1");
event.put("email", "alice@example.com");
event.put("event_type", "USER_CREATED");
try (KafkaProducer<String, GenericRecord> producer = new KafkaProducer<>(producerProps)) {
producer.send(new ProducerRecord<>("user-events", "u1", event)).get();
}
Properties consumerProps = new Properties();
consumerProps.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, kafka.getBootstrapServers());
consumerProps.put(ConsumerConfig.GROUP_ID_CONFIG, "test-group");
consumerProps.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
consumerProps.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
consumerProps.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, KafkaAvroDeserializer.class);
consumerProps.put("schema.registry.url", schemaRegistryUrl);
try (KafkaConsumer<String, GenericRecord> consumer = new KafkaConsumer<>(consumerProps)) {
consumer.subscribe(List.of("user-events"));
ConsumerRecords<String, GenericRecord> records = consumer.poll(Duration.ofSeconds(10));
assertFalse(records.isEmpty(), "Expected at least one record");
GenericRecord received = records.iterator().next().value();
assertEquals("alice@example.com", received.get("email").toString());
}
}
}Python: Testing with Confluent Kafka + Schema Registry
# pip install confluent-kafka fastavro requests-mock
import pytest
from unittest.mock import patch, MagicMock
from confluent_kafka import Producer, Consumer
from confluent_kafka.schema_registry import SchemaRegistryClient
from confluent_kafka.schema_registry.avro import AvroSerializer, AvroDeserializer
USER_SCHEMA_STR = '''
{
"type": "record",
"name": "UserEvent",
"fields": [
{"name": "user_id", "type": "string"},
{"name": "email", "type": "string"}
]
}
'''
@pytest.fixture
def mock_schema_registry():
"""Mock Schema Registry HTTP responses"""
with patch('requests.post') as mock_post, patch('requests.get') as mock_get:
# Mock schema registration response
mock_post.return_value = MagicMock(
status_code=200,
json=lambda: {"id": 1}
)
# Mock schema lookup response
mock_get.return_value = MagicMock(
status_code=200,
json=lambda: {"schema": USER_SCHEMA_STR, "id": 1}
)
yield
def test_avro_serialization_round_trip():
"""Test Avro serialization without a real Schema Registry"""
import fastavro
from io import BytesIO
import json
schema = json.loads(USER_SCHEMA_STR)
parsed_schema = fastavro.parse_schema(schema)
record = {"user_id": "u1", "email": "alice@example.com"}
# Serialize
buffer = BytesIO()
fastavro.schemaless_writer(buffer, parsed_schema, record)
serialized = buffer.getvalue()
# Deserialize
buffer = BytesIO(serialized)
deserialized = fastavro.schemaless_reader(buffer, parsed_schema)
assert deserialized["user_id"] == "u1"
assert deserialized["email"] == "alice@example.com"Schema Compatibility CI Check
Add a pre-commit or CI step that validates new schemas against the registry before merging.
#!/bin/bash
# scripts/check-schema-compatibility.sh
REGISTRY_URL="${SCHEMA_REGISTRY_URL:-http://localhost:8081}"
SCHEMAS_DIR="src/main/resources/schemas"
for schema_file in "$SCHEMAS_DIR"/*.avsc; do
subject=$(basename "$schema_file" .avsc)-value
schema_json=$(jq -Rs '{"schema": .}' "$schema_file")
response=$(curl -s -o /dev/null -w "%{http_code}" \
-X POST \
-H "Content-Type: application/json" \
-d "$schema_json" \
"$REGISTRY_URL/compatibility/subjects/$subject/versions/latest")
if [ "$response" != "200" ]; then
echo "FAIL: Schema $schema_file is not compatible with latest version of $subject"
exit 1
fi
echo "OK: $schema_file is compatible"
doneRun this in CI before deployment. It catches the "field removal" class of bug before it reaches production consumers.
The key insight: Schema Registry testing is mostly about catching breaking changes early, not about testing Kafka mechanics. Your most valuable tests are the schema compatibility checks that run before any deployment.