Testing Kafka with Schema Registry: Avro, Protobuf, and JSON Schema

Testing Kafka with Schema Registry: Avro, Protobuf, and JSON Schema

Kafka without Schema Registry passes raw bytes. With Schema Registry, your producers and consumers enforce serialization contracts — Avro, Protobuf, or JSON Schema. This adds testability (schemas are explicit) but also complexity (your tests need a Schema Registry).

What You're Testing

Schema Registry testing covers three areas:

  1. Producer tests: does the producer serialize the correct schema version?
  2. Consumer tests: does the consumer deserialize correctly and handle schema evolution?
  3. Schema compatibility tests: does a schema change break compatibility with the registry's configured strategy?

Most bugs live in (3) — a schema change that passes unit tests but breaks compatibility with existing consumers in production.

Unit Testing with MockSchemaRegistryClient

Confluent's Java client ships with a MockSchemaRegistryClient for unit tests — no real registry needed.

// pom.xml
<dependency>
    <groupId>io.confluent</groupId>
    <artifactId>kafka-avro-serializer</artifactId>
    <version>7.6.0</version>
    <scope>test</scope>
</dependency>
import io.confluent.kafka.schemaregistry.client.MockSchemaRegistryClient;
import io.confluent.kafka.serializers.KafkaAvroSerializer;
import io.confluent.kafka.serializers.KafkaAvroDeserializer;

@Test
void producerSerializesUserEventCorrectly() throws Exception {
    MockSchemaRegistryClient mockRegistry = new MockSchemaRegistryClient();
    
    // Register schema upfront (mimics what Schema Registry does)
    Schema userSchema = new Schema.Parser().parse(
        getClass().getResourceAsStream("/schemas/user-event-v1.avsc")
    );
    mockRegistry.register("user-events-value", userSchema);
    
    Map<String, Object> serializerConfig = Map.of(
        "schema.registry.url", "mock://",
        AbstractKafkaSchemaSerDeConfig.AUTO_REGISTER_SCHEMAS, false
    );
    
    KafkaAvroSerializer serializer = new KafkaAvroSerializer(mockRegistry);
    serializer.configure(serializerConfig, false);
    
    // Build an Avro GenericRecord
    GenericRecord userEvent = new GenericData.Record(userSchema);
    userEvent.put("user_id", "u1");
    userEvent.put("email", "alice@example.com");
    userEvent.put("event_type", "USER_CREATED");
    
    byte[] serialized = serializer.serialize("user-events", userEvent);
    
    // Deserialize and verify round-trip
    KafkaAvroDeserializer deserializer = new KafkaAvroDeserializer(mockRegistry);
    deserializer.configure(serializerConfig, false);
    
    GenericRecord deserialized = (GenericRecord) deserializer.deserialize("user-events", serialized);
    
    assertEquals("u1", deserialized.get("user_id").toString());
    assertEquals("alice@example.com", deserialized.get("email").toString());
}

Testing Schema Evolution

The most dangerous Kafka bugs: schema changes that break running consumers. Test backward and forward compatibility explicitly.

@Test
void newOptionalFieldIsBackwardCompatible() throws Exception {
    MockSchemaRegistryClient mockRegistry = new MockSchemaRegistryClient();
    
    // V1 schema — consumer is running this
    String v1SchemaJson = """
        {
          "type": "record",
          "name": "UserEvent",
          "fields": [
            {"name": "user_id", "type": "string"},
            {"name": "email", "type": "string"}
          ]
        }
        """;
    Schema v1Schema = new Schema.Parser().parse(v1SchemaJson);
    
    // V2 schema — producer wants to add a new field
    String v2SchemaJson = """
        {
          "type": "record",
          "name": "UserEvent",
          "fields": [
            {"name": "user_id", "type": "string"},
            {"name": "email", "type": "string"},
            {"name": "name", "type": ["null", "string"], "default": null}
          ]
        }
        """;
    Schema v2Schema = new Schema.Parser().parse(v2SchemaJson);
    
    // V1 consumer can read V2 messages — this is backward compatible
    boolean isCompatible = SchemaCompatibility
        .checkReaderWriterCompatibility(v1Schema, v2Schema)
        .getType() == SchemaCompatibility.SchemaCompatibilityType.COMPATIBLE;
    
    assertTrue(isCompatible, "V2 should be backward compatible with V1 reader");
}

@Test
void removingRequiredFieldBreaksCompatibility() {
    String v1SchemaJson = """
        {
          "type": "record",
          "name": "UserEvent", 
          "fields": [
            {"name": "user_id", "type": "string"},
            {"name": "email", "type": "string"}
          ]
        }
        """;
    
    // V2 removes email (required in V1)
    String v2SchemaJson = """
        {
          "type": "record",
          "name": "UserEvent",
          "fields": [
            {"name": "user_id", "type": "string"}
          ]
        }
        """;
    
    Schema v1Schema = new Schema.Parser().parse(v1SchemaJson);
    Schema v2Schema = new Schema.Parser().parse(v2SchemaJson);
    
    boolean isCompatible = SchemaCompatibility
        .checkReaderWriterCompatibility(v1Schema, v2Schema)
        .getType() == SchemaCompatibility.SchemaCompatibilityType.COMPATIBLE;
    
    assertFalse(isCompatible, "Removing a required field should be incompatible");
}

Integration Testing with Testcontainers

For integration tests, use Testcontainers to spin up Kafka + Schema Registry together.

@Testcontainers
class SchemaRegistryIntegrationTest {
    
    @Container
    static final Network network = Network.newNetwork();
    
    @Container
    static final KafkaContainer kafka = new KafkaContainer(
        DockerImageName.parse("confluentinc/cp-kafka:7.6.0")
    ).withNetwork(network).withNetworkAliases("kafka");
    
    @Container
    static final GenericContainer<?> schemaRegistry = new GenericContainer<>(
        DockerImageName.parse("confluentinc/cp-schema-registry:7.6.0")
    )
    .withNetwork(network)
    .withExposedPorts(8081)
    .withEnv("SCHEMA_REGISTRY_HOST_NAME", "schema-registry")
    .withEnv("SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS",
        "PLAINTEXT://kafka:9092")
    .dependsOn(kafka)
    .waitingFor(Wait.forHttp("/subjects").forStatusCode(200));
    
    static String schemaRegistryUrl;
    
    @BeforeAll
    static void setup() {
        schemaRegistryUrl = "http://localhost:" + schemaRegistry.getMappedPort(8081);
    }
    
    @Test
    void producerRegistersSchemaAndConsumerReceivesMessage() throws Exception {
        Properties producerProps = new Properties();
        producerProps.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, kafka.getBootstrapServers());
        producerProps.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
        producerProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, KafkaAvroSerializer.class);
        producerProps.put("schema.registry.url", schemaRegistryUrl);
        producerProps.put(AbstractKafkaSchemaSerDeConfig.AUTO_REGISTER_SCHEMAS, true);
        
        Schema schema = new Schema.Parser().parse(
            getClass().getResourceAsStream("/schemas/user-event-v1.avsc")
        );
        
        GenericRecord event = new GenericData.Record(schema);
        event.put("user_id", "u1");
        event.put("email", "alice@example.com");
        event.put("event_type", "USER_CREATED");
        
        try (KafkaProducer<String, GenericRecord> producer = new KafkaProducer<>(producerProps)) {
            producer.send(new ProducerRecord<>("user-events", "u1", event)).get();
        }
        
        Properties consumerProps = new Properties();
        consumerProps.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, kafka.getBootstrapServers());
        consumerProps.put(ConsumerConfig.GROUP_ID_CONFIG, "test-group");
        consumerProps.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
        consumerProps.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        consumerProps.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, KafkaAvroDeserializer.class);
        consumerProps.put("schema.registry.url", schemaRegistryUrl);
        
        try (KafkaConsumer<String, GenericRecord> consumer = new KafkaConsumer<>(consumerProps)) {
            consumer.subscribe(List.of("user-events"));
            
            ConsumerRecords<String, GenericRecord> records = consumer.poll(Duration.ofSeconds(10));
            
            assertFalse(records.isEmpty(), "Expected at least one record");
            GenericRecord received = records.iterator().next().value();
            assertEquals("alice@example.com", received.get("email").toString());
        }
    }
}

Python: Testing with Confluent Kafka + Schema Registry

# pip install confluent-kafka fastavro requests-mock

import pytest
from unittest.mock import patch, MagicMock
from confluent_kafka import Producer, Consumer
from confluent_kafka.schema_registry import SchemaRegistryClient
from confluent_kafka.schema_registry.avro import AvroSerializer, AvroDeserializer

USER_SCHEMA_STR = '''
{
  "type": "record",
  "name": "UserEvent",
  "fields": [
    {"name": "user_id", "type": "string"},
    {"name": "email", "type": "string"}
  ]
}
'''

@pytest.fixture
def mock_schema_registry():
    """Mock Schema Registry HTTP responses"""
    with patch('requests.post') as mock_post, patch('requests.get') as mock_get:
        # Mock schema registration response
        mock_post.return_value = MagicMock(
            status_code=200,
            json=lambda: {"id": 1}
        )
        # Mock schema lookup response
        mock_get.return_value = MagicMock(
            status_code=200,
            json=lambda: {"schema": USER_SCHEMA_STR, "id": 1}
        )
        yield

def test_avro_serialization_round_trip():
    """Test Avro serialization without a real Schema Registry"""
    import fastavro
    from io import BytesIO
    import json
    
    schema = json.loads(USER_SCHEMA_STR)
    parsed_schema = fastavro.parse_schema(schema)
    
    record = {"user_id": "u1", "email": "alice@example.com"}
    
    # Serialize
    buffer = BytesIO()
    fastavro.schemaless_writer(buffer, parsed_schema, record)
    serialized = buffer.getvalue()
    
    # Deserialize
    buffer = BytesIO(serialized)
    deserialized = fastavro.schemaless_reader(buffer, parsed_schema)
    
    assert deserialized["user_id"] == "u1"
    assert deserialized["email"] == "alice@example.com"

Schema Compatibility CI Check

Add a pre-commit or CI step that validates new schemas against the registry before merging.

#!/bin/bash
# scripts/check-schema-compatibility.sh

REGISTRY_URL="${SCHEMA_REGISTRY_URL:-http://localhost:8081}"
SCHEMAS_DIR="src/main/resources/schemas"

for schema_file in "$SCHEMAS_DIR"/*.avsc; do
    subject=$(basename "$schema_file" .avsc)-value
    schema_json=$(jq -Rs '{"schema": .}' "$schema_file")
    
    response=$(curl -s -o /dev/null -w "%{http_code}" \
        -X POST \
        -H "Content-Type: application/json" \
        -d "$schema_json" \
        "$REGISTRY_URL/compatibility/subjects/$subject/versions/latest")
    
    if [ "$response" != "200" ]; then
        echo "FAIL: Schema $schema_file is not compatible with latest version of $subject"
        exit 1
    fi
    
    echo "OK: $schema_file is compatible"
done

Run this in CI before deployment. It catches the "field removal" class of bug before it reaches production consumers.

The key insight: Schema Registry testing is mostly about catching breaking changes early, not about testing Kafka mechanics. Your most valuable tests are the schema compatibility checks that run before any deployment.

Read more

Start now free