gRPC

Protocol Buffer Schema Validation & Backward Compatibility Testing

HelpMeTest

HelpMeTest

— 6 min read
Protocol Buffer Schema Validation & Backward Compatibility Testing

Schema evolution is one of the hardest problems in distributed systems. When a gRPC service and its clients evolve independently — which they always do — a change to a .proto file can silently break production. This guide covers how to test Protocol Buffer schema compatibility before it becomes a production incident.

Why Protobuf Backward Compatibility Matters

Protocol Buffers are designed for forward and backward compatibility, but the rules are subtle. A "safe" proto change in isolation can break clients if you don't understand the wire format.

Safe changes (backward compatible):

  • Adding a new field with a new field number
  • Adding a new enum value
  • Adding a new message type
  • Marking a field as optional (already the default in proto3)

Breaking changes (incompatible):

  • Removing or renaming a field
  • Changing a field's type
  • Changing a field number
  • Changing a field from repeated to singular
  • Removing an enum value that clients depend on
  • Renaming a message used across services

The catch: proto3 won't tell you at compile time if you've made a breaking change. Clients with old schemas will silently drop unknown fields or use zero values for missing ones. This can cause data loss without any error.

Setting Up buf for Schema Validation

buf is the standard tool for Protobuf schema management, linting, and breaking change detection. It's the closest thing to a schema registry for gRPC.

Install buf:

# macOS
brew install bufbuild/buf/buf

# Linux
curl -sSL \
  "https://github.com/bufbuild/buf/releases/download/v1.28.1/buf-Linux-x86_64" \
  -o /usr/local/bin/buf && chmod +x /usr/local/bin/buf

# Verify
buf --version

Initialize buf in your proto directory:

cd proto/
buf config init

This creates buf.yaml:

version: v1
lint:
  use:
    - DEFAULT
breaking:
  use:
    - FILE

Linting Proto Files

Buf enforces a style guide that prevents common mistakes:

# Lint all proto files
buf lint

# Example output:
# proto/payment/v1/payment.proto:15:3:Field name "orderID" should be lower_snake_case, such as "order_id".
# proto/user/v1/user.proto:8:1:Message name "user_request" should be PascalCase, such as "UserRequest".

Configure lint rules in buf.yaml:

version: v1
lint:
  use:
    - DEFAULT
  except:
    - FIELD_LOWER_SNAKE_CASE  # if you need to exclude a rule
  ignore:
    - proto/vendor/           # ignore third-party protos

Run lint as part of CI to catch style violations before they reach code review.

Detecting Breaking Changes

This is where buf pays for itself. Run breaking change detection against a baseline:

# Compare against git main branch
buf breaking --against '.git#branch=main'

# Example output:
# proto/payment/v1/payment.proto:12:3:Field "1" with name "order_id" on message "PaymentRequest" changed option "json_name" from "orderId" to "orderIdentifier".
# proto/payment/v1/payment.proto:20:1:Message "PaymentRequest" had field "3" with name "currency" deleted.

Compare against a specific commit:

buf breaking --against '.git#tag=v2.3.0'

Compare against a remote BSR (Buf Schema Registry) module:

buf breaking --against 'buf.build/acme/paymentapi'

Understanding Breaking Change Categories

Buf checks breaking changes at different levels. Configure in buf.yaml:

breaking:
  use:
    - FILE        # checks within a single file (default)
    # - PACKAGE   # checks across a package
    # - WIRE      # only wire-format breaking changes
    # - WIRE_JSON # wire + JSON breaking changes

FILE level catches everything — renaming messages, moving fields between files, etc.

WIRE level is less strict — only catches changes that break the binary wire format. A renamed message doesn't break wire compatibility if clients already have the compiled code.

Choose based on your deployment model:

  • If clients compile their own protos: use FILE
  • If clients use binary protobuf only: use WIRE

Writing Compatibility Tests in Go

Unit tests that verify schema behavior catch issues at development time:

package payment_test

import (
    "testing"
    
    "github.com/stretchr/testify/assert"
    "github.com/stretchr/testify/require"
    "google.golang.org/protobuf/proto"
    
    paymentv1 "example.com/proto/payment/v1"
    paymentv2 "example.com/proto/payment/v2"
)

// Test that a v1 message can be decoded by v2 client
func TestBackwardCompatibility_V1MessageDecodedByV2(t *testing.T) {
    // Simulate a v1 client sending a message
    v1Msg := &paymentv1.PaymentRequest{
        OrderId:    "ord-123",
        Amount:     99.99,
        Currency:   "USD",
        CustomerId: "cust-456",
    }
    
    // Serialize as v1
    v1Bytes, err := proto.Marshal(v1Msg)
    require.NoError(t, err)
    
    // Deserialize as v2 (which has new fields)
    v2Msg := &paymentv2.PaymentRequest{}
    err = proto.Unmarshal(v1Bytes, v2Msg)
    require.NoError(t, err)
    
    // v2 should preserve existing fields
    assert.Equal(t, "ord-123", v2Msg.OrderId)
    assert.InDelta(t, 99.99, v2Msg.Amount, 0.001)
    assert.Equal(t, "USD", v2Msg.Currency)
    assert.Equal(t, "cust-456", v2Msg.CustomerId)
    
    // New v2 fields should have zero values
    assert.Equal(t, "", v2Msg.PromoCode)   // new field, should be empty string
    assert.False(t, v2Msg.IsSubscription)  // new field, should be false
}

// Test that a v2 message with new fields can be decoded by v1 client
func TestForwardCompatibility_V2MessageDecodedByV1(t *testing.T) {
    v2Msg := &paymentv2.PaymentRequest{
        OrderId:        "ord-123",
        Amount:         99.99,
        Currency:       "USD",
        CustomerId:     "cust-456",
        PromoCode:      "SAVE10",    // new field, v1 doesn't know about this
        IsSubscription: true,        // new field
    }
    
    v2Bytes, err := proto.Marshal(v2Msg)
    require.NoError(t, err)
    
    // V1 client receives this message — new fields should be silently ignored
    v1Msg := &paymentv1.PaymentRequest{}
    err = proto.Unmarshal(v2Bytes, v1Msg)
    require.NoError(t, err)
    
    // Known fields should be preserved
    assert.Equal(t, "ord-123", v1Msg.OrderId)
    assert.InDelta(t, 99.99, v1Msg.Amount, 0.001)
    
    // Unknown fields are stored in proto's unknown fields cache
    // v1 doesn't crash — it just ignores new fields
}

Testing Field Removal (The Dangerous Case)

The most dangerous schema change is removing a field. Test that your service handles missing fields correctly:

func TestMissingRequiredField_GracefulHandling(t *testing.T) {
    // Simulate old client that doesn't send currency (field 3)
    // by sending a message with currency omitted
    oldMsg := &paymentv1.PaymentRequest{
        OrderId:    "ord-123",
        Amount:     99.99,
        // Currency intentionally omitted
        CustomerId: "cust-456",
    }
    
    bytes, err := proto.Marshal(oldMsg)
    require.NoError(t, err)
    
    // Server receives message with missing currency
    serverMsg := &paymentv1.PaymentRequest{}
    err = proto.Unmarshal(bytes, serverMsg)
    require.NoError(t, err)
    
    // Server should validate and return a clear error, not panic
    // Test your validation logic here
    err = validatePaymentRequest(serverMsg)
    assert.Error(t, err)
    assert.Contains(t, err.Error(), "currency")
}

Testing Enum Compatibility

Enum changes are particularly tricky in proto3:

func TestEnumBackwardCompatibility(t *testing.T) {
    // Client sends an enum value that the server doesn't know about
    // (e.g., client has newer proto with more enum values)
    
    unknownEnumValue := int32(999) // some future enum value
    
    msg := &paymentv1.PaymentRequest{
        OrderId:    "ord-123",
        Amount:     99.99,
        Currency:   "USD",
        CustomerId: "cust-456",
    }
    
    bytes, err := proto.Marshal(msg)
    require.NoError(t, err)
    
    // Manually inject unknown enum value using raw proto manipulation
    // In practice, test with actual newer enum definitions
    
    received := &paymentv1.PaymentRequest{}
    err = proto.Unmarshal(bytes, received)
    require.NoError(t, err)
    
    // In proto3, unknown enum values are preserved as integers
    // Your code should handle this gracefully
    _ = unknownEnumValue
}

Setting Up a Schema Registry with Buf Schema Registry (BSR)

For teams managing multiple services, BSR provides centralized schema management:

# Push your schemas to BSR
buf push

# Other teams pull and depend on your schemas
buf dep update

Configure buf.yaml with a dependency:

version: v1
deps:
  - buf.build/acme/common-protos

BSR enforces breaking change rules on push — you can't push a breaking change without an explicit override. This is the safest way to manage schema evolution across teams.

CI Pipeline Integration

Add schema validation to your CI pipeline:

# .github/workflows/proto-check.yml
name: Protobuf Validation

on: [push, pull_request]

jobs:
  proto-lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0  # Need full history for breaking change detection
      
      - uses: bufbuild/buf-setup-action@v1
        with:
          version: '1.28.1'
      
      - name: Lint
        run: buf lint
      
      - name: Check for breaking changes
        run: buf breaking --against '.git#branch=main'
      
      - name: Generate (verify generated code is up to date)
        run: |
          buf generate
          git diff --exit-code  # Fail if generated code is out of sync

Practical Schema Evolution Workflow

When you need to make a breaking change:

  1. Create a new package version instead of modifying the existing one:
    • proto/payment/v1/payment.proto (keep as-is)
    • proto/payment/v2/payment.proto (new version with changes)
  2. Run both versions in parallel during migration:
    • New clients use v2
    • Old clients keep using v1
    • Server handles both
  3. Deprecate v1 once all clients have migrated
  4. Remove v1 only after confirming zero v1 traffic in logs

This multi-version approach avoids the "big bang" migration that always goes wrong in production.

Key Takeaways

  • Use buf lint to enforce proto style — catches naming mistakes before they become API contracts
  • Use buf breaking to detect compatibility violations — run in CI on every PR
  • Write explicit compatibility tests that marshal/unmarshal across versions
  • Use the reserved keyword to document removed fields and prevent reuse of their field numbers
  • Evolve schemas by adding versions (v2, v3), not by modifying existing packages in place
  • Set up BSR for centralized schema governance if you have multiple services

Schema validation is one of those things that feels unnecessary until it saves you from a 3am production incident.

Read more

Start now free