Testing

Protobuf Schema Evolution and Backward Compatibility Testing

HelpMeTest

HelpMeTest

— 6 min read
Protobuf Schema Evolution and Backward Compatibility Testing

Changing a Protobuf schema without breaking consumers is one of the most error-prone tasks in service maintenance. The rules seem simple, but they're easy to violate under deadline pressure. This guide covers which changes are safe, which are breaking, how to test for both, and how to automate compatibility checks in CI using the buf CLI.

Why Protobuf Compatibility Matters More Than JSON

With a JSON API, a client receiving an unknown field just ignores it. A client receiving a missing field gets undefined and might handle it gracefully. With Protobuf, the situation is more nuanced:

  • Unknown fields are preserved in proto3 (by default), but older implementations may not handle them
  • Field numbers are the real identity of a field — changing a field's name is safe, changing its number is breaking
  • Wire types must match for the field number — changing int32 to string on the same field number corrupts data silently
  • Enums and oneof have their own compatibility rules that differ from regular fields

The short version: Protobuf is safer than JSON for schema evolution, but only if you follow the rules. Breaking those rules produces silent data corruption, not immediate errors.

The Compatibility Matrix

Safe Changes (Non-Breaking)

// Before
message User {
    string id = 1;
    string email = 2;
}

// After — all of these are safe
message User {
    string id = 1;
    string email = 2;
    string display_name = 3;        // Adding a new field with a new number: SAFE
    repeated string roles = 4;      // Adding a repeated field: SAFE
    optional string phone = 5;      // Adding optional in proto3: SAFE
}
// Renaming a field (wire-compatible, but generated code breaks)
message User {
    string user_id = 1;     // renamed from 'id' — proto wire is identical, Go/Java generated API changes
    string email = 2;
}

Breaking Changes

// BREAKING: Changing field number
message User {
    string id = 2;      // was 1 — breaks all existing serialized data
    string email = 1;   // was 2
}

// BREAKING: Changing field type to incompatible wire type
message User {
    int32 id = 1;       // was 'string id = 1' — different wire type, silent corruption
    string email = 2;
}

// BREAKING: Removing a field without reserving the number
message User {
    // string id = 1;  REMOVED without reserve — number 1 can be reused, breaking existing clients
    string email = 2;
}

// CORRECT: Reserve removed field numbers and names
message User {
    reserved 1;
    reserved "id";
    string email = 2;
}

Conditionally Breaking Changes

// Changing from optional to repeated (or vice versa) depends on the wire type
// int32, int64, uint32, uint64, sint32, sint64, bool, enum: compatible with repeated
// string, bytes, embedded messages: compatible with repeated (uses length-delimited)
// fixed32, sfixed32, float: NOT compatible with repeated in packed encoding

Setting Up buf CLI for Breaking Change Detection

buf is the standard tool for Protobuf schema management. Install it and add breaking change detection to CI.

# Install buf
brew install bufbuild/buf/buf   # macOS
# or
curl -sSL https://github.com/bufbuild/buf/releases/latest/download/buf-Linux-x86_64 -o /usr/local/bin/buf
chmod +x /usr/local/bin/buf

Create buf.yaml in your proto directory:

version: v1
name: buf.build/yourorg/yourapi
deps:
  - buf.build/googleapis/googleapis
breaking:
  use:
    - FILE
lint:
  use:
    - DEFAULT
  except:
    - PACKAGE_VERSION_SUFFIX  # if you don't version your packages yet

The FILE breaking change category catches changes that break wire compatibility. There's also PACKAGE (catches package-level changes) and WIRE_JSON (also checks JSON field names).

# Check breaking changes against the local git HEAD
buf breaking --against '.git#branch=main'

# Check against a specific tag
buf breaking --against '.git#tag=v1.2.0'

# Check against a remote BSR repository
buf breaking --against 'buf.build/yourorg/yourapi:main'

# Dry run with output format
buf breaking --against '.git#branch=main' --error-format json

GitHub Actions Integration

# .github/workflows/proto-check.yaml
name: Proto Compatibility Check

on:
  pull_request:
    paths:
      - 'proto/**'

jobs:
  breaking:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0  # Required for git history

      - uses: bufbuild/buf-setup-action@v1
        with:
          version: '1.28.1'

      - name: Check for breaking changes
        run: |
          cd proto
          buf breaking --against '.git#branch=main,subdir=proto'

      - name: Lint proto files
        run: |
          cd proto
          buf lint

Testing Field Addition

Adding a field is safe, but you should test that:

  1. Old clients can still deserialize messages from the new server (they ignore the new field)
  2. New clients can deserialize messages from old servers (the new field has the zero value)
func TestFieldAddition_BackwardCompat(t *testing.T) {
    // Simulate: new server sends message with new field, old client receives it
    // Old client is simulated by decoding only the fields it knows about

    // New message (server-side)
    newMsg := &pb.User{
        Id:          "user-1",
        Email:       "alice@example.com",
        DisplayName: "Alice",  // new field
    }

    // Serialize as if new server is sending
    data, err := proto.Marshal(newMsg)
    if err != nil {
        t.Fatalf("marshal: %v", err)
    }

    // Deserialize as old message (without DisplayName field)
    type OldUser struct {
        Id    string `protobuf:"bytes,1,opt,name=id,proto3"`
        Email string `protobuf:"bytes,2,opt,name=email,proto3"`
    }

    // The cleaner way: use a snapshot of the old proto
    // Keep old .proto files in testdata/ and generate separate packages
    oldMsg := &testdata.UserV1{}  // generated from saved proto snapshot
    if err := proto.Unmarshal(data, oldMsg); err != nil {
        t.Fatalf("unmarshal into old type: %v", err)
    }

    if oldMsg.Id != "user-1" {
        t.Errorf("id not preserved: %q", oldMsg.Id)
    }
    if oldMsg.Email != "alice@example.com" {
        t.Errorf("email not preserved: %q", oldMsg.Email)
    }
    // DisplayName is silently ignored — that's correct behavior
}

func TestFieldAddition_ForwardCompat(t *testing.T) {
    // Simulate: old server sends message without new field, new client receives it
    oldMsg := &testdata.UserV1{
        Id:    "user-1",
        Email: "alice@example.com",
    }

    data, err := proto.Marshal(oldMsg)
    if err != nil {
        t.Fatalf("marshal: %v", err)
    }

    newMsg := &pb.User{}
    if err := proto.Unmarshal(data, newMsg); err != nil {
        t.Fatalf("unmarshal: %v", err)
    }

    if newMsg.DisplayName != "" {
        t.Errorf("expected zero value for new field, got %q", newMsg.DisplayName)
    }
    if newMsg.Id != "user-1" {
        t.Errorf("id not preserved: %q", newMsg.Id)
    }
}

Keeping Proto Snapshots for Compatibility Tests

The pattern is: save a copy of your proto file whenever you release a version, and test that current messages can round-trip through the old schema.

proto/
  user.proto          # current version
  testdata/
    user_v1.proto     # snapshot from v1.0
    user_v2.proto     # snapshot from v2.0
  testdata/
    gen/
      user_v1/        # generated Go code from snapshot
      user_v2/        # generated Go code from snapshot
# Generate code from snapshots (do this once when you take a snapshot)
protoc --go_out=testdata/gen/user_v1 --go_opt=paths=source_relative \
    proto/testdata/user_v1.proto

Testing Field Removal

Removing a field is the most dangerous operation. The safe procedure:

Step 1: Deprecate the field (ship this first, wait for all consumers to stop using it)

message User {
    string id = 1;
    string email = 2 [deprecated = true];  // mark deprecated but keep the number
    string primary_email = 5;              // new field for the replacement
}

Step 2: Remove and reserve (ship this only after all consumers are updated)

message User {
    string id = 1;
    reserved 2;         // prevent reuse of the field number
    reserved "email";   // prevent reuse of the field name
    string primary_email = 5;
}

Test that the reservation is in place:

func TestRemovedFieldIsReserved(t *testing.T) {
    desc := (&pb.User{}).ProtoReflect().Descriptor()

    // Check reserved ranges
    ranges := desc.ReservedRanges()
    fieldNumberReserved := false
    for i := 0; i < ranges.Len(); i++ {
        r := ranges.Get(i)
        if r.Start() <= 2 && 2 <= r.End() {
            fieldNumberReserved = true
        }
    }
    if !fieldNumberReserved {
        t.Error("field number 2 is not reserved — could be reused accidentally")
    }

    // Check reserved names
    names := desc.ReservedNames()
    nameReserved := false
    for i := 0; i < names.Len(); i++ {
        if names.Get(i) == "email" {
            nameReserved = true
        }
    }
    if !nameReserved {
        t.Error("field name 'email' is not reserved")
    }
}

Testing Enum Compatibility

Enums are a compatibility minefield. Proto3 clients are required to handle unknown enum values (they decode to 0), but that behavior can surprise callers.

// v1
enum UserStatus {
    UNKNOWN = 0;
    ACTIVE = 1;
    INACTIVE = 2;
}

// v2 — adding new values is safe
enum UserStatus {
    UNKNOWN = 0;
    ACTIVE = 1;
    INACTIVE = 2;
    SUSPENDED = 3;  // new value — old clients receive 0 (UNKNOWN)
}
func TestEnumAddition_OldClientReceivesUnknown(t *testing.T) {
    // New server sends SUSPENDED status
    newMsg := &pb.User{
        Id:     "user-1",
        Status: pb.UserStatus_SUSPENDED,
    }
    data, _ := proto.Marshal(newMsg)

    // Old client decodes it — unknown enum value becomes 0
    oldMsg := &testdata.UserV1{}
    proto.Unmarshal(data, oldMsg)

    // Old client sees UNKNOWN (0) — is your code safe when that happens?
    if oldMsg.Status != testdata.UserStatus_UNKNOWN {
        t.Logf("status: %v", oldMsg.Status)
        // Note: proto3 actually preserves the numeric value even if unknown
        // The generated code may or may not map it to a named constant
    }
}

Never remove an enum value — that's a breaking change. Reserve removed enum values:

enum UserStatus {
    UNKNOWN = 0;
    ACTIVE = 1;
    reserved 2;            // was INACTIVE
    reserved "INACTIVE";
    SUSPENDED = 3;
}

Automating Compatibility Tests in buf

Beyond breaking change detection, buf's buf lint catches common proto style violations that lead to compatibility problems:

# Run both lint and breaking change detection
buf lint && buf breaking --against '.git#branch=main'

Configure stricter rules in buf.yaml:

version: v1
breaking:
  use:
    - FILE
  except:
    - FIELD_SAME_DEFAULT   # if you intentionally change defaults
lint:
  use:
    - DEFAULT
    - COMMENTS             # require comments on all messages and fields
  rpc_allow_same_request_response: false
  rpc_allow_google_protobuf_empty_requests: false
  rpc_allow_google_protobuf_empty_responses: false

The Safe Schema Evolution Checklist

Before merging any proto change:

  • Run buf breaking --against '.git#branch=main' — zero breaking changes
  • If adding a field: verify it has a safe field number (not reusing a reserved number)
  • If removing a field: add reserved for both the number and the name
  • If changing a field type: verify wire type compatibility
  • If adding an enum value: verify consumers handle unknown enum values
  • If removing an enum value: add reserved for both the number and the name
  • Snapshot the current proto in testdata/ if this is a versioned release
  • Run the backward/forward compatibility tests

Schema evolution bugs are silent and show up days after deployment when old serialized data is read by new code, or when old clients receive new server responses. Automated checks catch them before that.

HelpMeTest can monitor your gRPC endpoints and alert on unexpected status codes or response shape changes after a deploy — useful as a last line of defense when schema changes slip through. If you're shipping proto changes frequently, that's the safety net worth having.

Read more

Start now free