Wire Compatibility & Schema Evolution (Protobuf field numbers).

🎬Friday, 16:40

order-service ships a tiny proto change. Two lines. CI green. Deploy.

By 17:10 payment-service is charging some orders 0.00 — and Sentry is quiet. No exceptions, just wrong numbers. Rolling back order-service doesn't fix the rows already written.

The diff:

message Order {
  string id       = 1;
  int32  total    = 2;
-  int32  discount = 3;
+  string coupon   = 3;    // reused tag 3
  string currency = 4;
}

🔍What the wire actually carries

Protobuf never sends field names. It sends (field_number, wire_type, value). To an old reader, coupon is not a new field — it IS field 3. The old payment-service pods still in the fleet decode tag 3 as int32.

string and int32 have different wire types (2 vs 0), so most readers skip it as "unknown field"… and a blob that happens to parse can land as garbage instead. Either way discount comes back as 0 → the proto3 default → no error, wrong money.

Names are free to change. Numbers are the contract. Renaming total → total_cents is a no-op on the wire.

🧯The fix that should have been in that diff

message Order {
  string id       = 1;
  int32  total    = 2;
  reserved 3;               // tombstone — nobody may reuse it
  reserved "discount";
  string coupon   = 7;      // a brand new number
  string currency = 4;
}

⚠️Two directions, and they decide your deploy order

• Backward compatible = NEW code reads OLD data.
• Forward compatible = OLD code reads NEW data. This is the one that bites, because during a rolling deploy both versions run at once for ~5 minutes.

Rule of thumb: deploy the reader first, the writer second. Always.

🕳️The proto3 trap you hit next

total = 0, currency = "" and "field absent" are the same bytes — proto3 omits defaults. So you cannot distinguish "customer set discount to 0" from "old client never sent it". When absence is meaningful, use explicit presence: optional string currency = 4; (available since protobuf 3.15).

Enums: always define FOO_UNSPECIFIED = 0; and never delete a value. An old reader given an unknown enum number keeps the raw int in Go/Java, but plenty of gateways and JSON mappers coerce it to 0 — so 0 must mean "I don't know", never "PENDING".

✅The 5-point checklist

1. Never reuse a field number. reserved it, forever.
2. New fields must be optional or have a sane default.
3. Additive only. Delete a field only in a later release, after every reader is gone.
4. Reader deploys before writer.
5. Put buf breaking --against '.git#branch=main' in CI so the diff above fails before a human ever reviews it.

🧪Your exercise

Take a proto — or a plain JSON API — you own. Add a field, then answer honestly: if an OLD consumer receives this message today, what exactly does it do? Then prove it: encode with the new schema, decode with the old one, print the result.

Bonus: run buf breaking against your last 3 merged schema commits and count how many would have been blocked.