Chapter 5: Protocol Buffers — The Serialization Layer

Protocol Buffers (Protobuf) is gRPC's default serialization format. It's what turns your structured data into compact bytes for the wire and back again. Understanding Protobuf is essential — it's half of what makes gRPC work.

What Problem Does Serialization Solve?

When you send data over a network, you need to convert in-memory data structures into a sequence of bytes (serialization) and back (deserialization). Every format makes different tradeoffs:

Format	Human Readable	Size	Speed	Schema
JSON	Yes	Large	Slow	Optional
XML	Yes	Very Large	Very Slow	XSD
Protobuf	No (binary)	Small	Fast	Required
MessagePack	No	Medium	Fast	No
FlatBuffers	No	Small	Zero-copy	Required

Protobuf by Example

Let's define a simple message and see how it encodes:

// user.proto — the schema definition
syntax = "proto3";

package myapp;

message User {
  int32  id    = 1;   // field number 1
  string name  = 2;   // field number 2
  string email = 3;   // field number 3
  int32  age   = 4;   // field number 4
}

Now let's compare the encoded size for a user {id: 123, name: "Alice", email: "alice@example.com", age: 30}:

// JSON (68 bytes):
{"id":123,"name":"Alice","email":"alice@example.com","age":30}

// Protobuf (33 bytes):
08 7B 12 05 41 6C 69 63 65 1A 11 61 6C 69 63 65
40 65 78 61 6D 70 6C 65 2E 63 6F 6D 20 1E

Protobuf is ~50% smaller. For large payloads or high-frequency messages, this adds up enormously.

How Protobuf Encoding Works

Each field is encoded as: [field_number + wire_type] [value]

Wire Types

Wire Type	ID	Used For
Varint	0	int32, int64, uint32, uint64, sint32, sint64, bool, enum
64-bit	1	fixed64, sfixed64, double
Length-delimited	2	string, bytes, embedded messages, repeated fields
32-bit	5	fixed32, sfixed32, float

Varint Encoding

Protobuf uses variable-length integers. Small numbers use fewer bytes:

// Number 1 → 1 byte:   01
// Number 127 → 1 byte: 7F
// Number 128 → 2 bytes: 80 01
// Number 300 → 2 bytes: AC 02
// Number 16383 → 2 bytes: FF 7F

// Each byte: MSB = "more bytes follow", lower 7 bits = data
// 300 = 0b100101100
//   → split into 7-bit groups: 0000010 0101100
//   → reverse (little-endian): 0101100 0000010
//   → add continuation bits:   10101100 00000010
//   → hex:                     AC       02

Encoding Our Example Step by Step

// User{id: 123, name: "Alice", email: "alice@example.com", age: 30}

// Field 1 (id=123): field_number=1, wire_type=0 (varint)
//   Tag: (1 << 3) | 0 = 0x08
//   Value: 123 = 0x7B (fits in one varint byte)
08 7B

// Field 2 (name="Alice"): field_number=2, wire_type=2 (length-delimited)
//   Tag: (2 << 3) | 2 = 0x12
//   Length: 5 bytes
//   Value: "Alice" in UTF-8
12 05 41 6C 69 63 65

// Field 3 (email="alice@example.com"): field_number=3, wire_type=2
//   Tag: (3 << 3) | 2 = 0x1A
//   Length: 17 bytes
//   Value: "alice@example.com" in UTF-8
1A 11 61 6C 69 63 65 40 65 78 61 6D 70 6C 65 2E 63 6F 6D

// Field 4 (age=30): field_number=4, wire_type=0 (varint)
//   Tag: (4 << 3) | 0 = 0x20
//   Value: 30 = 0x1E
20 1E

Key Design Decisions

Field Numbers, Not Field Names

Unlike JSON, Protobuf doesn't encode field names. It uses field numbers. This is why:

• Messages are smaller (no "email": prefix, just the number 3)
• You can rename fields without breaking compatibility
• Field numbers are the true identity — never reuse them!

Default Values Are Not Serialized

In proto3, if a field has its default value (0 for numbers, "" for strings, false for bools), it's not sent on the wire. This saves space but means you can't distinguish "field was set to 0" from "field was not set."

// This message with age=0:
User{id: 123, name: "Alice", email: "", age: 0}

// Encodes the same as:
User{id: 123, name: "Alice"}

// Only id and name are on the wire. email and age are omitted.

Proto3 Scalar Types

Proto Type	C Equivalent	Default	Notes
double	double	0.0	64-bit IEEE 754
float	float	0.0	32-bit IEEE 754
int32	int32_t	0	Varint, inefficient for negatives
int64	int64_t	0	Varint, inefficient for negatives
uint32	uint32_t	0	Varint
uint64	uint64_t	0	Varint
sint32	int32_t	0	ZigZag + Varint (efficient for negatives)
sint64	int64_t	0	ZigZag + Varint (efficient for negatives)
fixed32	uint32_t	0	Always 4 bytes (better if values > 2^28)
fixed64	uint64_t	0	Always 8 bytes
sfixed32	int32_t	0	Always 4 bytes, signed
sfixed64	int64_t	0	Always 8 bytes, signed
bool	bool	false	Varint (0 or 1)
string	char*	""	UTF-8 encoded
bytes	uint8_t*	empty	Arbitrary byte sequence

Composite Types

syntax = "proto3";

// Enum
enum Status {
  UNKNOWN = 0;    // First enum value MUST be 0 (default)
  ACTIVE  = 1;
  BANNED  = 2;
}

// Nested message
message Address {
  string street = 1;
  string city   = 2;
  string country = 3;
}

// Message with all composite types
message User {
  int32   id      = 1;
  string  name    = 2;
  Status  status  = 3;           // enum field
  Address address = 4;           // nested message
  repeated string tags = 5;    // list/array
  map<string, string> metadata = 6; // key-value map
}

The .proto File Structure

// 1. Syntax declaration (always first)
syntax = "proto3";

// 2. Package (namespace to avoid collisions)
package mycompany.myservice.v1;

// 3. Imports (other .proto files)
import "google/protobuf/timestamp.proto";
import "google/protobuf/duration.proto";

// 4. Options (language-specific settings)
option go_package = "github.com/mycompany/myservice/v1;myservicev1";
option java_package = "com.mycompany.myservice.v1";

// 5. Messages (data structures)
message CreateUserRequest {
  string name = 1;
  string email = 2;
}

message CreateUserResponse {
  User user = 1;
}

// 6. Service definition (for gRPC)
service UserService {
  rpc CreateUser(CreateUserRequest) returns (CreateUserResponse);
}

Proto2 vs Proto3

You'll encounter both in the wild:

Feature	Proto2	Proto3
Field presence	All fields have has_* methods	Only optional/message fields
Default values	Custom defaults allowed	Always zero/empty
Required fields	Yes (dangerous!)	No (removed)
Enums	First value can be anything	First value must be 0
Maps	No	Yes
JSON mapping	Limited	Well-defined

The protoc Compiler

protoc is the Protocol Buffer compiler. It reads .proto files and generates code:

# Generate C++ code
$ protoc --cpp_out=./gen user.proto

# Generate Python code
$ protoc --python_out=./gen user.proto

# Generate Go code
$ protoc --go_out=./gen user.proto

# Generate gRPC service code (needs plugin)
$ protoc --cpp_out=./gen --grpc_out=./gen \
    --plugin=protoc-gen-grpc=`which grpc_cpp_plugin` \
    user.proto

The generated code gives you:

• Classes/structs for each message
• Getters and setters for each field
• Serialize/deserialize methods
• For gRPC: client stubs and server base classes

Protobuf vs JSON: A C Programmer's Perspective

If you think of it in C terms:

// JSON is like sending a self-describing text file:
// You parse field names as strings, figure out types at runtime.
// Like receiving a config file and parsing it with sscanf().

// Protobuf is like sending a packed struct:
struct __attribute__((packed)) User {
    int32_t id;
    uint8_t name_len;
    char    name[5];    // "Alice"
    uint8_t email_len;
    char    email[17];  // "alice@example.com"
    int32_t age;
};
// Except smarter: variable-length integers, field tags for extensibility,
// and you can skip unknown fields.

Mark as Complete