Chapter 3: TCP Under the Hood — Nagle, TCP_NODELAY, TCP_CORK, Syscalls

Why TCP Details Matter Here

Every Redis command and every HTTP/2 frame travels over TCP. The number of write() syscalls directly determines how many TCP segments are sent. Fewer segments = less kernel overhead = less CPU = more throughput.

The write() Syscall

ssize_t n = write(fd, buffer, length);
// This is EXPENSIVE:
// 1. User→kernel context switch (~1µs)
// 2. Copy data to kernel send buffer
// 3. Kernel builds TCP segment (header + data)
// 4. Kernel→user context switch
// At 300K commands/sec, if each is a separate write():
//   300K × 2 context switches = 600K context switches/sec = significant CPU

Nagle's Algorithm

TCP's built-in batching: if there's unacknowledged data in flight, buffer small writes and send them together when the ACK arrives.

Without Nagle (TCP_NODELAY): write("SET") → [TCP segment: "SET"] → sent immediately write(" k") → [TCP segment: " k"] → sent immediately write(" v") → [TCP segment: " v"] → sent immediately = 3 tiny TCP segments (inefficient network usage) With Nagle (default): write("SET") → [TCP segment: "SET"] → sent (first write always goes) write(" k") → buffered (waiting for ACK of "SET") write(" v") → buffered ACK arrives → [TCP segment: " k v"] → sent as one = 2 segments (more efficient)

TCP_NODELAY — Disable Nagle

int flag = 1;
setsockopt(fd, IPPROTO_TCP, TCP_NODELAY, &flag, sizeof(flag));
// Every write() immediately becomes a TCP segment
// Used when: you control batching yourself (like PCG does)
// PCG has TCP_NODELAY ON for all Redis connections

⚠️ TCP_NODELAY + Multiple write() = Multiple Segments

With TCP_NODELAY enabled (as in PCG), EVERY write() syscall produces a separate TCP segment. If you call write() 5 times for 5 Redis commands, you get 5 TCP segments. This is why batching writes into ONE write() call matters so much.

TCP_CORK — Explicit Batching

// Cork: tell kernel "I'm going to do multiple writes, hold them"
int flag = 1;
setsockopt(fd, IPPROTO_TCP, TCP_CORK, &flag, sizeof(flag));

write(fd, header, header_len);   // buffered by kernel
write(fd, body, body_len);       // buffered by kernel
write(fd, trailer, trailer_len); // buffered by kernel

// Uncork: "OK, send everything now as one segment"
flag = 0;
setsockopt(fd, IPPROTO_TCP, TCP_CORK, &flag, sizeof(flag));
// Kernel sends all buffered data as one TCP segment

This is what the signaling path cork prototype does for HTTP/2 frames.

Userspace Buffering (Alternative to TCP_CORK)

// Instead of multiple write() calls, buffer in userspace:
char buffer[65536];
int offset = 0;

memcpy(buffer + offset, cmd1, cmd1_len); offset += cmd1_len;
memcpy(buffer + offset, cmd2, cmd2_len); offset += cmd2_len;
memcpy(buffer + offset, cmd3, cmd3_len); offset += cmd3_len;

write(fd, buffer, offset);  // ONE syscall, ONE TCP segment
// This is essentially what hiredis output buffer does!

The Cost Hierarchy

Operation	Cost	Why
memcpy to userspace buffer	~10 ns	Just memory copy, no kernel
write() syscall	~1 µs	Context switch to kernel and back
TCP segment on network	~50 µs (loopback)	Network stack processing
Redis processes command	~1 µs	Parse + execute + reply

At 300K ops/sec: reducing from 300K write() calls to 30K (by batching 10 commands per write) saves ~270K µs = 270ms of CPU time per second. That's significant.

How This Applies to PCG

Current state (TCP_NODELAY on, no explicit batching): Redis commands queued during EVL iteration → hiredis output buffer accumulates them → POLLOUT fires → ONE write() → ONE TCP segment = Already batched per EVL iteration (good!) Problem case (db-proxy with defers): Each command creates a defer → callback limit may split across iterations → Multiple POLLOUT events → Multiple write() calls → Multiple segments = Batching broken! db-mux solution: All commands go directly to hiredis buffer (no defers) → Guaranteed one write() per iteration = Batching preserved! Signaling path (HTTP/2 frames): nghttp2 sends frames one at a time → multiple SSL_write → multiple write() → Multiple TLS records → Multiple TCP segments Cork solution: buffer all frames → one SSL_write → one write() → one segment

Key Takeaways

Each write() syscall = ~1µs overhead + one TCP segment (with TCP_NODELAY)
TCP_NODELAY is ON in PCG — every write() immediately sends
Fewer write() calls = fewer segments = less CPU = more throughput
TCP_CORK: kernel-level batching (used in signaling path prototype)
Userspace buffering: accumulate in buffer, one write() (what hiredis does)
The goal: maximize data per write() call

← Previous Next: Redis Fundamentals →