Chapter 22: Consensus and Leader Election
Covered in DB guide (Raft, Paxos). System design perspective: you USE consensus systems, rarely build them.
When You Need Consensus
- Leader election: which node is the primary? (database failover)
- Distributed locks: only one worker processes a job
- Configuration: all nodes agree on current config
- Service discovery: which instances are alive?
Tools That Provide Consensus
| Tool | Algorithm | Use Case |
|---|---|---|
| etcd | Raft | Kubernetes config, service discovery |
| ZooKeeper | ZAB (Paxos-like) | Kafka coordination, distributed locks |
| Consul | Raft | Service discovery, KV config |
Distributed Locks (via Redis or etcd)
// Acquire lock (Redis SETNX with TTL):
SET lock:job123 owner=worker1 NX EX 30
// NX = only if not exists, EX 30 = expires in 30s (prevents deadlock)
// Release lock:
// Only if we still own it (Lua script for atomicity)
if redis.get("lock:job123") == "worker1" then redis.del("lock:job123")⚠️ Distributed Locks Are Hard
Clock skew, network partitions, and GC pauses can all cause a lock to be held by two nodes simultaneously. Use fencing tokens or accept that locks are "best effort" in distributed systems.