Chapter 18: Fault Tolerance — Failover, Circuit Breakers, Retries

Retry with Exponential Backoff

// Don't hammer a failing service. Back off exponentially:
attempt 1: wait 100ms
attempt 2: wait 200ms
attempt 3: wait 400ms
attempt 4: wait 800ms + random jitter
// Jitter prevents thundering herd (all clients retrying simultaneously)

Circuit Breaker Pattern

States: CLOSED ──(failures exceed threshold)──► OPEN ▲ │ │ (timeout) │ ▼ └──(success)──── HALF-OPEN ◄────────────┘ CLOSED: requests flow normally, count failures OPEN: reject immediately (don't call failing service), return fallback HALF-OPEN: allow one test request through to check if service recovered

Bulkhead Pattern

Isolate failures. Don't let one slow dependency consume all your threads/connections.

// Without bulkhead: Service B is slow → all threads waiting → entire system hangs
// With bulkhead: allocate max 10 threads to Service B calls
//   If all 10 busy → fail fast for B, other services unaffected

Timeout

Always set timeouts. A missing timeout = a thread/connection held forever if remote service hangs.

Failover Strategies

Strategy	How	Trade-off
Active-Passive	Standby takes over on failure	Simple but wastes standby resources
Active-Active	All nodes serve traffic	Efficient but complex (data sync)
Multi-region	Entire region fails over to another	Survives datacenter failure, complex

Key Takeaways

Retry with exponential backoff + jitter (never retry in tight loop)
Circuit breaker: stop calling a failing service, fail fast
Bulkhead: isolate failures, don't let one bad dependency sink everything
Always set timeouts on external calls
Design for failure: assume every dependency WILL fail

← Previous Next →