Chapter 20: Rate Limiting and Backpressure
Why Rate Limit?
- Prevent abuse (DDoS, scraping)
- Protect backend services from overload
- Fair usage across clients
- Cost control (expensive API calls)
Algorithms
| Algorithm | How | Pros/Cons |
|---|---|---|
| Token Bucket | Bucket fills at rate R, each request takes a token | Allows bursts up to bucket size. Most common. |
| Leaky Bucket | Requests queue, processed at fixed rate | Smooth output, no bursts. Good for APIs. |
| Fixed Window | Count requests per time window (e.g., per minute) | Simple but allows 2x burst at window boundary. |
| Sliding Window | Weighted count across current + previous window | Smooth, no boundary burst. Slightly more complex. |
Token Bucket (Most Common)
// Config: rate=10 req/sec, burst=20
// Bucket starts full (20 tokens)
// Each request consumes 1 token
// Bucket refills at 10 tokens/sec
// If empty → reject with 429 Too Many Requests
// Redis implementation (atomic):
// MULTI
// GET tokens:{client_id}
// if tokens > 0: DECR tokens:{client_id}; allow
// else: reject 429
// EXECDistributed Rate Limiting
With multiple app servers, rate limit state must be shared (Redis):
App Server 1 ──► Redis (shared counter) ◄── App Server 2
key: "ratelimit:user123"
value: tokens remaining
Backpressure
When a system is overloaded, push back on the caller rather than accepting and failing:
- Return 429 (Too Many Requests) with Retry-After header
- Queue fills up → reject new messages (bounded queue)
- TCP flow control (HTTP/2 window) — automatic backpressure
- gRPC: return RESOURCE_EXHAUSTED status
Key Takeaways
- Rate limiting protects services from overload and abuse
- Token bucket: most common, allows controlled bursts
- Distributed rate limiting needs shared state (Redis)
- Return 429 + Retry-After header to inform clients
- Backpressure: reject early rather than accept and fail later