Chapter 15: Message Queues — Kafka, RabbitMQ, Patterns
Why Message Queues?
- Decoupling: producer doesn't know/care about consumers
- Buffering: absorb traffic spikes (queue fills up, consumers process at their pace)
- Reliability: if consumer dies, messages wait in queue
- Async processing: respond to user immediately, process later
RabbitMQ vs Kafka
| Aspect | RabbitMQ | Kafka |
|---|---|---|
| Model | Message broker (smart broker, dumb consumer) | Distributed log (dumb broker, smart consumer) |
| Delivery | Push to consumers | Consumers pull from log |
| Retention | Delete after consumed | Retain for configured period (days/weeks) |
| Throughput | ~50K msg/sec | ~1M msg/sec |
| Ordering | Per-queue | Per-partition |
| Best for | Task queues, RPC, routing | Event streaming, log aggregation, high throughput |
Kafka Architecture
Producer ──► Topic (partitioned)
├── Partition 0: [msg1][msg2][msg5][msg8]
├── Partition 1: [msg3][msg4][msg6]
└── Partition 2: [msg7][msg9][msg10]
▲
Consumer Group A: ──────────────────┘
Consumer 1 reads Partition 0
Consumer 2 reads Partition 1
Consumer 3 reads Partition 2
(parallel processing, each message processed once)
Common Patterns
- Work queue: distribute tasks among workers (email sending, image processing)
- Event sourcing: store all state changes as events in Kafka
- CQRS: separate read/write models, sync via events
- Dead letter queue: failed messages go to separate queue for investigation
Delivery Guarantees
| Guarantee | Meaning | Use Case |
|---|---|---|
| At-most-once | May lose messages, never duplicate | Metrics (losing one data point is OK) |
| At-least-once | Never lose, may duplicate | Most systems (consumers must be idempotent) |
| Exactly-once | Never lose, never duplicate | Financial (hard to achieve, Kafka supports it) |
Key Takeaways
- Queues decouple services, buffer spikes, enable async processing
- RabbitMQ: traditional broker, good for task queues and routing
- Kafka: distributed log, massive throughput, event streaming
- At-least-once + idempotent consumers is the practical sweet spot
- Dead letter queues catch poison messages that can't be processed