Design Exercise: Object Storage (S3-like)
Requirements
- Store arbitrary blobs (bytes) by key
- PUT, GET, DELETE operations
- 99.999999999% durability (11 nines)
- Petabyte scale
Architecture
Client ──► API Gateway ──► Metadata Service ──► Metadata DB
│ (which nodes have this object?)
▼
Data Service
┌─────────┼─────────┐
▼ ▼ ▼
Data Node Data Node Data Node
(store chunks on local disks)
Key Design Decisions
- Chunking: split large objects into fixed-size chunks (e.g., 64MB)
- Replication: store each chunk on 3+ nodes (different racks/AZs)
- Erasure coding: more space-efficient than replication for cold data
- Metadata: separate metadata DB (object → chunk locations)
- Consistency: read-after-write consistency (quorum writes)
- Garbage collection: mark-and-sweep for deleted objects
Durability Math
// 3 replicas, each node has 0.1% annual failure rate
// P(all 3 fail) = 0.001^3 = 10^-9 = 99.9999999% durability
// With repair (replace failed replica quickly): even better