SRE Thailand Bootcamp

Week 7: Capacity Planning & Disaster Recovery

Theme: Load testing, right-sizing, DR strategies.

Learning Objectives

1. Run load tests with k6 (script-based) and hey (ad-hoc)
2. Analyze results to determine capacity limits
3. Design a disaster recovery plan with RTO/RPO targets
4. Implement backup/restore for a stateful service

Reading Materials

• k6 Documentation
• k6 Testing Guides
• hey: HTTP Load Generator
• Google SRE Book - Chapter 16: Managing Load
• Google SRE Book - Chapter 17: Handling Overload
• DR on Kubernetes (AWS)
• Velero Backup/Restore
• RTO and RPO Explained

Finger Exercises (choose 2)

1. Install k6. Write a test script that hits the Go app's /api/users endpoint with 100 concurrent VUs for 2 minutes. Analyze p50/p95/p99 latency.
2. Use hey for ad-hoc load testing: hey -n 10000 -c 50 http://localhost:8080/api/orders. Compare results with k6.
3. Install Velero and take a backup of a test namespace. Delete and restore the namespace.

Project

• Run a load test against the Phase 1 app:
  • Baseline: normal traffic (10 rps)
  • Spike: 10x traffic for 1 minute
  • Soak: sustained 50 rps for 10 minutes
• Document findings: bottleneck (which service/endpoint), latency degradation, error rate increase.
• Define RTO (1 hour) and RPO (15 minutes) for the app.
• Write a DR runbook: "How to recover the Phase 1 app from a full cluster failure."
• (Optional) Implement HPA based on Prometheus metrics.

Time Budget