Week 8: Chaos Engineering
Theme: Principles of chaos engineering, running experiments safely.
Learning Objectives
- Explain chaos engineering principles (steady state, hypothesis, blast radius)
- Run a chaos experiment with LitmusChaos
- Analyze system behavior under failure
- Write a game day plan
Reading Materials
- Principles of Chaos Engineering
- LitmusChaos Docs
- Gremlin Chaos Engineering Guides
- Chaos Monkey (Netflix)
- Google SRE Workbook - Game Days
- Awesome Chaos Engineering
Finger Exercises (choose 2)
- Install LitmusChaos in the kind cluster. Run a "pod-delete" experiment.
- Define a steady-state hypothesis: "The app maintains < 1% error rate when 1 pod is killed."
- Read about Netflix's Chaos Monkey and how they run game days internally.
Project
Design and run 3 chaos experiments on the Phase 1 app:
- Pod kill: Randomly kill the Go pod. Verify Grafana alerts fire.
- Network latency: Inject 2s latency between Go and Python services. Verify SLO burn rate alert triggers.
- CPU stress: Saturate CPU on the Python worker. Observe tracing spans showing increased duration.
For each experiment:
- State the hypothesis
- Run the experiment
- Measure steady state vs. post-experiment
- Document findings
After all experiments, run a "game day" retrospective and write a one-page game day plan template.
Time Budget
| Activity | Time |
|---|---|
| Reading | 1.5 hrs |
| Finger exercises | 1.5 hrs |
| Project | 3 hrs |
| Total | 6 hrs |