SRE Thailand Bootcamp

Week 6: Incident Management

Theme: On-call practices, incident response, blameless postmortems.

Learning Objectives

1. Define severity levels (SEV1-SEV4) and response SLAs
2. Design an on-call rotation schedule
3. Set up incident management tooling (PagerDuty / OpsGenie free tier)
4. Write a blameless postmortem using a standard template

Reading Materials

• Google SRE Book - Chapter 10: Practical Alerting
• Google SRE Book - Chapter 15: Postmortem Culture
• Google SRE Book - Chapter 11: Being On-Call
• PagerDuty Incident Response Docs
• Blameless Postmortem Template (Atlassian)
• OpsGenie Incident Management

Finger Exercises (choose 2)

1. Write a postmortem for a fictional SEV1 outage (e.g., database connection pool exhaustion). Include timeline, root cause, action items.
2. Set up PagerDuty Developer Plan (free tier). Create a service, escalation policy, and test alert.
3. Create an on-call schedule with 2 people for a 2-week rotation. Document handoff procedures.

Project

• Define the incident response process for the Phase 1 app:
  • Severity definitions with examples
  • On-call schedule (documented in ONCALL.md)
  • Escalation paths
  • Communication channels (Slack channel per severity)
• Simulate an incident:
  • Trigger a real alert from the Prometheus rules (e.g., kill the Go pod)
  • Respond using your process
  • Write a postmortem for the simulated outage
  • Create action items and track them as GitHub Issues

Time Budget