SRE Thailand Bootcamp

Week 2: Visualization & Alerting — Grafana

Theme: Build dashboards, configure alerting rules, manage Alertmanager.

Learning Objectives

1. Create Grafana dashboards with Prometheus data source
2. Use Grafana variables, template queries, and annotations
3. Write alerting rules in Prometheus and route them through Alertmanager
4. Configure Alertmanager receivers (Slack, email)

Reading Materials

• Grafana Getting Started
• Grafana Prometheus Data Source
• Grafana Dashboard Best Practices
• Prometheus Alerting Rules
• Alertmanager Configuration
• Google SRE Book - Chapter 19: Alerting on SLOs
• Grafana Play Dashboard Examples

Finger Exercises (choose 2)

1. Create a "RED" dashboard (Rate, Errors, Duration) for the Go app from Week 1.
2. Set up Alertmanager with Slack webhook. Create an alert that fires when error rate > 5%.
3. Build a dashboard with template variables for endpoint and method.

Project

• Deploy Grafana alongside Prometheus (same cluster).
• Import the Week 1 Go app metrics.
• Build 3 dashboards:
  1. Service Overview: RPS, error rate, p50/p95/p99 latency, up status
  2. Business Metrics: Orders created over time, success/failure ratio
  3. Infrastructure: CPU/memory per pod, scrape targets
• Write 3 alerting rules (high error rate, high latency P99 > 500ms, service down) and route to a Slack channel (or mock webhook).

Time Budget