SRE Thailand Bootcamp

Week 3: Logging — Loki + ELK Stack

Theme: Centralized logging with Loki (cloud-native) and ELK (traditional).

Learning Objectives

1. Deploy Loki + Promtail + Grafana (LPG stack) using helm
2. Write structured JSON logs from Go and Python apps
3. Query logs with LogQL
4. Understand ELK stack architecture (Elasticsearch, Logstash, Kibana)

Reading Materials

• Loki Overview
• Loki Architecture
• Promtail Configuration
• LogQL Reference
• LogQL Examples
• Structured Logging in Go (log/slog)
• Elasticsearch Getting Started
• Google SRE Book - Chapter 10: Practical Alerting (logging section)

Finger Exercises (choose 2)

1. Update the Go app to use slog with structured JSON output. Include request_id, user_id, latency, endpoint.
2. Write 5 LogQL queries on your own logs: line contains "error", rate of errors per minute, top 3 slowest endpoints.
3. Deploy ELK stack (using docker-compose from elastic) and ship logs from the Python worker.

Project

• Write a minimal Python worker that consumes a queue (in-memory list for now) and logs structured JSON.
• Deploy Loki + Promtail to the kind cluster.
• Configure Promtail to scrape logs from both Go and Python containers using pod labels.
• Create a Grafana dashboard panel showing log volume, error rate from logs, and a log search panel.
• Create a LogQL alert: rate({app="go-app"} |= "error" [5m]) > 0.1

Time Budget