SRE Thailand Bootcamp

Week 17: Kafka in Production

Theme: Operate Kafka at scale: deployment, monitoring, consumer patterns.

Learning Objectives

1. Deploy Kafka on Kubernetes using Strimzi operator
2. Create topics with appropriate partitions, replication factor, retention
3. Implement idempotent producers and consumer groups
4. Monitor Kafka: lag, throughput, disk usage, ISR status

Reading Materials

• Kafka Documentation
• Kafka Design (paper)
• Strimzi Kafka Operator
• Kafka on Kubernetes Best Practices
• Kafka Lag Monitoring
• Confluent Kafka Monitoring
• Kafka Consumer Group Protocol

Finger Exercises (choose 2)

1. Deploy a 3-broker Kafka cluster using Strimzi. Create a topic with 6 partitions and replication factor 3.
2. Write a Go Kafka producer that sends order events. Write a consumer that processes them. Use confluent-kafka-go or franz-go.
3. Monitor Kafka lag using Prometheus JMX exporter + Grafana dashboard.

Project

• Replace the Python worker's in-memory queue with Kafka:
  • Go app produces order events to orders topic
  • Python worker consumes from orders topic (consumer group order-processors)
  • Python worker produces to notifications topic after processing
• Configure:
  • Topic orders: 6 partitions, RF=3, retention=7 days, cleanup.policy=delete
  • Topic notifications: 3 partitions, RF=2, retention=24h, cleanup.policy=delete
  • Idempotent producers (enable.idempotence=true)
  • Exactly-once semantics for critical orders
• Monitor with:
  • Prometheus JMX exporter for broker metrics
  • Kafka lag dashboard (consumer lag per partition)
  • Alert when lag > 1000 messages for > 5 minutes
• Test failure scenarios:
  • Kill a Kafka broker, verify ISR shrinks and recovers
  • Restart a consumer, verify rebalance

Time Budget