Monitoring, Logging, and Observability
Three Pillars of Observability
| Pillar | What | Tools |
|---|---|---|
| Metrics | Numeric measurements over time (CPU, latency, error rate) | Prometheus, Grafana, Datadog |
| Logs | Discrete events with context (structured JSON logs) | ELK Stack, Loki, Splunk |
| Traces | Request path across services (distributed tracing) | Jaeger, Zipkin, OpenTelemetry |
Key Metrics (RED Method)
- Rate: requests per second
- Errors: error rate (% of failed requests)
- Duration: latency (p50, p95, p99)
Distributed Tracing
Request: GET /checkout
│
├── API Gateway (5ms)
│ └── Auth Service (10ms)
├── Order Service (50ms)
│ ├── Inventory Service (20ms)
│ └── Payment Service (100ms) ← BOTTLENECK!
└── Notification Service (5ms)
Trace shows: total 170ms, Payment is the bottleneck.
Without tracing: "checkout is slow" — which service? Unknown.
Alerting
Alert on symptoms (high error rate, high latency), not causes (high CPU). High CPU with no user impact = not urgent.
Key Takeaways
- Metrics for dashboards and alerting (Prometheus + Grafana)
- Logs for debugging specific issues (structured JSON, centralized)
- Traces for understanding request flow across services
- Alert on user-facing symptoms, not internal metrics
- OpenTelemetry: vendor-neutral standard for all three pillars