Chapter 40: Debugging, Troubleshooting, and War Stories
Debugging Tools
| Tool | Use For |
|---|---|
| grpcurl | Manual RPC calls (like curl for gRPC) |
| grpc-health-probe | Check health endpoint |
| GRPC_VERBOSITY=DEBUG | Enable verbose gRPC logging |
| GRPC_TRACE=all | Trace internal gRPC events |
| Wireshark | Inspect HTTP/2 frames on wire (with TLS keys) |
| Evans | Interactive gRPC REPL |
Common Problems
| Symptom | Likely Cause | Fix |
|---|---|---|
| UNAVAILABLE on first call | Channel not connected yet (lazy) | Wait for channel READY or use WaitForConnected() |
| DEADLINE_EXCEEDED | Server too slow or network issue | Increase deadline, check server logs, check DNS |
| All RPCs go to one pod | L4 LB + long-lived connection | Use L7 LB (Envoy) or client-side round_robin |
| Connection drops after idle | NAT/LB timeout (5-10 min) | Configure keepalive (30s ping interval) |
| RESOURCE_EXHAUSTED | Message too large (>4MB default) | Increase max_receive_message_length or chunk data |
| UNIMPLEMENTED | Method not registered or wrong service name | Check proto package, service registration |
Environment Variables for Debugging
# Enable all gRPC debug logging:
export GRPC_VERBOSITY=DEBUG
export GRPC_TRACE=all
# Trace specific subsystems:
export GRPC_TRACE=connectivity_state,channel,http
# Log to stderr:
export GRPC_GO_LOG_SEVERITY_LEVEL=info🎓 Congratulations!
You've completed the gRPC Zero to Hero guide. You now understand gRPC from protocol fundamentals through production deployment. The next step: build something. Pick a small service, define a .proto, generate code, implement it, deploy it, load test it.