Chapter 20: Reading Test Results — What the Numbers Mean
Example Result Table
Δ260514_185239
Create rate 7200 +0%
Create rate @80% vCPU 4207 +15% ← GOOD (more efficient)
Create avg DP CPU 68 -13% ← GOOD (less CPU)
Modify rate 21500 +0%
Modify rate @80% vCPU 18798 -5% ← BAD (less efficient)
Modify avg DP CPU 46 +5% ← BAD (more CPU)
Delete rate 14500 +0%
Delete rate @80% vCPU 8580 -6% ← BAD
Delete avg DP CPU 68 +6% ← BAD
avg CPU DP control * 68.5 % -10.5pp ← GOOD
avg CPU DP spare * 83.5 % +2.3pp ← WATCH (spare getting hot)
avg CPU KVDB * 16.3 % +0.4pp ← OKHow to Read the Delta Column
- Rate @80% vCPU: +15% = we can handle 15% more sessions at same CPU → improvement
- Rate @80% vCPU: -5% = we handle 5% fewer sessions at same CPU → regression
- avg CPU: -13% = using 13% less CPU at same rate → improvement
- avg CPU: +6% = using 6% more CPU at same rate → regression
- pp = percentage points (absolute change, not relative)
What to Watch For
⚠️ Red Flags
- DP spare CPU > 80% → approaching saturation (bottleneck)
- Rate@80% decreasing → capacity regression
- KVDB CPU increasing significantly → more load on Redis (bad batching?)
- Resendings > 100 → sessions timing out (latency too high)
Comparing Baseline vs Feature Build
// To evaluate the feature:
// 1. Run baseline (current code without feature)
// 2. Run feature build (with db-mux + MSET)
// 3. Compare at SAME rate:
// - CPU should be lower (or same) with feature
// - Rate@80% should be higher (or same) with feature
// - Latency should be same or better
// If CPU is lower AND rate@80% is higher → clear win
// If CPU is lower BUT latency is higher → trade-off (acceptable per architect)
// If CPU is higher → regression (need to investigate)Prometheus Counters
Internal counters (mentioned by Ferenc) can be used to double-check trends during test runs. These are exposed via Prometheus and visible in Grafana dashboards.
🎓 You Made It!
You now understand:
- How event loops and poll/POLLOUT drive I/O in PCG
- Why TCP_NODELAY + multiple writes = multiple segments (bad)
- How Redis pipelining and MSET reduce overhead
- How hiredis buffers commands and when they actually flush
- The PCG architecture (PEP, DP, Valkey, threads)
- Why db-mux is 2.5× faster than db-proxy (zero allocs, no defers)
- What the feature changes and what was rejected
- How to read performance test results
Next time your team discusses batching, POLLOUT, db-mux, or MSET — you'll follow along.