diff --git a/BACKLOG.md b/BACKLOG.md index cf7ebf6..57adb9f 100644 --- a/BACKLOG.md +++ b/BACKLOG.md @@ -1918,9 +1918,21 @@ Every client request goes through the leader's router today: the floating addres passes `/v1` on to it (a standby's router sees an empty cluster). That is failover, not scale: the cluster serves what one control plane's router can. In order: -- O1. Measure the ceiling first: one router in front of a fast fake engine (`sim-engine`, through an external model), - streams at 1 to 256 at once from another machine, the leader's CPU beside it. Is it the router, the guardrails, the - usage writes or TLS? Written down with the host's load (pve1 is oversubscribed). +- O1. Measured 2026-10-30/31 on the PoC: the leader (2 vCPU, 2 GB, pve1 at load 6–10 of 28 threads) in front of + a fake engine on node06 (`sim-engine`, 500 tok/s a stream, no batch slowdown, as an external model), 64-token + streams from mirror 146, the default `observe` guardrails and usage written to PostgreSQL: + + | at once | req/s | tok/s | first token p50/p95 | leader llmgw CPU | + |---|---|---|---|---| + | 64 | 465 | 29,752 | 9 / 34 ms | 159% | + | 128 | 572 | 36,607 | 91 / 167 ms | 181% | + | 256 | 577 | 36,900 | 310 / 392 ms | 182% | + | 512 | 551 | 35,240 | 793 / 883 ms | 181% | + + The ceiling is the router's CPU: ~575 requests/s, ~37k streamed tokens/s on two cores, no errors; past it the + first token waits. Found on the way and fixed: an external model held to 3–4 at once by admission; a valid key + throttled as "unrecognised" when its cache entry expired under load. Next: a profile of where the CPU goes + (pprof under the same load) before O2, and the same run on a machine the size of a customer's control plane. - O2. Every control plane answers `/v1` itself (active-active): - the leader's replica table followed live by the standbys, so their routers route; - rate limits and budgets kept cluster-wide, not per process (J6), or three routers allow three times as much;