diff --git a/BACKLOG.md b/BACKLOG.md index 57adb9f..2c229fa 100644 --- a/BACKLOG.md +++ b/BACKLOG.md @@ -1931,8 +1931,14 @@ one control plane's router can. In order: The ceiling is the router's CPU: ~575 requests/s, ~37k streamed tokens/s on two cores, no errors; past it the first token waits. Found on the way and fixed: an external model held to 3–4 at once by admission; a valid key - throttled as "unrecognised" when its cache entry expired under load. Next: a profile of where the CPU goes - (pprof under the same load) before O2, and the same run on a machine the size of a customer's control plane. + throttled as "unrecognised" when its cache entry expired under load. + + Profiled under 128 streams (2026-10-31): a third of the CPU was one write a token to the client; joining the + events already in hand into one write took the ceiling to ~700 req/s and ~45k tok/s, first token p50 45 ms at 128. + Now the largest part is the output guardrail of the default `observe` policy, ~30%: decoding each event (~20%; + a token-level reader with encoding/json/jsontext was measured slower than json.Unmarshal, 1.66 vs 1.27 µs) and + the PII patterns (~12%). Ways on, a guardrail decision: judge shadow output in larger windows, or a sample of it. + Next: the same run on a machine the size of a customer's control plane, then O2. - O2. Every control plane answers `/v1` itself (active-active): - the leader's replica table followed live by the standbys, so their routers route; - rate limits and budgets kept cluster-wide, not per process (J6), or three routers allow three times as much; diff --git a/CHANGELOG.md b/CHANGELOG.md index fb4dfc5..09d5a40 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -49,6 +49,10 @@ APIs change between minors without a deprecation period. request in flight missed the cache together and each took one of the address's allowance for unknown keys: past a hundred or so streams at once, a valid key got 429 "too many unrecognised api keys". Requests missing the same key at once now share one read and one allowance. +- **A streamed answer was one write to the client a token.** Each event was flushed alone, a system call each: a + third of a busy router's CPU. Events already received from the engine now go out in one write, with no token + waiting for another; on the PoC's two-core leader the ceiling rose from ~575 to ~700 streams' requests a second, + and the first token at 128 at once from 91 to 45 ms. ## v0.16.1 — 2026-10-30