BACKLOG: the router profiled, the write per token removed, what is left

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

BACKLOG.md

@@ -1931,8 +1931,14 @@ one control plane's router can. In order:
 
   The ceiling is the router's CPU: ~575 requests/s, ~37k streamed tokens/s on two cores, no errors; past it the
   first token waits. Found on the way and fixed: an external model held to 3–4 at once by admission; a valid key
-  throttled as "unrecognised" when its cache entry expired under load. Next: a profile of where the CPU goes
-  (pprof under the same load) before O2, and the same run on a machine the size of a customer's control plane.
+  throttled as "unrecognised" when its cache entry expired under load.
+
+  Profiled under 128 streams (2026-10-31): a third of the CPU was one write a token to the client; joining the
+  events already in hand into one write took the ceiling to ~700 req/s and ~45k tok/s, first token p50 45 ms at 128.
+  Now the largest part is the output guardrail of the default `observe` policy, ~30%: decoding each event (~20%;
+  a token-level reader with encoding/json/jsontext was measured slower than json.Unmarshal, 1.66 vs 1.27 µs) and
+  the PII patterns (~12%). Ways on, a guardrail decision: judge shadow output in larger windows, or a sample of it.
+  Next: the same run on a machine the size of a customer's control plane, then O2.
 - O2. Every control plane answers `/v1` itself (active-active):
   - the leader's replica table followed live by the standbys, so their routers route;
   - rate limits and budgets kept cluster-wide, not per process (J6), or three routers allow three times as much;

CHANGELOG.md

@@ -49,6 +49,10 @@ APIs change between minors without a deprecation period.
   request in flight missed the cache together and each took one of the address's allowance for unknown keys: past a
   hundred or so streams at once, a valid key got 429 "too many unrecognised api keys". Requests missing the same key
   at once now share one read and one allowance.
+- **A streamed answer was one write to the client a token.** Each event was flushed alone, a system call each: a
+  third of a busy router's CPU. Events already received from the engine now go out in one write, with no token
+  waiting for another; on the PoC's two-core leader the ceiling rose from ~575 to ~700 streams' requests a second,
+  and the first token at 128 at once from 91 to 45 ms.
 
 ## v0.16.1 — 2026-10-30