Standbys that answer the client API themselves, behind router.active_active
CI / test (push) Successful in 9m35s

A standby follows the leader's Ready replicas over the agent port, every
two seconds, and reports back the replicas its router used and the
on-demand models it was asked for. No engine keys travel: every control
plane derives them from the cluster secret. Budgets are raised to the
store's every 30 s. Unheard from the leader for a minute, a standby
passes /v1 on as before. Off by default.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-11-01 13:47:47 +01:00
1 parent f664592948
commit 8b7eefef16
22 files changed
+936 -15

No files matched your search

+17 -5
View File
@@ -1944,11 +1944,23 @@ one control plane's router can. In order:
the PII patterns (~12%). Ways on, a guardrail decision: judge shadow output in larger windows, or a sample of it.
Next: the same run on a machine the size of a customer's control plane, then O2.
- O2. Every control plane answers `/v1` itself (active-active):
- the leader's replica table followed live by the standbys, so their routers route;
- rate limits and budgets kept cluster-wide, not per process (J6), or three routers allow three times as much;
- a session's requests to one router and one replica (consistent hashing on the session), so the engines' prefix
caches keep paying;
- in front: IPVS on the keepalived the setup already installs, DNS round-robin, or the customer's balancer (J7, L9).
- [x] the leader's replica table followed live by the standbys, so their routers route (2026-11-01, behind
`router.active_active`, off by default). Every 2 s a standby posts what its router used to the leader's
`POST /api/v1/routing/activity` over the agent port with its control-plane certificate, and is answered with
the Ready replicas: no keys travel, each control plane derives a replica's key from the cluster secret. The
leader takes the last uses in (else a replica busy behind a standby is evicted as idle) and the on-demand models
asked for; a catalog changed on the leader is read again from the store. Unheard from the leader for a minute,
a standby routes nothing and passes `/v1` on as before. The portal, `/admin` and `/mcp` still go to the leader.
- [~] budgets: each router raises its counters to the store's every 30 s, so another's spend is seen within that.
Rate limits are still per process: up to N× a key's limit across N control planes (J6).
- [ ] a session's requests to one router and one replica: the session ring already picks the same replica on
every router for the same replica set; what is left is the same router, which is the balancer's (source hash).
- [ ] in front: IPVS on the keepalived the setup already installs, DNS round-robin, or the customer's balancer
(J7, L9). Until then the standbys' own addresses are what spread the load.
- [ ] tried on the PoC: 140 and 150 both taking `/v1`, the O1 load split between them.
- Known and accepted for now, per router: the precise KV index is empty on a standby (agents stream to the leader
only), a revoked key works on a standby until `auth.cache_ttl`, canary promotion sees the leader's share only,
the answer cache and prefix tree are split, an MCP caller's rate is per router.
- O3. Only if O2 is not enough: stateless gateway machines (`llmgw server --role gateway`) in front of the leader's
control plane, as many as the traffic needs.