Standbys that answer the client API themselves, behind router.active_active
CI / test (push) Successful in 9m35s
CI / test (push) Successful in 9m35s
A standby follows the leader's Ready replicas over the agent port, every two seconds, and reports back the replicas its router used and the on-demand models it was asked for. No engine keys travel: every control plane derives them from the cluster secret. Budgets are raised to the store's every 30 s. Unheard from the leader for a minute, a standby passes /v1 on as before. Off by default. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
f664592948
commit
8b7eefef16
22 files changed
+936
-15
No files matched your search
+17
-5
@@ -1944,11 +1944,23 @@ one control plane's router can. In order:
|
||||
the PII patterns (~12%). Ways on, a guardrail decision: judge shadow output in larger windows, or a sample of it.
|
||||
Next: the same run on a machine the size of a customer's control plane, then O2.
|
||||
- O2. Every control plane answers `/v1` itself (active-active):
|
||||
- the leader's replica table followed live by the standbys, so their routers route;
|
||||
- rate limits and budgets kept cluster-wide, not per process (J6), or three routers allow three times as much;
|
||||
- a session's requests to one router and one replica (consistent hashing on the session), so the engines' prefix
|
||||
caches keep paying;
|
||||
- in front: IPVS on the keepalived the setup already installs, DNS round-robin, or the customer's balancer (J7, L9).
|
||||
- [x] the leader's replica table followed live by the standbys, so their routers route (2026-11-01, behind
|
||||
`router.active_active`, off by default). Every 2 s a standby posts what its router used to the leader's
|
||||
`POST /api/v1/routing/activity` over the agent port with its control-plane certificate, and is answered with
|
||||
the Ready replicas: no keys travel, each control plane derives a replica's key from the cluster secret. The
|
||||
leader takes the last uses in (else a replica busy behind a standby is evicted as idle) and the on-demand models
|
||||
asked for; a catalog changed on the leader is read again from the store. Unheard from the leader for a minute,
|
||||
a standby routes nothing and passes `/v1` on as before. The portal, `/admin` and `/mcp` still go to the leader.
|
||||
- [~] budgets: each router raises its counters to the store's every 30 s, so another's spend is seen within that.
|
||||
Rate limits are still per process: up to N× a key's limit across N control planes (J6).
|
||||
- [ ] a session's requests to one router and one replica: the session ring already picks the same replica on
|
||||
every router for the same replica set; what is left is the same router, which is the balancer's (source hash).
|
||||
- [ ] in front: IPVS on the keepalived the setup already installs, DNS round-robin, or the customer's balancer
|
||||
(J7, L9). Until then the standbys' own addresses are what spread the load.
|
||||
- [ ] tried on the PoC: 140 and 150 both taking `/v1`, the O1 load split between them.
|
||||
- Known and accepted for now, per router: the precise KV index is empty on a standby (agents stream to the leader
|
||||
only), a revoked key works on a standby until `auth.cache_ttl`, canary promotion sees the leader's share only,
|
||||
the answer cache and prefix tree are split, an MCP caller's rate is per router.
|
||||
- O3. Only if O2 is not enough: stateless gateway machines (`llmgw server --role gateway`) in front of the leader's
|
||||
control plane, as many as the traffic needs.
|
||||
|
||||
|
||||
Reference in new issue
Block a user