External providers left out of admission's saturation; the scale-out section in the backlog

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-30 19:13:15 +01:00
1 parent be54f1617b
commit 524f029603
4 files changed
+49

No files matched your search

+20
View File
@@ -46,6 +46,8 @@ providers, of L5 cache-write and audio prices. L6 and L7 are llmgw-80's; L8 (A2A
cluster's hardware holds, with CPU and GPU tok/s). Next M2 (every download pinned and checked against Hugging
Face, done 2026-10-30), then M5 (engines without the internet), M3, M4, M6.
**Scale.** O: more than one control plane answering clients; O1 (measuring one router's ceiling) first.
**Governance.**
7. **C1**: a tenant-admin role, and forms hidden from viewers.
8. **Audit before and after**: what a change changed, and changes made with the CLI recorded too.
@@ -1910,6 +1912,24 @@ served only from a definition written by hand. In order:
- N16. Versions apart from llmgw's releases (Strata's signed update feed, `--rollback-engine`); offline bundles.
- N17. `cache_salt` for KB isolation if Strata caches prefixes; `ctl bench` with its metrics path.
## O. More than one place answering clients (written 2026-10-30)
Every client request goes through the leader's router today: the floating address is on the leader, and a standby
passes `/v1` on to it (a standby's router sees an empty cluster). That is failover, not scale: the cluster serves what
one control plane's router can. In order:
- O1. Measure the ceiling first: one router in front of a fast fake engine (`sim-engine`, through an external model),
streams at 1 to 256 at once from another machine, the leader's CPU beside it. Is it the router, the guardrails, the
usage writes or TLS? Written down with the host's load (pve1 is oversubscribed).
- O2. Every control plane answers `/v1` itself (active-active):
- the leader's replica table followed live by the standbys, so their routers route;
- rate limits and budgets kept cluster-wide, not per process (J6), or three routers allow three times as much;
- a session's requests to one router and one replica (consistent hashing on the session), so the engines' prefix
caches keep paying;
- in front: IPVS on the keepalived the setup already installs, DNS round-robin, or the customer's balancer (J7, L9).
- O3. Only if O2 is not enough: stateless gateway machines (`llmgw server --role gateway`) in front of the leader's
control plane, as many as the traffic needs.
## D. Facts worth knowing before starting
- Admin API: `127.0.0.1:9090`, behind the token in