Standbys that answer the client API themselves, behind router.active_active
CI / test (push) Successful in 9m35s

A standby follows the leader's Ready replicas over the agent port, every
two seconds, and reports back the replicas its router used and the
on-demand models it was asked for. No engine keys travel: every control
plane derives them from the cluster secret. Budgets are raised to the
store's every 30 s. Unheard from the leader for a minute, a standby
passes /v1 on as before. Off by default.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-11-01 13:47:47 +01:00
1 parent f664592948
commit 8b7eefef16
22 files changed
+936 -15

No files matched your search

+10
View File
@@ -30,6 +30,16 @@ APIs change between minors without a deprecation period.
### Added
- **Standbys that answer the client API themselves.** With
`router.active_active: true` in the cluster's settings, a standby control
plane routes `/v1` to the leader's replicas instead of passing every
request to the leader, so more machines carry the traffic. It hears the
replicas from the leader every two seconds over the agent port (no engine
keys travel) and reports back what it used, so the leader does not evict
a replica busy behind a standby, and the on-demand models it was asked
for. Budgets are brought up to the store's every 30 seconds; rate limits
are still counted per control plane. Off by default; the portal, `/admin`
and `/mcp` still go to the leader.
- **A served model's longest answer, changed in place.** `llmgw ctl model
context <model> --max-output N` (and `max_output` on `POST
/api/v1/models/{name}/context`, and the console's Context length… form)