Standbys that answer the client API themselves, behind router.active_active
CI / test (push) Successful in 9m35s
CI / test (push) Successful in 9m35s
A standby follows the leader's Ready replicas over the agent port, every two seconds, and reports back the replicas its router used and the on-demand models it was asked for. No engine keys travel: every control plane derives them from the cluster secret. Budgets are raised to the store's every 30 s. Unheard from the leader for a minute, a standby passes /v1 on as before. Off by default. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
f664592948
commit
8b7eefef16
22 files changed
+936
-15
No files matched your search
@@ -30,6 +30,16 @@ APIs change between minors without a deprecation period.
|
||||
|
||||
### Added
|
||||
|
||||
- **Standbys that answer the client API themselves.** With
|
||||
`router.active_active: true` in the cluster's settings, a standby control
|
||||
plane routes `/v1` to the leader's replicas instead of passing every
|
||||
request to the leader, so more machines carry the traffic. It hears the
|
||||
replicas from the leader every two seconds over the agent port (no engine
|
||||
keys travel) and reports back what it used, so the leader does not evict
|
||||
a replica busy behind a standby, and the on-demand models it was asked
|
||||
for. Budgets are brought up to the store's every 30 seconds; rate limits
|
||||
are still counted per control plane. Off by default; the portal, `/admin`
|
||||
and `/mcp` still go to the leader.
|
||||
- **A served model's longest answer, changed in place.** `llmgw ctl model
|
||||
context <model> --max-output N` (and `max_output` on `POST
|
||||
/api/v1/models/{name}/context`, and the console's Context length… form)
|
||||
|
||||
Reference in new issue
Block a user