Rate limits counted across the control planes with router.active_active
CI / test (push) Successful in 9m30s
CI / test (push) Successful in 9m30s
Each router's per-counter request and token counts, this minute's and the last, ride the standby's two-second exchange with the leader. The leader answers each standby with everyone else's sum, its own included, and puts the standbys' sum in force on its own router; each limiter adds the others' counts to its sliding window. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
95fb5eeb6e
commit
540b518fde
17 files changed
+494
-28
No files matched your search
+10
-3
@@ -2821,9 +2821,16 @@ Caddy doing that for three control planes, and
|
||||
`server add` checks, and says what to add to the mirror's unit if it
|
||||
refuses the new control plane — which would otherwise find out the first
|
||||
time it leads and fetches a model.
|
||||
- **Limits are per process.** Rate limits are counted in memory, and clients
|
||||
reach the leader through the floating address, so a failover starts each
|
||||
key's minute afresh.
|
||||
- **Standbys can answer clients too.** With `router.active_active: true`
|
||||
(`llmgw ctl config set router.active_active true`) a standby answers `/v1`
|
||||
itself, routing to the leader's replicas, instead of passing it on: more
|
||||
machines carry the traffic. Point clients at every control plane — DNS,
|
||||
or your own load balancer checking `/ready` — since the floating address
|
||||
stays on the leader. Rate limits stay the cluster's: each router's counts
|
||||
reach the others every two seconds, and a standby that becomes the leader
|
||||
keeps the minute it was counting. Without the setting, rate limits are
|
||||
counted in the leader's memory and a failover starts each key's minute
|
||||
afresh.
|
||||
|
||||
[deploy/ha/](../deploy/ha/) describes what each half does, for anyone putting
|
||||
their own load balancer in front.
|
||||
|
||||
Reference in new issue
Block a user