Rate limits counted across the control planes with router.active_active
CI / test (push) Successful in 9m30s

Each router's per-counter request and token counts, this minute's and the
last, ride the standby's two-second exchange with the leader. The leader
answers each standby with everyone else's sum, its own included, and puts
the standbys' sum in force on its own router; each limiter adds the
others' counts to its sliding window.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-11-01 17:30:08 +01:00
1 parent 95fb5eeb6e
commit 540b518fde
17 files changed
+494 -28

No files matched your search

+10 -3
View File
@@ -2821,9 +2821,16 @@ Caddy doing that for three control planes, and
`server add` checks, and says what to add to the mirror's unit if it
refuses the new control plane — which would otherwise find out the first
time it leads and fetches a model.
- **Limits are per process.** Rate limits are counted in memory, and clients
reach the leader through the floating address, so a failover starts each
key's minute afresh.
- **Standbys can answer clients too.** With `router.active_active: true`
(`llmgw ctl config set router.active_active true`) a standby answers `/v1`
itself, routing to the leader's replicas, instead of passing it on: more
machines carry the traffic. Point clients at every control plane — DNS,
or your own load balancer checking `/ready` — since the floating address
stays on the leader. Rate limits stay the cluster's: each router's counts
reach the others every two seconds, and a standby that becomes the leader
keeps the minute it was counting. Without the setting, rate limits are
counted in the leader's memory and a failover starts each key's minute
afresh.
[deploy/ha/](../deploy/ha/) describes what each half does, for anyone putting
their own load balancer in front.