Rate limits counted across the control planes with router.active_active
CI / test (push) Successful in 9m30s
CI / test (push) Successful in 9m30s
Each router's per-counter request and token counts, this minute's and the last, ride the standby's two-second exchange with the leader. The leader answers each standby with everyone else's sum, its own included, and puts the standbys' sum in force on its own router; each limiter adds the others' counts to its sliding window. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
95fb5eeb6e
commit
540b518fde
17 files changed
+494
-28
No files matched your search
+8
-2
@@ -1688,6 +1688,8 @@ how to fix it by hand (`ctl model resources`, then `ctl restart`).
|
||||
Rate limits are counted in each control plane's memory. A failover starts
|
||||
every key's minute afresh. Documented in install.md; worth a shared count in
|
||||
the store only if a customer's limits must hold across a failover.
|
||||
With `router.active_active` (O2, 2026-11-01) the counts are shared every 2 s,
|
||||
so a standby taking over keeps the minute; without it, as above.
|
||||
|
||||
### J7. nginx in front of several control planes (S)
|
||||
`deploy/proxy/Caddyfile` sends to whichever control plane answers `/leader`.
|
||||
@@ -1951,8 +1953,12 @@ one control plane's router can. In order:
|
||||
leader takes the last uses in (else a replica busy behind a standby is evicted as idle) and the on-demand models
|
||||
asked for; a catalog changed on the leader is read again from the store. Unheard from the leader for a minute,
|
||||
a standby routes nothing and passes `/v1` on as before. The portal, `/admin` and `/mcp` still go to the leader.
|
||||
- [~] budgets: each router raises its counters to the store's every 30 s, so another's spend is seen within that.
|
||||
Rate limits are still per process: up to N× a key's limit across N control planes (J6).
|
||||
- [x] budgets: each router raises its counters to the store's every 30 s, so another's spend is seen within that.
|
||||
- [x] rate limits cluster-wide (2026-11-01): each router's per-counter request and token counts, this minute's
|
||||
and the last as absolute figures, ride the same 2 s exchange; the leader answers each standby with everyone
|
||||
else's sum, its own included, and puts the standbys' sum in force on its own router. Each limiter adds the
|
||||
others' counts to its sliding window. Overshoot is bounded by what N routers allow in 2 s; a standby quiet for
|
||||
10 s stops counting. (J6's single-process case was never the problem.)
|
||||
- [ ] a session's requests to one router and one replica: the session ring already picks the same replica on
|
||||
every router for the same replica set; what is left is the same router, which is the balancer's (source hash).
|
||||
- [ ] in front: IPVS on the keepalived the setup already installs, DNS round-robin, or the customer's balancer
|
||||
|
||||
Reference in new issue
Block a user