Rate limits counted across the control planes with router.active_active

Each router's per-counter request and token counts, this minute's and the
last, ride the standby's two-second exchange with the leader. The leader
answers each standby with everyone else's sum, its own included, and puts
the standbys' sum in force on its own router; each limiter adds the
others' counts to its sliding window.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-11-01 17:30:08 +01:00
1 parent 95fb5eeb6e
commit 540b518fde
17 files changed
+494 -28

No files matched your search

+8 -2
View File
@@ -1688,6 +1688,8 @@ how to fix it by hand (`ctl model resources`, then `ctl restart`).
Rate limits are counted in each control plane's memory. A failover starts
every key's minute afresh. Documented in install.md; worth a shared count in
the store only if a customer's limits must hold across a failover.
With `router.active_active` (O2, 2026-11-01) the counts are shared every 2 s,
so a standby taking over keeps the minute; without it, as above.
### J7. nginx in front of several control planes (S)
`deploy/proxy/Caddyfile` sends to whichever control plane answers `/leader`.
@@ -1951,8 +1953,12 @@ one control plane's router can. In order:
leader takes the last uses in (else a replica busy behind a standby is evicted as idle) and the on-demand models
asked for; a catalog changed on the leader is read again from the store. Unheard from the leader for a minute,
a standby routes nothing and passes `/v1` on as before. The portal, `/admin` and `/mcp` still go to the leader.
- [~] budgets: each router raises its counters to the store's every 30 s, so another's spend is seen within that.
Rate limits are still per process: up to N× a key's limit across N control planes (J6).
- [x] budgets: each router raises its counters to the store's every 30 s, so another's spend is seen within that.
- [x] rate limits cluster-wide (2026-11-01): each router's per-counter request and token counts, this minute's
and the last as absolute figures, ride the same 2 s exchange; the leader answers each standby with everyone
else's sum, its own included, and puts the standbys' sum in force on its own router. Each limiter adds the
others' counts to its sliding window. Overshoot is bounded by what N routers allow in 2 s; a standby quiet for
10 s stops counting. (J6's single-process case was never the problem.)
- [ ] a session's requests to one router and one replica: the session ring already picks the same replica on
every router for the same replica set; what is left is the same router, which is the balancer's (source hash).
- [ ] in front: IPVS on the keepalived the setup already installs, DNS round-robin, or the customer's balancer