Rate limits counted across the control planes with router.active_active
CI / test (push) Successful in 9m30s

Each router's per-counter request and token counts, this minute's and the
last, ride the standby's two-second exchange with the leader. The leader
answers each standby with everyone else's sum, its own included, and puts
the standbys' sum in force on its own router; each limiter adds the
others' counts to its sliding window.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-11-01 17:30:08 +01:00
1 parent 95fb5eeb6e
commit 540b518fde
17 files changed
+494 -28

No files matched your search

+48
View File
@@ -9770,6 +9770,42 @@
}
}
},
"accounting.Counts": {
"type": "object",
"description": "Counts are one limiter's counts by counter: this minute's and the one before, as absolute figures, so reporting them twice changes nothing.",
"properties": {
"minute": {
"type": "string",
"format": "date-time"
},
"current": {
"type": "object",
"additionalProperties": {
"type": "integer",
"format": "int64"
}
},
"previous": {
"type": "object",
"additionalProperties": {
"type": "integer",
"format": "int64"
}
}
}
},
"accounting.Rates": {
"type": "object",
"description": "Rates are an accountant's two limiters' counts.",
"properties": {
"requests": {
"$ref": "#/components/schemas/accounting.Counts"
},
"tokens": {
"$ref": "#/components/schemas/accounting.Counts"
}
}
},
"bench.Capacity": {
"type": "object",
"description": "Capacity is what one replica was measured to serve, kept so the measurement outlives the terminal it was printed in.\n\nThe conclusion of a run, not the run. The ramp itself — every concurrency level, its latencies, its errors — is worth reading once, while deciding whether the run was sound; it is not worth carrying around afterwards, and storing it would grow without bound as a model is re-benchmarked. What a later reader needs is the number and enough provenance to judge whether it is still true.",
@@ -10632,6 +10668,10 @@
"type": "object",
"description": "Activity is what a standby's router reports back: when it last sent each replica a request, and the on-demand models it was asked for. The leader evicts and loads by these, and without them a replica busy behind a standby looks idle to the one machine that decides.",
"properties": {
"from": {
"type": "string",
"description": "From names the standby, so its counts replace its own last ones and are not given back to it."
},
"last_used": {
"type": "object",
"additionalProperties": {
@@ -10644,6 +10684,10 @@
"items": {
"type": "string"
}
},
"rates": {
"$ref": "#/components/schemas/accounting.Rates",
"description": "Rates are its router's own counts against each limit."
}
}
},
@@ -10684,6 +10728,10 @@
"catalog": {
"type": "string",
"description": "Catalog changes whenever the leader puts a catalog in force. Only compared with the last one a standby saw, never with its own: a model added, scaled or given a canary is in the store, and the standby reads it again from there."
},
"rates": {
"$ref": "#/components/schemas/accounting.Rates",
"description": "Rates are what every other control plane's router has counted against each limit, the leader's among them: added to the asker's own, they make a key's limits the cluster's."
}
}
},
+10 -3
View File
@@ -2821,9 +2821,16 @@ Caddy doing that for three control planes, and
`server add` checks, and says what to add to the mirror's unit if it
refuses the new control plane — which would otherwise find out the first
time it leads and fetches a model.
- **Limits are per process.** Rate limits are counted in memory, and clients
reach the leader through the floating address, so a failover starts each
key's minute afresh.
- **Standbys can answer clients too.** With `router.active_active: true`
(`llmgw ctl config set router.active_active true`) a standby answers `/v1`
itself, routing to the leader's replicas, instead of passing it on: more
machines carry the traffic. Point clients at every control plane — DNS,
or your own load balancer checking `/ready` — since the floating address
stays on the leader. Rate limits stay the cluster's: each router's counts
reach the others every two seconds, and a standby that becomes the leader
keeps the minute it was counting. Without the setting, rate limits are
counted in the leader's memory and a failover starts each key's minute
afresh.
[deploy/ha/](../deploy/ha/) describes what each half does, for anyone putting
their own load balancer in front.
+6 -5
View File
@@ -2818,11 +2818,12 @@ written into the setup scripts and whose attempts are counted where.
ActiveActive has standby control planes answer the client API
themselves, routing to the leader's replicas, rather than passing
every request to the leader. More machines carry the traffic, which
is the point; what each router keeps in memory is then its own:
rate limits are counted per control plane, so a key may make up to
as many times its limit as there are control planes taking its
requests; a budget is checked against the store every half minute;
cached answers and prefix affinity are split between them. Off by
is the point. Rate limits are the cluster's: each router's counts
reach the others through the leader every two seconds, so a key
spread over N control planes may pass its limit by what they allow
in those two seconds. A budget is checked against the store every
half minute; cached answers and prefix affinity are split between
the routers. Off by
default. Which machine a client reaches is the load balancer's or
DNS's to say; the floating address stays on the leader.