Rate limits counted across the control planes with router.active_active
CI / test (push) Successful in 9m30s
CI / test (push) Successful in 9m30s
Each router's per-counter request and token counts, this minute's and the last, ride the standby's two-second exchange with the leader. The leader answers each standby with everyone else's sum, its own included, and puts the standbys' sum in force on its own router; each limiter adds the others' counts to its sliding window. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
95fb5eeb6e
commit
540b518fde
17 files changed
+494
-28
No files matched your search
@@ -9770,6 +9770,42 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"accounting.Counts": {
|
||||
"type": "object",
|
||||
"description": "Counts are one limiter's counts by counter: this minute's and the one before, as absolute figures, so reporting them twice changes nothing.",
|
||||
"properties": {
|
||||
"minute": {
|
||||
"type": "string",
|
||||
"format": "date-time"
|
||||
},
|
||||
"current": {
|
||||
"type": "object",
|
||||
"additionalProperties": {
|
||||
"type": "integer",
|
||||
"format": "int64"
|
||||
}
|
||||
},
|
||||
"previous": {
|
||||
"type": "object",
|
||||
"additionalProperties": {
|
||||
"type": "integer",
|
||||
"format": "int64"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"accounting.Rates": {
|
||||
"type": "object",
|
||||
"description": "Rates are an accountant's two limiters' counts.",
|
||||
"properties": {
|
||||
"requests": {
|
||||
"$ref": "#/components/schemas/accounting.Counts"
|
||||
},
|
||||
"tokens": {
|
||||
"$ref": "#/components/schemas/accounting.Counts"
|
||||
}
|
||||
}
|
||||
},
|
||||
"bench.Capacity": {
|
||||
"type": "object",
|
||||
"description": "Capacity is what one replica was measured to serve, kept so the measurement outlives the terminal it was printed in.\n\nThe conclusion of a run, not the run. The ramp itself — every concurrency level, its latencies, its errors — is worth reading once, while deciding whether the run was sound; it is not worth carrying around afterwards, and storing it would grow without bound as a model is re-benchmarked. What a later reader needs is the number and enough provenance to judge whether it is still true.",
|
||||
@@ -10632,6 +10668,10 @@
|
||||
"type": "object",
|
||||
"description": "Activity is what a standby's router reports back: when it last sent each replica a request, and the on-demand models it was asked for. The leader evicts and loads by these, and without them a replica busy behind a standby looks idle to the one machine that decides.",
|
||||
"properties": {
|
||||
"from": {
|
||||
"type": "string",
|
||||
"description": "From names the standby, so its counts replace its own last ones and are not given back to it."
|
||||
},
|
||||
"last_used": {
|
||||
"type": "object",
|
||||
"additionalProperties": {
|
||||
@@ -10644,6 +10684,10 @@
|
||||
"items": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"rates": {
|
||||
"$ref": "#/components/schemas/accounting.Rates",
|
||||
"description": "Rates are its router's own counts against each limit."
|
||||
}
|
||||
}
|
||||
},
|
||||
@@ -10684,6 +10728,10 @@
|
||||
"catalog": {
|
||||
"type": "string",
|
||||
"description": "Catalog changes whenever the leader puts a catalog in force. Only compared with the last one a standby saw, never with its own: a model added, scaled or given a canary is in the store, and the standby reads it again from there."
|
||||
},
|
||||
"rates": {
|
||||
"$ref": "#/components/schemas/accounting.Rates",
|
||||
"description": "Rates are what every other control plane's router has counted against each limit, the leader's among them: added to the asker's own, they make a key's limits the cluster's."
|
||||
}
|
||||
}
|
||||
},
|
||||
|
||||
+10
-3
@@ -2821,9 +2821,16 @@ Caddy doing that for three control planes, and
|
||||
`server add` checks, and says what to add to the mirror's unit if it
|
||||
refuses the new control plane — which would otherwise find out the first
|
||||
time it leads and fetches a model.
|
||||
- **Limits are per process.** Rate limits are counted in memory, and clients
|
||||
reach the leader through the floating address, so a failover starts each
|
||||
key's minute afresh.
|
||||
- **Standbys can answer clients too.** With `router.active_active: true`
|
||||
(`llmgw ctl config set router.active_active true`) a standby answers `/v1`
|
||||
itself, routing to the leader's replicas, instead of passing it on: more
|
||||
machines carry the traffic. Point clients at every control plane — DNS,
|
||||
or your own load balancer checking `/ready` — since the floating address
|
||||
stays on the leader. Rate limits stay the cluster's: each router's counts
|
||||
reach the others every two seconds, and a standby that becomes the leader
|
||||
keeps the minute it was counting. Without the setting, rate limits are
|
||||
counted in the leader's memory and a failover starts each key's minute
|
||||
afresh.
|
||||
|
||||
[deploy/ha/](../deploy/ha/) describes what each half does, for anyone putting
|
||||
their own load balancer in front.
|
||||
|
||||
+6
-5
@@ -2818,11 +2818,12 @@ written into the setup scripts and whose attempts are counted where.
|
||||
ActiveActive has standby control planes answer the client API
|
||||
themselves, routing to the leader's replicas, rather than passing
|
||||
every request to the leader. More machines carry the traffic, which
|
||||
is the point; what each router keeps in memory is then its own:
|
||||
rate limits are counted per control plane, so a key may make up to
|
||||
as many times its limit as there are control planes taking its
|
||||
requests; a budget is checked against the store every half minute;
|
||||
cached answers and prefix affinity are split between them. Off by
|
||||
is the point. Rate limits are the cluster's: each router's counts
|
||||
reach the others through the leader every two seconds, so a key
|
||||
spread over N control planes may pass its limit by what they allow
|
||||
in those two seconds. A budget is checked against the store every
|
||||
half minute; cached answers and prefix affinity are split between
|
||||
the routers. Off by
|
||||
default. Which machine a client reaches is the load balancer's or
|
||||
DNS's to say; the floating address stays on the leader.
|
||||
|
||||
|
||||
Reference in new issue
Block a user