Rate limits counted across the control planes with router.active_active
CI / test (push) Successful in 9m30s

Each router's per-counter request and token counts, this minute's and the
last, ride the standby's two-second exchange with the leader. The leader
answers each standby with everyone else's sum, its own included, and puts
the standbys' sum in force on its own router; each limiter adds the
others' counts to its sliding window.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-11-01 17:30:08 +01:00
1 parent 95fb5eeb6e
commit 540b518fde
17 files changed
+494 -28

No files matched your search

+48
View File
@@ -9770,6 +9770,42 @@
}
}
},
"accounting.Counts": {
"type": "object",
"description": "Counts are one limiter's counts by counter: this minute's and the one before, as absolute figures, so reporting them twice changes nothing.",
"properties": {
"minute": {
"type": "string",
"format": "date-time"
},
"current": {
"type": "object",
"additionalProperties": {
"type": "integer",
"format": "int64"
}
},
"previous": {
"type": "object",
"additionalProperties": {
"type": "integer",
"format": "int64"
}
}
}
},
"accounting.Rates": {
"type": "object",
"description": "Rates are an accountant's two limiters' counts.",
"properties": {
"requests": {
"$ref": "#/components/schemas/accounting.Counts"
},
"tokens": {
"$ref": "#/components/schemas/accounting.Counts"
}
}
},
"bench.Capacity": {
"type": "object",
"description": "Capacity is what one replica was measured to serve, kept so the measurement outlives the terminal it was printed in.\n\nThe conclusion of a run, not the run. The ramp itself — every concurrency level, its latencies, its errors — is worth reading once, while deciding whether the run was sound; it is not worth carrying around afterwards, and storing it would grow without bound as a model is re-benchmarked. What a later reader needs is the number and enough provenance to judge whether it is still true.",
@@ -10632,6 +10668,10 @@
"type": "object",
"description": "Activity is what a standby's router reports back: when it last sent each replica a request, and the on-demand models it was asked for. The leader evicts and loads by these, and without them a replica busy behind a standby looks idle to the one machine that decides.",
"properties": {
"from": {
"type": "string",
"description": "From names the standby, so its counts replace its own last ones and are not given back to it."
},
"last_used": {
"type": "object",
"additionalProperties": {
@@ -10644,6 +10684,10 @@
"items": {
"type": "string"
}
},
"rates": {
"$ref": "#/components/schemas/accounting.Rates",
"description": "Rates are its router's own counts against each limit."
}
}
},
@@ -10684,6 +10728,10 @@
"catalog": {
"type": "string",
"description": "Catalog changes whenever the leader puts a catalog in force. Only compared with the last one a standby saw, never with its own: a model added, scaled or given a canary is in the store, and the standby reads it again from there."
},
"rates": {
"$ref": "#/components/schemas/accounting.Rates",
"description": "Rates are what every other control plane's router has counted against each limit, the leader's among them: added to the asker's own, they make a key's limits the cluster's."
}
}
},