Displace the newest lowest-band request when the gateway queue is full
A total bound across bands (admission.queue_capacity_total, default half of all bands' capacity together); reached, a request is not refused while a lower band has requests queued, and the newest of the lowest band gets a 503 with Retry-After instead. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
623cdb5ba3
commit
3d3c96e422
10 files changed
+199
-4
No files matched your search
@@ -21,6 +21,7 @@ rather than repeated.
|
||||
| `band_header` | `string` | [Admission](#admission) |
|
||||
| `queue_ttl` | `time.Duration` | [Admission](#admission) |
|
||||
| `queue_capacity_per_band` | `int` | [Admission](#admission) |
|
||||
| `queue_capacity_total` | `int` | [Admission](#admission) |
|
||||
| `interval` | `time.Duration` | [Admission](#admission) |
|
||||
| `enabled` | `bool` | [Auth](#auth) |
|
||||
| `store` | `string` | [Auth](#auth) |
|
||||
@@ -228,6 +229,15 @@ should not be able to promote themselves.
|
||||
|
||||
`int`
|
||||
|
||||
### `queue_capacity_total`
|
||||
|
||||
`int`
|
||||
|
||||
QueueCapacityTotal bounds everything queued across bands. Reached, a
|
||||
request is not refused while a lower band has any queued: the newest of
|
||||
the lowest band is turned away instead, with a 503 and Retry-After.
|
||||
Zero takes half of all the bands' capacities together.
|
||||
|
||||
### `interval`
|
||||
|
||||
`time.Duration`
|
||||
|
||||
Reference in new issue
Block a user