A band may wait longer than the rest; a busy refusal says when to return

A knowledge-base import sends far more than the cluster can take at once,
and every band waited the same 30 s. A band now has its own queue_ttl, so
one for bulk work waits behind people instead of being turned away.

And a request turned away because the cluster stayed busy — the queue
full (503) or waited as long as its band may (504) — now carries
Retry-After: 10. Without it llmgw-kb, which retries only what says when
to, counted each such passage as failed; that is what turned parts of the
metallurgy import away.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-10 16:11:09 +02:00
1 parent 9b7ceabd5b
commit 497e4b24a1
11 files changed
+113 -5

No files matched your search

+2 -1
View File
@@ -547,7 +547,8 @@ its usual exception with the gateway's message in it. The ones worth knowing:
| 401 | No key, a mistyped one, or one that was revoked or has expired. |
| 403 | The key may not use that model. |
| 429 | A rate limit or budget of the key, its project or its team was reached. `Retry-After` says when to try again, and is absent when waiting will not help — a spent monthly budget. |
| 503 | No model of that name is served (the message lists the ones that are), the model is still loading, or every replica is busy and the request waited as long as it may. The SDKs retry by themselves; a name that is not served will not start to be. |
| 503 | No model of that name is served (the message lists the ones that are), the model is still loading, or every replica is busy and the queue is full. The SDKs retry by themselves; a name that is not served will not start to be. |
| 504 | Every replica was busy for as long as the key's band may wait. `Retry-After` says when to send it again. |
## Priority
+18
View File
@@ -146,6 +146,24 @@ Rules that apply:
- Every document added or removed is in the audit log (`llmgw ctl audit`),
by name. The text is never recorded.
### A large import
A few thousand documents ask the embedding model for far more than it can
answer at once. Give the key that sends them a band of its own, below the
people's, that may wait for room for minutes rather than the 30 seconds the
others wait:
```sh
llmgw ctl config set router.admission.bands '[{name: gold, priority: 300, saturation_ceiling: 0.85},
{name: bronze, priority: 100, saturation_ceiling: 0.3}, {name: bulk, priority: 50, saturation_ceiling: 0.3, queue_ttl: 5m}]'
llmgw keys create import --project Acme/docs --band bulk
```
The people's questions then go first, and the import waits behind them. A
request that waits as long as its band allows is answered 504 with
`Retry-After`, and `llmgw-kb` sends it again rather than counting the
passage as failed. It gives up after 15 minutes.
## For everyone: `llmgw-kb`
### Get it
+19
View File
@@ -356,6 +356,7 @@ rather than repeated.
| `name` | `string` | [admission.Band](#admissionband) |
| `priority` | `int` | [admission.Band](#admissionband) |
| `saturation_ceiling` | `float64` | [admission.Band](#admissionband) |
| `queue_ttl` | `time.Duration` | [admission.Band](#admissionband) |
| `bands` | `[]Band` | [admission.Config](#admissionconfig) |
| `default_band` | `string` | [admission.Config](#admissionconfig) |
| `queue_ttl` | `time.Duration` | [admission.Config](#admissionconfig) |
@@ -385,6 +386,7 @@ rather than repeated.
| `name` | `string` | [config.Band](#configband) |
| `priority` | `int` | [config.Band](#configband) |
| `saturation_ceiling` | `float64` | [config.Band](#configband) |
| `queue_ttl` | `time.Duration` | [config.Band](#configband) |
| `prompt_per_1k` | `int64` | [config.Price](#configprice) |
| `completion_per_1k` | `int64` | [config.Price](#configprice) |
| `per_image` | `int64` | [config.Price](#configprice) |
@@ -3013,6 +3015,15 @@ Priority orders bands; higher is served first.
SaturationCeiling is the pool saturation above which this band waits.
1.0 means "never hold this band back".
### `queue_ttl`
`time.Duration`
QueueTTL, set, is how long this band's requests may wait instead of
the controller's: a band for bulk work — a knowledge-base import, a
backfill — whose requests had rather wait ten minutes behind people
than be turned away after thirty seconds.
## admission.Config
### `bands`
@@ -3226,6 +3237,14 @@ Takes a [Tracing](#tracing) block.
`float64`
### `queue_ttl`
`time.Duration`
QueueTTL is how long this band's requests may wait for room; the
admission's queue_ttl unless set. Long for a band of bulk work, which
had rather wait behind people than be turned away.
## config.Price
### `prompt_per_1k`