A band may wait longer than the rest; a busy refusal says when to return

A knowledge-base import sends far more than the cluster can take at once,
and every band waited the same 30 s. A band now has its own queue_ttl, so
one for bulk work waits behind people instead of being turned away.

And a request turned away because the cluster stayed busy — the queue
full (503) or waited as long as its band may (504) — now carries
Retry-After: 10. Without it llmgw-kb, which retries only what says when
to, counted each such passage as failed; that is what turned parts of the
metallurgy import away.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-10 16:11:09 +02:00
1 parent 9b7ceabd5b
commit 497e4b24a1
11 files changed
+113 -5

No files matched your search

+19
View File
@@ -356,6 +356,7 @@ rather than repeated.
| `name` | `string` | [admission.Band](#admissionband) |
| `priority` | `int` | [admission.Band](#admissionband) |
| `saturation_ceiling` | `float64` | [admission.Band](#admissionband) |
| `queue_ttl` | `time.Duration` | [admission.Band](#admissionband) |
| `bands` | `[]Band` | [admission.Config](#admissionconfig) |
| `default_band` | `string` | [admission.Config](#admissionconfig) |
| `queue_ttl` | `time.Duration` | [admission.Config](#admissionconfig) |
@@ -385,6 +386,7 @@ rather than repeated.
| `name` | `string` | [config.Band](#configband) |
| `priority` | `int` | [config.Band](#configband) |
| `saturation_ceiling` | `float64` | [config.Band](#configband) |
| `queue_ttl` | `time.Duration` | [config.Band](#configband) |
| `prompt_per_1k` | `int64` | [config.Price](#configprice) |
| `completion_per_1k` | `int64` | [config.Price](#configprice) |
| `per_image` | `int64` | [config.Price](#configprice) |
@@ -3013,6 +3015,15 @@ Priority orders bands; higher is served first.
SaturationCeiling is the pool saturation above which this band waits.
1.0 means "never hold this band back".
### `queue_ttl`
`time.Duration`
QueueTTL, set, is how long this band's requests may wait instead of
the controller's: a band for bulk work — a knowledge-base import, a
backfill — whose requests had rather wait ten minutes behind people
than be turned away after thirty seconds.
## admission.Config
### `bands`
@@ -3226,6 +3237,14 @@ Takes a [Tracing](#tracing) block.
`float64`
### `queue_ttl`
`time.Duration`
QueueTTL is how long this band's requests may wait for room; the
admission's queue_ttl unless set. Long for a band of bulk work, which
had rather wait behind people than be turned away.
## config.Price
### `prompt_per_1k`