A band may wait longer than the rest; a busy refusal says when to return

A knowledge-base import sends far more than the cluster can take at once,
and every band waited the same 30 s. A band now has its own queue_ttl, so
one for bulk work waits behind people instead of being turned away.

And a request turned away because the cluster stayed busy — the queue
full (503) or waited as long as its band may (504) — now carries
Retry-After: 10. Without it llmgw-kb, which retries only what says when
to, counted each such passage as failed; that is what turned parts of the
metallurgy import away.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-10 16:11:09 +02:00
1 parent 9b7ceabd5b
commit 497e4b24a1
11 files changed
+113 -5

No files matched your search

+18
View File
@@ -146,6 +146,24 @@ Rules that apply:
- Every document added or removed is in the audit log (`llmgw ctl audit`),
by name. The text is never recorded.
### A large import
A few thousand documents ask the embedding model for far more than it can
answer at once. Give the key that sends them a band of its own, below the
people's, that may wait for room for minutes rather than the 30 seconds the
others wait:
```sh
llmgw ctl config set router.admission.bands '[{name: gold, priority: 300, saturation_ceiling: 0.85},
{name: bronze, priority: 100, saturation_ceiling: 0.3}, {name: bulk, priority: 50, saturation_ceiling: 0.3, queue_ttl: 5m}]'
llmgw keys create import --project Acme/docs --band bulk
```
The people's questions then go first, and the import waits behind them. A
request that waits as long as its band allows is answered 504 with
`Retry-After`, and `llmgw-kb` sends it again rather than counting the
passage as failed. It gives up after 15 minutes.
## For everyone: `llmgw-kb`
### Get it