A band may wait longer than the rest; a busy refusal says when to return
A knowledge-base import sends far more than the cluster can take at once, and every band waited the same 30 s. A band now has its own queue_ttl, so one for bulk work waits behind people instead of being turned away. And a request turned away because the cluster stayed busy — the queue full (503) or waited as long as its band may (504) — now carries Retry-After: 10. Without it llmgw-kb, which retries only what says when to, counted each such passage as failed; that is what turned parts of the metallurgy import away. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
9b7ceabd5b
commit
497e4b24a1
11 files changed
+113
-5
No files matched your search
@@ -146,6 +146,24 @@ Rules that apply:
|
||||
- Every document added or removed is in the audit log (`llmgw ctl audit`),
|
||||
by name. The text is never recorded.
|
||||
|
||||
### A large import
|
||||
|
||||
A few thousand documents ask the embedding model for far more than it can
|
||||
answer at once. Give the key that sends them a band of its own, below the
|
||||
people's, that may wait for room for minutes rather than the 30 seconds the
|
||||
others wait:
|
||||
|
||||
```sh
|
||||
llmgw ctl config set router.admission.bands '[{name: gold, priority: 300, saturation_ceiling: 0.85},
|
||||
{name: bronze, priority: 100, saturation_ceiling: 0.3}, {name: bulk, priority: 50, saturation_ceiling: 0.3, queue_ttl: 5m}]'
|
||||
llmgw keys create import --project Acme/docs --band bulk
|
||||
```
|
||||
|
||||
The people's questions then go first, and the import waits behind them. A
|
||||
request that waits as long as its band allows is answered 504 with
|
||||
`Retry-After`, and `llmgw-kb` sends it again rather than counting the
|
||||
passage as failed. It gives up after 15 minutes.
|
||||
|
||||
## For everyone: `llmgw-kb`
|
||||
|
||||
### Get it
|
||||
|
||||
Reference in new issue
Block a user