A band may wait longer than the rest; a busy refusal says when to return

A knowledge-base import sends far more than the cluster can take at once,
and every band waited the same 30 s. A band now has its own queue_ttl, so
one for bulk work waits behind people instead of being turned away.

And a request turned away because the cluster stayed busy — the queue
full (503) or waited as long as its band may (504) — now carries
Retry-After: 10. Without it llmgw-kb, which retries only what says when
to, counted each such passage as failed; that is what turned parts of the
metallurgy import away.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-10 16:11:09 +02:00
1 parent 9b7ceabd5b
commit 497e4b24a1
11 files changed
+113 -5

No files matched your search

+2 -1
View File
@@ -547,7 +547,8 @@ its usual exception with the gateway's message in it. The ones worth knowing:
| 401 | No key, a mistyped one, or one that was revoked or has expired. |
| 403 | The key may not use that model. |
| 429 | A rate limit or budget of the key, its project or its team was reached. `Retry-After` says when to try again, and is absent when waiting will not help — a spent monthly budget. |
| 503 | No model of that name is served (the message lists the ones that are), the model is still loading, or every replica is busy and the request waited as long as it may. The SDKs retry by themselves; a name that is not served will not start to be. |
| 503 | No model of that name is served (the message lists the ones that are), the model is still loading, or every replica is busy and the queue is full. The SDKs retry by themselves; a name that is not served will not start to be. |
| 504 | Every replica was busy for as long as the key's band may wait. `Retry-After` says when to send it again. |
## Priority