Merge admission-bulk into jobs-3d

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-10 17:35:39 +02:00
commit ec233d3586
20 files changed
+328 -40

No files matched your search

+4
View File
@@ -8837,6 +8837,10 @@
"gpu_class": {
"type": "string"
},
"source": {
"type": "string",
"description": "Source is the weights this profile loads, when not the model's: an embedding model served by vLLM from its repository on a card and by llama.cpp from a GGUF on CPUs is one model with two sources. Empty loads the model's. Updates and canaries move the model's source; a profile with its own keeps it."
},
"tp": {
"type": "integer"
},
+2 -1
View File
@@ -547,7 +547,8 @@ its usual exception with the gateway's message in it. The ones worth knowing:
| 401 | No key, a mistyped one, or one that was revoked or has expired. |
| 403 | The key may not use that model. |
| 429 | A rate limit or budget of the key, its project or its team was reached. `Retry-After` says when to try again, and is absent when waiting will not help — a spent monthly budget. |
| 503 | No model of that name is served (the message lists the ones that are), the model is still loading, or every replica is busy and the request waited as long as it may. The SDKs retry by themselves; a name that is not served will not start to be. |
| 503 | No model of that name is served (the message lists the ones that are), the model is still loading, or every replica is busy and the queue is full. The SDKs retry by themselves; a name that is not served will not start to be. |
| 504 | Every replica was busy for as long as the key's band may wait. `Retry-After` says when to send it again. |
## Priority
+21
View File
@@ -1038,6 +1038,27 @@ qwen2.5-coder-3b-instruct yes 1/1 ready coder
No restart, and it survives one: models added this way are stored, while
anything in `server.yaml` is declarative and returns on every start.
A profile may load weights of its own. An embedding model, for example, can
be served by vLLM from its repository on a card and by llama.cpp from a GGUF
on CPUs, as one model:
```yaml
name: qwen3-embed-4b
source: Qwen/Qwen3-Embedding-4B # what a profile loads unless it says
capabilities: { embeddings: true }
profiles:
- name: gpu
engine: vllm
gpu_class: 6000-24g
args: ["--runner", "pooling"]
- name: cpu
engine: llamacpp
source: hf:Qwen/Qwen3-Embedding-4B-GGUF/Qwen3-Embedding-4B-Q8_0.gguf
```
Updates and canaries move the model's `source`; a profile with its own keeps
it.
### Models that load when someone asks
A cluster with more models than GPUs can hold every model in its catalog and
+18
View File
@@ -146,6 +146,24 @@ Rules that apply:
- Every document added or removed is in the audit log (`llmgw ctl audit`),
by name. The text is never recorded.
### A large import
A few thousand documents ask the embedding model for far more than it can
answer at once. Give the key that sends them a band of its own, below the
people's, that may wait for room for minutes rather than the 30 seconds the
others wait:
```sh
llmgw ctl config set router.admission.bands '[{name: gold, priority: 300, saturation_ceiling: 0.85},
{name: bronze, priority: 100, saturation_ceiling: 0.3}, {name: bulk, priority: 50, saturation_ceiling: 0.3, queue_ttl: 5m}]'
llmgw keys create import --project Acme/docs --band bulk
```
The people's questions then go first, and the import waits behind them. A
request that waits as long as its band allows is answered 504 with
`Retry-After`, and `llmgw-kb` sends it again rather than counting the
passage as failed. It gives up after 15 minutes.
## For everyone: `llmgw-kb`
### Get it
+30
View File
@@ -220,6 +220,7 @@ rather than repeated.
| `engine` | `string` | [Profile](#profile) |
| `engine_version` | `string` | [Profile](#profile) |
| `gpu_class` | `string` | [Profile](#profile) |
| `source` | `string` | [Profile](#profile) |
| `tp` | `int` | [Profile](#profile) |
| `pp` | `int` | [Profile](#profile) |
| `gpus_per_node` | `int` | [Profile](#profile) |
@@ -356,6 +357,7 @@ rather than repeated.
| `name` | `string` | [admission.Band](#admissionband) |
| `priority` | `int` | [admission.Band](#admissionband) |
| `saturation_ceiling` | `float64` | [admission.Band](#admissionband) |
| `queue_ttl` | `time.Duration` | [admission.Band](#admissionband) |
| `bands` | `[]Band` | [admission.Config](#admissionconfig) |
| `default_band` | `string` | [admission.Config](#admissionconfig) |
| `queue_ttl` | `time.Duration` | [admission.Config](#admissionconfig) |
@@ -385,6 +387,7 @@ rather than repeated.
| `name` | `string` | [config.Band](#configband) |
| `priority` | `int` | [config.Band](#configband) |
| `saturation_ceiling` | `float64` | [config.Band](#configband) |
| `queue_ttl` | `time.Duration` | [config.Band](#configband) |
| `prompt_per_1k` | `int64` | [config.Price](#configprice) |
| `completion_per_1k` | `int64` | [config.Price](#configprice) |
| `per_image` | `int64` | [config.Price](#configprice) |
@@ -1923,6 +1926,16 @@ version starts with it: "0.11" takes vLLM 0.11.0 and 0.11.1, not
`string`
### `source`
`string`
Source is the weights this profile loads, when not the model's: an
embedding model served by vLLM from its repository on a card and by
llama.cpp from a GGUF on CPUs is one model with two sources. Empty
loads the model's. Updates and canaries move the model's source; a
profile with its own keeps it.
### `tp`
`int`
@@ -3013,6 +3026,15 @@ Priority orders bands; higher is served first.
SaturationCeiling is the pool saturation above which this band waits.
1.0 means "never hold this band back".
### `queue_ttl`
`time.Duration`
QueueTTL, set, is how long this band's requests may wait instead of
the controller's: a band for bulk work — a knowledge-base import, a
backfill — whose requests had rather wait ten minutes behind people
than be turned away after thirty seconds.
## admission.Config
### `bands`
@@ -3226,6 +3248,14 @@ Takes a [Tracing](#tracing) block.
`float64`
### `queue_ttl`
`time.Duration`
QueueTTL is how long this band's requests may wait for room; the
admission's queue_ttl unless set. Long for a band of bulk work, which
had rather wait behind people than be turned away.
## config.Price
### `prompt_per_1k`