A served model's context can be changed, engine and memory with it

ctl model context MODEL TOKENS [--slots N], POST /api/v1/models/{name}/context
and the console's Context length…: the catalog, the engine's arguments and
the KV memory change together, refused past the trained context or what the
cards hold, then the replicas restart one by one.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-29 15:50:04 +01:00
1 parent 1908bfb4b2
commit 986981ac81
14 files changed
+644 -1

No files matched your search

+19
View File
@@ -1629,6 +1629,25 @@ Not yet: models declared in server.yaml cannot name a peer's model as a
fallback (the catalog in the file is checked before the peers are read), and
a peer behind a private CA needs that CA in the system's trust store.
### Changing a served model's context
How much one request may use is set when a model is served. To change it
afterwards:
```
llmgw ctl model context qwen-7b 32768 # tokens a request
llmgw ctl model context qwen-7b 32768 --slots 4 # llama.cpp: four at once
```
The catalog, the engine's arguments (`--ctx-size` and `--parallel` for
llama.cpp, `--max-model-len` for vLLM, `--context-length` for SGLang) and
the memory its KV cache takes change together. A context longer than the
model was trained for is refused, and so is one its cards cannot hold — with
the figures, so you know how far to go. Its replicas are then started again on
it, one by one, a new one ready before the old one stops. The console has the
same under a model's menu, Context length…. A model declared in the cluster's
settings is changed there (`llmgw ctl config edit`).
### Trying new weights on some of the traffic first
When a model's repository publishes new weights, `llmgw ctl model update`