A served model can be moved to more cards, or fewer
CI / test (push) Successful in 10m28s

llmgw ctl model gpus MODEL N [--profile P] [--in-place],
POST /api/v1/models/{name}/gpus and the console's GPUs…: how many cards of
one node each replica uses. Tensor parallel for vLLM, SGLang and
TensorRT-LLM, held to the model's KV heads (modelplan.SplitsHeads, now one
rule for the planner and this); the layers split for llama.cpp, its old
--tensor-split dropped. Refused for a count no node has, for cards that
would not hold the weights and the context, and for engines on one card.
The replicas start again beside the old ones; with no room beside them the
command says so, and --in-place stops each first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-29 16:40:49 +01:00
1 parent 51aeadae3e
commit e1cff0d4e3
16 files changed
+753 -13

No files matched your search

+30
View File
@@ -1648,6 +1648,36 @@ it, one by one, a new one ready before the old one stops. The console has the
same under a model's menu, Context length…. A model declared in the cluster's
settings is changed there (`llmgw ctl config edit`).
### Moving a served model to more cards
How many cards of one node each replica uses can be changed while the model
is served:
```
llmgw ctl model gpus qwen3-coder-30b 2 # two cards a replica
llmgw ctl model gpus llama-70b 4 --profile h100 # the model has several profiles on cards
llmgw ctl model gpus qwen3-coder-30b 2 --in-place # no room beside the running replica
```
vLLM, SGLang and TensorRT-LLM spread the model by tensor parallel (`tp`),
which needs the count to divide the model's KV heads or be divided by them:
a model with 8 splits 2, 4, 8 or 16 ways, not 3 or 12. llama.cpp has no
tensor parallel; it splits the layers over the cards (`gpus_per_node`), and
a `--tensor-split` written for the old cards is dropped. ComfyUI and Strata
run on one card. A count no node has is refused, and so is one whose cards
would not hold the weights and the context — with the figures; fewer cards
with a long context may need `llmgw ctl model context` first.
Its replicas are then started again on it, each new one ready beside the old
before the old stops. That needs the cards free beside it: on a node of two
cards with the model on one, two do not fit until it is gone, and the command
says so. `--in-place` stops each replica first and starts its replacement on
the cards it held and the free ones, and the model does not answer while the
new one loads — a model with several replicas keeps answering on the others.
A replica spread over several nodes is laid out by hand, with a fabric
(`nodes`, `gpus_per_node` in the profile). The console has the same under a
model's menu, GPUs….
### Trying new weights on some of the traffic first
When a model's repository publishes new weights, `llmgw ctl model update`