A served model can be moved to more cards, or fewer
llmgw ctl model gpus MODEL N [--profile P] [--in-place],
POST /api/v1/models/{name}/gpus and the console's GPUs…: how many cards of
one node each replica uses. Tensor parallel for vLLM, SGLang and
TensorRT-LLM, held to the model's KV heads (modelplan.SplitsHeads, now one
rule for the planner and this); the layers split for llama.cpp, its old
--tensor-split dropped. Refused for a count no node has, for cards that
would not hold the weights and the context, and for engines on one card.
The replicas start again beside the old ones; with no room beside them the
command says so, and --in-place stops each first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
51aeadae3e
commit
e1cff0d4e3
16 files changed
+753
-13
No files matched your search
@@ -2964,6 +2964,58 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/models/{name}/gpus": {
|
||||
"post": {
|
||||
"operationId": "postModelsNameGpus",
|
||||
"summary": "Spread a served model over more cards or fewer: its profile, checked against its KV heads, its cards' memory and the nodes, then its replicas restarted",
|
||||
"description": "Tensor parallel for vLLM, SGLang and TensorRT-LLM, a layer split for llama.cpp, on one node. With no room beside the running replicas, in_place stops each first.",
|
||||
"tags": [
|
||||
"models"
|
||||
],
|
||||
"parameters": [
|
||||
{
|
||||
"name": "name",
|
||||
"in": "path",
|
||||
"required": true,
|
||||
"description": "The model's name in the catalog.",
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
{
|
||||
"$ref": "#/components/parameters/Actor"
|
||||
}
|
||||
],
|
||||
"requestBody": {
|
||||
"required": true,
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/modelserve.GPUsRequest"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "OK",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/modelserve.GPUsChange"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"401": {
|
||||
"$ref": "#/components/responses/Unauthorized"
|
||||
},
|
||||
"default": {
|
||||
"$ref": "#/components/responses/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/models/{name}/headroom": {
|
||||
"get": {
|
||||
"operationId": "getModelsNameHeadroom",
|
||||
@@ -11133,6 +11185,67 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"modelserve.GPUsChange": {
|
||||
"type": "object",
|
||||
"description": "GPUsChange is what was changed, and the replicas started again on it.",
|
||||
"properties": {
|
||||
"model": {
|
||||
"type": "string"
|
||||
},
|
||||
"profile": {
|
||||
"type": "string"
|
||||
},
|
||||
"engine": {
|
||||
"type": "string"
|
||||
},
|
||||
"gpu_class": {
|
||||
"type": "string"
|
||||
},
|
||||
"before": {
|
||||
"type": "integer"
|
||||
},
|
||||
"after": {
|
||||
"type": "integer"
|
||||
},
|
||||
"args": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"per_card": {
|
||||
"type": "string",
|
||||
"description": "PerCard is what each card holds after, when it could be worked out."
|
||||
},
|
||||
"restarted": {
|
||||
"type": "integer"
|
||||
},
|
||||
"in_place": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"note": {
|
||||
"type": "string",
|
||||
"description": "Note is what could not be checked, said once."
|
||||
}
|
||||
}
|
||||
},
|
||||
"modelserve.GPUsRequest": {
|
||||
"type": "object",
|
||||
"description": "GPUsRequest is how many cards a replica is to use.",
|
||||
"properties": {
|
||||
"gpus": {
|
||||
"type": "integer"
|
||||
},
|
||||
"profile": {
|
||||
"type": "string",
|
||||
"description": "Profile is the one to change, when the model has more than one profile on cards."
|
||||
},
|
||||
"in_place": {
|
||||
"type": "boolean",
|
||||
"description": "InPlace stops each replica before its replacement starts: for when no node has the cards free beside it. It does not answer while the new one loads."
|
||||
}
|
||||
}
|
||||
},
|
||||
"modelserve.Job": {
|
||||
"type": "object",
|
||||
"description": "Job is one model being served, and where it stands.",
|
||||
|
||||
@@ -1648,6 +1648,36 @@ it, one by one, a new one ready before the old one stops. The console has the
|
||||
same under a model's menu, Context length…. A model declared in the cluster's
|
||||
settings is changed there (`llmgw ctl config edit`).
|
||||
|
||||
### Moving a served model to more cards
|
||||
|
||||
How many cards of one node each replica uses can be changed while the model
|
||||
is served:
|
||||
|
||||
```
|
||||
llmgw ctl model gpus qwen3-coder-30b 2 # two cards a replica
|
||||
llmgw ctl model gpus llama-70b 4 --profile h100 # the model has several profiles on cards
|
||||
llmgw ctl model gpus qwen3-coder-30b 2 --in-place # no room beside the running replica
|
||||
```
|
||||
|
||||
vLLM, SGLang and TensorRT-LLM spread the model by tensor parallel (`tp`),
|
||||
which needs the count to divide the model's KV heads or be divided by them:
|
||||
a model with 8 splits 2, 4, 8 or 16 ways, not 3 or 12. llama.cpp has no
|
||||
tensor parallel; it splits the layers over the cards (`gpus_per_node`), and
|
||||
a `--tensor-split` written for the old cards is dropped. ComfyUI and Strata
|
||||
run on one card. A count no node has is refused, and so is one whose cards
|
||||
would not hold the weights and the context — with the figures; fewer cards
|
||||
with a long context may need `llmgw ctl model context` first.
|
||||
|
||||
Its replicas are then started again on it, each new one ready beside the old
|
||||
before the old stops. That needs the cards free beside it: on a node of two
|
||||
cards with the model on one, two do not fit until it is gone, and the command
|
||||
says so. `--in-place` stops each replica first and starts its replacement on
|
||||
the cards it held and the free ones, and the model does not answer while the
|
||||
new one loads — a model with several replicas keeps answering on the others.
|
||||
A replica spread over several nodes is laid out by hand, with a fabric
|
||||
(`nodes`, `gpus_per_node` in the profile). The console has the same under a
|
||||
model's menu, GPUs….
|
||||
|
||||
### Trying new weights on some of the traffic first
|
||||
|
||||
When a model's repository publishes new weights, `llmgw ctl model update`
|
||||
|
||||
Reference in new issue
Block a user