llmgw ctl model gpus MODEL N [--profile P] [--in-place],
POST /api/v1/models/{name}/gpus and the console's GPUs…: how many cards of
one node each replica uses. Tensor parallel for vLLM, SGLang and
TensorRT-LLM, held to the model's KV heads (modelplan.SplitsHeads, now one
rule for the planner and this); the layers split for llama.cpp, its old
--tensor-split dropped. Refused for a count no node has, for cards that
would not hold the weights and the context, and for engines on one card.
The replicas start again beside the old ones; with no room beside them the
command says so, and --in-place stops each first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
51aeadae3e
commit
e1cff0d4e3
16 files changed
+753
-13
No files matched your search
@@ -25,6 +25,20 @@ is only released once its **exit criterion** is met, not when its code lands.
|
||||
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
|
||||
APIs change between minors without a deprecation period.
|
||||
|
||||
## Unreleased
|
||||
|
||||
### Added
|
||||
|
||||
- **A served model moved to more cards, or fewer.** `llmgw ctl model gpus
|
||||
MODEL N [--profile P] [--in-place]`, `POST /api/v1/models/{name}/gpus`, and
|
||||
GPUs… in the console's model menu. How many cards of one node each replica
|
||||
uses: tensor parallel for vLLM, SGLang and TensorRT-LLM, checked against
|
||||
the model's KV heads, and the layers split for llama.cpp. Refused for a
|
||||
count no node has, or whose cards would not hold the model and its
|
||||
context, with the figures; its replicas are then started again on it, one
|
||||
by one, beside the old — or, with `--in-place` when there is no room
|
||||
beside them, each stopped first.
|
||||
|
||||
## v0.15.1 — 2026-10-29
|
||||
|
||||
A 125B model on one card, a second mirror, and weights that run code
|
||||
|
||||
Reference in new issue
Block a user