A served model can be moved to more cards, or fewer
CI / test (push) Successful in 10m28s

llmgw ctl model gpus MODEL N [--profile P] [--in-place],
POST /api/v1/models/{name}/gpus and the console's GPUs…: how many cards of
one node each replica uses. Tensor parallel for vLLM, SGLang and
TensorRT-LLM, held to the model's KV heads (modelplan.SplitsHeads, now one
rule for the planner and this); the layers split for llama.cpp, its old
--tensor-split dropped. Refused for a count no node has, for cards that
would not hold the weights and the context, and for engines on one card.
The replicas start again beside the old ones; with no room beside them the
command says so, and --in-place stops each first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-29 16:40:49 +01:00
1 parent 51aeadae3e
commit e1cff0d4e3
16 files changed
+753 -13

No files matched your search

+1 -4
View File
@@ -155,10 +155,7 @@ func fitsOn(m Model, h Hardware, d Demand, p Precision, context int) []Fit {
if nodes > 1 && h.Fabric == "" {
continue
}
// The KV heads are split among the cards, or past their number each
// card holds a copy of one: the width divides them or they divide
// it. A model with 8 KV heads splits 2, 4, 8 or 16 ways, not 12.
if m.KVHeads > 0 && m.KVHeads%width != 0 && width%m.KVHeads != 0 {
if !SplitsHeads(m.KVHeads, width) {
continue
}