Commit Graph
9 Commits
Author SHA1 Message Date
Ilker ManapandClaude Opus 5.5 f41c96d30f Strata's DeepSeek not offered on cards its engine was not built for, and refused there by the agent
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-30 13:00:21 +01:00
Ilker Manap d6f8f433f4 CHANGELOG: one list 2026-10-30 10:39:30 +01:00
Ilker ManapandClaude Opus 5.5 e1cff0d4e3 A served model can be moved to more cards, or fewer
llmgw ctl model gpus MODEL N [--profile P] [--in-place],
POST /api/v1/models/{name}/gpus and the console's GPUs…: how many cards of
one node each replica uses. Tensor parallel for vLLM, SGLang and
TensorRT-LLM, held to the model's KV heads (modelplan.SplitsHeads, now one
rule for the planner and this); the layers split for llama.cpp, its old
--tensor-split dropped. Refused for a count no node has, for cards that
would not hold the weights and the context, and for engines on one card.
The replicas start again beside the old ones; with no room beside them the
command says so, and --in-place stops each first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 16:40:49 +01:00
Ilker ManapandClaude Opus 5.5 986981ac81 A served model's context can be changed, engine and memory with it
ctl model context MODEL TOKENS [--slots N], POST /api/v1/models/{name}/context
and the console's Context length…: the catalog, the engine's arguments and
the KV memory change together, refused past the trained context or what the
cards hold, then the replicas restart one by one.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-29 15:50:04 +01:00
Ilker ManapandClaude Opus 5.5 4fe9b46451 A plan for a big model: experts spread, any number of nodes, memory said
A mixture-of-experts model on more than one card is served with vLLM's
--enable-expert-parallel; config.json's experts are read by any of their
names, and a model that also reads pictures is sized by its text_config.
Tensor parallelism past the KV heads is allowed when it is a multiple of
them (Qwen3-235B, four KV heads, runs on eight cards), which the plan
had refused. Several nodes are offered in any number, not only powers of
two. Each way says what it needs of a card, part by part, and of all its
cards; the serve form checks a context longer than planned still fits.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-20 08:45:33 +02:00
Ilker ManapandClaude Opus 5.5 0fd55a862b MIG: a card in MIG mode is placed as its instances, each a GPU of class <card>-mig-<profile>
Discovery reports a card in MIG mode as its instances instead of the whole
card, on both paths: nvidia-smi (-L names them, -q -x gives their memory,
ids and processes) and NVML (GetMigMode, GetMaxMigDeviceCount,
GetMigDeviceHandleByIndex, behind an interface so it is tested with a fake).
Each instance has its MIG UUID, its own memory, and a class from the card's
and the profile: a100-80g-mig-3g.40gb. Placement numbers instances
1000 + 100 x card + instance and every view shows them as card/migN.

A replica on MIG runs on exactly one instance: a process opens one MIG
device and instances have no NVLink between them, so tp, pp, gpus or nodes
above one on a MIG class are refused in catalog validation and again in
placement, and the planner offers one instance or none. The agent gives
the engine CUDA_VISIBLE_DEVICES=MIG-<uuid> (no Vulkan device variable on
MIG). Memory and processes are per instance; heat, power and wear the
card's; utilisation is left out rather than reported as zero. A card in MIG
mode with nothing carved stays out of placement with the reason, and
llmgw doctor reports each card's mode and instances and warns of the states
in which nothing can run. ctl nodes, ctl status, the console and the plan
count devices by class. docs/install.md has a MIG section.

Written against fakes of nvidia-smi and NVML output; not yet run on MIG
hardware.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 00:01:14 +02:00
Ilker ManapandClaude Opus 5.5 7cb9d4fa01 Busy is not unfit: rank files by the hardware the cluster has, and let a model wait for its cards
Ranking what was free first made every file come out on the CPU on a cluster
whose one card was in use. The plan now ranks by hardware alone, free now
only breaking a tie; a model served onto cards that are busy is added and
waits for them, the job ending as waiting rather than failed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 01:17:32 +02:00
Ilker ManapandClaude Opus 5.5 0ea3008f64 On a CPU prefer a middling quantisation, on a card eight bits; past them a bigger file ranks lower
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 01:05:55 +02:00
Ilker ManapandClaude Opus 5.5 54db7aecb7 Plan a repository's files against the fleet, and serve the one that suits it the way it fits
The control plane reads the model's shape from one GGUF header or its
config.json and lists, for every file, the best way it runs here: one card,
several of a node's (by layer for llama.cpp, tensor-parallel for vLLM),
several nodes' over one fabric (vLLM, pipeline-parallel between them), or a
CPU node; what each card holds and whether it is free now. ctl model plan
prints it; model serve takes its file and way unless told, gives a GGUF on
GPUs a GPU profile, and with --fallbacks adds the next best ways. The
console shows it per file and asks for the way and the fallback.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 01:03:48 +02:00