MIG: a card in MIG mode is placed as its instances, each a GPU of class <card>-mig-<profile>

Discovery reports a card in MIG mode as its instances instead of the whole
card, on both paths: nvidia-smi (-L names them, -q -x gives their memory,
ids and processes) and NVML (GetMigMode, GetMaxMigDeviceCount,
GetMigDeviceHandleByIndex, behind an interface so it is tested with a fake).
Each instance has its MIG UUID, its own memory, and a class from the card's
and the profile: a100-80g-mig-3g.40gb. Placement numbers instances
1000 + 100 x card + instance and every view shows them as card/migN.

A replica on MIG runs on exactly one instance: a process opens one MIG
device and instances have no NVLink between them, so tp, pp, gpus or nodes
above one on a MIG class are refused in catalog validation and again in
placement, and the planner offers one instance or none. The agent gives
the engine CUDA_VISIBLE_DEVICES=MIG-<uuid> (no Vulkan device variable on
MIG). Memory and processes are per instance; heat, power and wear the
card's; utilisation is left out rather than reported as zero. A card in MIG
mode with nothing carved stays out of placement with the reason, and
llmgw doctor reports each card's mode and instances and warns of the states
in which nothing can run. ctl nodes, ctl status, the console and the plan
count devices by class. docs/install.md has a MIG section.

Written against fakes of nvidia-smi and NVML output; not yet run on MIG
hardware.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-05 00:01:14 +02:00
1 parent e81f1cba4c
commit 0fd55a862b
39 files changed
+2587 -327

No files matched your search

+65
View File
@@ -428,6 +428,71 @@ sudo llmgw doctor
A `-nvml` build is still published beside the portable one; it reads NVLink
from NVML directly, and needs a libc no older than the builder's.
### MIG, if you want to cut a card into pieces
An A100, A30, H100, H200 or later card can be split into MIG (Multi-Instance
GPU) instances, each with its own memory, its own share of the card's compute
and its own UUID. llmgw places a model on an instance as it would on a card,
which makes MIG the way to run several small models on one large card without
them competing for it.
**llmgw does not create the instances.** How a card is cut is your decision,
made once with `nvidia-smi` on the node; the agent reads what is there when
it starts:
```
sudo nvidia-smi -i 0 -mig 1 # MIG mode on card 0; may need a reset
sudo nvidia-smi mig -i 0 -lgip # the profiles this card offers
sudo nvidia-smi mig -i 0 -cgi 3g.40gb,3g.40gb -C # two instances, each with its compute instance
nvidia-smi -L # the instances, with their MIG- UUIDs
sudo systemctl restart llmgw-agent # so it registers the new devices
```
`-C` matters: a GPU instance with no compute instance inside it is not a
device yet. Turning MIG on takes effect at the card's next reset
(`nvidia-smi -i 0 -r` with nothing running on it, or a reboot), and instances
do not survive a reboot unless something recreates them — NVIDIA's
`mig-parted` or a systemd unit of your own.
**Classes.** A card in MIG mode is reported as its instances instead of as
the whole card, and each instance's class is the card's class and the
profile: two `3g.40gb` instances of an A100 80GB are two devices of class
`a100-80g-mig-3g.40gb`, an H100's `1g.10gb+me` is `h100-80g-mig-1g.10gb-me`.
`llmgw ctl nodes` lists the classes, and a profile names one like any other:
```yaml
profiles:
- name: mig
engine: vllm
gpu_class: a100-80g-mig-3g.40gb
```
**One instance a replica.** A process can open only one MIG device, and
instances have no NVLink or peer-to-peer access between them, so a replica on
MIG runs on exactly one instance: `tp: 1`, one GPU, one node. A profile on a
MIG class asking for more is refused when the model is added, with that
reason. A model too large for one instance wants a larger profile or a whole
card. `llmgw ctl model plan` and `ctl model serve` plan the same way: an
instance is offered for a model it holds, and never several together.
**How they are numbered and shown.** Placement numbers card 0's instance 2 as
device 1002 (1000 + 100 × card + instance), and every view shows it as
`0/mig2`. The engine is given the instance by its UUID —
`CUDA_VISIBLE_DEVICES=MIG-…` — which works for vLLM and for llama.cpp's CUDA
build. llama.cpp's Vulkan build cannot see MIG instances at all.
**What is measured.** Each instance's memory is its own; temperature, power
and wear are its card's, shown on each of its instances. Utilisation is not
reported for an instance, because no driver measures it per instance: the
console says `n/a on MIG`, and no `llmgw_gpu_utilization_percent` series is
published for it rather than a zero.
`sudo llmgw doctor` lists each card's MIG mode and instances, and warns of a
card in MIG mode with none created (nothing can run on it: CUDA will not open
the whole card while MIG is on), of GPU instances with no compute instance,
and of a mode change waiting for a reset. Such a card is also kept out of
placement, with the reason, in `llmgw ctl nodes`.
### InfiniBand, if you have it
```