Files
llmgw/notes/c22.md
T
Ilker ManapandClaude Opus 5.5 0fd55a862b MIG: a card in MIG mode is placed as its instances, each a GPU of class <card>-mig-<profile>
Discovery reports a card in MIG mode as its instances instead of the whole
card, on both paths: nvidia-smi (-L names them, -q -x gives their memory,
ids and processes) and NVML (GetMigMode, GetMaxMigDeviceCount,
GetMigDeviceHandleByIndex, behind an interface so it is tested with a fake).
Each instance has its MIG UUID, its own memory, and a class from the card's
and the profile: a100-80g-mig-3g.40gb. Placement numbers instances
1000 + 100 x card + instance and every view shows them as card/migN.

A replica on MIG runs on exactly one instance: a process opens one MIG
device and instances have no NVLink between them, so tp, pp, gpus or nodes
above one on a MIG class are refused in catalog validation and again in
placement, and the planner offers one instance or none. The agent gives
the engine CUDA_VISIBLE_DEVICES=MIG-<uuid> (no Vulkan device variable on
MIG). Memory and processes are per instance; heat, power and wear the
card's; utilisation is left out rather than reported as zero. A card in MIG
mode with nothing carved stays out of placement with the reason, and
llmgw doctor reports each card's mode and instances and warns of the states
in which nothing can run. ctl nodes, ctl status, the console and the plan
count devices by class. docs/install.md has a MIG section.

Written against fakes of nvidia-smi and NVML output; not yet run on MIG
hardware.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 00:01:14 +02:00

55 lines
4.5 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# C22: MIG
## CHANGELOG
### MIG instances as GPUs of their own class
- **A card in MIG mode is placed as its instances.** Discovery reports a card
in MIG mode as its MIG instances instead of the whole card, each a device of
its own with its UUID (`MIG-…`), its own memory, and a class named from the
card's and the profile: two `3g.40gb` instances of an A100 80GB are two
devices of class `a100-80g-mig-3g.40gb`. A profile names that class like
any other, and placement puts a replica on a free instance of it. Both
discovery paths do it: nvidia-smi (instances from `nvidia-smi -L`, their
memory and ids from `nvidia-smi -q -x`) and NVML (`GetMigMode`,
`GetMaxMigDeviceCount`, `GetMigDeviceHandleByIndex`). llmgw does not create
instances; `docs/install.md` has a MIG section on making them.
- **One instance a replica.** A process opens only one MIG device, and
instances have no NVLink or peer-to-peer between them, so a profile on a
MIG class asking for more than one GPU, tp or pp above 1, or more than one
node is refused when the model is added, with that reason; placement
refuses it again. `ctl model plan` and `ctl model serve` plan an instance
for a model one holds and never several together.
- **Engines are given the instance by UUID.** Placement numbers card 0's
instance 2 as device 1002 (1000 + 100 × card + instance); the agent turns
that into `CUDA_VISIBLE_DEVICES=MIG-<uuid>` for vLLM and llama.cpp, and
leaves `GGML_VK_VISIBLE_DEVICES` unset (Vulkan cannot see MIG). Every view
shows the device as `0/mig2`.
- **Memory per instance, no made-up utilisation.** Each instance's memory,
and the processes holding it, are its own — read per GPU and compute
instance from nvidia-smi's report, so a foreign process on one instance
keeps that instance out of placement and not its neighbours. Temperature,
power and wear are the card's. No driver reports utilisation per instance,
so none is published for one: the console says `n/a on MIG`, and
`llmgw_gpu_utilization_percent` has no series for it rather than a zero.
- **A card in MIG mode with nothing carved is kept out of placement**, with
the reason in `ctl nodes`, since CUDA will not open the whole card while MIG
is on.
- **`llmgw doctor` reports MIG**: each card's mode and instances, and warns
of a card in MIG mode with no instances, of GPU instances with no compute
instance, and of a mode change waiting for a reset, with the `nvidia-smi`
command for each.
- `ctl nodes` counts a node's devices by class when it has MIG instances or
more than one class (`a100-80g-mig-3g.40gb×2,a100-80g×1`); the console's
GPU list shows instances under their card with their own memory and the
card's heat and power, and a node's page lists its classes with counts.
`ctl status` and the plan count each device under its own class, where a
node's cards were all put under its first class before.
- The agent protocol's `GPU` gains `mig` (parent card, its memory, profile,
GPU and compute instance ids); the node API's GPUs gain `label`, `class`,
`card`, `mig_profile` and memory, and nodes gain `gpus_by_class`.
## BACKLOG
**Done (branch `features-c2`):** MIG instances discovered on both paths (nvidia-smi `-L` + `-q -x`; NVML through an interface with a faithful fake) and reported instead of their card, each with its MIG UUID, own memory and class `<card class>-mig-<profile>`; device numbers 1000 + 100 × card + instance, shown as `card/migN`; placement on MIG classes; tp/pp/gpus/nodes above 1 on a MIG class refused in catalog validation and again in placement; `CUDA_VISIBLE_DEVICES=MIG-<uuid>` for vLLM and llama.cpp (no Vulkan device variable on MIG); per-instance memory and processes in heartbeats and the GPU check, card sensors under each instance, utilisation absent rather than zero; doctor MIG findings (`mig -lgi`/`-lci` parsed); `ctl nodes`, console, `ctl status`, `ctl model plan`/`serve` by class; docs/install.md MIG section. Written and tested against fakes only — untested on real MIG hardware. Left: run it on an A100/H100 in MIG mode (check `-L`/`-q -x`/`mig -lgi`/`-lci` and NVML MIG device names against the parsers, and that `CUDA_VISIBLE_DEVICES=MIG-…` reaches vLLM's workers through Ray); instances changed while the agent runs are only seen when it restarts (no rescan); llmgw does not create or reshape instances (no mig-parted integration); quotas count an instance as one GPU, the same as a whole card; SGLang on MIG not checked; device numbers assume fewer than 100 instances per card and that instance numbers stay put while shards run on them.