Discovery reports a card in MIG mode as its instances instead of the whole card, on both paths: nvidia-smi (-L names them, -q -x gives their memory, ids and processes) and NVML (GetMigMode, GetMaxMigDeviceCount, GetMigDeviceHandleByIndex, behind an interface so it is tested with a fake). Each instance has its MIG UUID, its own memory, and a class from the card's and the profile: a100-80g-mig-3g.40gb. Placement numbers instances 1000 + 100 x card + instance and every view shows them as card/migN. A replica on MIG runs on exactly one instance: a process opens one MIG device and instances have no NVLink between them, so tp, pp, gpus or nodes above one on a MIG class are refused in catalog validation and again in placement, and the planner offers one instance or none. The agent gives the engine CUDA_VISIBLE_DEVICES=MIG-<uuid> (no Vulkan device variable on MIG). Memory and processes are per instance; heat, power and wear the card's; utilisation is left out rather than reported as zero. A card in MIG mode with nothing carved stays out of placement with the reason, and llmgw doctor reports each card's mode and instances and warns of the states in which nothing can run. ctl nodes, ctl status, the console and the plan count devices by class. docs/install.md has a MIG section. Written against fakes of nvidia-smi and NVML output; not yet run on MIG hardware. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
55 lines
4.5 KiB
Markdown
55 lines
4.5 KiB
Markdown
# C22: MIG
|
||
|
||
## CHANGELOG
|
||
|
||
### MIG instances as GPUs of their own class
|
||
|
||
- **A card in MIG mode is placed as its instances.** Discovery reports a card
|
||
in MIG mode as its MIG instances instead of the whole card, each a device of
|
||
its own with its UUID (`MIG-…`), its own memory, and a class named from the
|
||
card's and the profile: two `3g.40gb` instances of an A100 80GB are two
|
||
devices of class `a100-80g-mig-3g.40gb`. A profile names that class like
|
||
any other, and placement puts a replica on a free instance of it. Both
|
||
discovery paths do it: nvidia-smi (instances from `nvidia-smi -L`, their
|
||
memory and ids from `nvidia-smi -q -x`) and NVML (`GetMigMode`,
|
||
`GetMaxMigDeviceCount`, `GetMigDeviceHandleByIndex`). llmgw does not create
|
||
instances; `docs/install.md` has a MIG section on making them.
|
||
- **One instance a replica.** A process opens only one MIG device, and
|
||
instances have no NVLink or peer-to-peer between them, so a profile on a
|
||
MIG class asking for more than one GPU, tp or pp above 1, or more than one
|
||
node is refused when the model is added, with that reason; placement
|
||
refuses it again. `ctl model plan` and `ctl model serve` plan an instance
|
||
for a model one holds and never several together.
|
||
- **Engines are given the instance by UUID.** Placement numbers card 0's
|
||
instance 2 as device 1002 (1000 + 100 × card + instance); the agent turns
|
||
that into `CUDA_VISIBLE_DEVICES=MIG-<uuid>` for vLLM and llama.cpp, and
|
||
leaves `GGML_VK_VISIBLE_DEVICES` unset (Vulkan cannot see MIG). Every view
|
||
shows the device as `0/mig2`.
|
||
- **Memory per instance, no made-up utilisation.** Each instance's memory,
|
||
and the processes holding it, are its own — read per GPU and compute
|
||
instance from nvidia-smi's report, so a foreign process on one instance
|
||
keeps that instance out of placement and not its neighbours. Temperature,
|
||
power and wear are the card's. No driver reports utilisation per instance,
|
||
so none is published for one: the console says `n/a on MIG`, and
|
||
`llmgw_gpu_utilization_percent` has no series for it rather than a zero.
|
||
- **A card in MIG mode with nothing carved is kept out of placement**, with
|
||
the reason in `ctl nodes`, since CUDA will not open the whole card while MIG
|
||
is on.
|
||
- **`llmgw doctor` reports MIG**: each card's mode and instances, and warns
|
||
of a card in MIG mode with no instances, of GPU instances with no compute
|
||
instance, and of a mode change waiting for a reset, with the `nvidia-smi`
|
||
command for each.
|
||
- `ctl nodes` counts a node's devices by class when it has MIG instances or
|
||
more than one class (`a100-80g-mig-3g.40gb×2,a100-80g×1`); the console's
|
||
GPU list shows instances under their card with their own memory and the
|
||
card's heat and power, and a node's page lists its classes with counts.
|
||
`ctl status` and the plan count each device under its own class, where a
|
||
node's cards were all put under its first class before.
|
||
- The agent protocol's `GPU` gains `mig` (parent card, its memory, profile,
|
||
GPU and compute instance ids); the node API's GPUs gain `label`, `class`,
|
||
`card`, `mig_profile` and memory, and nodes gain `gpus_by_class`.
|
||
|
||
## BACKLOG
|
||
|
||
**Done (branch `features-c2`):** MIG instances discovered on both paths (nvidia-smi `-L` + `-q -x`; NVML through an interface with a faithful fake) and reported instead of their card, each with its MIG UUID, own memory and class `<card class>-mig-<profile>`; device numbers 1000 + 100 × card + instance, shown as `card/migN`; placement on MIG classes; tp/pp/gpus/nodes above 1 on a MIG class refused in catalog validation and again in placement; `CUDA_VISIBLE_DEVICES=MIG-<uuid>` for vLLM and llama.cpp (no Vulkan device variable on MIG); per-instance memory and processes in heartbeats and the GPU check, card sensors under each instance, utilisation absent rather than zero; doctor MIG findings (`mig -lgi`/`-lci` parsed); `ctl nodes`, console, `ctl status`, `ctl model plan`/`serve` by class; docs/install.md MIG section. Written and tested against fakes only — untested on real MIG hardware. Left: run it on an A100/H100 in MIG mode (check `-L`/`-q -x`/`mig -lgi`/`-lci` and NVML MIG device names against the parsers, and that `CUDA_VISIBLE_DEVICES=MIG-…` reaches vLLM's workers through Ray); instances changed while the agent runs are only seen when it restarts (no rescan); llmgw does not create or reshape instances (no mig-parted integration); quotas count an instance as one GPU, the same as a whole card; SGLang on MIG not checked; device numbers assume fewer than 100 instances per card and that instance numbers stay put while shards run on them.
|