llmgw/notes/c22.md

C22: MIG

CHANGELOG

MIG instances as GPUs of their own class

• A card in MIG mode is placed as its instances. Discovery reports a card in MIG mode as its MIG instances instead of the whole card, each a device of its own with its UUID (MIG-…), its own memory, and a class named from the card's and the profile: two 3g.40gb instances of an A100 80GB are two devices of class a100-80g-mig-3g.40gb. A profile names that class like any other, and placement puts a replica on a free instance of it. Both discovery paths do it: nvidia-smi (instances from nvidia-smi -L, their memory and ids from nvidia-smi -q -x) and NVML (GetMigMode, GetMaxMigDeviceCount, GetMigDeviceHandleByIndex). llmgw does not create instances; docs/install.md has a MIG section on making them.
• One instance a replica. A process opens only one MIG device, and instances have no NVLink or peer-to-peer between them, so a profile on a MIG class asking for more than one GPU, tp or pp above 1, or more than one node is refused when the model is added, with that reason; placement refuses it again. ctl model plan and ctl model serve plan an instance for a model one holds and never several together.
• Engines are given the instance by UUID. Placement numbers card 0's instance 2 as device 1002 (1000 + 100 × card + instance); the agent turns that into CUDA_VISIBLE_DEVICES=MIG-<uuid> for vLLM and llama.cpp, and leaves GGML_VK_VISIBLE_DEVICES unset (Vulkan cannot see MIG). Every view shows the device as 0/mig2.
• Memory per instance, no made-up utilisation. Each instance's memory, and the processes holding it, are its own — read per GPU and compute instance from nvidia-smi's report, so a foreign process on one instance keeps that instance out of placement and not its neighbours. Temperature, power and wear are the card's. No driver reports utilisation per instance, so none is published for one: the console says n/a on MIG, and llmgw_gpu_utilization_percent has no series for it rather than a zero.
• A card in MIG mode with nothing carved is kept out of placement, with the reason in ctl nodes, since CUDA will not open the whole card while MIG is on.
• llmgw doctor reports MIG: each card's mode and instances, and warns of a card in MIG mode with no instances, of GPU instances with no compute instance, and of a mode change waiting for a reset, with the nvidia-smi command for each.
• ctl nodes counts a node's devices by class when it has MIG instances or more than one class (a100-80g-mig-3g.40gb×2,a100-80g×1); the console's GPU list shows instances under their card with their own memory and the card's heat and power, and a node's page lists its classes with counts. ctl status and the plan count each device under its own class, where a node's cards were all put under its first class before.
• The agent protocol's GPU gains mig (parent card, its memory, profile, GPU and compute instance ids); the node API's GPUs gain label, class, card, mig_profile and memory, and nodes gain gpus_by_class.

BACKLOG

Done (branch features-c2): MIG instances discovered on both paths (nvidia-smi -L + -q -x; NVML through an interface with a faithful fake) and reported instead of their card, each with its MIG UUID, own memory and class <card class>-mig-<profile>; device numbers 1000 + 100 × card + instance, shown as card/migN; placement on MIG classes; tp/pp/gpus/nodes above 1 on a MIG class refused in catalog validation and again in placement; CUDA_VISIBLE_DEVICES=MIG-<uuid> for vLLM and llama.cpp (no Vulkan device variable on MIG); per-instance memory and processes in heartbeats and the GPU check, card sensors under each instance, utilisation absent rather than zero; doctor MIG findings (mig -lgi/-lci parsed); ctl nodes, console, ctl status, ctl model plan/serve by class; docs/install.md MIG section. Written and tested against fakes only — untested on real MIG hardware. Left: run it on an A100/H100 in MIG mode (check -L/-q -x/mig -lgi/-lci and NVML MIG device names against the parsers, and that CUDA_VISIBLE_DEVICES=MIG-… reaches vLLM's workers through Ray); instances changed while the agent runs are only seen when it restarts (no rescan); llmgw does not create or reshape instances (no mig-parted integration); quotas count an instance as one GPU, the same as a whole card; SGLang on MIG not checked; device numbers assume fewer than 100 instances per card and that instance numbers stay put while shards run on them.