MIG: a card in MIG mode is placed as its instances, each a GPU of class <card>-mig-<profile>
Discovery reports a card in MIG mode as its instances instead of the whole card, on both paths: nvidia-smi (-L names them, -q -x gives their memory, ids and processes) and NVML (GetMigMode, GetMaxMigDeviceCount, GetMigDeviceHandleByIndex, behind an interface so it is tested with a fake). Each instance has its MIG UUID, its own memory, and a class from the card's and the profile: a100-80g-mig-3g.40gb. Placement numbers instances 1000 + 100 x card + instance and every view shows them as card/migN. A replica on MIG runs on exactly one instance: a process opens one MIG device and instances have no NVLink between them, so tp, pp, gpus or nodes above one on a MIG class are refused in catalog validation and again in placement, and the planner offers one instance or none. The agent gives the engine CUDA_VISIBLE_DEVICES=MIG-<uuid> (no Vulkan device variable on MIG). Memory and processes are per instance; heat, power and wear the card's; utilisation is left out rather than reported as zero. A card in MIG mode with nothing carved stays out of placement with the reason, and llmgw doctor reports each card's mode and instances and warns of the states in which nothing can run. ctl nodes, ctl status, the console and the plan count devices by class. docs/install.md has a MIG section. Written against fakes of nvidia-smi and NVML output; not yet run on MIG hardware. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
e81f1cba4c
commit
0fd55a862b
39 files changed
+2587
-327
No files matched your search
@@ -428,6 +428,71 @@ sudo llmgw doctor
|
||||
A `-nvml` build is still published beside the portable one; it reads NVLink
|
||||
from NVML directly, and needs a libc no older than the builder's.
|
||||
|
||||
### MIG, if you want to cut a card into pieces
|
||||
|
||||
An A100, A30, H100, H200 or later card can be split into MIG (Multi-Instance
|
||||
GPU) instances, each with its own memory, its own share of the card's compute
|
||||
and its own UUID. llmgw places a model on an instance as it would on a card,
|
||||
which makes MIG the way to run several small models on one large card without
|
||||
them competing for it.
|
||||
|
||||
**llmgw does not create the instances.** How a card is cut is your decision,
|
||||
made once with `nvidia-smi` on the node; the agent reads what is there when
|
||||
it starts:
|
||||
|
||||
```
|
||||
sudo nvidia-smi -i 0 -mig 1 # MIG mode on card 0; may need a reset
|
||||
sudo nvidia-smi mig -i 0 -lgip # the profiles this card offers
|
||||
sudo nvidia-smi mig -i 0 -cgi 3g.40gb,3g.40gb -C # two instances, each with its compute instance
|
||||
nvidia-smi -L # the instances, with their MIG- UUIDs
|
||||
sudo systemctl restart llmgw-agent # so it registers the new devices
|
||||
```
|
||||
|
||||
`-C` matters: a GPU instance with no compute instance inside it is not a
|
||||
device yet. Turning MIG on takes effect at the card's next reset
|
||||
(`nvidia-smi -i 0 -r` with nothing running on it, or a reboot), and instances
|
||||
do not survive a reboot unless something recreates them — NVIDIA's
|
||||
`mig-parted` or a systemd unit of your own.
|
||||
|
||||
**Classes.** A card in MIG mode is reported as its instances instead of as
|
||||
the whole card, and each instance's class is the card's class and the
|
||||
profile: two `3g.40gb` instances of an A100 80GB are two devices of class
|
||||
`a100-80g-mig-3g.40gb`, an H100's `1g.10gb+me` is `h100-80g-mig-1g.10gb-me`.
|
||||
`llmgw ctl nodes` lists the classes, and a profile names one like any other:
|
||||
|
||||
```yaml
|
||||
profiles:
|
||||
- name: mig
|
||||
engine: vllm
|
||||
gpu_class: a100-80g-mig-3g.40gb
|
||||
```
|
||||
|
||||
**One instance a replica.** A process can open only one MIG device, and
|
||||
instances have no NVLink or peer-to-peer access between them, so a replica on
|
||||
MIG runs on exactly one instance: `tp: 1`, one GPU, one node. A profile on a
|
||||
MIG class asking for more is refused when the model is added, with that
|
||||
reason. A model too large for one instance wants a larger profile or a whole
|
||||
card. `llmgw ctl model plan` and `ctl model serve` plan the same way: an
|
||||
instance is offered for a model it holds, and never several together.
|
||||
|
||||
**How they are numbered and shown.** Placement numbers card 0's instance 2 as
|
||||
device 1002 (1000 + 100 × card + instance), and every view shows it as
|
||||
`0/mig2`. The engine is given the instance by its UUID —
|
||||
`CUDA_VISIBLE_DEVICES=MIG-…` — which works for vLLM and for llama.cpp's CUDA
|
||||
build. llama.cpp's Vulkan build cannot see MIG instances at all.
|
||||
|
||||
**What is measured.** Each instance's memory is its own; temperature, power
|
||||
and wear are its card's, shown on each of its instances. Utilisation is not
|
||||
reported for an instance, because no driver measures it per instance: the
|
||||
console says `n/a on MIG`, and no `llmgw_gpu_utilization_percent` series is
|
||||
published for it rather than a zero.
|
||||
|
||||
`sudo llmgw doctor` lists each card's MIG mode and instances, and warns of a
|
||||
card in MIG mode with none created (nothing can run on it: CUDA will not open
|
||||
the whole card while MIG is on), of GPU instances with no compute instance,
|
||||
and of a mode change waiting for a reset. Such a card is also kept out of
|
||||
placement, with the reason, in `llmgw ctl nodes`.
|
||||
|
||||
### InfiniBand, if you have it
|
||||
|
||||
```
|
||||
|
||||
Reference in new issue
Block a user