Discovery reports a card in MIG mode as its instances instead of the whole
card, on both paths: nvidia-smi (-L names them, -q -x gives their memory,
ids and processes) and NVML (GetMigMode, GetMaxMigDeviceCount,
GetMigDeviceHandleByIndex, behind an interface so it is tested with a fake).
Each instance has its MIG UUID, its own memory, and a class from the card's
and the profile: a100-80g-mig-3g.40gb. Placement numbers instances
1000 + 100 x card + instance and every view shows them as card/migN.
A replica on MIG runs on exactly one instance: a process opens one MIG
device and instances have no NVLink between them, so tp, pp, gpus or nodes
above one on a MIG class are refused in catalog validation and again in
placement, and the planner offers one instance or none. The agent gives
the engine CUDA_VISIBLE_DEVICES=MIG-<uuid> (no Vulkan device variable on
MIG). Memory and processes are per instance; heat, power and wear the
card's; utilisation is left out rather than reported as zero. A card in MIG
mode with nothing carved stays out of placement with the reason, and
llmgw doctor reports each card's mode and instances and warns of the states
in which nothing can run. ctl nodes, ctl status, the console and the plan
count devices by class. docs/install.md has a MIG section.
Written against fakes of nvidia-smi and NVML output; not yet run on MIG
hardware.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A RoCE port without a v2 GID and a missing ib_uverbs failed doctor, and
node add, whatever fabric used them. They are now failures only for a
declared fabric and warnings otherwise; node add passes its --fabric
list to doctor so those are checked as declared.
Refs #2 (item 5).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
engine_overrides give nodes picked by GPU class, id or labels their own
engine versions; node add --disable-engine keeps one off a node; what
vLLM cannot do on a node is a warning rather than a refused node; the
CUDA requirement follows the version; a hand-named binary is no longer
replaced by an installed one.
Refs #2 (items 4 and 7).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
vLLM was installed by hand and named in each agent's configuration. The
agent now installs it: a Python 3.12 environment built by uv, which is
itself a GitHub release checked against its checksum, from PyPI resolved as
it stood a day after that vLLM was published and installed with
--require-hashes. The resolved list is kept in the environment. The
environment is built relocatable under a scratch name and renamed into place
once it runs, like an archive.
The default is 0.30.0, whose flags were checked against the ones llmgw
builds. It is built for CUDA 13, so it needs driver 580 and a Turing GPU or
newer, and 25 GB free while it installs; a node lacking one says which, and a
node with no GPU skips it without being shown as a fault. gcc comes with it
because Triton compiles its launcher at the first model load. SGLang
installs the same way when the control plane names a version. llmgw doctor
reports whether vLLM can be installed on the node.
Checked by installing 0.30.0 this way on a test node: 7.7 GB, torch 2.13 for
CUDA 13.0, importing cleanly; the node has no GPU, so vLLM was not started.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
For machines nobody here can log into. It reads the GPUs and their
nvidia-smi topology, NVSwitch and the fabric manager, every InfiniBand
and RoCE port with its counters and RoCE v2 GID, the RDMA and
GPUDirect kernel modules, the agent's declared fabrics against the
adapters, the management interface NCCL will bootstrap over, the
agent unit's memlock limit and the engines installed - and says what
is wrong and how to fix it, most serious first, with the NCCL
environment a shard on each fabric would be given. --json writes every
fact it concluded from, to send back. It exits non-zero when something
fails.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
