Ilker ManapandClaude Opus 5
f68fbf7c49
The launch arguments had been read from documentation and never from an
engine. Now they have been:
vllm 0.29.0 --host --port --tensor-parallel-size --served-model-name
--distributed-executor-backend all accepted
llama.cpp b10991 --model --host --port --metrics --alias --parallel
--n-gpu-layers all accepted
Getting there found three faults in the checker rather than in llmgw.
It checked every command's flags against the engine's help, but a shard can be
several programs — vLLM under Ray is `ray start` and then `vllm serve` — and
they have different vocabularies. It reported --head and --num-gpus as
missing, which is true of vllm and irrelevant, because ray is what receives
them. Each command is now checked against its own help, and one that cannot be
run is reported as skipped rather than failed.
vLLM does not give up its whole help for --help: it prints a summary of
section names and expects --help=all. Checked against the summary, all five
flags looked missing. The warning about truncated help did not fire, because
the summary happens to mention --help= while listing how to ask for more.
And vLLM refuses to print anything at all on a machine with no accelerator
unless VLLM_TARGET_DEVICE says what to assume. Asking an engine about its
flags should not require the hardware it runs on.
Two things remain unchecked and are written down as such: the Ray sidecar,
because vLLM 0.29 does not install Ray, and SGLang, because it is not
installed anywhere. `launch_mode: mp` avoids Ray entirely and is worth
preferring until someone checks it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
6.6 KiB
Adding a node
Written for whoever runs the cluster. Two scripts, because the two halves of onboarding are done by different people with different access: one needs the hypervisor, the other needs the certificate authority.
1. The machine
On the Proxmox host:
llmgw-provision-node --vmid 144 --name llmgw-node04 --ip 10.0.0.144/24
Name it after the node id. The VM name, the hostname, the node id and the certificate's common name are all the same string. They do not have to be, and every time they are not, somebody spends an afternoon working out which of the four a message is talking about.
It imports a cloud image and lets cloud-init do the rest, which is why a node takes a minute rather than an afternoon. It refuses to touch an existing VMID: adopting a machine is not a decision a provisioning script should make.
Releases carry static linux/amd64 and linux/arm64 binaries. Download one and
pass it with --binary rather than copying whatever happens to be on the
machine you are running onboarding from — otherwise whoever runs the script
quietly becomes the build server, and the fleet ends up running whatever was in
their working tree.
2. The identity
On the host that holds the certificate authority — this is the part that cannot be delegated, because it issues the credential by which the control plane believes the node is who it says it is:
llmgw-onboard-node \
--node llmgw-node04 \
--addr 10.0.0.144 \
--server https://10.0.0.140:7443 \
--fabric ib-b
It issues a client certificate whose common name is the node id, installs it
along with the CA root, trusts that root system-wide so other tooling on the
machine stops treating llmgw's certificates as strangers, installs the binary
and the unit, and starts the agent. The node appears in llmgw ctl nodes within
a few seconds.
If it does not, the control plane's log says why. A certificate whose common name does not match the claimed node id is refused by design — that check is what makes a node's identity something it proves rather than something it picks.
It needs ssh from the CA host to the node. Provisioning copies that host's
authorized_keys into the new machine, so a node built by the script above
already satisfies it; an older machine may need the key added.
What onboarding does not do
It gives a machine an identity and an agent. It does not install inference
engines or model weights, and a node missing either looks healthy in
llmgw ctl nodes right up until a replica is placed on it and the shard dies
with "executable file not found".
So the node is not finished until:
- the engines its catalog profiles name are installed (
vllm,sglang,llama-server), at the paths the agent configuration gives - the weights are on disk, at the
sourceeach model declares
Both are deliberately outside this script. Engines are a platform decision with versions and drivers attached, and distributing weights is its own problem — hundreds of gigabytes that want a shared filesystem or a cache, not an scp from a laptop. llmgw has no answer for it yet; that is a known gap, not an oversight.
The sim engine needs neither, which is what makes it useful for exercising
everything else.
Checking an engine after an upgrade
Engines rename and remove flags between versions, and when that happens the failure arrives minutes into a model load, in the engine's own error message, on a node nobody was watching.
llmgw ctl engine-check vllm
llmgw ctl engine-check llamacpp --help-output "$(llama-server --help 2>&1)"
It builds the widest command llmgw would ever produce — a multi-node one, since
a flag that only appears there is exactly the one nobody notices has gone — and
checks every flag against the engine's own --help. No GPU required: a CLI
prints its help on any machine.
Some programs do not give up their whole help for --help. vLLM prints a
summary of section names and expects --help=all; checking against the summary
reports every flag as missing, confidently and wrongly. engine-check knows
that one. If you add an engine, check that what you get back is the full list.
vLLM also refuses to do anything at all on a machine with no accelerator,
including print its help, unless VLLM_TARGET_DEVICE says what to assume.
engine-check sets it.
Give it the whole help. Truncating llama.cpp's 725 lines to 400 reports five
flags as missing that the running cluster uses; the command warns when the text
it was given does not document --help itself, which is the cheapest sign it
was cut short.
Fabrics
--fabric is how a node says which InfiniBand fabric it is attached to, and
there is no way around stating it. OpenSM's default subnet prefix is fe80:: on
every fabric, so a matching prefix proves nothing and a wrong guess produces a
replica that starts, passes health checks, and then hangs on its first
all-reduce.
Omit it for a node with no fabric. Such a node is its own island: it can host single-node replicas and will never be given a multi-node one.
Two adapters per fabric is the default; --fabric ib-b:mlx5_2,mlx5_3 names them
explicitly. An adapter must never appear under two fabrics — the agent refuses
that configuration, because the consequence is the hang above.
Where the driver is
--nvidia-smi /opt/nvidia/bin/nvidia-smi
--ib-root /sys/class/infiniband
Both default to the usual places. Containerized driver installs routinely put
nvidia-smi somewhere else, and an agent that cannot find it registers with no
GPUs — which looks exactly like a node that has none.
Removing a node
llmgw ctl drain llmgw-node04 # move its replicas off, one at a time
llmgw ctl nodes # wait until it holds none
Then stop the agent. Draining brings each replacement up before stopping the original, which is the difference between a rolling move and an outage.
The node's certificate stays valid until it expires. There is no revocation list — if a machine is decommissioned rather than retired, reissue the authority or accept that a certificate naming a node that no longer exists can still connect and claim to be it.
Testing without hardware
Every part of this works on machines with no GPUs and no InfiniBand. Point
--nvidia-smi at a script that prints the CSV discovery asks for, point
--ib-root at a sysfs-shaped directory, and use engine: sim in the catalog.
The simulated engine rendezvouses for real: rank 0 binds the dist-init port and does not serve until every worker has joined, and workers serve nothing. That exercises the multi-node launch sequence, the fabric constraint, and failure and rebuild — which is most of what goes wrong — without exercising NCCL, which is the rest.