Ilker ManapandClaude Opus 5
5b425c32e5
A release was a git tag. Installing llmgw on a node meant having a Go toolchain, and the onboarding script copied a binary from wherever it was run — which works and quietly makes whoever runs it the build server, with the fleet ending up on whatever was in their working tree. Tags now build static linux/amd64, linux/arm64 and darwin binaries and attach them. CGO_ENABLED=0 is the point: it has to run on a node whose libc is older than the builder's, and llmgw has no C dependency because modernc.org/sqlite is pure Go precisely so this stays true. Verified: "not a dynamic executable", runs on a node, reports its version. `ctl engine-check` closes the other half of the phase 4 gap. llmgw's launch arguments are read from each engine's documentation, and engines rename flags between versions; when that happens the failure arrives minutes into a model load, in the engine's own error message, on a node nobody was watching. It builds the widest command llmgw would ever produce — multi-node, since a flag that only appears there is the one nobody notices has gone — and checks each flag against the engine's own --help. No GPU: a CLI prints its help anywhere. Verified against llama.cpp b10991, all seven flags accepted. Flags are matched on a word boundary, so --port is not satisfied by --port-range. Without that, a renamed flag would sail through the check that exists to catch renamed flags. It also warns when the help text looks truncated, which is how the first run went: llama.cpp's help cut to 400 of its 725 lines reported five flags as missing that the running cluster uses. TestSidecarRunsBeforeEngine was flaky under a full run, and worse than flaky — its wait loop fell through on timeout, so a slow machine turned the ordering assertion into a no-op and the failure surfaced later as something else. It now asserts, with a deadline that suits a machine running every other package at the same time. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
152 lines
6.2 KiB
Markdown
152 lines
6.2 KiB
Markdown
# Adding a node
|
|
|
|
Written for whoever runs the cluster. Two scripts, because the two halves of
|
|
onboarding are done by different people with different access: one needs the
|
|
hypervisor, the other needs the certificate authority.
|
|
|
|
## 1. The machine
|
|
|
|
On the Proxmox host:
|
|
|
|
```
|
|
llmgw-provision-node --vmid 144 --name llmgw-node04 --ip 10.0.0.144/24
|
|
```
|
|
|
|
Name it after the node id. **The VM name, the hostname, the node id and the
|
|
certificate's common name are all the same string.** They do not have to be, and
|
|
every time they are not, somebody spends an afternoon working out which of the
|
|
four a message is talking about.
|
|
|
|
It imports a cloud image and lets cloud-init do the rest, which is why a node
|
|
takes a minute rather than an afternoon. It refuses to touch an existing VMID:
|
|
adopting a machine is not a decision a provisioning script should make.
|
|
|
|
Releases carry static `linux/amd64` and `linux/arm64` binaries. Download one and
|
|
pass it with `--binary` rather than copying whatever happens to be on the
|
|
machine you are running onboarding from — otherwise whoever runs the script
|
|
quietly becomes the build server, and the fleet ends up running whatever was in
|
|
their working tree.
|
|
|
|
## 2. The identity
|
|
|
|
On the host that holds the certificate authority — this is the part that cannot
|
|
be delegated, because it issues the credential by which the control plane
|
|
believes the node is who it says it is:
|
|
|
|
```
|
|
llmgw-onboard-node \
|
|
--node llmgw-node04 \
|
|
--addr 10.0.0.144 \
|
|
--server https://10.0.0.140:7443 \
|
|
--fabric ib-b
|
|
```
|
|
|
|
It issues a client certificate whose common name is the node id, installs it
|
|
along with the CA root, trusts that root system-wide so other tooling on the
|
|
machine stops treating llmgw's certificates as strangers, installs the binary
|
|
and the unit, and starts the agent. The node appears in `llmgw ctl nodes` within
|
|
a few seconds.
|
|
|
|
If it does not, the control plane's log says why. A certificate whose common
|
|
name does not match the claimed node id is refused by design — that check is
|
|
what makes a node's identity something it proves rather than something it picks.
|
|
|
|
**It needs ssh from the CA host to the node.** Provisioning copies that host's
|
|
`authorized_keys` into the new machine, so a node built by the script above
|
|
already satisfies it; an older machine may need the key added.
|
|
|
|
## What onboarding does not do
|
|
|
|
It gives a machine an identity and an agent. It does **not** install inference
|
|
engines or model weights, and a node missing either looks healthy in
|
|
`llmgw ctl nodes` right up until a replica is placed on it and the shard dies
|
|
with "executable file not found".
|
|
|
|
So the node is not finished until:
|
|
|
|
- the engines its catalog profiles name are installed (`vllm`, `sglang`,
|
|
`llama-server`), at the paths the agent configuration gives
|
|
- the weights are on disk, at the `source` each model declares
|
|
|
|
Both are deliberately outside this script. Engines are a platform decision with
|
|
versions and drivers attached, and distributing weights is its own problem —
|
|
hundreds of gigabytes that want a shared filesystem or a cache, not an scp from
|
|
a laptop. llmgw has no answer for it yet; that is a known gap, not an oversight.
|
|
|
|
The `sim` engine needs neither, which is what makes it useful for exercising
|
|
everything else.
|
|
|
|
## Checking an engine after an upgrade
|
|
|
|
Engines rename and remove flags between versions, and when that happens the
|
|
failure arrives minutes into a model load, in the engine's own error message, on
|
|
a node nobody was watching.
|
|
|
|
```
|
|
llmgw ctl engine-check vllm
|
|
llmgw ctl engine-check llamacpp --help-output "$(llama-server --help 2>&1)"
|
|
```
|
|
|
|
It builds the widest command llmgw would ever produce — a multi-node one, since
|
|
a flag that only appears there is exactly the one nobody notices has gone — and
|
|
checks every flag against the engine's own `--help`. No GPU required: a CLI
|
|
prints its help on any machine.
|
|
|
|
Give it the **whole** help. Truncating llama.cpp's 725 lines to 400 reports five
|
|
flags as missing that the running cluster uses; the command warns when the text
|
|
it was given does not document `--help` itself, which is the cheapest sign it
|
|
was cut short.
|
|
|
|
## Fabrics
|
|
|
|
`--fabric` is how a node says which InfiniBand fabric it is attached to, and
|
|
there is no way around stating it. OpenSM's default subnet prefix is `fe80::` on
|
|
every fabric, so a matching prefix proves nothing and a wrong guess produces a
|
|
replica that starts, passes health checks, and then hangs on its first
|
|
all-reduce.
|
|
|
|
Omit it for a node with no fabric. Such a node is its own island: it can host
|
|
single-node replicas and will never be given a multi-node one.
|
|
|
|
Two adapters per fabric is the default; `--fabric ib-b:mlx5_2,mlx5_3` names them
|
|
explicitly. An adapter must never appear under two fabrics — the agent refuses
|
|
that configuration, because the consequence is the hang above.
|
|
|
|
## Where the driver is
|
|
|
|
```
|
|
--nvidia-smi /opt/nvidia/bin/nvidia-smi
|
|
--ib-root /sys/class/infiniband
|
|
```
|
|
|
|
Both default to the usual places. Containerized driver installs routinely put
|
|
`nvidia-smi` somewhere else, and an agent that cannot find it registers with no
|
|
GPUs — which looks exactly like a node that has none.
|
|
|
|
## Removing a node
|
|
|
|
```
|
|
llmgw ctl drain llmgw-node04 # move its replicas off, one at a time
|
|
llmgw ctl nodes # wait until it holds none
|
|
```
|
|
|
|
Then stop the agent. Draining brings each replacement up before stopping the
|
|
original, which is the difference between a rolling move and an outage.
|
|
|
|
The node's certificate stays valid until it expires. There is no revocation list
|
|
— if a machine is decommissioned rather than retired, reissue the authority or
|
|
accept that a certificate naming a node that no longer exists can still connect
|
|
and claim to be it.
|
|
|
|
## Testing without hardware
|
|
|
|
Every part of this works on machines with no GPUs and no InfiniBand. Point
|
|
`--nvidia-smi` at a script that prints the CSV discovery asks for, point
|
|
`--ib-root` at a sysfs-shaped directory, and use `engine: sim` in the catalog.
|
|
|
|
The simulated engine rendezvouses for real: rank 0 binds the dist-init port and
|
|
does not serve until every worker has joined, and workers serve nothing. That
|
|
exercises the multi-node launch sequence, the fabric constraint, and failure and
|
|
rebuild — which is most of what goes wrong — without exercising NCCL, which is
|
|
the rest.
|