Ilker ManapandClaude Opus 5
7b4acd9284
Restarting the control plane over a running two-node replica adopted half of it. Every shard it still had was healthy, so it was reported Ready — with one shard of two and no endpoint at all, because the missing one was rank 0. The router was handed a replica nothing could route to. The cause is that a replica had no idea how many shards it should have. It could tell "every shard I have is ready" and nothing else, which is the same sentence whether or not half of it is gone. It now carries the count from its profile, set at placement and again at adoption, and a replica short of shards fails rather than pretending. Onboarding also wrote no engine paths, so a node onboarded by the script could not find llama-server and failed every shard placed on it with "executable file not found". The script now takes them, and says so when none are given — a node with no engine paths can run the sim engine and nothing else. The cluster now runs both at once: two-node simulated replicas on each fabric, and real llama.cpp inference on the CPU, which is a better test bed than either alone. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
125 lines
5.0 KiB
Markdown
125 lines
5.0 KiB
Markdown
# Adding a node
|
|
|
|
Written for whoever runs the cluster. Two scripts, because the two halves of
|
|
onboarding are done by different people with different access: one needs the
|
|
hypervisor, the other needs the certificate authority.
|
|
|
|
## 1. The machine
|
|
|
|
On the Proxmox host:
|
|
|
|
```
|
|
llmgw-provision-node --vmid 144 --name llmgw-node04 --ip 10.0.0.144/24
|
|
```
|
|
|
|
Name it after the node id. **The VM name, the hostname, the node id and the
|
|
certificate's common name are all the same string.** They do not have to be, and
|
|
every time they are not, somebody spends an afternoon working out which of the
|
|
four a message is talking about.
|
|
|
|
It imports a cloud image and lets cloud-init do the rest, which is why a node
|
|
takes a minute rather than an afternoon. It refuses to touch an existing VMID:
|
|
adopting a machine is not a decision a provisioning script should make.
|
|
|
|
## 2. The identity
|
|
|
|
On the host that holds the certificate authority — this is the part that cannot
|
|
be delegated, because it issues the credential by which the control plane
|
|
believes the node is who it says it is:
|
|
|
|
```
|
|
llmgw-onboard-node \
|
|
--node llmgw-node04 \
|
|
--addr 10.0.0.144 \
|
|
--server https://10.0.0.140:7443 \
|
|
--fabric ib-b
|
|
```
|
|
|
|
It issues a client certificate whose common name is the node id, installs it
|
|
along with the CA root, trusts that root system-wide so other tooling on the
|
|
machine stops treating llmgw's certificates as strangers, installs the binary
|
|
and the unit, and starts the agent. The node appears in `llmgw ctl nodes` within
|
|
a few seconds.
|
|
|
|
If it does not, the control plane's log says why. A certificate whose common
|
|
name does not match the claimed node id is refused by design — that check is
|
|
what makes a node's identity something it proves rather than something it picks.
|
|
|
|
**It needs ssh from the CA host to the node.** Provisioning copies that host's
|
|
`authorized_keys` into the new machine, so a node built by the script above
|
|
already satisfies it; an older machine may need the key added.
|
|
|
|
## What onboarding does not do
|
|
|
|
It gives a machine an identity and an agent. It does **not** install inference
|
|
engines or model weights, and a node missing either looks healthy in
|
|
`llmgw ctl nodes` right up until a replica is placed on it and the shard dies
|
|
with "executable file not found".
|
|
|
|
So the node is not finished until:
|
|
|
|
- the engines its catalog profiles name are installed (`vllm`, `sglang`,
|
|
`llama-server`), at the paths the agent configuration gives
|
|
- the weights are on disk, at the `source` each model declares
|
|
|
|
Both are deliberately outside this script. Engines are a platform decision with
|
|
versions and drivers attached, and distributing weights is its own problem —
|
|
hundreds of gigabytes that want a shared filesystem or a cache, not an scp from
|
|
a laptop. llmgw has no answer for it yet; that is a known gap, not an oversight.
|
|
|
|
The `sim` engine needs neither, which is what makes it useful for exercising
|
|
everything else.
|
|
|
|
## Fabrics
|
|
|
|
`--fabric` is how a node says which InfiniBand fabric it is attached to, and
|
|
there is no way around stating it. OpenSM's default subnet prefix is `fe80::` on
|
|
every fabric, so a matching prefix proves nothing and a wrong guess produces a
|
|
replica that starts, passes health checks, and then hangs on its first
|
|
all-reduce.
|
|
|
|
Omit it for a node with no fabric. Such a node is its own island: it can host
|
|
single-node replicas and will never be given a multi-node one.
|
|
|
|
Two adapters per fabric is the default; `--fabric ib-b:mlx5_2,mlx5_3` names them
|
|
explicitly. An adapter must never appear under two fabrics — the agent refuses
|
|
that configuration, because the consequence is the hang above.
|
|
|
|
## Where the driver is
|
|
|
|
```
|
|
--nvidia-smi /opt/nvidia/bin/nvidia-smi
|
|
--ib-root /sys/class/infiniband
|
|
```
|
|
|
|
Both default to the usual places. Containerized driver installs routinely put
|
|
`nvidia-smi` somewhere else, and an agent that cannot find it registers with no
|
|
GPUs — which looks exactly like a node that has none.
|
|
|
|
## Removing a node
|
|
|
|
```
|
|
llmgw ctl drain llmgw-node04 # move its replicas off, one at a time
|
|
llmgw ctl nodes # wait until it holds none
|
|
```
|
|
|
|
Then stop the agent. Draining brings each replacement up before stopping the
|
|
original, which is the difference between a rolling move and an outage.
|
|
|
|
The node's certificate stays valid until it expires. There is no revocation list
|
|
— if a machine is decommissioned rather than retired, reissue the authority or
|
|
accept that a certificate naming a node that no longer exists can still connect
|
|
and claim to be it.
|
|
|
|
## Testing without hardware
|
|
|
|
Every part of this works on machines with no GPUs and no InfiniBand. Point
|
|
`--nvidia-smi` at a script that prints the CSV discovery asks for, point
|
|
`--ib-root` at a sysfs-shaped directory, and use `engine: sim` in the catalog.
|
|
|
|
The simulated engine rendezvouses for real: rank 0 binds the dist-init port and
|
|
does not serve until every worker has joined, and workers serve nothing. That
|
|
exercises the multi-node launch sequence, the fabric constraint, and failure and
|
|
rebuild — which is most of what goes wrong — without exercising NCCL, which is
|
|
the rest.
|