Make adding a node one command, and check the engine actually runs
Building node05 by hand turned up three things worth fixing, in the order they bit. Onboarding died on "Host key verification failed" — an error about ssh, in the middle of a script whose entire job is to talk to a machine created sixty seconds ago. It now uses accept-new, which learns an unknown host key and still refuses a changed one; the risk being guarded against is a substituted machine, not a new one. Then ssh started refusing connections on a node that had answered a moment before: cloud-init regenerates the host keys and restarts sshd partway through a new machine's first minute. The agent was registered and healthy throughout. Onboarding now waits for ssh rather than failing on it, which is also what makes running the two halves back to back safe. Then llama-server: present, executable, and unable to start for want of libgomp1. The node registered ready and would have stayed ready until a replica was placed on it and the shard died with a missing shared library three logs away. The docs already warned about this shape of failure; nothing checked for it. Onboarding now runs a configured engine and says plainly when it will not start. add-node.sh drives both halves. The reason is not keystrokes: the VM name, the hostname, the node id and the certificate's CN must all be the same string, and the address appears in three places. Split across two commands run minutes apart, that agreement is held together by whoever is typing. Here it is stated once, and the VMID defaults to the last octet of the address so qm list and ctl nodes stay in the same order. The two halves remain separately runnable, because needing the hypervisor and needing the CA really are different access. Verified by building a throwaway node06 with it: 45 seconds from nothing to registered, and the engine check caught that node06 had no llama-server. Destroyed afterwards. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
082d52bc97
commit
aa4da3264d
3 files changed
+218
-7
No files matched your search
+45
-3
@@ -1,8 +1,33 @@
|
||||
# Adding a node
|
||||
|
||||
Written for whoever runs the cluster. Two scripts, because the two halves of
|
||||
onboarding are done by different people with different access: one needs the
|
||||
hypervisor, the other needs the certificate authority.
|
||||
Written for whoever runs the cluster.
|
||||
|
||||
## The short version
|
||||
|
||||
Where one person holds both the hypervisor and the certificate authority —
|
||||
which is the common case — one command does the whole thing in about
|
||||
45 seconds:
|
||||
|
||||
```
|
||||
llmgw-add-node \
|
||||
--name llmgw-node05 --ip 10.0.0.145/24 \
|
||||
--server https://10.0.0.140:7443 \
|
||||
--cores 8 --memory 32768 --disk 60 \
|
||||
--llamacpp-bin /opt/llama.cpp/llama-server
|
||||
```
|
||||
|
||||
The VMID defaults to the last octet of the address, so `qm list` and
|
||||
`llmgw ctl nodes` stay in the same order. `--provision-only` and
|
||||
`--onboard-only` split it back into the two halves below when the two kinds
|
||||
of access belong to different people.
|
||||
|
||||
The rest of this page is what that command does, and what it still does not.
|
||||
|
||||
## The two halves
|
||||
|
||||
Two scripts, because the two halves of onboarding are done by different people
|
||||
with different access: one needs the hypervisor, the other needs the
|
||||
certificate authority.
|
||||
|
||||
## 1. The machine
|
||||
|
||||
@@ -55,6 +80,16 @@ what makes a node's identity something it proves rather than something it picks.
|
||||
`authorized_keys` into the new machine, so a node built by the script above
|
||||
already satisfies it; an older machine may need the key added.
|
||||
|
||||
It waits for ssh rather than failing on it, and learns the host key of a machine
|
||||
it has not seen (`accept-new`, which still refuses a *changed* key — the risk
|
||||
being guarded against is a substituted machine, not a new one). Both are needed
|
||||
to run the halves back to back: for the first minute of its life a new machine
|
||||
is still regenerating host keys and restarting sshd, so it refuses connections
|
||||
it accepted a moment earlier.
|
||||
|
||||
It also checks that a configured engine actually runs, which is cheap here and
|
||||
expensive later — see below.
|
||||
|
||||
## What onboarding does not do
|
||||
|
||||
It gives a machine an identity and an agent. It does **not** install inference
|
||||
@@ -62,6 +97,13 @@ engines or model weights, and a node missing either looks healthy in
|
||||
`llmgw ctl nodes` right up until a replica is placed on it and the shard dies
|
||||
with "executable file not found".
|
||||
|
||||
Onboarding does check that an engine you point it at can be executed, and says
|
||||
so loudly when it cannot. That check exists because the failure is invisible
|
||||
otherwise: `llama-server` on node05 was present, executable, and could not start
|
||||
for want of `libgomp1`, and the node reported `ready` in `llmgw ctl nodes` the
|
||||
whole time. Being told at onboarding beats finding out when a replica is placed
|
||||
and the reason is three logs away.
|
||||
|
||||
So the node is not finished until:
|
||||
|
||||
- the engines its catalog profiles name are installed (`vllm`, `sglang`,
|
||||
|
||||
Reference in new issue
Block a user