Ilker ManapandClaude Opus 5.5
7d97c6b74f
A node ran whatever llama-server somebody had once unpacked on it. Onboarding checked an engine only when told its path, installed none, and a node with nothing on it registered as ready. The one time that went wrong it went wrong invisibly: the binary was there, executable, and could not start for want of libgomp1, while the node reported healthy. The control plane now says which engines every node has, at which version — `engines:` in server.yaml, defaulting to the llama.cpp build this release was checked against — and each agent converges on it. What only the node can know, the node works out: the build for its processor and driver (CUDA with its runtime when an NVIDIA driver is loaded, CPU otherwise), the OS packages that build needs from its own package sources, and whether the result runs. Every archive is checked against the sha256 GitHub publishes for it and unpacked refusing any path or link that leaves the install directory. ldd names a missing library outright; running `--version` proves the rest. Only then is a directory marked installed, and it is checked again every ten minutes, because an unrelated upgrade can remove a library. An engine is reported only once installed and seen to run, or when the agent's configuration names it on purpose — never because something happened to be on PATH. The agent now says when it has probed, so an empty list means "has none" rather than "too old to say": a node rebuilt from nothing is no longer a candidate for every model at the moment it can run none of them. Placement's refusal carries the node's own words, "installing: downloading llama-b10991-bin-ubuntu-x64.tar.gz (16 MB)", and so do `ctl nodes` and the cluster page, along with a warning for a CPU without AVX2 — the VM CPU type that makes everything five times slower and nothing look wrong. Agents also report their OS, architecture, vector extensions and NVIDIA driver. Onboarding installs llmgw itself from the CI release for the node's architecture, checked against the release's checksums, rather than copying whatever binary the operator's machine had. vLLM and SGLang are not installed yet: they are Python environments of several gigabytes, not an archive, and still have to be named in the agent's configuration. Also fixes a race on the weights fingerprint, which two heartbeat loops touch during a failover. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
113 lines
4.4 KiB
Bash
Executable File
113 lines
4.4 KiB
Bash
Executable File
#!/bin/sh
|
|
# add-node.sh — build a node and put it in the fleet, in one command.
|
|
#
|
|
# provision-node.sh and onboard-node.sh stay separate because the two halves
|
|
# need different access: one the hypervisor, one the certificate authority.
|
|
# That is a real boundary and this script does not erase it — it drives both
|
|
# where one person holds both, which is the common case and was the case here.
|
|
#
|
|
# The reason to have it is not keystrokes. The VM name, the hostname, the node
|
|
# id and the certificate's common name must all be the same string, and the
|
|
# address appears in three places; split across two invocations run minutes
|
|
# apart, that agreement is maintained by whoever is typing. Here it is stated
|
|
# once.
|
|
set -eu
|
|
|
|
usage() {
|
|
cat >&2 <<'USAGE'
|
|
usage: add-node.sh --name NODE --ip ADDR/CIDR --server URL [options]
|
|
|
|
--name NODE node id, hostname and VM name; also the certificate's CN
|
|
--ip ADDR/CIDR e.g. 10.0.0.145/24
|
|
--server URL control plane, e.g. https://10.0.0.140:7443
|
|
|
|
--vmid N Proxmox VMID (default: the last octet of --ip)
|
|
--cores N (default 4)
|
|
--cpu TYPE QEMU CPU model (default host; see provision-node.sh)
|
|
--memory MB (default 8192)
|
|
--disk GB (default 20)
|
|
--fabric NAME InfiniBand fabric; repeatable as NAME:hca1,hca2.
|
|
Omit for a node with no fabric.
|
|
--llamacpp-bin P path to llama-server on the node
|
|
--vllm-bin P path to vllm
|
|
--sglang-python P path to the python that has sglang
|
|
--binary PATH llmgw binary to install (default: the one on PATH)
|
|
--pki DIR certificate authority (default /etc/llmgw/pki)
|
|
--ssh-user U (default ilker)
|
|
--provision-only build the VM and stop
|
|
--onboard-only skip provisioning; the machine already exists
|
|
USAGE
|
|
exit 2
|
|
}
|
|
|
|
NAME= IP= SERVER= VMID= CORES=4 MEMORY=8192 DISK=20 CPU=
|
|
BINARY= PKI=/etc/llmgw/pki SSH_USER=ilker
|
|
PROVISION=1 ONBOARD=1
|
|
set -- "$@"
|
|
PASS_THROUGH=
|
|
while [ $# -gt 0 ]; do
|
|
case "$1" in
|
|
--name) NAME="$2"; shift 2;;
|
|
--ip) IP="$2"; shift 2;;
|
|
--server) SERVER="$2"; shift 2;;
|
|
--vmid) VMID="$2"; shift 2;;
|
|
--cores) CORES="$2"; shift 2;;
|
|
--cpu) CPU="$2"; shift 2;;
|
|
--memory) MEMORY="$2"; shift 2;;
|
|
--disk) DISK="$2"; shift 2;;
|
|
--binary) BINARY="$2"; shift 2;;
|
|
--pki) PKI="$2"; shift 2;;
|
|
--ssh-user) SSH_USER="$2"; shift 2;;
|
|
--provision-only) ONBOARD=0; shift;;
|
|
--onboard-only) PROVISION=0; shift;;
|
|
# Everything the onboarding step understands and this one does not need to.
|
|
--fabric|--llamacpp-bin|--vllm-bin|--sglang-python|--ib-root|--nvidia-smi|--mgmt-addr|--mgmt-if)
|
|
PASS_THROUGH="$PASS_THROUGH $1 $2"; shift 2;;
|
|
-h|--help) usage;;
|
|
*) echo "unknown option: $1" >&2; usage;;
|
|
esac
|
|
done
|
|
[ -n "$NAME" ] && [ -n "$IP" ] && [ -n "$SERVER" ] || usage
|
|
|
|
# The address without its prefix, which is what onboarding and ssh want.
|
|
ADDR="${IP%%/*}"
|
|
# The last octet as the VMID by default: it keeps `qm list` and `ctl nodes` in
|
|
# the same order, and one fewer number to choose is one fewer to get wrong.
|
|
[ -n "$VMID" ] || VMID="${ADDR##*.}"
|
|
|
|
here() { dirname "$0"; }
|
|
PROVISION_SH="$(here)/provision-node.sh"
|
|
ONBOARD_SH="$(here)/onboard-node.sh"
|
|
[ -x "$PROVISION_SH" ] || PROVISION_SH="$(command -v llmgw-provision-node || true)"
|
|
[ -x "$ONBOARD_SH" ] || ONBOARD_SH="$(command -v llmgw-onboard-node || true)"
|
|
|
|
if [ "$PROVISION" = 1 ]; then
|
|
[ -x "$PROVISION_SH" ] || { echo "cannot find provision-node.sh" >&2; exit 1; }
|
|
echo "== 1/2 building $NAME as VM $VMID"
|
|
# shellcheck disable=SC2086
|
|
"$PROVISION_SH" --vmid "$VMID" --name "$NAME" --ip "$IP" \
|
|
--cores "$CORES" --memory "$MEMORY" --disk "$DISK" ${CPU:+--cpu "$CPU"}
|
|
fi
|
|
|
|
if [ "$ONBOARD" = 0 ]; then
|
|
echo "$NAME built. Onboard it from the CA host when you are ready."
|
|
exit 0
|
|
fi
|
|
|
|
[ -x "$ONBOARD_SH" ] || { echo "cannot find onboard-node.sh" >&2; exit 1; }
|
|
echo
|
|
echo "== 2/2 onboarding $NAME"
|
|
# Onboarding waits for ssh itself, which is what makes running the two halves
|
|
# back to back safe: cloud-init is still regenerating host keys and restarting
|
|
# sshd for the first minute of a new machine's life.
|
|
# shellcheck disable=SC2086
|
|
"$ONBOARD_SH" --node "$NAME" --addr "$ADDR" --server "$SERVER" \
|
|
--pki "$PKI" --ssh-user "$SSH_USER" \
|
|
${BINARY:+--binary "$BINARY"} $PASS_THROUGH
|
|
|
|
echo
|
|
echo "Check it with: llmgw ctl nodes"
|
|
echo
|
|
echo "The agent installs its engines itself; the node takes no model until they"
|
|
echo "are installed and seen to run. 'llmgw ctl nodes' shows where that stands."
|