Install a node's engines on the node, instead of trusting what is there
A node ran whatever llama-server somebody had once unpacked on it. Onboarding checked an engine only when told its path, installed none, and a node with nothing on it registered as ready. The one time that went wrong it went wrong invisibly: the binary was there, executable, and could not start for want of libgomp1, while the node reported healthy. The control plane now says which engines every node has, at which version — `engines:` in server.yaml, defaulting to the llama.cpp build this release was checked against — and each agent converges on it. What only the node can know, the node works out: the build for its processor and driver (CUDA with its runtime when an NVIDIA driver is loaded, CPU otherwise), the OS packages that build needs from its own package sources, and whether the result runs. Every archive is checked against the sha256 GitHub publishes for it and unpacked refusing any path or link that leaves the install directory. ldd names a missing library outright; running `--version` proves the rest. Only then is a directory marked installed, and it is checked again every ten minutes, because an unrelated upgrade can remove a library. An engine is reported only once installed and seen to run, or when the agent's configuration names it on purpose — never because something happened to be on PATH. The agent now says when it has probed, so an empty list means "has none" rather than "too old to say": a node rebuilt from nothing is no longer a candidate for every model at the moment it can run none of them. Placement's refusal carries the node's own words, "installing: downloading llama-b10991-bin-ubuntu-x64.tar.gz (16 MB)", and so do `ctl nodes` and the cluster page, along with a warning for a CPU without AVX2 — the VM CPU type that makes everything five times slower and nothing look wrong. Agents also report their OS, architecture, vector extensions and NVIDIA driver. Onboarding installs llmgw itself from the CI release for the node's architecture, checked against the release's checksums, rather than copying whatever binary the operator's machine had. vLLM and SGLang are not installed yet: they are Python environments of several gigabytes, not an archive, and still have to be named in the agent's configuration. Also fixes a race on the weights fingerprint, which two heartbeat loops touch during a failover. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
fa7f90880d
commit
7d97c6b74f
27 files changed
+2155
-192
No files matched your search
@@ -108,6 +108,5 @@ echo "== 2/2 onboarding $NAME"
|
||||
echo
|
||||
echo "Check it with: llmgw ctl nodes"
|
||||
echo
|
||||
echo "A node is not finished until the engines its profiles name are installed"
|
||||
echo "and the weights are on disk. Onboarding checks that a configured engine"
|
||||
echo "runs, but it installs neither — see docs/node-onboarding.md."
|
||||
echo "The agent installs its engines itself; the node takes no model until they"
|
||||
echo "are installed and seen to run. 'llmgw ctl nodes' shows where that stands."
|
||||
@@ -33,8 +33,8 @@ usage: $0 --node ID --addr HOST --server URL [options]
|
||||
address on the route to --server, found by the agent)
|
||||
--mgmt-if NAME interface NCCL bootstraps over (default: the one holding
|
||||
that address)
|
||||
--llamacpp-bin P path to llama-server (default: PATH, then /opt/llama.cpp)
|
||||
--vllm-bin P path to the vllm entry point
|
||||
--llamacpp-bin P use this llama-server instead of the one the agent installs
|
||||
--vllm-bin P path to the vllm entry point (not yet installed by the agent)
|
||||
--sglang-python P python interpreter of the SGLang virtualenv
|
||||
--nvidia-smi PATH where nvidia-smi lives, when it is not on PATH. Containerized
|
||||
driver installs routinely put it somewhere else.
|
||||
@@ -44,6 +44,8 @@ usage: $0 --node ID --addr HOST --server URL [options]
|
||||
Releases carry static linux/amd64 and linux/arm64 builds;
|
||||
downloading one is better than copying whatever happens to
|
||||
be on the machine you are running this from.
|
||||
--releases URL where release binaries are downloaded from (default the
|
||||
project's Gitea releases)
|
||||
--ssh-user NAME (default ilker)
|
||||
USAGE
|
||||
exit 2
|
||||
@@ -54,6 +56,7 @@ NVIDIA_SMI= IB_ROOT=/sys/class/infiniband
|
||||
LLAMACPP_BIN= VLLM_BIN= SGLANG_PYTHON=
|
||||
SSH_WAIT=180
|
||||
PKI=/etc/llmgw/pki BINARY= SSH_USER=ilker
|
||||
RELEASES=https://gitea.mynodes.xyz/ilker/llmgw/releases/download
|
||||
FABRIC_NAME= FABRIC_HCAS="mlx5_0,mlx5_1"
|
||||
|
||||
while [ $# -gt 0 ]; do
|
||||
@@ -71,6 +74,7 @@ while [ $# -gt 0 ]; do
|
||||
--ib-root) IB_ROOT="$2"; shift 2;;
|
||||
--pki) PKI="$2"; shift 2;;
|
||||
--binary) BINARY="$2"; shift 2;;
|
||||
--releases) RELEASES="$2"; shift 2;;
|
||||
--ssh-user) SSH_USER="$2"; shift 2;;
|
||||
*) usage;;
|
||||
esac
|
||||
@@ -131,9 +135,28 @@ $SSH "set -e
|
||||
rm -f /tmp/ca.pem /tmp/$NODE.pem /tmp/$NODE-key.pem"
|
||||
|
||||
# 4. The binary, the configuration and the unit.
|
||||
say "installing the binary"
|
||||
scp -q $SSHOPTS "$BINARY" "$SSH_USER@$ADDR:/tmp/llmgw"
|
||||
$SSH 'sudo install -m 0755 /tmp/llmgw /usr/local/bin/llmgw && rm -f /tmp/llmgw'
|
||||
# The node fetches the release CI built for this version, for its own
|
||||
# architecture, and checks it against the release's checksums — rather than
|
||||
# being handed whatever binary is on the machine running this script. A build
|
||||
# that is not a release (a developer's, with commits past the tag) has no such
|
||||
# file, and is copied as before.
|
||||
TAG="$("$BINARY" --version 2>/dev/null | awk '{print $NF}')"
|
||||
if printf '%s' "$TAG" | grep -Eq '^v[0-9]+\.[0-9]+\.[0-9]+$'; then
|
||||
say "installing llmgw $TAG from the release"
|
||||
$SSH "set -e
|
||||
arch=\$(dpkg --print-architecture)
|
||||
name=llmgw-$TAG-linux-\$arch
|
||||
cd /tmp && rm -f \$name checksums.txt
|
||||
curl -fsSL -o \$name '$RELEASES/$TAG/'\$name
|
||||
curl -fsSL -o checksums.txt '$RELEASES/$TAG/checksums.txt'
|
||||
grep \" \$name\\\$\" checksums.txt | sha256sum -c - >/dev/null
|
||||
sudo install -m 0755 \$name /usr/local/bin/llmgw
|
||||
rm -f \$name checksums.txt"
|
||||
else
|
||||
say "installing the binary (${TAG:-unversioned}, not a release)"
|
||||
scp -q $SSHOPTS "$BINARY" "$SSH_USER@$ADDR:/tmp/llmgw"
|
||||
$SSH 'sudo install -m 0755 /tmp/llmgw /usr/local/bin/llmgw && rm -f /tmp/llmgw'
|
||||
fi
|
||||
|
||||
say "writing the configuration"
|
||||
NODE_IS_HOSTNAME=no
|
||||
@@ -242,8 +265,9 @@ $SSH 'set -e
|
||||
|
||||
echo
|
||||
if [ -z "$LLAMACPP_BIN$VLLM_BIN$SGLANG_PYTHON" ]; then
|
||||
say "no engine paths given; this node can only run the 'sim' engine until"
|
||||
say "one is installed and the configuration points at it"
|
||||
say "the agent now installs the engines the control plane asks for: the"
|
||||
say "build for this machine, checked against its publisher's checksum and run"
|
||||
say "before the node is offered any model. 'llmgw ctl nodes' shows progress."
|
||||
fi
|
||||
if [ -n "$ENGINE_PROBLEMS" ]; then
|
||||
echo "$NODE is onboarded but an engine will not run:$ENGINE_PROBLEMS"
|
||||
|
||||
Reference in new issue
Block a user