Make adding a node one command, and check the engine actually runs
CI / test (push) Successful in 1m26s
CI / test (push) Successful in 1m26s
Building node05 by hand turned up three things worth fixing, in the order they bit. Onboarding died on "Host key verification failed" — an error about ssh, in the middle of a script whose entire job is to talk to a machine created sixty seconds ago. It now uses accept-new, which learns an unknown host key and still refuses a changed one; the risk being guarded against is a substituted machine, not a new one. Then ssh started refusing connections on a node that had answered a moment before: cloud-init regenerates the host keys and restarts sshd partway through a new machine's first minute. The agent was registered and healthy throughout. Onboarding now waits for ssh rather than failing on it, which is also what makes running the two halves back to back safe. Then llama-server: present, executable, and unable to start for want of libgomp1. The node registered ready and would have stayed ready until a replica was placed on it and the shard died with a missing shared library three logs away. The docs already warned about this shape of failure; nothing checked for it. Onboarding now runs a configured engine and says plainly when it will not start. add-node.sh drives both halves. The reason is not keystrokes: the VM name, the hostname, the node id and the certificate's CN must all be the same string, and the address appears in three places. Split across two commands run minutes apart, that agreement is held together by whoever is typing. Here it is stated once, and the VMID defaults to the last octet of the address so qm list and ctl nodes stay in the same order. The two halves remain separately runnable, because needing the hypervisor and needing the CA really are different access. Verified by building a throwaway node06 with it: 45 seconds from nothing to registered, and the engine check caught that node06 had no llama-server. Destroyed afterwards. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
082d52bc97
commit
aa4da3264d
3 files changed
+218
-7
No files matched your search
@@ -50,6 +50,7 @@ USAGE
|
||||
NODE= ADDR= SERVER= MGMT= MGMT_IF=eth0
|
||||
NVIDIA_SMI= IB_ROOT=/sys/class/infiniband
|
||||
LLAMACPP_BIN= VLLM_BIN= SGLANG_PYTHON=
|
||||
SSH_WAIT=180
|
||||
PKI=/etc/llmgw/pki BINARY= SSH_USER=ilker
|
||||
FABRIC_NAME= FABRIC_HCAS="mlx5_0,mlx5_1"
|
||||
|
||||
@@ -78,11 +79,33 @@ BINARY="${BINARY:-$(command -v llmgw)}"
|
||||
[ -x "$BINARY" ] || { echo "no llmgw binary; pass --binary" >&2; exit 1; }
|
||||
[ -f "$PKI/ca-key.pem" ] || { echo "no authority in $PKI; run 'llmgw ctl cert init-ca' first" >&2; exit 1; }
|
||||
|
||||
SSH="ssh -o BatchMode=yes -o ConnectTimeout=10 $SSH_USER@$ADDR"
|
||||
# accept-new, not "no": a machine provisioned a minute ago has a host key
|
||||
# nobody has seen, and refusing it stopped onboarding dead with "Host key
|
||||
# verification failed" — an error about ssh, in the middle of a script whose
|
||||
# whole job is to talk to a new machine. accept-new learns an unknown key and
|
||||
# still refuses a changed one, which is the distinction that matters: the risk
|
||||
# being guarded against is a substituted machine, not a new one.
|
||||
SSHOPTS="-o BatchMode=yes -o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new"
|
||||
SSH="ssh $SSHOPTS $SSH_USER@$ADDR"
|
||||
say() { printf ' %s\n' "$*"; }
|
||||
|
||||
echo "onboarding $NODE at $ADDR"
|
||||
$SSH true || { echo "cannot reach $SSH_USER@$ADDR over ssh" >&2; exit 1; }
|
||||
|
||||
# Waiting rather than failing: cloud-init restarts sshd after it regenerates
|
||||
# the host keys, so a machine that answered a moment ago refuses connections
|
||||
# for a few seconds. Observed while onboarding node05 — the agent was already
|
||||
# registered and healthy while ssh was refusing.
|
||||
wait_for_ssh() {
|
||||
deadline=$(( $(date +%s) + SSH_WAIT ))
|
||||
until $SSH true 2>/dev/null; do
|
||||
[ "$(date +%s)" -lt "$deadline" ] || {
|
||||
echo "cannot reach $SSH_USER@$ADDR over ssh after ${SSH_WAIT}s" >&2
|
||||
exit 1
|
||||
}
|
||||
sleep 5
|
||||
done
|
||||
}
|
||||
wait_for_ssh
|
||||
|
||||
# 1. The certificate. Issued fresh every time: re-onboarding a machine should
|
||||
# not depend on finding an old key, and the old one keeps working until it
|
||||
@@ -95,7 +118,7 @@ trap 'rm -rf "$TMP"' EXIT
|
||||
# 2 and 3. Identity, and trusting the root so future tooling on that machine
|
||||
# does not treat our certificates as strangers.
|
||||
say "installing the certificate and trusting the root"
|
||||
scp -q "$PKI/ca.pem" "$TMP/$NODE.pem" "$TMP/$NODE-key.pem" "$SSH_USER@$ADDR:/tmp/"
|
||||
scp -q $SSHOPTS "$PKI/ca.pem" "$TMP/$NODE.pem" "$TMP/$NODE-key.pem" "$SSH_USER@$ADDR:/tmp/"
|
||||
$SSH "set -e
|
||||
sudo install -d -m 0755 /etc/llmgw
|
||||
sudo install -m 0644 /tmp/ca.pem /etc/llmgw/ca.pem
|
||||
@@ -108,7 +131,7 @@ $SSH "set -e
|
||||
|
||||
# 4. The binary, the configuration and the unit.
|
||||
say "installing the binary"
|
||||
scp -q "$BINARY" "$SSH_USER@$ADDR:/tmp/llmgw"
|
||||
scp -q $SSHOPTS "$BINARY" "$SSH_USER@$ADDR:/tmp/llmgw"
|
||||
$SSH 'sudo install -m 0755 /tmp/llmgw /usr/local/bin/llmgw && rm -f /tmp/llmgw'
|
||||
|
||||
say "writing the configuration"
|
||||
@@ -192,6 +215,35 @@ WantedBy=multi-user.target
|
||||
UNIT
|
||||
scp -q "$TMP/llmgw-agent.service" "$SSH_USER@$ADDR:/tmp/"
|
||||
|
||||
# An engine path that is wrong, or right but unrunnable, is the failure this
|
||||
# node type is most likely to have, and it is invisible: the agent registers
|
||||
# healthy and every shard placed on it dies with "executable file not found"
|
||||
# or a missing shared library. Checked here because it is cheap here and
|
||||
# expensive later — llama-server on node05 was present, executable, and could
|
||||
# not start for want of libgomp1, and the node reported ready throughout.
|
||||
check_engine() {
|
||||
[ -n "$2" ] || return 0
|
||||
if $SSH "test -x '$2'" 2>/dev/null; then
|
||||
if $SSH "'$2' --version >/dev/null 2>&1 || '$2' --help >/dev/null 2>&1"; then
|
||||
say "$1: $2 runs"
|
||||
return 0
|
||||
fi
|
||||
ENGINE_PROBLEMS="$ENGINE_PROBLEMS
|
||||
$1 at $2 exists but will not run. Usually a missing shared library:
|
||||
ssh $SSH_USER@$ADDR ldd $2 | grep 'not found'"
|
||||
return 0
|
||||
fi
|
||||
ENGINE_PROBLEMS="$ENGINE_PROBLEMS
|
||||
$1 is configured at $2, which is not an executable on $NODE."
|
||||
}
|
||||
ENGINE_PROBLEMS=
|
||||
if [ -n "$LLAMACPP_BIN$VLLM_BIN$SGLANG_PYTHON" ]; then
|
||||
say "checking the engines"
|
||||
check_engine llamacpp "$LLAMACPP_BIN"
|
||||
check_engine vllm "$VLLM_BIN"
|
||||
check_engine sglang "$SGLANG_PYTHON"
|
||||
fi
|
||||
|
||||
say "starting the agent"
|
||||
$SSH 'set -e
|
||||
sudo install -m 0644 -o root -g root /tmp/agent.yaml /etc/llmgw/agent.yaml
|
||||
@@ -206,6 +258,13 @@ if [ -z "$LLAMACPP_BIN$VLLM_BIN$SGLANG_PYTHON" ]; then
|
||||
say "no engine paths given; this node can only run the 'sim' engine until"
|
||||
say "one is installed and the configuration points at it"
|
||||
fi
|
||||
if [ -n "$ENGINE_PROBLEMS" ]; then
|
||||
echo "$NODE is onboarded but an engine will not run:$ENGINE_PROBLEMS"
|
||||
echo
|
||||
echo "The node will register as ready regardless. Fix this before a replica"
|
||||
echo "is placed on it, or the shard fails and the reason is three logs away."
|
||||
echo
|
||||
fi
|
||||
echo "$NODE onboarded. It should appear in 'llmgw ctl nodes' within a few seconds."
|
||||
echo "If it does not, the control plane's log says why — a certificate whose"
|
||||
echo "common name does not match the claimed node id is refused by design."
|
||||
Reference in new issue
Block a user