Let a node describe itself instead of being described by hand
CI / test (push) Successful in 1m47s

Every agent file on the cluster named its own node, its own address, the
path to llama-server and two weight directories — different on every node,
kept true by hand, and not true: node05 carried "ram: 32g" beside a measured
31 GB, looked for weights in an empty directory while fetching them into
another, and the other four never looked in theirs at all, so four nodes
reported no disk. Growing a node's RAM would have gone unnoticed, because
memory was read once, at registration.

The agent now finds these out. Its id is the hostname, which is also the
certificate's common name. Its address is the one on the route to the
control plane, which the kernel picks for a UDP dial without sending
anything. llama-server is looked for on PATH and then where release archives
are unpacked. A missing nvidia-smi is a node without NVIDIA cards rather
than a failed discovery — which also cut discovery short before InfiniBand
on every CPU node not told to skip GPUs. Fetched weights land in
/var/lib/llmgw-weights by default, and wherever they land is always looked
in.

Memory is read on every heartbeat, and installed engines are probed again
each minute and sent when they change, so a VM given more RAM or a node
given vLLM becomes a candidate without anyone restarting its agent. The
control plane logs both changes, since "the node now qualifies" is the
answer to why a model placed an hour after it was asked for.

What is written still wins. The agent file shrinks to the control plane's
addresses and the TLS files, the same on every node; onboarding writes only
the overrides it was given. What cannot be found stays configuration: which
fabric an adapter is on, because OpenSM's default prefix is the same on all
of them.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-09-28 10:13:42 +02:00
1 parent 2cae92d113
commit 20deb22dfa
14 files changed
+552 -134

No files matched your search

+51 -64
View File
@@ -29,9 +29,11 @@ usage: $0 --node ID --addr HOST --server URL [options]
--fabric NAME:hca1,hca2 . Omit for a node with no fabric,
which can host single-node replicas and will never be given
a multi-node one.
--mgmt-addr ADDR address engines are reachable on (default: --addr)
--mgmt-if NAME interface NCCL bootstraps over (default eth0)
--llamacpp-bin P path to llama-server
--mgmt-addr ADDR address engines are reachable on (default: the node's
address on the route to --server, found by the agent)
--mgmt-if NAME interface NCCL bootstraps over (default: the one holding
that address)
--llamacpp-bin P path to llama-server (default: PATH, then /opt/llama.cpp)
--vllm-bin P path to the vllm entry point
--sglang-python P python interpreter of the SGLang virtualenv
--nvidia-smi PATH where nvidia-smi lives, when it is not on PATH. Containerized
@@ -47,7 +49,7 @@ USAGE
exit 2
}
NODE= ADDR= SERVER= MGMT= MGMT_IF=eth0
NODE= ADDR= SERVER= MGMT_OVERRIDE= MGMT_IF_OVERRIDE=
NVIDIA_SMI= IB_ROOT=/sys/class/infiniband
LLAMACPP_BIN= VLLM_BIN= SGLANG_PYTHON=
SSH_WAIT=180
@@ -60,8 +62,8 @@ while [ $# -gt 0 ]; do
--addr) ADDR="$2"; shift 2;;
--server) SERVER="$2"; shift 2;;
--fabric) FABRIC_NAME="${2%%:*}"; [ "${2#*:}" != "$2" ] && FABRIC_HCAS="${2#*:}"; shift 2;;
--mgmt-addr) MGMT="$2"; shift 2;;
--mgmt-if) MGMT_IF="$2"; shift 2;;
--mgmt-addr) MGMT_OVERRIDE="$2"; shift 2;;
--mgmt-if) MGMT_IF_OVERRIDE="$2"; shift 2;;
--llamacpp-bin) LLAMACPP_BIN="$2"; shift 2;;
--vllm-bin) VLLM_BIN="$2"; shift 2;;
--sglang-python) SGLANG_PYTHON="$2"; shift 2;;
@@ -74,7 +76,6 @@ while [ $# -gt 0 ]; do
esac
done
[ -n "$NODE" ] && [ -n "$ADDR" ] && [ -n "$SERVER" ] || usage
MGMT="${MGMT:-$ADDR}"
BINARY="${BINARY:-$(command -v llmgw)}"
[ -x "$BINARY" ] || { echo "no llmgw binary; pass --binary" >&2; exit 1; }
[ -f "$PKI/ca-key.pem" ] || { echo "no authority in $PKI; run 'llmgw ctl cert init-ca' first" >&2; exit 1; }
@@ -135,63 +136,49 @@ scp -q $SSHOPTS "$BINARY" "$SSH_USER@$ADDR:/tmp/llmgw"
$SSH 'sudo install -m 0755 /tmp/llmgw /usr/local/bin/llmgw && rm -f /tmp/llmgw'
say "writing the configuration"
FABRICS="fabrics: []"
if [ -n "$FABRIC_NAME" ]; then
FABRICS="fabrics:
- name: $FABRIC_NAME
hcas: [$(echo "$FABRIC_HCAS" | sed 's/,/, /g')]"
fi
cat > "$TMP/agent.yaml" <<YAML
# $NODE
#
# Written by onboard-node.sh. Fabric membership is configuration and there is no
# way around it: OpenSM's default subnet prefix is the same on every fabric, so
# a matching prefix proves nothing and a wrong guess produces a replica that
# starts, passes health checks and then hangs on its first all-reduce.
node_id: $NODE
server: "$SERVER"
tls:
ca: /etc/llmgw/ca.pem
cert: /etc/llmgw/agent.pem
key: /etc/llmgw/agent-key.pem
mgmt_addr: "$MGMT"
mgmt_interface: $MGMT_IF
heartbeat_interval: 5s
reconnect_base: 1s
reconnect_max: 30s
$FABRICS
discovery:
# Empty means look on PATH. Worth setting where the driver lives outside the
# default paths, which containerized installs routinely do.
nvidia_smi: "$NVIDIA_SMI"
ib_root: $IB_ROOT
prefer_nvml: false
skip_gpus: false
# Where the engines live. Empty means "on PATH", which is rarely true: a node
# that cannot find its engine registers healthy and then fails every shard
# placed on it with "executable file not found".
engines:
llamacpp:
bin: "$LLAMACPP_BIN"
parallel: 4
vllm:
bin: "$VLLM_BIN"
launch_mode: ray
sglang:
python: "$SGLANG_PYTHON"
supervisor:
work_dir: /var/lib/llmgw-agent
log_lines: 512
stop_timeout: 60s
health_interval: 2s
YAML
NODE_IS_HOSTNAME=no
[ "$($SSH hostname 2>/dev/null)" = "$NODE" ] && NODE_IS_HOSTNAME=yes
# Only what the node cannot find out for itself. Its name is its hostname, its
# address is the one on the route to the control plane, llama-server is looked
# for on PATH and where release archives are unpacked, GPUs are there when
# nvidia-smi is, and memory, disk and engines are re-read while it runs. Every
# setting written here is one that has to be kept true by hand — so an option
# below is written only when it was given, as an override.
{
echo "# $NODE, written by onboard-node.sh."
echo "server: \"$SERVER\""
echo
echo "tls:"
echo " ca: /etc/llmgw/ca.pem"
echo " cert: /etc/llmgw/agent.pem"
echo " key: /etc/llmgw/agent-key.pem"
# The certificate's common name is --node, and the control plane refuses a
# stream whose node id differs from it. Said explicitly only when the
# hostname would say something else.
[ "$NODE_IS_HOSTNAME" = yes ] || echo "node_id: $NODE"
[ -n "$MGMT_OVERRIDE" ] && echo "mgmt_addr: \"$MGMT_OVERRIDE\""
[ -n "$MGMT_IF_OVERRIDE" ] && echo "mgmt_interface: $MGMT_IF_OVERRIDE"
if [ -n "$FABRIC_NAME" ]; then
# Fabric membership is the one thing that cannot be found: OpenSM's default
# subnet prefix is the same on every fabric, so a matching prefix proves
# nothing and a wrong guess hangs the first all-reduce.
echo "fabrics:"
echo " - name: $FABRIC_NAME"
echo " hcas: [$(echo "$FABRIC_HCAS" | sed 's/,/, /g')]"
fi
if [ -n "$NVIDIA_SMI" ] || [ "$IB_ROOT" != /sys/class/infiniband ]; then
echo "discovery:"
[ -n "$NVIDIA_SMI" ] && echo " nvidia_smi: \"$NVIDIA_SMI\""
[ "$IB_ROOT" != /sys/class/infiniband ] && echo " ib_root: $IB_ROOT"
fi
if [ -n "$LLAMACPP_BIN$VLLM_BIN$SGLANG_PYTHON" ]; then
echo "engines:"
[ -n "$LLAMACPP_BIN" ] && printf ' llamacpp:\n bin: "%s"\n' "$LLAMACPP_BIN"
[ -n "$VLLM_BIN" ] && printf ' vllm:\n bin: "%s"\n' "$VLLM_BIN"
[ -n "$SGLANG_PYTHON" ] && printf ' sglang:\n python: "%s"\n' "$SGLANG_PYTHON"
fi
true
} > "$TMP/agent.yaml"
scp -q "$TMP/agent.yaml" "$SSH_USER@$ADDR:/tmp/agent.yaml"
cat > "$TMP/llmgw-agent.service" <<'UNIT'