Every agent file on the cluster named its own node, its own address, the path to llama-server and two weight directories — different on every node, kept true by hand, and not true: node05 carried "ram: 32g" beside a measured 31 GB, looked for weights in an empty directory while fetching them into another, and the other four never looked in theirs at all, so four nodes reported no disk. Growing a node's RAM would have gone unnoticed, because memory was read once, at registration. The agent now finds these out. Its id is the hostname, which is also the certificate's common name. Its address is the one on the route to the control plane, which the kernel picks for a UDP dial without sending anything. llama-server is looked for on PATH and then where release archives are unpacked. A missing nvidia-smi is a node without NVIDIA cards rather than a failed discovery — which also cut discovery short before InfiniBand on every CPU node not told to skip GPUs. Fetched weights land in /var/lib/llmgw-weights by default, and wherever they land is always looked in. Memory is read on every heartbeat, and installed engines are probed again each minute and sent when they change, so a VM given more RAM or a node given vLLM becomes a candidate without anyone restarting its agent. The control plane logs both changes, since "the node now qualifies" is the answer to why a model placed an hour after it was asked for. What is written still wins. The agent file shrinks to the control plane's addresses and the TLS files, the same on every node; onboarding writes only the overrides it was given. What cannot be found stays configuration: which fabric an adapter is on, because OpenSM's default prefix is the same on all of them. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
2cae92d113
commit
20deb22dfa
14 files changed
+552
-134
No files matched your search
+22
-12
@@ -4,12 +4,19 @@
|
||||
# keep serving if the agent restarts: the router talks to them directly, and
|
||||
# orphan adoption picks them up again on the next start.
|
||||
|
||||
node_id: gpu-a-01 # defaults to the hostname
|
||||
# Everything a node can find out about itself, it does: its id is its hostname,
|
||||
# its address is the one on the route to the control plane, llama-server is
|
||||
# looked for on PATH and in /opt/llama.cpp, GPUs are there when nvidia-smi is,
|
||||
# weights are fetched into /var/lib/llmgw-weights and looked for there, and
|
||||
# memory, disk and installed engines are re-read while the agent runs. So a
|
||||
# plain CPU node needs only the two blocks below, and the same file on every
|
||||
# node. What follows them are overrides, for when the defaults are wrong.
|
||||
|
||||
server: "http://llmgw-server.internal:7443"
|
||||
|
||||
# The agent's half of mutual TLS. The certificate's common name must be this
|
||||
# node's id, because the control plane compares the two and refuses the stream
|
||||
# when they disagree:
|
||||
# node's id — its hostname, unless node_id says otherwise — because the control
|
||||
# plane compares the two and refuses the stream when they disagree:
|
||||
#
|
||||
# llmgw ctl cert issue-client gpu-a-01 --dir /etc/llmgw/pki
|
||||
#
|
||||
@@ -22,13 +29,19 @@ server: "http://llmgw-server.internal:7443"
|
||||
# cert: /etc/llmgw/agent.pem
|
||||
# key: /etc/llmgw/agent-key.pem
|
||||
|
||||
# ---- overrides ---------------------------------------------------------------
|
||||
|
||||
# node_id: gpu-a-01 # when it is not the hostname
|
||||
|
||||
# The address engines are reachable on, and the address NCCL bootstraps over.
|
||||
# Never a fabric address: NCCL rendezvouses over TCP before moving to RDMA.
|
||||
mgmt_addr: "10.0.0.11"
|
||||
mgmt_interface: eth0
|
||||
# mgmt_addr: "10.0.0.11"
|
||||
# mgmt_interface: eth0
|
||||
|
||||
labels:
|
||||
rack: r12
|
||||
# Free-form, for a profile's node_selector. Nothing needs them; do not restate
|
||||
# what is measured (memory, GPUs, engines), because a label goes stale.
|
||||
# labels:
|
||||
# rack: r12
|
||||
|
||||
heartbeat_interval: 5s
|
||||
reconnect_base: 1s
|
||||
@@ -55,12 +68,9 @@ discovery:
|
||||
# tensor-parallel group off the PCIe path inside a node. It needs a binary
|
||||
# built with `-tags nvml`; otherwise this silently falls back to nvidia-smi.
|
||||
prefer_nvml: true
|
||||
skip_gpus: false
|
||||
|
||||
# Where this node keeps model weights.
|
||||
#
|
||||
# Unset by default, and unset means the node is not asked to look — which is
|
||||
# not the same as having nothing, and constrains nothing. Storage differs at
|
||||
# Where this node keeps model weights besides /var/lib/llmgw-weights, which is
|
||||
# where fetched weights land and is always looked in. Storage differs at
|
||||
# every installation and llmgw deliberately does not own it: a Lustre mount, an
|
||||
# image exported over the fabric with SRP or NVMe-oF, local NVMe, or a copy
|
||||
# made by hand all look identical from here.
|
||||
|
||||
Reference in new issue
Block a user