Every agent file on the cluster named its own node, its own address, the path to llama-server and two weight directories — different on every node, kept true by hand, and not true: node05 carried "ram: 32g" beside a measured 31 GB, looked for weights in an empty directory while fetching them into another, and the other four never looked in theirs at all, so four nodes reported no disk. Growing a node's RAM would have gone unnoticed, because memory was read once, at registration. The agent now finds these out. Its id is the hostname, which is also the certificate's common name. Its address is the one on the route to the control plane, which the kernel picks for a UDP dial without sending anything. llama-server is looked for on PATH and then where release archives are unpacked. A missing nvidia-smi is a node without NVIDIA cards rather than a failed discovery — which also cut discovery short before InfiniBand on every CPU node not told to skip GPUs. Fetched weights land in /var/lib/llmgw-weights by default, and wherever they land is always looked in. Memory is read on every heartbeat, and installed engines are probed again each minute and sent when they change, so a VM given more RAM or a node given vLLM becomes a candidate without anyone restarting its agent. The control plane logs both changes, since "the node now qualifies" is the answer to why a model placed an hour after it was asked for. What is written still wins. The agent file shrinks to the control plane's addresses and the TLS files, the same on every node; onboarding writes only the overrides it was given. What cannot be found stays configuration: which fabric an adapter is on, because OpenSM's default prefix is the same on all of them. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
2cae92d113
commit
20deb22dfa
14 files changed
+552
-134
No files matched your search
+23
-7
@@ -330,6 +330,9 @@ into an outage is worse.
|
||||
|
||||
`string`
|
||||
|
||||
NodeID defaults to the hostname. It must equal the common name of the
|
||||
agent's certificate, so set it only where the two differ.
|
||||
|
||||
### `server`
|
||||
|
||||
`string`
|
||||
@@ -352,19 +355,30 @@ edit to every node's configuration.
|
||||
|
||||
`string`
|
||||
|
||||
Hostname defaults to the machine's.
|
||||
|
||||
### `mgmt_addr`
|
||||
|
||||
`string`
|
||||
|
||||
MgmtAddr is the address engines are reached on. It defaults to this
|
||||
machine's address on the route to the control plane, which is the one
|
||||
the control plane and the router can reach.
|
||||
|
||||
### `labels`
|
||||
|
||||
`map[string]string`
|
||||
|
||||
Labels are free-form facts about a node for a profile's node_selector.
|
||||
Nothing needs them: memory, GPUs, engines and disk are measured, and a
|
||||
label restating one of those goes stale the day the machine changes.
|
||||
|
||||
### `mgmt_interface`
|
||||
|
||||
`string`
|
||||
|
||||
MgmtInterface is the ethernet interface NCCL bootstraps over.
|
||||
MgmtInterface is the ethernet interface NCCL bootstraps over. It
|
||||
defaults to the interface holding MgmtAddr, then eth0.
|
||||
|
||||
### `fabrics`
|
||||
|
||||
@@ -411,10 +425,8 @@ Takes a [Config](#config) block.
|
||||
|
||||
`[]string`
|
||||
|
||||
ModelDirs are the directories this node keeps model weights in.
|
||||
|
||||
Empty by default, and empty means the node is not asked to look — which
|
||||
is not the same as having nothing. Storage differs at every installation,
|
||||
ModelDirs are the directories this node keeps model weights in, beyond
|
||||
WeightsCache, which is always looked in. Storage differs at every installation,
|
||||
so llmgw observes it rather than owning it: a Lustre mount, an image
|
||||
exported over the fabric, local NVMe and a copy made by hand all look
|
||||
identical from here, and placement only needs to know whether the file
|
||||
@@ -426,7 +438,8 @@ is present.
|
||||
|
||||
WeightsCache is where a remote source is downloaded to. Defaults to the
|
||||
first entry of ModelDirs, because a node that says where its models live
|
||||
has already answered this question.
|
||||
has already answered this question, and otherwise to
|
||||
/var/lib/llmgw-weights.
|
||||
|
||||
Fetching is the simple arrangement: every node keeps its own copy. The
|
||||
exotic ones need no code — weights on a shared filer or an image exported
|
||||
@@ -576,6 +589,8 @@ know that this page cannot guess.
|
||||
|
||||
Bin is the llama-server binary. It is the server, not the CLI: the
|
||||
gateway talks OpenAI over HTTP and needs the one that serves it.
|
||||
Empty looks on PATH, then where release archives are unpacked
|
||||
(/opt/llama.cpp).
|
||||
|
||||
### `parallel`
|
||||
|
||||
@@ -761,7 +776,8 @@ source of NVLink topology.
|
||||
|
||||
`bool`
|
||||
|
||||
SkipGPUs omits GPU discovery, for nodes that have none.
|
||||
SkipGPUs omits GPU discovery. A node without nvidia-smi is already
|
||||
found to have none, so this is for one whose driver is broken.
|
||||
|
||||
## Prefix
|
||||
|
||||
|
||||
Reference in new issue
Block a user