Every agent file on the cluster named its own node, its own address, the
path to llama-server and two weight directories — different on every node,
kept true by hand, and not true: node05 carried "ram: 32g" beside a measured
31 GB, looked for weights in an empty directory while fetching them into
another, and the other four never looked in theirs at all, so four nodes
reported no disk. Growing a node's RAM would have gone unnoticed, because
memory was read once, at registration.
The agent now finds these out. Its id is the hostname, which is also the
certificate's common name. Its address is the one on the route to the
control plane, which the kernel picks for a UDP dial without sending
anything. llama-server is looked for on PATH and then where release archives
are unpacked. A missing nvidia-smi is a node without NVIDIA cards rather
than a failed discovery — which also cut discovery short before InfiniBand
on every CPU node not told to skip GPUs. Fetched weights land in
/var/lib/llmgw-weights by default, and wherever they land is always looked
in.
Memory is read on every heartbeat, and installed engines are probed again
each minute and sent when they change, so a VM given more RAM or a node
given vLLM becomes a candidate without anyone restarting its agent. The
control plane logs both changes, since "the node now qualifies" is the
answer to why a model placed an hour after it was asked for.
What is written still wins. The agent file shrinks to the control plane's
addresses and the TLS files, the same on every node; onboarding writes only
the overrides it was given. What cannot be found stays configuration: which
fabric an adapter is on, because OpenSM's default prefix is the same on all
of them.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
New nodes had been joined by hand, four times, by me. There was no procedure,
so there was nothing to be wrong — and nothing to be right either. Two scripts
now: provision-node.sh makes the VM on Proxmox, onboard-node.sh gives it an
identity. They are separate because they are run by different people with
different access, and the second must run where the certificate authority is,
which is the one part that cannot be delegated.
Running them turned up why phase 4 was never verified. FabricAttachment
carried no koanf tags, so the `name:` key the shipped example documents bound
to nothing and `rate_gbps` was never read at all. An agent configured exactly
as instructed refused to start with "fabrics[0]: name is required" — naming
the key the operator had just written. Every fabric in this project had been
configured in a Go test, where the tags do not matter.
The server example did not load either: its models had no source, so it failed
validation the moment anything tried to parse it.
Both examples are now parsed by a test. A published example that has never
been loaded is a guess, and these two had been guesses since they were
written.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
