Commit Graph
6 Commits
Author SHA1 Message Date
Ilker ManapandClaude Opus 5 753f4e3265 Let a node say how much room it has left
CI / test (push) Failing after 1m31s
Disk was the one resource nothing reported. `statfs` was already being called
on every model directory, but only to ask whether the filesystem was a network
one — the free space it returns in the same struct was thrown away. So the
only way to find out why a node was filling up was to ssh to it, which is what
this system exists to avoid.

Reported per filesystem rather than per configured directory. Free space
belongs to the disk, not the path: two model directories on one disk are one
pool of bytes, and counting them separately would let the control plane add
24 GB to 24 GB and place a 40 GB model on a node that has 24.

Shared filesystems are summed separately from local ones and left out of a
node's own total. Space on a filer belongs to everyone mounting it, so twenty
nodes would otherwise each report the same free terabyte, and a scheduler
would place twenty models into room for one.

Nothing reported means nothing is known, everywhere: an agent too old to send
storage and a node with no model directories both send nothing, and both must
read as unknown rather than as a full disk. `ctl nodes` and the dashboard show
a dash, the metric is absent rather than zero, and a missing directory — which
is what an unmounted filer looks like — is skipped rather than reported as
having no space.

Shown as free out of total, because the first is useless without the second:
8 GB free is comfortable on a 10 TB filer and nearly full on a 10 GB root
partition, and only the second stops a model being placed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 09:38:57 +02:00
Ilker ManapandClaude Opus 5 c13c567cc5 Know where the weights are, without owning the storage
CI / test (push) Successful in 1m32s
Storage will be different at every installation — Lustre, NFS over RDMA, an
image exported with SRP or NVMe-oF, local NVMe, an rsync — so llmgw observes
it rather than managing it. An agent told which directories to look in
reports what it can see; how the files got there is nobody's business here,
and a node told nothing reports nothing and behaves exactly as every node
did before this existed.

That fact buys two things. Placement prefers a node that already holds the
model over one that must first move tens of gigabytes, as a preference and
never a filter: fetching them is a valid answer, and refusing to place would
turn a slow start into no start at all. And ctl status shows what each
fabric holds.

Per fabric because fabrics are not connected to one another. Weights in
ib-a are not weights in ib-b — the only path between them is the management
network — so a model seeded into one fabric leaves the rest of the cluster
unable to serve it without a slow copy, and nothing said so before now. One
shared filer counts the same as twenty local copies, because "can this
fabric serve it" is the question being asked.

Three distinctions the tests hold to, because each is a way to get this
quietly wrong. A node that was never asked to look must rank with one that
looked and has nothing, or enabling the feature on a single node would
drain every other. An empty directory is not weights, because a
half-finished download looks exactly like one and would send placement to a
node whose engine then fails to load. And local storage is never mistaken
for shared: a filer read as local costs an unnecessary copy, while local
read as shared would have the cluster believe a model is everywhere when it
is on one node.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 08:21:16 +02:00
Ilker ManapandClaude Opus 5 d71d785598 Report node memory even when GPU discovery fails
CI / test (push) Successful in 1m31s
Deploying the memory-aware placement showed node01 to node04 reporting 7G
and node05 reporting nothing, on a machine whose /proc/meminfo plainly says
32 GB. node05 has no nvidia-smi, so GPU discovery returns an error, Discover
returns with it, and the memory read — which has nothing to do with cards —
never happened.

That is the node whose memory matters most, because a CPU profile is what a
GPU-less node will be asked to run, and min_memory_gb would have excluded it
from everything.

Read first now, before any path that can return early. The test fails
against the old order.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 23:50:53 +02:00
Ilker ManapandClaude Opus 5 3081c6f69b Place on nodes that can actually hold the model
CI / test (push) Successful in 1m23s
Placement had no idea how much memory a node has or a model needs. A CPU
profile has no card to be free, so eligibleNodes accepted every ready node:
Qwen2.5-Coder 14B, whose working set is about 24 GB, would have been placed
just as happily on one of the 8 GB nodes, which reports itself perfectly
healthy right up until the OOM killer arrives.

node_selector was yesterday's stopgap and it is the operator restating by
hand something the system can know. Agents now report MemTotal, profiles
declare min_memory_gb, and placement filters on it, naming what was short —
"has 8 GB of memory, needs 24" is actionable where "no candidates were
considered" is not. The label selector stays, because "this node is for
coding models" is a real thing to want to say that memory does not express.

Two deliberate non-exclusions. A profile with no requirement fits anywhere,
which is every profile that exists today. And a node that reports no memory
is still usable: an agent older than the field says nothing, and letting a
control plane upgrade quietly empty half a fleet is a worse failure than the
one this prevents.

ctl nodes grows a MEM column showing "-" rather than 0 for a node that did
not say, because an old agent is not a small machine.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 23:12:44 +02:00
ilkerandClaude Opus 5 2373726efd Agent stream: connect the fleet to the control plane
CI / test (push) Failing after 1m29s
One bidirectional connect-go stream per node, carrying registration, heartbeats
and shard status up, and start, stop and drain down. The agent dials out: GPU
nodes sit behind firewalls far more often than a control plane does, and an
outbound connection needs no inbound rule on every node in the fleet.

- registration carries the whole hardware inventory, so there is no separate
  discovery call and a reconnecting agent refreshes what the scheduler knows.
  The fabric labels travel with it; they cannot be inferred from the wire, so
  losing them in transit would silently disable multi-node placement.
- agentsvc is both the stream handler and the reconciler's dispatcher, because
  it holds the open streams and is therefore the only thing that can deliver a
  command to a node.
- the agent validates a profile before launching anything, since an engine that
  refuses a profile does so after minutes of loading, then supervises the
  process and reports every transition with the engine's output tail on failure.
- reconnects forever with jittered backoff. A control plane restart must not
  require touching every GPU node, and a fleet must not reconnect in lockstep
  and knock it over again. Engines keep serving throughout, because the router
  talks to them directly.

llmgw agent is real now, and llmgw server runs a control plane whenever any
model declares a profile. The static endpoint list stays for router-only runs.

Verified with real binaries as separate processes: the agent registers, a
replica is placed, an engine launches through the vLLM adapter's actual argument
construction, streaming traffic is served, and killing the agent moves the node
through unreachable to lost with placement explaining why nothing can run.

config.Model and catalog.Model were the same thing described twice, so the
router now reads catalog entries directly and the validation rules live in one
place.

Still missing: mTLS. The stream is plaintext HTTP/2 and must stay on a trusted
network until that lands.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 02:48:21 +02:00
ilkerandClaude Opus 5 55417546fc Phase 2 (2/2): NVML backend behind a build tag
CI / test (push) Successful in 1m37s
NVML is the only source of NVLink topology, and that topology is what keeps a
tensor-parallel group off the slow path inside a node. It needs cgo, so it sits
behind the nvml tag and the default build keeps using nvidia-smi.

Discover() picks the backend and falls back to nvidia-smi when NVML fails,
logging the fallback rather than performing it silently: on a node that is meant
to have working NVML, a quiet downgrade hides a broken driver installation and
costs the scheduler its topology.

CI now compiles the tagged build. That immediately paid for itself: ErrNoNVML
was declared only in the untagged file, so the tests did not compile under
-tags nvml. It is now declared where both builds can see it.

Not verified: the NVML path has never run against a driver, because there is no
GPU machine available. The NVLink domains it reports should be treated as
unverified until someone runs `llmgw agent inventory` on a real node.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 02:05:25 +02:00