ff86e52126e951269b1ae9e8c4d1e508bbaa382f
13
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Ilker ManapandClaude Opus 5.5
|
33e678ab84 |
Show a node's logs on the admin page
Seeing what a node did meant ssh through a jump host, to a machine whose address nobody remembers, to read a journal only root may open. When an engine install failed on a new node today, the reason was one line — and it was four steps away from the page that said the install had failed. Each node's row on the cluster page now links to its logs: the agent's own, and every engine running there, chosen from the list the node itself sends. The page can follow, refreshing every five seconds, and is on both the admin listener and /admin. `llmgw ctl logs NODE [--source SHARD] [-n N]` reads the same. They travel over the stream the agent already holds, so they need nothing the control plane does not already have: a request goes down, the lines come back, and a reply that does not come in ten seconds is an agent too old to answer. The agent keeps its last 2000 lines in memory as well as writing them wherever it did before; the engines' last few hundred lines were already kept and sent only when one crashed. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5.5
|
7d97c6b74f |
Install a node's engines on the node, instead of trusting what is there
A node ran whatever llama-server somebody had once unpacked on it. Onboarding checked an engine only when told its path, installed none, and a node with nothing on it registered as ready. The one time that went wrong it went wrong invisibly: the binary was there, executable, and could not start for want of libgomp1, while the node reported healthy. The control plane now says which engines every node has, at which version — `engines:` in server.yaml, defaulting to the llama.cpp build this release was checked against — and each agent converges on it. What only the node can know, the node works out: the build for its processor and driver (CUDA with its runtime when an NVIDIA driver is loaded, CPU otherwise), the OS packages that build needs from its own package sources, and whether the result runs. Every archive is checked against the sha256 GitHub publishes for it and unpacked refusing any path or link that leaves the install directory. ldd names a missing library outright; running `--version` proves the rest. Only then is a directory marked installed, and it is checked again every ten minutes, because an unrelated upgrade can remove a library. An engine is reported only once installed and seen to run, or when the agent's configuration names it on purpose — never because something happened to be on PATH. The agent now says when it has probed, so an empty list means "has none" rather than "too old to say": a node rebuilt from nothing is no longer a candidate for every model at the moment it can run none of them. Placement's refusal carries the node's own words, "installing: downloading llama-b10991-bin-ubuntu-x64.tar.gz (16 MB)", and so do `ctl nodes` and the cluster page, along with a warning for a CPU without AVX2 — the VM CPU type that makes everything five times slower and nothing look wrong. Agents also report their OS, architecture, vector extensions and NVIDIA driver. Onboarding installs llmgw itself from the CI release for the node's architecture, checked against the release's checksums, rather than copying whatever binary the operator's machine had. vLLM and SGLang are not installed yet: they are Python environments of several gigabytes, not an archive, and still have to be named in the agent's configuration. Also fixes a race on the weights fingerprint, which two heartbeat loops touch during a failover. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5.5
|
20deb22dfa |
Let a node describe itself instead of being described by hand
CI / test (push) Successful in 1m47s
Every agent file on the cluster named its own node, its own address, the path to llama-server and two weight directories — different on every node, kept true by hand, and not true: node05 carried "ram: 32g" beside a measured 31 GB, looked for weights in an empty directory while fetching them into another, and the other four never looked in theirs at all, so four nodes reported no disk. Growing a node's RAM would have gone unnoticed, because memory was read once, at registration. The agent now finds these out. Its id is the hostname, which is also the certificate's common name. Its address is the one on the route to the control plane, which the kernel picks for a UDP dial without sending anything. llama-server is looked for on PATH and then where release archives are unpacked. A missing nvidia-smi is a node without NVIDIA cards rather than a failed discovery — which also cut discovery short before InfiniBand on every CPU node not told to skip GPUs. Fetched weights land in /var/lib/llmgw-weights by default, and wherever they land is always looked in. Memory is read on every heartbeat, and installed engines are probed again each minute and sent when they change, so a VM given more RAM or a node given vLLM becomes a candidate without anyone restarting its agent. The control plane logs both changes, since "the node now qualifies" is the answer to why a model placed an hour after it was asked for. What is written still wins. The agent file shrinks to the control plane's addresses and the TLS files, the same on every node; onboarding writes only the overrides it was given. What cannot be found stays configuration: which fabric an adapter is on, because OpenSM's default prefix is the same on all of them. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
03e38e7dda |
Carry the profile fingerprint to the agent and back
CI / test (push) Successful in 1m39s
The stale-profile flag could not fire where it matters. The control plane keeps no record of a running replica across its own restart — it rebuilds them from what the agents report — and adoption left the fingerprint empty, so every replica read as unknown afterwards. That is precisely the moment the catalog is most likely to have been edited: change the configuration, restart the server, and nothing would ever say the fleet disagrees with it. The fingerprint now travels with StartShard, the agent keeps it beside the pool and fabric it already echoes back, and adoption takes the replica's version from the agent rather than from the catalog it is about to be compared against. An agent too old to report it sends nothing, which still reads as unknown rather than as matching. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
a35e8796d2 |
Carry a download's progress to where somebody is looking at it
CI / test (push) Successful in 1m28s
The agent has always logged a percentage as it fetches weights, with a comment saying a replica pulling a hundred gigabytes and one that is wedged look identical from the control plane. They still did: the log is on the node, and the node is not where anyone is looking when a replica has been starting for ten minutes. Progress now rides on the shard status and appears under the replica in `ctl replicas` — a line rather than a column, because most replicas are not downloading anything and a column would be empty for all of them. It is deliberately not a state transition. Treating each chunk as one would re-mark the shard as starting on every report and reset the timer that decides when a slow start has become a failed one, so an hour-long download would keep a doomed replica alive indefinitely. It arrives on its own method for that reason, and a real transition clears it — otherwise a finished download sits in the listing at ninety-nine percent under a shard that has been serving for an hour. Watchers are keyed by source rather than by shard, because a fetch is single-flight: eight shards starting together produce one download and all eight are waiting on it. No percentage is invented when the server sent no length. One that stops moving is worse than a byte count that keeps rising, so the field is absent and the listing says "so far". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
45e913bd10 |
Refresh disk and weights while the agent runs, not once when it starts
CI / test (push) Successful in 1m46s
Found by answering "where can we see this live". `ctl weights` was offering a file for deletion that I had deleted two days earlier, and the node's disk figure was the one it had when its agent started. Discover() is called at registration and nowhere else, so both were frozen. Disk is a fact about a moment, not about a machine. Sampled once, the figure is wrong on exactly the node worth watching — the one busy filling up — and it stays wrong until that agent restarts. The heartbeat already refreshes GPU memory for the same reason, with a comment about noticing a foreign process "after registration"; disk belonged beside it and I did not put it there. The stale list cut both ways. A model fetched after registration was invisible until a restart, so placement could not prefer the node that had just spent ten minutes downloading it, and the disk check could not see the space it used. Storage goes in every heartbeat, being a few numbers per filesystem. The file list goes only when a fingerprint of paths and sizes changes, with a flag saying the list is meant — without that, an empty list would be ambiguous between "this node now holds nothing" and "nothing to report", and the second reading would empty a node's inventory five seconds after it registered. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
774e134ed1 |
Refuse a node with no room, and take back only what llmgw put there
CI / test (push) Successful in 1m39s
Two halves of the same problem: nothing stopped a model being placed on a node that could not hold it, and nothing ever removed the weights of a model that had been deleted from the catalog. Placement now has a disk check, shaped exactly like the memory one it sits beside, with one addition: it is skipped on a node that already holds the weights. A node holding the file needs no room for it, and without that the node best placed to run a model — because it already has it — is the one most likely to be rejected, since holding the file is what used the space. Reclaiming is the dangerous half, so the whole design is about what it will not do. llmgw deletes only files it fetched itself, and that is a calculation rather than a heuristic: the cache name is a hash of the source, so the set of wanted files can be computed from the catalog exactly. Anything without that name was put there by an operator — scp, a shared mount, a copy by hand — and is none of this system's business however unreferenced it looks. A model that took a week to copy must not be removable by a command someone ran to tidy up. The node checks every path again before unlinking it, and that duplication is the point. The control plane knows the catalog; the node knows its disk. A control plane that is wrong, newer than the agent, or tampered with must not be able to talk a node into deleting something, so the agent refuses anything outside its own cache directory or without the cache name — including a path that tries to traverse out of it. Listing is the default and deleting takes --collect. The summary says "asked to remove", because each node decides, and one that refuses a path it cannot prove it fetched is this working rather than failing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
753f4e3265 |
Let a node say how much room it has left
CI / test (push) Failing after 1m31s
Disk was the one resource nothing reported. `statfs` was already being called on every model directory, but only to ask whether the filesystem was a network one — the free space it returns in the same struct was thrown away. So the only way to find out why a node was filling up was to ssh to it, which is what this system exists to avoid. Reported per filesystem rather than per configured directory. Free space belongs to the disk, not the path: two model directories on one disk are one pool of bytes, and counting them separately would let the control plane add 24 GB to 24 GB and place a 40 GB model on a node that has 24. Shared filesystems are summed separately from local ones and left out of a node's own total. Space on a filer belongs to everyone mounting it, so twenty nodes would otherwise each report the same free terabyte, and a scheduler would place twenty models into room for one. Nothing reported means nothing is known, everywhere: an agent too old to send storage and a node with no model directories both send nothing, and both must read as unknown rather than as a full disk. `ctl nodes` and the dashboard show a dash, the metric is absent rather than zero, and a missing directory — which is what an unmounted filer looks like — is skipped rather than reported as having no space. Shown as free out of total, because the first is useless without the second: 8 GB free is comfortable on a 10 TB filer and nearly full on a 10 GB root partition, and only the second stops a model being placed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
42fd0aa31e |
Export what the cluster is made of, not only what it is serving
CI / test (push) Successful in 1m28s
The router published twenty-two metric families about requests, endpoints, admission and scoring — everything about traffic and nothing about the machines carrying it. There was no way to graph GPU usage, watch a node go unhealthy, see a fabric fill up, or answer "was the cluster full at the time", which is the first question anyone asks after an incident. The agents exported nothing at all. Exported by the control plane rather than by each agent. It already holds every number: hardware arrives at registration and state on each heartbeat, so this is a map read away. One scrape target for a fleet of hundreds, no Prometheus configuration per node, no firewall rule per machine, and nothing new running on the nodes. On the same /metrics as the traffic figures, because correlating "the cluster was full" with "requests were queueing" should not need two systems. A collector rather than gauges kept current by a ticker: the numbers are derived from state that already exists, so computing them when asked is simpler than maintaining a copy that can drift, and nothing runs between scrapes. GPU utilisation is new, and is the number people actually watch. Memory used says a model is loaded; this says whether it is doing anything, and a fleet at ninety percent occupancy and two percent utilisation is a fleet holding models nobody is asking for. Adding it caught two things. utilization.gpu goes last in the nvidia-smi query and is optional, because a driver too old to know it returns a shorter row — requiring it made an llmgw upgrade silently stop discovering GPUs on an older node, which is far worse than not knowing how busy a card is. And I gave the new protobuf field number 14, which ecc_errors already had; buf lint is why that was a moment rather than a corrupted wire format. Absent is kept distinct from zero throughout, for the same reason as everywhere else here: an unreported utilisation shown as a flat zero reads as an idle fleet rather than a missing measurement, and a replica nothing has reported would be the most idle thing in the cluster. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
c13c567cc5 |
Know where the weights are, without owning the storage
CI / test (push) Successful in 1m32s
Storage will be different at every installation — Lustre, NFS over RDMA, an image exported with SRP or NVMe-oF, local NVMe, an rsync — so llmgw observes it rather than managing it. An agent told which directories to look in reports what it can see; how the files got there is nobody's business here, and a node told nothing reports nothing and behaves exactly as every node did before this existed. That fact buys two things. Placement prefers a node that already holds the model over one that must first move tens of gigabytes, as a preference and never a filter: fetching them is a valid answer, and refusing to place would turn a slow start into no start at all. And ctl status shows what each fabric holds. Per fabric because fabrics are not connected to one another. Weights in ib-a are not weights in ib-b — the only path between them is the management network — so a model seeded into one fabric leaves the rest of the cluster unable to serve it without a slow copy, and nothing said so before now. One shared filer counts the same as twenty local copies, because "can this fabric serve it" is the question being asked. Three distinctions the tests hold to, because each is a way to get this quietly wrong. A node that was never asked to look must rank with one that looked and has nothing, or enabling the feature on a single node would drain every other. An empty directory is not weights, because a half-finished download looks exactly like one and would send placement to a node whose engine then fails to load. And local storage is never mistaken for shared: a filer read as local costs an unnecessary copy, while local read as shared would have the cluster believe a model is everywhere when it is on one node. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
3081c6f69b |
Place on nodes that can actually hold the model
CI / test (push) Successful in 1m23s
Placement had no idea how much memory a node has or a model needs. A CPU profile has no card to be free, so eligibleNodes accepted every ready node: Qwen2.5-Coder 14B, whose working set is about 24 GB, would have been placed just as happily on one of the 8 GB nodes, which reports itself perfectly healthy right up until the OOM killer arrives. node_selector was yesterday's stopgap and it is the operator restating by hand something the system can know. Agents now report MemTotal, profiles declare min_memory_gb, and placement filters on it, naming what was short — "has 8 GB of memory, needs 24" is actionable where "no candidates were considered" is not. The label selector stays, because "this node is for coding models" is a real thing to want to say that memory does not express. Two deliberate non-exclusions. A profile with no requirement fits anywhere, which is every profile that exists today. And a node that reports no memory is still usable: an agent older than the field says nothing, and letting a control plane upgrade quietly empty half a fleet is a worse failure than the one this prevents. ctl nodes grows a MEM column showing "-" rather than 0 for a node that did not say, because an old agent is not a small machine. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
 ilkerandClaude Opus 5
|
75a8e76706 |
Survive a control plane restart without orphaning live engines
CI / test (push) Successful in 1m44s
Not persisting replica state left one hole: restarting the control plane made it forget every replica while the engines kept running. They held GPUs the inventory believed were free, so the next placement failed to allocate, with an error naming the new process rather than the old one. The fix belongs on the agent, not in a database. A row can say "running" about a process that died; the fleet cannot. - the supervisor writes the full shard spec, not just a pid and command line. A pid identifies a process; the spec is what lets it be described back to the control plane and matched to a replica. - Adopt takes over an engine a previous agent left running. It is not this process's child, so it cannot be waited on: it is watched by polling and stopped by signalling its group directly. Its earlier output is gone and the log says so rather than presenting emptiness as silence. - every Register reports what the agent is running, not only the first, because a reconnect may be a control plane restart and has to recover the same way. - the reconciler rebuilds replicas from those reports and reclaims their GPUs. Anything whose model or profile left the catalog is stopped; nothing will route to it and it is holding cards. - an engine the agent cannot identify is killed. A process holding GPUs that nothing can describe is worse than no process, because placement hands those GPUs out again. Stop was also wrong for adopted shards: it checked for an exec.Cmd, which they do not have, and reported them stopped without killing anything. TestAgentReconnects had been passing for no reason. httptest's CloseClientConnections does nothing to an h2c stream — prior-knowledge h2c hijacks the connection and httptest stops tracking hijacked ones — so the agent never reconnected, and the assertion on the connection map passed on a stale entry. Tests now sever connections through their own listener and count opened streams. Plan §11 also records what multi-node still needs, since placement landing early makes it look further along than it is: workers never become Ready because the agent health-checks every shard while only rank 0 serves HTTP; RayCommand is written and tested but never called; and startup is fire-and-forget. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
ilkerandClaude Opus 5
|
2373726efd |
Agent stream: connect the fleet to the control plane
CI / test (push) Failing after 1m29s
One bidirectional connect-go stream per node, carrying registration, heartbeats and shard status up, and start, stop and drain down. The agent dials out: GPU nodes sit behind firewalls far more often than a control plane does, and an outbound connection needs no inbound rule on every node in the fleet. - registration carries the whole hardware inventory, so there is no separate discovery call and a reconnecting agent refreshes what the scheduler knows. The fabric labels travel with it; they cannot be inferred from the wire, so losing them in transit would silently disable multi-node placement. - agentsvc is both the stream handler and the reconciler's dispatcher, because it holds the open streams and is therefore the only thing that can deliver a command to a node. - the agent validates a profile before launching anything, since an engine that refuses a profile does so after minutes of loading, then supervises the process and reports every transition with the engine's output tail on failure. - reconnects forever with jittered backoff. A control plane restart must not require touching every GPU node, and a fleet must not reconnect in lockstep and knock it over again. Engines keep serving throughout, because the router talks to them directly. llmgw agent is real now, and llmgw server runs a control plane whenever any model declares a profile. The static endpoint list stays for router-only runs. Verified with real binaries as separate processes: the agent registers, a replica is placed, an engine launches through the vLLM adapter's actual argument construction, streaming traffic is served, and killing the agent moves the node through unreachable to lost with placement explaining why nothing can run. config.Model and catalog.Model were the same thing described twice, so the router now reads catalog entries directly and the validation rules live in one place. Still missing: mTLS. The stream is plaintext HTTP/2 and must stay on a trusted network until that lands. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |