bf2d171e8941bbff7cbbfba078a5ceb18c388f9e
39
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Ilker ManapandClaude Opus 5.5
|
bf2d171e89 |
Read NVLink and GPU-to-adapter distance from nvidia-smi topo
The release binary is built without cgo, so without NVML, and NVML was the only source of NVLink peers: on every NVLink machine the portable agent reported none, and tensor-parallel groups were chosen by GPU index. Discovery now reads `nvidia-smi topo -m` after either path, for the NVLink peers (kept from NVML where it gave them), each card's NUMA node, and how far each network adapter is from it - PIX, PXB, PHB, NODE or SYS - which travels to the control plane as nic_paths for placement to use. Both the NICn-with-legend layout of newer drivers and the adapter-named columns of older ones are read. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5.5
|
daed835d88 |
Measure each node's CPU and memory, and give every node a page
The nodes are CPU machines, and nothing measured how busy one was: the heartbeat carried GPU memory and disk, and llmgw_node_memory_bytes is the total. The agent now reads /proc between heartbeats - CPU across all cores, load averages, memory used and available, and each engine's process group's cores and resident memory - and sends it with the heartbeat. Nothing new runs on the nodes and Prometheus needs no new target: the control plane exports it with the rest of the fleet, as llmgw_node_cpu_utilization_percent, llmgw_node_cpus, llmgw_node_load, llmgw_node_memory_used_bytes, llmgw_node_memory_available_bytes, llmgw_shard_cpu_cores and llmgw_shard_memory_bytes. A measurement more than a minute old is left out rather than repeated. The cluster page's node table shows CPU and RAM as bars, and each node links to a page of its own: current CPU, RAM, load and disk, the engines on it with their cores and memory, and an hour of CPU, RAM, load, per-model cores and memory and queue depth from Prometheus. It refreshes in place. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5.5
|
cba65f5be7 |
Move a control plane to PostgreSQL with server init --postgres
Installs PostgreSQL with the machine's own package manager when it has none (initdb with password auth on the RHEL family, whose default is ident), creates the llmgw role and database with a generated password kept in /etc/llmgw/secrets/store-dsn, and checks the address connects. On a machine already on SQLite it stops the gateway, copies the store so accounts and keys survive, points auth.store at the new address and starts it again. A move that fails starts the gateway on SQLite again. server.yaml is edited as text so its comments survive, and kept beside it. A PostgreSQL init did not install is not resized: it may serve something else. It refuses to move one PostgreSQL store into another. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5.5
|
33e678ab84 |
Show a node's logs on the admin page
Seeing what a node did meant ssh through a jump host, to a machine whose address nobody remembers, to read a journal only root may open. When an engine install failed on a new node today, the reason was one line — and it was four steps away from the page that said the install had failed. Each node's row on the cluster page now links to its logs: the agent's own, and every engine running there, chosen from the list the node itself sends. The page can follow, refreshing every five seconds, and is on both the admin listener and /admin. `llmgw ctl logs NODE [--source SHARD] [-n N]` reads the same. They travel over the stream the agent already holds, so they need nothing the control plane does not already have: a request goes down, the lines come back, and a reply that does not come in ten seconds is an agent too old to answer. The agent keeps its last 2000 lines in memory as well as writing them wherever it did before; the engines' last few hundred lines were already kept and sent only when one crashed. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5.5
|
a40fc53e09 |
Wait for a new machine's own apt, and retry a failed install sooner
Found by building a node from nothing. Its agent tried to install libgomp1 while cloud-init and unattended-upgrades were running apt, and failed with "Unable to lock directory /var/lib/apt/lists/": DPkg::Lock::Timeout covers dpkg's lock, not the one `apt-get update` takes on its lists. The node was correctly kept out of placement and said why, and then sat idle for the ten minutes until the next pass. A package command that fails only on a held lock is now retried for up to ten minutes, reported as "waiting for another apt-get to finish" rather than as a failure. A pass that does fail is retried after a minute, backing off to ten. The weights directory is created when the agent starts rather than by the first download, so a new node reports its disk from its first heartbeat. Onboarding takes several control plane addresses, comma separated, so a new node gets the same failover list as the rest. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5.5
|
f5d6210b86 |
Install engines on any glibc system, with its own package manager
The llama.cpp build is published from Ubuntu but is not Ubuntu's in any way that matters: it needs glibc 2.34 and GCC 12's libstdc++, which Debian 12 and RHEL 10 have and RHEL 9 does not. Refusing every system not called "ubuntu" turned away the first two for their name. The build is now tried everywhere, and ldd decides. What ldd says is made readable. "/lib64/libstdc++.so.6: version `GLIBCXX_3.4.30' not found" is a system too old for the build, and is reported as "libstdc++.so.6 is too old (needs GLIBCXX_3.4.30)", once — rather than as a missing library that is plainly installed. Packages are named per family — libgomp1 for Debian's, libgomp for Red Hat's — and installed with whichever of apt-get, dnf, yum or zypper the machine has, decided from ID_LIKE rather than a list of distribution names. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5.5
|
7d97c6b74f |
Install a node's engines on the node, instead of trusting what is there
A node ran whatever llama-server somebody had once unpacked on it. Onboarding checked an engine only when told its path, installed none, and a node with nothing on it registered as ready. The one time that went wrong it went wrong invisibly: the binary was there, executable, and could not start for want of libgomp1, while the node reported healthy. The control plane now says which engines every node has, at which version — `engines:` in server.yaml, defaulting to the llama.cpp build this release was checked against — and each agent converges on it. What only the node can know, the node works out: the build for its processor and driver (CUDA with its runtime when an NVIDIA driver is loaded, CPU otherwise), the OS packages that build needs from its own package sources, and whether the result runs. Every archive is checked against the sha256 GitHub publishes for it and unpacked refusing any path or link that leaves the install directory. ldd names a missing library outright; running `--version` proves the rest. Only then is a directory marked installed, and it is checked again every ten minutes, because an unrelated upgrade can remove a library. An engine is reported only once installed and seen to run, or when the agent's configuration names it on purpose — never because something happened to be on PATH. The agent now says when it has probed, so an empty list means "has none" rather than "too old to say": a node rebuilt from nothing is no longer a candidate for every model at the moment it can run none of them. Placement's refusal carries the node's own words, "installing: downloading llama-b10991-bin-ubuntu-x64.tar.gz (16 MB)", and so do `ctl nodes` and the cluster page, along with a warning for a CPU without AVX2 — the VM CPU type that makes everything five times slower and nothing look wrong. Agents also report their OS, architecture, vector extensions and NVIDIA driver. Onboarding installs llmgw itself from the CI release for the node's architecture, checked against the release's checksums, rather than copying whatever binary the operator's machine had. vLLM and SGLang are not installed yet: they are Python environments of several gigabytes, not an archive, and still have to be named in the agent's configuration. Also fixes a race on the weights fingerprint, which two heartbeat loops touch during a failover. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5.5
|
20deb22dfa |
Let a node describe itself instead of being described by hand
CI / test (push) Successful in 1m47s
Every agent file on the cluster named its own node, its own address, the path to llama-server and two weight directories — different on every node, kept true by hand, and not true: node05 carried "ram: 32g" beside a measured 31 GB, looked for weights in an empty directory while fetching them into another, and the other four never looked in theirs at all, so four nodes reported no disk. Growing a node's RAM would have gone unnoticed, because memory was read once, at registration. The agent now finds these out. Its id is the hostname, which is also the certificate's common name. Its address is the one on the route to the control plane, which the kernel picks for a UDP dial without sending anything. llama-server is looked for on PATH and then where release archives are unpacked. A missing nvidia-smi is a node without NVIDIA cards rather than a failed discovery — which also cut discovery short before InfiniBand on every CPU node not told to skip GPUs. Fetched weights land in /var/lib/llmgw-weights by default, and wherever they land is always looked in. Memory is read on every heartbeat, and installed engines are probed again each minute and sent when they change, so a VM given more RAM or a node given vLLM becomes a candidate without anyone restarting its agent. The control plane logs both changes, since "the node now qualifies" is the answer to why a model placed an hour after it was asked for. What is written still wins. The agent file shrinks to the control plane's addresses and the TLS files, the same on every node; onboarding writes only the overrides it was given. What cannot be found stays configuration: which fabric an adapter is on, because OpenSM's default prefix is the same on all of them. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
03e38e7dda |
Carry the profile fingerprint to the agent and back
CI / test (push) Successful in 1m39s
The stale-profile flag could not fire where it matters. The control plane keeps no record of a running replica across its own restart — it rebuilds them from what the agents report — and adoption left the fingerprint empty, so every replica read as unknown afterwards. That is precisely the moment the catalog is most likely to have been edited: change the configuration, restart the server, and nothing would ever say the fleet disagrees with it. The fingerprint now travels with StartShard, the agent keeps it beside the pool and fabric it already echoes back, and adoption takes the replica's version from the agent rather than from the catalog it is about to be compared against. An agent too old to report it sends nothing, which still reads as unknown rather than as matching. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
dfbb7557dd |
Serve a model made of several files
CI / test (push) Successful in 1m42s
The mirror would happily hold a four-shard model that nothing could serve. A catalog entry names one source, the agent fetched one URL, and so a version of several files had no way to be described — which is every safetensors model, and every GGUF too large to ship whole. A source ending in a slash names a version rather than a file. The agent fetches every file in it into one directory and hands the engine the directory: vLLM reads the shards and the config out of the folder, and llama.cpp is given the first part and finds the rest beside it. The trailing slash is the whole signal. A flag beside the source would be a second place for the two to disagree, and a URL already distinguishes a directory from a file everywhere else. The mirror answers a version with a manifest instead of a file listing. A plain file server gives HTML there, and something that has to know what a version consists of would be left parsing it. It is on the byte listener, not the management API: fetching weights is the one thing every node does, and making it depend on the administrative port would undo the split that keeps that port closeable. Built from the directory rather than a file written beside it. A manifest on disk is a second record of what is there, and the two disagree the first time somebody copies a file in by hand — which the mirror explicitly allows. The whole set or nothing. A directory with three of four shards loads as a broken model rather than failing to load, so a completion marker is written last and an interrupted fetch starts the missing files again rather than being found and served. Files already there at the right size are not fetched twice. Progress is reported under the version's address, not each file's: the shards waiting on it asked for the version, and progress under a file's own URL reaches nobody. A filename from the manifest is refused if it contains a separator. The mirror is a place an operator copies files into by hand, so what it reports is not automatically safe to join onto a path. An HTML listing where a manifest was expected says so. "invalid character '<'" sends somebody looking at the wrong thing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
2acd794022 |
Stop placing engines on nodes that do not have them
CI / test (push) Successful in 1m42s
A node's registry holds an adapter for every engine whether or not its binary exists, the agent never said which were installed, and placement carried an Engine field it had never once read. So a vLLM replica was as happily placed on a node with no vLLM: the node registered healthy, the shard failed minutes later with "executable file not found" in the engine's own words, and the reconciler tried again. That is how fifteen gigabytes of safetensors reached this cluster's mirror before anybody noticed it had no card and no vLLM to run them with. Agents now probe what is installed and report it — the protocol has carried engine_versions since it was written and nothing ever filled it in. Placement filters on the key set, `ctl nodes` has a column, and the mirror page's serve warning is a fact instead of a guess about cards, which would have waved through a fleet of eight-GPU nodes that never had vLLM. Probing resolves binaries and runs none of them. The first version asked each engine for --version, which is fine for an engine and not fine for the wrapper scripts people put in front of one: a script that shifts past flags it does not recognise and then execs the server will start an inference server, at agent startup, on every node. The agent's own test suite installs exactly such a script and caught it. So presence is a path lookup, and the version is left to `ctl engine-check`, which is run by somebody who meant to run the engine. SGLang is the exception and has to execute, because a module has no path to look up — but the command is llmgw's own one-line import rather than the engine's launcher. Its exit code is the answer: python3 is on almost every machine, and treating a failed import as "installed" reported SGLang on every node in a fleet that had none, which is the failure being fixed, inverted. Probed once, not per reconnect, and bounded by the context that registration waits on: an agent that cannot reach the control plane is worse than one reporting an incomplete list. A node that reports no engines at all is not excluded anywhere — by placement, or by the page. That is what every agent older than this reports, and emptying a fleet on a control plane upgrade is a worse failure than the one being fixed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
dde2bcab19 |
Stop asserting an ordering the supervisor does not promise
The test started a sidecar and an engine, each a shell appending its name to a file, and demanded the sidecar's line come first. The supervisor spawns the sidecar first, but spawning is not sequencing: without WaitForLeaderPort nothing holds the engine back, so two shells race and the scheduler picks the winner. It failed under full-suite load, and lately on its own. The wait that is a real guarantee already has its own test, so the fix is to check what this one is actually for — that both processes run under one shard, that the shard's pid is the engine's rather than the sidecar's, and that their output is distinguishable in a single log tail. Renamed to say so. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
a35e8796d2 |
Carry a download's progress to where somebody is looking at it
CI / test (push) Successful in 1m28s
The agent has always logged a percentage as it fetches weights, with a comment saying a replica pulling a hundred gigabytes and one that is wedged look identical from the control plane. They still did: the log is on the node, and the node is not where anyone is looking when a replica has been starting for ten minutes. Progress now rides on the shard status and appears under the replica in `ctl replicas` — a line rather than a column, because most replicas are not downloading anything and a column would be empty for all of them. It is deliberately not a state transition. Treating each chunk as one would re-mark the shard as starting on every report and reset the timer that decides when a slow start has become a failed one, so an hour-long download would keep a doomed replica alive indefinitely. It arrives on its own method for that reason, and a real transition clears it — otherwise a finished download sits in the listing at ninety-nine percent under a shard that has been serving for an hour. Watchers are keyed by source rather than by shard, because a fetch is single-flight: eight shards starting together produce one download and all eight are waiting on it. No percentage is invented when the server sent no length. One that stops moving is worse than a byte count that keeps rising, so the field is absent and the listing says "so far". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
4ba84fa90e |
Let an agent find the leader instead of being told one address
CI / test (push) Successful in 1m32s
The second half of leadership. A standby that refuses to place but still holds an agent's stream is worse than useless: commands travel down the stream the agent opened, so the leader's placements fail with "no agent connected" for a node that is plainly connected. So a standby refuses the stream, before reading the opening message, and says why. And an agent takes a list rather than an address, trying each in turn — a standby refusing, a stopped control plane refusing the connection and a crashed one never answering are the same thing from a node, and the answer to all three is somewhere else. `server` and `servers` are both accepted, with `server` first when both are given: every agent configuration written before there could be a second control plane names one address, and an operator adding a standby expects the one already working to stay preferred. With a single address the behaviour is exactly what it was, including the backoff — which is skipped only while there is another address that has not just been tried, because the delay exists to stop hammering one control plane and the next one has not been hammered. Removed a check that asked whether one field was empty, which is how the first run of these tests failed: the list was populated and `server` was not, and an agent configured entirely correctly was refused by its own validation. The tests run a real agent against a real service over h2c. The first attempt used a plain HTTP server, and the refusal arrived as an HTTP/2 framing error that said nothing about why — which is exactly what an operator would have seen. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
45e913bd10 |
Refresh disk and weights while the agent runs, not once when it starts
CI / test (push) Successful in 1m46s
Found by answering "where can we see this live". `ctl weights` was offering a file for deletion that I had deleted two days earlier, and the node's disk figure was the one it had when its agent started. Discover() is called at registration and nowhere else, so both were frozen. Disk is a fact about a moment, not about a machine. Sampled once, the figure is wrong on exactly the node worth watching — the one busy filling up — and it stays wrong until that agent restarts. The heartbeat already refreshes GPU memory for the same reason, with a comment about noticing a foreign process "after registration"; disk belonged beside it and I did not put it there. The stale list cut both ways. A model fetched after registration was invisible until a restart, so placement could not prefer the node that had just spent ten minutes downloading it, and the disk check could not see the space it used. Storage goes in every heartbeat, being a few numbers per filesystem. The file list goes only when a fingerprint of paths and sizes changes, with a flag saying the list is meant — without that, an empty list would be ambiguous between "this node now holds nothing" and "nothing to report", and the second reading would empty a node's inventory five seconds after it registered. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
774e134ed1 |
Refuse a node with no room, and take back only what llmgw put there
CI / test (push) Successful in 1m39s
Two halves of the same problem: nothing stopped a model being placed on a node that could not hold it, and nothing ever removed the weights of a model that had been deleted from the catalog. Placement now has a disk check, shaped exactly like the memory one it sits beside, with one addition: it is skipped on a node that already holds the weights. A node holding the file needs no room for it, and without that the node best placed to run a model — because it already has it — is the one most likely to be rejected, since holding the file is what used the space. Reclaiming is the dangerous half, so the whole design is about what it will not do. llmgw deletes only files it fetched itself, and that is a calculation rather than a heuristic: the cache name is a hash of the source, so the set of wanted files can be computed from the catalog exactly. Anything without that name was put there by an operator — scp, a shared mount, a copy by hand — and is none of this system's business however unreferenced it looks. A model that took a week to copy must not be removable by a command someone ran to tidy up. The node checks every path again before unlinking it, and that duplication is the point. The control plane knows the catalog; the node knows its disk. A control plane that is wrong, newer than the agent, or tampered with must not be able to talk a node into deleting something, so the agent refuses anything outside its own cache directory or without the cache name — including a path that tries to traverse out of it. Listing is the default and deleting takes --collect. The summary says "asked to remove", because each node decides, and one that refuses a path it cannot prove it fetched is this working rather than failing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
753f4e3265 |
Let a node say how much room it has left
CI / test (push) Failing after 1m31s
Disk was the one resource nothing reported. `statfs` was already being called on every model directory, but only to ask whether the filesystem was a network one — the free space it returns in the same struct was thrown away. So the only way to find out why a node was filling up was to ssh to it, which is what this system exists to avoid. Reported per filesystem rather than per configured directory. Free space belongs to the disk, not the path: two model directories on one disk are one pool of bytes, and counting them separately would let the control plane add 24 GB to 24 GB and place a 40 GB model on a node that has 24. Shared filesystems are summed separately from local ones and left out of a node's own total. Space on a filer belongs to everyone mounting it, so twenty nodes would otherwise each report the same free terabyte, and a scheduler would place twenty models into room for one. Nothing reported means nothing is known, everywhere: an agent too old to send storage and a node with no model directories both send nothing, and both must read as unknown rather than as a full disk. `ctl nodes` and the dashboard show a dash, the metric is absent rather than zero, and a missing directory — which is what an unmounted filer looks like — is skipped rather than reported as having no space. Shown as free out of total, because the first is useless without the second: 8 GB free is comfortable on a 10 TB filer and nearly full on a 10 GB root partition, and only the second stops a model being placed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
b619546516 |
Stop trusting hardware facts that were never checked
CI / test (push) Successful in 1m32s
Three places where the system took the infrastructure's word for something and would have failed silently on real hardware. A declared fabric adapter that is not on the node. Fabric membership has to be configuration — OpenSM's default subnet prefix is identical on every fabric, so a matching prefix proves nothing — but taking it entirely on trust was a different mistake. A typo, or a card that enumerated differently after a reboot, produced NCCL_IB_HCA naming an adapter that does not exist. An adapter with no ACTIVE port. Physically linked with no subnet manager sits in INIT: it looks healthy to anything counting adapters, carries no data, survives a reboot, and looks like a software problem. Discovery had recorded port state all along and placement only ever asked whether a fabric had any adapters listed. Both end the same way — the replica starts, passes its health checks, and hangs on its first all-reduce, which is the failure the fabric invariant exists to prevent arriving through a door nobody was watching. The agent now compares declared against discovered, where both are visible for the only time, and does not advertise a fabric that cannot carry traffic. The reason is logged on the machine that knows it, because the operator who typed the adapter name is the only person who can fix it. The fabric's rate is the slowest active link, since a collective runs at the speed of its slowest. And NVLink topology, which no published binary could see. NVML is its only source, the bindings need cgo, and CGO_ENABLED=0 is deliberate so the portable binary runs on an older libc. The cost was silent: tensor groups fall back to the lowest free indices, which is right on an SXM board with NVSwitch and wrong on a PCIe box where bridges pair specific cards — a group straddling two of them runs several times slower and nothing says so. Releases now also publish an nvml build for those nodes, and llmgw_gpu_nvlink_peers makes the difference visible: a flat zero across a fleet means the topology is unknown rather than absent. A test that declared mlx5_0 and mlx5_1 on a node whose sysfs held neither had been passing, because nothing compared the two. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
42fd0aa31e |
Export what the cluster is made of, not only what it is serving
CI / test (push) Successful in 1m28s
The router published twenty-two metric families about requests, endpoints, admission and scoring — everything about traffic and nothing about the machines carrying it. There was no way to graph GPU usage, watch a node go unhealthy, see a fabric fill up, or answer "was the cluster full at the time", which is the first question anyone asks after an incident. The agents exported nothing at all. Exported by the control plane rather than by each agent. It already holds every number: hardware arrives at registration and state on each heartbeat, so this is a map read away. One scrape target for a fleet of hundreds, no Prometheus configuration per node, no firewall rule per machine, and nothing new running on the nodes. On the same /metrics as the traffic figures, because correlating "the cluster was full" with "requests were queueing" should not need two systems. A collector rather than gauges kept current by a ticker: the numbers are derived from state that already exists, so computing them when asked is simpler than maintaining a copy that can drift, and nothing runs between scrapes. GPU utilisation is new, and is the number people actually watch. Memory used says a model is loaded; this says whether it is doing anything, and a fleet at ninety percent occupancy and two percent utilisation is a fleet holding models nobody is asking for. Adding it caught two things. utilization.gpu goes last in the nvidia-smi query and is optional, because a driver too old to know it returns a shorter row — requiring it made an llmgw upgrade silently stop discovering GPUs on an older node, which is far worse than not knowing how busy a card is. And I gave the new protobuf field number 14, which ecc_errors already had; buf lint is why that was a moment rather than a corrupted wire format. Absent is kept distinct from zero throughout, for the same reason as everywhere else here: an unreported utilisation shown as a flat zero reads as an idle fleet rather than a missing measurement, and a replica nothing has reported would be the most idle thing in the cluster. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
943c23a412 |
Fetch a model's weights when the simple arrangement needs it
CI / test (push) Successful in 1m26s
The default arrangement is the simple one: every node keeps its own copy. So a source naming a remote file is now fetched before the engine is launched, rather than left to fail as a missing path minutes into a load. Eight shards of a tensor-parallel replica start at the same instant and want the same file; they produce one download. An interrupted fetch resumes instead of starting over, which matters at forty gigabytes, and the download is written beside its target and renamed only when complete — a partial file under the model's name would be found by the next scan, reported as present, and handed to an engine that fails on it. The exotic arrangements need no code, which is the point. Weights on a Lustre mount or an image exported over the fabric are simply present, and present means nothing is fetched. The escape hatch from a distribution mechanism is a distribution mechanism that checks first. Wiring it up caught a regression in my own design. A bare model id — "Qwen/Qwen2.5-7B-Instruct" — is a documented source that vLLM and SGLang resolve themselves, and treating every non-remote source as a local path made every existing vLLM profile fail. Only an absolute or explicitly relative path is now checked as a file; anything else is passed through untouched. Three agent tests failed on the same change for a better reason: they used "/models/x", and the new check is exactly the one that catches that. Whole repositories of shards are deliberately not fetched here. vLLM and SGLang do that well already, and a second snapshot downloader would be work spent arriving back where we started. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
c13c567cc5 |
Know where the weights are, without owning the storage
CI / test (push) Successful in 1m32s
Storage will be different at every installation — Lustre, NFS over RDMA, an image exported with SRP or NVMe-oF, local NVMe, an rsync — so llmgw observes it rather than managing it. An agent told which directories to look in reports what it can see; how the files got there is nobody's business here, and a node told nothing reports nothing and behaves exactly as every node did before this existed. That fact buys two things. Placement prefers a node that already holds the model over one that must first move tens of gigabytes, as a preference and never a filter: fetching them is a valid answer, and refusing to place would turn a slow start into no start at all. And ctl status shows what each fabric holds. Per fabric because fabrics are not connected to one another. Weights in ib-a are not weights in ib-b — the only path between them is the management network — so a model seeded into one fabric leaves the rest of the cluster unable to serve it without a slow copy, and nothing said so before now. One shared filer counts the same as twenty local copies, because "can this fabric serve it" is the question being asked. Three distinctions the tests hold to, because each is a way to get this quietly wrong. A node that was never asked to look must rank with one that looked and has nothing, or enabling the feature on a single node would drain every other. An empty directory is not weights, because a half-finished download looks exactly like one and would send placement to a node whose engine then fails to load. And local storage is never mistaken for shared: a filer read as local costs an unnecessary copy, while local read as shared would have the cluster believe a model is everywhere when it is on one node. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
1431426db0 |
Finish adopting an engine instead of stopping at "it is alive"
CI / test (push) Successful in 1m26s
Adoption set a shard to Starting and started a liveness poll, and that was all. Nothing health-checked it, so nothing ever moved it to Ready. The control plane then waited out the profile's startup timeout and killed a perfectly healthy engine so it could start an identical one. An agent restart exists precisely so that engines are not interrupted. It was costing every model on the node a full reload, minutes of it, which is the failure adoption was written to prevent. Measured on node05 while deploying today's other changes: the engine answered /health the whole time and the replica sat in Starting for three minutes until the timeout reaped it. I had seen this earlier in the session and worked around it by killing the engine to force a clean restart, which made the symptom go away and left the cause in place. An adopted shard now takes the same two paths a freshly started one does: a rank that serves HTTP is health-checked, a worker is judged by staying alive. Run in the background, because recovery happens before the agent connects and a slow check must not delay registration. An engine that will not answer is left for the control plane to replace rather than killed locally, which would race the decision it already has a timeout for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
d71d785598 |
Report node memory even when GPU discovery fails
CI / test (push) Successful in 1m31s
Deploying the memory-aware placement showed node01 to node04 reporting 7G and node05 reporting nothing, on a machine whose /proc/meminfo plainly says 32 GB. node05 has no nvidia-smi, so GPU discovery returns an error, Discover returns with it, and the memory read — which has nothing to do with cards — never happened. That is the node whose memory matters most, because a CPU profile is what a GPU-less node will be asked to run, and min_memory_gb would have excluded it from everything. Read first now, before any path that can return early. The test fails against the old order. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
3081c6f69b |
Place on nodes that can actually hold the model
CI / test (push) Successful in 1m23s
Placement had no idea how much memory a node has or a model needs. A CPU profile has no card to be free, so eligibleNodes accepted every ready node: Qwen2.5-Coder 14B, whose working set is about 24 GB, would have been placed just as happily on one of the 8 GB nodes, which reports itself perfectly healthy right up until the OOM killer arrives. node_selector was yesterday's stopgap and it is the operator restating by hand something the system can know. Agents now report MemTotal, profiles declare min_memory_gb, and placement filters on it, naming what was short — "has 8 GB of memory, needs 24" is actionable where "no candidates were considered" is not. The label selector stays, because "this node is for coding models" is a real thing to want to say that memory does not express. Two deliberate non-exclusions. A profile with no requirement fits anywhere, which is every profile that exists today. And a node that reports no memory is still usable: an agent older than the field says nothing, and letting a control plane upgrade quietly empty half a fleet is a worse failure than the one this prevents. ctl nodes grows a MEM column showing "-" rather than 0 for a node that did not say, because an old agent is not a small machine. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
ea87021b5c |
Correct what the previous commit claimed about --jinja
CI / test (push) Successful in 1m30s
|
||
|
Ilker ManapandClaude Opus 5
|
9fc098fd5b |
Start llama.cpp with the model's own chat template
CI / test (push) Successful in 1m25s
Tool calling did not work, and it looked like the model's fault. Asked to
use a tool, qwen2.5-coder produced exactly the right call —
{"name": "list_files", "arguments": {"path": "."}} — as plain reply text,
with tool_calls null and finish_reason "stop". Every OpenAI-compatible
client, opencode among them, sees a model that ignores the tools it was
given.
The cause is that llama-server falls back to a generic chat template unless
--jinja is passed, and the generic one neither formats the tool definitions
the way the model was trained to read them nor parses what comes back into
tool_calls. It affects ordinary chat too, for the same reason.
A profile can still pass --no-jinja, because a model whose bundled template
is broken has to be serviceable without editing the adapter.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Ilker ManapandClaude Opus 5
|
5b425c32e5 |
Ship binaries with releases, and ask engines whether our flags still exist
CI / test (push) Successful in 1m31s
A release was a git tag. Installing llmgw on a node meant having a Go toolchain, and the onboarding script copied a binary from wherever it was run — which works and quietly makes whoever runs it the build server, with the fleet ending up on whatever was in their working tree. Tags now build static linux/amd64, linux/arm64 and darwin binaries and attach them. CGO_ENABLED=0 is the point: it has to run on a node whose libc is older than the builder's, and llmgw has no C dependency because modernc.org/sqlite is pure Go precisely so this stays true. Verified: "not a dynamic executable", runs on a node, reports its version. `ctl engine-check` closes the other half of the phase 4 gap. llmgw's launch arguments are read from each engine's documentation, and engines rename flags between versions; when that happens the failure arrives minutes into a model load, in the engine's own error message, on a node nobody was watching. It builds the widest command llmgw would ever produce — multi-node, since a flag that only appears there is the one nobody notices has gone — and checks each flag against the engine's own --help. No GPU: a CLI prints its help anywhere. Verified against llama.cpp b10991, all seven flags accepted. Flags are matched on a word boundary, so --port is not satisfied by --port-range. Without that, a renamed flag would sail through the check that exists to catch renamed flags. It also warns when the help text looks truncated, which is how the first run went: llama.cpp's help cut to 400 of its 725 lines reported five flags as missing that the running cluster uses. TestSidecarRunsBeforeEngine was flaky under a full run, and worse than flaky — its wait loop fell through on timeout, so a slow machine turned the ordering assertion into a no-op and the failure surfaced later as something else. It now asserts, with a deadline that suits a machine running every other package at the same time. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
c1ac2f022f |
Onboard a node with a script, and find two examples that never worked
New nodes had been joined by hand, four times, by me. There was no procedure, so there was nothing to be wrong — and nothing to be right either. Two scripts now: provision-node.sh makes the VM on Proxmox, onboard-node.sh gives it an identity. They are separate because they are run by different people with different access, and the second must run where the certificate authority is, which is the one part that cannot be delegated. Running them turned up why phase 4 was never verified. FabricAttachment carried no koanf tags, so the `name:` key the shipped example documents bound to nothing and `rate_gbps` was never read at all. An agent configured exactly as instructed refused to start with "fabrics[0]: name is required" — naming the key the operator had just written. Every fabric in this project had been configured in a Go test, where the tags do not matter. The server example did not load either: its models had no source, so it failed validation the moment anything tried to parse it. Both examples are now parsed by a test. A published example that has never been loaded is a guess, and these two had been guesses since they were written. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
058ee382fa |
An agent reporting nothing is reporting something
CI / test (push) Successful in 1m18s
The previous commit taught the reconciler to let go of shards an agent says it is not running. It made no difference on the cluster, because the service returned early when the agent's list was empty — which is precisely the case that carries the news. So the fix was dead code in production while its unit tests passed, and they passed because they called the reconciler directly. The test added here goes through the real stream instead, and asserts only that an agent running nothing reaches the reconciler at all. That is the lesson worth keeping: a test that calls the function under test directly cannot tell you the caller never calls it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
aef1c143c5 |
Let go of shards an agent says it is not running
CI / test (push) Successful in 1m19s
Adoption handled "the agent has something we do not know about". The mirror image was missing: "we think the agent has something it does not". Both follow from the same rule — the fleet is authoritative about what is running (plan §6.5) — and without the second half an engine that dies while its agent survives leaves the control plane believing a replica is Ready forever. Measured on the cluster: the engine was killed, the agent restarted and registered with nothing running, and the replica stayed Ready pointing at a dead port for two and a half minutes until I stopped watching. Traffic survived, because the breaker took the endpoint out and retries covered it — 24 requests, none failed — so the only symptom was a cluster quietly running a third short. A registration follows a fresh connection, so anything dispatched over a previous one is not running by definition; there is no window where a shard is legitimately started but unreported. The previous commit's disown rule was too broad, which this turned up: when a known replica had a live shard it disowned everything the agent reported for it, including the shards we already knew. A reconnecting agent would have been told to stop the replica that was serving. It now disowns only what it does not recognise. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
501db3d3e8 |
Add the llama.cpp engine, and let a profile ask for no GPU
llama.cpp is what makes a cluster with no cards useful: it serves the same OpenAI API on CPU, so the whole path — placement, supervision, scraping, scheduling, admission — can be exercised end to end on ordinary machines. Three things had to give way for it. A profile may now omit its GPU class, and that is how it says it runs on the CPU. Only engines that can actually serve without a card may omit it; for vLLM an omitted class stays an error, because otherwise a typo becomes a replica placed on a machine with no hardware and diagnosed minutes later during model loading. Placement spreads CPU replicas instead of bin packing them, since unlike GPUs there is no allocation to stop two of them sharing a machine's cores. Metric mappings are now per engine rather than per cluster. llama.cpp calls the same three numbers requests_processing, requests_deferred and kv_cache_usage_ratio, and a single global mapping would leave half a mixed fleet unmeasurable. An empty mapping still defaults to vLLM's names; one that names its own Running metric is taken as a complete statement, so the fields it leaves blank stay blank rather than being filled with names that engine cannot have. The adapter refuses multi-node and tensor-parallel profiles. llama.cpp splits across local GPUs by a different mechanism than this adapter drives, and accepting either would produce a replica that starts and then serves from one machine while the catalog believes it has several. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
ec1b6b04df |
Fix an agent that could neither explain itself nor stop
CI / test (push) Successful in 54s
Two bugs found while verifying mTLS against real machines. session cancelled its own derived context and then asked that context whether the agent was shutting down. The answer was always yes, so every stream error read as a clean close: the log said err=<nil>, the backoff reset, and an agent whose certificate the control plane refused hammered it forever at the base interval with nothing to diagnose. It now keeps the parent context to answer that question, and a refusal reaches the operator naming both the certificate and the claim. Cancelling the context also did not stop the agent. HTTP/2 watches the context while the request body is being written, and a bidirectional stream spends its life blocked on the read side instead, so the read never returned. Measured on a deployed node: `systemctl stop llmgw-agent` took 90 seconds and ended in SIGKILL, on every restart. Closing the response when the context ends is what actually unblocks it. TestAgentStopsWhenContextEnds hangs reliably without the second fix, which is the only reason to trust that it tests it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
1e3cafdc86 |
Add a private CA and mutual TLS for the agent stream
A node's identity was whatever the agent said it was. With mutual TLS the control plane compares the client certificate's common name against the claimed node id and refuses the connection when they disagree, so identity becomes something a node proves rather than something it picks. A public authority cannot supply these certificates: Let's Encrypt and its peers stopped issuing the clientAuth usage, and Go verifies client chains against it. Hence internal/tlsconf/ca.go, a small authority whose key lives on one machine and whose root is trusted everywhere, and `llmgw ctl cert` over it so issuing is reproducible instead of remembered openssl. The server certificate's hosts matter more than they look: hostname verification reads subject alternative names and ignores the common name entirely, so a certificate without the address agents dial — bare IP included — is rejected no matter what its subject says. There is a test for exactly that, because it is the failure everyone hits once. requireIdentity stays wired to false until the configuration lands, so this commit changes no running behaviour. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
 ilkerandClaude Opus 5
|
e8d90cf375 |
Treat a zombie as dead, and test fairness where it is deterministic
CI / test (push) Successful in 52s
CI went red on Linux with two failures that never appear on macOS. Both were worth the trip. The first is a real bug. processAlive used kill(pid, 0), which succeeds on a zombie — a process that has exited and is only waiting to be reaped. That happens for real whenever an engine's parent dies before it does and nothing reaps the orphan, which in a container is the normal case: PID 1 is usually the workload rather than an init. The supervisor would then report a dead adopted engine as running forever, and hold its GPUs against every future placement. Zombies are now detected through procfs on Linux, where the answer exists, and the check degrades to its old behaviour elsewhere rather than to something worse. The second was a bad test. TestTenantFairnessWithinBand asserted on the order goroutines sent to a channel after being admitted, which is the order the Go scheduler woke them, not the order the dispatcher chose. Under load it measured the runtime. The rotation itself lives in bandQueue.pop, which is pure, so it is tested there — including what happens when a tenant runs out mid-rotation and when a cancelled waiter is removed. The controller-level test now asserts the thing that is actually stable: nobody is starved. Verified with three consecutive `go test -race ./...` runs in a Linux container. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
ilkerandClaude Opus 5
|
1d1a6aebb0 |
Phase 4: make a multi-node replica able to start
CI / test (push) Failing after 53s
Placement and environment generation had landed early, which made multi-node look further along than it was. A replica spanning nodes could not come up at all, and all three reasons came down to assumptions that only hold on one node. Not every rank serves HTTP. The agent health-checked every shard at its own port, but only the leader answers; workers timed out, reported Failed and took the replica with them. Engine.ServesHTTP now says which ranks serve, and a worker is judged by staying alive. That is deliberately weaker evidence: the replica's real readiness is the leader answering, which only happens once every worker has joined. A shard is not always one process. Engine.Commands returns a sequence — every command but the last is a sidecar kept alive for the shard's lifetime and torn down with it, and the last one's exit is the shard's. vLLM under Ray is why: the leader runs a Ray head and then the server against it, while a worker runs only the join and no vLLM process at all. RayCommand had been written and tested but never called, so launch_mode: ray would have run vllm serve against a cluster that did not exist. Sidecars are killed with the shard, because a Ray head left behind holds the port the next replica needs. Start order was backwards. Workers went first, on the theory that the head should wait for them. Both engines put the rendezvous point on rank 0, so a worker started first has nothing to connect to. vLLM's server blocks until the Ray cluster has the GPUs it needs, so the leader can start immediately and waits on its own. A command can now also require that rendezvous point to be listening before the next starts, bounded by a timeout so a head that never appears fails the shard instead of hanging it. Stop was also killing only the engine, leaving sidecars behind. None of this is verified against real engines — it is tested with stub processes. The Ray sequencing, and which launch mode a given vLLM version wants, can only be confirmed on hardware. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
ilkerandClaude Opus 5
|
75a8e76706 |
Survive a control plane restart without orphaning live engines
CI / test (push) Successful in 1m44s
Not persisting replica state left one hole: restarting the control plane made it forget every replica while the engines kept running. They held GPUs the inventory believed were free, so the next placement failed to allocate, with an error naming the new process rather than the old one. The fix belongs on the agent, not in a database. A row can say "running" about a process that died; the fleet cannot. - the supervisor writes the full shard spec, not just a pid and command line. A pid identifies a process; the spec is what lets it be described back to the control plane and matched to a replica. - Adopt takes over an engine a previous agent left running. It is not this process's child, so it cannot be waited on: it is watched by polling and stopped by signalling its group directly. Its earlier output is gone and the log says so rather than presenting emptiness as silence. - every Register reports what the agent is running, not only the first, because a reconnect may be a control plane restart and has to recover the same way. - the reconciler rebuilds replicas from those reports and reclaims their GPUs. Anything whose model or profile left the catalog is stopped; nothing will route to it and it is holding cards. - an engine the agent cannot identify is killed. A process holding GPUs that nothing can describe is worse than no process, because placement hands those GPUs out again. Stop was also wrong for adopted shards: it checked for an exec.Cmd, which they do not have, and reported them stopped without killing anything. TestAgentReconnects had been passing for no reason. httptest's CloseClientConnections does nothing to an h2c stream — prior-knowledge h2c hijacks the connection and httptest stops tracking hijacked ones — so the agent never reconnected, and the assertion on the connection map passed on a stale entry. Tests now sever connections through their own listener and count opened streams. Plan §11 also records what multi-node still needs, since placement landing early makes it look further along than it is: workers never become Ready because the agent health-checks every shard while only rank 0 serves HTTP; RayCommand is written and tested but never called; and startup is fire-and-forget. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
ilkerandClaude Opus 5
|
2373726efd |
Agent stream: connect the fleet to the control plane
CI / test (push) Failing after 1m29s
One bidirectional connect-go stream per node, carrying registration, heartbeats and shard status up, and start, stop and drain down. The agent dials out: GPU nodes sit behind firewalls far more often than a control plane does, and an outbound connection needs no inbound rule on every node in the fleet. - registration carries the whole hardware inventory, so there is no separate discovery call and a reconnecting agent refreshes what the scheduler knows. The fabric labels travel with it; they cannot be inferred from the wire, so losing them in transit would silently disable multi-node placement. - agentsvc is both the stream handler and the reconciler's dispatcher, because it holds the open streams and is therefore the only thing that can deliver a command to a node. - the agent validates a profile before launching anything, since an engine that refuses a profile does so after minutes of loading, then supervises the process and reports every transition with the engine's output tail on failure. - reconnects forever with jittered backoff. A control plane restart must not require touching every GPU node, and a fleet must not reconnect in lockstep and knock it over again. Engines keep serving throughout, because the router talks to them directly. llmgw agent is real now, and llmgw server runs a control plane whenever any model declares a profile. The static endpoint list stays for router-only runs. Verified with real binaries as separate processes: the agent registers, a replica is placed, an engine launches through the vLLM adapter's actual argument construction, streaming traffic is served, and killing the agent moves the node through unreachable to lost with placement explaining why nothing can run. config.Model and catalog.Model were the same thing described twice, so the router now reads catalog entries directly and the validation rules live in one place. Still missing: mTLS. The stream is plaintext HTTP/2 and must stay on a trusted network until that lands. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
ilkerandClaude Opus 5
|
55417546fc |
Phase 2 (2/2): NVML backend behind a build tag
CI / test (push) Successful in 1m37s
NVML is the only source of NVLink topology, and that topology is what keeps a tensor-parallel group off the slow path inside a node. It needs cgo, so it sits behind the nvml tag and the default build keeps using nvidia-smi. Discover() picks the backend and falls back to nvidia-smi when NVML fails, logging the fallback rather than performing it silently: on a node that is meant to have working NVML, a quiet downgrade hides a broken driver installation and costs the scheduler its topology. CI now compiles the tagged build. That immediately paid for itself: ErrNoNVML was declared only in the untagged file, so the tests did not compile under -tags nvml. It is now declared where both builds can see it. Not verified: the NVML path has never run against a driver, because there is no GPU machine available. The NVLink domains it reports should be treated as unverified until someone runs `llmgw agent inventory` on a real node. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
ilkerandClaude Opus 5
|
6a13e3dddf |
Phase 2 (1/2): node discovery, engine adapters, process supervisor
CI / test (push) Successful in 45s
Everything here works without GPUs and is covered by tests. The control plane connection is not built yet, so llmgw agent itself is still a stub and the phase exit criterion is not met. - discovery: IB from /sys/class/infiniband, parsing the formats sysfs actually uses and telling native InfiniBand from RoCE; GPUs via nvidia-smi, the cgo-free path, with health flags for uncorrected ECC and memory already held by another process. A node without IB is not an error, it is a single-node island. - engines: vLLM and SGLang command and environment construction, kept free of process handling so it stays testable. The NCCL environment is the part the plan marks critical: NCCL_IB_HCA lists only the chosen fabric's adapters, because leaving it open lets NCCL pick an adapter on a fabric where the peer is unreachable, and the job then hangs instead of failing. - validation before launch: no mixed GPU types in a tensor-parallel group, the node/GPU/TP arithmetic has to add up, and fp8 is refused below compute capability 8.9 rather than silently falling back to a slower path. - supervisor: process groups, because engines fork workers that hold GPU memory and outlive a signal sent only to the leader; a bounded log tail, since engine output is the only diagnosis when loading fails; Starting separate from Loading, because a 70B model legitimately takes minutes; and orphan adoption through pid files fingerprinted with the command line, since pids get reused and adopting an unrelated process is worse than leaking the original. - readiness is two questions: /health says the process is up, /v1/models says the weights loaded. Routing on /health alone sends traffic into a loading model. - llmgw agent inventory reports what the scheduler would see on a node. One plan correction: GPU class sizes come from the driver in GiB, which does not always match the marketing name. An L40S is sold as 48GB and reports 44.99 GiB, so its class is l40s-45g, not l40s-48g. Consistency matters more than matching the box, because that number decides whether a model fits. Plan §2 and §5.1 updated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |