Adding a node was deploy/bootstrap/onboard-node.sh, which is not installed
anywhere and wanted the node's name, the control plane's address and a
dozen flags. It is now in the binary, run on the control plane:
sudo llmgw node add ubuntu@10.0.0.21
The node id is the machine's hostname, and the address its agent dials is
this machine's on the route to it, checked against the server certificate.
It installs llmgw there (the release for its architecture, fetched and
checked by the node; uname rather than dpkg, so RHEL works), runs doctor
there and stops on a problem before changing anything, issues and installs
the certificate, writes agent.yaml and the unit, starts the agent, and waits
for it to join and install llama.cpp. The authority is read first, so a
missing one is said before the node is touched.
The steps are events. In a terminal they draw a checklist (bubbletea) that
asks what a script cannot: which fabric each group of active adapters is on,
grouped by the subnet manager or RoCE network they report, and whether to go
on past a problem. Elsewhere, one line per step.
Found by running it on an empty VM: ssh's "Permanently added" warning on
stderr was read as the machine's hostname, so standard error is now kept
apart; and a node added again while cordoned never counted as joined, so
joining now means a heartbeat since the restart.
llmgw node remove forgets a lost node, which otherwise stays listed until
the control plane restarts. ctl nodes no longer lists vLLM as skipped on
every CPU node. The quick install, install and onboarding docs use it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
install.md explains every choice and is long for someone who wants the
commands. quick-install.md has only those, and what to check after each:
the binary, server init, onboarding GPU nodes (with doctor first and the
script fetched from the release tag, since it is not installed anywhere),
a model from the mirror's explorer, keys and editors, and Caddy. Every
button and path it names was checked against the pages. quickstart.md is
now said to be the from-source, one-workstation path it is; README and the
about page point at the new guide.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
vLLM was installed by hand and named in each agent's configuration. The
agent now installs it: a Python 3.12 environment built by uv, which is
itself a GitHub release checked against its checksum, from PyPI resolved as
it stood a day after that vLLM was published and installed with
--require-hashes. The resolved list is kept in the environment. The
environment is built relocatable under a scratch name and renamed into place
once it runs, like an archive.
The default is 0.30.0, whose flags were checked against the ones llmgw
builds. It is built for CUDA 13, so it needs driver 580 and a Turing GPU or
newer, and 25 GB free while it installs; a node lacking one says which, and a
node with no GPU skips it without being shown as a fault. gcc comes with it
because Triton compiles its launcher at the first model load. SGLang
installs the same way when the control plane names a version. llmgw doctor
reports whether vLLM can be installed on the node.
Checked by installing 0.30.0 this way on a test node: 7.7 GB, torch 2.13 for
CUDA 13.0, importing cleanly; the node has no GPU, so vLLM was not started.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The changelog's unreleased section had stopped at refusing unloadable
weights; it now covers the rest: telling independent fabrics apart,
their subnet managers and maps, NVLink on every build and across nodes,
the multi-node launch fixes and vLLM's new default, doctor, node load,
editors and the Responses API, the chat and mirror pages, a person's own
use, the admin pages, the new metrics, and how to upgrade.
docs/fabrics.md is new, for a site with InfiniBand or NVLink: what
llmgw does with them, what each node and fabric needs first, declaring
fabrics, checking a node with doctor, what the pages and metrics show
and which to alert on, what an engine is given, and where to look when
a replica spanning nodes will not run. It says plainly that none of it
has yet run on such hardware.
install.md, node-onboarding.md, onboarding.md, the README and the
example agent configuration no longer say the portable binary cannot
see NVLink or that a fabric's name cannot be checked, and describe the
account page and connecting an editor.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
server init now also:
- runs a model mirror beside the control plane (/srv/models, bytes on
:8081 at an address nodes reach, management API on loopback so it
needs no certificate) and adds it to server.yaml;
- makes the machine a node of its own control plane: issues its agent
certificate, writes agent.yaml labelled as the control plane, starts
llmgw-agent, and waits for it to register and install llama.cpp;
- checks for NVIDIA GPUs with nvidia-smi, and reports a card on the PCI
bus whose driver does not answer rather than ignoring it;
- fetches Qwen2.5 0.5B (Q4_K_M) into the mirror and adds it to the
catalog pinned to this node: a llama.cpp profile per GPU class found,
with a CPU profile as the fallback, then waits for it to serve.
--no-mirror, --no-node and --no-model leave these out. Every step looks
before it acts, so a rerun finishes what an interrupted one started.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Installs PostgreSQL with the machine's own package manager when it has
none (initdb with password auth on the RHEL family, whose default is
ident), creates the llmgw role and database with a generated password
kept in /etc/llmgw/secrets/store-dsn, and checks the address connects.
On a machine already on SQLite it stops the gateway, copies the store
so accounts and keys survive, points auth.store at the new address and
starts it again. A move that fails starts the gateway on SQLite again.
server.yaml is edited as text so its comments survive, and kept beside
it. A PostgreSQL init did not install is not resized: it may serve
something else. It refuses to move one PostgreSQL store into another.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
install.md said the client port belongs behind a reverse proxy and gave
nothing to put there. deploy/proxy/ has both, attached to every release.
What they get right is what a first attempt gets wrong: streamed replies
are not buffered, the read timeout outlasts a model's silence before its
first token (nginx's 60s default is a 504), the body limit matches the
router's, and X-Forwarded-Proto and -Host are set by the proxy rather
than passed through — llmgw marks session cookies Secure from the first
and builds the landing page's base URL from the second.
The Caddyfile was validated and run against a streaming upstream: events
arrive as sent, and forged forwarded headers are replaced. nginx uses
`listen ... http2` because `http2 on;` needs 1.25.1 and Ubuntu 22.04 and
Debian 12 ship older.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
`llmgw server init` does what install.md's steps 2 and 3 asked of the
operator: the llmgw user and directories, a certificate authority, the
control plane's certificate for every address the machine has, a
configuration, the default project, an administrator with a generated
password, and the systemd unit — then starts it and waits for /health.
Every step looks before it acts, so a second run changes nothing, and
--host later reissues the certificate with the old names kept. It never
overwrites server.yaml or an authority, and leaves alone certificates
issued by an authority kept on another machine.
The configuration install.md showed could not start: an empty catalog
without control.enabled is refused as having nothing to serve. The one
init writes sets it, and a test holds that.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Read start to finish as somebody standing up a new cluster, it sent them
down a path we no longer take.
The mirror was absent from all 821 lines, and the page still said that
distributing weights was "a known gap, not an oversight" — so the reader
would have gone away to solve, by hand, the problem the binary in front of
them solves. It is step 4 now, before the nodes, because where a node
fetches from is the difference between minutes and a day, and on a customer
network the nodes often have no route out at all.
Two of the listed gaps had been closed and were still listed: a fetch does
report its progress to the control plane, and there is an HA pattern. A
gap somebody believes in is worse than one they discover.
Added what this week made obvious was missing. Upgrading, including the
zero-byte copy that ssh -n silently produces and the size check that is the
only thing between that and a fleet on a truncated binary. Backing up, in
the order that matters, and the fact that API keys are hashed and do not
come back. Pricing in the first configuration, because a budget without a
price is displayed, counted against nothing, and never binds. And scale,
bench and the source check, none of which were written down anywhere.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Neither had a page. "We have GPUs, now what" was left to the reader, and what
llmgw stores was scattered across four documents and a changelog — which is
the same as not being written down when somebody is doing a security review.
docs/data.md describes the behaviour and leaves the legal mapping to whoever
holds the obligation. Every claim in it was checked against the source, and it
names where to look for each: usage rows carry no text field, keys are stored
hashed, conversations are off until a named project turns them on and cannot be
turned on without a retention window.
It also states the egress exactly, which took reading every outbound call to
say honestly: nothing on the serving path leaves the cluster, and the two
things that can reach outside — fetching weights, and ctl plan reading a model's
architecture — are both started by an operator. Weight fetching points wherever
you tell it, including an internal mirror.
docs/choosing-models.md is the opinion llmgw itself does not have. Three jobs
rather than one model, with the third — embeddings — being the one that gets
forgotten and the one that decides whether the deployment can answer questions
about your company or only questions in general.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The last piece of HA, and it is a deployment pattern rather than code: llmgw
picks one leader and agents follow it, but client traffic arrives in front of
both and llmgw is not in front of anything.
Writing the pattern found a bug in the pattern. The first draft said to
health-check `/health` and claimed a standby still routes to replicas the
leader placed, so sending it traffic was fine. It does not. A standby's
reconciler places nothing and it holds no agent streams to adopt from, so it
publishes an empty endpoint list — a load balancer following that advice would
have sent half the traffic to something that answers every request with 503.
So there is now a `/ready` that means "this process has somewhere to send a
request": 503 on a standby, 503 on a leader whose replicas have all failed,
200 when there is something to serve. `/health` keeps meaning the process is
up, which is a different and less useful question. Both are open, because a
load balancer has no key.
There is a gap during failover where neither passes — three seconds when the
leader stops cleanly, up to the lease TTL when it does not — and the document
says so rather than implying otherwise. Traffic cannot go to a control plane
that has nothing to route to.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was no reference. Four narrative documents explain how to install and
operate llmgw, and the only way to learn that `lease_ttl` exists was to read
the one that happened to mention it — or the struct.
Written by hand it would have been wrong within a fortnight. Six settings
arrived in the last two weeks alone: store, servers, lease_ttl, the two
retention windows, min_disk_gb. A reference nobody maintains is worse than
none, because it is the same list with the setting you came to look up quietly
missing from it.
So it is derived from the structs, with the field comments as the descriptions
— which means the only way for it to be wrong is for the code to be wrong, and
it makes the comments worth writing twice over. 1163 lines, 24 blocks, 87 keys.
Two tests keep that true after everyone has forgotten this exists. One
regenerates and compares, so adding a setting without regenerating is a
failure rather than an omission — checked by adding one and watching it fail.
The other walks the tree for koanf tags and fails on a configuration file
that is in neither the source list nor the short list of structs an operator
never writes, because a list is a thing to forget.
The generator finds the repository from its own path rather than the working
directory, so `go generate ./...` produces the same document from anywhere.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The store stops being a file on one machine, which is what stood between this
and more than one control plane.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The section leads with why SQLite is still the default rather than with how to
leave it: one file, no server, and a request path that barely touches it. Two
things change that, and both are recent — conversations and notes read and
write on every turn, and a second control plane is impossible while the store
is a file on one machine.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The gap list said a fetch "reports no progress". It does — the agent logs a
percentage as it downloads, with a comment saying a replica pulling a hundred
gigabytes and one that is wedged look identical from the control plane.
That comment describes the real gap, which is narrower and more useful than
the one written down: the progress exists, it just never leaves the node, so
the operator looking at ctl or the dashboard still cannot tell those two
apart.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Seventeen commits since v0.6.0, and not a phase of the roadmap — which is why
the table leaves the phase column empty rather than inventing one. What landed
is what a cluster needs to be lived with rather than demonstrated: somewhere
to keep a conversation, somewhere to keep a note, and an honest answer about
the disk underneath both.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The section opens on the default rather than on the feature, because the
default is the part someone deploying this needs to be sure of: nothing anyone
types is stored until a project asks, and the command to ask is an act rather
than a setting.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A command that deletes data needs its limits written down more than its
syntax. The section says plainly that llmgw only takes back files it fetched
itself, and then says the uncomfortable half out loud: that this means it will
not reclaim the storage most likely to be filling your disk, because weights
you placed yourself are yours to remove.
Also records the dash in the DISK column as "not asked" rather than "no
space", which is the reading that matters when a node will not schedule.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
It said "there is no distribution mechanism yet" and then, two paragraphs
later, documented the distribution mechanism. The known-gaps list — the part
someone reads at 2am — still said "no cache, no fetch, no progress". Fetching
landed in this cycle; both sentences were written before it and left standing.
node-onboarding.md carried the same claim.
What is actually still missing is narrower and worth naming precisely: a fetch
reports no progress, so a first start on a large model looks like a hang, and
every node fetches its own copy, so eight nodes on a new model are eight
downloads from one origin.
Also: the dashboard now shows who is using the cluster and what they are
running, which the sentence describing it did not mention; and one sentence
about idle timeouts had lost the words that made it a sentence.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The metrics were exported and the only way to look at them was to stand up
Prometheus beside the binary. For a project whose whole shape is one static
binary with no dependencies, that is the wrong default: the common case is
somebody wanting to know how full their cluster is, and that should not cost
a monitoring stack.
/admin now shows capacity per GPU class with a usage bar, free whole nodes
per fabric — the number that decides whether a replica spanning nodes can
land, since free cards on the wrong fabric are not capacity for a model that
needs this one — and which replicas are holding hardware nobody is asking
for. Nodes gain their memory and how many model files they can see.
Taken from the same Summary that ctl status uses rather than derived in the
page, because two implementations of the same arithmetic drift apart and the
one on the screen is the one people would believe.
A faulty card counts as neither used nor free, matching the scheduler.
Counting it free is how a cluster looks like it has room it does not, and a
page that disagreed with placement about that would be worse than no page.
Prometheus stays supported and is now the answer to a narrower question:
history, alerting, several clusters at once. The dashboard shows what is
true now and keeps nothing, which is the honest division between them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The metrics existed and nothing could record them. They shared a listener
with pprof, the management API and the dashboard, and pprof can dump the
process heap where API keys sit in plaintext — so a fleet could be monitored
or keep its heap private, and not both. That is not a choice anyone should
have to make, and it is why every previous answer to "how do I scrape this"
was a tunnel.
metrics_listen serves metrics and health and nothing else, so it can be
reached from wherever Prometheus lives. Metrics stay on the admin listener
too: adding a door is not the same as moving one, and anything already
scraping keeps working.
While there: admin_listen defaulted to :9090, every interface, with a
comment beside it saying not to expose it. A default that contradicts its
own documentation is a footgun, so it is loopback now and someone who wants
otherwise says so.
The handler is nil until the registry is built, and says so rather than
serving an empty page. Ordering has already caught this exact area once —
a collector registered after AdminHandler published nothing at all, silently
— and an empty scrape target looks like a working one.
deploy/monitoring/prometheus.yml has a starting configuration and the alerts
worth having: a node that has stopped reporting, a model that should be
serving and is not, a card kept out of placement by a fault, and a fleet
holding models nobody is asking for.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
On-demand loading worked right up to the moment the cluster was full and
then stopped meaning anything: the model somebody had just asked for could
not be placed, and the cards were held by a model nobody had asked for in an
hour. That is the only situation in which any of it matters.
Deliberately narrow. Only idle replicas of other on-demand models — a model
at min 1 is a promise that it stays up, and breaking it to serve a different
request would make every guarantee in the catalog conditional on load. Only
single-node replicas, since evicting a multi-node one frees several machines
and costs tens of minutes to rebuild. One at a time, because freeing
everything that qualifies would empty a cluster for a request that needed
one node. And never a replica with no reported usage: after a router restart
that is all of them, and unknown is not idle.
Writing the test found a gap the feature had rather than a bug in the test.
Two on-demand models on a cluster with room for one would take turns
evicting each other, at a model load per cycle — worse for both than one of
them waiting. A replica is now safe from eviction for a few minutes after it
starts serving, which bounds that.
Which in turn found a latent bug: deriveState stamped ReadyAt with
time.Now() inside a package that has an injectable clock, so the new
lifetime rule could not be tested at all — the fake clock said three minutes
and the wall clock said eight months. Invisible while nothing compared
ReadyAt to anything.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The feature the idle tracking was for. A cluster with more models than GPUs
can now hold every model in its catalog and only the wanted ones in its
cards: min 0 with a max above it means load on demand, and once nobody has
asked for idle_timeout the replicas stop and the cards go back.
No new field, because min 0 max 1 already reads correctly — keep none
running, allow up to one. Which made min 0 max 0 meaningful for the first
time: configured and deliberately not running. normalize() used to fill a
zero max in, on the reasonable ground that a model with no maximum cannot
run; that would now silently turn every switched-off model into an
on-demand one, including the ones already written that way in this
cluster's own configuration. A model with no scaling stanza stays off for
the same reason — defaulting it to on-demand would let any client request
start a 70B nobody intended to run.
The one real decision is what happens to the request that triggers a load.
Refusing makes the feature useless, since the first request for every model
fails; waiting without a bound turns a request into a job, and a client that
budgeted thirty seconds holds a connection for ten minutes. So it waits
briefly and then says what is happening — 503 with Retry-After and "the
model is loading", which is a different answer from "no endpoint serves
this" and produces different client behaviour. An unknown model still gets
the unknown-model error, or on-demand would turn every typo into "loading"
and the useful message would disappear.
Use keeps a model alive rather than only the request that started it,
without which the timer measures time since the load and a busy model is
evicted on schedule however hard it is working.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three places where the system took the infrastructure's word for something
and would have failed silently on real hardware.
A declared fabric adapter that is not on the node. Fabric membership has to
be configuration — OpenSM's default subnet prefix is identical on every
fabric, so a matching prefix proves nothing — but taking it entirely on
trust was a different mistake. A typo, or a card that enumerated
differently after a reboot, produced NCCL_IB_HCA naming an adapter that
does not exist.
An adapter with no ACTIVE port. Physically linked with no subnet manager
sits in INIT: it looks healthy to anything counting adapters, carries no
data, survives a reboot, and looks like a software problem. Discovery had
recorded port state all along and placement only ever asked whether a
fabric had any adapters listed.
Both end the same way — the replica starts, passes its health checks, and
hangs on its first all-reduce, which is the failure the fabric invariant
exists to prevent arriving through a door nobody was watching. The agent
now compares declared against discovered, where both are visible for the
only time, and does not advertise a fabric that cannot carry traffic. The
reason is logged on the machine that knows it, because the operator who
typed the adapter name is the only person who can fix it. The fabric's rate
is the slowest active link, since a collective runs at the speed of its
slowest.
And NVLink topology, which no published binary could see. NVML is its only
source, the bindings need cgo, and CGO_ENABLED=0 is deliberate so the
portable binary runs on an older libc. The cost was silent: tensor groups
fall back to the lowest free indices, which is right on an SXM board with
NVSwitch and wrong on a PCIe box where bridges pair specific cards — a
group straddling two of them runs several times slower and nothing says so.
Releases now also publish an nvml build for those nodes, and
llmgw_gpu_nvlink_peers makes the difference visible: a flat zero across a
fleet means the topology is unknown rather than absent.
A test that declared mlx5_0 and mlx5_1 on a node whose sysfs held neither
had been passing, because nothing compared the two.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The default arrangement is the simple one: every node keeps its own copy.
So a source naming a remote file is now fetched before the engine is
launched, rather than left to fail as a missing path minutes into a load.
Eight shards of a tensor-parallel replica start at the same instant and
want the same file; they produce one download. An interrupted fetch resumes
instead of starting over, which matters at forty gigabytes, and the
download is written beside its target and renamed only when complete — a
partial file under the model's name would be found by the next scan,
reported as present, and handed to an engine that fails on it.
The exotic arrangements need no code, which is the point. Weights on a
Lustre mount or an image exported over the fabric are simply present, and
present means nothing is fetched. The escape hatch from a distribution
mechanism is a distribution mechanism that checks first.
Wiring it up caught a regression in my own design. A bare model id —
"Qwen/Qwen2.5-7B-Instruct" — is a documented source that vLLM and SGLang
resolve themselves, and treating every non-remote source as a local path
made every existing vLLM profile fail. Only an absolute or explicitly
relative path is now checked as a file; anything else is passed through
untouched. Three agent tests failed on the same change for a better reason:
they used "/models/x", and the new check is exactly the one that catches
that.
Whole repositories of shards are deliberately not fetched here. vLLM and
SGLang do that well already, and a second snapshot downloader would be work
spent arriving back where we started.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Storage will be different at every installation — Lustre, NFS over RDMA, an
image exported with SRP or NVMe-oF, local NVMe, an rsync — so llmgw observes
it rather than managing it. An agent told which directories to look in
reports what it can see; how the files got there is nobody's business here,
and a node told nothing reports nothing and behaves exactly as every node
did before this existed.
That fact buys two things. Placement prefers a node that already holds the
model over one that must first move tens of gigabytes, as a preference and
never a filter: fetching them is a valid answer, and refusing to place would
turn a slow start into no start at all. And ctl status shows what each
fabric holds.
Per fabric because fabrics are not connected to one another. Weights in
ib-a are not weights in ib-b — the only path between them is the management
network — so a model seeded into one fabric leaves the rest of the cluster
unable to serve it without a slow copy, and nothing said so before now. One
shared filer counts the same as twenty local copies, because "can this
fabric serve it" is the question being asked.
Three distinctions the tests hold to, because each is a way to get this
quietly wrong. A node that was never asked to look must rank with one that
looked and has nothing, or enabling the feature on a single node would
drain every other. An empty directory is not weights, because a
half-finished download looks exactly like one and would send placement to a
node whose engine then fails to load. And local storage is never mistaken
for shared: a filer read as local costs an unnecessary copy, while local
read as shared would have the cluster believe a model is everywhere when it
is on one node.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Written for whoever is standing up the cluster with root on some servers and
nothing installed. Control plane, certificate authority, first node, first
model, first request.
It leads with what llmgw does not install — drivers, CUDA, engines, weights
— because a node missing any of them registers, reports ready, and fails
every replica placed on it, and the reason is three logs away. The useful
order is to make a node able to serve before onboarding it, which is the
opposite of what the tooling's convenience suggests.
The failures it warns about are ones this cluster actually hit: llama-server
present and unable to start for want of libgomp1 while the node reported
healthy; a Proxmox VM on the default kvm64 CPU, where inference is six to
nine times slower and nothing looks broken because every number is simply
small; requests dying at exactly sixty seconds on a liveness timeout that is
a generation budget in disguise for non-streaming calls.
Verified rather than recalled. Every command it prints was checked against
the binary, every config key against the struct tags, and its own
verification and troubleshooting commands were run against the live cluster.
That caught two errors in my draft: the health endpoint is /health and
returns an empty body, not /healthz, and the Go version needed is 1.26.
The known-gaps section is there so a reader finds them on a Tuesday
afternoon rather than at 2am: weights and engines are theirs to distribute,
there is one control plane and no HA, and multi-node replicas have never run
on real InfiniBand — only against simulated hardware.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
