The setup screens can reach a database machine by a join code instead of
ssh: they listen on 7444 and show a short code; llmgw join dials out,
checks the setup by fingerprint, admits itself with the code's secret, and
runs what the setup sends until it is done. A joined machine is one more
onboard.Remote, so checks and the Patroni install run over it unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
With nothing installed, sudo llmgw asks for one machine or three, takes how
to reach the other two (ssh agent, key or password, and a sudo password used
only for the install), checks every machine first, shows the plan and its
command line, and installs. On a control plane it opens a console with the
cluster's state and the next steps. Machines are reached with Go's ssh so a
password works; the store's etcd and access rules use this machine's routed
address rather than a client-facing name.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
init --patroni user@host ... installs PostgreSQL, Patroni and etcd on the
given machines over ssh, with this one as the third etcd vote, makes the role
and database, and moves the store in use into the cluster. Each step checks
first, so it changes nothing on a cluster that is up. For a store of several
hosts the control plane publishes each member's role, state and lag from
Patroni, a stopped member as missing, prints it in ctl status, and alerts on
it. Making an administrator on PostgreSQL sent a bool to an integer column;
it sends an integer now.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A team's members, quota and spend were on three pages that linked to each
other. They are three tabs of one page now, each at its old address, and
choosing anything in the tree opens its limits, capacity and use beside it,
editable in place.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Each engine has a list of builds, best first, from what its project
publishes and how each was compiled: llama.cpp CUDA 13.3, CUDA 12.8
(driver 570 for the GPUs it ships as PTX), then Vulkan, never the CPU
build on an NVIDIA node; vLLM CUDA 13, then CUDA 12.9, and for GPUs
older than every build of the version asked for, the newest built for
them (V100: 0.19.1). The node takes the first its hardware meets and
that installs and computes on the GPU, moving on when one does not, and
reports the version it runs. GPU processes are read from nvidia-smi's
full report, so Vulkan engines count as on their GPU.
Refs #2.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Read from vLLM's release pipeline: every x86 build from 0.20.0 has
kernels for compute 7.5 and up, the CUDA 12.9 one included, and 0.19.1
is the last built for 7.0. PyPI's build is CUDA 13 from 0.20, not 0.26.
A V100 asked for a newer vLLM is refused, naming 0.19.1 for
engine_overrides; the CUDA 12.9 fallback is for older drivers on
Turing and newer. The cu129 wheel is found among the release's assets,
since its manylinux tag differs between versions.
Refs #2.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A node that cannot run the CUDA 13 build installs the same version's
CUDA 12.9 build from vLLM's GitHub release, with PyTorch's cu129 torch;
the install computes on the GPU before it counts. The build can be
chosen per node — auto, cu13, cu129, none — on the cluster page, with
ctl engine-build, or over the admin API; the control plane keeps the
choice and tells the node at once.
Refs #2.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
engine_overrides give nodes picked by GPU class, id or labels their own
engine versions; node add --disable-engine keeps one off a node; what
vLLM cannot do on a node is a warning rather than a refused node; the
CUDA requirement follows the version; a hand-named binary is no longer
replaced by an installed one.
Refs #2 (items 4 and 7).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
llmgw ctl model adapters lists a model's adapters and adds or removes
them through POST /api/v1/models/{name}/adapters, which reads each new
file first and refuses one that is not an adapter for the model's
architecture, as the mirror page's card does.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A model's adapters are fetched by each node with its weights and loaded
unapplied at start; a request naming an adapter reaches the model's
replicas with llama.cpp's per-request lora field. Adapter names share the
model namespace, are listed, and follow allowlists on their own. The
router's config seed and pushed catalog now build the name map in one place.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A catalog model with external {url, model, api_key_file} and no profiles
is published as an endpoint by the control plane, the key read from its
file each time; the router rewrites the request's model to the provider's
name and sends its key. It is never scraped nor stale, and nothing is
placed for it. Named in fallbacks, it serves when the cluster has no room.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ed25519 over checksums.txt: llmgw release keygen makes the pair, CI signs
with the RELEASE_SIGNING_KEY secret when it is set, and release.Install
refuses an unsigned or wrongly signed release when the build embeds a
trusted key (internal/release/trusted.pub, empty for now) or
LLMGW_RELEASE_KEY names one.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Collects rc.1 to rc.14: the mirror page in tabs with a card per model,
one line per record in every admin table, reserved CPU cores and required
memory, capacity classes and preemption, tenant capacity quotas, sizing by
measured throughput, and node sensor readings. Adds `llmgw upgrade` for the
mirror host, which `ctl upgrade` does not reach, and says so when it ends.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
CPU replicas start on half the machine's CPUs, not all of them: two busy
llama.cpp servers each spinning on all 8 vCPUs fell to 0.8 tok/s apiece,
where at 4 threads they keep 5.5. llamacpp.threads sets it per node.
Inference requests can be traced over OTLP/HTTP (tracing.endpoint), with
admission, schedule and upstream spans and traceparent sent to the engine.
Security: onboarding no longer puts the cluster root in the system trust
store and removes it where it was; the mirror's mutual-TLS API can be
limited to named certificates (--api-client); init.log no longer keeps the
administrator's password; a warning when auth is off on a public listen;
the recorded actor is sanitised and sudo-aware; Ray's dashboard is off, and
the remaining engine gaps are documented.
reconciler.go and agent.go are split by topic, the status mapping of each
failure is tested, and the plan's checklist is brought up to date.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every upgrade so far has been a script of mine copying a binary to each
machine over ssh, restarting it and polling until it came back — which is
the job, but not something the product could do, and which needed ssh into
every node.
`sudo llmgw ctl upgrade VERSION` on the control plane does it. This machine
first: the release for its platform, checked against the release's
checksums, the old binary kept beside it, llmgw-server restarted, and a wait
until every node that was ready has reconnected. Then each node in turn: the
control plane tells its agent the version over the stream it already holds;
the agent fetches and checks the release itself, swaps it in with a rename,
answers, and exits with status 75 so systemd starts the new binary. Engines
keep running, as for any agent restart, and are adopted again.
A node is done when it is back on the new version, ready, with as many ready
replicas as before; one that is not within the timeout stops the run before
the next node, so a bad release reaches one machine. Nodes already on the
version are skipped, so running it again resumes.
The fetch-and-verify is one package now, shared with onboarding's idea of
what a release version looks like.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The first backup on the live control plane was written, and the control
plane reported none: /var/backups/llmgw already existed, made by an earlier
move to PostgreSQL for its own dumps, owned by postgres and 0700. MkdirAll
leaves an existing directory as it is, so the archive landed where the
llmgw user could not list it, and llmgw_backup_last_success_timestamp_seconds
said 0.
Opening that directory up would have exposed the dumps already in it, so
backups get their own, /var/backups/llmgw-server. And `llmgw server backup`
now warns when the directory it wrote to is one the control plane cannot
list, rather than leaving that to be found from an alert.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
There were no backups: docs/install.md gave a pg_dump command, and nothing
ran it. The store holds every key, person, quota and usage record, the models
added at runtime, and now the secret each engine's key is derived from; the
certificate authority's key sits in /etc/llmgw/pki beside it. Losing the disk
lost all of it.
`llmgw server backup` writes both to one archive: pg_dump's custom format, or
a VACUUM INTO snapshot for SQLite, plus /etc/llmgw. The archive holds the CA's
key, so it is 0600; the directory is listable, so the control plane — not
root — can see when the newest was written. pg_dump is given the connection
through its environment, since a DSN carries the password and a process's
arguments are readable by every user. Written beside its name and renamed,
so an interrupted backup is never the newest; the newest 14 are kept, and
nothing the command did not write is ever removed.
llmgw-backup.timer runs it nightly and catches up after downtime. The control
plane publishes the newest archive's time, and BackupTooOld fires past 26
hours — or when there has never been one. The install guide now covers
restoring, and says to copy archives off the machine, since they share its
disk.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Adding a node was deploy/bootstrap/onboard-node.sh, which is not installed
anywhere and wanted the node's name, the control plane's address and a
dozen flags. It is now in the binary, run on the control plane:
sudo llmgw node add ubuntu@10.0.0.21
The node id is the machine's hostname, and the address its agent dials is
this machine's on the route to it, checked against the server certificate.
It installs llmgw there (the release for its architecture, fetched and
checked by the node; uname rather than dpkg, so RHEL works), runs doctor
there and stops on a problem before changing anything, issues and installs
the certificate, writes agent.yaml and the unit, starts the agent, and waits
for it to join and install llama.cpp. The authority is read first, so a
missing one is said before the node is touched.
The steps are events. In a terminal they draw a checklist (bubbletea) that
asks what a script cannot: which fabric each group of active adapters is on,
grouped by the subnet manager or RoCE network they report, and whether to go
on past a problem. Elsewhere, one line per step.
Found by running it on an empty VM: ssh's "Permanently added" warning on
stderr was read as the machine's hostname, so standard error is now kept
apart; and a node added again while cordoned never counted as joined, so
joining now means a heartbeat since the restart.
llmgw node remove forgets a lost node, which otherwise stays listed until
the control plane restarts. ctl nodes no longer lists vLLM as skipped on
every CPU node. The quick install, install and onboarding docs use it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
install.md explains every choice and is long for someone who wants the
commands. quick-install.md has only those, and what to check after each:
the binary, server init, onboarding GPU nodes (with doctor first and the
script fetched from the release tag, since it is not installed anywhere),
a model from the mirror's explorer, keys and editors, and Caddy. Every
button and path it names was checked against the pages. quickstart.md is
now said to be the from-source, one-workstation path it is; README and the
about page point at the new guide.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
vLLM was installed by hand and named in each agent's configuration. The
agent now installs it: a Python 3.12 environment built by uv, which is
itself a GitHub release checked against its checksum, from PyPI resolved as
it stood a day after that vLLM was published and installed with
--require-hashes. The resolved list is kept in the environment. The
environment is built relocatable under a scratch name and renamed into place
once it runs, like an archive.
The default is 0.30.0, whose flags were checked against the ones llmgw
builds. It is built for CUDA 13, so it needs driver 580 and a Turing GPU or
newer, and 25 GB free while it installs; a node lacking one says which, and a
node with no GPU skips it without being shown as a fault. gcc comes with it
because Triton compiles its launcher at the first model load. SGLang
installs the same way when the control plane names a version. llmgw doctor
reports whether vLLM can be installed on the node.
Checked by installing 0.30.0 this way on a test node: 7.7 GB, torch 2.13 for
CUDA 13.0, importing cleanly; the node has no GPU, so vLLM was not started.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The changelog's unreleased section had stopped at refusing unloadable
weights; it now covers the rest: telling independent fabrics apart,
their subnet managers and maps, NVLink on every build and across nodes,
the multi-node launch fixes and vLLM's new default, doctor, node load,
editors and the Responses API, the chat and mirror pages, a person's own
use, the admin pages, the new metrics, and how to upgrade.
docs/fabrics.md is new, for a site with InfiniBand or NVLink: what
llmgw does with them, what each node and fabric needs first, declaring
fabrics, checking a node with doctor, what the pages and metrics show
and which to alert on, what an engine is given, and where to look when
a replica spanning nodes will not run. It says plainly that none of it
has yet run on such hardware.
install.md, node-onboarding.md, onboarding.md, the README and the
example agent configuration no longer say the portable binary cannot
see NVLink or that a fabric's name cannot be checked, and describe the
account page and connecting an editor.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
server init now also:
- runs a model mirror beside the control plane (/srv/models, bytes on
:8081 at an address nodes reach, management API on loopback so it
needs no certificate) and adds it to server.yaml;
- makes the machine a node of its own control plane: issues its agent
certificate, writes agent.yaml labelled as the control plane, starts
llmgw-agent, and waits for it to register and install llama.cpp;
- checks for NVIDIA GPUs with nvidia-smi, and reports a card on the PCI
bus whose driver does not answer rather than ignoring it;
- fetches Qwen2.5 0.5B (Q4_K_M) into the mirror and adds it to the
catalog pinned to this node: a llama.cpp profile per GPU class found,
with a CPU profile as the fallback, then waits for it to serve.
--no-mirror, --no-node and --no-model leave these out. Every step looks
before it acts, so a rerun finishes what an interrupted one started.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Installs PostgreSQL with the machine's own package manager when it has
none (initdb with password auth on the RHEL family, whose default is
ident), creates the llmgw role and database with a generated password
kept in /etc/llmgw/secrets/store-dsn, and checks the address connects.
On a machine already on SQLite it stops the gateway, copies the store
so accounts and keys survive, points auth.store at the new address and
starts it again. A move that fails starts the gateway on SQLite again.
server.yaml is edited as text so its comments survive, and kept beside
it. A PostgreSQL init did not install is not resized: it may serve
something else. It refuses to move one PostgreSQL store into another.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
install.md said the client port belongs behind a reverse proxy and gave
nothing to put there. deploy/proxy/ has both, attached to every release.
What they get right is what a first attempt gets wrong: streamed replies
are not buffered, the read timeout outlasts a model's silence before its
first token (nginx's 60s default is a 504), the body limit matches the
router's, and X-Forwarded-Proto and -Host are set by the proxy rather
than passed through — llmgw marks session cookies Secure from the first
and builds the landing page's base URL from the second.
The Caddyfile was validated and run against a streaming upstream: events
arrive as sent, and forged forwarded headers are replaced. nginx uses
`listen ... http2` because `http2 on;` needs 1.25.1 and Ubuntu 22.04 and
Debian 12 ship older.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
`llmgw server init` does what install.md's steps 2 and 3 asked of the
operator: the llmgw user and directories, a certificate authority, the
control plane's certificate for every address the machine has, a
configuration, the default project, an administrator with a generated
password, and the systemd unit — then starts it and waits for /health.
Every step looks before it acts, so a second run changes nothing, and
--host later reissues the certificate with the old names kept. It never
overwrites server.yaml or an authority, and leaves alone certificates
issued by an authority kept on another machine.
The configuration install.md showed could not start: an empty catalog
without control.enabled is refused as having nothing to serve. The one
init writes sets it, and a test holds that.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Read start to finish as somebody standing up a new cluster, it sent them
down a path we no longer take.
The mirror was absent from all 821 lines, and the page still said that
distributing weights was "a known gap, not an oversight" — so the reader
would have gone away to solve, by hand, the problem the binary in front of
them solves. It is step 4 now, before the nodes, because where a node
fetches from is the difference between minutes and a day, and on a customer
network the nodes often have no route out at all.
Two of the listed gaps had been closed and were still listed: a fetch does
report its progress to the control plane, and there is an HA pattern. A
gap somebody believes in is worse than one they discover.
Added what this week made obvious was missing. Upgrading, including the
zero-byte copy that ssh -n silently produces and the size check that is the
only thing between that and a fleet on a truncated binary. Backing up, in
the order that matters, and the fact that API keys are hashed and do not
come back. Pricing in the first configuration, because a budget without a
price is displayed, counted against nothing, and never binds. And scale,
bench and the source check, none of which were written down anywhere.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Neither had a page. "We have GPUs, now what" was left to the reader, and what
llmgw stores was scattered across four documents and a changelog — which is
the same as not being written down when somebody is doing a security review.
docs/data.md describes the behaviour and leaves the legal mapping to whoever
holds the obligation. Every claim in it was checked against the source, and it
names where to look for each: usage rows carry no text field, keys are stored
hashed, conversations are off until a named project turns them on and cannot be
turned on without a retention window.
It also states the egress exactly, which took reading every outbound call to
say honestly: nothing on the serving path leaves the cluster, and the two
things that can reach outside — fetching weights, and ctl plan reading a model's
architecture — are both started by an operator. Weight fetching points wherever
you tell it, including an internal mirror.
docs/choosing-models.md is the opinion llmgw itself does not have. Three jobs
rather than one model, with the third — embeddings — being the one that gets
forgotten and the one that decides whether the deployment can answer questions
about your company or only questions in general.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The last piece of HA, and it is a deployment pattern rather than code: llmgw
picks one leader and agents follow it, but client traffic arrives in front of
both and llmgw is not in front of anything.
Writing the pattern found a bug in the pattern. The first draft said to
health-check `/health` and claimed a standby still routes to replicas the
leader placed, so sending it traffic was fine. It does not. A standby's
reconciler places nothing and it holds no agent streams to adopt from, so it
publishes an empty endpoint list — a load balancer following that advice would
have sent half the traffic to something that answers every request with 503.
So there is now a `/ready` that means "this process has somewhere to send a
request": 503 on a standby, 503 on a leader whose replicas have all failed,
200 when there is something to serve. `/health` keeps meaning the process is
up, which is a different and less useful question. Both are open, because a
load balancer has no key.
There is a gap during failover where neither passes — three seconds when the
leader stops cleanly, up to the lease TTL when it does not — and the document
says so rather than implying otherwise. Traffic cannot go to a control plane
that has nothing to route to.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was no reference. Four narrative documents explain how to install and
operate llmgw, and the only way to learn that `lease_ttl` exists was to read
the one that happened to mention it — or the struct.
Written by hand it would have been wrong within a fortnight. Six settings
arrived in the last two weeks alone: store, servers, lease_ttl, the two
retention windows, min_disk_gb. A reference nobody maintains is worse than
none, because it is the same list with the setting you came to look up quietly
missing from it.
So it is derived from the structs, with the field comments as the descriptions
— which means the only way for it to be wrong is for the code to be wrong, and
it makes the comments worth writing twice over. 1163 lines, 24 blocks, 87 keys.
Two tests keep that true after everyone has forgotten this exists. One
regenerates and compares, so adding a setting without regenerating is a
failure rather than an omission — checked by adding one and watching it fail.
The other walks the tree for koanf tags and fails on a configuration file
that is in neither the source list nor the short list of structs an operator
never writes, because a list is a thing to forget.
The generator finds the repository from its own path rather than the working
directory, so `go generate ./...` produces the same document from anywhere.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The store stops being a file on one machine, which is what stood between this
and more than one control plane.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The section leads with why SQLite is still the default rather than with how to
leave it: one file, no server, and a request path that barely touches it. Two
things change that, and both are recent — conversations and notes read and
write on every turn, and a second control plane is impossible while the store
is a file on one machine.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The gap list said a fetch "reports no progress". It does — the agent logs a
percentage as it downloads, with a comment saying a replica pulling a hundred
gigabytes and one that is wedged look identical from the control plane.
That comment describes the real gap, which is narrower and more useful than
the one written down: the progress exists, it just never leaves the node, so
the operator looking at ctl or the dashboard still cannot tell those two
apart.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Seventeen commits since v0.6.0, and not a phase of the roadmap — which is why
the table leaves the phase column empty rather than inventing one. What landed
is what a cluster needs to be lived with rather than demonstrated: somewhere
to keep a conversation, somewhere to keep a note, and an honest answer about
the disk underneath both.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The section opens on the default rather than on the feature, because the
default is the part someone deploying this needs to be sure of: nothing anyone
types is stored until a project asks, and the command to ask is an act rather
than a setting.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A command that deletes data needs its limits written down more than its
syntax. The section says plainly that llmgw only takes back files it fetched
itself, and then says the uncomfortable half out loud: that this means it will
not reclaim the storage most likely to be filling your disk, because weights
you placed yourself are yours to remove.
Also records the dash in the DISK column as "not asked" rather than "no
space", which is the reading that matters when a node will not schedule.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
It said "there is no distribution mechanism yet" and then, two paragraphs
later, documented the distribution mechanism. The known-gaps list — the part
someone reads at 2am — still said "no cache, no fetch, no progress". Fetching
landed in this cycle; both sentences were written before it and left standing.
node-onboarding.md carried the same claim.
What is actually still missing is narrower and worth naming precisely: a fetch
reports no progress, so a first start on a large model looks like a hang, and
every node fetches its own copy, so eight nodes on a new model are eight
downloads from one origin.
Also: the dashboard now shows who is using the cluster and what they are
running, which the sentence describing it did not mention; and one sentence
about idle timeouts had lost the words that made it a sentence.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The metrics were exported and the only way to look at them was to stand up
Prometheus beside the binary. For a project whose whole shape is one static
binary with no dependencies, that is the wrong default: the common case is
somebody wanting to know how full their cluster is, and that should not cost
a monitoring stack.
/admin now shows capacity per GPU class with a usage bar, free whole nodes
per fabric — the number that decides whether a replica spanning nodes can
land, since free cards on the wrong fabric are not capacity for a model that
needs this one — and which replicas are holding hardware nobody is asking
for. Nodes gain their memory and how many model files they can see.
Taken from the same Summary that ctl status uses rather than derived in the
page, because two implementations of the same arithmetic drift apart and the
one on the screen is the one people would believe.
A faulty card counts as neither used nor free, matching the scheduler.
Counting it free is how a cluster looks like it has room it does not, and a
page that disagreed with placement about that would be worse than no page.
Prometheus stays supported and is now the answer to a narrower question:
history, alerting, several clusters at once. The dashboard shows what is
true now and keeps nothing, which is the honest division between them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The metrics existed and nothing could record them. They shared a listener
with pprof, the management API and the dashboard, and pprof can dump the
process heap where API keys sit in plaintext — so a fleet could be monitored
or keep its heap private, and not both. That is not a choice anyone should
have to make, and it is why every previous answer to "how do I scrape this"
was a tunnel.
metrics_listen serves metrics and health and nothing else, so it can be
reached from wherever Prometheus lives. Metrics stay on the admin listener
too: adding a door is not the same as moving one, and anything already
scraping keeps working.
While there: admin_listen defaulted to :9090, every interface, with a
comment beside it saying not to expose it. A default that contradicts its
own documentation is a footgun, so it is loopback now and someone who wants
otherwise says so.
The handler is nil until the registry is built, and says so rather than
serving an empty page. Ordering has already caught this exact area once —
a collector registered after AdminHandler published nothing at all, silently
— and an empty scrape target looks like a working one.
deploy/monitoring/prometheus.yml has a starting configuration and the alerts
worth having: a node that has stopped reporting, a model that should be
serving and is not, a card kept out of placement by a fault, and a fleet
holding models nobody is asking for.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
On-demand loading worked right up to the moment the cluster was full and
then stopped meaning anything: the model somebody had just asked for could
not be placed, and the cards were held by a model nobody had asked for in an
hour. That is the only situation in which any of it matters.
Deliberately narrow. Only idle replicas of other on-demand models — a model
at min 1 is a promise that it stays up, and breaking it to serve a different
request would make every guarantee in the catalog conditional on load. Only
single-node replicas, since evicting a multi-node one frees several machines
and costs tens of minutes to rebuild. One at a time, because freeing
everything that qualifies would empty a cluster for a request that needed
one node. And never a replica with no reported usage: after a router restart
that is all of them, and unknown is not idle.
Writing the test found a gap the feature had rather than a bug in the test.
Two on-demand models on a cluster with room for one would take turns
evicting each other, at a model load per cycle — worse for both than one of
them waiting. A replica is now safe from eviction for a few minutes after it
starts serving, which bounds that.
Which in turn found a latent bug: deriveState stamped ReadyAt with
time.Now() inside a package that has an injectable clock, so the new
lifetime rule could not be tested at all — the fake clock said three minutes
and the wall clock said eight months. Invisible while nothing compared
ReadyAt to anything.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The feature the idle tracking was for. A cluster with more models than GPUs
can now hold every model in its catalog and only the wanted ones in its
cards: min 0 with a max above it means load on demand, and once nobody has
asked for idle_timeout the replicas stop and the cards go back.
No new field, because min 0 max 1 already reads correctly — keep none
running, allow up to one. Which made min 0 max 0 meaningful for the first
time: configured and deliberately not running. normalize() used to fill a
zero max in, on the reasonable ground that a model with no maximum cannot
run; that would now silently turn every switched-off model into an
on-demand one, including the ones already written that way in this
cluster's own configuration. A model with no scaling stanza stays off for
the same reason — defaulting it to on-demand would let any client request
start a 70B nobody intended to run.
The one real decision is what happens to the request that triggers a load.
Refusing makes the feature useless, since the first request for every model
fails; waiting without a bound turns a request into a job, and a client that
budgeted thirty seconds holds a connection for ten minutes. So it waits
briefly and then says what is happening — 503 with Retry-After and "the
model is loading", which is a different answer from "no endpoint serves
this" and produces different client behaviour. An unknown model still gets
the unknown-model error, or on-demand would turn every typo into "loading"
and the useful message would disappear.
Use keeps a model alive rather than only the request that started it,
without which the timer measures time since the load and a busy model is
evicted on schedule however hard it is working.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three places where the system took the infrastructure's word for something
and would have failed silently on real hardware.
A declared fabric adapter that is not on the node. Fabric membership has to
be configuration — OpenSM's default subnet prefix is identical on every
fabric, so a matching prefix proves nothing — but taking it entirely on
trust was a different mistake. A typo, or a card that enumerated
differently after a reboot, produced NCCL_IB_HCA naming an adapter that
does not exist.
An adapter with no ACTIVE port. Physically linked with no subnet manager
sits in INIT: it looks healthy to anything counting adapters, carries no
data, survives a reboot, and looks like a software problem. Discovery had
recorded port state all along and placement only ever asked whether a
fabric had any adapters listed.
Both end the same way — the replica starts, passes its health checks, and
hangs on its first all-reduce, which is the failure the fabric invariant
exists to prevent arriving through a door nobody was watching. The agent
now compares declared against discovered, where both are visible for the
only time, and does not advertise a fabric that cannot carry traffic. The
reason is logged on the machine that knows it, because the operator who
typed the adapter name is the only person who can fix it. The fabric's rate
is the slowest active link, since a collective runs at the speed of its
slowest.
And NVLink topology, which no published binary could see. NVML is its only
source, the bindings need cgo, and CGO_ENABLED=0 is deliberate so the
portable binary runs on an older libc. The cost was silent: tensor groups
fall back to the lowest free indices, which is right on an SXM board with
NVSwitch and wrong on a PCIe box where bridges pair specific cards — a
group straddling two of them runs several times slower and nothing says so.
Releases now also publish an nvml build for those nodes, and
llmgw_gpu_nvlink_peers makes the difference visible: a flat zero across a
fleet means the topology is unknown rather than absent.
A test that declared mlx5_0 and mlx5_1 on a node whose sysfs held neither
had been passing, because nothing compared the two.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The default arrangement is the simple one: every node keeps its own copy.
So a source naming a remote file is now fetched before the engine is
launched, rather than left to fail as a missing path minutes into a load.
Eight shards of a tensor-parallel replica start at the same instant and
want the same file; they produce one download. An interrupted fetch resumes
instead of starting over, which matters at forty gigabytes, and the
download is written beside its target and renamed only when complete — a
partial file under the model's name would be found by the next scan,
reported as present, and handed to an engine that fails on it.
The exotic arrangements need no code, which is the point. Weights on a
Lustre mount or an image exported over the fabric are simply present, and
present means nothing is fetched. The escape hatch from a distribution
mechanism is a distribution mechanism that checks first.
Wiring it up caught a regression in my own design. A bare model id —
"Qwen/Qwen2.5-7B-Instruct" — is a documented source that vLLM and SGLang
resolve themselves, and treating every non-remote source as a local path
made every existing vLLM profile fail. Only an absolute or explicitly
relative path is now checked as a file; anything else is passed through
untouched. Three agent tests failed on the same change for a better reason:
they used "/models/x", and the new check is exactly the one that catches
that.
Whole repositories of shards are deliberately not fetched here. vLLM and
SGLang do that well already, and a second snapshot downloader would be work
spent arriving back where we started.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
