The agent's defaults said eth0 before normalize looked for the interface
holding the management address, so it never looked: a node on ens3 or
bond0 had NCCL and Gloo bootstrap over an interface it does not have. It
is now found, with eth0 only as the last resort.
The agent unit sets LimitMEMLOCK=infinity. RDMA pins what it registers,
and under systemd's 8 MB default an engine without the capability that
lets root ignore the limit fails its first collective over InfiniBand.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
GPUs were chosen by growing one card's NVLink peers, which took cards
each joined to that one but not to each other, and knew nothing of the
fabric. Every set of the right size is now scored - a node's handful of
cards makes that cheap - by NVLink pairs within it, then for a
multi-node shard by how near each card is to one of the fabric's
adapters (PIX before SYS), then by NUMA nodes spanned, with the lowest
indices breaking ties. A node without topology still gets the lowest
indices.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
vLLM's mp launch gave workers no rank and the leader's address as their
own VLLM_HOST_IP; every node now runs serve with --nnodes, --node-rank,
--master-addr and --master-port, and the workers --headless, as vLLM
documents it.
NCCL_SOCKET_IFNAME and GLOO_SOCKET_IFNAME were the control plane's one
interface name for every node; the agent already worked out its own -
the one holding its management address - and now uses it.
On RoCE, NCCL's default GID index is a v1 link-local entry on most hosts,
which does not route. The agent finds the RoCE v2 entry holding an IPv4
address and passes it as NCCL_IB_GID_INDEX; InfiniBand gets none.
A worker whose only process needs the leader - a Ray join - started
without waiting for it, since the wait was honoured for sidecars alone;
it found no head, exited and failed the replica. It now waits.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A port's state was read once, at registration: one that went down later
left its node on the fabric, offered multi-node work that would hang on
its first collective, until somebody restarted the agent. The agent now
re-reads every IB and RoCE port on each heartbeat - state, physical
state, rate and the error and data counters - and when the fabrics it
can carry traffic on change, says so once; the control plane replaces
the node's fabrics, which is what placement reads.
The ports are exported as llmgw_ib_port_active (with state and
phys_state, so an unplugged cable reads differently from a missing
subnet manager), llmgw_ib_port_rate_gbps and llmgw_ib_port_counter_total,
and a node's page lists them with the fabric each carries and any error
counter that is not zero.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The release binary is built without cgo, so without NVML, and NVML was
the only source of NVLink peers: on every NVLink machine the portable
agent reported none, and tensor-parallel groups were chosen by GPU index.
Discovery now reads `nvidia-smi topo -m` after either path, for the
NVLink peers (kept from NVML where it gave them), each card's NUMA node,
and how far each network adapter is from it - PIX, PXB, PHB, NODE or
SYS - which travels to the control plane as nic_paths for placement to
use. Both the NICn-with-legend layout of newer drivers and the
adapter-named columns of older ones are read.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Grouped by name, the four qwens in the mirror were four groups of one
and the table looked unchanged. A family is the letters a name starts
with - qwen2.5-coder-7b and qwen3-8b are qwen, llama-3.1-8b is llama -
and a family of two or more models gets a heading with how many and how
much disk; a family of one stays a plain row.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Each model is a heading - how many versions, how much disk, and whether
one is downloading, failed or in use - with its versions beneath it.
Sorted by state alone, versions of one model were scattered down the
table around unrelated downloads. A model with a download or a failure
still comes first.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A bytes axis read "20.0BB" - compact's billion with a B after it - and
a quiet node's CPU axis read 0% all the way up. Sizes are now binary like
every other on these pages, a percent under ten keeps a decimal, and a
window of an hour is "the last hour" rather than "1h0m0s".
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The nodes are CPU machines, and nothing measured how busy one was: the
heartbeat carried GPU memory and disk, and llmgw_node_memory_bytes is the
total. The agent now reads /proc between heartbeats - CPU across all
cores, load averages, memory used and available, and each engine's
process group's cores and resident memory - and sends it with the
heartbeat. Nothing new runs on the nodes and Prometheus needs no new
target: the control plane exports it with the rest of the fleet, as
llmgw_node_cpu_utilization_percent, llmgw_node_cpus, llmgw_node_load,
llmgw_node_memory_used_bytes, llmgw_node_memory_available_bytes,
llmgw_shard_cpu_cores and llmgw_shard_memory_bytes. A measurement more
than a minute old is left out rather than repeated.
The cluster page's node table shows CPU and RAM as bars, and each node
links to a page of its own: current CPU, RAM, load and disk, the engines
on it with their cores and memory, and an hour of CPU, RAM, load,
per-model cores and memory and queue depth from Prometheus. It refreshes
in place.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Capacity, nodes, models and replicas come first, then a rule, then
traffic, people, what is being run and usage. The people and per-model
bar charts, which repeated the tables under them, are gone.
The page swaps its content every few seconds with htmx instead of
reloading, so a reader keeps their place, and it stops while the tab is
hidden. Without script the meta refresh still reloads it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The account page (was "Your keys") shows the last thirty days of the
person's own use - requests, tokens, spend and tokens per day - and, for
each place they can spend, what they used there and a bar for every
budget that applies: their own, a per-member one, or a project's, team's
or organization's shared with others, named as such. Bars turn amber past
four-fifths and red when spent. It sits under the keys now, above the
password form.
Spend is read from the usage table for the pool each limit counts
against, from the start of its period.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A reply streams as text as before; once it ends, the server renders it
with the model card's rules - raw HTML dropped, dangerous links emptied,
images as links - and the page places that. Headings, lists, tables and
code blocks read as they were written.
Maths outside code is set aside before the markdown, so its underscores
and asterisks are not taken for emphasis, and marked for the page to
typeset. KaTeX no longer searches the reply for delimiters, which took
"$5 and $10" for a formula. A reply that cannot be rendered stays
plain text, with the old search.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The cluster page's usage table showed session:usr-... and proj-... ids;
it now shows the person's email, whether it came through a key or the
chat page, and the project's path or "personal". Times read "3 hours
ago" rather than "3h7m25s ago", everywhere.
The usage page's legend gave each line's latest slice, which was 0
beside a table of hundreds; it gives the window's total, and the table's
numbers are right-aligned like everywhere else.
The PostgreSQL placeholder rewriting skips line comments. An apostrophe
in one opened a string that swallowed every placeholder after it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Clicking a repository in the explorer shows its card and files in a row
beneath it, fetched from /admin/mirror/repo, and a second click closes
it; the list stays where it was. The link still leads to the
repository's own page without script or with a modifier key.
A repository's files now take one line each, and an importance matrix
published beside the quantisations is marked as not a model rather than
offered for fetching.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Opening a repository shows its model card, rendered from markdown, with
its base model, license and languages above it; the listing names each
repository's base model and license under its name. The card is a
stranger's text: raw HTML and dangerous links are dropped, images are
shown as links rather than loaded, and relative links point into the
repository.
Adds github.com/yuin/goldmark for the markdown.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The mirror page lists Hugging Face's trending models without anything
typed, with views for what is new this month and what is most
downloaded, filters for task and size, and each repository's parameter
count, architecture, context length and age from one registry call.
"Only what these nodes can run" is on by default and leaves out
architectures the nodes' llama.cpp cannot load, and adapters, saying how
many it hid. Listings are remembered for ten minutes.
The stylesheet is now linked by a hash of its content. It is cached for
a day, so under a fixed name a deploy's new styles reached people a day
after its markup.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A message to a model that was not running waited 45 seconds and came
back as "the model is loading; retry shortly". The page now asks for
the model with /chat/wake, shows where it stands from
/chat/model-status - waiting for a node, downloading with bytes and a
percentage, starting the engine - and sends the message when it answers.
Also passes the nodes' llama.cpp build to the pages, which the mirror's
checks for loadable weights needed and were never given.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
kev-0.5b declared architecture "ggmlc", which llama.cpp does not know.
It was fetched, catalogued and placed, and failed on every node — while
the answer was in the first kilobytes of the file.
internal/gguf reads a GGUF header (v1-v3) up to general.architecture,
over HTTP with a Range request, and judges it against the architectures
llama.cpp b10991 lists in LLM_ARCH_NAMES (152, copied from the source at
that tag); a vision projector is not a model on its own. An unknown
build is not refused, only not vouched for, and a header that cannot be
fetched does not refuse anything.
Checked in three places: the registry listing (from the architecture
Hugging Face reports with expand[]=gguf, so one request per repository)
shows it and withholds the fetch button; a fetch reads the header and
refuses unless "fetch anyway"; and the model service refuses to add
such a model, which covers the serve form and ctl model add.
Tried on real files: the mirror's qwen2.5-1.5b (qwen2) and bge-m3
(bert), and Qwen2.5-0.5B on Hugging Face through its redirect.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The run mode sat after a tag repeating the model's name, so each selector started where the name ended and long names wrapped the row. The column is now Runs: one fixed-width selector per model, with the catalog's name above it only when it differs from the mirror's. Sizes no longer wrap.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every mirrored version was in use on the cluster, so the serve link — hidden for a version already served — appeared nowhere, and changing a model between always on and on demand needed the CLI. Each model using a file now shows how it runs and offers always on, on demand (15 minutes idle) or stopped, through the model service's Scale; models declared in server.yaml show their mode and where to change it. The serve form asks the same question.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
With every organization removed, /org answered an administrator with a 404 while the menu kept offering it. An administrator now gets a page saying there are none and where to make one, and the menu offers Organizations only when there is one to show (to a manager, one they manage).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A member of an organization can be its manager (users join --org X
--role manager, or the star on the people page). /org shows a manager
their organization: usage over time by team, project or person, its
teams and projects with limits and 30-day spend, and its members, each
once with every place they belong. Read-only; administrators see every
organization. Only what was spent in the organization's projects is
shown — UsageBucketsIn filters by where a request was spent.
Found on the way:
- Usage by org, team and project, and the quotas page's spend, joined
through api_keys and so dropped every chat-page request, which has no
key row. They now read usage.project_id; keys no longer move between
projects, so the two agree.
- /org was not exempt from API-key auth and answered browsers with a
JSON error. A router test now drives the session pages through the
middleware; it fails without the fix.
Membership queries, org-scoped usage and the reworked joins pass the
PostgreSQL store tests on 16.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A person was in exactly one project, chosen at approval, and could not
use the gateway before that. People belong to several teams and to
none, so:
- memberships: a person is a member of any number of organizations,
teams and projects, or of none; a member of a team or organization
may use every project in it. users join/leave, and the people page
adds and takes out from either side. Approval no longer needs a
project.
- personal keys: a key may have an owner and no project. /profile asks
what a new key spends against; keys create --for does the same.
auth.personal sets default limits for personal use or turns it off.
- limits for people: --scope user holds a person across every key,
project and the chat page; --per-member on an org, team or project
gives each member their own share there. Resolution folds the tree,
each per-member limit for the key's owner, and the owner's own, each
in its own pool; the quotas page shows and sets both.
- the chat page asks what to spend against and checks it per message.
- usage records who spent it, so a person's budget survives a restart,
personal use and chat included; restore no longer drops rows with no
project.
Leaving a project revokes the keys made there unless another membership
still reaches it. A personal key keeps no threads or notes, which are
per project, and is refused rather than dropped into the anonymous pool.
Migration 0010 turns each person's project into a membership, adds
usage.user_id and backfills it, and makes api_keys.project_id nullable.
SQLite cannot drop NOT NULL, so the table is rebuilt; that needs foreign
keys off, which only works outside a transaction, so migrations marked
for it run on one connection with foreign_key_check before commit.
Tested on a populated SQLite store with claims, and on a copy of the
cluster's PostgreSQL store: 5 people became members, 17 keys and 3
claims intact, 145 of 577 usage rows and all 69 chat rows attributed.
The PostgreSQL store tests, including new membership cases, pass on 16.
users move is gone; join and leave replace it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A branch with no limit of its own read "—" in every column while its
organization held it to sixty requests a minute. Each cell now shows
the strictest value from above, muted and marked with an arrow, naming
its source on hover and in the form's placeholders. A larger value set
under a stricter one says what caps it, for rate and token limits as
well as budgets; only budgets were checked before.
The table follows the people page: indented tree, tinted organizations,
spend with a meter that turns red past 90%, a form that opens and saves
without reloading (htmx, falling back to plain links), and the
explanations folded under "How limits combine".
Rows are built from the store's Tree, so a project sits under its team
by id. Matching the start of its path put projects under the wrong team
when two organizations shared a name.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The ranked bars said "8825 ms" beside a table that said "8.8 s". One formatter, humanLatency, now serves both, with a test.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- Numeric column headers are right-aligned like their values; the
stylesheet only aligned the cells, so "REQUESTS" sat nowhere near its
numbers in every table that has them.
- Ranked bars are an HTML grid instead of a fixed-width SVG: the value
had 96 units and "34.2k tok 114 req" did not fit, so the figure a
reader came for was the one cut off. Labels ellipsise in CSS and give
their full text on hover.
- Mean latency reads as 5 ms, 8.8 s or 1m 51s rather than 110695 ms.
- The "What the cluster is doing" charts sat between the Models heading
and the models table; they now come before it.
- A fleet without GPUs shows its CPU capacity — nodes ready, memory,
disk free for weights — instead of "No GPUs have been reported", and
a node with no GPUs shows "—" rather than 0/0.
- The profile's forms are a form's width rather than the page's, and
the quotas page's price warning is a boxed banner rather than a line
of red text.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The profile page rendered after a password change carries the CSRF
token set on that same response, but the menu's sign-out form was left
without one — and it is the first form on the page. The e2e test that
signs in, changes the password and uses the next form caught it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Each page wrote its own nav: the same links in a different order, some
missing, the current page sometimes listed and sometimes not. A shared
"nav" template now draws it everywhere — Cluster, People, Quotas, Usage,
Mirror for an administrator, then Chat, Your keys and Sign out — with
the current page marked by aria-current. Pages rendered outside the
account pages (the dashboard, node logs) get it through the request
context their admin wrapper sets. The admin listener's own page keeps
its metrics/pprof/api links.
The chat and key pages said "project proj-3kq7…"; they now show the
project's path, and the key page no longer repeats the email.
A test renders several admin pages and requires the same menu on each.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
server init now also:
- runs a model mirror beside the control plane (/srv/models, bytes on
:8081 at an address nodes reach, management API on loopback so it
needs no certificate) and adds it to server.yaml;
- makes the machine a node of its own control plane: issues its agent
certificate, writes agent.yaml labelled as the control plane, starts
llmgw-agent, and waits for it to register and install llama.cpp;
- checks for NVIDIA GPUs with nvidia-smi, and reports a card on the PCI
bus whose driver does not answer rather than ignoring it;
- fetches Qwen2.5 0.5B (Q4_K_M) into the mirror and adds it to the
catalog pinned to this node: a llama.cpp profile per GPU class found,
with a CPU profile as the fallback, then waits for it to serve.
--no-mirror, --no-node and --no-model leave these out. Every step looks
before it acts, so a rerun finishes what an interrupted one started.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Each held file took two table rows — seven columns and its address
beneath — so one sharded model filled the screen. A version is now one
line: model and version, the file or "N files", total size, and the
catalog models that use it. The address is behind a "copy address"
button (a set's is its directory, trailing slash included) and the
checksum is in the file's tooltip.
The model service reports which models name each source, so a file in
use shows who uses it and is not offered for serving a second time;
removing one warns that the catalog uses it. Downloads and failures
stay one line per file, since each has its own progress or retry. The
header counts versions, not files, and the nav calls /admin/users
"people" everywhere.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Nothing in the tree could be renamed, moved or removed, and an approved
person stayed in the project they were first put in — from the web or
the CLI. Now:
- /admin/users is the tree, each project opening onto its people and
keys, with approve, move, suspend, reinstate and remove for people,
revoke for keys, and add, rename, move and remove at every level,
plus a search over every account. Forms post and get the page back,
so it works with no script; htmx 2.0.11 (0BSD, embedded, checked
against npm's integrity hash) swaps only the changed part, keeping
open projects open.
- llmgw org tree/add/rename/move/remove, users move/delete, and
keys list --all.
- Projects, teams and organizations are named by id, path
(Acme/platform/gateway) or unique name; users by id or email. With a
single project --project can be left out, instead of defaulting to
"default", which a tree built from the web does not have.
Moving a person moves the keys they made; operator keys stay with the
project. Only empty things are removed, and never a project that had a
key, since usage history refers to it.
Also: keys init no longer reparents an existing team or project;
reinstating an account with no project is refused (a rejected signup
used to come back active and fail at its first key); rejecting a
signup removes it; keys limits --budget takes whole units like the
quotas page.
Checked in headless Chrome: a move from an open project updates the
page in place, keeps the project open, and the search still filters.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
server init --postgres failed on Rocky with postgresql-setup not found.
The PostgreSQL project's packages (PGDG) provide "postgresql-server",
so dnf can satisfy that name with postgresql16-server — whose unit is
postgresql-16.service, whose tools are in /usr/pgsql-16/bin and whose
setup script is postgresql-16-setup. The code assumed the distribution's
names for all three.
The server unit is now picked from systemd's postgresql* unit files, its
data directory read from the unit's own PGDATA, and initdb run directly
as postgres when that directory is empty, with password logins over TCP.
psql, initdb and the restart use what was found. The unit names and
PGDATA values in the tests are the ones Rocky's and PGDG's packages ship.
Not yet handled: a server initialised with ident for TCP logins, one on a
port other than 5432, and several PGDG versions where the highest is not
the one on 5432.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Installs PostgreSQL with the machine's own package manager when it has
none (initdb with password auth on the RHEL family, whose default is
ident), creates the llmgw role and database with a generated password
kept in /etc/llmgw/secrets/store-dsn, and checks the address connects.
On a machine already on SQLite it stops the gateway, copies the store
so accounts and keys survive, points auth.store at the new address and
starts it again. A move that fails starts the gateway on SQLite again.
server.yaml is edited as text so its comments survive, and kept beside
it. A PostgreSQL init did not install is not resized: it may serve
something else. It refuses to move one PostgreSQL store into another.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A configuration with no static endpoints and no models was refused as
having nothing to serve. That is every new cluster: under systemd the
unit restarted every two seconds and nothing ever listened on 8080.
With no endpoints a router-only process has nothing it could serve, so
such a configuration is now a control plane with an empty catalog.
Static endpoints without profiled models remain router-only.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
`llmgw server init` does what install.md's steps 2 and 3 asked of the
operator: the llmgw user and directories, a certificate authority, the
control plane's certificate for every address the machine has, a
configuration, the default project, an administrator with a generated
password, and the systemd unit — then starts it and waits for /health.
Every step looks before it acts, so a second run changes nothing, and
--host later reissues the certificate with the old names kept. It never
overwrites server.yaml or an authority, and leaves alone certificates
issued by an authority kept on another machine.
The configuration install.md showed could not start: an empty catalog
without control.enabled is refused as having nothing to serve. The one
init writes sets it, and a test holds that.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Seeing what a node did meant ssh through a jump host, to a machine whose
address nobody remembers, to read a journal only root may open. When an
engine install failed on a new node today, the reason was one line — and it
was four steps away from the page that said the install had failed.
Each node's row on the cluster page now links to its logs: the agent's own,
and every engine running there, chosen from the list the node itself sends.
The page can follow, refreshing every five seconds, and is on both the admin
listener and /admin. `llmgw ctl logs NODE [--source SHARD] [-n N]` reads the
same.
They travel over the stream the agent already holds, so they need nothing
the control plane does not already have: a request goes down, the lines come
back, and a reply that does not come in ten seconds is an agent too old to
answer. The agent keeps its last 2000 lines in memory as well as writing them
wherever it did before; the engines' last few hundred lines were already kept
and sent only when one crashed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Found by building a node from nothing. Its agent tried to install libgomp1
while cloud-init and unattended-upgrades were running apt, and failed with
"Unable to lock directory /var/lib/apt/lists/": DPkg::Lock::Timeout covers
dpkg's lock, not the one `apt-get update` takes on its lists. The node was
correctly kept out of placement and said why, and then sat idle for the ten
minutes until the next pass.
A package command that fails only on a held lock is now retried for up to
ten minutes, reported as "waiting for another apt-get to finish" rather than
as a failure. A pass that does fail is retried after a minute, backing off to
ten.
The weights directory is created when the agent starts rather than by the
first download, so a new node reports its disk from its first heartbeat.
Onboarding takes several control plane addresses, comma separated, so a new
node gets the same failover list as the rest.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The llama.cpp build is published from Ubuntu but is not Ubuntu's in any way
that matters: it needs glibc 2.34 and GCC 12's libstdc++, which Debian 12
and RHEL 10 have and RHEL 9 does not. Refusing every system not called
"ubuntu" turned away the first two for their name. The build is now tried
everywhere, and ldd decides.
What ldd says is made readable. "/lib64/libstdc++.so.6: version
`GLIBCXX_3.4.30' not found" is a system too old for the build, and is
reported as "libstdc++.so.6 is too old (needs GLIBCXX_3.4.30)", once —
rather than as a missing library that is plainly installed.
Packages are named per family — libgomp1 for Debian's, libgomp for Red
Hat's — and installed with whichever of apt-get, dnf, yum or zypper the
machine has, decided from ID_LIKE rather than a list of distribution names.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A node ran whatever llama-server somebody had once unpacked on it. Onboarding
checked an engine only when told its path, installed none, and a node with
nothing on it registered as ready. The one time that went wrong it went wrong
invisibly: the binary was there, executable, and could not start for want of
libgomp1, while the node reported healthy.
The control plane now says which engines every node has, at which version —
`engines:` in server.yaml, defaulting to the llama.cpp build this release was
checked against — and each agent converges on it. What only the node can
know, the node works out: the build for its processor and driver (CUDA with
its runtime when an NVIDIA driver is loaded, CPU otherwise), the OS packages
that build needs from its own package sources, and whether the result runs.
Every archive is checked against the sha256 GitHub publishes for it and
unpacked refusing any path or link that leaves the install directory. ldd
names a missing library outright; running `--version` proves the rest. Only
then is a directory marked installed, and it is checked again every ten
minutes, because an unrelated upgrade can remove a library.
An engine is reported only once installed and seen to run, or when the
agent's configuration names it on purpose — never because something happened
to be on PATH. The agent now says when it has probed, so an empty list means
"has none" rather than "too old to say": a node rebuilt from nothing is no
longer a candidate for every model at the moment it can run none of them.
Placement's refusal carries the node's own words, "installing: downloading
llama-b10991-bin-ubuntu-x64.tar.gz (16 MB)", and so do `ctl nodes` and the
cluster page, along with a warning for a CPU without AVX2 — the VM CPU type
that makes everything five times slower and nothing look wrong.
Agents also report their OS, architecture, vector extensions and NVIDIA
driver. Onboarding installs llmgw itself from the CI release for the node's
architecture, checked against the release's checksums, rather than copying
whatever binary the operator's machine had.
vLLM and SGLang are not installed yet: they are Python environments of
several gigabytes, not an archive, and still have to be named in the agent's
configuration.
Also fixes a race on the weights fingerprint, which two heartbeat loops touch
during a failover.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every agent file on the cluster named its own node, its own address, the
path to llama-server and two weight directories — different on every node,
kept true by hand, and not true: node05 carried "ram: 32g" beside a measured
31 GB, looked for weights in an empty directory while fetching them into
another, and the other four never looked in theirs at all, so four nodes
reported no disk. Growing a node's RAM would have gone unnoticed, because
memory was read once, at registration.
The agent now finds these out. Its id is the hostname, which is also the
certificate's common name. Its address is the one on the route to the
control plane, which the kernel picks for a UDP dial without sending
anything. llama-server is looked for on PATH and then where release archives
are unpacked. A missing nvidia-smi is a node without NVIDIA cards rather
than a failed discovery — which also cut discovery short before InfiniBand
on every CPU node not told to skip GPUs. Fetched weights land in
/var/lib/llmgw-weights by default, and wherever they land is always looked
in.
Memory is read on every heartbeat, and installed engines are probed again
each minute and sent when they change, so a VM given more RAM or a node
given vLLM becomes a candidate without anyone restarting its agent. The
control plane logs both changes, since "the node now qualifies" is the
answer to why a model placed an hour after it was asked for.
What is written still wins. The agent file shrinks to the control plane's
addresses and the TLS files, the same on every node; onboarding writes only
the overrides it was given. What cannot be found stays configuration: which
fabric an adapter is on, because OpenSM's default prefix is the same on all
of them.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The picker listed what had an endpoint, and a model at min 0 has none until
somebody asks for it. On this page a message is the asking, so an on-demand
model could never be started from here: mistral-nemo-12b-instruct sat in the
catalog, served by nothing, and absent from the one place someone went to
try it.
Such a model is now offered, marked "not loaded", with a note saying the
first message fetches the weights and starts the engine, and that until it
is up the reply is "the model is loading". A model stopped outright — max 0 —
is still left out, because nothing would load it and offering it would only
produce an error.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A report nobody can reach from the page they land on is a report nobody
reads. It sits beside accounts, on the client port only — the admin
listener's copy of this page serves pprof and has no session to authorize
anybody.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Usage could be read as a total — per key, per user, per model, per group —
and the cluster's own charts covered the fleet's hardware. What nobody could
see was when a group used it. "Research doubled its traffic three weeks ago"
is not a question a column of totals can be asked, and it is the question
somebody asks before a quota conversation.
/admin/usage draws requests, tokens and spend over a chosen window, grouped
by any level of the tenancy tree or by person. Person is not a level of that
tree — one person holds keys in several projects — so the grouping is its
own type rather than store.Scope with a fourth value nothing else could use.
Three things learned from the charts already here. Buckets are gap-filled
per group, because a line drawn straight across an absence claims traffic
that did not happen, and because the chart sums its tail by timestamp and
can only do that if every series shares a clock. Series are ranked by total
rather than by peak — a counter's question is "who used the most", and
ranking by peak would colour the team that made one burst over the team that
ran steadily all week. And the spend chart is replaced by a sentence when no
model has a price, rather than drawn flat along the bottom.
The bucket query groups by ordinal and was planned against the cluster's
PostgreSQL before being believed, with a test alongside the others that run
there: three reports have shipped working on SQLite and failing on it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The page imposes its own ceiling on a turn, so a playground request cannot
occupy a replica for ten minutes. It sent that ceiling regardless of the
model, and a model declaring a smaller max_output then answered every turn
with "max_tokens is 2048, but this model allows at most 1024" — the gateway
refusing its own page, over a number the person never chose and has no way
to change.
The ceiling is the page's to impose on itself and the model's limit is not
negotiable, so it sends the smaller of the two.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An endpoint advertises every name a client may address it by — the model's
own and all of its aliases — because that is what the router matches a
request against. The picker took those strings as they came, so one model
appeared as several: "coder", "coder-7b" and "qwen2.5-coder-7b-instruct"
were three choices, and choosing between them is a question with no answer.
Each name is now resolved through the catalog and the model listed once,
under its catalog name — the one that also appears in the model list, the
usage table and the logs.
/v1/models still lists the aliases separately, which is deliberate: there a
client is discovering which ids it may send, and every alias is one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The price table is written against the catalog. The name reaching the
accountant was the one the caller typed, so every request made by an alias
missed the table and fell to the default price — the same request charged
two different rates depending on which of a model's names the client
happened to be given.
With no default price that is zero, which is silent. With one it is a wrong
number, which is worse: a budget spent at the rate of whatever the default
says rather than at the model's own.
The router already resolves the alias, two lines above, to decide whether
the key is allowed the model at all. It now passes the resolved name to the
accountant as well, in the ordinary path and in the budget fallback.
The test charges a real request through an aliased name and compares the
figure with arithmetic it states itself, because the first version of it
asserted "more than the fallback" and passed with the bug in place.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The stale-profile flag could not fire where it matters. The control plane
keeps no record of a running replica across its own restart — it rebuilds
them from what the agents report — and adoption left the fingerprint empty,
so every replica read as unknown afterwards. That is precisely the moment
the catalog is most likely to have been edited: change the configuration,
restart the server, and nothing would ever say the fleet disagrees with it.
The fingerprint now travels with StartShard, the agent keeps it beside the
pool and fabric it already echoes back, and adoption takes the replica's
version from the agent rather than from the catalog it is about to be
compared against. An agent too old to report it sends nothing, which still
reads as unknown rather than as matching.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
