A bytes axis read "20.0BB" - compact's billion with a B after it - and
a quiet node's CPU axis read 0% all the way up. Sizes are now binary like
every other on these pages, a percent under ten keeps a decimal, and a
window of an hour is "the last hour" rather than "1h0m0s".
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The nodes are CPU machines, and nothing measured how busy one was: the
heartbeat carried GPU memory and disk, and llmgw_node_memory_bytes is the
total. The agent now reads /proc between heartbeats - CPU across all
cores, load averages, memory used and available, and each engine's
process group's cores and resident memory - and sends it with the
heartbeat. Nothing new runs on the nodes and Prometheus needs no new
target: the control plane exports it with the rest of the fleet, as
llmgw_node_cpu_utilization_percent, llmgw_node_cpus, llmgw_node_load,
llmgw_node_memory_used_bytes, llmgw_node_memory_available_bytes,
llmgw_shard_cpu_cores and llmgw_shard_memory_bytes. A measurement more
than a minute old is left out rather than repeated.
The cluster page's node table shows CPU and RAM as bars, and each node
links to a page of its own: current CPU, RAM, load and disk, the engines
on it with their cores and memory, and an hour of CPU, RAM, load,
per-model cores and memory and queue depth from Prometheus. It refreshes
in place.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The cluster page's usage table showed session:usr-... and proj-... ids;
it now shows the person's email, whether it came through a key or the
chat page, and the project's path or "personal". Times read "3 hours
ago" rather than "3h7m25s ago", everywhere.
The usage page's legend gave each line's latest slice, which was 0
beside a table of hundreds; it gives the window's total, and the table's
numbers are right-aligned like everywhere else.
The PostgreSQL placeholder rewriting skips line comments. An apostrophe
in one opened a string that swallowed every placeholder after it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The mirror page lists Hugging Face's trending models without anything
typed, with views for what is new this month and what is most
downloaded, filters for task and size, and each repository's parameter
count, architecture, context length and age from one registry call.
"Only what these nodes can run" is on by default and leaves out
architectures the nodes' llama.cpp cannot load, and adapters, saying how
many it hid. Listings are remembered for ten minutes.
The stylesheet is now linked by a hash of its content. It is cached for
a day, so under a fixed name a deploy's new styles reached people a day
after its markup.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The ranked bars said "8825 ms" beside a table that said "8.8 s". One formatter, humanLatency, now serves both, with a test.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- Numeric column headers are right-aligned like their values; the
stylesheet only aligned the cells, so "REQUESTS" sat nowhere near its
numbers in every table that has them.
- Ranked bars are an HTML grid instead of a fixed-width SVG: the value
had 96 units and "34.2k tok 114 req" did not fit, so the figure a
reader came for was the one cut off. Labels ellipsise in CSS and give
their full text on hover.
- Mean latency reads as 5 ms, 8.8 s or 1m 51s rather than 110695 ms.
- The "What the cluster is doing" charts sat between the Models heading
and the models table; they now come before it.
- A fleet without GPUs shows its CPU capacity — nodes ready, memory,
disk free for weights — instead of "No GPUs have been reported", and
a node with no GPUs shows "—" rather than 0/0.
- The profile's forms are a form's width rather than the page's, and
the quotas page's price warning is a boxed banner rather than a line
of red text.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Nothing in the tree could be renamed, moved or removed, and an approved
person stayed in the project they were first put in — from the web or
the CLI. Now:
- /admin/users is the tree, each project opening onto its people and
keys, with approve, move, suspend, reinstate and remove for people,
revoke for keys, and add, rename, move and remove at every level,
plus a search over every account. Forms post and get the page back,
so it works with no script; htmx 2.0.11 (0BSD, embedded, checked
against npm's integrity hash) swaps only the changed part, keeping
open projects open.
- llmgw org tree/add/rename/move/remove, users move/delete, and
keys list --all.
- Projects, teams and organizations are named by id, path
(Acme/platform/gateway) or unique name; users by id or email. With a
single project --project can be left out, instead of defaulting to
"default", which a tree built from the web does not have.
Moving a person moves the keys they made; operator keys stay with the
project. Only empty things are removed, and never a project that had a
key, since usage history refers to it.
Also: keys init no longer reparents an existing team or project;
reinstating an account with no project is refused (a rejected signup
used to come back active and fail at its first key); rejecting a
signup removes it; keys limits --budget takes whole units like the
quotas page.
Checked in headless Chrome: a move from an open project updates the
page in place, keeps the project open, and the search still filters.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Usage could be read as a total — per key, per user, per model, per group —
and the cluster's own charts covered the fleet's hardware. What nobody could
see was when a group used it. "Research doubled its traffic three weeks ago"
is not a question a column of totals can be asked, and it is the question
somebody asks before a quota conversation.
/admin/usage draws requests, tokens and spend over a chosen window, grouped
by any level of the tenancy tree or by person. Person is not a level of that
tree — one person holds keys in several projects — so the grouping is its
own type rather than store.Scope with a fourth value nothing else could use.
Three things learned from the charts already here. Buckets are gap-filled
per group, because a line drawn straight across an absence claims traffic
that did not happen, and because the chart sums its tail by timestamp and
can only do that if every series shares a clock. Series are ranked by total
rather than by peak — a counter's question is "who used the most", and
ranking by peak would colour the team that made one burst over the team that
ran steadily all week. And the spend chart is replaced by a sentence when no
model has a price, rather than drawn flat along the bottom.
The bucket query groups by ordinal and was planned against the cluster's
PostgreSQL before being believed, with a test alongside the others that run
there: three reports have shipped working on SQLite and failing on it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The dashboard charted what the cluster had been billed for — requests, tokens,
spend, all from llmgw's own store — and nothing about whether it was healthy.
Saturation, queue depth, cache occupancy and replica counts were numbers in a
table at the instant the page loaded, with no way to see that one of them had
been climbing for an hour.
Four line charts now, read back from wherever the metrics were scraped to.
Read back rather than kept twice. llmgw publishes these already; what a gauge
cannot do is remember, and a ring buffer in this process would be a rival copy
of the same numbers with its own retention, lost on every restart and
disagreeing with the one an operator has on screen elsewhere. monitoring.prometheus
is empty by default and the panels are then absent rather than empty.
One axis per chart, always. Two measures of different scale belong in two
charts: a second y-axis lets any two lines be made to cross wherever the author
likes and the reader cannot tell it happened.
Four colours, handed out in fixed order, never cycled. A fifth series is summed
into "other" and drawn in grey — dropping it would make a chart that does not
add up, and inventing a hue for it would put two lines on the page that nobody
can tell apart. The fold is by timestamp, not by position: the tail's series do
not share sample times, and adding them by index slides points sideways onto
each other's clock.
A legend for every chart with more than one line, because colour alone is not
an encoding a reader can rely on.
A flat zero is drawn along the bottom rather than left blank: nothing happening
is a real answer, and an empty frame reads as a broken chart. A single sample
is not a line and says so instead of putting a dot in the corner.
One failed query fails all four panels. Three drawn and one blank invites
reading the blank one as "nothing is happening there".
NaN is dropped rather than plotted. Prometheus sends it as a string for a
sample it has no value for, Go's ParseFloat accepts it happily, and one NaN
coordinate is an SVG path that does not draw at all — taking the rest of the
line with it.
Axis maxima are rounded to a number a person would pick. An axis topping out at
0.8347 makes every gridline unreadable, and the shape is what a chart is for.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
/admin/mirror, behind the same admin session as the other admin pages. What is
held, what is arriving with its byte count, what failed and why; start a fetch,
retry a failed one, remove a version.
Not mounted at all without a mirror configured. The right page for a deployment
with no mirror is no page, rather than an empty one pointing at nothing.
It reloads itself only while something is downloading. A reload at any other
moment takes with it whatever is half-typed into the fetch form, and there is
nothing to watch when nothing is moving.
A failed row offers Retry, not Remove. A fetch that failed left nothing on disk,
so the remove could only answer "the mirror holds nothing for this" — a button
whose single outcome is a confusing error. Starting the same download again is
what somebody looking at that row wants, and the mirror allows it: only a
download still running is refused.
An unreachable mirror is said outright instead of rendering as an empty table.
The two look nearly identical and mean opposite things: one is a mirror waiting
to be filled, the other is a machine to go and look at.
Dashboard takes NavExtras rather than a second bool. Two adjacent booleans at a
call site get swapped silently and are discovered by somebody clicking the
wrong link.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The dashboard could say what the cluster was made of and not who was
spending it. Usage was per key, which is the right unit for revoking
something and the wrong one for the question actually being asked: one
person holds a laptop key, a CI key and a notebook key, and adding three
rows up by eye is a chore that gets skipped.
Per person now, joined through the keys they own, and per model — counted
by the model that answered rather than the one requested, since those differ
whenever a fallback serves and what actually ran is the question. Headline
figures are stat tiles with a sparkline: a single current value is a number,
and plotting one is how a chart ends up with a single bar in it. The
sparkline says only rising, falling or flat, which is the one thing the
number cannot.
Ranked horizontal bars, because the labels are email addresses and model
names and the label is the point — the question is which person, not what
shape the distribution has. Rotated or truncated axis labels say the chart
matters more than reading it. Trimmed where drawn and complete on hover, so
the plot keeps its geometry and a reader can still find out whose row it is.
The palette is validated rather than chosen: every adjacent pair clears the
colour-vision separation threshold, and the dark values are the same hues
re-stepped for the dark surface instead of an automatic flip. Run through a
validator, because that is the only way anyone can honestly claim a palette
is colourblind-safe.
Which turned up a bug in the existing chart code. cssVar is an allowlist of
four names that silently returns --accent for anything else, and callers
wrap it in var() themselves — so every new series would have rendered in one
colour with accent-coloured text. Caught by a test written from the rule
that chart text wears text tokens, never the series colour.
A key with no owner is labelled "operator-issued keys" rather than unknown:
it predates accounts and is doing its job, and "unknown" would imply
something is wrong.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The metrics were exported and the only way to look at them was to stand up
Prometheus beside the binary. For a project whose whole shape is one static
binary with no dependencies, that is the wrong default: the common case is
somebody wanting to know how full their cluster is, and that should not cost
a monitoring stack.
/admin now shows capacity per GPU class with a usage bar, free whole nodes
per fabric — the number that decides whether a replica spanning nodes can
land, since free cards on the wrong fabric are not capacity for a model that
needs this one — and which replicas are holding hardware nobody is asking
for. Nodes gain their memory and how many model files they can see.
Taken from the same Summary that ctl status uses rather than derived in the
page, because two implementations of the same arithmetic drift apart and the
one on the screen is the one people would believe.
A faulty card counts as neither used nor free, matching the scheduler.
Counting it free is how a cluster looks like it has room it does not, and a
page that disagreed with placement about that would be worse than no page.
Prometheus stays supported and is now the answer to a narrower question:
history, alerting, several clusters at once. The dashboard shows what is
true now and keeps nothing, which is the honest division between them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Replies are plain text, so a model asked about an integral answers with
\int_0^1 x^2 dx and the reader gets the source.
KaTeX is vendored into the binary rather than loaded from a CDN. This gateway
makes no outbound requests of its own and is expected to run where it could
not make one; a chat page that silently loses its maths on an air-gapped
cluster would be worse than one that never had it. 600 KB of JS, CSS and WOFF2
fonts, which the stripped binary absorbs without changing its rounded size.
The ordering is the safety argument, and there is a test for it rather than a
comment alone. The reply is put in place with textContent exactly as before,
and only then walked for maths: renderMathInElement replaces the spans it
recognises, so nothing the model wrote is ever parsed as HTML and the markup
that appears is KaTeX's. trust stays false — KaTeX's default, and what keeps
\href and \includegraphics out of a reply — and throwOnError is off so one
malformed expression does not take the rest with it.
Rendered once the reply is complete. Typesetting each chunk would try to set
half an expression on every token and flicker through the whole answer.
The static handler now serves a small embedded asset tree rather than one
file, so it moved to its own embed: a file server over the tree that also
carries the templates would serve those too. Directory listings are refused —
nothing links to one, so the only requests are from somebody looking around.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every key needed an operator with a shell. This adds the login system that
made that necessary, which is the thing I argued against building until it was
asked for — so it is built as if it will be attacked, because a public login
form is.
Signing up produces an account that can do nothing. Approval is what makes it
work and stays on the CLI: having a shell on the control plane is what makes
someone an administrator, and that is a smaller thing to get right than roles
and an admin UI that mints credentials.
The details that matter:
Passwords are argon2id, where a slow hash is correct — the input is chosen by
a human and has a dictionary, which is the opposite of an API key. Both kinds
of authentication now live in this codebase and they are in separate packages
so neither set of assumptions leaks into the other. Length is the only rule:
composition rules push people towards Password1! and cost more entropy than
they buy.
Signup never says whether an address already has an account, and sign-in
answers the same way for a wrong password and a missing account — verifying
against a dummy hash when there is no user, so a stopwatch cannot enumerate
accounts. Account state is checked after the password, not before, so guessing
an address reveals nothing about it. Guessing is bounded per address rather
than per IP: an attacker has many addresses to come from and one to guess
against, and limiting by IP locks out an office behind one NAT.
Sessions are server-side, so signing out and suspending take effect
immediately rather than whenever a token expires; suspension is checked in the
same query that resolves the session, because a second round trip is a check
somebody eventually forgets. Every state-changing form carries a CSRF token
verified against a hash in the session — a cookie authenticates the browser,
not the page that asked.
Key ownership is enforced in the UPDATE rather than by reading first, so a
guessed key id from another account changes nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Pasting the gateway's address into a browser returned 404, which looks like a
broken endpoint rather than one that tells you how to use it.
Two pages, and the split between them is a security boundary rather than a
layout choice. The landing page is served without a key — it has to be,
because the person reading it is the one who does not have one yet — so it
contains no data at all: it says how to connect, not what is here. Model names
stay behind /v1/models. It reconstructs the address from the forwarded headers
so the copy-pasteable example names the address the client actually used
rather than the internal one; those headers are reflected into displayed text
and never used for a decision.
The dashboard shows nodes, models, replicas, placement failures and usage, and
lives on the admin listener only. That listener is loopback-bound and also
serves pprof, which can dump the process heap — where API keys sit in
plaintext — so reaching the page from elsewhere is an SSH tunnel's job. It is
read-only: anything that mints a credential stays in `llmgw keys`, where it
needs a shell rather than a browser session this project has no way to
authenticate.
Mounting the landing page uncovered a hole in the exemption list. An entry
ending in "/" was a prefix rule, and "/" ends in "/", so exempting the landing
page would have exempted every path there is — authentication still installed,
still logging, and letting the whole API through. "/" is now exact, with a
test enumerating what must and must not be exempt.
Server-rendered templates and one stylesheet, embedded. The single-binary goal
rules out a JavaScript toolchain and nothing on either page needs one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
