Commit Graph
6 Commits
Author SHA1 Message Date
Ilker ManapandClaude Opus 5 753f4e3265 Let a node say how much room it has left
Disk was the one resource nothing reported. `statfs` was already being called
on every model directory, but only to ask whether the filesystem was a network
one — the free space it returns in the same struct was thrown away. So the
only way to find out why a node was filling up was to ssh to it, which is what
this system exists to avoid.

Reported per filesystem rather than per configured directory. Free space
belongs to the disk, not the path: two model directories on one disk are one
pool of bytes, and counting them separately would let the control plane add
24 GB to 24 GB and place a 40 GB model on a node that has 24.

Shared filesystems are summed separately from local ones and left out of a
node's own total. Space on a filer belongs to everyone mounting it, so twenty
nodes would otherwise each report the same free terabyte, and a scheduler
would place twenty models into room for one.

Nothing reported means nothing is known, everywhere: an agent too old to send
storage and a node with no model directories both send nothing, and both must
read as unknown rather than as a full disk. `ctl nodes` and the dashboard show
a dash, the metric is absent rather than zero, and a missing directory — which
is what an unmounted filer looks like — is skipped rather than reported as
having no space.

Shown as free out of total, because the first is useless without the second:
8 GB free is comfortable on a 10 TB filer and nearly full on a 10 GB root
partition, and only the second stops a model being placed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 09:38:57 +02:00
Ilker ManapandClaude Opus 5 eb9e9b348c Show who is using the cluster and what they are running
The dashboard could say what the cluster was made of and not who was
spending it. Usage was per key, which is the right unit for revoking
something and the wrong one for the question actually being asked: one
person holds a laptop key, a CI key and a notebook key, and adding three
rows up by eye is a chore that gets skipped.

Per person now, joined through the keys they own, and per model — counted
by the model that answered rather than the one requested, since those differ
whenever a fallback serves and what actually ran is the question. Headline
figures are stat tiles with a sparkline: a single current value is a number,
and plotting one is how a chart ends up with a single bar in it. The
sparkline says only rising, falling or flat, which is the one thing the
number cannot.

Ranked horizontal bars, because the labels are email addresses and model
names and the label is the point — the question is which person, not what
shape the distribution has. Rotated or truncated axis labels say the chart
matters more than reading it. Trimmed where drawn and complete on hover, so
the plot keeps its geometry and a reader can still find out whose row it is.

The palette is validated rather than chosen: every adjacent pair clears the
colour-vision separation threshold, and the dark values are the same hues
re-stepped for the dark surface instead of an automatic flip. Run through a
validator, because that is the only way anyone can honestly claim a palette
is colourblind-safe.

Which turned up a bug in the existing chart code. cssVar is an allowlist of
four names that silently returns --accent for anything else, and callers
wrap it in var() themselves — so every new series would have rendered in one
colour with accent-coloured text. Caught by a test written from the rule
that chart text wears text tokens, never the series colour.

A key with no owner is labelled "operator-issued keys" rather than unknown:
it predates accounts and is doing its job, and "unknown" would imply
something is wrong.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 18:38:21 +02:00
Ilker ManapandClaude Opus 5 c232f41a66 Put the fleet view in the dashboard, so nothing else is needed to see it
The metrics were exported and the only way to look at them was to stand up
Prometheus beside the binary. For a project whose whole shape is one static
binary with no dependencies, that is the wrong default: the common case is
somebody wanting to know how full their cluster is, and that should not cost
a monitoring stack.

/admin now shows capacity per GPU class with a usage bar, free whole nodes
per fabric — the number that decides whether a replica spanning nodes can
land, since free cards on the wrong fabric are not capacity for a model that
needs this one — and which replicas are holding hardware nobody is asking
for. Nodes gain their memory and how many model files they can see.

Taken from the same Summary that ctl status uses rather than derived in the
page, because two implementations of the same arithmetic drift apart and the
one on the screen is the one people would believe.

A faulty card counts as neither used nor free, matching the scheduler.
Counting it free is how a cluster looks like it has room it does not, and a
page that disagreed with placement about that would be worse than no page.

Prometheus stays supported and is now the answer to a narrower question:
history, alerting, several clusters at once. The dashboard shows what is
true now and keeps nothing, which is the honest division between them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 18:15:30 +02:00
Ilker ManapandClaude Opus 5 3b67b1fe42 Stop the dashboard linking to pages that are not where it is
Clicking "metrics" on the web dashboard answered "no api key was provided".

The page is served twice and its navigation assumed one of them. Metrics,
pprof and the management API exist on the admin listener; account
administration, the profile and the chat exist on the client-facing port. The
nav offered the first set in both places, so on the public port those links
fell through to the API-key middleware — which is a far more confusing way to
say "that is not here" than a 404 would have been.

Each page now offers what is actually reachable from it, and the public one
says where the rest lives rather than pretending it does not exist, including
why that port is not exposed: pprof can dump the process heap, and the heap
holds every API key in plaintext.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 16:57:58 +02:00
Ilker ManapandClaude Opus 5 859f08da9b Approve accounts from the browser, and draw the traffic
Two things the dashboard was missing: it could not act, and it was a column of
numbers.

Account administration moves to /admin/users — approving into a project,
suspending, reinstating. That is the work an operator does constantly and it
should not need a terminal. Three things stay on the CLI, and the line is not
arbitrary: promoting someone to admin, resetting a password, and issuing a key
on another person's behalf each turn a stolen admin session into more than it
was — permanent access, or somebody else's account. Approving and suspending
do not. Suspending signs the account out everywhere, or suspension is advisory
until the session happens to expire, and an admin cannot suspend themselves,
which is not a thing an interface should let you do by misclicking.

The charts are inline SVG generated in Go. A charting library would mean a
JavaScript toolchain in CI and a megabyte of bundle for what is a list of
rectangles, and the single-binary goal rules that out. Hourly over a day: fine
enough to show a working day, coarse enough that one slow minute is not an
incident. Empty hours are filled in, because a chart with hours missing where
nothing happened reads as a chart with hours missing from the data.

The SVG is marked as trusted HTML, so everything that reaches it is either a
number this package computed or escaped explicitly, and a bar's colour can only
be one of four names — a colour that could be anything is a way to write CSS
into every page the chart appears on. There are tests for both.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 12:01:42 +02:00
Ilker ManapandClaude Opus 5 1839832b8b Add a landing page and a read-only cluster dashboard
Pasting the gateway's address into a browser returned 404, which looks like a
broken endpoint rather than one that tells you how to use it.

Two pages, and the split between them is a security boundary rather than a
layout choice. The landing page is served without a key — it has to be,
because the person reading it is the one who does not have one yet — so it
contains no data at all: it says how to connect, not what is here. Model names
stay behind /v1/models. It reconstructs the address from the forwarded headers
so the copy-pasteable example names the address the client actually used
rather than the internal one; those headers are reflected into displayed text
and never used for a decision.

The dashboard shows nodes, models, replicas, placement failures and usage, and
lives on the admin listener only. That listener is loopback-bound and also
serves pprof, which can dump the process heap — where API keys sit in
plaintext — so reaching the page from elsewhere is an SSH tunnel's job. It is
read-only: anything that mints a credential stays in `llmgw keys`, where it
needs a shell rather than a browser session this project has no way to
authenticate.

Mounting the landing page uncovered a hole in the exemption list. An entry
ending in "/" was a prefix rule, and "/" ends in "/", so exempting the landing
page would have exempted every path there is — authentication still installed,
still logging, and letting the whole API through. "/" is now exact, with a
test enumerating what must and must not be exempt.

Server-rendered templates and one stylesheet, embedded. The single-binary goal
rules out a JavaScript toolchain and nothing on either page needs one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 08:58:05 +02:00