A review of v0.10.4-rc.4, and a look at the live network, found:
- Alertmanager listening on every interface: from any node its API took
silences, with no login. Bound to loopback on the monitoring host, and the
monitoring README now installs it that way; Prometheus beside it is its only
client, and one Alertmanager needs no cluster port.
- Login and signup unbounded in memory. Each attempt runs argon2 with 64 MiB,
and nothing limited how many ran at once; the per-address throttle kept one
entry per address ever tried. Hashing now takes one of a few slots and
waits for one otherwise. The throttle expires entries whose window has
passed when it fills, and if it is still full it refuses new addresses:
evicting live ones instead would let a flood of made-up addresses reset the
throttle on the address actually being guessed.
- Metric labels taken from the request. A refused request counted the model
name it asked for, so any key holder could mint a series per request.
Requests are counted under the catalog's name, or "unknown".
- Two in code written this week. A SQLite backup made its snapshot with the
default mode in the listable backup directory, a full copy of the store
readable while the backup ran; it is now made in a private directory, and
the unit sets UMask=0077. And archive links were checked by name only, so
"a -> ." then "a/a/b -> ../.." passed and a file written through them landed
above the install directory, as root. Archives now unpack through os.Root,
and a link may only point at a file beside it — which every llama.cpp
b10991 archive, CPU and CUDA, was checked to follow.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
There were no backups: docs/install.md gave a pg_dump command, and nothing
ran it. The store holds every key, person, quota and usage record, the models
added at runtime, and now the secret each engine's key is derived from; the
certificate authority's key sits in /etc/llmgw/pki beside it. Losing the disk
lost all of it.
`llmgw server backup` writes both to one archive: pg_dump's custom format, or
a VACUUM INTO snapshot for SQLite, plus /etc/llmgw. The archive holds the CA's
key, so it is 0600; the directory is listable, so the control plane — not
root — can see when the newest was written. pg_dump is given the connection
through its environment, since a DSN carries the password and a process's
arguments are readable by every user. Written beside its name and renamed,
so an interrupted backup is never the newest; the newest 14 are kept, and
nothing the command did not write is ever removed.
llmgw-backup.timer runs it nightly and catches up after downtime. The control
plane publishes the newest archive's time, and BackupTooOld fires past 26
hours — or when there has never been one. The install guide now covers
restoring, and says to copy archives off the machine, since they share its
disk.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
On the live cluster kev-0.5b — an architecture the nodes' llama.cpp did not
know — was started 155 times in eight hours, failing the same way each time,
and phi-4, which fitted on no node, logged "no placement" 250 times in two
days. Neither showed up anywhere an operator looked: the backoff never gave
up, no metric said a model was failing or unplaceable, and the one alert for
a model with nothing serving could not fire, because its series was published
only while a replica was ready.
After five failures in a row a model is parked: not placed again until its
definition changes — a new source or profile — or `llmgw ctl retry` is run.
Status, the model list and the admin API show it with its last error. The
placement warning is logged when its reason changes or every ten minutes.
A request for a parked or unplaceable model is answered at once with why,
rather than "loading; retry shortly", which a client retries forever. Each
request still clears the previous placement result and triggers a fresh
attempt, so a model that fits once room is made is not refused on a stale
answer; the router waits for that attempt, not for its full activation
window.
Metrics now cover model health (desired, ready at 0 for what should run,
consecutive start failures, parked, unplaceable), engine installs per node,
leadership, and certificate expiry read from disk at each scrape, with alerts
for each. `ctl status` counts a model against what it should have now, so an
on-demand model that was asked for and fitted nowhere no longer reads as
satisfied.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Where to put them turned out not to be a question about capacity. Measured on
this cluster: 2,359 series at 41 samples a second, about 7 MB a day, so a year
of history costs roughly 3 GB against 900 GB free. Retention was the package
default of fifteen days, which answers none of the questions this data is for —
what capacity looked like last quarter, whether it was always this slow, how
much the fleet grew.
A year, with a size cap as well. The time limit is the intent; the cap is what
stops Prometheus becoming the thing that fills the disk if somebody adds a
fleet and the series count grows tenfold.
The part that matters is what shares that disk. The model mirror and the
metrics database are on one filesystem, so a mirror that fills it stops
Prometheus writing — monitoring goes quiet exactly when the disk is the
problem, and the alert that would have said so is the one that died.
Nothing was watching it. node_exporter arrives with the Prometheus package and
was already running on that machine, unscraped, while llmgw's own disk metrics
cover only its nodes — and the mirror is not one. It is a target now, and the
alert fires at fifteen percent rather than five, because what is being
protected is the ability to be told.
No remote write and no second tier. At two thousand series that is
infrastructure for a problem this cluster does not have.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Prometheus, Alertmanager and Grafana, with every rule written against a series
this system actually publishes. Nothing here is aimed at a metric that might
exist.
One scrape target for the whole fleet. Agents publish nothing of their own —
the control plane gathers what they report and republishes it — which is worth
writing down before somebody goes looking for an exporter to install on each
node.
It scrapes metrics_listen, not the admin listener. That one is loopback because
it also serves pprof, and pprof can dump the heap where API keys sit in
plaintext. Metrics are the one thing on it something else genuinely needs, so
they get a port of their own rather than becoming a reason to open the other.
The README shows how to confirm the new port carries nothing else; it answers
404 to both pprof and the admin API.
Nothing is sent anywhere. Alertmanager groups and holds and has no receiver:
where an alert should go is a decision with credentials attached, and a
monitoring system that quietly starts mailing people is worse than one that is
visibly not wired up. critical already routes separately so it can be pointed
somewhere else without touching the rest.
The rules encode what the metrics mean rather than round numbers. A saturation
of 1 is the healthy limit measured by ctl bench, not the point of failure, so
the alert waits five minutes above it. A thirty-second p99 first token is the
threshold because this is CPU inference and would be absurd on a GPU. A model
that is deliberately stopped publishes no series at all, so "nothing serving"
cannot fire for one nobody asked for — which is what lets an on-demand
catalogue sit quiet.
Inhibitions, because a node that is down explains every replica on it and
reporting both buries the cause.
The dashboard is provisioned from a file rather than clicked together, so it
survives a reinstall and can be read in a diff.
Checked by firing one: cordoning a node moved llmgw_node_up to 0, the rule went
pending and then firing after its two minutes, the alert arrived at Alertmanager
with the node in its labels, and uncordoning resolved it. An alerting system
nobody has seen fire is an assumption.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The metrics existed and nothing could record them. They shared a listener
with pprof, the management API and the dashboard, and pprof can dump the
process heap where API keys sit in plaintext — so a fleet could be monitored
or keep its heap private, and not both. That is not a choice anyone should
have to make, and it is why every previous answer to "how do I scrape this"
was a tunnel.
metrics_listen serves metrics and health and nothing else, so it can be
reached from wherever Prometheus lives. Metrics stay on the admin listener
too: adding a door is not the same as moving one, and anything already
scraping keeps working.
While there: admin_listen defaulted to :9090, every interface, with a
comment beside it saying not to expose it. A default that contradicts its
own documentation is a footgun, so it is loopback now and someone who wants
otherwise says so.
The handler is nil until the registry is built, and says so rather than
serving an empty page. Ordering has already caught this exact area once —
a collector registered after AdminHandler published nothing at all, silently
— and an empty scrape target looks like a working one.
deploy/monitoring/prometheus.yml has a starting configuration and the alerts
worth having: a node that has stopped reporting, a model that should be
serving and is not, a card kept out of placement by a fault, and a fleet
holding models nobody is asking for.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
