init --patroni user@host ... installs PostgreSQL, Patroni and etcd on the
given machines over ssh, with this one as the third etcd vote, makes the role
and database, and moves the store in use into the cluster. Each step checks
first, so it changes nothing on a cluster that is up. For a store of several
hosts the control plane publishes each member's role, state and lag from
Patroni, a stopped member as missing, prints it in ctl status, and alerts on
it. Making an administrator on PostgreSQL sent a bool to an integer column;
it sends an integer now.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Requests by outcome, tokens and first-token time per tenant, at the level
router.tenant_label names (team by default), and every budget's share spent
this period. Rules alert on budgets at 80% and 100%, tenants hitting limits,
failing, spiking past their weekly average, or waiting on first tokens.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every minute the agent reads nvidia-smi's full report, its NVLink error
counters and the kernel's XID lines. They are published per card,
shown on the node page and the GPU map, charted in Grafana, and five
alerts watch them.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
They were read once, at agent start, and published unchanged since.
Each sample now carries utilisation, memory and bandwidth in use, the SM
clock and uncorrected ECC errors. The node page charts each card, the
cluster page the fleet by class, Grafana has a GPU row, and three GPU
alerts are added.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A model has an owner (org, team or project). Quotas guarantee a scope
GPUs, cores and memory and may limit it; what a scope holds is summed from
its models' replicas and those beneath it. A replica within its owner's
guarantee is not taken for another scope; what is held beyond a guarantee
is borrowed and taken back for a scope within its own. A limit stops a
model before it starts, and headroom stops there too. ctl quota and ctl
model owner set them; metrics and a CapacityAtLimit alert report them.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A model's class (critical, standard, opportunistic; unset follows its
scaling) decides what may be stopped to make room for it: a lower class
for a higher, and within a class only what another grew into under load,
for a model below its minimum. Victims are tried on a copy of the
inventory and stopped only if they make the room. Preempted replicas are
drained with a short grace and their model is held back until the room is
used. ctl model resources and class move under ctl model.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
There were no backups: docs/install.md gave a pg_dump command, and nothing
ran it. The store holds every key, person, quota and usage record, the models
added at runtime, and now the secret each engine's key is derived from; the
certificate authority's key sits in /etc/llmgw/pki beside it. Losing the disk
lost all of it.
`llmgw server backup` writes both to one archive: pg_dump's custom format, or
a VACUUM INTO snapshot for SQLite, plus /etc/llmgw. The archive holds the CA's
key, so it is 0600; the directory is listable, so the control plane — not
root — can see when the newest was written. pg_dump is given the connection
through its environment, since a DSN carries the password and a process's
arguments are readable by every user. Written beside its name and renamed,
so an interrupted backup is never the newest; the newest 14 are kept, and
nothing the command did not write is ever removed.
llmgw-backup.timer runs it nightly and catches up after downtime. The control
plane publishes the newest archive's time, and BackupTooOld fires past 26
hours — or when there has never been one. The install guide now covers
restoring, and says to copy archives off the machine, since they share its
disk.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
On the live cluster kev-0.5b — an architecture the nodes' llama.cpp did not
know — was started 155 times in eight hours, failing the same way each time,
and phi-4, which fitted on no node, logged "no placement" 250 times in two
days. Neither showed up anywhere an operator looked: the backoff never gave
up, no metric said a model was failing or unplaceable, and the one alert for
a model with nothing serving could not fire, because its series was published
only while a replica was ready.
After five failures in a row a model is parked: not placed again until its
definition changes — a new source or profile — or `llmgw ctl retry` is run.
Status, the model list and the admin API show it with its last error. The
placement warning is logged when its reason changes or every ten minutes.
A request for a parked or unplaceable model is answered at once with why,
rather than "loading; retry shortly", which a client retries forever. Each
request still clears the previous placement result and triggers a fresh
attempt, so a model that fits once room is made is not refused on a stale
answer; the router waits for that attempt, not for its full activation
window.
Metrics now cover model health (desired, ready at 0 for what should run,
consecutive start failures, parked, unplaceable), engine installs per node,
leadership, and certificate expiry read from disk at each scrape, with alerts
for each. `ctl status` counts a model against what it should have now, so an
on-demand model that was asked for and fitted nowhere no longer reads as
satisfied.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Where to put them turned out not to be a question about capacity. Measured on
this cluster: 2,359 series at 41 samples a second, about 7 MB a day, so a year
of history costs roughly 3 GB against 900 GB free. Retention was the package
default of fifteen days, which answers none of the questions this data is for —
what capacity looked like last quarter, whether it was always this slow, how
much the fleet grew.
A year, with a size cap as well. The time limit is the intent; the cap is what
stops Prometheus becoming the thing that fills the disk if somebody adds a
fleet and the series count grows tenfold.
The part that matters is what shares that disk. The model mirror and the
metrics database are on one filesystem, so a mirror that fills it stops
Prometheus writing — monitoring goes quiet exactly when the disk is the
problem, and the alert that would have said so is the one that died.
Nothing was watching it. node_exporter arrives with the Prometheus package and
was already running on that machine, unscraped, while llmgw's own disk metrics
cover only its nodes — and the mirror is not one. It is a target now, and the
alert fires at fifteen percent rather than five, because what is being
protected is the ability to be told.
No remote write and no second tier. At two thousand series that is
infrastructure for a problem this cluster does not have.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Prometheus, Alertmanager and Grafana, with every rule written against a series
this system actually publishes. Nothing here is aimed at a metric that might
exist.
One scrape target for the whole fleet. Agents publish nothing of their own —
the control plane gathers what they report and republishes it — which is worth
writing down before somebody goes looking for an exporter to install on each
node.
It scrapes metrics_listen, not the admin listener. That one is loopback because
it also serves pprof, and pprof can dump the heap where API keys sit in
plaintext. Metrics are the one thing on it something else genuinely needs, so
they get a port of their own rather than becoming a reason to open the other.
The README shows how to confirm the new port carries nothing else; it answers
404 to both pprof and the admin API.
Nothing is sent anywhere. Alertmanager groups and holds and has no receiver:
where an alert should go is a decision with credentials attached, and a
monitoring system that quietly starts mailing people is worse than one that is
visibly not wired up. critical already routes separately so it can be pointed
somewhere else without touching the rest.
The rules encode what the metrics mean rather than round numbers. A saturation
of 1 is the healthy limit measured by ctl bench, not the point of failure, so
the alert waits five minutes above it. A thirty-second p99 first token is the
threshold because this is CPU inference and would be absurd on a GPU. A model
that is deliberately stopped publishes no series at all, so "nothing serving"
cannot fire for one nobody asked for — which is what lets an on-demand
catalogue sit quiet.
Inhibitions, because a node that is down explains every replica on it and
reporting both buries the cause.
The dashboard is provisioned from a file rather than clicked together, so it
survives a reinstall and can be read in a diff.
Checked by firing one: cordoning a node moved llmgw_node_up to 0, the rule went
pending and then firing after its two minutes, the alert arrived at Alertmanager
with the node in its labels, and uncordoning resolved it. An alerting system
nobody has seen fire is an assumption.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
