235bf33dcd0dd0585e2602fd80a16352eaa93cda
7
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Ilker ManapandClaude Opus 5.5
|
62657f4c2a |
Release v0.10.4: CPU threads, tracing, and the rest of the review
CPU replicas start on half the machine's CPUs, not all of them: two busy llama.cpp servers each spinning on all 8 vCPUs fell to 0.8 tok/s apiece, where at 4 threads they keep 5.5. llamacpp.threads sets it per node. Inference requests can be traced over OTLP/HTTP (tracing.endpoint), with admission, schedule and upstream spans and traceparent sent to the engine. Security: onboarding no longer puts the cluster root in the system trust store and removes it where it was; the mirror's mutual-TLS API can be limited to named certificates (--api-client); init.log no longer keeps the administrator's password; a warning when auth is off on a public listen; the recorded actor is sanitised and sudo-aware; Ray's dashboard is off, and the remaining engine gaps are documented. reconciler.go and agent.go are split by topic, the status mapping of each failure is tested, and the plan's checklist is brought up to date. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5.5
|
91fe24a105 |
Close what a second security review found still open
A review of v0.10.4-rc.4, and a look at the live network, found: - Alertmanager listening on every interface: from any node its API took silences, with no login. Bound to loopback on the monitoring host, and the monitoring README now installs it that way; Prometheus beside it is its only client, and one Alertmanager needs no cluster port. - Login and signup unbounded in memory. Each attempt runs argon2 with 64 MiB, and nothing limited how many ran at once; the per-address throttle kept one entry per address ever tried. Hashing now takes one of a few slots and waits for one otherwise. The throttle expires entries whose window has passed when it fills, and if it is still full it refuses new addresses: evicting live ones instead would let a flood of made-up addresses reset the throttle on the address actually being guessed. - Metric labels taken from the request. A refused request counted the model name it asked for, so any key holder could mint a series per request. Requests are counted under the catalog's name, or "unknown". - Two in code written this week. A SQLite backup made its snapshot with the default mode in the listable backup directory, a full copy of the store readable while the backup ran; it is now made in a private directory, and the unit sets UMask=0077. And archive links were checked by name only, so "a -> ." then "a/a/b -> ../.." passed and a file written through them landed above the install directory, as root. Archives now unpack through os.Root, and a link may only point at a file beside it — which every llama.cpp b10991 archive, CPU and CUDA, was checked to follow. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5.5
|
1da4a2e1ef |
Back up the store and the certificate authority, and alert when it stops
There were no backups: docs/install.md gave a pg_dump command, and nothing ran it. The store holds every key, person, quota and usage record, the models added at runtime, and now the secret each engine's key is derived from; the certificate authority's key sits in /etc/llmgw/pki beside it. Losing the disk lost all of it. `llmgw server backup` writes both to one archive: pg_dump's custom format, or a VACUUM INTO snapshot for SQLite, plus /etc/llmgw. The archive holds the CA's key, so it is 0600; the directory is listable, so the control plane — not root — can see when the newest was written. pg_dump is given the connection through its environment, since a DSN carries the password and a process's arguments are readable by every user. Written beside its name and renamed, so an interrupted backup is never the newest; the newest 14 are kept, and nothing the command did not write is ever removed. llmgw-backup.timer runs it nightly and catches up after downtime. The control plane publishes the newest archive's time, and BackupTooOld fires past 26 hours — or when there has never been one. The install guide now covers restoring, and says to copy archives off the machine, since they share its disk. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5.5
|
37a80a9b44 |
Park a model that fails every time it starts, and say so everywhere
On the live cluster kev-0.5b — an architecture the nodes' llama.cpp did not know — was started 155 times in eight hours, failing the same way each time, and phi-4, which fitted on no node, logged "no placement" 250 times in two days. Neither showed up anywhere an operator looked: the backoff never gave up, no metric said a model was failing or unplaceable, and the one alert for a model with nothing serving could not fire, because its series was published only while a replica was ready. After five failures in a row a model is parked: not placed again until its definition changes — a new source or profile — or `llmgw ctl retry` is run. Status, the model list and the admin API show it with its last error. The placement warning is logged when its reason changes or every ten minutes. A request for a parked or unplaceable model is answered at once with why, rather than "loading; retry shortly", which a client retries forever. Each request still clears the previous placement result and triggers a fresh attempt, so a model that fits once room is made is not refused on a stale answer; the router waits for that attempt, not for its full activation window. Metrics now cover model health (desired, ready at 0 for what should run, consecutive start failures, parked, unplaceable), engine installs per node, leadership, and certificate expiry read from disk at each scrape, with alerts for each. `ctl status` counts a model against what it should have now, so an on-demand model that was asked for and fitted nowhere no longer reads as satisfied. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
c95cfbe426 |
Keep a year of metrics, and watch the disk they share with the models
CI / test (push) Successful in 1m48s
Where to put them turned out not to be a question about capacity. Measured on this cluster: 2,359 series at 41 samples a second, about 7 MB a day, so a year of history costs roughly 3 GB against 900 GB free. Retention was the package default of fifteen days, which answers none of the questions this data is for — what capacity looked like last quarter, whether it was always this slow, how much the fleet grew. A year, with a size cap as well. The time limit is the intent; the cap is what stops Prometheus becoming the thing that fills the disk if somebody adds a fleet and the series count grows tenfold. The part that matters is what shares that disk. The model mirror and the metrics database are on one filesystem, so a mirror that fills it stops Prometheus writing — monitoring goes quiet exactly when the disk is the problem, and the alert that would have said so is the one that died. Nothing was watching it. node_exporter arrives with the Prometheus package and was already running on that machine, unscraped, while llmgw's own disk metrics cover only its nodes — and the mirror is not one. It is a target now, and the alert fires at fifteen percent rather than five, because what is being protected is the ability to be told. No remote write and no second tier. At two thousand series that is infrastructure for a problem this cluster does not have. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
b921fa0d6b |
Watch the cluster, and prove the alerts fire
CI / test (push) Successful in 1m45s
Prometheus, Alertmanager and Grafana, with every rule written against a series this system actually publishes. Nothing here is aimed at a metric that might exist. One scrape target for the whole fleet. Agents publish nothing of their own — the control plane gathers what they report and republishes it — which is worth writing down before somebody goes looking for an exporter to install on each node. It scrapes metrics_listen, not the admin listener. That one is loopback because it also serves pprof, and pprof can dump the heap where API keys sit in plaintext. Metrics are the one thing on it something else genuinely needs, so they get a port of their own rather than becoming a reason to open the other. The README shows how to confirm the new port carries nothing else; it answers 404 to both pprof and the admin API. Nothing is sent anywhere. Alertmanager groups and holds and has no receiver: where an alert should go is a decision with credentials attached, and a monitoring system that quietly starts mailing people is worse than one that is visibly not wired up. critical already routes separately so it can be pointed somewhere else without touching the rest. The rules encode what the metrics mean rather than round numbers. A saturation of 1 is the healthy limit measured by ctl bench, not the point of failure, so the alert waits five minutes above it. A thirty-second p99 first token is the threshold because this is CPU inference and would be absurd on a GPU. A model that is deliberately stopped publishes no series at all, so "nothing serving" cannot fire for one nobody asked for — which is what lets an on-demand catalogue sit quiet. Inhibitions, because a node that is down explains every replica on it and reporting both buries the cause. The dashboard is provisioned from a file rather than clicked together, so it survives a reinstall and can be read in a diff. Checked by firing one: cordoning a node moved llmgw_node_up to 0, the rule went pending and then firing after its two minutes, the alert arrived at Alertmanager with the node in its labels, and uncordoning resolved it. An alerting system nobody has seen fire is an assumption. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
1ea067d08e |
Give metrics a door that does not also open onto pprof
CI / test (push) Successful in 1m38s
The metrics existed and nothing could record them. They shared a listener with pprof, the management API and the dashboard, and pprof can dump the process heap where API keys sit in plaintext — so a fleet could be monitored or keep its heap private, and not both. That is not a choice anyone should have to make, and it is why every previous answer to "how do I scrape this" was a tunnel. metrics_listen serves metrics and health and nothing else, so it can be reached from wherever Prometheus lives. Metrics stay on the admin listener too: adding a door is not the same as moving one, and anything already scraping keeps working. While there: admin_listen defaulted to :9090, every interface, with a comment beside it saying not to expose it. A default that contradicts its own documentation is a footgun, so it is loopback now and someone who wants otherwise says so. The handler is nil until the registry is built, and says so rather than serving an empty page. Ordering has already caught this exact area once — a collector registered after AdminHandler published nothing at all, silently — and an empty scrape target looks like a working one. deploy/monitoring/prometheus.yml has a starting configuration and the alerts worth having: a node that has stopped reporting, a model that should be serving and is not, a card kept out of placement by a fault, and a fleet holding models nobody is asking for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |