Ilker ManapandClaude Opus 5.5
91fe24a105
A review of v0.10.4-rc.4, and a look at the live network, found: - Alertmanager listening on every interface: from any node its API took silences, with no login. Bound to loopback on the monitoring host, and the monitoring README now installs it that way; Prometheus beside it is its only client, and one Alertmanager needs no cluster port. - Login and signup unbounded in memory. Each attempt runs argon2 with 64 MiB, and nothing limited how many ran at once; the per-address throttle kept one entry per address ever tried. Hashing now takes one of a few slots and waits for one otherwise. The throttle expires entries whose window has passed when it fills, and if it is still full it refuses new addresses: evicting live ones instead would let a flood of made-up addresses reset the throttle on the address actually being guessed. - Metric labels taken from the request. A refused request counted the model name it asked for, so any key holder could mint a series per request. Requests are counted under the catalog's name, or "unknown". - Two in code written this week. A SQLite backup made its snapshot with the default mode in the listable backup directory, a full copy of the store readable while the backup ran; it is now made in a private directory, and the unit sets UMask=0077. And archive links were checked by name only, so "a -> ." then "a/a/b -> ../.." passed and a file written through them landed above the install directory, as root. Archives now unpack through os.Root, and a link may only point at a file beside it — which every llama.cpp b10991 archive, CPU and CUDA, was checked to follow. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
178 lines
7.3 KiB
Markdown
178 lines
7.3 KiB
Markdown
# Watching a cluster
|
|
|
|
Prometheus, Alertmanager and Grafana, configured for what llmgw actually
|
|
publishes. Every alert here is written against a series this system emits;
|
|
nothing is aimed at a metric that might exist.
|
|
|
|
## What it is
|
|
|
|
One scrape target for the whole fleet.
|
|
|
|
Agents publish no metrics of their own. The control plane gathers what they
|
|
report — node state, memory, disk, engine queue depth, KV occupancy — and
|
|
republishes all of it, so a single endpoint describes every machine. That is
|
|
worth knowing before going looking for an exporter on each node.
|
|
|
|
## The port it scrapes
|
|
|
|
`metrics_listen`, not `admin_listen`.
|
|
|
|
The admin listener is bound to loopback for one reason: it also serves pprof,
|
|
and pprof can dump the process heap, where API keys sit in plaintext. Metrics
|
|
are the one thing on that listener something else genuinely needs to reach, so
|
|
they get a port of their own instead of becoming an argument for opening the
|
|
other one.
|
|
|
|
```yaml
|
|
# /etc/llmgw/server.yaml
|
|
metrics_listen: ":9091"
|
|
```
|
|
|
|
Confirm it carries nothing else:
|
|
|
|
```
|
|
curl -s -o /dev/null -w '%{http_code}\n' http://<control-plane>:9091/debug/pprof/ # 404
|
|
curl -s -o /dev/null -w '%{http_code}\n' http://<control-plane>:9091/api/v1/nodes # 404
|
|
```
|
|
|
|
## Installing
|
|
|
|
On Ubuntu 24.04 the first two come from the archive; Grafana needs its own
|
|
repository.
|
|
|
|
```
|
|
apt-get install prometheus prometheus-alertmanager
|
|
|
|
install -m 0644 prometheus.yml /etc/prometheus/prometheus.yml
|
|
install -m 0644 llmgw-rules.yml /etc/prometheus/rules/llmgw.yml
|
|
install -m 0644 alertmanager.yml /etc/prometheus/alertmanager.yml
|
|
# Alertmanager answers on loopback only. Its API takes silences from anyone who
|
|
# can reach it, with no login, and silencing BackupTooOld or ModelParked is how
|
|
# something that should be noticed is not. Prometheus beside it is its only
|
|
# client; one Alertmanager needs no cluster port.
|
|
echo 'ARGS="--web.listen-address=127.0.0.1:9093 --cluster.listen-address="' \
|
|
> /etc/default/prometheus-alertmanager
|
|
promtool check config /etc/prometheus/prometheus.yml
|
|
systemctl restart prometheus prometheus-alertmanager
|
|
|
|
install -m 0644 grafana/datasource.yaml /etc/grafana/provisioning/datasources/llmgw.yaml
|
|
install -m 0644 grafana/dashboards.yaml /etc/grafana/provisioning/dashboards/llmgw.yaml
|
|
install -m 0644 -o grafana -g grafana grafana/llmgw-overview.json \
|
|
/var/lib/grafana/dashboards/llmgw.json
|
|
systemctl restart grafana-server
|
|
```
|
|
|
|
The dashboard is provisioned from a file rather than clicked together, so it
|
|
survives a reinstall and can be reviewed in a diff.
|
|
|
|
## Where to run it
|
|
|
|
Not on the control plane. Monitoring that dies with the thing it monitors tells
|
|
you nothing at the moment you need it, and the control plane is also the busiest
|
|
machine in a small cluster.
|
|
|
|
Anywhere else on the management network will do. The mirror is a reasonable
|
|
host: it has the disk, and serving weights is bursty rather than constant.
|
|
|
|
## Nothing is sent anywhere
|
|
|
|
Alertmanager groups and holds alerts and has no receiver.
|
|
|
|
Where an alert should go — an address, a Slack workspace, a webhook — is a
|
|
decision with credentials attached, and a monitoring system that quietly starts
|
|
mailing people is worse than one that is visibly not wired up. Alerts are
|
|
visible in Prometheus, and at `:9093` on the monitoring host itself (`ssh -L
|
|
9093:127.0.0.1:9093 <host>` to open its page); add a receiver to `alertmanager.yml` to
|
|
make them travel. `critical` already routes separately so it can be pointed
|
|
somewhere else without touching the rest.
|
|
|
|
## Check that it fires
|
|
|
|
An alerting system nobody has seen fire is an assumption.
|
|
|
|
```
|
|
llmgw ctl cordon <node> # work already there keeps running
|
|
# ~2 minutes later
|
|
ssh <host> curl -s http://127.0.0.1:9093/api/v2/alerts | jq '.[].labels'
|
|
llmgw ctl uncordon <node> # the alert resolves on its own
|
|
```
|
|
|
|
Cordoning is the cheapest real test: it changes a metric the rules read, stops
|
|
nothing, and reverses in one command.
|
|
|
|
## What the alerts assume
|
|
|
|
**A stopped model publishes no series.** `llmgw_model_replicas_ready` exists
|
|
only for models with a replica placed, so `== 0` cannot fire for one that is
|
|
deliberately off. That is what makes an on-demand catalogue quiet here.
|
|
|
|
**Saturation of 1 is the healthy limit, not failure.** It is the worse of queue
|
|
depth and KV occupancy, measured by `ctl bench` rather than guessed. The alert
|
|
allows five minutes above it before saying anything.
|
|
|
|
**Latencies are in seconds.** These thresholds are written for CPU inference. A
|
|
p99 first token of thirty seconds is the alerting point here and would be
|
|
absurd on a GPU; change it when the hardware changes.
|
|
|
|
**Disk is only reported by nodes that hold weights.** A node with no model
|
|
directory reports nothing rather than zero, and the rule divides free by total
|
|
so a node that reports neither never fires.
|
|
|
|
## Where the metrics live
|
|
|
|
On the machine that scrapes them, in Prometheus's own store, and that is the
|
|
right answer at this size. Measured on this cluster: 2,359 series at 41 samples
|
|
a second, which compresses to roughly 7 MB a day — about 3 GB for a year.
|
|
|
|
```
|
|
# /etc/default/prometheus
|
|
ARGS="--storage.tsdb.retention.time=365d --storage.tsdb.retention.size=20GB"
|
|
```
|
|
|
|
A year because the questions worth asking of this data are seasonal: what did
|
|
capacity look like last quarter, was it always this slow, how much did the
|
|
fleet grow. Fifteen days answers none of them.
|
|
|
|
Both limits, not one. The time limit is the intent; the size cap is what stops
|
|
Prometheus becoming the thing that fills the disk if the series count grows
|
|
tenfold after somebody adds a fleet.
|
|
|
|
No remote write, no object store, no second tier. At two thousand series that
|
|
is infrastructure for a problem this cluster does not have, and every one of
|
|
those has its own failure modes to learn.
|
|
|
|
### The part that is not about capacity
|
|
|
|
The disk is not the constraint — 900 GB free against 3 GB a year. What matters
|
|
is what shares it.
|
|
|
|
On this cluster the model mirror and the metrics database are on the same
|
|
filesystem. A mirror that fills it stops Prometheus writing, so monitoring goes
|
|
quiet at exactly the moment the disk is the problem, and the alert that would
|
|
have said so is the one that died.
|
|
|
|
Two ways out, in order of how much they cost:
|
|
|
|
- Watch it, which is now done. `prometheus-node-exporter` arrives with the
|
|
Prometheus package and was running, unscraped, on this machine — so the disk
|
|
holding both was watched by nothing. It is a scrape target now, and
|
|
`MonitoringDiskFilling` fires at fifteen percent rather than five, because
|
|
the thing being protected is the ability to be told.
|
|
- Separate them: a volume of its own for `/var/lib/prometheus`, or monitoring
|
|
on its own machine. Worth doing before the mirror holds a few hundred
|
|
gigabytes; not worth doing today at seven percent used.
|
|
|
|
## What is not here
|
|
|
|
No exporter on the llmgw nodes, so no host CPU, load average or network for
|
|
them. The control plane publishes memory, disk and liveness for every node,
|
|
which is what placement decides on; the rest is a `node_exporter` away if
|
|
somebody wants it. The monitoring host itself is scraped, for the reason above.
|
|
|
|
No probe of the mirror. Its `/v1/health` is JSON rather than Prometheus text,
|
|
and checking it properly means `blackbox_exporter`.
|
|
|
|
No recording rules. At this size every query in the dashboard runs fast enough
|
|
raw, and a recording rule is a second definition of a number to keep in step
|
|
with the first.
|