Keep a year of metrics, and watch the disk they share with the models

Where to put them turned out not to be a question about capacity. Measured on
this cluster: 2,359 series at 41 samples a second, about 7 MB a day, so a year
of history costs roughly 3 GB against 900 GB free. Retention was the package
default of fifteen days, which answers none of the questions this data is for —
what capacity looked like last quarter, whether it was always this slow, how
much the fleet grew.

A year, with a size cap as well. The time limit is the intent; the cap is what
stops Prometheus becoming the thing that fills the disk if somebody adds a
fleet and the series count grows tenfold.

The part that matters is what shares that disk. The model mirror and the
metrics database are on one filesystem, so a mirror that fills it stops
Prometheus writing — monitoring goes quiet exactly when the disk is the
problem, and the alert that would have said so is the one that died.

Nothing was watching it. node_exporter arrives with the Prometheus package and
was already running on that machine, unscraped, while llmgw's own disk metrics
cover only its nodes — and the mirror is not one. It is a target now, and the
alert fires at fifteen percent rather than five, because what is being
protected is the ability to be told.

No remote write and no second tier. At two thousand series that is
infrastructure for a problem this cluster does not have.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5 committed 2026-09-27 18:57:06 +02:00
1 parent 9ef02dc9ec
commit c95cfbe426
3 files changed
+83 -3

No files matched your search

+48 -3
View File
@@ -111,11 +111,56 @@ absurd on a GPU; change it when the hardware changes.
directory reports nothing rather than zero, and the rule divides free by total
so a node that reports neither never fires.
## Where the metrics live
On the machine that scrapes them, in Prometheus's own store, and that is the
right answer at this size. Measured on this cluster: 2,359 series at 41 samples
a second, which compresses to roughly 7 MB a day — about 3 GB for a year.
```
# /etc/default/prometheus
ARGS="--storage.tsdb.retention.time=365d --storage.tsdb.retention.size=20GB"
```
A year because the questions worth asking of this data are seasonal: what did
capacity look like last quarter, was it always this slow, how much did the
fleet grow. Fifteen days answers none of them.
Both limits, not one. The time limit is the intent; the size cap is what stops
Prometheus becoming the thing that fills the disk if the series count grows
tenfold after somebody adds a fleet.
No remote write, no object store, no second tier. At two thousand series that
is infrastructure for a problem this cluster does not have, and every one of
those has its own failure modes to learn.
### The part that is not about capacity
The disk is not the constraint — 900 GB free against 3 GB a year. What matters
is what shares it.
On this cluster the model mirror and the metrics database are on the same
filesystem. A mirror that fills it stops Prometheus writing, so monitoring goes
quiet at exactly the moment the disk is the problem, and the alert that would
have said so is the one that died.
Two ways out, in order of how much they cost:
- Watch it, which is now done. `prometheus-node-exporter` arrives with the
Prometheus package and was running, unscraped, on this machine — so the disk
holding both was watched by nothing. It is a scrape target now, and
`MonitoringDiskFilling` fires at fifteen percent rather than five, because
the thing being protected is the ability to be told.
- Separate them: a volume of its own for `/var/lib/prometheus`, or monitoring
on its own machine. Worth doing before the mirror holds a few hundred
gigabytes; not worth doing today at seven percent used.
## What is not here
No exporter on each node, so no host CPU, load average or network. The control
plane publishes memory, disk and liveness for every node, which is what
placement decides on; the rest is a `node_exporter` away if somebody wants it.
No exporter on the llmgw nodes, so no host CPU, load average or network for
them. The control plane publishes memory, disk and liveness for every node,
which is what placement decides on; the rest is a `node_exporter` away if
somebody wants it. The monitoring host itself is scraped, for the reason above.
No probe of the mirror. Its `/v1/health` is JSON rather than Prometheus text,
and checking it properly means `blackbox_exporter`.
+25
View File
@@ -91,6 +91,31 @@ groups:
is skipped by placement rather than failing, so this is quiet until
somebody wonders why a model will not land there.
- name: monitoring-host
rules:
- alert: MonitoringDiskFilling
expr: >-
node_filesystem_avail_bytes{job="monitoring-host",fstype!~"tmpfs|overlay|squashfs"}
/ node_filesystem_size_bytes{job="monitoring-host",fstype!~"tmpfs|overlay|squashfs"} < 0.15
for: 10m
labels: { severity: warning }
annotations:
summary: "{{ $labels.mountpoint }} on the monitoring host is {{ $value | humanizePercentage }} free"
description: >-
This filesystem holds the model mirror and the metrics database at
once. Filling it stops new weights arriving and stops Prometheus
writing — so monitoring goes quiet at the moment it is most needed,
which is why this fires at fifteen percent rather than five.
- alert: MonitoringDiskAlmostFull
expr: >-
node_filesystem_avail_bytes{job="monitoring-host",fstype!~"tmpfs|overlay|squashfs"}
/ node_filesystem_size_bytes{job="monitoring-host",fstype!~"tmpfs|overlay|squashfs"} < 0.05
for: 5m
labels: { severity: critical }
annotations:
summary: "{{ $labels.mountpoint }} on the monitoring host has {{ $value | humanizePercentage }} left"
- name: llmgw-traffic
rules:
# These need traffic through the gateway to have any series at all. A
+10
View File
@@ -30,3 +30,13 @@ scrape_configs:
- job_name: prometheus
static_configs:
- targets: ['localhost:9090']
# The machine this runs on. It came with the Prometheus package and was not
# being scraped, which left the disk holding both the model mirror and the
# metrics themselves watched by nothing — so a mirror that filled it would
# take monitoring down and the alert that would have warned did not exist.
- job_name: monitoring-host
static_configs:
- targets: ['localhost:9100']
labels:
instance: monitoring