From c95cfbe426b8b84c29daed3cdd17778d1246323f Mon Sep 17 00:00:00 2001 From: Ilker Manap Date: Sun, 27 Sep 2026 18:57:06 +0200 Subject: [PATCH] Keep a year of metrics, and watch the disk they share with the models MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Where to put them turned out not to be a question about capacity. Measured on this cluster: 2,359 series at 41 samples a second, about 7 MB a day, so a year of history costs roughly 3 GB against 900 GB free. Retention was the package default of fifteen days, which answers none of the questions this data is for — what capacity looked like last quarter, whether it was always this slow, how much the fleet grew. A year, with a size cap as well. The time limit is the intent; the cap is what stops Prometheus becoming the thing that fills the disk if somebody adds a fleet and the series count grows tenfold. The part that matters is what shares that disk. The model mirror and the metrics database are on one filesystem, so a mirror that fills it stops Prometheus writing — monitoring goes quiet exactly when the disk is the problem, and the alert that would have said so is the one that died. Nothing was watching it. node_exporter arrives with the Prometheus package and was already running on that machine, unscraped, while llmgw's own disk metrics cover only its nodes — and the mirror is not one. It is a target now, and the alert fires at fifteen percent rather than five, because what is being protected is the ability to be told. No remote write and no second tier. At two thousand series that is infrastructure for a problem this cluster does not have. Co-Authored-By: Claude Opus 5 (1M context) --- deploy/monitoring/README.md | 51 +++++++++++++++++++++++++++++-- deploy/monitoring/llmgw-rules.yml | 25 +++++++++++++++ deploy/monitoring/prometheus.yml | 10 ++++++ 3 files changed, 83 insertions(+), 3 deletions(-) diff --git a/deploy/monitoring/README.md b/deploy/monitoring/README.md index efc68024..3947d740 100644 --- a/deploy/monitoring/README.md +++ b/deploy/monitoring/README.md @@ -111,11 +111,56 @@ absurd on a GPU; change it when the hardware changes. directory reports nothing rather than zero, and the rule divides free by total so a node that reports neither never fires. +## Where the metrics live + +On the machine that scrapes them, in Prometheus's own store, and that is the +right answer at this size. Measured on this cluster: 2,359 series at 41 samples +a second, which compresses to roughly 7 MB a day — about 3 GB for a year. + +``` +# /etc/default/prometheus +ARGS="--storage.tsdb.retention.time=365d --storage.tsdb.retention.size=20GB" +``` + +A year because the questions worth asking of this data are seasonal: what did +capacity look like last quarter, was it always this slow, how much did the +fleet grow. Fifteen days answers none of them. + +Both limits, not one. The time limit is the intent; the size cap is what stops +Prometheus becoming the thing that fills the disk if the series count grows +tenfold after somebody adds a fleet. + +No remote write, no object store, no second tier. At two thousand series that +is infrastructure for a problem this cluster does not have, and every one of +those has its own failure modes to learn. + +### The part that is not about capacity + +The disk is not the constraint — 900 GB free against 3 GB a year. What matters +is what shares it. + +On this cluster the model mirror and the metrics database are on the same +filesystem. A mirror that fills it stops Prometheus writing, so monitoring goes +quiet at exactly the moment the disk is the problem, and the alert that would +have said so is the one that died. + +Two ways out, in order of how much they cost: + +- Watch it, which is now done. `prometheus-node-exporter` arrives with the + Prometheus package and was running, unscraped, on this machine — so the disk + holding both was watched by nothing. It is a scrape target now, and + `MonitoringDiskFilling` fires at fifteen percent rather than five, because + the thing being protected is the ability to be told. +- Separate them: a volume of its own for `/var/lib/prometheus`, or monitoring + on its own machine. Worth doing before the mirror holds a few hundred + gigabytes; not worth doing today at seven percent used. + ## What is not here -No exporter on each node, so no host CPU, load average or network. The control -plane publishes memory, disk and liveness for every node, which is what -placement decides on; the rest is a `node_exporter` away if somebody wants it. +No exporter on the llmgw nodes, so no host CPU, load average or network for +them. The control plane publishes memory, disk and liveness for every node, +which is what placement decides on; the rest is a `node_exporter` away if +somebody wants it. The monitoring host itself is scraped, for the reason above. No probe of the mirror. Its `/v1/health` is JSON rather than Prometheus text, and checking it properly means `blackbox_exporter`. diff --git a/deploy/monitoring/llmgw-rules.yml b/deploy/monitoring/llmgw-rules.yml index 520efa0d..a35afc30 100644 --- a/deploy/monitoring/llmgw-rules.yml +++ b/deploy/monitoring/llmgw-rules.yml @@ -91,6 +91,31 @@ groups: is skipped by placement rather than failing, so this is quiet until somebody wonders why a model will not land there. + - name: monitoring-host + rules: + - alert: MonitoringDiskFilling + expr: >- + node_filesystem_avail_bytes{job="monitoring-host",fstype!~"tmpfs|overlay|squashfs"} + / node_filesystem_size_bytes{job="monitoring-host",fstype!~"tmpfs|overlay|squashfs"} < 0.15 + for: 10m + labels: { severity: warning } + annotations: + summary: "{{ $labels.mountpoint }} on the monitoring host is {{ $value | humanizePercentage }} free" + description: >- + This filesystem holds the model mirror and the metrics database at + once. Filling it stops new weights arriving and stops Prometheus + writing — so monitoring goes quiet at the moment it is most needed, + which is why this fires at fifteen percent rather than five. + + - alert: MonitoringDiskAlmostFull + expr: >- + node_filesystem_avail_bytes{job="monitoring-host",fstype!~"tmpfs|overlay|squashfs"} + / node_filesystem_size_bytes{job="monitoring-host",fstype!~"tmpfs|overlay|squashfs"} < 0.05 + for: 5m + labels: { severity: critical } + annotations: + summary: "{{ $labels.mountpoint }} on the monitoring host has {{ $value | humanizePercentage }} left" + - name: llmgw-traffic rules: # These need traffic through the gateway to have any series at all. A diff --git a/deploy/monitoring/prometheus.yml b/deploy/monitoring/prometheus.yml index bcddc97f..f69c501f 100644 --- a/deploy/monitoring/prometheus.yml +++ b/deploy/monitoring/prometheus.yml @@ -30,3 +30,13 @@ scrape_configs: - job_name: prometheus static_configs: - targets: ['localhost:9090'] + + # The machine this runs on. It came with the Prometheus package and was not + # being scraped, which left the disk holding both the model mirror and the + # metrics themselves watched by nothing — so a mirror that filled it would + # take monitoring down and the alert that would have warned did not exist. + - job_name: monitoring-host + static_configs: + - targets: ['localhost:9100'] + labels: + instance: monitoring