diff --git a/deploy/monitoring/README.md b/deploy/monitoring/README.md index efc6802..3947d74 100644 --- a/deploy/monitoring/README.md +++ b/deploy/monitoring/README.md @@ -111,11 +111,56 @@ absurd on a GPU; change it when the hardware changes. directory reports nothing rather than zero, and the rule divides free by total so a node that reports neither never fires. +## Where the metrics live + +On the machine that scrapes them, in Prometheus's own store, and that is the +right answer at this size. Measured on this cluster: 2,359 series at 41 samples +a second, which compresses to roughly 7 MB a day — about 3 GB for a year. + +``` +# /etc/default/prometheus +ARGS="--storage.tsdb.retention.time=365d --storage.tsdb.retention.size=20GB" +``` + +A year because the questions worth asking of this data are seasonal: what did +capacity look like last quarter, was it always this slow, how much did the +fleet grow. Fifteen days answers none of them. + +Both limits, not one. The time limit is the intent; the size cap is what stops +Prometheus becoming the thing that fills the disk if the series count grows +tenfold after somebody adds a fleet. + +No remote write, no object store, no second tier. At two thousand series that +is infrastructure for a problem this cluster does not have, and every one of +those has its own failure modes to learn. + +### The part that is not about capacity + +The disk is not the constraint — 900 GB free against 3 GB a year. What matters +is what shares it. + +On this cluster the model mirror and the metrics database are on the same +filesystem. A mirror that fills it stops Prometheus writing, so monitoring goes +quiet at exactly the moment the disk is the problem, and the alert that would +have said so is the one that died. + +Two ways out, in order of how much they cost: + +- Watch it, which is now done. `prometheus-node-exporter` arrives with the + Prometheus package and was running, unscraped, on this machine — so the disk + holding both was watched by nothing. It is a scrape target now, and + `MonitoringDiskFilling` fires at fifteen percent rather than five, because + the thing being protected is the ability to be told. +- Separate them: a volume of its own for `/var/lib/prometheus`, or monitoring + on its own machine. Worth doing before the mirror holds a few hundred + gigabytes; not worth doing today at seven percent used. + ## What is not here -No exporter on each node, so no host CPU, load average or network. The control -plane publishes memory, disk and liveness for every node, which is what -placement decides on; the rest is a `node_exporter` away if somebody wants it. +No exporter on the llmgw nodes, so no host CPU, load average or network for +them. The control plane publishes memory, disk and liveness for every node, +which is what placement decides on; the rest is a `node_exporter` away if +somebody wants it. The monitoring host itself is scraped, for the reason above. No probe of the mirror. Its `/v1/health` is JSON rather than Prometheus text, and checking it properly means `blackbox_exporter`. diff --git a/deploy/monitoring/llmgw-rules.yml b/deploy/monitoring/llmgw-rules.yml index 520efa0..a35afc3 100644 --- a/deploy/monitoring/llmgw-rules.yml +++ b/deploy/monitoring/llmgw-rules.yml @@ -91,6 +91,31 @@ groups: is skipped by placement rather than failing, so this is quiet until somebody wonders why a model will not land there. + - name: monitoring-host + rules: + - alert: MonitoringDiskFilling + expr: >- + node_filesystem_avail_bytes{job="monitoring-host",fstype!~"tmpfs|overlay|squashfs"} + / node_filesystem_size_bytes{job="monitoring-host",fstype!~"tmpfs|overlay|squashfs"} < 0.15 + for: 10m + labels: { severity: warning } + annotations: + summary: "{{ $labels.mountpoint }} on the monitoring host is {{ $value | humanizePercentage }} free" + description: >- + This filesystem holds the model mirror and the metrics database at + once. Filling it stops new weights arriving and stops Prometheus + writing — so monitoring goes quiet at the moment it is most needed, + which is why this fires at fifteen percent rather than five. + + - alert: MonitoringDiskAlmostFull + expr: >- + node_filesystem_avail_bytes{job="monitoring-host",fstype!~"tmpfs|overlay|squashfs"} + / node_filesystem_size_bytes{job="monitoring-host",fstype!~"tmpfs|overlay|squashfs"} < 0.05 + for: 5m + labels: { severity: critical } + annotations: + summary: "{{ $labels.mountpoint }} on the monitoring host has {{ $value | humanizePercentage }} left" + - name: llmgw-traffic rules: # These need traffic through the gateway to have any series at all. A diff --git a/deploy/monitoring/prometheus.yml b/deploy/monitoring/prometheus.yml index bcddc97..f69c501 100644 --- a/deploy/monitoring/prometheus.yml +++ b/deploy/monitoring/prometheus.yml @@ -30,3 +30,13 @@ scrape_configs: - job_name: prometheus static_configs: - targets: ['localhost:9090'] + + # The machine this runs on. It came with the Prometheus package and was not + # being scraped, which left the disk holding both the model mirror and the + # metrics themselves watched by nothing — so a mirror that filled it would + # take monitoring down and the alert that would have warned did not exist. + - job_name: monitoring-host + static_configs: + - targets: ['localhost:9100'] + labels: + instance: monitoring