Ilker ManapandClaude Opus 5
c95cfbe426
CI / test (push) Successful in 1m48s
Where to put them turned out not to be a question about capacity. Measured on this cluster: 2,359 series at 41 samples a second, about 7 MB a day, so a year of history costs roughly 3 GB against 900 GB free. Retention was the package default of fifteen days, which answers none of the questions this data is for — what capacity looked like last quarter, whether it was always this slow, how much the fleet grew. A year, with a size cap as well. The time limit is the intent; the cap is what stops Prometheus becoming the thing that fills the disk if somebody adds a fleet and the series count grows tenfold. The part that matters is what shares that disk. The model mirror and the metrics database are on one filesystem, so a mirror that fills it stops Prometheus writing — monitoring goes quiet exactly when the disk is the problem, and the alert that would have said so is the one that died. Nothing was watching it. node_exporter arrives with the Prometheus package and was already running on that machine, unscraped, while llmgw's own disk metrics cover only its nodes — and the mirror is not one. It is a target now, and the alert fires at fifteen percent rather than five, because what is being protected is the ability to be told. No remote write and no second tier. At two thousand series that is infrastructure for a problem this cluster does not have. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
43 lines
1.3 KiB
YAML
43 lines
1.3 KiB
YAML
# Prometheus for the llmgw cluster.
|
|
#
|
|
# One scrape target for the whole fleet. Agents publish nothing of their own —
|
|
# the control plane gathers what they report and republishes it, so node health,
|
|
# disk and endpoint state all arrive here from 10.0.0.140.
|
|
global:
|
|
scrape_interval: 15s
|
|
evaluation_interval: 15s
|
|
external_labels:
|
|
cluster: mynodes
|
|
|
|
alerting:
|
|
alertmanagers:
|
|
- static_configs:
|
|
- targets: ['localhost:9093']
|
|
|
|
rule_files:
|
|
- /etc/prometheus/rules/llmgw.yml
|
|
|
|
scrape_configs:
|
|
# The control plane's dedicated metrics port. Not its admin listener: that one
|
|
# is loopback because it also serves pprof, and pprof can dump the heap where
|
|
# API keys sit in plaintext.
|
|
- job_name: llmgw
|
|
static_configs:
|
|
- targets: ['10.0.0.140:9091']
|
|
labels:
|
|
instance: control-plane
|
|
|
|
- job_name: prometheus
|
|
static_configs:
|
|
- targets: ['localhost:9090']
|
|
|
|
# The machine this runs on. It came with the Prometheus package and was not
|
|
# being scraped, which left the disk holding both the model mirror and the
|
|
# metrics themselves watched by nothing — so a mirror that filled it would
|
|
# take monitoring down and the alert that would have warned did not exist.
|
|
- job_name: monitoring-host
|
|
static_configs:
|
|
- targets: ['localhost:9100']
|
|
labels:
|
|
instance: monitoring
|