Ilker ManapandClaude Opus 5
c95cfbe426
CI / test (push) Successful in 1m48s
Where to put them turned out not to be a question about capacity. Measured on this cluster: 2,359 series at 41 samples a second, about 7 MB a day, so a year of history costs roughly 3 GB against 900 GB free. Retention was the package default of fifteen days, which answers none of the questions this data is for — what capacity looked like last quarter, whether it was always this slow, how much the fleet grew. A year, with a size cap as well. The time limit is the intent; the cap is what stops Prometheus becoming the thing that fills the disk if somebody adds a fleet and the series count grows tenfold. The part that matters is what shares that disk. The model mirror and the metrics database are on one filesystem, so a mirror that fills it stops Prometheus writing — monitoring goes quiet exactly when the disk is the problem, and the alert that would have said so is the one that died. Nothing was watching it. node_exporter arrives with the Prometheus package and was already running on that machine, unscraped, while llmgw's own disk metrics cover only its nodes — and the mirror is not one. It is a target now, and the alert fires at fifteen percent rather than five, because what is being protected is the ability to be told. No remote write and no second tier. At two thousand series that is infrastructure for a problem this cluster does not have. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>