Where to put them turned out not to be a question about capacity. Measured on
this cluster: 2,359 series at 41 samples a second, about 7 MB a day, so a year
of history costs roughly 3 GB against 900 GB free. Retention was the package
default of fifteen days, which answers none of the questions this data is for —
what capacity looked like last quarter, whether it was always this slow, how
much the fleet grew.
A year, with a size cap as well. The time limit is the intent; the cap is what
stops Prometheus becoming the thing that fills the disk if somebody adds a
fleet and the series count grows tenfold.
The part that matters is what shares that disk. The model mirror and the
metrics database are on one filesystem, so a mirror that fills it stops
Prometheus writing — monitoring goes quiet exactly when the disk is the
problem, and the alert that would have said so is the one that died.
Nothing was watching it. node_exporter arrives with the Prometheus package and
was already running on that machine, unscraped, while llmgw's own disk metrics
cover only its nodes — and the mirror is not one. It is a target now, and the
alert fires at fifteen percent rather than five, because what is being
protected is the ability to be told.
No remote write and no second tier. At two thousand series that is
infrastructure for a problem this cluster does not have.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
