Put the cluster's own charts on its own admin page
CI / test (push) Successful in 1m50s

The dashboard charted what the cluster had been billed for — requests, tokens,
spend, all from llmgw's own store — and nothing about whether it was healthy.
Saturation, queue depth, cache occupancy and replica counts were numbers in a
table at the instant the page loaded, with no way to see that one of them had
been climbing for an hour.

Four line charts now, read back from wherever the metrics were scraped to.

Read back rather than kept twice. llmgw publishes these already; what a gauge
cannot do is remember, and a ring buffer in this process would be a rival copy
of the same numbers with its own retention, lost on every restart and
disagreeing with the one an operator has on screen elsewhere. monitoring.prometheus
is empty by default and the panels are then absent rather than empty.

One axis per chart, always. Two measures of different scale belong in two
charts: a second y-axis lets any two lines be made to cross wherever the author
likes and the reader cannot tell it happened.

Four colours, handed out in fixed order, never cycled. A fifth series is summed
into "other" and drawn in grey — dropping it would make a chart that does not
add up, and inventing a hue for it would put two lines on the page that nobody
can tell apart. The fold is by timestamp, not by position: the tail's series do
not share sample times, and adding them by index slides points sideways onto
each other's clock.

A legend for every chart with more than one line, because colour alone is not
an encoding a reader can rely on.

A flat zero is drawn along the bottom rather than left blank: nothing happening
is a real answer, and an empty frame reads as a broken chart. A single sample
is not a line and says so instead of putting a dot in the corner.

One failed query fails all four panels. Three drawn and one blank invites
reading the blank one as "nothing is happening there".

NaN is dropped rather than plotted. Prometheus sends it as a string for a
sample it has no value for, Go's ParseFloat accepts it happily, and one NaN
coordinate is an SVG path that does not draw at all — taking the rest of the
line with it.

Axis maxima are rounded to a number a person would pick. An axis topping out at
0.8347 makes every gridline unreadable, and the shape is what a chart is for.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5 committed 2026-09-27 18:25:56 +02:00
1 parent b921fa0d6b
commit c6df0659af
14 files changed
+975 -1

No files matched your search

+44
View File
@@ -120,6 +120,28 @@ requested whenever a fallback served it.</p>
{{else}}<p class="empty">No agents have registered.</p>{{end}}
<h2>Models</h2>
{{with .Fleet}}
<h2>What the cluster is doing <span class="opt">last {{.Window}}</span></h2>
{{if .Err}}
{{/* Said outright. An empty panel reads as a healthy cluster with nothing
happening, which is the opposite of not knowing. */}}
<p class="note error-row">No history: {{.Err}}</p>
{{else}}
<div class="chart-row">
{{template "tschart" dict "t" .Saturation "title" "Pool saturation, by model"
"note" "The worse of queue depth and cache pressure. 1 is the limit measured as healthy, not the point of failure."}}
{{template "tschart" dict "t" .Ready "title" "Replicas ready, by model"
"note" "A model deliberately stopped has no line at all, which is why an on-demand catalogue looks quiet here."}}
</div>
<div class="chart-row">
{{template "tschart" dict "t" .Queue "title" "Engine queue depth, by replica"
"note" "Requests accepted and not yet started. The first thing to move when a replica is falling behind."}}
{{template "tschart" dict "t" .KV "title" "KV cache occupancy, by replica"
"note" "How full the attention cache is. Long contexts fill it with an empty queue, which is why saturation reads both."}}
</div>
{{end}}
{{end}}
{{if .Models}}
<table><tr><th>Model</th><th>Aliases</th><th class="num">Ready</th><th class="num">Wanted</th><th>Profiles</th><th>Fallbacks</th></tr>
{{range .Models}}<tr>
@@ -205,3 +227,25 @@ exposed:
<span>{{.Now}}</span>
</footer>
</div></body></html>{{end}}
{{/* One chart: the SVG, its legend, and the sentence that says how to read it.
A legend is always present for two lines or more — colour alone is not an
encoding a reader can rely on — and a single line needs none, because the
title already names it. */}}
{{define "tschart"}}
<div class="chart-wrap">
<h3 class="chart-title">{{.title}}</h3>
{{if .t.Empty}}
<p class="empty">Nothing recorded yet.</p>
{{else}}
{{.t.SVG}}
<span class="chart-peak">peak {{.t.Peak}}</span>
{{if gt (len .t.Legend) 1}}
<div class="legend">
{{range .t.Legend}}<span><i style="background:var({{cssvar .Colour}})"></i>{{.Name}} <b>{{.Last}}</b></span>{{end}}
</div>
{{end}}
{{end}}
<p class="note">{{.note}}</p>
</div>
{{end}}