A btop-like monitor in the console, from Prometheus; node exporters discovered and installed

Monitor draws CPU, memory, GPUs, requests, tokens, latency, network and disk
as Braille charts over 5m/1h/24h/7d, then per node, card and model
sparklines; node and model pages have their own. History is read from
monitoring.prometheus through POST /api/v1/metrics/range, or without it
from what the console saw. The control plane serves the cluster's machines
for Prometheus http_sd at /prometheus/targets on its metrics port, and node
add installs prometheus-node-exporter.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-04 01:51:05 +02:00
1 parent 2e55487eca
commit a1b1663e9d
16 files changed
+1217 -4

No files matched your search

+20
View File
@@ -76,6 +76,26 @@ is not an update.
A model from `server.yaml` is reported but not changed: its source is edited
there.
### The monitor: the cluster drawn like btop
The console has a **Monitor** under Watch: CPU, memory, GPU busy and memory,
requests, tokens, time to first token, in flight and queued, network and
disk, as Braille charts filled and coloured by height, over 5 minutes, an
hour, a day or a week (1–4 or R); then each node's CPU, memory and network,
each card's busy, memory, heat and power, and each model's requests, as
sparklines. A node's and a model's own pages carry their charts too.
The history is read from the Prometheus in `monitoring.prometheus`, through
the admin API (`POST /api/v1/metrics/range`); without one, the charts draw
what the console has seen since it opened.
The control plane now lists the cluster's machines for Prometheus' service
discovery, at `/prometheus/targets` on its metrics port — nodes, itself and
the store's members, each labelled with its llmgw id — and `llmgw node add`
installs `prometheus-node-exporter` on the machine (`--no-node-exporter`
leaves it off). `deploy/monitoring/prometheus.yml` scrapes them as the
`nodes` job.
### Serving and updating run in the background
Serving a model or moving one to new weights from the console starts a job