A btop-like monitor in the console, from Prometheus; node exporters discovered and installed
Monitor draws CPU, memory, GPUs, requests, tokens, latency, network and disk as Braille charts over 5m/1h/24h/7d, then per node, card and model sparklines; node and model pages have their own. History is read from monitoring.prometheus through POST /api/v1/metrics/range, or without it from what the console saw. The control plane serves the cluster's machines for Prometheus http_sd at /prometheus/targets on its metrics port, and node add installs prometheus-node-exporter. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
2e55487eca
commit
a1b1663e9d
16 files changed
+1217
-4
No files matched your search
@@ -57,6 +57,19 @@ read, and Enter brings the console back.
|
||||
**Overview** is `llmgw ctl status`: nodes, GPUs by class, fabrics, replicas,
|
||||
models, the store's members, and anything parked, unplaced or idle.
|
||||
|
||||
**Monitor** draws the cluster the way btop draws a machine: CPU, memory,
|
||||
GPU busy and GPU memory, requests and tokens a second, time to first token,
|
||||
requests in flight and queued, network and disk — Braille charts, filled
|
||||
and coloured green to red by height — over 5 minutes, an hour, a day or a
|
||||
week (**1**–**4**, or **R** for the next). Below them each node's CPU,
|
||||
memory and network now and as a sparkline, each card's busy, memory, heat
|
||||
and power, and each model's requests a second. The history comes from the
|
||||
Prometheus named by `monitoring.prometheus` in server.yaml; without one the
|
||||
charts draw what this console has seen since it opened, and say so. Network
|
||||
and disk come from each machine's node exporter, which `llmgw node add`
|
||||
installs. A node's and a model's own pages have their charts too, over the
|
||||
same range.
|
||||
|
||||
**Nodes** lists every node with its state, load, GPUs, replicas, disk and
|
||||
agent version. On a row:
|
||||
|
||||
|
||||
Reference in new issue
Block a user