Measure each node's CPU and memory, and give every node a page
The nodes are CPU machines, and nothing measured how busy one was: the heartbeat carried GPU memory and disk, and llmgw_node_memory_bytes is the total. The agent now reads /proc between heartbeats - CPU across all cores, load averages, memory used and available, and each engine's process group's cores and resident memory - and sends it with the heartbeat. Nothing new runs on the nodes and Prometheus needs no new target: the control plane exports it with the rest of the fleet, as llmgw_node_cpu_utilization_percent, llmgw_node_cpus, llmgw_node_load, llmgw_node_memory_used_bytes, llmgw_node_memory_available_bytes, llmgw_shard_cpu_cores and llmgw_shard_memory_bytes. A measurement more than a minute old is left out rather than repeated. The cluster page's node table shows CPU and RAM as bars, and each node links to a page of its own: current CPU, RAM, load and disk, the engines on it with their cores and memory, and an hour of CPU, RAM, load, per-model cores and memory and queue depth from Prometheus. It refreshes in place. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
b06fb9f51f
commit
daed835d88
19 files changed
+1289
-102
No files matched your search
@@ -534,3 +534,11 @@ table.repo-files td:last-child .opt { display: block; margin-bottom: .2rem; }
|
||||
.own-use .tile { min-width: 9rem; }
|
||||
/* Between what the cluster is and how it is being used, on the cluster page. */
|
||||
hr.part { border: 0; border-top: 1px solid var(--line); margin: 2.25rem 0 .5rem; }
|
||||
/* How full a node is, in the cluster page's node table and on its own page. */
|
||||
td.load { white-space: nowrap; min-width: 8.5rem; }
|
||||
.minibar { display: inline-block; width: 3.2rem; height: .45rem; border-radius: 999px; background: var(--bg);
|
||||
border: 1px solid var(--line); overflow: hidden; vertical-align: middle; margin-right: .4rem; }
|
||||
.minibar > span { display: block; height: 100%; background: var(--accent); }
|
||||
.minibar.warm > span { background: var(--warn); }
|
||||
.minibar.hot > span { background: var(--bad); }
|
||||
.load-v { font-variant-numeric: tabular-nums; font-size: .85rem; }
|
||||
Reference in new issue
Block a user