Read nodes' heat and power: GPUs, CPUs, board inlet, RAPL
Release / build (push) Successful in 1m10s
Release / notes (push) Successful in 3s
CI / test (push) Successful in 3m51s

Every agent samples, at most every ten seconds: nvidia-smi's temperature,
HBM temperature, power draw, enforced limit and throttle reasons per GPU;
hwmon's CPU package and board inlet temperatures; and CPU package power
from RAPL's energy counters. Sent with heartbeats, published as metrics
and shown on the node page; unmeasured sensors are absent, never zero.
Observe-only, step 0 of docs/facility-aware-placement.md, which now
describes the agent source and what these readings can and cannot say.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-09-30 19:49:08 +02:00
1 parent 8724dc027a
commit cd10e086c8
15 files changed
+996 -172

No files matched your search

+48 -13
View File
@@ -1,9 +1,11 @@
# Facility-aware placement: rack power, temperature and cooling
**Status: planned, not built.** This is the design for step 5 of resource
allocation, after reserved CPU cores and memory (1), capacity classes and
preemption (2), tenant capacity quotas (3) and sizing by measured throughput
(4), all of which shipped in v0.10.5. Nothing here exists in the code yet.
**Status: planned; its first step, reading the nodes' own sensors, is built.**
This is the design for step 5 of resource allocation, after reserved CPU cores
and memory (1), capacity classes and preemption (2), tenant capacity quotas (3)
and sizing by measured throughput (4), all of which shipped in v0.10.5. Of
what follows, only the node sensors (rollout step 0) exist in the code: they
are read, published and shown, and change no decision.
## Why
@@ -35,7 +37,7 @@ and never let missing data stop the cluster.
| Headroom: the scheduler dry-run on a copy of the inventory | Room that respects power budgets, for free |
| Preemption with classes, minimums and a dry run (step 2) | Shedding load from a hot rack in the right order |
| Measured throughput per profile (step 4) | Tokens per watt as a placement preference |
| GPU temperature from NVML, per card | A node-level thermal signal that needs no facility data |
| Node sensors, read by every agent (step 0, built) | Heat and power with no facility integration: see below |
| `internal/promq`, a Prometheus client | Reading facility metrics that DCIM exporters publish |
## Concepts
@@ -116,8 +118,8 @@ starts nor ends anything:
| Condition | Enters when | Placement |
|---|---|---|
| `ok` | below the warm threshold for 5 min | normal |
| `warm` | inlet above the warm threshold for 2 min, or GPU temperatures near throttling | allowed, scored down: other domains first |
| `hot` | inlet above the hot threshold for 2 min, or a cooling unit in alarm | no new placements, as a cordon would, but automatic and reversible |
| `warm` | inlet above the warm threshold for 2 min, GPU temperatures near throttling, or any GPU thermally throttled | allowed, scored down: other domains first |
| `hot` | inlet above the hot threshold for 2 min, a cooling unit in alarm, or a sustained share of the domain's GPUs thermally throttled | no new placements, as a cordon would, but automatic and reversible |
| `unknown` | no fresh reading | as `ok`, and reported as unknown |
**Shedding** (optional, off by default): a domain `hot` for longer than its
@@ -186,9 +188,39 @@ type FacilitySource interface {
query: max by (row) (crac_alarm)
domain_label: row
```
- **Agents.** What the nodes can read themselves: GPU power and temperature from
NVML, node power from Redfish or IPMI where the BMC allows it. Needs no
facility integration, and gives node-level signals on day one.
- **Agents (built, step 0).** What each node reads about itself, every ten
seconds at most, sent with its heartbeat:
- **GPUs, from nvidia-smi:** temperature, memory (HBM) temperature where the
driver reports it, power draw, the enforced power limit, and what is
holding the clocks down now — `sw_power_cap`, `hw_slowdown`, `sw_thermal`,
`hw_thermal`, `power_brake`.
- **CPUs, from hwmon** (what lm_sensors reads, without it installed): the
package temperature, or the hottest core; and the board's inlet or ambient
sensor where it has one (`SYSTIN`, `Inlet`, `Ambient`).
- **CPU power, from RAPL:** the packages' energy counters over time,
subdomains not counted twice, counter wrap handled.
A machine without a sensor reports nothing for it, never a zero; a VM has
none of them. Published as `llmgw_gpu_temperature_celsius`,
`llmgw_gpu_memory_temperature_celsius`, `llmgw_gpu_power_watts`,
`llmgw_gpu_power_limit_watts`, `llmgw_gpu_throttled{reason}`,
`llmgw_node_cpu_temperature_celsius`, `llmgw_node_cpu_power_watts` and
`llmgw_node_inlet_temperature_celsius`, and on each node's page.
What they can and cannot say:
- **Throttling is the strongest signal there is.** A temperature is a proxy;
a card holding its clocks down is the fact that invalidates its measured
capacity, and it needs no threshold to be chosen.
- **Measured power** is GPU draw plus CPU packages; memory, disks, fans and
power-supply losses are not in it, so a node's draw is that plus an
estimated remainder. A rack's is its llmgw nodes' plus a declared figure
for everything else in it: switches and storage report nothing here.
- **CPU and GPU temperatures follow the load, not the room.** A hot rack is
inferred from many nodes running hot at comparable load, or from
throttling, not from one node's CPU. Only a board inlet sensor reads the
air, and most boards have none.
- **No budget comes from a sensor.** A reading is a draw; the cap it is
measured against still comes from the facility.
- **Simulated.** Readings from a script, for tests and for rehearsing a
thermal event on a cluster without facility data.
@@ -223,14 +255,17 @@ type FacilitySource interface {
Each step ships on its own and is useful without the next:
0. **Node sensors, observe-only (built).** GPU heat, power and throttling,
CPU heat and power, inlet air where the board has a sensor; published and
shown, no decision changed.
1. **Location and spreading.** Well-known labels, domains from labels, replicas
spread across racks and PDUs, locations on the node views. No facility data
needed.
2. **Power budgets from a static file.** Caps and margins, estimated draw per
replica (declared or derived), hard limit in placement and headroom.
3. **Readings, observe-only.** The Prometheus and agent sources, conditions
with hysteresis, measured power folded into headroom; decisions shown, not
enforced.
3. **Readings, observe-only.** The Prometheus source beside the agents',
conditions with hysteresis (throttling first among them), measured power
folded into headroom; decisions shown, not enforced.
4. **Thermal enforcement.** `warm` scored down, `hot` refused, manual hold and
release.
5. **Shedding and tokens per watt.** Opt-in shedding through step 2's