llmgw/docs/facility-aware-placement.md

Facility-aware placement: rack power, temperature and cooling

Status: planned; its first step, reading the nodes' own sensors, is built. This is the design for step 5 of resource allocation, after reserved CPU cores and memory (1), capacity classes and preemption (2), tenant capacity quotas (3) and sizing by measured throughput (4), all of which shipped in v0.10.5. Of what follows, only the node sensors (rollout step 0) exist in the code: they are read, published and shown, and change no decision.

Why

In a large installation the scarce thing is often not a GPU but the room it sits in. Three failures are invisible to placement today:

• A rack goes over its power budget. Every node in it reports healthy; placement sees free cards and fills them; the breaker, or the facility's power cap, decides the rest. A rack with A and B feeds is worse: it runs fine until one feed fails and the other cannot carry the load.
• Hot hardware is slow hardware. GPUs throttle long before they fail. A replica placed in a rack whose inlet air is too warm serves fewer tokens per second than its measured capacity says, and nothing connects the two.
• Correlated failure. Two replicas of a model on two nodes in the same rack or on the same PDU are one failure away from zero. Placement spreads replicas across nodes, not across racks.

The goal is for placement to treat facility limits the way it already treats memory and cores: reserve against a budget, refuse what does not fit, say why, and never let missing data stop the cluster.

What it builds on

Already in place	Used for
Node labels, reported by agents, matched by node_selector	Where a node is: its rack, row, PDU
inventory.Allocation carries GPUs, memory and cores per shard	Adding watts, summed per rack and PDU
Placement rejections (explain-placement, headroom's "why not")	Naming the rack that is full or hot
Headroom: the scheduler dry-run on a copy of the inventory	Room that respects power budgets, for free
Preemption with classes, minimums and a dry run (step 2)	Shedding load from a hot rack in the right order
Measured throughput per profile (step 4)	Tokens per watt as a placement preference
Node sensors, read by every agent (step 0, built)	Heat and power with no facility integration: see below
internal/promq, a Prometheus client	Reading facility metrics that DCIM exporters publish

Concepts

Location. Each node has a place in a tree: site, room, row, rack. Power has its own tree beside it: a rack draws from one or more PDUs, and each PDU from a feed. Both are read from node labels with well-known keys:

# agent.yaml
labels:
  llmgw.io/site: sto1
  llmgw.io/room: hall-b
  llmgw.io/row: "12"
  llmgw.io/rack: r12-07
  llmgw.io/pdu: r12-07-a,r12-07-b   # both feeds, for a dual-fed rack

A node without these labels is its own rack of one: placement behaves as it does today.

Domain. A rack, a row, a PDU, a room: anything with a budget or a condition. Domains are declared in a facility file (below) or discovered from node labels; a label naming a domain the file does not declare creates it with no budget and no condition.

Budget. A domain may have a power cap in watts. Placement reserves each replica's estimated draw against every domain its node belongs to and refuses a placement that would take any of them past its cap less a safety margin. For a dual-fed rack the cap that matters is what one feed can carry alone, since that is what the rack must survive.

Condition. A domain has a thermal and cooling condition: ok, warm, hot or unknown, derived from readings against thresholds with hysteresis. Conditions change what placement prefers and allows; they are not amounts to fill.

Reading. A measured value with a time: inlet temperature, power draw, a cooling unit's alarm state. Readings older than their staleness limit are treated as absent.

Estimating a replica's draw

What placement reserves has to be known before the replica runs, so it is an estimate first and a measurement later:

1. Declared. A profile may say power_watts. The operator's figure wins.
2. Derived. Otherwise GPU profiles take each card's power limit from NVML (the agent already reads the cards; the power limit is one more field) plus a per-node share for CPU, memory and fans; CPU profiles take their reserved cores times a per-core figure for the node's CPU model.
3. Measured. Once replicas run, the agent reports their cards' actual draw (NVML) and, where the BMC allows, the node's (Redfish or IPMI DCMI). A domain's headroom is computed from the larger of what is reserved and what is measured, so an estimate that was too low is corrected by the room it stops offering, not by an outage.

Decisions

Power

• Hard limit. A placement that would take any domain past cap × (1 − margin) is rejected, with the domain and the figures: rack r12-07: 14.2 of 16.0 kW reserved, replica needs ~2.8, margin 10%.
• Feeds. For a domain with two feeds the cap is the smaller feed's rating times its derating (typically 80%), not the sum.
• Already over. A domain found over its cap, because the estimate was low or the cap was lowered, gets no new placements, and an alert. It is not emptied automatically; shedding power is the facility's decision, and stopping inference to do it is ours only if asked (see shedding below).

Temperature and cooling

Thresholds are per domain with defaults, and each condition needs to hold for a while before it changes, in both directions, so a single reading neither starts nor ends anything:

Condition	Enters when	Placement
ok	below the warm threshold for 5 min	normal
warm	inlet above the warm threshold for 2 min, GPU temperatures near throttling, or any GPU thermally throttled	allowed, scored down: other domains first
hot	inlet above the hot threshold for 2 min, a cooling unit in alarm, or a sustained share of the domain's GPUs thermally throttled	no new placements, as a cordon would, but automatic and reversible
unknown	no fresh reading	as ok, and reported as unknown

Shedding (optional, off by default): a domain hot for longer than its shed delay has load moved off it using step 2's machinery. The lowest class goes first, and replicas above their model's minimum go before any at it. A critical minimum is never touched, and moves are rate limited to one replica per domain per interval. The replica is placed elsewhere first where there is room, so shedding moves work rather than dropping it. GPU power capping (nvidia-smi -pl) is a gentler alternative worth measuring against shedding: less throughput, no restart.

Spreading

Replicas of one model are spread across racks, and across PDUs, the way they are spread across nodes today. This is a preference, dropped when it is the only thing standing in the way, as node spreading already is.

Scoring

Placement today filters, then takes the first candidate in a fixed order (fewest free GPUs for GPU profiles, fewest replicas for CPU ones, nodes holding the weights first). Facility signals are better as a score than as more filters. Candidates that pass the hard limits are ordered by:

1. condition: ok before warm;
2. power headroom left in the rack after this replica, more first, so load levels across racks;
3. the existing order.

Once measurements include power, step 4's profile preference can use tokens per watt instead of tokens per GPU where both are known.

Where the data comes from

A FacilitySource interface, implemented several ways, chosen in server.yaml:

type FacilitySource interface {
    // Domains is what the source knows: budgets, thresholds, members.
    Domains(ctx) ([]Domain, error)
    // Readings is the latest value of each signal, per domain.
    Readings(ctx) ([]Reading, error)
}

• Static file. Domains, caps and thresholds, written by hand or generated from a DCIM export. Enough for power budgets with no live data at all.

• Prometheus. DCIM, PDU (SNMP exporter) and BMC (Redfish exporter) readings usually end up in Prometheus. The source runs configured instant queries and maps their labels onto domains:

  facility:
    source: prometheus
    prometheus: http://monitoring:9090
    signals:
      inlet_temp_c:
        query: max by (rack) (dcim_rack_inlet_temperature_celsius)
        domain_label: rack
        stale_after: 2m
      power_watts:
        query: sum by (pdu) (pdu_outlet_power_watts)
        domain_label: pdu
      cooling_alarm:
        query: max by (row) (crac_alarm)
        domain_label: row
  

• Agents (built, step 0). What each node reads about itself, every ten seconds at most, sent with its heartbeat:

  • GPUs, from nvidia-smi: temperature, memory (HBM) temperature where the driver reports it, power draw, the enforced power limit, and what is holding the clocks down now — sw_power_cap, hw_slowdown, sw_thermal, hw_thermal, power_brake.
  • CPUs, from hwmon (what lm_sensors reads, without it installed): the package temperature, or the hottest core; and the board's inlet or ambient sensor where it has one (SYSTIN, Inlet, Ambient).
  • CPU power, from RAPL: the packages' energy counters over time, subdomains not counted twice, counter wrap handled.

  A machine without a sensor reports nothing for it, never a zero; a VM has none of them. Published as llmgw_gpu_temperature_celsius, llmgw_gpu_memory_temperature_celsius, llmgw_gpu_power_watts, llmgw_gpu_power_limit_watts, llmgw_gpu_throttled{reason}, llmgw_node_cpu_temperature_celsius, llmgw_node_cpu_power_watts and llmgw_node_inlet_temperature_celsius, and on each node's page.

  What they can and cannot say:

  • Throttling is the strongest signal there is. A temperature is a proxy; a card holding its clocks down is the fact that invalidates its measured capacity, and it needs no threshold to be chosen.
  • Measured power is GPU draw plus CPU packages; memory, disks, fans and power-supply losses are not in it, so a node's draw is that plus an estimated remainder. A rack's is its llmgw nodes' plus a declared figure for everything else in it: switches and storage report nothing here.
  • CPU and GPU temperatures follow the load, not the room. A hot rack is inferred from many nodes running hot at comparable load, or from throttling, not from one node's CPU. Only a board inlet sensor reads the air, and most boards have none.
  • No budget comes from a sensor. A reading is a draw; the cap it is measured against still comes from the facility.

• Simulated. Readings from a script, for tests and for rehearsing a thermal event on a cluster without facility data.

Safety

• Missing data never blocks. A source that is down, a reading that is stale, a node without location labels: placement behaves as it does today, and says the data is missing (a metric, a line on the cluster page). Facility awareness can make placement more careful, never impossible.
• Hysteresis and hold times on every condition, so a noisy sensor does not flap a rack between allowed and forbidden.
• Observe-only first. A mode where every decision is computed, logged and shown ("would have refused: rack r12-07 hot") but nothing is enforced, run for long enough to trust the thresholds before turning enforcement on.
• Manual override. llmgw ctl facility hold RACK and release RACK, for the facility team to mark a domain unavailable, or to clear a condition they know to be a sensor fault.
• Shedding is opt-in, per domain or cluster-wide, and rate limited.

What people see

• explain-placement and headroom name the domain: rack r12-07 is hot (inlet 31°C for 6 min), pdu r12-07-a: 7.1 of 7.4 kW.
• llmgw ctl facility lists domains with budget, reserved, measured and condition. The cluster page gets a rack summary beside the nodes table.
• Metrics: llmgw_domain_power_budget_watts, _reserved_watts, _measured_watts, llmgw_domain_condition{condition}, llmgw_domain_reading_age_seconds; alerts for a domain near its cap, a domain hot, a domain over its cap, and facility data gone stale.

Rollout

Each step ships on its own and is useful without the next:

0. Node sensors, observe-only (built). GPU heat, power and throttling, CPU heat and power, inlet air where the board has a sensor; published and shown, no decision changed.
1. Location and spreading. Well-known labels, domains from labels, replicas spread across racks and PDUs, locations on the node views. No facility data needed.
2. Power budgets from a static file. Caps and margins, estimated draw per replica (declared or derived), hard limit in placement and headroom.
3. Readings, observe-only. The Prometheus source beside the agents', conditions with hysteresis (throttling first among them), measured power folded into headroom; decisions shown, not enforced.
4. Thermal enforcement. warm scored down, hot refused, manual hold and release.
5. Shedding and tokens per watt. Opt-in shedding through step 2's preemption; measured power in profile preference.

Testing

• The simulated source drives unit tests of every condition transition, the hysteresis, staleness, and the placement outcomes, with mutation checks as for steps 1–4.
• Property tests: whatever the readings, placement never exceeds a declared cap with fresh data, and never refuses everything because data is missing.
• On the PoC cluster, with no facility data: step 1 with labelled VMs, step 2 with a static file of made-up caps, steps 3–4 with the simulated source replaying a thermal event.
• Before enforcement anywhere real: a period in observe-only mode against the site's actual data.

Open questions

• What the sites have. Which DCIM, which PDUs, which BMCs, and whether any of it already reaches a Prometheus. The source design assumes Prometheus is the common ground; a site without it needs another source.
• Who owns the numbers. Caps, deratings and thresholds belong to the facility team. Whether they edit llmgw's file, or llmgw reads theirs.
• Phase balance. Three-phase PDUs have per-phase limits; whether placement should see phases or only the PDU total.
• Cooling granularity. Cooling is often per row or per aisle, not per rack; the domain tree allows either, but the defaults should match what sites actually measure.
• Power capping instead of shedding. Whether to lower GPU power limits in a warm rack rather than move replicas, and how that interacts with measured capacity from step 4, which would no longer hold at the lower limit.