llmgw/docs/facility-aware-placement.md

Facility-aware placement: rack power, temperature and cooling

Status: steps 0, 1 and 2 are built, with the manual hold of step 4; temperature conditions (steps 3 and 4) and shedding (step 5) are not, until the thermal data of a real site is at hand — without it thresholds would be designed against guesses. This is the design for step 5 of resource allocation, after reserved CPU cores and memory (1), capacity classes and preemption (2), tenant capacity quotas (3) and sizing by measured throughput (4). What is built says what works today and where it departs from the design below; the rest of the document is the design.

What is built

Where a node is. Five labels: llmgw.io/site, room, row, rack and pdu (a comma separated list, for a dual-fed rack). They come from the node's agent.yaml under labels:, or from the control plane:

llmgw ctl label gpu01 llmgw.io/rack=r12-07 llmgw.io/pdu=r12-07-a,r12-07-b
llmgw ctl label gpu01 llmgw.io/rack-        # back to agent.yaml's value, if any

Labels set with ctl label are kept in the store, win over the agent's, and may be set before the node joins. A misspelt llmgw.io/ key is refused. A domain is named level:name — rack:r12-07, pdu:r12-07-a — and names are taken as given, so two rooms that both have a rack r1 must label them apart. A node without location labels is in no domain and places as it always did.

Spreading. A model's next replica goes first to a node sharing no rack and no PDU with the nodes the model already runs on; nodes sharing fewer come before nodes sharing more. It is an ordering, never a filter: a rack holding the only free cards still takes the replica. It ranks above preferring a node that already holds the weights — a second replica one breaker away from the first is worth less than the copy saved — and above bin packing.

Keeping a replica together. A profile's keep_within: rack (or row, room, site) keeps a multi-node replica's nodes in one such domain, inside the fabric or NVLink domain placement already requires; nodes without the label are passed over for it, with the reason.

Holds. llmgw ctl facility hold rack:r12-07 --reason "PDU swap" places nothing new in the domain, as a cordon of each of its nodes would, including nodes labelled into it later. --drain also drains its nodes, one replica at a time with the replacement up first; llmgw ctl facility release undoes both. Holds are kept in the store across restarts (the drain itself is node state, which is not: after a restart the hold still keeps work out, and replicas not yet moved stay where they are). Any domain can be held: a PDU holds every rack on it.

Power budgets. In server.yaml:

facility:
  margin: 0.1            # share of every cap kept free
  gpu_watts: 700         # a card that reports no power limit
  cpu_core_watts: 10     # a core reserved by a CPU replica (cpu_cores)
  node_watts: 350        # each node's own draw beyond its cards
  budgets:
    - domain: rack:r12-07
      cap_watts: 16000   # what one feed carries alone, derated
      other_watts: 800   # switches, storage
    - domain: pdu:r12-07-a
      cap_watts: 7400

A replica's draw is the profile's power_watts (split over its nodes) when declared; otherwise each card at the power limit it reports through the agent's sensors (or gpu_watts), and each reserved core at cpu_core_watts. A domain's use is, node by node, node_watts plus the larger of what is placed there and what the node's sensors read in the last two minutes (measured_fresh), plus the domain's other_watts. A shard that would take any of its node's domains past cap × (1 − margin) is refused; a multi-node replica's shards are counted together, so two that each fit a rack alone do not both go into it. A draw nobody knows — no declared figure, no card limit, no gpu_watts — counts as nothing: missing data never blocks.

What people see. llmgw ctl explain-placement names the domain: rack r12-07 is held: PDU swap, rack r12-07: 14.2 kW of a 14.4 kW limit (16.0 kW cap less 10%) in use, the shard needs ~2.8 kW. The rejections of a placed replica are kept too, so a replica in an unexpected rack says why. llmgw ctl nodes has a LOCATION column, shows ready, held, and filters with --domain rack:r12-07. llmgw ctl facility lists every domain with its nodes, use, limit, reserved and measured draw, and holds. The console's Nodes view has a WHERE column; a node's page shows its location and hold, and its menu sets the location (Location…) and holds or releases its rack. The admin API: GET /api/v1/facility, POST /api/v1/facility/holds, DELETE /api/v1/facility/holds/{domain}, POST /api/v1/nodes/{id}/labels.

Where it departs from the design below, and why.

• Domains come from labels alone; there is no facility file declaring them. Budgets are in server.yaml, the one file the control plane already reads. A DCIM export would generate that block.
• Feeds are not modelled. The cap given is what one feed carries alone, derated — the figure the design says matters — so the operator gives the one number instead of llmgw deriving it from two.
• The manual hold of step 4 is built now, without conditions: it needs no sensor data and answers the commonest facility event, planned maintenance. Its command is ctl facility hold, with a --drain the design did not have.
• keep_within is new: the request that a multi-node replica stay in one rack where the cabling requires it.
• Spreading is an ordering of candidates, not the scoring of the design; power headroom does not yet order racks ("level load across racks"), and multi-node replicas are not spread across racks.
• No llmgw_domain_* metrics or alerts yet; ctl facility and the API carry the same figures.

Why

In a large installation the scarce thing is often not a GPU but the room it sits in. Three failures are invisible to placement today:

• A rack goes over its power budget. Every node in it reports healthy; placement sees free cards and fills them; the breaker, or the facility's power cap, decides the rest. A rack with A and B feeds is worse: it runs fine until one feed fails and the other cannot carry the load.
• Hot hardware is slow hardware. GPUs throttle long before they fail. A replica placed in a rack whose inlet air is too warm serves fewer tokens per second than its measured capacity says, and nothing connects the two.
• Correlated failure. Two replicas of a model on two nodes in the same rack or on the same PDU are one failure away from zero. Placement spreads replicas across nodes, not across racks.

The goal is for placement to treat facility limits the way it already treats memory and cores: reserve against a budget, refuse what does not fit, say why, and never let missing data stop the cluster.

What it builds on

Already in place	Used for
Node labels, reported by agents, matched by node_selector	Where a node is: its rack, row, PDU
inventory.Allocation carries GPUs, memory and cores per shard	Adding watts, summed per rack and PDU
Placement rejections (explain-placement, headroom's "why not")	Naming the rack that is full or hot
Headroom: the scheduler dry-run on a copy of the inventory	Room that respects power budgets, for free
Preemption with classes, minimums and a dry run (step 2)	Shedding load from a hot rack in the right order
Measured throughput per profile (step 4)	Tokens per watt as a placement preference
Node sensors, read by every agent (step 0, built)	Heat and power with no facility integration: see below
internal/promq, a Prometheus client	Reading facility metrics that DCIM exporters publish

Concepts

Location. Each node has a place in a tree: site, room, row, rack. Power has its own tree beside it: a rack draws from one or more PDUs, and each PDU from a feed. Both are read from node labels with well-known keys:

# agent.yaml
labels:
  llmgw.io/site: sto1
  llmgw.io/room: hall-b
  llmgw.io/row: "12"
  llmgw.io/rack: r12-07
  llmgw.io/pdu: r12-07-a,r12-07-b   # both feeds, for a dual-fed rack

A node without these labels is its own rack of one: placement behaves as it does today.

Domain. A rack, a row, a PDU, a room: anything with a budget or a condition. Domains are declared in a facility file (below) or discovered from node labels; a label naming a domain the file does not declare creates it with no budget and no condition.

Budget. A domain may have a power cap in watts. Placement reserves each replica's estimated draw against every domain its node belongs to and refuses a placement that would take any of them past its cap less a safety margin. For a dual-fed rack the cap that matters is what one feed can carry alone, since that is what the rack must survive.

Condition. A domain has a thermal and cooling condition: ok, warm, hot or unknown, derived from readings against thresholds with hysteresis. Conditions change what placement prefers and allows; they are not amounts to fill.

Reading. A measured value with a time: inlet temperature, power draw, a cooling unit's alarm state. Readings older than their staleness limit are treated as absent.

Estimating a replica's draw

What placement reserves has to be known before the replica runs, so it is an estimate first and a measurement later:

1. Declared. A profile may say power_watts. The operator's figure wins.
2. Derived. Otherwise GPU profiles take each card's power limit from NVML (the agent already reads the cards; the power limit is one more field) plus a per-node share for CPU, memory and fans; CPU profiles take their reserved cores times a per-core figure for the node's CPU model.
3. Measured. Once replicas run, the agent reports their cards' actual draw (NVML) and, where the BMC allows, the node's (Redfish or IPMI DCMI). A domain's headroom is computed from the larger of what is reserved and what is measured, so an estimate that was too low is corrected by the room it stops offering, not by an outage.

Decisions

Power

• Hard limit. A placement that would take any domain past cap × (1 − margin) is rejected, with the domain and the figures: rack r12-07: 14.2 of 16.0 kW reserved, replica needs ~2.8, margin 10%.
• Feeds. For a domain with two feeds the cap is the smaller feed's rating times its derating (typically 80%), not the sum.
• Already over. A domain found over its cap, because the estimate was low or the cap was lowered, gets no new placements, and an alert. It is not emptied automatically; shedding power is the facility's decision, and stopping inference to do it is ours only if asked (see shedding below).

Temperature and cooling

Thresholds are per domain with defaults, and each condition needs to hold for a while before it changes, in both directions, so a single reading neither starts nor ends anything:

Condition	Enters when	Placement
ok	below the warm threshold for 5 min	normal
warm	inlet above the warm threshold for 2 min, GPU temperatures near throttling, or any GPU thermally throttled	allowed, scored down: other domains first
hot	inlet above the hot threshold for 2 min, a cooling unit in alarm, or a sustained share of the domain's GPUs thermally throttled	no new placements, as a cordon would, but automatic and reversible
unknown	no fresh reading	as ok, and reported as unknown

Shedding (optional, off by default): a domain hot for longer than its shed delay has load moved off it using step 2's machinery. The lowest class goes first, and replicas above their model's minimum go before any at it. A critical minimum is never touched, and moves are rate limited to one replica per domain per interval. The replica is placed elsewhere first where there is room, so shedding moves work rather than dropping it. GPU power capping (nvidia-smi -pl) is a gentler alternative worth measuring against shedding: less throughput, no restart.

Spreading

Replicas of one model are spread across racks, and across PDUs, the way they are spread across nodes today. This is a preference, dropped when it is the only thing standing in the way, as node spreading already is.

Scoring

Placement today filters, then takes the first candidate in a fixed order (fewest free GPUs for GPU profiles, fewest replicas for CPU ones, nodes holding the weights first). Facility signals are better as a score than as more filters. Candidates that pass the hard limits are ordered by:

1. condition: ok before warm;
2. power headroom left in the rack after this replica, more first, so load levels across racks;
3. the existing order.

Once measurements include power, step 4's profile preference can use tokens per watt instead of tokens per GPU where both are known.

Where the data comes from

A FacilitySource interface, implemented several ways, chosen in server.yaml:

type FacilitySource interface {
    // Domains is what the source knows: budgets, thresholds, members.
    Domains(ctx) ([]Domain, error)
    // Readings is the latest value of each signal, per domain.
    Readings(ctx) ([]Reading, error)
}

• Static file. Domains, caps and thresholds, written by hand or generated from a DCIM export. Enough for power budgets with no live data at all.

• Prometheus. DCIM, PDU (SNMP exporter) and BMC (Redfish exporter) readings usually end up in Prometheus. The source runs configured instant queries and maps their labels onto domains:

  facility:
    source: prometheus
    prometheus: http://monitoring:9090
    signals:
      inlet_temp_c:
        query: max by (rack) (dcim_rack_inlet_temperature_celsius)
        domain_label: rack
        stale_after: 2m
      power_watts:
        query: sum by (pdu) (pdu_outlet_power_watts)
        domain_label: pdu
      cooling_alarm:
        query: max by (row) (crac_alarm)
        domain_label: row
  

• Agents (built, step 0). What each node reads about itself, every ten seconds at most, sent with its heartbeat:

  • GPUs, from nvidia-smi: temperature, memory (HBM) temperature where the driver reports it, power draw, the enforced power limit, and what is holding the clocks down now — sw_power_cap, hw_slowdown, sw_thermal, hw_thermal, power_brake.
  • CPUs, from hwmon (what lm_sensors reads, without it installed): the package temperature, or the hottest core; and the board's inlet or ambient sensor where it has one (SYSTIN, Inlet, Ambient).
  • CPU power, from RAPL: the packages' energy counters over time, subdomains not counted twice, counter wrap handled.

  A machine without a sensor reports nothing for it, never a zero; a VM has none of them. Published as llmgw_gpu_temperature_celsius, llmgw_gpu_memory_temperature_celsius, llmgw_gpu_power_watts, llmgw_gpu_power_limit_watts, llmgw_gpu_throttled{reason}, llmgw_node_cpu_temperature_celsius, llmgw_node_cpu_power_watts and llmgw_node_inlet_temperature_celsius, and on each node's page.

  What they can and cannot say:

  • Throttling is the strongest signal there is. A temperature is a proxy; a card holding its clocks down is the fact that invalidates its measured capacity, and it needs no threshold to be chosen.
  • Measured power is GPU draw plus CPU packages; memory, disks, fans and power-supply losses are not in it, so a node's draw is that plus an estimated remainder. A rack's is its llmgw nodes' plus a declared figure for everything else in it: switches and storage report nothing here.
  • CPU and GPU temperatures follow the load, not the room. A hot rack is inferred from many nodes running hot at comparable load, or from throttling, not from one node's CPU. Only a board inlet sensor reads the air, and most boards have none.
  • No budget comes from a sensor. A reading is a draw; the cap it is measured against still comes from the facility.

• Simulated. Readings from a script, for tests and for rehearsing a thermal event on a cluster without facility data.

Safety

• Missing data never blocks. A source that is down, a reading that is stale, a node without location labels: placement behaves as it does today, and says the data is missing (a metric, a line on the cluster page). Facility awareness can make placement more careful, never impossible.
• Hysteresis and hold times on every condition, so a noisy sensor does not flap a rack between allowed and forbidden.
• Observe-only first. A mode where every decision is computed, logged and shown ("would have refused: rack r12-07 hot") but nothing is enforced, run for long enough to trust the thresholds before turning enforcement on.
• Manual override. llmgw ctl facility hold RACK and release RACK, for the facility team to mark a domain unavailable, or to clear a condition they know to be a sensor fault.
• Shedding is opt-in, per domain or cluster-wide, and rate limited.

What people see

• explain-placement and headroom name the domain: rack r12-07 is hot (inlet 31°C for 6 min), pdu r12-07-a: 7.1 of 7.4 kW.
• llmgw ctl facility lists domains with budget, reserved, measured and condition. The cluster page gets a rack summary beside the nodes table.
• Metrics: llmgw_domain_power_budget_watts, _reserved_watts, _measured_watts, llmgw_domain_condition{condition}, llmgw_domain_reading_age_seconds; alerts for a domain near its cap, a domain hot, a domain over its cap, and facility data gone stale.

Rollout

Each step ships on its own and is useful without the next:

0. Node sensors, observe-only (built). GPU heat, power and throttling, CPU heat and power, inlet air where the board has a sensor; published and shown, no decision changed.
1. Location and spreading (built). Well-known labels, domains from labels, replicas spread across racks and PDUs, locations on the node views. No facility data needed.
2. Power budgets from a static file (built, without headroom scoring). Caps and margins, estimated draw per replica (declared or derived), hard limit in placement and headroom.
3. Readings, observe-only. The Prometheus source beside the agents', conditions with hysteresis (throttling first among them), measured power folded into headroom; decisions shown, not enforced.
4. Thermal enforcement. warm scored down, hot refused; the manual hold and release are built.
5. Shedding and tokens per watt. Opt-in shedding through step 2's preemption; measured power in profile preference.

Testing

• The simulated source drives unit tests of every condition transition, the hysteresis, staleness, and the placement outcomes, with mutation checks as for steps 1–4.
• Property tests: whatever the readings, placement never exceeds a declared cap with fresh data, and never refuses everything because data is missing.
• On the PoC cluster, with no facility data: step 1 with labelled VMs, step 2 with a static file of made-up caps, steps 3–4 with the simulated source replaying a thermal event.
• Before enforcement anywhere real: a period in observe-only mode against the site's actual data.

Open questions

• What the sites have. Which DCIM, which PDUs, which BMCs, and whether any of it already reaches a Prometheus. The source design assumes Prometheus is the common ground; a site without it needs another source.
• Who owns the numbers. Caps, deratings and thresholds belong to the facility team. Whether they edit llmgw's file, or llmgw reads theirs.
• Phase balance. Three-phase PDUs have per-phase limits; whether placement should see phases or only the PDU total.
• Cooling granularity. Cooling is often per row or per aisle, not per rack; the domain tree allows either, but the defaults should match what sites actually measure.
• Power capping instead of shedding. Whether to lower GPU power limits in a warm rack rather than move replicas, and how that interacts with measured capacity from step 4, which would no longer hold at the lower limit.