From 8724dc027ad15def32915df5c10f8a5cbec48336 Mon Sep 17 00:00:00 2001 From: Ilker Manap Date: Wed, 30 Sep 2026 19:25:31 +0200 Subject: [PATCH] Plan facility-aware placement: rack power, temperature and cooling A design document for step 5 of resource allocation, deferred: location labels and rack spreading, power budgets with estimated then measured draw, thermal and cooling conditions with hysteresis, opt-in shedding through preemption, pluggable facility sources, and a rollout that is useful at each step and safe when data is missing. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/facility-aware-placement.md | 266 +++++++++++++++++++++++++++++++ llmgw-plan.md | 5 +- 2 files changed, 269 insertions(+), 2 deletions(-) create mode 100644 docs/facility-aware-placement.md diff --git a/docs/facility-aware-placement.md b/docs/facility-aware-placement.md new file mode 100644 index 0000000..93388e0 --- /dev/null +++ b/docs/facility-aware-placement.md @@ -0,0 +1,266 @@ +# Facility-aware placement: rack power, temperature and cooling + +**Status: planned, not built.** This is the design for step 5 of resource +allocation, after reserved CPU cores and memory (1), capacity classes and +preemption (2), tenant capacity quotas (3) and sizing by measured throughput +(4), all of which shipped in v0.10.5. Nothing here exists in the code yet. + +## Why + +In a large installation the scarce thing is often not a GPU but the room it +sits in. Three failures are invisible to placement today: + +- **A rack goes over its power budget.** Every node in it reports healthy; + placement sees free cards and fills them; the breaker, or the facility's + power cap, decides the rest. A rack with A and B feeds is worse: it runs + fine until one feed fails and the other cannot carry the load. +- **Hot hardware is slow hardware.** GPUs throttle long before they fail. A + replica placed in a rack whose inlet air is too warm serves fewer tokens per + second than its measured capacity says, and nothing connects the two. +- **Correlated failure.** Two replicas of a model on two nodes in the same rack + or on the same PDU are one failure away from zero. Placement spreads replicas + across nodes, not across racks. + +The goal is for placement to treat facility limits the way it already treats +memory and cores: reserve against a budget, refuse what does not fit, say why, +and never let missing data stop the cluster. + +## What it builds on + +| Already in place | Used for | +|---|---| +| Node labels, reported by agents, matched by `node_selector` | Where a node is: its rack, row, PDU | +| `inventory.Allocation` carries GPUs, memory and cores per shard | Adding watts, summed per rack and PDU | +| Placement rejections (`explain-placement`, headroom's "why not") | Naming the rack that is full or hot | +| Headroom: the scheduler dry-run on a copy of the inventory | Room that respects power budgets, for free | +| Preemption with classes, minimums and a dry run (step 2) | Shedding load from a hot rack in the right order | +| Measured throughput per profile (step 4) | Tokens per watt as a placement preference | +| GPU temperature from NVML, per card | A node-level thermal signal that needs no facility data | +| `internal/promq`, a Prometheus client | Reading facility metrics that DCIM exporters publish | + +## Concepts + +**Location.** Each node has a place in a tree: site, room, row, rack. Power has +its own tree beside it: a rack draws from one or more PDUs, and each PDU from a +feed. Both are read from node labels with well-known keys: + +```yaml +# agent.yaml +labels: + llmgw.io/site: sto1 + llmgw.io/room: hall-b + llmgw.io/row: "12" + llmgw.io/rack: r12-07 + llmgw.io/pdu: r12-07-a,r12-07-b # both feeds, for a dual-fed rack +``` + +A node without these labels is its own rack of one: placement behaves as it +does today. + +**Domain.** A rack, a row, a PDU, a room: anything with a budget or a +condition. Domains are declared in a facility file (below) or discovered from +node labels; a label naming a domain the file does not declare creates it with +no budget and no condition. + +**Budget.** A domain may have a power cap in watts. Placement reserves each +replica's estimated draw against every domain its node belongs to and refuses a +placement that would take any of them past its cap less a safety margin. For a +dual-fed rack the cap that matters is what one feed can carry alone, since that +is what the rack must survive. + +**Condition.** A domain has a thermal and cooling condition: `ok`, `warm`, `hot` +or `unknown`, derived from readings against thresholds with hysteresis. +Conditions change what placement prefers and allows; they are not amounts to +fill. + +**Reading.** A measured value with a time: inlet temperature, power draw, a +cooling unit's alarm state. Readings older than their staleness limit are +treated as absent. + +## Estimating a replica's draw + +What placement reserves has to be known before the replica runs, so it is an +estimate first and a measurement later: + +1. **Declared.** A profile may say `power_watts`. The operator's figure wins. +2. **Derived.** Otherwise GPU profiles take each card's power limit from NVML + (the agent already reads the cards; the power limit is one more field) plus + a per-node share for CPU, memory and fans; CPU profiles take their reserved + cores times a per-core figure for the node's CPU model. +3. **Measured.** Once replicas run, the agent reports their cards' actual draw + (NVML) and, where the BMC allows, the node's (Redfish or IPMI DCMI). A + domain's headroom is computed from the larger of what is reserved and what + is measured, so an estimate that was too low is corrected by the room it + stops offering, not by an outage. + +## Decisions + +### Power + +- **Hard limit.** A placement that would take any domain past `cap × (1 − + margin)` is rejected, with the domain and the figures: *rack r12-07: 14.2 of + 16.0 kW reserved, replica needs ~2.8, margin 10%*. +- **Feeds.** For a domain with two feeds the cap is the smaller feed's rating + times its derating (typically 80%), not the sum. +- **Already over.** A domain found over its cap, because the estimate was low + or the cap was lowered, gets no new placements, and an alert. It is not + emptied automatically; shedding power is the facility's decision, and + stopping inference to do it is ours only if asked (see shedding below). + +### Temperature and cooling + +Thresholds are per domain with defaults, and each condition needs to hold for a +while before it changes, in both directions, so a single reading neither +starts nor ends anything: + +| Condition | Enters when | Placement | +|---|---|---| +| `ok` | below the warm threshold for 5 min | normal | +| `warm` | inlet above the warm threshold for 2 min, or GPU temperatures near throttling | allowed, scored down: other domains first | +| `hot` | inlet above the hot threshold for 2 min, or a cooling unit in alarm | no new placements, as a cordon would, but automatic and reversible | +| `unknown` | no fresh reading | as `ok`, and reported as unknown | + +**Shedding** (optional, off by default): a domain `hot` for longer than its +shed delay has load moved off it using step 2's machinery. The lowest class goes +first, and replicas above their model's minimum go before any at it. A critical +minimum is never touched, and moves are rate limited to one replica per domain +per interval. The replica is placed elsewhere first where there is room, so +shedding moves work rather than dropping it. GPU power capping (`nvidia-smi +-pl`) is a gentler alternative worth measuring against shedding: less +throughput, no restart. + +### Spreading + +Replicas of one model are spread across racks, and across PDUs, the way they +are spread across nodes today. This is a preference, dropped when it is the only +thing standing in the way, as node spreading already is. + +### Scoring + +Placement today filters, then takes the first candidate in a fixed order +(fewest free GPUs for GPU profiles, fewest replicas for CPU ones, nodes holding +the weights first). Facility signals are better as a score than as more +filters. Candidates that pass the hard limits are ordered by: + +1. condition: `ok` before `warm`; +2. power headroom left in the rack after this replica, more first, so load + levels across racks; +3. the existing order. + +Once measurements include power, step 4's profile preference can use tokens per +watt instead of tokens per GPU where both are known. + +## Where the data comes from + +A `FacilitySource` interface, implemented several ways, chosen in +`server.yaml`: + +```go +type FacilitySource interface { + // Domains is what the source knows: budgets, thresholds, members. + Domains(ctx) ([]Domain, error) + // Readings is the latest value of each signal, per domain. + Readings(ctx) ([]Reading, error) +} +``` + +- **Static file.** Domains, caps and thresholds, written by hand or generated + from a DCIM export. Enough for power budgets with no live data at all. +- **Prometheus.** DCIM, PDU (SNMP exporter) and BMC (Redfish exporter) + readings usually end up in Prometheus. The source runs configured instant + queries and maps their labels onto domains: + + ```yaml + facility: + source: prometheus + prometheus: http://monitoring:9090 + signals: + inlet_temp_c: + query: max by (rack) (dcim_rack_inlet_temperature_celsius) + domain_label: rack + stale_after: 2m + power_watts: + query: sum by (pdu) (pdu_outlet_power_watts) + domain_label: pdu + cooling_alarm: + query: max by (row) (crac_alarm) + domain_label: row + ``` +- **Agents.** What the nodes can read themselves: GPU power and temperature from + NVML, node power from Redfish or IPMI where the BMC allows it. Needs no + facility integration, and gives node-level signals on day one. +- **Simulated.** Readings from a script, for tests and for rehearsing a + thermal event on a cluster without facility data. + +## Safety + +- **Missing data never blocks.** A source that is down, a reading that is stale, + a node without location labels: placement behaves as it does today, and says + the data is missing (a metric, a line on the cluster page). Facility + awareness can make placement more careful, never impossible. +- **Hysteresis and hold times** on every condition, so a noisy sensor does not + flap a rack between allowed and forbidden. +- **Observe-only first.** A mode where every decision is computed, logged and + shown ("would have refused: rack r12-07 hot") but nothing is enforced, run + for long enough to trust the thresholds before turning enforcement on. +- **Manual override.** `llmgw ctl facility hold RACK` and `release RACK`, for the + facility team to mark a domain unavailable, or to clear a condition they know + to be a sensor fault. +- **Shedding is opt-in**, per domain or cluster-wide, and rate limited. + +## What people see + +- `explain-placement` and headroom name the domain: *rack r12-07 is hot (inlet + 31°C for 6 min)*, *pdu r12-07-a: 7.1 of 7.4 kW*. +- `llmgw ctl facility` lists domains with budget, reserved, measured and + condition. The cluster page gets a rack summary beside the nodes table. +- Metrics: `llmgw_domain_power_budget_watts`, `_reserved_watts`, + `_measured_watts`, `llmgw_domain_condition{condition}`, + `llmgw_domain_reading_age_seconds`; alerts for a domain near its cap, a + domain hot, a domain over its cap, and facility data gone stale. + +## Rollout + +Each step ships on its own and is useful without the next: + +1. **Location and spreading.** Well-known labels, domains from labels, replicas + spread across racks and PDUs, locations on the node views. No facility data + needed. +2. **Power budgets from a static file.** Caps and margins, estimated draw per + replica (declared or derived), hard limit in placement and headroom. +3. **Readings, observe-only.** The Prometheus and agent sources, conditions + with hysteresis, measured power folded into headroom; decisions shown, not + enforced. +4. **Thermal enforcement.** `warm` scored down, `hot` refused, manual hold and + release. +5. **Shedding and tokens per watt.** Opt-in shedding through step 2's + preemption; measured power in profile preference. + +## Testing + +- The simulated source drives unit tests of every condition transition, the + hysteresis, staleness, and the placement outcomes, with mutation checks as + for steps 1–4. +- Property tests: whatever the readings, placement never exceeds a declared + cap with fresh data, and never refuses everything because data is missing. +- On the PoC cluster, with no facility data: step 1 with labelled VMs, step 2 + with a static file of made-up caps, steps 3–4 with the simulated source + replaying a thermal event. +- Before enforcement anywhere real: a period in observe-only mode against the + site's actual data. + +## Open questions + +- **What the sites have.** Which DCIM, which PDUs, which BMCs, and whether any + of it already reaches a Prometheus. The source design assumes Prometheus is + the common ground; a site without it needs another source. +- **Who owns the numbers.** Caps, deratings and thresholds belong to the + facility team. Whether they edit llmgw's file, or llmgw reads theirs. +- **Phase balance.** Three-phase PDUs have per-phase limits; whether placement + should see phases or only the PDU total. +- **Cooling granularity.** Cooling is often per row or per aisle, not per rack; + the domain tree allows either, but the defaults should match what sites + actually measure. +- **Power capping instead of shedding.** Whether to lower GPU power limits in a + warm rack rather than move replicas, and how that interacts with measured + capacity from step 4, which would no longer hold at the lower limit. diff --git a/llmgw-plan.md b/llmgw-plan.md index 21f08cb..926b272 100644 --- a/llmgw-plan.md +++ b/llmgw-plan.md @@ -1485,8 +1485,9 @@ The launch path is now built too: - [x] Capacity quotas per organization, team and project: guarantees kept, borrowing taken back, limits enforced - [x] CPU cores reserved like memory, and memory required of CPU models -- [ ] Rack power, temperature and cooling as placement signals (a facility - source; not yet built) +- [ ] Rack power, temperature and cooling as placement signals: planned in + [docs/facility-aware-placement.md](docs/facility-aware-placement.md), + in five steps from rack spreading to opt-in shedding - [ ] Defragmentation for large tensor-parallel replicas across NVLink domains - [ ] Control plane HA (Raft) - [x] Weight downloading and cache management: the mirror, agent fetches