CI / test (push) Successful in 3m41s
A design document for step 5 of resource allocation, deferred: location labels and rack spreading, power budgets with estimated then measured draw, thermal and cooling conditions with hysteresis, opt-in shedding through preemption, pluggable facility sources, and a rollout that is useful at each step and safe when data is missing. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
267 lines
12 KiB
Markdown
267 lines
12 KiB
Markdown
# Facility-aware placement: rack power, temperature and cooling
|
||
|
||
**Status: planned, not built.** This is the design for step 5 of resource
|
||
allocation, after reserved CPU cores and memory (1), capacity classes and
|
||
preemption (2), tenant capacity quotas (3) and sizing by measured throughput
|
||
(4), all of which shipped in v0.10.5. Nothing here exists in the code yet.
|
||
|
||
## Why
|
||
|
||
In a large installation the scarce thing is often not a GPU but the room it
|
||
sits in. Three failures are invisible to placement today:
|
||
|
||
- **A rack goes over its power budget.** Every node in it reports healthy;
|
||
placement sees free cards and fills them; the breaker, or the facility's
|
||
power cap, decides the rest. A rack with A and B feeds is worse: it runs
|
||
fine until one feed fails and the other cannot carry the load.
|
||
- **Hot hardware is slow hardware.** GPUs throttle long before they fail. A
|
||
replica placed in a rack whose inlet air is too warm serves fewer tokens per
|
||
second than its measured capacity says, and nothing connects the two.
|
||
- **Correlated failure.** Two replicas of a model on two nodes in the same rack
|
||
or on the same PDU are one failure away from zero. Placement spreads replicas
|
||
across nodes, not across racks.
|
||
|
||
The goal is for placement to treat facility limits the way it already treats
|
||
memory and cores: reserve against a budget, refuse what does not fit, say why,
|
||
and never let missing data stop the cluster.
|
||
|
||
## What it builds on
|
||
|
||
| Already in place | Used for |
|
||
|---|---|
|
||
| Node labels, reported by agents, matched by `node_selector` | Where a node is: its rack, row, PDU |
|
||
| `inventory.Allocation` carries GPUs, memory and cores per shard | Adding watts, summed per rack and PDU |
|
||
| Placement rejections (`explain-placement`, headroom's "why not") | Naming the rack that is full or hot |
|
||
| Headroom: the scheduler dry-run on a copy of the inventory | Room that respects power budgets, for free |
|
||
| Preemption with classes, minimums and a dry run (step 2) | Shedding load from a hot rack in the right order |
|
||
| Measured throughput per profile (step 4) | Tokens per watt as a placement preference |
|
||
| GPU temperature from NVML, per card | A node-level thermal signal that needs no facility data |
|
||
| `internal/promq`, a Prometheus client | Reading facility metrics that DCIM exporters publish |
|
||
|
||
## Concepts
|
||
|
||
**Location.** Each node has a place in a tree: site, room, row, rack. Power has
|
||
its own tree beside it: a rack draws from one or more PDUs, and each PDU from a
|
||
feed. Both are read from node labels with well-known keys:
|
||
|
||
```yaml
|
||
# agent.yaml
|
||
labels:
|
||
llmgw.io/site: sto1
|
||
llmgw.io/room: hall-b
|
||
llmgw.io/row: "12"
|
||
llmgw.io/rack: r12-07
|
||
llmgw.io/pdu: r12-07-a,r12-07-b # both feeds, for a dual-fed rack
|
||
```
|
||
|
||
A node without these labels is its own rack of one: placement behaves as it
|
||
does today.
|
||
|
||
**Domain.** A rack, a row, a PDU, a room: anything with a budget or a
|
||
condition. Domains are declared in a facility file (below) or discovered from
|
||
node labels; a label naming a domain the file does not declare creates it with
|
||
no budget and no condition.
|
||
|
||
**Budget.** A domain may have a power cap in watts. Placement reserves each
|
||
replica's estimated draw against every domain its node belongs to and refuses a
|
||
placement that would take any of them past its cap less a safety margin. For a
|
||
dual-fed rack the cap that matters is what one feed can carry alone, since that
|
||
is what the rack must survive.
|
||
|
||
**Condition.** A domain has a thermal and cooling condition: `ok`, `warm`, `hot`
|
||
or `unknown`, derived from readings against thresholds with hysteresis.
|
||
Conditions change what placement prefers and allows; they are not amounts to
|
||
fill.
|
||
|
||
**Reading.** A measured value with a time: inlet temperature, power draw, a
|
||
cooling unit's alarm state. Readings older than their staleness limit are
|
||
treated as absent.
|
||
|
||
## Estimating a replica's draw
|
||
|
||
What placement reserves has to be known before the replica runs, so it is an
|
||
estimate first and a measurement later:
|
||
|
||
1. **Declared.** A profile may say `power_watts`. The operator's figure wins.
|
||
2. **Derived.** Otherwise GPU profiles take each card's power limit from NVML
|
||
(the agent already reads the cards; the power limit is one more field) plus
|
||
a per-node share for CPU, memory and fans; CPU profiles take their reserved
|
||
cores times a per-core figure for the node's CPU model.
|
||
3. **Measured.** Once replicas run, the agent reports their cards' actual draw
|
||
(NVML) and, where the BMC allows, the node's (Redfish or IPMI DCMI). A
|
||
domain's headroom is computed from the larger of what is reserved and what
|
||
is measured, so an estimate that was too low is corrected by the room it
|
||
stops offering, not by an outage.
|
||
|
||
## Decisions
|
||
|
||
### Power
|
||
|
||
- **Hard limit.** A placement that would take any domain past `cap × (1 −
|
||
margin)` is rejected, with the domain and the figures: *rack r12-07: 14.2 of
|
||
16.0 kW reserved, replica needs ~2.8, margin 10%*.
|
||
- **Feeds.** For a domain with two feeds the cap is the smaller feed's rating
|
||
times its derating (typically 80%), not the sum.
|
||
- **Already over.** A domain found over its cap, because the estimate was low
|
||
or the cap was lowered, gets no new placements, and an alert. It is not
|
||
emptied automatically; shedding power is the facility's decision, and
|
||
stopping inference to do it is ours only if asked (see shedding below).
|
||
|
||
### Temperature and cooling
|
||
|
||
Thresholds are per domain with defaults, and each condition needs to hold for a
|
||
while before it changes, in both directions, so a single reading neither
|
||
starts nor ends anything:
|
||
|
||
| Condition | Enters when | Placement |
|
||
|---|---|---|
|
||
| `ok` | below the warm threshold for 5 min | normal |
|
||
| `warm` | inlet above the warm threshold for 2 min, or GPU temperatures near throttling | allowed, scored down: other domains first |
|
||
| `hot` | inlet above the hot threshold for 2 min, or a cooling unit in alarm | no new placements, as a cordon would, but automatic and reversible |
|
||
| `unknown` | no fresh reading | as `ok`, and reported as unknown |
|
||
|
||
**Shedding** (optional, off by default): a domain `hot` for longer than its
|
||
shed delay has load moved off it using step 2's machinery. The lowest class goes
|
||
first, and replicas above their model's minimum go before any at it. A critical
|
||
minimum is never touched, and moves are rate limited to one replica per domain
|
||
per interval. The replica is placed elsewhere first where there is room, so
|
||
shedding moves work rather than dropping it. GPU power capping (`nvidia-smi
|
||
-pl`) is a gentler alternative worth measuring against shedding: less
|
||
throughput, no restart.
|
||
|
||
### Spreading
|
||
|
||
Replicas of one model are spread across racks, and across PDUs, the way they
|
||
are spread across nodes today. This is a preference, dropped when it is the only
|
||
thing standing in the way, as node spreading already is.
|
||
|
||
### Scoring
|
||
|
||
Placement today filters, then takes the first candidate in a fixed order
|
||
(fewest free GPUs for GPU profiles, fewest replicas for CPU ones, nodes holding
|
||
the weights first). Facility signals are better as a score than as more
|
||
filters. Candidates that pass the hard limits are ordered by:
|
||
|
||
1. condition: `ok` before `warm`;
|
||
2. power headroom left in the rack after this replica, more first, so load
|
||
levels across racks;
|
||
3. the existing order.
|
||
|
||
Once measurements include power, step 4's profile preference can use tokens per
|
||
watt instead of tokens per GPU where both are known.
|
||
|
||
## Where the data comes from
|
||
|
||
A `FacilitySource` interface, implemented several ways, chosen in
|
||
`server.yaml`:
|
||
|
||
```go
|
||
type FacilitySource interface {
|
||
// Domains is what the source knows: budgets, thresholds, members.
|
||
Domains(ctx) ([]Domain, error)
|
||
// Readings is the latest value of each signal, per domain.
|
||
Readings(ctx) ([]Reading, error)
|
||
}
|
||
```
|
||
|
||
- **Static file.** Domains, caps and thresholds, written by hand or generated
|
||
from a DCIM export. Enough for power budgets with no live data at all.
|
||
- **Prometheus.** DCIM, PDU (SNMP exporter) and BMC (Redfish exporter)
|
||
readings usually end up in Prometheus. The source runs configured instant
|
||
queries and maps their labels onto domains:
|
||
|
||
```yaml
|
||
facility:
|
||
source: prometheus
|
||
prometheus: http://monitoring:9090
|
||
signals:
|
||
inlet_temp_c:
|
||
query: max by (rack) (dcim_rack_inlet_temperature_celsius)
|
||
domain_label: rack
|
||
stale_after: 2m
|
||
power_watts:
|
||
query: sum by (pdu) (pdu_outlet_power_watts)
|
||
domain_label: pdu
|
||
cooling_alarm:
|
||
query: max by (row) (crac_alarm)
|
||
domain_label: row
|
||
```
|
||
- **Agents.** What the nodes can read themselves: GPU power and temperature from
|
||
NVML, node power from Redfish or IPMI where the BMC allows it. Needs no
|
||
facility integration, and gives node-level signals on day one.
|
||
- **Simulated.** Readings from a script, for tests and for rehearsing a
|
||
thermal event on a cluster without facility data.
|
||
|
||
## Safety
|
||
|
||
- **Missing data never blocks.** A source that is down, a reading that is stale,
|
||
a node without location labels: placement behaves as it does today, and says
|
||
the data is missing (a metric, a line on the cluster page). Facility
|
||
awareness can make placement more careful, never impossible.
|
||
- **Hysteresis and hold times** on every condition, so a noisy sensor does not
|
||
flap a rack between allowed and forbidden.
|
||
- **Observe-only first.** A mode where every decision is computed, logged and
|
||
shown ("would have refused: rack r12-07 hot") but nothing is enforced, run
|
||
for long enough to trust the thresholds before turning enforcement on.
|
||
- **Manual override.** `llmgw ctl facility hold RACK` and `release RACK`, for the
|
||
facility team to mark a domain unavailable, or to clear a condition they know
|
||
to be a sensor fault.
|
||
- **Shedding is opt-in**, per domain or cluster-wide, and rate limited.
|
||
|
||
## What people see
|
||
|
||
- `explain-placement` and headroom name the domain: *rack r12-07 is hot (inlet
|
||
31°C for 6 min)*, *pdu r12-07-a: 7.1 of 7.4 kW*.
|
||
- `llmgw ctl facility` lists domains with budget, reserved, measured and
|
||
condition. The cluster page gets a rack summary beside the nodes table.
|
||
- Metrics: `llmgw_domain_power_budget_watts`, `_reserved_watts`,
|
||
`_measured_watts`, `llmgw_domain_condition{condition}`,
|
||
`llmgw_domain_reading_age_seconds`; alerts for a domain near its cap, a
|
||
domain hot, a domain over its cap, and facility data gone stale.
|
||
|
||
## Rollout
|
||
|
||
Each step ships on its own and is useful without the next:
|
||
|
||
1. **Location and spreading.** Well-known labels, domains from labels, replicas
|
||
spread across racks and PDUs, locations on the node views. No facility data
|
||
needed.
|
||
2. **Power budgets from a static file.** Caps and margins, estimated draw per
|
||
replica (declared or derived), hard limit in placement and headroom.
|
||
3. **Readings, observe-only.** The Prometheus and agent sources, conditions
|
||
with hysteresis, measured power folded into headroom; decisions shown, not
|
||
enforced.
|
||
4. **Thermal enforcement.** `warm` scored down, `hot` refused, manual hold and
|
||
release.
|
||
5. **Shedding and tokens per watt.** Opt-in shedding through step 2's
|
||
preemption; measured power in profile preference.
|
||
|
||
## Testing
|
||
|
||
- The simulated source drives unit tests of every condition transition, the
|
||
hysteresis, staleness, and the placement outcomes, with mutation checks as
|
||
for steps 1–4.
|
||
- Property tests: whatever the readings, placement never exceeds a declared
|
||
cap with fresh data, and never refuses everything because data is missing.
|
||
- On the PoC cluster, with no facility data: step 1 with labelled VMs, step 2
|
||
with a static file of made-up caps, steps 3–4 with the simulated source
|
||
replaying a thermal event.
|
||
- Before enforcement anywhere real: a period in observe-only mode against the
|
||
site's actual data.
|
||
|
||
## Open questions
|
||
|
||
- **What the sites have.** Which DCIM, which PDUs, which BMCs, and whether any
|
||
of it already reaches a Prometheus. The source design assumes Prometheus is
|
||
the common ground; a site without it needs another source.
|
||
- **Who owns the numbers.** Caps, deratings and thresholds belong to the
|
||
facility team. Whether they edit llmgw's file, or llmgw reads theirs.
|
||
- **Phase balance.** Three-phase PDUs have per-phase limits; whether placement
|
||
should see phases or only the PDU total.
|
||
- **Cooling granularity.** Cooling is often per row or per aisle, not per rack;
|
||
the domain tree allows either, but the defaults should match what sites
|
||
actually measure.
|
||
- **Power capping instead of shedding.** Whether to lower GPU power limits in a
|
||
warm rack rather than move replicas, and how that interacts with measured
|
||
capacity from step 4, which would no longer hold at the lower limit.
|