Plan facility-aware placement: rack power, temperature and cooling
CI / test (push) Successful in 3m41s
CI / test (push) Successful in 3m41s
A design document for step 5 of resource allocation, deferred: location labels and rack spreading, power budgets with estimated then measured draw, thermal and cooling conditions with hysteresis, opt-in shedding through preemption, pluggable facility sources, and a rollout that is useful at each step and safe when data is missing. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
e7a8105aa4
commit
8724dc027a
2 files changed
+269
-2
No files matched your search
@@ -0,0 +1,266 @@
|
||||
# Facility-aware placement: rack power, temperature and cooling
|
||||
|
||||
**Status: planned, not built.** This is the design for step 5 of resource
|
||||
allocation, after reserved CPU cores and memory (1), capacity classes and
|
||||
preemption (2), tenant capacity quotas (3) and sizing by measured throughput
|
||||
(4), all of which shipped in v0.10.5. Nothing here exists in the code yet.
|
||||
|
||||
## Why
|
||||
|
||||
In a large installation the scarce thing is often not a GPU but the room it
|
||||
sits in. Three failures are invisible to placement today:
|
||||
|
||||
- **A rack goes over its power budget.** Every node in it reports healthy;
|
||||
placement sees free cards and fills them; the breaker, or the facility's
|
||||
power cap, decides the rest. A rack with A and B feeds is worse: it runs
|
||||
fine until one feed fails and the other cannot carry the load.
|
||||
- **Hot hardware is slow hardware.** GPUs throttle long before they fail. A
|
||||
replica placed in a rack whose inlet air is too warm serves fewer tokens per
|
||||
second than its measured capacity says, and nothing connects the two.
|
||||
- **Correlated failure.** Two replicas of a model on two nodes in the same rack
|
||||
or on the same PDU are one failure away from zero. Placement spreads replicas
|
||||
across nodes, not across racks.
|
||||
|
||||
The goal is for placement to treat facility limits the way it already treats
|
||||
memory and cores: reserve against a budget, refuse what does not fit, say why,
|
||||
and never let missing data stop the cluster.
|
||||
|
||||
## What it builds on
|
||||
|
||||
| Already in place | Used for |
|
||||
|---|---|
|
||||
| Node labels, reported by agents, matched by `node_selector` | Where a node is: its rack, row, PDU |
|
||||
| `inventory.Allocation` carries GPUs, memory and cores per shard | Adding watts, summed per rack and PDU |
|
||||
| Placement rejections (`explain-placement`, headroom's "why not") | Naming the rack that is full or hot |
|
||||
| Headroom: the scheduler dry-run on a copy of the inventory | Room that respects power budgets, for free |
|
||||
| Preemption with classes, minimums and a dry run (step 2) | Shedding load from a hot rack in the right order |
|
||||
| Measured throughput per profile (step 4) | Tokens per watt as a placement preference |
|
||||
| GPU temperature from NVML, per card | A node-level thermal signal that needs no facility data |
|
||||
| `internal/promq`, a Prometheus client | Reading facility metrics that DCIM exporters publish |
|
||||
|
||||
## Concepts
|
||||
|
||||
**Location.** Each node has a place in a tree: site, room, row, rack. Power has
|
||||
its own tree beside it: a rack draws from one or more PDUs, and each PDU from a
|
||||
feed. Both are read from node labels with well-known keys:
|
||||
|
||||
```yaml
|
||||
# agent.yaml
|
||||
labels:
|
||||
llmgw.io/site: sto1
|
||||
llmgw.io/room: hall-b
|
||||
llmgw.io/row: "12"
|
||||
llmgw.io/rack: r12-07
|
||||
llmgw.io/pdu: r12-07-a,r12-07-b # both feeds, for a dual-fed rack
|
||||
```
|
||||
|
||||
A node without these labels is its own rack of one: placement behaves as it
|
||||
does today.
|
||||
|
||||
**Domain.** A rack, a row, a PDU, a room: anything with a budget or a
|
||||
condition. Domains are declared in a facility file (below) or discovered from
|
||||
node labels; a label naming a domain the file does not declare creates it with
|
||||
no budget and no condition.
|
||||
|
||||
**Budget.** A domain may have a power cap in watts. Placement reserves each
|
||||
replica's estimated draw against every domain its node belongs to and refuses a
|
||||
placement that would take any of them past its cap less a safety margin. For a
|
||||
dual-fed rack the cap that matters is what one feed can carry alone, since that
|
||||
is what the rack must survive.
|
||||
|
||||
**Condition.** A domain has a thermal and cooling condition: `ok`, `warm`, `hot`
|
||||
or `unknown`, derived from readings against thresholds with hysteresis.
|
||||
Conditions change what placement prefers and allows; they are not amounts to
|
||||
fill.
|
||||
|
||||
**Reading.** A measured value with a time: inlet temperature, power draw, a
|
||||
cooling unit's alarm state. Readings older than their staleness limit are
|
||||
treated as absent.
|
||||
|
||||
## Estimating a replica's draw
|
||||
|
||||
What placement reserves has to be known before the replica runs, so it is an
|
||||
estimate first and a measurement later:
|
||||
|
||||
1. **Declared.** A profile may say `power_watts`. The operator's figure wins.
|
||||
2. **Derived.** Otherwise GPU profiles take each card's power limit from NVML
|
||||
(the agent already reads the cards; the power limit is one more field) plus
|
||||
a per-node share for CPU, memory and fans; CPU profiles take their reserved
|
||||
cores times a per-core figure for the node's CPU model.
|
||||
3. **Measured.** Once replicas run, the agent reports their cards' actual draw
|
||||
(NVML) and, where the BMC allows, the node's (Redfish or IPMI DCMI). A
|
||||
domain's headroom is computed from the larger of what is reserved and what
|
||||
is measured, so an estimate that was too low is corrected by the room it
|
||||
stops offering, not by an outage.
|
||||
|
||||
## Decisions
|
||||
|
||||
### Power
|
||||
|
||||
- **Hard limit.** A placement that would take any domain past `cap × (1 −
|
||||
margin)` is rejected, with the domain and the figures: *rack r12-07: 14.2 of
|
||||
16.0 kW reserved, replica needs ~2.8, margin 10%*.
|
||||
- **Feeds.** For a domain with two feeds the cap is the smaller feed's rating
|
||||
times its derating (typically 80%), not the sum.
|
||||
- **Already over.** A domain found over its cap, because the estimate was low
|
||||
or the cap was lowered, gets no new placements, and an alert. It is not
|
||||
emptied automatically; shedding power is the facility's decision, and
|
||||
stopping inference to do it is ours only if asked (see shedding below).
|
||||
|
||||
### Temperature and cooling
|
||||
|
||||
Thresholds are per domain with defaults, and each condition needs to hold for a
|
||||
while before it changes, in both directions, so a single reading neither
|
||||
starts nor ends anything:
|
||||
|
||||
| Condition | Enters when | Placement |
|
||||
|---|---|---|
|
||||
| `ok` | below the warm threshold for 5 min | normal |
|
||||
| `warm` | inlet above the warm threshold for 2 min, or GPU temperatures near throttling | allowed, scored down: other domains first |
|
||||
| `hot` | inlet above the hot threshold for 2 min, or a cooling unit in alarm | no new placements, as a cordon would, but automatic and reversible |
|
||||
| `unknown` | no fresh reading | as `ok`, and reported as unknown |
|
||||
|
||||
**Shedding** (optional, off by default): a domain `hot` for longer than its
|
||||
shed delay has load moved off it using step 2's machinery. The lowest class goes
|
||||
first, and replicas above their model's minimum go before any at it. A critical
|
||||
minimum is never touched, and moves are rate limited to one replica per domain
|
||||
per interval. The replica is placed elsewhere first where there is room, so
|
||||
shedding moves work rather than dropping it. GPU power capping (`nvidia-smi
|
||||
-pl`) is a gentler alternative worth measuring against shedding: less
|
||||
throughput, no restart.
|
||||
|
||||
### Spreading
|
||||
|
||||
Replicas of one model are spread across racks, and across PDUs, the way they
|
||||
are spread across nodes today. This is a preference, dropped when it is the only
|
||||
thing standing in the way, as node spreading already is.
|
||||
|
||||
### Scoring
|
||||
|
||||
Placement today filters, then takes the first candidate in a fixed order
|
||||
(fewest free GPUs for GPU profiles, fewest replicas for CPU ones, nodes holding
|
||||
the weights first). Facility signals are better as a score than as more
|
||||
filters. Candidates that pass the hard limits are ordered by:
|
||||
|
||||
1. condition: `ok` before `warm`;
|
||||
2. power headroom left in the rack after this replica, more first, so load
|
||||
levels across racks;
|
||||
3. the existing order.
|
||||
|
||||
Once measurements include power, step 4's profile preference can use tokens per
|
||||
watt instead of tokens per GPU where both are known.
|
||||
|
||||
## Where the data comes from
|
||||
|
||||
A `FacilitySource` interface, implemented several ways, chosen in
|
||||
`server.yaml`:
|
||||
|
||||
```go
|
||||
type FacilitySource interface {
|
||||
// Domains is what the source knows: budgets, thresholds, members.
|
||||
Domains(ctx) ([]Domain, error)
|
||||
// Readings is the latest value of each signal, per domain.
|
||||
Readings(ctx) ([]Reading, error)
|
||||
}
|
||||
```
|
||||
|
||||
- **Static file.** Domains, caps and thresholds, written by hand or generated
|
||||
from a DCIM export. Enough for power budgets with no live data at all.
|
||||
- **Prometheus.** DCIM, PDU (SNMP exporter) and BMC (Redfish exporter)
|
||||
readings usually end up in Prometheus. The source runs configured instant
|
||||
queries and maps their labels onto domains:
|
||||
|
||||
```yaml
|
||||
facility:
|
||||
source: prometheus
|
||||
prometheus: http://monitoring:9090
|
||||
signals:
|
||||
inlet_temp_c:
|
||||
query: max by (rack) (dcim_rack_inlet_temperature_celsius)
|
||||
domain_label: rack
|
||||
stale_after: 2m
|
||||
power_watts:
|
||||
query: sum by (pdu) (pdu_outlet_power_watts)
|
||||
domain_label: pdu
|
||||
cooling_alarm:
|
||||
query: max by (row) (crac_alarm)
|
||||
domain_label: row
|
||||
```
|
||||
- **Agents.** What the nodes can read themselves: GPU power and temperature from
|
||||
NVML, node power from Redfish or IPMI where the BMC allows it. Needs no
|
||||
facility integration, and gives node-level signals on day one.
|
||||
- **Simulated.** Readings from a script, for tests and for rehearsing a
|
||||
thermal event on a cluster without facility data.
|
||||
|
||||
## Safety
|
||||
|
||||
- **Missing data never blocks.** A source that is down, a reading that is stale,
|
||||
a node without location labels: placement behaves as it does today, and says
|
||||
the data is missing (a metric, a line on the cluster page). Facility
|
||||
awareness can make placement more careful, never impossible.
|
||||
- **Hysteresis and hold times** on every condition, so a noisy sensor does not
|
||||
flap a rack between allowed and forbidden.
|
||||
- **Observe-only first.** A mode where every decision is computed, logged and
|
||||
shown ("would have refused: rack r12-07 hot") but nothing is enforced, run
|
||||
for long enough to trust the thresholds before turning enforcement on.
|
||||
- **Manual override.** `llmgw ctl facility hold RACK` and `release RACK`, for the
|
||||
facility team to mark a domain unavailable, or to clear a condition they know
|
||||
to be a sensor fault.
|
||||
- **Shedding is opt-in**, per domain or cluster-wide, and rate limited.
|
||||
|
||||
## What people see
|
||||
|
||||
- `explain-placement` and headroom name the domain: *rack r12-07 is hot (inlet
|
||||
31°C for 6 min)*, *pdu r12-07-a: 7.1 of 7.4 kW*.
|
||||
- `llmgw ctl facility` lists domains with budget, reserved, measured and
|
||||
condition. The cluster page gets a rack summary beside the nodes table.
|
||||
- Metrics: `llmgw_domain_power_budget_watts`, `_reserved_watts`,
|
||||
`_measured_watts`, `llmgw_domain_condition{condition}`,
|
||||
`llmgw_domain_reading_age_seconds`; alerts for a domain near its cap, a
|
||||
domain hot, a domain over its cap, and facility data gone stale.
|
||||
|
||||
## Rollout
|
||||
|
||||
Each step ships on its own and is useful without the next:
|
||||
|
||||
1. **Location and spreading.** Well-known labels, domains from labels, replicas
|
||||
spread across racks and PDUs, locations on the node views. No facility data
|
||||
needed.
|
||||
2. **Power budgets from a static file.** Caps and margins, estimated draw per
|
||||
replica (declared or derived), hard limit in placement and headroom.
|
||||
3. **Readings, observe-only.** The Prometheus and agent sources, conditions
|
||||
with hysteresis, measured power folded into headroom; decisions shown, not
|
||||
enforced.
|
||||
4. **Thermal enforcement.** `warm` scored down, `hot` refused, manual hold and
|
||||
release.
|
||||
5. **Shedding and tokens per watt.** Opt-in shedding through step 2's
|
||||
preemption; measured power in profile preference.
|
||||
|
||||
## Testing
|
||||
|
||||
- The simulated source drives unit tests of every condition transition, the
|
||||
hysteresis, staleness, and the placement outcomes, with mutation checks as
|
||||
for steps 1–4.
|
||||
- Property tests: whatever the readings, placement never exceeds a declared
|
||||
cap with fresh data, and never refuses everything because data is missing.
|
||||
- On the PoC cluster, with no facility data: step 1 with labelled VMs, step 2
|
||||
with a static file of made-up caps, steps 3–4 with the simulated source
|
||||
replaying a thermal event.
|
||||
- Before enforcement anywhere real: a period in observe-only mode against the
|
||||
site's actual data.
|
||||
|
||||
## Open questions
|
||||
|
||||
- **What the sites have.** Which DCIM, which PDUs, which BMCs, and whether any
|
||||
of it already reaches a Prometheus. The source design assumes Prometheus is
|
||||
the common ground; a site without it needs another source.
|
||||
- **Who owns the numbers.** Caps, deratings and thresholds belong to the
|
||||
facility team. Whether they edit llmgw's file, or llmgw reads theirs.
|
||||
- **Phase balance.** Three-phase PDUs have per-phase limits; whether placement
|
||||
should see phases or only the PDU total.
|
||||
- **Cooling granularity.** Cooling is often per row or per aisle, not per rack;
|
||||
the domain tree allows either, but the defaults should match what sites
|
||||
actually measure.
|
||||
- **Power capping instead of shedding.** Whether to lower GPU power limits in a
|
||||
warm rack rather than move replicas, and how that interacts with measured
|
||||
capacity from step 4, which would no longer hold at the lower limit.
|
||||
+3
-2
@@ -1485,8 +1485,9 @@ The launch path is now built too:
|
||||
- [x] Capacity quotas per organization, team and project: guarantees kept,
|
||||
borrowing taken back, limits enforced
|
||||
- [x] CPU cores reserved like memory, and memory required of CPU models
|
||||
- [ ] Rack power, temperature and cooling as placement signals (a facility
|
||||
source; not yet built)
|
||||
- [ ] Rack power, temperature and cooling as placement signals: planned in
|
||||
[docs/facility-aware-placement.md](docs/facility-aware-placement.md),
|
||||
in five steps from rack spreading to opt-in shedding
|
||||
- [ ] Defragmentation for large tensor-parallel replicas across NVLink domains
|
||||
- [ ] Control plane HA (Raft)
|
||||
- [x] Weight downloading and cache management: the mirror, agent fetches
|
||||
|
||||
Reference in new issue
Block a user