Facility-aware placement: racks, PDUs, holds and power caps

Nodes say where they are with llmgw.io/site, room, row, rack and pdu labels,
from agent.yaml or `llmgw ctl label` (kept in the store, over the agent's).
Placement spreads a model's replicas across racks and PDUs, keeps a
multi-node replica within one domain when its profile says keep_within,
places nothing in a domain held with `llmgw ctl facility hold` (optionally
draining it), and refuses a shard that would take a domain past its power
cap from facility.budgets, counting declared, derived and measured draw.
Explain-placement names the domain, and keeps the rejections of a placed
replica; `ctl nodes` gains LOCATION and --domain; `ctl facility` lists the
domains; the console's Nodes view shows and sets location and holds.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-05 01:17:49 +02:00
1 parent d41104a452
commit 0d3eb82b9b
42 files changed
+3510 -63

No files matched your search

+109 -12
View File
@@ -1,13 +1,109 @@
# Facility-aware placement: rack power, temperature and cooling
**Status: step 0, reading the nodes' own sensors, is built; steps 1–5 are
deferred** until the rack, PDU and power information of a real site is at
hand — without it they would be designed against guesses.
This is the design for step 5 of resource allocation, after reserved CPU cores
and memory (1), capacity classes and preemption (2), tenant capacity quotas (3)
and sizing by measured throughput (4), all of which shipped in v0.10.5. Of
what follows, only the node sensors (rollout step 0) exist in the code: they
are read, published and shown, and change no decision.
**Status: steps 0, 1 and 2 are built, with the manual hold of step 4;
temperature conditions (steps 3 and 4) and shedding (step 5) are not**,
until the thermal data of a real site is at hand — without it thresholds would
be designed against guesses. This is the design for step 5 of resource
allocation, after reserved CPU cores and memory (1), capacity classes and
preemption (2), tenant capacity quotas (3) and sizing by measured throughput
(4). [What is built](#what-is-built) says what works today and where it
departs from the design below; the rest of the document is the design.
## What is built
**Where a node is.** Five labels: `llmgw.io/site`, `room`, `row`, `rack` and
`pdu` (a comma separated list, for a dual-fed rack). They come from the node's
`agent.yaml` under `labels:`, or from the control plane:
```
llmgw ctl label gpu01 llmgw.io/rack=r12-07 llmgw.io/pdu=r12-07-a,r12-07-b
llmgw ctl label gpu01 llmgw.io/rack- # back to agent.yaml's value, if any
```
Labels set with `ctl label` are kept in the store, win over the agent's, and
may be set before the node joins. A misspelt `llmgw.io/` key is refused. A
domain is named `level:name` — `rack:r12-07`, `pdu:r12-07-a` — and names are
taken as given, so two rooms that both have a rack `r1` must label them apart.
A node without location labels is in no domain and places as it always did.
**Spreading.** A model's next replica goes first to a node sharing no rack and
no PDU with the nodes the model already runs on; nodes sharing fewer come
before nodes sharing more. It is an ordering, never a filter: a rack holding
the only free cards still takes the replica. It ranks above preferring a node
that already holds the weights — a second replica one breaker away from the
first is worth less than the copy saved — and above bin packing.
**Keeping a replica together.** A profile's `keep_within: rack` (or `row`,
`room`, `site`) keeps a multi-node replica's nodes in one such domain, inside
the fabric or NVLink domain placement already requires; nodes without the
label are passed over for it, with the reason.
**Holds.** `llmgw ctl facility hold rack:r12-07 --reason "PDU swap"` places
nothing new in the domain, as a cordon of each of its nodes would, including
nodes labelled into it later. `--drain` also drains its nodes, one replica at
a time with the replacement up first; `llmgw ctl facility release` undoes
both. Holds are kept in the store across restarts (the drain itself is node
state, which is not: after a restart the hold still keeps work out, and
replicas not yet moved stay where they are). Any domain can be held: a PDU
holds every rack on it.
**Power budgets.** In `server.yaml`:
```yaml
facility:
margin: 0.1 # share of every cap kept free
gpu_watts: 700 # a card that reports no power limit
cpu_core_watts: 10 # a core reserved by a CPU replica (cpu_cores)
node_watts: 350 # each node's own draw beyond its cards
budgets:
- domain: rack:r12-07
cap_watts: 16000 # what one feed carries alone, derated
other_watts: 800 # switches, storage
- domain: pdu:r12-07-a
cap_watts: 7400
```
A replica's draw is the profile's `power_watts` (split over its nodes) when
declared; otherwise each card at the power limit it reports through the agent's
sensors (or `gpu_watts`), and each reserved core at `cpu_core_watts`. A
domain's use is, node by node, `node_watts` plus the larger of what is placed
there and what the node's sensors read in the last two minutes
(`measured_fresh`), plus the domain's `other_watts`. A shard that would take
any of its node's domains past `cap × (1 − margin)` is refused; a multi-node
replica's shards are counted together, so two that each fit a rack alone do not
both go into it. A draw nobody knows — no declared figure, no card limit, no
`gpu_watts` — counts as nothing: missing data never blocks.
**What people see.** `llmgw ctl explain-placement` names the domain: *rack
r12-07 is held: PDU swap*, *rack r12-07: 14.2 kW of a 14.4 kW limit (16.0 kW
cap less 10%) in use, the shard needs ~2.8 kW*. The rejections of a placed
replica are kept too, so a replica in an unexpected rack says why. `llmgw ctl
nodes` has a LOCATION column, shows *ready, held*, and filters with `--domain
rack:r12-07`. `llmgw ctl facility` lists every domain with its nodes, use,
limit, reserved and measured draw, and holds. The console's Nodes view has a
WHERE column; a node's page shows its location and hold, and its menu sets
the location (**Location…**) and holds or releases its rack. The admin API:
`GET /api/v1/facility`, `POST /api/v1/facility/holds`, `DELETE
/api/v1/facility/holds/{domain}`, `POST /api/v1/nodes/{id}/labels`.
**Where it departs from the design below, and why.**
- Domains come from labels alone; there is no facility file declaring them.
Budgets are in `server.yaml`, the one file the control plane already reads.
A DCIM export would generate that block.
- Feeds are not modelled. The cap given is what one feed carries alone,
derated — the figure the design says matters — so the operator gives the
one number instead of llmgw deriving it from two.
- The manual hold of step 4 is built now, without conditions: it needs no
sensor data and answers the commonest facility event, planned maintenance.
Its command is `ctl facility hold`, with a `--drain` the design did not have.
- `keep_within` is new: the request that a multi-node replica stay in one rack
where the cabling requires it.
- Spreading is an ordering of candidates, not the scoring of the design; power
headroom does not yet order racks ("level load across racks"), and
multi-node replicas are not spread across racks.
- No `llmgw_domain_*` metrics or alerts yet; `ctl facility` and the API carry
the same figures.
## Why
@@ -260,16 +356,17 @@ Each step ships on its own and is useful without the next:
0. **Node sensors, observe-only (built).** GPU heat, power and throttling,
CPU heat and power, inlet air where the board has a sensor; published and
shown, no decision changed.
1. **Location and spreading.** Well-known labels, domains from labels, replicas
1. **Location and spreading (built).** Well-known labels, domains from labels, replicas
spread across racks and PDUs, locations on the node views. No facility data
needed.
2. **Power budgets from a static file.** Caps and margins, estimated draw per
2. **Power budgets from a static file (built, without headroom
scoring).** Caps and margins, estimated draw per
replica (declared or derived), hard limit in placement and headroom.
3. **Readings, observe-only.** The Prometheus source beside the agents',
conditions with hysteresis (throttling first among them), measured power
folded into headroom; decisions shown, not enforced.
4. **Thermal enforcement.** `warm` scored down, `hot` refused, manual hold and
release.
4. **Thermal enforcement.** `warm` scored down, `hot` refused; the manual hold
and release are built.
5. **Shedding and tokens per watt.** Opt-in shedding through step 2's
preemption; measured power in profile preference.