Facility-aware placement: racks, PDUs, holds and power caps
Nodes say where they are with llmgw.io/site, room, row, rack and pdu labels, from agent.yaml or `llmgw ctl label` (kept in the store, over the agent's). Placement spreads a model's replicas across racks and PDUs, keeps a multi-node replica within one domain when its profile says keep_within, places nothing in a domain held with `llmgw ctl facility hold` (optionally draining it), and refuses a shard that would take a domain past its power cap from facility.budgets, counting declared, derived and measured draw. Explain-placement names the domain, and keeps the rejections of a placed replica; `ctl nodes` gains LOCATION and --domain; `ctl facility` lists the domains; the console's Nodes view shows and sets location and holds. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
d41104a452
commit
0d3eb82b9b
42 files changed
+3510
-63
No files matched your search
@@ -1,13 +1,109 @@
|
||||
# Facility-aware placement: rack power, temperature and cooling
|
||||
|
||||
**Status: step 0, reading the nodes' own sensors, is built; steps 1–5 are
|
||||
deferred** until the rack, PDU and power information of a real site is at
|
||||
hand — without it they would be designed against guesses.
|
||||
This is the design for step 5 of resource allocation, after reserved CPU cores
|
||||
and memory (1), capacity classes and preemption (2), tenant capacity quotas (3)
|
||||
and sizing by measured throughput (4), all of which shipped in v0.10.5. Of
|
||||
what follows, only the node sensors (rollout step 0) exist in the code: they
|
||||
are read, published and shown, and change no decision.
|
||||
**Status: steps 0, 1 and 2 are built, with the manual hold of step 4;
|
||||
temperature conditions (steps 3 and 4) and shedding (step 5) are not**,
|
||||
until the thermal data of a real site is at hand — without it thresholds would
|
||||
be designed against guesses. This is the design for step 5 of resource
|
||||
allocation, after reserved CPU cores and memory (1), capacity classes and
|
||||
preemption (2), tenant capacity quotas (3) and sizing by measured throughput
|
||||
(4). [What is built](#what-is-built) says what works today and where it
|
||||
departs from the design below; the rest of the document is the design.
|
||||
|
||||
## What is built
|
||||
|
||||
**Where a node is.** Five labels: `llmgw.io/site`, `room`, `row`, `rack` and
|
||||
`pdu` (a comma separated list, for a dual-fed rack). They come from the node's
|
||||
`agent.yaml` under `labels:`, or from the control plane:
|
||||
|
||||
```
|
||||
llmgw ctl label gpu01 llmgw.io/rack=r12-07 llmgw.io/pdu=r12-07-a,r12-07-b
|
||||
llmgw ctl label gpu01 llmgw.io/rack- # back to agent.yaml's value, if any
|
||||
```
|
||||
|
||||
Labels set with `ctl label` are kept in the store, win over the agent's, and
|
||||
may be set before the node joins. A misspelt `llmgw.io/` key is refused. A
|
||||
domain is named `level:name` — `rack:r12-07`, `pdu:r12-07-a` — and names are
|
||||
taken as given, so two rooms that both have a rack `r1` must label them apart.
|
||||
A node without location labels is in no domain and places as it always did.
|
||||
|
||||
**Spreading.** A model's next replica goes first to a node sharing no rack and
|
||||
no PDU with the nodes the model already runs on; nodes sharing fewer come
|
||||
before nodes sharing more. It is an ordering, never a filter: a rack holding
|
||||
the only free cards still takes the replica. It ranks above preferring a node
|
||||
that already holds the weights — a second replica one breaker away from the
|
||||
first is worth less than the copy saved — and above bin packing.
|
||||
|
||||
**Keeping a replica together.** A profile's `keep_within: rack` (or `row`,
|
||||
`room`, `site`) keeps a multi-node replica's nodes in one such domain, inside
|
||||
the fabric or NVLink domain placement already requires; nodes without the
|
||||
label are passed over for it, with the reason.
|
||||
|
||||
**Holds.** `llmgw ctl facility hold rack:r12-07 --reason "PDU swap"` places
|
||||
nothing new in the domain, as a cordon of each of its nodes would, including
|
||||
nodes labelled into it later. `--drain` also drains its nodes, one replica at
|
||||
a time with the replacement up first; `llmgw ctl facility release` undoes
|
||||
both. Holds are kept in the store across restarts (the drain itself is node
|
||||
state, which is not: after a restart the hold still keeps work out, and
|
||||
replicas not yet moved stay where they are). Any domain can be held: a PDU
|
||||
holds every rack on it.
|
||||
|
||||
**Power budgets.** In `server.yaml`:
|
||||
|
||||
```yaml
|
||||
facility:
|
||||
margin: 0.1 # share of every cap kept free
|
||||
gpu_watts: 700 # a card that reports no power limit
|
||||
cpu_core_watts: 10 # a core reserved by a CPU replica (cpu_cores)
|
||||
node_watts: 350 # each node's own draw beyond its cards
|
||||
budgets:
|
||||
- domain: rack:r12-07
|
||||
cap_watts: 16000 # what one feed carries alone, derated
|
||||
other_watts: 800 # switches, storage
|
||||
- domain: pdu:r12-07-a
|
||||
cap_watts: 7400
|
||||
```
|
||||
|
||||
A replica's draw is the profile's `power_watts` (split over its nodes) when
|
||||
declared; otherwise each card at the power limit it reports through the agent's
|
||||
sensors (or `gpu_watts`), and each reserved core at `cpu_core_watts`. A
|
||||
domain's use is, node by node, `node_watts` plus the larger of what is placed
|
||||
there and what the node's sensors read in the last two minutes
|
||||
(`measured_fresh`), plus the domain's `other_watts`. A shard that would take
|
||||
any of its node's domains past `cap × (1 − margin)` is refused; a multi-node
|
||||
replica's shards are counted together, so two that each fit a rack alone do not
|
||||
both go into it. A draw nobody knows — no declared figure, no card limit, no
|
||||
`gpu_watts` — counts as nothing: missing data never blocks.
|
||||
|
||||
**What people see.** `llmgw ctl explain-placement` names the domain: *rack
|
||||
r12-07 is held: PDU swap*, *rack r12-07: 14.2 kW of a 14.4 kW limit (16.0 kW
|
||||
cap less 10%) in use, the shard needs ~2.8 kW*. The rejections of a placed
|
||||
replica are kept too, so a replica in an unexpected rack says why. `llmgw ctl
|
||||
nodes` has a LOCATION column, shows *ready, held*, and filters with `--domain
|
||||
rack:r12-07`. `llmgw ctl facility` lists every domain with its nodes, use,
|
||||
limit, reserved and measured draw, and holds. The console's Nodes view has a
|
||||
WHERE column; a node's page shows its location and hold, and its menu sets
|
||||
the location (**Location…**) and holds or releases its rack. The admin API:
|
||||
`GET /api/v1/facility`, `POST /api/v1/facility/holds`, `DELETE
|
||||
/api/v1/facility/holds/{domain}`, `POST /api/v1/nodes/{id}/labels`.
|
||||
|
||||
**Where it departs from the design below, and why.**
|
||||
|
||||
- Domains come from labels alone; there is no facility file declaring them.
|
||||
Budgets are in `server.yaml`, the one file the control plane already reads.
|
||||
A DCIM export would generate that block.
|
||||
- Feeds are not modelled. The cap given is what one feed carries alone,
|
||||
derated — the figure the design says matters — so the operator gives the
|
||||
one number instead of llmgw deriving it from two.
|
||||
- The manual hold of step 4 is built now, without conditions: it needs no
|
||||
sensor data and answers the commonest facility event, planned maintenance.
|
||||
Its command is `ctl facility hold`, with a `--drain` the design did not have.
|
||||
- `keep_within` is new: the request that a multi-node replica stay in one rack
|
||||
where the cabling requires it.
|
||||
- Spreading is an ordering of candidates, not the scoring of the design; power
|
||||
headroom does not yet order racks ("level load across racks"), and
|
||||
multi-node replicas are not spread across racks.
|
||||
- No `llmgw_domain_*` metrics or alerts yet; `ctl facility` and the API carry
|
||||
the same figures.
|
||||
|
||||
## Why
|
||||
|
||||
@@ -260,16 +356,17 @@ Each step ships on its own and is useful without the next:
|
||||
0. **Node sensors, observe-only (built).** GPU heat, power and throttling,
|
||||
CPU heat and power, inlet air where the board has a sensor; published and
|
||||
shown, no decision changed.
|
||||
1. **Location and spreading.** Well-known labels, domains from labels, replicas
|
||||
1. **Location and spreading (built).** Well-known labels, domains from labels, replicas
|
||||
spread across racks and PDUs, locations on the node views. No facility data
|
||||
needed.
|
||||
2. **Power budgets from a static file.** Caps and margins, estimated draw per
|
||||
2. **Power budgets from a static file (built, without headroom
|
||||
scoring).** Caps and margins, estimated draw per
|
||||
replica (declared or derived), hard limit in placement and headroom.
|
||||
3. **Readings, observe-only.** The Prometheus source beside the agents',
|
||||
conditions with hysteresis (throttling first among them), measured power
|
||||
folded into headroom; decisions shown, not enforced.
|
||||
4. **Thermal enforcement.** `warm` scored down, `hot` refused, manual hold and
|
||||
release.
|
||||
4. **Thermal enforcement.** `warm` scored down, `hot` refused; the manual hold
|
||||
and release are built.
|
||||
5. **Shedding and tokens per watt.** Opt-in shedding through step 2's
|
||||
preemption; measured power in profile preference.
|
||||
|
||||
|
||||
Reference in new issue
Block a user