Files
llmgw/notes/c24-facility.md
T
Ilker ManapandClaude Opus 5.5 0d3eb82b9b Facility-aware placement: racks, PDUs, holds and power caps
Nodes say where they are with llmgw.io/site, room, row, rack and pdu labels,
from agent.yaml or `llmgw ctl label` (kept in the store, over the agent's).
Placement spreads a model's replicas across racks and PDUs, keeps a
multi-node replica within one domain when its profile says keep_within,
places nothing in a domain held with `llmgw ctl facility hold` (optionally
draining it), and refuses a shard that would take a domain past its power
cap from facility.budgets, counting declared, derived and measured draw.
Explain-placement names the domain, and keeps the rejections of a placed
replica; `ctl nodes` gains LOCATION and --domain; `ctl facility` lists the
domains; the console's Nodes view shows and sets location and holds.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 01:17:49 +02:00

41 lines
3.0 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
## CHANGELOG
### Facility-aware placement: racks, PDUs, holds and power caps
- **Where a node is**, from five labels: `llmgw.io/site`, `room`, `row`,
`rack` and `pdu` (a list, for a dual-fed rack), in `agent.yaml` or set from
the control plane with **`llmgw ctl label NODE key=value… [key-…]`**, which
is kept in the store (migration 0031, `node_labels`), wins over the agent's
labels, works before the node joins, and refuses a misspelt `llmgw.io/` key.
A node with no location places as before.
- **Replicas spread across racks and PDUs.** A model's next replica goes to a
node sharing no rack or PDU with its others when there is one: an ordering
above weights already on disk and bin packing, never a filter.
- **`keep_within: rack`** (or row, room, site) on a profile keeps a
multi-node replica's nodes in one such domain.
- **Holds.** **`llmgw ctl facility hold rack:r12-07 --reason … [--drain]`**
places nothing new in a rack, row, room, site or PDU; `--drain` also drains
its nodes. **`llmgw ctl facility release`** undoes both. Kept across
restarts (migration 0031, `facility_holds`).
- **Power caps.** `facility.budgets` in `server.yaml` gives a domain's
`cap_watts` and `other_watts`, with a `margin`; a replica's draw is the
profile's `power_watts`, or its cards at the power limit they report
(`gpu_watts` otherwise) and its cores at `cpu_core_watts`, with `node_watts`
per node. Use is the larger of reserved and measured, node by node; a shard
that would take a domain past its cap less the margin is refused, and a
multi-node replica's shards are counted together. Unknown draw never blocks.
- **Seen everywhere it matters.** `ctl explain-placement` names the held
domain or the kilowatts, and keeps the rejections of a replica that was
placed, so one in an unexpected rack says why. `ctl nodes` has LOCATION,
*ready, held* and `--domain`. **`llmgw ctl facility`** lists every domain
with nodes, use, limit, reserved, measured and holds. The console's Nodes
view has WHERE; a node's menu has **Location…** and **Hold rack…/Release**.
Admin API: `GET /api/v1/facility`, `POST /api/v1/facility/holds`,
`DELETE /api/v1/facility/holds/{domain}`, `POST /api/v1/nodes/{id}/labels`.
- Docs: `docs/facility-aware-placement.md` says what is built and where it
departs from the design; `docs/install.md` and `docs/console.md`.
## BACKLOG
- [x] 24 (facility) — built: location labels (`ctl label`, agent.yaml), spreading across racks/PDUs, `keep_within`, held domains with drain (`ctl facility hold/release`, kept in the store), power caps from `facility.budgets` against declared/derived/measured draw, shown in explain-placement, `ctl nodes`, `ctl facility`, the console and the API. Left: temperature conditions with hysteresis and their enforcement (steps 3–4), a Prometheus/DCIM source, power headroom as a rack-levelling score, spreading multi-node replicas across racks, `llmgw_domain_*` metrics and alerts (near cap, over cap, stale data), shedding and tokens per watt (step 5), and a drained node's state surviving a restart.