Nodes say where they are with llmgw.io/site, room, row, rack and pdu labels, from agent.yaml or `llmgw ctl label` (kept in the store, over the agent's). Placement spreads a model's replicas across racks and PDUs, keeps a multi-node replica within one domain when its profile says keep_within, places nothing in a domain held with `llmgw ctl facility hold` (optionally draining it), and refuses a shard that would take a domain past its power cap from facility.budgets, counting declared, derived and measured draw. Explain-placement names the domain, and keeps the rejections of a placed replica; `ctl nodes` gains LOCATION and --domain; `ctl facility` lists the domains; the console's Nodes view shows and sets location and holds. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
41 lines
3.0 KiB
Markdown
41 lines
3.0 KiB
Markdown
## CHANGELOG
|
||
|
||
### Facility-aware placement: racks, PDUs, holds and power caps
|
||
|
||
- **Where a node is**, from five labels: `llmgw.io/site`, `room`, `row`,
|
||
`rack` and `pdu` (a list, for a dual-fed rack), in `agent.yaml` or set from
|
||
the control plane with **`llmgw ctl label NODE key=value… [key-…]`**, which
|
||
is kept in the store (migration 0031, `node_labels`), wins over the agent's
|
||
labels, works before the node joins, and refuses a misspelt `llmgw.io/` key.
|
||
A node with no location places as before.
|
||
- **Replicas spread across racks and PDUs.** A model's next replica goes to a
|
||
node sharing no rack or PDU with its others when there is one: an ordering
|
||
above weights already on disk and bin packing, never a filter.
|
||
- **`keep_within: rack`** (or row, room, site) on a profile keeps a
|
||
multi-node replica's nodes in one such domain.
|
||
- **Holds.** **`llmgw ctl facility hold rack:r12-07 --reason … [--drain]`**
|
||
places nothing new in a rack, row, room, site or PDU; `--drain` also drains
|
||
its nodes. **`llmgw ctl facility release`** undoes both. Kept across
|
||
restarts (migration 0031, `facility_holds`).
|
||
- **Power caps.** `facility.budgets` in `server.yaml` gives a domain's
|
||
`cap_watts` and `other_watts`, with a `margin`; a replica's draw is the
|
||
profile's `power_watts`, or its cards at the power limit they report
|
||
(`gpu_watts` otherwise) and its cores at `cpu_core_watts`, with `node_watts`
|
||
per node. Use is the larger of reserved and measured, node by node; a shard
|
||
that would take a domain past its cap less the margin is refused, and a
|
||
multi-node replica's shards are counted together. Unknown draw never blocks.
|
||
- **Seen everywhere it matters.** `ctl explain-placement` names the held
|
||
domain or the kilowatts, and keeps the rejections of a replica that was
|
||
placed, so one in an unexpected rack says why. `ctl nodes` has LOCATION,
|
||
*ready, held* and `--domain`. **`llmgw ctl facility`** lists every domain
|
||
with nodes, use, limit, reserved, measured and holds. The console's Nodes
|
||
view has WHERE; a node's menu has **Location…** and **Hold rack…/Release**.
|
||
Admin API: `GET /api/v1/facility`, `POST /api/v1/facility/holds`,
|
||
`DELETE /api/v1/facility/holds/{domain}`, `POST /api/v1/nodes/{id}/labels`.
|
||
- Docs: `docs/facility-aware-placement.md` says what is built and where it
|
||
departs from the design; `docs/install.md` and `docs/console.md`.
|
||
|
||
## BACKLOG
|
||
|
||
- [x] 24 (facility) — built: location labels (`ctl label`, agent.yaml), spreading across racks/PDUs, `keep_within`, held domains with drain (`ctl facility hold/release`, kept in the store), power caps from `facility.budgets` against declared/derived/measured draw, shown in explain-placement, `ctl nodes`, `ctl facility`, the console and the API. Left: temperature conditions with hysteresis and their enforcement (steps 3–4), a Prometheus/DCIM source, power headroom as a rack-levelling score, spreading multi-node replicas across racks, `llmgw_domain_*` metrics and alerts (near cap, over cap, stale data), shedding and tokens per watt (step 5), and a drained node's state surviving a restart.
|