Files
llmgw/notes/c24-facility.md
T
Ilker ManapandClaude Opus 5.5 0d3eb82b9b Facility-aware placement: racks, PDUs, holds and power caps
Nodes say where they are with llmgw.io/site, room, row, rack and pdu labels,
from agent.yaml or `llmgw ctl label` (kept in the store, over the agent's).
Placement spreads a model's replicas across racks and PDUs, keeps a
multi-node replica within one domain when its profile says keep_within,
places nothing in a domain held with `llmgw ctl facility hold` (optionally
draining it), and refuses a shard that would take a domain past its power
cap from facility.budgets, counting declared, derived and measured draw.
Explain-placement names the domain, and keeps the rejections of a placed
replica; `ctl nodes` gains LOCATION and --domain; `ctl facility` lists the
domains; the console's Nodes view shows and sets location and holds.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 01:17:49 +02:00

3.0 KiB
Raw Blame History

CHANGELOG

Facility-aware placement: racks, PDUs, holds and power caps

  • Where a node is, from five labels: llmgw.io/site, room, row, rack and pdu (a list, for a dual-fed rack), in agent.yaml or set from the control plane with llmgw ctl label NODE key=value… [key-…], which is kept in the store (migration 0031, node_labels), wins over the agent's labels, works before the node joins, and refuses a misspelt llmgw.io/ key. A node with no location places as before.
  • Replicas spread across racks and PDUs. A model's next replica goes to a node sharing no rack or PDU with its others when there is one: an ordering above weights already on disk and bin packing, never a filter.
  • keep_within: rack (or row, room, site) on a profile keeps a multi-node replica's nodes in one such domain.
  • Holds. llmgw ctl facility hold rack:r12-07 --reason … [--drain] places nothing new in a rack, row, room, site or PDU; --drain also drains its nodes. llmgw ctl facility release undoes both. Kept across restarts (migration 0031, facility_holds).
  • Power caps. facility.budgets in server.yaml gives a domain's cap_watts and other_watts, with a margin; a replica's draw is the profile's power_watts, or its cards at the power limit they report (gpu_watts otherwise) and its cores at cpu_core_watts, with node_watts per node. Use is the larger of reserved and measured, node by node; a shard that would take a domain past its cap less the margin is refused, and a multi-node replica's shards are counted together. Unknown draw never blocks.
  • Seen everywhere it matters. ctl explain-placement names the held domain or the kilowatts, and keeps the rejections of a replica that was placed, so one in an unexpected rack says why. ctl nodes has LOCATION, ready, held and --domain. llmgw ctl facility lists every domain with nodes, use, limit, reserved, measured and holds. The console's Nodes view has WHERE; a node's menu has Location… and Hold rack…/Release. Admin API: GET /api/v1/facility, POST /api/v1/facility/holds, DELETE /api/v1/facility/holds/{domain}, POST /api/v1/nodes/{id}/labels.
  • Docs: docs/facility-aware-placement.md says what is built and where it departs from the design; docs/install.md and docs/console.md.

BACKLOG

  • 24 (facility) — built: location labels (ctl label, agent.yaml), spreading across racks/PDUs, keep_within, held domains with drain (ctl facility hold/release, kept in the store), power caps from facility.budgets against declared/derived/measured draw, shown in explain-placement, ctl nodes, ctl facility, the console and the API. Left: temperature conditions with hysteresis and their enforcement (steps 3–4), a Prometheus/DCIM source, power headroom as a rack-levelling score, spreading multi-node replicas across racks, llmgw_domain_* metrics and alerts (near cap, over cap, stale data), shedding and tokens per watt (step 5), and a drained node's state surviving a restart.