Facility-aware placement: racks, PDUs, holds and power caps
Nodes say where they are with llmgw.io/site, room, row, rack and pdu labels, from agent.yaml or `llmgw ctl label` (kept in the store, over the agent's). Placement spreads a model's replicas across racks and PDUs, keeps a multi-node replica within one domain when its profile says keep_within, places nothing in a domain held with `llmgw ctl facility hold` (optionally draining it), and refuses a shard that would take a domain past its power cap from facility.budgets, counting declared, derived and measured draw. Explain-placement names the domain, and keeps the rejections of a placed replica; `ctl nodes` gains LOCATION and --domain; `ctl facility` lists the domains; the console's Nodes view shows and sets location and holds. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
d41104a452
commit
0d3eb82b9b
42 files changed
+3510
-63
No files matched your search
@@ -86,6 +86,15 @@ rather than repeated.
|
||||
| `url` | `string` | [External](#external) |
|
||||
| `model` | `string` | [External](#external) |
|
||||
| `api_key_file` | `string` | [External](#external) |
|
||||
| `budgets` | `[]FacilityBudget` | [Facility](#facility) |
|
||||
| `margin` | `float64` | [Facility](#facility) |
|
||||
| `gpu_watts` | `float64` | [Facility](#facility) |
|
||||
| `cpu_core_watts` | `float64` | [Facility](#facility) |
|
||||
| `node_watts` | `float64` | [Facility](#facility) |
|
||||
| `measured_fresh` | `time.Duration` | [Facility](#facility) |
|
||||
| `domain` | `string` | [FacilityBudget](#facilitybudget) |
|
||||
| `cap_watts` | `float64` | [FacilityBudget](#facilitybudget) |
|
||||
| `other_watts` | `float64` | [FacilityBudget](#facilitybudget) |
|
||||
| `format` | `string` | [GuardModel](#guardmodel) |
|
||||
| `categories` | `[]string` | [GuardModel](#guardmodel) |
|
||||
| `model` | `string` | [JudgePolicy](#judgepolicy) |
|
||||
@@ -194,6 +203,8 @@ rather than repeated.
|
||||
| `cpu_cores` | `int` | [Profile](#profile) |
|
||||
| `min_disk_gb` | `int` | [Profile](#profile) |
|
||||
| `node_selector` | `map[string]string` | [Profile](#profile) |
|
||||
| `power_watts` | `float64` | [Profile](#profile) |
|
||||
| `keep_within` | `string` | [Profile](#profile) |
|
||||
| `weight` | `float64` | [Profile](#profile) |
|
||||
| `priority` | `int` | [Profile](#profile) |
|
||||
| `startup_timeout` | `time.Duration` | [Profile](#profile) |
|
||||
@@ -271,6 +282,7 @@ rather than repeated.
|
||||
| `router` | `Router` | [Server](#server) |
|
||||
| `control` | `Control` | [Server](#server) |
|
||||
| `models` | `[]catalog.Model` | [Server](#server) |
|
||||
| `facility` | `Facility` | [Server](#server) |
|
||||
| `endpoints` | `[]Endpoint` | [Server](#server) |
|
||||
| `bin` | `string` | [SimConfig](#simconfig) |
|
||||
| `python` | `string` | [TRTLLMConfig](#trtllmconfig) |
|
||||
@@ -839,6 +851,75 @@ APIKeyFile holds the provider's key, on the control plane. A file
|
||||
rather than the key itself, so the key is in neither the catalog, the
|
||||
store nor the audit log; empty for a provider that asks for none.
|
||||
|
||||
## Facility
|
||||
|
||||
### `budgets`
|
||||
|
||||
`[]FacilityBudget`
|
||||
|
||||
Budgets are the domains with a power cap.
|
||||
|
||||
### `margin`
|
||||
|
||||
`float64`
|
||||
|
||||
Margin is the share of every cap kept free, 0.1 for ten percent. Zero
|
||||
keeps none, for caps already given derated.
|
||||
|
||||
### `gpu_watts`
|
||||
|
||||
`float64`
|
||||
|
||||
GPUWatts is a card's draw when it does not report its power limit:
|
||||
an agent without nvidia-smi's power fields, or a node that has not
|
||||
sent a heartbeat yet. Zero counts such a card as drawing nothing.
|
||||
|
||||
### `cpu_core_watts`
|
||||
|
||||
`float64`
|
||||
|
||||
CPUCoreWatts is the draw of a core reserved by a replica with no GPU
|
||||
(cpu_cores). Zero counts such replicas as drawing nothing.
|
||||
|
||||
### `node_watts`
|
||||
|
||||
`float64`
|
||||
|
||||
NodeWatts is what a node draws besides its cards and reserved cores —
|
||||
idle CPUs, memory, fans, disks — counted for every node in a capped
|
||||
domain, busy or not.
|
||||
|
||||
### `measured_fresh`
|
||||
|
||||
`time.Duration`
|
||||
|
||||
MeasuredFresh is how old a node's power reading may be and still be
|
||||
counted against its domains. Zero takes two minutes.
|
||||
|
||||
## FacilityBudget
|
||||
|
||||
### `domain`
|
||||
|
||||
`string`
|
||||
|
||||
Domain is level:name, as the nodes' labels give it: rack:r12-07,
|
||||
pdu:r12-07-a, row:12.
|
||||
|
||||
### `cap_watts`
|
||||
|
||||
`float64`
|
||||
|
||||
CapWatts is what the domain may draw. For a dual-fed rack it is what
|
||||
one feed carries alone, derated as the facility derates it (usually
|
||||
80%), since that is what the rack has to survive.
|
||||
|
||||
### `other_watts`
|
||||
|
||||
`float64`
|
||||
|
||||
OtherWatts is drawn in the domain by what llmgw does not place on:
|
||||
switches, storage, machines outside the cluster.
|
||||
|
||||
## GuardModel
|
||||
|
||||
### `format`
|
||||
@@ -1638,6 +1719,27 @@ in the inventory says how much memory a node has, so the operator says
|
||||
which nodes are meant for a model instead of the scheduler inferring it.
|
||||
Agents have reported labels since the beginning; this is what reads them.
|
||||
|
||||
### `power_watts`
|
||||
|
||||
`float64`
|
||||
|
||||
PowerWatts is what one replica of this profile draws, in watts, split
|
||||
evenly over its nodes, counted against the power cap of every rack,
|
||||
PDU or other domain its nodes are in (facility.budgets in server.yaml).
|
||||
Zero derives it: each card at the power limit it reports, each
|
||||
reserved core at facility.cpu_core_watts. Only matters where a domain
|
||||
has a cap.
|
||||
|
||||
### `keep_within`
|
||||
|
||||
`string`
|
||||
|
||||
KeepWithin keeps a multi-node replica's nodes inside one rack, row,
|
||||
room or site, read from their llmgw.io/rack (and so on) labels: for a
|
||||
fabric whose leaves do not reach across racks, or cables that do not.
|
||||
Nodes without the label are passed over for such a profile. Empty is
|
||||
no constraint.
|
||||
|
||||
### `weight`
|
||||
|
||||
`float64`
|
||||
@@ -2249,6 +2351,17 @@ Takes a [Control](#control) block.
|
||||
|
||||
`[]catalog.Model`
|
||||
|
||||
### `facility`
|
||||
|
||||
`Facility`
|
||||
|
||||
Facility is the power caps of racks, PDUs and other domains, and how
|
||||
a replica's draw is estimated against them. Where a node is comes from
|
||||
its llmgw.io/site, room, row, rack and pdu labels; holding a domain for
|
||||
maintenance is `llmgw ctl facility hold`, not configuration.
|
||||
|
||||
Takes a [Facility](#facility) block.
|
||||
|
||||
### `endpoints`
|
||||
|
||||
`[]Endpoint`
|
||||
|
||||
Reference in new issue
Block a user