llmgw/docs/facility-aware-placement.md

# Facility-aware placement: rack power, temperature and cooling

**Status: step 0, reading the nodes' own sensors, is built; steps 1–5 are
deferred** until the rack, PDU and power information of a real site is at
hand — without it they would be designed against guesses.
This is the design for step 5 of resource allocation, after reserved CPU cores
and memory (1), capacity classes and preemption (2), tenant capacity quotas (3)
and sizing by measured throughput (4), all of which shipped in v0.10.5. Of
what follows, only the node sensors (rollout step 0) exist in the code: they
are read, published and shown, and change no decision.

## Why

In a large installation the scarce thing is often not a GPU but the room it
sits in. Three failures are invisible to placement today:

- **A rack goes over its power budget.** Every node in it reports healthy;
  placement sees free cards and fills them; the breaker, or the facility's
  power cap, decides the rest. A rack with A and B feeds is worse: it runs
  fine until one feed fails and the other cannot carry the load.
- **Hot hardware is slow hardware.** GPUs throttle long before they fail. A
  replica placed in a rack whose inlet air is too warm serves fewer tokens per
  second than its measured capacity says, and nothing connects the two.
- **Correlated failure.** Two replicas of a model on two nodes in the same rack
  or on the same PDU are one failure away from zero. Placement spreads replicas
  across nodes, not across racks.

The goal is for placement to treat facility limits the way it already treats
memory and cores: reserve against a budget, refuse what does not fit, say why,
and never let missing data stop the cluster.

## What it builds on

| Already in place | Used for |
|---|---|
| Node labels, reported by agents, matched by `node_selector` | Where a node is: its rack, row, PDU |
| `inventory.Allocation` carries GPUs, memory and cores per shard | Adding watts, summed per rack and PDU |
| Placement rejections (`explain-placement`, headroom's "why not") | Naming the rack that is full or hot |
| Headroom: the scheduler dry-run on a copy of the inventory | Room that respects power budgets, for free |
| Preemption with classes, minimums and a dry run (step 2) | Shedding load from a hot rack in the right order |
| Measured throughput per profile (step 4) | Tokens per watt as a placement preference |
| Node sensors, read by every agent (step 0, built) | Heat and power with no facility integration: see below |
| `internal/promq`, a Prometheus client | Reading facility metrics that DCIM exporters publish |

## Concepts

**Location.** Each node has a place in a tree: site, room, row, rack. Power has
its own tree beside it: a rack draws from one or more PDUs, and each PDU from a
feed. Both are read from node labels with well-known keys:

```yaml
# agent.yaml
labels:
  llmgw.io/site: sto1
  llmgw.io/room: hall-b
  llmgw.io/row: "12"
  llmgw.io/rack: r12-07
  llmgw.io/pdu: r12-07-a,r12-07-b   # both feeds, for a dual-fed rack
```

A node without these labels is its own rack of one: placement behaves as it
does today.

**Domain.** A rack, a row, a PDU, a room: anything with a budget or a
condition. Domains are declared in a facility file (below) or discovered from
node labels; a label naming a domain the file does not declare creates it with
no budget and no condition.

**Budget.** A domain may have a power cap in watts. Placement reserves each
replica's estimated draw against every domain its node belongs to and refuses a
placement that would take any of them past its cap less a safety margin. For a
dual-fed rack the cap that matters is what one feed can carry alone, since that
is what the rack must survive.

**Condition.** A domain has a thermal and cooling condition: `ok`, `warm`, `hot`
or `unknown`, derived from readings against thresholds with hysteresis.
Conditions change what placement prefers and allows; they are not amounts to
fill.

**Reading.** A measured value with a time: inlet temperature, power draw, a
cooling unit's alarm state. Readings older than their staleness limit are
treated as absent.

## Estimating a replica's draw

What placement reserves has to be known before the replica runs, so it is an
estimate first and a measurement later:

1. **Declared.** A profile may say `power_watts`. The operator's figure wins.
2. **Derived.** Otherwise GPU profiles take each card's power limit from NVML
   (the agent already reads the cards; the power limit is one more field) plus
   a per-node share for CPU, memory and fans; CPU profiles take their reserved
   cores times a per-core figure for the node's CPU model.
3. **Measured.** Once replicas run, the agent reports their cards' actual draw
   (NVML) and, where the BMC allows, the node's (Redfish or IPMI DCMI). A
   domain's headroom is computed from the larger of what is reserved and what
   is measured, so an estimate that was too low is corrected by the room it
   stops offering, not by an outage.

## Decisions

### Power

- **Hard limit.** A placement that would take any domain past `cap × (1 −
  margin)` is rejected, with the domain and the figures: *rack r12-07: 14.2 of
  16.0 kW reserved, replica needs ~2.8, margin 10%*.
- **Feeds.** For a domain with two feeds the cap is the smaller feed's rating
  times its derating (typically 80%), not the sum.
- **Already over.** A domain found over its cap, because the estimate was low
  or the cap was lowered, gets no new placements, and an alert. It is not
  emptied automatically; shedding power is the facility's decision, and
  stopping inference to do it is ours only if asked (see shedding below).

### Temperature and cooling

Thresholds are per domain with defaults, and each condition needs to hold for a
while before it changes, in both directions, so a single reading neither
starts nor ends anything:

| Condition | Enters when | Placement |
|---|---|---|
| `ok` | below the warm threshold for 5 min | normal |
| `warm` | inlet above the warm threshold for 2 min, GPU temperatures near throttling, or any GPU thermally throttled | allowed, scored down: other domains first |
| `hot` | inlet above the hot threshold for 2 min, a cooling unit in alarm, or a sustained share of the domain's GPUs thermally throttled | no new placements, as a cordon would, but automatic and reversible |
| `unknown` | no fresh reading | as `ok`, and reported as unknown |

**Shedding** (optional, off by default): a domain `hot` for longer than its
shed delay has load moved off it using step 2's machinery. The lowest class goes
first, and replicas above their model's minimum go before any at it. A critical
minimum is never touched, and moves are rate limited to one replica per domain
per interval. The replica is placed elsewhere first where there is room, so
shedding moves work rather than dropping it. GPU power capping (`nvidia-smi
-pl`) is a gentler alternative worth measuring against shedding: less
throughput, no restart.

### Spreading

Replicas of one model are spread across racks, and across PDUs, the way they
are spread across nodes today. This is a preference, dropped when it is the only
thing standing in the way, as node spreading already is.

### Scoring

Placement today filters, then takes the first candidate in a fixed order
(fewest free GPUs for GPU profiles, fewest replicas for CPU ones, nodes holding
the weights first). Facility signals are better as a score than as more
filters. Candidates that pass the hard limits are ordered by:

1. condition: `ok` before `warm`;
2. power headroom left in the rack after this replica, more first, so load
   levels across racks;
3. the existing order.

Once measurements include power, step 4's profile preference can use tokens per
watt instead of tokens per GPU where both are known.

## Where the data comes from

A `FacilitySource` interface, implemented several ways, chosen in
`server.yaml`:

```go
type FacilitySource interface {
    // Domains is what the source knows: budgets, thresholds, members.
    Domains(ctx) ([]Domain, error)
    // Readings is the latest value of each signal, per domain.
    Readings(ctx) ([]Reading, error)
}
```

- **Static file.** Domains, caps and thresholds, written by hand or generated
  from a DCIM export. Enough for power budgets with no live data at all.
- **Prometheus.** DCIM, PDU (SNMP exporter) and BMC (Redfish exporter)
  readings usually end up in Prometheus. The source runs configured instant
  queries and maps their labels onto domains:

  ```yaml
  facility:
    source: prometheus
    prometheus: http://monitoring:9090
    signals:
      inlet_temp_c:
        query: max by (rack) (dcim_rack_inlet_temperature_celsius)
        domain_label: rack
        stale_after: 2m
      power_watts:
        query: sum by (pdu) (pdu_outlet_power_watts)
        domain_label: pdu
      cooling_alarm:
        query: max by (row) (crac_alarm)
        domain_label: row
  ```
- **Agents (built, step 0).** What each node reads about itself, every ten
  seconds at most, sent with its heartbeat:
  - **GPUs, from nvidia-smi:** temperature, memory (HBM) temperature where the
    driver reports it, power draw, the enforced power limit, and what is
    holding the clocks down now — `sw_power_cap`, `hw_slowdown`, `sw_thermal`,
    `hw_thermal`, `power_brake`.
  - **CPUs, from hwmon** (what lm_sensors reads, without it installed): the
    package temperature, or the hottest core; and the board's inlet or ambient
    sensor where it has one (`SYSTIN`, `Inlet`, `Ambient`).
  - **CPU power, from RAPL:** the packages' energy counters over time,
    subdomains not counted twice, counter wrap handled.

  A machine without a sensor reports nothing for it, never a zero; a VM has
  none of them. Published as `llmgw_gpu_temperature_celsius`,
  `llmgw_gpu_memory_temperature_celsius`, `llmgw_gpu_power_watts`,
  `llmgw_gpu_power_limit_watts`, `llmgw_gpu_throttled{reason}`,
  `llmgw_node_cpu_temperature_celsius`, `llmgw_node_cpu_power_watts` and
  `llmgw_node_inlet_temperature_celsius`, and on each node's page.

  What they can and cannot say:
  - **Throttling is the strongest signal there is.** A temperature is a proxy;
    a card holding its clocks down is the fact that invalidates its measured
    capacity, and it needs no threshold to be chosen.
  - **Measured power** is GPU draw plus CPU packages; memory, disks, fans and
    power-supply losses are not in it, so a node's draw is that plus an
    estimated remainder. A rack's is its llmgw nodes' plus a declared figure
    for everything else in it: switches and storage report nothing here.
  - **CPU and GPU temperatures follow the load, not the room.** A hot rack is
    inferred from many nodes running hot at comparable load, or from
    throttling, not from one node's CPU. Only a board inlet sensor reads the
    air, and most boards have none.
  - **No budget comes from a sensor.** A reading is a draw; the cap it is
    measured against still comes from the facility.
- **Simulated.** Readings from a script, for tests and for rehearsing a
  thermal event on a cluster without facility data.

## Safety

- **Missing data never blocks.** A source that is down, a reading that is stale,
  a node without location labels: placement behaves as it does today, and says
  the data is missing (a metric, a line on the cluster page). Facility
  awareness can make placement more careful, never impossible.
- **Hysteresis and hold times** on every condition, so a noisy sensor does not
  flap a rack between allowed and forbidden.
- **Observe-only first.** A mode where every decision is computed, logged and
  shown ("would have refused: rack r12-07 hot") but nothing is enforced, run
  for long enough to trust the thresholds before turning enforcement on.
- **Manual override.** `llmgw ctl facility hold RACK` and `release RACK`, for the
  facility team to mark a domain unavailable, or to clear a condition they know
  to be a sensor fault.
- **Shedding is opt-in**, per domain or cluster-wide, and rate limited.

## What people see

- `explain-placement` and headroom name the domain: *rack r12-07 is hot (inlet
  31°C for 6 min)*, *pdu r12-07-a: 7.1 of 7.4 kW*.
- `llmgw ctl facility` lists domains with budget, reserved, measured and
  condition. The cluster page gets a rack summary beside the nodes table.
- Metrics: `llmgw_domain_power_budget_watts`, `_reserved_watts`,
  `_measured_watts`, `llmgw_domain_condition{condition}`,
  `llmgw_domain_reading_age_seconds`; alerts for a domain near its cap, a
  domain hot, a domain over its cap, and facility data gone stale.

## Rollout

Each step ships on its own and is useful without the next:

0. **Node sensors, observe-only (built).** GPU heat, power and throttling,
   CPU heat and power, inlet air where the board has a sensor; published and
   shown, no decision changed.
1. **Location and spreading.** Well-known labels, domains from labels, replicas
   spread across racks and PDUs, locations on the node views. No facility data
   needed.
2. **Power budgets from a static file.** Caps and margins, estimated draw per
   replica (declared or derived), hard limit in placement and headroom.
3. **Readings, observe-only.** The Prometheus source beside the agents',
   conditions with hysteresis (throttling first among them), measured power
   folded into headroom; decisions shown, not enforced.
4. **Thermal enforcement.** `warm` scored down, `hot` refused, manual hold and
   release.
5. **Shedding and tokens per watt.** Opt-in shedding through step 2's
   preemption; measured power in profile preference.

## Testing

- The simulated source drives unit tests of every condition transition, the
  hysteresis, staleness, and the placement outcomes, with mutation checks as
  for steps 1–4.
- Property tests: whatever the readings, placement never exceeds a declared
  cap with fresh data, and never refuses everything because data is missing.
- On the PoC cluster, with no facility data: step 1 with labelled VMs, step 2
  with a static file of made-up caps, steps 3–4 with the simulated source
  replaying a thermal event.
- Before enforcement anywhere real: a period in observe-only mode against the
  site's actual data.

## Open questions

- **What the sites have.** Which DCIM, which PDUs, which BMCs, and whether any
  of it already reaches a Prometheus. The source design assumes Prometheus is
  the common ground; a site without it needs another source.
- **Who owns the numbers.** Caps, deratings and thresholds belong to the
  facility team. Whether they edit llmgw's file, or llmgw reads theirs.
- **Phase balance.** Three-phase PDUs have per-phase limits; whether placement
  should see phases or only the PDU total.
- **Cooling granularity.** Cooling is often per row or per aisle, not per rack;
  the domain tree allows either, but the defaults should match what sites
  actually measure.
- **Power capping instead of shedding.** Whether to lower GPU power limits in a
  warm rack rather than move replicas, and how that interacts with measured
  capacity from step 4, which would no longer hold at the lower limit.