Release v0.10.5

Collects rc.1 to rc.14: the mirror page in tabs with a card per model,
one line per record in every admin table, reserved CPU cores and required
memory, capacity classes and preemption, tenant capacity quotas, sizing by
measured throughput, and node sensor readings. Adds `llmgw upgrade` for the
mirror host, which `ctl upgrade` does not reach, and says so when it ends.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

CHANGELOG.md

@@ -20,10 +20,156 @@ is only released once its **exit criterion** is met, not when its code lands.
 Until v1.0.0 the minor version is the breaking one: configuration formats and Go
 APIs change between minors without a deprecation period.
 
-## Unreleased
+## v0.10.5 — 2026-09-30
 
-Changes since v0.10.4, published as release candidates (v0.10.5-rc.N) until
-there are enough for v0.10.5.
+Published first as release candidates v0.10.5-rc.1 to rc.14; rc.5 failed its
+format check and has no binaries.
+
+After upgrading:
+
+- **Declare what CPU models need.** A CPU model added from now on must say its
+  memory; one stored before this still loads and is flagged in `ctl models`
+  until `llmgw ctl model resources MODEL --memory GB --cores N` is run. Cores
+  are then reserved per replica and become the engine's thread count, for
+  replicas started after the change: `llmgw ctl restart --model MODEL`.
+- **Upgrade the mirror on its own host**, now with one command:
+  `sudo llmgw upgrade v0.10.5 --service llmgw-mirror` (by hand once, for a
+  mirror older than this). `llmgw ctl upgrade` says so at the end.
+
+### CPU cores are reserved, and memory has to be declared
+
+The first step of resource allocation for larger clusters: what a replica
+holds of a node is reserved, not assumed.
+
+- **Cores.** A profile's `cpu_cores` is reserved out of its node's CPUs, as
+  memory is, and is what its engine computes on (llama.cpp's `--threads`). A
+  node holds as many CPU replicas as it has cores for, and two on one node no
+  longer spin against each other. Agents report their CPU count; a node that
+  does not, and a profile that declares no cores, constrain nothing.
+- **Memory is required of new CPU models.** A model with no GPU is all host
+  memory, and one declaring none looked to placement as if it fitted
+  everywhere, any number of times. Adding one without `min_memory_gb` is now
+  refused. Models stored before this still load, and say what is missing.
+- **Declared against measured.** Each model shows what one replica reserves
+  and the most one of its replicas uses — in `ctl models` (RESERVES) and on the
+  cluster page — with a warning when nothing is declared or when it uses more
+  than a tenth over what it reserves.
+- **Setting them.** `llmgw ctl model resources MODEL --memory GB --cores N`
+  (and `POST /api/v1/models/{name}/resources`) for a model added at runtime.
+  `ctl model serve` and the mirror's serve form fill in cores — half the
+  smallest node the model fits on — beside the memory they already estimate.
+  Running replicas keep their old figures until restarted.
+
+The commands are `llmgw ctl model resources` and `llmgw ctl model class`; in
+rc.7 the first was briefly `llmgw ctl resources`.
+
+### Capacity classes, and preemption
+
+A model has a class — `critical`, `standard` or `opportunistic` — saying what
+its capacity is worth against another's when there is not enough for both.
+Unset, an on-demand model is opportunistic and every other model standard,
+which is how the catalog behaved before. `llmgw ctl model class MODEL CLASS`
+sets it.
+
+When a model cannot be placed, replicas are stopped to make room for it, by
+these rules:
+
+- A lower class gives way to a higher one.
+- Within a class, only replicas above a model's minimum — room it grew into
+  under load — and only for a model still below its own minimum.
+- A critical model's minimum is never taken, nor a replica that has only just
+  loaded, nor one spanning several nodes.
+- Idle on-demand replicas go first, then what lower classes grew into, then
+  the same within the class, then lower classes' minimums; the longest unused
+  first within each, and of equally idle ones the smallest.
+- The set is tried on a copy of the inventory first, at most three replicas,
+  and nothing is stopped unless it makes the room. Of a set that does, each
+  replica whose stopping turns out not to be needed is left running, so an
+  idle replica tried first on the wrong node is not stopped with the one that
+  made the room.
+
+A replica stopped this way is drained and given 30 seconds to finish what it
+is serving. Its model is held back for up to five minutes, or until the model
+it gave way to has its replicas, so it does not take the room straight back.
+`ctl models` shows each model's class and "gave room to X" while held; the
+cluster page shows the class and how many replicas were lost; the metric is
+`llmgw_model_preempted_total{model,class}`, and every preemption is logged.
+
+### Capacity quotas for the tenancy tree
+
+An organization, team or project can be guaranteed GPUs, CPU cores and
+memory, and limited to at most some amount of each. What it holds is what the
+replicas of the models it owns hold — and those of the teams and projects
+beneath it. `llmgw ctl model owner MODEL team:Acme/ml` says whose a model is;
+one with no owner is shared and counts against nobody.
+
+- **A guarantee is kept.** A replica within its owner's guarantee is never
+  taken for another scope's model, whatever the classes: a guarantee a higher
+  class elsewhere could take would not be one.
+- **Beyond it is borrowing.** Free room is used; when a scope within its own
+  guarantee needs room, what others hold beyond theirs is taken back first,
+  whatever their class — except a critical model's minimum.
+- **A limit is a ceiling.** A model whose owner would go past a limit anywhere
+  up the tree gets no further replica, and says which limit; headroom and the
+  mirror card stop there too.
+
+`llmgw ctl quota set team:Acme/ml --gpus 8 --limit-gpus 16` (also `--cores`,
+`--memory` and their limits), `llmgw ctl quota list` with what each scope holds,
+`llmgw ctl quota clear`. Scopes are named by path or ID; a name that could mean
+two is refused. Changes take effect within half a minute. Metrics:
+`llmgw_capacity_holds`, `llmgw_capacity_guarantee` and `llmgw_capacity_limit`
+by scope and resource; alert: CapacityAtLimit.
+
+### Sized by what a replica was measured to serve
+
+What `llmgw ctl bench MODEL --save` measured — the most requests at once one
+replica served while meeting its SLO, and its throughput there — now drives
+decisions rather than only being printed:
+
+- **Scaling up** when requests per replica pass what was measured, not only
+  when the engines' queues fill: an engine takes requests into its slots long
+  after latency has gone.
+- **Scaling down** only when the replicas left would stay under 70% of what
+  they were measured to hold, so a replica taken back does not put the rest
+  straight over.
+- **Choosing a profile.** Among profiles of equal priority, the one measured
+  to serve the most per GPU — or per core — is tried first. Priority set by an
+  operator still wins.
+- **Sizing.** `llmgw ctl model target MODEL --tokens-per-sec N` says how many
+  replicas that takes, how many run, and whether the rest fit, with the reason
+  when they do not; `--apply` sets it if they do. Headroom reports what one
+  more replica would serve.
+
+A model never measured behaves exactly as before. Measurements are read from
+the store every half minute.
+
+### Upgrading a machine the control plane does not reach
+
+`sudo llmgw upgrade VERSION --service UNIT` upgrades the machine it runs on —
+the mirror's host above all, whose service runs unprivileged and cannot replace
+its own binary as an agent does. The release's checksums are checked, the old
+binary kept, the unit restarted and watched until it stays up.
+
+### Nodes report their own heat and power
+
+The first step of facility-aware placement (docs/facility-aware-placement.md),
+observe-only: nothing is decided on these yet. Every agent reads, at most every
+ten seconds, and sends with its heartbeat:
+
+- **GPUs** from nvidia-smi: temperature, HBM temperature where the driver has
+  it, power draw and the enforced limit, and what is holding the clocks down —
+  power cap, thermal slowdown, power brake. A GPU's temperature was read once,
+  at registration; it is now current.
+- **CPUs** from hwmon, which is what lm_sensors reads, with nothing to install:
+  the package temperature, and the board's inlet air sensor where it has one.
+- **CPU power** from RAPL's energy counters.
+
+A sensor a machine does not have is not reported, never shown as zero. New
+metrics: `llmgw_gpu_power_watts`, `llmgw_gpu_power_limit_watts`,
+`llmgw_gpu_memory_temperature_celsius`, `llmgw_gpu_throttled{reason}`,
+`llmgw_node_cpu_temperature_celsius`, `llmgw_node_cpu_power_watts`,
+`llmgw_node_inlet_temperature_celsius`. Each node's page shows them, with a GPU
+table of temperature, power against its limit, and what is throttling it.
 
 ### The mirror page in tabs
 

api/proto/llmgw/v1/agent.proto

@@ -120,6 +120,10 @@ message Register {
   // meant "an agent too old to say" and constrained nothing — so a freshly
   // built node with no engine installed yet was a candidate for every model.
   bool engines_probed = 14;
+  // cpu_count is the CPUs the agent may run on. Placement reserves cores
+  // for CPU replicas out of it, the way it reserves memory out of
+  // memory_total_mb; zero, from an older agent, reserves nothing.
+  int32 cpu_count = 18;
   // imex is the service that lets a multi-node NVLink domain share memory.
   IMEX imex = 15;
 
@@ -281,6 +285,25 @@ message NodeLoad {
   int64 memory_used_bytes = 6;
   int64 memory_available_bytes = 7;
   repeated ShardLoad shards = 8;
+  // What the machine's own sensors say; zero is not measured. cpu_temp_c is
+  // the hottest CPU sensor, cpu_power_w the CPU packages' draw from RAPL,
+  // inlet_temp_c the board's inlet or ambient sensor where it has one.
+  double cpu_temp_c = 9;
+  double cpu_power_w = 10;
+  double inlet_temp_c = 11;
+  repeated GPUSensor gpus = 12;
+}
+
+// GPUSensor is one card's heat, power, and what is holding it back.
+message GPUSensor {
+  int32 index = 1;
+  double temp_c = 2;
+  double memory_temp_c = 3;
+  double power_w = 4;
+  double power_limit_w = 5;
+  // throttle names what is holding the clocks down now: sw_power_cap,
+  // hw_slowdown, sw_thermal, hw_thermal, power_brake.
+  repeated string throttle = 6;
 }
 
 // ShardLoad is one engine process group's use.
@@ -420,6 +443,8 @@ message Heartbeat {
   // domain, and the node has to stop being offered work that spans it.
   repeated GPUFabricState gpu_fabric = 16;
   IMEX imex = 17;
+  // cpu_count as in Register, for a VM given more CPUs while it runs.
+  int32 cpu_count = 18;
 }
 
 // Provision is the software a node should have: which engines, at which
@@ -530,6 +555,9 @@ message StartShard {
   // control plane from the replica id. The router sends it; nothing else
   // that can reach the node's ports has it.
   string api_key = 16;
+  // cpu_threads is how many CPU threads a shard with no GPU computes on:
+  // the cores placement reserved for it. Zero leaves it to the agent.
+  int32 cpu_threads = 17;
 }
 
 message EngineProfile {

deploy/monitoring/llmgw-rules.yml

@@ -84,6 +84,28 @@ groups:
             <model>` says what each node lacks. For an on-demand model, requests
             for it are told this rather than asked to retry.
 
+      - alert: ModelPreempted
+        expr: increase(llmgw_model_preempted_total[15m]) > 0
+        labels: { severity: info }
+        annotations:
+          summary: "{{ $labels.model }} ({{ $labels.class }}) gave up replicas to a higher class"
+          description: >-
+            Capacity ran short and a model of a higher class took this one's
+            room. Expected now and then; if it keeps firing for the same model,
+            the cluster is too small for what is asked of it at these classes.
+            `llmgw ctl models` shows who it gave way to.
+
+      - alert: CapacityAtLimit
+        expr: llmgw_capacity_holds >= on(scope, resource) llmgw_capacity_limit
+        for: 15m
+        labels: { severity: info }
+        annotations:
+          summary: "{{ $labels.scope }} holds all the {{ $labels.resource }} it may"
+          description: >-
+            Its models get no further replicas, under load or not, until
+            something of theirs stops or the limit is raised: `llmgw ctl quota
+            list`, `llmgw ctl quota set`.
+
       - alert: EngineInstallFailed
         expr: llmgw_node_engine_install{state="failed"} == 1
         for: 15m

docs/facility-aware-placement.md

@@ -0,0 +1,303 @@
+# Facility-aware placement: rack power, temperature and cooling
+
+**Status: step 0, reading the nodes' own sensors, is built; steps 1–5 are
+deferred** until the rack, PDU and power information of a real site is at
+hand — without it they would be designed against guesses.
+This is the design for step 5 of resource allocation, after reserved CPU cores
+and memory (1), capacity classes and preemption (2), tenant capacity quotas (3)
+and sizing by measured throughput (4), all of which shipped in v0.10.5. Of
+what follows, only the node sensors (rollout step 0) exist in the code: they
+are read, published and shown, and change no decision.
+
+## Why
+
+In a large installation the scarce thing is often not a GPU but the room it
+sits in. Three failures are invisible to placement today:
+
+- **A rack goes over its power budget.** Every node in it reports healthy;
+  placement sees free cards and fills them; the breaker, or the facility's
+  power cap, decides the rest. A rack with A and B feeds is worse: it runs
+  fine until one feed fails and the other cannot carry the load.
+- **Hot hardware is slow hardware.** GPUs throttle long before they fail. A
+  replica placed in a rack whose inlet air is too warm serves fewer tokens per
+  second than its measured capacity says, and nothing connects the two.
+- **Correlated failure.** Two replicas of a model on two nodes in the same rack
+  or on the same PDU are one failure away from zero. Placement spreads replicas
+  across nodes, not across racks.
+
+The goal is for placement to treat facility limits the way it already treats
+memory and cores: reserve against a budget, refuse what does not fit, say why,
+and never let missing data stop the cluster.
+
+## What it builds on
+
+| Already in place | Used for |
+|---|---|
+| Node labels, reported by agents, matched by `node_selector` | Where a node is: its rack, row, PDU |
+| `inventory.Allocation` carries GPUs, memory and cores per shard | Adding watts, summed per rack and PDU |
+| Placement rejections (`explain-placement`, headroom's "why not") | Naming the rack that is full or hot |
+| Headroom: the scheduler dry-run on a copy of the inventory | Room that respects power budgets, for free |
+| Preemption with classes, minimums and a dry run (step 2) | Shedding load from a hot rack in the right order |
+| Measured throughput per profile (step 4) | Tokens per watt as a placement preference |
+| Node sensors, read by every agent (step 0, built) | Heat and power with no facility integration: see below |
+| `internal/promq`, a Prometheus client | Reading facility metrics that DCIM exporters publish |
+
+## Concepts
+
+**Location.** Each node has a place in a tree: site, room, row, rack. Power has
+its own tree beside it: a rack draws from one or more PDUs, and each PDU from a
+feed. Both are read from node labels with well-known keys:
+
+```yaml
+# agent.yaml
+labels:
+  llmgw.io/site: sto1
+  llmgw.io/room: hall-b
+  llmgw.io/row: "12"
+  llmgw.io/rack: r12-07
+  llmgw.io/pdu: r12-07-a,r12-07-b   # both feeds, for a dual-fed rack
+```
+
+A node without these labels is its own rack of one: placement behaves as it
+does today.
+
+**Domain.** A rack, a row, a PDU, a room: anything with a budget or a
+condition. Domains are declared in a facility file (below) or discovered from
+node labels; a label naming a domain the file does not declare creates it with
+no budget and no condition.
+
+**Budget.** A domain may have a power cap in watts. Placement reserves each
+replica's estimated draw against every domain its node belongs to and refuses a
+placement that would take any of them past its cap less a safety margin. For a
+dual-fed rack the cap that matters is what one feed can carry alone, since that
+is what the rack must survive.
+
+**Condition.** A domain has a thermal and cooling condition: `ok`, `warm`, `hot`
+or `unknown`, derived from readings against thresholds with hysteresis.
+Conditions change what placement prefers and allows; they are not amounts to
+fill.
+
+**Reading.** A measured value with a time: inlet temperature, power draw, a
+cooling unit's alarm state. Readings older than their staleness limit are
+treated as absent.
+
+## Estimating a replica's draw
+
+What placement reserves has to be known before the replica runs, so it is an
+estimate first and a measurement later:
+
+1. **Declared.** A profile may say `power_watts`. The operator's figure wins.
+2. **Derived.** Otherwise GPU profiles take each card's power limit from NVML
+   (the agent already reads the cards; the power limit is one more field) plus
+   a per-node share for CPU, memory and fans; CPU profiles take their reserved
+   cores times a per-core figure for the node's CPU model.
+3. **Measured.** Once replicas run, the agent reports their cards' actual draw
+   (NVML) and, where the BMC allows, the node's (Redfish or IPMI DCMI). A
+   domain's headroom is computed from the larger of what is reserved and what
+   is measured, so an estimate that was too low is corrected by the room it
+   stops offering, not by an outage.
+
+## Decisions
+
+### Power
+
+- **Hard limit.** A placement that would take any domain past `cap × (1 −
+  margin)` is rejected, with the domain and the figures: *rack r12-07: 14.2 of
+  16.0 kW reserved, replica needs ~2.8, margin 10%*.
+- **Feeds.** For a domain with two feeds the cap is the smaller feed's rating
+  times its derating (typically 80%), not the sum.
+- **Already over.** A domain found over its cap, because the estimate was low
+  or the cap was lowered, gets no new placements, and an alert. It is not
+  emptied automatically; shedding power is the facility's decision, and
+  stopping inference to do it is ours only if asked (see shedding below).
+
+### Temperature and cooling
+
+Thresholds are per domain with defaults, and each condition needs to hold for a
+while before it changes, in both directions, so a single reading neither
+starts nor ends anything:
+
+| Condition | Enters when | Placement |
+|---|---|---|
+| `ok` | below the warm threshold for 5 min | normal |
+| `warm` | inlet above the warm threshold for 2 min, GPU temperatures near throttling, or any GPU thermally throttled | allowed, scored down: other domains first |
+| `hot` | inlet above the hot threshold for 2 min, a cooling unit in alarm, or a sustained share of the domain's GPUs thermally throttled | no new placements, as a cordon would, but automatic and reversible |
+| `unknown` | no fresh reading | as `ok`, and reported as unknown |
+
+**Shedding** (optional, off by default): a domain `hot` for longer than its
+shed delay has load moved off it using step 2's machinery. The lowest class goes
+first, and replicas above their model's minimum go before any at it. A critical
+minimum is never touched, and moves are rate limited to one replica per domain
+per interval. The replica is placed elsewhere first where there is room, so
+shedding moves work rather than dropping it. GPU power capping (`nvidia-smi
+-pl`) is a gentler alternative worth measuring against shedding: less
+throughput, no restart.
+
+### Spreading
+
+Replicas of one model are spread across racks, and across PDUs, the way they
+are spread across nodes today. This is a preference, dropped when it is the only
+thing standing in the way, as node spreading already is.
+
+### Scoring
+
+Placement today filters, then takes the first candidate in a fixed order
+(fewest free GPUs for GPU profiles, fewest replicas for CPU ones, nodes holding
+the weights first). Facility signals are better as a score than as more
+filters. Candidates that pass the hard limits are ordered by:
+
+1. condition: `ok` before `warm`;
+2. power headroom left in the rack after this replica, more first, so load
+   levels across racks;
+3. the existing order.
+
+Once measurements include power, step 4's profile preference can use tokens per
+watt instead of tokens per GPU where both are known.
+
+## Where the data comes from
+
+A `FacilitySource` interface, implemented several ways, chosen in
+`server.yaml`:
+
+```go
+type FacilitySource interface {
+    // Domains is what the source knows: budgets, thresholds, members.
+    Domains(ctx) ([]Domain, error)
+    // Readings is the latest value of each signal, per domain.
+    Readings(ctx) ([]Reading, error)
+}
+```
+
+- **Static file.** Domains, caps and thresholds, written by hand or generated
+  from a DCIM export. Enough for power budgets with no live data at all.
+- **Prometheus.** DCIM, PDU (SNMP exporter) and BMC (Redfish exporter)
+  readings usually end up in Prometheus. The source runs configured instant
+  queries and maps their labels onto domains:
+
+  ```yaml
+  facility:
+    source: prometheus
+    prometheus: http://monitoring:9090
+    signals:
+      inlet_temp_c:
+        query: max by (rack) (dcim_rack_inlet_temperature_celsius)
+        domain_label: rack
+        stale_after: 2m
+      power_watts:
+        query: sum by (pdu) (pdu_outlet_power_watts)
+        domain_label: pdu
+      cooling_alarm:
+        query: max by (row) (crac_alarm)
+        domain_label: row
+  ```
+- **Agents (built, step 0).** What each node reads about itself, every ten
+  seconds at most, sent with its heartbeat:
+  - **GPUs, from nvidia-smi:** temperature, memory (HBM) temperature where the
+    driver reports it, power draw, the enforced power limit, and what is
+    holding the clocks down now — `sw_power_cap`, `hw_slowdown`, `sw_thermal`,
+    `hw_thermal`, `power_brake`.
+  - **CPUs, from hwmon** (what lm_sensors reads, without it installed): the
+    package temperature, or the hottest core; and the board's inlet or ambient
+    sensor where it has one (`SYSTIN`, `Inlet`, `Ambient`).
+  - **CPU power, from RAPL:** the packages' energy counters over time,
+    subdomains not counted twice, counter wrap handled.
+
+  A machine without a sensor reports nothing for it, never a zero; a VM has
+  none of them. Published as `llmgw_gpu_temperature_celsius`,
+  `llmgw_gpu_memory_temperature_celsius`, `llmgw_gpu_power_watts`,
+  `llmgw_gpu_power_limit_watts`, `llmgw_gpu_throttled{reason}`,
+  `llmgw_node_cpu_temperature_celsius`, `llmgw_node_cpu_power_watts` and
+  `llmgw_node_inlet_temperature_celsius`, and on each node's page.
+
+  What they can and cannot say:
+  - **Throttling is the strongest signal there is.** A temperature is a proxy;
+    a card holding its clocks down is the fact that invalidates its measured
+    capacity, and it needs no threshold to be chosen.
+  - **Measured power** is GPU draw plus CPU packages; memory, disks, fans and
+    power-supply losses are not in it, so a node's draw is that plus an
+    estimated remainder. A rack's is its llmgw nodes' plus a declared figure
+    for everything else in it: switches and storage report nothing here.
+  - **CPU and GPU temperatures follow the load, not the room.** A hot rack is
+    inferred from many nodes running hot at comparable load, or from
+    throttling, not from one node's CPU. Only a board inlet sensor reads the
+    air, and most boards have none.
+  - **No budget comes from a sensor.** A reading is a draw; the cap it is
+    measured against still comes from the facility.
+- **Simulated.** Readings from a script, for tests and for rehearsing a
+  thermal event on a cluster without facility data.
+
+## Safety
+
+- **Missing data never blocks.** A source that is down, a reading that is stale,
+  a node without location labels: placement behaves as it does today, and says
+  the data is missing (a metric, a line on the cluster page). Facility
+  awareness can make placement more careful, never impossible.
+- **Hysteresis and hold times** on every condition, so a noisy sensor does not
+  flap a rack between allowed and forbidden.
+- **Observe-only first.** A mode where every decision is computed, logged and
+  shown ("would have refused: rack r12-07 hot") but nothing is enforced, run
+  for long enough to trust the thresholds before turning enforcement on.
+- **Manual override.** `llmgw ctl facility hold RACK` and `release RACK`, for the
+  facility team to mark a domain unavailable, or to clear a condition they know
+  to be a sensor fault.
+- **Shedding is opt-in**, per domain or cluster-wide, and rate limited.
+
+## What people see
+
+- `explain-placement` and headroom name the domain: *rack r12-07 is hot (inlet
+  31°C for 6 min)*, *pdu r12-07-a: 7.1 of 7.4 kW*.
+- `llmgw ctl facility` lists domains with budget, reserved, measured and
+  condition. The cluster page gets a rack summary beside the nodes table.
+- Metrics: `llmgw_domain_power_budget_watts`, `_reserved_watts`,
+  `_measured_watts`, `llmgw_domain_condition{condition}`,
+  `llmgw_domain_reading_age_seconds`; alerts for a domain near its cap, a
+  domain hot, a domain over its cap, and facility data gone stale.
+
+## Rollout
+
+Each step ships on its own and is useful without the next:
+
+0. **Node sensors, observe-only (built).** GPU heat, power and throttling,
+   CPU heat and power, inlet air where the board has a sensor; published and
+   shown, no decision changed.
+1. **Location and spreading.** Well-known labels, domains from labels, replicas
+   spread across racks and PDUs, locations on the node views. No facility data
+   needed.
+2. **Power budgets from a static file.** Caps and margins, estimated draw per
+   replica (declared or derived), hard limit in placement and headroom.
+3. **Readings, observe-only.** The Prometheus source beside the agents',
+   conditions with hysteresis (throttling first among them), measured power
+   folded into headroom; decisions shown, not enforced.
+4. **Thermal enforcement.** `warm` scored down, `hot` refused, manual hold and
+   release.
+5. **Shedding and tokens per watt.** Opt-in shedding through step 2's
+   preemption; measured power in profile preference.
+
+## Testing
+
+- The simulated source drives unit tests of every condition transition, the
+  hysteresis, staleness, and the placement outcomes, with mutation checks as
+  for steps 1–4.
+- Property tests: whatever the readings, placement never exceeds a declared
+  cap with fresh data, and never refuses everything because data is missing.
+- On the PoC cluster, with no facility data: step 1 with labelled VMs, step 2
+  with a static file of made-up caps, steps 3–4 with the simulated source
+  replaying a thermal event.
+- Before enforcement anywhere real: a period in observe-only mode against the
+  site's actual data.
+
+## Open questions
+
+- **What the sites have.** Which DCIM, which PDUs, which BMCs, and whether any
+  of it already reaches a Prometheus. The source design assumes Prometheus is
+  the common ground; a site without it needs another source.
+- **Who owns the numbers.** Caps, deratings and thresholds belong to the
+  facility team. Whether they edit llmgw's file, or llmgw reads theirs.
+- **Phase balance.** Three-phase PDUs have per-phase limits; whether placement
+  should see phases or only the PDU total.
+- **Cooling granularity.** Cooling is often per row or per aisle, not per rack;
+  the domain tree allows either, but the defaults should match what sites
+  actually measure.
+- **Power capping instead of shedding.** Whether to lower GPU power limits in a
+  warm rack rather than move replicas, and how that interacts with measured
+  capacity from step 4, which would no longer hold at the lower limit.

docs/install.md

@@ -940,8 +940,18 @@ restart either — it rebuilds the fleet from what the agents report.
 fields, and both sides treat an absent one as unknown rather than as a default.
 Agents older than v0.10.4 cannot upgrade themselves: move those by hand once,
 installing the release binary and restarting `llmgw-agent`, and every upgrade
-after that is the command above. The mirror, which is not an agent, is
-upgraded the same way by hand.
+after that is the command above.
+
+**The mirror is upgraded on its own host**, since it is not an agent and its
+service cannot replace its own binary:
+
+```
+sudo llmgw upgrade v0.10.5 --service llmgw-mirror
+```
+
+The same checks — the release's checksums, the old binary kept beside the new —
+then a restart of the unit, waiting for it to stay up. A mirror older than
+v0.10.5 does not have the command: install the release binary by hand once.
 
 Then confirm what you have, rather than assuming:
 

docs/reference.md

@@ -92,6 +92,8 @@ rather than repeated.
 | `source` | `string` | [Model](#model) |
 | `profiles` | `[]Profile` | [Model](#model) |
 | `scaling` | `Scaling` | [Model](#model) |
+| `class` | `string` | [Model](#model) |
+| `owner` | `string` | [Model](#model) |
 | `fallbacks` | `[]string` | [Model](#model) |
 | `routing_profile` | `string` | [Model](#model) |
 | `slo` | `SLO` | [Model](#model) |
@@ -129,6 +131,7 @@ rather than repeated.
 | `args` | `[]string` | [Profile](#profile) |
 | `env` | `map[string]string` | [Profile](#profile) |
 | `min_memory_gb` | `int` | [Profile](#profile) |
+| `cpu_cores` | `int` | [Profile](#profile) |
 | `min_disk_gb` | `int` | [Profile](#profile) |
 | `node_selector` | `map[string]string` | [Profile](#profile) |
 | `weight` | `float64` | [Profile](#profile) |
@@ -735,6 +738,23 @@ Source is the weights path or a HuggingFace id.
 
 Takes a [Scaling](#scaling) block.
 
+### `class`
+
+`string`
+
+Class is how much this model's capacity is worth against another's,
+when there is not enough for both: critical, standard or opportunistic.
+Empty is standard, or opportunistic for a model loaded on demand. See
+EffectiveClass.
+
+### `owner`
+
+`string`
+
+Owner is the part of the tenancy tree whose capacity quota this model
+counts against, written org:ID, team:ID or project:ID. Empty is shared:
+held by the cluster, counted against nobody.
+
 ### `fallbacks`
 
 `[]string`
@@ -966,6 +986,20 @@ this. A node that does not report its memory is not excluded either —
 an older agent must not have its nodes quietly emptied by an upgrade of
 the control plane.
 
+### `cpu_cores`
+
+`int`
+
+CPUCores is the cores a replica of a profile with no GPU is given:
+reserved out of its node's CPUs by placement, as memory is, and what
+its engine computes on (llama.cpp's --threads).
+
+Reserved because an engine takes every core it is told about, and two
+on one node that each think they have all of them spin against each
+other: on an 8-vCPU node, two 7B replicas fell from 10 tokens a second
+to under one each. Zero reserves nothing and leaves the thread count to
+the node, which is every profile from before this.
+
 ### `min_disk_gb`
 
 `int`

internal/agent/agent.go

@@ -16,6 +16,7 @@ import (
 	"net/http"
 	"os"
 	"path/filepath"
+	"runtime"
 	"slices"
 	"sync"
 	"time"
@@ -100,6 +101,9 @@ type Agent struct {
 	smsErr map[string]string
 	// load measures the machine between heartbeats.
 	load *discovery.LoadSampler
+	// sensors reads heat and power: CPU and board from hwmon and RAPL, GPUs
+	// from nvidia-smi.
+	sensors *discovery.SensorSampler
 
 	// fetchWatchers are the shards waiting on each source, so progress on one
 	// download reaches every shard that asked for it. A fetch is single-flight
@@ -145,6 +149,7 @@ func New(cfg Config, httpClient connectHTTPClient, log *slog.Logger) (*Agent, er
 		logRing:      ring,
 		provKick:     make(chan struct{}, 1),
 		load:         discovery.NewLoadSampler(),
+		sensors:      discovery.NewSensorSampler(),
 	}
 	a.prov = &provision.Installer{Root: cfg.EnginesDir, Log: log, OnStatus: a.provisionStatus}
 	return a, nil
@@ -549,6 +554,7 @@ func (a *Agent) heartbeat() *llmgwv1.AgentMessage {
 	// and a node that grew from 7 GB to 32 GB has to be seen to have grown
 	// without anybody restarting anything.
 	hb.MemoryTotalMb = int64(discovery.MemoryTotalMB())
+	hb.CpuCount = int32(runtime.NumCPU())
 
 	a.reprobeEngines()
 	a.mu.Lock()
@@ -636,6 +642,16 @@ func (a *Agent) sampleLoad() *llmgwv1.NodeLoad {
 			ShardId: sh.ID, CpuCores: sh.CPUCores, MemoryRssBytes: sh.RSSBytes,
 		})
 	}
+	if a.sensors != nil {
+		se := a.sensors.Sample(context.Background())
+		out.CpuTempC, out.CpuPowerW, out.InletTempC = se.CPUTempC, se.CPUPowerW, se.InletTempC
+		for _, g := range se.GPUs {
+			out.Gpus = append(out.Gpus, &llmgwv1.GPUSensor{
+				Index: int32(g.Index), TempC: g.TempC, MemoryTempC: g.MemoryTempC,
+				PowerW: g.PowerW, PowerLimitW: g.PowerLimitW, Throttle: g.Throttle,
+			})
+		}
+	}
 	return out
 }
 

internal/agent/discovery/sensors.go

@@ -0,0 +1,242 @@
+package discovery
+
+import (
+	"context"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"strconv"
+	"strings"
+	"sync"
+	"time"
+)
+
+// Sensors is what a node's own hardware says about heat and power: the CPUs,
+// the board where it has an inlet sensor, and each GPU. A zero is not
+// measured, never a reading of zero — a VM has none of these, and must not
+// look like a machine running at 0 °C on no power.
+type Sensors struct {
+	// CPUTempC is the hottest CPU sensor: the package, or the hottest core
+	// where no package reading exists.
+	CPUTempC float64
+	// CPUPowerW is the CPU packages' draw, from RAPL: the change in their
+	// energy counters over the time between two samples.
+	CPUPowerW float64
+	// InletTempC is the air coming in, where the board has a sensor it calls
+	// that. Most do not, and a CPU's temperature is not a substitute: it
+	// follows the load, not the room.
+	InletTempC float64
+	GPUs       []GPUSensor
+}
+
+// GPUSensor is one card's heat, power and whether it is slowing itself down.
+type GPUSensor struct {
+	Index       int
+	TempC       float64
+	MemoryTempC float64
+	PowerW      float64
+	PowerLimitW float64
+	// Throttle names what is holding the card's clocks down now:
+	// sw_power_cap, hw_slowdown, sw_thermal, hw_thermal, power_brake. Empty
+	// is running free. Idle and application clock settings are left out;
+	// they are not the card being held back.
+	Throttle []string
+}
+
+// sensorEvery bounds how often nvidia-smi is run. Heartbeats come more often
+// than temperatures change, and each run costs a few hundred milliseconds.
+const sensorEvery = 10 * time.Second
+
+// SensorSampler reads Sensors. The first sample has no CPU power, which needs
+// two readings of the energy counters.
+type SensorSampler struct {
+	sys string // /sys, or a test's copy of it
+	smi func(ctx context.Context, fields string) ([]byte, error)
+	now func() time.Time
+
+	mu         sync.Mutex
+	last       Sensors
+	lastAt     time.Time
+	prevEnergy map[string]uint64
+	prevAt     time.Time
+}
+
+// NewSensorSampler reads this machine.
+func NewSensorSampler() *SensorSampler {
+	return &SensorSampler{sys: "/sys", smi: runSMI, now: time.Now}
+}
+
+func runSMI(ctx context.Context, fields string) ([]byte, error) {
+	ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
+	defer cancel()
+	//nolint:gosec // fixed binary, fields from this file
+	return exec.CommandContext(ctx, "nvidia-smi", "--query-gpu="+fields,
+		"--format=csv,noheader,nounits").Output()
+}
+
+// Sample reads the sensors, or returns the last reading when that is recent.
+func (s *SensorSampler) Sample(ctx context.Context) Sensors {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	now := s.now()
+	if !s.lastAt.IsZero() && now.Sub(s.lastAt) < sensorEvery {
+		return s.last
+	}
+	out := Sensors{}
+	out.CPUTempC, out.InletTempC = s.hwmon()
+	out.CPUPowerW = s.rapl(now)
+	out.GPUs = s.gpus(ctx)
+	s.last, s.lastAt = out, now
+	return out
+}
+
+// hwmon reads the hottest CPU sensor and an inlet sensor, from every hwmon
+// device the kernel exposes — what lm_sensors reads, without it installed.
+func (s *SensorSampler) hwmon() (cpu, inlet float64) {
+	devs, _ := filepath.Glob(filepath.Join(s.sys, "class/hwmon/hwmon*"))
+	var pkg, core float64
+	for _, d := range devs {
+		name := readTrim(filepath.Join(d, "name"))
+		inputs, _ := filepath.Glob(filepath.Join(d, "temp*_input"))
+		for _, in := range inputs {
+			milli, err := strconv.ParseFloat(readTrim(in), 64)
+			if err != nil || milli <= 0 {
+				continue
+			}
+			c := milli / 1000
+			label := strings.ToLower(readTrim(strings.TrimSuffix(in, "_input") + "_label"))
+			switch {
+			case isCPUSensor(name) && (strings.HasPrefix(label, "package") || label == "tctl" || label == "tdie"):
+				pkg = max(pkg, c)
+			case isCPUSensor(name):
+				core = max(core, c)
+			case label == "cpu":
+				// A board's own reading of the CPU, as Dell's SMM gives.
+				core = max(core, c)
+			case strings.Contains(label, "inlet") || strings.Contains(label, "ambient") ||
+				strings.Contains(label, "intake") || label == "systin":
+				inlet = max(inlet, c)
+			}
+		}
+	}
+	if pkg > 0 {
+		return pkg, inlet
+	}
+	return core, inlet
+}
+
+func isCPUSensor(name string) bool {
+	switch name {
+	case "coretemp", "k10temp", "zenpower", "cpu_thermal":
+		return true
+	}
+	return false
+}
+
+// rapl is the CPU packages' power from their energy counters. Package
+// domains only: the dram and core subdomains are parts of a package, and
+// adding them would count the same energy twice.
+func (s *SensorSampler) rapl(now time.Time) float64 {
+	doms, _ := filepath.Glob(filepath.Join(s.sys, "class/powercap/intel-rapl:*"))
+	energy := map[string]uint64{}
+	wrap := map[string]uint64{}
+	for _, d := range doms {
+		if strings.Count(filepath.Base(d), ":") != 1 {
+			continue // intel-rapl:0:0 is a subdomain of intel-rapl:0
+		}
+		e, err := strconv.ParseUint(readTrim(filepath.Join(d, "energy_uj")), 10, 64)
+		if err != nil {
+			continue // unreadable without root on newer kernels
+		}
+		energy[d] = e
+		wrap[d], _ = strconv.ParseUint(readTrim(filepath.Join(d, "max_energy_range_uj")), 10, 64)
+	}
+	prev, prevAt := s.prevEnergy, s.prevAt
+	s.prevEnergy, s.prevAt = energy, now
+	dt := now.Sub(prevAt).Seconds()
+	if prev == nil || dt <= 0 || len(energy) == 0 {
+		return 0
+	}
+	var joules float64
+	for d, e := range energy {
+		p, ok := prev[d]
+		if !ok {
+			return 0 // a domain appeared: no fair difference this time
+		}
+		if e < p { // the counter wrapped
+			e += wrap[d]
+		}
+		joules += float64(e-p) / 1e6
+	}
+	return joules / dt
+}
+
+// The throttle reasons nvidia-smi reports, as bits. Only those that mean the
+// card is being held back are named.
+var throttleBits = []struct {
+	bit  uint64
+	name string
+}{
+	{0x4, "sw_power_cap"},
+	{0x8, "hw_slowdown"},
+	{0x20, "sw_thermal"},
+	{0x40, "hw_thermal"},
+	{0x80, "power_brake"},
+}
+
+// gpus asks nvidia-smi for each card's heat and power. The memory temperature
+// is not known to every driver, and a field nvidia-smi does not know fails the
+// whole query, so it is asked for first and dropped on failure.
+func (s *SensorSampler) gpus(ctx context.Context) []GPUSensor {
+	withMem := "index,temperature.gpu,power.draw,enforced.power.limit,clocks_throttle_reasons.active,temperature.memory"
+	out, err := s.smi(ctx, withMem)
+	if err != nil {
+		if out, err = s.smi(ctx, strings.TrimSuffix(withMem, ",temperature.memory")); err != nil {
+			return nil
+		}
+	}
+	var gs []GPUSensor
+	for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
+		f := strings.Split(line, ",")
+		if len(f) < 5 {
+			continue
+		}
+		for i := range f {
+			f[i] = strings.TrimSpace(f[i])
+		}
+		idx, err := strconv.Atoi(f[0])
+		if err != nil {
+			continue
+		}
+		g := GPUSensor{Index: idx, TempC: num(f[1]), PowerW: num(f[2]), PowerLimitW: num(f[3])}
+		if bits, err := strconv.ParseUint(strings.TrimPrefix(f[4], "0x"), 16, 64); err == nil {
+			for _, b := range throttleBits {
+				if bits&b.bit != 0 {
+					g.Throttle = append(g.Throttle, b.name)
+				}
+			}
+		}
+		if len(f) > 5 {
+			g.MemoryTempC = num(f[5])
+		}
+		gs = append(gs, g)
+	}
+	return gs
+}
+
+// num reads a figure nvidia-smi may give as "[N/A]" or "N/A".
+func num(s string) float64 {
+	v, err := strconv.ParseFloat(s, 64)
+	if err != nil || v < 0 {
+		return 0
+	}
+	return v
+}
+
+func readTrim(path string) string {
+	b, err := os.ReadFile(path) //nolint:gosec // sysfs
+	if err != nil {
+		return ""
+	}
+	return strings.TrimSpace(string(b))
+}

internal/agent/discovery/sensors_test.go

@@ -0,0 +1,169 @@
+package discovery
+
+import (
+	"context"
+	"errors"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+	"time"
+)
+
+func put(t *testing.T, root, rel, body string) {
+	t.Helper()
+	p := filepath.Join(root, rel)
+	if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
+		t.Fatal(err)
+	}
+	if err := os.WriteFile(p, []byte(body+"\n"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+}
+
+// dellHost is the layout of the Proxmox host these tests were written
+// against: Dell's SMM with a CPU and SODIMM sensors, coretemp with a package
+// and cores, and RAPL with a package and its dram subdomain.
+func dellHost(t *testing.T) string {
+	root := t.TempDir()
+	put(t, root, "class/hwmon/hwmon0/name", "dell_smm")
+	put(t, root, "class/hwmon/hwmon0/temp1_input", "55000")
+	put(t, root, "class/hwmon/hwmon0/temp1_label", "CPU")
+	put(t, root, "class/hwmon/hwmon0/temp2_input", "47000")
+	put(t, root, "class/hwmon/hwmon0/temp2_label", "SODIMM")
+	put(t, root, "class/hwmon/hwmon1/name", "coretemp")
+	put(t, root, "class/hwmon/hwmon1/temp1_input", "58000")
+	put(t, root, "class/hwmon/hwmon1/temp1_label", "Package id 0")
+	put(t, root, "class/hwmon/hwmon1/temp10_input", "61000")
+	put(t, root, "class/hwmon/hwmon1/temp10_label", "Core 8")
+	put(t, root, "class/powercap/intel-rapl:0/energy_uj", "1000000000")
+	put(t, root, "class/powercap/intel-rapl:0/max_energy_range_uj", "262143328850")
+	put(t, root, "class/powercap/intel-rapl:0:0/energy_uj", "5000000")
+	return root
+}
+
+func sampler(sys string, smi func(context.Context, string) ([]byte, error), clock *time.Time) *SensorSampler {
+	return &SensorSampler{sys: sys, smi: smi, now: func() time.Time { return *clock }}
+}
+
+func noSMI(context.Context, string) ([]byte, error) { return nil, errors.New("not found") }
+
+func TestAHostsCPUHeatAndPowerAreReadAsLmSensorsWould(t *testing.T) {
+	root := dellHost(t)
+	now := time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC)
+	s := sampler(root, noSMI, &now)
+
+	first := s.Sample(t.Context())
+	// The package reading, not the hotter core: the package is what the
+	// chip itself reports as its temperature.
+	if first.CPUTempC != 58 {
+		t.Errorf("CPU %.1f °C, expected the package's 58", first.CPUTempC)
+	}
+	if first.CPUPowerW != 0 {
+		t.Error("power from one reading of a counter is invented")
+	}
+	if first.InletTempC != 0 || first.GPUs != nil {
+		t.Errorf("sensors this host does not have were reported: %+v", first)
+	}
+
+	// 10 s later the package has used 650 J, and its dram subdomain some of
+	// that — counted once.
+	put(t, root, "class/powercap/intel-rapl:0/energy_uj", "1650000000")
+	put(t, root, "class/powercap/intel-rapl:0:0/energy_uj", "99000000")
+	now = now.Add(10 * time.Second)
+	if got := s.Sample(t.Context()).CPUPowerW; got < 64.9 || got > 65.1 {
+		t.Errorf("CPU power %.2f W, expected 65", got)
+	}
+}
+
+func TestACounterThatWrapsIsNotANegativeDraw(t *testing.T) {
+	root := dellHost(t)
+	now := time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC)
+	put(t, root, "class/powercap/intel-rapl:0/energy_uj", "262143000000")
+	s := sampler(root, noSMI, &now)
+	s.Sample(t.Context())
+	// 328,850 µJ to the wrap, then 499,671,150 past it: 500 J in 10 s.
+	put(t, root, "class/powercap/intel-rapl:0/energy_uj", "499671150")
+	now = now.Add(10 * time.Second)
+	if got := s.Sample(t.Context()).CPUPowerW; got < 49.9 || got > 50.1 {
+		t.Errorf("across a wrap: %.2f W, expected 50", got)
+	}
+}
+
+func TestAVMHasNoSensorsAndSaysNothing(t *testing.T) {
+	root := t.TempDir()
+	put(t, root, "class/thermal/cooling_device0/type", "Processor")
+	now := time.Now()
+	got := sampler(root, noSMI, &now).Sample(t.Context())
+	if got.CPUTempC != 0 || got.CPUPowerW != 0 || got.InletTempC != 0 || got.GPUs != nil {
+		t.Errorf("a machine with no sensors reported %+v", got)
+	}
+}
+
+func TestABoardsInletSensorIsTheRoomNotTheCPU(t *testing.T) {
+	root := t.TempDir()
+	put(t, root, "class/hwmon/hwmon2/name", "nct6798")
+	put(t, root, "class/hwmon/hwmon2/temp1_input", "24500")
+	put(t, root, "class/hwmon/hwmon2/temp1_label", "SYSTIN")
+	put(t, root, "class/hwmon/hwmon3/name", "k10temp")
+	put(t, root, "class/hwmon/hwmon3/temp1_input", "71250")
+	put(t, root, "class/hwmon/hwmon3/temp1_label", "Tctl")
+	now := time.Now()
+	got := sampler(root, noSMI, &now).Sample(t.Context())
+	if got.InletTempC != 24.5 || got.CPUTempC != 71.25 {
+		t.Errorf("inlet %.2f, CPU %.2f", got.InletTempC, got.CPUTempC)
+	}
+}
+
+func TestEachGPUSaysHowHotHowHungryAndWhatHoldsItBack(t *testing.T) {
+	var asked []string
+	smi := func(_ context.Context, fields string) ([]byte, error) {
+		asked = append(asked, fields)
+		return []byte(
+			"0, 71, 612.45, 700.00, 0x0000000000000000, 84\n" +
+				"1, 88, 698.10, 700.00, 0x0000000000000024, 95\n" + // sw_power_cap + sw_thermal
+				"2, 45, [N/A], 700.00, 0x0000000000000001, [N/A]\n"), nil // idle only
+	}
+	now := time.Now()
+	ss := sampler(t.TempDir(), smi, &now)
+	got := ss.Sample(t.Context())
+	if len(got.GPUs) != 3 {
+		t.Fatalf("%+v", got.GPUs)
+	}
+	g0, g1, g2 := got.GPUs[0], got.GPUs[1], got.GPUs[2]
+	if g0.TempC != 71 || g0.PowerW != 612.45 || g0.PowerLimitW != 700 || g0.MemoryTempC != 84 || g0.Throttle != nil {
+		t.Errorf("card 0 %+v", g0)
+	}
+	if strings.Join(g1.Throttle, ",") != "sw_power_cap,sw_thermal" {
+		t.Errorf("card 1 throttled by %v", g1.Throttle)
+	}
+	if g2.PowerW != 0 || g2.MemoryTempC != 0 || g2.Throttle != nil {
+		t.Errorf("card 2: N/A is not a reading, and idle is not being held back: %+v", g2)
+	}
+	// Within ten seconds the last reading is reused rather than running
+	// nvidia-smi on every heartbeat; after, it is asked again.
+	now = now.Add(5 * time.Second)
+	ss.Sample(t.Context())
+	if len(asked) != 1 {
+		t.Errorf("nvidia-smi was asked %d times within ten seconds", len(asked))
+	}
+	now = now.Add(6 * time.Second)
+	ss.Sample(t.Context())
+	if len(asked) != 2 {
+		t.Errorf("nvidia-smi was asked %d times after eleven seconds, expected 2", len(asked))
+	}
+}
+
+func TestADriverThatDoesNotKnowMemoryTemperatureStillReportsTheRest(t *testing.T) {
+	smi := func(_ context.Context, fields string) ([]byte, error) {
+		if strings.Contains(fields, "temperature.memory") {
+			return nil, errors.New(`Field "temperature.memory" is not a valid field to query.`)
+		}
+		return []byte("0, 66, 250.0, 300.0, 0x0000000000000040\n"), nil
+	}
+	now := time.Now()
+	got := sampler(t.TempDir(), smi, &now).Sample(t.Context())
+	if len(got.GPUs) != 1 || got.GPUs[0].TempC != 66 || strings.Join(got.GPUs[0].Throttle, ",") != "hw_thermal" {
+		t.Errorf("%+v", got.GPUs)
+	}
+}

internal/agent/engines/engine.go

@@ -76,6 +76,10 @@ type ShardSpec struct {
 	// and accounting. Given to the engine in its environment rather than on
 	// its command line, which every user on the node can read.
 	APIKey string `json:"api_key,omitempty"`
+	// Threads is the CPU cores placement reserved for a shard with no GPU,
+	// and so what its engine computes on. Zero leaves it to the engine's own
+	// configuration.
+	Threads int `json:"threads,omitempty"`
 
 	// Pool and Fabric are control plane concepts the agent cannot derive. They
 	// are carried so the agent can report them back after a restart; Fabric

internal/agent/engines/llamacpp.go

@@ -133,7 +133,12 @@ func (l *LlamaCpp) Commands(s ShardSpec) ([]Command, error) {
 		args = append(args, "--n-gpu-layers", "999")
 	}
 	if len(s.GPUIndices) == 0 && !hasFlag(p.Args, "--threads", "-t") {
-		n := l.cfg.Threads
+		// What placement reserved first: it is what the node's other
+		// replicas were placed around.
+		n := s.Threads
+		if n <= 0 {
+			n = l.cfg.Threads
+		}
 		if n <= 0 {
 			n = max(numCPU()/2, 1)
 		}

internal/agent/engines/llamacpp_test.go

@@ -294,3 +294,15 @@ func TestACPUShardLeavesHalfTheMachineForAnother(t *testing.T) {
 		t.Errorf("--threads %q on one CPU", v)
 	}
 }
+
+func TestTheCoresPlacementReservedAreTheThreads(t *testing.T) {
+	s := llamaSpec(nil)
+	s.Threads = 3
+	cmds, err := (&LlamaCpp{cfg: LlamaCppConfig{Threads: 8}}).Commands(s)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if v, _ := flagValue(cmds[0].Args, "--threads"); v != "3" {
+		t.Errorf("--threads %q; what placement reserved should win over the node's setting", v)
+	}
+}

internal/cli/ctl.go

@@ -137,7 +137,7 @@ func newCtlCmd() *cobra.Command {
 		// apply to it; it lives here because this is where operators look.
 		newBenchCmd(client, &asJSON),
 		newStatusCmd(client, &asJSON),
-		newModelCmd(client),
+		newModelCmd(client, &asJSON),
 		newScaleCmd(client),
 		newRestartCmd(client),
 		newRetryCmd(client),
@@ -146,6 +146,7 @@ func newCtlCmd() *cobra.Command {
 		newToolCheckCmd(),
 		newWeightsCmd(client, &asJSON),
 		newCapacityCmd(client, &asJSON),
+		newQuotaCmd(client, &asJSON),
 		newUsageCmd(client, &asJSON),
 		newCertCmd(),
 	)
@@ -253,6 +254,8 @@ func modelStatus(m adminapi.ModelView) string {
 	switch {
 	case m.Parked:
 		return "parked"
+	case m.HeldFor != "":
+		return "gave room to " + m.HeldFor
 	case m.Unplaceable:
 		return "cannot place"
 	case m.Min == 0 && m.Total == 0:
@@ -323,17 +326,27 @@ func newModelsCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 			}
 
 			w := tab()
-			fmt.Fprintln(w, "MODEL\tSERVING\tREPLICAS\tSTATUS\tALIASES\tPROFILES\tFALLBACKS")
+			fmt.Fprintln(w, "MODEL\tSERVING\tREPLICAS\tSTATUS\tCLASS\tRESERVES\tALIASES\tPROFILES\tFALLBACKS")
+			var warnings []string
 			for _, m := range models {
 				serving := "no"
 				if m.Ready > 0 {
 					serving = "yes"
 				}
-				fmt.Fprintf(w, "%s\t%s\t%d/%d\t%s\t%s\t%s\t%s\n",
-					m.Name, serving, m.Ready, m.Min, modelStatus(m),
+				fmt.Fprintf(w, "%s\t%s\t%d/%d\t%s\t%s\t%s\t%s\t%s\t%s\n",
+					m.Name, serving, m.Ready, m.Min, modelStatus(m), m.Class, reserves(m),
 					join(m.Aliases), join(m.Profiles), join(m.Fallbacks))
+				for _, x := range m.Warnings {
+					warnings = append(warnings, m.Name+": "+x)
+				}
 			}
-			return w.Flush()
+			if err := w.Flush(); err != nil {
+				return err
+			}
+			for _, x := range warnings {
+				fmt.Println("warning:", x)
+			}
+			return nil
 		},
 	}
 }
@@ -511,3 +524,22 @@ func dash(s string) string {
 	}
 	return s
 }
+
+// reserves is what one replica of a model holds of a node, with what it was
+// measured using beside it: "6G 2c (5.1G used)".
+func reserves(m adminapi.ModelView) string {
+	if m.MemoryGB == 0 && m.CPUCores == 0 {
+		return "-"
+	}
+	out := fmt.Sprintf("%dG", m.MemoryGB)
+	if m.CPUCores > 0 {
+		out += fmt.Sprintf(" %dc", m.CPUCores)
+	}
+	if m.MeasuredMemoryGB > 0 {
+		out += fmt.Sprintf(" (%.1fG used)", m.MeasuredMemoryGB)
+	}
+	if len(m.Warnings) > 0 {
+		out += " !"
+	}
+	return out
+}

internal/cli/ctl_upgrade.go

@@ -172,6 +172,10 @@ func upgradeNodes(ctx context.Context, out io.Writer, client func() *ctlClient,
 		}
 	}
 	fmt.Fprintf(out, "%d node(s) upgraded to %s, %d already on it\n", done, v, skipped)
+	// The one machine this does not reach. Said every time, because it is
+	// exactly the kind of thing that stays three versions behind unnoticed.
+	fmt.Fprintf(out, "The model mirror's host is not upgraded by this; on it, run:\n"+
+		"  sudo llmgw upgrade %s --service llmgw-mirror\n", v)
 	return nil
 }
 

internal/cli/model.go

@@ -24,12 +24,14 @@ import (
 // minutes of outage. For a fixed production set that is a reasonable trade; for
 // a platform whose promise is that people can try things, it is the difference
 // between trying one and not bothering.
-func newModelCmd(client func() *ctlClient) *cobra.Command {
+func newModelCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 	c := &cobra.Command{
 		Use:   "model",
 		Short: "Add or remove models while the cluster is running",
 	}
 	c.AddCommand(newModelAddCmd(client), newModelRemoveCmd(client), newModelPlanCmd(client),
+		newResourcesCmd(client), newClassCmd(client), newOwnerCmd(client),
+		newTargetCmd(client, asJSON),
 		newModelServeCmd(client))
 	return c
 }
@@ -192,3 +194,73 @@ func newScaleCmd(client func() *ctlClient) *cobra.Command {
 		"how long an on-demand model stays up unasked-for; zero takes the cluster default")
 	return c
 }
+
+func newResourcesCmd(client func() *ctlClient) *cobra.Command {
+	var memory, cores int
+	var profile string
+	c := &cobra.Command{
+		Use:   "resources <model> --memory GB [--cores N]",
+		Short: "Set what a model reserves of a node: memory, and CPU cores",
+		Long: "--memory is the host RAM a replica needs: for a model with no GPU, its\n" +
+			"weights, KV cache and engine together. Placement reserves it on the node,\n" +
+			"so a node holds only as many replicas as it has memory for.\n\n" +
+			"--cores is the CPU cores a replica with no GPU computes on. They are\n" +
+			"reserved the same way, and become the engine's thread count, so two\n" +
+			"replicas on one node stop competing for the same cores. Zero reserves\n" +
+			"none and leaves the thread count to the node.\n\n" +
+			"Running replicas keep what they were started with; `llmgw ctl restart\n" +
+			"--model M` moves them to the new figures.",
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			if !cmd.Flags().Changed("memory") {
+				return fmt.Errorf("--memory is required")
+			}
+			body, err := json.Marshal(adminapi.ResourcesRequest{Profile: profile, MemoryGB: memory, CPUCores: cores})
+			if err != nil {
+				return err
+			}
+			var m catalog.Model
+			if err := client().post(cmd.Context(), "/api/v1/models/"+args[0]+"/resources", body, &m); err != nil {
+				return err
+			}
+			for _, p := range m.Profiles {
+				fmt.Fprintf(cmd.OutOrStdout(), "%s/%s: %d GB memory, %d cores\n", m.Name, p.Name, p.MinMemoryGB, p.CPUCores)
+			}
+			fmt.Fprintf(cmd.OutOrStdout(), "Running replicas keep their old figures until restarted: llmgw ctl restart --model %s\n", m.Name)
+			return nil
+		},
+	}
+	c.Flags().IntVar(&memory, "memory", 0, "host RAM a replica needs, in GB")
+	c.Flags().IntVar(&cores, "cores", 0, "CPU cores a replica with no GPU computes on; 0 reserves none")
+	c.Flags().StringVar(&profile, "profile", "", "one profile; every profile when left out")
+	return c
+}
+
+func newClassCmd(client func() *ctlClient) *cobra.Command {
+	return &cobra.Command{
+		Use:   "class <model> critical|standard|opportunistic|default",
+		Short: "Set how much a model's capacity is worth against another's",
+		Long: "When a model cannot be placed, replicas of a lower class are stopped to\n" +
+			"make room for it:\n\n" +
+			"  critical       never gives up the replicas of its minimum\n" +
+			"  standard       gives way to critical\n" +
+			"  opportunistic  gives way to both; the default for an on-demand model\n\n" +
+			"Within a class, a model below its minimum may take what another grew into\n" +
+			"under load, but not its minimum. \"default\" goes back to what the model's\n" +
+			"scaling implies: opportunistic when on demand, standard otherwise.",
+		Args: cobra.ExactArgs(2),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			class := args[1]
+			if class == "default" {
+				class = ""
+			}
+			body, _ := json.Marshal(map[string]string{"class": class})
+			var res map[string]string
+			if err := client().post(cmd.Context(), "/api/v1/models/"+args[0]+"/class", body, &res); err != nil {
+				return err
+			}
+			fmt.Fprintf(cmd.OutOrStdout(), "%s is %s\n", res["model"], res["class"])
+			return nil
+		},
+	}
+}

internal/cli/model_serve.go

@@ -75,6 +75,7 @@ func newModelServeCmd(client func() *ctlClient) *cobra.Command {
 	f.IntVar(&req.Replicas, "replicas", 1, "how many replicas to run")
 	f.BoolVar(&onDemand, "on-demand", false, "run none until first asked for")
 	f.IntVar(&req.CtxPool, "ctx-pool", 16384, "llama.cpp's --ctx-size, shared by its parallel slots")
+	f.IntVar(&req.CPUCores, "cores", 0, "CPU cores a replica computes on; 0 takes half the smallest node it fits")
 	f.IntVar(&req.ContextLength, "context-length", 4096, "what one request may use")
 	return c
 }

internal/cli/quota.go

@@ -0,0 +1,230 @@
+package cli
+
+import (
+	"encoding/json"
+	"fmt"
+	"net/url"
+	"strconv"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+func newQuotaCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
+	c := &cobra.Command{
+		Use:   "quota",
+		Short: "What parts of the tenancy tree are guaranteed of the cluster, and may hold",
+		Long: "A capacity quota belongs to an organization, team or project, and counts the\n" +
+			"GPUs, CPU cores and memory held by replicas of the models it owns (see\n" +
+			"`llmgw ctl model owner`), and of those its teams and projects own.\n\n" +
+			"The guarantee is kept for it: when it holds less and a model of its needs\n" +
+			"room, replicas other scopes hold beyond their own guarantees are stopped to\n" +
+			"make it — whatever their class. What it holds within its guarantee is\n" +
+			"never taken by another scope's model. Beyond it, it borrows: free room is\n" +
+			"used, and given back when asked for.\n\n" +
+			"The limit is the most it may hold, borrowing included; a model that would\n" +
+			"take it past is not given another replica.\n\n" +
+			"Scopes are written org:Acme, team:Acme/ml or project:Acme/ml/training, or\n" +
+			"with IDs. Changes take effect within half a minute.",
+	}
+	c.AddCommand(newQuotaListCmd(client, asJSON), newQuotaSetCmd(client), newQuotaClearCmd(client))
+	return c
+}
+
+func newQuotaListCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
+	return &cobra.Command{
+		Use:   "list",
+		Short: "Every quota, with what its scope holds now",
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			var qs []adminapi.QuotaView
+			if err := client().get(cmd.Context(), "/api/v1/quotas", &qs); err != nil {
+				return err
+			}
+			if *asJSON {
+				return json.NewEncoder(cmd.OutOrStdout()).Encode(qs)
+			}
+			if len(qs) == 0 {
+				fmt.Fprintln(cmd.OutOrStdout(), "No capacity quotas: everything shares the cluster by class alone.")
+				return nil
+			}
+			w := tab()
+			fmt.Fprintln(w, "SCOPE\tNAME\tGPUS\tCORES\tMEMORY GB")
+			for _, q := range qs {
+				fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\n", q.Scope, q.Name,
+					quotaCell(q.Holds.GPUs, q.Guarantee.GPUs, q.Limit.GPUs),
+					quotaCell(q.Holds.CPUCores, q.Guarantee.CPUCores, q.Limit.CPUCores),
+					quotaCell(q.Holds.MemoryGB, q.Guarantee.MemoryGB, q.Limit.MemoryGB))
+			}
+			if err := w.Flush(); err != nil {
+				return err
+			}
+			fmt.Fprintln(cmd.OutOrStdout(), "\nEach cell: holds / guaranteed (at most limit). - is not governed.")
+			return nil
+		},
+	}
+}
+
+// quotaCell is "6 / 8 (≤ 16)": what is held, what is guaranteed, the limit.
+func quotaCell(holds int, guarantee, limit *int) string {
+	if guarantee == nil && limit == nil {
+		return "-"
+	}
+	out := strconv.Itoa(holds) + " / "
+	if guarantee != nil {
+		out += strconv.Itoa(*guarantee)
+	} else {
+		out += "-"
+	}
+	if limit != nil {
+		out += fmt.Sprintf(" (≤ %d)", *limit)
+	}
+	return out
+}
+
+func newQuotaSetCmd(client func() *ctlClient) *cobra.Command {
+	var gpus, cores, mem, lgpus, lcores, lmem int
+	c := &cobra.Command{
+		Use:   "set SCOPE [--gpus N] [--cores N] [--memory GB] [--limit-gpus N] ...",
+		Short: "Set a scope's guarantee and limit, replacing what it had",
+		Args:  cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			f := cmd.Flags()
+			pick := func(name string, v int) *int {
+				if !f.Changed(name) {
+					return nil
+				}
+				return &v
+			}
+			req := adminapi.QuotaRequest{Scope: args[0],
+				Guarantee: store.Resources{GPUs: pick("gpus", gpus), CPUCores: pick("cores", cores), MemoryGB: pick("memory", mem)},
+				Limit:     store.Resources{GPUs: pick("limit-gpus", lgpus), CPUCores: pick("limit-cores", lcores), MemoryGB: pick("limit-memory", lmem)},
+			}
+			body, _ := json.Marshal(req)
+			var res map[string]string
+			if err := client().put(cmd.Context(), "/api/v1/quotas", body, &res); err != nil {
+				return err
+			}
+			fmt.Fprintf(cmd.OutOrStdout(), "%s: set; it takes effect within half a minute\n", res["scope"])
+			return nil
+		},
+	}
+	f := c.Flags()
+	f.IntVar(&gpus, "gpus", 0, "GPUs guaranteed")
+	f.IntVar(&cores, "cores", 0, "CPU cores guaranteed")
+	f.IntVar(&mem, "memory", 0, "memory guaranteed, in GB")
+	f.IntVar(&lgpus, "limit-gpus", 0, "the most GPUs it may hold")
+	f.IntVar(&lcores, "limit-cores", 0, "the most CPU cores it may hold")
+	f.IntVar(&lmem, "limit-memory", 0, "the most memory it may hold, in GB")
+	return c
+}
+
+func newQuotaClearCmd(client func() *ctlClient) *cobra.Command {
+	return &cobra.Command{
+		Use:   "clear SCOPE",
+		Short: "Remove a scope's quota",
+		Args:  cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			var res map[string]string
+			if err := client().delete(cmd.Context(), "/api/v1/quotas/"+url.PathEscape(args[0]), &res); err != nil {
+				return err
+			}
+			fmt.Fprintf(cmd.OutOrStdout(), "%s: quota removed\n", res["removed"])
+			return nil
+		},
+	}
+}
+
+func newOwnerCmd(client func() *ctlClient) *cobra.Command {
+	return &cobra.Command{
+		Use:   "owner <model> SCOPE|shared",
+		Short: "Set whose capacity quota a model counts against",
+		Long: "SCOPE is org:Acme, team:Acme/ml or project:Acme/ml/training. A shared\n" +
+			"model belongs to the cluster and counts against nobody's quota.",
+		Args: cobra.ExactArgs(2),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			owner := args[1]
+			if owner == "shared" {
+				owner = ""
+			}
+			body, _ := json.Marshal(map[string]string{"owner": owner})
+			var res map[string]string
+			if err := client().post(cmd.Context(), "/api/v1/models/"+args[0]+"/owner", body, &res); err != nil {
+				return err
+			}
+			if res["owner"] == "" {
+				fmt.Fprintf(cmd.OutOrStdout(), "%s is shared\n", res["model"])
+			} else {
+				fmt.Fprintf(cmd.OutOrStdout(), "%s counts against %s\n", res["model"], res["owner"])
+			}
+			return nil
+		},
+	}
+}
+
+func newTargetCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
+	var tps float64
+	var apply bool
+	c := &cobra.Command{
+		Use:   "target <model> --tokens-per-sec N [--apply]",
+		Short: "How many replicas a model needs for a throughput, and whether they fit",
+		Long: "Sized from what one replica was measured to serve while meeting its SLO\n" +
+			"(`llmgw ctl bench MODEL --save`), on the profile its replicas run on.\n" +
+			"--apply makes that the number it always runs, if the cluster has room.",
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			if tps <= 0 {
+				return fmt.Errorf("--tokens-per-sec is required")
+			}
+			var v adminapi.TargetView
+			path := "/api/v1/models/" + url.PathEscape(args[0]) + "/target?tokens_per_sec=" +
+				strconv.FormatFloat(tps, 'f', -1, 64)
+			if err := client().get(cmd.Context(), path, &v); err != nil {
+				return err
+			}
+			if *asJSON {
+				return json.NewEncoder(cmd.OutOrStdout()).Encode(v)
+			}
+			out := cmd.OutOrStdout()
+			fmt.Fprintf(out, "%s: %.0f tokens/s needs %d replicas at %.1f each (measured on %s); %d running\n",
+				v.Model, v.TokPerSec, v.Needed, v.PerReplica, v.Profile, v.Current)
+			switch {
+			case v.More == 0:
+				fmt.Fprintln(out, "It already has them.")
+			case v.Fits:
+				fmt.Fprintf(out, "%d more, and there is room for them.\n", v.More)
+			default:
+				fmt.Fprintf(out, "%d more, and there is room for %d:\n", v.More, v.Room)
+				for _, b := range v.Blocked {
+					fmt.Fprintln(out, "  "+b)
+				}
+			}
+			if !apply || v.More == 0 {
+				return nil
+			}
+			if !v.Fits {
+				return fmt.Errorf("not applied: the cluster has no room for them")
+			}
+			var models []adminapi.ModelView
+			if err := client().get(cmd.Context(), "/api/v1/models", &models); err != nil {
+				return err
+			}
+			top := v.Needed
+			for _, m := range models {
+				if m.Name == v.Model {
+					top = max(top, m.Max)
+				}
+			}
+			body, _ := json.Marshal(adminapi.ScaleRequest{Min: v.Needed, Max: top})
+			if err := client().post(cmd.Context(), "/api/v1/models/"+url.PathEscape(v.Model)+"/scale", body, nil); err != nil {
+				return err
+			}
+			fmt.Fprintf(out, "Applied: %s runs %d (up to %d under load).\n", v.Model, v.Needed, top)
+			return nil
+		},
+	}
+	c.Flags().Float64Var(&tps, "tokens-per-sec", 0, "the throughput to size for")
+	c.Flags().BoolVar(&apply, "apply", false, "set it as the number of replicas the model always runs")
+	return c
+}

internal/cli/root.go

@@ -47,6 +47,7 @@ func Root() *cobra.Command {
 		newSimEngineCmd(),
 		newDoctorCmd(),
 		newNodeCmd(),
+		newSelfUpgradeCmd(),
 	)
 	return root
 }

internal/cli/selfupgrade.go

@@ -0,0 +1,87 @@
+package cli
+
+import (
+	"errors"
+	"fmt"
+	"os"
+	"os/exec"
+	"runtime"
+	"strings"
+	"time"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/release"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/version"
+)
+
+// newSelfUpgradeCmd upgrades this machine's llmgw on its own, for the machines
+// `llmgw ctl upgrade` does not reach: a mirror host, above all, whose service
+// runs unprivileged with the filesystem read-only and so cannot replace its own
+// binary the way an agent does.
+func newSelfUpgradeCmd() *cobra.Command {
+	var releases, service string
+	c := &cobra.Command{
+		Use:   "upgrade VERSION [--service UNIT]",
+		Short: "Upgrade this machine's llmgw, and restart the service that runs it",
+		Long: "For a machine the control plane does not upgrade — the model mirror's host.\n" +
+			"The control plane and the nodes are upgraded with `llmgw ctl upgrade`.\n\n" +
+			"Fetches the release for this platform, checks it against the release's\n" +
+			"checksums, keeps the old binary beside the new one, and restarts --service,\n" +
+			"waiting for it to be running again. Run it with sudo.",
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			v := args[0]
+			out := cmd.OutOrStdout()
+			if !release.Tag.MatchString(v) {
+				return fmt.Errorf("%q is not a release version (v1.2.3 or v1.2.3-rc.4)", v)
+			}
+			if version.Version == v {
+				fmt.Fprintf(out, "this machine already runs %s\n", v)
+				return nil
+			}
+			if os.Geteuid() != 0 {
+				return errors.New("replacing this machine's binary needs root: run it with sudo")
+			}
+			self, err := os.Executable()
+			if err != nil {
+				return err
+			}
+			backup, err := release.Install(cmd.Context(), nil, releases, v, runtime.GOOS, runtime.GOARCH, self)
+			if err != nil {
+				return err
+			}
+			fmt.Fprintf(out, "%s -> %s: installed, checksum verified; the old binary is %s\n",
+				version.Version, v, backup)
+			if service == "" {
+				return nil
+			}
+			//nolint:gosec // the unit name is the operator's own flag
+			if b, err := exec.CommandContext(cmd.Context(), "systemctl", "restart", service).CombinedOutput(); err != nil {
+				return fmt.Errorf("restarting %s: %v: %s", service, err, strings.TrimSpace(string(b)))
+			}
+			// Running, and still running a few seconds on: a unit that starts
+			// and dies on a bad flag is "active" for a moment first.
+			deadline := time.Now().Add(30 * time.Second)
+			steady := 0
+			for time.Now().Before(deadline) {
+				//nolint:gosec // as above
+				b, _ := exec.CommandContext(cmd.Context(), "systemctl", "is-active", service).Output()
+				if strings.TrimSpace(string(b)) == "active" {
+					if steady++; steady >= 3 {
+						fmt.Fprintf(out, "%s restarted and running\n", service)
+						return nil
+					}
+				} else {
+					steady = 0
+				}
+				time.Sleep(2 * time.Second)
+			}
+			return fmt.Errorf("%s is not running after the upgrade; see journalctl -u %s, and %s holds the previous binary",
+				service, service, backup)
+		},
+	}
+	c.Flags().StringVar(&releases, "releases", release.DefaultBase, "where release binaries are fetched from")
+	c.Flags().StringVar(&service, "service", "", "the systemd unit to restart afterwards, e.g. llmgw-mirror")
+	return c
+}

internal/cli/server.go

@@ -226,6 +226,8 @@ func newServerCmd() *cobra.Command {
 					// And what each group of the tenancy tree spent, which is
 					// the other half of a quota being reviewable.
 					api.SetUsage(st)
+					// Capacity quotas, set and read over this API.
+					api.SetQuotas(st)
 				}
 				api.Register(adminMux)
 
@@ -483,6 +485,12 @@ func startControlPlane(ctx context.Context, cfg config.Server, cat *catalog.Live
 		if st := h.Store(); st != nil {
 			lead = control.NewLeadership(st, cfg.Control.LeaseTTL, log)
 			ctl.SetLeadership(lead)
+			// The tenancy tree's capacity quotas: what each part is
+			// guaranteed and may hold.
+			ctl.Reconciler().SetQuotas(control.QuotaPolicy(st, log))
+			// And what each model was measured to serve, for scaling and for
+			// choosing between profiles.
+			ctl.Reconciler().SetMeasurements(control.Measurements(st, log))
 			log.Info("competing to be the control plane that places work",
 				"owner", lead.Owner())
 		}

internal/control/adminapi/actor_test.go

@@ -4,6 +4,8 @@ import (
 	"net/http/httptest"
 	"strings"
 	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
 )
 
 func TestAClaimedActorIsOneShortName(t *testing.T) {
@@ -22,3 +24,22 @@ func TestAClaimedActorIsOneShortName(t *testing.T) {
 		}
 	}
 }
+
+func TestAReservationThatPlacementCannotTrustIsSaid(t *testing.T) {
+	cpu := func(mem int) catalog.Model {
+		return catalog.Model{Name: "m", Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp", MinMemoryGB: mem}}}
+	}
+	if w := reservationWarnings(cpu(0), 0); len(w) != 1 || !strings.Contains(w[0], "declares no memory") {
+		t.Errorf("no memory declared: %v", w)
+	}
+	if w := reservationWarnings(cpu(6), 8.2); len(w) != 1 || !strings.Contains(w[0], "uses 8.2 GB and reserves 6") {
+		t.Errorf("under-declared: %v", w)
+	}
+	if w := reservationWarnings(cpu(6), 6.4); len(w) != 0 {
+		t.Errorf("within a tenth is mapped weights, not a wrong figure: %v", w)
+	}
+	gpu := catalog.Model{Name: "g", Profiles: []catalog.Profile{{Name: "tp1", Engine: "vllm", GPUClass: "h100-80g"}}}
+	if w := reservationWarnings(gpu, 3); len(w) != 0 {
+		t.Errorf("a GPU model needs no host memory figure: %v", w)
+	}
+}

internal/control/adminapi/api.go

@@ -42,6 +42,7 @@ type API struct {
 	models   Models
 	capacity Capacity
 	usage    Usage
+	quotas   Quotas
 	ctl      *control.Controller
 }
 
@@ -57,6 +58,28 @@ func (a *API) Register(mux *http.ServeMux) {
 	mux.HandleFunc("POST /api/v1/models", a.addModel)
 	mux.HandleFunc("DELETE /api/v1/models/{name}", a.removeModel)
 	mux.HandleFunc("POST /api/v1/models/{name}/scale", a.scaleModel)
+	mux.HandleFunc("POST /api/v1/models/{name}/resources", a.setResources)
+	a.registerQuotas(mux)
+	mux.HandleFunc("GET /api/v1/models/{name}/target", a.target)
+	mux.HandleFunc("POST /api/v1/models/{name}/class", func(w http.ResponseWriter, r *http.Request) {
+		if a.models == nil {
+			writeError(w, http.StatusNotImplemented, "this control plane has no store")
+			return
+		}
+		var req struct {
+			Class string `json:"class"`
+		}
+		if err := json.NewDecoder(io.LimitReader(r.Body, 1<<12)).Decode(&req); err != nil {
+			writeError(w, http.StatusBadRequest, "the request is not valid JSON: "+err.Error())
+			return
+		}
+		m, err := a.models.SetClass(r.Context(), r.PathValue("name"), req.Class, actorOf(r))
+		if err != nil {
+			writeError(w, http.StatusBadRequest, err.Error())
+			return
+		}
+		writeJSON(w, http.StatusOK, map[string]string{"model": m.Name, "class": m.EffectiveClass()})
+	})
 	mux.HandleFunc("PUT /api/v1/models/{name}/capacity", a.putCapacity)
 	mux.HandleFunc("GET /api/v1/capacity", a.listCapacity)
 	mux.HandleFunc("GET /api/v1/usage", a.usageByScope)
@@ -119,6 +142,8 @@ type NodeView struct {
 	// what decides whether a CPU profile can be placed here, so it belongs
 	// where someone looks when a model will not schedule.
 	MemoryGB int `json:"memory_gb,omitempty"`
+	// CPUCount is the machine's CPUs, 0 when the agent did not report them.
+	CPUCount int `json:"cpu_count,omitempty"`
 	// Engines are the inference engines installed here, as the agent found
 	// them. Absent means the agent is older than the probe and said nothing,
 	// which is not the same as having none — placement treats it that way too,
@@ -224,6 +249,11 @@ type LoadView struct {
 	// MemPercent is used over used plus available.
 	MemPercent float64 `json:"memory_percent"`
 	AgeSec     int64   `json:"age_seconds"`
+	// What the node's sensors say; zero is not measured.
+	CPUTempC   float64               `json:"cpu_temp_c,omitempty"`
+	CPUPowerW  float64               `json:"cpu_power_w,omitempty"`
+	InletTempC float64               `json:"inlet_temp_c,omitempty"`
+	GPUs       []inventory.GPUSensor `json:"gpu_sensors,omitempty"`
 }
 
 // loadFresh is how old a measurement may be and still be shown as current.
@@ -237,7 +267,8 @@ func loadView(l *inventory.Load, now time.Time) *LoadView {
 		CPUPercent: l.CPUPercent, CPUs: l.CPUCount,
 		Load1: l.Load1, Load5: l.Load5, Load15: l.Load15,
 		MemUsed: l.MemUsedBytes, MemTotal: l.MemUsedBytes + l.MemAvailBytes,
-		AgeSec: int64(now.Sub(l.At).Seconds()),
+		AgeSec:   int64(now.Sub(l.At).Seconds()),
+		CPUTempC: l.CPUTempC, CPUPowerW: l.CPUPowerW, InletTempC: l.InletTempC, GPUs: l.GPUs,
 	}
 	if v.MemTotal > 0 {
 		v.MemPercent = float64(v.MemUsed) * 100 / float64(v.MemTotal)
@@ -270,6 +301,21 @@ type ModelView struct {
 	Unplaceable bool     `json:"unplaceable,omitempty"`
 	Fallbacks   []string `json:"fallbacks,omitempty"`
 	Profiles    []string `json:"profiles"`
+	// MemoryGB and CPUCores are what one replica reserves, from its first
+	// profile; MeasuredMemoryGB is the most one of its replicas uses now, 0
+	// when none is measured.
+	MemoryGB         int     `json:"memory_gb,omitempty"`
+	CPUCores         int     `json:"cpu_cores,omitempty"`
+	MeasuredMemoryGB float64 `json:"measured_memory_gb,omitempty"`
+	// Warnings are what makes its reservation wrong: nothing declared where
+	// placement needs a figure, or less declared than it uses.
+	Warnings []string `json:"warnings,omitempty"`
+	// Class is its capacity class; Preempted the replicas it lost to a higher
+	// class; HeldFor the model it is giving room to now, if any.
+	Class     string `json:"class"`
+	Owner     string `json:"owner,omitempty"`
+	Preempted int    `json:"preempted,omitempty"`
+	HeldFor   string `json:"held_for,omitempty"`
 }
 
 // ReplicaView is a replica as `ctl replicas` shows it.
@@ -360,6 +406,8 @@ type HeadroomView struct {
 	Limited bool `json:"limited,omitempty"`
 	// Blocked says why one more does not fit, a line per node considered.
 	Blocked []string `json:"blocked,omitempty"`
+	// TokPerSec is what one more replica was measured to serve; 0 unmeasured.
+	TokPerSec float64 `json:"tok_per_sec,omitempty"`
 }
 
 // RejectionView is one rejected candidate.
@@ -396,8 +444,8 @@ func (a *API) Nodes() []NodeView {
 			ID: n.ID, Hostname: n.Hostname, MgmtAddr: n.MgmtAddr,
 			State: n.State.String(), Labels: n.Labels,
 			GPUTotal: len(n.GPUs), GPUFree: len(inv.FreeGPUs(n.ID)),
-			MemoryGB: n.MemoryTotalMB / 1024,
-			Fabrics:  n.FabricNames(), AgentVersion: n.AgentVersion, Features: n.Features,
+			MemoryGB: n.MemoryTotalMB / 1024, CPUCount: n.CPUCount,
+			Fabrics: n.FabricNames(), AgentVersion: n.AgentVersion, Features: n.Features,
 			Replicas: replicasOn[n.ID],
 		}
 		if free, total := n.LocalDisk(); total > 0 {
@@ -497,6 +545,15 @@ func (a *API) Models() []ModelView {
 	for _, h := range a.ctl.Reconciler().ModelHealth() {
 		health[h.Model] = h
 	}
+	// The most one replica of each model uses, summed over its shards.
+	measured := map[string]int64{}
+	for _, rep := range a.Replicas("") {
+		var b int64
+		for _, sh := range rep.Shards {
+			b += sh.MemoryBytes
+		}
+		measured[rep.Model] = max(measured[rep.Model], b)
+	}
 	out := make([]ModelView, 0)
 	for _, m := range a.ctl.Catalog().Models() {
 		h := health[m.Name]
@@ -507,10 +564,16 @@ func (a *API) Models() []ModelView {
 			Fallbacks: m.Fallbacks,
 			Desired:   h.Desired, Failures: h.Failures,
 			Parked: h.Parked, Reason: h.Reason, Unplaceable: h.Unplaceable,
+			Class: m.EffectiveClass(), Owner: m.Owner, Preempted: h.Preempted, HeldFor: h.HeldFor,
 		}
 		for _, p := range m.Profiles {
 			v.Profiles = append(v.Profiles, p.Name)
 		}
+		if len(m.Profiles) > 0 {
+			v.MemoryGB, v.CPUCores = m.Profiles[0].MinMemoryGB, m.Profiles[0].CPUCores
+		}
+		v.MeasuredMemoryGB = float64(measured[m.Name]) / (1 << 30)
+		v.Warnings = reservationWarnings(m, v.MeasuredMemoryGB)
 		out = append(out, v)
 	}
 	return out
@@ -581,6 +644,17 @@ type Models interface {
 	Add(ctx context.Context, m catalog.Model, by string) error
 	Remove(ctx context.Context, name string) error
 	Scale(ctx context.Context, name string, s catalog.Scaling, by string) (catalog.Model, error)
+	Resources(ctx context.Context, name, profile string, memoryGB, cpuCores int, by string) (catalog.Model, error)
+	SetClass(ctx context.Context, name, class, by string) (catalog.Model, error)
+	SetOwner(ctx context.Context, name, owner, by string) (catalog.Model, error)
+}
+
+// ResourcesRequest sets what a model's profiles reserve of a node.
+type ResourcesRequest struct {
+	// Profile is one profile's name; empty is every profile.
+	Profile  string `json:"profile,omitempty"`
+	MemoryGB int    `json:"memory_gb"`
+	CPUCores int    `json:"cpu_cores"`
 }
 
 // Usage reports what each group of the tenancy tree spent. The store
@@ -826,6 +900,26 @@ func (a *API) scaleModel(w http.ResponseWriter, r *http.Request) {
 	})
 }
 
+// setResources sets a model's reserved memory and cores. Placed replicas keep
+// what they were started with until they are restarted.
+func (a *API) setResources(w http.ResponseWriter, r *http.Request) {
+	if a.models == nil {
+		writeError(w, http.StatusNotImplemented, "this control plane has no store")
+		return
+	}
+	var req ResourcesRequest
+	if err := json.NewDecoder(io.LimitReader(r.Body, 1<<16)).Decode(&req); err != nil {
+		writeError(w, http.StatusBadRequest, "the request is not valid JSON: "+err.Error())
+		return
+	}
+	m, err := a.models.Resources(r.Context(), r.PathValue("name"), req.Profile, req.MemoryGB, req.CPUCores, actorOf(r))
+	if err != nil {
+		writeError(w, http.StatusBadRequest, err.Error())
+		return
+	}
+	writeJSON(w, http.StatusOK, m)
+}
+
 func (a *API) removeModel(w http.ResponseWriter, r *http.Request) {
 	if a.models == nil {
 		writeError(w, http.StatusNotImplemented, "this control plane has no store")
@@ -873,7 +967,7 @@ func (a *API) Headroom(model string, limit int) (HeadroomView, bool) {
 		return HeadroomView{}, false
 	}
 	return HeadroomView{Model: model, Current: h.Current, More: h.More, Nodes: h.Nodes,
-		Limited: h.Limited, Blocked: h.Blocked}, true
+		Limited: h.Limited, Blocked: h.Blocked, TokPerSec: h.TokPerSec}, true
 }
 
 func (a *API) Explain(model string) ExplainView {
@@ -1307,3 +1401,40 @@ func actorOf(r *http.Request) string {
 	}
 	return b.String()
 }
+
+// NodeCPUs is the CPU counts of ready nodes with at least this much memory:
+// where a replica needing it could go.
+func (a *API) NodeCPUs(minMemoryGB int) []int {
+	var out []int
+	for _, n := range a.ctl.Inventory().Nodes() {
+		if n.State.String() != "ready" || n.CPUCount == 0 {
+			continue
+		}
+		if n.MemoryTotalMB > 0 && n.MemoryTotalMB/1024 < minMemoryGB {
+			continue
+		}
+		out = append(out, n.CPUCount)
+	}
+	return out
+}
+
+// reservationWarnings says what is wrong with what a model reserves.
+//
+// Resident memory includes the weights mapped from disk, which the kernel can
+// take back, so it overstates what a model needs by the part of the weights
+// not in use. A tenth over is let pass for that; more is a figure too low for
+// placement to be trusted with.
+func reservationWarnings(m catalog.Model, measuredGB float64) []string {
+	var out []string
+	for _, p := range m.Profiles {
+		if p.CPUOnly() && p.MinMemoryGB <= 0 {
+			out = append(out, fmt.Sprintf("profile %s declares no memory: placement cannot tell when a node is full "+
+				"(llmgw ctl model resources %s --memory GB)", p.Name, m.Name))
+		}
+	}
+	if len(m.Profiles) == 1 && m.Profiles[0].MinMemoryGB > 0 && measuredGB > float64(m.Profiles[0].MinMemoryGB)*1.1 {
+		out = append(out, fmt.Sprintf("uses %.1f GB and reserves %d: raise it, or nodes will be filled past their memory",
+			measuredGB, m.Profiles[0].MinMemoryGB))
+	}
+	return out
+}

internal/control/adminapi/api_test.go

@@ -258,6 +258,21 @@ type fakeModels struct {
 	err     error
 }
 
+func (f *fakeModels) SetOwner(_ context.Context, name, owner, _ string) (catalog.Model, error) {
+	return catalog.Model{Name: name, Owner: owner}, f.err
+}
+
+func (f *fakeModels) SetClass(_ context.Context, name, class, _ string) (catalog.Model, error) {
+	return catalog.Model{Name: name, Class: class}, f.err
+}
+
+func (f *fakeModels) Resources(_ context.Context, name, profile string, mem, cores int, _ string) (catalog.Model, error) {
+	if f.err != nil {
+		return catalog.Model{}, f.err
+	}
+	return catalog.Model{Name: name, Profiles: []catalog.Profile{{Name: "p", MinMemoryGB: mem, CPUCores: cores}}}, nil
+}
+
 func (f *fakeModels) Scale(_ context.Context, name string, s catalog.Scaling, by string) (catalog.Model, error) {
 	if f.err != nil {
 		return catalog.Model{}, f.err

internal/control/adminapi/quotas.go

@@ -0,0 +1,237 @@
+package adminapi
+
+import (
+	"context"
+	"encoding/json"
+	"fmt"
+	"io"
+	"net/http"
+	"strings"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// Quotas is where capacity quotas and the tenancy tree they name are kept.
+// The store implements it.
+type Quotas interface {
+	CapacityQuotas(ctx context.Context) ([]store.CapacityQuota, error)
+	PutCapacityQuota(ctx context.Context, q store.CapacityQuota) error
+	DeleteCapacityQuota(ctx context.Context, scope store.Scope, id string) error
+	ScopeParents(ctx context.Context) (map[string][]string, error)
+	AllOrgs(ctx context.Context) ([]store.Org, error)
+	AllTeams(ctx context.Context) ([]store.TeamPath, error)
+	AllProjects(ctx context.Context) ([]store.ProjectPath, error)
+}
+
+// SetQuotas supplies the quota store, which mounts the quota endpoints'
+// behaviour; without it they answer 501.
+func (a *API) SetQuotas(q Quotas) { a.quotas = q }
+
+// QuotaView is one scope's quota with what it holds now.
+type QuotaView struct {
+	// Scope is team:ID; Name is how people know it, "Acme / ml".
+	Scope     string          `json:"scope"`
+	Name      string          `json:"name"`
+	Guarantee store.Resources `json:"guarantee"`
+	Limit     store.Resources `json:"limit"`
+	Holds     quota.Amount    `json:"holds"`
+	UpdatedBy string          `json:"updated_by,omitempty"`
+	UpdatedAt time.Time       `json:"updated_at"`
+}
+
+// QuotaRequest sets a scope's quota. Scope may be written with an ID or a
+// name: org:Acme, team:Acme/ml, project:Acme/ml/training.
+type QuotaRequest struct {
+	Scope     string          `json:"scope"`
+	Guarantee store.Resources `json:"guarantee"`
+	Limit     store.Resources `json:"limit"`
+}
+
+func (a *API) registerQuotas(mux *http.ServeMux) {
+	mux.HandleFunc("GET /api/v1/quotas", func(w http.ResponseWriter, r *http.Request) {
+		if a.quotas == nil {
+			writeError(w, http.StatusNotImplemented, "this control plane has no store")
+			return
+		}
+		views, err := a.QuotaViews(r.Context())
+		if err != nil {
+			writeError(w, http.StatusInternalServerError, err.Error())
+			return
+		}
+		writeJSON(w, http.StatusOK, views)
+	})
+	mux.HandleFunc("PUT /api/v1/quotas", func(w http.ResponseWriter, r *http.Request) {
+		if a.quotas == nil {
+			writeError(w, http.StatusNotImplemented, "this control plane has no store")
+			return
+		}
+		var req QuotaRequest
+		if err := json.NewDecoder(io.LimitReader(r.Body, 1<<14)).Decode(&req); err != nil {
+			writeError(w, http.StatusBadRequest, "the request is not valid JSON: "+err.Error())
+			return
+		}
+		scope, id, _, err := a.resolveScope(r.Context(), req.Scope)
+		if err != nil {
+			writeError(w, http.StatusBadRequest, err.Error())
+			return
+		}
+		if !req.Guarantee.Any() && !req.Limit.Any() {
+			writeError(w, http.StatusBadRequest, "a quota needs a guarantee or a limit; to remove one, DELETE it")
+			return
+		}
+		q := store.CapacityQuota{Scope: scope, ScopeID: id, Guarantee: req.Guarantee, Limit: req.Limit,
+			UpdatedBy: actorOf(r)}
+		if err := a.quotas.PutCapacityQuota(r.Context(), q); err != nil {
+			writeError(w, http.StatusBadRequest, err.Error())
+			return
+		}
+		writeJSON(w, http.StatusOK, map[string]string{"scope": quota.Key(scope, id)})
+	})
+	mux.HandleFunc("DELETE /api/v1/quotas/{scope}", func(w http.ResponseWriter, r *http.Request) {
+		if a.quotas == nil {
+			writeError(w, http.StatusNotImplemented, "this control plane has no store")
+			return
+		}
+		scope, id, _, err := a.resolveScope(r.Context(), r.PathValue("scope"))
+		if err != nil {
+			writeError(w, http.StatusBadRequest, err.Error())
+			return
+		}
+		if err := a.quotas.DeleteCapacityQuota(r.Context(), scope, id); err != nil {
+			writeError(w, http.StatusInternalServerError, err.Error())
+			return
+		}
+		writeJSON(w, http.StatusOK, map[string]string{"removed": quota.Key(scope, id)})
+	})
+	mux.HandleFunc("POST /api/v1/models/{name}/owner", func(w http.ResponseWriter, r *http.Request) {
+		if a.models == nil || a.quotas == nil {
+			writeError(w, http.StatusNotImplemented, "this control plane has no store")
+			return
+		}
+		var req struct {
+			Owner string `json:"owner"`
+		}
+		if err := json.NewDecoder(io.LimitReader(r.Body, 1<<12)).Decode(&req); err != nil {
+			writeError(w, http.StatusBadRequest, "the request is not valid JSON: "+err.Error())
+			return
+		}
+		owner := ""
+		if req.Owner != "" {
+			scope, id, _, err := a.resolveScope(r.Context(), req.Owner)
+			if err != nil {
+				writeError(w, http.StatusBadRequest, err.Error())
+				return
+			}
+			owner = quota.Key(scope, id)
+		}
+		m, err := a.models.SetOwner(r.Context(), r.PathValue("name"), owner, actorOf(r))
+		if err != nil {
+			writeError(w, http.StatusBadRequest, err.Error())
+			return
+		}
+		writeJSON(w, http.StatusOK, map[string]string{"model": m.Name, "owner": m.Owner})
+	})
+}
+
+// QuotaViews lists every quota with what its scope holds now.
+func (a *API) QuotaViews(ctx context.Context) ([]QuotaView, error) {
+	qs, err := a.quotas.CapacityQuotas(ctx)
+	if err != nil {
+		return nil, err
+	}
+	parents, err := a.quotas.ScopeParents(ctx)
+	if err != nil {
+		return nil, err
+	}
+	names, err := a.scopeNames(ctx)
+	if err != nil {
+		return nil, err
+	}
+	usage := quota.New(qs, parents).Usage(a.ctl.Reconciler().Holdings())
+	out := make([]QuotaView, 0, len(qs))
+	for _, q := range qs {
+		key := quota.Key(q.Scope, q.ScopeID)
+		out = append(out, QuotaView{Scope: key, Name: names[key], Guarantee: q.Guarantee, Limit: q.Limit,
+			Holds: usage[key], UpdatedBy: q.UpdatedBy, UpdatedAt: q.UpdatedAt})
+	}
+	return out, nil
+}
+
+// scopeNames maps org:ID, team:ID and project:ID to their paths.
+func (a *API) scopeNames(ctx context.Context) (map[string]string, error) {
+	out := map[string]string{}
+	orgs, err := a.quotas.AllOrgs(ctx)
+	if err != nil {
+		return nil, err
+	}
+	for _, o := range orgs {
+		out["org:"+o.ID] = o.Name
+	}
+	teams, err := a.quotas.AllTeams(ctx)
+	if err != nil {
+		return nil, err
+	}
+	for _, t := range teams {
+		out["team:"+t.ID] = t.OrgName + " / " + t.Name
+	}
+	projects, err := a.quotas.AllProjects(ctx)
+	if err != nil {
+		return nil, err
+	}
+	for _, p := range projects {
+		out["project:"+p.ID] = p.Path
+	}
+	return out, nil
+}
+
+// resolveScope reads org:X, team:X or project:X, where X is an ID or the
+// scope's path — Acme, Acme/ml, Acme/ml/training — and returns its ID and
+// path. A path that could mean two scopes is refused rather than guessed.
+func (a *API) resolveScope(ctx context.Context, s string) (store.Scope, string, string, error) {
+	kind, what, ok := strings.Cut(strings.TrimSpace(s), ":")
+	scope := store.Scope(kind)
+	if !ok || what == "" || (scope != store.ScopeOrg && scope != store.ScopeTeam && scope != store.ScopeProject) {
+		return "", "", "", fmt.Errorf("%q is not org:NAME, team:ORG/TEAM or project:ORG/TEAM/PROJECT", s)
+	}
+	names, err := a.scopeNames(ctx)
+	if err != nil {
+		return "", "", "", err
+	}
+	norm := func(p string) string {
+		parts := strings.Split(p, "/")
+		for i := range parts {
+			parts[i] = strings.ToLower(strings.TrimSpace(parts[i]))
+		}
+		return strings.Join(parts, "/")
+	}
+	if name, ok := names[kind+":"+what]; ok {
+		return scope, what, name, nil
+	}
+	match := func(same func(name string) bool) []string {
+		var found []string
+		for key, name := range names {
+			if strings.HasPrefix(key, kind+":") && same(name) {
+				found = append(found, key)
+			}
+		}
+		return found
+	}
+	found := match(func(name string) bool { return norm(name) == norm(what) })
+	if len(found) == 0 {
+		// Its own name alone — ml rather than Acme/ml — when that is unique.
+		found = match(func(name string) bool {
+			parts := strings.Split(norm(name), "/")
+			return parts[len(parts)-1] == norm(what)
+		})
+	}
+	switch len(found) {
+	case 0:
+		return "", "", "", fmt.Errorf("there is no %s %q", kind, what)
+	case 1:
+		_, id, _ := strings.Cut(found[0], ":")
+		return scope, id, names[found[0]], nil
+	}
+	return "", "", "", fmt.Errorf("%q could be any of %d %ss; give its ID", what, len(found), kind)
+}

internal/control/adminapi/quotas_test.go

@@ -0,0 +1,107 @@
+package adminapi
+
+import (
+	"context"
+	"encoding/json"
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// fakeQuotaStore is Acme with teams ml and web, and Scania with another ml.
+type fakeQuotaStore struct {
+	quotas []store.CapacityQuota
+}
+
+func (f *fakeQuotaStore) CapacityQuotas(context.Context) ([]store.CapacityQuota, error) {
+	return f.quotas, nil
+}
+func (f *fakeQuotaStore) PutCapacityQuota(_ context.Context, q store.CapacityQuota) error {
+	for i := range f.quotas {
+		if f.quotas[i].Scope == q.Scope && f.quotas[i].ScopeID == q.ScopeID {
+			f.quotas[i] = q
+			return nil
+		}
+	}
+	f.quotas = append(f.quotas, q)
+	return nil
+}
+func (f *fakeQuotaStore) DeleteCapacityQuota(_ context.Context, s store.Scope, id string) error {
+	for i, q := range f.quotas {
+		if q.Scope == s && q.ScopeID == id {
+			f.quotas = append(f.quotas[:i], f.quotas[i+1:]...)
+		}
+	}
+	return nil
+}
+func (f *fakeQuotaStore) ScopeParents(context.Context) (map[string][]string, error) {
+	return map[string][]string{"team:t-ml": {"org:o-acme"}, "team:t-web": {"org:o-acme"},
+		"team:t-sml": {"org:o-scania"}}, nil
+}
+func (f *fakeQuotaStore) AllOrgs(context.Context) ([]store.Org, error) {
+	return []store.Org{{ID: "o-acme", Name: "Acme"}, {ID: "o-scania", Name: "Scania"}}, nil
+}
+func (f *fakeQuotaStore) AllTeams(context.Context) ([]store.TeamPath, error) {
+	return []store.TeamPath{{ID: "t-ml", Name: "ml", OrgID: "o-acme", OrgName: "Acme"},
+		{ID: "t-web", Name: "web", OrgID: "o-acme", OrgName: "Acme"},
+		{ID: "t-sml", Name: "ml", OrgID: "o-scania", OrgName: "Scania"}}, nil
+}
+func (f *fakeQuotaStore) AllProjects(context.Context) ([]store.ProjectPath, error) { return nil, nil }
+
+func TestAScopeIsNamedByPathOrIDAndNeverGuessed(t *testing.T) {
+	a, _ := setup(t, testModels())
+	a.SetQuotas(&fakeQuotaStore{})
+	for in, want := range map[string]string{
+		"org:Acme": "o-acme", "org:o-acme": "o-acme", "team:Acme/ml": "t-ml",
+		"team: acme / ML ": "t-ml", "team:Scania/ml": "t-sml", "team:web": "t-web",
+	} {
+		_, id, _, err := a.resolveScope(t.Context(), in)
+		if err != nil || id != want {
+			t.Errorf("%q: %q %v, expected %q", in, id, err, want)
+		}
+	}
+	for _, bad := range []string{"team:ml", "team:Acme/nope", "group:Acme", "Acme", "org:"} {
+		if _, _, _, err := a.resolveScope(t.Context(), bad); err == nil {
+			t.Errorf("%q was resolved", bad)
+		}
+	}
+	if _, _, _, err := a.resolveScope(t.Context(), "team:ml"); err == nil || !strings.Contains(err.Error(), "could be any of 2") {
+		t.Errorf("an ambiguous name should say so: %v", err)
+	}
+}
+
+func TestAQuotaIsSetListedAndCleared(t *testing.T) {
+	a, _ := setup(t, testModels())
+	qs := &fakeQuotaStore{}
+	a.SetQuotas(qs)
+	mux := http.NewServeMux()
+	a.Register(mux)
+	call := func(method, path, body string) *httptest.ResponseRecorder {
+		req := httptest.NewRequest(method, path, strings.NewReader(body))
+		req.Header.Set("Content-Type", "application/json")
+		req.Header.Set("X-Llmgw-Actor", "ops")
+		rec := httptest.NewRecorder()
+		mux.ServeHTTP(rec, req)
+		return rec
+	}
+	if rec := call("PUT", "/api/v1/quotas", `{"scope":"team:Acme/ml","guarantee":{"gpus":8},"limit":{"gpus":16}}`); rec.Code != 200 {
+		t.Fatalf("set: %d %s", rec.Code, rec.Body)
+	}
+	if len(qs.quotas) != 1 || qs.quotas[0].ScopeID != "t-ml" || *qs.quotas[0].Guarantee.GPUs != 8 || qs.quotas[0].UpdatedBy != "ops" {
+		t.Errorf("stored %+v", qs.quotas)
+	}
+	rec := call("GET", "/api/v1/quotas", "")
+	var views []QuotaView
+	if err := json.Unmarshal(rec.Body.Bytes(), &views); err != nil || len(views) != 1 || views[0].Name != "Acme / ml" {
+		t.Errorf("list %d %s", rec.Code, rec.Body)
+	}
+	if rec := call("PUT", "/api/v1/quotas", `{"scope":"team:Acme/ml"}`); rec.Code != 400 {
+		t.Errorf("an empty quota was accepted: %d", rec.Code)
+	}
+	if rec := call("DELETE", "/api/v1/quotas/team:Acme%2Fml", ""); rec.Code != 200 || len(qs.quotas) != 0 {
+		t.Errorf("clear: %d %s %+v", rec.Code, rec.Body, qs.quotas)
+	}
+}

internal/control/adminapi/target.go

@@ -0,0 +1,63 @@
+package adminapi
+
+import (
+	"fmt"
+	"math"
+	"net/http"
+	"strconv"
+)
+
+// TargetView is how many replicas a model needs to serve a throughput, from
+// what one replica was measured to serve, and whether the cluster has room.
+type TargetView struct {
+	Model string `json:"model"`
+	// TokPerSec is what was asked for; PerReplica what one replica was
+	// measured to serve at its SLO, on Profile.
+	TokPerSec  float64 `json:"tok_per_sec"`
+	PerReplica float64 `json:"per_replica_tok_per_sec"`
+	Profile    string  `json:"profile"`
+	// Needed replicas in all; Current hold capacity now; More is how many of
+	// Needed are not running, and Room how many more fit.
+	Needed  int      `json:"needed"`
+	Current int      `json:"current"`
+	More    int      `json:"more"`
+	Room    int      `json:"room"`
+	Fits    bool     `json:"fits"`
+	Blocked []string `json:"blocked,omitempty"`
+}
+
+// Target sizes a model for a throughput.
+func (a *API) Target(model string, tokPerSec float64) (TargetView, error) {
+	meas, profile, ok := a.ctl.Reconciler().MeasuredFor(model)
+	if !ok || meas.TokPerSec <= 0 {
+		return TargetView{}, fmt.Errorf("%s has not been measured; run `llmgw ctl bench %s --save` first", model, model)
+	}
+	v := TargetView{Model: model, TokPerSec: tokPerSec, PerReplica: meas.TokPerSec, Profile: profile,
+		Needed: max(int(math.Ceil(tokPerSec/meas.TokPerSec)), 1)}
+	h, ok := a.Headroom(model, v.Needed)
+	if !ok {
+		return TargetView{}, fmt.Errorf("no model %q", model)
+	}
+	v.Current = h.Current
+	v.More = max(v.Needed-h.Current, 0)
+	v.Room = h.More
+	v.Fits = v.More <= h.More || h.Limited
+	if !v.Fits {
+		v.Blocked = h.Blocked
+	}
+	return v, nil
+}
+
+func (a *API) target(w http.ResponseWriter, r *http.Request) {
+	tps, err := strconv.ParseFloat(r.URL.Query().Get("tokens_per_sec"), 64)
+	if err != nil || tps <= 0 {
+		writeError(w, http.StatusBadRequest, "tokens_per_sec must be a positive number")
+		return
+	}
+	v, err := a.Target(r.PathValue("name"), tps)
+	if err != nil {
+		writeError(w, http.StatusBadRequest, err.Error())
+		return
+	}
+	writeJSON(w, http.StatusOK, v)
+}

internal/control/adminapi/target_test.go

@@ -0,0 +1,30 @@
+package adminapi
+
+import (
+	"strings"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
+)
+
+func TestATargetIsSizedFromTheMeasurementAndTheRoom(t *testing.T) {
+	// Three 8-GPU nodes; one tp8 replica was measured at 500 tokens/s.
+	a, ctl := setup(t, testModels(), gpuNode("1", 8), gpuNode("2", 8), gpuNode("3", 8))
+	if _, err := a.Target("llama", 1000); err == nil || !strings.Contains(err.Error(), "ctl bench llama --save") {
+		t.Errorf("an unmeasured model should say how to measure it: %v", err)
+	}
+	ctl.Reconciler().SetMeasurements(func() map[string]reconciler.Measured {
+		return map[string]reconciler.Measured{"llama/tp8": {MaxConcurrency: 16, TokPerSec: 500}}
+	})
+	v, err := a.Target("llama", 1200)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if v.Needed != 3 || v.PerReplica != 500 || v.Profile != "tp8" || v.More != 3 || !v.Fits {
+		t.Errorf("1200 tok/s at 500 each is 3 replicas on 3 empty nodes: %+v", v)
+	}
+	v, _ = a.Target("llama", 2000)
+	if v.Needed != 4 || v.Fits || v.Room != 3 || len(v.Blocked) == 0 {
+		t.Errorf("a fourth does not fit on three nodes, and should say why: %+v", v)
+	}
+}

internal/control/agentsvc/convert.go

@@ -4,6 +4,7 @@ package agentsvc
 import (
 	"fmt"
 	"maps"
+	"runtime"
 	"time"
 
 	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
@@ -27,6 +28,7 @@ func NodeFromRegister(r *llmgwv1.Register) inventory.Node {
 		EngineVersions: r.GetEngineVersions(),
 		EnginesProbed:  r.GetEnginesProbed(),
 		MemoryTotalMB:  int(r.GetMemoryTotalMb()),
+		CPUCount:       int(r.GetCpuCount()),
 	}
 	if h := r.GetHost(); h != nil {
 		n.Host = discovery.Host{
@@ -90,6 +92,7 @@ func RegisterFromInventory(nodeID, hostname, mgmtAddr, agentVersion string,
 		NodeId: nodeID, Hostname: hostname, MgmtAddr: mgmtAddr,
 		Labels: labels, AgentVersion: agentVersion,
 		MemoryTotalMb: int64(inv.MemoryTotalMB),
+		CpuCount:      int32(runtime.NumCPU()),
 	}
 	for _, w := range inv.Weights {
 		r.Weights = append(r.Weights, &llmgwv1.LocalWeights{
@@ -192,6 +195,7 @@ func StartShardFromCommand(cmd reconciler.ShardCommand) *llmgwv1.StartShard {
 		Fabric:           cmd.Fabric,
 		ProfileFp:        cmd.ProfileFP,
 		ApiKey:           cmd.Spec.APIKey,
+		CpuThreads:       int32(cmd.Spec.Threads),
 	}
 }
 
@@ -213,6 +217,7 @@ func ShardSpecFromStart(m *llmgwv1.StartShard) (engines.ShardSpec, time.Duration
 		Fabric:        m.GetFabric(),
 		ProfileFP:     m.GetProfileFp(),
 		APIKey:        m.GetApiKey(),
+		Threads:       int(m.GetCpuThreads()),
 		Profile: engines.Profile{
 			Name: p.GetName(), Engine: p.GetEngine(), GPUClass: p.GetGpuClass(),
 			TP: int(p.GetTp()), PP: int(p.GetPp()),

internal/control/agentsvc/service.go

@@ -345,8 +345,9 @@ func (s *Service) applyHeartbeat(nodeID string, hb *llmgwv1.Heartbeat) {
 	problems := hb.GetNewProblems()
 	storage := hb.GetStorage()
 	memMB := int(hb.GetMemoryTotalMb())
+	cpus := int(hb.GetCpuCount())
 	if len(used) == 0 && len(problems) == 0 && len(storage) == 0 &&
-		!hb.GetWeightsChanged() && memMB == 0 && !hb.GetEnginesChanged() &&
+		!hb.GetWeightsChanged() && memMB == 0 && cpus == 0 && !hb.GetEnginesChanged() &&
 		!hb.GetProvisionChanged() && hb.GetLoad() == nil && len(hb.GetIbPorts()) == 0 &&
 		!hb.GetFabricsChanged() && len(hb.GetGpuFabric()) == 0 && hb.GetImex() == nil {
 		return
@@ -406,6 +407,12 @@ func (s *Service) applyHeartbeat(nodeID string, hb *llmgwv1.Heartbeat) {
 			"from_mb", n.MemoryTotalMB, "to_mb", memMB)
 		n.MemoryTotalMB = memMB
 	}
+	if cpus > 0 && cpus != n.CPUCount {
+		if n.CPUCount > 0 {
+			s.log.Info("node CPUs changed", "node", nodeID, "from", n.CPUCount, "to", cpus)
+		}
+		n.CPUCount = cpus
+	}
 	if hb.GetEnginesChanged() {
 		vers := maps.Clone(hb.GetEngineVersions())
 		if !maps.Equal(vers, n.EngineVersions) {
@@ -538,6 +545,13 @@ func loadFrom(l *llmgwv1.NodeLoad, at time.Time) *inventory.Load {
 			CPUCores: sh.GetCpuCores(), RSSBytes: sh.GetMemoryRssBytes(),
 		}
 	}
+	out.CPUTempC, out.CPUPowerW, out.InletTempC = l.GetCpuTempC(), l.GetCpuPowerW(), l.GetInletTempC()
+	for _, g := range l.GetGpus() {
+		out.GPUs = append(out.GPUs, inventory.GPUSensor{
+			Index: int(g.GetIndex()), TempC: g.GetTempC(), MemoryTempC: g.GetMemoryTempC(),
+			PowerW: g.GetPowerW(), PowerLimitW: g.GetPowerLimitW(), Throttle: g.GetThrottle(),
+		})
+	}
 	return out
 }
 

internal/control/catalog/catalog.go

@@ -74,6 +74,17 @@ type Profile struct {
 	// the control plane.
 	MinMemoryGB int `json:"min_memory_gb,omitempty" koanf:"min_memory_gb"`
 
+	// CPUCores is the cores a replica of a profile with no GPU is given:
+	// reserved out of its node's CPUs by placement, as memory is, and what
+	// its engine computes on (llama.cpp's --threads).
+	//
+	// Reserved because an engine takes every core it is told about, and two
+	// on one node that each think they have all of them spin against each
+	// other: on an 8-vCPU node, two 7B replicas fell from 10 tokens a second
+	// to under one each. Zero reserves nothing and leaves the thread count to
+	// the node, which is every profile from before this.
+	CPUCores int `json:"cpu_cores,omitempty" koanf:"cpu_cores"`
+
 	// MinDiskGB is the free space a node needs before it can hold these
 	// weights.
 	//
@@ -138,10 +149,19 @@ type Model struct {
 	// choice by guesswork.
 	Description string `json:"description,omitempty" koanf:"description"`
 	// Source is the weights path or a HuggingFace id.
-	Source    string    `json:"source,omitempty" koanf:"source"`
-	Profiles  []Profile `json:"profiles,omitempty" koanf:"profiles"`
-	Scaling   Scaling   `json:"scaling,omitempty" koanf:"scaling"`
-	Fallbacks []string  `json:"fallbacks,omitempty" koanf:"fallbacks"`
+	Source   string    `json:"source,omitempty" koanf:"source"`
+	Profiles []Profile `json:"profiles,omitempty" koanf:"profiles"`
+	Scaling  Scaling   `json:"scaling,omitempty" koanf:"scaling"`
+	// Class is how much this model's capacity is worth against another's,
+	// when there is not enough for both: critical, standard or opportunistic.
+	// Empty is standard, or opportunistic for a model loaded on demand. See
+	// EffectiveClass.
+	Class string `json:"class,omitempty" koanf:"class"`
+	// Owner is the part of the tenancy tree whose capacity quota this model
+	// counts against, written org:ID, team:ID or project:ID. Empty is shared:
+	// held by the cluster, counted against nobody.
+	Owner     string   `json:"owner,omitempty" koanf:"owner"`
+	Fallbacks []string `json:"fallbacks,omitempty" koanf:"fallbacks"`
 	// RoutingProfile selects how the router scores endpoints for this model.
 	RoutingProfile string `json:"routing_profile,omitempty" koanf:"routing_profile"`
 	SLO            SLO    `json:"slo,omitempty" koanf:"slo"`
@@ -288,6 +308,12 @@ func (m Model) validate() error {
 	if len(m.Profiles) == 0 {
 		return fmt.Errorf("model %q: at least one profile is required", m.Name)
 	}
+	if m.Owner != "" && !validOwner(m.Owner) {
+		return fmt.Errorf("model %q: owner %q is not org:ID, team:ID or project:ID", m.Name, m.Owner)
+	}
+	if !validClass(m.Class) {
+		return fmt.Errorf("model %q: class %q is not one of critical, standard, opportunistic", m.Name, m.Class)
+	}
 	if m.Scaling.Max < m.Scaling.Min {
 		return fmt.Errorf("model %q: scaling max %d is below min %d", m.Name, m.Scaling.Max, m.Scaling.Min)
 	}
@@ -380,6 +406,7 @@ func (m Model) PlacementProfiles() []placement.Profile {
 			NodeSelector: p.NodeSelector,
 			MinMemoryGB:  p.MinMemoryGB,
 			MinDiskGB:    p.MinDiskGB,
+			CPUCores:     p.CPUCores,
 			// The model's source, so placement can prefer nodes that already
 			// hold the weights rather than one that must first move them.
 			Source: m.Source,
@@ -475,3 +502,65 @@ func (p Profile) Fingerprint() string {
 	sum := sha256.Sum256(b)
 	return hex.EncodeToString(sum[:8])
 }
+
+// SuggestCores is the cores to give a replica with no GPU on nodes of these
+// CPU counts: half of the smallest, so two replicas share any of them without
+// competing. Zero when no node says, which reserves nothing.
+func SuggestCores(cpuCounts []int) int {
+	smallest := 0
+	for _, c := range cpuCounts {
+		if c > 0 && (smallest == 0 || c < smallest) {
+			smallest = c
+		}
+	}
+	if smallest == 0 {
+		return 0
+	}
+	return max(smallest/2, 1)
+}
+
+// The capacity classes, lowest first.
+const (
+	ClassOpportunistic = "opportunistic"
+	ClassStandard      = "standard"
+	ClassCritical      = "critical"
+)
+
+func validClass(c string) bool {
+	return c == "" || c == ClassOpportunistic || c == ClassStandard || c == ClassCritical
+}
+
+// EffectiveClass is the class a model is treated as: what it says, or, when
+// it says nothing, opportunistic for one loaded on demand — it holds capacity
+// only while asked for, and has always been the first to give it up — and
+// standard for everything else.
+func (m Model) EffectiveClass() string {
+	if m.Class != "" {
+		return m.Class
+	}
+	if m.Scaling.OnDemand() {
+		return ClassOpportunistic
+	}
+	return ClassStandard
+}
+
+// ClassRank orders classes: a higher rank may take capacity from a lower one.
+func ClassRank(c string) int {
+	switch c {
+	case ClassCritical:
+		return 3
+	case ClassStandard, "":
+		return 2
+	default:
+		return 1
+	}
+}
+
+func validOwner(o string) bool {
+	for _, p := range []string{"org:", "team:", "project:"} {
+		if strings.HasPrefix(o, p) && len(o) > len(p) {
+			return true
+		}
+	}
+	return false
+}

internal/control/control.go

@@ -375,3 +375,6 @@ func EngineKeys(secret []byte) func(replicaID string) string {
 		return "llmgw-" + hex.EncodeToString(m.Sum(nil))[:40]
 	}
 }
+
+// QuotaUse is every capacity quota beside what its scope holds.
+func (c *Controller) QuotaUse() []reconciler.QuotaUse { return c.rec.QuotaUse() }

internal/control/fleetmetrics/fleetmetrics.go

@@ -51,12 +51,16 @@ type HealthSource interface {
 type Collector struct {
 	src Source
 
-	modelDesired     *prometheus.Desc
-	modelFailures    *prometheus.Desc
-	modelParked      *prometheus.Desc
-	modelUnplaceable *prometheus.Desc
-	engineInstall    *prometheus.Desc
-	leader           *prometheus.Desc
+	modelDesired      *prometheus.Desc
+	modelFailures     *prometheus.Desc
+	modelParked       *prometheus.Desc
+	modelUnplaceable  *prometheus.Desc
+	modelPreempted    *prometheus.Desc
+	capacityHolds     *prometheus.Desc
+	capacityGuarantee *prometheus.Desc
+	capacityLimit     *prometheus.Desc
+	engineInstall     *prometheus.Desc
+	leader            *prometheus.Desc
 
 	nodeUp       *prometheus.Desc
 	nodeGPUs     *prometheus.Desc
@@ -70,6 +74,13 @@ type Collector struct {
 	gpuMemoryUsed  *prometheus.Desc
 	gpuUtil        *prometheus.Desc
 	gpuTemp        *prometheus.Desc
+	gpuMemTemp     *prometheus.Desc
+	gpuPower       *prometheus.Desc
+	gpuPowerLimit  *prometheus.Desc
+	gpuThrottled   *prometheus.Desc
+	nodeCPUTemp    *prometheus.Desc
+	nodeCPUPower   *prometheus.Desc
+	nodeInletTemp  *prometheus.Desc
 	gpuHealthy     *prometheus.Desc
 	gpuNVLink      *prometheus.Desc
 	gpuECC         *prometheus.Desc
@@ -138,6 +149,21 @@ func New(src Source) *Collector {
 			"How busy a GPU was over the last sample period. Absent when the driver does not "+
 				"report it, which is not the same as idle. Memory used says a model is loaded; "+
 				"this says whether it is doing anything.", gpu, nil),
+		gpuMemTemp: prometheus.NewDesc("llmgw_gpu_memory_temperature_celsius",
+			"A GPU's memory (HBM) temperature, where the driver reports it.", gpu, nil),
+		gpuPower: prometheus.NewDesc("llmgw_gpu_power_watts",
+			"A GPU's power draw now.", gpu, nil),
+		gpuPowerLimit: prometheus.NewDesc("llmgw_gpu_power_limit_watts",
+			"The power limit a GPU is held to.", gpu, nil),
+		gpuThrottled: prometheus.NewDesc("llmgw_gpu_throttled",
+			"1 while something holds a GPU's clocks down: sw_power_cap, hw_slowdown, sw_thermal, hw_thermal, power_brake.",
+			append(gpu, "reason"), nil),
+		nodeCPUTemp: prometheus.NewDesc("llmgw_node_cpu_temperature_celsius",
+			"A node's hottest CPU sensor.", node, nil),
+		nodeCPUPower: prometheus.NewDesc("llmgw_node_cpu_power_watts",
+			"A node's CPU packages' power draw, from RAPL.", node, nil),
+		nodeInletTemp: prometheus.NewDesc("llmgw_node_inlet_temperature_celsius",
+			"A node's inlet or ambient air sensor, where its board has one.", node, nil),
 		gpuTemp: prometheus.NewDesc("llmgw_gpu_temperature_celsius",
 			"A GPU's reported temperature.", gpu, nil),
 		gpuHealthy: prometheus.NewDesc("llmgw_gpu_healthy",
@@ -168,6 +194,18 @@ func New(src Source) *Collector {
 		modelUnplaceable: prometheus.NewDesc("llmgw_model_unplaceable",
 			"1 when a model is short of replicas and the last attempt found no node to place one on.",
 			[]string{"model"}, nil),
+		capacityHolds: prometheus.NewDesc("llmgw_capacity_holds",
+			"What a part of the tenancy tree holds of a resource, its own models' replicas and those beneath it.",
+			[]string{"scope", "resource"}, nil),
+		capacityGuarantee: prometheus.NewDesc("llmgw_capacity_guarantee",
+			"What a part of the tenancy tree is guaranteed of a resource.",
+			[]string{"scope", "resource"}, nil),
+		capacityLimit: prometheus.NewDesc("llmgw_capacity_limit",
+			"The most of a resource a part of the tenancy tree may hold.",
+			[]string{"scope", "resource"}, nil),
+		modelPreempted: prometheus.NewDesc("llmgw_model_preempted_total",
+			"Replicas of a model stopped to make room for a model of a higher capacity class.",
+			[]string{"model", "class"}, nil),
 		engineInstall: prometheus.NewDesc("llmgw_node_engine_install",
 			"Where installing an engine stands on a node: 1 for the current state.",
 			[]string{"node", "engine", "version", "state"}, nil),
@@ -234,9 +272,11 @@ func (c *Collector) Describe(ch chan<- *prometheus.Desc) {
 		c.ibActive, c.ibRate, c.ibCounter, c.fabricSMs, c.fabricMaster, c.fabricProblems,
 		c.gpuNVLJoined, c.imexUp,
 		c.gpuMemoryTotal, c.gpuMemoryUsed, c.gpuUtil, c.gpuTemp, c.gpuHealthy,
+		c.gpuMemTemp, c.gpuPower, c.gpuPowerLimit, c.gpuThrottled, c.nodeCPUTemp, c.nodeCPUPower, c.nodeInletTemp,
 		c.gpuNVLink, c.gpuECC,
 		c.replicaState, c.replicaReady, c.replicaIdle,
-		c.modelDesired, c.modelFailures, c.modelParked, c.modelUnplaceable,
+		c.modelDesired, c.modelFailures, c.modelParked, c.modelUnplaceable, c.modelPreempted,
+		c.capacityHolds, c.capacityGuarantee, c.capacityLimit,
 		c.engineInstall, c.leader,
 	} {
 		ch <- d
@@ -246,6 +286,7 @@ func (c *Collector) Describe(ch chan<- *prometheus.Desc) {
 // Collect implements prometheus.Collector.
 func (c *Collector) Collect(ch chan<- prometheus.Metric) {
 	now := time.Now()
+	c.collectQuotas(ch)
 
 	loads := map[string]*inventory.Load{}
 	for _, n := range c.src.Nodes() {
@@ -279,6 +320,16 @@ func (c *Collector) Collect(ch chan<- prometheus.Metric) {
 		}
 
 		if l := n.Load; l != nil && now.Sub(l.At) < loadFresh {
+			// Each only when measured: a VM has none of these, and a zero
+			// would read as a machine at 0 °C drawing nothing.
+			for _, v := range []struct {
+				d   *prometheus.Desc
+				val float64
+			}{{c.nodeCPUTemp, l.CPUTempC}, {c.nodeCPUPower, l.CPUPowerW}, {c.nodeInletTemp, l.InletTempC}} {
+				if v.val > 0 {
+					ch <- prometheus.MustNewConstMetric(v.d, prometheus.GaugeValue, v.val, n.ID)
+				}
+			}
 			ch <- prometheus.MustNewConstMetric(c.nodeCPU, prometheus.GaugeValue, l.CPUPercent, n.ID)
 			if l.CPUCount > 0 {
 				ch <- prometheus.MustNewConstMetric(c.nodeCPUs, prometheus.GaugeValue, float64(l.CPUCount), n.ID)
@@ -348,7 +399,11 @@ func (c *Collector) Collect(ch chan<- prometheus.Metric) {
 				ch <- prometheus.MustNewConstMetric(c.gpuUtil, prometheus.GaugeValue,
 					float64(g.UtilizationPercent), labels...)
 			}
-			if g.Health.TemperatureC > 0 {
+			// The sensors' fresh reading when there is one; the temperature
+			// read at registration otherwise, which is all an older agent sends.
+			if s, ok := sensorFor(n.Load, g.Index, now); ok {
+				c.gpuSensor(ch, s, labels)
+			} else if g.Health.TemperatureC > 0 {
 				ch <- prometheus.MustNewConstMetric(c.gpuTemp, prometheus.GaugeValue,
 					float64(g.Health.TemperatureC), labels...)
 			}
@@ -455,6 +510,7 @@ func (c *Collector) Collect(ch chan<- prometheus.Metric) {
 		ch <- prometheus.MustNewConstMetric(c.modelFailures, prometheus.GaugeValue, float64(h.Failures), h.Model)
 		ch <- prometheus.MustNewConstMetric(c.modelParked, prometheus.GaugeValue, b2f(h.Parked), h.Model)
 		ch <- prometheus.MustNewConstMetric(c.modelUnplaceable, prometheus.GaugeValue, b2f(h.Unplaceable), h.Model)
+		ch <- prometheus.MustNewConstMetric(c.modelPreempted, prometheus.CounterValue, float64(h.Preempted), h.Model, h.Class)
 	}
 }
 
@@ -464,3 +520,65 @@ func b2f(b bool) float64 {
 	}
 	return 0
 }
+
+// QuotaSource adds the tenancy tree's capacity quotas. Asked for by type
+// assertion, like HealthSource.
+type QuotaSource interface {
+	QuotaUse() []reconciler.QuotaUse
+}
+
+func (c *Collector) collectQuotas(ch chan<- prometheus.Metric) {
+	qs, ok := c.src.(QuotaSource)
+	if !ok {
+		return
+	}
+	for _, q := range qs.QuotaUse() {
+		for _, r := range []struct {
+			name     string
+			holds    int
+			gua, lim *int
+		}{
+			{"gpus", q.Holds.GPUs, q.Guarantee.GPUs, q.Limit.GPUs},
+			{"cpu_cores", q.Holds.CPUCores, q.Guarantee.CPUCores, q.Limit.CPUCores},
+			{"memory_gb", q.Holds.MemoryGB, q.Guarantee.MemoryGB, q.Limit.MemoryGB},
+		} {
+			if r.gua == nil && r.lim == nil {
+				continue
+			}
+			ch <- prometheus.MustNewConstMetric(c.capacityHolds, prometheus.GaugeValue, float64(r.holds), q.Scope, r.name)
+			if r.gua != nil {
+				ch <- prometheus.MustNewConstMetric(c.capacityGuarantee, prometheus.GaugeValue, float64(*r.gua), q.Scope, r.name)
+			}
+			if r.lim != nil {
+				ch <- prometheus.MustNewConstMetric(c.capacityLimit, prometheus.GaugeValue, float64(*r.lim), q.Scope, r.name)
+			}
+		}
+	}
+}
+
+// sensorFor is a GPU's fresh sensor reading, if the node sent one.
+func sensorFor(l *inventory.Load, index int, now time.Time) (inventory.GPUSensor, bool) {
+	if l == nil || now.Sub(l.At) >= loadFresh {
+		return inventory.GPUSensor{}, false
+	}
+	for _, g := range l.GPUs {
+		if g.Index == index {
+			return g, true
+		}
+	}
+	return inventory.GPUSensor{}, false
+}
+
+func (c *Collector) gpuSensor(ch chan<- prometheus.Metric, s inventory.GPUSensor, labels []string) {
+	for _, v := range []struct {
+		d   *prometheus.Desc
+		val float64
+	}{{c.gpuTemp, s.TempC}, {c.gpuMemTemp, s.MemoryTempC}, {c.gpuPower, s.PowerW}, {c.gpuPowerLimit, s.PowerLimitW}} {
+		if v.val > 0 {
+			ch <- prometheus.MustNewConstMetric(v.d, prometheus.GaugeValue, v.val, labels...)
+		}
+	}
+	for _, reason := range s.Throttle {
+		ch <- prometheus.MustNewConstMetric(c.gpuThrottled, prometheus.GaugeValue, 1, append(labels, reason)...)
+	}
+}

internal/control/fleetmetrics/sensors_test.go

@@ -0,0 +1,41 @@
+package fleetmetrics
+
+import (
+	"strings"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
+)
+
+func TestSensorsArePublishedFreshAndOnlyWhenMeasured(t *testing.T) {
+	gpuNode := inventory.Node{ID: "g1", State: inventory.Ready,
+		GPUs: []discovery.GPU{{Index: 0, Name: "NVIDIA H100 80GB HBM3", MemoryTotalMB: 81559, Health: discovery.GPUHealth{TemperatureC: 40}}},
+		Load: &inventory.Load{At: time.Now(), CPUTempC: 58, CPUPowerW: 65,
+			GPUs: []inventory.GPUSensor{{Index: 0, TempC: 88, PowerW: 698, PowerLimitW: 700,
+				Throttle: []string{"sw_power_cap", "sw_thermal"}}}}}
+	vm := inventory.Node{ID: "vm1", State: inventory.Ready, Load: &inventory.Load{At: time.Now(), CPUPercent: 3}}
+	out := gather(t, fakeSource{nodes: []inventory.Node{gpuNode, vm}})
+
+	for _, want := range []string{
+		"llmgw_node_cpu_temperature_celsius node=g1 -> 58",
+		"llmgw_node_cpu_power_watts node=g1 -> 65",
+		"llmgw_gpu_power_watts class=h100-80g gpu=0 node=g1 -> 698",
+		"llmgw_gpu_power_limit_watts class=h100-80g gpu=0 node=g1 -> 700",
+		"llmgw_gpu_throttled class=h100-80g gpu=0 node=g1 reason=sw_thermal -> 1",
+		// The fresh reading, not the 40 °C from registration.
+		"llmgw_gpu_temperature_celsius class=h100-80g gpu=0 node=g1 -> 88",
+	} {
+		if !strings.Contains(out, want) {
+			t.Errorf("missing %q", want)
+		}
+	}
+	for _, absent := range []string{"node=vm1 -> 0", "llmgw_node_inlet_temperature_celsius", "llmgw_gpu_memory_temperature_celsius"} {
+		for _, line := range strings.Split(out, "\n") {
+			if strings.Contains(line, absent) && (strings.Contains(line, "temperature") || strings.Contains(line, "power")) {
+				t.Errorf("published what was not measured: %s", line)
+			}
+		}
+	}
+}

internal/control/inventory/inventory.go

@@ -79,6 +79,9 @@ type Node struct {
 	// it, which is treated as no constraint — the same as every node before
 	// this field existed.
 	MemoryTotalMB int `json:"memory_total_mb,omitempty"`
+	// CPUCount is the CPUs the agent may run on. Zero is an agent too old to
+	// say, which, like memory, constrains nothing.
+	CPUCount int `json:"cpu_count,omitempty"`
 
 	// Weights are the model files this node can see, however they got there.
 	// Empty means the node was not asked to look, which is not the same as
@@ -301,6 +304,21 @@ type Load struct {
 	MemAvailBytes int64
 	// Shards is each engine's use, by shard ID.
 	Shards map[string]ShardLoad
+	// What the node's sensors say; zero is not measured.
+	CPUTempC   float64
+	CPUPowerW  float64
+	InletTempC float64
+	GPUs       []GPUSensor
+}
+
+// GPUSensor is one card's heat and power, and what is throttling it.
+type GPUSensor struct {
+	Index       int      `json:"index"`
+	TempC       float64  `json:"temp_c,omitempty"`
+	MemoryTempC float64  `json:"memory_temp_c,omitempty"`
+	PowerW      float64  `json:"power_w,omitempty"`
+	PowerLimitW float64  `json:"power_limit_w,omitempty"`
+	Throttle    []string `json:"throttle,omitempty"`
 }
 
 // ShardLoad is one engine's use: cores busy, and resident memory.
@@ -373,6 +391,9 @@ type Allocation struct {
 	// Zero for a profile that declared nothing, which constrains nothing and
 	// is counted as nothing.
 	MemoryGB int
+	// CPUCores is the cores this shard was given to compute on, for a shard
+	// with no GPU; zero for one that shares whatever is free.
+	CPUCores int
 }
 
 // Store holds the cluster inventory. It is safe for concurrent use.

internal/control/modelserve/modelserve.go

@@ -91,6 +91,9 @@ type Request struct {
 	// use. They default to 16384 and 4096, what four parallel slots divide.
 	CtxPool       int `json:"ctx_pool,omitempty"`
 	ContextLength int `json:"context_length,omitempty"`
+	// CPUCores is what a replica computes on; zero takes half the smallest
+	// node it fits on.
+	CPUCores int `json:"cpu_cores,omitempty"`
 	// By is who asked, for the catalog's record.
 	By string `json:"-"`
 }
@@ -278,13 +281,20 @@ func (s *Server) run(j *Job) {
 	}
 	s.step(j, "check", Done, fmt.Sprintf("%s, needs about %d GB (%s)", orUnknown(header.Architecture), memGB, basis))
 
+	cores := req.CPUCores
+	if cores <= 0 {
+		if nc, ok := s.deps.Fleet.(interface{ NodeCPUs(int) []int }); ok {
+			cores = catalog.SuggestCores(nc.NodeCPUs(memGB))
+		}
+	}
+
 	// 4. The catalog.
 	m := catalog.Model{
 		Name: name, Aliases: req.Aliases, Description: req.Description, Source: source,
 		RoutingProfile: "chat", ContextLength: req.ContextLength, MaxOutput: 1024,
 		Profiles: []catalog.Profile{{
 			Name: "cpu", Engine: "llamacpp", Weight: 1, Priority: 100,
-			Args: []string{"--ctx-size", strconv.Itoa(req.CtxPool)}, MinMemoryGB: memGB,
+			Args: []string{"--ctx-size", strconv.Itoa(req.CtxPool)}, MinMemoryGB: memGB, CPUCores: cores,
 		}},
 		Scaling: catalog.Scaling{Min: req.Replicas, Max: max(req.Replicas, 1)},
 	}

internal/control/modelsvc/modelsvc.go

@@ -18,6 +18,7 @@ package modelsvc
 import (
 	"context"
 	"encoding/json"
+	"errors"
 	"fmt"
 	"log/slog"
 	"sync"
@@ -104,6 +105,16 @@ func (s *Service) Add(ctx context.Context, m catalog.Model, by string) error {
 	if s.Declared(m.Name) {
 		return fmt.Errorf("model %q is declared in configuration; edit that rather than adding a second definition", m.Name)
 	}
+	// A model with no GPU is all host memory, and one that declares none
+	// fits everywhere as far as placement can tell — however many of it
+	// there already are. Required of what is added now; models stored before
+	// this still load, and say what they are missing.
+	for _, p := range m.Profiles {
+		if p.CPUOnly() && p.MinMemoryGB <= 0 {
+			return fmt.Errorf("model %q profile %q runs without a GPU and declares no memory (min_memory_gb): "+
+				"placement could not tell when a node is full", m.Name, p.Name)
+		}
+	}
 	if s.check != nil {
 		if err := s.check(ctx, m); err != nil {
 			return err
@@ -189,36 +200,98 @@ func (s *Service) Remove(ctx context.Context, name string) error {
 	return nil
 }
 
-// Scale changes how many replicas a model runs, without a restart.
-//
-// Only the scaling block moves. Everything else about the model — its source,
-// its profiles, the engine arguments somebody tuned — is read back from what
-// was stored and written out again untouched, so changing a number cannot
-// quietly become a redefinition.
-//
-// A model declared in configuration is refused, for the same reason Remove
-// refuses one: the YAML is read on every start, so the new number would hold
-// until the next restart and then silently revert. A cluster that disagrees
-// with its own config file is worse than an error message saying where to edit.
+// Scale changes how many replicas a model added at runtime runs.
 func (s *Service) Scale(ctx context.Context, name string, sc catalog.Scaling, by string) (catalog.Model, error) {
+	final, err := s.update(ctx, name, by, "change its scaling", func(m *catalog.Model) error {
+		m.Scaling = sc
+		return nil
+	})
+	if err == nil {
+		s.log.Info("model scaled", "model", name,
+			"min", final.Scaling.Min, "max", final.Scaling.Max, "by", by)
+	}
+	return final, err
+}
+
+// Resources sets what a model's profile reserves of a node: host memory, and
+// the CPU cores a replica without a GPU computes on. An empty profile name
+// sets every profile.
+func (s *Service) Resources(ctx context.Context, name, profile string, memoryGB, cpuCores int, by string) (catalog.Model, error) {
+	if memoryGB < 0 || cpuCores < 0 {
+		return catalog.Model{}, errors.New("memory and cores cannot be negative")
+	}
+	final, err := s.update(ctx, name, by, "change its resources", func(m *catalog.Model) error {
+		found := false
+		for i := range m.Profiles {
+			if profile != "" && m.Profiles[i].Name != profile {
+				continue
+			}
+			found = true
+			m.Profiles[i].MinMemoryGB = memoryGB
+			m.Profiles[i].CPUCores = cpuCores
+		}
+		if !found {
+			return fmt.Errorf("model %q has no profile %q", name, profile)
+		}
+		return nil
+	})
+	if err == nil {
+		s.log.Info("model resources set", "model", name, "profile", profile,
+			"memory_gb", memoryGB, "cpu_cores", cpuCores, "by", by)
+	}
+	return final, err
+}
+
+// SetClass sets a model's capacity class: critical, standard or
+// opportunistic, or empty to go back to what its scaling implies.
+func (s *Service) SetClass(ctx context.Context, name, class, by string) (catalog.Model, error) {
+	final, err := s.update(ctx, name, by, "change its class", func(m *catalog.Model) error {
+		m.Class = class
+		return nil
+	})
+	if err == nil {
+		s.log.Info("model class set", "model", name, "class", final.EffectiveClass(), "by", by)
+	}
+	return final, err
+}
+
+// SetOwner sets the part of the tenancy tree a model's capacity counts
+// against, org:ID, team:ID or project:ID; empty makes it shared.
+func (s *Service) SetOwner(ctx context.Context, name, owner, by string) (catalog.Model, error) {
+	final, err := s.update(ctx, name, by, "change its owner", func(m *catalog.Model) error {
+		m.Owner = owner
+		return nil
+	})
+	if err == nil {
+		s.log.Info("model owner set", "model", name, "owner", owner, "by", by)
+	}
+	return final, err
+}
+
+// update changes one model added at runtime and keeps the change.
+//
+// Built before anything is written, like Add: catalog.New is what applies the
+// defaults and refuses what does not validate, so the model that comes back
+// is the one that will run rather than the one that was asked for.
+func (s *Service) update(ctx context.Context, name, by, what string, change func(*catalog.Model) error) (catalog.Model, error) {
 	s.mu.Lock()
 	defer s.mu.Unlock()
 
 	if s.Declared(name) {
 		return catalog.Model{}, fmt.Errorf(
-			"model %q is declared in configuration; change its scaling there, or it reverts on the next restart", name)
+			"model %q is declared in configuration; %s there, or it reverts on the next restart", name, what)
 	}
-
 	stored, err := s.storedModels(ctx)
 	if err != nil {
 		return catalog.Model{}, err
 	}
-
 	next := append([]catalog.Model{}, s.declared...)
 	found := false
 	for _, e := range stored {
 		if e.Name == name {
-			e.Scaling = sc
+			if err := change(&e); err != nil {
+				return catalog.Model{}, err
+			}
 			found = true
 		}
 		next = append(next, e)
@@ -226,20 +299,14 @@ func (s *Service) Scale(ctx context.Context, name string, sc catalog.Scaling, by
 	if !found {
 		return catalog.Model{}, fmt.Errorf("model %q was not added at runtime", name)
 	}
-
-	// Built before anything is written, like Add: catalog.New is what applies
-	// the defaults (max follows min when it is left out) and what refuses a
-	// max below a min, so the model that comes back is the one that will run
-	// rather than the one that was asked for.
 	built, err := catalog.New(next)
 	if err != nil {
-		return catalog.Model{}, fmt.Errorf("scaling %q that way would make the catalog invalid: %w", name, err)
+		return catalog.Model{}, fmt.Errorf("that would make the catalog invalid: %w", err)
 	}
 	final, ok := built.Resolve(name)
 	if !ok {
 		return catalog.Model{}, fmt.Errorf("model %q vanished from the catalog it was just built into", name)
 	}
-
 	def, err := json.Marshal(final)
 	if err != nil {
 		return catalog.Model{}, err
@@ -249,10 +316,7 @@ func (s *Service) Scale(ctx context.Context, name string, sc catalog.Scaling, by
 	}); err != nil {
 		return catalog.Model{}, err
 	}
-
 	s.apply.ReplaceCatalog(ctx, built)
-	s.log.Info("model scaled", "model", name,
-		"min", final.Scaling.Min, "max", final.Scaling.Max, "by", by)
 	return final, nil
 }
 

internal/control/modelsvc/modelsvc_test.go

@@ -48,7 +48,7 @@ func model(name string) catalog.Model {
 	return catalog.Model{
 		Name: name, Source: "/models/" + name,
 		Scaling:  catalog.Scaling{Min: 1, Max: 1},
-		Profiles: []catalog.Profile{{Name: "p", Engine: "llamacpp", Weight: 1}},
+		Profiles: []catalog.Profile{{Name: "p", Engine: "llamacpp", Weight: 1, MinMemoryGB: 2}},
 	}
 }
 
@@ -337,3 +337,43 @@ func TestMinZeroWithAMaxIsOnDemand(t *testing.T) {
 		t.Errorf("the idle timeout was lost; got %v", got.Scaling.IdleTimeout)
 	}
 }
+
+func TestACPUModelAddedNowMustSayHowMuchMemoryItNeeds(t *testing.T) {
+	s, st, _ := newService(t)
+	m := model("bare")
+	m.Profiles[0].MinMemoryGB = 0
+	err := s.Add(t.Context(), m, "tester")
+	if err == nil || !strings.Contains(err.Error(), "declares no memory") {
+		t.Fatalf("a CPU model with no memory was added: %v", err)
+	}
+	if len(st.rows) != 0 {
+		t.Error("it was stored anyway")
+	}
+}
+
+func TestResourcesAreSetOnAStoredModelAndSurviveARestart(t *testing.T) {
+	s, _, _ := newService(t)
+	if err := s.Add(t.Context(), model("m"), "tester"); err != nil {
+		t.Fatal(err)
+	}
+	got, err := s.Resources(t.Context(), "m", "", 6, 3, "tester")
+	if err != nil {
+		t.Fatal(err)
+	}
+	if p := got.Profiles[0]; p.MinMemoryGB != 6 || p.CPUCores != 3 {
+		t.Errorf("profile %+v", p)
+	}
+	if _, err := s.Resources(t.Context(), "m", "nope", 6, 3, "tester"); err == nil {
+		t.Error("a profile the model does not have was accepted")
+	}
+	if _, err := s.Resources(t.Context(), "m", "", -1, 0, "tester"); err == nil {
+		t.Error("negative memory was accepted")
+	}
+	cat, err := s.Load(t.Context())
+	if err != nil {
+		t.Fatal(err)
+	}
+	if m, _ := cat.Resolve("m"); m.Profiles[0].CPUCores != 3 {
+		t.Error("the cores were lost on reload")
+	}
+}

internal/control/placement/cpu_test.go

@@ -1,6 +1,7 @@
 package placement
 
 import (
+	"strings"
 	"testing"
 
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
@@ -100,3 +101,38 @@ func TestCPUProfileRespectsAvoid(t *testing.T) {
 		t.Errorf("placed on %q despite being told to avoid it", got)
 	}
 }
+
+func TestCPUCoresAreReservedAsMemoryIs(t *testing.T) {
+	n := node("n1", "", 0)
+	n.CPUCount = 8
+	st := store(n)
+	s := New(st)
+	p := cpuProfile("cpu")
+	p.CPUCores = 3
+
+	for i := range 2 {
+		res := s.Place([]Profile{p}, nil)
+		if res.Placement == nil {
+			t.Fatalf("replica %d did not fit in 8 cores:\n%s", i+1, res.Explain())
+		}
+		st.Allocate("n1", inventory.Allocation{ShardID: string(rune('a' + i)), CPUCores: 3})
+	}
+	// Six of eight given away: a third would need three.
+	res := s.Place([]Profile{p}, nil)
+	if res.Placement != nil {
+		t.Fatal("a third replica was given cores the node does not have")
+	}
+	if got := res.Explain(); !strings.Contains(got, "has 8 CPUs with 2 free, needs 3") {
+		t.Errorf("the refusal should say how many cores are left:\n%s", got)
+	}
+
+	// Unknown counts constrain nothing: an older agent, or a profile that
+	// reserves no cores.
+	old := node("n2", "", 0)
+	if res := New(store(old)).Place([]Profile{p}, nil); res.Placement == nil {
+		t.Error("a node that does not report its CPUs was excluded")
+	}
+	if res := s.Place([]Profile{cpuProfile("cpu")}, nil); res.Placement == nil {
+		t.Error("a profile reserving no cores was refused")
+	}
+}

internal/control/placement/placement.go

@@ -39,6 +39,9 @@ type Profile struct {
 	// MinDiskGB is the free space needed to hold the weights, checked only on
 	// nodes that do not already have them. Zero is unconstrained.
 	MinDiskGB int
+	// CPUCores is the cores a profile with no GPU reserves. Zero is
+	// unconstrained.
+	CPUCores int
 	// Source is the model's weights, as the engine would be given it. Used to
 	// prefer nodes that already hold them; empty asks for no preference.
 	Source string
@@ -582,6 +585,18 @@ func (s *Scheduler) eligibleNodes(p Profile, fabric string, avoid []string, reje
 			}
 		}
 
+		// Cores, for a replica that computes on them: what is left after the
+		// cores already given to the node's other replicas. Same exemptions
+		// as memory: zero declares nothing, and a node that does not report
+		// its CPUs is an older agent, not one with none.
+		if p.CPUOnly() && p.CPUCores > 0 && n.CPUCount > 0 {
+			free := n.CPUCount - s.claimedCores(n.ID)
+			if free < p.CPUCores {
+				reject(fmt.Sprintf("has %d CPUs with %d free, needs %d", n.CPUCount, free, p.CPUCores))
+				continue
+			}
+		}
+
 		if p.CPUOnly() {
 			// Every ready node can run it; there is no card to be free.
 			out = append(out, candidate{node: n})
@@ -723,6 +738,15 @@ func (s *Scheduler) claimedMemory(nodeID string) int {
 	return total
 }
 
+// claimedCores is the CPU cores already given to replicas on a node.
+func (s *Scheduler) claimedCores(nodeID string) int {
+	total := 0
+	for _, a := range s.inv.Allocations(nodeID) {
+		total += a.CPUCores
+	}
+	return total
+}
+
 // installing says why an engine is missing, when the node has said.
 func installing(n inventory.Node, engine string) string {
 	for _, p := range n.Provision {

internal/control/quota/quota.go

@@ -0,0 +1,192 @@
+// Package quota decides what the tenancy tree is guaranteed of the cluster
+// and how much of it each part may hold.
+//
+// A scope — an organization, team or project — may be guaranteed GPUs, CPU
+// cores and memory, and limited to at most some amount of each. What a scope
+// holds is what the replicas of the models it owns, and of the models its
+// teams and projects own, have reserved. Holding more than its guarantee is
+// borrowing: allowed while the room is free, and taken back when another
+// scope within its own guarantee needs it.
+package quota
+
+import (
+	"fmt"
+	"sort"
+	"strings"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// Amount is how much of each resource something holds or needs.
+type Amount struct {
+	GPUs, CPUCores, MemoryGB int
+}
+
+// Plus adds two amounts.
+func (a Amount) Plus(b Amount) Amount {
+	return Amount{a.GPUs + b.GPUs, a.CPUCores + b.CPUCores, a.MemoryGB + b.MemoryGB}
+}
+
+// Over names the first resource in which a is above the governed amounts of
+// l, with both figures; empty when it is above none.
+func Over(a Amount, l store.Resources) string {
+	switch {
+	case l.GPUs != nil && a.GPUs > *l.GPUs:
+		return fmt.Sprintf("%d GPUs of %d", a.GPUs, *l.GPUs)
+	case l.CPUCores != nil && a.CPUCores > *l.CPUCores:
+		return fmt.Sprintf("%d cores of %d", a.CPUCores, *l.CPUCores)
+	case l.MemoryGB != nil && a.MemoryGB > *l.MemoryGB:
+		return fmt.Sprintf("%d GB of memory of %d", a.MemoryGB, *l.MemoryGB)
+	}
+	return ""
+}
+
+// Holding is one replica's hold on the cluster, for working out who holds
+// what.
+type Holding struct {
+	Replica string
+	Owner   string
+	Amount  Amount
+	Since   time.Time
+}
+
+// Policy is the quotas and the tree they sit in, read at one moment.
+type Policy struct {
+	quotas  map[string]store.CapacityQuota
+	parents map[string][]string
+}
+
+// New builds a policy from stored quotas and each scope's parents.
+func New(quotas []store.CapacityQuota, parents map[string][]string) *Policy {
+	p := &Policy{quotas: map[string]store.CapacityQuota{}, parents: parents}
+	for _, q := range quotas {
+		p.quotas[Key(q.Scope, q.ScopeID)] = q
+	}
+	return p
+}
+
+// Key writes a scope the way owners are written: team:ID.
+func Key(s store.Scope, id string) string { return string(s) + ":" + id }
+
+// Chain is an owner and every scope above it, nearest first.
+func (p *Policy) Chain(owner string) []string {
+	if owner == "" {
+		return nil
+	}
+	return append([]string{owner}, p.parents[owner]...)
+}
+
+// Governing is the quotas along an owner's chain, nearest first.
+func (p *Policy) Governing(owner string) []store.CapacityQuota {
+	if p == nil {
+		return nil
+	}
+	var out []store.CapacityQuota
+	for _, s := range p.Chain(owner) {
+		if q, ok := p.quotas[s]; ok {
+			out = append(out, q)
+		}
+	}
+	return out
+}
+
+// Quotas lists every quota.
+func (p *Policy) Quotas() []store.CapacityQuota {
+	if p == nil {
+		return nil
+	}
+	out := make([]store.CapacityQuota, 0, len(p.quotas))
+	for _, q := range p.quotas {
+		out = append(out, q)
+	}
+	sort.Slice(out, func(i, j int) bool {
+		return Key(out[i].Scope, out[i].ScopeID) < Key(out[j].Scope, out[j].ScopeID)
+	})
+	return out
+}
+
+// Usage is what every scope holds, its own models' replicas and those of
+// everything beneath it.
+func (p *Policy) Usage(hs []Holding) map[string]Amount {
+	out := map[string]Amount{}
+	if p == nil {
+		return out
+	}
+	for _, h := range hs {
+		for _, s := range p.Chain(h.Owner) {
+			out[s] = out[s].Plus(h.Amount)
+		}
+	}
+	return out
+}
+
+// Borrowed is the replicas held beyond a guarantee: of each governed scope's
+// replicas, oldest first, the ones that take it past what it is guaranteed.
+// A replica beyond any guarantee along its chain is borrowed.
+func (p *Policy) Borrowed(hs []Holding) map[string]bool {
+	out := map[string]bool{}
+	if p == nil || len(p.quotas) == 0 {
+		return out
+	}
+	sorted := append([]Holding(nil), hs...)
+	sort.Slice(sorted, func(i, j int) bool { return sorted[i].Since.Before(sorted[j].Since) })
+	for key, q := range p.quotas {
+		if !q.Guarantee.Any() {
+			continue
+		}
+		var sum Amount
+		for _, h := range sorted {
+			if !contains(p.Chain(h.Owner), key) {
+				continue
+			}
+			sum = sum.Plus(h.Amount)
+			if Over(sum, q.Guarantee) != "" {
+				out[h.Replica] = true
+			}
+		}
+	}
+	return out
+}
+
+// Guaranteed reports whether an owner, holding what it holds, is still within
+// every guarantee along its chain with need added — so it may take back what
+// others borrowed. An owner with no guarantee anywhere is never guaranteed.
+func (p *Policy) Guaranteed(owner string, usage map[string]Amount, need Amount) bool {
+	gov := p.Governing(owner)
+	any := false
+	for _, q := range gov {
+		if !q.Guarantee.Any() {
+			continue
+		}
+		any = true
+		if Over(usage[Key(q.Scope, q.ScopeID)].Plus(need), q.Guarantee) != "" {
+			return false
+		}
+	}
+	return any
+}
+
+// AtLimit says which limit along an owner's chain need would take it past,
+// empty when none.
+func (p *Policy) AtLimit(owner string, usage map[string]Amount, need Amount) string {
+	for _, q := range p.Governing(owner) {
+		if !q.Limit.Any() {
+			continue
+		}
+		key := Key(q.Scope, q.ScopeID)
+		if over := Over(usage[key].Plus(need), q.Limit); over != "" {
+			return fmt.Sprintf("%s is at its capacity limit (%s)", key, strings.TrimSpace(over))
+		}
+	}
+	return ""
+}
+
+func contains(list []string, v string) bool {
+	for _, x := range list {
+		if x == v {
+			return true
+		}
+	}
+	return false
+}

internal/control/quota/quota_test.go

@@ -0,0 +1,83 @@
+package quota
+
+import (
+	"strings"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+func n(v int) *int { return &v }
+
+// acme has two teams; ml is guaranteed 8 GPUs, the org as a whole 12 and
+// no more than 16.
+func policy() *Policy {
+	return New([]store.CapacityQuota{
+		{Scope: store.ScopeTeam, ScopeID: "ml", Guarantee: store.Resources{GPUs: n(8)}},
+		{Scope: store.ScopeOrg, ScopeID: "acme", Guarantee: store.Resources{GPUs: n(12)}, Limit: store.Resources{GPUs: n(16)}},
+	}, map[string][]string{
+		"team:ml": {"org:acme"}, "team:web": {"org:acme"},
+		"project:train": {"team:ml", "org:acme"},
+	})
+}
+
+func at(min int) time.Time { return time.Date(2026, 1, 1, 0, min, 0, 0, time.UTC) }
+
+func TestAScopeHoldsWhatEverythingBeneathItHolds(t *testing.T) {
+	p := policy()
+	u := p.Usage([]Holding{
+		{Replica: "a", Owner: "project:train", Amount: Amount{GPUs: 4}},
+		{Replica: "b", Owner: "team:web", Amount: Amount{GPUs: 2, CPUCores: 3}},
+		{Replica: "c", Owner: "", Amount: Amount{GPUs: 8}}, // shared: nobody's
+	})
+	if u["team:ml"].GPUs != 4 || u["org:acme"].GPUs != 6 || u["org:acme"].CPUCores != 3 || u["project:train"].GPUs != 4 {
+		t.Errorf("usage %+v", u)
+	}
+	if _, ok := u[""]; ok {
+		t.Error("a shared model was counted against somebody")
+	}
+}
+
+func TestWhatGoesPastAGuaranteeIsBorrowedNewestFirst(t *testing.T) {
+	p := policy()
+	b := p.Borrowed([]Holding{
+		{Replica: "old", Owner: "team:ml", Amount: Amount{GPUs: 8}, Since: at(1)},
+		{Replica: "new", Owner: "team:ml", Amount: Amount{GPUs: 4}, Since: at(2)},
+		{Replica: "web", Owner: "team:web", Amount: Amount{GPUs: 4}, Since: at(3)},
+	})
+	// ml: 8 within its 8, then 4 past it. acme: 8 + 4 = 12 within, web's 4
+	// takes the org to 16, past its 12.
+	if b["old"] || !b["new"] || !b["web"] {
+		t.Errorf("borrowed %v", b)
+	}
+}
+
+func TestAGuaranteeIsOnlyForWhoHasOne(t *testing.T) {
+	p := policy()
+	u := map[string]Amount{"team:ml": {GPUs: 4}, "org:acme": {GPUs: 4}}
+	if !p.Guaranteed("project:train", u, Amount{GPUs: 4}) {
+		t.Error("8 of ml's 8 and 8 of acme's 12 is within both")
+	}
+	if p.Guaranteed("project:train", u, Amount{GPUs: 5}) {
+		t.Error("9 GPUs is past ml's 8")
+	}
+	if p.Guaranteed("", u, Amount{GPUs: 1}) || New(nil, nil).Guaranteed("team:ml", u, Amount{}) {
+		t.Error("an owner with no guarantee is not guaranteed anything")
+	}
+}
+
+func TestALimitStopsGrowthWhereverItIsSet(t *testing.T) {
+	p := policy()
+	u := map[string]Amount{"org:acme": {GPUs: 14}}
+	if why := p.AtLimit("team:web", u, Amount{GPUs: 2}); why != "" {
+		t.Errorf("16 of 16 is at the limit, not past it: %s", why)
+	}
+	why := p.AtLimit("team:web", u, Amount{GPUs: 4})
+	if !strings.Contains(why, "org:acme is at its capacity limit (18 GPUs of 16)") {
+		t.Errorf("the refusal should name the scope and the figures: %q", why)
+	}
+	if p.AtLimit("", u, Amount{GPUs: 99}) != "" {
+		t.Error("a shared model was limited")
+	}
+}

internal/control/quotas.go

@@ -0,0 +1,95 @@
+package control
+
+import (
+	"context"
+	"encoding/json"
+	"log/slog"
+	"sync"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/bench"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// QuotaStore is where capacity quotas and the tenancy tree are kept.
+type QuotaStore interface {
+	CapacityQuotas(ctx context.Context) ([]store.CapacityQuota, error)
+	ScopeParents(ctx context.Context) (map[string][]string, error)
+}
+
+// quotaRefresh is how stale the reconciler's view of quotas may get. A quota
+// changed by an operator takes effect within this; the reconciler asks many
+// times a pass and the store is not asked each time.
+const quotaRefresh = 30 * time.Second
+
+// QuotaPolicy returns what the reconciler asks for its quotas: the last
+// policy read from the store, read again once it is older than quotaRefresh.
+// A store that fails keeps the last good policy, so a database hiccup does
+// not drop every guarantee and limit at once.
+func QuotaPolicy(st QuotaStore, log *slog.Logger) func() *quota.Policy {
+	var mu sync.Mutex
+	var cur *quota.Policy
+	var at time.Time
+	return func() *quota.Policy {
+		mu.Lock()
+		defer mu.Unlock()
+		if cur != nil && time.Since(at) < quotaRefresh {
+			return cur
+		}
+		ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+		defer cancel()
+		qs, err := st.CapacityQuotas(ctx)
+		if err == nil {
+			var parents map[string][]string
+			if parents, err = st.ScopeParents(ctx); err == nil {
+				cur, at = quota.New(qs, parents), time.Now()
+				return cur
+			}
+		}
+		log.Warn("could not read capacity quotas; keeping the last ones", "err", err)
+		at = time.Now() // not again on every call while the store is down
+		return cur
+	}
+}
+
+// MeasurementStore is where benchmark results are kept.
+type MeasurementStore interface {
+	Capacities(ctx context.Context) ([]store.StoredCapacity, error)
+}
+
+// Measurements returns what the reconciler asks for benchmark results, keyed
+// model/profile, read again once older than quotaRefresh; like QuotaPolicy it
+// keeps the last good answer when the store fails.
+func Measurements(st MeasurementStore, log *slog.Logger) func() map[string]reconciler.Measured {
+	var mu sync.Mutex
+	var cur map[string]reconciler.Measured
+	var at time.Time
+	return func() map[string]reconciler.Measured {
+		mu.Lock()
+		defer mu.Unlock()
+		if cur != nil && time.Since(at) < quotaRefresh {
+			return cur
+		}
+		ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+		defer cancel()
+		at = time.Now()
+		rows, err := st.Capacities(ctx)
+		if err != nil {
+			log.Warn("could not read benchmark results; keeping the last ones", "err", err)
+			return cur
+		}
+		next := map[string]reconciler.Measured{}
+		for _, row := range rows {
+			var c bench.Capacity
+			if json.Unmarshal(row.Measurement, &c) != nil {
+				continue
+			}
+			next[row.Model+"/"+row.Profile] = reconciler.Measured{
+				MaxConcurrency: c.MaxConcurrencyAtSLO, TokPerSec: c.ThroughputTokPerSec}
+		}
+		cur = next
+		return cur
+	}
+}

internal/control/reconciler/adopt.go

@@ -144,6 +144,7 @@ func (r *Reconciler) mergeAdopted(rep *Replica, group []AdoptedShard) {
 		r.inv.Allocate(s.NodeID, inventory.Allocation{
 			ShardID: s.ShardID, ReplicaID: rep.ID, GPUIndices: s.GPUIndices,
 			MemoryGB: r.profileMemory(rep.Model, rep.Profile),
+			CPUCores: r.adoptedCores(rep.Model, rep.Profile),
 		})
 		// The endpoint comes from the leader, which may be in any of the
 		// groups — the rank 0 node is not necessarily the first to register.
@@ -253,3 +254,9 @@ type AdoptedShard struct {
 	// Endpoint is the leader's base URL, set on the rank 0 shard.
 	Endpoint string
 }
+
+// adoptedCores is what an adopted shard is taken to hold of its node's CPUs.
+func (r *Reconciler) adoptedCores(model, profile string) int {
+	p, _ := r.profileOf(model, profile)
+	return cpuCores(p)
+}

internal/control/reconciler/demand.go

@@ -157,8 +157,31 @@ type Load struct {
 // the last one is still starting: on a small CPU fleet each replica is minutes
 // of weights and gigabytes of memory, and a spike that is over by the time it
 // loads should not leave three behind.
+//
+// With a benchmark stored for the profile a model runs on, its measured
+// capacity is used too: more requests per replica than were measured to meet
+// the SLO is busy, even with the engines' queues empty — an engine will take
+// requests into its slots long after latency has gone — and a replica is only
+// taken back when the rest would still be under scaleDownHeadroom of what they
+// were measured to hold, so scaling down does not land on the edge.
 func (r *Reconciler) RecordLoad(loads map[string]Load) {
 	now := r.now()
+	// The profile each model's replicas run on, for its measurement; read
+	// before the loop, which holds the lock.
+	running := map[string]string{}
+	r.mu.Lock()
+	for _, rep := range r.replicas {
+		if rep.State.Counts() {
+			running[rep.Model] = rep.Profile
+		}
+	}
+	r.mu.Unlock()
+	measured := map[string]Measured{}
+	for model, profile := range running {
+		if m, ok := r.measured(model, profile); ok {
+			measured[model] = m
+		}
+	}
 	for _, m := range r.cat.Load().Models() {
 		base := r.baseFor(m)
 		r.mu.Lock()
@@ -178,8 +201,13 @@ func (r *Reconciler) RecordLoad(loads map[string]Load) {
 		l := loads[m.Name]
 		perReplica := float64(l.Waiting) / float64(max(l.Replicas, 1))
 		extra := r.extra[m.Name]
+		meas, hasMeas := measured[m.Name]
+		overMeasured := hasMeas &&
+			float64(l.Running+l.Waiting) > float64(meas.MaxConcurrency*max(l.Replicas, 1))
+		fitsInFewer := !hasMeas ||
+			float64(l.Running) <= scaleDownHeadroom*float64(meas.MaxConcurrency*max(l.Replicas-1, 0))
 		switch {
-		case perReplica >= r.cfg.ScaleUpWaiting:
+		case perReplica >= r.cfg.ScaleUpWaiting || overMeasured:
 			delete(r.quietSince, m.Name)
 			since, ok := r.busySince[m.Name]
 			if !ok {
@@ -190,9 +218,10 @@ func (r *Reconciler) RecordLoad(loads map[string]Load) {
 				r.extra[m.Name] = extra + 1
 				r.busySince[m.Name] = now
 				r.log.Info("scaling up for load", "model", m.Name, "replicas", base+extra+1,
-					"waiting_per_replica", perReplica, "for", now.Sub(since).Round(time.Second))
+					"waiting_per_replica", perReplica, "in_flight", l.Running+l.Waiting,
+					"measured_per_replica", meas.MaxConcurrency, "for", now.Sub(since).Round(time.Second))
 			}
-		case l.Waiting == 0 && extra > 0:
+		case l.Waiting == 0 && extra > 0 && fitsInFewer:
 			delete(r.busySince, m.Name)
 			since, ok := r.quietSince[m.Name]
 			if !ok {
@@ -212,3 +241,8 @@ func (r *Reconciler) RecordLoad(loads map[string]Load) {
 		r.mu.Unlock()
 	}
 }
+
+// scaleDownHeadroom is how full, of their measured capacity, the replicas left
+// after a scale-down may be. Below one so that the load that was there a
+// minute ago does not put them straight back over.
+const scaleDownHeadroom = 0.7

internal/control/reconciler/headroom.go

@@ -5,6 +5,7 @@ import (
 
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/placement"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
 )
 
 // Headroom is how many more replicas of a model the cluster has room for now.
@@ -24,6 +25,9 @@ type Headroom struct {
 	// Blocked is why one more than More does not fit, a line per node that
 	// was considered. Empty when Limited.
 	Blocked []string
+	// TokPerSec is what one more replica was measured to serve, on the
+	// profile placement would give it; zero when it was never measured.
+	TokPerSec float64
 }
 
 // Headroom tries placing up to limit more replicas of a model on a copy of
@@ -52,11 +56,16 @@ func (r *Reconciler) Headroom(model string, limit int) (Headroom, bool) {
 
 	inv := r.inv.Clone()
 	sched := placement.New(inv)
+	pol := r.policy()
+	var usage map[string]quota.Amount
+	if pol != nil {
+		usage = pol.Usage(r.Holdings())
+	}
 	for h.More < limit {
-		res := sched.Place(m.PlacementProfiles(), avoid)
+		res := sched.Place(r.orderedProfiles(m), avoid)
 		if res.Placement == nil && len(avoid) > 0 {
 			// As scaleUp does: spreading is a preference, not a rule.
-			res = sched.Place(m.PlacementProfiles(), nil)
+			res = sched.Place(r.orderedProfiles(m), nil)
 		}
 		if res.Placement == nil {
 			for _, rej := range res.Rejections {
@@ -66,14 +75,30 @@ func (r *Reconciler) Headroom(model string, limit int) (Headroom, bool) {
 			h.Blocked = slices.Compact(h.Blocked)
 			return h, true
 		}
-		mem := 0
+		// A quota limit is as much an edge as a full node.
+		if pol != nil && m.Owner != "" {
+			need := placementNeed(m, res.Placement)
+			if why := pol.AtLimit(m.Owner, usage, need); why != "" {
+				h.Blocked = []string{why}
+				return h, true
+			}
+			for _, s := range pol.Chain(m.Owner) {
+				usage[s] = usage[s].Plus(need)
+			}
+		}
+		mem, cores := 0, 0
 		if prof, ok := m.ProfileByName(res.Placement.Profile.Name); ok {
-			mem = prof.MinMemoryGB
+			mem, cores = prof.MinMemoryGB, cpuCores(prof)
+		}
+		if h.More == 0 {
+			if meas, ok := r.measured(m.Name, res.Placement.Profile.Name); ok {
+				h.TokPerSec = meas.TokPerSec
+			}
 		}
 		var nodes []string
 		for _, a := range res.Placement.Nodes {
 			inv.Allocate(a.NodeID, inventory.Allocation{
-				ShardID: "headroom", ReplicaID: "headroom", GPUIndices: a.GPUIndices, MemoryGB: mem,
+				ShardID: "headroom", ReplicaID: "headroom", GPUIndices: a.GPUIndices, MemoryGB: mem, CPUCores: cores,
 			})
 			nodes = append(nodes, a.NodeID)
 		}

internal/control/reconciler/headroom_test.go

@@ -57,3 +57,24 @@ func TestHeadroomCountsMemoryAlreadyClaimed(t *testing.T) {
 		t.Errorf("headroom %+v, expected room for 3 more", got)
 	}
 }
+
+func TestReservedCoresAreTheEnginesThreadsAndBoundTheRoom(t *testing.T) {
+	cpu := inventory.Node{ID: "a", MgmtAddr: "10.0.0.1", State: inventory.Ready, MemoryTotalMB: 64 * 1024, CPUCount: 8}
+	m := catalog.Model{Name: "coder", Source: "/m.gguf", Scaling: catalog.Scaling{Min: 1, Max: 1},
+		Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 6, CPUCores: 3}}}
+	h := newHarness(t, m, cpu)
+	h.r.Reconcile(t.Context())
+
+	started := h.disp.startedShards()
+	if len(started) != 1 || started[0].Spec.Threads != 3 {
+		t.Fatalf("the engine was not told its 3 cores: %+v", started)
+	}
+	if a := h.inv.Allocations("a"); len(a) != 1 || a[0].CPUCores != 3 {
+		t.Errorf("the cores were not reserved: %+v", a)
+	}
+	// Memory would allow nine more; cores allow one: 8 - 3 = 5, then 2.
+	got, _ := h.r.Headroom("coder", 10)
+	if got.More != 1 {
+		t.Errorf("room for %d more, expected 1 by cores: %+v", got.More, got)
+	}
+}

internal/control/reconciler/health.go

@@ -83,6 +83,12 @@ type ModelHealth struct {
 	// place one found nowhere; Placement says why.
 	Unplaceable bool
 	Placement   string
+	// Class is the capacity class it is treated as. Preempted counts the
+	// replicas it has lost to models of a higher class since the control
+	// plane started; HeldFor names the model it is giving room to now.
+	Class     string
+	Preempted int
+	HeldFor   string
 }
 
 // ModelHealth reports every model in the catalog.
@@ -106,7 +112,10 @@ func (r *Reconciler) ModelHealth() []ModelHealth {
 	out := make([]ModelHealth, 0, len(models))
 	for _, m := range models {
 		h := ModelHealth{Model: m.Name, Desired: desired[m.Name], Ready: ready[m.Name],
-			Failures: r.attempts[m.Name]}
+			Failures: r.attempts[m.Name], Class: m.EffectiveClass(), Preempted: r.preempted[m.Name]}
+		if hd, ok := r.held[m.Name]; ok && r.now().Before(hd.Until) {
+			h.HeldFor = hd.For
+		}
 		if p, ok := r.parked[m.Name]; ok {
 			h.Parked, h.ParkedSince, h.Reason = true, p.Since, p.Reason
 		}

internal/control/reconciler/measured.go

@@ -0,0 +1,108 @@
+package reconciler
+
+import (
+	"sort"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/placement"
+)
+
+// Measured is what one replica of a model on a profile was measured to serve
+// while meeting its SLO, by `llmgw ctl bench --save`.
+type Measured struct {
+	// MaxConcurrency is the most requests at once that still met the SLO.
+	MaxConcurrency int
+	// TokPerSec is the throughput at that concurrency.
+	TokPerSec float64
+}
+
+// SetMeasurements supplies benchmark results, keyed model/profile, read
+// afresh each time it is called. Without them scaling goes by queue alone and
+// profiles are chosen by priority alone.
+func (r *Reconciler) SetMeasurements(f func() map[string]Measured) {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	r.measurements = f
+}
+
+// measured is a model's measurement on one profile.
+func (r *Reconciler) measured(model, profile string) (Measured, bool) {
+	r.mu.Lock()
+	f := r.measurements
+	r.mu.Unlock()
+	if f == nil {
+		return Measured{}, false
+	}
+	m, ok := f()[model+"/"+profile]
+	return m, ok && m.MaxConcurrency > 0
+}
+
+// orderedProfiles is a model's profiles as placement should try them: by
+// priority, and among equal priorities the one measured to serve the most per
+// GPU — or per core, for a profile with none — first. Unmeasured profiles keep
+// their catalog order behind measured ones of the same priority.
+//
+// Placement sorts by priority itself, so the order found here is written back
+// as priorities, descending, for it to keep.
+func (r *Reconciler) orderedProfiles(m catalog.Model) []placement.Profile {
+	ps := m.PlacementProfiles()
+	eff := make([]float64, len(ps))
+	for i, p := range ps {
+		meas, ok := r.measured(m.Name, p.Name)
+		if !ok {
+			continue
+		}
+		units := float64(max(p.GPUsPerNode, 0) * max(p.Nodes, 1))
+		if units == 0 {
+			units = float64(max(p.CPUCores, 1))
+		}
+		eff[i] = meas.TokPerSec / units
+	}
+	idx := make([]int, len(ps))
+	for i := range idx {
+		idx[i] = i
+	}
+	sort.SliceStable(idx, func(a, b int) bool {
+		pa, pb := ps[idx[a]], ps[idx[b]]
+		if pa.Priority != pb.Priority {
+			return pa.Priority > pb.Priority
+		}
+		return eff[idx[a]] > eff[idx[b]]
+	})
+	out := make([]placement.Profile, len(ps))
+	for rank, i := range idx {
+		out[rank] = ps[i]
+		out[rank].Priority = len(ps) - rank
+	}
+	return out
+}
+
+// MeasuredFor is the measurement that sizes a model: that of the profile its
+// replicas run on, or, with none running, of the first profile placement
+// would try that was measured.
+func (r *Reconciler) MeasuredFor(model string) (Measured, string, bool) {
+	r.mu.Lock()
+	profile := ""
+	for _, rep := range r.replicas {
+		if rep.Model == model && rep.State.Counts() {
+			profile = rep.Profile
+			break
+		}
+	}
+	r.mu.Unlock()
+	if profile != "" {
+		if m, ok := r.measured(model, profile); ok {
+			return m, profile, true
+		}
+	}
+	cm, ok := r.cat.Load().Resolve(model)
+	if !ok {
+		return Measured{}, "", false
+	}
+	for _, p := range r.orderedProfiles(cm) {
+		if m, ok := r.measured(cm.Name, p.Name); ok {
+			return m, p.Name, true
+		}
+	}
+	return Measured{}, "", false
+}

internal/control/reconciler/measured_test.go

@@ -0,0 +1,96 @@
+package reconciler
+
+import (
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+)
+
+func measure(h *harness, byKey map[string]Measured) {
+	h.r.SetMeasurements(func() map[string]Measured { return byKey })
+}
+
+func TestMoreRequestsThanWereMeasuredToMeetTheSLOIsBusy(t *testing.T) {
+	m := testModel(1)
+	m.Scaling.Max = 2
+	h := newHarness(t, m, gpuNode("1", 8), gpuNode("2", 8))
+	h.r.Reconcile(t.Context())
+	h.markReady()
+	// Six in flight, nothing waiting at the engine: by the queue it is fine.
+	load := map[string]Load{"llama": {Replicas: 1, Running: 6}}
+	hold := func() {
+		h.r.RecordLoad(load)
+		h.advance(61 * time.Second)
+		h.r.RecordLoad(load)
+		h.r.Reconcile(t.Context())
+	}
+	hold()
+	if n := liveCount(h); n != 1 {
+		t.Fatalf("scaled up with no measurement and an empty queue: %d", n)
+	}
+	// Measured to meet its SLO at four at once: six is past it.
+	measure(h, map[string]Measured{"llama/tp4": {MaxConcurrency: 4, TokPerSec: 200}})
+	hold()
+	if n := liveCount(h); n != 2 {
+		t.Errorf("%d replicas with 6 in flight against a measured 4", n)
+	}
+}
+
+func TestAReplicaIsTakenBackOnlyIfTheRestWouldHaveRoomToSpare(t *testing.T) {
+	m := testModel(1)
+	m.Scaling.Max = 2
+	h := newHarness(t, m, gpuNode("1", 8), gpuNode("2", 8))
+	measure(h, map[string]Measured{"llama/tp4": {MaxConcurrency: 4, TokPerSec: 200}})
+	h.r.Reconcile(t.Context())
+	h.markReady()
+	h.r.mu.Lock()
+	h.r.extra["llama"] = 1
+	h.r.mu.Unlock()
+	h.r.Reconcile(t.Context())
+	h.markReady()
+	if liveCount(h) != 2 {
+		t.Fatal("setup: not at two replicas")
+	}
+	quietFor := func(l Load) {
+		for range 3 {
+			h.r.RecordLoad(map[string]Load{"llama": l})
+			h.advance(DefaultConfig().ScaleDownAfter + time.Second)
+			h.r.RecordLoad(map[string]Load{"llama": l})
+			h.r.Reconcile(t.Context())
+		}
+	}
+	// Nothing waiting, but three running: one replica left would be at 3 of
+	// its 4, past 70%.
+	quietFor(Load{Replicas: 2, Running: 3})
+	if n := liveCount(h); n != 2 {
+		t.Errorf("scaled down to the edge of what one replica holds: %d", n)
+	}
+	quietFor(Load{Replicas: 2, Running: 2})
+	if n := liveCount(h); n != 1 {
+		t.Errorf("two running fits one replica's 70%% of 4; still %d", n)
+	}
+}
+
+func TestTheProfileMeasuredToServeMostPerGPUIsTriedFirst(t *testing.T) {
+	m := catalog.Model{Name: "llama", Source: "/models/llama", Scaling: catalog.Scaling{Min: 1, Max: 1},
+		Profiles: []catalog.Profile{
+			{Name: "tp4", GPUClass: "h100-80g", TP: 4},
+			{Name: "tp2", GPUClass: "h100-80g", TP: 2},
+			{Name: "tp8", GPUClass: "h100-80g", TP: 8, Priority: 10},
+		}}
+	h := newHarness(t, m, gpuNode("1", 8))
+	// tp4: 400/4 = 100 per GPU; tp2: 300/2 = 150. tp8 is unmeasured but has
+	// the higher priority, which an operator set and a benchmark does not
+	// overrule.
+	measure(h, map[string]Measured{"llama/tp4": {MaxConcurrency: 8, TokPerSec: 400},
+		"llama/tp2": {MaxConcurrency: 6, TokPerSec: 300}})
+	cm, _ := h.r.cat.Load().Resolve("llama")
+	var order []string
+	for _, p := range h.r.orderedProfiles(cm) {
+		order = append(order, p.Name)
+	}
+	if len(order) != 3 || order[0] != "tp8" || order[1] != "tp2" || order[2] != "tp4" {
+		t.Errorf("order %v, expected tp8 (priority), then tp2 (150/GPU), then tp4 (100/GPU)", order)
+	}
+}

internal/control/reconciler/preempt.go

@@ -0,0 +1,324 @@
+package reconciler
+
+import (
+	"context"
+	"slices"
+	"sort"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/placement"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
+)
+
+// hold keeps a model that gave up room from taking it back.
+type hold struct {
+	Until time.Time
+	For   string
+}
+
+// maxVictims bounds how many replicas one placement may stop. Past this the
+// model is better reported as not fitting than paid for with half a cluster.
+const maxVictims = 3
+
+// victim is a replica that could be stopped for another model.
+type victim struct {
+	ID       string
+	Model    string
+	Class    string
+	AboveMin bool
+	Tier     int
+	Idle     time.Duration
+	// Size is what stopping it frees, for choosing the smaller of two
+	// otherwise equal victims.
+	Size quota.Amount
+}
+
+// preemptFor stops replicas of lower classes when a model cannot otherwise be
+// placed, and says whether it did.
+//
+// What may be taken from whom:
+//
+//   - A lower class gives way to a higher one: opportunistic to standard,
+//     standard to critical.
+//   - Within a class, only a replica above its model's minimum, and only for
+//     a model still below its own. Room a model grew into under load is not a
+//     promise; its minimum is.
+//   - A critical model's minimum is never taken.
+//
+// Victims only ever come from a lower rank, or from above a minimum for a
+// model below one, so two models cannot take turns stopping each other.
+//
+// The set is tried on a copy of the inventory first, and only stopped if it
+// would let the model be placed: stopping replicas that do not make room is
+// the one outcome worse than not preempting.
+func (r *Reconciler) preemptFor(ctx context.Context, m catalog.Model, avoid []string) bool {
+	cands := r.victims(m)
+	if len(cands) == 0 {
+		return false
+	}
+	// Candidates in order until the model fits; then every one whose
+	// stopping turns out not to be needed is kept after all. Without that
+	// second step an idle replica on the wrong node, tried first, would be
+	// stopped along with the one that actually made the room.
+	var chosen []victim
+	for _, v := range cands {
+		if len(chosen) == maxVictims {
+			break
+		}
+		chosen = append(chosen, v)
+		if !r.fitsWithout(m, avoid, chosen) {
+			continue
+		}
+		for i := len(chosen) - 1; i >= 0 && len(chosen) > 1; i-- {
+			fewer := append(append([]victim(nil), chosen[:i]...), chosen[i+1:]...)
+			if r.fitsWithout(m, avoid, fewer) {
+				chosen = fewer
+			}
+		}
+		r.preempt(ctx, m, chosen)
+		return true
+	}
+	return false
+}
+
+// fitsWithout reports whether m could be placed with these replicas gone,
+// tried on a copy of the inventory.
+func (r *Reconciler) fitsWithout(m catalog.Model, avoid []string, gone []victim) bool {
+	trial := r.inv.Clone()
+	for _, v := range gone {
+		trial.ReleaseReplica(v.ID)
+	}
+	sched := placement.New(trial)
+	res := sched.Place(r.orderedProfiles(m), avoid)
+	if res.Placement == nil && len(avoid) > 0 {
+		res = sched.Place(r.orderedProfiles(m), nil)
+	}
+	return res.Placement != nil
+}
+
+// preempt drains the chosen replicas and holds their models back.
+func (r *Reconciler) preempt(ctx context.Context, m catalog.Model, chosen []victim) {
+	now := r.now()
+	var ids []string
+	r.mu.Lock()
+	for _, v := range chosen {
+		if rep, ok := r.replicas[v.ID]; ok {
+			rep.PreemptedFor = m.Name
+		}
+		ids = append(ids, v.ID)
+		r.held[v.Model] = hold{Until: now.Add(r.cfg.PreemptHold), For: m.Name}
+		r.preempted[v.Model]++
+	}
+	r.preempting[m.Name] = ids
+	r.mu.Unlock()
+
+	for _, v := range chosen {
+		r.log.Warn("preempting a replica to make room",
+			"replica", v.ID, "model", v.Model, "class", v.Class, "above_min", v.AboveMin,
+			"for", m.Name, "for_class", m.EffectiveClass())
+	}
+	r.scaleDown(ctx, ids)
+}
+
+// awaitingPreemption reports whether replicas stopped for a model are still
+// on their way out, so it waits for them rather than stopping more.
+func (r *Reconciler) awaitingPreemption(model string) bool {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	ids := r.preempting[model]
+	ids = slices.DeleteFunc(slices.Clone(ids), func(id string) bool {
+		_, ok := r.replicas[id]
+		return !ok
+	})
+	if len(ids) == 0 {
+		delete(r.preempting, model)
+		return false
+	}
+	r.preempting[model] = ids
+	return true
+}
+
+// isHeld reports whether a model gave up room and may not take it back yet.
+// The hold ends early once the model it was given to has what it asked for.
+//
+// Today a preempted replica is only removed by its own model's drain loop,
+// which runs after that model has decided not to scale up, so the room it
+// frees reaches the model it was freed for first anyway. The hold is what
+// keeps that true for any other way a replica can go — a stop reported
+// between passes, a reap at the start of one — rather than it resting on the
+// order of lines in reconcileModel.
+func (r *Reconciler) isHeld(model string) bool {
+	r.mu.Lock()
+	h, ok := r.held[model]
+	r.mu.Unlock()
+	if !ok {
+		return false
+	}
+	if r.now().After(h.Until) {
+		r.mu.Lock()
+		delete(r.held, model)
+		r.mu.Unlock()
+		return false
+	}
+	if to, ok := r.cat.Load().Resolve(h.For); ok && r.liveCount(h.For) >= r.desiredFor(to) {
+		r.mu.Lock()
+		delete(r.held, model)
+		r.mu.Unlock()
+		return false
+	}
+	return true
+}
+
+// liveCount is how many replicas of a model hold capacity.
+func (r *Reconciler) liveCount(model string) int {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	n := 0
+	for _, rep := range r.replicas {
+		if rep.Model == model && rep.State.Counts() && rep.RetiringAt.IsZero() {
+			n++
+		}
+	}
+	return n
+}
+
+// victims lists the replicas that may be stopped for m, in the order they
+// should be: idle on-demand ones first, then what other tenants borrowed past
+// their guarantees when m's owner is within its own, then what lower classes
+// grew into under load, then the same within m's class, then lower classes'
+// minimums; within each, the longest unused first.
+//
+// A replica within its owner's guarantee is taken only for a model of the
+// same owner: a guarantee that another tenant's higher class could take is
+// not one.
+func (r *Reconciler) victims(m catalog.Model) []victim {
+	cat := r.cat.Load()
+	now := r.now()
+	myRank := catalog.ClassRank(m.EffectiveClass())
+	belowMin := r.liveCount(m.Name) < m.Scaling.Min
+
+	pol := r.policy()
+	var borrowed map[string]bool
+	reclaim := false
+	if pol != nil {
+		hs := r.Holdings()
+		borrowed = pol.Borrowed(hs)
+		reclaim = pol.Guaranteed(m.Owner, pol.Usage(hs), profileNeed(m))
+	}
+
+	r.mu.Lock()
+	defer r.mu.Unlock()
+
+	// Per model, its live replicas newest first: the newest are the ones
+	// above its minimum.
+	byModel := map[string][]*Replica{}
+	for _, rep := range r.replicas {
+		if rep.Model == m.Name || !rep.State.Counts() || !rep.RetiringAt.IsZero() || rep.PreemptedFor != "" {
+			continue
+		}
+		byModel[rep.Model] = append(byModel[rep.Model], rep)
+	}
+
+	var out []victim
+	for model, reps := range byModel {
+		other, ok := cat.Resolve(model)
+		if !ok {
+			continue
+		}
+		class := other.EffectiveClass()
+		rank := catalog.ClassRank(class)
+		sort.Slice(reps, func(i, j int) bool { return reps[i].CreatedAt.After(reps[j].CreatedAt) })
+		above := max(len(reps)-other.Scaling.Min, 0)
+		for i, rep := range reps {
+			aboveMin := i < above
+			tier := -1
+			switch {
+			case rank < myRank && other.Scaling.OnDemand() && idleFor(rep, now) >= r.cfg.EvictionGrace:
+				tier = 0
+			case rank < myRank && aboveMin:
+				tier = 2
+			case rank == myRank && aboveMin && belowMin:
+				tier = 3
+			case rank < myRank && class != catalog.ClassCritical:
+				tier = 4
+			}
+			if pol != nil && other.Owner != m.Owner {
+				switch {
+				case borrowed[rep.ID] && reclaim && (class != catalog.ClassCritical || aboveMin):
+					// Past its owner's guarantee, wanted by an owner within
+					// its own: taken back whatever the classes.
+					if tier != 0 {
+						tier = 1
+					}
+				case !borrowed[rep.ID] && guaranteed(pol, other.Owner):
+					continue // held within its owner's guarantee
+				}
+			}
+			if tier < 0 {
+				continue
+			}
+			// Just started: stopping it throws away a model load that has
+			// only now been paid for.
+			if !rep.ReadyAt.IsZero() && now.Sub(rep.ReadyAt) < r.cfg.MinLifetime {
+				continue
+			}
+			if len(nodesOf(rep)) > 1 {
+				continue // see makeRoomFor: a multi-node replica is its own decision
+			}
+			prof, _ := other.ProfileByName(rep.Profile)
+			var size quota.Amount
+			for _, sh := range rep.Shards {
+				size = size.Plus(quota.Amount{GPUs: len(sh.GPUIndices), CPUCores: cpuCores(prof), MemoryGB: prof.MinMemoryGB})
+			}
+			out = append(out, victim{ID: rep.ID, Model: model, Class: class,
+				AboveMin: aboveMin, Tier: tier, Idle: idleFor(rep, now), Size: size})
+		}
+	}
+	sort.SliceStable(out, func(i, j int) bool {
+		a, b := out[i], out[j]
+		if a.Tier != b.Tier {
+			return a.Tier < b.Tier
+		}
+		if a.Idle != b.Idle {
+			return a.Idle > b.Idle
+		}
+		// The smaller first: the least served is lost if it is enough, and
+		// the pruning after keeps a larger one only when it is needed.
+		if a.Size.GPUs != b.Size.GPUs {
+			return a.Size.GPUs < b.Size.GPUs
+		}
+		if a.Size.MemoryGB != b.Size.MemoryGB {
+			return a.Size.MemoryGB < b.Size.MemoryGB
+		}
+		return a.ID < b.ID
+	})
+	return out
+}
+
+// idleFor is how long a replica has gone unused; zero when nobody has said,
+// which is not the same as idle.
+func idleFor(rep *Replica, now time.Time) time.Duration {
+	if rep.LastUsedAt.IsZero() {
+		return 0
+	}
+	return now.Sub(rep.LastUsedAt)
+}
+
+func nodesOf(rep *Replica) map[string]struct{} {
+	out := map[string]struct{}{}
+	for _, sh := range rep.Shards {
+		out[sh.NodeID] = struct{}{}
+	}
+	return out
+}
+
+// guaranteed reports whether an owner has a guarantee anywhere up its tree.
+func guaranteed(pol *quota.Policy, owner string) bool {
+	for _, q := range pol.Governing(owner) {
+		if q.Guarantee.Any() {
+			return true
+		}
+	}
+	return false
+}

internal/control/reconciler/preempt_test.go

@@ -0,0 +1,314 @@
+package reconciler
+
+import (
+	"log/slog"
+	"strings"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
+)
+
+// newFleetHarness is newHarness with a catalog of several models.
+func newFleetHarness(t *testing.T, models []catalog.Model, nodes ...inventory.Node) *harness {
+	t.Helper()
+	cat, err := catalog.New(models)
+	if err != nil {
+		t.Fatal(err)
+	}
+	inv := inventory.NewStore()
+	c := newClock()
+	for _, n := range nodes {
+		inv.Upsert(n)
+	}
+	disp := &fakeDispatcher{}
+	cfg := DefaultConfig()
+	cfg.UnreachableAfter = 15 * time.Second
+	cfg.LostAfter = 60 * time.Second
+	r := New(cfg, catalog.NewLive(cat), inv, disp, slog.New(slog.DiscardHandler))
+	r.SetClock(c.now)
+	c.advance(cfg.SettleFor)
+	for _, n := range nodes {
+		touchAt(inv, n.ID, c.now())
+	}
+	return &harness{r: r, inv: inv, disp: disp, clock: c}
+}
+
+// gpuModel runs on one whole 8-GPU node.
+func gpuModel(name, class string, min int) catalog.Model {
+	return catalog.Model{Name: name, Source: "/models/" + name, Class: class,
+		Scaling:  catalog.Scaling{Min: min, Max: min},
+		Profiles: []catalog.Profile{{Name: "tp8", Engine: "vllm", GPUClass: "h100-80g", TP: 8}}}
+}
+
+// settle runs passes until nothing changes, marking what starts ready and
+// letting preempted replicas run out their grace.
+func (h *harness) settle(t *testing.T, passes int) {
+	t.Helper()
+	for range passes {
+		h.r.Reconcile(t.Context())
+		h.markReady()
+		h.advance(h.r.cfg.PreemptGrace + time.Second)
+	}
+}
+
+func replicasOf(h *harness, model string) (live int) {
+	for _, rep := range h.r.Replicas() {
+		if rep.Model == model && rep.State.Counts() {
+			live++
+		}
+	}
+	return live
+}
+
+// fill places the standard model, then ages it past MinLifetime so it may be
+// taken from, and only then adds the model that wants its node.
+func fillThenWant(t *testing.T, want catalog.Model, have catalog.Model, nodes ...inventory.Node) *harness {
+	t.Helper()
+	h := newFleetHarness(t, []catalog.Model{have, withMin(want, 0)}, nodes...)
+	h.settle(t, 3)
+	if got := replicasOf(h, have.Name); got != have.Scaling.Min {
+		t.Fatalf("%s has %d replicas before anything wanted its room", have.Name, got)
+	}
+	h.advance(h.r.cfg.MinLifetime + time.Minute)
+	cat, err := catalog.New([]catalog.Model{have, want})
+	if err != nil {
+		t.Fatal(err)
+	}
+	h.r.cat.Store(cat)
+	return h
+}
+
+func withMin(m catalog.Model, n int) catalog.Model {
+	m.Scaling = catalog.Scaling{Min: n, Max: n}
+	return m
+}
+
+func TestACriticalModelTakesAStandardModelsNode(t *testing.T) {
+	h := fillThenWant(t, gpuModel("vital", catalog.ClassCritical, 1), gpuModel("chat", "", 1), gpuNode("1", 8))
+
+	h.r.Reconcile(t.Context())
+	var draining int
+	for _, rep := range h.r.Replicas() {
+		if rep.Model == "chat" && rep.State == Draining && rep.PreemptedFor == "vital" {
+			draining++
+		}
+	}
+	if draining != 1 {
+		t.Fatalf("the standard replica was not drained for the critical one: %+v", h.r.Replicas())
+	}
+	if hs := health(h, "chat"); hs.HeldFor != "vital" || hs.Preempted != 1 {
+		t.Errorf("the victim's health does not say it gave way: %+v", hs)
+	}
+
+	h.settle(t, 4)
+	if replicasOf(h, "vital") != 1 {
+		t.Fatalf("the critical model was not placed once room was made: %+v", h.r.Replicas())
+	}
+	// Held back until the critical model had its replica, then short again
+	// because there is no room: it must not have taken the node back — not
+	// even for a moment, which would load a model only to preempt it again.
+	if replicasOf(h, "chat") != 0 {
+		t.Error("the standard model took the node back")
+	}
+	started := 0
+	for _, cmd := range h.disp.startedShards() {
+		if strings.HasPrefix(cmd.ReplicaID, "chat-") {
+			started++
+		}
+	}
+	if started != 1 {
+		t.Errorf("the standard model started %d replicas; it should not have tried again", started)
+	}
+}
+
+func TestAStandardModelDoesNotTakeAnothersMinimum(t *testing.T) {
+	h := fillThenWant(t, gpuModel("second", "", 1), gpuModel("first", "", 1), gpuNode("1", 8))
+	h.settle(t, 3)
+	if replicasOf(h, "first") != 1 || replicasOf(h, "second") != 0 {
+		t.Errorf("a model of the same class took another's minimum: %+v", h.r.Replicas())
+	}
+	if hs := health(h, "second"); !hs.Unplaceable {
+		t.Error("the model left without room should say it cannot be placed")
+	}
+}
+
+func TestWithinAClassOnlyWhatWasGrownIntoIsTaken(t *testing.T) {
+	grown := gpuModel("busy", "", 1)
+	grown.Scaling = catalog.Scaling{Min: 1, Max: 2}
+	h := newFleetHarness(t, []catalog.Model{grown, withMin(gpuModel("new", "", 1), 0)}, gpuNode("1", 8), gpuNode("2", 8))
+	h.settle(t, 2)
+	// Load grows it to its max: the second replica is above its minimum.
+	h.r.mu.Lock()
+	h.r.extra["busy"] = 1
+	h.r.mu.Unlock()
+	h.settle(t, 3)
+	if replicasOf(h, "busy") != 2 {
+		t.Fatalf("setup: busy did not grow to 2: %+v", h.r.Replicas())
+	}
+	h.advance(h.r.cfg.MinLifetime + time.Minute)
+	cat, _ := catalog.New([]catalog.Model{grown, gpuModel("new", "", 1)})
+	h.r.cat.Store(cat)
+
+	h.settle(t, 5)
+	if replicasOf(h, "new") != 1 || replicasOf(h, "busy") != 1 {
+		t.Errorf("the grown replica should give way to a minimum: busy %d, new %d",
+			replicasOf(h, "busy"), replicasOf(h, "new"))
+	}
+}
+
+func TestNothingIsStoppedThatWouldNotMakeRoom(t *testing.T) {
+	// Two nodes, each half used by an opportunistic model; the critical one
+	// needs a whole node. Stopping one frees half a node, which is no use —
+	// and both together are on two nodes, which is no use either.
+	half := catalog.Model{Name: "half", Source: "/m", Class: catalog.ClassOpportunistic,
+		Scaling:  catalog.Scaling{Min: 2, Max: 2},
+		Profiles: []catalog.Profile{{Name: "tp4", Engine: "vllm", GPUClass: "h100-80g", TP: 4}}}
+	five := func(id string) inventory.Node { return gpuNode(id, 5) }
+	h := fillThenWant(t, gpuModel("vital", catalog.ClassCritical, 1), half, five("1"), five("2"))
+	h.r.Reconcile(t.Context())
+	for _, rep := range h.r.Replicas() {
+		if rep.PreemptedFor != "" || rep.State == Draining {
+			t.Fatalf("a replica was stopped without making room: %+v", rep)
+		}
+	}
+}
+
+func TestACriticalMinimumIsNeverTaken(t *testing.T) {
+	h := fillThenWant(t, gpuModel("also-vital", catalog.ClassCritical, 1), gpuModel("vital", catalog.ClassCritical, 1), gpuNode("1", 8))
+	h.settle(t, 3)
+	if replicasOf(h, "vital") != 1 {
+		t.Error("a critical model's minimum was taken")
+	}
+}
+
+func TestAReplicaThatJustStartedIsLeftAlone(t *testing.T) {
+	h := newFleetHarness(t, []catalog.Model{gpuModel("chat", "", 1), withMin(gpuModel("vital", catalog.ClassCritical, 1), 0)}, gpuNode("1", 8))
+	h.settle(t, 2)
+	cat, _ := catalog.New([]catalog.Model{gpuModel("chat", "", 1), gpuModel("vital", catalog.ClassCritical, 1)})
+	h.r.cat.Store(cat)
+	h.r.Reconcile(t.Context())
+	for _, rep := range h.r.Replicas() {
+		if rep.State == Draining {
+			t.Fatal("a replica that had only just loaded was preempted")
+		}
+	}
+}
+
+func health(h *harness, model string) ModelHealth {
+	for _, x := range h.r.ModelHealth() {
+		if x.Model == model {
+			return x
+		}
+	}
+	return ModelHealth{}
+}
+
+func TestAHoldLastsUntilTheRoomIsUsedOrItExpires(t *testing.T) {
+	h := fillThenWant(t, gpuModel("vital", catalog.ClassCritical, 1), gpuModel("chat", "", 1), gpuNode("1", 8))
+	h.r.Reconcile(t.Context())
+	if !h.r.isHeld("chat") {
+		t.Fatal("the model that gave way is not held")
+	}
+	if h.r.isHeld("vital") {
+		t.Error("the model room was made for is held")
+	}
+	// Released as soon as the model it was given to has what it asked for.
+	h.settle(t, 3)
+	if replicasOf(h, "vital") != 1 {
+		t.Fatalf("setup: vital was not placed: %+v", h.r.Replicas())
+	}
+	if h.r.isHeld("chat") {
+		t.Error("the hold outlasted its purpose")
+	}
+
+	// And, whatever happens, after PreemptHold.
+	h.r.mu.Lock()
+	h.r.held["other"] = hold{Until: h.r.now().Add(time.Minute), For: "never-placed"}
+	h.r.mu.Unlock()
+	if !h.r.isHeld("other") {
+		t.Fatal("a fresh hold is not in force")
+	}
+	h.advance(2 * time.Minute)
+	if h.r.isHeld("other") {
+		t.Error("a hold outlived its time")
+	}
+}
+
+func TestOnlyTheReplicasThatMakeTheRoomAreStopped(t *testing.T) {
+	// Two half-used nodes. "idle" on node 1 is on demand and long unused, so
+	// it is first in line; but the critical model needs node 2's other half,
+	// which only stopping "chat" frees.
+	idle := catalog.Model{Name: "idle", Source: "/m", Scaling: catalog.Scaling{Min: 0, Max: 1},
+		Profiles: []catalog.Profile{{Name: "tp4", Engine: "vllm", GPUClass: "h100-80g", TP: 4}}}
+	chat := catalog.Model{Name: "chat", Source: "/m", Scaling: catalog.Scaling{Min: 1, Max: 1},
+		Profiles: []catalog.Profile{{Name: "tp6", Engine: "vllm", GPUClass: "h100-80g", TP: 6}}}
+	vital := catalog.Model{Name: "vital", Source: "/m", Class: catalog.ClassCritical,
+		Scaling:  catalog.Scaling{Min: 1, Max: 1},
+		Profiles: []catalog.Profile{{Name: "tp8", Engine: "vllm", GPUClass: "h100-80g", TP: 8}}}
+	h := newFleetHarness(t, []catalog.Model{idle, chat}, gpuNode("1", 6), gpuNode("2", 8))
+	h.r.Want("idle")
+	h.settle(t, 4)
+	var idleNode, chatNode string
+	for _, rep := range h.r.Replicas() {
+		switch rep.Model {
+		case "idle":
+			idleNode = rep.Shards[0].NodeID
+		case "chat":
+			chatNode = rep.Shards[0].NodeID
+		}
+	}
+	if idleNode == "" || chatNode != "2" {
+		t.Fatalf("setup: idle on %q, chat on %q; chat must hold node 2: %+v", idleNode, chatNode, h.r.Replicas())
+	}
+	h.r.mu.Lock()
+	for _, rep := range h.r.replicas {
+		rep.LastUsedAt = h.r.now().Add(-time.Hour)
+	}
+	h.r.mu.Unlock()
+	h.advance(h.r.cfg.MinLifetime + time.Minute)
+	cat, _ := catalog.New([]catalog.Model{idle, chat, vital})
+	h.r.cat.Store(cat)
+
+	h.r.Reconcile(t.Context())
+	var stopped []string
+	for _, rep := range h.r.Replicas() {
+		if rep.PreemptedFor == "vital" {
+			stopped = append(stopped, rep.Model)
+		}
+	}
+	if len(stopped) != 1 || stopped[0] != "chat" {
+		t.Errorf("stopped %v; only chat's replica makes room for a whole node", stopped)
+	}
+}
+
+func TestOfTwoThatWouldEachMakeRoomTheSmallerIsStopped(t *testing.T) {
+	cpu := inventory.Node{ID: "a", MgmtAddr: "10.0.0.1", State: inventory.Ready, MemoryTotalMB: 31 * 1024, CPUCount: 8}
+	big := catalog.Model{Name: "big", Source: "/m", Scaling: catalog.Scaling{Min: 1, Max: 1},
+		Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 12, CPUCores: 4}}}
+	small := catalog.Model{Name: "small", Source: "/m", Scaling: catalog.Scaling{Min: 1, Max: 1},
+		Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 2, CPUCores: 1}}}
+	vital := catalog.Model{Name: "vital", Source: "/m", Class: catalog.ClassCritical,
+		Scaling:  catalog.Scaling{Min: 1, Max: 1},
+		Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 15, CPUCores: 4}}}
+	// 8 cores: big 4 + small 1 leaves 3, and vital needs 4. Stopping either
+	// is enough; stopping big loses far more.
+	for range 5 { // the choice must not depend on map order
+		h := newFleetHarness(t, []catalog.Model{big, small}, cpu)
+		h.settle(t, 3)
+		h.advance(h.r.cfg.MinLifetime + time.Minute)
+		cat, _ := catalog.New([]catalog.Model{big, small, vital})
+		h.r.cat.Store(cat)
+		h.r.Reconcile(t.Context())
+		var stopped []string
+		for _, rep := range h.r.Replicas() {
+			if rep.PreemptedFor == "vital" {
+				stopped = append(stopped, rep.Model)
+			}
+		}
+		if len(stopped) != 1 || stopped[0] != "small" {
+			t.Fatalf("stopped %v; the small replica alone makes room", stopped)
+		}
+	}
+}

internal/control/reconciler/quotas.go

@@ -0,0 +1,105 @@
+package reconciler
+
+import (
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/placement"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// SetQuotas supplies the tenancy tree's capacity quotas, read afresh each
+// time it is called. Without it nothing is guaranteed or limited, and
+// preemption goes by class alone.
+func (r *Reconciler) SetQuotas(f func() *quota.Policy) {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	r.quotas = f
+}
+
+func (r *Reconciler) policy() *quota.Policy {
+	r.mu.Lock()
+	f := r.quotas
+	r.mu.Unlock()
+	if f == nil {
+		return nil
+	}
+	return f()
+}
+
+// Holdings is what every replica holds, and for whom.
+func (r *Reconciler) Holdings() []quota.Holding {
+	cat := r.cat.Load()
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	out := make([]quota.Holding, 0, len(r.replicas))
+	for _, rep := range r.replicas {
+		m, ok := cat.Resolve(rep.Model)
+		if !ok {
+			continue
+		}
+		prof, _ := m.ProfileByName(rep.Profile)
+		var a quota.Amount
+		for _, sh := range rep.Shards {
+			a = a.Plus(quota.Amount{GPUs: len(sh.GPUIndices), CPUCores: cpuCores(prof), MemoryGB: prof.MinMemoryGB})
+		}
+		out = append(out, quota.Holding{Replica: rep.ID, Owner: m.Owner, Amount: a, Since: rep.CreatedAt})
+	}
+	return out
+}
+
+// placementNeed is what a placement would hold.
+func placementNeed(m catalog.Model, pl *placement.Placement) quota.Amount {
+	prof, _ := m.ProfileByName(pl.Profile.Name)
+	var a quota.Amount
+	for _, n := range pl.Nodes {
+		a = a.Plus(quota.Amount{GPUs: len(n.GPUIndices), CPUCores: cpuCores(prof), MemoryGB: prof.MinMemoryGB})
+	}
+	return a
+}
+
+// profileNeed is what a replica of a model's first profile would hold, for
+// deciding before there is a placement.
+func profileNeed(m catalog.Model) quota.Amount {
+	if len(m.Profiles) == 0 {
+		return quota.Amount{}
+	}
+	p := m.Profiles[0]
+	nodes := max(p.Nodes, 1)
+	gpus := p.GPUsPerNode
+	if gpus == 0 && !p.CPUOnly() {
+		gpus = max(p.TP, 1)
+	}
+	return quota.Amount{GPUs: gpus * nodes, CPUCores: cpuCores(p) * nodes, MemoryGB: p.MinMemoryGB * nodes}
+}
+
+// overLimit says why a placement would take its model's owner past a limit.
+func (r *Reconciler) overLimit(m catalog.Model, pl *placement.Placement) string {
+	pol := r.policy()
+	if pol == nil || m.Owner == "" {
+		return ""
+	}
+	return pol.AtLimit(m.Owner, pol.Usage(r.Holdings()), placementNeed(m, pl))
+}
+
+// QuotaUse is one scope's quota beside what it holds.
+type QuotaUse struct {
+	Scope     string
+	Guarantee store.Resources
+	Limit     store.Resources
+	Holds     quota.Amount
+}
+
+// QuotaUse lists every quota with what its scope holds now.
+func (r *Reconciler) QuotaUse() []QuotaUse {
+	pol := r.policy()
+	if pol == nil {
+		return nil
+	}
+	usage := pol.Usage(r.Holdings())
+	var out []QuotaUse
+	for _, q := range pol.Quotas() {
+		key := quota.Key(q.Scope, q.ScopeID)
+		out = append(out, QuotaUse{Scope: key, Guarantee: q.Guarantee, Limit: q.Limit, Holds: usage[key]})
+	}
+	return out
+}

internal/control/reconciler/quotas_test.go

@@ -0,0 +1,81 @@
+package reconciler
+
+import (
+	"strings"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+func owned(m catalog.Model, owner string) catalog.Model { m.Owner = owner; return m }
+
+func gpus(n int) store.Resources { return store.Resources{GPUs: &n} }
+
+// Two teams in one org; each 8-GPU node holds one tp8 replica.
+func withQuotas(h *harness, qs ...store.CapacityQuota) {
+	pol := quota.New(qs, map[string][]string{"team:ml": {"org:acme"}, "team:web": {"org:acme"}})
+	h.r.SetQuotas(func() *quota.Policy { return pol })
+}
+
+func TestATenantWithinItsGuaranteeTakesBackWhatAnotherBorrowed(t *testing.T) {
+	// web is guaranteed one node and borrowed the second; ml is guaranteed
+	// one node and wants it. Same class, and web's replica is at its minimum
+	// — by class alone nothing could be taken.
+	web := owned(gpuModel("web-chat", "", 2), "team:web")
+	ml := owned(gpuModel("ml-train", "", 1), "team:ml")
+	h := newFleetHarness(t, []catalog.Model{web, withMin(ml, 0)}, gpuNode("1", 8), gpuNode("2", 8))
+	withQuotas(h,
+		store.CapacityQuota{Scope: store.ScopeTeam, ScopeID: "web", Guarantee: gpus(8)},
+		store.CapacityQuota{Scope: store.ScopeTeam, ScopeID: "ml", Guarantee: gpus(8)})
+	h.settle(t, 4)
+	if replicasOf(h, "web-chat") != 2 {
+		t.Fatalf("setup: web did not borrow the second node: %+v", h.r.Replicas())
+	}
+	h.advance(h.r.cfg.MinLifetime + time.Minute)
+	cat, _ := catalog.New([]catalog.Model{web, ml})
+	h.r.cat.Store(cat)
+
+	h.settle(t, 5)
+	if replicasOf(h, "ml-train") != 1 || replicasOf(h, "web-chat") != 1 {
+		t.Errorf("the borrowed node was not taken back: web %d, ml %d",
+			replicasOf(h, "web-chat"), replicasOf(h, "ml-train"))
+	}
+}
+
+func TestAReplicaWithinItsGuaranteeIsNotTakenByAHigherClassElsewhere(t *testing.T) {
+	web := owned(gpuModel("web-chat", "", 1), "team:web")
+	vital := owned(gpuModel("vital", catalog.ClassCritical, 1), "team:ml")
+	h := fillThenWant(t, vital, web, gpuNode("1", 8))
+	withQuotas(h, store.CapacityQuota{Scope: store.ScopeTeam, ScopeID: "web", Guarantee: gpus(8)})
+	h.settle(t, 3)
+	if replicasOf(h, "web-chat") != 1 || replicasOf(h, "vital") != 0 {
+		t.Errorf("a guaranteed replica was preempted by another tenant's class: %+v", h.r.Replicas())
+	}
+	// Without the guarantee, critical takes it: the guarantee was the reason.
+	h2 := fillThenWant(t, vital, web, gpuNode("1", 8))
+	h2.settle(t, 3)
+	if replicasOf(h2, "vital") != 1 {
+		t.Error("setup: without a guarantee the critical model should have taken the node")
+	}
+}
+
+func TestALimitStopsAModelBeforeItStarts(t *testing.T) {
+	web := owned(gpuModel("web-chat", "", 2), "team:web")
+	h := newFleetHarness(t, []catalog.Model{web}, gpuNode("1", 8), gpuNode("2", 8))
+	withQuotas(h, store.CapacityQuota{Scope: store.ScopeOrg, ScopeID: "acme", Limit: gpus(8)})
+	h.settle(t, 4)
+	if replicasOf(h, "web-chat") != 1 {
+		t.Errorf("a second replica went past the org's 8-GPU limit: %+v", h.r.Replicas())
+	}
+	hs := health(h, "web-chat")
+	if !hs.Unplaceable || !strings.Contains(hs.Placement, "org:acme is at its capacity limit (16 GPUs of 8)") {
+		t.Errorf("the refusal should name the limit: %+v", hs)
+	}
+	room, _ := h.r.Headroom("web-chat", 5)
+	if room.More != 0 || len(room.Blocked) != 1 || !strings.Contains(room.Blocked[0], "capacity limit") {
+		t.Errorf("headroom should stop at the limit, not the empty node: %+v", room)
+	}
+}

internal/control/reconciler/reconciler.go

@@ -15,6 +15,7 @@ import (
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/placement"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
 )
 
 // ShardCommand is everything an agent needs to launch one shard.
@@ -65,6 +66,12 @@ type Config struct {
 	// Without a bound, draining a node whose model cannot be placed anywhere
 	// else would never finish.
 	DrainGrace time.Duration
+	// PreemptGrace is how long a replica stopped to make room for another
+	// is given to finish what it is serving: a request, not a replacement.
+	PreemptGrace time.Duration
+	// PreemptHold is how long a model that gave up room may not take it back,
+	// so the replica it was given up for is placed first.
+	PreemptHold time.Duration
 	// ScaleUpWaiting is the average number of requests waiting per ready
 	// replica, held for ScaleUpAfter, that adds a replica — up to the model's
 	// max. ScaleDownAfter is how long with nothing waiting before one of the
@@ -141,6 +148,8 @@ func DefaultConfig() Config {
 		EvictionGrace:    2 * time.Minute,
 		MinLifetime:      5 * time.Minute,
 		DrainGrace:       5 * time.Minute,
+		PreemptGrace:     30 * time.Second,
+		PreemptHold:      5 * time.Minute,
 		ParkAfter:        5,
 		ScaleUpWaiting:   1,
 		ScaleUpAfter:     time.Minute,
@@ -158,6 +167,12 @@ func (c *Config) normalize() {
 	if c.BaseBackoff <= 0 {
 		c.BaseBackoff = d.BaseBackoff
 	}
+	if c.PreemptGrace <= 0 {
+		c.PreemptGrace = d.PreemptGrace
+	}
+	if c.PreemptHold <= 0 {
+		c.PreemptHold = d.PreemptHold
+	}
 	if c.MaxBackoff < c.BaseBackoff {
 		c.MaxBackoff = max(c.BaseBackoff, d.MaxBackoff)
 	}
@@ -250,6 +265,16 @@ type Reconciler struct {
 	// placementLogged remembers the last "no placement" said per model, so it
 	// is said when it changes rather than on every five-second pass.
 	placementLogged map[string]placementNote
+	// preempting is, per model, the replicas stopped to make room for it and
+	// not gone yet; held is, per model that gave them up, until when it may
+	// not take the room back; preempted counts, per model, replicas it lost.
+	preempting map[string][]string
+	held       map[string]hold
+	preempted  map[string]int
+	// quotas is the tenancy tree's capacity quotas as last read; nil is none.
+	quotas func() *quota.Policy
+	// measurements are benchmark results by model/profile; nil is none.
+	measurements func() map[string]Measured
 
 	wake chan struct{}
 	stop chan struct{}
@@ -285,6 +310,9 @@ func New(cfg Config, cat *catalog.Live, inv *inventory.Store, disp Dispatcher, l
 		quietSince:      map[string]time.Time{},
 		placementLogged: map[string]placementNote{},
 		lastExplain:     map[string]placement.Result{},
+		preempting:      map[string][]string{},
+		held:            map[string]hold{},
+		preempted:       map[string]int{},
 		wake:            make(chan struct{}, 1),
 		stop:            make(chan struct{}),
 		done:            make(chan struct{}),
@@ -704,6 +732,9 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
 	switch {
 	case len(liveIDs) < desired && isParked:
 		// Not placed. Its failure is on every status the operator reads.
+	case len(liveIDs) < desired && r.isHeld(m.Name):
+		// It gave up room to a model of a higher class, which is placed
+		// first; taking it back now would undo the preemption.
 	case len(liveIDs) < desired:
 		if r.now().Before(blockedUntil) {
 			// Still in backoff. Rebuilding immediately would hot-loop on a
@@ -724,8 +755,12 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
 		r.mu.Lock()
 		rep, ok := r.replicas[id]
 		var expired bool
+		grace, preemptedFor := r.cfg.DrainGrace, ""
 		if ok {
-			expired = !rep.DrainingAt.IsZero() && r.now().Sub(rep.DrainingAt) > r.cfg.DrainGrace
+			if rep.PreemptedFor != "" {
+				grace, preemptedFor = r.cfg.PreemptGrace, rep.PreemptedFor
+			}
+			expired = !rep.DrainingAt.IsZero() && r.now().Sub(rep.DrainingAt) > grace
 		}
 		r.mu.Unlock()
 
@@ -733,11 +768,15 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
 		case !ok:
 		case replaced:
 			r.stopReplica(ctx, id)
+		case expired && preemptedFor != "":
+			r.log.Info("stopping a preempted replica", "replica", id,
+				"for", preemptedFor, "after", grace)
+			r.stopReplica(ctx, id)
 		case expired:
 			// The replacement never arrived — often because nowhere else can
 			// host it. Draining forever would leave the node undrainable.
 			r.log.Warn("stopping a drained replica without a replacement",
-				"replica", id, "after", r.cfg.DrainGrace)
+				"replica", id, "after", grace)
 			r.stopReplica(ctx, id)
 		}
 	}
@@ -751,12 +790,13 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
 // allocated, and starting several at once from the same inventory snapshot
 // would hand them the same GPUs.
 func (r *Reconciler) scaleUp(ctx context.Context, m catalog.Model, avoid []string) {
-	res := r.sched.Place(m.PlacementProfiles(), avoid)
+	profiles := r.orderedProfiles(m)
+	res := r.sched.Place(profiles, avoid)
 	if res.Placement == nil && len(avoid) > 0 {
 		// Spreading across failure domains is a preference, not a rule. If it
 		// is the only thing standing in the way, take the placement anyway:
 		// a second replica on one node still survives an engine crash.
-		res = r.sched.Place(m.PlacementProfiles(), nil)
+		res = r.sched.Place(profiles, nil)
 	}
 
 	r.mu.Lock()
@@ -775,9 +815,22 @@ func (r *Reconciler) scaleUp(ctx context.Context, m catalog.Model, avoid []strin
 		if m.Scaling.OnDemand() && r.makeRoomFor(ctx, m) {
 			return
 		}
+		if r.awaitingPreemption(m.Name) || r.preemptFor(ctx, m, avoid) {
+			return
+		}
 		r.notePlacementFailure(m.Name, res.Explain())
 		return
 	}
+	if why := r.overLimit(m, res.Placement); why != "" {
+		// Recorded as the placement's answer, so the model reads as not
+		// placeable, and says why, wherever that is shown.
+		r.mu.Lock()
+		r.lastExplain[m.Name] = placement.Result{Rejections: []placement.Rejection{
+			{Profile: res.Placement.Profile.Name, Reason: why}}}
+		r.mu.Unlock()
+		r.notePlacementFailure(m.Name, why)
+		return
+	}
 	if err := r.startReplica(ctx, m, res.Placement); err != nil {
 		r.log.Error("starting replica failed", "model", m.Name, "err", err)
 	}
@@ -825,7 +878,7 @@ func (r *Reconciler) startReplica(ctx context.Context, m catalog.Model, pl *plac
 		})
 		r.inv.Allocate(a.NodeID, inventory.Allocation{
 			ShardID: shardID, ReplicaID: id, GPUIndices: a.GPUIndices,
-			MemoryGB: prof.MinMemoryGB,
+			MemoryGB: prof.MinMemoryGB, CPUCores: cpuCores(prof),
 		})
 	}
 	rep.Endpoint = fmt.Sprintf("http://%s:%d", addrOf(leaderNode), basePort)
@@ -869,6 +922,7 @@ func (r *Reconciler) startReplica(ctx context.Context, m catalog.Model, pl *plac
 				HTTPPort:      s.HTTPPort,
 				Profile:       espec,
 				APIKey:        r.engineKey(id),
+				Threads:       cpuCores(prof),
 			},
 			StartupTimeout: prof.StartupTimeout,
 			Pool:           pl.Pool,
@@ -1108,16 +1162,31 @@ func addrOf(n inventory.Node) string {
 // claim on restart would place a second large model onto a node already full of
 // the first.
 func (r *Reconciler) profileMemory(model, profile string) int {
+	p, _ := r.profileOf(model, profile)
+	return p.MinMemoryGB
+}
+
+// profileOf is a replica's profile as the catalog holds it now.
+func (r *Reconciler) profileOf(model, profile string) (catalog.Profile, bool) {
 	m, ok := r.cat.Load().Resolve(model)
 	if !ok {
-		return 0
+		return catalog.Profile{}, false
 	}
 	for _, p := range m.Profiles {
 		if p.Name == profile {
-			return p.MinMemoryGB
+			return p, true
 		}
 	}
-	return 0
+	return catalog.Profile{}, false
+}
+
+// cpuCores is what a profile reserves of a node's CPUs: its cores when it
+// computes on them, nothing when a GPU does the work.
+func cpuCores(p catalog.Profile) int {
+	if !p.CPUOnly() {
+		return 0
+	}
+	return p.CPUCores
 }
 
 func (r *Reconciler) engineKey(replicaID string) string {

internal/control/reconciler/replica.go

@@ -156,6 +156,8 @@ type Replica struct {
 	// DrainingAt is when the replica stopped taking new work, so the wait for a
 	// replacement can be bounded.
 	DrainingAt time.Time
+	// PreemptedFor is the model this replica was stopped to make room for.
+	PreemptedFor string
 	// RetiringAt is set while the replica is being restarted: it keeps serving
 	// but no longer counts toward the model's replicas, so a replacement is
 	// placed beside it, and it is drained only once that replacement is ready.

internal/store/copy.go

@@ -29,6 +29,9 @@ var copyOrder = []string{
 	"key_claims",
 	"sessions",
 	"limits",
+	// Beside limits, and like them keyed by scope with no foreign key: a
+	// quota on a scope that is later removed is inert, not an error.
+	"capacity_quotas",
 	"usage",
 	"models",
 	// No foreign key to models, deliberately: a measurement outlives the

internal/store/migrations/postgres/0013_capacity_quotas.sql

@@ -0,0 +1,19 @@
+-- Capacity a part of the tenancy tree is guaranteed, and the most it may hold.
+--
+-- Beside limits rather than in it: limits are counted on the request path, per
+-- key, in requests and tokens; these are counted by the reconciler, per model,
+-- in the GPUs, cores and memory its replicas hold. A NULL is a resource this
+-- row does not govern: guaranteed nothing, or limited by nothing.
+CREATE TABLE capacity_quotas (
+  scope_type      TEXT NOT NULL,
+  scope_id        TEXT NOT NULL,
+  gpus            INTEGER,
+  cpu_cores       INTEGER,
+  memory_gb       INTEGER,
+  limit_gpus      INTEGER,
+  limit_cpu_cores INTEGER,
+  limit_memory_gb INTEGER,
+  updated_at      BIGINT NOT NULL,
+  updated_by      TEXT NOT NULL,
+  PRIMARY KEY (scope_type, scope_id)
+);

internal/store/migrations/sqlite/0013_capacity_quotas.sql

@@ -0,0 +1,19 @@
+-- Capacity a part of the tenancy tree is guaranteed, and the most it may hold.
+--
+-- Beside limits rather than in it: limits are counted on the request path, per
+-- key, in requests and tokens; these are counted by the reconciler, per model,
+-- in the GPUs, cores and memory its replicas hold. A NULL is a resource this
+-- row does not govern: guaranteed nothing, or limited by nothing.
+CREATE TABLE capacity_quotas (
+  scope_type      TEXT NOT NULL,
+  scope_id        TEXT NOT NULL,
+  gpus            INTEGER,
+  cpu_cores       INTEGER,
+  memory_gb       INTEGER,
+  limit_gpus      INTEGER,
+  limit_cpu_cores INTEGER,
+  limit_memory_gb INTEGER,
+  updated_at      BIGINT NOT NULL,
+  updated_by      TEXT NOT NULL,
+  PRIMARY KEY (scope_type, scope_id)
+);

internal/store/quotas.go

@@ -0,0 +1,127 @@
+package store
+
+import (
+	"context"
+	"database/sql"
+	"fmt"
+	"time"
+)
+
+// Resources are amounts of the three things a replica holds of a cluster.
+// Nil is not governed: nothing guaranteed, or nothing limited.
+type Resources struct {
+	GPUs     *int `json:"gpus,omitempty"`
+	CPUCores *int `json:"cpu_cores,omitempty"`
+	MemoryGB *int `json:"memory_gb,omitempty"`
+}
+
+// Any reports whether any resource is governed.
+func (r Resources) Any() bool { return r.GPUs != nil || r.CPUCores != nil || r.MemoryGB != nil }
+
+// CapacityQuota is what one part of the tenancy tree is guaranteed of the
+// cluster, and the most it may hold.
+type CapacityQuota struct {
+	Scope   Scope  `json:"scope"`
+	ScopeID string `json:"scope_id"`
+	// Guarantee is held for the scope: while it holds less, what others
+	// borrowed beyond their own guarantees is taken back for it.
+	Guarantee Resources `json:"guarantee"`
+	// Limit is the most it may hold, borrowing included.
+	Limit     Resources `json:"limit"`
+	UpdatedAt time.Time `json:"updated_at"`
+	UpdatedBy string    `json:"updated_by"`
+}
+
+// PutCapacityQuota sets a scope's quota, replacing what was there.
+func (s *DB) PutCapacityQuota(ctx context.Context, q CapacityQuota) error {
+	if !validMembershipScope(q.Scope) || q.ScopeID == "" {
+		return fmt.Errorf("a capacity quota belongs to an organization, team or project")
+	}
+	for _, v := range []*int{q.Guarantee.GPUs, q.Guarantee.CPUCores, q.Guarantee.MemoryGB,
+		q.Limit.GPUs, q.Limit.CPUCores, q.Limit.MemoryGB} {
+		if v != nil && *v < 0 {
+			return fmt.Errorf("a quota cannot be negative")
+		}
+	}
+	if q.UpdatedAt.IsZero() {
+		q.UpdatedAt = time.Now()
+	}
+	_, err := s.db.ExecContext(ctx,
+		`INSERT INTO capacity_quotas (scope_type, scope_id, gpus, cpu_cores, memory_gb,
+		   limit_gpus, limit_cpu_cores, limit_memory_gb, updated_at, updated_by)
+		 VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
+		 ON CONFLICT (scope_type, scope_id) DO UPDATE SET
+		   gpus = excluded.gpus, cpu_cores = excluded.cpu_cores, memory_gb = excluded.memory_gb,
+		   limit_gpus = excluded.limit_gpus, limit_cpu_cores = excluded.limit_cpu_cores,
+		   limit_memory_gb = excluded.limit_memory_gb,
+		   updated_at = excluded.updated_at, updated_by = excluded.updated_by`,
+		string(q.Scope), q.ScopeID,
+		nullInt(q.Guarantee.GPUs), nullInt(q.Guarantee.CPUCores), nullInt(q.Guarantee.MemoryGB),
+		nullInt(q.Limit.GPUs), nullInt(q.Limit.CPUCores), nullInt(q.Limit.MemoryGB),
+		q.UpdatedAt.UnixMilli(), q.UpdatedBy)
+	return err
+}
+
+// CapacityQuotas lists every quota.
+func (s *DB) CapacityQuotas(ctx context.Context) ([]CapacityQuota, error) {
+	return queryAll(ctx, s.db,
+		`SELECT scope_type, scope_id, gpus, cpu_cores, memory_gb,
+		   limit_gpus, limit_cpu_cores, limit_memory_gb, updated_at, updated_by
+		 FROM capacity_quotas ORDER BY scope_type, scope_id`, nil,
+		func(rows *sql.Rows) (CapacityQuota, error) {
+			var q CapacityQuota
+			var scope string
+			var g [6]sql.NullInt64
+			var at int64
+			err := rows.Scan(&scope, &q.ScopeID, &g[0], &g[1], &g[2], &g[3], &g[4], &g[5], &at, &q.UpdatedBy)
+			q.Scope = Scope(scope)
+			q.Guarantee = Resources{GPUs: intOf(g[0]), CPUCores: intOf(g[1]), MemoryGB: intOf(g[2])}
+			q.Limit = Resources{GPUs: intOf(g[3]), CPUCores: intOf(g[4]), MemoryGB: intOf(g[5])}
+			q.UpdatedAt = time.UnixMilli(at)
+			return q, err
+		})
+}
+
+// DeleteCapacityQuota removes a scope's quota.
+func (s *DB) DeleteCapacityQuota(ctx context.Context, scope Scope, id string) error {
+	_, err := s.db.ExecContext(ctx,
+		`DELETE FROM capacity_quotas WHERE scope_type = ? AND scope_id = ?`, string(scope), id)
+	return err
+}
+
+// ScopeParents maps every team and project to the scopes above it, nearest
+// first: a project to its team and organization, a team to its organization.
+// Keys and values are written scope:id.
+func (s *DB) ScopeParents(ctx context.Context) (map[string][]string, error) {
+	out := map[string][]string{}
+	teams, err := s.AllTeams(ctx)
+	if err != nil {
+		return nil, err
+	}
+	orgOf := map[string]string{}
+	for _, t := range teams {
+		orgOf[t.ID] = t.OrgID
+		out["team:"+t.ID] = []string{"org:" + t.OrgID}
+	}
+	type pt struct{ project, team string }
+	rows, err := queryAll(ctx, s.db, `SELECT id, team_id FROM projects`, nil,
+		func(r *sql.Rows) (pt, error) {
+			var p pt
+			return p, r.Scan(&p.project, &p.team)
+		})
+	if err != nil {
+		return nil, err
+	}
+	for _, p := range rows {
+		out["project:"+p.project] = []string{"team:" + p.team, "org:" + orgOf[p.team]}
+	}
+	return out, nil
+}
+
+func intOf(n sql.NullInt64) *int {
+	if !n.Valid {
+		return nil
+	}
+	v := int(n.Int64)
+	return &v
+}

internal/store/quotas_test.go

@@ -0,0 +1,63 @@
+package store
+
+import "testing"
+
+func ip(v int) *int { return &v }
+
+func TestAQuotaKeepsWhatItGovernsAndNothingElse(t *testing.T) {
+	s := tenancyStore(t)
+	org, _ := s.CreateOrg(t.Context(), "Acme")
+	q := CapacityQuota{Scope: ScopeOrg, ScopeID: org, UpdatedBy: "ops",
+		Guarantee: Resources{GPUs: ip(8)}, Limit: Resources{GPUs: ip(16), CPUCores: ip(0)}}
+	if err := s.PutCapacityQuota(t.Context(), q); err != nil {
+		t.Fatal(err)
+	}
+	all, err := s.CapacityQuotas(t.Context())
+	if err != nil || len(all) != 1 {
+		t.Fatalf("%v %+v", err, all)
+	}
+	got := all[0]
+	if *got.Guarantee.GPUs != 8 || got.Guarantee.CPUCores != nil || got.Guarantee.MemoryGB != nil {
+		t.Errorf("guarantee %+v: only GPUs were set", got.Guarantee)
+	}
+	// Zero is a limit of nothing, not an absent one.
+	if *got.Limit.GPUs != 16 || got.Limit.CPUCores == nil || *got.Limit.CPUCores != 0 {
+		t.Errorf("limit %+v", got.Limit)
+	}
+	q.Guarantee = Resources{MemoryGB: ip(64)}
+	if err := s.PutCapacityQuota(t.Context(), q); err != nil {
+		t.Fatal(err)
+	}
+	all, _ = s.CapacityQuotas(t.Context())
+	if len(all) != 1 || all[0].Guarantee.GPUs != nil || *all[0].Guarantee.MemoryGB != 64 {
+		t.Errorf("a second put should replace the first: %+v", all)
+	}
+	if err := s.DeleteCapacityQuota(t.Context(), ScopeOrg, org); err != nil {
+		t.Fatal(err)
+	}
+	if all, _ = s.CapacityQuotas(t.Context()); len(all) != 0 {
+		t.Error("the quota survived its deletion")
+	}
+	if err := s.PutCapacityQuota(t.Context(), CapacityQuota{Scope: ScopeOrg, ScopeID: org,
+		Guarantee: Resources{GPUs: ip(-1)}}); err == nil {
+		t.Error("a negative quota was stored")
+	}
+}
+
+func TestEveryScopeKnowsWhatIsAboveIt(t *testing.T) {
+	s := tenancyStore(t)
+	org, _ := s.CreateOrg(t.Context(), "Acme")
+	team, _ := s.CreateTeam(t.Context(), org, "ml")
+	proj, _ := s.CreateProject(t.Context(), team, "training")
+	p, err := s.ScopeParents(t.Context())
+	if err != nil {
+		t.Fatal(err)
+	}
+	want := []string{"team:" + team, "org:" + org}
+	if got := p["project:"+proj]; len(got) != 2 || got[0] != want[0] || got[1] != want[1] {
+		t.Errorf("project's parents %v, expected %v", got, want)
+	}
+	if got := p["team:"+team]; len(got) != 1 || got[0] != "org:"+org {
+		t.Errorf("team's parents %v", got)
+	}
+}

internal/web/admin_serve.go

@@ -49,6 +49,9 @@ type serveForm struct {
 	// 3.5 GB for a KV pool. Nothing was wrong with placement — it was never
 	// told what the model needed.
 	MinMemoryGB int
+	// CPUCores is the cores a replica computes on: half the smallest node it
+	// fits on, or zero when no node says how many it has.
+	CPUCores int
 	// MemoryBasis says how MinMemoryGB was arrived at, for the form to show.
 	MemoryBasis string
 	// CtxPool is the engine context pool the form starts with, and the one
@@ -152,6 +155,15 @@ func planServe(st mirror.Status, files []mirror.Status, base string,
 			defaultCtxPool, header.BlockCount, kvHeads,
 			float64(int64(1<<30)+weights/20)/(1<<30))
 	}
+	if fleet != nil {
+		var cpus []int
+		for _, n := range fleet.Nodes() {
+			if n.State == "ready" && (n.MemoryGB == 0 || n.MemoryGB >= f.MinMemoryGB) {
+				cpus = append(cpus, n.CPUCount)
+			}
+		}
+		f.CPUCores = catalog.SuggestCores(cpus)
+	}
 	if biggest := biggestNode(fleet); biggest > 0 && f.MinMemoryGB > biggest {
 		f.Warning = strings.TrimSuffix(f.Warning, ".") +
 			fmt.Sprintf("%sthe largest node has %d GB and this needs about %d",
@@ -277,6 +289,11 @@ func buildModel(get func(string) string) (catalog.Model, error) {
 			return m, err
 		}
 	}
+	if v := strings.TrimSpace(get("cpu_cores")); v != "" {
+		if p.CPUCores, err = positive(v, "the cores"); err != nil {
+			return m, err
+		}
+	}
 	if k, v := strings.TrimSpace(get("selector_key")), strings.TrimSpace(get("selector_value")); k != "" && v != "" {
 		// What keeps a large model off the small nodes. A CPU profile has no
 		// card to be short of, so without this every ready node looks like a

internal/web/node_page_test.go

@@ -7,6 +7,7 @@ import (
 	"time"
 
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/nvlink"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/topology"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/promq"
@@ -174,3 +175,33 @@ func TestTheNVLinkPageDrawsEachDomainAsARack(t *testing.T) {
 		t.Error("the cluster page does not lead to the racks")
 	}
 }
+
+// hotSource is a GPU node whose sensors report, one card held back.
+type hotSource struct{ fakeSource }
+
+func (hotSource) Nodes() []adminapi.NodeView {
+	n := fakeSource{}.Nodes()[0]
+	n.Load = &adminapi.LoadView{CPUPercent: 40, CPUs: 64, MemTotal: 1 << 40, MemUsed: 1 << 39,
+		CPUTempC: 58, CPUPowerW: 312, InletTempC: 24.5,
+		GPUs: []inventory.GPUSensor{
+			{Index: 0, TempC: 71, MemoryTempC: 84, PowerW: 612, PowerLimitW: 700},
+			{Index: 1, TempC: 88, PowerW: 698, PowerLimitW: 700, Throttle: []string{"sw_power_cap", "sw_thermal"}},
+		}}
+	return []adminapi.NodeView{n}
+}
+
+func TestANodesSensorsAreOnItsPageAndNothingItCannotMeasure(t *testing.T) {
+	_, body := get(t, nodeMux(hotSource{}, nil), "http://x/admin/nodes/llmgw-node01", nil)
+	for _, want := range []string{"58 °C", "312 W", "24.5 °C", "inlet air", "84 °C", "612 W", "of 700",
+		`<span class="bad">sw_power_cap, sw_thermal</span>`} {
+		if !strings.Contains(body, want) {
+			t.Errorf("the page lacks %q", want)
+		}
+	}
+	_, vm := get(t, nodeMux(busySource{}, nil), "http://x/admin/nodes/llmgw-node01", nil)
+	for _, absent := range []string{"°C", "<h2>GPUs</h2>", "inlet air"} {
+		if strings.Contains(vm, absent) {
+			t.Errorf("a node with no sensors shows %q", absent)
+		}
+	}
+}

internal/web/static/base.css

@@ -854,3 +854,4 @@ table.explore td:first-child { max-width: 34rem; overflow: hidden; text-overflow
 table.explore .from { margin-left: .5rem; font-size: .78rem; color: var(--muted); }
 .clip-inline { display: inline-block; max-width: 24rem; overflow: hidden; text-overflow: ellipsis;
                vertical-align: bottom; font-size: .8rem; color: var(--muted); margin-left: .4rem; }
+table.rows td.warn { color: var(--warn); cursor: help; }

internal/web/templates/admin_mirror.html

@@ -165,7 +165,9 @@ reachable from this machine.</p>
   <div class="field-row">
     <label>Profile name <input type="text" name="profile" value="{{.Profile}}"></label>
     <label>Needs RAM (GB) <span class="opt">for the context pool beside it</span>
-      <input type="text" name="min_memory_gb" value="{{.MinMemoryGB}}"></label>
+      <input type="text" name="min_memory_gb" value="{{.MinMemoryGB}}" required></label>
+    <label>CPU cores <span class="opt">reserved; the engine's threads</span>
+      <input type="text" name="cpu_cores" value="{{.CPUCores}}"></label>
     <label>Only nodes labelled <span class="opt">key / value, optional</span>
       <input type="text" name="selector_key" placeholder="role" size="8">
       <input type="text" name="selector_value" placeholder="cpu-large" size="10"></label>

internal/web/templates/dashboard.html

@@ -80,12 +80,15 @@
 <h2>Models</h2>
 
 {{if .Models}}
-<table class="rows"><tr><th>Model</th><th>Aliases</th><th class="num">Ready</th><th class="num">Wanted</th><th>Profiles</th>{{if .AnyFallbacks}}<th>Fallbacks</th>{{end}}<th>What it is for</th></tr>
+<table class="rows"><tr><th>Model</th><th>Aliases</th><th class="num">Ready</th><th class="num">Wanted</th><th>Class</th><th>Reserves</th><th>Profiles</th>{{if .AnyFallbacks}}<th>Fallbacks</th>{{end}}<th>What it is for</th></tr>
 {{range .Models}}<tr>
   <td><code>{{.Name}}</code></td>
   <td>{{join .Aliases}}</td>
   <td class="num">{{.Ready}}</td>
   <td class="num">{{.Min}}–{{.Max}}</td>
+  <td{{if .HeldFor}} class="warn" title="gave room to {{.HeldFor}}"{{end}}>{{.Class}}{{if .Preempted}} <span class="note" title="replicas lost to a higher class">−{{.Preempted}}</span>{{end}}</td>
+  <td{{with .Warnings}} class="warn" title="{{join .}}"{{end}}>{{if or .MemoryGB .CPUCores}}{{.MemoryGB}}G{{if .CPUCores}} · {{.CPUCores}}c{{end}}{{else}}—{{end}}
+    {{- if .MeasuredMemoryGB}} <span class="note">{{printf "%.1f" .MeasuredMemoryGB}}G used</span>{{end}}{{if .Warnings}} ⚠{{end}}</td>
   <td>{{join .Profiles}}</td>
   {{if $.AnyFallbacks}}<td>{{join .Fallbacks}}</td>{{end}}
   {{/* Cut to the column, the whole sentence on hover: a row of its own for

internal/web/templates/node.html

@@ -21,6 +21,10 @@
   <div class="tile"><div class="v">{{printf "%.0f" .CPUPercent}}%</div><div class="k">CPU{{if .CPUs}} · {{.CPUs}} cores{{end}}</div>{{template "minibar" pctbar .CPUPercent}}</div>
   <div class="tile"><div class="v">{{bytes .MemUsed}}</div><div class="k">RAM in use · of {{bytes .MemTotal}}</div>{{template "minibar" pctbar .MemPercent}}</div>
   <div class="tile"><div class="v">{{printf "%.2f" .Load1}}</div><div class="k">load · {{printf "%.2f" .Load5}} 5m · {{printf "%.2f" .Load15}} 15m</div></div>
+  {{/* Only what this machine has sensors for: a VM has none, and a tile
+       saying 0 °C would be a reading nobody took. */}}
+  {{if or .CPUTempC .CPUPowerW}}<div class="tile"><div class="v">{{if .CPUTempC}}{{printf "%.0f" .CPUTempC}} °C{{else}}—{{end}}</div><div class="k">CPU{{if .CPUPowerW}} · {{printf "%.0f" .CPUPowerW}} W{{end}}</div></div>{{end}}
+  {{if .InletTempC}}<div class="tile"><div class="v">{{printf "%.1f" .InletTempC}} °C</div><div class="k">inlet air</div></div>{{end}}
   {{else}}
   <div class="tile"><div class="v">—</div><div class="k">CPU and RAM not measured lately</div></div>
   {{end}}
@@ -29,6 +33,17 @@
 </div>
 {{if .Warnings}}{{range .Warnings}}<p class="note bad">{{.}}</p>{{end}}{{end}}
 {{if .Unhealthy}}<p class="note bad">{{join .Unhealthy}}</p>{{end}}
+{{with .Load}}{{if .GPUs}}
+<h2>GPUs</h2>
+<table class="rows"><tr><th class="num">GPU</th><th class="num">Temperature</th><th class="num">Memory</th><th>Power</th><th>Held back by</th></tr>
+{{range .GPUs}}<tr>
+  <td class="num">{{.Index}}</td>
+  <td class="num">{{if .TempC}}{{printf "%.0f" .TempC}} °C{{else}}—{{end}}</td>
+  <td class="num">{{if .MemoryTempC}}{{printf "%.0f" .MemoryTempC}} °C{{else}}—{{end}}</td>
+  <td class="load">{{if and .PowerW .PowerLimitW}}{{template "minibar" pctbar (percent .PowerW .PowerLimitW)}}<span class="load-v">{{printf "%.0f" .PowerW}} W <span class="note">of {{printf "%.0f" .PowerLimitW}}</span></span>{{else if .PowerW}}{{printf "%.0f" .PowerW}} W{{else}}—{{end}}</td>
+  <td>{{if .Throttle}}<span class="bad">{{join .Throttle}}</span>{{else}}<span class="note">nothing</span>{{end}}</td>
+</tr>{{end}}</table>
+{{end}}{{end}}
 {{end}}
 
 {{if or .Node.IBPorts .Node.Fabrics}}

internal/web/web.go

@@ -109,6 +109,13 @@ var funcs = template.FuncMap{
 		return p
 	},
 	"ago": ago,
+	// percent is part of whole, 0 to 100, for a bar.
+	"percent": func(part, whole float64) float64 {
+		if whole <= 0 {
+			return 0
+		}
+		return part * 100 / whole
+	},
 	// shortNode is a node's name without what every node's name starts with.
 	"shortNode": strings.TrimPrefix,
 	"age":       age,

llmgw-plan.md

@@ -1463,8 +1463,8 @@ The launch path is now built too:
 - [x] `ctl bench`: a load ramp that reports capacity as the last concurrency
       level meeting the SLO, plus the queue depth and KV usage observed there,
       so `weight`, `target_waiting` and `kv_ceiling` stop being guesses
-- [ ] `ctl plan <model> --target-rps N`, which needs capacity profiles stored
-      rather than printed
+- [x] Sizing from stored capacity: `ctl model target <model> --tokens-per-sec N`,
+      and the autoscaler and profile choice use what `ctl bench --save` measured
 - [x] OTel tracing: an inference span with admission, schedule and each
       upstream attempt under it, over OTLP/HTTP when `tracing.endpoint` is
       set, and the traceparent carried on to the engine
@@ -1480,7 +1480,17 @@ The launch path is now built too:
 - [ ] Admission displacement (dropping requests from lower bands)
 - [x] Autoscaling on load: a replica added while requests wait, up to a
       model's max, and taken back after ten quiet minutes
-- [ ] Priority-based preemption
+- [x] Priority-based preemption: capacity classes (critical, standard,
+      opportunistic), preemption tried on a copy of the inventory first
+- [x] Capacity quotas per organization, team and project: guarantees kept,
+      borrowing taken back, limits enforced
+- [x] CPU cores reserved like memory, and memory required of CPU models
+- [x] Node sensors collected: GPU temperature, power and throttling, CPU
+      temperature and RAPL power, board inlet air; published and shown
+- [ ] Rack power, temperature and cooling as placement signals: planned in
+      [docs/facility-aware-placement.md](docs/facility-aware-placement.md),
+      deferred — it needs each site's rack, PDU and power data first
+- [ ] Defragmentation for large tensor-parallel replicas across NVLink domains
 - [ ] Control plane HA (Raft)
 - [x] Weight downloading and cache management: the mirror, agent fetches
       checked against their digest, and reclaiming what nothing uses