Displace the newest lowest-band request when the gateway queue is full

A total bound across bands (admission.queue_capacity_total, default half of
all bands' capacity together); reached, a request is not refused while a
lower band has requests queued, and the newest of the lowest band gets a 503
with Retry-After instead.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

.gitea/workflows/release.yml

@@ -93,6 +93,24 @@ jobs:
           ./dist/llmgw-${version}-linux-amd64 --version
           cat dist/checksums.txt
 
+      # Signed when the repository has a signing key, so a machine that trusts
+      # it installs nothing the server alone could have put there. Without the
+      # secret nothing is signed and releases are checked on their checksums,
+      # as before; a build that trusts a key refuses those, so the secret has
+      # to be in place before trusted.pub is filled in.
+      - name: Sign the checksums
+        env:
+          RELEASE_SIGNING_KEY: ${{ secrets.RELEASE_SIGNING_KEY }}
+        run: |
+          set -euo pipefail
+          version="${GITHUB_REF_NAME:-$(git describe --tags --always)}"
+          if [ -z "${RELEASE_SIGNING_KEY:-}" ]; then
+            echo "no RELEASE_SIGNING_KEY secret; releases are not signed"
+            exit 0
+          fi
+          ./dist/llmgw-${version}-linux-amd64 release sign dist/checksums.txt
+          echo "signed: dist/checksums.txt.sig"
+
       - name: Attach to the release
         env:
           TOKEN: ${{ secrets.GITEA_TOKEN }}

CHANGELOG.md

@@ -20,6 +20,251 @@ is only released once its **exit criterion** is met, not when its code lands.
 Until v1.0.0 the minor version is the breaking one: configuration formats and Go
 APIs change between minors without a deprecation period.
 
+## Unreleased
+
+Changes since v0.10.5, published as release candidates (v0.10.6-rc.N) until
+there are enough for v0.10.6.
+
+### Capacity from the admin pages
+
+What v0.10.5 made settable from the command line is now on the pages too:
+
+- **Quotas** has a Capacity section: every scope's guaranteed and most GPUs,
+  cores and memory beside what it holds, with a form to set, change or remove
+  one. It goes through the same checks as `llmgw ctl quota`.
+- **A model's card** on the mirror page sets its class, its owner (when there
+  are quotas to count against), and the memory and cores a replica reserves,
+  beside its replica counts. Only what changed is applied, and each change is
+  said.
+- **Room counts what a class may take.** The card, `llmgw ctl model target` and
+  the headroom API count the replicas a model could get by stopping those its
+  class outranks, separately from free room, and name whose they would be. A
+  critical model's card no longer refuses a replica that preemption would make
+  room for.
+
+### Displacement: a higher band is not refused while a lower one waits
+
+Each band's queue was bounded, and nothing bounded them together, so a full
+gateway turned an interactive request away while batch requests sat queued
+behind it. `admission.queue_capacity_total` now bounds everything queued —
+by default half of every band's capacity together. Reached, a request is not
+refused while a lower band has any queued: the newest request of the lowest
+band is turned away instead, with a 503 and `Retry-After: 5`, so it backs off
+and returns. The newest, because the oldest have waited longest and are
+nearest their TTL. Counted as `displaced` in the admission rejection metric.
+
+### Releases can be signed
+
+`llmgw release keygen` makes an ed25519 key pair. With the private key in the
+CI secret `RELEASE_SIGNING_KEY`, releases carry `checksums.txt.sig`; with the
+public key in `internal/release/trusted.pub`, every binary built after refuses
+a release not signed with it, wherever it installs one. Off until then, and
+docs/install.md says in which order to turn it on.
+
+### Restarts borrow a lower class's room
+
+A restart places the replacement beside the replica it replaces, and where
+there is no free room it now makes some the way any placement does: by
+stopping a replica its class outranks, which gets its room back once the old
+replica has gone. The in-place restart — the model unserved while its
+replacement loads — is left for when nothing of a lower class is running. On a
+cluster where every model is standard, that is every full one: mark the models
+that can wait `opportunistic` (`llmgw ctl model class`, or the model's card).
+
+## v0.10.5 — 2026-09-30
+
+Published first as release candidates v0.10.5-rc.1 to rc.14; rc.5 failed its
+format check and has no binaries.
+
+After upgrading:
+
+- **Declare what CPU models need.** A CPU model added from now on must say its
+  memory; one stored before this still loads and is flagged in `ctl models`
+  until `llmgw ctl model resources MODEL --memory GB --cores N` is run. Cores
+  are then reserved per replica and become the engine's thread count, for
+  replicas started after the change: `llmgw ctl restart --model MODEL`.
+- **Upgrade the mirror on its own host**, now with one command:
+  `sudo llmgw upgrade v0.10.5 --service llmgw-mirror` (by hand once, for a
+  mirror older than this). `llmgw ctl upgrade` says so at the end.
+
+### CPU cores are reserved, and memory has to be declared
+
+The first step of resource allocation for larger clusters: what a replica
+holds of a node is reserved, not assumed.
+
+- **Cores.** A profile's `cpu_cores` is reserved out of its node's CPUs, as
+  memory is, and is what its engine computes on (llama.cpp's `--threads`). A
+  node holds as many CPU replicas as it has cores for, and two on one node no
+  longer spin against each other. Agents report their CPU count; a node that
+  does not, and a profile that declares no cores, constrain nothing.
+- **Memory is required of new CPU models.** A model with no GPU is all host
+  memory, and one declaring none looked to placement as if it fitted
+  everywhere, any number of times. Adding one without `min_memory_gb` is now
+  refused. Models stored before this still load, and say what is missing.
+- **Declared against measured.** Each model shows what one replica reserves
+  and the most one of its replicas uses — in `ctl models` (RESERVES) and on the
+  cluster page — with a warning when nothing is declared or when it uses more
+  than a tenth over what it reserves.
+- **Setting them.** `llmgw ctl model resources MODEL --memory GB --cores N`
+  (and `POST /api/v1/models/{name}/resources`) for a model added at runtime.
+  `ctl model serve` and the mirror's serve form fill in cores — half the
+  smallest node the model fits on — beside the memory they already estimate.
+  Running replicas keep their old figures until restarted.
+
+The commands are `llmgw ctl model resources` and `llmgw ctl model class`; in
+rc.7 the first was briefly `llmgw ctl resources`.
+
+### Capacity classes, and preemption
+
+A model has a class — `critical`, `standard` or `opportunistic` — saying what
+its capacity is worth against another's when there is not enough for both.
+Unset, an on-demand model is opportunistic and every other model standard,
+which is how the catalog behaved before. `llmgw ctl model class MODEL CLASS`
+sets it.
+
+When a model cannot be placed, replicas are stopped to make room for it, by
+these rules:
+
+- A lower class gives way to a higher one.
+- Within a class, only replicas above a model's minimum — room it grew into
+  under load — and only for a model still below its own minimum.
+- A critical model's minimum is never taken, nor a replica that has only just
+  loaded, nor one spanning several nodes.
+- Idle on-demand replicas go first, then what lower classes grew into, then
+  the same within the class, then lower classes' minimums; the longest unused
+  first within each, and of equally idle ones the smallest.
+- The set is tried on a copy of the inventory first, at most three replicas,
+  and nothing is stopped unless it makes the room. Of a set that does, each
+  replica whose stopping turns out not to be needed is left running, so an
+  idle replica tried first on the wrong node is not stopped with the one that
+  made the room.
+
+A replica stopped this way is drained and given 30 seconds to finish what it
+is serving. Its model is held back for up to five minutes, or until the model
+it gave way to has its replicas, so it does not take the room straight back.
+`ctl models` shows each model's class and "gave room to X" while held; the
+cluster page shows the class and how many replicas were lost; the metric is
+`llmgw_model_preempted_total{model,class}`, and every preemption is logged.
+
+### Capacity quotas for the tenancy tree
+
+An organization, team or project can be guaranteed GPUs, CPU cores and
+memory, and limited to at most some amount of each. What it holds is what the
+replicas of the models it owns hold — and those of the teams and projects
+beneath it. `llmgw ctl model owner MODEL team:Acme/ml` says whose a model is;
+one with no owner is shared and counts against nobody.
+
+- **A guarantee is kept.** A replica within its owner's guarantee is never
+  taken for another scope's model, whatever the classes: a guarantee a higher
+  class elsewhere could take would not be one.
+- **Beyond it is borrowing.** Free room is used; when a scope within its own
+  guarantee needs room, what others hold beyond theirs is taken back first,
+  whatever their class — except a critical model's minimum.
+- **A limit is a ceiling.** A model whose owner would go past a limit anywhere
+  up the tree gets no further replica, and says which limit; headroom and the
+  mirror card stop there too.
+
+`llmgw ctl quota set team:Acme/ml --gpus 8 --limit-gpus 16` (also `--cores`,
+`--memory` and their limits), `llmgw ctl quota list` with what each scope holds,
+`llmgw ctl quota clear`. Scopes are named by path or ID; a name that could mean
+two is refused. Changes take effect within half a minute. Metrics:
+`llmgw_capacity_holds`, `llmgw_capacity_guarantee` and `llmgw_capacity_limit`
+by scope and resource; alert: CapacityAtLimit.
+
+### Sized by what a replica was measured to serve
+
+What `llmgw ctl bench MODEL --save` measured — the most requests at once one
+replica served while meeting its SLO, and its throughput there — now drives
+decisions rather than only being printed:
+
+- **Scaling up** when requests per replica pass what was measured, not only
+  when the engines' queues fill: an engine takes requests into its slots long
+  after latency has gone.
+- **Scaling down** only when the replicas left would stay under 70% of what
+  they were measured to hold, so a replica taken back does not put the rest
+  straight over.
+- **Choosing a profile.** Among profiles of equal priority, the one measured
+  to serve the most per GPU — or per core — is tried first. Priority set by an
+  operator still wins.
+- **Sizing.** `llmgw ctl model target MODEL --tokens-per-sec N` says how many
+  replicas that takes, how many run, and whether the rest fit, with the reason
+  when they do not; `--apply` sets it if they do. Headroom reports what one
+  more replica would serve.
+
+A model never measured behaves exactly as before. Measurements are read from
+the store every half minute.
+
+### Upgrading a machine the control plane does not reach
+
+`sudo llmgw upgrade VERSION --service UNIT` upgrades the machine it runs on —
+the mirror's host above all, whose service runs unprivileged and cannot replace
+its own binary as an agent does. The release's checksums are checked, the old
+binary kept, the unit restarted and watched until it stays up.
+
+### Nodes report their own heat and power
+
+The first step of facility-aware placement (docs/facility-aware-placement.md),
+observe-only: nothing is decided on these yet. Every agent reads, at most every
+ten seconds, and sends with its heartbeat:
+
+- **GPUs** from nvidia-smi: temperature, HBM temperature where the driver has
+  it, power draw and the enforced limit, and what is holding the clocks down —
+  power cap, thermal slowdown, power brake. A GPU's temperature was read once,
+  at registration; it is now current.
+- **CPUs** from hwmon, which is what lm_sensors reads, with nothing to install:
+  the package temperature, and the board's inlet air sensor where it has one.
+- **CPU power** from RAPL's energy counters.
+
+A sensor a machine does not have is not reported, never shown as zero. New
+metrics: `llmgw_gpu_power_watts`, `llmgw_gpu_power_limit_watts`,
+`llmgw_gpu_memory_temperature_celsius`, `llmgw_gpu_throttled{reason}`,
+`llmgw_node_cpu_temperature_celsius`, `llmgw_node_cpu_power_watts`,
+`llmgw_node_inlet_temperature_celsius`. Each node's page shows them, with a GPU
+table of temperature, power against its limit, and what is throttling it.
+
+### The mirror page in tabs
+
+/admin/mirror was one long page: what the mirror holds, the Hugging Face
+explorer and the fetch form, one under another, with every visit waiting on
+Hugging Face for a list it might not show. It is now three tabs — Held (with
+how many are held, downloading and failed), Explore Hugging Face, and Fetch by
+URL — each with its own address. Only the tab shown does its work, and only
+Held reloads while something downloads. A download started from the explorer
+links to Held, where its progress is; a fetch the mirror refused comes back on
+the form it was sent from.
+
+### A held model opens to where it runs, and how many
+
+Clicking a model under Held opens a card beneath it: each replica with its
+node, state, age and the CPU and memory its engine uses, and how many more
+replicas the cluster has room for now, on which nodes, and why not more. The
+replicas it always runs and the most it grows to under load are changed there.
+Room is found by trying the scheduler itself on a copy of the inventory — the
+same decisions the next passes would make, not an estimate beside them — and a
+number of always-running replicas it cannot place is refused with the reason,
+rather than accepted and left short. A ceiling above the room is taken, and
+said. The same answer is at `GET /api/v1/models/{name}/headroom` on the admin
+API.
+
+Each row also says how its model runs without being opened: replicas ready of
+those it should have, how many are still starting, and the nodes they are on —
+in amber when short, in red when parked or when it does not fit. It is one line
+per model beside its mode, and nodes drop the part of the name every node
+shares (node05 for llmgw-node05, the full name on hover).
+
+### One line per record, in every table
+
+The same rule across the admin pages. Headers do not wrap; a record is one
+line; free text is cut to its column with the whole of it on hover rather than
+given a row of its own; a row across the whole table is only a note or an open
+form. On the cluster page that means a node is one line (its address and logs
+beside its name, not under it), a model's description is a column rather than a
+second row, node names drop the prefix they all share, and the GPUs, classes,
+fabrics and fallbacks columns appear only when some row has something in them —
+on a CPU-only cluster they were three columns of dashes pushing the table past
+the page. On People, a person in many teams shows two and a count, the rest on
+hover.
+
 ## v0.10.4 — 2026-09-30
 
 Published first as release candidates v0.10.4-rc.1 to rc.9.

api/proto/llmgw/v1/agent.proto

@@ -120,6 +120,10 @@ message Register {
   // meant "an agent too old to say" and constrained nothing — so a freshly
   // built node with no engine installed yet was a candidate for every model.
   bool engines_probed = 14;
+  // cpu_count is the CPUs the agent may run on. Placement reserves cores
+  // for CPU replicas out of it, the way it reserves memory out of
+  // memory_total_mb; zero, from an older agent, reserves nothing.
+  int32 cpu_count = 18;
   // imex is the service that lets a multi-node NVLink domain share memory.
   IMEX imex = 15;
 
@@ -281,6 +285,25 @@ message NodeLoad {
   int64 memory_used_bytes = 6;
   int64 memory_available_bytes = 7;
   repeated ShardLoad shards = 8;
+  // What the machine's own sensors say; zero is not measured. cpu_temp_c is
+  // the hottest CPU sensor, cpu_power_w the CPU packages' draw from RAPL,
+  // inlet_temp_c the board's inlet or ambient sensor where it has one.
+  double cpu_temp_c = 9;
+  double cpu_power_w = 10;
+  double inlet_temp_c = 11;
+  repeated GPUSensor gpus = 12;
+}
+
+// GPUSensor is one card's heat, power, and what is holding it back.
+message GPUSensor {
+  int32 index = 1;
+  double temp_c = 2;
+  double memory_temp_c = 3;
+  double power_w = 4;
+  double power_limit_w = 5;
+  // throttle names what is holding the clocks down now: sw_power_cap,
+  // hw_slowdown, sw_thermal, hw_thermal, power_brake.
+  repeated string throttle = 6;
 }
 
 // ShardLoad is one engine process group's use.
@@ -420,6 +443,8 @@ message Heartbeat {
   // domain, and the node has to stop being offered work that spans it.
   repeated GPUFabricState gpu_fabric = 16;
   IMEX imex = 17;
+  // cpu_count as in Register, for a VM given more CPUs while it runs.
+  int32 cpu_count = 18;
 }
 
 // Provision is the software a node should have: which engines, at which
@@ -530,6 +555,9 @@ message StartShard {
   // control plane from the replica id. The router sends it; nothing else
   // that can reach the node's ports has it.
   string api_key = 16;
+  // cpu_threads is how many CPU threads a shard with no GPU computes on:
+  // the cores placement reserved for it. Zero leaves it to the agent.
+  int32 cpu_threads = 17;
 }
 
 message EngineProfile {

deploy/monitoring/llmgw-rules.yml

@@ -84,6 +84,28 @@ groups:
             <model>` says what each node lacks. For an on-demand model, requests
             for it are told this rather than asked to retry.
 
+      - alert: ModelPreempted
+        expr: increase(llmgw_model_preempted_total[15m]) > 0
+        labels: { severity: info }
+        annotations:
+          summary: "{{ $labels.model }} ({{ $labels.class }}) gave up replicas to a higher class"
+          description: >-
+            Capacity ran short and a model of a higher class took this one's
+            room. Expected now and then; if it keeps firing for the same model,
+            the cluster is too small for what is asked of it at these classes.
+            `llmgw ctl models` shows who it gave way to.
+
+      - alert: CapacityAtLimit
+        expr: llmgw_capacity_holds >= on(scope, resource) llmgw_capacity_limit
+        for: 15m
+        labels: { severity: info }
+        annotations:
+          summary: "{{ $labels.scope }} holds all the {{ $labels.resource }} it may"
+          description: >-
+            Its models get no further replicas, under load or not, until
+            something of theirs stops or the limit is raised: `llmgw ctl quota
+            list`, `llmgw ctl quota set`.
+
       - alert: EngineInstallFailed
         expr: llmgw_node_engine_install{state="failed"} == 1
         for: 15m

docs/facility-aware-placement.md

@@ -0,0 +1,303 @@
+# Facility-aware placement: rack power, temperature and cooling
+
+**Status: step 0, reading the nodes' own sensors, is built; steps 1–5 are
+deferred** until the rack, PDU and power information of a real site is at
+hand — without it they would be designed against guesses.
+This is the design for step 5 of resource allocation, after reserved CPU cores
+and memory (1), capacity classes and preemption (2), tenant capacity quotas (3)
+and sizing by measured throughput (4), all of which shipped in v0.10.5. Of
+what follows, only the node sensors (rollout step 0) exist in the code: they
+are read, published and shown, and change no decision.
+
+## Why
+
+In a large installation the scarce thing is often not a GPU but the room it
+sits in. Three failures are invisible to placement today:
+
+- **A rack goes over its power budget.** Every node in it reports healthy;
+  placement sees free cards and fills them; the breaker, or the facility's
+  power cap, decides the rest. A rack with A and B feeds is worse: it runs
+  fine until one feed fails and the other cannot carry the load.
+- **Hot hardware is slow hardware.** GPUs throttle long before they fail. A
+  replica placed in a rack whose inlet air is too warm serves fewer tokens per
+  second than its measured capacity says, and nothing connects the two.
+- **Correlated failure.** Two replicas of a model on two nodes in the same rack
+  or on the same PDU are one failure away from zero. Placement spreads replicas
+  across nodes, not across racks.
+
+The goal is for placement to treat facility limits the way it already treats
+memory and cores: reserve against a budget, refuse what does not fit, say why,
+and never let missing data stop the cluster.
+
+## What it builds on
+
+| Already in place | Used for |
+|---|---|
+| Node labels, reported by agents, matched by `node_selector` | Where a node is: its rack, row, PDU |
+| `inventory.Allocation` carries GPUs, memory and cores per shard | Adding watts, summed per rack and PDU |
+| Placement rejections (`explain-placement`, headroom's "why not") | Naming the rack that is full or hot |
+| Headroom: the scheduler dry-run on a copy of the inventory | Room that respects power budgets, for free |
+| Preemption with classes, minimums and a dry run (step 2) | Shedding load from a hot rack in the right order |
+| Measured throughput per profile (step 4) | Tokens per watt as a placement preference |
+| Node sensors, read by every agent (step 0, built) | Heat and power with no facility integration: see below |
+| `internal/promq`, a Prometheus client | Reading facility metrics that DCIM exporters publish |
+
+## Concepts
+
+**Location.** Each node has a place in a tree: site, room, row, rack. Power has
+its own tree beside it: a rack draws from one or more PDUs, and each PDU from a
+feed. Both are read from node labels with well-known keys:
+
+```yaml
+# agent.yaml
+labels:
+  llmgw.io/site: sto1
+  llmgw.io/room: hall-b
+  llmgw.io/row: "12"
+  llmgw.io/rack: r12-07
+  llmgw.io/pdu: r12-07-a,r12-07-b   # both feeds, for a dual-fed rack
+```
+
+A node without these labels is its own rack of one: placement behaves as it
+does today.
+
+**Domain.** A rack, a row, a PDU, a room: anything with a budget or a
+condition. Domains are declared in a facility file (below) or discovered from
+node labels; a label naming a domain the file does not declare creates it with
+no budget and no condition.
+
+**Budget.** A domain may have a power cap in watts. Placement reserves each
+replica's estimated draw against every domain its node belongs to and refuses a
+placement that would take any of them past its cap less a safety margin. For a
+dual-fed rack the cap that matters is what one feed can carry alone, since that
+is what the rack must survive.
+
+**Condition.** A domain has a thermal and cooling condition: `ok`, `warm`, `hot`
+or `unknown`, derived from readings against thresholds with hysteresis.
+Conditions change what placement prefers and allows; they are not amounts to
+fill.
+
+**Reading.** A measured value with a time: inlet temperature, power draw, a
+cooling unit's alarm state. Readings older than their staleness limit are
+treated as absent.
+
+## Estimating a replica's draw
+
+What placement reserves has to be known before the replica runs, so it is an
+estimate first and a measurement later:
+
+1. **Declared.** A profile may say `power_watts`. The operator's figure wins.
+2. **Derived.** Otherwise GPU profiles take each card's power limit from NVML
+   (the agent already reads the cards; the power limit is one more field) plus
+   a per-node share for CPU, memory and fans; CPU profiles take their reserved
+   cores times a per-core figure for the node's CPU model.
+3. **Measured.** Once replicas run, the agent reports their cards' actual draw
+   (NVML) and, where the BMC allows, the node's (Redfish or IPMI DCMI). A
+   domain's headroom is computed from the larger of what is reserved and what
+   is measured, so an estimate that was too low is corrected by the room it
+   stops offering, not by an outage.
+
+## Decisions
+
+### Power
+
+- **Hard limit.** A placement that would take any domain past `cap × (1 −
+  margin)` is rejected, with the domain and the figures: *rack r12-07: 14.2 of
+  16.0 kW reserved, replica needs ~2.8, margin 10%*.
+- **Feeds.** For a domain with two feeds the cap is the smaller feed's rating
+  times its derating (typically 80%), not the sum.
+- **Already over.** A domain found over its cap, because the estimate was low
+  or the cap was lowered, gets no new placements, and an alert. It is not
+  emptied automatically; shedding power is the facility's decision, and
+  stopping inference to do it is ours only if asked (see shedding below).
+
+### Temperature and cooling
+
+Thresholds are per domain with defaults, and each condition needs to hold for a
+while before it changes, in both directions, so a single reading neither
+starts nor ends anything:
+
+| Condition | Enters when | Placement |
+|---|---|---|
+| `ok` | below the warm threshold for 5 min | normal |
+| `warm` | inlet above the warm threshold for 2 min, GPU temperatures near throttling, or any GPU thermally throttled | allowed, scored down: other domains first |
+| `hot` | inlet above the hot threshold for 2 min, a cooling unit in alarm, or a sustained share of the domain's GPUs thermally throttled | no new placements, as a cordon would, but automatic and reversible |
+| `unknown` | no fresh reading | as `ok`, and reported as unknown |
+
+**Shedding** (optional, off by default): a domain `hot` for longer than its
+shed delay has load moved off it using step 2's machinery. The lowest class goes
+first, and replicas above their model's minimum go before any at it. A critical
+minimum is never touched, and moves are rate limited to one replica per domain
+per interval. The replica is placed elsewhere first where there is room, so
+shedding moves work rather than dropping it. GPU power capping (`nvidia-smi
+-pl`) is a gentler alternative worth measuring against shedding: less
+throughput, no restart.
+
+### Spreading
+
+Replicas of one model are spread across racks, and across PDUs, the way they
+are spread across nodes today. This is a preference, dropped when it is the only
+thing standing in the way, as node spreading already is.
+
+### Scoring
+
+Placement today filters, then takes the first candidate in a fixed order
+(fewest free GPUs for GPU profiles, fewest replicas for CPU ones, nodes holding
+the weights first). Facility signals are better as a score than as more
+filters. Candidates that pass the hard limits are ordered by:
+
+1. condition: `ok` before `warm`;
+2. power headroom left in the rack after this replica, more first, so load
+   levels across racks;
+3. the existing order.
+
+Once measurements include power, step 4's profile preference can use tokens per
+watt instead of tokens per GPU where both are known.
+
+## Where the data comes from
+
+A `FacilitySource` interface, implemented several ways, chosen in
+`server.yaml`:
+
+```go
+type FacilitySource interface {
+    // Domains is what the source knows: budgets, thresholds, members.
+    Domains(ctx) ([]Domain, error)
+    // Readings is the latest value of each signal, per domain.
+    Readings(ctx) ([]Reading, error)
+}
+```
+
+- **Static file.** Domains, caps and thresholds, written by hand or generated
+  from a DCIM export. Enough for power budgets with no live data at all.
+- **Prometheus.** DCIM, PDU (SNMP exporter) and BMC (Redfish exporter)
+  readings usually end up in Prometheus. The source runs configured instant
+  queries and maps their labels onto domains:
+
+  ```yaml
+  facility:
+    source: prometheus
+    prometheus: http://monitoring:9090
+    signals:
+      inlet_temp_c:
+        query: max by (rack) (dcim_rack_inlet_temperature_celsius)
+        domain_label: rack
+        stale_after: 2m
+      power_watts:
+        query: sum by (pdu) (pdu_outlet_power_watts)
+        domain_label: pdu
+      cooling_alarm:
+        query: max by (row) (crac_alarm)
+        domain_label: row
+  ```
+- **Agents (built, step 0).** What each node reads about itself, every ten
+  seconds at most, sent with its heartbeat:
+  - **GPUs, from nvidia-smi:** temperature, memory (HBM) temperature where the
+    driver reports it, power draw, the enforced power limit, and what is
+    holding the clocks down now — `sw_power_cap`, `hw_slowdown`, `sw_thermal`,
+    `hw_thermal`, `power_brake`.
+  - **CPUs, from hwmon** (what lm_sensors reads, without it installed): the
+    package temperature, or the hottest core; and the board's inlet or ambient
+    sensor where it has one (`SYSTIN`, `Inlet`, `Ambient`).
+  - **CPU power, from RAPL:** the packages' energy counters over time,
+    subdomains not counted twice, counter wrap handled.
+
+  A machine without a sensor reports nothing for it, never a zero; a VM has
+  none of them. Published as `llmgw_gpu_temperature_celsius`,
+  `llmgw_gpu_memory_temperature_celsius`, `llmgw_gpu_power_watts`,
+  `llmgw_gpu_power_limit_watts`, `llmgw_gpu_throttled{reason}`,
+  `llmgw_node_cpu_temperature_celsius`, `llmgw_node_cpu_power_watts` and
+  `llmgw_node_inlet_temperature_celsius`, and on each node's page.
+
+  What they can and cannot say:
+  - **Throttling is the strongest signal there is.** A temperature is a proxy;
+    a card holding its clocks down is the fact that invalidates its measured
+    capacity, and it needs no threshold to be chosen.
+  - **Measured power** is GPU draw plus CPU packages; memory, disks, fans and
+    power-supply losses are not in it, so a node's draw is that plus an
+    estimated remainder. A rack's is its llmgw nodes' plus a declared figure
+    for everything else in it: switches and storage report nothing here.
+  - **CPU and GPU temperatures follow the load, not the room.** A hot rack is
+    inferred from many nodes running hot at comparable load, or from
+    throttling, not from one node's CPU. Only a board inlet sensor reads the
+    air, and most boards have none.
+  - **No budget comes from a sensor.** A reading is a draw; the cap it is
+    measured against still comes from the facility.
+- **Simulated.** Readings from a script, for tests and for rehearsing a
+  thermal event on a cluster without facility data.
+
+## Safety
+
+- **Missing data never blocks.** A source that is down, a reading that is stale,
+  a node without location labels: placement behaves as it does today, and says
+  the data is missing (a metric, a line on the cluster page). Facility
+  awareness can make placement more careful, never impossible.
+- **Hysteresis and hold times** on every condition, so a noisy sensor does not
+  flap a rack between allowed and forbidden.
+- **Observe-only first.** A mode where every decision is computed, logged and
+  shown ("would have refused: rack r12-07 hot") but nothing is enforced, run
+  for long enough to trust the thresholds before turning enforcement on.
+- **Manual override.** `llmgw ctl facility hold RACK` and `release RACK`, for the
+  facility team to mark a domain unavailable, or to clear a condition they know
+  to be a sensor fault.
+- **Shedding is opt-in**, per domain or cluster-wide, and rate limited.
+
+## What people see
+
+- `explain-placement` and headroom name the domain: *rack r12-07 is hot (inlet
+  31°C for 6 min)*, *pdu r12-07-a: 7.1 of 7.4 kW*.
+- `llmgw ctl facility` lists domains with budget, reserved, measured and
+  condition. The cluster page gets a rack summary beside the nodes table.
+- Metrics: `llmgw_domain_power_budget_watts`, `_reserved_watts`,
+  `_measured_watts`, `llmgw_domain_condition{condition}`,
+  `llmgw_domain_reading_age_seconds`; alerts for a domain near its cap, a
+  domain hot, a domain over its cap, and facility data gone stale.
+
+## Rollout
+
+Each step ships on its own and is useful without the next:
+
+0. **Node sensors, observe-only (built).** GPU heat, power and throttling,
+   CPU heat and power, inlet air where the board has a sensor; published and
+   shown, no decision changed.
+1. **Location and spreading.** Well-known labels, domains from labels, replicas
+   spread across racks and PDUs, locations on the node views. No facility data
+   needed.
+2. **Power budgets from a static file.** Caps and margins, estimated draw per
+   replica (declared or derived), hard limit in placement and headroom.
+3. **Readings, observe-only.** The Prometheus source beside the agents',
+   conditions with hysteresis (throttling first among them), measured power
+   folded into headroom; decisions shown, not enforced.
+4. **Thermal enforcement.** `warm` scored down, `hot` refused, manual hold and
+   release.
+5. **Shedding and tokens per watt.** Opt-in shedding through step 2's
+   preemption; measured power in profile preference.
+
+## Testing
+
+- The simulated source drives unit tests of every condition transition, the
+  hysteresis, staleness, and the placement outcomes, with mutation checks as
+  for steps 1–4.
+- Property tests: whatever the readings, placement never exceeds a declared
+  cap with fresh data, and never refuses everything because data is missing.
+- On the PoC cluster, with no facility data: step 1 with labelled VMs, step 2
+  with a static file of made-up caps, steps 3–4 with the simulated source
+  replaying a thermal event.
+- Before enforcement anywhere real: a period in observe-only mode against the
+  site's actual data.
+
+## Open questions
+
+- **What the sites have.** Which DCIM, which PDUs, which BMCs, and whether any
+  of it already reaches a Prometheus. The source design assumes Prometheus is
+  the common ground; a site without it needs another source.
+- **Who owns the numbers.** Caps, deratings and thresholds belong to the
+  facility team. Whether they edit llmgw's file, or llmgw reads theirs.
+- **Phase balance.** Three-phase PDUs have per-phase limits; whether placement
+  should see phases or only the PDU total.
+- **Cooling granularity.** Cooling is often per row or per aisle, not per rack;
+  the domain tree allows either, but the defaults should match what sites
+  actually measure.
+- **Power capping instead of shedding.** Whether to lower GPU power limits in a
+  warm rack rather than move replicas, and how that interacts with measured
+  capacity from step 4, which would no longer hold at the lower limit.

docs/install.md

@@ -940,8 +940,18 @@ restart either — it rebuilds the fleet from what the agents report.
 fields, and both sides treat an absent one as unknown rather than as a default.
 Agents older than v0.10.4 cannot upgrade themselves: move those by hand once,
 installing the release binary and restarting `llmgw-agent`, and every upgrade
-after that is the command above. The mirror, which is not an agent, is
-upgraded the same way by hand.
+after that is the command above.
+
+**The mirror is upgraded on its own host**, since it is not an agent and its
+service cannot replace its own binary:
+
+```
+sudo llmgw upgrade v0.10.5 --service llmgw-mirror
+```
+
+The same checks — the release's checksums, the old binary kept beside the new —
+then a restart of the unit, waiting for it to stay up. A mirror older than
+v0.10.5 does not have the command: install the release binary by hand once.
 
 Then confirm what you have, rather than assuming:
 
@@ -956,6 +966,33 @@ replica alone on purpose, because churning one costs more than the difference.
 `llmgw ctl restart <replica>` replaces it with one on the new profile, starting
 the new one before stopping the old.
 
+### Signed releases
+
+Every install above checks a release against its published checksums, which
+proves the download is what the server holds — not that the server holds what
+was built. Signing closes that: CI signs the checksums, and a binary that
+trusts a key installs only releases signed with it. Off until a key exists;
+turning it on, once:
+
+```
+llmgw release keygen
+```
+
+1. Put the **private** key in the repository's Actions secret
+   `RELEASE_SIGNING_KEY`. From the next tag, releases carry
+   `checksums.txt.sig`.
+2. Put the **public** key in `internal/release/trusted.pub` and release. Every
+   binary built from then on carries it and refuses releases that are not
+   signed with it — `ctl upgrade`, agents upgrading themselves, and
+   `llmgw upgrade` alike.
+
+In that order: a build that trusts a key refuses unsigned releases, so the
+secret has to be signing before the key is trusted. The first trusting release
+is itself installed by a binary that trusts nothing; every one after is checked
+by the one before. `LLMGW_RELEASE_KEY` overrides the built-in key, for rotating
+it. The onboarding script fetches with `curl` and checks checksums only; a node
+is checked from its first upgrade on.
+
 ## Backing up
 
 `llmgw server backup` writes one archive holding everything a control plane

docs/reference.md

@@ -21,6 +21,7 @@ rather than repeated.
 | `band_header` | `string` | [Admission](#admission) |
 | `queue_ttl` | `time.Duration` | [Admission](#admission) |
 | `queue_capacity_per_band` | `int` | [Admission](#admission) |
+| `queue_capacity_total` | `int` | [Admission](#admission) |
 | `interval` | `time.Duration` | [Admission](#admission) |
 | `enabled` | `bool` | [Auth](#auth) |
 | `store` | `string` | [Auth](#auth) |
@@ -92,6 +93,8 @@ rather than repeated.
 | `source` | `string` | [Model](#model) |
 | `profiles` | `[]Profile` | [Model](#model) |
 | `scaling` | `Scaling` | [Model](#model) |
+| `class` | `string` | [Model](#model) |
+| `owner` | `string` | [Model](#model) |
 | `fallbacks` | `[]string` | [Model](#model) |
 | `routing_profile` | `string` | [Model](#model) |
 | `slo` | `SLO` | [Model](#model) |
@@ -129,6 +132,7 @@ rather than repeated.
 | `args` | `[]string` | [Profile](#profile) |
 | `env` | `map[string]string` | [Profile](#profile) |
 | `min_memory_gb` | `int` | [Profile](#profile) |
+| `cpu_cores` | `int` | [Profile](#profile) |
 | `min_disk_gb` | `int` | [Profile](#profile) |
 | `node_selector` | `map[string]string` | [Profile](#profile) |
 | `weight` | `float64` | [Profile](#profile) |
@@ -225,6 +229,15 @@ should not be able to promote themselves.
 
 `int`
 
+### `queue_capacity_total`
+
+`int`
+
+QueueCapacityTotal bounds everything queued across bands. Reached, a
+request is not refused while a lower band has any queued: the newest of
+the lowest band is turned away instead, with a 503 and Retry-After.
+Zero takes half of all the bands' capacities together.
+
 ### `interval`
 
 `time.Duration`
@@ -735,6 +748,23 @@ Source is the weights path or a HuggingFace id.
 
 Takes a [Scaling](#scaling) block.
 
+### `class`
+
+`string`
+
+Class is how much this model's capacity is worth against another's,
+when there is not enough for both: critical, standard or opportunistic.
+Empty is standard, or opportunistic for a model loaded on demand. See
+EffectiveClass.
+
+### `owner`
+
+`string`
+
+Owner is the part of the tenancy tree whose capacity quota this model
+counts against, written org:ID, team:ID or project:ID. Empty is shared:
+held by the cluster, counted against nobody.
+
 ### `fallbacks`
 
 `[]string`
@@ -966,6 +996,20 @@ this. A node that does not report its memory is not excluded either —
 an older agent must not have its nodes quietly emptied by an upgrade of
 the control plane.
 
+### `cpu_cores`
+
+`int`
+
+CPUCores is the cores a replica of a profile with no GPU is given:
+reserved out of its node's CPUs by placement, as memory is, and what
+its engine computes on (llama.cpp's --threads).
+
+Reserved because an engine takes every core it is told about, and two
+on one node that each think they have all of them spin against each
+other: on an 8-vCPU node, two 7B replicas fell from 10 tokens a second
+to under one each. Zero reserves nothing and leaves the thread count to
+the node, which is every profile from before this.
+
 ### `min_disk_gb`
 
 `int`

internal/agent/agent.go

@@ -16,6 +16,7 @@ import (
 	"net/http"
 	"os"
 	"path/filepath"
+	"runtime"
 	"slices"
 	"sync"
 	"time"
@@ -100,6 +101,9 @@ type Agent struct {
 	smsErr map[string]string
 	// load measures the machine between heartbeats.
 	load *discovery.LoadSampler
+	// sensors reads heat and power: CPU and board from hwmon and RAPL, GPUs
+	// from nvidia-smi.
+	sensors *discovery.SensorSampler
 
 	// fetchWatchers are the shards waiting on each source, so progress on one
 	// download reaches every shard that asked for it. A fetch is single-flight
@@ -145,6 +149,7 @@ func New(cfg Config, httpClient connectHTTPClient, log *slog.Logger) (*Agent, er
 		logRing:      ring,
 		provKick:     make(chan struct{}, 1),
 		load:         discovery.NewLoadSampler(),
+		sensors:      discovery.NewSensorSampler(),
 	}
 	a.prov = &provision.Installer{Root: cfg.EnginesDir, Log: log, OnStatus: a.provisionStatus}
 	return a, nil
@@ -549,6 +554,7 @@ func (a *Agent) heartbeat() *llmgwv1.AgentMessage {
 	// and a node that grew from 7 GB to 32 GB has to be seen to have grown
 	// without anybody restarting anything.
 	hb.MemoryTotalMb = int64(discovery.MemoryTotalMB())
+	hb.CpuCount = int32(runtime.NumCPU())
 
 	a.reprobeEngines()
 	a.mu.Lock()
@@ -636,6 +642,16 @@ func (a *Agent) sampleLoad() *llmgwv1.NodeLoad {
 			ShardId: sh.ID, CpuCores: sh.CPUCores, MemoryRssBytes: sh.RSSBytes,
 		})
 	}
+	if a.sensors != nil {
+		se := a.sensors.Sample(context.Background())
+		out.CpuTempC, out.CpuPowerW, out.InletTempC = se.CPUTempC, se.CPUPowerW, se.InletTempC
+		for _, g := range se.GPUs {
+			out.Gpus = append(out.Gpus, &llmgwv1.GPUSensor{
+				Index: int32(g.Index), TempC: g.TempC, MemoryTempC: g.MemoryTempC,
+				PowerW: g.PowerW, PowerLimitW: g.PowerLimitW, Throttle: g.Throttle,
+			})
+		}
+	}
 	return out
 }
 

internal/agent/discovery/sensors.go

@@ -0,0 +1,242 @@
+package discovery
+
+import (
+	"context"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"strconv"
+	"strings"
+	"sync"
+	"time"
+)
+
+// Sensors is what a node's own hardware says about heat and power: the CPUs,
+// the board where it has an inlet sensor, and each GPU. A zero is not
+// measured, never a reading of zero — a VM has none of these, and must not
+// look like a machine running at 0 °C on no power.
+type Sensors struct {
+	// CPUTempC is the hottest CPU sensor: the package, or the hottest core
+	// where no package reading exists.
+	CPUTempC float64
+	// CPUPowerW is the CPU packages' draw, from RAPL: the change in their
+	// energy counters over the time between two samples.
+	CPUPowerW float64
+	// InletTempC is the air coming in, where the board has a sensor it calls
+	// that. Most do not, and a CPU's temperature is not a substitute: it
+	// follows the load, not the room.
+	InletTempC float64
+	GPUs       []GPUSensor
+}
+
+// GPUSensor is one card's heat, power and whether it is slowing itself down.
+type GPUSensor struct {
+	Index       int
+	TempC       float64
+	MemoryTempC float64
+	PowerW      float64
+	PowerLimitW float64
+	// Throttle names what is holding the card's clocks down now:
+	// sw_power_cap, hw_slowdown, sw_thermal, hw_thermal, power_brake. Empty
+	// is running free. Idle and application clock settings are left out;
+	// they are not the card being held back.
+	Throttle []string
+}
+
+// sensorEvery bounds how often nvidia-smi is run. Heartbeats come more often
+// than temperatures change, and each run costs a few hundred milliseconds.
+const sensorEvery = 10 * time.Second
+
+// SensorSampler reads Sensors. The first sample has no CPU power, which needs
+// two readings of the energy counters.
+type SensorSampler struct {
+	sys string // /sys, or a test's copy of it
+	smi func(ctx context.Context, fields string) ([]byte, error)
+	now func() time.Time
+
+	mu         sync.Mutex
+	last       Sensors
+	lastAt     time.Time
+	prevEnergy map[string]uint64
+	prevAt     time.Time
+}
+
+// NewSensorSampler reads this machine.
+func NewSensorSampler() *SensorSampler {
+	return &SensorSampler{sys: "/sys", smi: runSMI, now: time.Now}
+}
+
+func runSMI(ctx context.Context, fields string) ([]byte, error) {
+	ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
+	defer cancel()
+	//nolint:gosec // fixed binary, fields from this file
+	return exec.CommandContext(ctx, "nvidia-smi", "--query-gpu="+fields,
+		"--format=csv,noheader,nounits").Output()
+}
+
+// Sample reads the sensors, or returns the last reading when that is recent.
+func (s *SensorSampler) Sample(ctx context.Context) Sensors {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	now := s.now()
+	if !s.lastAt.IsZero() && now.Sub(s.lastAt) < sensorEvery {
+		return s.last
+	}
+	out := Sensors{}
+	out.CPUTempC, out.InletTempC = s.hwmon()
+	out.CPUPowerW = s.rapl(now)
+	out.GPUs = s.gpus(ctx)
+	s.last, s.lastAt = out, now
+	return out
+}
+
+// hwmon reads the hottest CPU sensor and an inlet sensor, from every hwmon
+// device the kernel exposes — what lm_sensors reads, without it installed.
+func (s *SensorSampler) hwmon() (cpu, inlet float64) {
+	devs, _ := filepath.Glob(filepath.Join(s.sys, "class/hwmon/hwmon*"))
+	var pkg, core float64
+	for _, d := range devs {
+		name := readTrim(filepath.Join(d, "name"))
+		inputs, _ := filepath.Glob(filepath.Join(d, "temp*_input"))
+		for _, in := range inputs {
+			milli, err := strconv.ParseFloat(readTrim(in), 64)
+			if err != nil || milli <= 0 {
+				continue
+			}
+			c := milli / 1000
+			label := strings.ToLower(readTrim(strings.TrimSuffix(in, "_input") + "_label"))
+			switch {
+			case isCPUSensor(name) && (strings.HasPrefix(label, "package") || label == "tctl" || label == "tdie"):
+				pkg = max(pkg, c)
+			case isCPUSensor(name):
+				core = max(core, c)
+			case label == "cpu":
+				// A board's own reading of the CPU, as Dell's SMM gives.
+				core = max(core, c)
+			case strings.Contains(label, "inlet") || strings.Contains(label, "ambient") ||
+				strings.Contains(label, "intake") || label == "systin":
+				inlet = max(inlet, c)
+			}
+		}
+	}
+	if pkg > 0 {
+		return pkg, inlet
+	}
+	return core, inlet
+}
+
+func isCPUSensor(name string) bool {
+	switch name {
+	case "coretemp", "k10temp", "zenpower", "cpu_thermal":
+		return true
+	}
+	return false
+}
+
+// rapl is the CPU packages' power from their energy counters. Package
+// domains only: the dram and core subdomains are parts of a package, and
+// adding them would count the same energy twice.
+func (s *SensorSampler) rapl(now time.Time) float64 {
+	doms, _ := filepath.Glob(filepath.Join(s.sys, "class/powercap/intel-rapl:*"))
+	energy := map[string]uint64{}
+	wrap := map[string]uint64{}
+	for _, d := range doms {
+		if strings.Count(filepath.Base(d), ":") != 1 {
+			continue // intel-rapl:0:0 is a subdomain of intel-rapl:0
+		}
+		e, err := strconv.ParseUint(readTrim(filepath.Join(d, "energy_uj")), 10, 64)
+		if err != nil {
+			continue // unreadable without root on newer kernels
+		}
+		energy[d] = e
+		wrap[d], _ = strconv.ParseUint(readTrim(filepath.Join(d, "max_energy_range_uj")), 10, 64)
+	}
+	prev, prevAt := s.prevEnergy, s.prevAt
+	s.prevEnergy, s.prevAt = energy, now
+	dt := now.Sub(prevAt).Seconds()
+	if prev == nil || dt <= 0 || len(energy) == 0 {
+		return 0
+	}
+	var joules float64
+	for d, e := range energy {
+		p, ok := prev[d]
+		if !ok {
+			return 0 // a domain appeared: no fair difference this time
+		}
+		if e < p { // the counter wrapped
+			e += wrap[d]
+		}
+		joules += float64(e-p) / 1e6
+	}
+	return joules / dt
+}
+
+// The throttle reasons nvidia-smi reports, as bits. Only those that mean the
+// card is being held back are named.
+var throttleBits = []struct {
+	bit  uint64
+	name string
+}{
+	{0x4, "sw_power_cap"},
+	{0x8, "hw_slowdown"},
+	{0x20, "sw_thermal"},
+	{0x40, "hw_thermal"},
+	{0x80, "power_brake"},
+}
+
+// gpus asks nvidia-smi for each card's heat and power. The memory temperature
+// is not known to every driver, and a field nvidia-smi does not know fails the
+// whole query, so it is asked for first and dropped on failure.
+func (s *SensorSampler) gpus(ctx context.Context) []GPUSensor {
+	withMem := "index,temperature.gpu,power.draw,enforced.power.limit,clocks_throttle_reasons.active,temperature.memory"
+	out, err := s.smi(ctx, withMem)
+	if err != nil {
+		if out, err = s.smi(ctx, strings.TrimSuffix(withMem, ",temperature.memory")); err != nil {
+			return nil
+		}
+	}
+	var gs []GPUSensor
+	for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
+		f := strings.Split(line, ",")
+		if len(f) < 5 {
+			continue
+		}
+		for i := range f {
+			f[i] = strings.TrimSpace(f[i])
+		}
+		idx, err := strconv.Atoi(f[0])
+		if err != nil {
+			continue
+		}
+		g := GPUSensor{Index: idx, TempC: num(f[1]), PowerW: num(f[2]), PowerLimitW: num(f[3])}
+		if bits, err := strconv.ParseUint(strings.TrimPrefix(f[4], "0x"), 16, 64); err == nil {
+			for _, b := range throttleBits {
+				if bits&b.bit != 0 {
+					g.Throttle = append(g.Throttle, b.name)
+				}
+			}
+		}
+		if len(f) > 5 {
+			g.MemoryTempC = num(f[5])
+		}
+		gs = append(gs, g)
+	}
+	return gs
+}
+
+// num reads a figure nvidia-smi may give as "[N/A]" or "N/A".
+func num(s string) float64 {
+	v, err := strconv.ParseFloat(s, 64)
+	if err != nil || v < 0 {
+		return 0
+	}
+	return v
+}
+
+func readTrim(path string) string {
+	b, err := os.ReadFile(path) //nolint:gosec // sysfs
+	if err != nil {
+		return ""
+	}
+	return strings.TrimSpace(string(b))
+}

internal/agent/discovery/sensors_test.go

@@ -0,0 +1,169 @@
+package discovery
+
+import (
+	"context"
+	"errors"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+	"time"
+)
+
+func put(t *testing.T, root, rel, body string) {
+	t.Helper()
+	p := filepath.Join(root, rel)
+	if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
+		t.Fatal(err)
+	}
+	if err := os.WriteFile(p, []byte(body+"\n"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+}
+
+// dellHost is the layout of the Proxmox host these tests were written
+// against: Dell's SMM with a CPU and SODIMM sensors, coretemp with a package
+// and cores, and RAPL with a package and its dram subdomain.
+func dellHost(t *testing.T) string {
+	root := t.TempDir()
+	put(t, root, "class/hwmon/hwmon0/name", "dell_smm")
+	put(t, root, "class/hwmon/hwmon0/temp1_input", "55000")
+	put(t, root, "class/hwmon/hwmon0/temp1_label", "CPU")
+	put(t, root, "class/hwmon/hwmon0/temp2_input", "47000")
+	put(t, root, "class/hwmon/hwmon0/temp2_label", "SODIMM")
+	put(t, root, "class/hwmon/hwmon1/name", "coretemp")
+	put(t, root, "class/hwmon/hwmon1/temp1_input", "58000")
+	put(t, root, "class/hwmon/hwmon1/temp1_label", "Package id 0")
+	put(t, root, "class/hwmon/hwmon1/temp10_input", "61000")
+	put(t, root, "class/hwmon/hwmon1/temp10_label", "Core 8")
+	put(t, root, "class/powercap/intel-rapl:0/energy_uj", "1000000000")
+	put(t, root, "class/powercap/intel-rapl:0/max_energy_range_uj", "262143328850")
+	put(t, root, "class/powercap/intel-rapl:0:0/energy_uj", "5000000")
+	return root
+}
+
+func sampler(sys string, smi func(context.Context, string) ([]byte, error), clock *time.Time) *SensorSampler {
+	return &SensorSampler{sys: sys, smi: smi, now: func() time.Time { return *clock }}
+}
+
+func noSMI(context.Context, string) ([]byte, error) { return nil, errors.New("not found") }
+
+func TestAHostsCPUHeatAndPowerAreReadAsLmSensorsWould(t *testing.T) {
+	root := dellHost(t)
+	now := time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC)
+	s := sampler(root, noSMI, &now)
+
+	first := s.Sample(t.Context())
+	// The package reading, not the hotter core: the package is what the
+	// chip itself reports as its temperature.
+	if first.CPUTempC != 58 {
+		t.Errorf("CPU %.1f °C, expected the package's 58", first.CPUTempC)
+	}
+	if first.CPUPowerW != 0 {
+		t.Error("power from one reading of a counter is invented")
+	}
+	if first.InletTempC != 0 || first.GPUs != nil {
+		t.Errorf("sensors this host does not have were reported: %+v", first)
+	}
+
+	// 10 s later the package has used 650 J, and its dram subdomain some of
+	// that — counted once.
+	put(t, root, "class/powercap/intel-rapl:0/energy_uj", "1650000000")
+	put(t, root, "class/powercap/intel-rapl:0:0/energy_uj", "99000000")
+	now = now.Add(10 * time.Second)
+	if got := s.Sample(t.Context()).CPUPowerW; got < 64.9 || got > 65.1 {
+		t.Errorf("CPU power %.2f W, expected 65", got)
+	}
+}
+
+func TestACounterThatWrapsIsNotANegativeDraw(t *testing.T) {
+	root := dellHost(t)
+	now := time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC)
+	put(t, root, "class/powercap/intel-rapl:0/energy_uj", "262143000000")
+	s := sampler(root, noSMI, &now)
+	s.Sample(t.Context())
+	// 328,850 µJ to the wrap, then 499,671,150 past it: 500 J in 10 s.
+	put(t, root, "class/powercap/intel-rapl:0/energy_uj", "499671150")
+	now = now.Add(10 * time.Second)
+	if got := s.Sample(t.Context()).CPUPowerW; got < 49.9 || got > 50.1 {
+		t.Errorf("across a wrap: %.2f W, expected 50", got)
+	}
+}
+
+func TestAVMHasNoSensorsAndSaysNothing(t *testing.T) {
+	root := t.TempDir()
+	put(t, root, "class/thermal/cooling_device0/type", "Processor")
+	now := time.Now()
+	got := sampler(root, noSMI, &now).Sample(t.Context())
+	if got.CPUTempC != 0 || got.CPUPowerW != 0 || got.InletTempC != 0 || got.GPUs != nil {
+		t.Errorf("a machine with no sensors reported %+v", got)
+	}
+}
+
+func TestABoardsInletSensorIsTheRoomNotTheCPU(t *testing.T) {
+	root := t.TempDir()
+	put(t, root, "class/hwmon/hwmon2/name", "nct6798")
+	put(t, root, "class/hwmon/hwmon2/temp1_input", "24500")
+	put(t, root, "class/hwmon/hwmon2/temp1_label", "SYSTIN")
+	put(t, root, "class/hwmon/hwmon3/name", "k10temp")
+	put(t, root, "class/hwmon/hwmon3/temp1_input", "71250")
+	put(t, root, "class/hwmon/hwmon3/temp1_label", "Tctl")
+	now := time.Now()
+	got := sampler(root, noSMI, &now).Sample(t.Context())
+	if got.InletTempC != 24.5 || got.CPUTempC != 71.25 {
+		t.Errorf("inlet %.2f, CPU %.2f", got.InletTempC, got.CPUTempC)
+	}
+}
+
+func TestEachGPUSaysHowHotHowHungryAndWhatHoldsItBack(t *testing.T) {
+	var asked []string
+	smi := func(_ context.Context, fields string) ([]byte, error) {
+		asked = append(asked, fields)
+		return []byte(
+			"0, 71, 612.45, 700.00, 0x0000000000000000, 84\n" +
+				"1, 88, 698.10, 700.00, 0x0000000000000024, 95\n" + // sw_power_cap + sw_thermal
+				"2, 45, [N/A], 700.00, 0x0000000000000001, [N/A]\n"), nil // idle only
+	}
+	now := time.Now()
+	ss := sampler(t.TempDir(), smi, &now)
+	got := ss.Sample(t.Context())
+	if len(got.GPUs) != 3 {
+		t.Fatalf("%+v", got.GPUs)
+	}
+	g0, g1, g2 := got.GPUs[0], got.GPUs[1], got.GPUs[2]
+	if g0.TempC != 71 || g0.PowerW != 612.45 || g0.PowerLimitW != 700 || g0.MemoryTempC != 84 || g0.Throttle != nil {
+		t.Errorf("card 0 %+v", g0)
+	}
+	if strings.Join(g1.Throttle, ",") != "sw_power_cap,sw_thermal" {
+		t.Errorf("card 1 throttled by %v", g1.Throttle)
+	}
+	if g2.PowerW != 0 || g2.MemoryTempC != 0 || g2.Throttle != nil {
+		t.Errorf("card 2: N/A is not a reading, and idle is not being held back: %+v", g2)
+	}
+	// Within ten seconds the last reading is reused rather than running
+	// nvidia-smi on every heartbeat; after, it is asked again.
+	now = now.Add(5 * time.Second)
+	ss.Sample(t.Context())
+	if len(asked) != 1 {
+		t.Errorf("nvidia-smi was asked %d times within ten seconds", len(asked))
+	}
+	now = now.Add(6 * time.Second)
+	ss.Sample(t.Context())
+	if len(asked) != 2 {
+		t.Errorf("nvidia-smi was asked %d times after eleven seconds, expected 2", len(asked))
+	}
+}
+
+func TestADriverThatDoesNotKnowMemoryTemperatureStillReportsTheRest(t *testing.T) {
+	smi := func(_ context.Context, fields string) ([]byte, error) {
+		if strings.Contains(fields, "temperature.memory") {
+			return nil, errors.New(`Field "temperature.memory" is not a valid field to query.`)
+		}
+		return []byte("0, 66, 250.0, 300.0, 0x0000000000000040\n"), nil
+	}
+	now := time.Now()
+	got := sampler(t.TempDir(), smi, &now).Sample(t.Context())
+	if len(got.GPUs) != 1 || got.GPUs[0].TempC != 66 || strings.Join(got.GPUs[0].Throttle, ",") != "hw_thermal" {
+		t.Errorf("%+v", got.GPUs)
+	}
+}

internal/agent/engines/engine.go

@@ -76,6 +76,10 @@ type ShardSpec struct {
 	// and accounting. Given to the engine in its environment rather than on
 	// its command line, which every user on the node can read.
 	APIKey string `json:"api_key,omitempty"`
+	// Threads is the CPU cores placement reserved for a shard with no GPU,
+	// and so what its engine computes on. Zero leaves it to the engine's own
+	// configuration.
+	Threads int `json:"threads,omitempty"`
 
 	// Pool and Fabric are control plane concepts the agent cannot derive. They
 	// are carried so the agent can report them back after a restart; Fabric

internal/agent/engines/llamacpp.go

@@ -133,7 +133,12 @@ func (l *LlamaCpp) Commands(s ShardSpec) ([]Command, error) {
 		args = append(args, "--n-gpu-layers", "999")
 	}
 	if len(s.GPUIndices) == 0 && !hasFlag(p.Args, "--threads", "-t") {
-		n := l.cfg.Threads
+		// What placement reserved first: it is what the node's other
+		// replicas were placed around.
+		n := s.Threads
+		if n <= 0 {
+			n = l.cfg.Threads
+		}
 		if n <= 0 {
 			n = max(numCPU()/2, 1)
 		}

internal/agent/engines/llamacpp_test.go

@@ -294,3 +294,15 @@ func TestACPUShardLeavesHalfTheMachineForAnother(t *testing.T) {
 		t.Errorf("--threads %q on one CPU", v)
 	}
 }
+
+func TestTheCoresPlacementReservedAreTheThreads(t *testing.T) {
+	s := llamaSpec(nil)
+	s.Threads = 3
+	cmds, err := (&LlamaCpp{cfg: LlamaCppConfig{Threads: 8}}).Commands(s)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if v, _ := flagValue(cmds[0].Args, "--threads"); v != "3" {
+		t.Errorf("--threads %q; what placement reserved should win over the node's setting", v)
+	}
+}

internal/cli/ctl.go

@@ -137,7 +137,7 @@ func newCtlCmd() *cobra.Command {
 		// apply to it; it lives here because this is where operators look.
 		newBenchCmd(client, &asJSON),
 		newStatusCmd(client, &asJSON),
-		newModelCmd(client),
+		newModelCmd(client, &asJSON),
 		newScaleCmd(client),
 		newRestartCmd(client),
 		newRetryCmd(client),
@@ -146,6 +146,7 @@ func newCtlCmd() *cobra.Command {
 		newToolCheckCmd(),
 		newWeightsCmd(client, &asJSON),
 		newCapacityCmd(client, &asJSON),
+		newQuotaCmd(client, &asJSON),
 		newUsageCmd(client, &asJSON),
 		newCertCmd(),
 	)
@@ -253,6 +254,8 @@ func modelStatus(m adminapi.ModelView) string {
 	switch {
 	case m.Parked:
 		return "parked"
+	case m.HeldFor != "":
+		return "gave room to " + m.HeldFor
 	case m.Unplaceable:
 		return "cannot place"
 	case m.Min == 0 && m.Total == 0:
@@ -323,17 +326,27 @@ func newModelsCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 			}
 
 			w := tab()
-			fmt.Fprintln(w, "MODEL\tSERVING\tREPLICAS\tSTATUS\tALIASES\tPROFILES\tFALLBACKS")
+			fmt.Fprintln(w, "MODEL\tSERVING\tREPLICAS\tSTATUS\tCLASS\tRESERVES\tALIASES\tPROFILES\tFALLBACKS")
+			var warnings []string
 			for _, m := range models {
 				serving := "no"
 				if m.Ready > 0 {
 					serving = "yes"
 				}
-				fmt.Fprintf(w, "%s\t%s\t%d/%d\t%s\t%s\t%s\t%s\n",
-					m.Name, serving, m.Ready, m.Min, modelStatus(m),
+				fmt.Fprintf(w, "%s\t%s\t%d/%d\t%s\t%s\t%s\t%s\t%s\t%s\n",
+					m.Name, serving, m.Ready, m.Min, modelStatus(m), m.Class, reserves(m),
 					join(m.Aliases), join(m.Profiles), join(m.Fallbacks))
+				for _, x := range m.Warnings {
+					warnings = append(warnings, m.Name+": "+x)
+				}
 			}
-			return w.Flush()
+			if err := w.Flush(); err != nil {
+				return err
+			}
+			for _, x := range warnings {
+				fmt.Println("warning:", x)
+			}
+			return nil
 		},
 	}
 }
@@ -511,3 +524,22 @@ func dash(s string) string {
 	}
 	return s
 }
+
+// reserves is what one replica of a model holds of a node, with what it was
+// measured using beside it: "6G 2c (5.1G used)".
+func reserves(m adminapi.ModelView) string {
+	if m.MemoryGB == 0 && m.CPUCores == 0 {
+		return "-"
+	}
+	out := fmt.Sprintf("%dG", m.MemoryGB)
+	if m.CPUCores > 0 {
+		out += fmt.Sprintf(" %dc", m.CPUCores)
+	}
+	if m.MeasuredMemoryGB > 0 {
+		out += fmt.Sprintf(" (%.1fG used)", m.MeasuredMemoryGB)
+	}
+	if len(m.Warnings) > 0 {
+		out += " !"
+	}
+	return out
+}

internal/cli/ctl_upgrade.go

@@ -172,6 +172,10 @@ func upgradeNodes(ctx context.Context, out io.Writer, client func() *ctlClient,
 		}
 	}
 	fmt.Fprintf(out, "%d node(s) upgraded to %s, %d already on it\n", done, v, skipped)
+	// The one machine this does not reach. Said every time, because it is
+	// exactly the kind of thing that stays three versions behind unnoticed.
+	fmt.Fprintf(out, "The model mirror's host is not upgraded by this; on it, run:\n"+
+		"  sudo llmgw upgrade %s --service llmgw-mirror\n", v)
 	return nil
 }
 

internal/cli/model.go

@@ -24,12 +24,14 @@ import (
 // minutes of outage. For a fixed production set that is a reasonable trade; for
 // a platform whose promise is that people can try things, it is the difference
 // between trying one and not bothering.
-func newModelCmd(client func() *ctlClient) *cobra.Command {
+func newModelCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 	c := &cobra.Command{
 		Use:   "model",
 		Short: "Add or remove models while the cluster is running",
 	}
 	c.AddCommand(newModelAddCmd(client), newModelRemoveCmd(client), newModelPlanCmd(client),
+		newResourcesCmd(client), newClassCmd(client), newOwnerCmd(client),
+		newTargetCmd(client, asJSON),
 		newModelServeCmd(client))
 	return c
 }
@@ -192,3 +194,73 @@ func newScaleCmd(client func() *ctlClient) *cobra.Command {
 		"how long an on-demand model stays up unasked-for; zero takes the cluster default")
 	return c
 }
+
+func newResourcesCmd(client func() *ctlClient) *cobra.Command {
+	var memory, cores int
+	var profile string
+	c := &cobra.Command{
+		Use:   "resources <model> --memory GB [--cores N]",
+		Short: "Set what a model reserves of a node: memory, and CPU cores",
+		Long: "--memory is the host RAM a replica needs: for a model with no GPU, its\n" +
+			"weights, KV cache and engine together. Placement reserves it on the node,\n" +
+			"so a node holds only as many replicas as it has memory for.\n\n" +
+			"--cores is the CPU cores a replica with no GPU computes on. They are\n" +
+			"reserved the same way, and become the engine's thread count, so two\n" +
+			"replicas on one node stop competing for the same cores. Zero reserves\n" +
+			"none and leaves the thread count to the node.\n\n" +
+			"Running replicas keep what they were started with; `llmgw ctl restart\n" +
+			"--model M` moves them to the new figures.",
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			if !cmd.Flags().Changed("memory") {
+				return fmt.Errorf("--memory is required")
+			}
+			body, err := json.Marshal(adminapi.ResourcesRequest{Profile: profile, MemoryGB: memory, CPUCores: cores})
+			if err != nil {
+				return err
+			}
+			var m catalog.Model
+			if err := client().post(cmd.Context(), "/api/v1/models/"+args[0]+"/resources", body, &m); err != nil {
+				return err
+			}
+			for _, p := range m.Profiles {
+				fmt.Fprintf(cmd.OutOrStdout(), "%s/%s: %d GB memory, %d cores\n", m.Name, p.Name, p.MinMemoryGB, p.CPUCores)
+			}
+			fmt.Fprintf(cmd.OutOrStdout(), "Running replicas keep their old figures until restarted: llmgw ctl restart --model %s\n", m.Name)
+			return nil
+		},
+	}
+	c.Flags().IntVar(&memory, "memory", 0, "host RAM a replica needs, in GB")
+	c.Flags().IntVar(&cores, "cores", 0, "CPU cores a replica with no GPU computes on; 0 reserves none")
+	c.Flags().StringVar(&profile, "profile", "", "one profile; every profile when left out")
+	return c
+}
+
+func newClassCmd(client func() *ctlClient) *cobra.Command {
+	return &cobra.Command{
+		Use:   "class <model> critical|standard|opportunistic|default",
+		Short: "Set how much a model's capacity is worth against another's",
+		Long: "When a model cannot be placed, replicas of a lower class are stopped to\n" +
+			"make room for it:\n\n" +
+			"  critical       never gives up the replicas of its minimum\n" +
+			"  standard       gives way to critical\n" +
+			"  opportunistic  gives way to both; the default for an on-demand model\n\n" +
+			"Within a class, a model below its minimum may take what another grew into\n" +
+			"under load, but not its minimum. \"default\" goes back to what the model's\n" +
+			"scaling implies: opportunistic when on demand, standard otherwise.",
+		Args: cobra.ExactArgs(2),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			class := args[1]
+			if class == "default" {
+				class = ""
+			}
+			body, _ := json.Marshal(map[string]string{"class": class})
+			var res map[string]string
+			if err := client().post(cmd.Context(), "/api/v1/models/"+args[0]+"/class", body, &res); err != nil {
+				return err
+			}
+			fmt.Fprintf(cmd.OutOrStdout(), "%s is %s\n", res["model"], res["class"])
+			return nil
+		},
+	}
+}

internal/cli/model_serve.go

@@ -75,6 +75,7 @@ func newModelServeCmd(client func() *ctlClient) *cobra.Command {
 	f.IntVar(&req.Replicas, "replicas", 1, "how many replicas to run")
 	f.BoolVar(&onDemand, "on-demand", false, "run none until first asked for")
 	f.IntVar(&req.CtxPool, "ctx-pool", 16384, "llama.cpp's --ctx-size, shared by its parallel slots")
+	f.IntVar(&req.CPUCores, "cores", 0, "CPU cores a replica computes on; 0 takes half the smallest node it fits")
 	f.IntVar(&req.ContextLength, "context-length", 4096, "what one request may use")
 	return c
 }

internal/cli/quota.go

@@ -0,0 +1,234 @@
+package cli
+
+import (
+	"encoding/json"
+	"fmt"
+	"net/url"
+	"strconv"
+	"strings"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+func newQuotaCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
+	c := &cobra.Command{
+		Use:   "quota",
+		Short: "What parts of the tenancy tree are guaranteed of the cluster, and may hold",
+		Long: "A capacity quota belongs to an organization, team or project, and counts the\n" +
+			"GPUs, CPU cores and memory held by replicas of the models it owns (see\n" +
+			"`llmgw ctl model owner`), and of those its teams and projects own.\n\n" +
+			"The guarantee is kept for it: when it holds less and a model of its needs\n" +
+			"room, replicas other scopes hold beyond their own guarantees are stopped to\n" +
+			"make it — whatever their class. What it holds within its guarantee is\n" +
+			"never taken by another scope's model. Beyond it, it borrows: free room is\n" +
+			"used, and given back when asked for.\n\n" +
+			"The limit is the most it may hold, borrowing included; a model that would\n" +
+			"take it past is not given another replica.\n\n" +
+			"Scopes are written org:Acme, team:Acme/ml or project:Acme/ml/training, or\n" +
+			"with IDs. Changes take effect within half a minute.",
+	}
+	c.AddCommand(newQuotaListCmd(client, asJSON), newQuotaSetCmd(client), newQuotaClearCmd(client))
+	return c
+}
+
+func newQuotaListCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
+	return &cobra.Command{
+		Use:   "list",
+		Short: "Every quota, with what its scope holds now",
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			var qs []adminapi.QuotaView
+			if err := client().get(cmd.Context(), "/api/v1/quotas", &qs); err != nil {
+				return err
+			}
+			if *asJSON {
+				return json.NewEncoder(cmd.OutOrStdout()).Encode(qs)
+			}
+			if len(qs) == 0 {
+				fmt.Fprintln(cmd.OutOrStdout(), "No capacity quotas: everything shares the cluster by class alone.")
+				return nil
+			}
+			w := tab()
+			fmt.Fprintln(w, "SCOPE\tNAME\tGPUS\tCORES\tMEMORY GB")
+			for _, q := range qs {
+				fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\n", q.Scope, q.Name,
+					quotaCell(q.Holds.GPUs, q.Guarantee.GPUs, q.Limit.GPUs),
+					quotaCell(q.Holds.CPUCores, q.Guarantee.CPUCores, q.Limit.CPUCores),
+					quotaCell(q.Holds.MemoryGB, q.Guarantee.MemoryGB, q.Limit.MemoryGB))
+			}
+			if err := w.Flush(); err != nil {
+				return err
+			}
+			fmt.Fprintln(cmd.OutOrStdout(), "\nEach cell: holds / guaranteed (at most limit). - is not governed.")
+			return nil
+		},
+	}
+}
+
+// quotaCell is "6 / 8 (≤ 16)": what is held, what is guaranteed, the limit.
+func quotaCell(holds int, guarantee, limit *int) string {
+	if guarantee == nil && limit == nil {
+		return "-"
+	}
+	out := strconv.Itoa(holds) + " / "
+	if guarantee != nil {
+		out += strconv.Itoa(*guarantee)
+	} else {
+		out += "-"
+	}
+	if limit != nil {
+		out += fmt.Sprintf(" (≤ %d)", *limit)
+	}
+	return out
+}
+
+func newQuotaSetCmd(client func() *ctlClient) *cobra.Command {
+	var gpus, cores, mem, lgpus, lcores, lmem int
+	c := &cobra.Command{
+		Use:   "set SCOPE [--gpus N] [--cores N] [--memory GB] [--limit-gpus N] ...",
+		Short: "Set a scope's guarantee and limit, replacing what it had",
+		Args:  cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			f := cmd.Flags()
+			pick := func(name string, v int) *int {
+				if !f.Changed(name) {
+					return nil
+				}
+				return &v
+			}
+			req := adminapi.QuotaRequest{Scope: args[0],
+				Guarantee: store.Resources{GPUs: pick("gpus", gpus), CPUCores: pick("cores", cores), MemoryGB: pick("memory", mem)},
+				Limit:     store.Resources{GPUs: pick("limit-gpus", lgpus), CPUCores: pick("limit-cores", lcores), MemoryGB: pick("limit-memory", lmem)},
+			}
+			body, _ := json.Marshal(req)
+			var res map[string]string
+			if err := client().put(cmd.Context(), "/api/v1/quotas", body, &res); err != nil {
+				return err
+			}
+			fmt.Fprintf(cmd.OutOrStdout(), "%s: set; it takes effect within half a minute\n", res["scope"])
+			return nil
+		},
+	}
+	f := c.Flags()
+	f.IntVar(&gpus, "gpus", 0, "GPUs guaranteed")
+	f.IntVar(&cores, "cores", 0, "CPU cores guaranteed")
+	f.IntVar(&mem, "memory", 0, "memory guaranteed, in GB")
+	f.IntVar(&lgpus, "limit-gpus", 0, "the most GPUs it may hold")
+	f.IntVar(&lcores, "limit-cores", 0, "the most CPU cores it may hold")
+	f.IntVar(&lmem, "limit-memory", 0, "the most memory it may hold, in GB")
+	return c
+}
+
+func newQuotaClearCmd(client func() *ctlClient) *cobra.Command {
+	return &cobra.Command{
+		Use:   "clear SCOPE",
+		Short: "Remove a scope's quota",
+		Args:  cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			var res map[string]string
+			if err := client().delete(cmd.Context(), "/api/v1/quotas/"+url.PathEscape(args[0]), &res); err != nil {
+				return err
+			}
+			fmt.Fprintf(cmd.OutOrStdout(), "%s: quota removed\n", res["removed"])
+			return nil
+		},
+	}
+}
+
+func newOwnerCmd(client func() *ctlClient) *cobra.Command {
+	return &cobra.Command{
+		Use:   "owner <model> SCOPE|shared",
+		Short: "Set whose capacity quota a model counts against",
+		Long: "SCOPE is org:Acme, team:Acme/ml or project:Acme/ml/training. A shared\n" +
+			"model belongs to the cluster and counts against nobody's quota.",
+		Args: cobra.ExactArgs(2),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			owner := args[1]
+			if owner == "shared" {
+				owner = ""
+			}
+			body, _ := json.Marshal(map[string]string{"owner": owner})
+			var res map[string]string
+			if err := client().post(cmd.Context(), "/api/v1/models/"+args[0]+"/owner", body, &res); err != nil {
+				return err
+			}
+			if res["owner"] == "" {
+				fmt.Fprintf(cmd.OutOrStdout(), "%s is shared\n", res["model"])
+			} else {
+				fmt.Fprintf(cmd.OutOrStdout(), "%s counts against %s\n", res["model"], res["owner"])
+			}
+			return nil
+		},
+	}
+}
+
+func newTargetCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
+	var tps float64
+	var apply bool
+	c := &cobra.Command{
+		Use:   "target <model> --tokens-per-sec N [--apply]",
+		Short: "How many replicas a model needs for a throughput, and whether they fit",
+		Long: "Sized from what one replica was measured to serve while meeting its SLO\n" +
+			"(`llmgw ctl bench MODEL --save`), on the profile its replicas run on.\n" +
+			"--apply makes that the number it always runs, if the cluster has room.",
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			if tps <= 0 {
+				return fmt.Errorf("--tokens-per-sec is required")
+			}
+			var v adminapi.TargetView
+			path := "/api/v1/models/" + url.PathEscape(args[0]) + "/target?tokens_per_sec=" +
+				strconv.FormatFloat(tps, 'f', -1, 64)
+			if err := client().get(cmd.Context(), path, &v); err != nil {
+				return err
+			}
+			if *asJSON {
+				return json.NewEncoder(cmd.OutOrStdout()).Encode(v)
+			}
+			out := cmd.OutOrStdout()
+			fmt.Fprintf(out, "%s: %.0f tokens/s needs %d replicas at %.1f each (measured on %s); %d running\n",
+				v.Model, v.TokPerSec, v.Needed, v.PerReplica, v.Profile, v.Current)
+			switch {
+			case v.More == 0:
+				fmt.Fprintln(out, "It already has them.")
+			case v.Fits && len(v.Preempts) > 0:
+				fmt.Fprintf(out, "%d more; there is room once replicas of %s are stopped, which its class may do.\n",
+					v.More, strings.Join(v.Preempts, ", "))
+			case v.Fits:
+				fmt.Fprintf(out, "%d more, and there is room for them.\n", v.More)
+			default:
+				fmt.Fprintf(out, "%d more, and there is room for %d:\n", v.More, v.Room)
+				for _, b := range v.Blocked {
+					fmt.Fprintln(out, "  "+b)
+				}
+			}
+			if !apply || v.More == 0 {
+				return nil
+			}
+			if !v.Fits {
+				return fmt.Errorf("not applied: the cluster has no room for them")
+			}
+			var models []adminapi.ModelView
+			if err := client().get(cmd.Context(), "/api/v1/models", &models); err != nil {
+				return err
+			}
+			top := v.Needed
+			for _, m := range models {
+				if m.Name == v.Model {
+					top = max(top, m.Max)
+				}
+			}
+			body, _ := json.Marshal(adminapi.ScaleRequest{Min: v.Needed, Max: top})
+			if err := client().post(cmd.Context(), "/api/v1/models/"+url.PathEscape(v.Model)+"/scale", body, nil); err != nil {
+				return err
+			}
+			fmt.Fprintf(out, "Applied: %s runs %d (up to %d under load).\n", v.Model, v.Needed, top)
+			return nil
+		},
+	}
+	c.Flags().Float64Var(&tps, "tokens-per-sec", 0, "the throughput to size for")
+	c.Flags().BoolVar(&apply, "apply", false, "set it as the number of replicas the model always runs")
+	return c
+}

internal/cli/release.go

@@ -0,0 +1,67 @@
+package cli
+
+import (
+	"errors"
+	"fmt"
+	"os"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/release"
+)
+
+func newReleaseCmd() *cobra.Command {
+	c := &cobra.Command{
+		Use:   "release",
+		Short: "Sign releases, and make the key they are signed with",
+	}
+	c.AddCommand(&cobra.Command{
+		Use:   "keygen",
+		Short: "Make a release signing key pair",
+		Long: "Prints a public key and a private key. The private key goes into the CI\n" +
+			"secret RELEASE_SIGNING_KEY and nowhere else; the public key into\n" +
+			"internal/release/trusted.pub, which every build from then on carries\n" +
+			"and checks releases against. Run once, on a machine you trust.",
+		Args: cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			pub, priv, err := release.GenerateKey()
+			if err != nil {
+				return err
+			}
+			fmt.Fprintf(cmd.OutOrStdout(), "public:  %s\nprivate: %s\n", pub, priv)
+			fmt.Fprintln(cmd.ErrOrStderr(), "\nThe private key signs every release: put it in the CI secret "+
+				"RELEASE_SIGNING_KEY, then clear this terminal's scrollback.")
+			return nil
+		},
+	})
+	c.AddCommand(&cobra.Command{
+		Use:   "sign FILE",
+		Short: "Sign a release's checksums, writing FILE.sig",
+		Long: "Reads the private key from RELEASE_SIGNING_KEY, never from an argument,\n" +
+			"so it does not land in a shell history or a process list.",
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			key := os.Getenv("RELEASE_SIGNING_KEY")
+			if key == "" {
+				return errors.New("RELEASE_SIGNING_KEY is not set")
+			}
+			data, err := os.ReadFile(args[0])
+			if err != nil {
+				return err
+			}
+			sig, err := release.Sign(key, data)
+			if err != nil {
+				return err
+			}
+			if pub := release.TrustedKey(); pub != "" {
+				// Signed with a key this build would not accept is a release
+				// every upgraded machine refuses: stopped here instead.
+				if err := release.Verify(pub, data, sig); err != nil {
+					return fmt.Errorf("the signing key does not match this build's trusted key: %w", err)
+				}
+			}
+			return os.WriteFile(args[0]+".sig", []byte(sig+"\n"), 0o644) //nolint:gosec // public
+		},
+	})
+	return c
+}

internal/cli/root.go

@@ -47,6 +47,8 @@ func Root() *cobra.Command {
 		newSimEngineCmd(),
 		newDoctorCmd(),
 		newNodeCmd(),
+		newSelfUpgradeCmd(),
+		newReleaseCmd(),
 	)
 	return root
 }

internal/cli/selfupgrade.go

@@ -0,0 +1,87 @@
+package cli
+
+import (
+	"errors"
+	"fmt"
+	"os"
+	"os/exec"
+	"runtime"
+	"strings"
+	"time"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/release"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/version"
+)
+
+// newSelfUpgradeCmd upgrades this machine's llmgw on its own, for the machines
+// `llmgw ctl upgrade` does not reach: a mirror host, above all, whose service
+// runs unprivileged with the filesystem read-only and so cannot replace its own
+// binary the way an agent does.
+func newSelfUpgradeCmd() *cobra.Command {
+	var releases, service string
+	c := &cobra.Command{
+		Use:   "upgrade VERSION [--service UNIT]",
+		Short: "Upgrade this machine's llmgw, and restart the service that runs it",
+		Long: "For a machine the control plane does not upgrade — the model mirror's host.\n" +
+			"The control plane and the nodes are upgraded with `llmgw ctl upgrade`.\n\n" +
+			"Fetches the release for this platform, checks it against the release's\n" +
+			"checksums, keeps the old binary beside the new one, and restarts --service,\n" +
+			"waiting for it to be running again. Run it with sudo.",
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			v := args[0]
+			out := cmd.OutOrStdout()
+			if !release.Tag.MatchString(v) {
+				return fmt.Errorf("%q is not a release version (v1.2.3 or v1.2.3-rc.4)", v)
+			}
+			if version.Version == v {
+				fmt.Fprintf(out, "this machine already runs %s\n", v)
+				return nil
+			}
+			if os.Geteuid() != 0 {
+				return errors.New("replacing this machine's binary needs root: run it with sudo")
+			}
+			self, err := os.Executable()
+			if err != nil {
+				return err
+			}
+			backup, err := release.Install(cmd.Context(), nil, releases, v, runtime.GOOS, runtime.GOARCH, self)
+			if err != nil {
+				return err
+			}
+			fmt.Fprintf(out, "%s -> %s: installed, checksum verified; the old binary is %s\n",
+				version.Version, v, backup)
+			if service == "" {
+				return nil
+			}
+			//nolint:gosec // the unit name is the operator's own flag
+			if b, err := exec.CommandContext(cmd.Context(), "systemctl", "restart", service).CombinedOutput(); err != nil {
+				return fmt.Errorf("restarting %s: %v: %s", service, err, strings.TrimSpace(string(b)))
+			}
+			// Running, and still running a few seconds on: a unit that starts
+			// and dies on a bad flag is "active" for a moment first.
+			deadline := time.Now().Add(30 * time.Second)
+			steady := 0
+			for time.Now().Before(deadline) {
+				//nolint:gosec // as above
+				b, _ := exec.CommandContext(cmd.Context(), "systemctl", "is-active", service).Output()
+				if strings.TrimSpace(string(b)) == "active" {
+					if steady++; steady >= 3 {
+						fmt.Fprintf(out, "%s restarted and running\n", service)
+						return nil
+					}
+				} else {
+					steady = 0
+				}
+				time.Sleep(2 * time.Second)
+			}
+			return fmt.Errorf("%s is not running after the upgrade; see journalctl -u %s, and %s holds the previous binary",
+				service, service, backup)
+		},
+	}
+	c.Flags().StringVar(&releases, "releases", release.DefaultBase, "where release binaries are fetched from")
+	c.Flags().StringVar(&service, "service", "", "the systemd unit to restart afterwards, e.g. llmgw-mirror")
+	return c
+}

internal/cli/server.go

@@ -226,6 +226,8 @@ func newServerCmd() *cobra.Command {
 					// And what each group of the tenancy tree spent, which is
 					// the other half of a quota being reviewable.
 					api.SetUsage(st)
+					// Capacity quotas, set and read over this API.
+					api.SetQuotas(st)
 				}
 				api.Register(adminMux)
 
@@ -483,6 +485,12 @@ func startControlPlane(ctx context.Context, cfg config.Server, cat *catalog.Live
 		if st := h.Store(); st != nil {
 			lead = control.NewLeadership(st, cfg.Control.LeaseTTL, log)
 			ctl.SetLeadership(lead)
+			// The tenancy tree's capacity quotas: what each part is
+			// guaranteed and may hold.
+			ctl.Reconciler().SetQuotas(control.QuotaPolicy(st, log))
+			// And what each model was measured to serve, for scaling and for
+			// choosing between profiles.
+			ctl.Reconciler().SetMeasurements(control.Measurements(st, log))
 			log.Info("competing to be the control plane that places work",
 				"owner", lead.Owner())
 		}

internal/config/config.go

@@ -378,7 +378,12 @@ type Admission struct {
 
 	QueueTTL             time.Duration `koanf:"queue_ttl"`
 	QueueCapacityPerBand int           `koanf:"queue_capacity_per_band"`
-	Interval             time.Duration `koanf:"interval"`
+	// QueueCapacityTotal bounds everything queued across bands. Reached, a
+	// request is not refused while a lower band has any queued: the newest of
+	// the lowest band is turned away instead, with a 503 and Retry-After.
+	// Zero takes half of all the bands' capacities together.
+	QueueCapacityTotal int           `koanf:"queue_capacity_total"`
+	Interval           time.Duration `koanf:"interval"`
 }
 
 // Prefix configures approximate prefix routing.

internal/control/adminapi/actor_test.go

@@ -4,6 +4,8 @@ import (
 	"net/http/httptest"
 	"strings"
 	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
 )
 
 func TestAClaimedActorIsOneShortName(t *testing.T) {
@@ -22,3 +24,22 @@ func TestAClaimedActorIsOneShortName(t *testing.T) {
 		}
 	}
 }
+
+func TestAReservationThatPlacementCannotTrustIsSaid(t *testing.T) {
+	cpu := func(mem int) catalog.Model {
+		return catalog.Model{Name: "m", Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp", MinMemoryGB: mem}}}
+	}
+	if w := reservationWarnings(cpu(0), 0); len(w) != 1 || !strings.Contains(w[0], "declares no memory") {
+		t.Errorf("no memory declared: %v", w)
+	}
+	if w := reservationWarnings(cpu(6), 8.2); len(w) != 1 || !strings.Contains(w[0], "uses 8.2 GB and reserves 6") {
+		t.Errorf("under-declared: %v", w)
+	}
+	if w := reservationWarnings(cpu(6), 6.4); len(w) != 0 {
+		t.Errorf("within a tenth is mapped weights, not a wrong figure: %v", w)
+	}
+	gpu := catalog.Model{Name: "g", Profiles: []catalog.Profile{{Name: "tp1", Engine: "vllm", GPUClass: "h100-80g"}}}
+	if w := reservationWarnings(gpu, 3); len(w) != 0 {
+		t.Errorf("a GPU model needs no host memory figure: %v", w)
+	}
+}

internal/control/adminapi/api.go

@@ -15,6 +15,7 @@ import (
 	"io"
 	"net/http"
 	"slices"
+	"strconv"
 	"strings"
 	"time"
 	"unicode"
@@ -41,6 +42,7 @@ type API struct {
 	models   Models
 	capacity Capacity
 	usage    Usage
+	quotas   Quotas
 	ctl      *control.Controller
 }
 
@@ -56,6 +58,28 @@ func (a *API) Register(mux *http.ServeMux) {
 	mux.HandleFunc("POST /api/v1/models", a.addModel)
 	mux.HandleFunc("DELETE /api/v1/models/{name}", a.removeModel)
 	mux.HandleFunc("POST /api/v1/models/{name}/scale", a.scaleModel)
+	mux.HandleFunc("POST /api/v1/models/{name}/resources", a.setResources)
+	a.registerQuotas(mux)
+	mux.HandleFunc("GET /api/v1/models/{name}/target", a.target)
+	mux.HandleFunc("POST /api/v1/models/{name}/class", func(w http.ResponseWriter, r *http.Request) {
+		if a.models == nil {
+			writeError(w, http.StatusNotImplemented, "this control plane has no store")
+			return
+		}
+		var req struct {
+			Class string `json:"class"`
+		}
+		if err := json.NewDecoder(io.LimitReader(r.Body, 1<<12)).Decode(&req); err != nil {
+			writeError(w, http.StatusBadRequest, "the request is not valid JSON: "+err.Error())
+			return
+		}
+		m, err := a.models.SetClass(r.Context(), r.PathValue("name"), req.Class, actorOf(r))
+		if err != nil {
+			writeError(w, http.StatusBadRequest, err.Error())
+			return
+		}
+		writeJSON(w, http.StatusOK, map[string]string{"model": m.Name, "class": m.EffectiveClass()})
+	})
 	mux.HandleFunc("PUT /api/v1/models/{name}/capacity", a.putCapacity)
 	mux.HandleFunc("GET /api/v1/capacity", a.listCapacity)
 	mux.HandleFunc("GET /api/v1/usage", a.usageByScope)
@@ -91,6 +115,15 @@ func (a *API) Register(mux *http.ServeMux) {
 	mux.HandleFunc("GET /api/v1/jobs/{id}", a.job)
 	mux.HandleFunc("GET /api/v1/requests/{id}", a.request)
 	mux.HandleFunc("POST /api/v1/models/{name}/retry", a.retryModel)
+	mux.HandleFunc("GET /api/v1/models/{name}/headroom", func(w http.ResponseWriter, r *http.Request) {
+		limit, _ := strconv.Atoi(r.URL.Query().Get("limit"))
+		v, ok := a.Headroom(r.PathValue("name"), limit)
+		if !ok {
+			http.Error(w, "no such model", http.StatusNotFound)
+			return
+		}
+		writeJSON(w, http.StatusOK, v)
+	})
 }
 
 // --- responses ---
@@ -109,6 +142,8 @@ type NodeView struct {
 	// what decides whether a CPU profile can be placed here, so it belongs
 	// where someone looks when a model will not schedule.
 	MemoryGB int `json:"memory_gb,omitempty"`
+	// CPUCount is the machine's CPUs, 0 when the agent did not report them.
+	CPUCount int `json:"cpu_count,omitempty"`
 	// Engines are the inference engines installed here, as the agent found
 	// them. Absent means the agent is older than the probe and said nothing,
 	// which is not the same as having none — placement treats it that way too,
@@ -214,6 +249,11 @@ type LoadView struct {
 	// MemPercent is used over used plus available.
 	MemPercent float64 `json:"memory_percent"`
 	AgeSec     int64   `json:"age_seconds"`
+	// What the node's sensors say; zero is not measured.
+	CPUTempC   float64               `json:"cpu_temp_c,omitempty"`
+	CPUPowerW  float64               `json:"cpu_power_w,omitempty"`
+	InletTempC float64               `json:"inlet_temp_c,omitempty"`
+	GPUs       []inventory.GPUSensor `json:"gpu_sensors,omitempty"`
 }
 
 // loadFresh is how old a measurement may be and still be shown as current.
@@ -227,7 +267,8 @@ func loadView(l *inventory.Load, now time.Time) *LoadView {
 		CPUPercent: l.CPUPercent, CPUs: l.CPUCount,
 		Load1: l.Load1, Load5: l.Load5, Load15: l.Load15,
 		MemUsed: l.MemUsedBytes, MemTotal: l.MemUsedBytes + l.MemAvailBytes,
-		AgeSec: int64(now.Sub(l.At).Seconds()),
+		AgeSec:   int64(now.Sub(l.At).Seconds()),
+		CPUTempC: l.CPUTempC, CPUPowerW: l.CPUPowerW, InletTempC: l.InletTempC, GPUs: l.GPUs,
 	}
 	if v.MemTotal > 0 {
 		v.MemPercent = float64(v.MemUsed) * 100 / float64(v.MemTotal)
@@ -260,6 +301,21 @@ type ModelView struct {
 	Unplaceable bool     `json:"unplaceable,omitempty"`
 	Fallbacks   []string `json:"fallbacks,omitempty"`
 	Profiles    []string `json:"profiles"`
+	// MemoryGB and CPUCores are what one replica reserves, from its first
+	// profile; MeasuredMemoryGB is the most one of its replicas uses now, 0
+	// when none is measured.
+	MemoryGB         int     `json:"memory_gb,omitempty"`
+	CPUCores         int     `json:"cpu_cores,omitempty"`
+	MeasuredMemoryGB float64 `json:"measured_memory_gb,omitempty"`
+	// Warnings are what makes its reservation wrong: nothing declared where
+	// placement needs a figure, or less declared than it uses.
+	Warnings []string `json:"warnings,omitempty"`
+	// Class is its capacity class; Preempted the replicas it lost to a higher
+	// class; HeldFor the model it is giving room to now, if any.
+	Class     string `json:"class"`
+	Owner     string `json:"owner,omitempty"`
+	Preempted int    `json:"preempted,omitempty"`
+	HeldFor   string `json:"held_for,omitempty"`
 }
 
 // ReplicaView is a replica as `ctl replicas` shows it.
@@ -336,6 +392,28 @@ type ExplainView struct {
 	Rejections []RejectionView `json:"rejections,omitempty"`
 }
 
+// HeadroomView is how many more replicas of a model fit now, and where.
+type HeadroomView struct {
+	Model string `json:"model"`
+	// Current is the replicas holding capacity: starting or ready.
+	Current int `json:"current"`
+	// More is how many more placement would find room for now.
+	More int `json:"more"`
+	// Nodes is where each of those would go.
+	Nodes [][]string `json:"nodes,omitempty"`
+	// Limited is set when More stopped at the limit asked rather than at the
+	// cluster's edge: there may be room for more.
+	Limited bool `json:"limited,omitempty"`
+	// Blocked says why one more does not fit, a line per node considered.
+	Blocked []string `json:"blocked,omitempty"`
+	// TokPerSec is what one more replica was measured to serve; 0 unmeasured.
+	TokPerSec float64 `json:"tok_per_sec,omitempty"`
+	// ByPreempting is how many more would fit by stopping replicas this
+	// model's class may take; Preempts the models those belong to.
+	ByPreempting int      `json:"by_preempting,omitempty"`
+	Preempts     []string `json:"preempts,omitempty"`
+}
+
 // RejectionView is one rejected candidate.
 type RejectionView struct {
 	Profile string `json:"profile"`
@@ -370,8 +448,8 @@ func (a *API) Nodes() []NodeView {
 			ID: n.ID, Hostname: n.Hostname, MgmtAddr: n.MgmtAddr,
 			State: n.State.String(), Labels: n.Labels,
 			GPUTotal: len(n.GPUs), GPUFree: len(inv.FreeGPUs(n.ID)),
-			MemoryGB: n.MemoryTotalMB / 1024,
-			Fabrics:  n.FabricNames(), AgentVersion: n.AgentVersion, Features: n.Features,
+			MemoryGB: n.MemoryTotalMB / 1024, CPUCount: n.CPUCount,
+			Fabrics: n.FabricNames(), AgentVersion: n.AgentVersion, Features: n.Features,
 			Replicas: replicasOn[n.ID],
 		}
 		if free, total := n.LocalDisk(); total > 0 {
@@ -471,6 +549,15 @@ func (a *API) Models() []ModelView {
 	for _, h := range a.ctl.Reconciler().ModelHealth() {
 		health[h.Model] = h
 	}
+	// The most one replica of each model uses, summed over its shards.
+	measured := map[string]int64{}
+	for _, rep := range a.Replicas("") {
+		var b int64
+		for _, sh := range rep.Shards {
+			b += sh.MemoryBytes
+		}
+		measured[rep.Model] = max(measured[rep.Model], b)
+	}
 	out := make([]ModelView, 0)
 	for _, m := range a.ctl.Catalog().Models() {
 		h := health[m.Name]
@@ -481,10 +568,16 @@ func (a *API) Models() []ModelView {
 			Fallbacks: m.Fallbacks,
 			Desired:   h.Desired, Failures: h.Failures,
 			Parked: h.Parked, Reason: h.Reason, Unplaceable: h.Unplaceable,
+			Class: m.EffectiveClass(), Owner: m.Owner, Preempted: h.Preempted, HeldFor: h.HeldFor,
 		}
 		for _, p := range m.Profiles {
 			v.Profiles = append(v.Profiles, p.Name)
 		}
+		if len(m.Profiles) > 0 {
+			v.MemoryGB, v.CPUCores = m.Profiles[0].MinMemoryGB, m.Profiles[0].CPUCores
+		}
+		v.MeasuredMemoryGB = float64(measured[m.Name]) / (1 << 30)
+		v.Warnings = reservationWarnings(m, v.MeasuredMemoryGB)
 		out = append(out, v)
 	}
 	return out
@@ -555,6 +648,17 @@ type Models interface {
 	Add(ctx context.Context, m catalog.Model, by string) error
 	Remove(ctx context.Context, name string) error
 	Scale(ctx context.Context, name string, s catalog.Scaling, by string) (catalog.Model, error)
+	Resources(ctx context.Context, name, profile string, memoryGB, cpuCores int, by string) (catalog.Model, error)
+	SetClass(ctx context.Context, name, class, by string) (catalog.Model, error)
+	SetOwner(ctx context.Context, name, owner, by string) (catalog.Model, error)
+}
+
+// ResourcesRequest sets what a model's profiles reserve of a node.
+type ResourcesRequest struct {
+	// Profile is one profile's name; empty is every profile.
+	Profile  string `json:"profile,omitempty"`
+	MemoryGB int    `json:"memory_gb"`
+	CPUCores int    `json:"cpu_cores"`
 }
 
 // Usage reports what each group of the tenancy tree spent. The store
@@ -800,6 +904,26 @@ func (a *API) scaleModel(w http.ResponseWriter, r *http.Request) {
 	})
 }
 
+// setResources sets a model's reserved memory and cores. Placed replicas keep
+// what they were started with until they are restarted.
+func (a *API) setResources(w http.ResponseWriter, r *http.Request) {
+	if a.models == nil {
+		writeError(w, http.StatusNotImplemented, "this control plane has no store")
+		return
+	}
+	var req ResourcesRequest
+	if err := json.NewDecoder(io.LimitReader(r.Body, 1<<16)).Decode(&req); err != nil {
+		writeError(w, http.StatusBadRequest, "the request is not valid JSON: "+err.Error())
+		return
+	}
+	m, err := a.models.Resources(r.Context(), r.PathValue("name"), req.Profile, req.MemoryGB, req.CPUCores, actorOf(r))
+	if err != nil {
+		writeError(w, http.StatusBadRequest, err.Error())
+		return
+	}
+	writeJSON(w, http.StatusOK, m)
+}
+
 func (a *API) removeModel(w http.ResponseWriter, r *http.Request) {
 	if a.models == nil {
 		writeError(w, http.StatusNotImplemented, "this control plane has no store")
@@ -836,6 +960,21 @@ func (a *API) explainPlacement(w http.ResponseWriter, r *http.Request) {
 // It does not check that the model exists; the handler above does that so it
 // can answer 404, while the dashboard only ever asks about models it just read
 // out of the catalog.
+// Headroom tries placing up to limit more replicas of a model, on a copy of
+// the inventory, and says how many fit. A limit of zero or less takes 16.
+func (a *API) Headroom(model string, limit int) (HeadroomView, bool) {
+	if limit <= 0 || limit > 64 {
+		limit = 16
+	}
+	h, ok := a.ctl.Reconciler().Headroom(model, limit)
+	if !ok {
+		return HeadroomView{}, false
+	}
+	return HeadroomView{Model: model, Current: h.Current, More: h.More, Nodes: h.Nodes,
+		Limited: h.Limited, Blocked: h.Blocked, TokPerSec: h.TokPerSec,
+		ByPreempting: h.ByPreempting, Preempts: h.Preempts}, true
+}
+
 func (a *API) Explain(model string) ExplainView {
 	v := ExplainView{Model: model}
 	for _, rep := range a.ctl.Reconciler().Replicas() {
@@ -1267,3 +1406,40 @@ func actorOf(r *http.Request) string {
 	}
 	return b.String()
 }
+
+// NodeCPUs is the CPU counts of ready nodes with at least this much memory:
+// where a replica needing it could go.
+func (a *API) NodeCPUs(minMemoryGB int) []int {
+	var out []int
+	for _, n := range a.ctl.Inventory().Nodes() {
+		if n.State.String() != "ready" || n.CPUCount == 0 {
+			continue
+		}
+		if n.MemoryTotalMB > 0 && n.MemoryTotalMB/1024 < minMemoryGB {
+			continue
+		}
+		out = append(out, n.CPUCount)
+	}
+	return out
+}
+
+// reservationWarnings says what is wrong with what a model reserves.
+//
+// Resident memory includes the weights mapped from disk, which the kernel can
+// take back, so it overstates what a model needs by the part of the weights
+// not in use. A tenth over is let pass for that; more is a figure too low for
+// placement to be trusted with.
+func reservationWarnings(m catalog.Model, measuredGB float64) []string {
+	var out []string
+	for _, p := range m.Profiles {
+		if p.CPUOnly() && p.MinMemoryGB <= 0 {
+			out = append(out, fmt.Sprintf("profile %s declares no memory: placement cannot tell when a node is full "+
+				"(llmgw ctl model resources %s --memory GB)", p.Name, m.Name))
+		}
+	}
+	if len(m.Profiles) == 1 && m.Profiles[0].MinMemoryGB > 0 && measuredGB > float64(m.Profiles[0].MinMemoryGB)*1.1 {
+		out = append(out, fmt.Sprintf("uses %.1f GB and reserves %d: raise it, or nodes will be filled past their memory",
+			measuredGB, m.Profiles[0].MinMemoryGB))
+	}
+	return out
+}

internal/control/adminapi/api_test.go

@@ -258,6 +258,21 @@ type fakeModels struct {
 	err     error
 }
 
+func (f *fakeModels) SetOwner(_ context.Context, name, owner, _ string) (catalog.Model, error) {
+	return catalog.Model{Name: name, Owner: owner}, f.err
+}
+
+func (f *fakeModels) SetClass(_ context.Context, name, class, _ string) (catalog.Model, error) {
+	return catalog.Model{Name: name, Class: class}, f.err
+}
+
+func (f *fakeModels) Resources(_ context.Context, name, profile string, mem, cores int, _ string) (catalog.Model, error) {
+	if f.err != nil {
+		return catalog.Model{}, f.err
+	}
+	return catalog.Model{Name: name, Profiles: []catalog.Profile{{Name: "p", MinMemoryGB: mem, CPUCores: cores}}}, nil
+}
+
 func (f *fakeModels) Scale(_ context.Context, name string, s catalog.Scaling, by string) (catalog.Model, error) {
 	if f.err != nil {
 		return catalog.Model{}, f.err

internal/control/adminapi/quotas.go

@@ -0,0 +1,285 @@
+package adminapi
+
+import (
+	"context"
+	"encoding/json"
+	"fmt"
+	"io"
+	"net/http"
+	"sort"
+	"strings"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// Quotas is where capacity quotas and the tenancy tree they name are kept.
+// The store implements it.
+type Quotas interface {
+	CapacityQuotas(ctx context.Context) ([]store.CapacityQuota, error)
+	PutCapacityQuota(ctx context.Context, q store.CapacityQuota) error
+	DeleteCapacityQuota(ctx context.Context, scope store.Scope, id string) error
+	ScopeParents(ctx context.Context) (map[string][]string, error)
+	AllOrgs(ctx context.Context) ([]store.Org, error)
+	AllTeams(ctx context.Context) ([]store.TeamPath, error)
+	AllProjects(ctx context.Context) ([]store.ProjectPath, error)
+}
+
+// SetQuotas supplies the quota store, which mounts the quota endpoints'
+// behaviour; without it they answer 501.
+func (a *API) SetQuotas(q Quotas) { a.quotas = q }
+
+// QuotaView is one scope's quota with what it holds now.
+type QuotaView struct {
+	// Scope is team:ID; Name is how people know it, "Acme / ml".
+	Scope     string          `json:"scope"`
+	Name      string          `json:"name"`
+	Guarantee store.Resources `json:"guarantee"`
+	Limit     store.Resources `json:"limit"`
+	Holds     quota.Amount    `json:"holds"`
+	UpdatedBy string          `json:"updated_by,omitempty"`
+	UpdatedAt time.Time       `json:"updated_at"`
+}
+
+// QuotaRequest sets a scope's quota. Scope may be written with an ID or a
+// name: org:Acme, team:Acme/ml, project:Acme/ml/training.
+type QuotaRequest struct {
+	Scope     string          `json:"scope"`
+	Guarantee store.Resources `json:"guarantee"`
+	Limit     store.Resources `json:"limit"`
+}
+
+func (a *API) registerQuotas(mux *http.ServeMux) {
+	mux.HandleFunc("GET /api/v1/quotas", func(w http.ResponseWriter, r *http.Request) {
+		if a.quotas == nil {
+			writeError(w, http.StatusNotImplemented, "this control plane has no store")
+			return
+		}
+		views, err := a.QuotaViews(r.Context())
+		if err != nil {
+			writeError(w, http.StatusInternalServerError, err.Error())
+			return
+		}
+		writeJSON(w, http.StatusOK, views)
+	})
+	mux.HandleFunc("PUT /api/v1/quotas", func(w http.ResponseWriter, r *http.Request) {
+		if a.quotas == nil {
+			writeError(w, http.StatusNotImplemented, "this control plane has no store")
+			return
+		}
+		var req QuotaRequest
+		if err := json.NewDecoder(io.LimitReader(r.Body, 1<<14)).Decode(&req); err != nil {
+			writeError(w, http.StatusBadRequest, "the request is not valid JSON: "+err.Error())
+			return
+		}
+		key, err := a.SetQuota(r.Context(), req, actorOf(r))
+		if err != nil {
+			writeError(w, http.StatusBadRequest, err.Error())
+			return
+		}
+		writeJSON(w, http.StatusOK, map[string]string{"scope": key})
+	})
+	mux.HandleFunc("DELETE /api/v1/quotas/{scope}", func(w http.ResponseWriter, r *http.Request) {
+		if a.quotas == nil {
+			writeError(w, http.StatusNotImplemented, "this control plane has no store")
+			return
+		}
+		key, err := a.ClearQuota(r.Context(), r.PathValue("scope"))
+		if err != nil {
+			writeError(w, http.StatusBadRequest, err.Error())
+			return
+		}
+		writeJSON(w, http.StatusOK, map[string]string{"removed": key})
+	})
+	mux.HandleFunc("POST /api/v1/models/{name}/owner", func(w http.ResponseWriter, r *http.Request) {
+		if a.models == nil || a.quotas == nil {
+			writeError(w, http.StatusNotImplemented, "this control plane has no store")
+			return
+		}
+		var req struct {
+			Owner string `json:"owner"`
+		}
+		if err := json.NewDecoder(io.LimitReader(r.Body, 1<<12)).Decode(&req); err != nil {
+			writeError(w, http.StatusBadRequest, "the request is not valid JSON: "+err.Error())
+			return
+		}
+		owner := ""
+		if req.Owner != "" {
+			scope, id, _, err := a.resolveScope(r.Context(), req.Owner)
+			if err != nil {
+				writeError(w, http.StatusBadRequest, err.Error())
+				return
+			}
+			owner = quota.Key(scope, id)
+		}
+		m, err := a.models.SetOwner(r.Context(), r.PathValue("name"), owner, actorOf(r))
+		if err != nil {
+			writeError(w, http.StatusBadRequest, err.Error())
+			return
+		}
+		writeJSON(w, http.StatusOK, map[string]string{"model": m.Name, "owner": m.Owner})
+	})
+}
+
+// QuotaViews lists every quota with what its scope holds now.
+func (a *API) QuotaViews(ctx context.Context) ([]QuotaView, error) {
+	if a.quotas == nil {
+		return nil, nil
+	}
+	qs, err := a.quotas.CapacityQuotas(ctx)
+	if err != nil {
+		return nil, err
+	}
+	parents, err := a.quotas.ScopeParents(ctx)
+	if err != nil {
+		return nil, err
+	}
+	names, err := a.scopeNames(ctx)
+	if err != nil {
+		return nil, err
+	}
+	usage := quota.New(qs, parents).Usage(a.ctl.Reconciler().Holdings())
+	out := make([]QuotaView, 0, len(qs))
+	for _, q := range qs {
+		key := quota.Key(q.Scope, q.ScopeID)
+		out = append(out, QuotaView{Scope: key, Name: names[key], Guarantee: q.Guarantee, Limit: q.Limit,
+			Holds: usage[key], UpdatedBy: q.UpdatedBy, UpdatedAt: q.UpdatedAt})
+	}
+	return out, nil
+}
+
+// scopeNames maps org:ID, team:ID and project:ID to their paths.
+func (a *API) scopeNames(ctx context.Context) (map[string]string, error) {
+	out := map[string]string{}
+	orgs, err := a.quotas.AllOrgs(ctx)
+	if err != nil {
+		return nil, err
+	}
+	for _, o := range orgs {
+		out["org:"+o.ID] = o.Name
+	}
+	teams, err := a.quotas.AllTeams(ctx)
+	if err != nil {
+		return nil, err
+	}
+	for _, t := range teams {
+		out["team:"+t.ID] = t.OrgName + " / " + t.Name
+	}
+	projects, err := a.quotas.AllProjects(ctx)
+	if err != nil {
+		return nil, err
+	}
+	for _, p := range projects {
+		out["project:"+p.ID] = p.Path
+	}
+	return out, nil
+}
+
+// resolveScope reads org:X, team:X or project:X, where X is an ID or the
+// scope's path — Acme, Acme/ml, Acme/ml/training — and returns its ID and
+// path. A path that could mean two scopes is refused rather than guessed.
+func (a *API) resolveScope(ctx context.Context, s string) (store.Scope, string, string, error) {
+	kind, what, ok := strings.Cut(strings.TrimSpace(s), ":")
+	scope := store.Scope(kind)
+	if !ok || what == "" || (scope != store.ScopeOrg && scope != store.ScopeTeam && scope != store.ScopeProject) {
+		return "", "", "", fmt.Errorf("%q is not org:NAME, team:ORG/TEAM or project:ORG/TEAM/PROJECT", s)
+	}
+	names, err := a.scopeNames(ctx)
+	if err != nil {
+		return "", "", "", err
+	}
+	norm := func(p string) string {
+		parts := strings.Split(p, "/")
+		for i := range parts {
+			parts[i] = strings.ToLower(strings.TrimSpace(parts[i]))
+		}
+		return strings.Join(parts, "/")
+	}
+	if name, ok := names[kind+":"+what]; ok {
+		return scope, what, name, nil
+	}
+	match := func(same func(name string) bool) []string {
+		var found []string
+		for key, name := range names {
+			if strings.HasPrefix(key, kind+":") && same(name) {
+				found = append(found, key)
+			}
+		}
+		return found
+	}
+	found := match(func(name string) bool { return norm(name) == norm(what) })
+	if len(found) == 0 {
+		// Its own name alone — ml rather than Acme/ml — when that is unique.
+		found = match(func(name string) bool {
+			parts := strings.Split(norm(name), "/")
+			return parts[len(parts)-1] == norm(what)
+		})
+	}
+	switch len(found) {
+	case 0:
+		return "", "", "", fmt.Errorf("there is no %s %q", kind, what)
+	case 1:
+		_, id, _ := strings.Cut(found[0], ":")
+		return scope, id, names[found[0]], nil
+	}
+	return "", "", "", fmt.Errorf("%q could be any of %d %ss; give its ID", what, len(found), kind)
+}
+
+// SetQuota sets a scope's capacity quota, named by path or ID, and returns
+// the scope as org:ID.
+func (a *API) SetQuota(ctx context.Context, req QuotaRequest, by string) (string, error) {
+	if a.quotas == nil {
+		return "", fmt.Errorf("this control plane has no store")
+	}
+	scope, id, _, err := a.resolveScope(ctx, req.Scope)
+	if err != nil {
+		return "", err
+	}
+	if !req.Guarantee.Any() && !req.Limit.Any() {
+		return "", fmt.Errorf("a quota needs a guarantee or a limit; to remove one, clear it")
+	}
+	q := store.CapacityQuota{Scope: scope, ScopeID: id, Guarantee: req.Guarantee, Limit: req.Limit, UpdatedBy: by}
+	if err := a.quotas.PutCapacityQuota(ctx, q); err != nil {
+		return "", err
+	}
+	return quota.Key(scope, id), nil
+}
+
+// ClearQuota removes a scope's capacity quota.
+func (a *API) ClearQuota(ctx context.Context, scope string) (string, error) {
+	if a.quotas == nil {
+		return "", fmt.Errorf("this control plane has no store")
+	}
+	s, id, _, err := a.resolveScope(ctx, scope)
+	if err != nil {
+		return "", err
+	}
+	if err := a.quotas.DeleteCapacityQuota(ctx, s, id); err != nil {
+		return "", err
+	}
+	return quota.Key(s, id), nil
+}
+
+// ScopeName is a part of the tenancy tree: org:ID and its path.
+type ScopeName struct {
+	Key  string `json:"scope"`
+	Name string `json:"name"`
+}
+
+// Scopes lists every organization, team and project, by path.
+func (a *API) Scopes(ctx context.Context) ([]ScopeName, error) {
+	if a.quotas == nil {
+		return nil, nil
+	}
+	names, err := a.scopeNames(ctx)
+	if err != nil {
+		return nil, err
+	}
+	out := make([]ScopeName, 0, len(names))
+	for k, n := range names {
+		out = append(out, ScopeName{Key: k, Name: n})
+	}
+	sort.Slice(out, func(i, j int) bool { return out[i].Name < out[j].Name })
+	return out, nil
+}

internal/control/adminapi/quotas_test.go

@@ -0,0 +1,107 @@
+package adminapi
+
+import (
+	"context"
+	"encoding/json"
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// fakeQuotaStore is Acme with teams ml and web, and Scania with another ml.
+type fakeQuotaStore struct {
+	quotas []store.CapacityQuota
+}
+
+func (f *fakeQuotaStore) CapacityQuotas(context.Context) ([]store.CapacityQuota, error) {
+	return f.quotas, nil
+}
+func (f *fakeQuotaStore) PutCapacityQuota(_ context.Context, q store.CapacityQuota) error {
+	for i := range f.quotas {
+		if f.quotas[i].Scope == q.Scope && f.quotas[i].ScopeID == q.ScopeID {
+			f.quotas[i] = q
+			return nil
+		}
+	}
+	f.quotas = append(f.quotas, q)
+	return nil
+}
+func (f *fakeQuotaStore) DeleteCapacityQuota(_ context.Context, s store.Scope, id string) error {
+	for i, q := range f.quotas {
+		if q.Scope == s && q.ScopeID == id {
+			f.quotas = append(f.quotas[:i], f.quotas[i+1:]...)
+		}
+	}
+	return nil
+}
+func (f *fakeQuotaStore) ScopeParents(context.Context) (map[string][]string, error) {
+	return map[string][]string{"team:t-ml": {"org:o-acme"}, "team:t-web": {"org:o-acme"},
+		"team:t-sml": {"org:o-scania"}}, nil
+}
+func (f *fakeQuotaStore) AllOrgs(context.Context) ([]store.Org, error) {
+	return []store.Org{{ID: "o-acme", Name: "Acme"}, {ID: "o-scania", Name: "Scania"}}, nil
+}
+func (f *fakeQuotaStore) AllTeams(context.Context) ([]store.TeamPath, error) {
+	return []store.TeamPath{{ID: "t-ml", Name: "ml", OrgID: "o-acme", OrgName: "Acme"},
+		{ID: "t-web", Name: "web", OrgID: "o-acme", OrgName: "Acme"},
+		{ID: "t-sml", Name: "ml", OrgID: "o-scania", OrgName: "Scania"}}, nil
+}
+func (f *fakeQuotaStore) AllProjects(context.Context) ([]store.ProjectPath, error) { return nil, nil }
+
+func TestAScopeIsNamedByPathOrIDAndNeverGuessed(t *testing.T) {
+	a, _ := setup(t, testModels())
+	a.SetQuotas(&fakeQuotaStore{})
+	for in, want := range map[string]string{
+		"org:Acme": "o-acme", "org:o-acme": "o-acme", "team:Acme/ml": "t-ml",
+		"team: acme / ML ": "t-ml", "team:Scania/ml": "t-sml", "team:web": "t-web",
+	} {
+		_, id, _, err := a.resolveScope(t.Context(), in)
+		if err != nil || id != want {
+			t.Errorf("%q: %q %v, expected %q", in, id, err, want)
+		}
+	}
+	for _, bad := range []string{"team:ml", "team:Acme/nope", "group:Acme", "Acme", "org:"} {
+		if _, _, _, err := a.resolveScope(t.Context(), bad); err == nil {
+			t.Errorf("%q was resolved", bad)
+		}
+	}
+	if _, _, _, err := a.resolveScope(t.Context(), "team:ml"); err == nil || !strings.Contains(err.Error(), "could be any of 2") {
+		t.Errorf("an ambiguous name should say so: %v", err)
+	}
+}
+
+func TestAQuotaIsSetListedAndCleared(t *testing.T) {
+	a, _ := setup(t, testModels())
+	qs := &fakeQuotaStore{}
+	a.SetQuotas(qs)
+	mux := http.NewServeMux()
+	a.Register(mux)
+	call := func(method, path, body string) *httptest.ResponseRecorder {
+		req := httptest.NewRequest(method, path, strings.NewReader(body))
+		req.Header.Set("Content-Type", "application/json")
+		req.Header.Set("X-Llmgw-Actor", "ops")
+		rec := httptest.NewRecorder()
+		mux.ServeHTTP(rec, req)
+		return rec
+	}
+	if rec := call("PUT", "/api/v1/quotas", `{"scope":"team:Acme/ml","guarantee":{"gpus":8},"limit":{"gpus":16}}`); rec.Code != 200 {
+		t.Fatalf("set: %d %s", rec.Code, rec.Body)
+	}
+	if len(qs.quotas) != 1 || qs.quotas[0].ScopeID != "t-ml" || *qs.quotas[0].Guarantee.GPUs != 8 || qs.quotas[0].UpdatedBy != "ops" {
+		t.Errorf("stored %+v", qs.quotas)
+	}
+	rec := call("GET", "/api/v1/quotas", "")
+	var views []QuotaView
+	if err := json.Unmarshal(rec.Body.Bytes(), &views); err != nil || len(views) != 1 || views[0].Name != "Acme / ml" {
+		t.Errorf("list %d %s", rec.Code, rec.Body)
+	}
+	if rec := call("PUT", "/api/v1/quotas", `{"scope":"team:Acme/ml"}`); rec.Code != 400 {
+		t.Errorf("an empty quota was accepted: %d", rec.Code)
+	}
+	if rec := call("DELETE", "/api/v1/quotas/team:Acme%2Fml", ""); rec.Code != 200 || len(qs.quotas) != 0 {
+		t.Errorf("clear: %d %s %+v", rec.Code, rec.Body, qs.quotas)
+	}
+}

internal/control/adminapi/target.go

@@ -0,0 +1,69 @@
+package adminapi
+
+import (
+	"fmt"
+	"math"
+	"net/http"
+	"strconv"
+)
+
+// TargetView is how many replicas a model needs to serve a throughput, from
+// what one replica was measured to serve, and whether the cluster has room.
+type TargetView struct {
+	Model string `json:"model"`
+	// TokPerSec is what was asked for; PerReplica what one replica was
+	// measured to serve at its SLO, on Profile.
+	TokPerSec  float64 `json:"tok_per_sec"`
+	PerReplica float64 `json:"per_replica_tok_per_sec"`
+	Profile    string  `json:"profile"`
+	// Needed replicas in all; Current hold capacity now; More is how many of
+	// Needed are not running, and Room how many more fit.
+	Needed  int      `json:"needed"`
+	Current int      `json:"current"`
+	More    int      `json:"more"`
+	Room    int      `json:"room"`
+	Fits    bool     `json:"fits"`
+	Blocked []string `json:"blocked,omitempty"`
+	// Preempts is whose replicas would be stopped to make the room, when
+	// free room alone is not enough; Room counts that room too.
+	Preempts []string `json:"preempts,omitempty"`
+}
+
+// Target sizes a model for a throughput.
+func (a *API) Target(model string, tokPerSec float64) (TargetView, error) {
+	meas, profile, ok := a.ctl.Reconciler().MeasuredFor(model)
+	if !ok || meas.TokPerSec <= 0 {
+		return TargetView{}, fmt.Errorf("%s has not been measured; run `llmgw ctl bench %s --save` first", model, model)
+	}
+	v := TargetView{Model: model, TokPerSec: tokPerSec, PerReplica: meas.TokPerSec, Profile: profile,
+		Needed: max(int(math.Ceil(tokPerSec/meas.TokPerSec)), 1)}
+	h, ok := a.Headroom(model, v.Needed)
+	if !ok {
+		return TargetView{}, fmt.Errorf("no model %q", model)
+	}
+	v.Current = h.Current
+	v.More = max(v.Needed-h.Current, 0)
+	v.Room = h.More + h.ByPreempting
+	v.Fits = v.More <= v.Room || h.Limited
+	if v.More > h.More {
+		v.Preempts = h.Preempts
+	}
+	if !v.Fits {
+		v.Blocked = h.Blocked
+	}
+	return v, nil
+}
+
+func (a *API) target(w http.ResponseWriter, r *http.Request) {
+	tps, err := strconv.ParseFloat(r.URL.Query().Get("tokens_per_sec"), 64)
+	if err != nil || tps <= 0 {
+		writeError(w, http.StatusBadRequest, "tokens_per_sec must be a positive number")
+		return
+	}
+	v, err := a.Target(r.PathValue("name"), tps)
+	if err != nil {
+		writeError(w, http.StatusBadRequest, err.Error())
+		return
+	}
+	writeJSON(w, http.StatusOK, v)
+}

internal/control/adminapi/target_test.go

@@ -0,0 +1,30 @@
+package adminapi
+
+import (
+	"strings"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
+)
+
+func TestATargetIsSizedFromTheMeasurementAndTheRoom(t *testing.T) {
+	// Three 8-GPU nodes; one tp8 replica was measured at 500 tokens/s.
+	a, ctl := setup(t, testModels(), gpuNode("1", 8), gpuNode("2", 8), gpuNode("3", 8))
+	if _, err := a.Target("llama", 1000); err == nil || !strings.Contains(err.Error(), "ctl bench llama --save") {
+		t.Errorf("an unmeasured model should say how to measure it: %v", err)
+	}
+	ctl.Reconciler().SetMeasurements(func() map[string]reconciler.Measured {
+		return map[string]reconciler.Measured{"llama/tp8": {MaxConcurrency: 16, TokPerSec: 500}}
+	})
+	v, err := a.Target("llama", 1200)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if v.Needed != 3 || v.PerReplica != 500 || v.Profile != "tp8" || v.More != 3 || !v.Fits {
+		t.Errorf("1200 tok/s at 500 each is 3 replicas on 3 empty nodes: %+v", v)
+	}
+	v, _ = a.Target("llama", 2000)
+	if v.Needed != 4 || v.Fits || v.Room != 3 || len(v.Blocked) == 0 {
+		t.Errorf("a fourth does not fit on three nodes, and should say why: %+v", v)
+	}
+}

internal/control/agentsvc/convert.go

@@ -4,6 +4,7 @@ package agentsvc
 import (
 	"fmt"
 	"maps"
+	"runtime"
 	"time"
 
 	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
@@ -27,6 +28,7 @@ func NodeFromRegister(r *llmgwv1.Register) inventory.Node {
 		EngineVersions: r.GetEngineVersions(),
 		EnginesProbed:  r.GetEnginesProbed(),
 		MemoryTotalMB:  int(r.GetMemoryTotalMb()),
+		CPUCount:       int(r.GetCpuCount()),
 	}
 	if h := r.GetHost(); h != nil {
 		n.Host = discovery.Host{
@@ -90,6 +92,7 @@ func RegisterFromInventory(nodeID, hostname, mgmtAddr, agentVersion string,
 		NodeId: nodeID, Hostname: hostname, MgmtAddr: mgmtAddr,
 		Labels: labels, AgentVersion: agentVersion,
 		MemoryTotalMb: int64(inv.MemoryTotalMB),
+		CpuCount:      int32(runtime.NumCPU()),
 	}
 	for _, w := range inv.Weights {
 		r.Weights = append(r.Weights, &llmgwv1.LocalWeights{
@@ -192,6 +195,7 @@ func StartShardFromCommand(cmd reconciler.ShardCommand) *llmgwv1.StartShard {
 		Fabric:           cmd.Fabric,
 		ProfileFp:        cmd.ProfileFP,
 		ApiKey:           cmd.Spec.APIKey,
+		CpuThreads:       int32(cmd.Spec.Threads),
 	}
 }
 
@@ -213,6 +217,7 @@ func ShardSpecFromStart(m *llmgwv1.StartShard) (engines.ShardSpec, time.Duration
 		Fabric:        m.GetFabric(),
 		ProfileFP:     m.GetProfileFp(),
 		APIKey:        m.GetApiKey(),
+		Threads:       int(m.GetCpuThreads()),
 		Profile: engines.Profile{
 			Name: p.GetName(), Engine: p.GetEngine(), GPUClass: p.GetGpuClass(),
 			TP: int(p.GetTp()), PP: int(p.GetPp()),

internal/control/agentsvc/service.go

@@ -345,8 +345,9 @@ func (s *Service) applyHeartbeat(nodeID string, hb *llmgwv1.Heartbeat) {
 	problems := hb.GetNewProblems()
 	storage := hb.GetStorage()
 	memMB := int(hb.GetMemoryTotalMb())
+	cpus := int(hb.GetCpuCount())
 	if len(used) == 0 && len(problems) == 0 && len(storage) == 0 &&
-		!hb.GetWeightsChanged() && memMB == 0 && !hb.GetEnginesChanged() &&
+		!hb.GetWeightsChanged() && memMB == 0 && cpus == 0 && !hb.GetEnginesChanged() &&
 		!hb.GetProvisionChanged() && hb.GetLoad() == nil && len(hb.GetIbPorts()) == 0 &&
 		!hb.GetFabricsChanged() && len(hb.GetGpuFabric()) == 0 && hb.GetImex() == nil {
 		return
@@ -406,6 +407,12 @@ func (s *Service) applyHeartbeat(nodeID string, hb *llmgwv1.Heartbeat) {
 			"from_mb", n.MemoryTotalMB, "to_mb", memMB)
 		n.MemoryTotalMB = memMB
 	}
+	if cpus > 0 && cpus != n.CPUCount {
+		if n.CPUCount > 0 {
+			s.log.Info("node CPUs changed", "node", nodeID, "from", n.CPUCount, "to", cpus)
+		}
+		n.CPUCount = cpus
+	}
 	if hb.GetEnginesChanged() {
 		vers := maps.Clone(hb.GetEngineVersions())
 		if !maps.Equal(vers, n.EngineVersions) {
@@ -538,6 +545,13 @@ func loadFrom(l *llmgwv1.NodeLoad, at time.Time) *inventory.Load {
 			CPUCores: sh.GetCpuCores(), RSSBytes: sh.GetMemoryRssBytes(),
 		}
 	}
+	out.CPUTempC, out.CPUPowerW, out.InletTempC = l.GetCpuTempC(), l.GetCpuPowerW(), l.GetInletTempC()
+	for _, g := range l.GetGpus() {
+		out.GPUs = append(out.GPUs, inventory.GPUSensor{
+			Index: int(g.GetIndex()), TempC: g.GetTempC(), MemoryTempC: g.GetMemoryTempC(),
+			PowerW: g.GetPowerW(), PowerLimitW: g.GetPowerLimitW(), Throttle: g.GetThrottle(),
+		})
+	}
 	return out
 }
 

internal/control/catalog/catalog.go

@@ -74,6 +74,17 @@ type Profile struct {
 	// the control plane.
 	MinMemoryGB int `json:"min_memory_gb,omitempty" koanf:"min_memory_gb"`
 
+	// CPUCores is the cores a replica of a profile with no GPU is given:
+	// reserved out of its node's CPUs by placement, as memory is, and what
+	// its engine computes on (llama.cpp's --threads).
+	//
+	// Reserved because an engine takes every core it is told about, and two
+	// on one node that each think they have all of them spin against each
+	// other: on an 8-vCPU node, two 7B replicas fell from 10 tokens a second
+	// to under one each. Zero reserves nothing and leaves the thread count to
+	// the node, which is every profile from before this.
+	CPUCores int `json:"cpu_cores,omitempty" koanf:"cpu_cores"`
+
 	// MinDiskGB is the free space a node needs before it can hold these
 	// weights.
 	//
@@ -138,10 +149,19 @@ type Model struct {
 	// choice by guesswork.
 	Description string `json:"description,omitempty" koanf:"description"`
 	// Source is the weights path or a HuggingFace id.
-	Source    string    `json:"source,omitempty" koanf:"source"`
-	Profiles  []Profile `json:"profiles,omitempty" koanf:"profiles"`
-	Scaling   Scaling   `json:"scaling,omitempty" koanf:"scaling"`
-	Fallbacks []string  `json:"fallbacks,omitempty" koanf:"fallbacks"`
+	Source   string    `json:"source,omitempty" koanf:"source"`
+	Profiles []Profile `json:"profiles,omitempty" koanf:"profiles"`
+	Scaling  Scaling   `json:"scaling,omitempty" koanf:"scaling"`
+	// Class is how much this model's capacity is worth against another's,
+	// when there is not enough for both: critical, standard or opportunistic.
+	// Empty is standard, or opportunistic for a model loaded on demand. See
+	// EffectiveClass.
+	Class string `json:"class,omitempty" koanf:"class"`
+	// Owner is the part of the tenancy tree whose capacity quota this model
+	// counts against, written org:ID, team:ID or project:ID. Empty is shared:
+	// held by the cluster, counted against nobody.
+	Owner     string   `json:"owner,omitempty" koanf:"owner"`
+	Fallbacks []string `json:"fallbacks,omitempty" koanf:"fallbacks"`
 	// RoutingProfile selects how the router scores endpoints for this model.
 	RoutingProfile string `json:"routing_profile,omitempty" koanf:"routing_profile"`
 	SLO            SLO    `json:"slo,omitempty" koanf:"slo"`
@@ -288,6 +308,12 @@ func (m Model) validate() error {
 	if len(m.Profiles) == 0 {
 		return fmt.Errorf("model %q: at least one profile is required", m.Name)
 	}
+	if m.Owner != "" && !validOwner(m.Owner) {
+		return fmt.Errorf("model %q: owner %q is not org:ID, team:ID or project:ID", m.Name, m.Owner)
+	}
+	if !validClass(m.Class) {
+		return fmt.Errorf("model %q: class %q is not one of critical, standard, opportunistic", m.Name, m.Class)
+	}
 	if m.Scaling.Max < m.Scaling.Min {
 		return fmt.Errorf("model %q: scaling max %d is below min %d", m.Name, m.Scaling.Max, m.Scaling.Min)
 	}
@@ -380,6 +406,7 @@ func (m Model) PlacementProfiles() []placement.Profile {
 			NodeSelector: p.NodeSelector,
 			MinMemoryGB:  p.MinMemoryGB,
 			MinDiskGB:    p.MinDiskGB,
+			CPUCores:     p.CPUCores,
 			// The model's source, so placement can prefer nodes that already
 			// hold the weights rather than one that must first move them.
 			Source: m.Source,
@@ -475,3 +502,65 @@ func (p Profile) Fingerprint() string {
 	sum := sha256.Sum256(b)
 	return hex.EncodeToString(sum[:8])
 }
+
+// SuggestCores is the cores to give a replica with no GPU on nodes of these
+// CPU counts: half of the smallest, so two replicas share any of them without
+// competing. Zero when no node says, which reserves nothing.
+func SuggestCores(cpuCounts []int) int {
+	smallest := 0
+	for _, c := range cpuCounts {
+		if c > 0 && (smallest == 0 || c < smallest) {
+			smallest = c
+		}
+	}
+	if smallest == 0 {
+		return 0
+	}
+	return max(smallest/2, 1)
+}
+
+// The capacity classes, lowest first.
+const (
+	ClassOpportunistic = "opportunistic"
+	ClassStandard      = "standard"
+	ClassCritical      = "critical"
+)
+
+func validClass(c string) bool {
+	return c == "" || c == ClassOpportunistic || c == ClassStandard || c == ClassCritical
+}
+
+// EffectiveClass is the class a model is treated as: what it says, or, when
+// it says nothing, opportunistic for one loaded on demand — it holds capacity
+// only while asked for, and has always been the first to give it up — and
+// standard for everything else.
+func (m Model) EffectiveClass() string {
+	if m.Class != "" {
+		return m.Class
+	}
+	if m.Scaling.OnDemand() {
+		return ClassOpportunistic
+	}
+	return ClassStandard
+}
+
+// ClassRank orders classes: a higher rank may take capacity from a lower one.
+func ClassRank(c string) int {
+	switch c {
+	case ClassCritical:
+		return 3
+	case ClassStandard, "":
+		return 2
+	default:
+		return 1
+	}
+}
+
+func validOwner(o string) bool {
+	for _, p := range []string{"org:", "team:", "project:"} {
+		if strings.HasPrefix(o, p) && len(o) > len(p) {
+			return true
+		}
+	}
+	return false
+}

internal/control/control.go

@@ -375,3 +375,6 @@ func EngineKeys(secret []byte) func(replicaID string) string {
 		return "llmgw-" + hex.EncodeToString(m.Sum(nil))[:40]
 	}
 }
+
+// QuotaUse is every capacity quota beside what its scope holds.
+func (c *Controller) QuotaUse() []reconciler.QuotaUse { return c.rec.QuotaUse() }

internal/control/fleetmetrics/fleetmetrics.go

@@ -51,12 +51,16 @@ type HealthSource interface {
 type Collector struct {
 	src Source
 
-	modelDesired     *prometheus.Desc
-	modelFailures    *prometheus.Desc
-	modelParked      *prometheus.Desc
-	modelUnplaceable *prometheus.Desc
-	engineInstall    *prometheus.Desc
-	leader           *prometheus.Desc
+	modelDesired      *prometheus.Desc
+	modelFailures     *prometheus.Desc
+	modelParked       *prometheus.Desc
+	modelUnplaceable  *prometheus.Desc
+	modelPreempted    *prometheus.Desc
+	capacityHolds     *prometheus.Desc
+	capacityGuarantee *prometheus.Desc
+	capacityLimit     *prometheus.Desc
+	engineInstall     *prometheus.Desc
+	leader            *prometheus.Desc
 
 	nodeUp       *prometheus.Desc
 	nodeGPUs     *prometheus.Desc
@@ -70,6 +74,13 @@ type Collector struct {
 	gpuMemoryUsed  *prometheus.Desc
 	gpuUtil        *prometheus.Desc
 	gpuTemp        *prometheus.Desc
+	gpuMemTemp     *prometheus.Desc
+	gpuPower       *prometheus.Desc
+	gpuPowerLimit  *prometheus.Desc
+	gpuThrottled   *prometheus.Desc
+	nodeCPUTemp    *prometheus.Desc
+	nodeCPUPower   *prometheus.Desc
+	nodeInletTemp  *prometheus.Desc
 	gpuHealthy     *prometheus.Desc
 	gpuNVLink      *prometheus.Desc
 	gpuECC         *prometheus.Desc
@@ -138,6 +149,21 @@ func New(src Source) *Collector {
 			"How busy a GPU was over the last sample period. Absent when the driver does not "+
 				"report it, which is not the same as idle. Memory used says a model is loaded; "+
 				"this says whether it is doing anything.", gpu, nil),
+		gpuMemTemp: prometheus.NewDesc("llmgw_gpu_memory_temperature_celsius",
+			"A GPU's memory (HBM) temperature, where the driver reports it.", gpu, nil),
+		gpuPower: prometheus.NewDesc("llmgw_gpu_power_watts",
+			"A GPU's power draw now.", gpu, nil),
+		gpuPowerLimit: prometheus.NewDesc("llmgw_gpu_power_limit_watts",
+			"The power limit a GPU is held to.", gpu, nil),
+		gpuThrottled: prometheus.NewDesc("llmgw_gpu_throttled",
+			"1 while something holds a GPU's clocks down: sw_power_cap, hw_slowdown, sw_thermal, hw_thermal, power_brake.",
+			append(gpu, "reason"), nil),
+		nodeCPUTemp: prometheus.NewDesc("llmgw_node_cpu_temperature_celsius",
+			"A node's hottest CPU sensor.", node, nil),
+		nodeCPUPower: prometheus.NewDesc("llmgw_node_cpu_power_watts",
+			"A node's CPU packages' power draw, from RAPL.", node, nil),
+		nodeInletTemp: prometheus.NewDesc("llmgw_node_inlet_temperature_celsius",
+			"A node's inlet or ambient air sensor, where its board has one.", node, nil),
 		gpuTemp: prometheus.NewDesc("llmgw_gpu_temperature_celsius",
 			"A GPU's reported temperature.", gpu, nil),
 		gpuHealthy: prometheus.NewDesc("llmgw_gpu_healthy",
@@ -168,6 +194,18 @@ func New(src Source) *Collector {
 		modelUnplaceable: prometheus.NewDesc("llmgw_model_unplaceable",
 			"1 when a model is short of replicas and the last attempt found no node to place one on.",
 			[]string{"model"}, nil),
+		capacityHolds: prometheus.NewDesc("llmgw_capacity_holds",
+			"What a part of the tenancy tree holds of a resource, its own models' replicas and those beneath it.",
+			[]string{"scope", "resource"}, nil),
+		capacityGuarantee: prometheus.NewDesc("llmgw_capacity_guarantee",
+			"What a part of the tenancy tree is guaranteed of a resource.",
+			[]string{"scope", "resource"}, nil),
+		capacityLimit: prometheus.NewDesc("llmgw_capacity_limit",
+			"The most of a resource a part of the tenancy tree may hold.",
+			[]string{"scope", "resource"}, nil),
+		modelPreempted: prometheus.NewDesc("llmgw_model_preempted_total",
+			"Replicas of a model stopped to make room for a model of a higher capacity class.",
+			[]string{"model", "class"}, nil),
 		engineInstall: prometheus.NewDesc("llmgw_node_engine_install",
 			"Where installing an engine stands on a node: 1 for the current state.",
 			[]string{"node", "engine", "version", "state"}, nil),
@@ -234,9 +272,11 @@ func (c *Collector) Describe(ch chan<- *prometheus.Desc) {
 		c.ibActive, c.ibRate, c.ibCounter, c.fabricSMs, c.fabricMaster, c.fabricProblems,
 		c.gpuNVLJoined, c.imexUp,
 		c.gpuMemoryTotal, c.gpuMemoryUsed, c.gpuUtil, c.gpuTemp, c.gpuHealthy,
+		c.gpuMemTemp, c.gpuPower, c.gpuPowerLimit, c.gpuThrottled, c.nodeCPUTemp, c.nodeCPUPower, c.nodeInletTemp,
 		c.gpuNVLink, c.gpuECC,
 		c.replicaState, c.replicaReady, c.replicaIdle,
-		c.modelDesired, c.modelFailures, c.modelParked, c.modelUnplaceable,
+		c.modelDesired, c.modelFailures, c.modelParked, c.modelUnplaceable, c.modelPreempted,
+		c.capacityHolds, c.capacityGuarantee, c.capacityLimit,
 		c.engineInstall, c.leader,
 	} {
 		ch <- d
@@ -246,6 +286,7 @@ func (c *Collector) Describe(ch chan<- *prometheus.Desc) {
 // Collect implements prometheus.Collector.
 func (c *Collector) Collect(ch chan<- prometheus.Metric) {
 	now := time.Now()
+	c.collectQuotas(ch)
 
 	loads := map[string]*inventory.Load{}
 	for _, n := range c.src.Nodes() {
@@ -279,6 +320,16 @@ func (c *Collector) Collect(ch chan<- prometheus.Metric) {
 		}
 
 		if l := n.Load; l != nil && now.Sub(l.At) < loadFresh {
+			// Each only when measured: a VM has none of these, and a zero
+			// would read as a machine at 0 °C drawing nothing.
+			for _, v := range []struct {
+				d   *prometheus.Desc
+				val float64
+			}{{c.nodeCPUTemp, l.CPUTempC}, {c.nodeCPUPower, l.CPUPowerW}, {c.nodeInletTemp, l.InletTempC}} {
+				if v.val > 0 {
+					ch <- prometheus.MustNewConstMetric(v.d, prometheus.GaugeValue, v.val, n.ID)
+				}
+			}
 			ch <- prometheus.MustNewConstMetric(c.nodeCPU, prometheus.GaugeValue, l.CPUPercent, n.ID)
 			if l.CPUCount > 0 {
 				ch <- prometheus.MustNewConstMetric(c.nodeCPUs, prometheus.GaugeValue, float64(l.CPUCount), n.ID)
@@ -348,7 +399,11 @@ func (c *Collector) Collect(ch chan<- prometheus.Metric) {
 				ch <- prometheus.MustNewConstMetric(c.gpuUtil, prometheus.GaugeValue,
 					float64(g.UtilizationPercent), labels...)
 			}
-			if g.Health.TemperatureC > 0 {
+			// The sensors' fresh reading when there is one; the temperature
+			// read at registration otherwise, which is all an older agent sends.
+			if s, ok := sensorFor(n.Load, g.Index, now); ok {
+				c.gpuSensor(ch, s, labels)
+			} else if g.Health.TemperatureC > 0 {
 				ch <- prometheus.MustNewConstMetric(c.gpuTemp, prometheus.GaugeValue,
 					float64(g.Health.TemperatureC), labels...)
 			}
@@ -455,6 +510,7 @@ func (c *Collector) Collect(ch chan<- prometheus.Metric) {
 		ch <- prometheus.MustNewConstMetric(c.modelFailures, prometheus.GaugeValue, float64(h.Failures), h.Model)
 		ch <- prometheus.MustNewConstMetric(c.modelParked, prometheus.GaugeValue, b2f(h.Parked), h.Model)
 		ch <- prometheus.MustNewConstMetric(c.modelUnplaceable, prometheus.GaugeValue, b2f(h.Unplaceable), h.Model)
+		ch <- prometheus.MustNewConstMetric(c.modelPreempted, prometheus.CounterValue, float64(h.Preempted), h.Model, h.Class)
 	}
 }
 
@@ -464,3 +520,65 @@ func b2f(b bool) float64 {
 	}
 	return 0
 }
+
+// QuotaSource adds the tenancy tree's capacity quotas. Asked for by type
+// assertion, like HealthSource.
+type QuotaSource interface {
+	QuotaUse() []reconciler.QuotaUse
+}
+
+func (c *Collector) collectQuotas(ch chan<- prometheus.Metric) {
+	qs, ok := c.src.(QuotaSource)
+	if !ok {
+		return
+	}
+	for _, q := range qs.QuotaUse() {
+		for _, r := range []struct {
+			name     string
+			holds    int
+			gua, lim *int
+		}{
+			{"gpus", q.Holds.GPUs, q.Guarantee.GPUs, q.Limit.GPUs},
+			{"cpu_cores", q.Holds.CPUCores, q.Guarantee.CPUCores, q.Limit.CPUCores},
+			{"memory_gb", q.Holds.MemoryGB, q.Guarantee.MemoryGB, q.Limit.MemoryGB},
+		} {
+			if r.gua == nil && r.lim == nil {
+				continue
+			}
+			ch <- prometheus.MustNewConstMetric(c.capacityHolds, prometheus.GaugeValue, float64(r.holds), q.Scope, r.name)
+			if r.gua != nil {
+				ch <- prometheus.MustNewConstMetric(c.capacityGuarantee, prometheus.GaugeValue, float64(*r.gua), q.Scope, r.name)
+			}
+			if r.lim != nil {
+				ch <- prometheus.MustNewConstMetric(c.capacityLimit, prometheus.GaugeValue, float64(*r.lim), q.Scope, r.name)
+			}
+		}
+	}
+}
+
+// sensorFor is a GPU's fresh sensor reading, if the node sent one.
+func sensorFor(l *inventory.Load, index int, now time.Time) (inventory.GPUSensor, bool) {
+	if l == nil || now.Sub(l.At) >= loadFresh {
+		return inventory.GPUSensor{}, false
+	}
+	for _, g := range l.GPUs {
+		if g.Index == index {
+			return g, true
+		}
+	}
+	return inventory.GPUSensor{}, false
+}
+
+func (c *Collector) gpuSensor(ch chan<- prometheus.Metric, s inventory.GPUSensor, labels []string) {
+	for _, v := range []struct {
+		d   *prometheus.Desc
+		val float64
+	}{{c.gpuTemp, s.TempC}, {c.gpuMemTemp, s.MemoryTempC}, {c.gpuPower, s.PowerW}, {c.gpuPowerLimit, s.PowerLimitW}} {
+		if v.val > 0 {
+			ch <- prometheus.MustNewConstMetric(v.d, prometheus.GaugeValue, v.val, labels...)
+		}
+	}
+	for _, reason := range s.Throttle {
+		ch <- prometheus.MustNewConstMetric(c.gpuThrottled, prometheus.GaugeValue, 1, append(labels, reason)...)
+	}
+}

internal/control/fleetmetrics/sensors_test.go

@@ -0,0 +1,41 @@
+package fleetmetrics
+
+import (
+	"strings"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
+)
+
+func TestSensorsArePublishedFreshAndOnlyWhenMeasured(t *testing.T) {
+	gpuNode := inventory.Node{ID: "g1", State: inventory.Ready,
+		GPUs: []discovery.GPU{{Index: 0, Name: "NVIDIA H100 80GB HBM3", MemoryTotalMB: 81559, Health: discovery.GPUHealth{TemperatureC: 40}}},
+		Load: &inventory.Load{At: time.Now(), CPUTempC: 58, CPUPowerW: 65,
+			GPUs: []inventory.GPUSensor{{Index: 0, TempC: 88, PowerW: 698, PowerLimitW: 700,
+				Throttle: []string{"sw_power_cap", "sw_thermal"}}}}}
+	vm := inventory.Node{ID: "vm1", State: inventory.Ready, Load: &inventory.Load{At: time.Now(), CPUPercent: 3}}
+	out := gather(t, fakeSource{nodes: []inventory.Node{gpuNode, vm}})
+
+	for _, want := range []string{
+		"llmgw_node_cpu_temperature_celsius node=g1 -> 58",
+		"llmgw_node_cpu_power_watts node=g1 -> 65",
+		"llmgw_gpu_power_watts class=h100-80g gpu=0 node=g1 -> 698",
+		"llmgw_gpu_power_limit_watts class=h100-80g gpu=0 node=g1 -> 700",
+		"llmgw_gpu_throttled class=h100-80g gpu=0 node=g1 reason=sw_thermal -> 1",
+		// The fresh reading, not the 40 °C from registration.
+		"llmgw_gpu_temperature_celsius class=h100-80g gpu=0 node=g1 -> 88",
+	} {
+		if !strings.Contains(out, want) {
+			t.Errorf("missing %q", want)
+		}
+	}
+	for _, absent := range []string{"node=vm1 -> 0", "llmgw_node_inlet_temperature_celsius", "llmgw_gpu_memory_temperature_celsius"} {
+		for _, line := range strings.Split(out, "\n") {
+			if strings.Contains(line, absent) && (strings.Contains(line, "temperature") || strings.Contains(line, "power")) {
+				t.Errorf("published what was not measured: %s", line)
+			}
+		}
+	}
+}

internal/control/inventory/inventory.go

@@ -79,6 +79,9 @@ type Node struct {
 	// it, which is treated as no constraint — the same as every node before
 	// this field existed.
 	MemoryTotalMB int `json:"memory_total_mb,omitempty"`
+	// CPUCount is the CPUs the agent may run on. Zero is an agent too old to
+	// say, which, like memory, constrains nothing.
+	CPUCount int `json:"cpu_count,omitempty"`
 
 	// Weights are the model files this node can see, however they got there.
 	// Empty means the node was not asked to look, which is not the same as
@@ -301,6 +304,21 @@ type Load struct {
 	MemAvailBytes int64
 	// Shards is each engine's use, by shard ID.
 	Shards map[string]ShardLoad
+	// What the node's sensors say; zero is not measured.
+	CPUTempC   float64
+	CPUPowerW  float64
+	InletTempC float64
+	GPUs       []GPUSensor
+}
+
+// GPUSensor is one card's heat and power, and what is throttling it.
+type GPUSensor struct {
+	Index       int      `json:"index"`
+	TempC       float64  `json:"temp_c,omitempty"`
+	MemoryTempC float64  `json:"memory_temp_c,omitempty"`
+	PowerW      float64  `json:"power_w,omitempty"`
+	PowerLimitW float64  `json:"power_limit_w,omitempty"`
+	Throttle    []string `json:"throttle,omitempty"`
 }
 
 // ShardLoad is one engine's use: cores busy, and resident memory.
@@ -373,6 +391,9 @@ type Allocation struct {
 	// Zero for a profile that declared nothing, which constrains nothing and
 	// is counted as nothing.
 	MemoryGB int
+	// CPUCores is the cores this shard was given to compute on, for a shard
+	// with no GPU; zero for one that shares whatever is free.
+	CPUCores int
 }
 
 // Store holds the cluster inventory. It is safe for concurrent use.
@@ -392,6 +413,23 @@ func NewStore() *Store {
 	return &Store{nodes: map[string]*Node{}, allocs: map[string][]Allocation{}}
 }
 
+// Clone is a copy to try decisions on without making them: allocations made
+// on it are its own. Nodes are copied by value; the slices inside them are
+// shared, and a clone must not write through them.
+func (s *Store) Clone() *Store {
+	s.mu.RLock()
+	defer s.mu.RUnlock()
+	c := NewStore()
+	for id, n := range s.nodes {
+		cp := *n
+		c.nodes[id] = &cp
+	}
+	for id, list := range s.allocs {
+		c.allocs[id] = slices.Clone(list)
+	}
+	return c
+}
+
 // Upsert records a node's registration, preserving operator-set state such as
 // a cordon: an agent reconnecting must not silently undo an operator's decision.
 // RecordSMChange keeps a port's change of master subnet manager, the last

internal/control/modelserve/modelserve.go

@@ -91,6 +91,9 @@ type Request struct {
 	// use. They default to 16384 and 4096, what four parallel slots divide.
 	CtxPool       int `json:"ctx_pool,omitempty"`
 	ContextLength int `json:"context_length,omitempty"`
+	// CPUCores is what a replica computes on; zero takes half the smallest
+	// node it fits on.
+	CPUCores int `json:"cpu_cores,omitempty"`
 	// By is who asked, for the catalog's record.
 	By string `json:"-"`
 }
@@ -278,13 +281,20 @@ func (s *Server) run(j *Job) {
 	}
 	s.step(j, "check", Done, fmt.Sprintf("%s, needs about %d GB (%s)", orUnknown(header.Architecture), memGB, basis))
 
+	cores := req.CPUCores
+	if cores <= 0 {
+		if nc, ok := s.deps.Fleet.(interface{ NodeCPUs(int) []int }); ok {
+			cores = catalog.SuggestCores(nc.NodeCPUs(memGB))
+		}
+	}
+
 	// 4. The catalog.
 	m := catalog.Model{
 		Name: name, Aliases: req.Aliases, Description: req.Description, Source: source,
 		RoutingProfile: "chat", ContextLength: req.ContextLength, MaxOutput: 1024,
 		Profiles: []catalog.Profile{{
 			Name: "cpu", Engine: "llamacpp", Weight: 1, Priority: 100,
-			Args: []string{"--ctx-size", strconv.Itoa(req.CtxPool)}, MinMemoryGB: memGB,
+			Args: []string{"--ctx-size", strconv.Itoa(req.CtxPool)}, MinMemoryGB: memGB, CPUCores: cores,
 		}},
 		Scaling: catalog.Scaling{Min: req.Replicas, Max: max(req.Replicas, 1)},
 	}

internal/control/modelsvc/modelsvc.go

@@ -18,6 +18,7 @@ package modelsvc
 import (
 	"context"
 	"encoding/json"
+	"errors"
 	"fmt"
 	"log/slog"
 	"sync"
@@ -104,6 +105,16 @@ func (s *Service) Add(ctx context.Context, m catalog.Model, by string) error {
 	if s.Declared(m.Name) {
 		return fmt.Errorf("model %q is declared in configuration; edit that rather than adding a second definition", m.Name)
 	}
+	// A model with no GPU is all host memory, and one that declares none
+	// fits everywhere as far as placement can tell — however many of it
+	// there already are. Required of what is added now; models stored before
+	// this still load, and say what they are missing.
+	for _, p := range m.Profiles {
+		if p.CPUOnly() && p.MinMemoryGB <= 0 {
+			return fmt.Errorf("model %q profile %q runs without a GPU and declares no memory (min_memory_gb): "+
+				"placement could not tell when a node is full", m.Name, p.Name)
+		}
+	}
 	if s.check != nil {
 		if err := s.check(ctx, m); err != nil {
 			return err
@@ -189,36 +200,98 @@ func (s *Service) Remove(ctx context.Context, name string) error {
 	return nil
 }
 
-// Scale changes how many replicas a model runs, without a restart.
-//
-// Only the scaling block moves. Everything else about the model — its source,
-// its profiles, the engine arguments somebody tuned — is read back from what
-// was stored and written out again untouched, so changing a number cannot
-// quietly become a redefinition.
-//
-// A model declared in configuration is refused, for the same reason Remove
-// refuses one: the YAML is read on every start, so the new number would hold
-// until the next restart and then silently revert. A cluster that disagrees
-// with its own config file is worse than an error message saying where to edit.
+// Scale changes how many replicas a model added at runtime runs.
 func (s *Service) Scale(ctx context.Context, name string, sc catalog.Scaling, by string) (catalog.Model, error) {
+	final, err := s.update(ctx, name, by, "change its scaling", func(m *catalog.Model) error {
+		m.Scaling = sc
+		return nil
+	})
+	if err == nil {
+		s.log.Info("model scaled", "model", name,
+			"min", final.Scaling.Min, "max", final.Scaling.Max, "by", by)
+	}
+	return final, err
+}
+
+// Resources sets what a model's profile reserves of a node: host memory, and
+// the CPU cores a replica without a GPU computes on. An empty profile name
+// sets every profile.
+func (s *Service) Resources(ctx context.Context, name, profile string, memoryGB, cpuCores int, by string) (catalog.Model, error) {
+	if memoryGB < 0 || cpuCores < 0 {
+		return catalog.Model{}, errors.New("memory and cores cannot be negative")
+	}
+	final, err := s.update(ctx, name, by, "change its resources", func(m *catalog.Model) error {
+		found := false
+		for i := range m.Profiles {
+			if profile != "" && m.Profiles[i].Name != profile {
+				continue
+			}
+			found = true
+			m.Profiles[i].MinMemoryGB = memoryGB
+			m.Profiles[i].CPUCores = cpuCores
+		}
+		if !found {
+			return fmt.Errorf("model %q has no profile %q", name, profile)
+		}
+		return nil
+	})
+	if err == nil {
+		s.log.Info("model resources set", "model", name, "profile", profile,
+			"memory_gb", memoryGB, "cpu_cores", cpuCores, "by", by)
+	}
+	return final, err
+}
+
+// SetClass sets a model's capacity class: critical, standard or
+// opportunistic, or empty to go back to what its scaling implies.
+func (s *Service) SetClass(ctx context.Context, name, class, by string) (catalog.Model, error) {
+	final, err := s.update(ctx, name, by, "change its class", func(m *catalog.Model) error {
+		m.Class = class
+		return nil
+	})
+	if err == nil {
+		s.log.Info("model class set", "model", name, "class", final.EffectiveClass(), "by", by)
+	}
+	return final, err
+}
+
+// SetOwner sets the part of the tenancy tree a model's capacity counts
+// against, org:ID, team:ID or project:ID; empty makes it shared.
+func (s *Service) SetOwner(ctx context.Context, name, owner, by string) (catalog.Model, error) {
+	final, err := s.update(ctx, name, by, "change its owner", func(m *catalog.Model) error {
+		m.Owner = owner
+		return nil
+	})
+	if err == nil {
+		s.log.Info("model owner set", "model", name, "owner", owner, "by", by)
+	}
+	return final, err
+}
+
+// update changes one model added at runtime and keeps the change.
+//
+// Built before anything is written, like Add: catalog.New is what applies the
+// defaults and refuses what does not validate, so the model that comes back
+// is the one that will run rather than the one that was asked for.
+func (s *Service) update(ctx context.Context, name, by, what string, change func(*catalog.Model) error) (catalog.Model, error) {
 	s.mu.Lock()
 	defer s.mu.Unlock()
 
 	if s.Declared(name) {
 		return catalog.Model{}, fmt.Errorf(
-			"model %q is declared in configuration; change its scaling there, or it reverts on the next restart", name)
+			"model %q is declared in configuration; %s there, or it reverts on the next restart", name, what)
 	}
-
 	stored, err := s.storedModels(ctx)
 	if err != nil {
 		return catalog.Model{}, err
 	}
-
 	next := append([]catalog.Model{}, s.declared...)
 	found := false
 	for _, e := range stored {
 		if e.Name == name {
-			e.Scaling = sc
+			if err := change(&e); err != nil {
+				return catalog.Model{}, err
+			}
 			found = true
 		}
 		next = append(next, e)
@@ -226,20 +299,14 @@ func (s *Service) Scale(ctx context.Context, name string, sc catalog.Scaling, by
 	if !found {
 		return catalog.Model{}, fmt.Errorf("model %q was not added at runtime", name)
 	}
-
-	// Built before anything is written, like Add: catalog.New is what applies
-	// the defaults (max follows min when it is left out) and what refuses a
-	// max below a min, so the model that comes back is the one that will run
-	// rather than the one that was asked for.
 	built, err := catalog.New(next)
 	if err != nil {
-		return catalog.Model{}, fmt.Errorf("scaling %q that way would make the catalog invalid: %w", name, err)
+		return catalog.Model{}, fmt.Errorf("that would make the catalog invalid: %w", err)
 	}
 	final, ok := built.Resolve(name)
 	if !ok {
 		return catalog.Model{}, fmt.Errorf("model %q vanished from the catalog it was just built into", name)
 	}
-
 	def, err := json.Marshal(final)
 	if err != nil {
 		return catalog.Model{}, err
@@ -249,10 +316,7 @@ func (s *Service) Scale(ctx context.Context, name string, sc catalog.Scaling, by
 	}); err != nil {
 		return catalog.Model{}, err
 	}
-
 	s.apply.ReplaceCatalog(ctx, built)
-	s.log.Info("model scaled", "model", name,
-		"min", final.Scaling.Min, "max", final.Scaling.Max, "by", by)
 	return final, nil
 }
 

internal/control/modelsvc/modelsvc_test.go

@@ -48,7 +48,7 @@ func model(name string) catalog.Model {
 	return catalog.Model{
 		Name: name, Source: "/models/" + name,
 		Scaling:  catalog.Scaling{Min: 1, Max: 1},
-		Profiles: []catalog.Profile{{Name: "p", Engine: "llamacpp", Weight: 1}},
+		Profiles: []catalog.Profile{{Name: "p", Engine: "llamacpp", Weight: 1, MinMemoryGB: 2}},
 	}
 }
 
@@ -337,3 +337,43 @@ func TestMinZeroWithAMaxIsOnDemand(t *testing.T) {
 		t.Errorf("the idle timeout was lost; got %v", got.Scaling.IdleTimeout)
 	}
 }
+
+func TestACPUModelAddedNowMustSayHowMuchMemoryItNeeds(t *testing.T) {
+	s, st, _ := newService(t)
+	m := model("bare")
+	m.Profiles[0].MinMemoryGB = 0
+	err := s.Add(t.Context(), m, "tester")
+	if err == nil || !strings.Contains(err.Error(), "declares no memory") {
+		t.Fatalf("a CPU model with no memory was added: %v", err)
+	}
+	if len(st.rows) != 0 {
+		t.Error("it was stored anyway")
+	}
+}
+
+func TestResourcesAreSetOnAStoredModelAndSurviveARestart(t *testing.T) {
+	s, _, _ := newService(t)
+	if err := s.Add(t.Context(), model("m"), "tester"); err != nil {
+		t.Fatal(err)
+	}
+	got, err := s.Resources(t.Context(), "m", "", 6, 3, "tester")
+	if err != nil {
+		t.Fatal(err)
+	}
+	if p := got.Profiles[0]; p.MinMemoryGB != 6 || p.CPUCores != 3 {
+		t.Errorf("profile %+v", p)
+	}
+	if _, err := s.Resources(t.Context(), "m", "nope", 6, 3, "tester"); err == nil {
+		t.Error("a profile the model does not have was accepted")
+	}
+	if _, err := s.Resources(t.Context(), "m", "", -1, 0, "tester"); err == nil {
+		t.Error("negative memory was accepted")
+	}
+	cat, err := s.Load(t.Context())
+	if err != nil {
+		t.Fatal(err)
+	}
+	if m, _ := cat.Resolve("m"); m.Profiles[0].CPUCores != 3 {
+		t.Error("the cores were lost on reload")
+	}
+}

internal/control/placement/cpu_test.go

@@ -1,6 +1,7 @@
 package placement
 
 import (
+	"strings"
 	"testing"
 
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
@@ -100,3 +101,38 @@ func TestCPUProfileRespectsAvoid(t *testing.T) {
 		t.Errorf("placed on %q despite being told to avoid it", got)
 	}
 }
+
+func TestCPUCoresAreReservedAsMemoryIs(t *testing.T) {
+	n := node("n1", "", 0)
+	n.CPUCount = 8
+	st := store(n)
+	s := New(st)
+	p := cpuProfile("cpu")
+	p.CPUCores = 3
+
+	for i := range 2 {
+		res := s.Place([]Profile{p}, nil)
+		if res.Placement == nil {
+			t.Fatalf("replica %d did not fit in 8 cores:\n%s", i+1, res.Explain())
+		}
+		st.Allocate("n1", inventory.Allocation{ShardID: string(rune('a' + i)), CPUCores: 3})
+	}
+	// Six of eight given away: a third would need three.
+	res := s.Place([]Profile{p}, nil)
+	if res.Placement != nil {
+		t.Fatal("a third replica was given cores the node does not have")
+	}
+	if got := res.Explain(); !strings.Contains(got, "has 8 CPUs with 2 free, needs 3") {
+		t.Errorf("the refusal should say how many cores are left:\n%s", got)
+	}
+
+	// Unknown counts constrain nothing: an older agent, or a profile that
+	// reserves no cores.
+	old := node("n2", "", 0)
+	if res := New(store(old)).Place([]Profile{p}, nil); res.Placement == nil {
+		t.Error("a node that does not report its CPUs was excluded")
+	}
+	if res := s.Place([]Profile{cpuProfile("cpu")}, nil); res.Placement == nil {
+		t.Error("a profile reserving no cores was refused")
+	}
+}

internal/control/placement/placement.go

@@ -39,6 +39,9 @@ type Profile struct {
 	// MinDiskGB is the free space needed to hold the weights, checked only on
 	// nodes that do not already have them. Zero is unconstrained.
 	MinDiskGB int
+	// CPUCores is the cores a profile with no GPU reserves. Zero is
+	// unconstrained.
+	CPUCores int
 	// Source is the model's weights, as the engine would be given it. Used to
 	// prefer nodes that already hold them; empty asks for no preference.
 	Source string
@@ -582,6 +585,18 @@ func (s *Scheduler) eligibleNodes(p Profile, fabric string, avoid []string, reje
 			}
 		}
 
+		// Cores, for a replica that computes on them: what is left after the
+		// cores already given to the node's other replicas. Same exemptions
+		// as memory: zero declares nothing, and a node that does not report
+		// its CPUs is an older agent, not one with none.
+		if p.CPUOnly() && p.CPUCores > 0 && n.CPUCount > 0 {
+			free := n.CPUCount - s.claimedCores(n.ID)
+			if free < p.CPUCores {
+				reject(fmt.Sprintf("has %d CPUs with %d free, needs %d", n.CPUCount, free, p.CPUCores))
+				continue
+			}
+		}
+
 		if p.CPUOnly() {
 			// Every ready node can run it; there is no card to be free.
 			out = append(out, candidate{node: n})
@@ -723,6 +738,15 @@ func (s *Scheduler) claimedMemory(nodeID string) int {
 	return total
 }
 
+// claimedCores is the CPU cores already given to replicas on a node.
+func (s *Scheduler) claimedCores(nodeID string) int {
+	total := 0
+	for _, a := range s.inv.Allocations(nodeID) {
+		total += a.CPUCores
+	}
+	return total
+}
+
 // installing says why an engine is missing, when the node has said.
 func installing(n inventory.Node, engine string) string {
 	for _, p := range n.Provision {

internal/control/quota/quota.go

@@ -0,0 +1,192 @@
+// Package quota decides what the tenancy tree is guaranteed of the cluster
+// and how much of it each part may hold.
+//
+// A scope — an organization, team or project — may be guaranteed GPUs, CPU
+// cores and memory, and limited to at most some amount of each. What a scope
+// holds is what the replicas of the models it owns, and of the models its
+// teams and projects own, have reserved. Holding more than its guarantee is
+// borrowing: allowed while the room is free, and taken back when another
+// scope within its own guarantee needs it.
+package quota
+
+import (
+	"fmt"
+	"sort"
+	"strings"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// Amount is how much of each resource something holds or needs.
+type Amount struct {
+	GPUs, CPUCores, MemoryGB int
+}
+
+// Plus adds two amounts.
+func (a Amount) Plus(b Amount) Amount {
+	return Amount{a.GPUs + b.GPUs, a.CPUCores + b.CPUCores, a.MemoryGB + b.MemoryGB}
+}
+
+// Over names the first resource in which a is above the governed amounts of
+// l, with both figures; empty when it is above none.
+func Over(a Amount, l store.Resources) string {
+	switch {
+	case l.GPUs != nil && a.GPUs > *l.GPUs:
+		return fmt.Sprintf("%d GPUs of %d", a.GPUs, *l.GPUs)
+	case l.CPUCores != nil && a.CPUCores > *l.CPUCores:
+		return fmt.Sprintf("%d cores of %d", a.CPUCores, *l.CPUCores)
+	case l.MemoryGB != nil && a.MemoryGB > *l.MemoryGB:
+		return fmt.Sprintf("%d GB of memory of %d", a.MemoryGB, *l.MemoryGB)
+	}
+	return ""
+}
+
+// Holding is one replica's hold on the cluster, for working out who holds
+// what.
+type Holding struct {
+	Replica string
+	Owner   string
+	Amount  Amount
+	Since   time.Time
+}
+
+// Policy is the quotas and the tree they sit in, read at one moment.
+type Policy struct {
+	quotas  map[string]store.CapacityQuota
+	parents map[string][]string
+}
+
+// New builds a policy from stored quotas and each scope's parents.
+func New(quotas []store.CapacityQuota, parents map[string][]string) *Policy {
+	p := &Policy{quotas: map[string]store.CapacityQuota{}, parents: parents}
+	for _, q := range quotas {
+		p.quotas[Key(q.Scope, q.ScopeID)] = q
+	}
+	return p
+}
+
+// Key writes a scope the way owners are written: team:ID.
+func Key(s store.Scope, id string) string { return string(s) + ":" + id }
+
+// Chain is an owner and every scope above it, nearest first.
+func (p *Policy) Chain(owner string) []string {
+	if owner == "" {
+		return nil
+	}
+	return append([]string{owner}, p.parents[owner]...)
+}
+
+// Governing is the quotas along an owner's chain, nearest first.
+func (p *Policy) Governing(owner string) []store.CapacityQuota {
+	if p == nil {
+		return nil
+	}
+	var out []store.CapacityQuota
+	for _, s := range p.Chain(owner) {
+		if q, ok := p.quotas[s]; ok {
+			out = append(out, q)
+		}
+	}
+	return out
+}
+
+// Quotas lists every quota.
+func (p *Policy) Quotas() []store.CapacityQuota {
+	if p == nil {
+		return nil
+	}
+	out := make([]store.CapacityQuota, 0, len(p.quotas))
+	for _, q := range p.quotas {
+		out = append(out, q)
+	}
+	sort.Slice(out, func(i, j int) bool {
+		return Key(out[i].Scope, out[i].ScopeID) < Key(out[j].Scope, out[j].ScopeID)
+	})
+	return out
+}
+
+// Usage is what every scope holds, its own models' replicas and those of
+// everything beneath it.
+func (p *Policy) Usage(hs []Holding) map[string]Amount {
+	out := map[string]Amount{}
+	if p == nil {
+		return out
+	}
+	for _, h := range hs {
+		for _, s := range p.Chain(h.Owner) {
+			out[s] = out[s].Plus(h.Amount)
+		}
+	}
+	return out
+}
+
+// Borrowed is the replicas held beyond a guarantee: of each governed scope's
+// replicas, oldest first, the ones that take it past what it is guaranteed.
+// A replica beyond any guarantee along its chain is borrowed.
+func (p *Policy) Borrowed(hs []Holding) map[string]bool {
+	out := map[string]bool{}
+	if p == nil || len(p.quotas) == 0 {
+		return out
+	}
+	sorted := append([]Holding(nil), hs...)
+	sort.Slice(sorted, func(i, j int) bool { return sorted[i].Since.Before(sorted[j].Since) })
+	for key, q := range p.quotas {
+		if !q.Guarantee.Any() {
+			continue
+		}
+		var sum Amount
+		for _, h := range sorted {
+			if !contains(p.Chain(h.Owner), key) {
+				continue
+			}
+			sum = sum.Plus(h.Amount)
+			if Over(sum, q.Guarantee) != "" {
+				out[h.Replica] = true
+			}
+		}
+	}
+	return out
+}
+
+// Guaranteed reports whether an owner, holding what it holds, is still within
+// every guarantee along its chain with need added — so it may take back what
+// others borrowed. An owner with no guarantee anywhere is never guaranteed.
+func (p *Policy) Guaranteed(owner string, usage map[string]Amount, need Amount) bool {
+	gov := p.Governing(owner)
+	any := false
+	for _, q := range gov {
+		if !q.Guarantee.Any() {
+			continue
+		}
+		any = true
+		if Over(usage[Key(q.Scope, q.ScopeID)].Plus(need), q.Guarantee) != "" {
+			return false
+		}
+	}
+	return any
+}
+
+// AtLimit says which limit along an owner's chain need would take it past,
+// empty when none.
+func (p *Policy) AtLimit(owner string, usage map[string]Amount, need Amount) string {
+	for _, q := range p.Governing(owner) {
+		if !q.Limit.Any() {
+			continue
+		}
+		key := Key(q.Scope, q.ScopeID)
+		if over := Over(usage[key].Plus(need), q.Limit); over != "" {
+			return fmt.Sprintf("%s is at its capacity limit (%s)", key, strings.TrimSpace(over))
+		}
+	}
+	return ""
+}
+
+func contains(list []string, v string) bool {
+	for _, x := range list {
+		if x == v {
+			return true
+		}
+	}
+	return false
+}

internal/control/quota/quota_test.go

@@ -0,0 +1,83 @@
+package quota
+
+import (
+	"strings"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+func n(v int) *int { return &v }
+
+// acme has two teams; ml is guaranteed 8 GPUs, the org as a whole 12 and
+// no more than 16.
+func policy() *Policy {
+	return New([]store.CapacityQuota{
+		{Scope: store.ScopeTeam, ScopeID: "ml", Guarantee: store.Resources{GPUs: n(8)}},
+		{Scope: store.ScopeOrg, ScopeID: "acme", Guarantee: store.Resources{GPUs: n(12)}, Limit: store.Resources{GPUs: n(16)}},
+	}, map[string][]string{
+		"team:ml": {"org:acme"}, "team:web": {"org:acme"},
+		"project:train": {"team:ml", "org:acme"},
+	})
+}
+
+func at(min int) time.Time { return time.Date(2026, 1, 1, 0, min, 0, 0, time.UTC) }
+
+func TestAScopeHoldsWhatEverythingBeneathItHolds(t *testing.T) {
+	p := policy()
+	u := p.Usage([]Holding{
+		{Replica: "a", Owner: "project:train", Amount: Amount{GPUs: 4}},
+		{Replica: "b", Owner: "team:web", Amount: Amount{GPUs: 2, CPUCores: 3}},
+		{Replica: "c", Owner: "", Amount: Amount{GPUs: 8}}, // shared: nobody's
+	})
+	if u["team:ml"].GPUs != 4 || u["org:acme"].GPUs != 6 || u["org:acme"].CPUCores != 3 || u["project:train"].GPUs != 4 {
+		t.Errorf("usage %+v", u)
+	}
+	if _, ok := u[""]; ok {
+		t.Error("a shared model was counted against somebody")
+	}
+}
+
+func TestWhatGoesPastAGuaranteeIsBorrowedNewestFirst(t *testing.T) {
+	p := policy()
+	b := p.Borrowed([]Holding{
+		{Replica: "old", Owner: "team:ml", Amount: Amount{GPUs: 8}, Since: at(1)},
+		{Replica: "new", Owner: "team:ml", Amount: Amount{GPUs: 4}, Since: at(2)},
+		{Replica: "web", Owner: "team:web", Amount: Amount{GPUs: 4}, Since: at(3)},
+	})
+	// ml: 8 within its 8, then 4 past it. acme: 8 + 4 = 12 within, web's 4
+	// takes the org to 16, past its 12.
+	if b["old"] || !b["new"] || !b["web"] {
+		t.Errorf("borrowed %v", b)
+	}
+}
+
+func TestAGuaranteeIsOnlyForWhoHasOne(t *testing.T) {
+	p := policy()
+	u := map[string]Amount{"team:ml": {GPUs: 4}, "org:acme": {GPUs: 4}}
+	if !p.Guaranteed("project:train", u, Amount{GPUs: 4}) {
+		t.Error("8 of ml's 8 and 8 of acme's 12 is within both")
+	}
+	if p.Guaranteed("project:train", u, Amount{GPUs: 5}) {
+		t.Error("9 GPUs is past ml's 8")
+	}
+	if p.Guaranteed("", u, Amount{GPUs: 1}) || New(nil, nil).Guaranteed("team:ml", u, Amount{}) {
+		t.Error("an owner with no guarantee is not guaranteed anything")
+	}
+}
+
+func TestALimitStopsGrowthWhereverItIsSet(t *testing.T) {
+	p := policy()
+	u := map[string]Amount{"org:acme": {GPUs: 14}}
+	if why := p.AtLimit("team:web", u, Amount{GPUs: 2}); why != "" {
+		t.Errorf("16 of 16 is at the limit, not past it: %s", why)
+	}
+	why := p.AtLimit("team:web", u, Amount{GPUs: 4})
+	if !strings.Contains(why, "org:acme is at its capacity limit (18 GPUs of 16)") {
+		t.Errorf("the refusal should name the scope and the figures: %q", why)
+	}
+	if p.AtLimit("", u, Amount{GPUs: 99}) != "" {
+		t.Error("a shared model was limited")
+	}
+}

internal/control/quotas.go

@@ -0,0 +1,95 @@
+package control
+
+import (
+	"context"
+	"encoding/json"
+	"log/slog"
+	"sync"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/bench"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// QuotaStore is where capacity quotas and the tenancy tree are kept.
+type QuotaStore interface {
+	CapacityQuotas(ctx context.Context) ([]store.CapacityQuota, error)
+	ScopeParents(ctx context.Context) (map[string][]string, error)
+}
+
+// quotaRefresh is how stale the reconciler's view of quotas may get. A quota
+// changed by an operator takes effect within this; the reconciler asks many
+// times a pass and the store is not asked each time.
+const quotaRefresh = 30 * time.Second
+
+// QuotaPolicy returns what the reconciler asks for its quotas: the last
+// policy read from the store, read again once it is older than quotaRefresh.
+// A store that fails keeps the last good policy, so a database hiccup does
+// not drop every guarantee and limit at once.
+func QuotaPolicy(st QuotaStore, log *slog.Logger) func() *quota.Policy {
+	var mu sync.Mutex
+	var cur *quota.Policy
+	var at time.Time
+	return func() *quota.Policy {
+		mu.Lock()
+		defer mu.Unlock()
+		if cur != nil && time.Since(at) < quotaRefresh {
+			return cur
+		}
+		ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+		defer cancel()
+		qs, err := st.CapacityQuotas(ctx)
+		if err == nil {
+			var parents map[string][]string
+			if parents, err = st.ScopeParents(ctx); err == nil {
+				cur, at = quota.New(qs, parents), time.Now()
+				return cur
+			}
+		}
+		log.Warn("could not read capacity quotas; keeping the last ones", "err", err)
+		at = time.Now() // not again on every call while the store is down
+		return cur
+	}
+}
+
+// MeasurementStore is where benchmark results are kept.
+type MeasurementStore interface {
+	Capacities(ctx context.Context) ([]store.StoredCapacity, error)
+}
+
+// Measurements returns what the reconciler asks for benchmark results, keyed
+// model/profile, read again once older than quotaRefresh; like QuotaPolicy it
+// keeps the last good answer when the store fails.
+func Measurements(st MeasurementStore, log *slog.Logger) func() map[string]reconciler.Measured {
+	var mu sync.Mutex
+	var cur map[string]reconciler.Measured
+	var at time.Time
+	return func() map[string]reconciler.Measured {
+		mu.Lock()
+		defer mu.Unlock()
+		if cur != nil && time.Since(at) < quotaRefresh {
+			return cur
+		}
+		ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+		defer cancel()
+		at = time.Now()
+		rows, err := st.Capacities(ctx)
+		if err != nil {
+			log.Warn("could not read benchmark results; keeping the last ones", "err", err)
+			return cur
+		}
+		next := map[string]reconciler.Measured{}
+		for _, row := range rows {
+			var c bench.Capacity
+			if json.Unmarshal(row.Measurement, &c) != nil {
+				continue
+			}
+			next[row.Model+"/"+row.Profile] = reconciler.Measured{
+				MaxConcurrency: c.MaxConcurrencyAtSLO, TokPerSec: c.ThroughputTokPerSec}
+		}
+		cur = next
+		return cur
+	}
+}

internal/control/reconciler/adopt.go

@@ -144,6 +144,7 @@ func (r *Reconciler) mergeAdopted(rep *Replica, group []AdoptedShard) {
 		r.inv.Allocate(s.NodeID, inventory.Allocation{
 			ShardID: s.ShardID, ReplicaID: rep.ID, GPUIndices: s.GPUIndices,
 			MemoryGB: r.profileMemory(rep.Model, rep.Profile),
+			CPUCores: r.adoptedCores(rep.Model, rep.Profile),
 		})
 		// The endpoint comes from the leader, which may be in any of the
 		// groups — the rank 0 node is not necessarily the first to register.
@@ -253,3 +254,9 @@ type AdoptedShard struct {
 	// Endpoint is the leader's base URL, set on the rank 0 shard.
 	Endpoint string
 }
+
+// adoptedCores is what an adopted shard is taken to hold of its node's CPUs.
+func (r *Reconciler) adoptedCores(model, profile string) int {
+	p, _ := r.profileOf(model, profile)
+	return cpuCores(p)
+}

internal/control/reconciler/demand.go

@@ -157,8 +157,31 @@ type Load struct {
 // the last one is still starting: on a small CPU fleet each replica is minutes
 // of weights and gigabytes of memory, and a spike that is over by the time it
 // loads should not leave three behind.
+//
+// With a benchmark stored for the profile a model runs on, its measured
+// capacity is used too: more requests per replica than were measured to meet
+// the SLO is busy, even with the engines' queues empty — an engine will take
+// requests into its slots long after latency has gone — and a replica is only
+// taken back when the rest would still be under scaleDownHeadroom of what they
+// were measured to hold, so scaling down does not land on the edge.
 func (r *Reconciler) RecordLoad(loads map[string]Load) {
 	now := r.now()
+	// The profile each model's replicas run on, for its measurement; read
+	// before the loop, which holds the lock.
+	running := map[string]string{}
+	r.mu.Lock()
+	for _, rep := range r.replicas {
+		if rep.State.Counts() {
+			running[rep.Model] = rep.Profile
+		}
+	}
+	r.mu.Unlock()
+	measured := map[string]Measured{}
+	for model, profile := range running {
+		if m, ok := r.measured(model, profile); ok {
+			measured[model] = m
+		}
+	}
 	for _, m := range r.cat.Load().Models() {
 		base := r.baseFor(m)
 		r.mu.Lock()
@@ -178,8 +201,13 @@ func (r *Reconciler) RecordLoad(loads map[string]Load) {
 		l := loads[m.Name]
 		perReplica := float64(l.Waiting) / float64(max(l.Replicas, 1))
 		extra := r.extra[m.Name]
+		meas, hasMeas := measured[m.Name]
+		overMeasured := hasMeas &&
+			float64(l.Running+l.Waiting) > float64(meas.MaxConcurrency*max(l.Replicas, 1))
+		fitsInFewer := !hasMeas ||
+			float64(l.Running) <= scaleDownHeadroom*float64(meas.MaxConcurrency*max(l.Replicas-1, 0))
 		switch {
-		case perReplica >= r.cfg.ScaleUpWaiting:
+		case perReplica >= r.cfg.ScaleUpWaiting || overMeasured:
 			delete(r.quietSince, m.Name)
 			since, ok := r.busySince[m.Name]
 			if !ok {
@@ -190,9 +218,10 @@ func (r *Reconciler) RecordLoad(loads map[string]Load) {
 				r.extra[m.Name] = extra + 1
 				r.busySince[m.Name] = now
 				r.log.Info("scaling up for load", "model", m.Name, "replicas", base+extra+1,
-					"waiting_per_replica", perReplica, "for", now.Sub(since).Round(time.Second))
+					"waiting_per_replica", perReplica, "in_flight", l.Running+l.Waiting,
+					"measured_per_replica", meas.MaxConcurrency, "for", now.Sub(since).Round(time.Second))
 			}
-		case l.Waiting == 0 && extra > 0:
+		case l.Waiting == 0 && extra > 0 && fitsInFewer:
 			delete(r.busySince, m.Name)
 			since, ok := r.quietSince[m.Name]
 			if !ok {
@@ -212,3 +241,8 @@ func (r *Reconciler) RecordLoad(loads map[string]Load) {
 		r.mu.Unlock()
 	}
 }
+
+// scaleDownHeadroom is how full, of their measured capacity, the replicas left
+// after a scale-down may be. Below one so that the load that was there a
+// minute ago does not put them straight back over.
+const scaleDownHeadroom = 0.7

internal/control/reconciler/headroom.go

@@ -0,0 +1,148 @@
+package reconciler
+
+import (
+	"slices"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/placement"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
+)
+
+// Headroom is how many more replicas of a model the cluster has room for now.
+type Headroom struct {
+	// Current is the replicas the model has that hold capacity: starting or
+	// ready, not those on their way out.
+	Current int
+	// More is how many more could be placed now, at most the limit asked.
+	More int
+	// Nodes is where each of those would go, in order, a replica's nodes
+	// joined when it spans several.
+	Nodes [][]string
+	// Limited is set when More reached the limit asked, so there may be room
+	// for more than that. A CPU profile that declares no memory is always
+	// limited: nothing it asks for runs out.
+	Limited bool
+	// Blocked is why one more than More does not fit, a line per node that
+	// was considered. Empty when Limited.
+	Blocked []string
+	// TokPerSec is what one more replica was measured to serve, on the
+	// profile placement would give it; zero when it was never measured.
+	TokPerSec float64
+	// ByPreempting is how many more, beyond More, would fit by stopping
+	// replicas the model's class may take, and Preempts the models those
+	// replicas belong to: what "room" means for a model that outranks others.
+	ByPreempting int
+	Preempts     []string
+}
+
+// Headroom tries placing up to limit more replicas of a model on a copy of
+// the inventory, each one's allocation made on the copy before the next is
+// tried, exactly as the reconciler would place them one pass at a time.
+//
+// The scheduler's own decisions, not an estimate beside them: what the page
+// offers is then what the next passes will do, and a count that the
+// arithmetic of a second implementation allowed but placement refuses would
+// leave a model short with nobody told why.
+func (r *Reconciler) Headroom(model string, limit int) (Headroom, bool) {
+	m, ok := r.cat.Load().Resolve(model)
+	if !ok {
+		return Headroom{}, false
+	}
+	var h Headroom
+	var avoid []string
+	r.mu.Lock()
+	for _, rep := range r.replicas {
+		if rep.Model == m.Name && rep.State.Counts() && rep.RetiringAt.IsZero() {
+			h.Current++
+			avoid = append(avoid, rep.NodeIDs()...)
+		}
+	}
+	r.mu.Unlock()
+
+	inv := r.inv.Clone()
+	sched := placement.New(inv)
+	pol := r.policy()
+	var usage map[string]quota.Amount
+	if pol != nil {
+		usage = pol.Usage(r.Holdings())
+	}
+	place := func() placement.Result {
+		res := sched.Place(r.orderedProfiles(m), avoid)
+		if res.Placement == nil && len(avoid) > 0 {
+			// As scaleUp does: spreading is a preference, not a rule.
+			res = sched.Place(r.orderedProfiles(m), nil)
+		}
+		return res
+	}
+	// What the model's class may take, in the order preemption would, each
+	// released on the copy only when free room has run out.
+	victims, nextVictim, preempting := r.victims(m), 0, false
+	for h.More+h.ByPreempting < limit {
+		res := place()
+		// Named only once they have made room: candidates tried on the way
+		// to finding there is none would otherwise read as whose replicas a
+		// placement would stop.
+		var tried []string
+		for res.Placement == nil && nextVictim < len(victims) {
+			v := victims[nextVictim]
+			nextVictim++
+			inv.ReleaseReplica(v.ID)
+			tried = append(tried, v.Model)
+			res = place()
+		}
+		if res.Placement != nil && len(tried) > 0 {
+			preempting = true
+			for _, m := range tried {
+				if !slices.Contains(h.Preempts, m) {
+					h.Preempts = append(h.Preempts, m)
+				}
+			}
+		}
+		if res.Placement == nil {
+			for _, rej := range res.Rejections {
+				h.Blocked = append(h.Blocked, rej.String())
+			}
+			slices.Sort(h.Blocked)
+			h.Blocked = slices.Compact(h.Blocked)
+			return h, true
+		}
+		// A quota limit is as much an edge as a full node.
+		if pol != nil && m.Owner != "" {
+			need := placementNeed(m, res.Placement)
+			if why := pol.AtLimit(m.Owner, usage, need); why != "" {
+				h.Blocked = []string{why}
+				return h, true
+			}
+			for _, s := range pol.Chain(m.Owner) {
+				usage[s] = usage[s].Plus(need)
+			}
+		}
+		mem, cores := 0, 0
+		if prof, ok := m.ProfileByName(res.Placement.Profile.Name); ok {
+			mem, cores = prof.MinMemoryGB, cpuCores(prof)
+		}
+		if h.More == 0 {
+			if meas, ok := r.measured(m.Name, res.Placement.Profile.Name); ok {
+				h.TokPerSec = meas.TokPerSec
+			}
+		}
+		var nodes []string
+		for _, a := range res.Placement.Nodes {
+			inv.Allocate(a.NodeID, inventory.Allocation{
+				ShardID: "headroom", ReplicaID: "headroom", GPUIndices: a.GPUIndices, MemoryGB: mem, CPUCores: cores,
+			})
+			nodes = append(nodes, a.NodeID)
+		}
+		avoid = append(avoid, nodes...)
+		// Once anything was stopped, what fits after is counted as fitting
+		// by preempting: it may be sitting in the room that was freed.
+		if preempting {
+			h.ByPreempting++
+		} else {
+			h.Nodes = append(h.Nodes, nodes)
+			h.More++
+		}
+	}
+	h.Limited = true
+	return h, true
+}

internal/control/reconciler/headroom_test.go

@@ -0,0 +1,80 @@
+package reconciler
+
+import (
+	"strings"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
+)
+
+func TestHeadroomIsWhatPlacementWouldDoNext(t *testing.T) {
+	// Two 8-GPU nodes and a 4-GPU profile: four replicas in all, one placed.
+	h := newHarness(t, testModel(1), gpuNode("1", 8), gpuNode("2", 8))
+	h.r.Reconcile(t.Context())
+	h.markReady()
+
+	got, ok := h.r.Headroom("llama", 10)
+	if !ok {
+		t.Fatal("the model was not found")
+	}
+	if got.Current != 1 || got.More != 3 || got.Limited {
+		t.Fatalf("headroom %+v, expected 1 placed and room for 3 more", got)
+	}
+	// Spread first: the next one goes to the node without one.
+	if first := got.Nodes[0][0]; first == h.r.Replicas()[0].NodeIDs()[0] {
+		t.Errorf("the next replica would stack on %s beside the first", first)
+	}
+	if len(got.Blocked) == 0 || !strings.Contains(strings.Join(got.Blocked, "\n"), "GPU") {
+		t.Errorf("no reason given for the fifth: %v", got.Blocked)
+	}
+	// Tried on a copy: nothing was taken from the real inventory.
+	if free := len(h.inv.FreeGPUs("1")) + len(h.inv.FreeGPUs("2")); free != 12 {
+		t.Errorf("%d GPUs free after asking, expected the 12 the one replica leaves", free)
+	}
+
+	capped, _ := h.r.Headroom("llama", 2)
+	if capped.More != 2 || !capped.Limited || capped.Blocked != nil {
+		t.Errorf("asked for at most 2: %+v", capped)
+	}
+	if _, ok := h.r.Headroom("nope", 5); ok {
+		t.Error("an unknown model has headroom")
+	}
+}
+
+func TestHeadroomCountsMemoryAlreadyClaimed(t *testing.T) {
+	cpu := func(id string) inventory.Node {
+		return inventory.Node{ID: id, MgmtAddr: "10.0.0." + id, State: inventory.Ready, MemoryTotalMB: 16 * 1024}
+	}
+	m := catalog.Model{Name: "small", Source: "/m.gguf", Scaling: catalog.Scaling{Min: 1, Max: 1},
+		Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 6}}}
+	h := newHarness(t, m, cpu("a"), cpu("b"))
+	h.r.Reconcile(t.Context())
+
+	// 16 GB each at 6 GB a replica: two per node, one of them taken.
+	got, _ := h.r.Headroom("small", 10)
+	if got.Current != 1 || got.More != 3 {
+		t.Errorf("headroom %+v, expected room for 3 more", got)
+	}
+}
+
+func TestReservedCoresAreTheEnginesThreadsAndBoundTheRoom(t *testing.T) {
+	cpu := inventory.Node{ID: "a", MgmtAddr: "10.0.0.1", State: inventory.Ready, MemoryTotalMB: 64 * 1024, CPUCount: 8}
+	m := catalog.Model{Name: "coder", Source: "/m.gguf", Scaling: catalog.Scaling{Min: 1, Max: 1},
+		Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 6, CPUCores: 3}}}
+	h := newHarness(t, m, cpu)
+	h.r.Reconcile(t.Context())
+
+	started := h.disp.startedShards()
+	if len(started) != 1 || started[0].Spec.Threads != 3 {
+		t.Fatalf("the engine was not told its 3 cores: %+v", started)
+	}
+	if a := h.inv.Allocations("a"); len(a) != 1 || a[0].CPUCores != 3 {
+		t.Errorf("the cores were not reserved: %+v", a)
+	}
+	// Memory would allow nine more; cores allow one: 8 - 3 = 5, then 2.
+	got, _ := h.r.Headroom("coder", 10)
+	if got.More != 1 {
+		t.Errorf("room for %d more, expected 1 by cores: %+v", got.More, got)
+	}
+}

internal/control/reconciler/health.go

@@ -83,6 +83,12 @@ type ModelHealth struct {
 	// place one found nowhere; Placement says why.
 	Unplaceable bool
 	Placement   string
+	// Class is the capacity class it is treated as. Preempted counts the
+	// replicas it has lost to models of a higher class since the control
+	// plane started; HeldFor names the model it is giving room to now.
+	Class     string
+	Preempted int
+	HeldFor   string
 }
 
 // ModelHealth reports every model in the catalog.
@@ -106,7 +112,10 @@ func (r *Reconciler) ModelHealth() []ModelHealth {
 	out := make([]ModelHealth, 0, len(models))
 	for _, m := range models {
 		h := ModelHealth{Model: m.Name, Desired: desired[m.Name], Ready: ready[m.Name],
-			Failures: r.attempts[m.Name]}
+			Failures: r.attempts[m.Name], Class: m.EffectiveClass(), Preempted: r.preempted[m.Name]}
+		if hd, ok := r.held[m.Name]; ok && r.now().Before(hd.Until) {
+			h.HeldFor = hd.For
+		}
 		if p, ok := r.parked[m.Name]; ok {
 			h.Parked, h.ParkedSince, h.Reason = true, p.Since, p.Reason
 		}

internal/control/reconciler/measured.go

@@ -0,0 +1,108 @@
+package reconciler
+
+import (
+	"sort"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/placement"
+)
+
+// Measured is what one replica of a model on a profile was measured to serve
+// while meeting its SLO, by `llmgw ctl bench --save`.
+type Measured struct {
+	// MaxConcurrency is the most requests at once that still met the SLO.
+	MaxConcurrency int
+	// TokPerSec is the throughput at that concurrency.
+	TokPerSec float64
+}
+
+// SetMeasurements supplies benchmark results, keyed model/profile, read
+// afresh each time it is called. Without them scaling goes by queue alone and
+// profiles are chosen by priority alone.
+func (r *Reconciler) SetMeasurements(f func() map[string]Measured) {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	r.measurements = f
+}
+
+// measured is a model's measurement on one profile.
+func (r *Reconciler) measured(model, profile string) (Measured, bool) {
+	r.mu.Lock()
+	f := r.measurements
+	r.mu.Unlock()
+	if f == nil {
+		return Measured{}, false
+	}
+	m, ok := f()[model+"/"+profile]
+	return m, ok && m.MaxConcurrency > 0
+}
+
+// orderedProfiles is a model's profiles as placement should try them: by
+// priority, and among equal priorities the one measured to serve the most per
+// GPU — or per core, for a profile with none — first. Unmeasured profiles keep
+// their catalog order behind measured ones of the same priority.
+//
+// Placement sorts by priority itself, so the order found here is written back
+// as priorities, descending, for it to keep.
+func (r *Reconciler) orderedProfiles(m catalog.Model) []placement.Profile {
+	ps := m.PlacementProfiles()
+	eff := make([]float64, len(ps))
+	for i, p := range ps {
+		meas, ok := r.measured(m.Name, p.Name)
+		if !ok {
+			continue
+		}
+		units := float64(max(p.GPUsPerNode, 0) * max(p.Nodes, 1))
+		if units == 0 {
+			units = float64(max(p.CPUCores, 1))
+		}
+		eff[i] = meas.TokPerSec / units
+	}
+	idx := make([]int, len(ps))
+	for i := range idx {
+		idx[i] = i
+	}
+	sort.SliceStable(idx, func(a, b int) bool {
+		pa, pb := ps[idx[a]], ps[idx[b]]
+		if pa.Priority != pb.Priority {
+			return pa.Priority > pb.Priority
+		}
+		return eff[idx[a]] > eff[idx[b]]
+	})
+	out := make([]placement.Profile, len(ps))
+	for rank, i := range idx {
+		out[rank] = ps[i]
+		out[rank].Priority = len(ps) - rank
+	}
+	return out
+}
+
+// MeasuredFor is the measurement that sizes a model: that of the profile its
+// replicas run on, or, with none running, of the first profile placement
+// would try that was measured.
+func (r *Reconciler) MeasuredFor(model string) (Measured, string, bool) {
+	r.mu.Lock()
+	profile := ""
+	for _, rep := range r.replicas {
+		if rep.Model == model && rep.State.Counts() {
+			profile = rep.Profile
+			break
+		}
+	}
+	r.mu.Unlock()
+	if profile != "" {
+		if m, ok := r.measured(model, profile); ok {
+			return m, profile, true
+		}
+	}
+	cm, ok := r.cat.Load().Resolve(model)
+	if !ok {
+		return Measured{}, "", false
+	}
+	for _, p := range r.orderedProfiles(cm) {
+		if m, ok := r.measured(cm.Name, p.Name); ok {
+			return m, p.Name, true
+		}
+	}
+	return Measured{}, "", false
+}

internal/control/reconciler/measured_test.go

@@ -0,0 +1,96 @@
+package reconciler
+
+import (
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+)
+
+func measure(h *harness, byKey map[string]Measured) {
+	h.r.SetMeasurements(func() map[string]Measured { return byKey })
+}
+
+func TestMoreRequestsThanWereMeasuredToMeetTheSLOIsBusy(t *testing.T) {
+	m := testModel(1)
+	m.Scaling.Max = 2
+	h := newHarness(t, m, gpuNode("1", 8), gpuNode("2", 8))
+	h.r.Reconcile(t.Context())
+	h.markReady()
+	// Six in flight, nothing waiting at the engine: by the queue it is fine.
+	load := map[string]Load{"llama": {Replicas: 1, Running: 6}}
+	hold := func() {
+		h.r.RecordLoad(load)
+		h.advance(61 * time.Second)
+		h.r.RecordLoad(load)
+		h.r.Reconcile(t.Context())
+	}
+	hold()
+	if n := liveCount(h); n != 1 {
+		t.Fatalf("scaled up with no measurement and an empty queue: %d", n)
+	}
+	// Measured to meet its SLO at four at once: six is past it.
+	measure(h, map[string]Measured{"llama/tp4": {MaxConcurrency: 4, TokPerSec: 200}})
+	hold()
+	if n := liveCount(h); n != 2 {
+		t.Errorf("%d replicas with 6 in flight against a measured 4", n)
+	}
+}
+
+func TestAReplicaIsTakenBackOnlyIfTheRestWouldHaveRoomToSpare(t *testing.T) {
+	m := testModel(1)
+	m.Scaling.Max = 2
+	h := newHarness(t, m, gpuNode("1", 8), gpuNode("2", 8))
+	measure(h, map[string]Measured{"llama/tp4": {MaxConcurrency: 4, TokPerSec: 200}})
+	h.r.Reconcile(t.Context())
+	h.markReady()
+	h.r.mu.Lock()
+	h.r.extra["llama"] = 1
+	h.r.mu.Unlock()
+	h.r.Reconcile(t.Context())
+	h.markReady()
+	if liveCount(h) != 2 {
+		t.Fatal("setup: not at two replicas")
+	}
+	quietFor := func(l Load) {
+		for range 3 {
+			h.r.RecordLoad(map[string]Load{"llama": l})
+			h.advance(DefaultConfig().ScaleDownAfter + time.Second)
+			h.r.RecordLoad(map[string]Load{"llama": l})
+			h.r.Reconcile(t.Context())
+		}
+	}
+	// Nothing waiting, but three running: one replica left would be at 3 of
+	// its 4, past 70%.
+	quietFor(Load{Replicas: 2, Running: 3})
+	if n := liveCount(h); n != 2 {
+		t.Errorf("scaled down to the edge of what one replica holds: %d", n)
+	}
+	quietFor(Load{Replicas: 2, Running: 2})
+	if n := liveCount(h); n != 1 {
+		t.Errorf("two running fits one replica's 70%% of 4; still %d", n)
+	}
+}
+
+func TestTheProfileMeasuredToServeMostPerGPUIsTriedFirst(t *testing.T) {
+	m := catalog.Model{Name: "llama", Source: "/models/llama", Scaling: catalog.Scaling{Min: 1, Max: 1},
+		Profiles: []catalog.Profile{
+			{Name: "tp4", GPUClass: "h100-80g", TP: 4},
+			{Name: "tp2", GPUClass: "h100-80g", TP: 2},
+			{Name: "tp8", GPUClass: "h100-80g", TP: 8, Priority: 10},
+		}}
+	h := newHarness(t, m, gpuNode("1", 8))
+	// tp4: 400/4 = 100 per GPU; tp2: 300/2 = 150. tp8 is unmeasured but has
+	// the higher priority, which an operator set and a benchmark does not
+	// overrule.
+	measure(h, map[string]Measured{"llama/tp4": {MaxConcurrency: 8, TokPerSec: 400},
+		"llama/tp2": {MaxConcurrency: 6, TokPerSec: 300}})
+	cm, _ := h.r.cat.Load().Resolve("llama")
+	var order []string
+	for _, p := range h.r.orderedProfiles(cm) {
+		order = append(order, p.Name)
+	}
+	if len(order) != 3 || order[0] != "tp8" || order[1] != "tp2" || order[2] != "tp4" {
+		t.Errorf("order %v, expected tp8 (priority), then tp2 (150/GPU), then tp4 (100/GPU)", order)
+	}
+}

internal/control/reconciler/preempt.go

@@ -0,0 +1,324 @@
+package reconciler
+
+import (
+	"context"
+	"slices"
+	"sort"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/placement"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
+)
+
+// hold keeps a model that gave up room from taking it back.
+type hold struct {
+	Until time.Time
+	For   string
+}
+
+// maxVictims bounds how many replicas one placement may stop. Past this the
+// model is better reported as not fitting than paid for with half a cluster.
+const maxVictims = 3
+
+// victim is a replica that could be stopped for another model.
+type victim struct {
+	ID       string
+	Model    string
+	Class    string
+	AboveMin bool
+	Tier     int
+	Idle     time.Duration
+	// Size is what stopping it frees, for choosing the smaller of two
+	// otherwise equal victims.
+	Size quota.Amount
+}
+
+// preemptFor stops replicas of lower classes when a model cannot otherwise be
+// placed, and says whether it did.
+//
+// What may be taken from whom:
+//
+//   - A lower class gives way to a higher one: opportunistic to standard,
+//     standard to critical.
+//   - Within a class, only a replica above its model's minimum, and only for
+//     a model still below its own. Room a model grew into under load is not a
+//     promise; its minimum is.
+//   - A critical model's minimum is never taken.
+//
+// Victims only ever come from a lower rank, or from above a minimum for a
+// model below one, so two models cannot take turns stopping each other.
+//
+// The set is tried on a copy of the inventory first, and only stopped if it
+// would let the model be placed: stopping replicas that do not make room is
+// the one outcome worse than not preempting.
+func (r *Reconciler) preemptFor(ctx context.Context, m catalog.Model, avoid []string) bool {
+	cands := r.victims(m)
+	if len(cands) == 0 {
+		return false
+	}
+	// Candidates in order until the model fits; then every one whose
+	// stopping turns out not to be needed is kept after all. Without that
+	// second step an idle replica on the wrong node, tried first, would be
+	// stopped along with the one that actually made the room.
+	var chosen []victim
+	for _, v := range cands {
+		if len(chosen) == maxVictims {
+			break
+		}
+		chosen = append(chosen, v)
+		if !r.fitsWithout(m, avoid, chosen) {
+			continue
+		}
+		for i := len(chosen) - 1; i >= 0 && len(chosen) > 1; i-- {
+			fewer := append(append([]victim(nil), chosen[:i]...), chosen[i+1:]...)
+			if r.fitsWithout(m, avoid, fewer) {
+				chosen = fewer
+			}
+		}
+		r.preempt(ctx, m, chosen)
+		return true
+	}
+	return false
+}
+
+// fitsWithout reports whether m could be placed with these replicas gone,
+// tried on a copy of the inventory.
+func (r *Reconciler) fitsWithout(m catalog.Model, avoid []string, gone []victim) bool {
+	trial := r.inv.Clone()
+	for _, v := range gone {
+		trial.ReleaseReplica(v.ID)
+	}
+	sched := placement.New(trial)
+	res := sched.Place(r.orderedProfiles(m), avoid)
+	if res.Placement == nil && len(avoid) > 0 {
+		res = sched.Place(r.orderedProfiles(m), nil)
+	}
+	return res.Placement != nil
+}
+
+// preempt drains the chosen replicas and holds their models back.
+func (r *Reconciler) preempt(ctx context.Context, m catalog.Model, chosen []victim) {
+	now := r.now()
+	var ids []string
+	r.mu.Lock()
+	for _, v := range chosen {
+		if rep, ok := r.replicas[v.ID]; ok {
+			rep.PreemptedFor = m.Name
+		}
+		ids = append(ids, v.ID)
+		r.held[v.Model] = hold{Until: now.Add(r.cfg.PreemptHold), For: m.Name}
+		r.preempted[v.Model]++
+	}
+	r.preempting[m.Name] = ids
+	r.mu.Unlock()
+
+	for _, v := range chosen {
+		r.log.Warn("preempting a replica to make room",
+			"replica", v.ID, "model", v.Model, "class", v.Class, "above_min", v.AboveMin,
+			"for", m.Name, "for_class", m.EffectiveClass())
+	}
+	r.scaleDown(ctx, ids)
+}
+
+// awaitingPreemption reports whether replicas stopped for a model are still
+// on their way out, so it waits for them rather than stopping more.
+func (r *Reconciler) awaitingPreemption(model string) bool {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	ids := r.preempting[model]
+	ids = slices.DeleteFunc(slices.Clone(ids), func(id string) bool {
+		_, ok := r.replicas[id]
+		return !ok
+	})
+	if len(ids) == 0 {
+		delete(r.preempting, model)
+		return false
+	}
+	r.preempting[model] = ids
+	return true
+}
+
+// isHeld reports whether a model gave up room and may not take it back yet.
+// The hold ends early once the model it was given to has what it asked for.
+//
+// Today a preempted replica is only removed by its own model's drain loop,
+// which runs after that model has decided not to scale up, so the room it
+// frees reaches the model it was freed for first anyway. The hold is what
+// keeps that true for any other way a replica can go — a stop reported
+// between passes, a reap at the start of one — rather than it resting on the
+// order of lines in reconcileModel.
+func (r *Reconciler) isHeld(model string) bool {
+	r.mu.Lock()
+	h, ok := r.held[model]
+	r.mu.Unlock()
+	if !ok {
+		return false
+	}
+	if r.now().After(h.Until) {
+		r.mu.Lock()
+		delete(r.held, model)
+		r.mu.Unlock()
+		return false
+	}
+	if to, ok := r.cat.Load().Resolve(h.For); ok && r.liveCount(h.For) >= r.desiredFor(to) {
+		r.mu.Lock()
+		delete(r.held, model)
+		r.mu.Unlock()
+		return false
+	}
+	return true
+}
+
+// liveCount is how many replicas of a model hold capacity.
+func (r *Reconciler) liveCount(model string) int {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	n := 0
+	for _, rep := range r.replicas {
+		if rep.Model == model && rep.State.Counts() && rep.RetiringAt.IsZero() {
+			n++
+		}
+	}
+	return n
+}
+
+// victims lists the replicas that may be stopped for m, in the order they
+// should be: idle on-demand ones first, then what other tenants borrowed past
+// their guarantees when m's owner is within its own, then what lower classes
+// grew into under load, then the same within m's class, then lower classes'
+// minimums; within each, the longest unused first.
+//
+// A replica within its owner's guarantee is taken only for a model of the
+// same owner: a guarantee that another tenant's higher class could take is
+// not one.
+func (r *Reconciler) victims(m catalog.Model) []victim {
+	cat := r.cat.Load()
+	now := r.now()
+	myRank := catalog.ClassRank(m.EffectiveClass())
+	belowMin := r.liveCount(m.Name) < m.Scaling.Min
+
+	pol := r.policy()
+	var borrowed map[string]bool
+	reclaim := false
+	if pol != nil {
+		hs := r.Holdings()
+		borrowed = pol.Borrowed(hs)
+		reclaim = pol.Guaranteed(m.Owner, pol.Usage(hs), profileNeed(m))
+	}
+
+	r.mu.Lock()
+	defer r.mu.Unlock()
+
+	// Per model, its live replicas newest first: the newest are the ones
+	// above its minimum.
+	byModel := map[string][]*Replica{}
+	for _, rep := range r.replicas {
+		if rep.Model == m.Name || !rep.State.Counts() || !rep.RetiringAt.IsZero() || rep.PreemptedFor != "" {
+			continue
+		}
+		byModel[rep.Model] = append(byModel[rep.Model], rep)
+	}
+
+	var out []victim
+	for model, reps := range byModel {
+		other, ok := cat.Resolve(model)
+		if !ok {
+			continue
+		}
+		class := other.EffectiveClass()
+		rank := catalog.ClassRank(class)
+		sort.Slice(reps, func(i, j int) bool { return reps[i].CreatedAt.After(reps[j].CreatedAt) })
+		above := max(len(reps)-other.Scaling.Min, 0)
+		for i, rep := range reps {
+			aboveMin := i < above
+			tier := -1
+			switch {
+			case rank < myRank && other.Scaling.OnDemand() && idleFor(rep, now) >= r.cfg.EvictionGrace:
+				tier = 0
+			case rank < myRank && aboveMin:
+				tier = 2
+			case rank == myRank && aboveMin && belowMin:
+				tier = 3
+			case rank < myRank && class != catalog.ClassCritical:
+				tier = 4
+			}
+			if pol != nil && other.Owner != m.Owner {
+				switch {
+				case borrowed[rep.ID] && reclaim && (class != catalog.ClassCritical || aboveMin):
+					// Past its owner's guarantee, wanted by an owner within
+					// its own: taken back whatever the classes.
+					if tier != 0 {
+						tier = 1
+					}
+				case !borrowed[rep.ID] && guaranteed(pol, other.Owner):
+					continue // held within its owner's guarantee
+				}
+			}
+			if tier < 0 {
+				continue
+			}
+			// Just started: stopping it throws away a model load that has
+			// only now been paid for.
+			if !rep.ReadyAt.IsZero() && now.Sub(rep.ReadyAt) < r.cfg.MinLifetime {
+				continue
+			}
+			if len(nodesOf(rep)) > 1 {
+				continue // see makeRoomFor: a multi-node replica is its own decision
+			}
+			prof, _ := other.ProfileByName(rep.Profile)
+			var size quota.Amount
+			for _, sh := range rep.Shards {
+				size = size.Plus(quota.Amount{GPUs: len(sh.GPUIndices), CPUCores: cpuCores(prof), MemoryGB: prof.MinMemoryGB})
+			}
+			out = append(out, victim{ID: rep.ID, Model: model, Class: class,
+				AboveMin: aboveMin, Tier: tier, Idle: idleFor(rep, now), Size: size})
+		}
+	}
+	sort.SliceStable(out, func(i, j int) bool {
+		a, b := out[i], out[j]
+		if a.Tier != b.Tier {
+			return a.Tier < b.Tier
+		}
+		if a.Idle != b.Idle {
+			return a.Idle > b.Idle
+		}
+		// The smaller first: the least served is lost if it is enough, and
+		// the pruning after keeps a larger one only when it is needed.
+		if a.Size.GPUs != b.Size.GPUs {
+			return a.Size.GPUs < b.Size.GPUs
+		}
+		if a.Size.MemoryGB != b.Size.MemoryGB {
+			return a.Size.MemoryGB < b.Size.MemoryGB
+		}
+		return a.ID < b.ID
+	})
+	return out
+}
+
+// idleFor is how long a replica has gone unused; zero when nobody has said,
+// which is not the same as idle.
+func idleFor(rep *Replica, now time.Time) time.Duration {
+	if rep.LastUsedAt.IsZero() {
+		return 0
+	}
+	return now.Sub(rep.LastUsedAt)
+}
+
+func nodesOf(rep *Replica) map[string]struct{} {
+	out := map[string]struct{}{}
+	for _, sh := range rep.Shards {
+		out[sh.NodeID] = struct{}{}
+	}
+	return out
+}
+
+// guaranteed reports whether an owner has a guarantee anywhere up its tree.
+func guaranteed(pol *quota.Policy, owner string) bool {
+	for _, q := range pol.Governing(owner) {
+		if q.Guarantee.Any() {
+			return true
+		}
+	}
+	return false
+}

internal/control/reconciler/preempt_test.go

@@ -0,0 +1,418 @@
+package reconciler
+
+import (
+	"log/slog"
+	"strings"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
+)
+
+// newFleetHarness is newHarness with a catalog of several models.
+func newFleetHarness(t *testing.T, models []catalog.Model, nodes ...inventory.Node) *harness {
+	t.Helper()
+	cat, err := catalog.New(models)
+	if err != nil {
+		t.Fatal(err)
+	}
+	inv := inventory.NewStore()
+	c := newClock()
+	for _, n := range nodes {
+		inv.Upsert(n)
+	}
+	disp := &fakeDispatcher{}
+	cfg := DefaultConfig()
+	cfg.UnreachableAfter = 15 * time.Second
+	cfg.LostAfter = 60 * time.Second
+	r := New(cfg, catalog.NewLive(cat), inv, disp, slog.New(slog.DiscardHandler))
+	r.SetClock(c.now)
+	c.advance(cfg.SettleFor)
+	for _, n := range nodes {
+		touchAt(inv, n.ID, c.now())
+	}
+	return &harness{r: r, inv: inv, disp: disp, clock: c}
+}
+
+// gpuModel runs on one whole 8-GPU node.
+func gpuModel(name, class string, min int) catalog.Model {
+	return catalog.Model{Name: name, Source: "/models/" + name, Class: class,
+		Scaling:  catalog.Scaling{Min: min, Max: min},
+		Profiles: []catalog.Profile{{Name: "tp8", Engine: "vllm", GPUClass: "h100-80g", TP: 8}}}
+}
+
+// settle runs passes until nothing changes, marking what starts ready and
+// letting preempted replicas run out their grace.
+func (h *harness) settle(t *testing.T, passes int) {
+	t.Helper()
+	for range passes {
+		h.r.Reconcile(t.Context())
+		h.markReady()
+		h.advance(h.r.cfg.PreemptGrace + time.Second)
+	}
+}
+
+func replicasOf(h *harness, model string) (live int) {
+	for _, rep := range h.r.Replicas() {
+		if rep.Model == model && rep.State.Counts() {
+			live++
+		}
+	}
+	return live
+}
+
+// fill places the standard model, then ages it past MinLifetime so it may be
+// taken from, and only then adds the model that wants its node.
+func fillThenWant(t *testing.T, want catalog.Model, have catalog.Model, nodes ...inventory.Node) *harness {
+	t.Helper()
+	h := newFleetHarness(t, []catalog.Model{have, withMin(want, 0)}, nodes...)
+	h.settle(t, 3)
+	if got := replicasOf(h, have.Name); got != have.Scaling.Min {
+		t.Fatalf("%s has %d replicas before anything wanted its room", have.Name, got)
+	}
+	h.advance(h.r.cfg.MinLifetime + time.Minute)
+	cat, err := catalog.New([]catalog.Model{have, want})
+	if err != nil {
+		t.Fatal(err)
+	}
+	h.r.cat.Store(cat)
+	return h
+}
+
+func withMin(m catalog.Model, n int) catalog.Model {
+	m.Scaling = catalog.Scaling{Min: n, Max: n}
+	return m
+}
+
+func TestACriticalModelTakesAStandardModelsNode(t *testing.T) {
+	h := fillThenWant(t, gpuModel("vital", catalog.ClassCritical, 1), gpuModel("chat", "", 1), gpuNode("1", 8))
+
+	h.r.Reconcile(t.Context())
+	var draining int
+	for _, rep := range h.r.Replicas() {
+		if rep.Model == "chat" && rep.State == Draining && rep.PreemptedFor == "vital" {
+			draining++
+		}
+	}
+	if draining != 1 {
+		t.Fatalf("the standard replica was not drained for the critical one: %+v", h.r.Replicas())
+	}
+	if hs := health(h, "chat"); hs.HeldFor != "vital" || hs.Preempted != 1 {
+		t.Errorf("the victim's health does not say it gave way: %+v", hs)
+	}
+
+	h.settle(t, 4)
+	if replicasOf(h, "vital") != 1 {
+		t.Fatalf("the critical model was not placed once room was made: %+v", h.r.Replicas())
+	}
+	// Held back until the critical model had its replica, then short again
+	// because there is no room: it must not have taken the node back — not
+	// even for a moment, which would load a model only to preempt it again.
+	if replicasOf(h, "chat") != 0 {
+		t.Error("the standard model took the node back")
+	}
+	started := 0
+	for _, cmd := range h.disp.startedShards() {
+		if strings.HasPrefix(cmd.ReplicaID, "chat-") {
+			started++
+		}
+	}
+	if started != 1 {
+		t.Errorf("the standard model started %d replicas; it should not have tried again", started)
+	}
+}
+
+func TestAStandardModelDoesNotTakeAnothersMinimum(t *testing.T) {
+	h := fillThenWant(t, gpuModel("second", "", 1), gpuModel("first", "", 1), gpuNode("1", 8))
+	h.settle(t, 3)
+	if replicasOf(h, "first") != 1 || replicasOf(h, "second") != 0 {
+		t.Errorf("a model of the same class took another's minimum: %+v", h.r.Replicas())
+	}
+	if hs := health(h, "second"); !hs.Unplaceable {
+		t.Error("the model left without room should say it cannot be placed")
+	}
+}
+
+func TestWithinAClassOnlyWhatWasGrownIntoIsTaken(t *testing.T) {
+	grown := gpuModel("busy", "", 1)
+	grown.Scaling = catalog.Scaling{Min: 1, Max: 2}
+	h := newFleetHarness(t, []catalog.Model{grown, withMin(gpuModel("new", "", 1), 0)}, gpuNode("1", 8), gpuNode("2", 8))
+	h.settle(t, 2)
+	// Load grows it to its max: the second replica is above its minimum.
+	h.r.mu.Lock()
+	h.r.extra["busy"] = 1
+	h.r.mu.Unlock()
+	h.settle(t, 3)
+	if replicasOf(h, "busy") != 2 {
+		t.Fatalf("setup: busy did not grow to 2: %+v", h.r.Replicas())
+	}
+	h.advance(h.r.cfg.MinLifetime + time.Minute)
+	cat, _ := catalog.New([]catalog.Model{grown, gpuModel("new", "", 1)})
+	h.r.cat.Store(cat)
+
+	h.settle(t, 5)
+	if replicasOf(h, "new") != 1 || replicasOf(h, "busy") != 1 {
+		t.Errorf("the grown replica should give way to a minimum: busy %d, new %d",
+			replicasOf(h, "busy"), replicasOf(h, "new"))
+	}
+}
+
+func TestNothingIsStoppedThatWouldNotMakeRoom(t *testing.T) {
+	// Two nodes, each half used by an opportunistic model; the critical one
+	// needs a whole node. Stopping one frees half a node, which is no use —
+	// and both together are on two nodes, which is no use either.
+	half := catalog.Model{Name: "half", Source: "/m", Class: catalog.ClassOpportunistic,
+		Scaling:  catalog.Scaling{Min: 2, Max: 2},
+		Profiles: []catalog.Profile{{Name: "tp4", Engine: "vllm", GPUClass: "h100-80g", TP: 4}}}
+	five := func(id string) inventory.Node { return gpuNode(id, 5) }
+	h := fillThenWant(t, gpuModel("vital", catalog.ClassCritical, 1), half, five("1"), five("2"))
+	h.r.Reconcile(t.Context())
+	for _, rep := range h.r.Replicas() {
+		if rep.PreemptedFor != "" || rep.State == Draining {
+			t.Fatalf("a replica was stopped without making room: %+v", rep)
+		}
+	}
+}
+
+func TestACriticalMinimumIsNeverTaken(t *testing.T) {
+	h := fillThenWant(t, gpuModel("also-vital", catalog.ClassCritical, 1), gpuModel("vital", catalog.ClassCritical, 1), gpuNode("1", 8))
+	h.settle(t, 3)
+	if replicasOf(h, "vital") != 1 {
+		t.Error("a critical model's minimum was taken")
+	}
+}
+
+func TestAReplicaThatJustStartedIsLeftAlone(t *testing.T) {
+	h := newFleetHarness(t, []catalog.Model{gpuModel("chat", "", 1), withMin(gpuModel("vital", catalog.ClassCritical, 1), 0)}, gpuNode("1", 8))
+	h.settle(t, 2)
+	cat, _ := catalog.New([]catalog.Model{gpuModel("chat", "", 1), gpuModel("vital", catalog.ClassCritical, 1)})
+	h.r.cat.Store(cat)
+	h.r.Reconcile(t.Context())
+	for _, rep := range h.r.Replicas() {
+		if rep.State == Draining {
+			t.Fatal("a replica that had only just loaded was preempted")
+		}
+	}
+}
+
+func health(h *harness, model string) ModelHealth {
+	for _, x := range h.r.ModelHealth() {
+		if x.Model == model {
+			return x
+		}
+	}
+	return ModelHealth{}
+}
+
+func TestAHoldLastsUntilTheRoomIsUsedOrItExpires(t *testing.T) {
+	h := fillThenWant(t, gpuModel("vital", catalog.ClassCritical, 1), gpuModel("chat", "", 1), gpuNode("1", 8))
+	h.r.Reconcile(t.Context())
+	if !h.r.isHeld("chat") {
+		t.Fatal("the model that gave way is not held")
+	}
+	if h.r.isHeld("vital") {
+		t.Error("the model room was made for is held")
+	}
+	// Released as soon as the model it was given to has what it asked for.
+	h.settle(t, 3)
+	if replicasOf(h, "vital") != 1 {
+		t.Fatalf("setup: vital was not placed: %+v", h.r.Replicas())
+	}
+	if h.r.isHeld("chat") {
+		t.Error("the hold outlasted its purpose")
+	}
+
+	// And, whatever happens, after PreemptHold.
+	h.r.mu.Lock()
+	h.r.held["other"] = hold{Until: h.r.now().Add(time.Minute), For: "never-placed"}
+	h.r.mu.Unlock()
+	if !h.r.isHeld("other") {
+		t.Fatal("a fresh hold is not in force")
+	}
+	h.advance(2 * time.Minute)
+	if h.r.isHeld("other") {
+		t.Error("a hold outlived its time")
+	}
+}
+
+func TestOnlyTheReplicasThatMakeTheRoomAreStopped(t *testing.T) {
+	// Two half-used nodes. "idle" on node 1 is on demand and long unused, so
+	// it is first in line; but the critical model needs node 2's other half,
+	// which only stopping "chat" frees.
+	idle := catalog.Model{Name: "idle", Source: "/m", Scaling: catalog.Scaling{Min: 0, Max: 1},
+		Profiles: []catalog.Profile{{Name: "tp4", Engine: "vllm", GPUClass: "h100-80g", TP: 4}}}
+	chat := catalog.Model{Name: "chat", Source: "/m", Scaling: catalog.Scaling{Min: 1, Max: 1},
+		Profiles: []catalog.Profile{{Name: "tp6", Engine: "vllm", GPUClass: "h100-80g", TP: 6}}}
+	vital := catalog.Model{Name: "vital", Source: "/m", Class: catalog.ClassCritical,
+		Scaling:  catalog.Scaling{Min: 1, Max: 1},
+		Profiles: []catalog.Profile{{Name: "tp8", Engine: "vllm", GPUClass: "h100-80g", TP: 8}}}
+	h := newFleetHarness(t, []catalog.Model{idle, chat}, gpuNode("1", 6), gpuNode("2", 8))
+	h.r.Want("idle")
+	h.settle(t, 4)
+	var idleNode, chatNode string
+	for _, rep := range h.r.Replicas() {
+		switch rep.Model {
+		case "idle":
+			idleNode = rep.Shards[0].NodeID
+		case "chat":
+			chatNode = rep.Shards[0].NodeID
+		}
+	}
+	if idleNode == "" || chatNode != "2" {
+		t.Fatalf("setup: idle on %q, chat on %q; chat must hold node 2: %+v", idleNode, chatNode, h.r.Replicas())
+	}
+	h.r.mu.Lock()
+	for _, rep := range h.r.replicas {
+		rep.LastUsedAt = h.r.now().Add(-time.Hour)
+	}
+	h.r.mu.Unlock()
+	h.advance(h.r.cfg.MinLifetime + time.Minute)
+	cat, _ := catalog.New([]catalog.Model{idle, chat, vital})
+	h.r.cat.Store(cat)
+
+	h.r.Reconcile(t.Context())
+	var stopped []string
+	for _, rep := range h.r.Replicas() {
+		if rep.PreemptedFor == "vital" {
+			stopped = append(stopped, rep.Model)
+		}
+	}
+	if len(stopped) != 1 || stopped[0] != "chat" {
+		t.Errorf("stopped %v; only chat's replica makes room for a whole node", stopped)
+	}
+}
+
+func TestOfTwoThatWouldEachMakeRoomTheSmallerIsStopped(t *testing.T) {
+	cpu := inventory.Node{ID: "a", MgmtAddr: "10.0.0.1", State: inventory.Ready, MemoryTotalMB: 31 * 1024, CPUCount: 8}
+	big := catalog.Model{Name: "big", Source: "/m", Scaling: catalog.Scaling{Min: 1, Max: 1},
+		Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 12, CPUCores: 4}}}
+	small := catalog.Model{Name: "small", Source: "/m", Scaling: catalog.Scaling{Min: 1, Max: 1},
+		Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 2, CPUCores: 1}}}
+	vital := catalog.Model{Name: "vital", Source: "/m", Class: catalog.ClassCritical,
+		Scaling:  catalog.Scaling{Min: 1, Max: 1},
+		Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 15, CPUCores: 4}}}
+	// 8 cores: big 4 + small 1 leaves 3, and vital needs 4. Stopping either
+	// is enough; stopping big loses far more.
+	for range 5 { // the choice must not depend on map order
+		h := newFleetHarness(t, []catalog.Model{big, small}, cpu)
+		h.settle(t, 3)
+		h.advance(h.r.cfg.MinLifetime + time.Minute)
+		cat, _ := catalog.New([]catalog.Model{big, small, vital})
+		h.r.cat.Store(cat)
+		h.r.Reconcile(t.Context())
+		var stopped []string
+		for _, rep := range h.r.Replicas() {
+			if rep.PreemptedFor == "vital" {
+				stopped = append(stopped, rep.Model)
+			}
+		}
+		if len(stopped) != 1 || stopped[0] != "small" {
+			t.Fatalf("stopped %v; the small replica alone makes room", stopped)
+		}
+	}
+}
+
+func TestHeadroomCountsWhatAHigherClassCouldTake(t *testing.T) {
+	h := fillThenWant(t, gpuModel("vital", catalog.ClassCritical, 1), gpuModel("chat", "", 1), gpuNode("1", 8))
+	got, _ := h.r.Headroom("vital", 3)
+	if got.More != 0 || got.ByPreempting != 1 || len(got.Preempts) != 1 || got.Preempts[0] != "chat" {
+		t.Errorf("a critical model should see one more by stopping chat: %+v", got)
+	}
+	// Tried on a copy: nothing was stopped.
+	if replicasOf(h, "chat") != 1 {
+		t.Fatal("asking about room stopped a replica")
+	}
+	// A model of chat's own class may not take its minimum, so has no such room.
+	peer := newFleetHarness(t, []catalog.Model{gpuModel("chat", "", 1), withMin(gpuModel("peer", "", 1), 0)}, gpuNode("1", 8))
+	peer.settle(t, 3)
+	peer.advance(peer.r.cfg.MinLifetime + time.Minute)
+	if got, _ := peer.r.Headroom("peer", 3); got.ByPreempting != 0 {
+		t.Errorf("a standard model counted room in another's minimum: %+v", got)
+	}
+}
+
+func TestARestartBorrowsALowerClassesRoomAndGivesItBack(t *testing.T) {
+	// chat fills node 1; the only other node holds an opportunistic replica.
+	chat := gpuModel("chat", "", 1)
+	extra := gpuModel("extra", catalog.ClassOpportunistic, 1)
+	h := newFleetHarness(t, []catalog.Model{chat, extra}, gpuNode("1", 8), gpuNode("2", 8))
+	h.settle(t, 3)
+	h.advance(h.r.cfg.MinLifetime + time.Minute)
+	var old string
+	for _, rep := range h.r.Replicas() {
+		if rep.Model == "chat" {
+			old = rep.ID
+		}
+	}
+	if old == "" || replicasOf(h, "extra") != 1 {
+		t.Fatalf("setup: %+v", h.r.Replicas())
+	}
+
+	if err := h.r.Restart(old); err != nil {
+		t.Fatal(err)
+	}
+	// Replacement first, beside the old one, in the room extra gives up;
+	// the old one keeps serving until then.
+	servedThroughout := true
+	for range 8 {
+		h.r.Reconcile(t.Context())
+		h.markReady()
+		ready := 0
+		for _, rep := range h.r.Replicas() {
+			if rep.Model == "chat" && rep.State == Ready {
+				ready++
+			}
+		}
+		if ready == 0 {
+			servedThroughout = false
+		}
+		h.advance(h.r.cfg.PreemptGrace + time.Second)
+	}
+	if !servedThroughout {
+		t.Error("chat had no ready replica during its restart")
+	}
+	var newChat bool
+	for _, rep := range h.r.Replicas() {
+		if rep.Model == "chat" && rep.ID != old && rep.State == Ready {
+			newChat = true
+		}
+		if rep.ID == old {
+			t.Errorf("the old replica is still there: %+v", rep)
+		}
+	}
+	if !newChat {
+		t.Fatalf("no new chat replica: %+v", h.r.Replicas())
+	}
+	// And extra has its replica back, in the room the old chat left.
+	h.settle(t, 4)
+	if replicasOf(h, "extra") != 1 {
+		t.Errorf("the opportunistic model did not get its room back: %+v", h.r.Replicas())
+	}
+}
+
+func TestHeadroomNamesOnlyTheReplicasThatWouldMakeRoom(t *testing.T) {
+	// vital needs a whole node and chat is on the only one; a second
+	// critical can never fit. Asking for 3: one by stopping chat, then no
+	// candidates left — nothing else may be named.
+	h := fillThenWant(t, gpuModel("vital", catalog.ClassCritical, 1), gpuModel("chat", "", 1), gpuNode("1", 8))
+	got, _ := h.r.Headroom("vital", 3)
+	if got.ByPreempting != 1 || len(got.Preempts) != 1 {
+		t.Errorf("%+v", got)
+	}
+	// Free room for one, and candidates that cannot make room for a second:
+	// none of them may be listed.
+	small := catalog.Model{Name: "small", Source: "/m", Class: catalog.ClassOpportunistic,
+		Scaling:  catalog.Scaling{Min: 1, Max: 1},
+		Profiles: []catalog.Profile{{Name: "tp2", Engine: "vllm", GPUClass: "h100-80g", TP: 2}}}
+	h2 := fillThenWant(t, gpuModel("big", "", 1), small, gpuNode("1", 8), gpuNode("2", 8))
+	got2, _ := h2.r.Headroom("big", 3)
+	if got2.More != 1 || got2.ByPreempting != 1 || len(got2.Preempts) != 1 {
+		t.Errorf("one free, one by stopping small: %+v", got2)
+	}
+	h3 := fillThenWant(t, gpuModel("big", "", 1), small, gpuNode("1", 8), gpuNode("2", 6))
+	got3, _ := h3.r.Headroom("big", 3)
+	if got3.ByPreempting != 0 || len(got3.Preempts) != 0 {
+		t.Errorf("stopping small frees a 6-GPU node, too small for big; nobody may be named: %+v", got3)
+	}
+}

internal/control/reconciler/quotas.go

@@ -0,0 +1,105 @@
+package reconciler
+
+import (
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/placement"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// SetQuotas supplies the tenancy tree's capacity quotas, read afresh each
+// time it is called. Without it nothing is guaranteed or limited, and
+// preemption goes by class alone.
+func (r *Reconciler) SetQuotas(f func() *quota.Policy) {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	r.quotas = f
+}
+
+func (r *Reconciler) policy() *quota.Policy {
+	r.mu.Lock()
+	f := r.quotas
+	r.mu.Unlock()
+	if f == nil {
+		return nil
+	}
+	return f()
+}
+
+// Holdings is what every replica holds, and for whom.
+func (r *Reconciler) Holdings() []quota.Holding {
+	cat := r.cat.Load()
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	out := make([]quota.Holding, 0, len(r.replicas))
+	for _, rep := range r.replicas {
+		m, ok := cat.Resolve(rep.Model)
+		if !ok {
+			continue
+		}
+		prof, _ := m.ProfileByName(rep.Profile)
+		var a quota.Amount
+		for _, sh := range rep.Shards {
+			a = a.Plus(quota.Amount{GPUs: len(sh.GPUIndices), CPUCores: cpuCores(prof), MemoryGB: prof.MinMemoryGB})
+		}
+		out = append(out, quota.Holding{Replica: rep.ID, Owner: m.Owner, Amount: a, Since: rep.CreatedAt})
+	}
+	return out
+}
+
+// placementNeed is what a placement would hold.
+func placementNeed(m catalog.Model, pl *placement.Placement) quota.Amount {
+	prof, _ := m.ProfileByName(pl.Profile.Name)
+	var a quota.Amount
+	for _, n := range pl.Nodes {
+		a = a.Plus(quota.Amount{GPUs: len(n.GPUIndices), CPUCores: cpuCores(prof), MemoryGB: prof.MinMemoryGB})
+	}
+	return a
+}
+
+// profileNeed is what a replica of a model's first profile would hold, for
+// deciding before there is a placement.
+func profileNeed(m catalog.Model) quota.Amount {
+	if len(m.Profiles) == 0 {
+		return quota.Amount{}
+	}
+	p := m.Profiles[0]
+	nodes := max(p.Nodes, 1)
+	gpus := p.GPUsPerNode
+	if gpus == 0 && !p.CPUOnly() {
+		gpus = max(p.TP, 1)
+	}
+	return quota.Amount{GPUs: gpus * nodes, CPUCores: cpuCores(p) * nodes, MemoryGB: p.MinMemoryGB * nodes}
+}
+
+// overLimit says why a placement would take its model's owner past a limit.
+func (r *Reconciler) overLimit(m catalog.Model, pl *placement.Placement) string {
+	pol := r.policy()
+	if pol == nil || m.Owner == "" {
+		return ""
+	}
+	return pol.AtLimit(m.Owner, pol.Usage(r.Holdings()), placementNeed(m, pl))
+}
+
+// QuotaUse is one scope's quota beside what it holds.
+type QuotaUse struct {
+	Scope     string
+	Guarantee store.Resources
+	Limit     store.Resources
+	Holds     quota.Amount
+}
+
+// QuotaUse lists every quota with what its scope holds now.
+func (r *Reconciler) QuotaUse() []QuotaUse {
+	pol := r.policy()
+	if pol == nil {
+		return nil
+	}
+	usage := pol.Usage(r.Holdings())
+	var out []QuotaUse
+	for _, q := range pol.Quotas() {
+		key := quota.Key(q.Scope, q.ScopeID)
+		out = append(out, QuotaUse{Scope: key, Guarantee: q.Guarantee, Limit: q.Limit, Holds: usage[key]})
+	}
+	return out
+}

internal/control/reconciler/quotas_test.go

@@ -0,0 +1,81 @@
+package reconciler
+
+import (
+	"strings"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+func owned(m catalog.Model, owner string) catalog.Model { m.Owner = owner; return m }
+
+func gpus(n int) store.Resources { return store.Resources{GPUs: &n} }
+
+// Two teams in one org; each 8-GPU node holds one tp8 replica.
+func withQuotas(h *harness, qs ...store.CapacityQuota) {
+	pol := quota.New(qs, map[string][]string{"team:ml": {"org:acme"}, "team:web": {"org:acme"}})
+	h.r.SetQuotas(func() *quota.Policy { return pol })
+}
+
+func TestATenantWithinItsGuaranteeTakesBackWhatAnotherBorrowed(t *testing.T) {
+	// web is guaranteed one node and borrowed the second; ml is guaranteed
+	// one node and wants it. Same class, and web's replica is at its minimum
+	// — by class alone nothing could be taken.
+	web := owned(gpuModel("web-chat", "", 2), "team:web")
+	ml := owned(gpuModel("ml-train", "", 1), "team:ml")
+	h := newFleetHarness(t, []catalog.Model{web, withMin(ml, 0)}, gpuNode("1", 8), gpuNode("2", 8))
+	withQuotas(h,
+		store.CapacityQuota{Scope: store.ScopeTeam, ScopeID: "web", Guarantee: gpus(8)},
+		store.CapacityQuota{Scope: store.ScopeTeam, ScopeID: "ml", Guarantee: gpus(8)})
+	h.settle(t, 4)
+	if replicasOf(h, "web-chat") != 2 {
+		t.Fatalf("setup: web did not borrow the second node: %+v", h.r.Replicas())
+	}
+	h.advance(h.r.cfg.MinLifetime + time.Minute)
+	cat, _ := catalog.New([]catalog.Model{web, ml})
+	h.r.cat.Store(cat)
+
+	h.settle(t, 5)
+	if replicasOf(h, "ml-train") != 1 || replicasOf(h, "web-chat") != 1 {
+		t.Errorf("the borrowed node was not taken back: web %d, ml %d",
+			replicasOf(h, "web-chat"), replicasOf(h, "ml-train"))
+	}
+}
+
+func TestAReplicaWithinItsGuaranteeIsNotTakenByAHigherClassElsewhere(t *testing.T) {
+	web := owned(gpuModel("web-chat", "", 1), "team:web")
+	vital := owned(gpuModel("vital", catalog.ClassCritical, 1), "team:ml")
+	h := fillThenWant(t, vital, web, gpuNode("1", 8))
+	withQuotas(h, store.CapacityQuota{Scope: store.ScopeTeam, ScopeID: "web", Guarantee: gpus(8)})
+	h.settle(t, 3)
+	if replicasOf(h, "web-chat") != 1 || replicasOf(h, "vital") != 0 {
+		t.Errorf("a guaranteed replica was preempted by another tenant's class: %+v", h.r.Replicas())
+	}
+	// Without the guarantee, critical takes it: the guarantee was the reason.
+	h2 := fillThenWant(t, vital, web, gpuNode("1", 8))
+	h2.settle(t, 3)
+	if replicasOf(h2, "vital") != 1 {
+		t.Error("setup: without a guarantee the critical model should have taken the node")
+	}
+}
+
+func TestALimitStopsAModelBeforeItStarts(t *testing.T) {
+	web := owned(gpuModel("web-chat", "", 2), "team:web")
+	h := newFleetHarness(t, []catalog.Model{web}, gpuNode("1", 8), gpuNode("2", 8))
+	withQuotas(h, store.CapacityQuota{Scope: store.ScopeOrg, ScopeID: "acme", Limit: gpus(8)})
+	h.settle(t, 4)
+	if replicasOf(h, "web-chat") != 1 {
+		t.Errorf("a second replica went past the org's 8-GPU limit: %+v", h.r.Replicas())
+	}
+	hs := health(h, "web-chat")
+	if !hs.Unplaceable || !strings.Contains(hs.Placement, "org:acme is at its capacity limit (16 GPUs of 8)") {
+		t.Errorf("the refusal should name the limit: %+v", hs)
+	}
+	room, _ := h.r.Headroom("web-chat", 5)
+	if room.More != 0 || len(room.Blocked) != 1 || !strings.Contains(room.Blocked[0], "capacity limit") {
+		t.Errorf("headroom should stop at the limit, not the empty node: %+v", room)
+	}
+}

internal/control/reconciler/reconciler.go

@@ -15,6 +15,7 @@ import (
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/placement"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
 )
 
 // ShardCommand is everything an agent needs to launch one shard.
@@ -65,6 +66,12 @@ type Config struct {
 	// Without a bound, draining a node whose model cannot be placed anywhere
 	// else would never finish.
 	DrainGrace time.Duration
+	// PreemptGrace is how long a replica stopped to make room for another
+	// is given to finish what it is serving: a request, not a replacement.
+	PreemptGrace time.Duration
+	// PreemptHold is how long a model that gave up room may not take it back,
+	// so the replica it was given up for is placed first.
+	PreemptHold time.Duration
 	// ScaleUpWaiting is the average number of requests waiting per ready
 	// replica, held for ScaleUpAfter, that adds a replica — up to the model's
 	// max. ScaleDownAfter is how long with nothing waiting before one of the
@@ -141,6 +148,8 @@ func DefaultConfig() Config {
 		EvictionGrace:    2 * time.Minute,
 		MinLifetime:      5 * time.Minute,
 		DrainGrace:       5 * time.Minute,
+		PreemptGrace:     30 * time.Second,
+		PreemptHold:      5 * time.Minute,
 		ParkAfter:        5,
 		ScaleUpWaiting:   1,
 		ScaleUpAfter:     time.Minute,
@@ -158,6 +167,12 @@ func (c *Config) normalize() {
 	if c.BaseBackoff <= 0 {
 		c.BaseBackoff = d.BaseBackoff
 	}
+	if c.PreemptGrace <= 0 {
+		c.PreemptGrace = d.PreemptGrace
+	}
+	if c.PreemptHold <= 0 {
+		c.PreemptHold = d.PreemptHold
+	}
 	if c.MaxBackoff < c.BaseBackoff {
 		c.MaxBackoff = max(c.BaseBackoff, d.MaxBackoff)
 	}
@@ -250,6 +265,16 @@ type Reconciler struct {
 	// placementLogged remembers the last "no placement" said per model, so it
 	// is said when it changes rather than on every five-second pass.
 	placementLogged map[string]placementNote
+	// preempting is, per model, the replicas stopped to make room for it and
+	// not gone yet; held is, per model that gave them up, until when it may
+	// not take the room back; preempted counts, per model, replicas it lost.
+	preempting map[string][]string
+	held       map[string]hold
+	preempted  map[string]int
+	// quotas is the tenancy tree's capacity quotas as last read; nil is none.
+	quotas func() *quota.Policy
+	// measurements are benchmark results by model/profile; nil is none.
+	measurements func() map[string]Measured
 
 	wake chan struct{}
 	stop chan struct{}
@@ -285,6 +310,9 @@ func New(cfg Config, cat *catalog.Live, inv *inventory.Store, disp Dispatcher, l
 		quietSince:      map[string]time.Time{},
 		placementLogged: map[string]placementNote{},
 		lastExplain:     map[string]placement.Result{},
+		preempting:      map[string][]string{},
+		held:            map[string]hold{},
+		preempted:       map[string]int{},
 		wake:            make(chan struct{}, 1),
 		stop:            make(chan struct{}),
 		done:            make(chan struct{}),
@@ -704,6 +732,9 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
 	switch {
 	case len(liveIDs) < desired && isParked:
 		// Not placed. Its failure is on every status the operator reads.
+	case len(liveIDs) < desired && r.isHeld(m.Name):
+		// It gave up room to a model of a higher class, which is placed
+		// first; taking it back now would undo the preemption.
 	case len(liveIDs) < desired:
 		if r.now().Before(blockedUntil) {
 			// Still in backoff. Rebuilding immediately would hot-loop on a
@@ -724,8 +755,12 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
 		r.mu.Lock()
 		rep, ok := r.replicas[id]
 		var expired bool
+		grace, preemptedFor := r.cfg.DrainGrace, ""
 		if ok {
-			expired = !rep.DrainingAt.IsZero() && r.now().Sub(rep.DrainingAt) > r.cfg.DrainGrace
+			if rep.PreemptedFor != "" {
+				grace, preemptedFor = r.cfg.PreemptGrace, rep.PreemptedFor
+			}
+			expired = !rep.DrainingAt.IsZero() && r.now().Sub(rep.DrainingAt) > grace
 		}
 		r.mu.Unlock()
 
@@ -733,11 +768,15 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
 		case !ok:
 		case replaced:
 			r.stopReplica(ctx, id)
+		case expired && preemptedFor != "":
+			r.log.Info("stopping a preempted replica", "replica", id,
+				"for", preemptedFor, "after", grace)
+			r.stopReplica(ctx, id)
 		case expired:
 			// The replacement never arrived — often because nowhere else can
 			// host it. Draining forever would leave the node undrainable.
 			r.log.Warn("stopping a drained replica without a replacement",
-				"replica", id, "after", r.cfg.DrainGrace)
+				"replica", id, "after", grace)
 			r.stopReplica(ctx, id)
 		}
 	}
@@ -751,12 +790,13 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
 // allocated, and starting several at once from the same inventory snapshot
 // would hand them the same GPUs.
 func (r *Reconciler) scaleUp(ctx context.Context, m catalog.Model, avoid []string) {
-	res := r.sched.Place(m.PlacementProfiles(), avoid)
+	profiles := r.orderedProfiles(m)
+	res := r.sched.Place(profiles, avoid)
 	if res.Placement == nil && len(avoid) > 0 {
 		// Spreading across failure domains is a preference, not a rule. If it
 		// is the only thing standing in the way, take the placement anyway:
 		// a second replica on one node still survives an engine crash.
-		res = r.sched.Place(m.PlacementProfiles(), nil)
+		res = r.sched.Place(profiles, nil)
 	}
 
 	r.mu.Lock()
@@ -775,9 +815,22 @@ func (r *Reconciler) scaleUp(ctx context.Context, m catalog.Model, avoid []strin
 		if m.Scaling.OnDemand() && r.makeRoomFor(ctx, m) {
 			return
 		}
+		if r.awaitingPreemption(m.Name) || r.preemptFor(ctx, m, avoid) {
+			return
+		}
 		r.notePlacementFailure(m.Name, res.Explain())
 		return
 	}
+	if why := r.overLimit(m, res.Placement); why != "" {
+		// Recorded as the placement's answer, so the model reads as not
+		// placeable, and says why, wherever that is shown.
+		r.mu.Lock()
+		r.lastExplain[m.Name] = placement.Result{Rejections: []placement.Rejection{
+			{Profile: res.Placement.Profile.Name, Reason: why}}}
+		r.mu.Unlock()
+		r.notePlacementFailure(m.Name, why)
+		return
+	}
 	if err := r.startReplica(ctx, m, res.Placement); err != nil {
 		r.log.Error("starting replica failed", "model", m.Name, "err", err)
 	}
@@ -825,7 +878,7 @@ func (r *Reconciler) startReplica(ctx context.Context, m catalog.Model, pl *plac
 		})
 		r.inv.Allocate(a.NodeID, inventory.Allocation{
 			ShardID: shardID, ReplicaID: id, GPUIndices: a.GPUIndices,
-			MemoryGB: prof.MinMemoryGB,
+			MemoryGB: prof.MinMemoryGB, CPUCores: cpuCores(prof),
 		})
 	}
 	rep.Endpoint = fmt.Sprintf("http://%s:%d", addrOf(leaderNode), basePort)
@@ -869,6 +922,7 @@ func (r *Reconciler) startReplica(ctx context.Context, m catalog.Model, pl *plac
 				HTTPPort:      s.HTTPPort,
 				Profile:       espec,
 				APIKey:        r.engineKey(id),
+				Threads:       cpuCores(prof),
 			},
 			StartupTimeout: prof.StartupTimeout,
 			Pool:           pl.Pool,
@@ -1108,16 +1162,31 @@ func addrOf(n inventory.Node) string {
 // claim on restart would place a second large model onto a node already full of
 // the first.
 func (r *Reconciler) profileMemory(model, profile string) int {
+	p, _ := r.profileOf(model, profile)
+	return p.MinMemoryGB
+}
+
+// profileOf is a replica's profile as the catalog holds it now.
+func (r *Reconciler) profileOf(model, profile string) (catalog.Profile, bool) {
 	m, ok := r.cat.Load().Resolve(model)
 	if !ok {
-		return 0
+		return catalog.Profile{}, false
 	}
 	for _, p := range m.Profiles {
 		if p.Name == profile {
-			return p.MinMemoryGB
+			return p, true
 		}
 	}
-	return 0
+	return catalog.Profile{}, false
+}
+
+// cpuCores is what a profile reserves of a node's CPUs: its cores when it
+// computes on them, nothing when a GPU does the work.
+func cpuCores(p catalog.Profile) int {
+	if !p.CPUOnly() {
+		return 0
+	}
+	return p.CPUCores
 }
 
 func (r *Reconciler) engineKey(replicaID string) string {

internal/control/reconciler/replica.go

@@ -156,6 +156,8 @@ type Replica struct {
 	// DrainingAt is when the replica stopped taking new work, so the wait for a
 	// replacement can be bounded.
 	DrainingAt time.Time
+	// PreemptedFor is the model this replica was stopped to make room for.
+	PreemptedFor string
 	// RetiringAt is set while the replica is being restarted: it keeps serving
 	// but no longer counts toward the model's replicas, so a replacement is
 	// placed beside it, and it is drained only once that replacement is ready.

internal/release/release.go

@@ -9,6 +9,7 @@ package release
 
 import (
 	"bufio"
+	"bytes"
 	"context"
 	"crypto/sha256"
 	"encoding/hex"
@@ -51,7 +52,7 @@ func Install(ctx context.Context, client *http.Client, base, version, goos, goar
 	asset := AssetName(version, goos, goarch)
 	root := strings.TrimSuffix(base, "/") + "/" + version + "/"
 
-	want, err := checksum(ctx, client, root+"checksums.txt", asset)
+	want, err := checksum(ctx, client, root, asset)
 	if err != nil {
 		return "", err
 	}
@@ -85,13 +86,35 @@ func Install(ctx context.Context, client *http.Client, base, version, goos, goar
 	return backup, nil
 }
 
-func checksum(ctx context.Context, client *http.Client, url, asset string) (string, error) {
-	body, err := get(ctx, client, url)
+// checksum reads the release's checksum for asset — after checking the
+// checksums' signature, when this build trusts a signing key. A release with
+// no signature is then refused rather than taken on its checksums: a missing
+// signature is exactly what replacing a release would look like.
+func checksum(ctx context.Context, client *http.Client, root, asset string) (string, error) {
+	body, err := get(ctx, client, root+"checksums.txt")
 	if err != nil {
 		return "", err
 	}
 	defer body.Close()
-	sc := bufio.NewScanner(io.LimitReader(body, 1<<20))
+	sums, err := io.ReadAll(io.LimitReader(body, 1<<20))
+	if err != nil {
+		return "", err
+	}
+	if key := TrustedKey(); key != "" {
+		sigBody, err := get(ctx, client, root+SignatureName)
+		if err != nil {
+			return "", fmt.Errorf("this build only installs signed releases, and %s could not be read: %w", SignatureName, err)
+		}
+		sig, err := io.ReadAll(io.LimitReader(sigBody, 4096))
+		sigBody.Close()
+		if err != nil {
+			return "", err
+		}
+		if err := Verify(key, sums, string(sig)); err != nil {
+			return "", err
+		}
+	}
+	sc := bufio.NewScanner(bytes.NewReader(sums))
 	for sc.Scan() {
 		f := strings.Fields(sc.Text())
 		if len(f) == 2 && f[1] == asset && len(f[0]) == 64 {

internal/release/sign.go

@@ -0,0 +1,72 @@
+package release
+
+import (
+	"crypto/ed25519"
+	"crypto/rand"
+	_ "embed"
+	"encoding/base64"
+	"errors"
+	"fmt"
+	"os"
+	"strings"
+)
+
+// trustedPub is the public key releases are signed with, base64, built into
+// every binary from the repository. Empty — as it is until a key is made —
+// trusts a release on its checksums alone, as every release before this did.
+//
+// Built in rather than fetched, because the point of a signature is that the
+// server the release comes from cannot vouch for it: a machine that already
+// runs llmgw checks the next release against the key the release it runs was
+// built with, so replacing both the binaries and the checksums on the server
+// is no longer enough.
+//
+//go:embed trusted.pub
+var trustedPub string
+
+// TrustedKey is the key releases are verified with: LLMGW_RELEASE_KEY when set
+// (for rotating a key, or testing), the built-in one otherwise; empty is none.
+func TrustedKey() string {
+	if k := strings.TrimSpace(os.Getenv("LLMGW_RELEASE_KEY")); k != "" {
+		return k
+	}
+	return strings.TrimSpace(trustedPub)
+}
+
+// SignatureName is the file a release's checksums are signed in.
+const SignatureName = "checksums.txt.sig"
+
+// GenerateKey makes a signing key pair, both base64: the private key is the
+// 32-byte seed, which is what goes into the CI secret.
+func GenerateKey() (public, private string, err error) {
+	pub, priv, err := ed25519.GenerateKey(rand.Reader)
+	if err != nil {
+		return "", "", err
+	}
+	return base64.StdEncoding.EncodeToString(pub), base64.StdEncoding.EncodeToString(priv.Seed()), nil
+}
+
+// Sign signs data with a base64 seed, returning the signature in base64.
+func Sign(privateSeed string, data []byte) (string, error) {
+	seed, err := base64.StdEncoding.DecodeString(strings.TrimSpace(privateSeed))
+	if err != nil || len(seed) != ed25519.SeedSize {
+		return "", errors.New("the signing key is not a base64 ed25519 seed")
+	}
+	return base64.StdEncoding.EncodeToString(ed25519.Sign(ed25519.NewKeyFromSeed(seed), data)), nil
+}
+
+// Verify checks a base64 signature of data against a base64 public key.
+func Verify(public string, data []byte, signature string) error {
+	pub, err := base64.StdEncoding.DecodeString(strings.TrimSpace(public))
+	if err != nil || len(pub) != ed25519.PublicKeySize {
+		return errors.New("the trusted release key is not a base64 ed25519 public key")
+	}
+	sig, err := base64.StdEncoding.DecodeString(strings.TrimSpace(signature))
+	if err != nil {
+		return fmt.Errorf("the release signature is not base64: %w", err)
+	}
+	if !ed25519.Verify(pub, data, sig) {
+		return errors.New("the release's checksums are not signed by the trusted key")
+	}
+	return nil
+}

internal/release/sign_test.go

@@ -0,0 +1,99 @@
+package release
+
+import (
+	"crypto/sha256"
+	"encoding/hex"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+)
+
+// signedServer publishes a release whose checksums are signed with seed,
+// or not signed at all when seed is empty; tamper rewrites the checksums
+// after signing.
+func signedServer(t *testing.T, seed string, body []byte, tamper bool) (*httptest.Server, string) {
+	t.Helper()
+	asset := AssetName("v2.0.0", "linux", "amd64")
+	sum := sha256.Sum256(body)
+	sums := hex.EncodeToString(sum[:]) + "  " + asset + "\n"
+	sig := ""
+	if seed != "" {
+		var err error
+		if sig, err = Sign(seed, []byte(sums)); err != nil {
+			t.Fatal(err)
+		}
+	}
+	if tamper {
+		sums += strings.Repeat("0", 64) + "  extra-file\n"
+	}
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		switch r.URL.Path {
+		case "/v2.0.0/checksums.txt":
+			_, _ = w.Write([]byte(sums))
+		case "/v2.0.0/" + SignatureName:
+			if sig == "" {
+				http.NotFound(w, r)
+				return
+			}
+			_, _ = w.Write([]byte(sig + "\n"))
+		case "/v2.0.0/" + asset:
+			_, _ = w.Write(body)
+		default:
+			http.NotFound(w, r)
+		}
+	}))
+	t.Cleanup(srv.Close)
+	return srv, asset
+}
+
+func TestABuildThatTrustsAKeyInstallsOnlyReleasesSignedWithIt(t *testing.T) {
+	pub, priv, err := GenerateKey()
+	if err != nil {
+		t.Fatal(err)
+	}
+	_, otherPriv, _ := GenerateKey()
+	t.Setenv("LLMGW_RELEASE_KEY", pub)
+	install := func(srv *httptest.Server) error {
+		target := filepath.Join(t.TempDir(), "llmgw")
+		_ = os.WriteFile(target, []byte("old"), 0o755)
+		_, err := Install(t.Context(), srv.Client(), srv.URL, "v2.0.0", "linux", "amd64", target)
+		return err
+	}
+	body := []byte("new binary")
+
+	good, _ := signedServer(t, priv, body, false)
+	if err := install(good); err != nil {
+		t.Fatalf("a release signed with the trusted key was refused: %v", err)
+	}
+	for name, srv := range map[string]*httptest.Server{
+		"unsigned":            func() *httptest.Server { s, _ := signedServer(t, "", body, false); return s }(),
+		"signed by another":   func() *httptest.Server { s, _ := signedServer(t, otherPriv, body, false); return s }(),
+		"checksums rewritten": func() *httptest.Server { s, _ := signedServer(t, priv, body, true); return s }(),
+	} {
+		if err := install(srv); err == nil {
+			t.Errorf("%s: installed", name)
+		}
+	}
+
+	// A build that trusts no key installs on checksums, as before.
+	t.Setenv("LLMGW_RELEASE_KEY", "")
+	if TrustedKey() != "" {
+		t.Skip("this build embeds a trusted key; the unsigned path cannot be tested here")
+	}
+	unsigned, _ := signedServer(t, "", body, false)
+	if err := install(unsigned); err != nil {
+		t.Errorf("without a trusted key an unsigned release was refused: %v", err)
+	}
+}
+
+func TestAKeyThatIsNotOneIsSaid(t *testing.T) {
+	if _, err := Sign("not-a-key", []byte("x")); err == nil {
+		t.Error("signed with a malformed key")
+	}
+	if err := Verify("bm9wZQ==", []byte("x"), "c2ln"); err == nil || !strings.Contains(err.Error(), "public key") {
+		t.Errorf("a malformed trusted key: %v", err)
+	}
+}

internal/router/admission/controller.go

@@ -34,6 +34,12 @@ type Config struct {
 	// QueueCapacityPerBand bounds each band's queue. Rejecting immediately
 	// beats accumulating silently.
 	QueueCapacityPerBand int `koanf:"queue_capacity_per_band"`
+	// QueueCapacityTotal bounds everything queued, across bands. When it is
+	// reached, a request is not refused while a lower band has requests
+	// queued: the newest of the lowest band is turned away instead
+	// (displacement), since the oldest have waited longest and are nearest
+	// their TTL. Zero takes half of every band's capacity together.
+	QueueCapacityTotal int `koanf:"queue_capacity_total"`
 	// Interval is how often the dispatcher re-checks saturation. Releases also
 	// wake it, so this is only the backstop for saturation falling on its own.
 	Interval time.Duration `koanf:"interval"`
@@ -94,6 +100,9 @@ func (c *Config) normalize() error {
 	if c.QueueCapacityPerBand <= 0 {
 		c.QueueCapacityPerBand = 2000
 	}
+	if c.QueueCapacityTotal <= 0 {
+		c.QueueCapacityTotal = max(c.QueueCapacityPerBand*len(c.Bands)/2, 1)
+	}
 	if c.Interval <= 0 {
 		c.Interval = 10 * time.Millisecond
 	}
@@ -123,6 +132,10 @@ var (
 	ErrQueueTimeout = errors.New("admission queue timeout")
 	// ErrClosed means the controller is shutting down.
 	ErrClosed = errors.New("admission controller closed")
+	// ErrDisplaced means a queued request was turned away to make room for
+	// one of a higher band. Worth retrying: the queue is full now, not for
+	// good.
+	ErrDisplaced = errors.New("displaced from the admission queue by higher-priority requests")
 )
 
 // Stats is a snapshot of one band's queue, for metrics.
@@ -259,6 +272,12 @@ func (c *Controller) Acquire(ctx context.Context, r Request) (*Lease, error) {
 		c.mu.Unlock()
 		return nil, fmt.Errorf("%w: band %q", ErrQueueFull, band.Name)
 	}
+	if c.queuedLocked() >= c.cfg.QueueCapacityTotal && !c.displaceBelowLocked(idx) {
+		c.rejected[band.Name]++
+		c.mu.Unlock()
+		return nil, fmt.Errorf("%w: %d queued in all, and nothing of a lower band to make room",
+			ErrQueueFull, c.cfg.QueueCapacityTotal)
+	}
 
 	w := &waiter{
 		tenant:   tenantKey(r),
@@ -279,6 +298,10 @@ func (c *Controller) Acquire(ctx context.Context, r Request) (*Lease, error) {
 			c.mu.Unlock()
 			return nil, ErrClosed
 		}
+		if w.err != nil {
+			c.mu.Unlock()
+			return nil, w.err
+		}
 		c.waitTotal[band.Name] += time.Since(w.enqueued)
 		c.mu.Unlock()
 		return &Lease{c: c, band: band.Name, model: r.Model}, nil
@@ -297,8 +320,9 @@ func (c *Controller) Acquire(ctx context.Context, r Request) (*Lease, error) {
 func (c *Controller) remove(idx int, w *waiter, reason string) {
 	c.mu.Lock()
 	defer c.mu.Unlock()
-	if w.admitted {
-		// It won the race; the caller will see ready closed next time round.
+	if w.admitted || w.err != nil {
+		// It won the race, or was displaced; the caller will see ready
+		// closed next time round.
 		return
 	}
 	c.queues[idx].remove(w)
@@ -410,3 +434,31 @@ func tenantKey(r Request) string {
 	}
 	return "anonymous"
 }
+
+// queuedLocked is how many requests wait in every band. Called with mu held.
+func (c *Controller) queuedLocked() int {
+	n := 0
+	for _, q := range c.queues {
+		n += q.len()
+	}
+	return n
+}
+
+// displaceBelowLocked turns away the newest request of the lowest band below
+// idx that has any queued, and says whether there was one. Called with mu held.
+func (c *Controller) displaceBelowLocked(idx int) bool {
+	// Bands are sorted highest priority first, so lower bands have larger
+	// indices, and the lowest is last.
+	for j := len(c.queues) - 1; j > idx; j-- {
+		w := c.queues[j].newest()
+		if w == nil {
+			continue
+		}
+		c.queues[j].remove(w)
+		w.err = fmt.Errorf("%w (band %q)", ErrDisplaced, c.cfg.Bands[j].Name)
+		close(w.ready)
+		c.rejected[c.cfg.Bands[j].Name+"/displaced"]++
+		return true
+	}
+	return false
+}

internal/router/admission/displace_test.go

@@ -0,0 +1,89 @@
+package admission
+
+import (
+	"context"
+	"errors"
+	"testing"
+	"time"
+)
+
+// queueUntil starts an Acquire in the background and waits until the band
+// holds depth requests, returning where its result will arrive.
+func queueUntil(t *testing.T, c *Controller, band string, bandIdx, depth int) <-chan error {
+	t.Helper()
+	out := make(chan error, 1)
+	go func() {
+		l, err := c.Acquire(context.Background(), Request{Model: "m", Band: band})
+		if err == nil {
+			l.Release()
+		}
+		out <- err
+	}()
+	deadline := time.Now().Add(2 * time.Second)
+	for c.Depths()[bandIdx].Depth < depth && time.Now().Before(deadline) {
+		time.Sleep(time.Millisecond)
+	}
+	if got := c.Depths()[bandIdx].Depth; got != depth {
+		t.Fatalf("band %s holds %d, expected %d", band, got, depth)
+	}
+	return out
+}
+
+func TestAHigherBandDisplacesTheNewestOfTheLowest(t *testing.T) {
+	s := &fixedSat{}
+	s.set(5) // past every ceiling: everything queues
+	cfg := testConfig()
+	cfg.QueueCapacityTotal = 3
+	c := newController(t, cfg, s.fn)
+	// Bands, highest first: platinum 0, gold 1, silver 2, bronze 3.
+	oldest := queueUntil(t, c, "bronze", 3, 1)
+	time.Sleep(2 * time.Millisecond)
+	silver := queueUntil(t, c, "silver", 2, 1)
+	time.Sleep(2 * time.Millisecond)
+	newest := queueUntil(t, c, "bronze", 3, 2)
+
+	// Full at three. Gold arrives: the newest bronze goes, not the oldest,
+	// and not silver, which outranks bronze.
+	gold := queueUntil(t, c, "gold", 1, 1)
+	select {
+	case err := <-newest:
+		if !errors.Is(err, ErrDisplaced) {
+			t.Fatalf("the newest bronze got %v", err)
+		}
+	case <-time.After(time.Second):
+		t.Fatal("nothing was displaced")
+	}
+	if d := c.Depths(); d[3].Depth != 1 || d[2].Depth != 1 || d[1].Depth != 1 {
+		t.Errorf("depths %+v: one bronze, the silver and the gold should remain", d)
+	}
+
+	// Bronze cannot displace anything: nothing is below it.
+	_, err := c.Acquire(t.Context(), Request{Model: "m", Band: "bronze"})
+	if !errors.Is(err, ErrQueueFull) {
+		t.Errorf("a bronze request at a full queue got %v", err)
+	}
+	// Another gold takes the remaining bronze, then silver.
+	queueUntil(t, c, "gold", 1, 2)
+	if err := <-oldest; !errors.Is(err, ErrDisplaced) {
+		t.Errorf("the last bronze got %v", err)
+	}
+	queueUntil(t, c, "gold", 1, 3)
+	if err := <-silver; !errors.Is(err, ErrDisplaced) {
+		t.Errorf("silver got %v", err)
+	}
+	// All gold now: a fourth has nobody lower to displace.
+	if _, err := c.Acquire(t.Context(), Request{Model: "m", Band: "gold"}); !errors.Is(err, ErrQueueFull) {
+		t.Errorf("a gold request with only gold queued got %v", err)
+	}
+
+	// And they are served once there is room.
+	s.set(0)
+	select {
+	case err := <-gold:
+		if err != nil {
+			t.Errorf("the first gold was not served: %v", err)
+		}
+	case <-time.After(2 * time.Second):
+		t.Error("the queue did not drain")
+	}
+}

internal/router/admission/queue.go

@@ -12,6 +12,9 @@ type waiter struct {
 	// admitted guards the race between the dispatcher admitting a waiter and
 	// the waiter giving up on its own.
 	admitted bool
+	// err is why the waiter was turned away while queued — displaced by a
+	// higher band — set before ready is closed.
+	err error
 }
 
 // bandQueue holds one band's waiters, grouped by tenant.
@@ -166,3 +169,17 @@ func (q *bandQueue) dropExpired(now time.Time, onDrop func()) {
 		q.byTenant[tenant] = kept
 	}
 }
+
+// newest is the most recently queued waiter, across every tenant: the one
+// displacement takes, since it has invested the least waiting.
+func (q *bandQueue) newest() *waiter {
+	var out *waiter
+	for _, ws := range q.byTenant {
+		for _, w := range ws {
+			if out == nil || w.enqueued.After(out.enqueued) {
+				out = w
+			}
+		}
+	}
+	return out
+}

internal/router/finish_test.go

@@ -88,6 +88,9 @@ func TestARefusalBeforeDispatchSaysWhetherWaitingHelps(t *testing.T) {
 		{"queued too long", func(w http.ResponseWriter) {
 			rt.rejectAdmission(w, &parsedRequest{Model: "m"}, admission.ErrQueueTimeout, time.Now())
 		}, 504, ""},
+		{"displaced for a higher band", func(w http.ResponseWriter) {
+			rt.rejectAdmission(w, &parsedRequest{Model: "m"}, admission.ErrDisplaced, time.Now())
+		}, 503, "5"},
 		{"shutting down", func(w http.ResponseWriter) {
 			rt.rejectAdmission(w, &parsedRequest{Model: "m"}, admission.ErrClosed, time.Now())
 		}, 503, ""},

internal/router/handlers.go

@@ -60,9 +60,15 @@ func (r *Router) rejectAdmission(w http.ResponseWriter, req *parsedRequest, err
 		return
 	case errors.Is(err, admission.ErrQueueTimeout):
 		status = http.StatusGatewayTimeout
+	case errors.Is(err, admission.ErrDisplaced):
+		// Turned away for higher-priority work, not for good: said so the
+		// client backs off and comes back rather than giving up.
+		w.Header().Set("Retry-After", "5")
 	}
 	reason := "unavailable"
 	switch {
+	case errors.Is(err, admission.ErrDisplaced):
+		reason = "displaced"
 	case errors.Is(err, admission.ErrQueueFull):
 		reason = "queue_full"
 	case errors.Is(err, admission.ErrQueueTimeout):
@@ -185,6 +191,11 @@ func (r *Router) Handler() http.Handler {
 			pages.RegisterAdminUsers(mux, r.store)
 			// Quotas are written on the same tree the accounts page builds, so
 			// they are mounted together.
+			// Capacity quotas are the control plane's, reached through the
+			// same admin API the CLI uses.
+			if c, ok := r.clusterView.(web.CapacityAdmin); ok {
+				pages.SetCapacity(c)
+			}
 			pages.RegisterAdminQuotas(mux, r.store, r.acct)
 			// And the report of what that tree actually used.
 			pages.RegisterAdminUsage(mux, r.store, r.acct)

internal/router/router.go

@@ -347,6 +347,7 @@ func admissionConfig(rc config.Router) admission.Config {
 		},
 		QueueTTL:             rc.Admission.QueueTTL,
 		QueueCapacityPerBand: rc.Admission.QueueCapacityPerBand,
+		QueueCapacityTotal:   rc.Admission.QueueCapacityTotal,
 		Interval:             rc.Admission.Interval,
 	}
 }

internal/store/copy.go

@@ -29,6 +29,9 @@ var copyOrder = []string{
 	"key_claims",
 	"sessions",
 	"limits",
+	// Beside limits, and like them keyed by scope with no foreign key: a
+	// quota on a scope that is later removed is inert, not an error.
+	"capacity_quotas",
 	"usage",
 	"models",
 	// No foreign key to models, deliberately: a measurement outlives the

internal/store/migrations/postgres/0013_capacity_quotas.sql

@@ -0,0 +1,19 @@
+-- Capacity a part of the tenancy tree is guaranteed, and the most it may hold.
+--
+-- Beside limits rather than in it: limits are counted on the request path, per
+-- key, in requests and tokens; these are counted by the reconciler, per model,
+-- in the GPUs, cores and memory its replicas hold. A NULL is a resource this
+-- row does not govern: guaranteed nothing, or limited by nothing.
+CREATE TABLE capacity_quotas (
+  scope_type      TEXT NOT NULL,
+  scope_id        TEXT NOT NULL,
+  gpus            INTEGER,
+  cpu_cores       INTEGER,
+  memory_gb       INTEGER,
+  limit_gpus      INTEGER,
+  limit_cpu_cores INTEGER,
+  limit_memory_gb INTEGER,
+  updated_at      BIGINT NOT NULL,
+  updated_by      TEXT NOT NULL,
+  PRIMARY KEY (scope_type, scope_id)
+);

internal/store/migrations/sqlite/0013_capacity_quotas.sql

@@ -0,0 +1,19 @@
+-- Capacity a part of the tenancy tree is guaranteed, and the most it may hold.
+--
+-- Beside limits rather than in it: limits are counted on the request path, per
+-- key, in requests and tokens; these are counted by the reconciler, per model,
+-- in the GPUs, cores and memory its replicas hold. A NULL is a resource this
+-- row does not govern: guaranteed nothing, or limited by nothing.
+CREATE TABLE capacity_quotas (
+  scope_type      TEXT NOT NULL,
+  scope_id        TEXT NOT NULL,
+  gpus            INTEGER,
+  cpu_cores       INTEGER,
+  memory_gb       INTEGER,
+  limit_gpus      INTEGER,
+  limit_cpu_cores INTEGER,
+  limit_memory_gb INTEGER,
+  updated_at      BIGINT NOT NULL,
+  updated_by      TEXT NOT NULL,
+  PRIMARY KEY (scope_type, scope_id)
+);

internal/store/quotas.go

@@ -0,0 +1,127 @@
+package store
+
+import (
+	"context"
+	"database/sql"
+	"fmt"
+	"time"
+)
+
+// Resources are amounts of the three things a replica holds of a cluster.
+// Nil is not governed: nothing guaranteed, or nothing limited.
+type Resources struct {
+	GPUs     *int `json:"gpus,omitempty"`
+	CPUCores *int `json:"cpu_cores,omitempty"`
+	MemoryGB *int `json:"memory_gb,omitempty"`
+}
+
+// Any reports whether any resource is governed.
+func (r Resources) Any() bool { return r.GPUs != nil || r.CPUCores != nil || r.MemoryGB != nil }
+
+// CapacityQuota is what one part of the tenancy tree is guaranteed of the
+// cluster, and the most it may hold.
+type CapacityQuota struct {
+	Scope   Scope  `json:"scope"`
+	ScopeID string `json:"scope_id"`
+	// Guarantee is held for the scope: while it holds less, what others
+	// borrowed beyond their own guarantees is taken back for it.
+	Guarantee Resources `json:"guarantee"`
+	// Limit is the most it may hold, borrowing included.
+	Limit     Resources `json:"limit"`
+	UpdatedAt time.Time `json:"updated_at"`
+	UpdatedBy string    `json:"updated_by"`
+}
+
+// PutCapacityQuota sets a scope's quota, replacing what was there.
+func (s *DB) PutCapacityQuota(ctx context.Context, q CapacityQuota) error {
+	if !validMembershipScope(q.Scope) || q.ScopeID == "" {
+		return fmt.Errorf("a capacity quota belongs to an organization, team or project")
+	}
+	for _, v := range []*int{q.Guarantee.GPUs, q.Guarantee.CPUCores, q.Guarantee.MemoryGB,
+		q.Limit.GPUs, q.Limit.CPUCores, q.Limit.MemoryGB} {
+		if v != nil && *v < 0 {
+			return fmt.Errorf("a quota cannot be negative")
+		}
+	}
+	if q.UpdatedAt.IsZero() {
+		q.UpdatedAt = time.Now()
+	}
+	_, err := s.db.ExecContext(ctx,
+		`INSERT INTO capacity_quotas (scope_type, scope_id, gpus, cpu_cores, memory_gb,
+		   limit_gpus, limit_cpu_cores, limit_memory_gb, updated_at, updated_by)
+		 VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
+		 ON CONFLICT (scope_type, scope_id) DO UPDATE SET
+		   gpus = excluded.gpus, cpu_cores = excluded.cpu_cores, memory_gb = excluded.memory_gb,
+		   limit_gpus = excluded.limit_gpus, limit_cpu_cores = excluded.limit_cpu_cores,
+		   limit_memory_gb = excluded.limit_memory_gb,
+		   updated_at = excluded.updated_at, updated_by = excluded.updated_by`,
+		string(q.Scope), q.ScopeID,
+		nullInt(q.Guarantee.GPUs), nullInt(q.Guarantee.CPUCores), nullInt(q.Guarantee.MemoryGB),
+		nullInt(q.Limit.GPUs), nullInt(q.Limit.CPUCores), nullInt(q.Limit.MemoryGB),
+		q.UpdatedAt.UnixMilli(), q.UpdatedBy)
+	return err
+}
+
+// CapacityQuotas lists every quota.
+func (s *DB) CapacityQuotas(ctx context.Context) ([]CapacityQuota, error) {
+	return queryAll(ctx, s.db,
+		`SELECT scope_type, scope_id, gpus, cpu_cores, memory_gb,
+		   limit_gpus, limit_cpu_cores, limit_memory_gb, updated_at, updated_by
+		 FROM capacity_quotas ORDER BY scope_type, scope_id`, nil,
+		func(rows *sql.Rows) (CapacityQuota, error) {
+			var q CapacityQuota
+			var scope string
+			var g [6]sql.NullInt64
+			var at int64
+			err := rows.Scan(&scope, &q.ScopeID, &g[0], &g[1], &g[2], &g[3], &g[4], &g[5], &at, &q.UpdatedBy)
+			q.Scope = Scope(scope)
+			q.Guarantee = Resources{GPUs: intOf(g[0]), CPUCores: intOf(g[1]), MemoryGB: intOf(g[2])}
+			q.Limit = Resources{GPUs: intOf(g[3]), CPUCores: intOf(g[4]), MemoryGB: intOf(g[5])}
+			q.UpdatedAt = time.UnixMilli(at)
+			return q, err
+		})
+}
+
+// DeleteCapacityQuota removes a scope's quota.
+func (s *DB) DeleteCapacityQuota(ctx context.Context, scope Scope, id string) error {
+	_, err := s.db.ExecContext(ctx,
+		`DELETE FROM capacity_quotas WHERE scope_type = ? AND scope_id = ?`, string(scope), id)
+	return err
+}
+
+// ScopeParents maps every team and project to the scopes above it, nearest
+// first: a project to its team and organization, a team to its organization.
+// Keys and values are written scope:id.
+func (s *DB) ScopeParents(ctx context.Context) (map[string][]string, error) {
+	out := map[string][]string{}
+	teams, err := s.AllTeams(ctx)
+	if err != nil {
+		return nil, err
+	}
+	orgOf := map[string]string{}
+	for _, t := range teams {
+		orgOf[t.ID] = t.OrgID
+		out["team:"+t.ID] = []string{"org:" + t.OrgID}
+	}
+	type pt struct{ project, team string }
+	rows, err := queryAll(ctx, s.db, `SELECT id, team_id FROM projects`, nil,
+		func(r *sql.Rows) (pt, error) {
+			var p pt
+			return p, r.Scan(&p.project, &p.team)
+		})
+	if err != nil {
+		return nil, err
+	}
+	for _, p := range rows {
+		out["project:"+p.project] = []string{"team:" + p.team, "org:" + orgOf[p.team]}
+	}
+	return out, nil
+}
+
+func intOf(n sql.NullInt64) *int {
+	if !n.Valid {
+		return nil
+	}
+	v := int(n.Int64)
+	return &v
+}

internal/store/quotas_test.go

@@ -0,0 +1,63 @@
+package store
+
+import "testing"
+
+func ip(v int) *int { return &v }
+
+func TestAQuotaKeepsWhatItGovernsAndNothingElse(t *testing.T) {
+	s := tenancyStore(t)
+	org, _ := s.CreateOrg(t.Context(), "Acme")
+	q := CapacityQuota{Scope: ScopeOrg, ScopeID: org, UpdatedBy: "ops",
+		Guarantee: Resources{GPUs: ip(8)}, Limit: Resources{GPUs: ip(16), CPUCores: ip(0)}}
+	if err := s.PutCapacityQuota(t.Context(), q); err != nil {
+		t.Fatal(err)
+	}
+	all, err := s.CapacityQuotas(t.Context())
+	if err != nil || len(all) != 1 {
+		t.Fatalf("%v %+v", err, all)
+	}
+	got := all[0]
+	if *got.Guarantee.GPUs != 8 || got.Guarantee.CPUCores != nil || got.Guarantee.MemoryGB != nil {
+		t.Errorf("guarantee %+v: only GPUs were set", got.Guarantee)
+	}
+	// Zero is a limit of nothing, not an absent one.
+	if *got.Limit.GPUs != 16 || got.Limit.CPUCores == nil || *got.Limit.CPUCores != 0 {
+		t.Errorf("limit %+v", got.Limit)
+	}
+	q.Guarantee = Resources{MemoryGB: ip(64)}
+	if err := s.PutCapacityQuota(t.Context(), q); err != nil {
+		t.Fatal(err)
+	}
+	all, _ = s.CapacityQuotas(t.Context())
+	if len(all) != 1 || all[0].Guarantee.GPUs != nil || *all[0].Guarantee.MemoryGB != 64 {
+		t.Errorf("a second put should replace the first: %+v", all)
+	}
+	if err := s.DeleteCapacityQuota(t.Context(), ScopeOrg, org); err != nil {
+		t.Fatal(err)
+	}
+	if all, _ = s.CapacityQuotas(t.Context()); len(all) != 0 {
+		t.Error("the quota survived its deletion")
+	}
+	if err := s.PutCapacityQuota(t.Context(), CapacityQuota{Scope: ScopeOrg, ScopeID: org,
+		Guarantee: Resources{GPUs: ip(-1)}}); err == nil {
+		t.Error("a negative quota was stored")
+	}
+}
+
+func TestEveryScopeKnowsWhatIsAboveIt(t *testing.T) {
+	s := tenancyStore(t)
+	org, _ := s.CreateOrg(t.Context(), "Acme")
+	team, _ := s.CreateTeam(t.Context(), org, "ml")
+	proj, _ := s.CreateProject(t.Context(), team, "training")
+	p, err := s.ScopeParents(t.Context())
+	if err != nil {
+		t.Fatal(err)
+	}
+	want := []string{"team:" + team, "org:" + org}
+	if got := p["project:"+proj]; len(got) != 2 || got[0] != want[0] || got[1] != want[1] {
+		t.Errorf("project's parents %v, expected %v", got, want)
+	}
+	if got := p["team:"+team]; len(got) != 1 || got[0] != "org:"+org {
+		t.Errorf("team's parents %v", got)
+	}
+}

internal/web/account.go

@@ -61,6 +61,8 @@ type AccountPages struct {
 	hasMirror bool
 	// hasQuotas is set when /admin/quotas is registered.
 	hasQuotas bool
+	// capacity sets capacity quotas, from the quotas page; nil leaves them off.
+	capacity CapacityAdmin
 	// personalOff refuses personal keys; see SetPersonalOff.
 	personalOff bool
 	// personalDefaults is what personal use falls under by default.

internal/web/admin_capacity.go

@@ -0,0 +1,138 @@
+package web
+
+import (
+	"context"
+	"fmt"
+	"net/http"
+	"strconv"
+	"strings"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// CapacityAdmin sets what parts of the tenancy tree are guaranteed of the
+// cluster and may hold. The admin API implements it; the page and the CLI go
+// through the same checks.
+type CapacityAdmin interface {
+	QuotaViews(ctx context.Context) ([]adminapi.QuotaView, error)
+	SetQuota(ctx context.Context, req adminapi.QuotaRequest, by string) (string, error)
+	ClearQuota(ctx context.Context, scope string) (string, error)
+	Scopes(ctx context.Context) ([]adminapi.ScopeName, error)
+}
+
+// SetCapacity gives the quotas page capacity quotas to show and set. Without
+// it the section is left off.
+func (p *AccountPages) SetCapacity(c CapacityAdmin) { p.capacity = c }
+
+// capacityRow is one scope's capacity quota as the page shows it.
+type capacityRow struct {
+	adminapi.QuotaView
+	GPUs, Cores, Memory string
+}
+
+// capacityForm is what the form starts with: a scope's quota to change, or
+// nothing for a new one.
+type capacityForm struct {
+	Scope               string
+	GPUs, Cores, Memory string
+	LGPUs, LCores, LMem string
+}
+
+func (p *AccountPages) capacitySection(ctx context.Context, d *adminQuotasData, editing string) {
+	if p.capacity == nil {
+		return
+	}
+	d.HasCapacity = true
+	views, err := p.capacity.QuotaViews(ctx)
+	if err != nil {
+		d.Error = err.Error()
+		return
+	}
+	for _, v := range views {
+		d.Capacity = append(d.Capacity, capacityRow{QuotaView: v,
+			GPUs:   capCell(v.Holds.GPUs, v.Guarantee.GPUs, v.Limit.GPUs),
+			Cores:  capCell(v.Holds.CPUCores, v.Guarantee.CPUCores, v.Limit.CPUCores),
+			Memory: capCell(v.Holds.MemoryGB, v.Guarantee.MemoryGB, v.Limit.MemoryGB)})
+		if v.Scope == editing {
+			d.CapForm = capacityForm{Scope: v.Scope,
+				GPUs: intStr(v.Guarantee.GPUs), Cores: intStr(v.Guarantee.CPUCores), Memory: intStr(v.Guarantee.MemoryGB),
+				LGPUs: intStr(v.Limit.GPUs), LCores: intStr(v.Limit.CPUCores), LMem: intStr(v.Limit.MemoryGB)}
+		}
+	}
+	if d.Scopes, err = p.capacity.Scopes(ctx); err != nil {
+		d.Error = err.Error()
+	}
+}
+
+// capCell is "6 of 8 (at most 16)": held, guaranteed, limit.
+func capCell(holds int, g, l *int) string {
+	if g == nil && l == nil {
+		return ""
+	}
+	out := strconv.Itoa(holds)
+	if g != nil {
+		out += " of " + strconv.Itoa(*g)
+	}
+	if l != nil {
+		out += fmt.Sprintf(" (at most %d)", *l)
+	}
+	return out
+}
+
+func intStr(v *int) string {
+	if v == nil {
+		return ""
+	}
+	return strconv.Itoa(*v)
+}
+
+// applyCapacity sets or clears one scope's capacity quota from the form.
+func (p *AccountPages) applyCapacity(r *http.Request, admin store.User) (string, error) {
+	scope := strings.TrimSpace(r.FormValue("scope"))
+	if scope == "" {
+		return "", fmt.Errorf("choose where the quota applies")
+	}
+	if r.FormValue("clear") != "" {
+		key, err := p.capacity.ClearQuota(r.Context(), scope)
+		if err != nil {
+			return "", err
+		}
+		return "capacity quota removed from " + key, nil
+	}
+	field := func(name, what string) (*int, error) {
+		v := strings.TrimSpace(r.FormValue(name))
+		if v == "" {
+			return nil, nil
+		}
+		n, err := strconv.Atoi(v)
+		if err != nil || n < 0 {
+			return nil, fmt.Errorf("%s must be a whole number, or blank for none", what)
+		}
+		return &n, nil
+	}
+	var req adminapi.QuotaRequest
+	req.Scope = scope
+	var err error
+	for _, f := range []struct {
+		dst  **int
+		name string
+		what string
+	}{
+		{&req.Guarantee.GPUs, "gpus", "guaranteed GPUs"},
+		{&req.Guarantee.CPUCores, "cores", "guaranteed cores"},
+		{&req.Guarantee.MemoryGB, "memory", "guaranteed memory"},
+		{&req.Limit.GPUs, "limit_gpus", "the GPU limit"},
+		{&req.Limit.CPUCores, "limit_cores", "the core limit"},
+		{&req.Limit.MemoryGB, "limit_memory", "the memory limit"},
+	} {
+		if *f.dst, err = field(f.name, f.what); err != nil {
+			return "", err
+		}
+	}
+	key, err := p.capacity.SetQuota(r.Context(), req, admin.Email)
+	if err != nil {
+		return "", err
+	}
+	return "capacity quota set on " + key + "; it takes effect within half a minute", nil
+}

internal/web/admin_capacity_test.go

@@ -0,0 +1,88 @@
+package web
+
+import (
+	"context"
+	"errors"
+	"log/slog"
+	"net/http"
+	"net/url"
+	"strings"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+type fakeCapacity struct {
+	views   []adminapi.QuotaView
+	set     []adminapi.QuotaRequest
+	cleared []string
+}
+
+func (f *fakeCapacity) QuotaViews(context.Context) ([]adminapi.QuotaView, error) { return f.views, nil }
+func (f *fakeCapacity) SetQuota(_ context.Context, r adminapi.QuotaRequest, _ string) (string, error) {
+	if r.Scope == "team:nope" {
+		return "", errors.New(`there is no team "nope"`)
+	}
+	f.set = append(f.set, r)
+	return r.Scope, nil
+}
+func (f *fakeCapacity) ClearQuota(_ context.Context, s string) (string, error) {
+	f.cleared = append(f.cleared, s)
+	return s, nil
+}
+func (f *fakeCapacity) Scopes(context.Context) ([]adminapi.ScopeName, error) {
+	return []adminapi.ScopeName{{Key: "org:o1", Name: "Acme"}, {Key: "team:t1", Name: "Acme / ml"}}, nil
+}
+
+func TestCapacityQuotasAreShownAndSetFromTheQuotasPage(t *testing.T) {
+	eight, sixteen := 8, 16
+	fc := &fakeCapacity{views: []adminapi.QuotaView{{Scope: "team:t1", Name: "Acme / ml",
+		Guarantee: store.Resources{GPUs: &eight}, Limit: store.Resources{GPUs: &sixteen},
+		Holds: quota.Amount{GPUs: 6}, UpdatedBy: "ops@acme"}}}
+	a := fakeAccounts{user: store.User{ID: "u1", Email: "a@example.com", IsAdmin: true}}
+	p := NewAccountPages(a, nil, nil, "v0", slog.New(slog.DiscardHandler))
+	p.SetCapacity(fc)
+	mux := http.NewServeMux()
+	p.RegisterAdminQuotas(mux, &fakeQuotas{}, pricedYes{})
+
+	body := mirrorGet(t, mux, "/admin/quotas").Body.String()
+	for _, want := range []string{`id="capacity"`, "Acme / ml", "6 of 8 (at most 16)", "ops@acme",
+		`<option value="team:t1">Acme / ml</option>`} {
+		if !strings.Contains(body, want) {
+			t.Errorf("the page lacks %q", want)
+		}
+	}
+	// "change" fills the form with the scope's figures.
+	edit := mirrorGet(t, mux, "/admin/quotas?cap=team:t1").Body.String()
+	if !strings.Contains(edit, `<option value="team:t1" selected>`) || !strings.Contains(edit, `name="limit_gpus" value="16"`) {
+		t.Error("change did not open the form on the scope's quota")
+	}
+
+	res := mirrorPost(t, mux, "/admin/quotas/capacity", url.Values{"csrf": {"tok"}, "scope": {"team:t1"},
+		"gpus": {"8"}, "cores": {""}, "limit_memory": {"64"}}).Body.String()
+	if len(fc.set) != 1 || *fc.set[0].Guarantee.GPUs != 8 || fc.set[0].Guarantee.CPUCores != nil || *fc.set[0].Limit.MemoryGB != 64 {
+		t.Errorf("sent %+v", fc.set)
+	}
+	if !strings.Contains(res, "capacity quota set on team:t1") {
+		t.Errorf("no word of it: %s", res)
+	}
+	for _, bad := range []url.Values{{"scope": {"team:t1"}, "gpus": {"eight"}}, {"scope": {"team:nope"}, "gpus": {"1"}}, {"gpus": {"1"}}} {
+		bad.Set("csrf", "tok")
+		n := len(fc.set)
+		out := mirrorPost(t, mux, "/admin/quotas/capacity", bad).Body.String()
+		if len(fc.set) != n || !strings.Contains(out, `class="error"`) {
+			t.Errorf("%v was not refused with a reason", bad)
+		}
+	}
+	mirrorPost(t, mux, "/admin/quotas/capacity", url.Values{"csrf": {"tok"}, "scope": {"team:t1"}, "clear": {"1"}})
+	if len(fc.cleared) != 1 || fc.cleared[0] != "team:t1" {
+		t.Errorf("cleared %v", fc.cleared)
+	}
+	// Without a CSRF token nothing happens.
+	mirrorPost(t, mux, "/admin/quotas/capacity", url.Values{"scope": {"team:t1"}, "clear": {"1"}})
+	if len(fc.cleared) != 1 {
+		t.Error("a change was made without the CSRF token")
+	}
+}

internal/web/admin_mirror.go

@@ -7,11 +7,13 @@ import (
 	"fmt"
 	"html/template"
 	"net/http"
+	"slices"
 	"sort"
 	"strconv"
 	"strings"
 	"time"
 
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
@@ -67,6 +69,23 @@ type modelUse struct {
 	Name     string
 	Mode     string // always, ondemand or stopped
 	Replicas int
+	// Min and Max are its scaling as the catalog holds it.
+	Min, Max int
+	// Class is what it set, empty for its default; EffClass what that comes
+	// to. Owner is org:ID or empty for shared. MemoryGB and CPUCores are what
+	// a replica of its first profile reserves.
+	Class, EffClass, Owner string
+	MemoryGB, CPUCores     int
+	// Ready, Wanted and Nodes are how it runs now, for the row itself:
+	// replicas ready of those it should have, and the nodes they are on.
+	// Fleet is false where nothing could be asked, so the row says nothing
+	// rather than "0 of 0".
+	Fleet         bool
+	Ready, Wanted int
+	Starting      int
+	Nodes         []nodeName
+	Parked        bool
+	Unplaceable   bool
 	// Declared models are read from server.yaml on every start, so a mode
 	// changed here would revert; the page says where to change it instead.
 	Declared bool
@@ -83,6 +102,11 @@ type mirrorRow struct {
 	// operator clicked. Nil on a row that is still arriving: a catalog entry
 	// pointing at a partial file is the thing versioned paths exist to prevent.
 	Serve *serveForm
+	// Detail is the card opened beneath the row: what runs it, on which
+	// nodes, and how many replicas it runs.
+	Detail *heldDetail
+	// Open is the row's ?open= value.
+	Open string
 
 	// HasPercent is false when the upstream never sent a length. The bar is
 	// then not drawn at all: a progress bar that has to invent its own total is
@@ -100,7 +124,11 @@ type mirrorRow struct {
 }
 
 type adminMirrorData struct {
-	Nav  Nav
+	Nav Nav
+	// Tab is the part of the page shown: held, explore or fetch. Tabs are the
+	// links to each.
+	Tab  string
+	Tabs []mirrorTab
 	Addr string
 	Base string
 	Rows []mirrorRow
@@ -270,6 +298,8 @@ func (p *AccountPages) RegisterAdminMirror(mux *http.ServeMux, d MirrorDeps) {
 				return name + " now runs all the time", nil
 			}))
 
+		mux.HandleFunc("POST /admin/mirror/replicas", p.mirrorAction(m, p.setModel))
+
 		mux.HandleFunc("POST /admin/mirror/serve", p.mirrorAction(m,
 			func(ctx context.Context, admin store.User, r *http.Request) (string, error) {
 				model, err := buildModel(r.FormValue)
@@ -361,7 +391,14 @@ func (p *AccountPages) renderMirror(w http.ResponseWriter, r *http.Request,
 	}
 
 	d.HasCatalog = p.catalog != nil
-	p.explore(r, &d)
+	d.HasRegistry = p.registry != nil
+	d.Tab = mirrorTabFor(r, errMsg)
+	// Only the tab shown does its work: the registry is a server across the
+	// internet, and asking it for sixty repositories to show somebody who came
+	// to look at a download's progress made every visit wait on it.
+	if d.Tab == tabExplore {
+		p.explore(r, &d)
+	}
 
 	got, err := m.List(r.Context())
 	if err != nil {
@@ -370,6 +407,7 @@ func (p *AccountPages) renderMirror(w http.ResponseWriter, r *http.Request,
 		if d.Error == "" {
 			d.Error = err.Error()
 		}
+		d.Tabs = mirrorTabs(d)
 		render(w, "admin_mirror", d)
 		return
 	}
@@ -412,12 +450,50 @@ func (p *AccountPages) renderMirror(w http.ResponseWriter, r *http.Request,
 		}
 		d.Rows = append(d.Rows, row)
 	}
+	var views map[string]adminapi.ModelView
+	prefix := ""
+	if p.fleet != nil && len(sources) > 0 {
+		views = map[string]adminapi.ModelView{}
+		for _, v := range p.fleet.Models() {
+			views[v.Name] = v
+		}
+		var ids []string
+		for _, n := range p.fleet.Nodes() {
+			ids = append(ids, n.ID)
+		}
+		prefix = sharedPrefix(ids)
+	}
 	for i := range d.Rows {
 		for _, m := range sources[d.Rows[i].URL] {
-			d.Rows[i].UsedBy = append(d.Rows[i].UsedBy, modelUse{
+			u := modelUse{
 				Name: m.Name, Mode: modeOf(m.Scaling), Replicas: max(m.Scaling.Max, 1),
+				Min: m.Scaling.Min, Max: m.Scaling.Max,
+				Class: m.Class, EffClass: m.EffectiveClass(), Owner: m.Owner,
 				Declared: p.catalog.Declared(m.Name),
-			})
+			}
+			if len(m.Profiles) > 0 {
+				u.MemoryGB, u.CPUCores = m.Profiles[0].MinMemoryGB, m.Profiles[0].CPUCores
+			}
+			if v, ok := views[m.Name]; ok {
+				u.Fleet, u.Ready, u.Wanted = true, v.Ready, v.Desired
+				u.Parked, u.Unplaceable = v.Parked, v.Unplaceable
+				var nodes []string
+				nodes, u.Starting = runningOn(p.fleet.Replicas(m.Name))
+				for _, n := range nodes {
+					u.Nodes = append(u.Nodes, nodeName{Full: n, Short: strings.TrimPrefix(n, prefix)})
+				}
+			}
+			d.Rows[i].UsedBy = append(d.Rows[i].UsedBy, u)
+		}
+	}
+
+	// The one row whose card is open. From the form too, so a change made in
+	// the card comes back with the card still open to show it.
+	open := r.FormValue("open")
+	for i := range d.Rows {
+		d.Rows[i].Open = openKey(d.Rows[i].Model, d.Rows[i].Version)
+		if open != "" && d.Rows[i].Open == open && d.Rows[i].State == mirror.Present {
+			d.Rows[i].Detail = p.heldDetailFor(r.Context(), d.Rows[i])
 		}
 	}
 
@@ -462,12 +538,78 @@ func (p *AccountPages) renderMirror(w http.ResponseWriter, r *http.Request,
 
 	d.Groups = groupByModel(d.Rows)
 
-	if d.Fetching > 0 {
+	// Only on the list: a reload anywhere else takes with it a search, an
+	// open repository or a URL half typed.
+	if d.Fetching > 0 && d.Tab == tabHeld && open == "" {
 		d.RefreshSec = 5
 	}
+	d.Tabs = mirrorTabs(d)
 	render(w, "admin_mirror", d)
 }
 
+const (
+	tabHeld    = "held"
+	tabExplore = "explore"
+	tabFetch   = "fetch"
+)
+
+// mirrorTab is one of the page's tabs.
+type mirrorTab struct {
+	Label  string
+	Count  string // beside the label, empty for none
+	Href   string
+	Active bool
+}
+
+// mirrorTabFor is the tab a request is for.
+//
+// Named in the address when it is chosen, and otherwise read from what the
+// address carries, so links made before the page had tabs — a search, an
+// open repository — still land where they meant. After a change the list is
+// shown, where its effect is, unless it was refused: then the form it came
+// from, to try again.
+func mirrorTabFor(r *http.Request, errMsg string) string {
+	valid := func(t string) bool { return t == tabHeld || t == tabExplore || t == tabFetch }
+	if r.Method == http.MethodPost {
+		if t := r.PostFormValue("tab"); errMsg != "" && valid(t) {
+			return t
+		}
+		return tabHeld
+	}
+	q := r.URL.Query()
+	if t := q.Get("tab"); valid(t) {
+		return t
+	}
+	for _, k := range []string{"q", "repo", "view", "task", "size", "x"} {
+		if q.Has(k) {
+			return tabExplore
+		}
+	}
+	return tabHeld
+}
+
+func mirrorTabs(d adminMirrorData) []mirrorTab {
+	held := mirrorTab{Label: "Held", Href: "/admin/mirror?tab=held", Active: d.Tab == tabHeld}
+	switch {
+	case d.Unreachable:
+		held.Count = "unreachable"
+	case d.Fetching > 0:
+		held.Count = fmt.Sprintf("%d · %d downloading", d.Present, d.Fetching)
+	default:
+		held.Count = strconv.Itoa(d.Present)
+	}
+	if d.Failed > 0 {
+		held.Count += fmt.Sprintf(" · %d failed", d.Failed)
+	}
+	tabs := []mirrorTab{held}
+	if d.HasRegistry {
+		tabs = append(tabs, mirrorTab{Label: "Explore Hugging Face",
+			Href: "/admin/mirror?tab=explore", Active: d.Tab == tabExplore})
+	}
+	return append(tabs, mirrorTab{Label: "Fetch by URL",
+		Href: "/admin/mirror?tab=fetch", Active: d.Tab == tabFetch})
+}
+
 // mirrorGroup is a family of models the mirror holds — every qwen, every
 // llama — and their versions.
 type mirrorGroup struct {
@@ -663,3 +805,43 @@ func writeJSONStatus(w http.ResponseWriter, code int, v any) {
 	w.WriteHeader(code)
 	_ = json.NewEncoder(w).Encode(v)
 }
+
+// runningOn is the nodes a model's replicas are on, each once and in order,
+// and how many of its replicas are not ready yet.
+func runningOn(reps []adminapi.ReplicaView) (nodes []string, starting int) {
+	for _, r := range reps {
+		if r.State != "ready" {
+			starting++
+		}
+		for _, sh := range r.Shards {
+			if !slices.Contains(nodes, sh.Node) {
+				nodes = append(nodes, sh.Node)
+			}
+		}
+	}
+	slices.Sort(nodes)
+	return nodes, starting
+}
+
+// nodeName is a node as a row names it: short, with the full name beside.
+type nodeName struct{ Short, Full string }
+
+// sharedPrefix is what every node's name starts with, up to and including a
+// separator — "llmgw-" of llmgw-node01 and llmgw-node02 — so a row can say
+// node01 and fit on a line. Nothing when the names share no such part, or
+// when there is one node and dropping it would leave only a number.
+func sharedPrefix(ids []string) string {
+	if len(ids) < 2 {
+		return ""
+	}
+	p := ids[0]
+	for _, id := range ids[1:] {
+		for !strings.HasPrefix(id, p) {
+			p = p[:len(p)-1]
+		}
+	}
+	if i := strings.LastIndexAny(p, "-._"); i >= 0 {
+		return p[:i+1]
+	}
+	return ""
+}

internal/web/admin_mirror_test.go

@@ -341,16 +341,18 @@ func TestAFailedRowOffersRetryRatherThanRemove(t *testing.T) {
 }
 
 type fakeRegistry struct {
-	results []hf.Model
-	repo    hf.Repo
-	err     error
-	asked   string
-	query   hf.Query
+	results  []hf.Model
+	repo     hf.Repo
+	err      error
+	asked    string
+	query    hf.Query
+	explored int
 }
 
 func (f *fakeRegistry) Explore(_ context.Context, q hf.Query) ([]hf.Model, error) {
 	f.asked = q.Search
 	f.query = q
+	f.explored++
 	return f.results, f.err
 }
 
@@ -441,7 +443,11 @@ func TestWithNoRegistryThereIsNoSearchBox(t *testing.T) {
 	if strings.Contains(body, "Search Hugging Face") {
 		t.Error("the explorer should be absent without a registry")
 	}
-	if !strings.Contains(body, "Fetch a model") {
+	if strings.Contains(body, "Explore Hugging Face") {
+		t.Error("a tab for an explorer that is not there")
+	}
+	fetch := mirrorGet(t, mirrorPage(t, &fakeMirror{}, true), "/admin/mirror?tab=fetch").Body.String()
+	if !strings.Contains(fetch, `name="source" required`) {
 		t.Error("fetching by hand must still be there")
 	}
 }
@@ -464,8 +470,7 @@ func TestAShardedModelIsOneRowAndOneButton(t *testing.T) {
 		"/admin/mirror?repo=Qwen/Qwen2.5-7B-Instruct").Body.String()
 
 	// Fifteen shards is one decision, not fifteen. Four here, and one form.
-	if n := strings.Count(body, `action="/admin/mirror/fetch"`); n != 2 {
-		// One for the set, one for the manual form at the bottom.
+	if n := strings.Count(body, `action="/admin/mirror/fetch"`); n != 1 {
 		t.Errorf("expected the four shards as a single form, found %d forms", n)
 	}
 	if !strings.Contains(body, "4 shards, fetched together") {
@@ -534,8 +539,8 @@ func TestAlternativeQuantisationsStaySeparateRows(t *testing.T) {
 
 	// Two GGUFs are alternatives, not halves. Offering "fetch all" here would
 	// spend gigabytes on a copy nobody asked for.
-	if n := strings.Count(body, `action="/admin/mirror/fetch"`); n != 3 {
-		t.Errorf("expected a form each plus the manual one, found %d", n)
+	if n := strings.Count(body, `action="/admin/mirror/fetch"`); n != 2 {
+		t.Errorf("expected a form each, found %d", n)
 	}
 }
 
@@ -587,7 +592,7 @@ func TestHowAModelRunsIsChangedWhereItsFileIsShown(t *testing.T) {
 		t.Error("the running model's mode is not shown as chosen")
 	}
 	// Declared in server.yaml: shown, not offered, since it would revert.
-	if !strings.Contains(body, "on demand<br>set in server.yaml") {
+	if !strings.Contains(body, `title="set in server.yaml, which is read again on every start">on demand · yaml</span>`) {
 		t.Error("a declared model should say where its mode is set")
 	}
 	if strings.Count(body, `name="mode" class="mode"`) != 1 {
@@ -655,3 +660,53 @@ func privateOK(m *mirror.Manager) *mirror.Manager {
 	m.AllowPrivateSources()
 	return m
 }
+
+func TestEachTabDoesOnlyItsOwnWork(t *testing.T) {
+	reg := &fakeRegistry{}
+	mux := mirrorPageWith(t, &fakeMirror{models: []mirror.Status{
+		{Model: "m", Version: "v", File: "m.gguf", State: mirror.Present, BytesDone: 1},
+		{Model: "n", Version: "v", File: "n.gguf", State: mirror.Fetching, BytesDone: 1},
+	}}, reg, true)
+
+	held := mirrorGet(t, mux, "/admin/mirror").Body.String()
+	if reg.explored != 0 {
+		t.Error("the list asked the registry for repositories it does not show")
+	}
+	if !strings.Contains(held, `class="on" aria-current="page">Held <span class="count">1 · 1 downloading</span>`) {
+		t.Errorf("the Held tab should be chosen and count what it holds: %s", held)
+	}
+	if strings.Contains(held, `name="source" required`) || strings.Contains(held, "Search Hugging Face") {
+		t.Error("other tabs' forms are on the list")
+	}
+	if !strings.Contains(held, `http-equiv="refresh"`) {
+		t.Error("the list stopped following a download")
+	}
+
+	explore := mirrorGet(t, mux, "/admin/mirror?tab=explore").Body.String()
+	if reg.explored != 1 || !strings.Contains(explore, "Search Hugging Face") {
+		t.Error("the explorer tab did not explore")
+	}
+	if strings.Contains(explore, `id="held"`) || strings.Contains(explore, `http-equiv="refresh"`) {
+		t.Error("the explorer shows the list, or reloads away a search")
+	}
+	// A link from before there were tabs still lands on the explorer.
+	if old := mirrorGet(t, mux, "/admin/mirror?q=qwen").Body.String(); !strings.Contains(old, "Search Hugging Face") {
+		t.Error("a search address no longer opens the explorer")
+	}
+}
+
+func TestARefusedFetchReturnsToItsFormAndAStartedOneToTheList(t *testing.T) {
+	m := &fakeMirror{fetchErr: errors.New("upstream said 404")}
+	mux := mirrorPage(t, m, true)
+	body := mirrorPost(t, mux, "/admin/mirror/fetch", url.Values{"csrf": {"tok"}, "source": {"https://h.example/m.gguf"}, "model": {"m"},
+		"version": {"v"}, "force": {"1"}, "tab": {"fetch"}}).Body.String()
+	if !strings.Contains(body, "upstream said 404") || !strings.Contains(body, `name="source" required`) {
+		t.Errorf("a refusal should come back on the form it came from: %s", body)
+	}
+	m.fetchErr = nil
+	body = mirrorPost(t, mux, "/admin/mirror/fetch", url.Values{"csrf": {"tok"}, "source": {"https://h.example/m.gguf"}, "model": {"m"},
+		"version": {"v"}, "force": {"1"}, "tab": {"fetch"}}).Body.String()
+	if !strings.Contains(body, `id="held"`) {
+		t.Errorf("a started download should be shown where it is followed: %s", body)
+	}
+}

internal/web/admin_quotas.go

@@ -9,6 +9,7 @@ import (
 	"strings"
 	"time"
 
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
 )
 
@@ -129,12 +130,17 @@ type adminQuotasData struct {
 	// Editing is the "scope:id" whose form is open, so the template can keep
 	// the link on every other row.
 	Editing string
-	Since   string
-	Message string
-	Error   string
-	CSRF    string
-	Version string
-	Now     string
+	// The capacity section: quotas on GPUs, cores and memory.
+	HasCapacity bool
+	Capacity    []capacityRow
+	Scopes      []adminapi.ScopeName
+	CapForm     capacityForm
+	Since       string
+	Message     string
+	Error       string
+	CSRF        string
+	Version     string
+	Now         string
 }
 
 // RegisterAdminQuotas mounts quota administration under /admin/quotas.
@@ -176,6 +182,24 @@ func (p *AccountPages) RegisterAdminQuotas(mux *http.ServeMux, s QuotaAdmin, pri
 			}
 			p.renderQuotas(w, r, s, msg, "")
 		}))
+
+	mux.HandleFunc("POST /admin/quotas/capacity", p.adminOnly(
+		func(w http.ResponseWriter, r *http.Request, admin store.User) {
+			sess, _, _ := p.session(r)
+			if p.capacity == nil || !p.accounts.CheckCSRF(sess, r.FormValue("csrf")) {
+				http.Error(w, "bad request", http.StatusBadRequest)
+				return
+			}
+			msg, err := p.applyCapacity(r, admin)
+			if err != nil {
+				p.log.Warn("a capacity quota change was refused",
+					"admin", admin.Email, "scope", r.FormValue("scope"), "err", err)
+				p.renderQuotas(w, r, s, "", err.Error())
+				return
+			}
+			p.log.Info("capacity quota changed", "admin", admin.Email, "what", msg)
+			p.renderQuotas(w, r, s, msg, "")
+		}))
 }
 
 // applyQuota writes one scope's limits, and says what it did.
@@ -259,6 +283,7 @@ func (p *AccountPages) renderQuotas(w http.ResponseWriter, r *http.Request,
 
 	d.BudgetsBind = p.priced != nil && p.priced.AnythingPriced()
 	d.Editing = r.URL.Query().Get("edit")
+	p.capacitySection(r.Context(), &d, r.FormValue("cap"))
 
 	limits, err := s.AllLimits(r.Context())
 	if err != nil {

internal/web/admin_serve.go

@@ -22,6 +22,11 @@ type CatalogAdmin interface {
 	// Scale changes how a model added at runtime runs: always, on demand, or
 	// not at all. One declared in configuration is refused.
 	Scale(ctx context.Context, name string, sc catalog.Scaling, by string) (catalog.Model, error)
+	// SetClass, SetOwner and Resources change what a model's capacity is
+	// worth, whose quota it counts against, and what a replica reserves.
+	SetClass(ctx context.Context, name, class, by string) (catalog.Model, error)
+	SetOwner(ctx context.Context, name, owner, by string) (catalog.Model, error)
+	Resources(ctx context.Context, name, profile string, memoryGB, cpuCores int, by string) (catalog.Model, error)
 }
 
 // serveForm is the offer to put one mirrored version into service.
@@ -49,6 +54,9 @@ type serveForm struct {
 	// 3.5 GB for a KV pool. Nothing was wrong with placement — it was never
 	// told what the model needed.
 	MinMemoryGB int
+	// CPUCores is the cores a replica computes on: half the smallest node it
+	// fits on, or zero when no node says how many it has.
+	CPUCores int
 	// MemoryBasis says how MinMemoryGB was arrived at, for the form to show.
 	MemoryBasis string
 	// CtxPool is the engine context pool the form starts with, and the one
@@ -152,6 +160,15 @@ func planServe(st mirror.Status, files []mirror.Status, base string,
 			defaultCtxPool, header.BlockCount, kvHeads,
 			float64(int64(1<<30)+weights/20)/(1<<30))
 	}
+	if fleet != nil {
+		var cpus []int
+		for _, n := range fleet.Nodes() {
+			if n.State == "ready" && (n.MemoryGB == 0 || n.MemoryGB >= f.MinMemoryGB) {
+				cpus = append(cpus, n.CPUCount)
+			}
+		}
+		f.CPUCores = catalog.SuggestCores(cpus)
+	}
 	if biggest := biggestNode(fleet); biggest > 0 && f.MinMemoryGB > biggest {
 		f.Warning = strings.TrimSuffix(f.Warning, ".") +
 			fmt.Sprintf("%sthe largest node has %d GB and this needs about %d",
@@ -277,6 +294,11 @@ func buildModel(get func(string) string) (catalog.Model, error) {
 			return m, err
 		}
 	}
+	if v := strings.TrimSpace(get("cpu_cores")); v != "" {
+		if p.CPUCores, err = positive(v, "the cores"); err != nil {
+			return m, err
+		}
+	}
 	if k, v := strings.TrimSpace(get("selector_key")), strings.TrimSpace(get("selector_value")); k != "" && v != "" {
 		// What keeps a large model off the small nodes. A CPU profile has no
 		// card to be short of, so without this every ready node looks like a

internal/web/admin_serve_test.go

@@ -3,6 +3,7 @@ package web
 import (
 	"context"
 	"errors"
+	"fmt"
 	"net/http"
 	"net/http/httptest"
 	"net/url"
@@ -21,6 +22,9 @@ type fakeCatalog struct {
 	scaled    []string
 	lastScale catalog.Scaling
 	err       error
+	classes   []string
+	owners    []string
+	resources []string
 }
 
 func (f *fakeCatalog) Add(_ context.Context, m catalog.Model, _ string) error {
@@ -35,6 +39,21 @@ func (f *fakeCatalog) Sources(context.Context) (map[string][]catalog.Model, erro
 	return f.sources, nil
 }
 
+func (f *fakeCatalog) SetClass(_ context.Context, name, class, _ string) (catalog.Model, error) {
+	f.classes = append(f.classes, name+"="+class)
+	return catalog.Model{Name: name, Class: class}, f.err
+}
+
+func (f *fakeCatalog) SetOwner(_ context.Context, name, owner, _ string) (catalog.Model, error) {
+	f.owners = append(f.owners, name+"="+owner)
+	return catalog.Model{Name: name, Owner: owner}, f.err
+}
+
+func (f *fakeCatalog) Resources(_ context.Context, name, _ string, mem, cores int, _ string) (catalog.Model, error) {
+	f.resources = append(f.resources, fmt.Sprintf("%s=%d/%d", name, mem, cores))
+	return catalog.Model{Name: name}, f.err
+}
+
 func (f *fakeCatalog) Scale(_ context.Context, name string, sc catalog.Scaling, _ string) (catalog.Model, error) {
 	f.scaled = append(f.scaled, name)
 	f.lastScale = sc

internal/web/dashboard.go

@@ -45,6 +45,13 @@ type DashboardData struct {
 
 	HasPeople bool
 
+	// NodePrefix is what every node's name starts with, left off in tables
+	// so a row fits on a line; the full name is on hover.
+	NodePrefix string
+	// Columns shown only when some row has something in them: a CPU-only
+	// cluster's nodes table was three columns of dashes wide.
+	AnyGPUs, AnyClasses, AnyFabrics, AnyFallbacks bool
+
 	// Charts are pre-rendered rather than assembled in the template: building
 	// SVG out of template fragments is how a chart ends up unescaped.
 	RequestChart template.HTML
@@ -177,6 +184,17 @@ func Dashboard(src Source, usage UsageSource, version string, refresh time.Durat
 				}
 			}
 			d.Unplaced = unplaced(src, d.Models)
+			ids := make([]string, 0, len(d.Nodes))
+			for _, n := range d.Nodes {
+				ids = append(ids, n.ID)
+				d.AnyGPUs = d.AnyGPUs || n.GPUTotal > 0
+				d.AnyClasses = d.AnyClasses || len(n.Classes) > 0
+				d.AnyFabrics = d.AnyFabrics || len(n.Fabrics) > 0
+			}
+			d.NodePrefix = sharedPrefix(ids)
+			for _, m := range d.Models {
+				d.AnyFallbacks = d.AnyFallbacks || len(m.Fallbacks) > 0
+			}
 			d.Summary = src.Summary()
 			if fs, ok := src.(FabricSource); ok {
 				r := fs.Fabrics()

internal/web/mirror_detail.go

@@ -0,0 +1,284 @@
+package web
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"net/http"
+	"strconv"
+	"strings"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// HeadroomSource is a fleet that can say how many more replicas of a model
+// fit now. Optional: without it the card shows what runs and offers no change
+// it cannot check.
+type HeadroomSource interface {
+	Headroom(model string, limit int) (adminapi.HeadroomView, bool)
+}
+
+// maxReplicas bounds what the form accepts. Not a cluster limit, a typing one:
+// 300 where 3 was meant would otherwise be taken as a ceiling.
+const maxReplicas = 64
+
+// heldDetail is the card opened beneath a held version: what runs it, where,
+// and room to change it.
+type heldDetail struct {
+	Models []heldModel
+	// Scopes are the parts of the tenancy tree a model can belong to; empty
+	// when capacity quotas are not set up, and the owner choice is left off.
+	Scopes []adminapi.ScopeName
+}
+
+// heldModel is one catalog model running a held version.
+type heldModel struct {
+	modelUse
+	View     adminapi.ModelView
+	Replicas []heldReplica
+	Headroom *adminapi.HeadroomView
+	// MostNow is how many replicas could always run, Current plus the room
+	// found; zero when the room was not measured or had no end in sight.
+	MostNow int
+}
+
+// heldReplica is one replica, by the nodes it runs on.
+type heldReplica struct {
+	ID       string
+	State    string
+	Restart  string
+	Age      int64
+	Endpoint string
+	Nodes    []heldShard
+}
+
+type heldShard struct {
+	Node     string
+	MemoryGB int // the node's RAM, 0 when unknown
+	CPU      float64
+	MemBytes int64
+}
+
+// openKey names a row the way ?open= does.
+func openKey(model, version string) string { return model + ":" + version }
+
+// heldDetailFor reads, for each model using a row, how it runs and how much
+// more room it has.
+func (p *AccountPages) heldDetailFor(ctx context.Context, row mirrorRow) *heldDetail {
+	d := &heldDetail{}
+	if p.capacity != nil {
+		d.Scopes, _ = p.capacity.Scopes(ctx)
+	}
+	if p.fleet == nil {
+		for _, u := range row.UsedBy {
+			d.Models = append(d.Models, heldModel{modelUse: u})
+		}
+		return d
+	}
+	views := map[string]adminapi.ModelView{}
+	for _, v := range p.fleet.Models() {
+		views[v.Name] = v
+	}
+	memory := map[string]int{}
+	for _, n := range p.fleet.Nodes() {
+		memory[n.ID] = n.MemoryGB
+	}
+	hs, _ := p.fleet.(HeadroomSource)
+	for _, u := range row.UsedBy {
+		hm := heldModel{modelUse: u, View: views[u.Name]}
+		for _, rep := range p.fleet.Replicas(u.Name) {
+			// Built field by field: a ReplicaView carries the engine's key,
+			// which is for the admin listener and never for a page.
+			hr := heldReplica{ID: rep.ID, State: rep.State, Restart: rep.Restart,
+				Age: rep.AgeSec, Endpoint: rep.Endpoint}
+			for _, sh := range rep.Shards {
+				hr.Nodes = append(hr.Nodes, heldShard{Node: sh.Node, MemoryGB: memory[sh.Node],
+					CPU: sh.CPUCores, MemBytes: sh.MemoryBytes})
+			}
+			hm.Replicas = append(hm.Replicas, hr)
+		}
+		if hs != nil {
+			if h, ok := hs.Headroom(u.Name, 16); ok {
+				hm.Headroom = &h
+				if !h.Limited {
+					hm.MostNow = h.Current + h.More + h.ByPreempting
+				}
+			}
+		}
+		d.Models = append(d.Models, hm)
+	}
+	return d
+}
+
+// setReplicas changes how many replicas a model always runs and how far it
+// may grow, refusing an always-on count the cluster has no room for.
+//
+// Refused rather than taken and left short: a minimum placement cannot meet
+// is a model reported as unplaceable on every pass from then on, which is the
+// same answer given later and less plainly. A maximum above the room is
+// accepted and said: it is reached only under load, by which time something
+// may have made room, and refusing it would make the ceiling depend on what
+// happened to be running when it was set.
+func (p *AccountPages) setReplicas(ctx context.Context, admin store.User, r *http.Request) (string, error) {
+	name := r.FormValue("model")
+	lo, err1 := strconv.Atoi(strings.TrimSpace(r.FormValue("min")))
+	hi, err2 := strconv.Atoi(strings.TrimSpace(r.FormValue("max")))
+	switch {
+	case name == "":
+		return "", errors.New("which model?")
+	case err1 != nil || err2 != nil:
+		return "", errors.New("replicas are whole numbers")
+	case lo < 0 || hi < lo:
+		return "", fmt.Errorf("always running (%d) cannot be more than the most under load (%d)", lo, hi)
+	case hi > maxReplicas:
+		return "", fmt.Errorf("at most %d replicas", maxReplicas)
+	}
+	current, err := p.currentModel(ctx, name)
+	if err != nil {
+		return "", err
+	}
+	if p.catalog.Declared(name) {
+		return "", fmt.Errorf("%s is set in server.yaml; change it there", name)
+	}
+
+	note := ""
+	if hs, ok := p.fleet.(HeadroomSource); ok && hi > 0 {
+		h, ok := hs.Headroom(name, hi)
+		if !ok {
+			return "", fmt.Errorf("no model %q", name)
+		}
+		fits := h.Current + h.More + h.ByPreempting
+		if lo > fits && !h.Limited {
+			why := "nothing more fits"
+			if len(h.Blocked) > 0 {
+				why = strings.Join(h.Blocked, "; ")
+			}
+			return "", fmt.Errorf("%s can always run at most %d now (%d running, room for %d more): %s",
+				name, fits, h.Current, h.More, why)
+		}
+		if hi > fits && !h.Limited {
+			note = fmt.Sprintf("; there is room for %d now, so it grows past that only if room is made", fits)
+		}
+		// Room its class would make by stopping others: said, since it is
+		// somebody else's replica that goes.
+		if lo > h.Current+h.More && len(h.Preempts) > 0 {
+			note += "; to make room, replicas of " + strings.Join(h.Preempts, ", ") + " will be stopped"
+		}
+	}
+
+	sc := catalog.Scaling{Min: lo, Max: hi, IdleTimeout: current.Scaling.IdleTimeout}
+	if sc.OnDemand() && sc.IdleTimeout <= 0 {
+		sc.IdleTimeout = defaultIdle
+	}
+	if _, err := p.catalog.Scale(ctx, name, sc, admin.Email); err != nil {
+		return "", err
+	}
+	switch {
+	case hi == 0:
+		return name + " is stopped; its replicas are being taken down", nil
+	case lo == 0:
+		return fmt.Sprintf("%s loads when asked for, up to %d under load%s", name, hi, note), nil
+	case lo == hi:
+		return fmt.Sprintf("%s runs %s%s", name, plural(lo, "replica", "replicas"), note), nil
+	}
+	return fmt.Sprintf("%s runs %d, and up to %d under load%s", name, lo, hi, note), nil
+}
+
+// currentModel is a model as the catalog holds it now.
+func (p *AccountPages) currentModel(ctx context.Context, name string) (catalog.Model, error) {
+	sources, err := p.catalog.Sources(ctx)
+	if err != nil {
+		return catalog.Model{}, err
+	}
+	for _, ms := range sources {
+		for _, m := range ms {
+			if m.Name == name {
+				return m, nil
+			}
+		}
+	}
+	return catalog.Model{}, fmt.Errorf("no model %q runs from the mirror", name)
+}
+
+// age renders seconds the way the replicas table does.
+func age(sec int64) string {
+	d := time.Duration(sec) * time.Second
+	switch {
+	case d < time.Minute:
+		return fmt.Sprintf("%ds", sec)
+	case d < time.Hour:
+		return fmt.Sprintf("%dm", int(d.Minutes()))
+	case d < 48*time.Hour:
+		return fmt.Sprintf("%dh%02dm", int(d.Hours()), int(d.Minutes())%60)
+	}
+	return fmt.Sprintf("%dd", int(d.Hours()/24))
+}
+
+// setModel applies the card's settings form: what a replica reserves, the
+// model's class and owner, then how many replicas it runs — in that order, so
+// the room checked for the replicas is the room its new class and reservation
+// would get. Only what changed is applied, and each change is said.
+func (p *AccountPages) setModel(ctx context.Context, admin store.User, r *http.Request) (string, error) {
+	name := r.FormValue("model")
+	current, err := p.currentModel(ctx, name)
+	if err != nil {
+		return "", err
+	}
+	if p.catalog.Declared(name) {
+		return "", fmt.Errorf("%s is set in server.yaml; change it there", name)
+	}
+	var done []string
+	if r.Form.Has("memory") && len(current.Profiles) > 0 {
+		mem, err1 := strconv.Atoi(strings.TrimSpace(r.FormValue("memory")))
+		cores, err2 := strconv.Atoi(strings.TrimSpace(r.FormValue("cores")))
+		if err1 != nil || err2 != nil || mem < 0 || cores < 0 {
+			return "", errors.New("memory and cores are whole numbers")
+		}
+		if cur := current.Profiles[0]; mem != cur.MinMemoryGB || cores != cur.CPUCores {
+			if _, err := p.catalog.Resources(ctx, name, "", mem, cores, admin.Email); err != nil {
+				return "", err
+			}
+			done = append(done, fmt.Sprintf("a replica reserves %d GB and %d cores (running ones keep theirs until restarted)", mem, cores))
+		}
+	}
+	if r.Form.Has("class") {
+		if c := r.FormValue("class"); c != current.Class {
+			m, err := p.catalog.SetClass(ctx, name, c, admin.Email)
+			if err != nil {
+				return "", err
+			}
+			done = append(done, "is "+m.EffectiveClass())
+		}
+	}
+	if r.Form.Has("owner") && p.capacity != nil {
+		if o := r.FormValue("owner"); o != current.Owner {
+			if _, err := p.catalog.SetOwner(ctx, name, o, admin.Email); err != nil {
+				return "", err
+			}
+			if o == "" {
+				done = append(done, "is shared")
+			} else {
+				done = append(done, "counts against "+o)
+			}
+		}
+	}
+	lo, _ := strconv.Atoi(strings.TrimSpace(r.FormValue("min")))
+	hi, _ := strconv.Atoi(strings.TrimSpace(r.FormValue("max")))
+	if r.Form.Has("min") && (lo != current.Scaling.Min || hi != current.Scaling.Max) {
+		msg, err := p.setReplicas(ctx, admin, r)
+		if err != nil {
+			if len(done) > 0 {
+				return "", fmt.Errorf("%s %s, but not the replicas: %w", name, strings.Join(done, ", "), err)
+			}
+			return "", err
+		}
+		done = append(done, strings.TrimPrefix(msg, name+" "))
+	}
+	if len(done) == 0 {
+		return name + ": nothing changed", nil
+	}
+	return name + " " + strings.Join(done, "; "), nil
+}

internal/web/mirror_detail_test.go

@@ -0,0 +1,271 @@
+package web
+
+import (
+	"log/slog"
+	"net/http"
+	"net/url"
+	"strings"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// roomyFleet runs one replica of "coder" on n2 and has room for `more`.
+type roomyFleet struct {
+	fakeFleet
+	more     int
+	blocked  []string
+	asked    int
+	preempt  int
+	preempts []string
+}
+
+func (f *roomyFleet) Nodes() []adminapi.NodeView {
+	return []adminapi.NodeView{{ID: "n1", MemoryGB: 16}, {ID: "n2", MemoryGB: 32}}
+}
+func (f *roomyFleet) Models() []adminapi.ModelView {
+	return []adminapi.ModelView{{Name: "coder", Min: 1, Max: 1, Ready: 1, Total: 1, Desired: 1}}
+}
+func (f *roomyFleet) Replicas(string) []adminapi.ReplicaView {
+	return []adminapi.ReplicaView{{ID: "coder-abc", Model: "coder", State: "ready", AgeSec: 3700,
+		EngineKey: "secret-engine-key",
+		Shards:    []adminapi.ShardView{{Node: "n2", CPUCores: 3.5, MemoryBytes: 6 << 30}}}}
+}
+func (f *roomyFleet) Headroom(model string, limit int) (adminapi.HeadroomView, bool) {
+	f.asked = limit
+	more := min(f.more, limit)
+	h := adminapi.HeadroomView{Model: model, Current: 1, More: more}
+	for range more {
+		h.Nodes = append(h.Nodes, []string{"n1"})
+	}
+	if more < limit {
+		h.Blocked = f.blocked
+	} else {
+		h.Limited = true
+	}
+	h.ByPreempting, h.Preempts = f.preempt, f.preempts
+	return h, true
+}
+
+func detailPage(t *testing.T, fleet *roomyFleet, cat *fakeCatalog) func(string) string {
+	t.Helper()
+	m := &fakeMirror{models: []mirror.Status{
+		{Model: "coder", Version: "v1", File: "c.gguf", State: mirror.Present, BytesDone: 5 << 30},
+	}}
+	mux := mirrorPageDeps(t, MirrorDeps{Mirror: m, Catalog: cat, Fleet: fleet}, true)
+	return func(path string) string { return mirrorGet(t, mux, path).Body.String() }
+}
+
+func coderCatalog() *fakeCatalog {
+	return &fakeCatalog{sources: map[string][]catalog.Model{"http://mirror:8081/coder/v1/c.gguf": {
+		{Name: "coder", Scaling: catalog.Scaling{Min: 1, Max: 1}}}}}
+}
+
+func TestAHeldModelOpensToWhereItRunsAndHowMuchRoomIsLeft(t *testing.T) {
+	fleet := &roomyFleet{more: 2, blocked: []string{"n2: 2 GB free of 32, profile needs 6"}}
+	get := detailPage(t, fleet, coderCatalog())
+
+	closed := get("/admin/mirror")
+	if strings.Contains(closed, "held-card") {
+		t.Error("a card is open before anything was clicked")
+	}
+	if !strings.Contains(closed, `href="/admin/mirror?tab=held&amp;open=coder%3av1"`) {
+		t.Error("the model's name does not open its card")
+	}
+
+	body := get("/admin/mirror?tab=held&open=coder:v1")
+	for _, want := range []string{
+		"held-card", `<a href="/admin/nodes/n2">n2</a>`, "32 GB", "coder-abc", "1h01m", "3.5 cores", "6.0 GiB",
+		"Room for 2 more now, on n1, n1.", "n2: 2 GB free of 32, profile needs 6",
+		`name="min" value="1" min="0" max="3"`, `action="/admin/mirror/replicas"`,
+		`name="open" value="coder:v1"`,
+	} {
+		if !strings.Contains(body, want) {
+			t.Errorf("the card is missing %q", want)
+		}
+	}
+	if strings.Contains(body, "secret-engine-key") {
+		t.Error("the engine's key reached the page")
+	}
+	if strings.Contains(body, `http-equiv="refresh"`) {
+		t.Error("a page with an open form reloads under the operator")
+	}
+}
+
+func TestAReplicaCountTheClusterHasNoRoomForIsRefused(t *testing.T) {
+	fleet := &roomyFleet{more: 1, blocked: []string{"n1: 1 GB free of 16, profile needs 6"}}
+	cat := coderCatalog()
+	mux := mirrorPageDeps(t, MirrorDeps{Mirror: &fakeMirror{models: []mirror.Status{
+		{Model: "coder", Version: "v1", File: "c.gguf", State: mirror.Present, BytesDone: 1},
+	}}, Catalog: cat, Fleet: fleet}, true)
+	post := func(lo, hi string) string {
+		return mirrorPost(t, mux, "/admin/mirror/replicas", url.Values{"csrf": {"tok"},
+			"model": {"coder"}, "min": {lo}, "max": {hi}, "open": {"coder:v1"}}).Body.String()
+	}
+
+	body := post("3", "3")
+	if len(cat.scaled) != 0 {
+		t.Fatalf("three always-on replicas were set with room for two: %+v", cat.lastScale)
+	}
+	if !strings.Contains(body, "can always run at most 2 now") || !strings.Contains(body, "1 GB free of 16") {
+		t.Errorf("the refusal should say how many fit and why: %s", body)
+	}
+	if !strings.Contains(body, "held-card") {
+		t.Error("the card closed on the refusal")
+	}
+
+	body = post("2", "4")
+	if len(cat.scaled) != 1 || cat.lastScale.Min != 2 || cat.lastScale.Max != 4 {
+		t.Fatalf("what fits was not set: %+v", cat.lastScale)
+	}
+	if !strings.Contains(body, "coder runs 2, and up to 4 under load; there is room for 2 now") {
+		t.Errorf("a ceiling above the room should be taken and said: %s", body)
+	}
+
+	post("0", "2")
+	if cat.lastScale.Min != 0 || cat.lastScale.Max != 2 || cat.lastScale.IdleTimeout == 0 {
+		t.Errorf("on demand was set as %+v", cat.lastScale)
+	}
+	for _, bad := range [][2]string{{"3", "2"}, {"-1", "1"}, {"x", "1"}, {"1", "100"}} {
+		n := len(cat.scaled)
+		post(bad[0], bad[1])
+		if len(cat.scaled) != n {
+			t.Errorf("min %s max %s was accepted", bad[0], bad[1])
+		}
+	}
+}
+
+func TestAModelSetInConfigurationIsShownButNotChanged(t *testing.T) {
+	cat := coderCatalog()
+	cat.declared = "coder"
+	body := detailPage(t, &roomyFleet{more: 20}, cat)("/admin/mirror?open=coder:v1")
+	if strings.Contains(body, `action="/admin/mirror/replicas"`) || !strings.Contains(body, "change its replicas there") {
+		t.Error("a declared model offered a change that would revert")
+	}
+	if !strings.Contains(body, "Room for at least 16 more") {
+		t.Error("room that reached the limit asked should read as at least")
+	}
+}
+
+func TestAHeldRowSaysHowItsModelRunsWithoutOpening(t *testing.T) {
+	body := detailPage(t, &roomyFleet{more: 1}, coderCatalog())("/admin/mirror")
+	if strings.Contains(body, "held-card") {
+		t.Fatal("the card opened by itself")
+	}
+	if !strings.Contains(body, `title="replicas ready / wanted">1/1 · <span class="node" title="n2">n2</span></span>`) {
+		t.Errorf("the row does not say where its replicas run: %s", body)
+	}
+}
+
+func TestRunningOnNamesEachNodeOnceAndCountsWhatIsStarting(t *testing.T) {
+	nodes, starting := runningOn([]adminapi.ReplicaView{
+		{State: "ready", Shards: []adminapi.ShardView{{Node: "n2"}}},
+		{State: "starting", Shards: []adminapi.ShardView{{Node: "n1"}}},
+		{State: "ready", Shards: []adminapi.ShardView{{Node: "n2"}}},
+	})
+	if strings.Join(nodes, ",") != "n1,n2" || starting != 1 {
+		t.Errorf("nodes %v, starting %d", nodes, starting)
+	}
+}
+
+func TestNodeNamesLoseOnlyWhatEveryNodeShares(t *testing.T) {
+	for _, c := range []struct {
+		ids  []string
+		want string
+	}{
+		{[]string{"llmgw-node01", "llmgw-node02", "llmgw-node06"}, "llmgw-"},
+		{[]string{"gpu-a.dc1", "gpu-b.dc1"}, "gpu-"},
+		{[]string{"n1", "n2"}, ""},
+		{[]string{"llmgw-node01"}, ""},
+		{[]string{"alpha", "beta"}, ""},
+	} {
+		if got := sharedPrefix(c.ids); got != c.want {
+			t.Errorf("%v: %q, expected %q", c.ids, got, c.want)
+		}
+	}
+}
+
+func settingsPage(t *testing.T, cat *fakeCatalog, withQuotas bool) *http.ServeMux {
+	t.Helper()
+	a := fakeAccounts{user: store.User{ID: "u1", Email: "a@example.com", IsAdmin: true}}
+	p := NewAccountPages(a, nil, nil, "v0", slog.New(slog.DiscardHandler))
+	if withQuotas {
+		p.SetCapacity(&fakeCapacity{})
+	}
+	mux := http.NewServeMux()
+	p.RegisterAdminMirror(mux, MirrorDeps{Mirror: &fakeMirror{models: []mirror.Status{
+		{Model: "coder", Version: "v1", File: "c.gguf", State: mirror.Present, BytesDone: 1},
+	}}, Catalog: cat, Fleet: &roomyFleet{more: 3}})
+	return mux
+}
+
+func TestACardSetsClassOwnerAndReservationAndOnlyWhatChanged(t *testing.T) {
+	cat := coderCatalog()
+	cat.sources["http://mirror:8081/coder/v1/c.gguf"][0].Profiles = []catalog.Profile{{Name: "cpu", MinMemoryGB: 12, CPUCores: 4}}
+	mux := settingsPage(t, cat, true)
+
+	card := mirrorGet(t, mux, "/admin/mirror?open=coder:v1").Body.String()
+	for _, want := range []string{`name="class"`, `<option value="" selected>default (standard)</option>`,
+		`name="owner"`, `<option value="team:t1">Acme / ml</option>`, `name="memory" value="12"`, `name="cores" value="4"`} {
+		if !strings.Contains(card, want) {
+			t.Errorf("the card lacks %q", want)
+		}
+	}
+	post := func(v url.Values) string {
+		v.Set("csrf", "tok")
+		v.Set("model", "coder")
+		v.Set("open", "coder:v1")
+		return mirrorPost(t, mux, "/admin/mirror/replicas", v).Body.String()
+	}
+	same := url.Values{"min": {"1"}, "max": {"1"}, "class": {""}, "owner": {""}, "memory": {"12"}, "cores": {"4"}}
+	if body := post(same); !strings.Contains(body, "coder: nothing changed") ||
+		len(cat.classes)+len(cat.owners)+len(cat.resources)+len(cat.scaled) != 0 {
+		t.Errorf("an unchanged form changed something: %v %v %v %v", cat.classes, cat.owners, cat.resources, cat.scaled)
+	}
+	changed := url.Values{"min": {"1"}, "max": {"1"}, "class": {"critical"}, "owner": {"team:t1"}, "memory": {"14"}, "cores": {"4"}}
+	body := post(changed)
+	if strings.Join(cat.resources, ",") != "coder=14/4" || strings.Join(cat.classes, ",") != "coder=critical" ||
+		strings.Join(cat.owners, ",") != "coder=team:t1" || len(cat.scaled) != 0 {
+		t.Errorf("applied %v %v %v %v", cat.resources, cat.classes, cat.owners, cat.scaled)
+	}
+	if !strings.Contains(body, "a replica reserves 14 GB and 4 cores") || !strings.Contains(body, "counts against team:t1") {
+		t.Errorf("the change was not said: %s", body)
+	}
+
+	// Without capacity quotas there is no owner to choose, and none is sent.
+	plain := mirrorGet(t, settingsPage(t, coderCatalog(), false), "/admin/mirror?open=coder:v1").Body.String()
+	if strings.Contains(plain, `name="owner"`) {
+		t.Error("an owner choice with no quotas to count against")
+	}
+}
+
+func TestRoomAHigherClassWouldMakeIsOfferedAndSaid(t *testing.T) {
+	fleet := &roomyFleet{more: 0, preempt: 1, preempts: []string{"chat"}, blocked: []string{"n1: full"}}
+	cat := coderCatalog()
+	mux := mirrorPageDeps(t, MirrorDeps{Mirror: &fakeMirror{models: []mirror.Status{
+		{Model: "coder", Version: "v1", File: "c.gguf", State: mirror.Present, BytesDone: 1},
+	}}, Catalog: cat, Fleet: fleet}, true)
+
+	card := mirrorGet(t, mux, "/admin/mirror?open=coder:v1").Body.String()
+	if !strings.Contains(card, "No free room for another replica now.") ||
+		!strings.Contains(card, "1 by stopping replicas its class outranks: chat.") ||
+		!strings.Contains(card, `name="min" value="1" min="0" max="2"`) {
+		t.Errorf("the card should offer the room preemption makes, and say whose")
+	}
+	body := mirrorPost(t, mux, "/admin/mirror/replicas", url.Values{"csrf": {"tok"},
+		"model": {"coder"}, "min": {"2"}, "max": {"2"}, "open": {"coder:v1"}}).Body.String()
+	if len(cat.scaled) != 1 || cat.lastScale.Min != 2 {
+		t.Fatalf("two were refused though preemption makes room: %s", body)
+	}
+	if !strings.Contains(body, "replicas of chat will be stopped") {
+		t.Errorf("stopping somebody else's replica should be said: %s", body)
+	}
+	body = mirrorPost(t, mux, "/admin/mirror/replicas", url.Values{"csrf": {"tok"},
+		"model": {"coder"}, "min": {"3"}, "max": {"3"}, "open": {"coder:v1"}}).Body.String()
+	if len(cat.scaled) != 1 || !strings.Contains(body, "at most 2") {
+		t.Errorf("three is past even what preemption makes: %s", body)
+	}
+}

internal/web/mirror_explore.go

@@ -155,7 +155,7 @@ func oneOf(s string, options ...string) bool {
 
 // href is the page with these settings and one of them changed.
 func (e explorer) href(query, key, value string) template.URL {
-	v := url.Values{"view": {e.View}, "task": {e.Task}, "size": {e.Size}, "x": {"1"}}
+	v := url.Values{"tab": {tabExplore}, "view": {e.View}, "task": {e.Task}, "size": {e.Size}, "x": {"1"}}
 	if e.Here {
 		v.Set("here", "1")
 	}

internal/web/mirror_explore_test.go

@@ -27,7 +27,7 @@ func TestTheExplorerOpensOnTrendingModelsTheNodesCanRun(t *testing.T) {
 		ggufModel("odd/Kev-GGUF", "ggmlc", 5e8, now.Add(-2*24*time.Hour)),
 	}}
 	body := mirrorGet(t, mirrorPageLlama(t, MirrorDeps{Mirror: &fakeMirror{}, Registry: reg}),
-		"/admin/mirror").Body.String()
+		"/admin/mirror?tab=explore").Body.String()
 
 	q := reg.query
 	if q.Sort != hf.SortTrending || q.Task != "text-generation" || !q.GGUF || q.Search != "" {
@@ -96,7 +96,7 @@ func TestAnAdapterIsNotListedAsAModel(t *testing.T) {
 		ggufModel("a/Tiny-GGUF", "qwen2", 135e6, time.Now()),
 	}}
 	body := mirrorGet(t, mirrorPageLlama(t, MirrorDeps{Mirror: &fakeMirror{}, Registry: reg}),
-		"/admin/mirror").Body.String()
+		"/admin/mirror?tab=explore").Body.String()
 	if strings.Contains(body, "Some-LoRA") || !strings.Contains(body, "a/Tiny-GGUF") {
 		t.Errorf("adapters should be left out and small models kept: %s", body)
 	}

internal/web/node_page_test.go

@@ -7,6 +7,7 @@ import (
 	"time"
 
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/nvlink"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/topology"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/promq"
@@ -174,3 +175,33 @@ func TestTheNVLinkPageDrawsEachDomainAsARack(t *testing.T) {
 		t.Error("the cluster page does not lead to the racks")
 	}
 }
+
+// hotSource is a GPU node whose sensors report, one card held back.
+type hotSource struct{ fakeSource }
+
+func (hotSource) Nodes() []adminapi.NodeView {
+	n := fakeSource{}.Nodes()[0]
+	n.Load = &adminapi.LoadView{CPUPercent: 40, CPUs: 64, MemTotal: 1 << 40, MemUsed: 1 << 39,
+		CPUTempC: 58, CPUPowerW: 312, InletTempC: 24.5,
+		GPUs: []inventory.GPUSensor{
+			{Index: 0, TempC: 71, MemoryTempC: 84, PowerW: 612, PowerLimitW: 700},
+			{Index: 1, TempC: 88, PowerW: 698, PowerLimitW: 700, Throttle: []string{"sw_power_cap", "sw_thermal"}},
+		}}
+	return []adminapi.NodeView{n}
+}
+
+func TestANodesSensorsAreOnItsPageAndNothingItCannotMeasure(t *testing.T) {
+	_, body := get(t, nodeMux(hotSource{}, nil), "http://x/admin/nodes/llmgw-node01", nil)
+	for _, want := range []string{"58 °C", "312 W", "24.5 °C", "inlet air", "84 °C", "612 W", "of 700",
+		`<span class="bad">sw_power_cap, sw_thermal</span>`} {
+		if !strings.Contains(body, want) {
+			t.Errorf("the page lacks %q", want)
+		}
+	}
+	_, vm := get(t, nodeMux(busySource{}, nil), "http://x/admin/nodes/llmgw-node01", nil)
+	for _, absent := range []string{"°C", "<h2>GPUs</h2>", "inlet air"} {
+		if strings.Contains(vm, absent) {
+			t.Errorf("a node with no sensors shows %q", absent)
+		}
+	}
+}

internal/web/rows_test.go

@@ -0,0 +1,54 @@
+package web
+
+import (
+	"strings"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+)
+
+// cpuFleet is a CPU-only cluster whose nodes share a name prefix.
+type cpuFleet struct{ fakeSource }
+
+func (cpuFleet) Nodes() []adminapi.NodeView {
+	return []adminapi.NodeView{
+		{ID: "llmgw-node01", State: "ready", MgmtAddr: "10.0.0.141", MemoryGB: 8},
+		{ID: "llmgw-node02", State: "ready", MgmtAddr: "10.0.0.142", MemoryGB: 8},
+	}
+}
+func (cpuFleet) Summary() adminapi.Summary { return adminapi.Summary{} }
+func (cpuFleet) Models() []adminapi.ModelView {
+	return []adminapi.ModelView{{Name: "qwen", Min: 1, Max: 1, Ready: 1,
+		Description: "A sentence long enough that it once had a row of its own under the model."}}
+}
+func (cpuFleet) Replicas(string) []adminapi.ReplicaView {
+	return []adminapi.ReplicaView{{ID: "qwen-1", Model: "qwen", State: "ready",
+		Shards: []adminapi.ShardView{{Node: "llmgw-node02"}}}}
+}
+
+func TestTheClusterPageKeepsEachRecordOnOneLine(t *testing.T) {
+	h := Dashboard(cpuFleet{}, nil, "v1", time.Second, true, NavExtras{}, nil)
+	_, body := get(t, h, "http://127.0.0.1:9090/", nil)
+
+	for _, gone := range []string{"<th>Classes</th>", "<th>Fabrics</th>", `<th class="num">GPUs</th>`, "<th>Fallbacks</th>"} {
+		if strings.Contains(body, gone) {
+			t.Errorf("%s is shown with nothing in it on any row", gone)
+		}
+	}
+	if !strings.Contains(body, `title="llmgw-node01"><code>node01</code>`) {
+		t.Error("node names should drop what every node shares, keeping the whole on hover")
+	}
+	if !strings.Contains(body, `<code title="llmgw-node02">node02</code>`) {
+		t.Error("the replicas table should name nodes the same way")
+	}
+	if strings.Contains(body, `class="url-row"`) || !strings.Contains(body, `<td class="clip" title="A sentence long enough`) {
+		t.Error("a model's description should be a clipped cell, not a row of its own")
+	}
+
+	// A cluster with GPUs still gets its GPU column.
+	_, gpu := get(t, Dashboard(fakeSource{}, nil, "v1", time.Second, true, NavExtras{}, nil), "http://127.0.0.1:9090/", nil)
+	if !strings.Contains(gpu, `<th class="num">GPUs</th>`) || !strings.Contains(gpu, "<th>Fabrics</th>") {
+		t.Error("columns with something in them were hidden")
+	}
+}

internal/web/static/base.css

@@ -60,7 +60,7 @@ li { margin: .25rem 0; }
 a { color: var(--accent); }
 .wrap.narrow { max-width: 30rem; }
 label { display: block; margin: .9rem 0; font-size: .9rem; font-weight: 600; }
-input[type=text], input[type=email], input[type=password], input[type=search] {
+input[type=text], input[type=email], input[type=password], input[type=search], input[type=number] {
   display: block; width: 100%; margin-top: .3rem; font: inherit; font-weight: 400;
   padding: .5rem .6rem; border: 1px solid var(--line); border-radius: 6px;
   background: var(--card); color: var(--fg);
@@ -444,18 +444,29 @@ h3.limits-for { font-size: .92rem; margin: 1rem 0 .4rem; }
 table.narrow-table { max-width: 30rem; }
 .q-act a + a { margin-left: .8rem; }
 .chip-role { font-size: .68rem; font-weight: 600; color: var(--accent); text-transform: uppercase; letter-spacing: .04em; }
-/* Runs: one fixed-width selector per model, so they line up down the column
-   whatever the names; a catalog name differing from the mirror's sits above. */
-table.held .use { width: 9.5rem; margin: .1rem 0; }
-table.held .use .as { display: block; font-size: .72rem; color: var(--muted);
-                      overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
-table.held .use .opt { font-size: .75rem; line-height: 1.3; }
+/* Runs: one line per model — a catalog name that differs from the mirror's,
+   how it runs, and where — with the selector at a fixed width so the
+   replica counts line up down the column. */
+table.held .use { display: flex; align-items: center; gap: .5rem; margin: .1rem 0; white-space: nowrap; }
+table.held .use .as { font-size: .72rem; color: var(--muted); max-width: 10rem;
+                      overflow: hidden; text-overflow: ellipsis; }
+table.held .use form.inline { margin: 0; }
+table.held .use .mode-fixed { width: 7.5rem; font-size: .8rem; color: var(--muted); }
 table.held td.num { white-space: nowrap; }
-table.held select.mode { width: 100%; padding: .15rem .3rem; font-size: .8rem; }
+table.held select.mode { width: 7.5rem; margin: 0; padding: .1rem .3rem; font-size: .8rem; }
 
 /* Link pills: the same look as .tabs, for choices that are addresses rather
    than panels on one page — a filtered listing should survive a reload and be
    worth sending to somebody. */
+/* A page's own sections, one address each. Underlined rather than pills so
+ * they do not read as one more filter beside the ones inside a section. */
+.page-tabs { display: flex; flex-wrap: wrap; gap: 0 1.5rem; margin: -1rem 0 1.5rem;
+             border-bottom: 1px solid var(--line); }
+.page-tabs a { padding: .55rem 0; margin-bottom: -1px; font-weight: 600; color: var(--muted);
+               text-decoration: none; border-bottom: 2px solid transparent; }
+.page-tabs a:hover { color: var(--fg); }
+.page-tabs a.on { color: var(--fg); border-bottom-color: var(--accent); }
+.page-tabs .count { font-weight: 500; font-size: .8rem; color: var(--muted); margin-left: .25rem; }
 .pills { display: flex; flex-wrap: wrap; gap: .35rem; margin: .5rem 0; }
 .pills a { font-size: .85rem; font-weight: 600; color: var(--muted); padding: .35rem .85rem;
            border: 1px solid transparent; border-radius: 999px; text-decoration: none; }
@@ -804,3 +815,46 @@ details.chat-settings output { font-variant-numeric: tabular-nums; font-weight:
 .copyable button { margin-right: .75rem; font-size: .8rem; }
 .connect-form .field-row label { flex: 1 1 12rem; }
 table.connect-fields th { white-space: nowrap; }
+
+/* The card opened beneath a held version on the mirror page. */
+tr.held-detail > td { background: var(--bg); padding: .75rem 1rem 1rem; }
+.held-card .held-model { margin-top: .75rem; padding-top: .75rem; border-top: 1px solid var(--line); }
+.held-card .held-model:first-of-type { border-top: 0; padding-top: 0; }
+.held-card h3 { font-size: 1rem; margin: 0 0 .5rem; display: flex; gap: .6rem; align-items: baseline; flex-wrap: wrap; }
+.held-card table.replicas { margin: .25rem 0 .5rem; }
+.held-card .headroom { margin: .5rem 0 .25rem; font-weight: 600; }
+.held-card details.blocked { font-size: .85rem; color: var(--muted); margin-bottom: .5rem; }
+.held-card details.blocked ul { margin: .35rem 0 0 1.1rem; padding: 0; }
+.held-card .replicas-form .field-row { align-items: flex-end; }
+.held-card .replicas-form .field-row label { flex: 0 0 auto; }
+.held-card .replicas-form input[type=number] { width: 8rem; margin: .25rem 0 0; }
+.held-card .replicas-form button { margin: 0; }
+a.open-held { text-decoration: none; color: inherit; }
+a.open-held:hover strong { text-decoration: underline; }
+a.open-held[aria-expanded=true]::before { content: "▾ "; color: var(--muted); }
+a.open-held[aria-expanded=false]::before { content: "▸ "; color: var(--muted); }
+.use .runs-now { font-size: .78rem; color: var(--muted); }
+.use .runs-now.short { color: var(--warn); }
+.use .runs-now.bad { color: var(--bad); }
+.use .runs-now .node { white-space: nowrap; }
+
+/* One line per record, in every table. A header never wraps; a cell of free
+ * text is cut to its column with the whole of it on hover, instead of a row of
+ * its own or a line break that makes one record look like two. */
+th { white-space: nowrap; }
+table.rows td { white-space: nowrap; }
+td.clip { max-width: 22rem; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
+table.rows td.clip { color: var(--muted); font-size: .85rem; }
+/* A row across the whole table is a note or an open form, not a record: it
+ * wraps as prose does. */
+table.rows td[colspan] { white-space: normal; }
+table.rows td.where { max-width: none; }
+table.rows .more { cursor: help; margin-left: .2rem; }
+table.explore td:first-child { max-width: 34rem; overflow: hidden; text-overflow: ellipsis; }
+table.explore .from { margin-left: .5rem; font-size: .78rem; color: var(--muted); }
+.clip-inline { display: inline-block; max-width: 24rem; overflow: hidden; text-overflow: ellipsis;
+               vertical-align: bottom; font-size: .8rem; color: var(--muted); margin-left: .4rem; }
+table.rows td.warn { color: var(--warn); cursor: help; }
+.held-card .replicas-form select, .capacity-form select { display: block; margin-top: .25rem; }
+.held-card .replicas-form .field-row { flex-wrap: wrap; gap: .75rem 1rem; }
+.quota-tree td.q-act form.inline { margin-left: .75rem; }

internal/web/templates/account.html

@@ -84,7 +84,7 @@ Tell whoever you asked for access that you have signed up.</p>
 {{end}}
 {{if .Fields}}
 <h3>Settings to fill in</h3>
-<table class="compact narrow-table connect-fields">{{range .Fields}}
+<table class="rows compact narrow-table connect-fields">{{range .Fields}}
   <tr><th>{{.Label}}</th><td><code>{{.Value}}</code></td><td><button type="button" class="link copy" data-value="{{.Value}}">copy</button></td></tr>{{end}}
 </table>
 {{end}}
@@ -149,7 +149,7 @@ document.addEventListener("click", (e) => {
 
 <h2>Keys</h2>
 {{if .Keys}}
-<table><tr><th>Name</th><th>Key</th><th>Spends against</th><th>Created</th><th>Last used</th><th>State</th><th></th></tr>
+<table class="rows"><tr><th>Name</th><th>Key</th><th>Spends against</th><th>Created</th><th>Last used</th><th>State</th><th></th></tr>
 {{range .Keys}}<tr>
   <td>{{.Name}}</td>
   <td><code>{{.Prefix}}</code></td>
@@ -192,7 +192,7 @@ old one to stop moving, then revoke it.</p>
 </div>
 {{end}}
 {{if .Rows}}
-<table class="compact narrow-table"><tr><th>Limit</th><th class="num">Value</th></tr>
+<table class="rows compact narrow-table"><tr><th>Limit</th><th class="num">Value</th></tr>
 {{range .Rows}}<tr><td>{{.Name}}</td><td class="num">{{.Value}}</td></tr>{{end}}</table>
 {{else}}<p class="empty">No limits here.</p>{{end}}
 {{else}}<p class="empty">Nowhere to spend yet.</p>{{end}}

internal/web/templates/admin_mirror.html

@@ -8,16 +8,20 @@
 </head><body><div class="wrap">
 <header>
   <h1>Model mirror</h1>
-  <p class="sub">
-    {{if .Unreachable}}{{.Addr}} did not answer{{else}}{{.Present}} held ·
-    {{.Fetching}} downloading · {{.Failed}} failed · {{.Addr}}{{end}}
-  </p>
+  <p class="sub">{{.Addr}}{{if .Unreachable}} did not answer{{end}}</p>
   {{template "nav" .Nav}}
 </header>
 
+{{/* Links rather than a script's panels: each tab has an address, the back
+     button works, and only the one shown asks anything of the registry. */}}
+<nav class="page-tabs" aria-label="Mirror">
+{{range .Tabs}}<a href="{{.Href}}"{{if .Active}} class="on" aria-current="page"{{end}}>{{.Label}}{{with .Count}} <span class="count">{{.}}</span>{{end}}</a>
+{{end}}</nav>
+
 {{if .Error}}<p class="error">{{.Error}}</p>{{end}}
 {{if .Message}}<p class="ok">{{.Message}}</p>{{end}}
 
+{{if eq .Tab "held"}}
 {{if .Unreachable}}
 {{/* Said outright rather than shown as an empty table. A mirror holding
      nothing and a mirror that is not there look the same in a list of zero
@@ -29,9 +33,8 @@ reachable from this machine.</p>
 {{else}}
 
 <div id="held">
-<h2>What the mirror holds</h2>
 {{if .Rows}}
-<table class="compact held">
+<table class="rows compact held">
 <tr><th>Model</th><th>Files</th><th class="num">Size</th><th>Runs</th><th></th></tr>
 {{range .Groups}}{{$grouped := gt .Models 1}}
 {{if $grouped}}
@@ -41,8 +44,10 @@ reachable from this machine.</p>
   {{if .Failed}}<span class="tag bad">failed</span>{{end}}
   {{if .InUse}}<span class="tag">in use</span>{{end}}</td></tr>
 {{end}}
-{{range .Rows}}<tr{{if $grouped}} class="model-version"{{end}}>
-  <td><strong>{{.Model}}</strong> <span class="opt">{{.Version}}</span></td>
+{{range .Rows}}{{$row0 := .}}<tr{{if $grouped}} class="model-version"{{end}}{{if .Detail}} class="open"{{end}}>
+  <td>{{if eq (printf "%s" .State) "present"}}<a class="open-held" aria-expanded="{{if .Detail}}true{{else}}false{{end}}"
+       href="/admin/mirror?tab=held{{if not .Detail}}&amp;open={{.Open}}{{end}}"><strong>{{.Model}}</strong></a>
+    {{else}}<strong>{{.Model}}</strong>{{end}} <span class="opt">{{.Version}}</span></td>
   <td class="file" title="{{.File}}{{if .SHA256}} · sha256 {{.SHA256}}{{end}}">
     {{if gt .Files 1}}{{.Files}} files{{else}}<code>{{.File}}</code>{{end}}</td>
   <td class="num">
@@ -62,7 +67,8 @@ reachable from this machine.</p>
       {{/* Named only when the catalog calls it something other than the
            mirror does; otherwise it is the model column again. */}}
       {{if ne .Name $row.Model}}<span class="as" title="{{.Name}}">as {{.Name}}</span>{{end}}
-      {{if .Declared}}<span class="opt">{{if eq .Mode "always"}}always on{{else if eq .Mode "ondemand"}}on demand{{else}}stopped{{end}}<br>set in server.yaml</span>
+      {{if .Declared}}<span class="mode-fixed" title="set in server.yaml, which is read again on every start">
+        {{- if eq .Mode "always"}}always on{{else if eq .Mode "ondemand"}}on demand{{else}}stopped{{end}} · yaml</span>
       {{else}}
       {{/* Submitted on change: the choice is the action, and a second click
            to confirm it would only be a way to forget to. */}}
@@ -78,6 +84,12 @@ reachable from this machine.</p>
         <noscript><button type="submit" class="link">set</button></noscript>
       </form>
       {{end}}
+      {{if .Fleet}}<span class="runs-now{{if or .Parked .Unplaceable}} bad{{else if lt .Ready .Wanted}} short{{end}}" title="replicas ready / wanted{{if .Starting}}, + starting{{end}}">
+        {{- if .Parked}}parked
+        {{- else if and (eq .Wanted 0) (eq .Ready 0)}}none running
+        {{- else}}{{.Ready}}/{{.Wanted}}{{if .Starting}} +{{.Starting}}{{end}}{{end}}
+        {{- if .Unplaceable}} · no room{{end}}
+        {{- with .Nodes}} · {{range $i, $n := .}}{{if $i}}, {{end}}<span class="node" title="{{$n.Full}}">{{$n.Short}}</span>{{end}}{{end}}</span>{{end}}
     </div>{{end}}
     {{else}}<span class="opt">not served</span>{{end}}
   </td>
@@ -109,6 +121,7 @@ reachable from this machine.</p>
   </td>
 </tr>
 {{if .Err}}<tr><td colspan="5" class="error-row">{{.Err}}</td></tr>{{end}}
+{{with .Detail}}<tr class="held-detail"><td colspan="5">{{template "mirror_held_detail" dict "D" . "Row" $row0 "CSRF" $.CSRF}}</td></tr>{{end}}
 
 {{with .Serve}}
 <tr class="editing"><td colspan="5">
@@ -152,7 +165,9 @@ reachable from this machine.</p>
   <div class="field-row">
     <label>Profile name <input type="text" name="profile" value="{{.Profile}}"></label>
     <label>Needs RAM (GB) <span class="opt">for the context pool beside it</span>
-      <input type="text" name="min_memory_gb" value="{{.MinMemoryGB}}"></label>
+      <input type="text" name="min_memory_gb" value="{{.MinMemoryGB}}" required></label>
+    <label>CPU cores <span class="opt">reserved; the engine's threads</span>
+      <input type="text" name="cpu_cores" value="{{.CPUCores}}"></label>
     <label>Only nodes labelled <span class="opt">key / value, optional</span>
       <input type="text" name="selector_key" placeholder="role" size="8">
       <input type="text" name="selector_value" placeholder="cpu-large" size="10"></label>
@@ -190,11 +205,14 @@ document.addEventListener('click', function (e) {
   } else { window.prompt('Address', b.dataset.url); }
 });
 </script>
-{{else}}<p class="empty">The mirror is reachable and holds nothing yet.</p>{{end}}
+{{else}}<p class="empty">The mirror is reachable and holds nothing yet.
+{{if .HasRegistry}}<a href="/admin/mirror?tab=explore">Explore Hugging Face</a> for a model, or
+{{end}}<a href="/admin/mirror?tab=fetch">fetch one by URL</a>.</p>{{end}}
 </div>
+{{end}}
+{{end}}
 
-{{if .HasRegistry}}
-<h2>Explore models</h2>
+{{if and (eq .Tab "explore") .HasRegistry}}
 {{if not .Repo}}{{with .Explore}}
 <nav class="pills" aria-label="Which models">{{range .Views}}<a href="{{.Href}}"{{if .Active}} class="on" aria-current="page"{{end}}>{{.Label}}</a>{{end}}</nav>
 <div class="explore-filters">
@@ -203,6 +221,7 @@ document.addEventListener('click', function (e) {
 </div>
 {{end}}
 <form method="get" action="/admin/mirror" class="explore-search">
+  <input type="hidden" name="tab" value="explore">
   {{with .Explore}}<input type="hidden" name="view" value="{{.View}}">
   <input type="hidden" name="task" value="{{.Task}}">
   <input type="hidden" name="size" value="{{.Size}}">
@@ -222,13 +241,13 @@ document.addEventListener('click', function (e) {
 
 {{if not .Repo}}{{with .Explore}}
 {{if .Rows}}
-<table class="explore">
+<table class="rows explore">
 <tr><th>Repository</th><th class="num">Size</th><th>Architecture</th><th class="num">Context</th>
 <th class="num">Downloads</th><th class="num">Likes</th><th class="num">Updated</th></tr>
 {{range .Rows}}<tr class="explore-row">
   <td><a href="{{.Href}}" class="open-repo" data-repo="{{.ID}}" aria-expanded="false"><code>{{.ID}}</code></a>
     {{if .New}}<span class="tag new">new</span>{{end}}{{if .Gated.Is}}<span class="tag">gated</span>{{end}}
-    {{if or .BaseModel .License}}<div class="from">{{with .BaseModel}}from {{.}}{{end}}{{if and .BaseModel .License}} · {{end}}{{.License}}</div>{{end}}</td>
+    {{if or .BaseModel .License}}<span class="from" title="{{with .BaseModel}}from {{.}}{{end}}{{if and .BaseModel .License}} · {{end}}{{.License}}">{{with .BaseModel}}from {{.}}{{end}}{{if and .BaseModel .License}} · {{end}}{{.License}}</span>{{end}}</td>
   <td class="num">{{.ParamsLabel}}</td>
   <td>{{if .Architecture}}<span{{if not .Verdict.Servable}} class="cannot"{{end}} title="{{.Verdict.Reason}}">{{.Architecture}}{{if .Verdict.Checked}}{{if .Verdict.Servable}} ✓{{else}} ✗{{end}}{{end}}</span>{{else}}—{{end}}</td>
   <td class="num">{{.ContextLabel}}</td>
@@ -307,9 +326,12 @@ document.querySelector("table.explore").addEventListener("submit", async functio
       return;
     }
     note.classList.remove("bad");
-    note.textContent = "✓ " + out.message + " — its progress is in the table above.";
+    note.textContent = "✓ " + out.message + " — ";
+    const link = document.createElement("a");
+    link.href = "/admin/mirror?tab=held";
+    link.textContent = "follow it under Held";
+    note.appendChild(link);
     btn.textContent = "Started";
-    followHeld();
   } catch (err) {
     note.textContent = "Could not reach the gateway: " + err;
     note.classList.add("bad");
@@ -317,28 +339,6 @@ document.querySelector("table.explore").addEventListener("submit", async functio
     btn.textContent = was;
   }
 });
-
-// Refreshes only the mirror's own table, every five seconds while anything
-// in it is downloading.
-let following = false;
-async function followHeld() {
-  if (following) return;
-  following = true;
-  try {
-    for (;;) {
-      const r = await fetch(location.pathname + location.search, { headers: { Accept: "text/html" } });
-      if (!r.ok) break;
-      const doc = new DOMParser().parseFromString(await r.text(), "text/html");
-      const fresh = doc.getElementById("held"), held = document.getElementById("held");
-      if (!fresh || !held) break;
-      held.replaceWith(fresh);
-      if (!fresh.querySelector(".state-starting")) break;
-      await new Promise((ok) => setTimeout(ok, 5000));
-    }
-  } finally {
-    following = false;
-  }
-}
 </script>
 {{if .Hidden}}<p class="note">{{.Hidden}} more {{if eq .Hidden 1}}is{{else}}are{{end}} hidden: their architecture is one the nodes' llama.cpp cannot load.
 Untick “only what these nodes can run” to see them.</p>{{end}}
@@ -354,10 +354,11 @@ directory, or give the mirror a token.</p>
 {{if .Repo}}{{template "mirror_repo" .}}{{end}}
 {{end}}
 
-<h2>Fetch a model</h2>
+{{if eq .Tab "fetch"}}
 <p class="note">By hand, for anything the search does not reach.</p>
 <form method="post" action="/admin/mirror/fetch">
   <input type="hidden" name="csrf" value="{{.CSRF}}">
+  <input type="hidden" name="tab" value="fetch">
   <label>Source URL
     <input type="text" name="source" required
            placeholder="https://huggingface.co/…/resolve/main/model.gguf"></label>
@@ -373,7 +374,7 @@ directory, or give the mirror a token.</p>
 </form>
 <p class="note">The mirror answers as soon as the download is under way, not when
 it finishes — a model is tens of gigabytes and nothing here holds a connection
-open for that. The row above will show it arriving. A version is never
+open for that. The Held tab shows it arriving. A version is never
 overwritten: fetching into one that exists is refused, so a new build is a new
 version and the old weights stay where they are.</p>
 {{end}}
@@ -400,7 +401,7 @@ version and the old weights stay where they are.</p>
 </details>
 {{end}}
 {{if .RepoSets}}
-<table class="repo-files">
+<table class="rows repo-files">
 <tr><th>What</th><th class="num">Size</th><th>Published checksum</th><th></th></tr>
 {{range .RepoSets}}<tr>
   <td><code>{{.Name}}</code>
@@ -410,11 +411,12 @@ version and the old weights stay where they are.</p>
   <td>
     {{if not .Verdict.Servable}}
     <span class="state state-failed" title="{{.Verdict.Reason}}">cannot be served</span>
-    <div class="note">{{.Verdict.Reason}}</div>
+    <span class="clip-inline" title="{{.Verdict.Reason}}">{{.Verdict.Reason}}</span>
     {{else}}
     {{if .Verdict.Architecture}}<span class="opt" title="{{.Verdict.Reason}}">{{.Verdict.Architecture}}{{if .Verdict.Checked}} ✓{{end}}</span>{{end}}
     <form method="post" action="/admin/mirror/fetch" class="inline">
       <input type="hidden" name="csrf" value="{{$.CSRF}}">
+      <input type="hidden" name="tab" value="explore">
       {{range .Sources}}<input type="hidden" name="source" value="{{.URL}}">
       <input type="hidden" name="file" value="{{.Path}}">
       {{end}}
@@ -439,3 +441,93 @@ computes once it has it.</p>
 <p class="empty">No weight files in {{.Repo.ID}}.</p>
 {{end}}
 {{end}}
+
+{{/* The card beneath a held version: what runs it, where, and how many. */}}
+{{define "mirror_held_detail"}}{{$row := .Row}}{{$csrf := .CSRF}}{{$scopes := .D.Scopes}}
+<div class="held-card">
+<p class="repo-meta">
+  <span>{{if gt $row.Files 1}}{{$row.Files}} files{{else}}<code>{{$row.File}}</code>{{end}}</span>
+  <span>{{bytes $row.BytesDone}}</span>
+  {{with $row.SHA256}}<span>sha256 <code title="{{.}}">{{slice . 0 12}}</code></span>{{end}}
+  <span><code>{{$row.URL}}</code></span>
+</p>
+{{if not .D.Models}}
+<p class="note">Not in the catalog: nothing runs this version.
+{{if $row.Serve}}{{else}}Use <strong>serve</strong> on its row to put it into service.{{end}}</p>
+{{end}}
+{{range .D.Models}}
+<section class="held-model">
+<h3>{{.Name}}
+  {{with .View}}<span class="opt">{{.Ready}} of {{.Desired}} ready{{if ne .Total .Ready}} · {{.Total}} in all{{end}}</span>
+  {{if .Parked}}<span class="tag bad">parked</span>{{else if .Unplaceable}}<span class="tag bad">does not fit</span>{{end}}{{end}}
+</h3>
+{{with .View.Reason}}<p class="note error-row">{{.}}</p>{{end}}
+
+{{if .Replicas}}
+<table class="rows compact replicas">
+<tr><th>Replica</th><th>Node</th><th>State</th><th class="num">Age</th><th class="num">CPU</th><th class="num">Memory</th></tr>
+{{range .Replicas}}{{$rep := .}}{{range $i, $s := .Nodes}}<tr>
+  <td>{{if eq $i 0}}<code>{{$rep.ID}}</code>{{end}}</td>
+  <td><a href="/admin/nodes/{{$s.Node}}">{{$s.Node}}</a>{{if $s.MemoryGB}} <span class="opt">{{$s.MemoryGB}} GB</span>{{end}}</td>
+  <td>{{if eq $i 0}}<span class="state state-{{if eq $rep.State "ready"}}ready{{else if eq $rep.State "failed"}}failed{{else}}starting{{end}}">{{$rep.State}}</span>
+    {{with $rep.Restart}}<span class="opt">restart {{.}}</span>{{end}}{{end}}</td>
+  <td class="num">{{if eq $i 0}}{{age $rep.Age}}{{end}}</td>
+  <td class="num">{{if $s.CPU}}{{printf "%.1f" $s.CPU}} cores{{else}}—{{end}}</td>
+  <td class="num">{{if $s.MemBytes}}{{bytes $s.MemBytes}}{{else}}—{{end}}</td>
+</tr>{{end}}{{end}}
+</table>
+{{else}}<p class="note">No replica runs now{{if eq .Mode "ondemand"}}: it starts when somebody asks for it{{end}}.</p>{{end}}
+
+{{with .Headroom}}
+<p class="headroom">
+{{if .Limited}}Room for at least {{.More}} more replicas.
+{{else if .More}}Room for {{.More}} more now{{if .Nodes}}, on {{range $i, $n := .Nodes}}{{if $i}}, {{end}}{{join $n}}{{end}}{{end}}.
+{{else}}No free room for another replica now.{{end}}
+{{if .ByPreempting}}<br><span class="note">{{if .More}}And {{end}}{{.ByPreempting}} {{if .More}}more {{end}}by stopping replicas its class outranks: {{join .Preempts}}.</span>{{end}}
+</p>
+{{if and (not .Limited) .Blocked}}<details class="blocked"><summary>Why not {{if .More}}more{{else}}another{{end}}</summary>
+<ul>{{range .Blocked}}<li>{{.}}</li>{{end}}</ul></details>{{end}}
+{{end}}
+
+{{if .Declared}}
+<p class="note">Set in server.yaml, which is read again on every start: change its replicas there.</p>
+{{else}}
+<form method="post" action="/admin/mirror/replicas" class="replicas-form">
+  <input type="hidden" name="csrf" value="{{$csrf}}">
+  <input type="hidden" name="model" value="{{.Name}}">
+  <input type="hidden" name="open" value="{{$row.Open}}">
+  <div class="field-row">
+    <label>Always running
+      <input type="number" name="min" value="{{.Min}}" min="0"{{if .MostNow}} max="{{.MostNow}}"{{end}} required></label>
+    <label>Up to, under load
+      <input type="number" name="max" value="{{.Max}}" min="0" max="64" required></label>
+    <label>Class
+      <select name="class">
+        <option value=""{{if eq .Class ""}} selected{{end}}>default ({{if eq .Mode "ondemand"}}opportunistic{{else}}standard{{end}})</option>
+        <option value="critical"{{if eq .Class "critical"}} selected{{end}}>critical</option>
+        <option value="standard"{{if eq .Class "standard"}} selected{{end}}>standard</option>
+        <option value="opportunistic"{{if eq .Class "opportunistic"}} selected{{end}}>opportunistic</option>
+      </select></label>
+    {{if $scopes}}{{$owner := .Owner}}<label>Owner
+      <select name="owner">
+        <option value=""{{if eq $owner ""}} selected{{end}}>shared</option>
+        {{range $scopes}}<option value="{{.Key}}"{{if eq .Key $owner}} selected{{end}}>{{.Name}}</option>{{end}}
+      </select></label>{{end}}
+  </div>
+  <div class="field-row">
+    <label>Memory GB <span class="opt">per replica</span>
+      <input type="number" name="memory" value="{{.MemoryGB}}" min="0"></label>
+    <label>Cores <span class="opt">per replica, its threads</span>
+      <input type="number" name="cores" value="{{.CPUCores}}" min="0"></label>
+    <button type="submit">Save</button>
+  </div>
+  <p class="note">{{if .MostNow}}At most {{.MostNow}} can always run now{{if .Headroom}}{{if .Headroom.ByPreempting}}, counting what its class may stop to make room{{end}}{{end}}. {{end}}0 always running with a
+  higher limit loads it when asked for; 0 and 0 stops it. A replica is added under load when requests wait, and
+  taken away after ten quiet minutes. A higher class may stop replicas of lower ones to make room; memory and
+  cores apply to replicas started after the change.</p>
+</form>
+{{end}}
+</section>
+{{end}}
+</div>
+{{end}}

internal/web/templates/admin_quotas.html

@@ -34,7 +34,7 @@ budgets bind.</p>
 {{/* The tree is the interface. A quota belongs to a branch, so the form to set
      one opens on that branch rather than asking somebody to name it again in a
      box at the bottom. */}}
-<table class="quota-tree">
+<table class="rows quota-tree">
 <tr><th>Where</th><th class="num">Requests/min</th><th class="num">Tokens/min</th>
     <th class="num">Budget</th><th>Spent</th><th></th></tr>
 {{range .Rows}}
@@ -119,7 +119,7 @@ up — hover it to see where. A value set here is in normal type.</p>
 <p class="note">A person's own limit covers everything they do, across all their
 keys, projects and the chat page. {{.Personal}}</p>
 {{if .People}}
-<table class="quota-tree people-limits">
+<table class="rows quota-tree people-limits">
 <tr><th>Person</th><th class="num">Requests/min</th><th class="num">Tokens/min</th>
     <th class="num">Budget</th><th>Spent</th><th></th></tr>
 {{range .People}}
@@ -147,6 +147,60 @@ keys, projects and the chat page. {{.Personal}}</p>
 </table>
 {{else}}<p class="empty">Nobody has been approved yet.</p>{{end}}
 
+{{if .HasCapacity}}
+<h2 id="capacity">Capacity</h2>
+<p class="note">What a part of the tree is guaranteed of the cluster, and the most
+it may hold: GPUs, CPU cores and memory, counted from the replicas of the models
+it owns and of those beneath it. What it holds within its guarantee is never taken
+for someone else's model; beyond it, it borrows free room and gives it back when
+asked. A model's owner is set on its card on the mirror page.</p>
+{{if .Capacity}}
+<table class="rows quota-tree">
+<tr><th>Where</th><th class="num">GPUs</th><th class="num">Cores</th><th class="num">Memory GB</th><th>Set by</th><th></th></tr>
+{{range .Capacity}}<tr>
+  <td class="q-where"><span class="q-name">{{or .Name .Scope}}</span></td>
+  <td class="num">{{or .GPUs "—"}}</td>
+  <td class="num">{{or .Cores "—"}}</td>
+  <td class="num">{{or .Memory "—"}}</td>
+  <td class="note">{{.UpdatedBy}}</td>
+  <td class="q-act"><a href="/admin/quotas?cap={{.Scope}}#capacity">change</a>
+    <form method="post" action="/admin/quotas/capacity" class="inline">
+      <input type="hidden" name="csrf" value="{{$.CSRF}}">
+      <input type="hidden" name="scope" value="{{.Scope}}">
+      <button type="submit" name="clear" value="1" class="link danger">remove</button>
+    </form></td>
+</tr>{{end}}
+</table>
+<p class="q-legend">Each figure is held of guaranteed (at most the limit); — is not governed.</p>
+{{else}}<p class="empty">No capacity quotas: every model shares the cluster by its class alone.</p>{{end}}
+
+{{if .Scopes}}
+<form method="post" action="/admin/quotas/capacity" class="q-form capacity-form">
+  <input type="hidden" name="csrf" value="{{.CSRF}}">
+  <p class="q-form-title"><strong>{{if .CapForm.Scope}}Change{{else}}Set{{end}} a capacity quota.</strong>
+    A blank field is not governed; 0 is none at all.</p>
+  <label>Where
+    <select name="scope" required>
+      <option value="">choose…</option>
+      {{range .Scopes}}<option value="{{.Key}}"{{if eq .Key $.CapForm.Scope}} selected{{end}}>{{.Name}}</option>{{end}}
+    </select></label>
+  <div class="field-row">
+    <label>Guaranteed GPUs <input type="text" name="gpus" value="{{.CapForm.GPUs}}" placeholder="—"></label>
+    <label>Cores <input type="text" name="cores" value="{{.CapForm.Cores}}" placeholder="—"></label>
+    <label>Memory GB <input type="text" name="memory" value="{{.CapForm.Memory}}" placeholder="—"></label>
+  </div>
+  <div class="field-row">
+    <label>At most GPUs <input type="text" name="limit_gpus" value="{{.CapForm.LGPUs}}" placeholder="—"></label>
+    <label>Cores <input type="text" name="limit_cores" value="{{.CapForm.LCores}}" placeholder="—"></label>
+    <label>Memory GB <input type="text" name="limit_memory" value="{{.CapForm.LMem}}" placeholder="—"></label>
+  </div>
+  <div class="q-form-actions"><button type="submit">Save</button>
+    {{if .CapForm.Scope}}<a href="/admin/quotas#capacity">Cancel</a>{{end}}</div>
+</form>
+{{else}}<p class="empty">Nothing to give a capacity quota to yet. Make an organization on the
+<a href="/admin/users">people page</a> first.</p>{{end}}
+{{end}}
+
 <details class="explain">
   <summary>How limits combine</summary>
   <p>Along a branch the most restrictive value wins, so a team cannot raise what

internal/web/templates/admin_usage.html

@@ -47,7 +47,7 @@
 </div>
 
 {{if .Rows}}
-<table>
+<table class="rows">
   <tr><th>Group</th><th class="num">Requests</th><th class="num">Tokens</th><th class="num">Spend</th><th class="num">Errors</th></tr>
   {{range .Rows}}
   <tr>

internal/web/templates/admin_users.html

@@ -22,7 +22,7 @@
 
 {{if .Pending}}
 <h2>Waiting for approval</h2>
-<table><tr><th>Email</th><th>Name</th><th>Signed up</th><th>Approve, and put them in</th><th></th></tr>
+<table class="rows"><tr><th>Email</th><th>Name</th><th>Signed up</th><th>Approve, and put them in</th><th></th></tr>
 {{range .Pending}}<tr>
   <td>{{.Email}}</td><td>{{.Name}}</td><td>{{ago .CreatedAt}}</td>
   <td>
@@ -149,14 +149,14 @@ project in the team — an account can only be approved into a project.</p>
         </div>
 
         {{with index $.Members (printf "project:%s" $p.ID)}}
-        <table class="compact"><tr><th>Member</th><th>State</th><th>Last sign-in</th><th></th></tr>
+        <table class="rows compact"><tr><th>Member</th><th>State</th><th>Last sign-in</th><th></th></tr>
         {{range .}}{{template "member-row" (dict "U" . "D" $ "Scope" (printf "project:%s" $p.ID) "Open" $p.ID)}}{{end}}
         </table>
         {{else}}<p class="empty">No members of its own. Members of its team or organization can use it too.</p>{{end}}
         {{template "add-member" (dict "Kind" "project" "ID" $p.ID "D" $ "Open" $p.ID)}}
 
         {{with index $.Keys $p.ID}}
-        <table class="compact"><tr><th>Key</th><th>Owner</th><th>State</th><th>Last used</th><th></th></tr>
+        <table class="rows compact"><tr><th>Key</th><th>Owner</th><th>State</th><th>Last used</th><th></th></tr>
         {{range .}}<tr>
           <td>{{.Name}} <code>{{.Prefix}}</code></td>
           <td>{{or .OwnerEmail "issued by an operator"}}</td>
@@ -210,7 +210,7 @@ project in the team — an account can only be approved into a project.</p>
 
 <h2 id="accounts">All accounts</h2>
 {{if .Accounts}}
-<table id="account-list"><tr><th>Person</th><th>State</th><th>Member of</th><th>Last sign-in</th><th>Add to</th><th></th></tr>
+<table class="rows" id="account-list"><tr><th>Person</th><th>State</th><th>Member of</th><th>Last sign-in</th><th>Add to</th><th></th></tr>
 {{range .Accounts}}{{template "account-row" (dict "U" . "D" $)}}{{end}}
 </table>
 <p class="note">Someone who is a member of nothing uses the gateway on their own:
@@ -298,7 +298,7 @@ access than it had, so they need a shell on the control plane —
   <td>{{$u.Email}}{{if $u.IsAdmin}} <span class="tag">admin</span>{{end}}</td>
   <td class="state state-{{$u.State}}">{{$u.State}}</td>
   <td class="where">
-    {{range $u.Memberships}}<span class="chip-m">{{.Path}}{{if eq .Role "manager"}} <b class="chip-role">manager</b>{{end}}
+    {{range $i, $m := $u.Memberships}}{{if lt $i 2}}{{with $m}}<span class="chip-m">{{.Path}}{{if eq .Role "manager"}} <b class="chip-role">manager</b>{{end}}
       {{if eq (printf "%s" .Scope) "org"}}
       <form method="post" action="/admin/users/{{$u.ID}}/role" class="inline">
         <input type="hidden" name="csrf" value="{{$d.CSRF}}">
@@ -313,8 +313,12 @@ access than it had, so they need a shell on the control plane —
         <input type="hidden" name="csrf" value="{{$d.CSRF}}">
         <input type="hidden" name="scope" value="{{.Scope}}:{{.ScopeID}}">
         <button type="submit" class="chip-x" aria-label="Take out of {{.Path}}" title="Take out of {{.Path}}">×</button>
-      </form></span>
+      </form></span>{{end}}{{end}}
     {{else}}<span class="opt">on their own</span>{{end}}
+    {{/* Two, then a count: a person in six teams was six chips wrapping the
+         row into a paragraph. The rest are named on hover and in their own
+         scopes' member lists, where taking them out is done. */}}
+    {{if gt (len $u.Memberships) 2}}<span class="opt more" title="{{range $i, $m := $u.Memberships}}{{if ge $i 2}}{{$m.Path}}&#10;{{end}}{{end}}">+{{sub (len $u.Memberships) 2}}</span>{{end}}
   </td>
   <td>{{ago $u.LastLoginAt}}</td>
   <td>
@@ -365,7 +369,7 @@ access than it had, so they need a shell on the control plane —
 <details class="edit"><summary>{{len (index $d.Members $k)}} member{{if ne (len (index $d.Members $k)) 1}}s{{end}}</summary>
   <div class="scope-members">
   {{with index $d.Members $k}}
-  <table class="compact">
+  <table class="rows compact">
   {{range .}}{{template "member-row" (dict "U" . "D" $d "Scope" $k "Open" "")}}{{end}}
   </table>
   {{else}}<p class="note">Nobody is a member of {{.Name}} itself; a member here could use every project in it.</p>{{end}}

internal/web/templates/dashboard.html

@@ -51,23 +51,23 @@
 
 <h2>Nodes</h2>
 {{if .Nodes}}
-<table class="nodes"><tr><th>Node</th><th>State</th><th>CPU</th><th>RAM</th><th class="num">GPUs</th><th class="num">Disk</th><th>Engines</th><th>Classes</th><th>Fabrics</th><th class="num">Models</th><th class="num">Replicas</th><th>Last seen</th></tr>
+<table class="nodes rows"><tr><th>Node</th><th>State</th><th>CPU</th><th>RAM</th>{{if .AnyGPUs}}<th class="num">GPUs</th>{{end}}<th class="num">Disk</th><th>Engines</th>{{if .AnyClasses}}<th>Classes</th>{{end}}{{if .AnyFabrics}}<th>Fabrics</th>{{end}}<th class="num">Models</th><th class="num">Replicas</th><th>Seen</th></tr>
 {{range .Nodes}}<tr>
-  <td><a href="{{if $.OnAdminListener}}/nodes/{{.ID}}{{else}}/admin/nodes/{{.ID}}{{end}}"><code>{{.ID}}</code></a>{{if .MgmtAddr}} <span class="note">{{.MgmtAddr}}</span>{{end}}
+  <td><a href="{{if $.OnAdminListener}}/nodes/{{.ID}}{{else}}/admin/nodes/{{.ID}}{{end}}" title="{{.ID}}"><code>{{shortNode .ID $.NodePrefix}}</code></a>{{if .MgmtAddr}} <span class="note">{{.MgmtAddr}}</span>{{end}}
     <a class="note" href="{{if $.OnAdminListener}}/nodes/{{.ID}}/logs{{else}}/admin/nodes/{{.ID}}/logs{{end}}">logs</a></td>
   <td class="state state-{{.State}}">{{.State}}</td>
   {{/* A dash, not an empty bar, when the agent has not measured it lately:
        unknown is not idle. */}}
   <td class="load">{{with .Load}}{{template "minibar" pctbar .CPUPercent}}<span class="load-v">{{printf "%.0f" .CPUPercent}}%{{if .CPUs}} <span class="note">of {{.CPUs}}</span>{{end}}</span>{{else}}—{{end}}</td>
   <td class="load">{{with .Load}}{{template "minibar" pctbar .MemPercent}}<span class="load-v">{{bytes .MemUsed}} <span class="note">of {{bytes .MemTotal}}</span></span>{{else}}{{if .MemoryGB}}{{.MemoryGB}}G{{else}}—{{end}}{{end}}</td>
-  <td class="num">{{if .GPUTotal}}{{.GPUFree}}/{{.GPUTotal}}{{else}}—{{end}}</td>
+  {{if $.AnyGPUs}}<td class="num">{{if .GPUTotal}}{{.GPUFree}}/{{.GPUTotal}}{{else}}—{{end}}</td>{{end}}
   {{/* Free out of total, because the first is meaningless without the
        second, and an em dash where nothing was reported: an agent that
        was never asked about its disk must not read as a full one. */}}
   <td class="num">{{if .DiskTotalGB}}{{.DiskFreeGB}}G/{{.DiskTotalGB}}G{{else}}—{{end}}</td>
   <td>{{if .Engines}}{{join .Engines}}{{else}}<span class="note">none yet</span>{{end}}</td>
-  <td>{{join .Classes}}</td>
-  <td>{{join .Fabrics}}</td>
+  {{if $.AnyClasses}}<td>{{join .Classes}}</td>{{end}}
+  {{if $.AnyFabrics}}<td>{{join .Fabrics}}</td>{{end}}
   <td class="num">{{if .Models}}{{len .Models}}{{else}}—{{end}}</td>
   <td class="num">{{.Replicas}}</td>
   <td>{{or .LastSeenAgo "—"}}</td>
@@ -80,18 +80,21 @@
 <h2>Models</h2>
 
 {{if .Models}}
-<table><tr><th>Model</th><th>Aliases</th><th class="num">Ready</th><th class="num">Wanted</th><th>Profiles</th><th>Fallbacks</th></tr>
+<table class="rows"><tr><th>Model</th><th>Aliases</th><th class="num">Ready</th><th class="num">Wanted</th><th>Class</th><th>Reserves</th><th>Profiles</th>{{if .AnyFallbacks}}<th>Fallbacks</th>{{end}}<th>What it is for</th></tr>
 {{range .Models}}<tr>
   <td><code>{{.Name}}</code></td>
   <td>{{join .Aliases}}</td>
   <td class="num">{{.Ready}}</td>
   <td class="num">{{.Min}}–{{.Max}}</td>
+  <td{{if .HeldFor}} class="warn" title="gave room to {{.HeldFor}}"{{end}}>{{.Class}}{{if .Preempted}} <span class="note" title="replicas lost to a higher class">−{{.Preempted}}</span>{{end}}</td>
+  <td{{with .Warnings}} class="warn" title="{{join .}}"{{end}}>{{if or .MemoryGB .CPUCores}}{{.MemoryGB}}G{{if .CPUCores}} · {{.CPUCores}}c{{end}}{{else}}—{{end}}
+    {{- if .MeasuredMemoryGB}} <span class="note">{{printf "%.1f" .MeasuredMemoryGB}}G used</span>{{end}}{{if .Warnings}} ⚠{{end}}</td>
   <td>{{join .Profiles}}</td>
-  <td>{{join .Fallbacks}}</td>
+  {{if $.AnyFallbacks}}<td>{{join .Fallbacks}}</td>{{end}}
+  {{/* Cut to the column, the whole sentence on hover: a row of its own for
+       each description made the table twice as long as the catalog. */}}
+  <td class="clip"{{with .Description}} title="{{.}}"{{end}}>{{or .Description "—"}}</td>
 </tr>
-{{/* Under the row rather than in a column: a sentence in a table cell either
-     wraps every other column into a stripe or gets cut off. */}}
-{{if .Description}}<tr><td colspan="6" class="url-row">{{.Description}}</td></tr>{{end}}
 {{end}}</table>
 {{else}}<p class="empty">The catalog is empty.</p>{{end}}
 
@@ -106,7 +109,7 @@
 
 <h2>Replicas</h2>
 {{if .Replicas}}
-<table><tr><th>Replica</th><th>Model</th><th>State</th><th>Pool</th><th>Endpoint</th><th class="num">Age</th><th>Nodes</th></tr>
+<table class="rows"><tr><th>Replica</th><th>Model</th><th>State</th><th>Pool</th><th>Endpoint</th><th class="num">Age</th><th>Nodes</th></tr>
 {{range .Replicas}}<tr>
   <td><code>{{.ID}}</code></td>
   <td>{{.Model}}</td>
@@ -114,7 +117,7 @@
   <td>{{or .Pool "—"}}</td>
   <td>{{if .Endpoint}}<code>{{.Endpoint}}</code>{{else}}—{{end}}</td>
   <td class="num">{{duration .AgeSec}}</td>
-  <td>{{range .Shards}}<code>{{.Node}}</code> {{end}}</td>
+  <td>{{range .Shards}}<code title="{{.Node}}">{{shortNode .Node $.NodePrefix}}</code> {{end}}</td>
 </tr>{{if .Err}}<tr><td colspan="7" class="note">{{.Err}}</td></tr>{{end}}
 {{end}}</table>
 {{else}}<p class="empty">No replicas.</p>{{end}}

internal/web/templates/node.html

@@ -21,6 +21,10 @@
   <div class="tile"><div class="v">{{printf "%.0f" .CPUPercent}}%</div><div class="k">CPU{{if .CPUs}} · {{.CPUs}} cores{{end}}</div>{{template "minibar" pctbar .CPUPercent}}</div>
   <div class="tile"><div class="v">{{bytes .MemUsed}}</div><div class="k">RAM in use · of {{bytes .MemTotal}}</div>{{template "minibar" pctbar .MemPercent}}</div>
   <div class="tile"><div class="v">{{printf "%.2f" .Load1}}</div><div class="k">load · {{printf "%.2f" .Load5}} 5m · {{printf "%.2f" .Load15}} 15m</div></div>
+  {{/* Only what this machine has sensors for: a VM has none, and a tile
+       saying 0 °C would be a reading nobody took. */}}
+  {{if or .CPUTempC .CPUPowerW}}<div class="tile"><div class="v">{{if .CPUTempC}}{{printf "%.0f" .CPUTempC}} °C{{else}}—{{end}}</div><div class="k">CPU{{if .CPUPowerW}} · {{printf "%.0f" .CPUPowerW}} W{{end}}</div></div>{{end}}
+  {{if .InletTempC}}<div class="tile"><div class="v">{{printf "%.1f" .InletTempC}} °C</div><div class="k">inlet air</div></div>{{end}}
   {{else}}
   <div class="tile"><div class="v">—</div><div class="k">CPU and RAM not measured lately</div></div>
   {{end}}
@@ -29,13 +33,24 @@
 </div>
 {{if .Warnings}}{{range .Warnings}}<p class="note bad">{{.}}</p>{{end}}{{end}}
 {{if .Unhealthy}}<p class="note bad">{{join .Unhealthy}}</p>{{end}}
+{{with .Load}}{{if .GPUs}}
+<h2>GPUs</h2>
+<table class="rows"><tr><th class="num">GPU</th><th class="num">Temperature</th><th class="num">Memory</th><th>Power</th><th>Held back by</th></tr>
+{{range .GPUs}}<tr>
+  <td class="num">{{.Index}}</td>
+  <td class="num">{{if .TempC}}{{printf "%.0f" .TempC}} °C{{else}}—{{end}}</td>
+  <td class="num">{{if .MemoryTempC}}{{printf "%.0f" .MemoryTempC}} °C{{else}}—{{end}}</td>
+  <td class="load">{{if and .PowerW .PowerLimitW}}{{template "minibar" pctbar (percent .PowerW .PowerLimitW)}}<span class="load-v">{{printf "%.0f" .PowerW}} W <span class="note">of {{printf "%.0f" .PowerLimitW}}</span></span>{{else if .PowerW}}{{printf "%.0f" .PowerW}} W{{else}}—{{end}}</td>
+  <td>{{if .Throttle}}<span class="bad">{{join .Throttle}}</span>{{else}}<span class="note">nothing</span>{{end}}</td>
+</tr>{{end}}</table>
+{{end}}{{end}}
 {{end}}
 
 {{if or .Node.IBPorts .Node.Fabrics}}
 <h2>Fabric</h2>
 <p class="sub">{{if .Node.Fabrics}}On {{join .Node.Fabrics}} · <a href="{{if $.OnAdminListener}}/fabrics{{else}}/admin/fabrics{{end}}">the map →</a>{{else}}<span class="bad">On no fabric: no port a declared fabric names can carry traffic, so no multi-node replica is placed here.</span>{{end}}</p>
 {{if .Node.IBPorts}}
-<table><tr><th>Adapter</th><th>State</th><th class="num">Rate</th><th>Fabric</th><th>Physically on</th><th>Errors since boot</th><th class="num">Sent</th><th class="num">Received</th></tr>
+<table class="rows"><tr><th>Adapter</th><th>State</th><th class="num">Rate</th><th>Fabric</th><th>Physically on</th><th>Errors since boot</th><th class="num">Sent</th><th class="num">Received</th></tr>
 {{range .Node.IBPorts}}<tr>
   <td><code>{{.HCA}}/{{.Port}}</code>{{if eq .LinkLayer "Ethernet"}} <span class="tag">RoCE</span>{{end}}</td>
   <td class="{{if .Active}}state-ready{{else}}bad{{end}}">{{.State}}{{with .PhysState}} <span class="note">{{.}}</span>{{end}}</td>
@@ -60,13 +75,13 @@ the adapter came up: a number that grows is the one to look at.</p>
   {{if or .Node.NVLinkDomain .Node.IMEX}} · IMEX {{if eq .Node.IMEX "active"}}<span class="state-ready">active</span>, {{.Node.IMEXChannels}} channel{{if ne .Node.IMEXChannels 1}}s{{end}}{{else}}<span class="bad">{{or .Node.IMEX "not installed"}}</span>{{end}}{{end}}
 </p>
 <div class="gpu-wiring">
-<table class="nvl-matrix" aria-label="NVLink between GPUs">
+<table class="rows nvl-matrix" aria-label="NVLink between GPUs">
   <tr><th></th>{{range .Node.GPUs}}<th>{{.Index}}</th>{{end}}</tr>
   {{range $a := .Node.GPUs}}<tr><th>GPU {{$a.Index}}</th>
     {{range $b := $.Node.GPUs}}{{if eq $a.Index $b.Index}}<td class="self"></td>{{else if hasInt $a.NVLinkPeers $b.Index}}<td class="nv" title="GPU {{$a.Index}} ↔ GPU {{$b.Index}} over NVLink">NV</td>{{else}}<td class="pcie" title="GPU {{$a.Index}} ↔ GPU {{$b.Index}} over PCIe">·</td>{{end}}{{end}}
   </tr>{{end}}
 </table>
-<table class="compact gpu-list">
+<table class="rows compact gpu-list">
   <tr><th>GPU</th><th>Name</th><th class="num">NUMA</th><th>Nearest adapter</th><th>Multi-node NVLink</th></tr>
   {{range .Node.GPUs}}<tr>
     <td>{{.Index}}</td><td>{{.Name}}</td><td class="num">{{.NUMA}}</td>
@@ -82,7 +97,7 @@ traffic between nodes should go: PIX and PXB are under the same PCIe switch, SYS
 
 <h2>What runs here</h2>
 {{if .Engines}}
-<table><tr><th>Model</th><th>Replica</th><th>State</th><th>CPU</th><th class="num">Memory</th></tr>
+<table class="rows"><tr><th>Model</th><th>Replica</th><th>State</th><th>CPU</th><th class="num">Memory</th></tr>
 {{range .Engines}}<tr>
   <td>{{.Model}}</td>
   <td><code>{{.Replica}}</code></td>

internal/web/templates/org.html

@@ -42,7 +42,7 @@
 {{end}}
 
 {{if .Rows}}
-<table class="compact">
+<table class="rows compact">
   <tr><th>{{if eq .By "user"}}Person{{else if eq .By "project"}}Project{{else}}Team{{end}}</th>
       <th class="num">Requests</th><th class="num">Tokens</th><th class="num">Spend</th><th class="num">Errors</th></tr>
   {{range .Rows}}<tr>
@@ -56,7 +56,7 @@ people do elsewhere — in another organization, or on their own — is not show
 
 <h2>Teams, projects and limits</h2>
 {{if .Quotas}}
-<table class="quota-tree">
+<table class="rows quota-tree">
 <tr><th>Where</th><th class="num">Requests/min</th><th class="num">Tokens/min</th>
     <th class="num">Budget</th><th>Spent · 30 days</th></tr>
 {{range .Quotas}}
@@ -79,7 +79,7 @@ people do elsewhere — in another organization, or on their own — is not show
 
 <h2>Members</h2>
 {{if .Members}}
-<table class="compact">
+<table class="rows compact">
 <tr><th>Person</th><th>Member of</th><th>State</th></tr>
 {{range .Members}}<tr>
   <td>{{.Email}}</td>

internal/web/web.go

@@ -109,6 +109,16 @@ var funcs = template.FuncMap{
 		return p
 	},
 	"ago": ago,
+	// percent is part of whole, 0 to 100, for a bar.
+	"percent": func(part, whole float64) float64 {
+		if whole <= 0 {
+			return 0
+		}
+		return part * 100 / whole
+	},
+	// shortNode is a node's name without what every node's name starts with.
+	"shortNode": strings.TrimPrefix,
+	"age":       age,
 	// pctbar is a percentage as the minibar template takes it: whole, and
 	// held to 0..100 so a measurement a little over does not draw past its box.
 	"pctbar": func(v float64) struct{ P int } {

llmgw-plan.md

@@ -1463,8 +1463,8 @@ The launch path is now built too:
 - [x] `ctl bench`: a load ramp that reports capacity as the last concurrency
       level meeting the SLO, plus the queue depth and KV usage observed there,
       so `weight`, `target_waiting` and `kv_ceiling` stop being guesses
-- [ ] `ctl plan <model> --target-rps N`, which needs capacity profiles stored
-      rather than printed
+- [x] Sizing from stored capacity: `ctl model target <model> --tokens-per-sec N`,
+      and the autoscaler and profile choice use what `ctl bench --save` measured
 - [x] OTel tracing: an inference span with admission, schedule and each
       upstream attempt under it, over OTLP/HTTP when `tracing.endpoint` is
       set, and the traceparent carried on to the engine
@@ -1477,10 +1477,21 @@ The launch path is now built too:
 ### Phase 6 — v2 ideas
 - [ ] Precise KV index (§7.5 tier 2) — vLLM KV events plus tokenizer integration
 - [ ] Dynamo-style block cost function, tuning the `softmax` picker
-- [ ] Admission displacement (dropping requests from lower bands)
+- [x] Admission displacement: a total queue bound, and the newest request of
+      the lowest band turned away (503, Retry-After) for a higher one
 - [x] Autoscaling on load: a replica added while requests wait, up to a
       model's max, and taken back after ten quiet minutes
-- [ ] Priority-based preemption
+- [x] Priority-based preemption: capacity classes (critical, standard,
+      opportunistic), preemption tried on a copy of the inventory first
+- [x] Capacity quotas per organization, team and project: guarantees kept,
+      borrowing taken back, limits enforced
+- [x] CPU cores reserved like memory, and memory required of CPU models
+- [x] Node sensors collected: GPU temperature, power and throttling, CPU
+      temperature and RAPL power, board inlet air; published and shown
+- [ ] Rack power, temperature and cooling as placement signals: planned in
+      [docs/facility-aware-placement.md](docs/facility-aware-placement.md),
+      deferred — it needs each site's rack, PDU and power data first
+- [ ] Defragmentation for large tensor-parallel replicas across NVLink domains
 - [ ] Control plane HA (Raft)
 - [x] Weight downloading and cache management: the mirror, agent fetches
       checked against their digest, and reclaiming what nothing uses