v0.10.6

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Load LoRA adapters at scale zero
2026-10-01 08:52:01 +02:00 · 2026-10-01 07:39:12 +02:00 · 2026-10-01 07:30:13 +02:00 · 2026-10-01 07:20:35 +02:00 · 2026-10-01 07:19:08 +02:00 · 2026-09-30 23:40:47 +02:00
139 changed files with 12608 additions and 1927 deletions
@@ -93,6 +93,24 @@ jobs:
          ./dist/llmgw-${version}-linux-amd64 --version
          cat dist/checksums.txt

+      # Signed when the repository has a signing key, so a machine that trusts
+      # it installs nothing the server alone could have put there. Without the
+      # secret nothing is signed and releases are checked on their checksums,
+      # as before; a build that trusts a key refuses those, so the secret has
+      # to be in place before trusted.pub is filled in.
+      - name: Sign the checksums
+        env:
+          RELEASE_SIGNING_KEY: ${{ secrets.RELEASE_SIGNING_KEY }}
+        run: |
+          set -euo pipefail
+          version="${GITHUB_REF_NAME:-$(git describe --tags --always)}"
+          if [ -z "${RELEASE_SIGNING_KEY:-}" ]; then
+            echo "no RELEASE_SIGNING_KEY secret; releases are not signed"
+            exit 0
+          fi
+          ./dist/llmgw-${version}-linux-amd64 release sign dist/checksums.txt
+          echo "signed: dist/checksums.txt.sig"
+
      - name: Attach to the release
        env:
          TOKEN: ${{ secrets.GITEA_TOKEN }}
@@ -20,10 +20,336 @@ is only released once its **exit criterion** is met, not when its code lands.
 Until v1.0.0 the minor version is the breaking one: configuration formats and Go
 APIs change between minors without a deprecation period.

-## Unreleased
+## v0.10.6 — 2026-10-01

-Changes since v0.10.3, published as release candidates (v0.10.4-rc.N) until
-there are enough for v0.10.4.
+Published first as release candidates v0.10.6-rc.1 to rc.8; rc.5 failed its
+format check and has no binaries.
+
+Tried on the PoC cluster before release: LoRA adapters asked for by name
+through the gateway, two on one model, each answering as trained and the model
+itself unchanged; a saturated pool spilling to the next (8 of 12 concurrent
+requests on the preferred replica, 4 on the other); and a bronze request
+displaced from a full queue by a gold one, with 503 and Retry-After 5. Not tried
+live: a model served from an external provider, which needs a provider key.
+
+### Capacity from the admin pages
+
+What v0.10.5 made settable from the command line is now on the pages too:
+
+- **Quotas** has a Capacity section: every scope's guaranteed and most GPUs,
+  cores and memory beside what it holds, with a form to set, change or remove
+  one. It goes through the same checks as `llmgw ctl quota`.
+- **A model's card** on the mirror page sets its class, its owner (when there
+  are quotas to count against), and the memory and cores a replica reserves,
+  beside its replica counts. Only what changed is applied, and each change is
+  said.
+- **Room counts what a class may take.** The card, `llmgw ctl model target` and
+  the headroom API count the replicas a model could get by stopping those its
+  class outranks, separately from free room, and name whose they would be. A
+  critical model's card no longer refuses a replica that preemption would make
+  room for.
+
+### LoRA adapters
+
+A llama.cpp model can carry `adapters`, each a name and a GGUF LoRA file. The
+nodes fetch them with the model's weights and llama.cpp loads them all at start,
+applied to nothing (`--lora-scaled FILE:0`); a request that names an
+adapter goes to the model's replicas with `"lora": [{"id": i, "scale": s}]`,
+so one replica serves the model and all its adapters. Adapter names share the
+namespace of models, are listed in `/v1/models`, and follow key allowlists on
+their own — a key allowed an adapter does not reach the model under it. Adapter
+files are kept from weight cleanup like the model's own. Not yet for vLLM or
+SGLang profiles, whose adapters are loaded differently.
+
+### Finding and attaching adapters from the mirror page
+
+- **Explore** has a *LoRA adapters* view: adapters for any model, or — from a
+  held model's card, *find adapters on Hugging Face* — those made for the model
+  its repository was made from, since a GGUF repository is a quantisation and
+  adapters are made for the original.
+- **Opening a repository** reads the first bytes of each GGUF file and marks it
+  a *LoRA adapter* or a *model*. Repositories tagged "lora" hold a model with
+  the adapter merged in about as often as an adapter, and only the file says.
+- **Held adapter files** are marked in the list, and a model's card lists its
+  adapters, offers the held adapter files of its architecture to attach under a
+  name and scale, and removes them. A file that is not an adapter, or is for
+  another architecture, is refused before it reaches the catalog.
+- **`llmgw ctl model adapters MODEL [--add NAME=URL[@SCALE]] [--remove NAME]`**
+  lists and changes them from the command line, through
+  `POST /api/v1/models/{name}/adapters`, with the same checks.
+
+### Models served outside the cluster
+
+A catalog model can be an OpenAI-compatible API elsewhere: `external: {url,
+model, api_key_file}` and no profiles. Nothing is placed for it; the router
+sends its requests to the provider, rewritten to the provider's name for the
+model and with its key, read from the file on every publish so a rotated key is
+picked up. Named in another model's `fallbacks`, it is used only when the
+cluster has no capacity left. It is never scraped for metrics, and never reads
+as stale. Prices, budgets, limits, usage and key allowlists apply as to any
+model — and since its prompts leave the cluster, the allowlist is how to decide
+who may send them (docs/install.md). `ctl models` and the cluster page mark it
+external.
+
+### Pool spillover
+
+A model served from pools of different priority — H100 before A100, say — was
+routed to the preferred pool however full it was. Now a pool whose replicas are
+saturated, by the same measure admission uses (queue past its healthy depth, or
+KV cache past its ceiling), passes requests to the next pool that is not. When
+every pool is saturated the preferred one is kept: a pool that is just as full
+gains nothing.
+
+### Displacement: a higher band is not refused while a lower one waits
+
+Each band's queue was bounded, and nothing bounded them together, so a full
+gateway turned an interactive request away while batch requests sat queued
+behind it. `admission.queue_capacity_total` now bounds everything queued —
+by default half of every band's capacity together. Reached, a request is not
+refused while a lower band has any queued: the newest request of the lowest
+band is turned away instead, with a 503 and `Retry-After: 5`, so it backs off
+and returns. The newest, because the oldest have waited longest and are
+nearest their TTL. Counted as `displaced` in the admission rejection metric.
+
+### Releases can be signed
+
+`llmgw release keygen` makes an ed25519 key pair. With the private key in the
+CI secret `RELEASE_SIGNING_KEY`, releases carry `checksums.txt.sig`; with the
+public key in `internal/release/trusted.pub`, every binary built after refuses
+a release not signed with it, wherever it installs one. Off until then, and
+docs/install.md says in which order to turn it on.
+
+### Restarts borrow a lower class's room
+
+A restart places the replacement beside the replica it replaces, and where
+there is no free room it now makes some the way any placement does: by
+stopping a replica its class outranks, which gets its room back once the old
+replica has gone. The in-place restart — the model unserved while its
+replacement loads — is left for when nothing of a lower class is running. On a
+cluster where every model is standard, that is every full one: mark the models
+that can wait `opportunistic` (`llmgw ctl model class`, or the model's card).
+
+## v0.10.5 — 2026-09-30
+
+Published first as release candidates v0.10.5-rc.1 to rc.14; rc.5 failed its
+format check and has no binaries.
+
+After upgrading:
+
+- **Declare what CPU models need.** A CPU model added from now on must say its
+  memory; one stored before this still loads and is flagged in `ctl models`
+  until `llmgw ctl model resources MODEL --memory GB --cores N` is run. Cores
+  are then reserved per replica and become the engine's thread count, for
+  replicas started after the change: `llmgw ctl restart --model MODEL`.
+- **Upgrade the mirror on its own host**, now with one command:
+  `sudo llmgw upgrade v0.10.5 --service llmgw-mirror` (by hand once, for a
+  mirror older than this). `llmgw ctl upgrade` says so at the end.
+
+### CPU cores are reserved, and memory has to be declared
+
+The first step of resource allocation for larger clusters: what a replica
+holds of a node is reserved, not assumed.
+
+- **Cores.** A profile's `cpu_cores` is reserved out of its node's CPUs, as
+  memory is, and is what its engine computes on (llama.cpp's `--threads`). A
+  node holds as many CPU replicas as it has cores for, and two on one node no
+  longer spin against each other. Agents report their CPU count; a node that
+  does not, and a profile that declares no cores, constrain nothing.
+- **Memory is required of new CPU models.** A model with no GPU is all host
+  memory, and one declaring none looked to placement as if it fitted
+  everywhere, any number of times. Adding one without `min_memory_gb` is now
+  refused. Models stored before this still load, and say what is missing.
+- **Declared against measured.** Each model shows what one replica reserves
+  and the most one of its replicas uses — in `ctl models` (RESERVES) and on the
+  cluster page — with a warning when nothing is declared or when it uses more
+  than a tenth over what it reserves.
+- **Setting them.** `llmgw ctl model resources MODEL --memory GB --cores N`
+  (and `POST /api/v1/models/{name}/resources`) for a model added at runtime.
+  `ctl model serve` and the mirror's serve form fill in cores — half the
+  smallest node the model fits on — beside the memory they already estimate.
+  Running replicas keep their old figures until restarted.
+
+The commands are `llmgw ctl model resources` and `llmgw ctl model class`; in
+rc.7 the first was briefly `llmgw ctl resources`.
+
+### Capacity classes, and preemption
+
+A model has a class — `critical`, `standard` or `opportunistic` — saying what
+its capacity is worth against another's when there is not enough for both.
+Unset, an on-demand model is opportunistic and every other model standard,
+which is how the catalog behaved before. `llmgw ctl model class MODEL CLASS`
+sets it.
+
+When a model cannot be placed, replicas are stopped to make room for it, by
+these rules:
+
+- A lower class gives way to a higher one.
+- Within a class, only replicas above a model's minimum — room it grew into
+  under load — and only for a model still below its own minimum.
+- A critical model's minimum is never taken, nor a replica that has only just
+  loaded, nor one spanning several nodes.
+- Idle on-demand replicas go first, then what lower classes grew into, then
+  the same within the class, then lower classes' minimums; the longest unused
+  first within each, and of equally idle ones the smallest.
+- The set is tried on a copy of the inventory first, at most three replicas,
+  and nothing is stopped unless it makes the room. Of a set that does, each
+  replica whose stopping turns out not to be needed is left running, so an
+  idle replica tried first on the wrong node is not stopped with the one that
+  made the room.
+
+A replica stopped this way is drained and given 30 seconds to finish what it
+is serving. Its model is held back for up to five minutes, or until the model
+it gave way to has its replicas, so it does not take the room straight back.
+`ctl models` shows each model's class and "gave room to X" while held; the
+cluster page shows the class and how many replicas were lost; the metric is
+`llmgw_model_preempted_total{model,class}`, and every preemption is logged.
+
+### Capacity quotas for the tenancy tree
+
+An organization, team or project can be guaranteed GPUs, CPU cores and
+memory, and limited to at most some amount of each. What it holds is what the
+replicas of the models it owns hold — and those of the teams and projects
+beneath it. `llmgw ctl model owner MODEL team:Acme/ml` says whose a model is;
+one with no owner is shared and counts against nobody.
+
+- **A guarantee is kept.** A replica within its owner's guarantee is never
+  taken for another scope's model, whatever the classes: a guarantee a higher
+  class elsewhere could take would not be one.
+- **Beyond it is borrowing.** Free room is used; when a scope within its own
+  guarantee needs room, what others hold beyond theirs is taken back first,
+  whatever their class — except a critical model's minimum.
+- **A limit is a ceiling.** A model whose owner would go past a limit anywhere
+  up the tree gets no further replica, and says which limit; headroom and the
+  mirror card stop there too.
+
+`llmgw ctl quota set team:Acme/ml --gpus 8 --limit-gpus 16` (also `--cores`,
+`--memory` and their limits), `llmgw ctl quota list` with what each scope holds,
+`llmgw ctl quota clear`. Scopes are named by path or ID; a name that could mean
+two is refused. Changes take effect within half a minute. Metrics:
+`llmgw_capacity_holds`, `llmgw_capacity_guarantee` and `llmgw_capacity_limit`
+by scope and resource; alert: CapacityAtLimit.
+
+### Sized by what a replica was measured to serve
+
+What `llmgw ctl bench MODEL --save` measured — the most requests at once one
+replica served while meeting its SLO, and its throughput there — now drives
+decisions rather than only being printed:
+
+- **Scaling up** when requests per replica pass what was measured, not only
+  when the engines' queues fill: an engine takes requests into its slots long
+  after latency has gone.
+- **Scaling down** only when the replicas left would stay under 70% of what
+  they were measured to hold, so a replica taken back does not put the rest
+  straight over.
+- **Choosing a profile.** Among profiles of equal priority, the one measured
+  to serve the most per GPU — or per core — is tried first. Priority set by an
+  operator still wins.
+- **Sizing.** `llmgw ctl model target MODEL --tokens-per-sec N` says how many
+  replicas that takes, how many run, and whether the rest fit, with the reason
+  when they do not; `--apply` sets it if they do. Headroom reports what one
+  more replica would serve.
+
+A model never measured behaves exactly as before. Measurements are read from
+the store every half minute.
+
+### Upgrading a machine the control plane does not reach
+
+`sudo llmgw upgrade VERSION --service UNIT` upgrades the machine it runs on —
+the mirror's host above all, whose service runs unprivileged and cannot replace
+its own binary as an agent does. The release's checksums are checked, the old
+binary kept, the unit restarted and watched until it stays up.
+
+### Nodes report their own heat and power
+
+The first step of facility-aware placement (docs/facility-aware-placement.md),
+observe-only: nothing is decided on these yet. Every agent reads, at most every
+ten seconds, and sends with its heartbeat:
+
+- **GPUs** from nvidia-smi: temperature, HBM temperature where the driver has
+  it, power draw and the enforced limit, and what is holding the clocks down —
+  power cap, thermal slowdown, power brake. A GPU's temperature was read once,
+  at registration; it is now current.
+- **CPUs** from hwmon, which is what lm_sensors reads, with nothing to install:
+  the package temperature, and the board's inlet air sensor where it has one.
+- **CPU power** from RAPL's energy counters.
+
+A sensor a machine does not have is not reported, never shown as zero. New
+metrics: `llmgw_gpu_power_watts`, `llmgw_gpu_power_limit_watts`,
+`llmgw_gpu_memory_temperature_celsius`, `llmgw_gpu_throttled{reason}`,
+`llmgw_node_cpu_temperature_celsius`, `llmgw_node_cpu_power_watts`,
+`llmgw_node_inlet_temperature_celsius`. Each node's page shows them, with a GPU
+table of temperature, power against its limit, and what is throttling it.
+
+### The mirror page in tabs
+
+/admin/mirror was one long page: what the mirror holds, the Hugging Face
+explorer and the fetch form, one under another, with every visit waiting on
+Hugging Face for a list it might not show. It is now three tabs — Held (with
+how many are held, downloading and failed), Explore Hugging Face, and Fetch by
+URL — each with its own address. Only the tab shown does its work, and only
+Held reloads while something downloads. A download started from the explorer
+links to Held, where its progress is; a fetch the mirror refused comes back on
+the form it was sent from.
+
+### A held model opens to where it runs, and how many
+
+Clicking a model under Held opens a card beneath it: each replica with its
+node, state, age and the CPU and memory its engine uses, and how many more
+replicas the cluster has room for now, on which nodes, and why not more. The
+replicas it always runs and the most it grows to under load are changed there.
+Room is found by trying the scheduler itself on a copy of the inventory — the
+same decisions the next passes would make, not an estimate beside them — and a
+number of always-running replicas it cannot place is refused with the reason,
+rather than accepted and left short. A ceiling above the room is taken, and
+said. The same answer is at `GET /api/v1/models/{name}/headroom` on the admin
+API.
+
+Each row also says how its model runs without being opened: replicas ready of
+those it should have, how many are still starting, and the nodes they are on —
+in amber when short, in red when parked or when it does not fit. It is one line
+per model beside its mode, and nodes drop the part of the name every node
+shares (node05 for llmgw-node05, the full name on hover).
+
+### One line per record, in every table
+
+The same rule across the admin pages. Headers do not wrap; a record is one
+line; free text is cut to its column with the whole of it on hover rather than
+given a row of its own; a row across the whole table is only a note or an open
+form. On the cluster page that means a node is one line (its address and logs
+beside its name, not under it), a model's description is a column rather than a
+second row, node names drop the prefix they all share, and the GPUs, classes,
+fabrics and fallbacks columns appear only when some row has something in them —
+on a CPU-only cluster they were three columns of dashes pushing the table past
+the page. On People, a person in many teams shows two and a count, the rest on
+hover.
+
+## v0.10.4 — 2026-09-30
+
+Published first as release candidates v0.10.4-rc.1 to rc.9.
+
+After upgrading, run `llmgw ctl restart --all` once: CPU replicas take the new
+thread count below only when they start.
+
+### CPU replicas no longer fight over the CPUs
+
+llama.cpp computes on every CPU of the machine unless told otherwise, and the
+agent did not tell it. Two busy CPU replicas on one node then spin against each
+other: on an 8-vCPU node, 7B generation fell from 10 tokens a second alone to
+0.8 each with both busy, in two rounds of the same test. The agent now starts a
+CPU replica on half the machine's CPUs, where one alone makes 6.5 and two busy
+make 5.5 each. `llamacpp.threads` in the agent's configuration sets it for a
+node — all of them suits a node that only ever runs one CPU replica — and a
+profile's own `--threads` still wins. Measured on VMs whose host is
+oversubscribed; the proportions, not the numbers, are the point.
+
+### Tracing
+
+With `tracing.endpoint` set (or `OTEL_EXPORTER_OTLP_ENDPOINT`), each inference
+request is traced over OTLP/HTTP: a span for the request, carrying its
+`X-Request-Id`, with admission, scheduling and each attempt at an engine under
+it. A caller's own `traceparent` is continued, and the engine is sent one for
+its attempt. `tracing.sample_ratio` traces a share of the requests that arrive
+without a decision. Off, it costs nothing.

 ### Restarting replicas

@@ -77,6 +403,29 @@ more than 26 hours old or there has never been one. docs/install.md now says how
 to restore, including why running engines survive it: the engine-key secret
 comes back with the store.

+### Replicas follow load between min and max
+
+A model's `max` was a ceiling nothing reached: every model ran exactly its
+minimum however long its queue. Now, when requests are waiting at the engines —
+every slot of every replica busy — at one or more per replica for a minute, one
+replica is added, up to `max`; ten minutes with nothing waiting takes one of
+those away, down to `min`. One step at a time, and not while the previous one
+is still starting, so a spike that is over before a replica loads does not
+leave several behind. Each decision is logged with the load that made it.
+`llmgw ctl scale MODEL --min 1 --max 3` turns it on for a model.
+
+### A model from the registry in one command
+
+`llmgw ctl model serve OWNER/REPO` takes a model from the registry to a replica
+that answers. It picks the GGUF file (`--quant`, Q4_K_M by default), has the
+mirror fetch it and checks what arrived against the registry's published
+sha256, checks that the nodes' llama.cpp can load it, sizes its memory from its
+header, adds it to the catalog, waits for a replica — stopping early with the
+reason if it fits nowhere or keeps failing — and asks it to say hello. Each step
+is shown as it happens. It runs in the control plane as a job, so a download of
+hours does not depend on a terminal staying open. `--on-demand` adds it without
+starting a replica.
+
 ### Upgrading from one command

 `sudo llmgw ctl upgrade VERSION`, run on the control plane, upgrades that
@@ -134,6 +483,27 @@ half again plus two" could not see.
  links.** Archives are unpacked through `os.Root`, and a link may only point
  at a file beside it. Every llama.cpp b10991 archive, CPU and CUDA, already
  follows that rule.
+- **Onboarding made every node trust the cluster's root for everything.** It
+  was added to the system store, so a certificate the control plane signed
+  would pass for any site with every TLS client on the node. The agent trusts
+  it through its own configuration only; onboarding no longer adds it, and
+  takes it out of a node that an older version onboarded.
+- **Any node could manage a mirror running mutual TLS**, since one root signs
+  every node's certificate. `llmgw mirror serve --api-client NAME` answers only
+  the certificate with that common name, and the mirror warns when mutual TLS
+  runs without one.
+- **The administrator's password stayed in `init.log`.** It is shown once, on
+  the screen, and the log keeps a placeholder.
+- **Auth off on a public address was silent.** The server now warns at start
+  when the API listens beyond loopback without keys.
+- **The actor recorded with a change was whatever the caller sent**, line
+  breaks included. It is cut to a short printable name, and `sudo llmgw ctl`
+  records the person who ran sudo rather than root.
+- **vLLM's Ray head ran its dashboard**, whose job API runs code for anyone who
+  reaches it. It is off. The head itself still has no authentication, and
+  SGLang still takes its key as a flag; docs/install.md says what to do about
+  both. The monitoring README says how to keep Prometheus's queries to the
+  control plane.

 ### Fixes

@@ -142,6 +512,8 @@ half again plus two" could not see.
  loaded runner.
 - `ctl` reported a change the API had queued (202 Accepted) as an error, as
  `ctl restart` did on its first real use.
+- The status each failure is answered with — 499, 502, 503 with and without
+  Retry-After, 504, 429 — is now tested case by case.

 ### Releases

@@ -120,6 +120,10 @@ message Register {
  // meant "an agent too old to say" and constrained nothing — so a freshly
  // built node with no engine installed yet was a candidate for every model.
  bool engines_probed = 14;
+  // cpu_count is the CPUs the agent may run on. Placement reserves cores
+  // for CPU replicas out of it, the way it reserves memory out of
+  // memory_total_mb; zero, from an older agent, reserves nothing.
+  int32 cpu_count = 18;
  // imex is the service that lets a multi-node NVLink domain share memory.
  IMEX imex = 15;

@@ -281,6 +285,25 @@ message NodeLoad {
  int64 memory_used_bytes = 6;
  int64 memory_available_bytes = 7;
  repeated ShardLoad shards = 8;
+  // What the machine's own sensors say; zero is not measured. cpu_temp_c is
+  // the hottest CPU sensor, cpu_power_w the CPU packages' draw from RAPL,
+  // inlet_temp_c the board's inlet or ambient sensor where it has one.
+  double cpu_temp_c = 9;
+  double cpu_power_w = 10;
+  double inlet_temp_c = 11;
+  repeated GPUSensor gpus = 12;
+}
+
+// GPUSensor is one card's heat, power, and what is holding it back.
+message GPUSensor {
+  int32 index = 1;
+  double temp_c = 2;
+  double memory_temp_c = 3;
+  double power_w = 4;
+  double power_limit_w = 5;
+  // throttle names what is holding the clocks down now: sw_power_cap,
+  // hw_slowdown, sw_thermal, hw_thermal, power_brake.
+  repeated string throttle = 6;
 }

 // ShardLoad is one engine process group's use.
@@ -420,6 +443,8 @@ message Heartbeat {
  // domain, and the node has to stop being offered work that spans it.
  repeated GPUFabricState gpu_fabric = 16;
  IMEX imex = 17;
+  // cpu_count as in Register, for a VM given more CPUs while it runs.
+  int32 cpu_count = 18;
 }

 // Provision is the software a node should have: which engines, at which
@@ -530,6 +555,9 @@ message StartShard {
  // control plane from the replica id. The router sends it; nothing else
  // that can reach the node's ports has it.
  string api_key = 16;
+  // cpu_threads is how many CPU threads a shard with no GPU computes on:
+  // the cores placement reserved for it. Zero leaves it to the agent.
+  int32 cpu_threads = 17;
 }

 message EngineProfile {
@@ -544,6 +572,9 @@ message EngineProfile {
  map<string, string> env = 9;
  string model_path = 10;
  string served_model_name = 11;
+  // adapters are LoRA adapters loaded beside the model, in order: the index
+  // is the id a request applies one by.
+  repeated LoraAdapter adapters = 12;
 }

 // RemoveWeights deletes cached model files a node no longer needs.
@@ -676,3 +707,10 @@ message UpgradeReply {
  string version = 2;
  string error = 3;
 }
+
+// LoraAdapter is one adapter: the name clients ask for it by, and its source,
+// which the agent fetches as it does the model's weights.
+message LoraAdapter {
+  string name = 1;
+  string source = 2;
+}
@@ -122,9 +122,10 @@ TMP="$(mktemp -d)"
 trap 'rm -rf "$TMP"' EXIT
 "$BINARY" ctl cert issue-client "$NODE" --dir "$PKI" --out "$TMP" >/dev/null

-# 2 and 3. Identity, and trusting the root so future tooling on that machine
-#    does not treat our certificates as strangers.
-say "installing the certificate and trusting the root"
+# 2 and 3. Identity. The root is trusted by the agent's configuration only,
+#    never system-wide: it signs whatever the control plane asks, and every
+#    TLS client on the machine would take it for any site.
+say "installing the certificate"
 scp -q $SSHOPTS "$PKI/ca.pem" "$TMP/$NODE.pem" "$TMP/$NODE-key.pem" "$SSH_USER@$ADDR:/tmp/"
 $SSH "set -e
  sudo install -d -m 0755 /etc/llmgw
@@ -132,8 +133,10 @@ $SSH "set -e
  sudo install -m 0644 /tmp/$NODE.pem /etc/llmgw/agent.pem
  sudo install -m 0600 /tmp/$NODE-key.pem /etc/llmgw/agent-key.pem
  sudo chown root:root /etc/llmgw/*
-  sudo install -m 0644 /tmp/ca.pem /usr/local/share/ca-certificates/llmgw-ca.crt
-  sudo update-ca-certificates >/dev/null 2>&1 || true
+  if [ -f /usr/local/share/ca-certificates/llmgw-ca.crt ]; then
+    sudo rm -f /usr/local/share/ca-certificates/llmgw-ca.crt
+    sudo update-ca-certificates --fresh >/dev/null 2>&1 || true
+  fi
  rm -f /tmp/ca.pem /tmp/$NODE.pem /tmp/$NODE-key.pem"

 # 4. The binary, the configuration and the unit.
@@ -52,6 +52,13 @@ install -m 0644 alertmanager.yml          /etc/prometheus/alertmanager.yml
 # client; one Alertmanager needs no cluster port.
 echo 'ARGS="--web.listen-address=127.0.0.1:9093 --cluster.listen-address="' \
    > /etc/default/prometheus-alertmanager
+# Prometheus itself is read-only as installed: its admin and lifecycle APIs
+# stay off (no --web.enable-admin-api, no --web.enable-lifecycle). It still
+# answers any query from anyone who reaches :9090, which says which models and
+# nodes there are and how busy each is. Its readers are the control plane
+# (monitoring.prometheus) and Grafana beside it; to keep it to those, allow
+# the control plane and drop the rest, e.g. with nftables:
+#   nft add rule inet filter input tcp dport 9090 ip saddr != { 127.0.0.1, CONTROL_PLANE_IP } drop
 promtool check config /etc/prometheus/prometheus.yml
 systemctl restart prometheus prometheus-alertmanager

@@ -84,6 +84,28 @@ groups:
            <model>` says what each node lacks. For an on-demand model, requests
            for it are told this rather than asked to retry.

+      - alert: ModelPreempted
+        expr: increase(llmgw_model_preempted_total[15m]) > 0
+        labels: { severity: info }
+        annotations:
+          summary: "{{ $labels.model }} ({{ $labels.class }}) gave up replicas to a higher class"
+          description: >-
+            Capacity ran short and a model of a higher class took this one's
+            room. Expected now and then; if it keeps firing for the same model,
+            the cluster is too small for what is asked of it at these classes.
+            `llmgw ctl models` shows who it gave way to.
+
+      - alert: CapacityAtLimit
+        expr: llmgw_capacity_holds >= on(scope, resource) llmgw_capacity_limit
+        for: 15m
+        labels: { severity: info }
+        annotations:
+          summary: "{{ $labels.scope }} holds all the {{ $labels.resource }} it may"
+          description: >-
+            Its models get no further replicas, under load or not, until
+            something of theirs stops or the limit is raised: `llmgw ctl quota
+            list`, `llmgw ctl quota set`.
+
      - alert: EngineInstallFailed
        expr: llmgw_node_engine_install{state="failed"} == 1
        for: 15m
@@ -0,0 +1,303 @@
+# Facility-aware placement: rack power, temperature and cooling
+
+**Status: step 0, reading the nodes' own sensors, is built; steps 1–5 are
+deferred** until the rack, PDU and power information of a real site is at
+hand — without it they would be designed against guesses.
+This is the design for step 5 of resource allocation, after reserved CPU cores
+and memory (1), capacity classes and preemption (2), tenant capacity quotas (3)
+and sizing by measured throughput (4), all of which shipped in v0.10.5. Of
+what follows, only the node sensors (rollout step 0) exist in the code: they
+are read, published and shown, and change no decision.
+
+## Why
+
+In a large installation the scarce thing is often not a GPU but the room it
+sits in. Three failures are invisible to placement today:
+
+- **A rack goes over its power budget.** Every node in it reports healthy;
+  placement sees free cards and fills them; the breaker, or the facility's
+  power cap, decides the rest. A rack with A and B feeds is worse: it runs
+  fine until one feed fails and the other cannot carry the load.
+- **Hot hardware is slow hardware.** GPUs throttle long before they fail. A
+  replica placed in a rack whose inlet air is too warm serves fewer tokens per
+  second than its measured capacity says, and nothing connects the two.
+- **Correlated failure.** Two replicas of a model on two nodes in the same rack
+  or on the same PDU are one failure away from zero. Placement spreads replicas
+  across nodes, not across racks.
+
+The goal is for placement to treat facility limits the way it already treats
+memory and cores: reserve against a budget, refuse what does not fit, say why,
+and never let missing data stop the cluster.
+
+## What it builds on
+
+| Already in place | Used for |
+|---|---|
+| Node labels, reported by agents, matched by `node_selector` | Where a node is: its rack, row, PDU |
+| `inventory.Allocation` carries GPUs, memory and cores per shard | Adding watts, summed per rack and PDU |
+| Placement rejections (`explain-placement`, headroom's "why not") | Naming the rack that is full or hot |
+| Headroom: the scheduler dry-run on a copy of the inventory | Room that respects power budgets, for free |
+| Preemption with classes, minimums and a dry run (step 2) | Shedding load from a hot rack in the right order |
+| Measured throughput per profile (step 4) | Tokens per watt as a placement preference |
+| Node sensors, read by every agent (step 0, built) | Heat and power with no facility integration: see below |
+| `internal/promq`, a Prometheus client | Reading facility metrics that DCIM exporters publish |
+
+## Concepts
+
+**Location.** Each node has a place in a tree: site, room, row, rack. Power has
+its own tree beside it: a rack draws from one or more PDUs, and each PDU from a
+feed. Both are read from node labels with well-known keys:
+
+```yaml
+# agent.yaml
+labels:
+  llmgw.io/site: sto1
+  llmgw.io/room: hall-b
+  llmgw.io/row: "12"
+  llmgw.io/rack: r12-07
+  llmgw.io/pdu: r12-07-a,r12-07-b   # both feeds, for a dual-fed rack
+```
+
+A node without these labels is its own rack of one: placement behaves as it
+does today.
+
+**Domain.** A rack, a row, a PDU, a room: anything with a budget or a
+condition. Domains are declared in a facility file (below) or discovered from
+node labels; a label naming a domain the file does not declare creates it with
+no budget and no condition.
+
+**Budget.** A domain may have a power cap in watts. Placement reserves each
+replica's estimated draw against every domain its node belongs to and refuses a
+placement that would take any of them past its cap less a safety margin. For a
+dual-fed rack the cap that matters is what one feed can carry alone, since that
+is what the rack must survive.
+
+**Condition.** A domain has a thermal and cooling condition: `ok`, `warm`, `hot`
+or `unknown`, derived from readings against thresholds with hysteresis.
+Conditions change what placement prefers and allows; they are not amounts to
+fill.
+
+**Reading.** A measured value with a time: inlet temperature, power draw, a
+cooling unit's alarm state. Readings older than their staleness limit are
+treated as absent.
+
+## Estimating a replica's draw
+
+What placement reserves has to be known before the replica runs, so it is an
+estimate first and a measurement later:
+
+1. **Declared.** A profile may say `power_watts`. The operator's figure wins.
+2. **Derived.** Otherwise GPU profiles take each card's power limit from NVML
+   (the agent already reads the cards; the power limit is one more field) plus
+   a per-node share for CPU, memory and fans; CPU profiles take their reserved
+   cores times a per-core figure for the node's CPU model.
+3. **Measured.** Once replicas run, the agent reports their cards' actual draw
+   (NVML) and, where the BMC allows, the node's (Redfish or IPMI DCMI). A
+   domain's headroom is computed from the larger of what is reserved and what
+   is measured, so an estimate that was too low is corrected by the room it
+   stops offering, not by an outage.
+
+## Decisions
+
+### Power
+
+- **Hard limit.** A placement that would take any domain past `cap × (1 −
+  margin)` is rejected, with the domain and the figures: *rack r12-07: 14.2 of
+  16.0 kW reserved, replica needs ~2.8, margin 10%*.
+- **Feeds.** For a domain with two feeds the cap is the smaller feed's rating
+  times its derating (typically 80%), not the sum.
+- **Already over.** A domain found over its cap, because the estimate was low
+  or the cap was lowered, gets no new placements, and an alert. It is not
+  emptied automatically; shedding power is the facility's decision, and
+  stopping inference to do it is ours only if asked (see shedding below).
+
+### Temperature and cooling
+
+Thresholds are per domain with defaults, and each condition needs to hold for a
+while before it changes, in both directions, so a single reading neither
+starts nor ends anything:
+
+| Condition | Enters when | Placement |
+|---|---|---|
+| `ok` | below the warm threshold for 5 min | normal |
+| `warm` | inlet above the warm threshold for 2 min, GPU temperatures near throttling, or any GPU thermally throttled | allowed, scored down: other domains first |
+| `hot` | inlet above the hot threshold for 2 min, a cooling unit in alarm, or a sustained share of the domain's GPUs thermally throttled | no new placements, as a cordon would, but automatic and reversible |
+| `unknown` | no fresh reading | as `ok`, and reported as unknown |
+
+**Shedding** (optional, off by default): a domain `hot` for longer than its
+shed delay has load moved off it using step 2's machinery. The lowest class goes
+first, and replicas above their model's minimum go before any at it. A critical
+minimum is never touched, and moves are rate limited to one replica per domain
+per interval. The replica is placed elsewhere first where there is room, so
+shedding moves work rather than dropping it. GPU power capping (`nvidia-smi
+-pl`) is a gentler alternative worth measuring against shedding: less
+throughput, no restart.
+
+### Spreading
+
+Replicas of one model are spread across racks, and across PDUs, the way they
+are spread across nodes today. This is a preference, dropped when it is the only
+thing standing in the way, as node spreading already is.
+
+### Scoring
+
+Placement today filters, then takes the first candidate in a fixed order
+(fewest free GPUs for GPU profiles, fewest replicas for CPU ones, nodes holding
+the weights first). Facility signals are better as a score than as more
+filters. Candidates that pass the hard limits are ordered by:
+
+1. condition: `ok` before `warm`;
+2. power headroom left in the rack after this replica, more first, so load
+   levels across racks;
+3. the existing order.
+
+Once measurements include power, step 4's profile preference can use tokens per
+watt instead of tokens per GPU where both are known.
+
+## Where the data comes from
+
+A `FacilitySource` interface, implemented several ways, chosen in
+`server.yaml`:
+
+```go
+type FacilitySource interface {
+    // Domains is what the source knows: budgets, thresholds, members.
+    Domains(ctx) ([]Domain, error)
+    // Readings is the latest value of each signal, per domain.
+    Readings(ctx) ([]Reading, error)
+}
+```
+
+- **Static file.** Domains, caps and thresholds, written by hand or generated
+  from a DCIM export. Enough for power budgets with no live data at all.
+- **Prometheus.** DCIM, PDU (SNMP exporter) and BMC (Redfish exporter)
+  readings usually end up in Prometheus. The source runs configured instant
+  queries and maps their labels onto domains:
+
+  ```yaml
+  facility:
+    source: prometheus
+    prometheus: http://monitoring:9090
+    signals:
+      inlet_temp_c:
+        query: max by (rack) (dcim_rack_inlet_temperature_celsius)
+        domain_label: rack
+        stale_after: 2m
+      power_watts:
+        query: sum by (pdu) (pdu_outlet_power_watts)
+        domain_label: pdu
+      cooling_alarm:
+        query: max by (row) (crac_alarm)
+        domain_label: row
+  ```
+- **Agents (built, step 0).** What each node reads about itself, every ten
+  seconds at most, sent with its heartbeat:
+  - **GPUs, from nvidia-smi:** temperature, memory (HBM) temperature where the
+    driver reports it, power draw, the enforced power limit, and what is
+    holding the clocks down now — `sw_power_cap`, `hw_slowdown`, `sw_thermal`,
+    `hw_thermal`, `power_brake`.
+  - **CPUs, from hwmon** (what lm_sensors reads, without it installed): the
+    package temperature, or the hottest core; and the board's inlet or ambient
+    sensor where it has one (`SYSTIN`, `Inlet`, `Ambient`).
+  - **CPU power, from RAPL:** the packages' energy counters over time,
+    subdomains not counted twice, counter wrap handled.
+
+  A machine without a sensor reports nothing for it, never a zero; a VM has
+  none of them. Published as `llmgw_gpu_temperature_celsius`,
+  `llmgw_gpu_memory_temperature_celsius`, `llmgw_gpu_power_watts`,
+  `llmgw_gpu_power_limit_watts`, `llmgw_gpu_throttled{reason}`,
+  `llmgw_node_cpu_temperature_celsius`, `llmgw_node_cpu_power_watts` and
+  `llmgw_node_inlet_temperature_celsius`, and on each node's page.
+
+  What they can and cannot say:
+  - **Throttling is the strongest signal there is.** A temperature is a proxy;
+    a card holding its clocks down is the fact that invalidates its measured
+    capacity, and it needs no threshold to be chosen.
+  - **Measured power** is GPU draw plus CPU packages; memory, disks, fans and
+    power-supply losses are not in it, so a node's draw is that plus an
+    estimated remainder. A rack's is its llmgw nodes' plus a declared figure
+    for everything else in it: switches and storage report nothing here.
+  - **CPU and GPU temperatures follow the load, not the room.** A hot rack is
+    inferred from many nodes running hot at comparable load, or from
+    throttling, not from one node's CPU. Only a board inlet sensor reads the
+    air, and most boards have none.
+  - **No budget comes from a sensor.** A reading is a draw; the cap it is
+    measured against still comes from the facility.
+- **Simulated.** Readings from a script, for tests and for rehearsing a
+  thermal event on a cluster without facility data.
+
+## Safety
+
+- **Missing data never blocks.** A source that is down, a reading that is stale,
+  a node without location labels: placement behaves as it does today, and says
+  the data is missing (a metric, a line on the cluster page). Facility
+  awareness can make placement more careful, never impossible.
+- **Hysteresis and hold times** on every condition, so a noisy sensor does not
+  flap a rack between allowed and forbidden.
+- **Observe-only first.** A mode where every decision is computed, logged and
+  shown ("would have refused: rack r12-07 hot") but nothing is enforced, run
+  for long enough to trust the thresholds before turning enforcement on.
+- **Manual override.** `llmgw ctl facility hold RACK` and `release RACK`, for the
+  facility team to mark a domain unavailable, or to clear a condition they know
+  to be a sensor fault.
+- **Shedding is opt-in**, per domain or cluster-wide, and rate limited.
+
+## What people see
+
+- `explain-placement` and headroom name the domain: *rack r12-07 is hot (inlet
+  31°C for 6 min)*, *pdu r12-07-a: 7.1 of 7.4 kW*.
+- `llmgw ctl facility` lists domains with budget, reserved, measured and
+  condition. The cluster page gets a rack summary beside the nodes table.
+- Metrics: `llmgw_domain_power_budget_watts`, `_reserved_watts`,
+  `_measured_watts`, `llmgw_domain_condition{condition}`,
+  `llmgw_domain_reading_age_seconds`; alerts for a domain near its cap, a
+  domain hot, a domain over its cap, and facility data gone stale.
+
+## Rollout
+
+Each step ships on its own and is useful without the next:
+
+0. **Node sensors, observe-only (built).** GPU heat, power and throttling,
+   CPU heat and power, inlet air where the board has a sensor; published and
+   shown, no decision changed.
+1. **Location and spreading.** Well-known labels, domains from labels, replicas
+   spread across racks and PDUs, locations on the node views. No facility data
+   needed.
+2. **Power budgets from a static file.** Caps and margins, estimated draw per
+   replica (declared or derived), hard limit in placement and headroom.
+3. **Readings, observe-only.** The Prometheus source beside the agents',
+   conditions with hysteresis (throttling first among them), measured power
+   folded into headroom; decisions shown, not enforced.
+4. **Thermal enforcement.** `warm` scored down, `hot` refused, manual hold and
+   release.
+5. **Shedding and tokens per watt.** Opt-in shedding through step 2's
+   preemption; measured power in profile preference.
+
+## Testing
+
+- The simulated source drives unit tests of every condition transition, the
+  hysteresis, staleness, and the placement outcomes, with mutation checks as
+  for steps 1–4.
+- Property tests: whatever the readings, placement never exceeds a declared
+  cap with fresh data, and never refuses everything because data is missing.
+- On the PoC cluster, with no facility data: step 1 with labelled VMs, step 2
+  with a static file of made-up caps, steps 3–4 with the simulated source
+  replaying a thermal event.
+- Before enforcement anywhere real: a period in observe-only mode against the
+  site's actual data.
+
+## Open questions
+
+- **What the sites have.** Which DCIM, which PDUs, which BMCs, and whether any
+  of it already reaches a Prometheus. The source design assumes Prometheus is
+  the common ground; a site without it needs another source.
+- **Who owns the numbers.** Caps, deratings and thresholds belong to the
+  facility team. Whether they edit llmgw's file, or llmgw reads theirs.
+- **Phase balance.** Three-phase PDUs have per-phase limits; whether placement
+  should see phases or only the PDU total.
+- **Cooling granularity.** Cooling is often per row or per aisle, not per rack;
+  the domain tree allows either, but the defaults should match what sites
+  actually measure.
+- **Power capping instead of shedding.** Whether to lower GPU power limits in a
+  warm rack rather than move replicas, and how that interacts with measured
+  capacity from step 4, which would no longer hold at the lower limit.
@@ -671,6 +671,74 @@ llmgw ctl explain-placement qwen2.5-coder-3b-instruct
 which names every candidate and why each was rejected, rather than leaving you
 to infer it.

+### LoRA adapters
+
+A llama.cpp model can carry LoRA adapters, each asked for by its own name as
+though it were a model:
+
+```yaml
+- name: qwen2.5-coder-7b-instruct
+  source: http://mirror:8081/qwen2.5-coder-7b-instruct/5d567d26/model.gguf
+  adapters:
+    - name: coder-sql
+      source: http://mirror:8081/coder-sql/v1/adapter.gguf
+      scale: 1.0          # optional; 1 when left out
+  profiles: [...]
+```
+
+Every replica of the model fetches every adapter and loads it at start, applied
+to nothing. A request for `coder-sql` goes to the model's replicas with the
+adapter applied to it alone; a request for the model gets the model. Adapter
+names share the namespace of models and aliases, appear in `/v1/models`, and
+follow key allowlists on their own: a key allowed `coder-sql` reaches that
+adapter and not the model under it, while a key allowed the model reaches its
+adapters too. Adding or removing an adapter is a change to the model; running
+replicas load the new set when restarted (`llmgw ctl restart --model M`).
+Besides the mirror page's model card, adapters are attached from the command
+line, which reads the file first and refuses one that is not an adapter for
+this model's architecture:
+
+    llmgw ctl model adapters coder --add coder-sql=http://mirror:8081/sql-lora/v1/sql.gguf@0.8
+    llmgw ctl model adapters coder --remove coder-sql
+
+The adapter file must be a GGUF LoRA made for this model — llama.cpp's
+`convert_lora_to_gguf.py` makes one from a PEFT adapter — and not a model with
+the adapter already merged in, which is what most "LoRA GGUF" repositories hold.
+
+### Models served outside the cluster
+
+A model can be an OpenAI-compatible API elsewhere — a hosted provider, or
+another gateway — rather than replicas here. Nothing is placed for it; the
+router sends its requests on, with the provider's key and the provider's name
+for the model:
+
+```yaml
+- name: cloud-mini
+  source: external
+  external:
+    url: https://api.openai.com          # without /v1
+    model: gpt-4o-mini                   # what the provider calls it
+    api_key_file: /etc/llmgw/secrets/openai.key
+```
+
+Named in another model's `fallbacks`, it is used only when the cluster has no
+capacity left for that model:
+
+```yaml
+- name: qwen2.5-coder-7b-instruct
+  fallbacks: [cloud-mini]
+  ...
+```
+
+The key is read from its file on the control plane, never stored in the
+catalog, and a key rotated in the file is used within seconds. A model whose key
+file cannot be read is not offered. Everything else applies as to any model:
+prices and budgets, rate limits, usage records, and key allowlists.
+
+**Its prompts leave the cluster.** Decide who may send them: a key or project
+whose allowlist leaves the external model out cannot reach it, as a fallback or
+by name. The cluster page and `ctl models` mark it `external`.
+
 ### How many replicas, and how much each one carries

 Replica count is changed while the cluster runs, without editing the
@@ -940,8 +1008,18 @@ restart either — it rebuilds the fleet from what the agents report.
 fields, and both sides treat an absent one as unknown rather than as a default.
 Agents older than v0.10.4 cannot upgrade themselves: move those by hand once,
 installing the release binary and restarting `llmgw-agent`, and every upgrade
-after that is the command above. The mirror, which is not an agent, is
-upgraded the same way by hand.
+after that is the command above.
+
+**The mirror is upgraded on its own host**, since it is not an agent and its
+service cannot replace its own binary:
+
+```
+sudo llmgw upgrade v0.10.5 --service llmgw-mirror
+```
+
+The same checks — the release's checksums, the old binary kept beside the new —
+then a restart of the unit, waiting for it to stay up. A mirror older than
+v0.10.5 does not have the command: install the release binary by hand once.

 Then confirm what you have, rather than assuming:

@@ -956,6 +1034,33 @@ replica alone on purpose, because churning one costs more than the difference.
 `llmgw ctl restart <replica>` replaces it with one on the new profile, starting
 the new one before stopping the old.

+### Signed releases
+
+Every install above checks a release against its published checksums, which
+proves the download is what the server holds — not that the server holds what
+was built. Signing closes that: CI signs the checksums, and a binary that
+trusts a key installs only releases signed with it. Off until a key exists;
+turning it on, once:
+
+```
+llmgw release keygen
+```
+
+1. Put the **private** key in the repository's Actions secret
+   `RELEASE_SIGNING_KEY`. From the next tag, releases carry
+   `checksums.txt.sig`.
+2. Put the **public** key in `internal/release/trusted.pub` and release. Every
+   binary built from then on carries it and refuses releases that are not
+   signed with it — `ctl upgrade`, agents upgrading themselves, and
+   `llmgw upgrade` alike.
+
+In that order: a build that trusts a key refuses unsigned releases, so the
+secret has to be signing before the key is trusted. The first trusting release
+is itself installed by a binary that trusts nothing; every one after is checked
+by the one before. `LLMGW_RELEASE_KEY` overrides the built-in key, for rotating
+it. The onboarding script fetches with `curl` and checks checksums only; a node
+is checked from its first upgrade on.
+
 ## Backing up

 `llmgw server backup` writes one archive holding everything a control plane
@@ -1124,3 +1229,11 @@ requests. On a slow model, set `timeouts.generation` higher.
  has them. Single-node placement on real GPUs is the well-trodden path; if you
  are the first to run a TP group across real cards, run `llmgw doctor --json`
  on each node first and keep the reports.
+- **Two engine setups are weaker than the rest.** An engine's key is passed
+  in its environment, where only root and its own user can read it, except
+  SGLang's, which takes it only as a flag: other users on the node can see it
+  in the process list. Mount `/proc` with `hidepid=2` on nodes where that
+  matters. vLLM's `launch_mode: ray` starts a Ray head without
+  authentication, so anyone who can reach its port can run code on the
+  node. The dashboard is off, but firewall the port to the other nodes in the
+  replica, or use the default `mp` launch mode, which needs no Ray at all.
@@ -15,12 +15,16 @@ rather than repeated.

 | Key | Type | In |
 |---|---|---|
+| `name` | `string` | [Adapter](#adapter) |
+| `source` | `string` | [Adapter](#adapter) |
+| `scale` | `float64` | [Adapter](#adapter) |
 | `enabled` | `bool` | [Admission](#admission) |
 | `bands` | `[]Band` | [Admission](#admission) |
 | `default_band` | `string` | [Admission](#admission) |
 | `band_header` | `string` | [Admission](#admission) |
 | `queue_ttl` | `time.Duration` | [Admission](#admission) |
 | `queue_capacity_per_band` | `int` | [Admission](#admission) |
+| `queue_capacity_total` | `int` | [Admission](#admission) |
 | `interval` | `time.Duration` | [Admission](#admission) |
 | `enabled` | `bool` | [Auth](#auth) |
 | `store` | `string` | [Auth](#auth) |
@@ -77,10 +81,14 @@ rather than repeated.
 | `pool` | `string` | [Endpoint](#endpoint) |
 | `weight` | `float64` | [Endpoint](#endpoint) |
 | `priority` | `int` | [Endpoint](#endpoint) |
+| `url` | `string` | [External](#external) |
+| `model` | `string` | [External](#external) |
+| `api_key_file` | `string` | [External](#external) |
 | `contact` | `string` | [Landing](#landing) |
 | `notice` | `string` | [Landing](#landing) |
 | `bin` | `string` | [LlamaCppConfig](#llamacppconfig) |
 | `parallel` | `int` | [LlamaCppConfig](#llamacppconfig) |
+| `threads` | `int` | [LlamaCppConfig](#llamacppconfig) |
 | `api` | `string` | [Mirror](#mirror) |
 | `tls` | `tlsconf.Config` | [Mirror](#mirror) |
 | `registry` | `string` | [Mirror](#mirror) |
@@ -91,6 +99,10 @@ rather than repeated.
 | `source` | `string` | [Model](#model) |
 | `profiles` | `[]Profile` | [Model](#model) |
 | `scaling` | `Scaling` | [Model](#model) |
+| `class` | `string` | [Model](#model) |
+| `owner` | `string` | [Model](#model) |
+| `external` | `*External` | [Model](#model) |
+| `adapters` | `[]Adapter` | [Model](#model) |
 | `fallbacks` | `[]string` | [Model](#model) |
 | `routing_profile` | `string` | [Model](#model) |
 | `slo` | `SLO` | [Model](#model) |
@@ -128,6 +140,7 @@ rather than repeated.
 | `args` | `[]string` | [Profile](#profile) |
 | `env` | `map[string]string` | [Profile](#profile) |
 | `min_memory_gb` | `int` | [Profile](#profile) |
+| `cpu_cores` | `int` | [Profile](#profile) |
 | `min_disk_gb` | `int` | [Profile](#profile) |
 | `node_selector` | `map[string]string` | [Profile](#profile) |
 | `weight` | `float64` | [Profile](#profile) |
@@ -167,6 +180,7 @@ rather than repeated.
 | `landing` | `Landing` | [Server](#server) |
 | `agent_tls` | `tlsconf.Config` | [Server](#server) |
 | `monitoring` | `Monitoring` | [Server](#server) |
+| `tracing` | `Tracing` | [Server](#server) |
 | `mirror` | `Mirror` | [Server](#server) |
 | `engines` | `map[string]string` | [Server](#server) |
 | `backup_dir` | `string` | [Server](#server) |
@@ -184,10 +198,33 @@ rather than repeated.
 | `first_token` | `time.Duration` | [Timeouts](#timeouts) |
 | `idle` | `time.Duration` | [Timeouts](#timeouts) |
 | `total` | `time.Duration` | [Timeouts](#timeouts) |
+| `endpoint` | `string` | [Tracing](#tracing) |
+| `sample_ratio` | `float64` | [Tracing](#tracing) |
 | `bin` | `string` | [VLLMConfig](#vllmconfig) |
 | `launch_mode` | `string` | [VLLMConfig](#vllmconfig) |
 | `ray_bin` | `string` | [VLLMConfig](#vllmconfig) |

+## Adapter
+
+### `name`
+
+`string`
+
+Name is what clients ask for, like a model name.
+
+### `source`
+
+`string`
+
+Source is the adapter file (GGUF for llama.cpp), fetched by each node
+as the model's weights are.
+
+### `scale`
+
+`float64`
+
+Scale is how strongly it is applied; zero takes 1.
+
 ## Admission

 ### `enabled`
@@ -221,6 +258,15 @@ should not be able to promote themselves.

 `int`

+### `queue_capacity_total`
+
+`int`
+
+QueueCapacityTotal bounds everything queued across bands. Reached, a
+request is not refused while a lower band has any queued: the newest of
+the lowest band is turned away instead, with a 503 and Retry-After.
+Zero takes half of all the bands' capacities together.
+
 ### `interval`

 `time.Duration`
@@ -599,6 +645,28 @@ called. Empty means vLLM's names.

 `int`

+## External
+
+### `url`
+
+`string`
+
+URL is the provider's base address, without /v1: https://api.openai.com.
+
+### `model`
+
+`string`
+
+Model is what the provider calls it; requests are rewritten to it.
+
+### `api_key_file`
+
+`string`
+
+APIKeyFile holds the provider's key, on the control plane. A file
+rather than the key itself, so the key is in neither the catalog, the
+store nor the audit log; empty for a provider that asks for none.
+
 ## Landing

 ### `contact`
@@ -634,6 +702,20 @@ Parallel is how many requests a server handles at once, when a profile
 does not say. llama.cpp defaults to one, which would make every second
 request wait behind the first no matter what the router decided.

+### `threads`
+
+`int`
+
+Threads is how many CPU threads a shard with no GPU computes on. Zero
+takes half this machine's CPUs.
+
+Not all of them, which is llama.cpp's own default: two busy servers
+that each spin on every CPU fight for them, and on an 8-vCPU node
+that made 7B generation fall from 10 tokens a second to under 1 each.
+At half, one server alone is about a third slower and two together
+keep 5.5 each. A node that only ever runs one CPU replica does better
+with all of them; say so here.
+
 ## Mirror

 ### `api`
@@ -717,6 +799,45 @@ Source is the weights path or a HuggingFace id.

 Takes a [Scaling](#scaling) block.

+### `class`
+
+`string`
+
+Class is how much this model's capacity is worth against another's,
+when there is not enough for both: critical, standard or opportunistic.
+Empty is standard, or opportunistic for a model loaded on demand. See
+EffectiveClass.
+
+### `owner`
+
+`string`
+
+Owner is the part of the tenancy tree whose capacity quota this model
+counts against, written org:ID, team:ID or project:ID. Empty is shared:
+held by the cluster, counted against nobody.
+
+### `external`
+
+`*External`
+
+External is a model served by someone else — an OpenAI-compatible API
+outside the cluster — rather than by replicas placed here. Such a model
+has no profiles and nothing is placed for it; the router sends its
+requests to the provider. Most useful named in another model's
+fallbacks: used only when the cluster has no capacity left. Its
+prompts leave the cluster, so which keys may name it is worth an
+allowlist.
+
+Takes a [External](#external) block.
+
+### `adapters`
+
+`[]Adapter`
+
+Adapters are LoRA adapters served on this model's replicas, each asked
+for by its own name as though it were a model. Every replica loads
+every adapter, applying one only to a request that names it.
+
 ### `fallbacks`

 `[]string`
@@ -948,6 +1069,20 @@ this. A node that does not report its memory is not excluded either —
 an older agent must not have its nodes quietly emptied by an upgrade of
 the control plane.

+### `cpu_cores`
+
+`int`
+
+CPUCores is the cores a replica of a profile with no GPU is given:
+reserved out of its node's CPUs by placement, as memory is, and what
+its engine computes on (llama.cpp's --threads).
+
+Reserved because an engine takes every core it is told about, and two
+on one node that each think they have all of them spin against each
+other: on an 8-vCPU node, two 7B replicas fell from 10 tokens a second
+to under one each. Zero reserves nothing and leaves the thread count to
+the node, which is every profile from before this.
+
 ### `min_disk_gb`

 `int`
@@ -1246,6 +1381,16 @@ Monitoring is where the dashboard reads its history from.

 Takes a [Monitoring](#monitoring) block.

+### `tracing`
+
+`Tracing`
+
+Tracing sends a trace of each inference request to an OpenTelemetry
+collector. Off unless an endpoint is set here or in
+OTEL_EXPORTER_OTLP_ENDPOINT.
+
+Takes a [Tracing](#tracing) block.
+
 ### `mirror`

 `Mirror`
@@ -1366,6 +1511,22 @@ about a hundred tokens on a slow model.

 `time.Duration`

+## Tracing
+
+### `endpoint`
+
+`string`
+
+Endpoint is the collector, for example http://otel.internal:4318.
+Empty falls back to OTEL_EXPORTER_OTLP_ENDPOINT, and to no tracing.
+
+### `sample_ratio`
+
+`float64`
+
+SampleRatio is the share of requests traced, 0 to 1, when the caller
+did not decide it with a sampled traceparent of its own. Zero takes 1.
+
 ## VLLMConfig

 ### `bin`
@@ -19,6 +19,10 @@ require (
 	github.com/prometheus/common v0.71.0
 	github.com/spf13/cobra v1.10.2
 	github.com/yuin/goldmark v1.8.6
+	go.opentelemetry.io/otel v1.46.0
+	go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.46.0
+	go.opentelemetry.io/otel/sdk v1.46.0
+	go.opentelemetry.io/otel/trace v1.46.0
 	golang.org/x/crypto v0.57.0
 	golang.org/x/net v0.59.0
 	golang.org/x/sync v0.23.0
@@ -29,6 +33,7 @@ require (
 require (
 	github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
 	github.com/beorn7/perks v1.0.1 // indirect
+	github.com/cenkalti/backoff/v5 v5.0.3 // indirect
 	github.com/charmbracelet/colorprofile v0.4.1 // indirect
 	github.com/charmbracelet/x/ansi v0.11.6 // indirect
 	github.com/charmbracelet/x/cellbuf v0.0.15 // indirect
@@ -39,8 +44,11 @@ require (
 	github.com/dustin/go-humanize v1.0.1 // indirect
 	github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f // indirect
 	github.com/fsnotify/fsnotify v1.9.0 // indirect
-	github.com/go-viper/mapstructure/v2 v2.4.0 // indirect
+	github.com/go-logr/logr v1.4.4 // indirect
+	github.com/go-logr/stdr v1.2.2 // indirect
+	github.com/go-viper/mapstructure/v2 v2.5.0 // indirect
 	github.com/google/uuid v1.6.0 // indirect
+	github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0 // indirect
 	github.com/inconshreveable/mousetrap v1.1.0 // indirect
 	github.com/jackc/pgpassfile v1.0.0 // indirect
 	github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
@@ -61,9 +69,16 @@ require (
 	github.com/rivo/uniseg v0.4.7 // indirect
 	github.com/spf13/pflag v1.0.9 // indirect
 	github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e // indirect
+	go.opentelemetry.io/auto/sdk v1.2.1 // indirect
+	go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.46.0 // indirect
+	go.opentelemetry.io/otel/metric v1.46.0 // indirect
+	go.opentelemetry.io/proto/otlp v1.11.0 // indirect
 	go.yaml.in/yaml/v3 v3.0.5 // indirect
 	golang.org/x/sys v0.48.0 // indirect
 	golang.org/x/text v0.42.0 // indirect
+	google.golang.org/genproto/googleapis/api v0.0.0-20260819154853-08b0e4226688 // indirect
+	google.golang.org/genproto/googleapis/rpc v0.0.0-20260819154853-08b0e4226688 // indirect
+	google.golang.org/grpc v1.83.1 // indirect
 	modernc.org/libc v1.75.7 // indirect
 	modernc.org/mathutil v1.7.1 // indirect
 	modernc.org/memory v1.12.1 // indirect
@@ -6,6 +6,8 @@ github.com/aymanbagabas/go-osc52/v2 v2.0.1 h1:HwpRHbFMcZLEVr42D4p7XBqjyuxQH5SMiE
 github.com/aymanbagabas/go-osc52/v2 v2.0.1/go.mod h1:uYgXzlJ7ZpABp8OJ+exZzJJhRNQ2ASbcXHWsFqH8hp8=
 github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
 github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
+github.com/cenkalti/backoff/v5 v5.0.3 h1:ZN+IMa753KfX5hd8vVaMixjnqRZ3y8CuJKRKj1xcsSM=
+github.com/cenkalti/backoff/v5 v5.0.3/go.mod h1:rkhZdG3JZukswDf7f0cwqPNk4K0sa+F97BxZthm/crw=
 github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
 github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
 github.com/charmbracelet/bubbletea v1.3.10 h1:otUDHWMMzQSB0Pkc87rm691KZ3SWa4KUlvF9nRvCICw=
@@ -34,14 +36,23 @@ github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f h1:Y/CXytFA4m6
 github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f/go.mod h1:vw97MGsxSvLiUE2X8qFplwetxpGLQrlU1Q9AUEIzCaM=
 github.com/fsnotify/fsnotify v1.9.0 h1:2Ml+OJNzbYCTzsxtv8vKSFD9PbJjmhYF14k/jKC7S9k=
 github.com/fsnotify/fsnotify v1.9.0/go.mod h1:8jBTzvmWwFyi3Pb8djgCCO5IBqzKJ/Jwo8TRcHyHii0=
-github.com/go-viper/mapstructure/v2 v2.4.0 h1:EBsztssimR/CONLSZZ04E8qAkxNYq4Qp9LvH92wZUgs=
-github.com/go-viper/mapstructure/v2 v2.4.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM=
+github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
+github.com/go-logr/logr v1.4.4 h1:tG4xh9yMsRCAiodLVTxyrkzSZ9+o0L1Kg/+cPVcbP/8=
+github.com/go-logr/logr v1.4.4/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
+github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
+github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
+github.com/go-viper/mapstructure/v2 v2.5.0 h1:vM5IJoUAy3d7zRSVtIwQgBj7BiWtMPfmPEgAXnvj1Ro=
+github.com/go-viper/mapstructure/v2 v2.5.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM=
+github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek=
+github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps=
 github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
 github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
 github.com/google/pprof v0.0.0-20260802141513-ef3492d7dac3 h1:LMLX+LgTNWpfvCBdFebv6EsYotImrt/Ppc5cXIriCSo=
 github.com/google/pprof v0.0.0-20260802141513-ef3492d7dac3/go.mod h1:jl5iWTm0/hd5PjEYEOuwAJ57L/CibdZfrqZ5XA5GrCk=
 github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
 github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
+github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0 h1:/Tnpcb2E0Pz/tN9s3bfEY2Q8ePCEX9iuS+cneUwncnw=
+github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0/go.mod h1:zOBXOsUaBSjKgmH4OGzV1esUpR3oUSCPYVd2cUBjKYY=
 github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
 github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
 github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
@@ -117,6 +128,24 @@ github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e h1:JVG44RsyaB9T2KIHavM
 github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e/go.mod h1:RbqR21r5mrJuqunuUZ/Dhy/avygyECGrLceyNeo4LiM=
 github.com/yuin/goldmark v1.8.6 h1:d0VcaP1sx9GkFVkoW+KtggpGi2KZ965i14b0+bDQST4=
 github.com/yuin/goldmark v1.8.6/go.mod h1:ip/1k0VRfGynBgxOz0yCqHrbZXhcjxyuS66Brc7iBKg=
+go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64=
+go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y=
+go.opentelemetry.io/otel v1.46.0 h1:FHt5/CDyVxi/8IM1CH7VE/rRgq3kLHa2mSTVMO8AWyc=
+go.opentelemetry.io/otel v1.46.0/go.mod h1:Gj3SEScelsNC45tp4nSxRYlS+f5iez7W8XPMCt905kE=
+go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.46.0 h1:OFnwLJr+pF3iHrlGSzbxyuo6/6HyBlnlN1CWEJmBVcw=
+go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.46.0/go.mod h1:716wFneO0ov19A2beH5hjfh9AK5z/VWNAtDijp1Y0/g=
+go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.46.0 h1:KrC1YrQeSt46ITMWAbgQx1M1eV1/1TKzttrBzymPmss=
+go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.46.0/go.mod h1:zDSEzoEqsOrgBeGvH66KRgxh90VonFyJqBHA0Pk3+rM=
+go.opentelemetry.io/otel/metric v1.46.0 h1:yBnkXvgV7AXFILZc5K6IZe/CBFF3OS7BJ8ov6/lj0K8=
+go.opentelemetry.io/otel/metric v1.46.0/go.mod h1:iPmdWqifKUdzziPkvvzIJXITl56fQx2mGM/DHLB3/2o=
+go.opentelemetry.io/otel/sdk v1.46.0 h1:h5CNQQjEbuQXY/JfZtgt3i7HVFV3aHPO2OAwO2eTYPI=
+go.opentelemetry.io/otel/sdk v1.46.0/go.mod h1:GAERFXFt5SYCEB+YiKUbMBeza6UaDH7GmGOZEfh2gSM=
+go.opentelemetry.io/otel/sdk/metric v1.46.0 h1:0piZ26EG4RBfebb2jhDH6ERCYHoVWduc3kLgPCwSnSE=
+go.opentelemetry.io/otel/sdk/metric v1.46.0/go.mod h1:I1PbKrdVc8Qu8HYVDNtqVIwLwjNrhsV/uFuxfwg8mO4=
+go.opentelemetry.io/otel/trace v1.46.0 h1:OULy7ccdJnZtJ0UDYFOIGaCmiWzJ8Vi2G/Rsu60qs1c=
+go.opentelemetry.io/otel/trace v1.46.0/go.mod h1:J7GAXweO77XSFkB/rmAqk9D6ihszhFjLU+d9WuUxDLI=
+go.opentelemetry.io/proto/otlp v1.11.0 h1:5rrYs0Ykyj50sdU/JU0x8etU+LubXWb+gED6TbEdMIk=
+go.opentelemetry.io/proto/otlp v1.11.0/go.mod h1:SmVizdCOAm3XBtG1g1NnOdhW6jtddT72hLMhv8VwA8E=
 go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
 go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
 go.yaml.in/yaml/v2 v2.4.4 h1:tuyd0P+2Ont/d6e2rl3be67goVK4R6deVxCUX5vyPaQ=
@@ -141,6 +170,14 @@ golang.org/x/text v0.42.0 h1:JbOZXgfeCPU9gacVtYliJqOhD+zhrEqK4LfdpmlUZqI=
 golang.org/x/text v0.42.0/go.mod h1:ojzP1Z+2QtioaF8DTtO8K5q7JWVVYwZKenzujK0Zd0E=
 golang.org/x/tools v0.49.0 h1:3NI7VXzL9+1WZD52Dx2ttoPwD5DWrFGpl9mFZDlmisI=
 golang.org/x/tools v0.49.0/go.mod h1:SJNXV9DBKT0UbdttsQjbfJlAE/q+y36++zo3uL3N0Oo=
+gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4=
+gonum.org/v1/gonum v0.17.0/go.mod h1:El3tOrEuMpv2UdMrbNlKEh9vd86bmQ6vqIcDwxEOc1E=
+google.golang.org/genproto/googleapis/api v0.0.0-20260819154853-08b0e4226688 h1:ax2KzoSRIZU/M0cIxri3pKxy99vniH1PVxWC6si/eZI=
+google.golang.org/genproto/googleapis/api v0.0.0-20260819154853-08b0e4226688/go.mod h1:1RJ9BQGyNdZwkGc1eTqkErfRZ6RJyYPHZo73BZ1vQqI=
+google.golang.org/genproto/googleapis/rpc v0.0.0-20260819154853-08b0e4226688 h1:cYNAzI2sUwhmCcoj9TxvihSrqsxt6uIkj3rDRhSDmW4=
+google.golang.org/genproto/googleapis/rpc v0.0.0-20260819154853-08b0e4226688/go.mod h1:DjtHYE8FKJLivXcBEjGwndXfIC23G0VpXiXKqG179uA=
+google.golang.org/grpc v1.83.1 h1:HIO0+BEtBP6soyqvqC8sNUjZ7bTs+0hFQuFF+RAy++Y=
+google.golang.org/grpc v1.83.1/go.mod h1:kDyl6SKsiHKt0uylY5gtn5cEjkrIOhQOGDgIc4JGwzQ=
 google.golang.org/protobuf v1.36.12 h1:pJOKDDOyeXErUroCihFAd5LQuwXBSpVnKGrj5o/fwxc=
 google.golang.org/protobuf v1.36.12/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
@@ -8,21 +8,16 @@ import (
 	"encoding/hex"
 	"errors"
 	"fmt"
-	"gitea.mynodes.xyz/ilker/llmgw/internal/release"
 	"io"
 	"log/slog"
 	"maps"
 	"math/rand/v2"
 	"net"
 	"net/http"
-	"net/url"
 	"os"
-	"os/exec"
 	"path/filepath"
 	"runtime"
 	"slices"
-	"strconv"
-	"strings"
 	"sync"
 	"time"

@@ -36,278 +31,9 @@ import (
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/supervisor"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/weights"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/agentsvc"
-	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
 )

-// Config configures the agent.
-type Config struct {
-	// NodeID defaults to the hostname. It must equal the common name of the
-	// agent's certificate, so set it only where the two differ.
-	NodeID string `koanf:"node_id"`
-	// Server is the control plane, and Servers is the same thing when there is
-	// more than one. Both are accepted: every configuration written before
-	// there could be a second control plane names one address, and none of
-	// them should have to change.
-	//
-	// The order is tried in order, and a refusal moves to the next. Only the
-	// leader accepts an agent, so a standby is skipped in the time it takes to
-	// be told so — which is what makes failover a reconnect rather than an
-	// edit to every node's configuration.
-	Server  string   `koanf:"server"`
-	Servers []string `koanf:"servers"`
-	// Hostname defaults to the machine's.
-	Hostname string `koanf:"hostname"`
-	// MgmtAddr is the address engines are reached on. It defaults to this
-	// machine's address on the route to the control plane, which is the one
-	// the control plane and the router can reach.
-	MgmtAddr string `koanf:"mgmt_addr"`
-	// Labels are free-form facts about a node for a profile's node_selector.
-	// Nothing needs them: memory, GPUs, engines and disk are measured, and a
-	// label restating one of those goes stale the day the machine changes.
-	Labels map[string]string `koanf:"labels"`
-
-	// MgmtInterface is the ethernet interface NCCL bootstraps over. It
-	// defaults to the interface holding MgmtAddr, then eth0.
-	MgmtInterface string `koanf:"mgmt_interface"`
-	// Fabrics labels this node's HCAs. It is explicit on purpose: OpenSM's
-	// default subnet prefix is the same on every fabric, so nothing can be
-	// inferred from the wire (plan §5.2).
-	Fabrics []inventory.FabricAttachment `koanf:"fabrics"`
-
-	HeartbeatInterval time.Duration `koanf:"heartbeat_interval"`
-	// ReconnectBase is the first reconnect delay; it backs off from there.
-	ReconnectBase time.Duration `koanf:"reconnect_base"`
-	ReconnectMax  time.Duration `koanf:"reconnect_max"`
-
-	// AdoptReadyTimeout bounds how long an adopted engine is given to answer a
-	// health check before the agent stops waiting on it.
-	//
-	// Shorter than a startup timeout on purpose: this engine is already
-	// running, so it is not loading a model, it is either serving or it is not.
-	AdoptReadyTimeout time.Duration `koanf:"adopt_ready_timeout"`
-
-	// TLS authenticates both ends of the control-plane stream. Leaving it unset
-	// keeps the plaintext behaviour, which only belongs on a trusted network.
-	TLS tlsconf.Config `koanf:"tls"`
-
-	// ModelDirs are the directories this node keeps model weights in, beyond
-	// WeightsCache, which is always looked in. Storage differs at every installation,
-	// so llmgw observes it rather than owning it: a Lustre mount, an image
-	// exported over the fabric, local NVMe and a copy made by hand all look
-	// identical from here, and placement only needs to know whether the file
-	// is present.
-	ModelDirs []string `koanf:"model_dirs"`
-
-	// EnginesDir is where the agent installs engines, one directory per
-	// build. Defaults to /opt/llmgw/engines.
-	EnginesDir string `koanf:"engines_dir"`
-
-	// WeightsCache is where a remote source is downloaded to. Defaults to the
-	// first entry of ModelDirs, because a node that says where its models live
-	// has already answered this question, and otherwise to
-	// /var/lib/llmgw-weights.
-	//
-	// Fetching is the simple arrangement: every node keeps its own copy. The
-	// exotic ones need no code — weights on a shared filer or an image exported
-	// over the fabric are simply present, and present means nothing is fetched.
-	WeightsCache string `koanf:"weights_cache"`
-
-	Discovery discovery.Options `koanf:"discovery"`
-	Engines   engines.Config    `koanf:"engines"`
-	Superv    supervisor.Config `koanf:"supervisor"`
-
-	Version string
-}
-
-// DefaultConfig returns sensible defaults.
-func DefaultConfig() Config {
-	// No MgmtInterface: normalize finds the one holding the management
-	// address, and falls back to eth0 only when it cannot. Defaulted here, it
-	// was never found — every node said eth0 unless told otherwise, and a node
-	// on ens3 or bond0 had NCCL bootstrap over an interface it does not have.
-	return Config{
-		HeartbeatInterval: 5 * time.Second,
-		ReconnectBase:     time.Second,
-		ReconnectMax:      30 * time.Second,
-		AdoptReadyTimeout: 60 * time.Second,
-		Superv:            supervisor.DefaultConfig(),
-		// NVML gives NVLink topology, so prefer it when this build has it.
-		Discovery: discovery.Options{PreferNVML: true},
-	}
-}
-
-func (c *Config) normalize() error {
-	d := DefaultConfig()
-	if err := c.TLS.Validate(); err != nil {
-		return err
-	}
-	if err := c.normalizeServerURL(); err != nil {
-		return err
-	}
-	// Everything below is found rather than written. A file that has to say
-	// which machine it is on, and at what address, is a file that differs on
-	// every node — and a copy taken from one node to the next registers the
-	// second as the first.
-	host, _ := os.Hostname()
-	if c.NodeID == "" {
-		c.NodeID = host
-	}
-	if c.NodeID == "" {
-		return errors.New("node_id is not set and the hostname could not be read")
-	}
-	if c.Hostname == "" {
-		c.Hostname = host
-	}
-	if c.MgmtAddr == "" {
-		c.MgmtAddr = addrToward(c.Server)
-	}
-	if c.MgmtInterface == "" {
-		c.MgmtInterface = interfaceOf(c.MgmtAddr)
-	}
-	if c.MgmtInterface == "" {
-		c.MgmtInterface = "eth0"
-	}
-	// The directory downloads land in is always one that is looked in. When
-	// they were two settings, a node could fill its disk with fetched weights
-	// that placement never saw — which is what every node here was doing.
-	if c.WeightsCache == "" && len(c.ModelDirs) > 0 {
-		c.WeightsCache = c.ModelDirs[0]
-	}
-	if c.WeightsCache == "" {
-		c.WeightsCache = DefaultWeightsCache
-	}
-	if !slices.Contains(c.ModelDirs, c.WeightsCache) {
-		c.ModelDirs = append(c.ModelDirs, c.WeightsCache)
-	}
-	if c.HeartbeatInterval <= 0 {
-		c.HeartbeatInterval = d.HeartbeatInterval
-	}
-	if c.ReconnectBase <= 0 {
-		c.ReconnectBase = d.ReconnectBase
-	}
-	if c.AdoptReadyTimeout <= 0 {
-		c.AdoptReadyTimeout = d.AdoptReadyTimeout
-	}
-	if c.ReconnectMax < c.ReconnectBase {
-		c.ReconnectMax = max(c.ReconnectBase, d.ReconnectMax)
-	}
-	return nil
-}
-
-// normalizeServerURL makes the configured address a URL connect can use, and
-// refuses the combination that would quietly undo the certificates: a scheme
-// that says plaintext while TLS is configured. Silently connecting anyway would
-// mean a cluster believing it has mutual authentication when it has none.
-func (c *Config) normalizeServerURL() error {
-	// One list, however it was written. Server first when both are given,
-	// because an operator adding a standby to an existing file expects the
-	// one already there to stay preferred.
-	all := make([]string, 0, 1+len(c.Servers))
-	if c.Server != "" {
-		all = append(all, c.Server)
-	}
-	all = append(all, c.Servers...)
-
-	seen := map[string]struct{}{}
-	out := make([]string, 0, len(all))
-	for _, s := range all {
-		fixed, err := normalizeOne(s, c.TLS.Enabled())
-		if err != nil {
-			return err
-		}
-		// A duplicate would be tried twice in a row on failover, which reads
-		// as the rotation not working.
-		if _, dup := seen[fixed]; dup {
-			continue
-		}
-		seen[fixed] = struct{}{}
-		out = append(out, fixed)
-	}
-	if len(out) == 0 {
-		return errors.New("server is required")
-	}
-	c.Servers = out
-	c.Server = out[0]
-	return nil
-}
-
-// DefaultWeightsCache is where fetched weights go when nothing says otherwise.
-const DefaultWeightsCache = "/var/lib/llmgw-weights"
-
-// addrToward is this machine's address on the route to the control plane.
-//
-// The address engines are reached on is whichever one the control plane can
-// reach, and the kernel already knows which that is: a UDP "connection" picks
-// the source address for the route without sending a packet. Asking the
-// interfaces instead would have to guess among docker bridges, VPNs and
-// link-local addresses.
-func addrToward(server string) string {
-	u, err := url.Parse(server)
-	if err != nil || u.Host == "" {
-		return ""
-	}
-	host := u.Hostname()
-	port := u.Port()
-	if port == "" {
-		port = "443"
-	}
-	conn, err := net.Dial("udp", net.JoinHostPort(host, port))
-	if err != nil {
-		return ""
-	}
-	defer conn.Close()
-	if a, ok := conn.LocalAddr().(*net.UDPAddr); ok {
-		return a.IP.String()
-	}
-	return ""
-}
-
-// interfaceOf names the interface that holds an address, or "" when none does.
-func interfaceOf(addr string) string {
-	ip := net.ParseIP(addr)
-	if ip == nil {
-		return ""
-	}
-	ifaces, err := net.Interfaces()
-	if err != nil {
-		return ""
-	}
-	for _, ifc := range ifaces {
-		addrs, err := ifc.Addrs()
-		if err != nil {
-			continue
-		}
-		for _, a := range addrs {
-			if n, ok := a.(*net.IPNet); ok && n.IP.Equal(ip) {
-				return ifc.Name
-			}
-		}
-	}
-	return ""
-}
-
-// normalizeOne applies the scheme rules to a single address.
-func normalizeOne(addr string, secure bool) (string, error) {
-	switch {
-	case strings.HasPrefix(addr, "https://"):
-		if !secure {
-			return "", fmt.Errorf("%s is an https:// address but no tls certificates are configured", addr)
-		}
-	case strings.HasPrefix(addr, "http://"):
-		if secure {
-			return "", fmt.Errorf("tls is configured but %s is an http:// address; use https://", addr)
-		}
-	case strings.Contains(addr, "://"):
-		return "", fmt.Errorf("server %q must be an http:// or https:// address", addr)
-	case secure:
-		return "https://" + addr, nil
-	default:
-		return "http://" + addr, nil
-	}
-	return addr, nil
-}
-
 // Agent runs on a GPU node.
 type Agent struct {
 	cfg  Config
@@ -375,6 +101,9 @@ type Agent struct {
 	smsErr map[string]string
 	// load measures the machine between heartbeats.
 	load *discovery.LoadSampler
+	// sensors reads heat and power: CPU and board from hwmon and RAPL, GPUs
+	// from nvidia-smi.
+	sensors *discovery.SensorSampler

 	// fetchWatchers are the shards waiting on each source, so progress on one
 	// download reaches every shard that asked for it. A fetch is single-flight
@@ -420,6 +149,7 @@ func New(cfg Config, httpClient connectHTTPClient, log *slog.Logger) (*Agent, er
 		logRing:      ring,
 		provKick:     make(chan struct{}, 1),
 		load:         discovery.NewLoadSampler(),
+		sensors:      discovery.NewSensorSampler(),
 	}
 	a.prov = &provision.Installer{Root: cfg.EnginesDir, Log: log, OnStatus: a.provisionStatus}
 	return a, nil
@@ -588,131 +318,6 @@ func (a *Agent) settleAdopted(sh *supervisor.Shard, o supervisor.Orphan) {
 		"shard", o.ShardID, "replica", o.ReplicaID)
 }

-// usableFabrics reports the fabrics this node can actually carry traffic on.
-//
-// Loud about what it drops. A fabric silently missing from a registration looks
-// identical to one never configured, and the operator who typed the adapter
-// name is the only person who can fix it — so the reason is logged here, on the
-// machine where both halves are visible, rather than inferred later from a
-// replica that will not converge.
-func (a *Agent) usableFabrics(inv discovery.Inventory) []inventory.FabricAttachment {
-	out := a.checkFabrics(inv.HCAs, true)
-	a.mu.Lock()
-	a.fabricsSent = fabricsFingerprint(out)
-	a.mu.Unlock()
-	return out
-}
-
-// checkFabrics is the declared fabrics this node can carry traffic on, from
-// the adapters as they are now. Loud logs every problem; the heartbeat checks
-// quietly and speaks only when the answer changes.
-func (a *Agent) checkFabrics(hcas []discovery.HCA, loud bool) []inventory.FabricAttachment {
-	declared := map[string][]string{}
-	for _, f := range a.cfg.Fabrics {
-		declared[f.Fabric] = f.HCAs
-	}
-
-	var out []inventory.FabricAttachment
-	for _, c := range discovery.CheckFabrics(declared, hcas) {
-		if loud {
-			for _, p := range c.Problems {
-				a.log.Error("a declared fabric adapter cannot carry traffic",
-					"fabric", c.Fabric, "problem", p)
-			}
-		}
-		if !c.OK() {
-			if loud {
-				a.log.Error("fabric not advertised; no declared adapter on it can carry traffic",
-					"fabric", c.Fabric,
-					"consequence", "no multi-node replica will be placed here")
-			}
-			continue
-		}
-		out = append(out, inventory.FabricAttachment{
-			Fabric: c.Fabric, HCAs: c.Usable, RateGbps: c.RateGbps,
-		})
-	}
-	return out
-}
-
-// fabricsFingerprint is enough to notice a fabric, an adapter or a rate
-// changing.
-func fabricsFingerprint(fs []inventory.FabricAttachment) string {
-	var b strings.Builder
-	for _, f := range fs {
-		fmt.Fprintf(&b, "%s:%s:%d;", f.Fabric, strings.Join(f.HCAs, ","), f.RateGbps)
-	}
-	return b.String()
-}
-
-// sampleIB re-reads the adapters for the heartbeat: every port's state and
-// counters, and the fabrics again when they no longer match what the control
-// plane was told.
-//
-// Every heartbeat, because it is a few small files per port, and because a
-// port that went down used to be invisible until the agent restarted — the
-// node went on being offered multi-node work that would hang on its first
-// collective.
-func (a *Agent) sampleIB(hb *llmgwv1.Heartbeat) {
-	a.mu.Lock()
-	had := len(a.inv.HCAs) > 0
-	a.mu.Unlock()
-	if !had && len(a.cfg.Fabrics) == 0 {
-		return
-	}
-	hcas, err := discovery.DiscoverIB(a.cfg.Discovery.IBRoot)
-	if err != nil {
-		return
-	}
-	a.attachSM(hcas)
-	for _, h := range hcas {
-		for _, p := range h.Ports {
-			ip := &llmgwv1.IBPort{
-				Hca: h.Name, Port: int32(p.Num), State: p.State, PhysState: p.PhysState,
-				RateGbps: int32(p.RateGbps), LinkLayer: string(p.LinkLayer), Counters: p.Counters,
-				NodeGuid: h.NodeGUID, PortGuid: p.PortGUID, Network: p.Network, GidPrefix: p.GIDPrefix,
-			}
-			a.mu.Lock()
-			if p.SM != nil {
-				ip.SmGuid, ip.SmLid, ip.SmPriority, ip.SmState = p.SM.GUID, int32(p.SM.LID), int32(p.SM.Priority), p.SM.State
-			} else {
-				ip.SmError = a.smErr[smKey(h.Name, p.Num)]
-			}
-			for _, r := range a.sms[smKey(h.Name, p.Num)] {
-				ip.SubnetManagers = append(ip.SubnetManagers, &llmgwv1.SubnetManager{
-					Guid: r.GUID, Lid: int32(r.LID), Priority: int32(r.Priority), State: r.State})
-			}
-			ip.SubnetManagersError = a.smsErr[smKey(h.Name, p.Num)]
-			a.mu.Unlock()
-			hb.IbPorts = append(hb.IbPorts, ip)
-		}
-	}
-	a.mu.Lock()
-	a.inv.HCAs = hcas // a reconnect registers what is true now
-	a.mu.Unlock()
-	if len(a.cfg.Fabrics) == 0 {
-		return
-	}
-	now := a.checkFabrics(hcas, false)
-	fp := fabricsFingerprint(now)
-	a.mu.Lock()
-	changed := fp != a.fabricsSent
-	a.fabricsSent = fp
-	a.mu.Unlock()
-	if !changed {
-		return
-	}
-	// Said once, with the reasons, the moment it happens.
-	a.checkFabrics(hcas, true)
-	a.log.Warn("the fabrics this node can carry traffic on changed", "now", fp)
-	hb.FabricsChanged = true
-	for _, f := range now {
-		hb.Fabrics = append(hb.Fabrics, &llmgwv1.FabricAttachment{
-			Fabric: f.Fabric, Hcas: f.HCAs, RateGbps: int32(f.RateGbps),
-		})
-	}
-}
-
 // weights builds the fetcher lazily, so an agent that never starts a shard
 // never needs a cache directory.
 func (a *Agent) weights() *weights.Fetcher {
@@ -949,6 +554,7 @@ func (a *Agent) heartbeat() *llmgwv1.AgentMessage {
 	// and a node that grew from 7 GB to 32 GB has to be seen to have grown
 	// without anybody restarting anything.
 	hb.MemoryTotalMb = int64(discovery.MemoryTotalMB())
+	hb.CpuCount = int32(runtime.NumCPU())

 	a.reprobeEngines()
 	a.mu.Lock()
@@ -1036,6 +642,16 @@ func (a *Agent) sampleLoad() *llmgwv1.NodeLoad {
 			ShardId: sh.ID, CpuCores: sh.CPUCores, MemoryRssBytes: sh.RSSBytes,
 		})
 	}
+	if a.sensors != nil {
+		se := a.sensors.Sample(context.Background())
+		out.CpuTempC, out.CpuPowerW, out.InletTempC = se.CPUTempC, se.CPUPowerW, se.InletTempC
+		for _, g := range se.GPUs {
+			out.Gpus = append(out.Gpus, &llmgwv1.GPUSensor{
+				Index: int32(g.Index), TempC: g.TempC, MemoryTempC: g.MemoryTempC,
+				PowerW: g.PowerW, PowerLimitW: g.PowerLimitW, Throttle: g.Throttle,
+			})
+		}
+	}
 	return out
 }

@@ -1275,6 +891,17 @@ func (a *Agent) startShard(ctx context.Context, m *llmgwv1.StartShard, out chan<
 		stop()
 		spec.Profile.ModelPath = path
 	}
+	// Adapters are fetched like the model, and before the engine starts: it
+	// loads them at start, and one missing is a replica that answers for the
+	// base model under the adapter's name.
+	for i, ad := range spec.Profile.Adapters {
+		path, err := a.weights().Ensure(ctx, ad.Path)
+		if err != nil {
+			report(supervisor.Failed, fmt.Sprintf("adapter %s: %v", ad.Name, err), nil)
+			return
+		}
+		spec.Profile.Adapters[i].Path = path
+	}

 	cmds, err := eng.Commands(spec)
 	if err != nil {
@@ -1414,215 +1041,6 @@ func Client(cfg tlsconf.Config, timeout time.Duration) (*http.Client, error) {
 	return &http.Client{Transport: &http2.Transport{TLSClientConfig: tc}}, nil
 }

-// engines is what this node can run: what the agent installed and saw run,
-// and what the configuration names and answers. Nothing merely found on the
-// machine — a llama-server on PATH is somebody's, not necessarily one that
-// works, and placing on it is how a replica fails on a node nobody watches.
-func (a *Agent) engines(ctx context.Context) map[string]string {
-	a.engineOnce.Do(func() {
-		vers := a.reg.ProbeConfigured(ctx)
-		a.mu.Lock()
-		a.configuredVers, a.lastEngineProbe = vers, time.Now()
-		a.mu.Unlock()
-		if len(vers) > 0 {
-			a.log.Info("engines named in the configuration",
-				"engines", engines.ProbedNames(vers))
-		}
-	})
-	a.mu.Lock()
-	defer a.mu.Unlock()
-	vers := a.engineReportLocked()
-	// Whatever registration sends is what the control plane now knows, so a
-	// heartbeat repeats it only once it differs.
-	a.enginesSent = enginesFingerprint(vers)
-	return vers
-}
-
-// engineReportLocked merges the two sources. Installed wins: it is the one
-// that was seen to run. Called with mu held.
-func (a *Agent) engineReportLocked() map[string]string {
-	out := maps.Clone(a.configuredVers)
-	if out == nil {
-		out = map[string]string{}
-	}
-	maps.Copy(out, a.installedVers)
-	return out
-}
-
-// provisionEvery is how often installed engines are checked again. A library
-// removed by an unrelated upgrade is found here rather than by the next
-// replica to start.
-const provisionEvery = 10 * time.Minute
-
-// setProvision takes the control plane's list of wanted engines.
-func (a *Agent) setProvision(p *llmgwv1.Provision) {
-	wants := []provision.Want{}
-	for _, e := range p.GetEngines() {
-		wants = append(wants, provision.Want{Engine: e.GetEngine(), Version: e.GetVersion()})
-	}
-	a.mu.Lock()
-	a.provWants = wants
-	a.mu.Unlock()
-	select {
-	case a.provKick <- struct{}{}:
-	default:
-	}
-}
-
-// provisionLoop installs and re-checks the wanted engines, one pass at a time.
-func (a *Agent) provisionLoop(ctx context.Context) {
-	// After a failure the next pass comes sooner, and backs off: a lock held
-	// by a new machine's own updates clears in a minute or two, while ten
-	// minutes' wait left a node idle for no reason.
-	next := provisionEvery
-	retry := time.Minute
-	for {
-		select {
-		case <-ctx.Done():
-			return
-		case <-a.provKick:
-		case <-time.After(next):
-		}
-		a.mu.Lock()
-		wants := slices.Clone(a.provWants)
-		managed := a.provWants != nil
-		a.mu.Unlock()
-		if !managed {
-			continue
-		}
-
-		statuses := a.prov.Converge(ctx, wants)
-		next = provisionEvery
-		for _, st := range statuses {
-			if st.State == provision.Failed {
-				next, retry = retry, min(retry*2, provisionEvery)
-				break
-			}
-		}
-		if next == provisionEvery {
-			retry = time.Minute
-		}
-		installed := map[string]string{}
-		keep := map[string]bool{}
-		for _, st := range statuses {
-			keep[st.Engine] = true
-			if st.State == provision.Ready {
-				a.reg.UseInstalled(st.Engine, st.Binary)
-				installed[st.Engine] = st.Version
-			}
-		}
-		a.mu.Lock()
-		if enginesFingerprint(installed) != enginesFingerprint(a.installedVers) {
-			a.log.Info("engines installed and running", "engines", engines.ProbedNames(installed))
-		}
-		a.installedVers = installed
-		// An engine no longer asked for is no longer reported on.
-		for e := range a.provStatus {
-			if !keep[e] {
-				delete(a.provStatus, e)
-			}
-		}
-		a.mu.Unlock()
-	}
-}
-
-// provisionStatus records a change in one engine's installation.
-func (a *Agent) provisionStatus(st provision.Status) {
-	a.mu.Lock()
-	prev, had := a.provStatus[st.Engine]
-	a.provStatus[st.Engine] = st
-	a.mu.Unlock()
-	if !had || prev.State != st.State || prev.Detail != st.Detail {
-		lvl := slog.LevelInfo
-		if st.State == provision.Failed || st.State == provision.Unsupported {
-			lvl = slog.LevelWarn
-		}
-		a.log.Log(context.Background(), lvl, "engine installation",
-			"engine", st.Engine, "version", st.Version, "variant", st.Variant,
-			"state", st.State, "detail", st.Detail)
-	}
-}
-
-func provisionFingerprint(m map[string]provision.Status) string {
-	var b strings.Builder
-	for _, k := range slices.Sorted(maps.Keys(m)) {
-		st := m[k]
-		fmt.Fprintf(&b, "%s|%s|%s|%s|%s\n", st.Engine, st.Version, st.Variant, st.State, st.Detail)
-	}
-	return b.String()
-}
-
-// engineReprobe is how often the installed engines are looked for again.
-//
-// Minutes, not seconds: an engine is installed by somebody at a keyboard, so a
-// minute's delay in noticing is invisible, while SGLang's probe starts a
-// Python interpreter that imports it.
-const engineReprobe = time.Minute
-
-// reprobeEngines looks for installed engines again when it is time, off the
-// heartbeat goroutine so a slow probe never delays a heartbeat.
-func (a *Agent) reprobeEngines() {
-	a.mu.Lock()
-	due := !a.probingEngines && !a.lastEngineProbe.IsZero() &&
-		time.Since(a.lastEngineProbe) >= engineReprobe
-	if due {
-		a.probingEngines = true
-	}
-	a.mu.Unlock()
-	if !due {
-		return
-	}
-	go func() {
-		ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
-		defer cancel()
-		vers := a.reg.ProbeConfigured(ctx)
-
-		a.mu.Lock()
-		changed := enginesFingerprint(vers) != enginesFingerprint(a.configuredVers)
-		a.configuredVers, a.lastEngineProbe, a.probingEngines = vers, time.Now(), false
-		a.mu.Unlock()
-		if changed {
-			a.log.Info("engines named in the configuration changed",
-				"engines", engines.ProbedNames(vers))
-		}
-	}()
-}
-
-func enginesFingerprint(vers map[string]string) string {
-	var b strings.Builder
-	for _, k := range slices.Sorted(maps.Keys(vers)) {
-		fmt.Fprintf(&b, "%s=%s\n", k, vers[k])
-	}
-	return b.String()
-}
-
-// logsReply answers a request for one of this node's logs.
-func (a *Agent) logsReply(req *llmgwv1.LogsRequest) *llmgwv1.LogsReply {
-	n := int(req.GetLines())
-	if n <= 0 {
-		n = 200
-	}
-	n = min(n, agentLogLines)
-	shards := a.sup.IDs()
-	slices.Sort(shards)
-	out := &llmgwv1.LogsReply{
-		RequestId: req.GetRequestId(),
-		Sources:   append([]string{"agent"}, shards...),
-	}
-	switch src := req.GetSource(); src {
-	case "", "agent":
-		out.Lines = a.logRing.tail(n)
-	default:
-		sh, err := a.sup.Get(src)
-		if err != nil {
-			out.Error = "no engine " + src + " runs on this node"
-			return out
-		}
-		out.Lines = sh.Logs(n)
-	}
-	return out
-}
-
 // localizeSpec fills in what only this node knows.
 //
 // The interface NCCL bootstraps over is this node's, not the cluster's: the
@@ -1673,254 +1091,3 @@ func ResolveManagement(c Config) (addr, iface string) {
 	}
 	return addr, iface
 }
-
-func smKey(hca string, port int) string { return fmt.Sprintf("%s/%d", hca, port) }
-
-// scanFabric is discovery.ScanFabric, replaced in tests.
-var scanFabric = discovery.ScanFabric
-
-// querySMs is discovery.QuerySMs, replaced in tests.
-var querySMs = func(hca string, port int) ([]discovery.SMRecord, error) {
-	return discovery.QuerySMs(context.Background(), hca, port)
-}
-
-func splitSMKey(k string) (string, int) {
-	i := strings.LastIndexByte(k, '/')
-	port, _ := strconv.Atoi(k[i+1:])
-	return k[:i], port
-}
-
-// querySM is discovery.QuerySM, replaced in tests.
-var querySM = func(hca string, port int) (*discovery.SMInfo, error) {
-	return discovery.QuerySM(context.Background(), hca, port)
-}
-
-// attachSM puts on each active InfiniBand port the subnet manager it sees,
-// asking again once a minute and otherwise reusing the last answer.
-func (a *Agent) attachSM(hcas []discovery.HCA) {
-	a.mu.Lock()
-	stale := time.Since(a.smAsked) > time.Minute || a.sm == nil
-	a.mu.Unlock()
-	if stale {
-		sm, errs := map[string]*discovery.SMInfo{}, map[string]string{}
-		_ = discovery.AttachSM(hcas, func(hca string, port int) (*discovery.SMInfo, error) {
-			got, err := querySM(hca, port)
-			if err != nil {
-				errs[smKey(hca, port)] = err.Error()
-				return nil, err
-			}
-			sm[smKey(hca, port)] = got
-			return got, nil
-		})
-		// Every subnet manager on each fabric, through one port per master:
-		// the same list through every port of one fabric is one query too
-		// many for each.
-		sms, smsErr := map[string][]discovery.SMRecord{}, map[string]string{}
-		asked := map[string]bool{}
-		for k, v := range sm {
-			if asked[v.GUID] {
-				continue
-			}
-			asked[v.GUID] = true
-			hca, port := splitSMKey(k)
-			if recs, err := querySMs(hca, port); err != nil {
-				smsErr[k] = err.Error()
-			} else {
-				sms[k] = recs
-			}
-		}
-		a.mu.Lock()
-		for k, v := range sm {
-			if old := a.sm[k]; old != nil && old.GUID != v.GUID {
-				a.log.Warn("the master subnet manager changed", "port", k, "from", old.GUID, "to", v.GUID)
-			}
-		}
-		a.sm, a.smErr, a.smAsked = sm, errs, time.Now()
-		a.sms, a.smsErr = sms, smsErr
-		a.mu.Unlock()
-		return
-	}
-	a.mu.Lock()
-	defer a.mu.Unlock()
-	for i := range hcas {
-		for j := range hcas[i].Ports {
-			if sm := a.sm[smKey(hcas[i].Name, hcas[i].Ports[j].Num)]; sm != nil && hcas[i].Ports[j].Active() {
-				hcas[i].Ports[j].SM = sm
-			}
-		}
-	}
-}
-
-// ensureDiags installs infiniband-diags on a node with adapters and without
-// it. sminfo is what tells which physical fabric a port is on; without it,
-// two separate fabrics given one name cannot be caught. Installed the way
-// engine dependencies are, from the machine's own package sources.
-func (a *Agent) ensureDiags(ctx context.Context) {
-	a.mu.Lock()
-	hasIB := len(a.inv.HCAs) > 0
-	a.mu.Unlock()
-	if !hasIB || a.prov == nil {
-		return
-	}
-	if _, err := exec.LookPath("sminfo"); err == nil {
-		return
-	}
-	a.log.Info("installing infiniband-diags, to tell which fabric each port is on")
-	pkgs := map[string][]string{"deb": {"infiniband-diags"}, "rpm": {"infiniband-diags"}}
-	if err := a.prov.InstallPackages(ctx, pkgs); err != nil {
-		a.log.Warn("could not install infiniband-diags; fabrics cannot be told apart by their subnet manager",
-			"err", err)
-	}
-}
-
-// FeatureFabricTest is the agent answering FabricTestRequest.
-const FeatureFabricTest = "fabric-test"
-
-// runWriteBW is discovery.RunWriteBW, and haveWriteBW whether it is
-// installed; both replaced in tests.
-var (
-	runWriteBW  = discovery.RunWriteBW
-	haveWriteBW = func() bool { _, err := exec.LookPath("ib_write_bw"); return err == nil }
-)
-
-// fabricTest runs this node's side of a bandwidth test with another.
-func (a *Agent) fabricTest(ctx context.Context, req *llmgwv1.FabricTestRequest) *llmgwv1.FabricTestReply {
-	start := time.Now()
-	reply := &llmgwv1.FabricTestReply{RequestId: req.GetRequestId()}
-	defer func() { reply.TookMs = time.Since(start).Milliseconds() }()
-
-	w := discovery.WriteBW{HCA: req.GetHca(), Port: int(req.GetPort()), GIDIndex: -1,
-		TCPPort: int(req.GetTcpPort()), Seconds: int(req.GetSeconds())}
-	if req.GetRole() == "client" {
-		w.Peer = req.GetPeer()
-	}
-	a.mu.Lock()
-	found := false
-	for _, h := range a.inv.HCAs {
-		for _, p := range h.Ports {
-			if h.Name == w.HCA && p.Num == w.Port {
-				found = true
-				// RoCE needs its v2 GID named, as NCCL does; InfiniBand uses the LID.
-				if p.LinkLayer == discovery.LinkEthernet && p.RoCEv2GID >= 0 {
-					w.GIDIndex = p.RoCEv2GID
-				}
-			}
-		}
-	}
-	a.mu.Unlock()
-	if !found {
-		reply.Error = fmt.Sprintf("this node has no port %s/%d", w.HCA, w.Port)
-		return reply
-	}
-	if !haveWriteBW() && a.prov != nil {
-		a.log.Info("installing perftest, for a bandwidth test with another node")
-		pkgs := map[string][]string{"deb": {"perftest"}, "rpm": {"perftest"}}
-		if err := a.prov.InstallPackages(ctx, pkgs); err != nil {
-			reply.Error = "installing perftest: " + err.Error()
-			return reply
-		}
-	}
-	gbps, out, err := runWriteBW(ctx, w)
-	reply.Gbps = gbps
-	reply.Output = tailLines(out, 12)
-	if err != nil {
-		reply.Error = err.Error()
-	}
-	return reply
-}
-
-func tailLines(s string, n int) string {
-	lines := strings.Split(strings.TrimRight(s, "\n"), "\n")
-	if len(lines) > n {
-		lines = lines[len(lines)-n:]
-	}
-	return strings.Join(lines, "\n")
-}
-
-// discoverGPUFabric and discoverIMEX are replaced in tests.
-var (
-	discoverGPUFabric = discovery.DiscoverGPUFabric
-	discoverIMEX      = discovery.DiscoverIMEX
-)
-
-// sampleNVLink reads, once a minute, which NVLink domain each GPU is joined
-// to and whether IMEX is up. nvidia-smi -q takes a second or two, too long
-// for every heartbeat, and a fabric that was joined an hour ago rarely
-// changes in five seconds — but when a fabric manager restarts it does, for
-// every card at once.
-func (a *Agent) sampleNVLink(hb *llmgwv1.Heartbeat) {
-	a.mu.Lock()
-	gpus := slices.Clone(a.inv.GPUs)
-	due := time.Since(a.nvlAsked) > time.Minute
-	if due {
-		a.nvlAsked = time.Now()
-	}
-	a.mu.Unlock()
-	if len(gpus) == 0 || !due {
-		return
-	}
-	ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
-	defer cancel()
-	fab, err := discoverGPUFabric(ctx, a.cfg.Discovery.NvidiaSMI)
-	if err != nil || len(fab) == 0 {
-		return
-	}
-	discovery.ApplyFabric(gpus, fab)
-	for _, g := range gpus {
-		if g.Fabric == nil {
-			continue
-		}
-		hb.GpuFabric = append(hb.GpuFabric, &llmgwv1.GPUFabricState{
-			Index: int32(g.Index), State: g.Fabric.State, Status: g.Fabric.Status,
-			ClusterUuid: g.Fabric.ClusterUUID, CliqueId: g.Fabric.CliqueID, Bandwidth: g.Fabric.Bandwidth,
-		})
-	}
-	m := discoverIMEX(ctx)
-	hb.Imex = &llmgwv1.IMEX{State: m.State, Channels: int32(m.Channels)}
-}
-
-// upgradeExit is the exit status after an upgrade: non-zero, so systemd's
-// Restart=on-failure starts the new binary, and distinct, so the journal says
-// why the agent went.
-const upgradeExit = 75
-
-// exitForUpgrade ends the process; a variable so a test can see it called
-// rather than have its own process end.
-var exitForUpgrade = func() { os.Exit(upgradeExit) }
-
-// upgradeTarget is the binary an upgrade replaces; a variable so a test
-// replaces a file of its own rather than the test binary running it.
-var upgradeTarget = os.Executable
-
-// upgrade replaces this agent's binary with a release and restarts into it.
-//
-// Engines are not touched: an agent restart leaves them running and adopts
-// them again, which is what makes upgrading a node something that can be done
-// while it serves.
-func (a *Agent) upgrade(ctx context.Context, m *llmgwv1.UpgradeAgent, out chan<- *llmgwv1.AgentMessage) {
-	reply := &llmgwv1.UpgradeReply{RequestId: m.GetRequestId(), Version: m.GetVersion()}
-	send := func() {
-		select {
-		case out <- &llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_Upgrade{Upgrade: reply}}:
-		case <-ctx.Done():
-		}
-	}
-	a.log.Info("upgrading", "to", m.GetVersion(), "from", a.cfg.Version)
-	self, err := upgradeTarget()
-	if err == nil {
-		dctx, cancel := context.WithTimeout(ctx, 10*time.Minute)
-		_, err = release.Install(dctx, nil, m.GetReleases(), m.GetVersion(), runtime.GOOS, runtime.GOARCH, self)
-		cancel()
-	}
-	if err != nil {
-		reply.Error = err.Error()
-		a.log.Error("upgrade failed; still running the old version", "to", m.GetVersion(), "err", err)
-		send()
-		return
-	}
-	send()
-	a.log.Info("upgraded; restarting into the new version", "to", m.GetVersion())
-	// A moment for the reply to leave before the stream goes.
-	time.Sleep(500 * time.Millisecond)
-	exitForUpgrade()
-}
@@ -0,0 +1,286 @@
+package agent
+
+import (
+	"errors"
+	"fmt"
+	"net"
+	"net/url"
+	"os"
+	"slices"
+	"strings"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/supervisor"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
+)
+
+// Config configures the agent.
+type Config struct {
+	// NodeID defaults to the hostname. It must equal the common name of the
+	// agent's certificate, so set it only where the two differ.
+	NodeID string `koanf:"node_id"`
+	// Server is the control plane, and Servers is the same thing when there is
+	// more than one. Both are accepted: every configuration written before
+	// there could be a second control plane names one address, and none of
+	// them should have to change.
+	//
+	// The order is tried in order, and a refusal moves to the next. Only the
+	// leader accepts an agent, so a standby is skipped in the time it takes to
+	// be told so — which is what makes failover a reconnect rather than an
+	// edit to every node's configuration.
+	Server  string   `koanf:"server"`
+	Servers []string `koanf:"servers"`
+	// Hostname defaults to the machine's.
+	Hostname string `koanf:"hostname"`
+	// MgmtAddr is the address engines are reached on. It defaults to this
+	// machine's address on the route to the control plane, which is the one
+	// the control plane and the router can reach.
+	MgmtAddr string `koanf:"mgmt_addr"`
+	// Labels are free-form facts about a node for a profile's node_selector.
+	// Nothing needs them: memory, GPUs, engines and disk are measured, and a
+	// label restating one of those goes stale the day the machine changes.
+	Labels map[string]string `koanf:"labels"`
+
+	// MgmtInterface is the ethernet interface NCCL bootstraps over. It
+	// defaults to the interface holding MgmtAddr, then eth0.
+	MgmtInterface string `koanf:"mgmt_interface"`
+	// Fabrics labels this node's HCAs. It is explicit on purpose: OpenSM's
+	// default subnet prefix is the same on every fabric, so nothing can be
+	// inferred from the wire (plan §5.2).
+	Fabrics []inventory.FabricAttachment `koanf:"fabrics"`
+
+	HeartbeatInterval time.Duration `koanf:"heartbeat_interval"`
+	// ReconnectBase is the first reconnect delay; it backs off from there.
+	ReconnectBase time.Duration `koanf:"reconnect_base"`
+	ReconnectMax  time.Duration `koanf:"reconnect_max"`
+
+	// AdoptReadyTimeout bounds how long an adopted engine is given to answer a
+	// health check before the agent stops waiting on it.
+	//
+	// Shorter than a startup timeout on purpose: this engine is already
+	// running, so it is not loading a model, it is either serving or it is not.
+	AdoptReadyTimeout time.Duration `koanf:"adopt_ready_timeout"`
+
+	// TLS authenticates both ends of the control-plane stream. Leaving it unset
+	// keeps the plaintext behaviour, which only belongs on a trusted network.
+	TLS tlsconf.Config `koanf:"tls"`
+
+	// ModelDirs are the directories this node keeps model weights in, beyond
+	// WeightsCache, which is always looked in. Storage differs at every installation,
+	// so llmgw observes it rather than owning it: a Lustre mount, an image
+	// exported over the fabric, local NVMe and a copy made by hand all look
+	// identical from here, and placement only needs to know whether the file
+	// is present.
+	ModelDirs []string `koanf:"model_dirs"`
+
+	// EnginesDir is where the agent installs engines, one directory per
+	// build. Defaults to /opt/llmgw/engines.
+	EnginesDir string `koanf:"engines_dir"`
+
+	// WeightsCache is where a remote source is downloaded to. Defaults to the
+	// first entry of ModelDirs, because a node that says where its models live
+	// has already answered this question, and otherwise to
+	// /var/lib/llmgw-weights.
+	//
+	// Fetching is the simple arrangement: every node keeps its own copy. The
+	// exotic ones need no code — weights on a shared filer or an image exported
+	// over the fabric are simply present, and present means nothing is fetched.
+	WeightsCache string `koanf:"weights_cache"`
+
+	Discovery discovery.Options `koanf:"discovery"`
+	Engines   engines.Config    `koanf:"engines"`
+	Superv    supervisor.Config `koanf:"supervisor"`
+
+	Version string
+}
+
+// DefaultConfig returns sensible defaults.
+func DefaultConfig() Config {
+	// No MgmtInterface: normalize finds the one holding the management
+	// address, and falls back to eth0 only when it cannot. Defaulted here, it
+	// was never found — every node said eth0 unless told otherwise, and a node
+	// on ens3 or bond0 had NCCL bootstrap over an interface it does not have.
+	return Config{
+		HeartbeatInterval: 5 * time.Second,
+		ReconnectBase:     time.Second,
+		ReconnectMax:      30 * time.Second,
+		AdoptReadyTimeout: 60 * time.Second,
+		Superv:            supervisor.DefaultConfig(),
+		// NVML gives NVLink topology, so prefer it when this build has it.
+		Discovery: discovery.Options{PreferNVML: true},
+	}
+}
+
+func (c *Config) normalize() error {
+	d := DefaultConfig()
+	if err := c.TLS.Validate(); err != nil {
+		return err
+	}
+	if err := c.normalizeServerURL(); err != nil {
+		return err
+	}
+	// Everything below is found rather than written. A file that has to say
+	// which machine it is on, and at what address, is a file that differs on
+	// every node — and a copy taken from one node to the next registers the
+	// second as the first.
+	host, _ := os.Hostname()
+	if c.NodeID == "" {
+		c.NodeID = host
+	}
+	if c.NodeID == "" {
+		return errors.New("node_id is not set and the hostname could not be read")
+	}
+	if c.Hostname == "" {
+		c.Hostname = host
+	}
+	if c.MgmtAddr == "" {
+		c.MgmtAddr = addrToward(c.Server)
+	}
+	if c.MgmtInterface == "" {
+		c.MgmtInterface = interfaceOf(c.MgmtAddr)
+	}
+	if c.MgmtInterface == "" {
+		c.MgmtInterface = "eth0"
+	}
+	// The directory downloads land in is always one that is looked in. When
+	// they were two settings, a node could fill its disk with fetched weights
+	// that placement never saw — which is what every node here was doing.
+	if c.WeightsCache == "" && len(c.ModelDirs) > 0 {
+		c.WeightsCache = c.ModelDirs[0]
+	}
+	if c.WeightsCache == "" {
+		c.WeightsCache = DefaultWeightsCache
+	}
+	if !slices.Contains(c.ModelDirs, c.WeightsCache) {
+		c.ModelDirs = append(c.ModelDirs, c.WeightsCache)
+	}
+	if c.HeartbeatInterval <= 0 {
+		c.HeartbeatInterval = d.HeartbeatInterval
+	}
+	if c.ReconnectBase <= 0 {
+		c.ReconnectBase = d.ReconnectBase
+	}
+	if c.AdoptReadyTimeout <= 0 {
+		c.AdoptReadyTimeout = d.AdoptReadyTimeout
+	}
+	if c.ReconnectMax < c.ReconnectBase {
+		c.ReconnectMax = max(c.ReconnectBase, d.ReconnectMax)
+	}
+	return nil
+}
+
+// normalizeServerURL makes the configured address a URL connect can use, and
+// refuses the combination that would quietly undo the certificates: a scheme
+// that says plaintext while TLS is configured. Silently connecting anyway would
+// mean a cluster believing it has mutual authentication when it has none.
+func (c *Config) normalizeServerURL() error {
+	// One list, however it was written. Server first when both are given,
+	// because an operator adding a standby to an existing file expects the
+	// one already there to stay preferred.
+	all := make([]string, 0, 1+len(c.Servers))
+	if c.Server != "" {
+		all = append(all, c.Server)
+	}
+	all = append(all, c.Servers...)
+
+	seen := map[string]struct{}{}
+	out := make([]string, 0, len(all))
+	for _, s := range all {
+		fixed, err := normalizeOne(s, c.TLS.Enabled())
+		if err != nil {
+			return err
+		}
+		// A duplicate would be tried twice in a row on failover, which reads
+		// as the rotation not working.
+		if _, dup := seen[fixed]; dup {
+			continue
+		}
+		seen[fixed] = struct{}{}
+		out = append(out, fixed)
+	}
+	if len(out) == 0 {
+		return errors.New("server is required")
+	}
+	c.Servers = out
+	c.Server = out[0]
+	return nil
+}
+
+// DefaultWeightsCache is where fetched weights go when nothing says otherwise.
+const DefaultWeightsCache = "/var/lib/llmgw-weights"
+
+// addrToward is this machine's address on the route to the control plane.
+//
+// The address engines are reached on is whichever one the control plane can
+// reach, and the kernel already knows which that is: a UDP "connection" picks
+// the source address for the route without sending a packet. Asking the
+// interfaces instead would have to guess among docker bridges, VPNs and
+// link-local addresses.
+func addrToward(server string) string {
+	u, err := url.Parse(server)
+	if err != nil || u.Host == "" {
+		return ""
+	}
+	host := u.Hostname()
+	port := u.Port()
+	if port == "" {
+		port = "443"
+	}
+	conn, err := net.Dial("udp", net.JoinHostPort(host, port))
+	if err != nil {
+		return ""
+	}
+	defer conn.Close()
+	if a, ok := conn.LocalAddr().(*net.UDPAddr); ok {
+		return a.IP.String()
+	}
+	return ""
+}
+
+// interfaceOf names the interface that holds an address, or "" when none does.
+func interfaceOf(addr string) string {
+	ip := net.ParseIP(addr)
+	if ip == nil {
+		return ""
+	}
+	ifaces, err := net.Interfaces()
+	if err != nil {
+		return ""
+	}
+	for _, ifc := range ifaces {
+		addrs, err := ifc.Addrs()
+		if err != nil {
+			continue
+		}
+		for _, a := range addrs {
+			if n, ok := a.(*net.IPNet); ok && n.IP.Equal(ip) {
+				return ifc.Name
+			}
+		}
+	}
+	return ""
+}
+
+// normalizeOne applies the scheme rules to a single address.
+func normalizeOne(addr string, secure bool) (string, error) {
+	switch {
+	case strings.HasPrefix(addr, "https://"):
+		if !secure {
+			return "", fmt.Errorf("%s is an https:// address but no tls certificates are configured", addr)
+		}
+	case strings.HasPrefix(addr, "http://"):
+		if secure {
+			return "", fmt.Errorf("tls is configured but %s is an http:// address; use https://", addr)
+		}
+	case strings.Contains(addr, "://"):
+		return "", fmt.Errorf("server %q must be an http:// or https:// address", addr)
+	case secure:
+		return "https://" + addr, nil
+	default:
+		return "http://" + addr, nil
+	}
+	return addr, nil
+}
@@ -0,0 +1,242 @@
+package discovery
+
+import (
+	"context"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"strconv"
+	"strings"
+	"sync"
+	"time"
+)
+
+// Sensors is what a node's own hardware says about heat and power: the CPUs,
+// the board where it has an inlet sensor, and each GPU. A zero is not
+// measured, never a reading of zero — a VM has none of these, and must not
+// look like a machine running at 0 °C on no power.
+type Sensors struct {
+	// CPUTempC is the hottest CPU sensor: the package, or the hottest core
+	// where no package reading exists.
+	CPUTempC float64
+	// CPUPowerW is the CPU packages' draw, from RAPL: the change in their
+	// energy counters over the time between two samples.
+	CPUPowerW float64
+	// InletTempC is the air coming in, where the board has a sensor it calls
+	// that. Most do not, and a CPU's temperature is not a substitute: it
+	// follows the load, not the room.
+	InletTempC float64
+	GPUs       []GPUSensor
+}
+
+// GPUSensor is one card's heat, power and whether it is slowing itself down.
+type GPUSensor struct {
+	Index       int
+	TempC       float64
+	MemoryTempC float64
+	PowerW      float64
+	PowerLimitW float64
+	// Throttle names what is holding the card's clocks down now:
+	// sw_power_cap, hw_slowdown, sw_thermal, hw_thermal, power_brake. Empty
+	// is running free. Idle and application clock settings are left out;
+	// they are not the card being held back.
+	Throttle []string
+}
+
+// sensorEvery bounds how often nvidia-smi is run. Heartbeats come more often
+// than temperatures change, and each run costs a few hundred milliseconds.
+const sensorEvery = 10 * time.Second
+
+// SensorSampler reads Sensors. The first sample has no CPU power, which needs
+// two readings of the energy counters.
+type SensorSampler struct {
+	sys string // /sys, or a test's copy of it
+	smi func(ctx context.Context, fields string) ([]byte, error)
+	now func() time.Time
+
+	mu         sync.Mutex
+	last       Sensors
+	lastAt     time.Time
+	prevEnergy map[string]uint64
+	prevAt     time.Time
+}
+
+// NewSensorSampler reads this machine.
+func NewSensorSampler() *SensorSampler {
+	return &SensorSampler{sys: "/sys", smi: runSMI, now: time.Now}
+}
+
+func runSMI(ctx context.Context, fields string) ([]byte, error) {
+	ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
+	defer cancel()
+	//nolint:gosec // fixed binary, fields from this file
+	return exec.CommandContext(ctx, "nvidia-smi", "--query-gpu="+fields,
+		"--format=csv,noheader,nounits").Output()
+}
+
+// Sample reads the sensors, or returns the last reading when that is recent.
+func (s *SensorSampler) Sample(ctx context.Context) Sensors {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	now := s.now()
+	if !s.lastAt.IsZero() && now.Sub(s.lastAt) < sensorEvery {
+		return s.last
+	}
+	out := Sensors{}
+	out.CPUTempC, out.InletTempC = s.hwmon()
+	out.CPUPowerW = s.rapl(now)
+	out.GPUs = s.gpus(ctx)
+	s.last, s.lastAt = out, now
+	return out
+}
+
+// hwmon reads the hottest CPU sensor and an inlet sensor, from every hwmon
+// device the kernel exposes — what lm_sensors reads, without it installed.
+func (s *SensorSampler) hwmon() (cpu, inlet float64) {
+	devs, _ := filepath.Glob(filepath.Join(s.sys, "class/hwmon/hwmon*"))
+	var pkg, core float64
+	for _, d := range devs {
+		name := readTrim(filepath.Join(d, "name"))
+		inputs, _ := filepath.Glob(filepath.Join(d, "temp*_input"))
+		for _, in := range inputs {
+			milli, err := strconv.ParseFloat(readTrim(in), 64)
+			if err != nil || milli <= 0 {
+				continue
+			}
+			c := milli / 1000
+			label := strings.ToLower(readTrim(strings.TrimSuffix(in, "_input") + "_label"))
+			switch {
+			case isCPUSensor(name) && (strings.HasPrefix(label, "package") || label == "tctl" || label == "tdie"):
+				pkg = max(pkg, c)
+			case isCPUSensor(name):
+				core = max(core, c)
+			case label == "cpu":
+				// A board's own reading of the CPU, as Dell's SMM gives.
+				core = max(core, c)
+			case strings.Contains(label, "inlet") || strings.Contains(label, "ambient") ||
+				strings.Contains(label, "intake") || label == "systin":
+				inlet = max(inlet, c)
+			}
+		}
+	}
+	if pkg > 0 {
+		return pkg, inlet
+	}
+	return core, inlet
+}
+
+func isCPUSensor(name string) bool {
+	switch name {
+	case "coretemp", "k10temp", "zenpower", "cpu_thermal":
+		return true
+	}
+	return false
+}
+
+// rapl is the CPU packages' power from their energy counters. Package
+// domains only: the dram and core subdomains are parts of a package, and
+// adding them would count the same energy twice.
+func (s *SensorSampler) rapl(now time.Time) float64 {
+	doms, _ := filepath.Glob(filepath.Join(s.sys, "class/powercap/intel-rapl:*"))
+	energy := map[string]uint64{}
+	wrap := map[string]uint64{}
+	for _, d := range doms {
+		if strings.Count(filepath.Base(d), ":") != 1 {
+			continue // intel-rapl:0:0 is a subdomain of intel-rapl:0
+		}
+		e, err := strconv.ParseUint(readTrim(filepath.Join(d, "energy_uj")), 10, 64)
+		if err != nil {
+			continue // unreadable without root on newer kernels
+		}
+		energy[d] = e
+		wrap[d], _ = strconv.ParseUint(readTrim(filepath.Join(d, "max_energy_range_uj")), 10, 64)
+	}
+	prev, prevAt := s.prevEnergy, s.prevAt
+	s.prevEnergy, s.prevAt = energy, now
+	dt := now.Sub(prevAt).Seconds()
+	if prev == nil || dt <= 0 || len(energy) == 0 {
+		return 0
+	}
+	var joules float64
+	for d, e := range energy {
+		p, ok := prev[d]
+		if !ok {
+			return 0 // a domain appeared: no fair difference this time
+		}
+		if e < p { // the counter wrapped
+			e += wrap[d]
+		}
+		joules += float64(e-p) / 1e6
+	}
+	return joules / dt
+}
+
+// The throttle reasons nvidia-smi reports, as bits. Only those that mean the
+// card is being held back are named.
+var throttleBits = []struct {
+	bit  uint64
+	name string
+}{
+	{0x4, "sw_power_cap"},
+	{0x8, "hw_slowdown"},
+	{0x20, "sw_thermal"},
+	{0x40, "hw_thermal"},
+	{0x80, "power_brake"},
+}
+
+// gpus asks nvidia-smi for each card's heat and power. The memory temperature
+// is not known to every driver, and a field nvidia-smi does not know fails the
+// whole query, so it is asked for first and dropped on failure.
+func (s *SensorSampler) gpus(ctx context.Context) []GPUSensor {
+	withMem := "index,temperature.gpu,power.draw,enforced.power.limit,clocks_throttle_reasons.active,temperature.memory"
+	out, err := s.smi(ctx, withMem)
+	if err != nil {
+		if out, err = s.smi(ctx, strings.TrimSuffix(withMem, ",temperature.memory")); err != nil {
+			return nil
+		}
+	}
+	var gs []GPUSensor
+	for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
+		f := strings.Split(line, ",")
+		if len(f) < 5 {
+			continue
+		}
+		for i := range f {
+			f[i] = strings.TrimSpace(f[i])
+		}
+		idx, err := strconv.Atoi(f[0])
+		if err != nil {
+			continue
+		}
+		g := GPUSensor{Index: idx, TempC: num(f[1]), PowerW: num(f[2]), PowerLimitW: num(f[3])}
+		if bits, err := strconv.ParseUint(strings.TrimPrefix(f[4], "0x"), 16, 64); err == nil {
+			for _, b := range throttleBits {
+				if bits&b.bit != 0 {
+					g.Throttle = append(g.Throttle, b.name)
+				}
+			}
+		}
+		if len(f) > 5 {
+			g.MemoryTempC = num(f[5])
+		}
+		gs = append(gs, g)
+	}
+	return gs
+}
+
+// num reads a figure nvidia-smi may give as "[N/A]" or "N/A".
+func num(s string) float64 {
+	v, err := strconv.ParseFloat(s, 64)
+	if err != nil || v < 0 {
+		return 0
+	}
+	return v
+}
+
+func readTrim(path string) string {
+	b, err := os.ReadFile(path) //nolint:gosec // sysfs
+	if err != nil {
+		return ""
+	}
+	return strings.TrimSpace(string(b))
+}
@@ -0,0 +1,169 @@
+package discovery
+
+import (
+	"context"
+	"errors"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+	"time"
+)
+
+func put(t *testing.T, root, rel, body string) {
+	t.Helper()
+	p := filepath.Join(root, rel)
+	if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
+		t.Fatal(err)
+	}
+	if err := os.WriteFile(p, []byte(body+"\n"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+}
+
+// dellHost is the layout of the Proxmox host these tests were written
+// against: Dell's SMM with a CPU and SODIMM sensors, coretemp with a package
+// and cores, and RAPL with a package and its dram subdomain.
+func dellHost(t *testing.T) string {
+	root := t.TempDir()
+	put(t, root, "class/hwmon/hwmon0/name", "dell_smm")
+	put(t, root, "class/hwmon/hwmon0/temp1_input", "55000")
+	put(t, root, "class/hwmon/hwmon0/temp1_label", "CPU")
+	put(t, root, "class/hwmon/hwmon0/temp2_input", "47000")
+	put(t, root, "class/hwmon/hwmon0/temp2_label", "SODIMM")
+	put(t, root, "class/hwmon/hwmon1/name", "coretemp")
+	put(t, root, "class/hwmon/hwmon1/temp1_input", "58000")
+	put(t, root, "class/hwmon/hwmon1/temp1_label", "Package id 0")
+	put(t, root, "class/hwmon/hwmon1/temp10_input", "61000")
+	put(t, root, "class/hwmon/hwmon1/temp10_label", "Core 8")
+	put(t, root, "class/powercap/intel-rapl:0/energy_uj", "1000000000")
+	put(t, root, "class/powercap/intel-rapl:0/max_energy_range_uj", "262143328850")
+	put(t, root, "class/powercap/intel-rapl:0:0/energy_uj", "5000000")
+	return root
+}
+
+func sampler(sys string, smi func(context.Context, string) ([]byte, error), clock *time.Time) *SensorSampler {
+	return &SensorSampler{sys: sys, smi: smi, now: func() time.Time { return *clock }}
+}
+
+func noSMI(context.Context, string) ([]byte, error) { return nil, errors.New("not found") }
+
+func TestAHostsCPUHeatAndPowerAreReadAsLmSensorsWould(t *testing.T) {
+	root := dellHost(t)
+	now := time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC)
+	s := sampler(root, noSMI, &now)
+
+	first := s.Sample(t.Context())
+	// The package reading, not the hotter core: the package is what the
+	// chip itself reports as its temperature.
+	if first.CPUTempC != 58 {
+		t.Errorf("CPU %.1f °C, expected the package's 58", first.CPUTempC)
+	}
+	if first.CPUPowerW != 0 {
+		t.Error("power from one reading of a counter is invented")
+	}
+	if first.InletTempC != 0 || first.GPUs != nil {
+		t.Errorf("sensors this host does not have were reported: %+v", first)
+	}
+
+	// 10 s later the package has used 650 J, and its dram subdomain some of
+	// that — counted once.
+	put(t, root, "class/powercap/intel-rapl:0/energy_uj", "1650000000")
+	put(t, root, "class/powercap/intel-rapl:0:0/energy_uj", "99000000")
+	now = now.Add(10 * time.Second)
+	if got := s.Sample(t.Context()).CPUPowerW; got < 64.9 || got > 65.1 {
+		t.Errorf("CPU power %.2f W, expected 65", got)
+	}
+}
+
+func TestACounterThatWrapsIsNotANegativeDraw(t *testing.T) {
+	root := dellHost(t)
+	now := time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC)
+	put(t, root, "class/powercap/intel-rapl:0/energy_uj", "262143000000")
+	s := sampler(root, noSMI, &now)
+	s.Sample(t.Context())
+	// 328,850 µJ to the wrap, then 499,671,150 past it: 500 J in 10 s.
+	put(t, root, "class/powercap/intel-rapl:0/energy_uj", "499671150")
+	now = now.Add(10 * time.Second)
+	if got := s.Sample(t.Context()).CPUPowerW; got < 49.9 || got > 50.1 {
+		t.Errorf("across a wrap: %.2f W, expected 50", got)
+	}
+}
+
+func TestAVMHasNoSensorsAndSaysNothing(t *testing.T) {
+	root := t.TempDir()
+	put(t, root, "class/thermal/cooling_device0/type", "Processor")
+	now := time.Now()
+	got := sampler(root, noSMI, &now).Sample(t.Context())
+	if got.CPUTempC != 0 || got.CPUPowerW != 0 || got.InletTempC != 0 || got.GPUs != nil {
+		t.Errorf("a machine with no sensors reported %+v", got)
+	}
+}
+
+func TestABoardsInletSensorIsTheRoomNotTheCPU(t *testing.T) {
+	root := t.TempDir()
+	put(t, root, "class/hwmon/hwmon2/name", "nct6798")
+	put(t, root, "class/hwmon/hwmon2/temp1_input", "24500")
+	put(t, root, "class/hwmon/hwmon2/temp1_label", "SYSTIN")
+	put(t, root, "class/hwmon/hwmon3/name", "k10temp")
+	put(t, root, "class/hwmon/hwmon3/temp1_input", "71250")
+	put(t, root, "class/hwmon/hwmon3/temp1_label", "Tctl")
+	now := time.Now()
+	got := sampler(root, noSMI, &now).Sample(t.Context())
+	if got.InletTempC != 24.5 || got.CPUTempC != 71.25 {
+		t.Errorf("inlet %.2f, CPU %.2f", got.InletTempC, got.CPUTempC)
+	}
+}
+
+func TestEachGPUSaysHowHotHowHungryAndWhatHoldsItBack(t *testing.T) {
+	var asked []string
+	smi := func(_ context.Context, fields string) ([]byte, error) {
+		asked = append(asked, fields)
+		return []byte(
+			"0, 71, 612.45, 700.00, 0x0000000000000000, 84\n" +
+				"1, 88, 698.10, 700.00, 0x0000000000000024, 95\n" + // sw_power_cap + sw_thermal
+				"2, 45, [N/A], 700.00, 0x0000000000000001, [N/A]\n"), nil // idle only
+	}
+	now := time.Now()
+	ss := sampler(t.TempDir(), smi, &now)
+	got := ss.Sample(t.Context())
+	if len(got.GPUs) != 3 {
+		t.Fatalf("%+v", got.GPUs)
+	}
+	g0, g1, g2 := got.GPUs[0], got.GPUs[1], got.GPUs[2]
+	if g0.TempC != 71 || g0.PowerW != 612.45 || g0.PowerLimitW != 700 || g0.MemoryTempC != 84 || g0.Throttle != nil {
+		t.Errorf("card 0 %+v", g0)
+	}
+	if strings.Join(g1.Throttle, ",") != "sw_power_cap,sw_thermal" {
+		t.Errorf("card 1 throttled by %v", g1.Throttle)
+	}
+	if g2.PowerW != 0 || g2.MemoryTempC != 0 || g2.Throttle != nil {
+		t.Errorf("card 2: N/A is not a reading, and idle is not being held back: %+v", g2)
+	}
+	// Within ten seconds the last reading is reused rather than running
+	// nvidia-smi on every heartbeat; after, it is asked again.
+	now = now.Add(5 * time.Second)
+	ss.Sample(t.Context())
+	if len(asked) != 1 {
+		t.Errorf("nvidia-smi was asked %d times within ten seconds", len(asked))
+	}
+	now = now.Add(6 * time.Second)
+	ss.Sample(t.Context())
+	if len(asked) != 2 {
+		t.Errorf("nvidia-smi was asked %d times after eleven seconds, expected 2", len(asked))
+	}
+}
+
+func TestADriverThatDoesNotKnowMemoryTemperatureStillReportsTheRest(t *testing.T) {
+	smi := func(_ context.Context, fields string) ([]byte, error) {
+		if strings.Contains(fields, "temperature.memory") {
+			return nil, errors.New(`Field "temperature.memory" is not a valid field to query.`)
+		}
+		return []byte("0, 66, 250.0, 300.0, 0x0000000000000040\n"), nil
+	}
+	now := time.Now()
+	got := sampler(t.TempDir(), smi, &now).Sample(t.Context())
+	if len(got.GPUs) != 1 || got.GPUs[0].TempC != 66 || strings.Join(got.GPUs[0].Throttle, ",") != "hw_thermal" {
+		t.Errorf("%+v", got.GPUs)
+	}
+}
@@ -40,6 +40,9 @@ type Profile struct {
 	ModelPath string
 	// ServedModelName is the name clients use, which need not match the path.
 	ServedModelName string
+	// Adapters are LoRA adapters to load beside the model, in order: an
+	// adapter's index is the id requests apply it by.
+	Adapters []Adapter
 }

 // ShardSpec is one shard of one replica on this node.
@@ -76,6 +79,10 @@ type ShardSpec struct {
 	// and accounting. Given to the engine in its environment rather than on
 	// its command line, which every user on the node can read.
 	APIKey string `json:"api_key,omitempty"`
+	// Threads is the CPU cores placement reserved for a shard with no GPU,
+	// and so what its engine computes on. Zero leaves it to the engine's own
+	// configuration.
+	Threads int `json:"threads,omitempty"`

 	// Pool and Fabric are control plane concepts the agent cannot derive. They
 	// are carried so the agent can report them back after a restart; Fabric
@@ -369,3 +376,10 @@ func quantizationOf(args []string) string {
 	}
 	return ""
 }
+
+// Adapter is a LoRA adapter an engine loads beside its model. Path is a
+// source until the agent has fetched it, a local file after.
+type Adapter struct {
+	Name string `json:"name"`
+	Path string `json:"path"`
+}
@@ -4,6 +4,7 @@ import (
 	"fmt"
 	"os"
 	"path/filepath"
+	"runtime"
 	"sort"
 	"strconv"
 	"strings"
@@ -21,8 +22,21 @@ type LlamaCppConfig struct {
 	// does not say. llama.cpp defaults to one, which would make every second
 	// request wait behind the first no matter what the router decided.
 	Parallel int `koanf:"parallel"`
+	// Threads is how many CPU threads a shard with no GPU computes on. Zero
+	// takes half this machine's CPUs.
+	//
+	// Not all of them, which is llama.cpp's own default: two busy servers
+	// that each spin on every CPU fight for them, and on an 8-vCPU node
+	// that made 7B generation fall from 10 tokens a second to under 1 each.
+	// At half, one server alone is about a third slower and two together
+	// keep 5.5 each. A node that only ever runs one CPU replica does better
+	// with all of them; say so here.
+	Threads int `koanf:"threads"`
 }

+// numCPU is the CPUs this process may run on; a variable for tests.
+var numCPU = runtime.NumCPU
+
 // LlamaCpp builds llama.cpp launch commands.
 //
 // It is the engine that makes a cluster without GPUs useful: llama.cpp runs a
@@ -118,6 +132,27 @@ func (l *LlamaCpp) Commands(s ShardSpec) ([]Command, error) {
 	if len(s.GPUIndices) > 0 && !hasFlag(p.Args, "--n-gpu-layers", "-ngl") {
 		args = append(args, "--n-gpu-layers", "999")
 	}
+	if len(s.GPUIndices) == 0 && !hasFlag(p.Args, "--threads", "-t") {
+		// What placement reserved first: it is what the node's other
+		// replicas were placed around.
+		n := s.Threads
+		if n <= 0 {
+			n = l.cfg.Threads
+		}
+		if n <= 0 {
+			n = max(numCPU()/2, 1)
+		}
+		args = append(args, "--threads", strconv.Itoa(n))
+	}
+	// Loaded at start, applied per request: a request that names an adapter
+	// carries "lora": [{"id": i, "scale": s}], and one that names none is the
+	// base model. Each is loaded at scale zero, not with --lora and
+	// --lora-init-without-apply: that leaves an adapter's scale at 1, and
+	// llama-server applies it at that scale to every request with no "lora"
+	// of its own — every request for the model itself.
+	for _, a := range p.Adapters {
+		args = append(args, "--lora-scaled", a.Path+":0")
+	}
 	args = append(args, p.Args...)

 	env := baseEnv(s)
@@ -261,3 +261,65 @@ func TestLlamaCppIsGivenItsKeyOutOfSightOfTheProcessList(t *testing.T) {
 		t.Error("llama-server was not given its key")
 	}
 }
+
+func TestACPUShardLeavesHalfTheMachineForAnother(t *testing.T) {
+	prev := numCPU
+	numCPU = func() int { return 8 }
+	t.Cleanup(func() { numCPU = prev })
+
+	threads := func(l *LlamaCpp, s ShardSpec) (string, bool) {
+		cmds, err := l.Commands(s)
+		if err != nil {
+			t.Fatal(err)
+		}
+		return flagValue(cmds[0].Args, "--threads")
+	}
+	if v, _ := threads(&LlamaCpp{}, llamaSpec(nil)); v != "4" {
+		t.Errorf("--threads %q on 8 CPUs, expected 4", v)
+	}
+	if v, _ := threads(&LlamaCpp{cfg: LlamaCppConfig{Threads: 8}}, llamaSpec(nil)); v != "8" {
+		t.Errorf("--threads %q with 8 configured", v)
+	}
+	if _, ok := threads(&LlamaCpp{}, llamaSpec([]int{0})); ok {
+		t.Error("a GPU shard was given a thread count")
+	}
+	s := llamaSpec(nil)
+	s.Profile.Args = []string{"-t", "2"}
+	cmds, _ := (&LlamaCpp{}).Commands(s)
+	if slices.Contains(cmds[0].Args, "--threads") {
+		t.Error("the profile's own -t was overridden")
+	}
+	numCPU = func() int { return 1 }
+	if v, _ := threads(&LlamaCpp{}, llamaSpec(nil)); v != "1" {
+		t.Errorf("--threads %q on one CPU", v)
+	}
+}
+
+func TestTheCoresPlacementReservedAreTheThreads(t *testing.T) {
+	s := llamaSpec(nil)
+	s.Threads = 3
+	cmds, err := (&LlamaCpp{cfg: LlamaCppConfig{Threads: 8}}).Commands(s)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if v, _ := flagValue(cmds[0].Args, "--threads"); v != "3" {
+		t.Errorf("--threads %q; what placement reserved should win over the node's setting", v)
+	}
+}
+
+func TestAdaptersAreLoadedAtStartAndAppliedToNoneByDefault(t *testing.T) {
+	s := llamaSpec(nil)
+	s.Profile.Adapters = []Adapter{{Name: "sql", Path: "/w/sql.gguf"}, {Name: "docs", Path: "/w/docs.gguf"}}
+	cmds, err := (&LlamaCpp{}).Commands(s)
+	if err != nil {
+		t.Fatal(err)
+	}
+	args := strings.Join(cmds[0].Args, " ")
+	if !strings.Contains(args, "--lora-scaled /w/sql.gguf:0 --lora-scaled /w/docs.gguf:0") {
+		t.Errorf("args %s: both adapters, in order, loaded at scale zero", args)
+	}
+	plain, _ := (&LlamaCpp{}).Commands(llamaSpec(nil))
+	if strings.Contains(strings.Join(plain[0].Args, " "), "--lora") {
+		t.Error("a model with no adapters was given lora flags")
+	}
+}
@@ -169,7 +169,10 @@ func (v *VLLM) rayCommand(s ShardSpec) (Command, bool) {

 	args := []string{"start", "--block"}
 	if s.Leader() {
-		args = append(args, "--head", "--port", strconv.Itoa(s.LeaderPort))
+		// Ray's head has no authentication: whoever reaches its port can run
+		// code on the node. Its dashboard, and the job API behind it, is the
+		// easiest way in, and nothing here uses it.
+		args = append(args, "--head", "--port", strconv.Itoa(s.LeaderPort), "--include-dashboard=false")
 	} else {
 		args = append(args, "--address", fmt.Sprintf("%s:%d", s.LeaderAddr, s.LeaderPort))
 	}
@@ -0,0 +1,345 @@
+package agent
+
+import (
+	"context"
+	"fmt"
+	"os/exec"
+	"slices"
+	"strconv"
+	"strings"
+	"time"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
+)
+
+// usableFabrics reports the fabrics this node can actually carry traffic on.
+//
+// Loud about what it drops. A fabric silently missing from a registration looks
+// identical to one never configured, and the operator who typed the adapter
+// name is the only person who can fix it — so the reason is logged here, on the
+// machine where both halves are visible, rather than inferred later from a
+// replica that will not converge.
+func (a *Agent) usableFabrics(inv discovery.Inventory) []inventory.FabricAttachment {
+	out := a.checkFabrics(inv.HCAs, true)
+	a.mu.Lock()
+	a.fabricsSent = fabricsFingerprint(out)
+	a.mu.Unlock()
+	return out
+}
+
+// checkFabrics is the declared fabrics this node can carry traffic on, from
+// the adapters as they are now. Loud logs every problem; the heartbeat checks
+// quietly and speaks only when the answer changes.
+func (a *Agent) checkFabrics(hcas []discovery.HCA, loud bool) []inventory.FabricAttachment {
+	declared := map[string][]string{}
+	for _, f := range a.cfg.Fabrics {
+		declared[f.Fabric] = f.HCAs
+	}
+
+	var out []inventory.FabricAttachment
+	for _, c := range discovery.CheckFabrics(declared, hcas) {
+		if loud {
+			for _, p := range c.Problems {
+				a.log.Error("a declared fabric adapter cannot carry traffic",
+					"fabric", c.Fabric, "problem", p)
+			}
+		}
+		if !c.OK() {
+			if loud {
+				a.log.Error("fabric not advertised; no declared adapter on it can carry traffic",
+					"fabric", c.Fabric,
+					"consequence", "no multi-node replica will be placed here")
+			}
+			continue
+		}
+		out = append(out, inventory.FabricAttachment{
+			Fabric: c.Fabric, HCAs: c.Usable, RateGbps: c.RateGbps,
+		})
+	}
+	return out
+}
+
+// fabricsFingerprint is enough to notice a fabric, an adapter or a rate
+// changing.
+func fabricsFingerprint(fs []inventory.FabricAttachment) string {
+	var b strings.Builder
+	for _, f := range fs {
+		fmt.Fprintf(&b, "%s:%s:%d;", f.Fabric, strings.Join(f.HCAs, ","), f.RateGbps)
+	}
+	return b.String()
+}
+
+// sampleIB re-reads the adapters for the heartbeat: every port's state and
+// counters, and the fabrics again when they no longer match what the control
+// plane was told.
+//
+// Every heartbeat, because it is a few small files per port, and because a
+// port that went down used to be invisible until the agent restarted — the
+// node went on being offered multi-node work that would hang on its first
+// collective.
+func (a *Agent) sampleIB(hb *llmgwv1.Heartbeat) {
+	a.mu.Lock()
+	had := len(a.inv.HCAs) > 0
+	a.mu.Unlock()
+	if !had && len(a.cfg.Fabrics) == 0 {
+		return
+	}
+	hcas, err := discovery.DiscoverIB(a.cfg.Discovery.IBRoot)
+	if err != nil {
+		return
+	}
+	a.attachSM(hcas)
+	for _, h := range hcas {
+		for _, p := range h.Ports {
+			ip := &llmgwv1.IBPort{
+				Hca: h.Name, Port: int32(p.Num), State: p.State, PhysState: p.PhysState,
+				RateGbps: int32(p.RateGbps), LinkLayer: string(p.LinkLayer), Counters: p.Counters,
+				NodeGuid: h.NodeGUID, PortGuid: p.PortGUID, Network: p.Network, GidPrefix: p.GIDPrefix,
+			}
+			a.mu.Lock()
+			if p.SM != nil {
+				ip.SmGuid, ip.SmLid, ip.SmPriority, ip.SmState = p.SM.GUID, int32(p.SM.LID), int32(p.SM.Priority), p.SM.State
+			} else {
+				ip.SmError = a.smErr[smKey(h.Name, p.Num)]
+			}
+			for _, r := range a.sms[smKey(h.Name, p.Num)] {
+				ip.SubnetManagers = append(ip.SubnetManagers, &llmgwv1.SubnetManager{
+					Guid: r.GUID, Lid: int32(r.LID), Priority: int32(r.Priority), State: r.State})
+			}
+			ip.SubnetManagersError = a.smsErr[smKey(h.Name, p.Num)]
+			a.mu.Unlock()
+			hb.IbPorts = append(hb.IbPorts, ip)
+		}
+	}
+	a.mu.Lock()
+	a.inv.HCAs = hcas // a reconnect registers what is true now
+	a.mu.Unlock()
+	if len(a.cfg.Fabrics) == 0 {
+		return
+	}
+	now := a.checkFabrics(hcas, false)
+	fp := fabricsFingerprint(now)
+	a.mu.Lock()
+	changed := fp != a.fabricsSent
+	a.fabricsSent = fp
+	a.mu.Unlock()
+	if !changed {
+		return
+	}
+	// Said once, with the reasons, the moment it happens.
+	a.checkFabrics(hcas, true)
+	a.log.Warn("the fabrics this node can carry traffic on changed", "now", fp)
+	hb.FabricsChanged = true
+	for _, f := range now {
+		hb.Fabrics = append(hb.Fabrics, &llmgwv1.FabricAttachment{
+			Fabric: f.Fabric, Hcas: f.HCAs, RateGbps: int32(f.RateGbps),
+		})
+	}
+}
+
+func smKey(hca string, port int) string { return fmt.Sprintf("%s/%d", hca, port) }
+
+// scanFabric is discovery.ScanFabric, replaced in tests.
+var scanFabric = discovery.ScanFabric
+
+// querySMs is discovery.QuerySMs, replaced in tests.
+var querySMs = func(hca string, port int) ([]discovery.SMRecord, error) {
+	return discovery.QuerySMs(context.Background(), hca, port)
+}
+
+func splitSMKey(k string) (string, int) {
+	i := strings.LastIndexByte(k, '/')
+	port, _ := strconv.Atoi(k[i+1:])
+	return k[:i], port
+}
+
+// querySM is discovery.QuerySM, replaced in tests.
+var querySM = func(hca string, port int) (*discovery.SMInfo, error) {
+	return discovery.QuerySM(context.Background(), hca, port)
+}
+
+// attachSM puts on each active InfiniBand port the subnet manager it sees,
+// asking again once a minute and otherwise reusing the last answer.
+func (a *Agent) attachSM(hcas []discovery.HCA) {
+	a.mu.Lock()
+	stale := time.Since(a.smAsked) > time.Minute || a.sm == nil
+	a.mu.Unlock()
+	if stale {
+		sm, errs := map[string]*discovery.SMInfo{}, map[string]string{}
+		_ = discovery.AttachSM(hcas, func(hca string, port int) (*discovery.SMInfo, error) {
+			got, err := querySM(hca, port)
+			if err != nil {
+				errs[smKey(hca, port)] = err.Error()
+				return nil, err
+			}
+			sm[smKey(hca, port)] = got
+			return got, nil
+		})
+		// Every subnet manager on each fabric, through one port per master:
+		// the same list through every port of one fabric is one query too
+		// many for each.
+		sms, smsErr := map[string][]discovery.SMRecord{}, map[string]string{}
+		asked := map[string]bool{}
+		for k, v := range sm {
+			if asked[v.GUID] {
+				continue
+			}
+			asked[v.GUID] = true
+			hca, port := splitSMKey(k)
+			if recs, err := querySMs(hca, port); err != nil {
+				smsErr[k] = err.Error()
+			} else {
+				sms[k] = recs
+			}
+		}
+		a.mu.Lock()
+		for k, v := range sm {
+			if old := a.sm[k]; old != nil && old.GUID != v.GUID {
+				a.log.Warn("the master subnet manager changed", "port", k, "from", old.GUID, "to", v.GUID)
+			}
+		}
+		a.sm, a.smErr, a.smAsked = sm, errs, time.Now()
+		a.sms, a.smsErr = sms, smsErr
+		a.mu.Unlock()
+		return
+	}
+	a.mu.Lock()
+	defer a.mu.Unlock()
+	for i := range hcas {
+		for j := range hcas[i].Ports {
+			if sm := a.sm[smKey(hcas[i].Name, hcas[i].Ports[j].Num)]; sm != nil && hcas[i].Ports[j].Active() {
+				hcas[i].Ports[j].SM = sm
+			}
+		}
+	}
+}
+
+// ensureDiags installs infiniband-diags on a node with adapters and without
+// it. sminfo is what tells which physical fabric a port is on; without it,
+// two separate fabrics given one name cannot be caught. Installed the way
+// engine dependencies are, from the machine's own package sources.
+func (a *Agent) ensureDiags(ctx context.Context) {
+	a.mu.Lock()
+	hasIB := len(a.inv.HCAs) > 0
+	a.mu.Unlock()
+	if !hasIB || a.prov == nil {
+		return
+	}
+	if _, err := exec.LookPath("sminfo"); err == nil {
+		return
+	}
+	a.log.Info("installing infiniband-diags, to tell which fabric each port is on")
+	pkgs := map[string][]string{"deb": {"infiniband-diags"}, "rpm": {"infiniband-diags"}}
+	if err := a.prov.InstallPackages(ctx, pkgs); err != nil {
+		a.log.Warn("could not install infiniband-diags; fabrics cannot be told apart by their subnet manager",
+			"err", err)
+	}
+}
+
+// FeatureFabricTest is the agent answering FabricTestRequest.
+const FeatureFabricTest = "fabric-test"
+
+// runWriteBW is discovery.RunWriteBW, and haveWriteBW whether it is
+// installed; both replaced in tests.
+var (
+	runWriteBW  = discovery.RunWriteBW
+	haveWriteBW = func() bool { _, err := exec.LookPath("ib_write_bw"); return err == nil }
+)
+
+// fabricTest runs this node's side of a bandwidth test with another.
+func (a *Agent) fabricTest(ctx context.Context, req *llmgwv1.FabricTestRequest) *llmgwv1.FabricTestReply {
+	start := time.Now()
+	reply := &llmgwv1.FabricTestReply{RequestId: req.GetRequestId()}
+	defer func() { reply.TookMs = time.Since(start).Milliseconds() }()
+
+	w := discovery.WriteBW{HCA: req.GetHca(), Port: int(req.GetPort()), GIDIndex: -1,
+		TCPPort: int(req.GetTcpPort()), Seconds: int(req.GetSeconds())}
+	if req.GetRole() == "client" {
+		w.Peer = req.GetPeer()
+	}
+	a.mu.Lock()
+	found := false
+	for _, h := range a.inv.HCAs {
+		for _, p := range h.Ports {
+			if h.Name == w.HCA && p.Num == w.Port {
+				found = true
+				// RoCE needs its v2 GID named, as NCCL does; InfiniBand uses the LID.
+				if p.LinkLayer == discovery.LinkEthernet && p.RoCEv2GID >= 0 {
+					w.GIDIndex = p.RoCEv2GID
+				}
+			}
+		}
+	}
+	a.mu.Unlock()
+	if !found {
+		reply.Error = fmt.Sprintf("this node has no port %s/%d", w.HCA, w.Port)
+		return reply
+	}
+	if !haveWriteBW() && a.prov != nil {
+		a.log.Info("installing perftest, for a bandwidth test with another node")
+		pkgs := map[string][]string{"deb": {"perftest"}, "rpm": {"perftest"}}
+		if err := a.prov.InstallPackages(ctx, pkgs); err != nil {
+			reply.Error = "installing perftest: " + err.Error()
+			return reply
+		}
+	}
+	gbps, out, err := runWriteBW(ctx, w)
+	reply.Gbps = gbps
+	reply.Output = tailLines(out, 12)
+	if err != nil {
+		reply.Error = err.Error()
+	}
+	return reply
+}
+
+func tailLines(s string, n int) string {
+	lines := strings.Split(strings.TrimRight(s, "\n"), "\n")
+	if len(lines) > n {
+		lines = lines[len(lines)-n:]
+	}
+	return strings.Join(lines, "\n")
+}
+
+// discoverGPUFabric and discoverIMEX are replaced in tests.
+var (
+	discoverGPUFabric = discovery.DiscoverGPUFabric
+	discoverIMEX      = discovery.DiscoverIMEX
+)
+
+// sampleNVLink reads, once a minute, which NVLink domain each GPU is joined
+// to and whether IMEX is up. nvidia-smi -q takes a second or two, too long
+// for every heartbeat, and a fabric that was joined an hour ago rarely
+// changes in five seconds — but when a fabric manager restarts it does, for
+// every card at once.
+func (a *Agent) sampleNVLink(hb *llmgwv1.Heartbeat) {
+	a.mu.Lock()
+	gpus := slices.Clone(a.inv.GPUs)
+	due := time.Since(a.nvlAsked) > time.Minute
+	if due {
+		a.nvlAsked = time.Now()
+	}
+	a.mu.Unlock()
+	if len(gpus) == 0 || !due {
+		return
+	}
+	ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
+	defer cancel()
+	fab, err := discoverGPUFabric(ctx, a.cfg.Discovery.NvidiaSMI)
+	if err != nil || len(fab) == 0 {
+		return
+	}
+	discovery.ApplyFabric(gpus, fab)
+	for _, g := range gpus {
+		if g.Fabric == nil {
+			continue
+		}
+		hb.GpuFabric = append(hb.GpuFabric, &llmgwv1.GPUFabricState{
+			Index: int32(g.Index), State: g.Fabric.State, Status: g.Fabric.Status,
+			ClusterUuid: g.Fabric.ClusterUUID, CliqueId: g.Fabric.CliqueID, Bandwidth: g.Fabric.Bandwidth,
+		})
+	}
+	m := discoverIMEX(ctx)
+	hb.Imex = &llmgwv1.IMEX{State: m.State, Channels: int32(m.Channels)}
+}
@@ -0,0 +1,34 @@
+package agent
+
+import (
+	"slices"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+)
+
+// logsReply answers a request for one of this node's logs.
+func (a *Agent) logsReply(req *llmgwv1.LogsRequest) *llmgwv1.LogsReply {
+	n := int(req.GetLines())
+	if n <= 0 {
+		n = 200
+	}
+	n = min(n, agentLogLines)
+	shards := a.sup.IDs()
+	slices.Sort(shards)
+	out := &llmgwv1.LogsReply{
+		RequestId: req.GetRequestId(),
+		Sources:   append([]string{"agent"}, shards...),
+	}
+	switch src := req.GetSource(); src {
+	case "", "agent":
+		out.Lines = a.logRing.tail(n)
+	default:
+		sh, err := a.sup.Get(src)
+		if err != nil {
+			out.Error = "no engine " + src + " runs on this node"
+			return out
+		}
+		out.Lines = sh.Logs(n)
+	}
+	return out
+}
@@ -0,0 +1,197 @@
+package agent
+
+import (
+	"context"
+	"fmt"
+	"log/slog"
+	"maps"
+	"slices"
+	"strings"
+	"time"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
+)
+
+// engines is what this node can run: what the agent installed and saw run,
+// and what the configuration names and answers. Nothing merely found on the
+// machine — a llama-server on PATH is somebody's, not necessarily one that
+// works, and placing on it is how a replica fails on a node nobody watches.
+func (a *Agent) engines(ctx context.Context) map[string]string {
+	a.engineOnce.Do(func() {
+		vers := a.reg.ProbeConfigured(ctx)
+		a.mu.Lock()
+		a.configuredVers, a.lastEngineProbe = vers, time.Now()
+		a.mu.Unlock()
+		if len(vers) > 0 {
+			a.log.Info("engines named in the configuration",
+				"engines", engines.ProbedNames(vers))
+		}
+	})
+	a.mu.Lock()
+	defer a.mu.Unlock()
+	vers := a.engineReportLocked()
+	// Whatever registration sends is what the control plane now knows, so a
+	// heartbeat repeats it only once it differs.
+	a.enginesSent = enginesFingerprint(vers)
+	return vers
+}
+
+// engineReportLocked merges the two sources. Installed wins: it is the one
+// that was seen to run. Called with mu held.
+func (a *Agent) engineReportLocked() map[string]string {
+	out := maps.Clone(a.configuredVers)
+	if out == nil {
+		out = map[string]string{}
+	}
+	maps.Copy(out, a.installedVers)
+	return out
+}
+
+// provisionEvery is how often installed engines are checked again. A library
+// removed by an unrelated upgrade is found here rather than by the next
+// replica to start.
+const provisionEvery = 10 * time.Minute
+
+// setProvision takes the control plane's list of wanted engines.
+func (a *Agent) setProvision(p *llmgwv1.Provision) {
+	wants := []provision.Want{}
+	for _, e := range p.GetEngines() {
+		wants = append(wants, provision.Want{Engine: e.GetEngine(), Version: e.GetVersion()})
+	}
+	a.mu.Lock()
+	a.provWants = wants
+	a.mu.Unlock()
+	select {
+	case a.provKick <- struct{}{}:
+	default:
+	}
+}
+
+// provisionLoop installs and re-checks the wanted engines, one pass at a time.
+func (a *Agent) provisionLoop(ctx context.Context) {
+	// After a failure the next pass comes sooner, and backs off: a lock held
+	// by a new machine's own updates clears in a minute or two, while ten
+	// minutes' wait left a node idle for no reason.
+	next := provisionEvery
+	retry := time.Minute
+	for {
+		select {
+		case <-ctx.Done():
+			return
+		case <-a.provKick:
+		case <-time.After(next):
+		}
+		a.mu.Lock()
+		wants := slices.Clone(a.provWants)
+		managed := a.provWants != nil
+		a.mu.Unlock()
+		if !managed {
+			continue
+		}
+
+		statuses := a.prov.Converge(ctx, wants)
+		next = provisionEvery
+		for _, st := range statuses {
+			if st.State == provision.Failed {
+				next, retry = retry, min(retry*2, provisionEvery)
+				break
+			}
+		}
+		if next == provisionEvery {
+			retry = time.Minute
+		}
+		installed := map[string]string{}
+		keep := map[string]bool{}
+		for _, st := range statuses {
+			keep[st.Engine] = true
+			if st.State == provision.Ready {
+				a.reg.UseInstalled(st.Engine, st.Binary)
+				installed[st.Engine] = st.Version
+			}
+		}
+		a.mu.Lock()
+		if enginesFingerprint(installed) != enginesFingerprint(a.installedVers) {
+			a.log.Info("engines installed and running", "engines", engines.ProbedNames(installed))
+		}
+		a.installedVers = installed
+		// An engine no longer asked for is no longer reported on.
+		for e := range a.provStatus {
+			if !keep[e] {
+				delete(a.provStatus, e)
+			}
+		}
+		a.mu.Unlock()
+	}
+}
+
+// provisionStatus records a change in one engine's installation.
+func (a *Agent) provisionStatus(st provision.Status) {
+	a.mu.Lock()
+	prev, had := a.provStatus[st.Engine]
+	a.provStatus[st.Engine] = st
+	a.mu.Unlock()
+	if !had || prev.State != st.State || prev.Detail != st.Detail {
+		lvl := slog.LevelInfo
+		if st.State == provision.Failed || st.State == provision.Unsupported {
+			lvl = slog.LevelWarn
+		}
+		a.log.Log(context.Background(), lvl, "engine installation",
+			"engine", st.Engine, "version", st.Version, "variant", st.Variant,
+			"state", st.State, "detail", st.Detail)
+	}
+}
+
+func provisionFingerprint(m map[string]provision.Status) string {
+	var b strings.Builder
+	for _, k := range slices.Sorted(maps.Keys(m)) {
+		st := m[k]
+		fmt.Fprintf(&b, "%s|%s|%s|%s|%s\n", st.Engine, st.Version, st.Variant, st.State, st.Detail)
+	}
+	return b.String()
+}
+
+// engineReprobe is how often the installed engines are looked for again.
+//
+// Minutes, not seconds: an engine is installed by somebody at a keyboard, so a
+// minute's delay in noticing is invisible, while SGLang's probe starts a
+// Python interpreter that imports it.
+const engineReprobe = time.Minute
+
+// reprobeEngines looks for installed engines again when it is time, off the
+// heartbeat goroutine so a slow probe never delays a heartbeat.
+func (a *Agent) reprobeEngines() {
+	a.mu.Lock()
+	due := !a.probingEngines && !a.lastEngineProbe.IsZero() &&
+		time.Since(a.lastEngineProbe) >= engineReprobe
+	if due {
+		a.probingEngines = true
+	}
+	a.mu.Unlock()
+	if !due {
+		return
+	}
+	go func() {
+		ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
+		defer cancel()
+		vers := a.reg.ProbeConfigured(ctx)
+
+		a.mu.Lock()
+		changed := enginesFingerprint(vers) != enginesFingerprint(a.configuredVers)
+		a.configuredVers, a.lastEngineProbe, a.probingEngines = vers, time.Now(), false
+		a.mu.Unlock()
+		if changed {
+			a.log.Info("engines named in the configuration changed",
+				"engines", engines.ProbedNames(vers))
+		}
+	}()
+}
+
+func enginesFingerprint(vers map[string]string) string {
+	var b strings.Builder
+	for _, k := range slices.Sorted(maps.Keys(vers)) {
+		fmt.Fprintf(&b, "%s=%s\n", k, vers[k])
+	}
+	return b.String()
+}
@@ -0,0 +1,58 @@
+package agent
+
+import (
+	"context"
+	"os"
+	"runtime"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/release"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+)
+
+// upgradeExit is the exit status after an upgrade: non-zero, so systemd's
+// Restart=on-failure starts the new binary, and distinct, so the journal says
+// why the agent went.
+const upgradeExit = 75
+
+// exitForUpgrade ends the process; a variable so a test can see it called
+// rather than have its own process end.
+var exitForUpgrade = func() { os.Exit(upgradeExit) }
+
+// upgradeTarget is the binary an upgrade replaces; a variable so a test
+// replaces a file of its own rather than the test binary running it.
+var upgradeTarget = os.Executable
+
+// upgrade replaces this agent's binary with a release and restarts into it.
+//
+// Engines are not touched: an agent restart leaves them running and adopts
+// them again, which is what makes upgrading a node something that can be done
+// while it serves.
+func (a *Agent) upgrade(ctx context.Context, m *llmgwv1.UpgradeAgent, out chan<- *llmgwv1.AgentMessage) {
+	reply := &llmgwv1.UpgradeReply{RequestId: m.GetRequestId(), Version: m.GetVersion()}
+	send := func() {
+		select {
+		case out <- &llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_Upgrade{Upgrade: reply}}:
+		case <-ctx.Done():
+		}
+	}
+	a.log.Info("upgrading", "to", m.GetVersion(), "from", a.cfg.Version)
+	self, err := upgradeTarget()
+	if err == nil {
+		dctx, cancel := context.WithTimeout(ctx, 10*time.Minute)
+		_, err = release.Install(dctx, nil, m.GetReleases(), m.GetVersion(), runtime.GOOS, runtime.GOARCH, self)
+		cancel()
+	}
+	if err != nil {
+		reply.Error = err.Error()
+		a.log.Error("upgrade failed; still running the old version", "to", m.GetVersion(), "err", err)
+		send()
+		return
+	}
+	send()
+	a.log.Info("upgraded; restarting into the new version", "to", m.GetVersion())
+	// A moment for the reply to leave before the stream goes.
+	time.Sleep(500 * time.Millisecond)
+	exitForUpgrade()
+}
@@ -137,7 +137,7 @@ func newCtlCmd() *cobra.Command {
 		// apply to it; it lives here because this is where operators look.
 		newBenchCmd(client, &asJSON),
 		newStatusCmd(client, &asJSON),
-		newModelCmd(client),
+		newModelCmd(client, &asJSON),
 		newScaleCmd(client),
 		newRestartCmd(client),
 		newRetryCmd(client),
@@ -146,6 +146,7 @@ func newCtlCmd() *cobra.Command {
 		newToolCheckCmd(),
 		newWeightsCmd(client, &asJSON),
 		newCapacityCmd(client, &asJSON),
+		newQuotaCmd(client, &asJSON),
 		newUsageCmd(client, &asJSON),
 		newCertCmd(),
 	)
@@ -253,6 +254,10 @@ func modelStatus(m adminapi.ModelView) string {
 	switch {
 	case m.Parked:
 		return "parked"
+	case m.External != "":
+		return "external: " + m.External
+	case m.HeldFor != "":
+		return "gave room to " + m.HeldFor
 	case m.Unplaceable:
 		return "cannot place"
 	case m.Min == 0 && m.Total == 0:
@@ -323,17 +328,27 @@ func newModelsCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 			}

 			w := tab()
-			fmt.Fprintln(w, "MODEL\tSERVING\tREPLICAS\tSTATUS\tALIASES\tPROFILES\tFALLBACKS")
+			fmt.Fprintln(w, "MODEL\tSERVING\tREPLICAS\tSTATUS\tCLASS\tRESERVES\tALIASES\tPROFILES\tFALLBACKS")
+			var warnings []string
 			for _, m := range models {
 				serving := "no"
 				if m.Ready > 0 {
 					serving = "yes"
 				}
-				fmt.Fprintf(w, "%s\t%s\t%d/%d\t%s\t%s\t%s\t%s\n",
-					m.Name, serving, m.Ready, m.Min, modelStatus(m),
+				fmt.Fprintf(w, "%s\t%s\t%d/%d\t%s\t%s\t%s\t%s\t%s\t%s\n",
+					m.Name, serving, m.Ready, m.Min, modelStatus(m), m.Class, reserves(m),
 					join(m.Aliases), join(m.Profiles), join(m.Fallbacks))
+				for _, x := range m.Warnings {
+					warnings = append(warnings, m.Name+": "+x)
+				}
 			}
-			return w.Flush()
+			if err := w.Flush(); err != nil {
+				return err
+			}
+			for _, x := range warnings {
+				fmt.Println("warning:", x)
+			}
+			return nil
 		},
 	}
 }
@@ -511,3 +526,22 @@ func dash(s string) string {
 	}
 	return s
 }
+
+// reserves is what one replica of a model holds of a node, with what it was
+// measured using beside it: "6G 2c (5.1G used)".
+func reserves(m adminapi.ModelView) string {
+	if m.MemoryGB == 0 && m.CPUCores == 0 {
+		return "-"
+	}
+	out := fmt.Sprintf("%dG", m.MemoryGB)
+	if m.CPUCores > 0 {
+		out += fmt.Sprintf(" %dc", m.CPUCores)
+	}
+	if m.MeasuredMemoryGB > 0 {
+		out += fmt.Sprintf(" (%.1fG used)", m.MeasuredMemoryGB)
+	}
+	if len(m.Warnings) > 0 {
+		out += " !"
+	}
+	return out
+}
@@ -172,6 +172,10 @@ func upgradeNodes(ctx context.Context, out io.Writer, client func() *ctlClient,
 		}
 	}
 	fmt.Fprintf(out, "%d node(s) upgraded to %s, %d already on it\n", done, v, skipped)
+	// The one machine this does not reach. Said every time, because it is
+	// exactly the kind of thing that stays three versions behind unnoticed.
+	fmt.Fprintf(out, "The model mirror's host is not upgraded by this; on it, run:\n"+
+		"  sudo llmgw upgrade %s --service llmgw-mirror\n", v)
 	return nil
 }

@@ -8,6 +8,7 @@ import (
 	"net/http"
 	"net/netip"
 	"os"
+	"slices"
 	"strings"
 	"time"

@@ -232,7 +233,7 @@ func newMirrorRemoveCmd(dir *string) *cobra.Command {
 func newMirrorServeCmd(dir *string) *cobra.Command {
 	var listen, apiListen, base string
 	var tlsc tlsconf.Config
-	var allow []string
+	var allow, clients []string
 	var allowPrivate bool

 	c := &cobra.Command{
@@ -259,6 +260,9 @@ func newMirrorServeCmd(dir *string) *cobra.Command {
 			if err := tlsc.Validate(); err != nil {
 				return err
 			}
+			if len(clients) > 0 && !tlsc.Enabled() {
+				return fmt.Errorf("--api-client names a certificate, and needs --tls-ca, --tls-cert and --tls-key")
+			}

 			out := cmd.OutOrStdout()
 			log := slog.Default()
@@ -282,7 +286,7 @@ func newMirrorServeCmd(dir *string) *cobra.Command {
 			}
 			admin := &http.Server{
 				Addr:              apiListen,
-				Handler:           apiGate(tlsc.Enabled(), allowed, api.Handler()),
+				Handler:           onlyClients(clients, apiGate(tlsc.Enabled(), allowed, api.Handler())),
 				ReadHeaderTimeout: 10 * time.Second,
 			}

@@ -294,6 +298,12 @@ func newMirrorServeCmd(dir *string) *cobra.Command {
 			switch {
 			case tlsc.Enabled():
 				fmt.Fprintf(out, "management API on %s, mutual TLS against %s\n", apiListen, tlsc.CA)
+				if len(clients) == 0 {
+					// The same root signs every node's certificate, so without a
+					// name any node could delete models here.
+					log.Warn("the management API takes any certificate from " + tlsc.CA +
+						", every node's included; name the control plane's with --api-client")
+				}
 			case len(allowed) > 0:
 				fmt.Fprintf(out, "management API on %s, for %s and this machine only\n", apiListen, strings.Join(allow, ", "))
 			default:
@@ -317,6 +327,8 @@ func newMirrorServeCmd(dir *string) *cobra.Command {
 		"the address nodes use to reach this mirror, as it should appear in catalog sources")
 	c.Flags().StringArrayVar(&allow, "api-allow", nil,
 		"an address or network the management API answers, besides this machine; repeatable (default: this machine only)")
+	c.Flags().StringArrayVar(&clients, "api-client", nil,
+		"with mutual TLS, the common name of a certificate the management API answers; repeatable (default: any from --tls-ca)")
 	c.Flags().BoolVar(&allowPrivate, "allow-private-sources", false,
 		"let downloads reach loopback, private and link-local addresses")
 	c.Flags().StringVar(&tlsc.CA, "tls-ca", "", "CA bundle that management clients must present a certificate from")
@@ -452,6 +464,23 @@ func apiGate(mtls bool, allowed []netip.Prefix, next http.Handler) http.Handler
 	return onlyFrom(allowed, next)
 }

+// onlyClients answers only certificates with one of these common names, when
+// any are given. The handshake has already checked the certificate against
+// the root; this narrows which of the root's certificates may manage.
+func onlyClients(names []string, next http.Handler) http.Handler {
+	if len(names) == 0 {
+		return next
+	}
+	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		if r.TLS == nil || len(r.TLS.VerifiedChains) == 0 ||
+			!slices.Contains(names, r.TLS.VerifiedChains[0][0].Subject.CommonName) {
+			http.Error(w, "this mirror is managed by its control plane only", http.StatusForbidden)
+			return
+		}
+		next.ServeHTTP(w, r)
+	})
+}
+
 // onlyFrom answers only this machine and the addresses allowed: the management
 // API can delete models and start downloads, and a mirror on its own machine
 // is reached over the network by one control plane and no one else.
@@ -1,6 +1,9 @@
 package cli

 import (
+	"crypto/tls"
+	"crypto/x509"
+	"crypto/x509/pkix"
 	"net/http"
 	"net/http/httptest"
 	"net/netip"
@@ -37,3 +40,27 @@ func TestTheMirrorAPIAnswersOnlyThisMachineUnlessTold(t *testing.T) {
 		t.Errorf("mTLS without an allowlist refused a client: %d", got)
 	}
 }
+
+func TestOnlyANamedCertificateManagesTheMirror(t *testing.T) {
+	ok := http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {})
+	with := func(cn string) *http.Request {
+		r := httptest.NewRequest("GET", "/", nil)
+		if cn != "" {
+			r.TLS = &tls.ConnectionState{VerifiedChains: [][]*x509.Certificate{{{Subject: pkix.Name{CommonName: cn}}}}}
+		}
+		return r
+	}
+	h := onlyClients([]string{"control-plane"}, ok)
+	for cn, want := range map[string]int{"control-plane": 200, "node01": 403, "": 403} {
+		w := httptest.NewRecorder()
+		h.ServeHTTP(w, with(cn))
+		if w.Code != want {
+			t.Errorf("%q: %d, expected %d", cn, w.Code, want)
+		}
+	}
+	w := httptest.NewRecorder()
+	onlyClients(nil, ok).ServeHTTP(w, with("node01"))
+	if w.Code != 200 {
+		t.Error("no names given narrowed it anyway")
+	}
+}
@@ -3,7 +3,11 @@ package cli
 import (
 	"encoding/json"
 	"fmt"
+	"os"
 	"os/user"
+	"slices"
+	"strconv"
+	"strings"
 	"time"

 	"github.com/knadh/koanf/parsers/yaml"
@@ -23,12 +27,15 @@ import (
 // minutes of outage. For a fixed production set that is a reasonable trade; for
 // a platform whose promise is that people can try things, it is the difference
 // between trying one and not bothering.
-func newModelCmd(client func() *ctlClient) *cobra.Command {
+func newModelCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 	c := &cobra.Command{
 		Use:   "model",
 		Short: "Add or remove models while the cluster is running",
 	}
-	c.AddCommand(newModelAddCmd(client), newModelRemoveCmd(client), newModelPlanCmd(client))
+	c.AddCommand(newModelAddCmd(client), newModelRemoveCmd(client), newModelPlanCmd(client),
+		newResourcesCmd(client), newClassCmd(client), newOwnerCmd(client),
+		newTargetCmd(client, asJSON), newAdaptersCmd(client),
+		newModelServeCmd(client))
 	return c
 }

@@ -126,6 +133,10 @@ func newModelRemoveCmd(client func() *ctlClient) *cobra.Command {

 // actor names who made a change, for the audit trail on a stored model.
 func actor() string {
+	// Under sudo, the person who ran it rather than root.
+	if u := os.Getenv("SUDO_USER"); u != "" && os.Geteuid() == 0 {
+		return u
+	}
 	if u, err := user.Current(); err == nil && u.Username != "" {
 		return u.Username
 	}
@@ -186,3 +197,139 @@ func newScaleCmd(client func() *ctlClient) *cobra.Command {
 		"how long an on-demand model stays up unasked-for; zero takes the cluster default")
 	return c
 }
+
+func newResourcesCmd(client func() *ctlClient) *cobra.Command {
+	var memory, cores int
+	var profile string
+	c := &cobra.Command{
+		Use:   "resources <model> --memory GB [--cores N]",
+		Short: "Set what a model reserves of a node: memory, and CPU cores",
+		Long: "--memory is the host RAM a replica needs: for a model with no GPU, its\n" +
+			"weights, KV cache and engine together. Placement reserves it on the node,\n" +
+			"so a node holds only as many replicas as it has memory for.\n\n" +
+			"--cores is the CPU cores a replica with no GPU computes on. They are\n" +
+			"reserved the same way, and become the engine's thread count, so two\n" +
+			"replicas on one node stop competing for the same cores. Zero reserves\n" +
+			"none and leaves the thread count to the node.\n\n" +
+			"Running replicas keep what they were started with; `llmgw ctl restart\n" +
+			"--model M` moves them to the new figures.",
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			if !cmd.Flags().Changed("memory") {
+				return fmt.Errorf("--memory is required")
+			}
+			body, err := json.Marshal(adminapi.ResourcesRequest{Profile: profile, MemoryGB: memory, CPUCores: cores})
+			if err != nil {
+				return err
+			}
+			var m catalog.Model
+			if err := client().post(cmd.Context(), "/api/v1/models/"+args[0]+"/resources", body, &m); err != nil {
+				return err
+			}
+			for _, p := range m.Profiles {
+				fmt.Fprintf(cmd.OutOrStdout(), "%s/%s: %d GB memory, %d cores\n", m.Name, p.Name, p.MinMemoryGB, p.CPUCores)
+			}
+			fmt.Fprintf(cmd.OutOrStdout(), "Running replicas keep their old figures until restarted: llmgw ctl restart --model %s\n", m.Name)
+			return nil
+		},
+	}
+	c.Flags().IntVar(&memory, "memory", 0, "host RAM a replica needs, in GB")
+	c.Flags().IntVar(&cores, "cores", 0, "CPU cores a replica with no GPU computes on; 0 reserves none")
+	c.Flags().StringVar(&profile, "profile", "", "one profile; every profile when left out")
+	return c
+}
+
+func newClassCmd(client func() *ctlClient) *cobra.Command {
+	return &cobra.Command{
+		Use:   "class <model> critical|standard|opportunistic|default",
+		Short: "Set how much a model's capacity is worth against another's",
+		Long: "When a model cannot be placed, replicas of a lower class are stopped to\n" +
+			"make room for it:\n\n" +
+			"  critical       never gives up the replicas of its minimum\n" +
+			"  standard       gives way to critical\n" +
+			"  opportunistic  gives way to both; the default for an on-demand model\n\n" +
+			"Within a class, a model below its minimum may take what another grew into\n" +
+			"under load, but not its minimum. \"default\" goes back to what the model's\n" +
+			"scaling implies: opportunistic when on demand, standard otherwise.",
+		Args: cobra.ExactArgs(2),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			class := args[1]
+			if class == "default" {
+				class = ""
+			}
+			body, _ := json.Marshal(map[string]string{"class": class})
+			var res map[string]string
+			if err := client().post(cmd.Context(), "/api/v1/models/"+args[0]+"/class", body, &res); err != nil {
+				return err
+			}
+			fmt.Fprintf(cmd.OutOrStdout(), "%s is %s\n", res["model"], res["class"])
+			return nil
+		},
+	}
+}
+
+func newAdaptersCmd(client func() *ctlClient) *cobra.Command {
+	var add, remove []string
+	c := &cobra.Command{
+		Use:   "adapters <model> [--add NAME=URL[@SCALE]] [--remove NAME]",
+		Short: "List, add or remove a model's LoRA adapters",
+		Long: "An adapter is asked for by its NAME as though it were a model, and is served\n" +
+			"by this model's replicas with it applied. URL is the adapter file, usually in\n" +
+			"the mirror; it must be a GGUF LoRA adapter for this model's architecture,\n" +
+			"which is checked before it is attached. Replicas load a changed set when\n" +
+			"restarted: llmgw ctl restart --model MODEL.",
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			name := args[0]
+			var models []adminapi.ModelView
+			if err := client().get(cmd.Context(), "/api/v1/models", &models); err != nil {
+				return err
+			}
+			var current []catalog.Adapter
+			found := false
+			for _, m := range models {
+				if m.Name == name {
+					current, found = m.Adapters, true
+				}
+			}
+			if !found {
+				return fmt.Errorf("no model %q", name)
+			}
+			out := cmd.OutOrStdout()
+			if len(add) == 0 && len(remove) == 0 {
+				if len(current) == 0 {
+					fmt.Fprintf(out, "%s has no adapters\n", name)
+				}
+				for _, a := range current {
+					fmt.Fprintf(out, "%s\tscale %g\t%s\n", a.Name, a.EffectiveScale(), a.Source)
+				}
+				return nil
+			}
+			next := slices.DeleteFunc(slices.Clone(current), func(a catalog.Adapter) bool { return slices.Contains(remove, a.Name) })
+			for _, spec := range add {
+				n, rest, ok := strings.Cut(spec, "=")
+				if !ok || n == "" || rest == "" {
+					return fmt.Errorf("--add %q is not NAME=URL[@SCALE]", spec)
+				}
+				ad := catalog.Adapter{Name: n, Source: rest}
+				if i := strings.LastIndex(rest, "@"); i > 0 {
+					if s, err := strconv.ParseFloat(rest[i+1:], 64); err == nil {
+						ad.Source, ad.Scale = rest[:i], s
+					}
+				}
+				next = append(next, ad)
+			}
+			body, _ := json.Marshal(adminapi.AdaptersRequest{Adapters: next})
+			var res adminapi.AdaptersRequest
+			if err := client().post(cmd.Context(), "/api/v1/models/"+name+"/adapters", body, &res); err != nil {
+				return err
+			}
+			fmt.Fprintf(out, "%s serves %d adapter(s); restart its replicas to load them: llmgw ctl restart --model %s\n",
+				name, len(res.Adapters), name)
+			return nil
+		},
+	}
+	c.Flags().StringArrayVar(&add, "add", nil, "NAME=URL[@SCALE]; repeatable")
+	c.Flags().StringArrayVar(&remove, "remove", nil, "an adapter's name; repeatable")
+	return c
+}
@@ -0,0 +1,81 @@
+package cli
+
+import (
+	"encoding/json"
+	"errors"
+	"fmt"
+	"time"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
+)
+
+func newModelServeCmd(client func() *ctlClient) *cobra.Command {
+	var (
+		req      modelserve.Request
+		onDemand bool
+	)
+	c := &cobra.Command{
+		Use:   "serve OWNER/REPO",
+		Short: "Take a model from the registry to a replica that answers, in one command",
+		Long: "Picks the GGUF file (--quant, Q4_K_M by default, or --file), has the mirror\n" +
+			"fetch it and checks what arrived against the registry's checksum, checks\n" +
+			"that the nodes' llama.cpp can load it and sizes its memory from its own\n" +
+			"header, adds it to the catalog, waits for a replica, and asks it to say\n" +
+			"hello. Each step is shown as it happens; the first that fails stops it.\n\n" +
+			"  llmgw ctl model serve bartowski/Qwen2.5-3B-Instruct-GGUF\n" +
+			"  llmgw ctl model serve Qwen/Qwen2.5-7B-Instruct-GGUF --quant Q5_K_M --name qwen-7b\n" +
+			"  llmgw ctl model serve owner/big-GGUF --on-demand",
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			req.Repo = args[0]
+			if onDemand {
+				req.Replicas = 0
+			}
+			body, _ := json.Marshal(req)
+			var job modelserve.Job
+			if err := client().post(cmd.Context(), "/api/v1/models/serve", body, &job); err != nil {
+				return err
+			}
+			out := cmd.OutOrStdout()
+			fmt.Fprintf(out, "job %s\n", job.ID)
+			shown := map[string]string{}
+			for {
+				for _, s := range job.Steps {
+					line := s.State + " " + s.Detail
+					if shown[s.Name] == line {
+						continue
+					}
+					shown[s.Name] = line
+					mark := map[string]string{modelserve.Running: "..", modelserve.Done: "ok",
+						modelserve.Failed: "!!"}[s.State]
+					fmt.Fprintf(out, "%-3s %-11s %s\n", mark, s.Name, s.Detail)
+				}
+				switch job.State {
+				case modelserve.Done:
+					fmt.Fprintf(out, "\n%s is served.\n", job.Model)
+					return nil
+				case modelserve.Failed:
+					return errors.New(job.Error)
+				}
+				time.Sleep(2 * time.Second)
+				if err := client().get(cmd.Context(), "/api/v1/jobs/"+job.ID, &job); err != nil {
+					return err
+				}
+			}
+		},
+	}
+	f := c.Flags()
+	f.StringVar(&req.Quant, "quant", "", "the quantisation to serve (default Q4_K_M)")
+	f.StringVar(&req.File, "file", "", "the file to serve, instead of choosing by --quant")
+	f.StringVar(&req.Name, "name", "", "the catalog name (default: from the repository)")
+	f.StringSliceVar(&req.Aliases, "alias", nil, "other names clients may use; repeatable")
+	f.StringVar(&req.Description, "description", "", "what it is for, shown to clients in /v1/models")
+	f.IntVar(&req.Replicas, "replicas", 1, "how many replicas to run")
+	f.BoolVar(&onDemand, "on-demand", false, "run none until first asked for")
+	f.IntVar(&req.CtxPool, "ctx-pool", 16384, "llama.cpp's --ctx-size, shared by its parallel slots")
+	f.IntVar(&req.CPUCores, "cores", 0, "CPU cores a replica computes on; 0 takes half the smallest node it fits")
+	f.IntVar(&req.ContextLength, "context-length", 4096, "what one request may use")
+	return c
+}
@@ -0,0 +1,234 @@
+package cli
+
+import (
+	"encoding/json"
+	"fmt"
+	"net/url"
+	"strconv"
+	"strings"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+func newQuotaCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
+	c := &cobra.Command{
+		Use:   "quota",
+		Short: "What parts of the tenancy tree are guaranteed of the cluster, and may hold",
+		Long: "A capacity quota belongs to an organization, team or project, and counts the\n" +
+			"GPUs, CPU cores and memory held by replicas of the models it owns (see\n" +
+			"`llmgw ctl model owner`), and of those its teams and projects own.\n\n" +
+			"The guarantee is kept for it: when it holds less and a model of its needs\n" +
+			"room, replicas other scopes hold beyond their own guarantees are stopped to\n" +
+			"make it — whatever their class. What it holds within its guarantee is\n" +
+			"never taken by another scope's model. Beyond it, it borrows: free room is\n" +
+			"used, and given back when asked for.\n\n" +
+			"The limit is the most it may hold, borrowing included; a model that would\n" +
+			"take it past is not given another replica.\n\n" +
+			"Scopes are written org:Acme, team:Acme/ml or project:Acme/ml/training, or\n" +
+			"with IDs. Changes take effect within half a minute.",
+	}
+	c.AddCommand(newQuotaListCmd(client, asJSON), newQuotaSetCmd(client), newQuotaClearCmd(client))
+	return c
+}
+
+func newQuotaListCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
+	return &cobra.Command{
+		Use:   "list",
+		Short: "Every quota, with what its scope holds now",
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			var qs []adminapi.QuotaView
+			if err := client().get(cmd.Context(), "/api/v1/quotas", &qs); err != nil {
+				return err
+			}
+			if *asJSON {
+				return json.NewEncoder(cmd.OutOrStdout()).Encode(qs)
+			}
+			if len(qs) == 0 {
+				fmt.Fprintln(cmd.OutOrStdout(), "No capacity quotas: everything shares the cluster by class alone.")
+				return nil
+			}
+			w := tab()
+			fmt.Fprintln(w, "SCOPE\tNAME\tGPUS\tCORES\tMEMORY GB")
+			for _, q := range qs {
+				fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\n", q.Scope, q.Name,
+					quotaCell(q.Holds.GPUs, q.Guarantee.GPUs, q.Limit.GPUs),
+					quotaCell(q.Holds.CPUCores, q.Guarantee.CPUCores, q.Limit.CPUCores),
+					quotaCell(q.Holds.MemoryGB, q.Guarantee.MemoryGB, q.Limit.MemoryGB))
+			}
+			if err := w.Flush(); err != nil {
+				return err
+			}
+			fmt.Fprintln(cmd.OutOrStdout(), "\nEach cell: holds / guaranteed (at most limit). - is not governed.")
+			return nil
+		},
+	}
+}
+
+// quotaCell is "6 / 8 (≤ 16)": what is held, what is guaranteed, the limit.
+func quotaCell(holds int, guarantee, limit *int) string {
+	if guarantee == nil && limit == nil {
+		return "-"
+	}
+	out := strconv.Itoa(holds) + " / "
+	if guarantee != nil {
+		out += strconv.Itoa(*guarantee)
+	} else {
+		out += "-"
+	}
+	if limit != nil {
+		out += fmt.Sprintf(" (≤ %d)", *limit)
+	}
+	return out
+}
+
+func newQuotaSetCmd(client func() *ctlClient) *cobra.Command {
+	var gpus, cores, mem, lgpus, lcores, lmem int
+	c := &cobra.Command{
+		Use:   "set SCOPE [--gpus N] [--cores N] [--memory GB] [--limit-gpus N] ...",
+		Short: "Set a scope's guarantee and limit, replacing what it had",
+		Args:  cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			f := cmd.Flags()
+			pick := func(name string, v int) *int {
+				if !f.Changed(name) {
+					return nil
+				}
+				return &v
+			}
+			req := adminapi.QuotaRequest{Scope: args[0],
+				Guarantee: store.Resources{GPUs: pick("gpus", gpus), CPUCores: pick("cores", cores), MemoryGB: pick("memory", mem)},
+				Limit:     store.Resources{GPUs: pick("limit-gpus", lgpus), CPUCores: pick("limit-cores", lcores), MemoryGB: pick("limit-memory", lmem)},
+			}
+			body, _ := json.Marshal(req)
+			var res map[string]string
+			if err := client().put(cmd.Context(), "/api/v1/quotas", body, &res); err != nil {
+				return err
+			}
+			fmt.Fprintf(cmd.OutOrStdout(), "%s: set; it takes effect within half a minute\n", res["scope"])
+			return nil
+		},
+	}
+	f := c.Flags()
+	f.IntVar(&gpus, "gpus", 0, "GPUs guaranteed")
+	f.IntVar(&cores, "cores", 0, "CPU cores guaranteed")
+	f.IntVar(&mem, "memory", 0, "memory guaranteed, in GB")
+	f.IntVar(&lgpus, "limit-gpus", 0, "the most GPUs it may hold")
+	f.IntVar(&lcores, "limit-cores", 0, "the most CPU cores it may hold")
+	f.IntVar(&lmem, "limit-memory", 0, "the most memory it may hold, in GB")
+	return c
+}
+
+func newQuotaClearCmd(client func() *ctlClient) *cobra.Command {
+	return &cobra.Command{
+		Use:   "clear SCOPE",
+		Short: "Remove a scope's quota",
+		Args:  cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			var res map[string]string
+			if err := client().delete(cmd.Context(), "/api/v1/quotas/"+url.PathEscape(args[0]), &res); err != nil {
+				return err
+			}
+			fmt.Fprintf(cmd.OutOrStdout(), "%s: quota removed\n", res["removed"])
+			return nil
+		},
+	}
+}
+
+func newOwnerCmd(client func() *ctlClient) *cobra.Command {
+	return &cobra.Command{
+		Use:   "owner <model> SCOPE|shared",
+		Short: "Set whose capacity quota a model counts against",
+		Long: "SCOPE is org:Acme, team:Acme/ml or project:Acme/ml/training. A shared\n" +
+			"model belongs to the cluster and counts against nobody's quota.",
+		Args: cobra.ExactArgs(2),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			owner := args[1]
+			if owner == "shared" {
+				owner = ""
+			}
+			body, _ := json.Marshal(map[string]string{"owner": owner})
+			var res map[string]string
+			if err := client().post(cmd.Context(), "/api/v1/models/"+args[0]+"/owner", body, &res); err != nil {
+				return err
+			}
+			if res["owner"] == "" {
+				fmt.Fprintf(cmd.OutOrStdout(), "%s is shared\n", res["model"])
+			} else {
+				fmt.Fprintf(cmd.OutOrStdout(), "%s counts against %s\n", res["model"], res["owner"])
+			}
+			return nil
+		},
+	}
+}
+
+func newTargetCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
+	var tps float64
+	var apply bool
+	c := &cobra.Command{
+		Use:   "target <model> --tokens-per-sec N [--apply]",
+		Short: "How many replicas a model needs for a throughput, and whether they fit",
+		Long: "Sized from what one replica was measured to serve while meeting its SLO\n" +
+			"(`llmgw ctl bench MODEL --save`), on the profile its replicas run on.\n" +
+			"--apply makes that the number it always runs, if the cluster has room.",
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			if tps <= 0 {
+				return fmt.Errorf("--tokens-per-sec is required")
+			}
+			var v adminapi.TargetView
+			path := "/api/v1/models/" + url.PathEscape(args[0]) + "/target?tokens_per_sec=" +
+				strconv.FormatFloat(tps, 'f', -1, 64)
+			if err := client().get(cmd.Context(), path, &v); err != nil {
+				return err
+			}
+			if *asJSON {
+				return json.NewEncoder(cmd.OutOrStdout()).Encode(v)
+			}
+			out := cmd.OutOrStdout()
+			fmt.Fprintf(out, "%s: %.0f tokens/s needs %d replicas at %.1f each (measured on %s); %d running\n",
+				v.Model, v.TokPerSec, v.Needed, v.PerReplica, v.Profile, v.Current)
+			switch {
+			case v.More == 0:
+				fmt.Fprintln(out, "It already has them.")
+			case v.Fits && len(v.Preempts) > 0:
+				fmt.Fprintf(out, "%d more; there is room once replicas of %s are stopped, which its class may do.\n",
+					v.More, strings.Join(v.Preempts, ", "))
+			case v.Fits:
+				fmt.Fprintf(out, "%d more, and there is room for them.\n", v.More)
+			default:
+				fmt.Fprintf(out, "%d more, and there is room for %d:\n", v.More, v.Room)
+				for _, b := range v.Blocked {
+					fmt.Fprintln(out, "  "+b)
+				}
+			}
+			if !apply || v.More == 0 {
+				return nil
+			}
+			if !v.Fits {
+				return fmt.Errorf("not applied: the cluster has no room for them")
+			}
+			var models []adminapi.ModelView
+			if err := client().get(cmd.Context(), "/api/v1/models", &models); err != nil {
+				return err
+			}
+			top := v.Needed
+			for _, m := range models {
+				if m.Name == v.Model {
+					top = max(top, m.Max)
+				}
+			}
+			body, _ := json.Marshal(adminapi.ScaleRequest{Min: v.Needed, Max: top})
+			if err := client().post(cmd.Context(), "/api/v1/models/"+url.PathEscape(v.Model)+"/scale", body, nil); err != nil {
+				return err
+			}
+			fmt.Fprintf(out, "Applied: %s runs %d (up to %d under load).\n", v.Model, v.Needed, top)
+			return nil
+		},
+	}
+	c.Flags().Float64Var(&tps, "tokens-per-sec", 0, "the throughput to size for")
+	c.Flags().BoolVar(&apply, "apply", false, "set it as the number of replicas the model always runs")
+	return c
+}
@@ -0,0 +1,67 @@
+package cli
+
+import (
+	"errors"
+	"fmt"
+	"os"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/release"
+)
+
+func newReleaseCmd() *cobra.Command {
+	c := &cobra.Command{
+		Use:   "release",
+		Short: "Sign releases, and make the key they are signed with",
+	}
+	c.AddCommand(&cobra.Command{
+		Use:   "keygen",
+		Short: "Make a release signing key pair",
+		Long: "Prints a public key and a private key. The private key goes into the CI\n" +
+			"secret RELEASE_SIGNING_KEY and nowhere else; the public key into\n" +
+			"internal/release/trusted.pub, which every build from then on carries\n" +
+			"and checks releases against. Run once, on a machine you trust.",
+		Args: cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			pub, priv, err := release.GenerateKey()
+			if err != nil {
+				return err
+			}
+			fmt.Fprintf(cmd.OutOrStdout(), "public:  %s\nprivate: %s\n", pub, priv)
+			fmt.Fprintln(cmd.ErrOrStderr(), "\nThe private key signs every release: put it in the CI secret "+
+				"RELEASE_SIGNING_KEY, then clear this terminal's scrollback.")
+			return nil
+		},
+	})
+	c.AddCommand(&cobra.Command{
+		Use:   "sign FILE",
+		Short: "Sign a release's checksums, writing FILE.sig",
+		Long: "Reads the private key from RELEASE_SIGNING_KEY, never from an argument,\n" +
+			"so it does not land in a shell history or a process list.",
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			key := os.Getenv("RELEASE_SIGNING_KEY")
+			if key == "" {
+				return errors.New("RELEASE_SIGNING_KEY is not set")
+			}
+			data, err := os.ReadFile(args[0])
+			if err != nil {
+				return err
+			}
+			sig, err := release.Sign(key, data)
+			if err != nil {
+				return err
+			}
+			if pub := release.TrustedKey(); pub != "" {
+				// Signed with a key this build would not accept is a release
+				// every upgraded machine refuses: stopped here instead.
+				if err := release.Verify(pub, data, sig); err != nil {
+					return fmt.Errorf("the signing key does not match this build's trusted key: %w", err)
+				}
+			}
+			return os.WriteFile(args[0]+".sig", []byte(sig+"\n"), 0o644) //nolint:gosec // public
+		},
+	})
+	return c
+}
@@ -47,6 +47,8 @@ func Root() *cobra.Command {
 		newSimEngineCmd(),
 		newDoctorCmd(),
 		newNodeCmd(),
+		newSelfUpgradeCmd(),
+		newReleaseCmd(),
 	)
 	return root
 }
@@ -0,0 +1,87 @@
+package cli
+
+import (
+	"errors"
+	"fmt"
+	"os"
+	"os/exec"
+	"runtime"
+	"strings"
+	"time"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/release"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/version"
+)
+
+// newSelfUpgradeCmd upgrades this machine's llmgw on its own, for the machines
+// `llmgw ctl upgrade` does not reach: a mirror host, above all, whose service
+// runs unprivileged with the filesystem read-only and so cannot replace its own
+// binary the way an agent does.
+func newSelfUpgradeCmd() *cobra.Command {
+	var releases, service string
+	c := &cobra.Command{
+		Use:   "upgrade VERSION [--service UNIT]",
+		Short: "Upgrade this machine's llmgw, and restart the service that runs it",
+		Long: "For a machine the control plane does not upgrade — the model mirror's host.\n" +
+			"The control plane and the nodes are upgraded with `llmgw ctl upgrade`.\n\n" +
+			"Fetches the release for this platform, checks it against the release's\n" +
+			"checksums, keeps the old binary beside the new one, and restarts --service,\n" +
+			"waiting for it to be running again. Run it with sudo.",
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			v := args[0]
+			out := cmd.OutOrStdout()
+			if !release.Tag.MatchString(v) {
+				return fmt.Errorf("%q is not a release version (v1.2.3 or v1.2.3-rc.4)", v)
+			}
+			if version.Version == v {
+				fmt.Fprintf(out, "this machine already runs %s\n", v)
+				return nil
+			}
+			if os.Geteuid() != 0 {
+				return errors.New("replacing this machine's binary needs root: run it with sudo")
+			}
+			self, err := os.Executable()
+			if err != nil {
+				return err
+			}
+			backup, err := release.Install(cmd.Context(), nil, releases, v, runtime.GOOS, runtime.GOARCH, self)
+			if err != nil {
+				return err
+			}
+			fmt.Fprintf(out, "%s -> %s: installed, checksum verified; the old binary is %s\n",
+				version.Version, v, backup)
+			if service == "" {
+				return nil
+			}
+			//nolint:gosec // the unit name is the operator's own flag
+			if b, err := exec.CommandContext(cmd.Context(), "systemctl", "restart", service).CombinedOutput(); err != nil {
+				return fmt.Errorf("restarting %s: %v: %s", service, err, strings.TrimSpace(string(b)))
+			}
+			// Running, and still running a few seconds on: a unit that starts
+			// and dies on a bad flag is "active" for a moment first.
+			deadline := time.Now().Add(30 * time.Second)
+			steady := 0
+			for time.Now().Before(deadline) {
+				//nolint:gosec // as above
+				b, _ := exec.CommandContext(cmd.Context(), "systemctl", "is-active", service).Output()
+				if strings.TrimSpace(string(b)) == "active" {
+					if steady++; steady >= 3 {
+						fmt.Fprintf(out, "%s restarted and running\n", service)
+						return nil
+					}
+				} else {
+					steady = 0
+				}
+				time.Sleep(2 * time.Second)
+			}
+			return fmt.Errorf("%s is not running after the upgrade; see journalctl -u %s, and %s holds the previous binary",
+				service, service, backup)
+		},
+	}
+	c.Flags().StringVar(&releases, "releases", release.DefaultBase, "where release binaries are fetched from")
+	c.Flags().StringVar(&service, "service", "", "the systemd unit to restart afterwards, e.g. llmgw-mirror")
+	return c
+}
@@ -5,6 +5,7 @@ import (
 	"crypto/tls"
 	"errors"
 	"fmt"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/sourcecheck"
 	"github.com/prometheus/client_golang/prometheus"
@@ -34,6 +35,7 @@ import (
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/obs"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/promq"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/router"
 	storepkg "gitea.mynodes.xyz/ilker/llmgw/internal/store"
@@ -89,6 +91,21 @@ func newServerCmd() *cobra.Command {
 					"or give --endpoint for a router-only run")
 			}

+			// Before the router, so its tracer is the configured one.
+			shutdownTracing, err := obs.SetupTracing(cmd.Context(), obs.TracingOptions{
+				Endpoint: cfg.Tracing.Endpoint, SampleRatio: cfg.Tracing.SampleRatio,
+				Service: "llmgw-server", Version: version.Version,
+			})
+			if err != nil {
+				return err
+			}
+			defer func() {
+				// Bounded: a collector that is gone must not hold up a shutdown.
+				sctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+				defer cancel()
+				_ = shutdownTracing(sctx)
+			}()
+
 			rt, err := router.New(cfg, log)
 			if err != nil {
 				return err
@@ -171,8 +188,16 @@ func newServerCmd() *cobra.Command {
 			if err != nil {
 				return err
 			}
+			// Kept past this block: serving a model from the registry needs
+			// both the admin API and the mirror, and the mirror is set up
+			// below.
+			var (
+				adminAPI   *adminapi.API
+				catalogSvc *modelsvc.Service
+			)
 			if admin != nil {
 				api := adminapi.New(admin)
+				adminAPI = api
 				api.SetReleases(cfg.Releases)
 				// Catalog changes need somewhere to persist, so this is wired
 				// only when there is a store. Without one, the endpoints say so
@@ -190,6 +215,7 @@ func newServerCmd() *cobra.Command {
 						admin.ReplaceCatalog(ctx, built)
 					}
 					api.SetModels(svc)
+					catalogSvc = svc
 					// And the mirror page, so a model that has just been
 					// fetched can be served without writing YAML by hand.
 					rt.SetCatalogAdmin(svc)
@@ -200,6 +226,8 @@ func newServerCmd() *cobra.Command {
 					// And what each group of the tenancy tree spent, which is
 					// the other half of a quota being reviewable.
 					api.SetUsage(st)
+					// Capacity quotas, set and read over this API.
+					api.SetQuotas(st)
 				}
 				api.Register(adminMux)

@@ -259,7 +287,16 @@ func newServerCmd() *cobra.Command {
 				// Browsing the registry is what turns "paste a resolve URL"
 				// into "find the model". It needs a way out to the internet
 				// from this machine, which the mirror's own fetching does not.
-				rt.SetRegistry(hf.New(cfg.Mirror.Registry, cfg.Mirror.Timeout))
+				registry := hf.New(cfg.Mirror.Registry, cfg.Mirror.Timeout)
+				rt.SetRegistry(registry)
+				// One command from a registry repository to a replica that
+				// answers: `llmgw ctl model serve`.
+				if adminAPI != nil && catalogSvc != nil {
+					adminAPI.SetServe(modelserve.New(modelserve.Deps{
+						Registry: registry, Mirror: mc, Catalog: catalogSvc, Fleet: admin,
+						LlamaVersion: cfg.WantedEngines()["llamacpp"], Log: log,
+					}))
+				}
 				log.Info("model mirror configured",
 					"api", cfg.Mirror.API, "mtls", cfg.Mirror.TLS.Enabled())
 			}
@@ -448,6 +485,12 @@ func startControlPlane(ctx context.Context, cfg config.Server, cat *catalog.Live
 		if st := h.Store(); st != nil {
 			lead = control.NewLeadership(st, cfg.Control.LeaseTTL, log)
 			ctl.SetLeadership(lead)
+			// The tenancy tree's capacity quotas: what each part is
+			// guaranteed and may hold.
+			ctl.Reconciler().SetQuotas(control.QuotaPolicy(st, log))
+			// And what each model was measured to serve, for scaling and for
+			// choosing between profiles.
+			ctl.Reconciler().SetMeasurements(control.Measurements(st, log))
 			log.Info("competing to be the control plane that places work",
 				"owner", lead.Owner())
 		}
@@ -389,17 +389,22 @@ func (in *serverInit) runInitTUI(ctx context.Context, out io.Writer) error {
 		return fmt.Errorf("stopped; run init again to finish what was started")
 	}

-	// The whole run is kept, root's only, since it holds the password. On a
-	// failure it is printed too, for the lines the screen had no room for;
-	// otherwise the screen's last frame says it, and only what must outlive
-	// the screen is printed again: the password, and what to do next.
+	// The whole run is kept, root's only, with the password taken out: it is
+	// shown once, on the screen, and a file on disk would outlive that. On a
+	// failure the run is printed too, for the lines the screen had no room
+	// for; otherwise the screen's last frame says it, and only what must
+	// outlive the screen is printed again: the password, and what to do next.
 	logMu.Lock()
 	defer logMu.Unlock()
 	whole := strings.Join(log, "\n") + "\n"
+	kept := whole
+	if fm.password != "" {
+		kept = strings.ReplaceAll(kept, fm.password, "[password shown once, not kept]")
+	}
 	saved := ""
 	if in.stateDir != "" {
 		p := filepath.Join(in.stateDir, "init.log")
-		if err := os.WriteFile(p, []byte(whole), 0o600); err == nil {
+		if err := os.WriteFile(p, []byte(kept), 0o600); err == nil {
 			saved = p
 		}
 	}
@@ -67,6 +67,10 @@ type Server struct {

 	// Monitoring is where the dashboard reads its history from.
 	Monitoring Monitoring `koanf:"monitoring"`
+	// Tracing sends a trace of each inference request to an OpenTelemetry
+	// collector. Off unless an endpoint is set here or in
+	// OTEL_EXPORTER_OTLP_ENDPOINT.
+	Tracing Tracing `koanf:"tracing"`

 	// Mirror points the control plane at a model mirror it can administer.
 	// Empty means there is none, and /admin/mirror stays unmounted.
@@ -120,6 +124,21 @@ type Monitoring struct {
 	Timeout time.Duration `koanf:"timeout"`
 }

+// Tracing is where request traces go, over OTLP/HTTP.
+//
+// One span per inference request, with admission, scheduling and each attempt
+// at an engine beneath it, and the trace carried on to the engine in a
+// traceparent header. The request's X-Request-Id is on the span, so a trace
+// and `llmgw ctl request` name the same request.
+type Tracing struct {
+	// Endpoint is the collector, for example http://otel.internal:4318.
+	// Empty falls back to OTEL_EXPORTER_OTLP_ENDPOINT, and to no tracing.
+	Endpoint string `koanf:"endpoint"`
+	// SampleRatio is the share of requests traced, 0 to 1, when the caller
+	// did not decide it with a sampled traceparent of its own. Zero takes 1.
+	SampleRatio float64 `koanf:"sample_ratio"`
+}
+
 // Mirror is a model mirror this control plane administers over its API.
 //
 // The mirror is a separate machine with a large disk, not part of the gateway,
@@ -359,7 +378,12 @@ type Admission struct {

 	QueueTTL             time.Duration `koanf:"queue_ttl"`
 	QueueCapacityPerBand int           `koanf:"queue_capacity_per_band"`
-	Interval             time.Duration `koanf:"interval"`
+	// QueueCapacityTotal bounds everything queued across bands. Reached, a
+	// request is not refused while a lower band has any queued: the newest of
+	// the lowest band is turned away instead, with a 503 and Retry-After.
+	// Zero takes half of all the bands' capacities together.
+	QueueCapacityTotal int           `koanf:"queue_capacity_total"`
+	Interval           time.Duration `koanf:"interval"`
 }

 // Prefix configures approximate prefix routing.
@@ -0,0 +1,45 @@
+package adminapi
+
+import (
+	"net/http/httptest"
+	"strings"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+)
+
+func TestAClaimedActorIsOneShortName(t *testing.T) {
+	for claimed, want := range map[string]string{
+		"ilker":                    "ilker",
+		"":                         "unknown",
+		"ilker\nlevel=ERROR msg=x": "ilkerlevelERRORmsgx",
+		strings.Repeat("a", 500):   strings.Repeat("a", 64),
+		"ops@example.org":          "ops@example.org",
+		"\x1b[31m":                 "31m",
+	} {
+		r := httptest.NewRequest("POST", "/", nil)
+		r.Header.Set("X-Llmgw-Actor", claimed)
+		if got := actorOf(r); got != want {
+			t.Errorf("%q: %q, expected %q", claimed, got, want)
+		}
+	}
+}
+
+func TestAReservationThatPlacementCannotTrustIsSaid(t *testing.T) {
+	cpu := func(mem int) catalog.Model {
+		return catalog.Model{Name: "m", Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp", MinMemoryGB: mem}}}
+	}
+	if w := reservationWarnings(cpu(0), 0); len(w) != 1 || !strings.Contains(w[0], "declares no memory") {
+		t.Errorf("no memory declared: %v", w)
+	}
+	if w := reservationWarnings(cpu(6), 8.2); len(w) != 1 || !strings.Contains(w[0], "uses 8.2 GB and reserves 6") {
+		t.Errorf("under-declared: %v", w)
+	}
+	if w := reservationWarnings(cpu(6), 6.4); len(w) != 0 {
+		t.Errorf("within a tenth is mapped weights, not a wrong figure: %v", w)
+	}
+	gpu := catalog.Model{Name: "g", Profiles: []catalog.Profile{{Name: "tp1", Engine: "vllm", GPUClass: "h100-80g"}}}
+	if w := reservationWarnings(gpu, 3); len(w) != 0 {
+		t.Errorf("a GPU model needs no host memory figure: %v", w)
+	}
+}
@@ -0,0 +1,71 @@
+package adminapi
+
+import (
+	"bytes"
+	"encoding/binary"
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+)
+
+// ggufStart is the start of a GGUF file with these string keys.
+func ggufStart(kv ...string) []byte {
+	var b bytes.Buffer
+	w := func(v any) { _ = binary.Write(&b, binary.LittleEndian, v) }
+	str := func(s string) { w(uint64(len(s))); b.WriteString(s) }
+	b.WriteString("GGUF")
+	w(uint32(3))
+	w(uint64(0))
+	w(uint64(len(kv) / 2))
+	for i := 0; i < len(kv); i += 2 {
+		str(kv[i])
+		w(uint32(8)) // string
+		str(kv[i+1])
+	}
+	b.Write(make([]byte, 4096))
+	return b.Bytes()
+}
+
+func TestOnlyAnAdapterForTheModelsArchitectureIsAttached(t *testing.T) {
+	files := map[string][]byte{
+		"/base.gguf":  ggufStart("general.architecture", "qwen2"),
+		"/model.gguf": ggufStart("general.architecture", "qwen2", "general.type", "model"),
+		"/llama.gguf": ggufStart("general.architecture", "llama", "general.type", "adapter"),
+		"/lora.gguf":  ggufStart("general.architecture", "qwen2", "general.type", "adapter"),
+	}
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.WriteHeader(http.StatusPartialContent)
+		_, _ = w.Write(files[r.URL.Path])
+	}))
+	defer srv.Close()
+
+	models := []catalog.Model{{
+		Name: "coder", Source: srv.URL + "/base.gguf",
+		Scaling:  catalog.Scaling{Min: 1, Max: 1},
+		Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 8}},
+	}}
+	for _, tc := range []struct {
+		file, want string
+		code       int
+	}{
+		{"/model.gguf", "not a LoRA adapter", http.StatusBadRequest},
+		{"/llama.gguf", "for llama models", http.StatusBadRequest},
+		{"/lora.gguf", "", http.StatusOK},
+	} {
+		a, _ := setup(t, models)
+		fm := &fakeModels{}
+		a.SetModels(fm)
+		mux := http.NewServeMux()
+		a.Register(mux)
+		req := httptest.NewRequest(http.MethodPost, "/api/v1/models/coder/adapters",
+			strings.NewReader(`{"adapters":[{"name":"fix","source":"`+srv.URL+tc.file+`"}]}`))
+		rec := httptest.NewRecorder()
+		mux.ServeHTTP(rec, req)
+		if rec.Code != tc.code || !strings.Contains(rec.Body.String(), tc.want) {
+			t.Errorf("%s: %d %s; want %d mentioning %q", tc.file, rec.Code, rec.Body.String(), tc.code, tc.want)
+		}
+	}
+}
@@ -10,12 +10,18 @@ import (
 	"context"
 	"encoding/json"
 	"fmt"
-	"gitea.mynodes.xyz/ilker/llmgw/internal/release"
 	"io"
 	"net/http"
+	"net/url"
 	"slices"
+	"strconv"
 	"strings"
 	"time"
+	"unicode"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/release"

 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/bench"
@@ -33,10 +39,13 @@ import (
 type API struct {
 	// releases is where release binaries are fetched from, for upgrades.
 	releases string
+	// serve runs registry-to-replica jobs; nil without a mirror and a store.
+	serve *modelserve.Server

 	models   Models
 	capacity Capacity
 	usage    Usage
+	quotas   Quotas
 	ctl      *control.Controller
 }

@@ -52,6 +61,29 @@ func (a *API) Register(mux *http.ServeMux) {
 	mux.HandleFunc("POST /api/v1/models", a.addModel)
 	mux.HandleFunc("DELETE /api/v1/models/{name}", a.removeModel)
 	mux.HandleFunc("POST /api/v1/models/{name}/scale", a.scaleModel)
+	mux.HandleFunc("POST /api/v1/models/{name}/resources", a.setResources)
+	a.registerQuotas(mux)
+	mux.HandleFunc("POST /api/v1/models/{name}/adapters", a.setAdapters)
+	mux.HandleFunc("GET /api/v1/models/{name}/target", a.target)
+	mux.HandleFunc("POST /api/v1/models/{name}/class", func(w http.ResponseWriter, r *http.Request) {
+		if a.models == nil {
+			writeError(w, http.StatusNotImplemented, "this control plane has no store")
+			return
+		}
+		var req struct {
+			Class string `json:"class"`
+		}
+		if err := json.NewDecoder(io.LimitReader(r.Body, 1<<12)).Decode(&req); err != nil {
+			writeError(w, http.StatusBadRequest, "the request is not valid JSON: "+err.Error())
+			return
+		}
+		m, err := a.models.SetClass(r.Context(), r.PathValue("name"), req.Class, actorOf(r))
+		if err != nil {
+			writeError(w, http.StatusBadRequest, err.Error())
+			return
+		}
+		writeJSON(w, http.StatusOK, map[string]string{"model": m.Name, "class": m.EffectiveClass()})
+	})
 	mux.HandleFunc("PUT /api/v1/models/{name}/capacity", a.putCapacity)
 	mux.HandleFunc("GET /api/v1/capacity", a.listCapacity)
 	mux.HandleFunc("GET /api/v1/usage", a.usageByScope)
@@ -83,8 +115,19 @@ func (a *API) Register(mux *http.ServeMux) {
 	mux.HandleFunc("POST /api/v1/fabric-test", a.fabricTest)
 	mux.HandleFunc("GET /api/v1/nodes/{id}/logs", a.nodeLogs)
 	mux.HandleFunc("POST /api/v1/restart", a.restart)
+	mux.HandleFunc("POST /api/v1/models/serve", a.serveModel)
+	mux.HandleFunc("GET /api/v1/jobs/{id}", a.job)
 	mux.HandleFunc("GET /api/v1/requests/{id}", a.request)
 	mux.HandleFunc("POST /api/v1/models/{name}/retry", a.retryModel)
+	mux.HandleFunc("GET /api/v1/models/{name}/headroom", func(w http.ResponseWriter, r *http.Request) {
+		limit, _ := strconv.Atoi(r.URL.Query().Get("limit"))
+		v, ok := a.Headroom(r.PathValue("name"), limit)
+		if !ok {
+			http.Error(w, "no such model", http.StatusNotFound)
+			return
+		}
+		writeJSON(w, http.StatusOK, v)
+	})
 }

 // --- responses ---
@@ -103,6 +146,8 @@ type NodeView struct {
 	// what decides whether a CPU profile can be placed here, so it belongs
 	// where someone looks when a model will not schedule.
 	MemoryGB int `json:"memory_gb,omitempty"`
+	// CPUCount is the machine's CPUs, 0 when the agent did not report them.
+	CPUCount int `json:"cpu_count,omitempty"`
 	// Engines are the inference engines installed here, as the agent found
 	// them. Absent means the agent is older than the probe and said nothing,
 	// which is not the same as having none — placement treats it that way too,
@@ -208,6 +253,11 @@ type LoadView struct {
 	// MemPercent is used over used plus available.
 	MemPercent float64 `json:"memory_percent"`
 	AgeSec     int64   `json:"age_seconds"`
+	// What the node's sensors say; zero is not measured.
+	CPUTempC   float64               `json:"cpu_temp_c,omitempty"`
+	CPUPowerW  float64               `json:"cpu_power_w,omitempty"`
+	InletTempC float64               `json:"inlet_temp_c,omitempty"`
+	GPUs       []inventory.GPUSensor `json:"gpu_sensors,omitempty"`
 }

 // loadFresh is how old a measurement may be and still be shown as current.
@@ -221,7 +271,8 @@ func loadView(l *inventory.Load, now time.Time) *LoadView {
 		CPUPercent: l.CPUPercent, CPUs: l.CPUCount,
 		Load1: l.Load1, Load5: l.Load5, Load15: l.Load15,
 		MemUsed: l.MemUsedBytes, MemTotal: l.MemUsedBytes + l.MemAvailBytes,
-		AgeSec: int64(now.Sub(l.At).Seconds()),
+		AgeSec:   int64(now.Sub(l.At).Seconds()),
+		CPUTempC: l.CPUTempC, CPUPowerW: l.CPUPowerW, InletTempC: l.InletTempC, GPUs: l.GPUs,
 	}
 	if v.MemTotal > 0 {
 		v.MemPercent = float64(v.MemUsed) * 100 / float64(v.MemTotal)
@@ -254,6 +305,26 @@ type ModelView struct {
 	Unplaceable bool     `json:"unplaceable,omitempty"`
 	Fallbacks   []string `json:"fallbacks,omitempty"`
 	Profiles    []string `json:"profiles"`
+	// MemoryGB and CPUCores are what one replica reserves, from its first
+	// profile; MeasuredMemoryGB is the most one of its replicas uses now, 0
+	// when none is measured.
+	MemoryGB         int     `json:"memory_gb,omitempty"`
+	CPUCores         int     `json:"cpu_cores,omitempty"`
+	MeasuredMemoryGB float64 `json:"measured_memory_gb,omitempty"`
+	// Warnings are what makes its reservation wrong: nothing declared where
+	// placement needs a figure, or less declared than it uses.
+	Warnings []string `json:"warnings,omitempty"`
+	// Class is its capacity class; Preempted the replicas it lost to a higher
+	// class; HeldFor the model it is giving room to now, if any.
+	Class string `json:"class"`
+	Owner string `json:"owner,omitempty"`
+	// External is where a model served outside the cluster goes, provider
+	// host and its name for the model; empty for one served here.
+	External string `json:"external,omitempty"`
+	// Adapters are the LoRA adapters it serves.
+	Adapters  []catalog.Adapter `json:"adapters,omitempty"`
+	Preempted int               `json:"preempted,omitempty"`
+	HeldFor   string            `json:"held_for,omitempty"`
 }

 // ReplicaView is a replica as `ctl replicas` shows it.
@@ -330,6 +401,28 @@ type ExplainView struct {
 	Rejections []RejectionView `json:"rejections,omitempty"`
 }

+// HeadroomView is how many more replicas of a model fit now, and where.
+type HeadroomView struct {
+	Model string `json:"model"`
+	// Current is the replicas holding capacity: starting or ready.
+	Current int `json:"current"`
+	// More is how many more placement would find room for now.
+	More int `json:"more"`
+	// Nodes is where each of those would go.
+	Nodes [][]string `json:"nodes,omitempty"`
+	// Limited is set when More stopped at the limit asked rather than at the
+	// cluster's edge: there may be room for more.
+	Limited bool `json:"limited,omitempty"`
+	// Blocked says why one more does not fit, a line per node considered.
+	Blocked []string `json:"blocked,omitempty"`
+	// TokPerSec is what one more replica was measured to serve; 0 unmeasured.
+	TokPerSec float64 `json:"tok_per_sec,omitempty"`
+	// ByPreempting is how many more would fit by stopping replicas this
+	// model's class may take; Preempts the models those belong to.
+	ByPreempting int      `json:"by_preempting,omitempty"`
+	Preempts     []string `json:"preempts,omitempty"`
+}
+
 // RejectionView is one rejected candidate.
 type RejectionView struct {
 	Profile string `json:"profile"`
@@ -364,8 +457,8 @@ func (a *API) Nodes() []NodeView {
 			ID: n.ID, Hostname: n.Hostname, MgmtAddr: n.MgmtAddr,
 			State: n.State.String(), Labels: n.Labels,
 			GPUTotal: len(n.GPUs), GPUFree: len(inv.FreeGPUs(n.ID)),
-			MemoryGB: n.MemoryTotalMB / 1024,
-			Fabrics:  n.FabricNames(), AgentVersion: n.AgentVersion, Features: n.Features,
+			MemoryGB: n.MemoryTotalMB / 1024, CPUCount: n.CPUCount,
+			Fabrics: n.FabricNames(), AgentVersion: n.AgentVersion, Features: n.Features,
 			Replicas: replicasOn[n.ID],
 		}
 		if free, total := n.LocalDisk(); total > 0 {
@@ -465,6 +558,15 @@ func (a *API) Models() []ModelView {
 	for _, h := range a.ctl.Reconciler().ModelHealth() {
 		health[h.Model] = h
 	}
+	// The most one replica of each model uses, summed over its shards.
+	measured := map[string]int64{}
+	for _, rep := range a.Replicas("") {
+		var b int64
+		for _, sh := range rep.Shards {
+			b += sh.MemoryBytes
+		}
+		measured[rep.Model] = max(measured[rep.Model], b)
+	}
 	out := make([]ModelView, 0)
 	for _, m := range a.ctl.Catalog().Models() {
 		h := health[m.Name]
@@ -475,10 +577,17 @@ func (a *API) Models() []ModelView {
 			Fallbacks: m.Fallbacks,
 			Desired:   h.Desired, Failures: h.Failures,
 			Parked: h.Parked, Reason: h.Reason, Unplaceable: h.Unplaceable,
+			Class: m.EffectiveClass(), Owner: m.Owner, Preempted: h.Preempted, HeldFor: h.HeldFor,
+			External: externalLabel(m), Adapters: m.Adapters,
 		}
 		for _, p := range m.Profiles {
 			v.Profiles = append(v.Profiles, p.Name)
 		}
+		if len(m.Profiles) > 0 {
+			v.MemoryGB, v.CPUCores = m.Profiles[0].MinMemoryGB, m.Profiles[0].CPUCores
+		}
+		v.MeasuredMemoryGB = float64(measured[m.Name]) / (1 << 30)
+		v.Warnings = reservationWarnings(m, v.MeasuredMemoryGB)
 		out = append(out, v)
 	}
 	return out
@@ -549,6 +658,18 @@ type Models interface {
 	Add(ctx context.Context, m catalog.Model, by string) error
 	Remove(ctx context.Context, name string) error
 	Scale(ctx context.Context, name string, s catalog.Scaling, by string) (catalog.Model, error)
+	Resources(ctx context.Context, name, profile string, memoryGB, cpuCores int, by string) (catalog.Model, error)
+	SetClass(ctx context.Context, name, class, by string) (catalog.Model, error)
+	SetOwner(ctx context.Context, name, owner, by string) (catalog.Model, error)
+	SetAdapters(ctx context.Context, name string, adapters []catalog.Adapter, by string) (catalog.Model, error)
+}
+
+// ResourcesRequest sets what a model's profiles reserve of a node.
+type ResourcesRequest struct {
+	// Profile is one profile's name; empty is every profile.
+	Profile  string `json:"profile,omitempty"`
+	MemoryGB int    `json:"memory_gb"`
+	CPUCores int    `json:"cpu_cores"`
 }

 // Usage reports what each group of the tenancy tree spent. The store
@@ -680,7 +801,7 @@ func (a *API) putCapacity(w http.ResponseWriter, r *http.Request) {
 	}
 	if err := a.capacity.PutCapacity(r.Context(), store.StoredCapacity{
 		Model: c.Model, Profile: c.Profile, Measurement: body,
-		MeasuredAt: c.MeasuredAt, MeasuredBy: r.Header.Get("X-Llmgw-Actor"),
+		MeasuredAt: c.MeasuredAt, MeasuredBy: actorOf(r),
 	}); err != nil {
 		writeError(w, http.StatusInternalServerError, err.Error())
 		return
@@ -731,7 +852,7 @@ func (a *API) addModel(w http.ResponseWriter, r *http.Request) {
 		writeError(w, http.StatusBadRequest, "the model definition is not valid JSON: "+err.Error())
 		return
 	}
-	if err := a.models.Add(r.Context(), m, r.Header.Get("X-Llmgw-Actor")); err != nil {
+	if err := a.models.Add(r.Context(), m, actorOf(r)); err != nil {
 		// Every failure here is the definition being wrong, which is the
 		// caller's to fix — a collision, a missing fallback, a profile that
 		// cannot be placed. None of them are server faults.
@@ -780,7 +901,7 @@ func (a *API) scaleModel(w http.ResponseWriter, r *http.Request) {
 	name := r.PathValue("name")
 	m, err := a.models.Scale(r.Context(), name,
 		catalog.Scaling{Min: req.Min, Max: req.Max, IdleTimeout: req.IdleTimeout},
-		r.Header.Get("X-Llmgw-Actor"))
+		actorOf(r))
 	if err != nil {
 		// Like addModel: every refusal here is the request being wrong — a
 		// model declared in configuration, one that was never added, a max
@@ -794,6 +915,26 @@ func (a *API) scaleModel(w http.ResponseWriter, r *http.Request) {
 	})
 }

+// setResources sets a model's reserved memory and cores. Placed replicas keep
+// what they were started with until they are restarted.
+func (a *API) setResources(w http.ResponseWriter, r *http.Request) {
+	if a.models == nil {
+		writeError(w, http.StatusNotImplemented, "this control plane has no store")
+		return
+	}
+	var req ResourcesRequest
+	if err := json.NewDecoder(io.LimitReader(r.Body, 1<<16)).Decode(&req); err != nil {
+		writeError(w, http.StatusBadRequest, "the request is not valid JSON: "+err.Error())
+		return
+	}
+	m, err := a.models.Resources(r.Context(), r.PathValue("name"), req.Profile, req.MemoryGB, req.CPUCores, actorOf(r))
+	if err != nil {
+		writeError(w, http.StatusBadRequest, err.Error())
+		return
+	}
+	writeJSON(w, http.StatusOK, m)
+}
+
 func (a *API) removeModel(w http.ResponseWriter, r *http.Request) {
 	if a.models == nil {
 		writeError(w, http.StatusNotImplemented, "this control plane has no store")
@@ -830,6 +971,21 @@ func (a *API) explainPlacement(w http.ResponseWriter, r *http.Request) {
 // It does not check that the model exists; the handler above does that so it
 // can answer 404, while the dashboard only ever asks about models it just read
 // out of the catalog.
+// Headroom tries placing up to limit more replicas of a model, on a copy of
+// the inventory, and says how many fit. A limit of zero or less takes 16.
+func (a *API) Headroom(model string, limit int) (HeadroomView, bool) {
+	if limit <= 0 || limit > 64 {
+		limit = 16
+	}
+	h, ok := a.ctl.Reconciler().Headroom(model, limit)
+	if !ok {
+		return HeadroomView{}, false
+	}
+	return HeadroomView{Model: model, Current: h.Current, More: h.More, Nodes: h.Nodes,
+		Limited: h.Limited, Blocked: h.Blocked, TokPerSec: h.TokPerSec,
+		ByPreempting: h.ByPreempting, Preempts: h.Preempts}, true
+}
+
 func (a *API) Explain(model string) ExplainView {
 	v := ExplainView{Model: model}
 	for _, rep := range a.ctl.Reconciler().Replicas() {
@@ -1204,3 +1360,164 @@ func (a *API) upgradeNode(w http.ResponseWriter, r *http.Request) {
 	}
 	writeJSON(w, http.StatusOK, map[string]string{"node": r.PathValue("id"), "version": req.Version})
 }
+
+// SetServe supplies what serves a model from the registry.
+func (a *API) SetServe(s *modelserve.Server) { a.serve = s }
+
+func (a *API) serveModel(w http.ResponseWriter, r *http.Request) {
+	if a.serve == nil {
+		writeError(w, http.StatusNotImplemented,
+			"serving from the registry needs a mirror and a store, and this control plane lacks one")
+		return
+	}
+	var req modelserve.Request
+	if err := json.NewDecoder(io.LimitReader(r.Body, 1<<16)).Decode(&req); err != nil {
+		writeError(w, http.StatusBadRequest, "the request is not valid JSON: "+err.Error())
+		return
+	}
+	req.By = actorOf(r)
+	job, err := a.serve.Start(req)
+	if err != nil {
+		writeError(w, http.StatusBadRequest, err.Error())
+		return
+	}
+	writeJSON(w, http.StatusAccepted, job)
+}
+
+func (a *API) job(w http.ResponseWriter, r *http.Request) {
+	if a.serve == nil {
+		writeError(w, http.StatusNotFound, "no such job")
+		return
+	}
+	job, ok := a.serve.Get(r.PathValue("id"))
+	if !ok {
+		writeError(w, http.StatusNotFound, "no such job; jobs are kept only until the control plane restarts")
+		return
+	}
+	writeJSON(w, http.StatusOK, job)
+}
+
+// actorOf is who a change is recorded as made by: what the caller says, since
+// the admin API answers only this machine and trusts whoever reaches it. Kept
+// to a short printable name, so a claim cannot pass for more than one — no
+// line breaks to forge a second log line, no length to bury the rest.
+func actorOf(r *http.Request) string {
+	claimed := r.Header.Get("X-Llmgw-Actor")
+	var b strings.Builder
+	for _, c := range claimed {
+		if b.Len() >= 64 {
+			break
+		}
+		if unicode.IsLetter(c) || unicode.IsDigit(c) || strings.ContainsRune("._-@", c) {
+			b.WriteRune(c)
+		}
+	}
+	if b.Len() == 0 {
+		return "unknown"
+	}
+	return b.String()
+}
+
+// NodeCPUs is the CPU counts of ready nodes with at least this much memory:
+// where a replica needing it could go.
+func (a *API) NodeCPUs(minMemoryGB int) []int {
+	var out []int
+	for _, n := range a.ctl.Inventory().Nodes() {
+		if n.State.String() != "ready" || n.CPUCount == 0 {
+			continue
+		}
+		if n.MemoryTotalMB > 0 && n.MemoryTotalMB/1024 < minMemoryGB {
+			continue
+		}
+		out = append(out, n.CPUCount)
+	}
+	return out
+}
+
+// reservationWarnings says what is wrong with what a model reserves.
+//
+// Resident memory includes the weights mapped from disk, which the kernel can
+// take back, so it overstates what a model needs by the part of the weights
+// not in use. A tenth over is let pass for that; more is a figure too low for
+// placement to be trusted with.
+func reservationWarnings(m catalog.Model, measuredGB float64) []string {
+	var out []string
+	for _, p := range m.Profiles {
+		if p.CPUOnly() && p.MinMemoryGB <= 0 {
+			out = append(out, fmt.Sprintf("profile %s declares no memory: placement cannot tell when a node is full "+
+				"(llmgw ctl model resources %s --memory GB)", p.Name, m.Name))
+		}
+	}
+	if len(m.Profiles) == 1 && m.Profiles[0].MinMemoryGB > 0 && measuredGB > float64(m.Profiles[0].MinMemoryGB)*1.1 {
+		out = append(out, fmt.Sprintf("uses %.1f GB and reserves %d: raise it, or nodes will be filled past their memory",
+			measuredGB, m.Profiles[0].MinMemoryGB))
+	}
+	return out
+}
+
+// externalLabel is "api.openai.com · gpt-4o-mini" for a model served elsewhere.
+func externalLabel(m catalog.Model) string {
+	if !m.IsExternal() {
+		return ""
+	}
+	host := m.External.URL
+	if u, err := url.Parse(host); err == nil && u.Host != "" {
+		host = u.Host
+	}
+	return host + " · " + m.External.Model
+}
+
+// AdaptersRequest replaces a model's LoRA adapters with these.
+type AdaptersRequest struct {
+	Adapters []catalog.Adapter `json:"adapters"`
+}
+
+// setAdapters replaces a model's adapters, after reading each new file to
+// check it is an adapter for the model's architecture: one that is not is a
+// replica that fails to start, found minutes later on a node.
+func (a *API) setAdapters(w http.ResponseWriter, r *http.Request) {
+	if a.models == nil {
+		writeError(w, http.StatusNotImplemented, "this control plane has no store")
+		return
+	}
+	var req AdaptersRequest
+	if err := json.NewDecoder(io.LimitReader(r.Body, 1<<16)).Decode(&req); err != nil {
+		writeError(w, http.StatusBadRequest, "the request is not valid JSON: "+err.Error())
+		return
+	}
+	name := r.PathValue("name")
+	m, ok := a.ctl.Catalog().Resolve(name)
+	if !ok {
+		writeError(w, http.StatusBadRequest, "no model "+name)
+		return
+	}
+	base, baseErr := gguf.KindFromURL(r.Context(), nil, m.Source)
+	have := map[string]bool{}
+	for _, ad := range m.Adapters {
+		have[ad.Source] = true
+	}
+	for _, ad := range req.Adapters {
+		if have[ad.Source] {
+			continue // already checked when it was attached
+		}
+		h, err := gguf.KindFromURL(r.Context(), nil, ad.Source)
+		switch {
+		case err != nil:
+			writeError(w, http.StatusBadRequest, fmt.Sprintf("adapter %s: %v", ad.Name, err))
+			return
+		case !h.IsAdapter():
+			writeError(w, http.StatusBadRequest, fmt.Sprintf("adapter %s: %s is a model, not a LoRA adapter", ad.Name, ad.Source))
+			return
+		case baseErr == nil && h.Architecture != base.Architecture:
+			writeError(w, http.StatusBadRequest, fmt.Sprintf("adapter %s is for %s models and %s is %s",
+				ad.Name, h.Architecture, m.Name, base.Architecture))
+			return
+		}
+	}
+	out, err := a.models.SetAdapters(r.Context(), m.Name, req.Adapters, actorOf(r))
+	if err != nil {
+		writeError(w, http.StatusBadRequest, err.Error())
+		return
+	}
+	writeJSON(w, http.StatusOK, AdaptersRequest{Adapters: out.Adapters})
+}
@@ -258,6 +258,25 @@ type fakeModels struct {
 	err     error
 }

+func (f *fakeModels) SetAdapters(_ context.Context, name string, as []catalog.Adapter, _ string) (catalog.Model, error) {
+	return catalog.Model{Name: name, Adapters: as}, f.err
+}
+
+func (f *fakeModels) SetOwner(_ context.Context, name, owner, _ string) (catalog.Model, error) {
+	return catalog.Model{Name: name, Owner: owner}, f.err
+}
+
+func (f *fakeModels) SetClass(_ context.Context, name, class, _ string) (catalog.Model, error) {
+	return catalog.Model{Name: name, Class: class}, f.err
+}
+
+func (f *fakeModels) Resources(_ context.Context, name, profile string, mem, cores int, _ string) (catalog.Model, error) {
+	if f.err != nil {
+		return catalog.Model{}, f.err
+	}
+	return catalog.Model{Name: name, Profiles: []catalog.Profile{{Name: "p", MinMemoryGB: mem, CPUCores: cores}}}, nil
+}
+
 func (f *fakeModels) Scale(_ context.Context, name string, s catalog.Scaling, by string) (catalog.Model, error) {
 	if f.err != nil {
 		return catalog.Model{}, f.err
@@ -0,0 +1,285 @@
+package adminapi
+
+import (
+	"context"
+	"encoding/json"
+	"fmt"
+	"io"
+	"net/http"
+	"sort"
+	"strings"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// Quotas is where capacity quotas and the tenancy tree they name are kept.
+// The store implements it.
+type Quotas interface {
+	CapacityQuotas(ctx context.Context) ([]store.CapacityQuota, error)
+	PutCapacityQuota(ctx context.Context, q store.CapacityQuota) error
+	DeleteCapacityQuota(ctx context.Context, scope store.Scope, id string) error
+	ScopeParents(ctx context.Context) (map[string][]string, error)
+	AllOrgs(ctx context.Context) ([]store.Org, error)
+	AllTeams(ctx context.Context) ([]store.TeamPath, error)
+	AllProjects(ctx context.Context) ([]store.ProjectPath, error)
+}
+
+// SetQuotas supplies the quota store, which mounts the quota endpoints'
+// behaviour; without it they answer 501.
+func (a *API) SetQuotas(q Quotas) { a.quotas = q }
+
+// QuotaView is one scope's quota with what it holds now.
+type QuotaView struct {
+	// Scope is team:ID; Name is how people know it, "Acme / ml".
+	Scope     string          `json:"scope"`
+	Name      string          `json:"name"`
+	Guarantee store.Resources `json:"guarantee"`
+	Limit     store.Resources `json:"limit"`
+	Holds     quota.Amount    `json:"holds"`
+	UpdatedBy string          `json:"updated_by,omitempty"`
+	UpdatedAt time.Time       `json:"updated_at"`
+}
+
+// QuotaRequest sets a scope's quota. Scope may be written with an ID or a
+// name: org:Acme, team:Acme/ml, project:Acme/ml/training.
+type QuotaRequest struct {
+	Scope     string          `json:"scope"`
+	Guarantee store.Resources `json:"guarantee"`
+	Limit     store.Resources `json:"limit"`
+}
+
+func (a *API) registerQuotas(mux *http.ServeMux) {
+	mux.HandleFunc("GET /api/v1/quotas", func(w http.ResponseWriter, r *http.Request) {
+		if a.quotas == nil {
+			writeError(w, http.StatusNotImplemented, "this control plane has no store")
+			return
+		}
+		views, err := a.QuotaViews(r.Context())
+		if err != nil {
+			writeError(w, http.StatusInternalServerError, err.Error())
+			return
+		}
+		writeJSON(w, http.StatusOK, views)
+	})
+	mux.HandleFunc("PUT /api/v1/quotas", func(w http.ResponseWriter, r *http.Request) {
+		if a.quotas == nil {
+			writeError(w, http.StatusNotImplemented, "this control plane has no store")
+			return
+		}
+		var req QuotaRequest
+		if err := json.NewDecoder(io.LimitReader(r.Body, 1<<14)).Decode(&req); err != nil {
+			writeError(w, http.StatusBadRequest, "the request is not valid JSON: "+err.Error())
+			return
+		}
+		key, err := a.SetQuota(r.Context(), req, actorOf(r))
+		if err != nil {
+			writeError(w, http.StatusBadRequest, err.Error())
+			return
+		}
+		writeJSON(w, http.StatusOK, map[string]string{"scope": key})
+	})
+	mux.HandleFunc("DELETE /api/v1/quotas/{scope}", func(w http.ResponseWriter, r *http.Request) {
+		if a.quotas == nil {
+			writeError(w, http.StatusNotImplemented, "this control plane has no store")
+			return
+		}
+		key, err := a.ClearQuota(r.Context(), r.PathValue("scope"))
+		if err != nil {
+			writeError(w, http.StatusBadRequest, err.Error())
+			return
+		}
+		writeJSON(w, http.StatusOK, map[string]string{"removed": key})
+	})
+	mux.HandleFunc("POST /api/v1/models/{name}/owner", func(w http.ResponseWriter, r *http.Request) {
+		if a.models == nil || a.quotas == nil {
+			writeError(w, http.StatusNotImplemented, "this control plane has no store")
+			return
+		}
+		var req struct {
+			Owner string `json:"owner"`
+		}
+		if err := json.NewDecoder(io.LimitReader(r.Body, 1<<12)).Decode(&req); err != nil {
+			writeError(w, http.StatusBadRequest, "the request is not valid JSON: "+err.Error())
+			return
+		}
+		owner := ""
+		if req.Owner != "" {
+			scope, id, _, err := a.resolveScope(r.Context(), req.Owner)
+			if err != nil {
+				writeError(w, http.StatusBadRequest, err.Error())
+				return
+			}
+			owner = quota.Key(scope, id)
+		}
+		m, err := a.models.SetOwner(r.Context(), r.PathValue("name"), owner, actorOf(r))
+		if err != nil {
+			writeError(w, http.StatusBadRequest, err.Error())
+			return
+		}
+		writeJSON(w, http.StatusOK, map[string]string{"model": m.Name, "owner": m.Owner})
+	})
+}
+
+// QuotaViews lists every quota with what its scope holds now.
+func (a *API) QuotaViews(ctx context.Context) ([]QuotaView, error) {
+	if a.quotas == nil {
+		return nil, nil
+	}
+	qs, err := a.quotas.CapacityQuotas(ctx)
+	if err != nil {
+		return nil, err
+	}
+	parents, err := a.quotas.ScopeParents(ctx)
+	if err != nil {
+		return nil, err
+	}
+	names, err := a.scopeNames(ctx)
+	if err != nil {
+		return nil, err
+	}
+	usage := quota.New(qs, parents).Usage(a.ctl.Reconciler().Holdings())
+	out := make([]QuotaView, 0, len(qs))
+	for _, q := range qs {
+		key := quota.Key(q.Scope, q.ScopeID)
+		out = append(out, QuotaView{Scope: key, Name: names[key], Guarantee: q.Guarantee, Limit: q.Limit,
+			Holds: usage[key], UpdatedBy: q.UpdatedBy, UpdatedAt: q.UpdatedAt})
+	}
+	return out, nil
+}
+
+// scopeNames maps org:ID, team:ID and project:ID to their paths.
+func (a *API) scopeNames(ctx context.Context) (map[string]string, error) {
+	out := map[string]string{}
+	orgs, err := a.quotas.AllOrgs(ctx)
+	if err != nil {
+		return nil, err
+	}
+	for _, o := range orgs {
+		out["org:"+o.ID] = o.Name
+	}
+	teams, err := a.quotas.AllTeams(ctx)
+	if err != nil {
+		return nil, err
+	}
+	for _, t := range teams {
+		out["team:"+t.ID] = t.OrgName + " / " + t.Name
+	}
+	projects, err := a.quotas.AllProjects(ctx)
+	if err != nil {
+		return nil, err
+	}
+	for _, p := range projects {
+		out["project:"+p.ID] = p.Path
+	}
+	return out, nil
+}
+
+// resolveScope reads org:X, team:X or project:X, where X is an ID or the
+// scope's path — Acme, Acme/ml, Acme/ml/training — and returns its ID and
+// path. A path that could mean two scopes is refused rather than guessed.
+func (a *API) resolveScope(ctx context.Context, s string) (store.Scope, string, string, error) {
+	kind, what, ok := strings.Cut(strings.TrimSpace(s), ":")
+	scope := store.Scope(kind)
+	if !ok || what == "" || (scope != store.ScopeOrg && scope != store.ScopeTeam && scope != store.ScopeProject) {
+		return "", "", "", fmt.Errorf("%q is not org:NAME, team:ORG/TEAM or project:ORG/TEAM/PROJECT", s)
+	}
+	names, err := a.scopeNames(ctx)
+	if err != nil {
+		return "", "", "", err
+	}
+	norm := func(p string) string {
+		parts := strings.Split(p, "/")
+		for i := range parts {
+			parts[i] = strings.ToLower(strings.TrimSpace(parts[i]))
+		}
+		return strings.Join(parts, "/")
+	}
+	if name, ok := names[kind+":"+what]; ok {
+		return scope, what, name, nil
+	}
+	match := func(same func(name string) bool) []string {
+		var found []string
+		for key, name := range names {
+			if strings.HasPrefix(key, kind+":") && same(name) {
+				found = append(found, key)
+			}
+		}
+		return found
+	}
+	found := match(func(name string) bool { return norm(name) == norm(what) })
+	if len(found) == 0 {
+		// Its own name alone — ml rather than Acme/ml — when that is unique.
+		found = match(func(name string) bool {
+			parts := strings.Split(norm(name), "/")
+			return parts[len(parts)-1] == norm(what)
+		})
+	}
+	switch len(found) {
+	case 0:
+		return "", "", "", fmt.Errorf("there is no %s %q", kind, what)
+	case 1:
+		_, id, _ := strings.Cut(found[0], ":")
+		return scope, id, names[found[0]], nil
+	}
+	return "", "", "", fmt.Errorf("%q could be any of %d %ss; give its ID", what, len(found), kind)
+}
+
+// SetQuota sets a scope's capacity quota, named by path or ID, and returns
+// the scope as org:ID.
+func (a *API) SetQuota(ctx context.Context, req QuotaRequest, by string) (string, error) {
+	if a.quotas == nil {
+		return "", fmt.Errorf("this control plane has no store")
+	}
+	scope, id, _, err := a.resolveScope(ctx, req.Scope)
+	if err != nil {
+		return "", err
+	}
+	if !req.Guarantee.Any() && !req.Limit.Any() {
+		return "", fmt.Errorf("a quota needs a guarantee or a limit; to remove one, clear it")
+	}
+	q := store.CapacityQuota{Scope: scope, ScopeID: id, Guarantee: req.Guarantee, Limit: req.Limit, UpdatedBy: by}
+	if err := a.quotas.PutCapacityQuota(ctx, q); err != nil {
+		return "", err
+	}
+	return quota.Key(scope, id), nil
+}
+
+// ClearQuota removes a scope's capacity quota.
+func (a *API) ClearQuota(ctx context.Context, scope string) (string, error) {
+	if a.quotas == nil {
+		return "", fmt.Errorf("this control plane has no store")
+	}
+	s, id, _, err := a.resolveScope(ctx, scope)
+	if err != nil {
+		return "", err
+	}
+	if err := a.quotas.DeleteCapacityQuota(ctx, s, id); err != nil {
+		return "", err
+	}
+	return quota.Key(s, id), nil
+}
+
+// ScopeName is a part of the tenancy tree: org:ID and its path.
+type ScopeName struct {
+	Key  string `json:"scope"`
+	Name string `json:"name"`
+}
+
+// Scopes lists every organization, team and project, by path.
+func (a *API) Scopes(ctx context.Context) ([]ScopeName, error) {
+	if a.quotas == nil {
+		return nil, nil
+	}
+	names, err := a.scopeNames(ctx)
+	if err != nil {
+		return nil, err
+	}
+	out := make([]ScopeName, 0, len(names))
+	for k, n := range names {
+		out = append(out, ScopeName{Key: k, Name: n})
+	}
+	sort.Slice(out, func(i, j int) bool { return out[i].Name < out[j].Name })
+	return out, nil
+}
@@ -0,0 +1,107 @@
+package adminapi
+
+import (
+	"context"
+	"encoding/json"
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// fakeQuotaStore is Acme with teams ml and web, and Scania with another ml.
+type fakeQuotaStore struct {
+	quotas []store.CapacityQuota
+}
+
+func (f *fakeQuotaStore) CapacityQuotas(context.Context) ([]store.CapacityQuota, error) {
+	return f.quotas, nil
+}
+func (f *fakeQuotaStore) PutCapacityQuota(_ context.Context, q store.CapacityQuota) error {
+	for i := range f.quotas {
+		if f.quotas[i].Scope == q.Scope && f.quotas[i].ScopeID == q.ScopeID {
+			f.quotas[i] = q
+			return nil
+		}
+	}
+	f.quotas = append(f.quotas, q)
+	return nil
+}
+func (f *fakeQuotaStore) DeleteCapacityQuota(_ context.Context, s store.Scope, id string) error {
+	for i, q := range f.quotas {
+		if q.Scope == s && q.ScopeID == id {
+			f.quotas = append(f.quotas[:i], f.quotas[i+1:]...)
+		}
+	}
+	return nil
+}
+func (f *fakeQuotaStore) ScopeParents(context.Context) (map[string][]string, error) {
+	return map[string][]string{"team:t-ml": {"org:o-acme"}, "team:t-web": {"org:o-acme"},
+		"team:t-sml": {"org:o-scania"}}, nil
+}
+func (f *fakeQuotaStore) AllOrgs(context.Context) ([]store.Org, error) {
+	return []store.Org{{ID: "o-acme", Name: "Acme"}, {ID: "o-scania", Name: "Scania"}}, nil
+}
+func (f *fakeQuotaStore) AllTeams(context.Context) ([]store.TeamPath, error) {
+	return []store.TeamPath{{ID: "t-ml", Name: "ml", OrgID: "o-acme", OrgName: "Acme"},
+		{ID: "t-web", Name: "web", OrgID: "o-acme", OrgName: "Acme"},
+		{ID: "t-sml", Name: "ml", OrgID: "o-scania", OrgName: "Scania"}}, nil
+}
+func (f *fakeQuotaStore) AllProjects(context.Context) ([]store.ProjectPath, error) { return nil, nil }
+
+func TestAScopeIsNamedByPathOrIDAndNeverGuessed(t *testing.T) {
+	a, _ := setup(t, testModels())
+	a.SetQuotas(&fakeQuotaStore{})
+	for in, want := range map[string]string{
+		"org:Acme": "o-acme", "org:o-acme": "o-acme", "team:Acme/ml": "t-ml",
+		"team: acme / ML ": "t-ml", "team:Scania/ml": "t-sml", "team:web": "t-web",
+	} {
+		_, id, _, err := a.resolveScope(t.Context(), in)
+		if err != nil || id != want {
+			t.Errorf("%q: %q %v, expected %q", in, id, err, want)
+		}
+	}
+	for _, bad := range []string{"team:ml", "team:Acme/nope", "group:Acme", "Acme", "org:"} {
+		if _, _, _, err := a.resolveScope(t.Context(), bad); err == nil {
+			t.Errorf("%q was resolved", bad)
+		}
+	}
+	if _, _, _, err := a.resolveScope(t.Context(), "team:ml"); err == nil || !strings.Contains(err.Error(), "could be any of 2") {
+		t.Errorf("an ambiguous name should say so: %v", err)
+	}
+}
+
+func TestAQuotaIsSetListedAndCleared(t *testing.T) {
+	a, _ := setup(t, testModels())
+	qs := &fakeQuotaStore{}
+	a.SetQuotas(qs)
+	mux := http.NewServeMux()
+	a.Register(mux)
+	call := func(method, path, body string) *httptest.ResponseRecorder {
+		req := httptest.NewRequest(method, path, strings.NewReader(body))
+		req.Header.Set("Content-Type", "application/json")
+		req.Header.Set("X-Llmgw-Actor", "ops")
+		rec := httptest.NewRecorder()
+		mux.ServeHTTP(rec, req)
+		return rec
+	}
+	if rec := call("PUT", "/api/v1/quotas", `{"scope":"team:Acme/ml","guarantee":{"gpus":8},"limit":{"gpus":16}}`); rec.Code != 200 {
+		t.Fatalf("set: %d %s", rec.Code, rec.Body)
+	}
+	if len(qs.quotas) != 1 || qs.quotas[0].ScopeID != "t-ml" || *qs.quotas[0].Guarantee.GPUs != 8 || qs.quotas[0].UpdatedBy != "ops" {
+		t.Errorf("stored %+v", qs.quotas)
+	}
+	rec := call("GET", "/api/v1/quotas", "")
+	var views []QuotaView
+	if err := json.Unmarshal(rec.Body.Bytes(), &views); err != nil || len(views) != 1 || views[0].Name != "Acme / ml" {
+		t.Errorf("list %d %s", rec.Code, rec.Body)
+	}
+	if rec := call("PUT", "/api/v1/quotas", `{"scope":"team:Acme/ml"}`); rec.Code != 400 {
+		t.Errorf("an empty quota was accepted: %d", rec.Code)
+	}
+	if rec := call("DELETE", "/api/v1/quotas/team:Acme%2Fml", ""); rec.Code != 200 || len(qs.quotas) != 0 {
+		t.Errorf("clear: %d %s %+v", rec.Code, rec.Body, qs.quotas)
+	}
+}
@@ -0,0 +1,69 @@
+package adminapi
+
+import (
+	"fmt"
+	"math"
+	"net/http"
+	"strconv"
+)
+
+// TargetView is how many replicas a model needs to serve a throughput, from
+// what one replica was measured to serve, and whether the cluster has room.
+type TargetView struct {
+	Model string `json:"model"`
+	// TokPerSec is what was asked for; PerReplica what one replica was
+	// measured to serve at its SLO, on Profile.
+	TokPerSec  float64 `json:"tok_per_sec"`
+	PerReplica float64 `json:"per_replica_tok_per_sec"`
+	Profile    string  `json:"profile"`
+	// Needed replicas in all; Current hold capacity now; More is how many of
+	// Needed are not running, and Room how many more fit.
+	Needed  int      `json:"needed"`
+	Current int      `json:"current"`
+	More    int      `json:"more"`
+	Room    int      `json:"room"`
+	Fits    bool     `json:"fits"`
+	Blocked []string `json:"blocked,omitempty"`
+	// Preempts is whose replicas would be stopped to make the room, when
+	// free room alone is not enough; Room counts that room too.
+	Preempts []string `json:"preempts,omitempty"`
+}
+
+// Target sizes a model for a throughput.
+func (a *API) Target(model string, tokPerSec float64) (TargetView, error) {
+	meas, profile, ok := a.ctl.Reconciler().MeasuredFor(model)
+	if !ok || meas.TokPerSec <= 0 {
+		return TargetView{}, fmt.Errorf("%s has not been measured; run `llmgw ctl bench %s --save` first", model, model)
+	}
+	v := TargetView{Model: model, TokPerSec: tokPerSec, PerReplica: meas.TokPerSec, Profile: profile,
+		Needed: max(int(math.Ceil(tokPerSec/meas.TokPerSec)), 1)}
+	h, ok := a.Headroom(model, v.Needed)
+	if !ok {
+		return TargetView{}, fmt.Errorf("no model %q", model)
+	}
+	v.Current = h.Current
+	v.More = max(v.Needed-h.Current, 0)
+	v.Room = h.More + h.ByPreempting
+	v.Fits = v.More <= v.Room || h.Limited
+	if v.More > h.More {
+		v.Preempts = h.Preempts
+	}
+	if !v.Fits {
+		v.Blocked = h.Blocked
+	}
+	return v, nil
+}
+
+func (a *API) target(w http.ResponseWriter, r *http.Request) {
+	tps, err := strconv.ParseFloat(r.URL.Query().Get("tokens_per_sec"), 64)
+	if err != nil || tps <= 0 {
+		writeError(w, http.StatusBadRequest, "tokens_per_sec must be a positive number")
+		return
+	}
+	v, err := a.Target(r.PathValue("name"), tps)
+	if err != nil {
+		writeError(w, http.StatusBadRequest, err.Error())
+		return
+	}
+	writeJSON(w, http.StatusOK, v)
+}
@@ -0,0 +1,30 @@
+package adminapi
+
+import (
+	"strings"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
+)
+
+func TestATargetIsSizedFromTheMeasurementAndTheRoom(t *testing.T) {
+	// Three 8-GPU nodes; one tp8 replica was measured at 500 tokens/s.
+	a, ctl := setup(t, testModels(), gpuNode("1", 8), gpuNode("2", 8), gpuNode("3", 8))
+	if _, err := a.Target("llama", 1000); err == nil || !strings.Contains(err.Error(), "ctl bench llama --save") {
+		t.Errorf("an unmeasured model should say how to measure it: %v", err)
+	}
+	ctl.Reconciler().SetMeasurements(func() map[string]reconciler.Measured {
+		return map[string]reconciler.Measured{"llama/tp8": {MaxConcurrency: 16, TokPerSec: 500}}
+	})
+	v, err := a.Target("llama", 1200)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if v.Needed != 3 || v.PerReplica != 500 || v.Profile != "tp8" || v.More != 3 || !v.Fits {
+		t.Errorf("1200 tok/s at 500 each is 3 replicas on 3 empty nodes: %+v", v)
+	}
+	v, _ = a.Target("llama", 2000)
+	if v.Needed != 4 || v.Fits || v.Room != 3 || len(v.Blocked) == 0 {
+		t.Errorf("a fourth does not fit on three nodes, and should say why: %+v", v)
+	}
+}
@@ -31,6 +31,11 @@ func (a *API) Orphans() []OrphanView {
 	var wanted []string
 	for _, m := range a.ctl.Catalog().Models() {
 		wanted = append(wanted, m.Source)
+		// A model's adapters are its weights too; collecting them would
+		// leave replicas that cannot restart.
+		for _, ad := range m.Adapters {
+			wanted = append(wanted, ad.Source)
+		}
 	}

 	out := make([]OrphanView, 0)
@@ -4,6 +4,7 @@ package agentsvc
 import (
 	"fmt"
 	"maps"
+	"runtime"
 	"time"

 	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
@@ -27,6 +28,7 @@ func NodeFromRegister(r *llmgwv1.Register) inventory.Node {
 		EngineVersions: r.GetEngineVersions(),
 		EnginesProbed:  r.GetEnginesProbed(),
 		MemoryTotalMB:  int(r.GetMemoryTotalMb()),
+		CPUCount:       int(r.GetCpuCount()),
 	}
 	if h := r.GetHost(); h != nil {
 		n.Host = discovery.Host{
@@ -90,6 +92,7 @@ func RegisterFromInventory(nodeID, hostname, mgmtAddr, agentVersion string,
 		NodeId: nodeID, Hostname: hostname, MgmtAddr: mgmtAddr,
 		Labels: labels, AgentVersion: agentVersion,
 		MemoryTotalMb: int64(inv.MemoryTotalMB),
+		CpuCount:      int32(runtime.NumCPU()),
 	}
 	for _, w := range inv.Weights {
 		r.Weights = append(r.Weights, &llmgwv1.LocalWeights{
@@ -186,12 +189,14 @@ func StartShardFromCommand(cmd reconciler.ShardCommand) *llmgwv1.StartShard {
 			GpusPerNode: int32(p.GPUsPerNode), Nodes: int32(p.Nodes),
 			Args: p.Args, Env: p.Env,
 			ModelPath: p.ModelPath, ServedModelName: p.ServedModelName,
+			Adapters: adaptersToWire(p.Adapters),
 		},
 		StartupTimeoutMs: cmd.StartupTimeout.Milliseconds(),
 		Pool:             cmd.Pool,
 		Fabric:           cmd.Fabric,
 		ProfileFp:        cmd.ProfileFP,
 		ApiKey:           cmd.Spec.APIKey,
+		CpuThreads:       int32(cmd.Spec.Threads),
 	}
 }

@@ -213,12 +218,14 @@ func ShardSpecFromStart(m *llmgwv1.StartShard) (engines.ShardSpec, time.Duration
 		Fabric:        m.GetFabric(),
 		ProfileFP:     m.GetProfileFp(),
 		APIKey:        m.GetApiKey(),
+		Threads:       int(m.GetCpuThreads()),
 		Profile: engines.Profile{
 			Name: p.GetName(), Engine: p.GetEngine(), GPUClass: p.GetGpuClass(),
 			TP: int(p.GetTp()), PP: int(p.GetPp()),
 			GPUsPerNode: int(p.GetGpusPerNode()), Nodes: int(p.GetNodes()),
 			Args: p.GetArgs(), Env: p.GetEnv(),
 			ModelPath: p.GetModelPath(), ServedModelName: p.GetServedModelName(),
+			Adapters: adaptersFromWire(p.GetAdapters()),
 		},
 	}
 	return spec, time.Duration(m.GetStartupTimeoutMs()) * time.Millisecond
@@ -287,3 +294,19 @@ func fabricFrom(state, status, cluster, clique, bandwidth string) *discovery.GPU
 	}
 	return &discovery.GPUFabric{State: state, Status: status, ClusterUUID: cluster, CliqueID: clique, Bandwidth: bandwidth}
 }
+
+func adaptersToWire(as []engines.Adapter) []*llmgwv1.LoraAdapter {
+	var out []*llmgwv1.LoraAdapter
+	for _, a := range as {
+		out = append(out, &llmgwv1.LoraAdapter{Name: a.Name, Source: a.Path})
+	}
+	return out
+}
+
+func adaptersFromWire(as []*llmgwv1.LoraAdapter) []engines.Adapter {
+	var out []engines.Adapter
+	for _, a := range as {
+		out = append(out, engines.Adapter{Name: a.GetName(), Path: a.GetSource()})
+	}
+	return out
+}
@@ -0,0 +1,20 @@
+package agentsvc
+
+import (
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
+)
+
+func TestAdaptersTravelToTheAgentInOrder(t *testing.T) {
+	cmd := reconciler.ShardCommand{ReplicaID: "r", ShardID: "s", Spec: engines.ShardSpec{
+		Profile: engines.Profile{Name: "cpu", Engine: "llamacpp", ModelPath: "/m.gguf",
+			Adapters: []engines.Adapter{{Name: "sql", Path: "http://mirror/sql.gguf"}, {Name: "docs", Path: "http://mirror/docs.gguf"}}},
+	}}
+	spec, _ := ShardSpecFromStart(StartShardFromCommand(cmd))
+	got := spec.Profile.Adapters
+	if len(got) != 2 || got[0].Name != "sql" || got[1].Path != "http://mirror/docs.gguf" {
+		t.Errorf("adapters %+v", got)
+	}
+}
@@ -345,8 +345,9 @@ func (s *Service) applyHeartbeat(nodeID string, hb *llmgwv1.Heartbeat) {
 	problems := hb.GetNewProblems()
 	storage := hb.GetStorage()
 	memMB := int(hb.GetMemoryTotalMb())
+	cpus := int(hb.GetCpuCount())
 	if len(used) == 0 && len(problems) == 0 && len(storage) == 0 &&
-		!hb.GetWeightsChanged() && memMB == 0 && !hb.GetEnginesChanged() &&
+		!hb.GetWeightsChanged() && memMB == 0 && cpus == 0 && !hb.GetEnginesChanged() &&
 		!hb.GetProvisionChanged() && hb.GetLoad() == nil && len(hb.GetIbPorts()) == 0 &&
 		!hb.GetFabricsChanged() && len(hb.GetGpuFabric()) == 0 && hb.GetImex() == nil {
 		return
@@ -406,6 +407,12 @@ func (s *Service) applyHeartbeat(nodeID string, hb *llmgwv1.Heartbeat) {
 			"from_mb", n.MemoryTotalMB, "to_mb", memMB)
 		n.MemoryTotalMB = memMB
 	}
+	if cpus > 0 && cpus != n.CPUCount {
+		if n.CPUCount > 0 {
+			s.log.Info("node CPUs changed", "node", nodeID, "from", n.CPUCount, "to", cpus)
+		}
+		n.CPUCount = cpus
+	}
 	if hb.GetEnginesChanged() {
 		vers := maps.Clone(hb.GetEngineVersions())
 		if !maps.Equal(vers, n.EngineVersions) {
@@ -538,6 +545,13 @@ func loadFrom(l *llmgwv1.NodeLoad, at time.Time) *inventory.Load {
 			CPUCores: sh.GetCpuCores(), RSSBytes: sh.GetMemoryRssBytes(),
 		}
 	}
+	out.CPUTempC, out.CPUPowerW, out.InletTempC = l.GetCpuTempC(), l.GetCpuPowerW(), l.GetInletTempC()
+	for _, g := range l.GetGpus() {
+		out.GPUs = append(out.GPUs, inventory.GPUSensor{
+			Index: int(g.GetIndex()), TempC: g.GetTempC(), MemoryTempC: g.GetMemoryTempC(),
+			PowerW: g.GetPowerW(), PowerLimitW: g.GetPowerLimitW(), Throttle: g.GetThrottle(),
+		})
+	}
 	return out
 }

@@ -7,6 +7,7 @@ import (
 	"encoding/hex"
 	"encoding/json"
 	"fmt"
+	"net/url"
 	"slices"
 	"strings"
 	"time"
@@ -74,6 +75,17 @@ type Profile struct {
 	// the control plane.
 	MinMemoryGB int `json:"min_memory_gb,omitempty" koanf:"min_memory_gb"`

+	// CPUCores is the cores a replica of a profile with no GPU is given:
+	// reserved out of its node's CPUs by placement, as memory is, and what
+	// its engine computes on (llama.cpp's --threads).
+	//
+	// Reserved because an engine takes every core it is told about, and two
+	// on one node that each think they have all of them spin against each
+	// other: on an 8-vCPU node, two 7B replicas fell from 10 tokens a second
+	// to under one each. Zero reserves nothing and leaves the thread count to
+	// the node, which is every profile from before this.
+	CPUCores int `json:"cpu_cores,omitempty" koanf:"cpu_cores"`
+
 	// MinDiskGB is the free space a node needs before it can hold these
 	// weights.
 	//
@@ -138,9 +150,30 @@ type Model struct {
 	// choice by guesswork.
 	Description string `json:"description,omitempty" koanf:"description"`
 	// Source is the weights path or a HuggingFace id.
-	Source    string    `json:"source,omitempty" koanf:"source"`
-	Profiles  []Profile `json:"profiles,omitempty" koanf:"profiles"`
-	Scaling   Scaling   `json:"scaling,omitempty" koanf:"scaling"`
+	Source   string    `json:"source,omitempty" koanf:"source"`
+	Profiles []Profile `json:"profiles,omitempty" koanf:"profiles"`
+	Scaling  Scaling   `json:"scaling,omitempty" koanf:"scaling"`
+	// Class is how much this model's capacity is worth against another's,
+	// when there is not enough for both: critical, standard or opportunistic.
+	// Empty is standard, or opportunistic for a model loaded on demand. See
+	// EffectiveClass.
+	Class string `json:"class,omitempty" koanf:"class"`
+	// Owner is the part of the tenancy tree whose capacity quota this model
+	// counts against, written org:ID, team:ID or project:ID. Empty is shared:
+	// held by the cluster, counted against nobody.
+	Owner string `json:"owner,omitempty" koanf:"owner"`
+	// External is a model served by someone else — an OpenAI-compatible API
+	// outside the cluster — rather than by replicas placed here. Such a model
+	// has no profiles and nothing is placed for it; the router sends its
+	// requests to the provider. Most useful named in another model's
+	// fallbacks: used only when the cluster has no capacity left. Its
+	// prompts leave the cluster, so which keys may name it is worth an
+	// allowlist.
+	External *External `json:"external,omitempty" koanf:"external"`
+	// Adapters are LoRA adapters served on this model's replicas, each asked
+	// for by its own name as though it were a model. Every replica loads
+	// every adapter, applying one only to a request that names it.
+	Adapters  []Adapter `json:"adapters,omitempty" koanf:"adapters"`
 	Fallbacks []string  `json:"fallbacks,omitempty" koanf:"fallbacks"`
 	// RoutingProfile selects how the router scores endpoints for this model.
 	RoutingProfile string `json:"routing_profile,omitempty" koanf:"routing_profile"`
@@ -184,14 +217,17 @@ type Capabilities struct {
 type Catalog struct {
 	models  map[string]Model
 	aliases map[string]string
-	order   []string
+	// adapters maps an adapter's name to the model it runs on.
+	adapters map[string]string
+	order    []string
 }

 // New builds a catalog, rejecting definitions that would fail confusingly later.
 func New(models []Model) (*Catalog, error) {
 	c := &Catalog{
-		models:  make(map[string]Model, len(models)),
-		aliases: map[string]string{},
+		models:   make(map[string]Model, len(models)),
+		aliases:  map[string]string{},
+		adapters: map[string]string{},
 	}

 	for i := range models {
@@ -220,6 +256,18 @@ func New(models []Model) (*Catalog, error) {
 		}
 	}

+	// Adapters are named like models, so they share the namespace too.
+	for _, name := range c.order {
+		for _, a := range c.models[name].Adapters {
+			_, isModel := c.models[a.Name]
+			_, isAlias := c.aliases[a.Name]
+			if _, isAdapter := c.adapters[a.Name]; isModel || isAlias || isAdapter {
+				return nil, fmt.Errorf("model %q: adapter %q collides with a model, an alias or another adapter", name, a.Name)
+			}
+			c.adapters[a.Name] = name
+		}
+	}
+
 	// Fallbacks are resolved now rather than at request time: a typo in a
 	// fallback should stop a deployment, not surface during an outage when the
 	// fallback is finally needed.
@@ -285,9 +333,22 @@ func (m Model) validate() error {
 	if m.Source == "" {
 		return fmt.Errorf("model %q: source is required", m.Name)
 	}
-	if len(m.Profiles) == 0 {
+	if err := validateAdapters(m); err != nil {
+		return err
+	}
+	if m.External != nil {
+		if err := m.External.validate(m.Name, len(m.Profiles)); err != nil {
+			return err
+		}
+	} else if len(m.Profiles) == 0 {
 		return fmt.Errorf("model %q: at least one profile is required", m.Name)
 	}
+	if m.Owner != "" && !validOwner(m.Owner) {
+		return fmt.Errorf("model %q: owner %q is not org:ID, team:ID or project:ID", m.Name, m.Owner)
+	}
+	if !validClass(m.Class) {
+		return fmt.Errorf("model %q: class %q is not one of critical, standard, opportunistic", m.Name, m.Class)
+	}
 	if m.Scaling.Max < m.Scaling.Min {
 		return fmt.Errorf("model %q: scaling max %d is below min %d", m.Name, m.Scaling.Max, m.Scaling.Min)
 	}
@@ -331,6 +392,23 @@ func (m Model) validate() error {
 }

 // Resolve maps a client-supplied name or alias onto a model.
+// ResolveAdapter finds an adapter by its name: the model it runs on, the
+// adapter, and its index among the model's adapters — which is the id the
+// engine knows it by.
+func (c *Catalog) ResolveAdapter(name string) (Model, Adapter, int, bool) {
+	base, ok := c.adapters[name]
+	if !ok {
+		return Model{}, Adapter{}, 0, false
+	}
+	m := c.models[base]
+	for i, a := range m.Adapters {
+		if a.Name == name {
+			return m, a, i, true
+		}
+	}
+	return Model{}, Adapter{}, 0, false
+}
+
 func (c *Catalog) Resolve(name string) (Model, bool) {
 	if m, ok := c.models[name]; ok {
 		return m, true
@@ -380,6 +458,7 @@ func (m Model) PlacementProfiles() []placement.Profile {
 			NodeSelector: p.NodeSelector,
 			MinMemoryGB:  p.MinMemoryGB,
 			MinDiskGB:    p.MinDiskGB,
+			CPUCores:     p.CPUCores,
 			// The model's source, so placement can prefer nodes that already
 			// hold the weights rather than one that must first move them.
 			Source: m.Source,
@@ -406,6 +485,7 @@ func (m Model) EngineProfile(name string) (engines.Profile, bool) {
 			Env:             p.Env,
 			ModelPath:       m.Source,
 			ServedModelName: m.Name,
+			Adapters:        m.engineAdapters(),
 		}, true
 	}
 	return engines.Profile{}, false
@@ -475,3 +555,152 @@ func (p Profile) Fingerprint() string {
 	sum := sha256.Sum256(b)
 	return hex.EncodeToString(sum[:8])
 }
+
+// SuggestCores is the cores to give a replica with no GPU on nodes of these
+// CPU counts: half of the smallest, so two replicas share any of them without
+// competing. Zero when no node says, which reserves nothing.
+func SuggestCores(cpuCounts []int) int {
+	smallest := 0
+	for _, c := range cpuCounts {
+		if c > 0 && (smallest == 0 || c < smallest) {
+			smallest = c
+		}
+	}
+	if smallest == 0 {
+		return 0
+	}
+	return max(smallest/2, 1)
+}
+
+// The capacity classes, lowest first.
+const (
+	ClassOpportunistic = "opportunistic"
+	ClassStandard      = "standard"
+	ClassCritical      = "critical"
+)
+
+func validClass(c string) bool {
+	return c == "" || c == ClassOpportunistic || c == ClassStandard || c == ClassCritical
+}
+
+// EffectiveClass is the class a model is treated as: what it says, or, when
+// it says nothing, opportunistic for one loaded on demand — it holds capacity
+// only while asked for, and has always been the first to give it up — and
+// standard for everything else.
+func (m Model) EffectiveClass() string {
+	if m.Class != "" {
+		return m.Class
+	}
+	if m.Scaling.OnDemand() {
+		return ClassOpportunistic
+	}
+	return ClassStandard
+}
+
+// ClassRank orders classes: a higher rank may take capacity from a lower one.
+func ClassRank(c string) int {
+	switch c {
+	case ClassCritical:
+		return 3
+	case ClassStandard, "":
+		return 2
+	default:
+		return 1
+	}
+}
+
+func validOwner(o string) bool {
+	for _, p := range []string{"org:", "team:", "project:"} {
+		if strings.HasPrefix(o, p) && len(o) > len(p) {
+			return true
+		}
+	}
+	return false
+}
+
+// External is a provider outside the cluster that serves a model.
+type External struct {
+	// URL is the provider's base address, without /v1: https://api.openai.com.
+	URL string `json:"url" koanf:"url"`
+	// Model is what the provider calls it; requests are rewritten to it.
+	Model string `json:"model" koanf:"model"`
+	// APIKeyFile holds the provider's key, on the control plane. A file
+	// rather than the key itself, so the key is in neither the catalog, the
+	// store nor the audit log; empty for a provider that asks for none.
+	APIKeyFile string `json:"api_key_file,omitempty" koanf:"api_key_file"`
+}
+
+func (e *External) validate(model string, profiles int) error {
+	if profiles > 0 {
+		return fmt.Errorf("model %q: an external model is served elsewhere and has no profiles", model)
+	}
+	u, err := url.Parse(e.URL)
+	if err != nil || (u.Scheme != "https" && u.Scheme != "http") || u.Host == "" {
+		return fmt.Errorf("model %q: external url %q is not an http(s) address", model, e.URL)
+	}
+	if strings.TrimSpace(e.Model) == "" {
+		return fmt.Errorf("model %q: external model needs the name the provider knows it by", model)
+	}
+	return nil
+}
+
+// IsExternal reports whether a model is served outside the cluster.
+func (m Model) IsExternal() bool { return m.External != nil }
+
+// Adapter is a LoRA adapter served on a model's replicas.
+type Adapter struct {
+	// Name is what clients ask for, like a model name.
+	Name string `json:"name" koanf:"name"`
+	// Source is the adapter file (GGUF for llama.cpp), fetched by each node
+	// as the model's weights are.
+	Source string `json:"source" koanf:"source"`
+	// Scale is how strongly it is applied; zero takes 1.
+	Scale float64 `json:"scale,omitempty" koanf:"scale"`
+}
+
+// EffectiveScale is the scale an adapter is applied at.
+func (a Adapter) EffectiveScale() float64 {
+	if a.Scale <= 0 {
+		return 1
+	}
+	return a.Scale
+}
+
+func validateAdapters(m Model) error {
+	if len(m.Adapters) == 0 {
+		return nil
+	}
+	if m.External != nil {
+		return fmt.Errorf("model %q: an external model cannot carry adapters", m.Name)
+	}
+	for _, p := range m.Profiles {
+		// llama.cpp loads adapters at start and applies them per request;
+		// the other engines are not driven that way yet.
+		if p.Engine != "llamacpp" && p.Engine != "sim" {
+			return fmt.Errorf("model %q profile %q: adapters are served by llamacpp profiles only", m.Name, p.Name)
+		}
+	}
+	seen := map[string]bool{}
+	for _, a := range m.Adapters {
+		switch {
+		case a.Name == "" || a.Source == "":
+			return fmt.Errorf("model %q: an adapter needs a name and a source", m.Name)
+		case a.Name == m.Name:
+			return fmt.Errorf("model %q: an adapter cannot have the model's own name", m.Name)
+		case seen[a.Name]:
+			return fmt.Errorf("model %q: adapter %q is declared twice", m.Name, a.Name)
+		case a.Scale < 0:
+			return fmt.Errorf("model %q: adapter %q has a negative scale", m.Name, a.Name)
+		}
+		seen[a.Name] = true
+	}
+	return nil
+}
+
+func (m Model) engineAdapters() []engines.Adapter {
+	var out []engines.Adapter
+	for _, a := range m.Adapters {
+		out = append(out, engines.Adapter{Name: a.Name, Path: a.Source})
+	}
+	return out
+}
@@ -9,6 +9,7 @@ import (
 	"encoding/hex"
 	"fmt"
 	"log/slog"
+	"os"
 	"strings"
 	"sync"
 	"time"
@@ -134,6 +135,16 @@ func (c *Controller) collectUsage() {
 		return
 	}
 	c.rec.RecordUsage(c.usage.LastUsed())
+	// And what the replicas are doing, when the source can say: what adds
+	// and removes replicas between a model's min and max.
+	if l, ok := c.usage.(LoadSource); ok {
+		c.rec.RecordLoad(l.ModelLoad())
+	}
+}
+
+// LoadSource reports each model's replicas' load, keyed by catalog name.
+type LoadSource interface {
+	ModelLoad() map[string]reconciler.Load
 }

 // ReplaceCatalog puts a new catalog in force, without a restart.
@@ -328,6 +339,34 @@ func (c *Controller) endpoints() []endpoint.Info {
 			Priority: prof.Priority,
 		})
 	}
+	return append(out, c.externalEndpoints()...)
+}
+
+// externalEndpoints are the models served outside the cluster, as endpoints:
+// one each, with the provider's key read from its file every time, so a key
+// rotated on disk is used from the next publish on.
+func (c *Controller) externalEndpoints() []endpoint.Info {
+	var out []endpoint.Info
+	for _, m := range c.cat.Load().Models() {
+		if !m.IsExternal() {
+			continue
+		}
+		key := ""
+		if f := m.External.APIKeyFile; f != "" {
+			b, err := os.ReadFile(f) //nolint:gosec // the operator's own path
+			if err != nil {
+				c.log.Warn("an external model's key could not be read; not offered", "model", m.Name, "err", err)
+				continue
+			}
+			key = strings.TrimSpace(string(b))
+		}
+		out = append(out, endpoint.Info{
+			ID: "external:" + m.Name, URL: strings.TrimRight(m.External.URL, "/"),
+			Models: append([]string{m.Name}, m.Aliases...), APIKey: key,
+			Engine: endpoint.EngineExternal, Pool: "external", Weight: 1,
+			UpstreamModel: m.External.Model,
+		})
+	}
 	return out
 }

@@ -335,7 +374,8 @@ func (c *Controller) endpoints() []endpoint.Info {
 func fingerprint(infos []endpoint.Info) string {
 	var b strings.Builder
 	for _, i := range infos {
-		fmt.Fprintf(&b, "%s|%s|%s|%s|%v|%g|%d;", i.ID, i.URL, i.Engine, i.Pool, i.Models, i.Weight, i.Priority)
+		fmt.Fprintf(&b, "%s|%s|%s|%s|%v|%g|%d|%s|%x;", i.ID, i.URL, i.Engine, i.Pool, i.Models, i.Weight, i.Priority,
+			i.UpstreamModel, sha256.Sum256([]byte(i.APIKey)))
 	}
 	return b.String()
 }
@@ -365,3 +405,6 @@ func EngineKeys(secret []byte) func(replicaID string) string {
 		return "llmgw-" + hex.EncodeToString(m.Sum(nil))[:40]
 	}
 }
+
+// QuotaUse is every capacity quota beside what its scope holds.
+func (c *Controller) QuotaUse() []reconciler.QuotaUse { return c.rec.QuotaUse() }
@@ -0,0 +1,75 @@
+package control
+
+import (
+	"context"
+	"log/slog"
+	"os"
+	"path/filepath"
+	"sync"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint"
+)
+
+type recordSink struct {
+	mu    sync.Mutex
+	infos []endpoint.Info
+	times int
+}
+
+func (s *recordSink) SetEndpoints(_ context.Context, infos []endpoint.Info) {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	s.infos, s.times = infos, s.times+1
+}
+
+type noDispatch struct{}
+
+func (noDispatch) StartShard(context.Context, reconciler.ShardCommand) error { return nil }
+func (noDispatch) StopShard(context.Context, string, string) error           { return nil }
+func (noDispatch) DrainShard(context.Context, string, string) error          { return nil }
+
+func TestAnExternalModelIsPublishedWithItsKeyAndNothingIsPlacedForIt(t *testing.T) {
+	dir := t.TempDir()
+	keyFile := filepath.Join(dir, "provider.key")
+	if err := os.WriteFile(keyFile, []byte("sk-provider-1\n"), 0o600); err != nil {
+		t.Fatal(err)
+	}
+	models := []catalog.Model{
+		{Name: "cloud", Source: "external", Aliases: []string{"gpt"}, Scaling: catalog.Scaling{Min: 3, Max: 3},
+			External: &catalog.External{URL: "https://api.example.com/", Model: "gpt-4o-mini", APIKeyFile: keyFile}},
+		{Name: "gone", Source: "external",
+			External: &catalog.External{URL: "https://api.example.com", Model: "m", APIKeyFile: filepath.Join(dir, "missing")}},
+	}
+	cat, err := catalog.New(models)
+	if err != nil {
+		t.Fatal(err)
+	}
+	sink := &recordSink{}
+	c := New(DefaultConfig(), catalog.NewLive(cat), noDispatch{}, sink, slog.New(slog.DiscardHandler))
+	c.Publish(t.Context())
+
+	if len(sink.infos) != 1 {
+		t.Fatalf("published %+v; the model whose key is missing must not be offered", sink.infos)
+	}
+	e := sink.infos[0]
+	if e.ID != "external:cloud" || e.URL != "https://api.example.com" || e.APIKey != "sk-provider-1" ||
+		e.UpstreamModel != "gpt-4o-mini" || e.Engine != endpoint.EngineExternal || len(e.Models) != 2 {
+		t.Errorf("endpoint %+v", e)
+	}
+	// A key rotated on disk is published on the next pass.
+	_ = os.WriteFile(keyFile, []byte("sk-provider-2"), 0o600)
+	c.Publish(t.Context())
+	if sink.times != 2 || sink.infos[0].APIKey != "sk-provider-2" {
+		t.Errorf("the rotated key was not published: %d pushes, key %q", sink.times, sink.infos[0].APIKey)
+	}
+	// Scaling of 3 or not, nothing is wanted of the cluster for a model
+	// served elsewhere, and so it never reads as short or unplaceable.
+	for _, h := range c.Reconciler().ModelHealth() {
+		if h.Model == "cloud" && (h.Desired != 0 || h.Unplaceable) {
+			t.Errorf("an external model wants replicas: %+v", h)
+		}
+	}
+}
@@ -51,12 +51,16 @@ type HealthSource interface {
 type Collector struct {
 	src Source

-	modelDesired     *prometheus.Desc
-	modelFailures    *prometheus.Desc
-	modelParked      *prometheus.Desc
-	modelUnplaceable *prometheus.Desc
-	engineInstall    *prometheus.Desc
-	leader           *prometheus.Desc
+	modelDesired      *prometheus.Desc
+	modelFailures     *prometheus.Desc
+	modelParked       *prometheus.Desc
+	modelUnplaceable  *prometheus.Desc
+	modelPreempted    *prometheus.Desc
+	capacityHolds     *prometheus.Desc
+	capacityGuarantee *prometheus.Desc
+	capacityLimit     *prometheus.Desc
+	engineInstall     *prometheus.Desc
+	leader            *prometheus.Desc

 	nodeUp       *prometheus.Desc
 	nodeGPUs     *prometheus.Desc
@@ -70,6 +74,13 @@ type Collector struct {
 	gpuMemoryUsed  *prometheus.Desc
 	gpuUtil        *prometheus.Desc
 	gpuTemp        *prometheus.Desc
+	gpuMemTemp     *prometheus.Desc
+	gpuPower       *prometheus.Desc
+	gpuPowerLimit  *prometheus.Desc
+	gpuThrottled   *prometheus.Desc
+	nodeCPUTemp    *prometheus.Desc
+	nodeCPUPower   *prometheus.Desc
+	nodeInletTemp  *prometheus.Desc
 	gpuHealthy     *prometheus.Desc
 	gpuNVLink      *prometheus.Desc
 	gpuECC         *prometheus.Desc
@@ -138,6 +149,21 @@ func New(src Source) *Collector {
 			"How busy a GPU was over the last sample period. Absent when the driver does not "+
 				"report it, which is not the same as idle. Memory used says a model is loaded; "+
 				"this says whether it is doing anything.", gpu, nil),
+		gpuMemTemp: prometheus.NewDesc("llmgw_gpu_memory_temperature_celsius",
+			"A GPU's memory (HBM) temperature, where the driver reports it.", gpu, nil),
+		gpuPower: prometheus.NewDesc("llmgw_gpu_power_watts",
+			"A GPU's power draw now.", gpu, nil),
+		gpuPowerLimit: prometheus.NewDesc("llmgw_gpu_power_limit_watts",
+			"The power limit a GPU is held to.", gpu, nil),
+		gpuThrottled: prometheus.NewDesc("llmgw_gpu_throttled",
+			"1 while something holds a GPU's clocks down: sw_power_cap, hw_slowdown, sw_thermal, hw_thermal, power_brake.",
+			append(gpu, "reason"), nil),
+		nodeCPUTemp: prometheus.NewDesc("llmgw_node_cpu_temperature_celsius",
+			"A node's hottest CPU sensor.", node, nil),
+		nodeCPUPower: prometheus.NewDesc("llmgw_node_cpu_power_watts",
+			"A node's CPU packages' power draw, from RAPL.", node, nil),
+		nodeInletTemp: prometheus.NewDesc("llmgw_node_inlet_temperature_celsius",
+			"A node's inlet or ambient air sensor, where its board has one.", node, nil),
 		gpuTemp: prometheus.NewDesc("llmgw_gpu_temperature_celsius",
 			"A GPU's reported temperature.", gpu, nil),
 		gpuHealthy: prometheus.NewDesc("llmgw_gpu_healthy",
@@ -168,6 +194,18 @@ func New(src Source) *Collector {
 		modelUnplaceable: prometheus.NewDesc("llmgw_model_unplaceable",
 			"1 when a model is short of replicas and the last attempt found no node to place one on.",
 			[]string{"model"}, nil),
+		capacityHolds: prometheus.NewDesc("llmgw_capacity_holds",
+			"What a part of the tenancy tree holds of a resource, its own models' replicas and those beneath it.",
+			[]string{"scope", "resource"}, nil),
+		capacityGuarantee: prometheus.NewDesc("llmgw_capacity_guarantee",
+			"What a part of the tenancy tree is guaranteed of a resource.",
+			[]string{"scope", "resource"}, nil),
+		capacityLimit: prometheus.NewDesc("llmgw_capacity_limit",
+			"The most of a resource a part of the tenancy tree may hold.",
+			[]string{"scope", "resource"}, nil),
+		modelPreempted: prometheus.NewDesc("llmgw_model_preempted_total",
+			"Replicas of a model stopped to make room for a model of a higher capacity class.",
+			[]string{"model", "class"}, nil),
 		engineInstall: prometheus.NewDesc("llmgw_node_engine_install",
 			"Where installing an engine stands on a node: 1 for the current state.",
 			[]string{"node", "engine", "version", "state"}, nil),
@@ -234,9 +272,11 @@ func (c *Collector) Describe(ch chan<- *prometheus.Desc) {
 		c.ibActive, c.ibRate, c.ibCounter, c.fabricSMs, c.fabricMaster, c.fabricProblems,
 		c.gpuNVLJoined, c.imexUp,
 		c.gpuMemoryTotal, c.gpuMemoryUsed, c.gpuUtil, c.gpuTemp, c.gpuHealthy,
+		c.gpuMemTemp, c.gpuPower, c.gpuPowerLimit, c.gpuThrottled, c.nodeCPUTemp, c.nodeCPUPower, c.nodeInletTemp,
 		c.gpuNVLink, c.gpuECC,
 		c.replicaState, c.replicaReady, c.replicaIdle,
-		c.modelDesired, c.modelFailures, c.modelParked, c.modelUnplaceable,
+		c.modelDesired, c.modelFailures, c.modelParked, c.modelUnplaceable, c.modelPreempted,
+		c.capacityHolds, c.capacityGuarantee, c.capacityLimit,
 		c.engineInstall, c.leader,
 	} {
 		ch <- d
@@ -246,6 +286,7 @@ func (c *Collector) Describe(ch chan<- *prometheus.Desc) {
 // Collect implements prometheus.Collector.
 func (c *Collector) Collect(ch chan<- prometheus.Metric) {
 	now := time.Now()
+	c.collectQuotas(ch)

 	loads := map[string]*inventory.Load{}
 	for _, n := range c.src.Nodes() {
@@ -279,6 +320,16 @@ func (c *Collector) Collect(ch chan<- prometheus.Metric) {
 		}

 		if l := n.Load; l != nil && now.Sub(l.At) < loadFresh {
+			// Each only when measured: a VM has none of these, and a zero
+			// would read as a machine at 0 °C drawing nothing.
+			for _, v := range []struct {
+				d   *prometheus.Desc
+				val float64
+			}{{c.nodeCPUTemp, l.CPUTempC}, {c.nodeCPUPower, l.CPUPowerW}, {c.nodeInletTemp, l.InletTempC}} {
+				if v.val > 0 {
+					ch <- prometheus.MustNewConstMetric(v.d, prometheus.GaugeValue, v.val, n.ID)
+				}
+			}
 			ch <- prometheus.MustNewConstMetric(c.nodeCPU, prometheus.GaugeValue, l.CPUPercent, n.ID)
 			if l.CPUCount > 0 {
 				ch <- prometheus.MustNewConstMetric(c.nodeCPUs, prometheus.GaugeValue, float64(l.CPUCount), n.ID)
@@ -348,7 +399,11 @@ func (c *Collector) Collect(ch chan<- prometheus.Metric) {
 				ch <- prometheus.MustNewConstMetric(c.gpuUtil, prometheus.GaugeValue,
 					float64(g.UtilizationPercent), labels...)
 			}
-			if g.Health.TemperatureC > 0 {
+			// The sensors' fresh reading when there is one; the temperature
+			// read at registration otherwise, which is all an older agent sends.
+			if s, ok := sensorFor(n.Load, g.Index, now); ok {
+				c.gpuSensor(ch, s, labels)
+			} else if g.Health.TemperatureC > 0 {
 				ch <- prometheus.MustNewConstMetric(c.gpuTemp, prometheus.GaugeValue,
 					float64(g.Health.TemperatureC), labels...)
 			}
@@ -455,6 +510,7 @@ func (c *Collector) Collect(ch chan<- prometheus.Metric) {
 		ch <- prometheus.MustNewConstMetric(c.modelFailures, prometheus.GaugeValue, float64(h.Failures), h.Model)
 		ch <- prometheus.MustNewConstMetric(c.modelParked, prometheus.GaugeValue, b2f(h.Parked), h.Model)
 		ch <- prometheus.MustNewConstMetric(c.modelUnplaceable, prometheus.GaugeValue, b2f(h.Unplaceable), h.Model)
+		ch <- prometheus.MustNewConstMetric(c.modelPreempted, prometheus.CounterValue, float64(h.Preempted), h.Model, h.Class)
 	}
 }

@@ -464,3 +520,65 @@ func b2f(b bool) float64 {
 	}
 	return 0
 }
+
+// QuotaSource adds the tenancy tree's capacity quotas. Asked for by type
+// assertion, like HealthSource.
+type QuotaSource interface {
+	QuotaUse() []reconciler.QuotaUse
+}
+
+func (c *Collector) collectQuotas(ch chan<- prometheus.Metric) {
+	qs, ok := c.src.(QuotaSource)
+	if !ok {
+		return
+	}
+	for _, q := range qs.QuotaUse() {
+		for _, r := range []struct {
+			name     string
+			holds    int
+			gua, lim *int
+		}{
+			{"gpus", q.Holds.GPUs, q.Guarantee.GPUs, q.Limit.GPUs},
+			{"cpu_cores", q.Holds.CPUCores, q.Guarantee.CPUCores, q.Limit.CPUCores},
+			{"memory_gb", q.Holds.MemoryGB, q.Guarantee.MemoryGB, q.Limit.MemoryGB},
+		} {
+			if r.gua == nil && r.lim == nil {
+				continue
+			}
+			ch <- prometheus.MustNewConstMetric(c.capacityHolds, prometheus.GaugeValue, float64(r.holds), q.Scope, r.name)
+			if r.gua != nil {
+				ch <- prometheus.MustNewConstMetric(c.capacityGuarantee, prometheus.GaugeValue, float64(*r.gua), q.Scope, r.name)
+			}
+			if r.lim != nil {
+				ch <- prometheus.MustNewConstMetric(c.capacityLimit, prometheus.GaugeValue, float64(*r.lim), q.Scope, r.name)
+			}
+		}
+	}
+}
+
+// sensorFor is a GPU's fresh sensor reading, if the node sent one.
+func sensorFor(l *inventory.Load, index int, now time.Time) (inventory.GPUSensor, bool) {
+	if l == nil || now.Sub(l.At) >= loadFresh {
+		return inventory.GPUSensor{}, false
+	}
+	for _, g := range l.GPUs {
+		if g.Index == index {
+			return g, true
+		}
+	}
+	return inventory.GPUSensor{}, false
+}
+
+func (c *Collector) gpuSensor(ch chan<- prometheus.Metric, s inventory.GPUSensor, labels []string) {
+	for _, v := range []struct {
+		d   *prometheus.Desc
+		val float64
+	}{{c.gpuTemp, s.TempC}, {c.gpuMemTemp, s.MemoryTempC}, {c.gpuPower, s.PowerW}, {c.gpuPowerLimit, s.PowerLimitW}} {
+		if v.val > 0 {
+			ch <- prometheus.MustNewConstMetric(v.d, prometheus.GaugeValue, v.val, labels...)
+		}
+	}
+	for _, reason := range s.Throttle {
+		ch <- prometheus.MustNewConstMetric(c.gpuThrottled, prometheus.GaugeValue, 1, append(labels, reason)...)
+	}
+}
@@ -0,0 +1,41 @@
+package fleetmetrics
+
+import (
+	"strings"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
+)
+
+func TestSensorsArePublishedFreshAndOnlyWhenMeasured(t *testing.T) {
+	gpuNode := inventory.Node{ID: "g1", State: inventory.Ready,
+		GPUs: []discovery.GPU{{Index: 0, Name: "NVIDIA H100 80GB HBM3", MemoryTotalMB: 81559, Health: discovery.GPUHealth{TemperatureC: 40}}},
+		Load: &inventory.Load{At: time.Now(), CPUTempC: 58, CPUPowerW: 65,
+			GPUs: []inventory.GPUSensor{{Index: 0, TempC: 88, PowerW: 698, PowerLimitW: 700,
+				Throttle: []string{"sw_power_cap", "sw_thermal"}}}}}
+	vm := inventory.Node{ID: "vm1", State: inventory.Ready, Load: &inventory.Load{At: time.Now(), CPUPercent: 3}}
+	out := gather(t, fakeSource{nodes: []inventory.Node{gpuNode, vm}})
+
+	for _, want := range []string{
+		"llmgw_node_cpu_temperature_celsius node=g1 -> 58",
+		"llmgw_node_cpu_power_watts node=g1 -> 65",
+		"llmgw_gpu_power_watts class=h100-80g gpu=0 node=g1 -> 698",
+		"llmgw_gpu_power_limit_watts class=h100-80g gpu=0 node=g1 -> 700",
+		"llmgw_gpu_throttled class=h100-80g gpu=0 node=g1 reason=sw_thermal -> 1",
+		// The fresh reading, not the 40 °C from registration.
+		"llmgw_gpu_temperature_celsius class=h100-80g gpu=0 node=g1 -> 88",
+	} {
+		if !strings.Contains(out, want) {
+			t.Errorf("missing %q", want)
+		}
+	}
+	for _, absent := range []string{"node=vm1 -> 0", "llmgw_node_inlet_temperature_celsius", "llmgw_gpu_memory_temperature_celsius"} {
+		for _, line := range strings.Split(out, "\n") {
+			if strings.Contains(line, absent) && (strings.Contains(line, "temperature") || strings.Contains(line, "power")) {
+				t.Errorf("published what was not measured: %s", line)
+			}
+		}
+	}
+}
@@ -79,6 +79,9 @@ type Node struct {
 	// it, which is treated as no constraint — the same as every node before
 	// this field existed.
 	MemoryTotalMB int `json:"memory_total_mb,omitempty"`
+	// CPUCount is the CPUs the agent may run on. Zero is an agent too old to
+	// say, which, like memory, constrains nothing.
+	CPUCount int `json:"cpu_count,omitempty"`

 	// Weights are the model files this node can see, however they got there.
 	// Empty means the node was not asked to look, which is not the same as
@@ -301,6 +304,21 @@ type Load struct {
 	MemAvailBytes int64
 	// Shards is each engine's use, by shard ID.
 	Shards map[string]ShardLoad
+	// What the node's sensors say; zero is not measured.
+	CPUTempC   float64
+	CPUPowerW  float64
+	InletTempC float64
+	GPUs       []GPUSensor
+}
+
+// GPUSensor is one card's heat and power, and what is throttling it.
+type GPUSensor struct {
+	Index       int      `json:"index"`
+	TempC       float64  `json:"temp_c,omitempty"`
+	MemoryTempC float64  `json:"memory_temp_c,omitempty"`
+	PowerW      float64  `json:"power_w,omitempty"`
+	PowerLimitW float64  `json:"power_limit_w,omitempty"`
+	Throttle    []string `json:"throttle,omitempty"`
 }

 // ShardLoad is one engine's use: cores busy, and resident memory.
@@ -373,6 +391,9 @@ type Allocation struct {
 	// Zero for a profile that declared nothing, which constrains nothing and
 	// is counted as nothing.
 	MemoryGB int
+	// CPUCores is the cores this shard was given to compute on, for a shard
+	// with no GPU; zero for one that shares whatever is free.
+	CPUCores int
 }

 // Store holds the cluster inventory. It is safe for concurrent use.
@@ -392,6 +413,23 @@ func NewStore() *Store {
 	return &Store{nodes: map[string]*Node{}, allocs: map[string][]Allocation{}}
 }

+// Clone is a copy to try decisions on without making them: allocations made
+// on it are its own. Nodes are copied by value; the slices inside them are
+// shared, and a clone must not write through them.
+func (s *Store) Clone() *Store {
+	s.mu.RLock()
+	defer s.mu.RUnlock()
+	c := NewStore()
+	for id, n := range s.nodes {
+		cp := *n
+		c.nodes[id] = &cp
+	}
+	for id, list := range s.allocs {
+		c.allocs[id] = slices.Clone(list)
+	}
+	return c
+}
+
 // Upsert records a node's registration, preserving operator-set state such as
 // a cordon: an agent reconnecting must not silently undo an operator's decision.
 // RecordSMChange keeps a port's change of master subnet manager, the last
@@ -0,0 +1,562 @@
+// Package modelserve takes a model from a registry repository to a replica
+// answering requests, as one job.
+//
+// Adding a model was four commands and a hand-written YAML file: plan it,
+// fetch it into the mirror, write its catalog entry with a memory figure
+// somebody guessed, scale it — and then find out whether it loads. This runs
+// those steps in the control plane, where the mirror, the registry, the
+// catalog and the fleet already are, and reports each one as it goes, so
+// whatever asks — ctl, a page — only has to watch.
+package modelserve
+
+import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"log/slog"
+	"net/http"
+	"regexp"
+	"slices"
+	"strconv"
+	"strings"
+	"sync"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
+)
+
+// Registry reads a repository.
+type Registry interface {
+	Repo(ctx context.Context, id string) (hf.Repo, error)
+}
+
+// Mirror fetches files and reports on them.
+type Mirror interface {
+	Fetch(ctx context.Context, req mirror.FetchRequest) (mirror.FetchResponse, error)
+	List(ctx context.Context) (mirror.ModelsResponse, error)
+}
+
+// Catalog adds a model.
+type Catalog interface {
+	Add(ctx context.Context, m catalog.Model, by string) error
+}
+
+// Fleet is what the job watches once the model is in the catalog.
+type Fleet interface {
+	Replicas() []reconciler.Replica
+	ModelHealth() []reconciler.ModelHealth
+	EngineKey(replicaID string) string
+}
+
+// Deps are what a job needs.
+type Deps struct {
+	Registry Registry
+	Mirror   Mirror
+	Catalog  Catalog
+	Fleet    Fleet
+	// LlamaVersion is the llama.cpp build the nodes install, which decides
+	// which architectures load.
+	LlamaVersion string
+	HTTP         *http.Client
+	Log          *slog.Logger
+	// Poll is how often the job looks at the mirror and the fleet.
+	Poll time.Duration
+	// ReadyTimeout bounds the wait for a replica: the node fetching the
+	// weights from the mirror and the engine loading them.
+	ReadyTimeout time.Duration
+}
+
+// Request is what to serve.
+type Request struct {
+	// Repo is the registry repository, owner/name.
+	Repo string `json:"repo"`
+	// Quant picks the file by its quantisation, Q4_K_M unless given.
+	Quant string `json:"quant,omitempty"`
+	// File picks it by name instead.
+	File string `json:"file,omitempty"`
+	// Name is the catalog name; the repository suggests one otherwise.
+	Name        string   `json:"name,omitempty"`
+	Aliases     []string `json:"aliases,omitempty"`
+	Description string   `json:"description,omitempty"`
+	// Replicas to run; 0 makes it on demand, loaded when first asked for.
+	Replicas int `json:"replicas"`
+	// CtxPool is llama.cpp's --ctx-size; ContextLength what one request may
+	// use. They default to 16384 and 4096, what four parallel slots divide.
+	CtxPool       int `json:"ctx_pool,omitempty"`
+	ContextLength int `json:"context_length,omitempty"`
+	// CPUCores is what a replica computes on; zero takes half the smallest
+	// node it fits on.
+	CPUCores int `json:"cpu_cores,omitempty"`
+	// By is who asked, for the catalog's record.
+	By string `json:"-"`
+}
+
+// The states a job and its steps can be in.
+const (
+	Running = "running"
+	Done    = "done"
+	Failed  = "failed"
+	Skipped = "skipped"
+)
+
+// Step is one stage of a job.
+type Step struct {
+	Name   string    `json:"name"`
+	State  string    `json:"state"`
+	Detail string    `json:"detail,omitempty"`
+	At     time.Time `json:"at"`
+}
+
+// Job is one model being served, and where it stands.
+type Job struct {
+	ID      string    `json:"id"`
+	Request Request   `json:"request"`
+	State   string    `json:"state"`
+	Error   string    `json:"error,omitempty"`
+	Model   string    `json:"model,omitempty"`
+	Steps   []Step    `json:"steps"`
+	Started time.Time `json:"started"`
+	// Reply is what the model said to the smoke test, and TokensPerSecond
+	// how fast it said it.
+	Reply           string  `json:"reply,omitempty"`
+	TokensPerSecond float64 `json:"tokens_per_second,omitempty"`
+}
+
+// Server runs jobs and remembers them.
+type Server struct {
+	deps Deps
+	mu   sync.Mutex
+	jobs map[string]*Job
+	seq  int
+}
+
+// New builds a server.
+func New(d Deps) *Server {
+	if d.Poll <= 0 {
+		d.Poll = 2 * time.Second
+	}
+	if d.ReadyTimeout <= 0 {
+		d.ReadyTimeout = 30 * time.Minute
+	}
+	if d.HTTP == nil {
+		d.HTTP = &http.Client{Timeout: 5 * time.Minute}
+	}
+	if d.Log == nil {
+		d.Log = slog.Default()
+	}
+	return &Server{deps: d, jobs: map[string]*Job{}}
+}
+
+var repoPattern = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._-]*/[A-Za-z0-9][A-Za-z0-9._-]*$`)
+
+// Start begins a job and returns at once; the job runs on its own.
+func (s *Server) Start(req Request) (Job, error) {
+	req.Repo = strings.Trim(strings.TrimSpace(req.Repo), "/")
+	if !repoPattern.MatchString(req.Repo) {
+		return Job{}, fmt.Errorf("%q is not a repository; it looks like owner/name", req.Repo)
+	}
+	if req.Replicas < 0 {
+		return Job{}, errors.New("replicas cannot be negative")
+	}
+	if req.Quant == "" && req.File == "" {
+		req.Quant = "Q4_K_M"
+	}
+	if req.CtxPool <= 0 {
+		req.CtxPool = 16384
+	}
+	if req.ContextLength <= 0 {
+		req.ContextLength = 4096
+	}
+	s.mu.Lock()
+	s.seq++
+	j := &Job{ID: fmt.Sprintf("serve-%d-%d", time.Now().Unix(), s.seq), Request: req,
+		State: Running, Started: time.Now()}
+	s.jobs[j.ID] = j
+	s.mu.Unlock()
+	go s.run(j)
+	return s.snapshot(j), nil
+}
+
+// Get returns a job as it stands.
+func (s *Server) Get(id string) (Job, bool) {
+	s.mu.Lock()
+	j, ok := s.jobs[id]
+	s.mu.Unlock()
+	if !ok {
+		return Job{}, false
+	}
+	return s.snapshot(j), true
+}
+
+func (s *Server) snapshot(j *Job) Job {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	out := *j
+	out.Steps = slices.Clone(j.Steps)
+	return out
+}
+
+// step starts a new step, or updates the detail of the current one.
+func (s *Server) step(j *Job, name, state, detail string) {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	if n := len(j.Steps); n > 0 && j.Steps[n-1].Name == name {
+		j.Steps[n-1].State, j.Steps[n-1].Detail, j.Steps[n-1].At = state, detail, time.Now()
+		return
+	}
+	j.Steps = append(j.Steps, Step{Name: name, State: state, Detail: detail, At: time.Now()})
+}
+
+func (s *Server) fail(j *Job, name string, err error) {
+	s.step(j, name, Failed, err.Error())
+	s.mu.Lock()
+	j.State, j.Error = Failed, err.Error()
+	s.mu.Unlock()
+	s.deps.Log.Warn("serving a model failed", "job", j.ID, "repo", j.Request.Repo, "step", name, "err", err)
+}
+
+func (s *Server) run(j *Job) {
+	ctx := context.Background()
+	req := j.Request
+
+	// 1. The registry: which files.
+	s.step(j, "registry", Running, "reading "+req.Repo)
+	repo, err := s.deps.Registry.Repo(ctx, req.Repo)
+	if err != nil {
+		s.fail(j, "registry", err)
+		return
+	}
+	if repo.Gated.Is() {
+		s.fail(j, "registry", fmt.Errorf("%s is gated: it needs an account's approval the mirror does not have", req.Repo))
+		return
+	}
+	set, err := pick(repo, req)
+	if err != nil {
+		s.fail(j, "registry", err)
+		return
+	}
+	name := req.Name
+	if name == "" {
+		name = hf.ModelName(repo.ID)
+	}
+	s.mu.Lock()
+	j.Model = name
+	s.mu.Unlock()
+	s.step(j, "registry", Done, fmt.Sprintf("%s, %.1f GB in %d file(s), revision %s",
+		set.Name, gb(set.Bytes), len(set.Files), repo.ShortRevision()))
+
+	// 2. The mirror: fetch, and check what arrived against what was published.
+	source, err := s.mirror(ctx, j, repo, set, name)
+	if err != nil {
+		s.fail(j, "mirror", err)
+		return
+	}
+
+	// 3. Whether the nodes' llama.cpp loads it, and how much memory it needs.
+	s.step(j, "check", Running, "reading its header from the mirror")
+	var header gguf.Header
+	if !strings.HasSuffix(source, "/") {
+		v := gguf.Check(ctx, source, s.deps.LlamaVersion)
+		if !v.Servable {
+			s.fail(j, "check", fmt.Errorf("llama.cpp %s on the nodes cannot run this: %s", s.deps.LlamaVersion, v.Reason))
+			return
+		}
+		header = v.Header
+	}
+	memGB := int((header.MemoryBytes(set.Bytes, req.CtxPool) + 1<<30 - 1) >> 30)
+	basis := fmt.Sprintf("weights %.1f GB + KV cache %.1f GB for %d tokens + engine",
+		gb(set.Bytes), gb(int64(req.CtxPool)*header.KVBytesPerToken()), req.CtxPool)
+	if memGB == 0 {
+		// A set of shards, or a header that did not say: weights plus half
+		// again plus two, the rough figure from before shapes were read.
+		memGB = int(gb(set.Bytes)*1.5) + 2
+		basis = "weights plus half again plus two; the header gave no shape"
+	}
+	s.step(j, "check", Done, fmt.Sprintf("%s, needs about %d GB (%s)", orUnknown(header.Architecture), memGB, basis))
+
+	cores := req.CPUCores
+	if cores <= 0 {
+		if nc, ok := s.deps.Fleet.(interface{ NodeCPUs(int) []int }); ok {
+			cores = catalog.SuggestCores(nc.NodeCPUs(memGB))
+		}
+	}
+
+	// 4. The catalog.
+	m := catalog.Model{
+		Name: name, Aliases: req.Aliases, Description: req.Description, Source: source,
+		RoutingProfile: "chat", ContextLength: req.ContextLength, MaxOutput: 1024,
+		Profiles: []catalog.Profile{{
+			Name: "cpu", Engine: "llamacpp", Weight: 1, Priority: 100,
+			Args: []string{"--ctx-size", strconv.Itoa(req.CtxPool)}, MinMemoryGB: memGB, CPUCores: cores,
+		}},
+		Scaling: catalog.Scaling{Min: req.Replicas, Max: max(req.Replicas, 1)},
+	}
+	s.step(j, "catalog", Running, "adding "+name)
+	if err := s.deps.Catalog.Add(ctx, m, req.By); err != nil {
+		s.fail(j, "catalog", err)
+		return
+	}
+	if req.Replicas == 0 {
+		s.step(j, "catalog", Done, name+" added on demand: it loads when first asked for")
+		s.finish(j)
+		return
+	}
+	s.step(j, "catalog", Done, fmt.Sprintf("%s added, %d replica(s)", name, req.Replicas))
+
+	// 5. A replica, ready.
+	rep, err := s.waitReady(ctx, j, name)
+	if err != nil {
+		s.fail(j, "placement", err)
+		return
+	}
+
+	// 6. Does it answer.
+	s.step(j, "smoke test", Running, "asking "+rep.ID+" to say hello")
+	reply, tps, err := s.smoke(ctx, rep, name)
+	if err != nil {
+		s.fail(j, "smoke test", err)
+		return
+	}
+	s.mu.Lock()
+	j.Reply, j.TokensPerSecond = reply, tps
+	s.mu.Unlock()
+	s.step(j, "smoke test", Done, fmt.Sprintf("answered at %.1f tokens/s: %q", tps, reply))
+	s.finish(j)
+}
+
+func (s *Server) finish(j *Job) {
+	s.mu.Lock()
+	j.State = Done
+	s.mu.Unlock()
+	s.deps.Log.Info("model served", "job", j.ID, "repo", j.Request.Repo, "model", j.Model)
+}
+
+// pick chooses the file set to serve.
+func pick(repo hf.Repo, req Request) (hf.FileSet, error) {
+	var weights []hf.File
+	for _, f := range repo.Files {
+		lower := strings.ToLower(f.Path)
+		if !strings.HasSuffix(lower, ".gguf") || strings.Contains(lower, "mmproj") ||
+			strings.Contains(lower, "imatrix") {
+			continue
+		}
+		weights = append(weights, f)
+	}
+	sets := hf.Group(weights)
+	if len(sets) == 0 {
+		return hf.FileSet{}, fmt.Errorf("%s has no GGUF files; the nodes run llama.cpp, which needs one", repo.ID)
+	}
+	var names []string
+	for _, st := range sets {
+		names = append(names, st.Name)
+		if req.File != "" && (st.Name == req.File || slices.ContainsFunc(st.Files, func(f hf.File) bool { return f.Path == req.File })) {
+			return st, nil
+		}
+	}
+	if req.File == "" {
+		want := strings.ToLower(req.Quant)
+		var match []hf.FileSet
+		for _, st := range sets {
+			if strings.Contains(strings.ToLower(st.Name), want) {
+				match = append(match, st)
+			}
+		}
+		if len(match) == 1 {
+			return match[0], nil
+		}
+		if len(match) > 1 {
+			// Both a split and a joined copy of one quantisation: the joined
+			// one, which is one download to verify rather than several.
+			slices.SortFunc(match, func(a, b hf.FileSet) int { return a.Shards - b.Shards })
+			return match[0], nil
+		}
+	}
+	return hf.FileSet{}, fmt.Errorf("%s has no file matching %q; it has: %s",
+		repo.ID, or(req.File, req.Quant), strings.Join(names, ", "))
+}
+
+// mirror fetches the set into the mirror and returns the source a catalog
+// entry names: the file, or the directory for a set of shards.
+func (s *Server) mirror(ctx context.Context, j *Job, repo hf.Repo, set hf.FileSet, name string) (string, error) {
+	version := repo.ShortRevision()
+	s.step(j, "mirror", Running, "asking the mirror for "+set.Name)
+	var base string
+	for _, f := range set.Files {
+		file := mirror.LastPathElement(f.Path)
+		resp, err := s.deps.Mirror.Fetch(ctx, mirror.FetchRequest{
+			Source: repo.DownloadURL("", f.Path), Model: name, Version: version, File: file,
+		})
+		// Already there is not a failure: the same commit fetched before.
+		if err != nil && !strings.Contains(err.Error(), "already") {
+			return "", err
+		}
+		base = resp.URL
+	}
+	deadline := time.Now().Add(6 * time.Hour)
+	for {
+		got, err := s.deps.Mirror.List(ctx)
+		if err != nil {
+			return "", err
+		}
+		var done, total int64
+		present, failed := 0, ""
+		sums := map[string]string{}
+		for _, st := range got.Models {
+			if st.Model != name || st.Version != version {
+				continue
+			}
+			done += st.BytesDone
+			total += max(st.BytesTotal, st.BytesDone)
+			switch st.State {
+			case mirror.Present:
+				present++
+				sums[st.File] = st.SHA256
+			case mirror.Failed:
+				failed = st.Err
+			}
+		}
+		if failed != "" {
+			return "", fmt.Errorf("the mirror could not fetch it: %s", failed)
+		}
+		if present >= len(set.Files) {
+			// What arrived against what the registry published: a mirror
+			// copies one file to every node, and is the one place a corrupted
+			// download is caught once instead of loaded everywhere.
+			for _, f := range set.Files {
+				file := mirror.LastPathElement(f.Path)
+				if f.SHA256 != "" && sums[file] != "" && !strings.EqualFold(f.SHA256, sums[file]) {
+					return "", fmt.Errorf("%s arrived with sha256 %s, and the registry published %s",
+						file, sums[file][:12], f.SHA256[:12])
+				}
+			}
+			source := base
+			if len(set.Files) > 1 {
+				source = strings.TrimSuffix(source, mirror.LastPathElement(source))
+			}
+			s.step(j, "mirror", Done, fmt.Sprintf("%.1f GB on the mirror, checksums match the registry", gb(total)))
+			return source, nil
+		}
+		if total > 0 {
+			s.step(j, "mirror", Running, fmt.Sprintf("fetching: %.1f of %.1f GB (%d%%)",
+				gb(done), gb(total), done*100/total))
+		}
+		if time.Now().After(deadline) {
+			return "", errors.New("the mirror had not finished after six hours")
+		}
+		time.Sleep(s.deps.Poll)
+	}
+}
+
+// waitReady waits for a replica of the model to serve, reporting what it is
+// doing and giving up early when it cannot be placed or keeps failing.
+func (s *Server) waitReady(ctx context.Context, j *Job, name string) (reconciler.Replica, error) {
+	s.step(j, "placement", Running, "waiting for the control plane to place it")
+	started := time.Now()
+	deadline := started.Add(s.deps.ReadyTimeout)
+	for {
+		for _, h := range s.deps.Fleet.ModelHealth() {
+			if h.Model != name {
+				continue
+			}
+			if h.Parked {
+				return reconciler.Replica{}, fmt.Errorf("it failed to start %d times and is parked: %s", h.Failures, h.Reason)
+			}
+			// A minute's grace from when this step began, not from the job:
+			// the download before it can take hours, and the first pass may
+			// not have run yet.
+			if h.Unplaceable && time.Since(started) > time.Minute {
+				return reconciler.Replica{}, fmt.Errorf("no node can host it:\n%s", h.Placement)
+			}
+		}
+		for _, r := range s.deps.Fleet.Replicas() {
+			if r.Model != name {
+				continue
+			}
+			if r.State == reconciler.Ready {
+				s.step(j, "placement", Done, fmt.Sprintf("%s ready on %s at %s", r.ID,
+					strings.Join(r.NodeIDs(), ","), r.Endpoint))
+				return r, nil
+			}
+			s.step(j, "placement", Running, fmt.Sprintf("%s on %s is %s", r.ID,
+				strings.Join(r.NodeIDs(), ","), r.State))
+		}
+		if time.Now().After(deadline) {
+			return reconciler.Replica{}, fmt.Errorf("no replica was ready after %s", s.deps.ReadyTimeout)
+		}
+		select {
+		case <-ctx.Done():
+			return reconciler.Replica{}, ctx.Err()
+		case <-time.After(s.deps.Poll):
+		}
+	}
+}
+
+// smoke sends one short request straight to the replica, with its engine key.
+func (s *Server) smoke(ctx context.Context, r reconciler.Replica, name string) (string, float64, error) {
+	body, _ := json.Marshal(map[string]any{
+		"model":      name,
+		"max_tokens": 32,
+		"messages":   []map[string]string{{"role": "user", "content": "Say hello in one short sentence."}},
+	})
+	req, err := http.NewRequestWithContext(ctx, http.MethodPost,
+		strings.TrimRight(r.Endpoint, "/")+"/v1/chat/completions", bytes.NewReader(body))
+	if err != nil {
+		return "", 0, err
+	}
+	req.Header.Set("Content-Type", "application/json")
+	if k := s.deps.Fleet.EngineKey(r.ID); k != "" {
+		req.Header.Set("Authorization", "Bearer "+k)
+	}
+	start := time.Now()
+	resp, err := s.deps.HTTP.Do(req)
+	if err != nil {
+		return "", 0, err
+	}
+	defer resp.Body.Close()
+	b, _ := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
+	if resp.StatusCode != http.StatusOK {
+		return "", 0, fmt.Errorf("the replica answered %s: %s", resp.Status, strings.TrimSpace(string(b)))
+	}
+	var out struct {
+		Choices []struct {
+			Message struct {
+				Content string `json:"content"`
+			} `json:"message"`
+		} `json:"choices"`
+		Usage struct {
+			CompletionTokens int `json:"completion_tokens"`
+		} `json:"usage"`
+	}
+	if err := json.Unmarshal(b, &out); err != nil || len(out.Choices) == 0 {
+		return "", 0, fmt.Errorf("the replica's answer was not a completion: %s", strings.TrimSpace(string(b)))
+	}
+	secs := time.Since(start).Seconds()
+	tps := 0.0
+	if secs > 0 {
+		tps = float64(out.Usage.CompletionTokens) / secs
+	}
+	return strings.TrimSpace(out.Choices[0].Message.Content), tps, nil
+}
+
+func gb(b int64) float64 { return float64(b) / (1 << 30) }
+
+func or(a, b string) string {
+	if a != "" {
+		return a
+	}
+	return b
+}
+
+func orUnknown(arch string) string {
+	if arch == "" {
+		return "architecture not read"
+	}
+	return arch
+}
@@ -0,0 +1,225 @@
+package modelserve
+
+import (
+	"bytes"
+	"context"
+	"encoding/binary"
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"sync"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
+)
+
+// ggufHeader is enough of a GGUF file for its architecture and shape.
+func ggufHeader() []byte {
+	var b bytes.Buffer
+	w := func(v any) { _ = binary.Write(&b, binary.LittleEndian, v) }
+	str := func(s string) { w(uint64(len(s))); b.WriteString(s) }
+	u32 := func(k string, v uint32) { str(k); w(uint32(4)); w(v) }
+	b.WriteString("GGUF")
+	w(uint32(3))
+	w(uint64(0))
+	w(uint64(5))
+	str("general.architecture")
+	w(uint32(8))
+	str("qwen2")
+	u32("qwen2.block_count", 28)
+	u32("qwen2.attention.head_count", 12)
+	u32("qwen2.attention.head_count_kv", 2)
+	u32("qwen2.embedding_length", 1536)
+	return b.Bytes()
+}
+
+type fakeRegistry struct{ repo hf.Repo }
+
+func (f fakeRegistry) Repo(context.Context, string) (hf.Repo, error) { return f.repo, nil }
+
+// fakeMirror serves the header at every file URL and reports each fetched
+// file present, with the checksum it was told to.
+type fakeMirror struct {
+	mu   sync.Mutex
+	srv  *httptest.Server
+	got  []mirror.FetchRequest
+	sums map[string]string
+}
+
+func (m *fakeMirror) Fetch(_ context.Context, r mirror.FetchRequest) (mirror.FetchResponse, error) {
+	m.mu.Lock()
+	defer m.mu.Unlock()
+	m.got = append(m.got, r)
+	return mirror.FetchResponse{URL: m.srv.URL + "/" + r.Model + "/" + r.Version + "/" + r.File}, nil
+}
+
+func (m *fakeMirror) List(context.Context) (mirror.ModelsResponse, error) {
+	m.mu.Lock()
+	defer m.mu.Unlock()
+	var out mirror.ModelsResponse
+	for _, r := range m.got {
+		out.Models = append(out.Models, mirror.Status{Model: r.Model, Version: r.Version, File: r.File,
+			State: mirror.Present, BytesDone: 1 << 30, BytesTotal: 1 << 30, SHA256: m.sums[r.File]})
+	}
+	return out, nil
+}
+
+type fakeCatalog struct {
+	mu    sync.Mutex
+	added []catalog.Model
+}
+
+func (c *fakeCatalog) Add(_ context.Context, m catalog.Model, _ string) error {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+	c.added = append(c.added, m)
+	return nil
+}
+
+// fakeFleet has a replica ready at engine once the catalog holds the model.
+type fakeFleet struct {
+	cat    *fakeCatalog
+	engine string
+	health []reconciler.ModelHealth
+}
+
+func (f *fakeFleet) Replicas() []reconciler.Replica {
+	f.cat.mu.Lock()
+	defer f.cat.mu.Unlock()
+	var out []reconciler.Replica
+	for _, m := range f.cat.added {
+		out = append(out, reconciler.Replica{ID: m.Name + "-r1", Model: m.Name,
+			State: reconciler.Ready, Endpoint: f.engine,
+			Shards: []reconciler.Shard{{NodeID: "node01"}}})
+	}
+	return out
+}
+func (f *fakeFleet) ModelHealth() []reconciler.ModelHealth { return f.health }
+func (f *fakeFleet) EngineKey(string) string               { return "k" }
+
+func setup(t *testing.T, files []hf.File, sums map[string]string) (*Server, *fakeMirror, *fakeCatalog, *fakeFleet) {
+	t.Helper()
+	header := ggufHeader()
+	files0 := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+		_, _ = w.Write(header)
+	}))
+	t.Cleanup(files0.Close)
+	engine := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		if r.Header.Get("Authorization") != "Bearer k" {
+			w.WriteHeader(http.StatusUnauthorized)
+			return
+		}
+		_, _ = w.Write([]byte(`{"choices":[{"message":{"content":"Hello!"}}],"usage":{"completion_tokens":3}}`))
+	}))
+	t.Cleanup(engine.Close)
+
+	m := &fakeMirror{srv: files0, sums: sums}
+	c := &fakeCatalog{}
+	f := &fakeFleet{cat: c, engine: engine.URL}
+	s := New(Deps{
+		Registry: fakeRegistry{repo: hf.Repo{ID: "bartowski/Qwen2.5-1.5B-Instruct-GGUF", Revision: "abcdef123456", Files: files}},
+		Mirror:   m, Catalog: c, Fleet: f, Poll: 10 * time.Millisecond, ReadyTimeout: 5 * time.Second,
+	})
+	return s, m, c, f
+}
+
+func wait(t *testing.T, s *Server, id string) Job {
+	t.Helper()
+	deadline := time.Now().Add(10 * time.Second)
+	for time.Now().Before(deadline) {
+		if j, _ := s.Get(id); j.State != Running {
+			return j
+		}
+		time.Sleep(10 * time.Millisecond)
+	}
+	t.Fatal("the job never finished")
+	return Job{}
+}
+
+func TestAModelGoesFromTheRegistryToAnAnsweringReplica(t *testing.T) {
+	files := []hf.File{
+		{Path: "Qwen2.5-1.5B-Instruct-Q8_0.gguf", Bytes: 2 << 30, SHA256: "aa"},
+		{Path: "Qwen2.5-1.5B-Instruct-Q4_K_M.gguf", Bytes: 1 << 30, SHA256: "bb"},
+		{Path: "mmproj-Q4_K_M.gguf", Bytes: 1 << 20},
+	}
+	s, m, c, _ := setup(t, files, map[string]string{"Qwen2.5-1.5B-Instruct-Q4_K_M.gguf": "BB"})
+	job, err := s.Start(Request{Repo: "bartowski/Qwen2.5-1.5B-Instruct-GGUF", Replicas: 1})
+	if err != nil {
+		t.Fatal(err)
+	}
+	j := wait(t, s, job.ID)
+	if j.State != Done {
+		t.Fatalf("job %s: %s\n%+v", j.State, j.Error, j.Steps)
+	}
+	if len(m.got) != 1 || m.got[0].File != "Qwen2.5-1.5B-Instruct-Q4_K_M.gguf" || m.got[0].Version != "abcdef12" {
+		t.Errorf("fetched %+v; expected the Q4_K_M file at its revision", m.got)
+	}
+	mod := c.added[0]
+	if mod.Name != "qwen2.5-1.5b-instruct" || mod.Profiles[0].MinMemoryGB == 0 || mod.Scaling.Min != 1 {
+		t.Errorf("catalog entry %+v", mod)
+	}
+	if j.Reply != "Hello!" {
+		t.Errorf("the smoke test's answer was %q", j.Reply)
+	}
+	var names []string
+	for _, st := range j.Steps {
+		names = append(names, st.Name)
+	}
+	if strings.Join(names, ",") != "registry,mirror,check,catalog,placement,smoke test" {
+		t.Errorf("steps %v", names)
+	}
+}
+
+func TestAQuantTheRepositoryDoesNotHaveIsNamedWithWhatItHas(t *testing.T) {
+	s, _, c, _ := setup(t, []hf.File{{Path: "m-Q8_0.gguf", Bytes: 1}}, nil)
+	job, _ := s.Start(Request{Repo: "a/b", Replicas: 1})
+	j := wait(t, s, job.ID)
+	if j.State != Failed || !strings.Contains(j.Error, "m-Q8_0.gguf") {
+		t.Errorf("%s: %s", j.State, j.Error)
+	}
+	if len(c.added) != 0 {
+		t.Error("something was added to the catalog")
+	}
+}
+
+func TestADownloadThatDoesNotMatchTheRegistryIsNotServed(t *testing.T) {
+	files := []hf.File{{Path: "m-Q4_K_M.gguf", Bytes: 1 << 30, SHA256: "aaaaaaaaaaaaaaaa"}}
+	s, _, c, _ := setup(t, files, map[string]string{"m-Q4_K_M.gguf": "bbbbbbbbbbbbbbbb"})
+	job, _ := s.Start(Request{Repo: "a/b", Replicas: 1})
+	j := wait(t, s, job.ID)
+	if j.State != Failed || !strings.Contains(j.Error, "registry published") {
+		t.Errorf("%s: %s", j.State, j.Error)
+	}
+	if len(c.added) != 0 {
+		t.Error("a corrupted download reached the catalog")
+	}
+}
+
+func TestAParkedModelStopsTheWait(t *testing.T) {
+	s, _, _, f := setup(t, []hf.File{{Path: "m-Q4_K_M.gguf", Bytes: 1 << 30}}, nil)
+	f.health = []reconciler.ModelHealth{{Model: "qwen2.5-1.5b-instruct", Parked: true, Failures: 5, Reason: "unknown model architecture"}}
+	// A fleet where it never becomes ready, only parked.
+	s.deps.Fleet = parkedFleet{f}
+	job, _ := s.Start(Request{Repo: "a/b", Replicas: 1})
+	j := wait(t, s, job.ID)
+	if j.State != Failed || !strings.Contains(j.Error, "parked") {
+		t.Errorf("%s: %s", j.State, j.Error)
+	}
+}
+
+type parkedFleet struct{ *fakeFleet }
+
+func (parkedFleet) Replicas() []reconciler.Replica { return nil }
+
+func TestARepositoryThatIsNotOneIsRefused(t *testing.T) {
+	s := New(Deps{})
+	for _, r := range []string{"", "noslash", "../etc/passwd", "a/b/c"} {
+		if _, err := s.Start(Request{Repo: r}); err == nil {
+			t.Errorf("%q accepted", r)
+		}
+	}
+}
@@ -18,6 +18,7 @@ package modelsvc
 import (
 	"context"
 	"encoding/json"
+	"errors"
 	"fmt"
 	"log/slog"
 	"sync"
@@ -86,6 +87,9 @@ func (s *Service) Sources(ctx context.Context) (map[string][]catalog.Model, erro
 		if m.Source != "" {
 			out[m.Source] = append(out[m.Source], m)
 		}
+		for _, ad := range m.Adapters {
+			out[ad.Source] = append(out[ad.Source], m)
+		}
 	}
 	return out, nil
 }
@@ -104,6 +108,16 @@ func (s *Service) Add(ctx context.Context, m catalog.Model, by string) error {
 	if s.Declared(m.Name) {
 		return fmt.Errorf("model %q is declared in configuration; edit that rather than adding a second definition", m.Name)
 	}
+	// A model with no GPU is all host memory, and one that declares none
+	// fits everywhere as far as placement can tell — however many of it
+	// there already are. Required of what is added now; models stored before
+	// this still load, and say what they are missing.
+	for _, p := range m.Profiles {
+		if p.CPUOnly() && p.MinMemoryGB <= 0 {
+			return fmt.Errorf("model %q profile %q runs without a GPU and declares no memory (min_memory_gb): "+
+				"placement could not tell when a node is full", m.Name, p.Name)
+		}
+	}
 	if s.check != nil {
 		if err := s.check(ctx, m); err != nil {
 			return err
@@ -189,36 +203,115 @@ func (s *Service) Remove(ctx context.Context, name string) error {
 	return nil
 }

-// Scale changes how many replicas a model runs, without a restart.
-//
-// Only the scaling block moves. Everything else about the model — its source,
-// its profiles, the engine arguments somebody tuned — is read back from what
-// was stored and written out again untouched, so changing a number cannot
-// quietly become a redefinition.
-//
-// A model declared in configuration is refused, for the same reason Remove
-// refuses one: the YAML is read on every start, so the new number would hold
-// until the next restart and then silently revert. A cluster that disagrees
-// with its own config file is worse than an error message saying where to edit.
+// Scale changes how many replicas a model added at runtime runs.
 func (s *Service) Scale(ctx context.Context, name string, sc catalog.Scaling, by string) (catalog.Model, error) {
+	final, err := s.update(ctx, name, by, "change its scaling", func(m *catalog.Model) error {
+		m.Scaling = sc
+		return nil
+	})
+	if err == nil {
+		s.log.Info("model scaled", "model", name,
+			"min", final.Scaling.Min, "max", final.Scaling.Max, "by", by)
+	}
+	return final, err
+}
+
+// Resources sets what a model's profile reserves of a node: host memory, and
+// the CPU cores a replica without a GPU computes on. An empty profile name
+// sets every profile.
+func (s *Service) Resources(ctx context.Context, name, profile string, memoryGB, cpuCores int, by string) (catalog.Model, error) {
+	if memoryGB < 0 || cpuCores < 0 {
+		return catalog.Model{}, errors.New("memory and cores cannot be negative")
+	}
+	final, err := s.update(ctx, name, by, "change its resources", func(m *catalog.Model) error {
+		found := false
+		for i := range m.Profiles {
+			if profile != "" && m.Profiles[i].Name != profile {
+				continue
+			}
+			found = true
+			m.Profiles[i].MinMemoryGB = memoryGB
+			m.Profiles[i].CPUCores = cpuCores
+		}
+		if !found {
+			return fmt.Errorf("model %q has no profile %q", name, profile)
+		}
+		return nil
+	})
+	if err == nil {
+		s.log.Info("model resources set", "model", name, "profile", profile,
+			"memory_gb", memoryGB, "cpu_cores", cpuCores, "by", by)
+	}
+	return final, err
+}
+
+// SetClass sets a model's capacity class: critical, standard or
+// opportunistic, or empty to go back to what its scaling implies.
+func (s *Service) SetClass(ctx context.Context, name, class, by string) (catalog.Model, error) {
+	final, err := s.update(ctx, name, by, "change its class", func(m *catalog.Model) error {
+		m.Class = class
+		return nil
+	})
+	if err == nil {
+		s.log.Info("model class set", "model", name, "class", final.EffectiveClass(), "by", by)
+	}
+	return final, err
+}
+
+// SetOwner sets the part of the tenancy tree a model's capacity counts
+// against, org:ID, team:ID or project:ID; empty makes it shared.
+func (s *Service) SetOwner(ctx context.Context, name, owner, by string) (catalog.Model, error) {
+	final, err := s.update(ctx, name, by, "change its owner", func(m *catalog.Model) error {
+		m.Owner = owner
+		return nil
+	})
+	if err == nil {
+		s.log.Info("model owner set", "model", name, "owner", owner, "by", by)
+	}
+	return final, err
+}
+
+// SetAdapters replaces the LoRA adapters a model serves. Replicas load the
+// new set when they next start.
+func (s *Service) SetAdapters(ctx context.Context, name string, adapters []catalog.Adapter, by string) (catalog.Model, error) {
+	final, err := s.update(ctx, name, by, "change its adapters", func(m *catalog.Model) error {
+		m.Adapters = adapters
+		return nil
+	})
+	if err == nil {
+		names := make([]string, 0, len(adapters))
+		for _, a := range adapters {
+			names = append(names, a.Name)
+		}
+		s.log.Info("model adapters set", "model", name, "adapters", names, "by", by)
+	}
+	return final, err
+}
+
+// update changes one model added at runtime and keeps the change.
+//
+// Built before anything is written, like Add: catalog.New is what applies the
+// defaults and refuses what does not validate, so the model that comes back
+// is the one that will run rather than the one that was asked for.
+func (s *Service) update(ctx context.Context, name, by, what string, change func(*catalog.Model) error) (catalog.Model, error) {
 	s.mu.Lock()
 	defer s.mu.Unlock()

 	if s.Declared(name) {
 		return catalog.Model{}, fmt.Errorf(
-			"model %q is declared in configuration; change its scaling there, or it reverts on the next restart", name)
+			"model %q is declared in configuration; %s there, or it reverts on the next restart", name, what)
 	}
-
 	stored, err := s.storedModels(ctx)
 	if err != nil {
 		return catalog.Model{}, err
 	}
-
 	next := append([]catalog.Model{}, s.declared...)
 	found := false
 	for _, e := range stored {
 		if e.Name == name {
-			e.Scaling = sc
+			if err := change(&e); err != nil {
+				return catalog.Model{}, err
+			}
 			found = true
 		}
 		next = append(next, e)
@@ -226,20 +319,14 @@ func (s *Service) Scale(ctx context.Context, name string, sc catalog.Scaling, by
 	if !found {
 		return catalog.Model{}, fmt.Errorf("model %q was not added at runtime", name)
 	}
-
-	// Built before anything is written, like Add: catalog.New is what applies
-	// the defaults (max follows min when it is left out) and what refuses a
-	// max below a min, so the model that comes back is the one that will run
-	// rather than the one that was asked for.
 	built, err := catalog.New(next)
 	if err != nil {
-		return catalog.Model{}, fmt.Errorf("scaling %q that way would make the catalog invalid: %w", name, err)
+		return catalog.Model{}, fmt.Errorf("that would make the catalog invalid: %w", err)
 	}
 	final, ok := built.Resolve(name)
 	if !ok {
 		return catalog.Model{}, fmt.Errorf("model %q vanished from the catalog it was just built into", name)
 	}
-
 	def, err := json.Marshal(final)
 	if err != nil {
 		return catalog.Model{}, err
@@ -249,10 +336,7 @@ func (s *Service) Scale(ctx context.Context, name string, sc catalog.Scaling, by
 	}); err != nil {
 		return catalog.Model{}, err
 	}
-
 	s.apply.ReplaceCatalog(ctx, built)
-	s.log.Info("model scaled", "model", name,
-		"min", final.Scaling.Min, "max", final.Scaling.Max, "by", by)
 	return final, nil
 }

@@ -48,7 +48,7 @@ func model(name string) catalog.Model {
 	return catalog.Model{
 		Name: name, Source: "/models/" + name,
 		Scaling:  catalog.Scaling{Min: 1, Max: 1},
-		Profiles: []catalog.Profile{{Name: "p", Engine: "llamacpp", Weight: 1}},
+		Profiles: []catalog.Profile{{Name: "p", Engine: "llamacpp", Weight: 1, MinMemoryGB: 2}},
 	}
 }

@@ -337,3 +337,43 @@ func TestMinZeroWithAMaxIsOnDemand(t *testing.T) {
 		t.Errorf("the idle timeout was lost; got %v", got.Scaling.IdleTimeout)
 	}
 }
+
+func TestACPUModelAddedNowMustSayHowMuchMemoryItNeeds(t *testing.T) {
+	s, st, _ := newService(t)
+	m := model("bare")
+	m.Profiles[0].MinMemoryGB = 0
+	err := s.Add(t.Context(), m, "tester")
+	if err == nil || !strings.Contains(err.Error(), "declares no memory") {
+		t.Fatalf("a CPU model with no memory was added: %v", err)
+	}
+	if len(st.rows) != 0 {
+		t.Error("it was stored anyway")
+	}
+}
+
+func TestResourcesAreSetOnAStoredModelAndSurviveARestart(t *testing.T) {
+	s, _, _ := newService(t)
+	if err := s.Add(t.Context(), model("m"), "tester"); err != nil {
+		t.Fatal(err)
+	}
+	got, err := s.Resources(t.Context(), "m", "", 6, 3, "tester")
+	if err != nil {
+		t.Fatal(err)
+	}
+	if p := got.Profiles[0]; p.MinMemoryGB != 6 || p.CPUCores != 3 {
+		t.Errorf("profile %+v", p)
+	}
+	if _, err := s.Resources(t.Context(), "m", "nope", 6, 3, "tester"); err == nil {
+		t.Error("a profile the model does not have was accepted")
+	}
+	if _, err := s.Resources(t.Context(), "m", "", -1, 0, "tester"); err == nil {
+		t.Error("negative memory was accepted")
+	}
+	cat, err := s.Load(t.Context())
+	if err != nil {
+		t.Fatal(err)
+	}
+	if m, _ := cat.Resolve("m"); m.Profiles[0].CPUCores != 3 {
+		t.Error("the cores were lost on reload")
+	}
+}
@@ -1,6 +1,7 @@
 package placement

 import (
+	"strings"
 	"testing"

 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
@@ -100,3 +101,38 @@ func TestCPUProfileRespectsAvoid(t *testing.T) {
 		t.Errorf("placed on %q despite being told to avoid it", got)
 	}
 }
+
+func TestCPUCoresAreReservedAsMemoryIs(t *testing.T) {
+	n := node("n1", "", 0)
+	n.CPUCount = 8
+	st := store(n)
+	s := New(st)
+	p := cpuProfile("cpu")
+	p.CPUCores = 3
+
+	for i := range 2 {
+		res := s.Place([]Profile{p}, nil)
+		if res.Placement == nil {
+			t.Fatalf("replica %d did not fit in 8 cores:\n%s", i+1, res.Explain())
+		}
+		st.Allocate("n1", inventory.Allocation{ShardID: string(rune('a' + i)), CPUCores: 3})
+	}
+	// Six of eight given away: a third would need three.
+	res := s.Place([]Profile{p}, nil)
+	if res.Placement != nil {
+		t.Fatal("a third replica was given cores the node does not have")
+	}
+	if got := res.Explain(); !strings.Contains(got, "has 8 CPUs with 2 free, needs 3") {
+		t.Errorf("the refusal should say how many cores are left:\n%s", got)
+	}
+
+	// Unknown counts constrain nothing: an older agent, or a profile that
+	// reserves no cores.
+	old := node("n2", "", 0)
+	if res := New(store(old)).Place([]Profile{p}, nil); res.Placement == nil {
+		t.Error("a node that does not report its CPUs was excluded")
+	}
+	if res := s.Place([]Profile{cpuProfile("cpu")}, nil); res.Placement == nil {
+		t.Error("a profile reserving no cores was refused")
+	}
+}
@@ -39,6 +39,9 @@ type Profile struct {
 	// MinDiskGB is the free space needed to hold the weights, checked only on
 	// nodes that do not already have them. Zero is unconstrained.
 	MinDiskGB int
+	// CPUCores is the cores a profile with no GPU reserves. Zero is
+	// unconstrained.
+	CPUCores int
 	// Source is the model's weights, as the engine would be given it. Used to
 	// prefer nodes that already hold them; empty asks for no preference.
 	Source string
@@ -582,6 +585,18 @@ func (s *Scheduler) eligibleNodes(p Profile, fabric string, avoid []string, reje
 			}
 		}

+		// Cores, for a replica that computes on them: what is left after the
+		// cores already given to the node's other replicas. Same exemptions
+		// as memory: zero declares nothing, and a node that does not report
+		// its CPUs is an older agent, not one with none.
+		if p.CPUOnly() && p.CPUCores > 0 && n.CPUCount > 0 {
+			free := n.CPUCount - s.claimedCores(n.ID)
+			if free < p.CPUCores {
+				reject(fmt.Sprintf("has %d CPUs with %d free, needs %d", n.CPUCount, free, p.CPUCores))
+				continue
+			}
+		}
+
 		if p.CPUOnly() {
 			// Every ready node can run it; there is no card to be free.
 			out = append(out, candidate{node: n})
@@ -723,6 +738,15 @@ func (s *Scheduler) claimedMemory(nodeID string) int {
 	return total
 }

+// claimedCores is the CPU cores already given to replicas on a node.
+func (s *Scheduler) claimedCores(nodeID string) int {
+	total := 0
+	for _, a := range s.inv.Allocations(nodeID) {
+		total += a.CPUCores
+	}
+	return total
+}
+
 // installing says why an engine is missing, when the node has said.
 func installing(n inventory.Node, engine string) string {
 	for _, p := range n.Provision {
@@ -0,0 +1,192 @@
+// Package quota decides what the tenancy tree is guaranteed of the cluster
+// and how much of it each part may hold.
+//
+// A scope — an organization, team or project — may be guaranteed GPUs, CPU
+// cores and memory, and limited to at most some amount of each. What a scope
+// holds is what the replicas of the models it owns, and of the models its
+// teams and projects own, have reserved. Holding more than its guarantee is
+// borrowing: allowed while the room is free, and taken back when another
+// scope within its own guarantee needs it.
+package quota
+
+import (
+	"fmt"
+	"sort"
+	"strings"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// Amount is how much of each resource something holds or needs.
+type Amount struct {
+	GPUs, CPUCores, MemoryGB int
+}
+
+// Plus adds two amounts.
+func (a Amount) Plus(b Amount) Amount {
+	return Amount{a.GPUs + b.GPUs, a.CPUCores + b.CPUCores, a.MemoryGB + b.MemoryGB}
+}
+
+// Over names the first resource in which a is above the governed amounts of
+// l, with both figures; empty when it is above none.
+func Over(a Amount, l store.Resources) string {
+	switch {
+	case l.GPUs != nil && a.GPUs > *l.GPUs:
+		return fmt.Sprintf("%d GPUs of %d", a.GPUs, *l.GPUs)
+	case l.CPUCores != nil && a.CPUCores > *l.CPUCores:
+		return fmt.Sprintf("%d cores of %d", a.CPUCores, *l.CPUCores)
+	case l.MemoryGB != nil && a.MemoryGB > *l.MemoryGB:
+		return fmt.Sprintf("%d GB of memory of %d", a.MemoryGB, *l.MemoryGB)
+	}
+	return ""
+}
+
+// Holding is one replica's hold on the cluster, for working out who holds
+// what.
+type Holding struct {
+	Replica string
+	Owner   string
+	Amount  Amount
+	Since   time.Time
+}
+
+// Policy is the quotas and the tree they sit in, read at one moment.
+type Policy struct {
+	quotas  map[string]store.CapacityQuota
+	parents map[string][]string
+}
+
+// New builds a policy from stored quotas and each scope's parents.
+func New(quotas []store.CapacityQuota, parents map[string][]string) *Policy {
+	p := &Policy{quotas: map[string]store.CapacityQuota{}, parents: parents}
+	for _, q := range quotas {
+		p.quotas[Key(q.Scope, q.ScopeID)] = q
+	}
+	return p
+}
+
+// Key writes a scope the way owners are written: team:ID.
+func Key(s store.Scope, id string) string { return string(s) + ":" + id }
+
+// Chain is an owner and every scope above it, nearest first.
+func (p *Policy) Chain(owner string) []string {
+	if owner == "" {
+		return nil
+	}
+	return append([]string{owner}, p.parents[owner]...)
+}
+
+// Governing is the quotas along an owner's chain, nearest first.
+func (p *Policy) Governing(owner string) []store.CapacityQuota {
+	if p == nil {
+		return nil
+	}
+	var out []store.CapacityQuota
+	for _, s := range p.Chain(owner) {
+		if q, ok := p.quotas[s]; ok {
+			out = append(out, q)
+		}
+	}
+	return out
+}
+
+// Quotas lists every quota.
+func (p *Policy) Quotas() []store.CapacityQuota {
+	if p == nil {
+		return nil
+	}
+	out := make([]store.CapacityQuota, 0, len(p.quotas))
+	for _, q := range p.quotas {
+		out = append(out, q)
+	}
+	sort.Slice(out, func(i, j int) bool {
+		return Key(out[i].Scope, out[i].ScopeID) < Key(out[j].Scope, out[j].ScopeID)
+	})
+	return out
+}
+
+// Usage is what every scope holds, its own models' replicas and those of
+// everything beneath it.
+func (p *Policy) Usage(hs []Holding) map[string]Amount {
+	out := map[string]Amount{}
+	if p == nil {
+		return out
+	}
+	for _, h := range hs {
+		for _, s := range p.Chain(h.Owner) {
+			out[s] = out[s].Plus(h.Amount)
+		}
+	}
+	return out
+}
+
+// Borrowed is the replicas held beyond a guarantee: of each governed scope's
+// replicas, oldest first, the ones that take it past what it is guaranteed.
+// A replica beyond any guarantee along its chain is borrowed.
+func (p *Policy) Borrowed(hs []Holding) map[string]bool {
+	out := map[string]bool{}
+	if p == nil || len(p.quotas) == 0 {
+		return out
+	}
+	sorted := append([]Holding(nil), hs...)
+	sort.Slice(sorted, func(i, j int) bool { return sorted[i].Since.Before(sorted[j].Since) })
+	for key, q := range p.quotas {
+		if !q.Guarantee.Any() {
+			continue
+		}
+		var sum Amount
+		for _, h := range sorted {
+			if !contains(p.Chain(h.Owner), key) {
+				continue
+			}
+			sum = sum.Plus(h.Amount)
+			if Over(sum, q.Guarantee) != "" {
+				out[h.Replica] = true
+			}
+		}
+	}
+	return out
+}
+
+// Guaranteed reports whether an owner, holding what it holds, is still within
+// every guarantee along its chain with need added — so it may take back what
+// others borrowed. An owner with no guarantee anywhere is never guaranteed.
+func (p *Policy) Guaranteed(owner string, usage map[string]Amount, need Amount) bool {
+	gov := p.Governing(owner)
+	any := false
+	for _, q := range gov {
+		if !q.Guarantee.Any() {
+			continue
+		}
+		any = true
+		if Over(usage[Key(q.Scope, q.ScopeID)].Plus(need), q.Guarantee) != "" {
+			return false
+		}
+	}
+	return any
+}
+
+// AtLimit says which limit along an owner's chain need would take it past,
+// empty when none.
+func (p *Policy) AtLimit(owner string, usage map[string]Amount, need Amount) string {
+	for _, q := range p.Governing(owner) {
+		if !q.Limit.Any() {
+			continue
+		}
+		key := Key(q.Scope, q.ScopeID)
+		if over := Over(usage[key].Plus(need), q.Limit); over != "" {
+			return fmt.Sprintf("%s is at its capacity limit (%s)", key, strings.TrimSpace(over))
+		}
+	}
+	return ""
+}
+
+func contains(list []string, v string) bool {
+	for _, x := range list {
+		if x == v {
+			return true
+		}
+	}
+	return false
+}
@@ -0,0 +1,83 @@
+package quota
+
+import (
+	"strings"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+func n(v int) *int { return &v }
+
+// acme has two teams; ml is guaranteed 8 GPUs, the org as a whole 12 and
+// no more than 16.
+func policy() *Policy {
+	return New([]store.CapacityQuota{
+		{Scope: store.ScopeTeam, ScopeID: "ml", Guarantee: store.Resources{GPUs: n(8)}},
+		{Scope: store.ScopeOrg, ScopeID: "acme", Guarantee: store.Resources{GPUs: n(12)}, Limit: store.Resources{GPUs: n(16)}},
+	}, map[string][]string{
+		"team:ml": {"org:acme"}, "team:web": {"org:acme"},
+		"project:train": {"team:ml", "org:acme"},
+	})
+}
+
+func at(min int) time.Time { return time.Date(2026, 1, 1, 0, min, 0, 0, time.UTC) }
+
+func TestAScopeHoldsWhatEverythingBeneathItHolds(t *testing.T) {
+	p := policy()
+	u := p.Usage([]Holding{
+		{Replica: "a", Owner: "project:train", Amount: Amount{GPUs: 4}},
+		{Replica: "b", Owner: "team:web", Amount: Amount{GPUs: 2, CPUCores: 3}},
+		{Replica: "c", Owner: "", Amount: Amount{GPUs: 8}}, // shared: nobody's
+	})
+	if u["team:ml"].GPUs != 4 || u["org:acme"].GPUs != 6 || u["org:acme"].CPUCores != 3 || u["project:train"].GPUs != 4 {
+		t.Errorf("usage %+v", u)
+	}
+	if _, ok := u[""]; ok {
+		t.Error("a shared model was counted against somebody")
+	}
+}
+
+func TestWhatGoesPastAGuaranteeIsBorrowedNewestFirst(t *testing.T) {
+	p := policy()
+	b := p.Borrowed([]Holding{
+		{Replica: "old", Owner: "team:ml", Amount: Amount{GPUs: 8}, Since: at(1)},
+		{Replica: "new", Owner: "team:ml", Amount: Amount{GPUs: 4}, Since: at(2)},
+		{Replica: "web", Owner: "team:web", Amount: Amount{GPUs: 4}, Since: at(3)},
+	})
+	// ml: 8 within its 8, then 4 past it. acme: 8 + 4 = 12 within, web's 4
+	// takes the org to 16, past its 12.
+	if b["old"] || !b["new"] || !b["web"] {
+		t.Errorf("borrowed %v", b)
+	}
+}
+
+func TestAGuaranteeIsOnlyForWhoHasOne(t *testing.T) {
+	p := policy()
+	u := map[string]Amount{"team:ml": {GPUs: 4}, "org:acme": {GPUs: 4}}
+	if !p.Guaranteed("project:train", u, Amount{GPUs: 4}) {
+		t.Error("8 of ml's 8 and 8 of acme's 12 is within both")
+	}
+	if p.Guaranteed("project:train", u, Amount{GPUs: 5}) {
+		t.Error("9 GPUs is past ml's 8")
+	}
+	if p.Guaranteed("", u, Amount{GPUs: 1}) || New(nil, nil).Guaranteed("team:ml", u, Amount{}) {
+		t.Error("an owner with no guarantee is not guaranteed anything")
+	}
+}
+
+func TestALimitStopsGrowthWhereverItIsSet(t *testing.T) {
+	p := policy()
+	u := map[string]Amount{"org:acme": {GPUs: 14}}
+	if why := p.AtLimit("team:web", u, Amount{GPUs: 2}); why != "" {
+		t.Errorf("16 of 16 is at the limit, not past it: %s", why)
+	}
+	why := p.AtLimit("team:web", u, Amount{GPUs: 4})
+	if !strings.Contains(why, "org:acme is at its capacity limit (18 GPUs of 16)") {
+		t.Errorf("the refusal should name the scope and the figures: %q", why)
+	}
+	if p.AtLimit("", u, Amount{GPUs: 99}) != "" {
+		t.Error("a shared model was limited")
+	}
+}
@@ -0,0 +1,95 @@
+package control
+
+import (
+	"context"
+	"encoding/json"
+	"log/slog"
+	"sync"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/bench"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// QuotaStore is where capacity quotas and the tenancy tree are kept.
+type QuotaStore interface {
+	CapacityQuotas(ctx context.Context) ([]store.CapacityQuota, error)
+	ScopeParents(ctx context.Context) (map[string][]string, error)
+}
+
+// quotaRefresh is how stale the reconciler's view of quotas may get. A quota
+// changed by an operator takes effect within this; the reconciler asks many
+// times a pass and the store is not asked each time.
+const quotaRefresh = 30 * time.Second
+
+// QuotaPolicy returns what the reconciler asks for its quotas: the last
+// policy read from the store, read again once it is older than quotaRefresh.
+// A store that fails keeps the last good policy, so a database hiccup does
+// not drop every guarantee and limit at once.
+func QuotaPolicy(st QuotaStore, log *slog.Logger) func() *quota.Policy {
+	var mu sync.Mutex
+	var cur *quota.Policy
+	var at time.Time
+	return func() *quota.Policy {
+		mu.Lock()
+		defer mu.Unlock()
+		if cur != nil && time.Since(at) < quotaRefresh {
+			return cur
+		}
+		ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+		defer cancel()
+		qs, err := st.CapacityQuotas(ctx)
+		if err == nil {
+			var parents map[string][]string
+			if parents, err = st.ScopeParents(ctx); err == nil {
+				cur, at = quota.New(qs, parents), time.Now()
+				return cur
+			}
+		}
+		log.Warn("could not read capacity quotas; keeping the last ones", "err", err)
+		at = time.Now() // not again on every call while the store is down
+		return cur
+	}
+}
+
+// MeasurementStore is where benchmark results are kept.
+type MeasurementStore interface {
+	Capacities(ctx context.Context) ([]store.StoredCapacity, error)
+}
+
+// Measurements returns what the reconciler asks for benchmark results, keyed
+// model/profile, read again once older than quotaRefresh; like QuotaPolicy it
+// keeps the last good answer when the store fails.
+func Measurements(st MeasurementStore, log *slog.Logger) func() map[string]reconciler.Measured {
+	var mu sync.Mutex
+	var cur map[string]reconciler.Measured
+	var at time.Time
+	return func() map[string]reconciler.Measured {
+		mu.Lock()
+		defer mu.Unlock()
+		if cur != nil && time.Since(at) < quotaRefresh {
+			return cur
+		}
+		ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+		defer cancel()
+		at = time.Now()
+		rows, err := st.Capacities(ctx)
+		if err != nil {
+			log.Warn("could not read benchmark results; keeping the last ones", "err", err)
+			return cur
+		}
+		next := map[string]reconciler.Measured{}
+		for _, row := range rows {
+			var c bench.Capacity
+			if json.Unmarshal(row.Measurement, &c) != nil {
+				continue
+			}
+			next[row.Model+"/"+row.Profile] = reconciler.Measured{
+				MaxConcurrency: c.MaxConcurrencyAtSLO, TokPerSec: c.ThroughputTokPerSec}
+		}
+		cur = next
+		return cur
+	}
+}
@@ -0,0 +1,262 @@
+package reconciler
+
+import (
+	"slices"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/supervisor"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
+)
+
+// Adopt rebuilds replica state from shards an agent reports it is already
+// running (plan §6.5).
+//
+// This is what makes a control plane restart survivable. Without it the
+// reconciler sees an empty cluster, places duplicates, and the engines already
+// running keep holding GPUs the inventory believes are free — so the duplicates
+// fail to allocate, with an error naming the new process rather than the old.
+//
+// Shards whose model or profile is no longer in the catalog are returned for
+// the caller to stop: nothing will ever claim them.
+func (r *Reconciler) Adopt(nodeID string, shards []AdoptedShard) (unclaimed []AdoptedShard) {
+	defer r.forgetShardsTheAgentDoesNotHave(nodeID, shards)
+
+	byReplica := map[string][]AdoptedShard{}
+	for _, s := range shards {
+		byReplica[s.ReplicaID] = append(byReplica[s.ReplicaID], s)
+	}
+
+	r.mu.Lock()
+	defer r.mu.Unlock()
+
+	for replicaID, group := range byReplica {
+		if existing, known := r.replicas[replicaID]; known {
+			// The other half of a replica we are still assembling. Merge rather
+			// than choose: these shards belong to the same replica as the ones
+			// already adopted, they are simply on a different node whose agent
+			// registered a moment later. Treating them as a competing claim is
+			// what tore multi-node replicas apart on every control plane
+			// restart — one half adopted and failed for being short, the other
+			// stopped as unclaimed when it finally arrived.
+			if !existing.AdoptedAt.IsZero() && len(existing.Shards) < existing.WantShards {
+				r.mergeAdopted(existing, group)
+				continue
+			}
+			// Usually this is an agent reconnecting rather than a control plane
+			// restart, and its status reports keep the state current.
+			//
+			// Not always. Replica ids are derived from the model name and a
+			// counter, so a control plane that restarts and places before the
+			// agents reconnect produces a *different* replica under the same
+			// id, on a freshly allocated port. Skipping the adoption then
+			// leaves the cluster pointing at a port nothing listens on, while
+			// the engine that is actually serving is treated as a stray.
+			//
+			// The fleet is authoritative about what is running (plan §6.5), so
+			// what we merely placed loses to what an agent reports — unless our
+			// copy has a shard that has actually come up, in which case two
+			// things really are running and the one we did not ask for is the
+			// stray.
+			if existing.hasLiveShard() {
+				// Only what we do not recognise. A reconnecting agent reports
+				// the shards we already know, and disowning those would tell it
+				// to stop the very replica that is serving.
+				known := map[string]struct{}{}
+				for _, sh := range existing.Shards {
+					known[sh.ID] = struct{}{}
+				}
+				var strays []AdoptedShard
+				for _, s := range group {
+					if _, ok := known[s.ShardID]; !ok {
+						strays = append(strays, s)
+					}
+				}
+				unclaimed = append(unclaimed, strays...)
+				r.reservePorts(strays)
+				continue
+			}
+			r.log.Warn("a running replica reported by the fleet replaces one we placed",
+				"replica", replicaID, "placed_endpoint", existing.Endpoint)
+			for _, s := range existing.Shards {
+				r.inv.Release(s.ID)
+			}
+			delete(r.replicas, replicaID)
+		}
+
+		first := group[0]
+		m, ok := r.cat.Load().Resolve(first.Model)
+		if !ok {
+			unclaimed = append(unclaimed, group...)
+			r.reservePorts(group)
+			continue
+		}
+		prof, ok := m.ProfileByName(first.Profile)
+		if !ok {
+			unclaimed = append(unclaimed, group...)
+			r.reservePorts(group)
+			continue
+		}
+
+		rep := &Replica{
+			ID: replicaID, Model: m.Name, Profile: prof.Name,
+			Fabric: first.Fabric, Pool: first.Pool,
+			// From the agent, which was told it when the shard started. Empty
+			// from an agent too old to report it, which reads as unknown.
+			ProfileFP: first.ProfileFP,
+			Weight:    prof.Weight, CreatedAt: r.now(),
+			Endpoint: first.Endpoint,
+			// From the profile, not from what was reported: the point is to
+			// notice when what was reported is short.
+			WantShards: max(prof.Nodes, 1),
+			AdoptedAt:  r.now(),
+		}
+		r.mergeAdopted(rep, group)
+		r.replicas[replicaID] = rep
+
+	}
+	return unclaimed
+}
+
+// mergeAdopted folds shards an agent reports into a replica being reassembled.
+//
+// Called both for the first group, which creates the replica, and for every
+// later group belonging to it, so a replica spanning N nodes is built up over N
+// registrations. Shard ids already present are skipped: an agent that
+// reconnects repeats what it is running, and counting those again would carry a
+// replica past its shard count without a second node ever appearing.
+func (r *Reconciler) mergeAdopted(rep *Replica, group []AdoptedShard) {
+	have := make(map[string]struct{}, len(rep.Shards))
+	for _, sh := range rep.Shards {
+		have[sh.ID] = struct{}{}
+	}
+	for _, s := range group {
+		if _, dup := have[s.ShardID]; dup {
+			continue
+		}
+		have[s.ShardID] = struct{}{}
+		rep.Shards = append(rep.Shards, Shard{
+			ID: s.ShardID, NodeID: s.NodeID, Rank: s.Rank,
+			GPUIndices: s.GPUIndices, HCAs: s.HCAs,
+			HTTPPort: s.HTTPPort, State: s.State,
+		})
+		// Reclaim the GPUs and the memory, or the next placement hands them
+		// out twice. The memory comes from the catalog rather than the agent:
+		// what a profile asked for is not something a running process reports.
+		r.inv.Allocate(s.NodeID, inventory.Allocation{
+			ShardID: s.ShardID, ReplicaID: rep.ID, GPUIndices: s.GPUIndices,
+			MemoryGB: r.profileMemory(rep.Model, rep.Profile),
+			CPUCores: r.adoptedCores(rep.Model, rep.Profile),
+		})
+		// The endpoint comes from the leader, which may be in any of the
+		// groups — the rank 0 node is not necessarily the first to register.
+		if s.Rank == 0 && s.Endpoint != "" {
+			rep.Endpoint = s.Endpoint
+		}
+	}
+	slices.SortFunc(rep.Shards, func(a, b Shard) int { return a.Rank - b.Rank })
+	rep.deriveState(r.now())
+
+	r.log.Info("adopted replica from the fleet",
+		"replica", rep.ID, "model", rep.Model,
+		"shards", len(rep.Shards), "want", rep.WantShards, "state", rep.State)
+}
+
+// forgetShardsTheAgentDoesNotHave fails shards we believe are on a node that the
+// agent, having just told us everything it is running, did not mention.
+//
+// The mirror image of adoption, and the half that was missing. Adoption handles
+// "the agent has something we do not know about"; this handles "we think the
+// agent has something it does not". Both follow from the same rule: the fleet is
+// authoritative about what is running (plan §6.5).
+//
+// Without it, an engine that dies while its agent survives leaves the control
+// plane believing a replica is Ready forever. Observed on a real cluster: the
+// engine was killed, the agent restarted and registered with nothing running,
+// and the replica stayed Ready pointing at a dead port for as long as anyone
+// watched. Traffic survived — the breaker took the endpoint out — but the
+// cluster ran a third short with no sign of it.
+//
+// Safe against the obvious race. A registration follows a fresh connection, so
+// anything we dispatched over a previous one is not running by definition;
+// there is no window where a shard is legitimately started but unreported.
+func (r *Reconciler) forgetShardsTheAgentDoesNotHave(nodeID string, running []AdoptedShard) {
+	if nodeID == "" {
+		return
+	}
+	reported := make(map[string]struct{}, len(running))
+	for _, s := range running {
+		reported[s.ShardID] = struct{}{}
+	}
+
+	var lost []string
+	r.mu.Lock()
+	for _, rep := range r.replicas {
+		if rep.State == Stopped {
+			continue
+		}
+		changed := false
+		for i := range rep.Shards {
+			sh := &rep.Shards[i]
+			if sh.NodeID != nodeID || sh.State.Terminal() {
+				continue
+			}
+			if _, ok := reported[sh.ID]; ok {
+				continue
+			}
+			sh.State = supervisor.Failed
+			sh.Err = "the agent reports it is not running"
+			r.inv.Release(sh.ID)
+			changed = true
+			lost = append(lost, sh.ID)
+		}
+		if changed {
+			rep.deriveState(r.now())
+		}
+	}
+	r.mu.Unlock()
+
+	for _, id := range lost {
+		r.log.Warn("a shard we believed was running is not; the agent does not have it",
+			"node", nodeID, "shard", id)
+	}
+}
+
+// reservePorts records ports held by shards nothing owns.
+//
+// They are being stopped, but stopping is not instant and the engine keeps its
+// socket until it exits. Handing the port to a new shard in the meantime makes
+// that shard fail to bind, which fails its replica, which places another one on
+// the same port — a loop that does not end on its own.
+//
+// Called with the lock held.
+func (r *Reconciler) reservePorts(shards []AdoptedShard) {
+	for _, s := range shards {
+		if s.HTTPPort > 0 {
+			r.reservedPorts[s.HTTPPort] = struct{}{}
+		}
+	}
+}
+
+// AdoptedShard is one running shard an agent reported at registration.
+type AdoptedShard struct {
+	NodeID     string
+	ReplicaID  string
+	ShardID    string
+	Rank       int
+	GPUIndices []int
+	HCAs       []string
+	HTTPPort   int
+	State      supervisor.State
+	Model      string
+	Profile    string
+	Pool       string
+	Fabric     string
+	ProfileFP  string
+	// Endpoint is the leader's base URL, set on the rank 0 shard.
+	Endpoint string
+}
+
+// adoptedCores is what an adopted shard is taken to hold of its node's CPUs.
+func (r *Reconciler) adoptedCores(model, profile string) int {
+	p, _ := r.profileOf(model, profile)
+	return cpuCores(p)
+}
@@ -0,0 +1,251 @@
+package reconciler
+
+import (
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+)
+
+// RecordUsage stamps replicas with when each was last sent a request.
+//
+// Replicas absent from the map are left alone rather than zeroed: a router that
+// has just restarted knows nothing about what happened before it, and treating
+// that as "never used" would make every replica look abandoned at once —
+// exactly when a fleet is least able to afford acting on that.
+func (r *Reconciler) RecordUsage(lastUsed map[string]time.Time) {
+	if len(lastUsed) == 0 {
+		return
+	}
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	for id, at := range lastUsed {
+		rep, ok := r.replicas[id]
+		if !ok {
+			continue
+		}
+		// Only forwards. Two routers reporting, or a stale snapshot arriving
+		// late, must not move a replica's last use backwards.
+		if at.After(rep.LastUsedAt) {
+			rep.LastUsedAt = at
+		}
+	}
+	r.noteUsage(lastUsed)
+}
+
+// IdleFor reports how long a replica has gone without a request, and whether
+// that is known at all.
+//
+// The second return is not a detail. "Idle for an hour" and "nobody has told me
+// yet" are different states, and anything that acts on idleness — eviction,
+// most obviously — has to refuse to act on the second.
+func (r *Reconciler) IdleFor(id string) (time.Duration, bool) {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	rep, ok := r.replicas[id]
+	if !ok || rep.LastUsedAt.IsZero() {
+		return 0, false
+	}
+	return r.now().Sub(rep.LastUsedAt), true
+}
+
+// Want marks an on-demand model as asked for, so the next pass places it.
+//
+// Called by whatever fields the request — the router, in practice. It returns
+// whether the model is one this applies to, so a caller can tell "being loaded"
+// from "there is genuinely nothing serving this", which are different answers
+// to give a client.
+func (r *Reconciler) Want(model string) bool {
+	m, ok := r.cat.Load().Resolve(model)
+	if !ok || !m.Scaling.OnDemand() {
+		return false
+	}
+	r.mu.Lock()
+	r.wanted[m.Name] = r.now()
+	// A placement that failed before this request says nothing about now:
+	// forgotten, so the pass this triggers decides afresh and Unservable
+	// does not answer from a stale result.
+	if res, ok := r.lastExplain[m.Name]; ok && res.Placement == nil {
+		delete(r.lastExplain, m.Name)
+	}
+	r.mu.Unlock()
+	// Immediately, not on the next tick. Someone is waiting on this request,
+	// and five seconds of ticker is five seconds they spend staring at nothing.
+	r.Notify()
+	return true
+}
+
+// desiredFor is how many replicas a model should have right now.
+//
+// For an ordinary model that is its configured minimum. For an on-demand one it
+// is one while somebody is asking and none once they stop, which is the whole
+// feature: a cluster with more models than GPUs can hold every model in its
+// catalog and only the wanted ones in its cards.
+func (r *Reconciler) desiredFor(m catalog.Model) int {
+	base := r.baseFor(m)
+	if base == 0 {
+		return 0
+	}
+	r.mu.Lock()
+	extra := r.extra[m.Name]
+	r.mu.Unlock()
+	return min(base+extra, max(m.Scaling.Max, base))
+}
+
+// baseFor is what a model runs without load: its minimum, or for an on-demand
+// model one while somebody is asking and none once they stop.
+func (r *Reconciler) baseFor(m catalog.Model) int {
+	if m.IsExternal() {
+		return 0 // served elsewhere: nothing to place, whatever its scaling says
+	}
+	if !m.Scaling.OnDemand() {
+		return m.Scaling.Min
+	}
+	r.mu.Lock()
+	last, asked := r.wanted[m.Name]
+	r.mu.Unlock()
+	if !asked {
+		return 0
+	}
+	if r.now().Sub(last) > r.idleTimeout(m) {
+		return 0
+	}
+	return 1
+}
+
+// idleTimeout is how long an on-demand model stays up unasked-for.
+func (r *Reconciler) idleTimeout(m catalog.Model) time.Duration {
+	if m.Scaling.IdleTimeout > 0 {
+		return m.Scaling.IdleTimeout
+	}
+	return r.cfg.IdleTimeout
+}
+
+// noteUsage keeps an on-demand model alive while it is being used.
+//
+// Without this a model would be evicted exactly IdleTimeout after the request
+// that loaded it, however busy it had been since — the timer would measure
+// time since the first request rather than the last.
+func (r *Reconciler) noteUsage(lastUsed map[string]time.Time) {
+	byModel := map[string]time.Time{}
+	for id, at := range lastUsed {
+		rep, ok := r.replicas[id]
+		if !ok {
+			continue
+		}
+		if at.After(byModel[rep.Model]) {
+			byModel[rep.Model] = at
+		}
+	}
+	for model, at := range byModel {
+		if cur, ok := r.wanted[model]; ok && at.After(cur) {
+			r.wanted[model] = at
+		}
+	}
+}
+
+// Load is what a model's replicas are doing, as the router sees it.
+type Load struct {
+	Replicas int
+	Running  int
+	Waiting  int
+}
+
+// RecordLoad adds or takes away replicas between a model's min and max as its
+// load says.
+//
+// Max was a ceiling nothing reached: every model ran exactly its minimum
+// however long its queue. Now requests waiting at the engines — every slot of
+// every replica busy — held for ScaleUpAfter add one replica, and a quiet spell
+// of ScaleDownAfter takes one of those back. One step at a time, and not while
+// the last one is still starting: on a small CPU fleet each replica is minutes
+// of weights and gigabytes of memory, and a spike that is over by the time it
+// loads should not leave three behind.
+//
+// With a benchmark stored for the profile a model runs on, its measured
+// capacity is used too: more requests per replica than were measured to meet
+// the SLO is busy, even with the engines' queues empty — an engine will take
+// requests into its slots long after latency has gone — and a replica is only
+// taken back when the rest would still be under scaleDownHeadroom of what they
+// were measured to hold, so scaling down does not land on the edge.
+func (r *Reconciler) RecordLoad(loads map[string]Load) {
+	now := r.now()
+	// The profile each model's replicas run on, for its measurement; read
+	// before the loop, which holds the lock.
+	running := map[string]string{}
+	r.mu.Lock()
+	for _, rep := range r.replicas {
+		if rep.State.Counts() {
+			running[rep.Model] = rep.Profile
+		}
+	}
+	r.mu.Unlock()
+	measured := map[string]Measured{}
+	for model, profile := range running {
+		if m, ok := r.measured(model, profile); ok {
+			measured[model] = m
+		}
+	}
+	for _, m := range r.cat.Load().Models() {
+		base := r.baseFor(m)
+		r.mu.Lock()
+		if base == 0 || m.Scaling.Max <= base {
+			delete(r.extra, m.Name)
+			delete(r.busySince, m.Name)
+			delete(r.quietSince, m.Name)
+			r.mu.Unlock()
+			continue
+		}
+		starting := false
+		for _, rep := range r.replicas {
+			if rep.Model == m.Name && (rep.State == Placing || rep.State == Starting) {
+				starting = true
+			}
+		}
+		l := loads[m.Name]
+		perReplica := float64(l.Waiting) / float64(max(l.Replicas, 1))
+		extra := r.extra[m.Name]
+		meas, hasMeas := measured[m.Name]
+		overMeasured := hasMeas &&
+			float64(l.Running+l.Waiting) > float64(meas.MaxConcurrency*max(l.Replicas, 1))
+		fitsInFewer := !hasMeas ||
+			float64(l.Running) <= scaleDownHeadroom*float64(meas.MaxConcurrency*max(l.Replicas-1, 0))
+		switch {
+		case perReplica >= r.cfg.ScaleUpWaiting || overMeasured:
+			delete(r.quietSince, m.Name)
+			since, ok := r.busySince[m.Name]
+			if !ok {
+				r.busySince[m.Name] = now
+				break
+			}
+			if now.Sub(since) >= r.cfg.ScaleUpAfter && base+extra < m.Scaling.Max && !starting {
+				r.extra[m.Name] = extra + 1
+				r.busySince[m.Name] = now
+				r.log.Info("scaling up for load", "model", m.Name, "replicas", base+extra+1,
+					"waiting_per_replica", perReplica, "in_flight", l.Running+l.Waiting,
+					"measured_per_replica", meas.MaxConcurrency, "for", now.Sub(since).Round(time.Second))
+			}
+		case l.Waiting == 0 && extra > 0 && fitsInFewer:
+			delete(r.busySince, m.Name)
+			since, ok := r.quietSince[m.Name]
+			if !ok {
+				r.quietSince[m.Name] = now
+				break
+			}
+			if now.Sub(since) >= r.cfg.ScaleDownAfter {
+				r.extra[m.Name] = extra - 1
+				r.quietSince[m.Name] = now
+				r.log.Info("scaling down after a quiet spell", "model", m.Name,
+					"replicas", base+extra-1, "quiet_for", now.Sub(since).Round(time.Second))
+			}
+		default:
+			delete(r.busySince, m.Name)
+			delete(r.quietSince, m.Name)
+		}
+		r.mu.Unlock()
+	}
+}
+
+// scaleDownHeadroom is how full, of their measured capacity, the replicas left
+// after a scale-down may be. Below one so that the load that was there a
+// minute ago does not put them straight back over.
+const scaleDownHeadroom = 0.7
@@ -0,0 +1,148 @@
+package reconciler
+
+import (
+	"slices"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/placement"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
+)
+
+// Headroom is how many more replicas of a model the cluster has room for now.
+type Headroom struct {
+	// Current is the replicas the model has that hold capacity: starting or
+	// ready, not those on their way out.
+	Current int
+	// More is how many more could be placed now, at most the limit asked.
+	More int
+	// Nodes is where each of those would go, in order, a replica's nodes
+	// joined when it spans several.
+	Nodes [][]string
+	// Limited is set when More reached the limit asked, so there may be room
+	// for more than that. A CPU profile that declares no memory is always
+	// limited: nothing it asks for runs out.
+	Limited bool
+	// Blocked is why one more than More does not fit, a line per node that
+	// was considered. Empty when Limited.
+	Blocked []string
+	// TokPerSec is what one more replica was measured to serve, on the
+	// profile placement would give it; zero when it was never measured.
+	TokPerSec float64
+	// ByPreempting is how many more, beyond More, would fit by stopping
+	// replicas the model's class may take, and Preempts the models those
+	// replicas belong to: what "room" means for a model that outranks others.
+	ByPreempting int
+	Preempts     []string
+}
+
+// Headroom tries placing up to limit more replicas of a model on a copy of
+// the inventory, each one's allocation made on the copy before the next is
+// tried, exactly as the reconciler would place them one pass at a time.
+//
+// The scheduler's own decisions, not an estimate beside them: what the page
+// offers is then what the next passes will do, and a count that the
+// arithmetic of a second implementation allowed but placement refuses would
+// leave a model short with nobody told why.
+func (r *Reconciler) Headroom(model string, limit int) (Headroom, bool) {
+	m, ok := r.cat.Load().Resolve(model)
+	if !ok {
+		return Headroom{}, false
+	}
+	var h Headroom
+	var avoid []string
+	r.mu.Lock()
+	for _, rep := range r.replicas {
+		if rep.Model == m.Name && rep.State.Counts() && rep.RetiringAt.IsZero() {
+			h.Current++
+			avoid = append(avoid, rep.NodeIDs()...)
+		}
+	}
+	r.mu.Unlock()
+
+	inv := r.inv.Clone()
+	sched := placement.New(inv)
+	pol := r.policy()
+	var usage map[string]quota.Amount
+	if pol != nil {
+		usage = pol.Usage(r.Holdings())
+	}
+	place := func() placement.Result {
+		res := sched.Place(r.orderedProfiles(m), avoid)
+		if res.Placement == nil && len(avoid) > 0 {
+			// As scaleUp does: spreading is a preference, not a rule.
+			res = sched.Place(r.orderedProfiles(m), nil)
+		}
+		return res
+	}
+	// What the model's class may take, in the order preemption would, each
+	// released on the copy only when free room has run out.
+	victims, nextVictim, preempting := r.victims(m), 0, false
+	for h.More+h.ByPreempting < limit {
+		res := place()
+		// Named only once they have made room: candidates tried on the way
+		// to finding there is none would otherwise read as whose replicas a
+		// placement would stop.
+		var tried []string
+		for res.Placement == nil && nextVictim < len(victims) {
+			v := victims[nextVictim]
+			nextVictim++
+			inv.ReleaseReplica(v.ID)
+			tried = append(tried, v.Model)
+			res = place()
+		}
+		if res.Placement != nil && len(tried) > 0 {
+			preempting = true
+			for _, m := range tried {
+				if !slices.Contains(h.Preempts, m) {
+					h.Preempts = append(h.Preempts, m)
+				}
+			}
+		}
+		if res.Placement == nil {
+			for _, rej := range res.Rejections {
+				h.Blocked = append(h.Blocked, rej.String())
+			}
+			slices.Sort(h.Blocked)
+			h.Blocked = slices.Compact(h.Blocked)
+			return h, true
+		}
+		// A quota limit is as much an edge as a full node.
+		if pol != nil && m.Owner != "" {
+			need := placementNeed(m, res.Placement)
+			if why := pol.AtLimit(m.Owner, usage, need); why != "" {
+				h.Blocked = []string{why}
+				return h, true
+			}
+			for _, s := range pol.Chain(m.Owner) {
+				usage[s] = usage[s].Plus(need)
+			}
+		}
+		mem, cores := 0, 0
+		if prof, ok := m.ProfileByName(res.Placement.Profile.Name); ok {
+			mem, cores = prof.MinMemoryGB, cpuCores(prof)
+		}
+		if h.More == 0 {
+			if meas, ok := r.measured(m.Name, res.Placement.Profile.Name); ok {
+				h.TokPerSec = meas.TokPerSec
+			}
+		}
+		var nodes []string
+		for _, a := range res.Placement.Nodes {
+			inv.Allocate(a.NodeID, inventory.Allocation{
+				ShardID: "headroom", ReplicaID: "headroom", GPUIndices: a.GPUIndices, MemoryGB: mem, CPUCores: cores,
+			})
+			nodes = append(nodes, a.NodeID)
+		}
+		avoid = append(avoid, nodes...)
+		// Once anything was stopped, what fits after is counted as fitting
+		// by preempting: it may be sitting in the room that was freed.
+		if preempting {
+			h.ByPreempting++
+		} else {
+			h.Nodes = append(h.Nodes, nodes)
+			h.More++
+		}
+	}
+	h.Limited = true
+	return h, true
+}
@@ -0,0 +1,80 @@
+package reconciler
+
+import (
+	"strings"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
+)
+
+func TestHeadroomIsWhatPlacementWouldDoNext(t *testing.T) {
+	// Two 8-GPU nodes and a 4-GPU profile: four replicas in all, one placed.
+	h := newHarness(t, testModel(1), gpuNode("1", 8), gpuNode("2", 8))
+	h.r.Reconcile(t.Context())
+	h.markReady()
+
+	got, ok := h.r.Headroom("llama", 10)
+	if !ok {
+		t.Fatal("the model was not found")
+	}
+	if got.Current != 1 || got.More != 3 || got.Limited {
+		t.Fatalf("headroom %+v, expected 1 placed and room for 3 more", got)
+	}
+	// Spread first: the next one goes to the node without one.
+	if first := got.Nodes[0][0]; first == h.r.Replicas()[0].NodeIDs()[0] {
+		t.Errorf("the next replica would stack on %s beside the first", first)
+	}
+	if len(got.Blocked) == 0 || !strings.Contains(strings.Join(got.Blocked, "\n"), "GPU") {
+		t.Errorf("no reason given for the fifth: %v", got.Blocked)
+	}
+	// Tried on a copy: nothing was taken from the real inventory.
+	if free := len(h.inv.FreeGPUs("1")) + len(h.inv.FreeGPUs("2")); free != 12 {
+		t.Errorf("%d GPUs free after asking, expected the 12 the one replica leaves", free)
+	}
+
+	capped, _ := h.r.Headroom("llama", 2)
+	if capped.More != 2 || !capped.Limited || capped.Blocked != nil {
+		t.Errorf("asked for at most 2: %+v", capped)
+	}
+	if _, ok := h.r.Headroom("nope", 5); ok {
+		t.Error("an unknown model has headroom")
+	}
+}
+
+func TestHeadroomCountsMemoryAlreadyClaimed(t *testing.T) {
+	cpu := func(id string) inventory.Node {
+		return inventory.Node{ID: id, MgmtAddr: "10.0.0." + id, State: inventory.Ready, MemoryTotalMB: 16 * 1024}
+	}
+	m := catalog.Model{Name: "small", Source: "/m.gguf", Scaling: catalog.Scaling{Min: 1, Max: 1},
+		Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 6}}}
+	h := newHarness(t, m, cpu("a"), cpu("b"))
+	h.r.Reconcile(t.Context())
+
+	// 16 GB each at 6 GB a replica: two per node, one of them taken.
+	got, _ := h.r.Headroom("small", 10)
+	if got.Current != 1 || got.More != 3 {
+		t.Errorf("headroom %+v, expected room for 3 more", got)
+	}
+}
+
+func TestReservedCoresAreTheEnginesThreadsAndBoundTheRoom(t *testing.T) {
+	cpu := inventory.Node{ID: "a", MgmtAddr: "10.0.0.1", State: inventory.Ready, MemoryTotalMB: 64 * 1024, CPUCount: 8}
+	m := catalog.Model{Name: "coder", Source: "/m.gguf", Scaling: catalog.Scaling{Min: 1, Max: 1},
+		Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 6, CPUCores: 3}}}
+	h := newHarness(t, m, cpu)
+	h.r.Reconcile(t.Context())
+
+	started := h.disp.startedShards()
+	if len(started) != 1 || started[0].Spec.Threads != 3 {
+		t.Fatalf("the engine was not told its 3 cores: %+v", started)
+	}
+	if a := h.inv.Allocations("a"); len(a) != 1 || a[0].CPUCores != 3 {
+		t.Errorf("the cores were not reserved: %+v", a)
+	}
+	// Memory would allow nine more; cores allow one: 8 - 3 = 5, then 2.
+	got, _ := h.r.Headroom("coder", 10)
+	if got.More != 1 {
+		t.Errorf("room for %d more, expected 1 by cores: %+v", got.More, got)
+	}
+}
@@ -0,0 +1,160 @@
+package reconciler
+
+import (
+	"crypto/sha256"
+	"encoding/hex"
+	"encoding/json"
+	"fmt"
+	"strings"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+)
+
+// parkRecord is why a model stopped being retried.
+type parkRecord struct {
+	Since    time.Time
+	Reason   string
+	Failures int
+	// fp is the model's definition when it was parked; a different one
+	// unparks it.
+	fp string
+}
+
+type placementNote struct {
+	text string
+	at   time.Time
+}
+
+// modelFingerprint identifies a model's definition, so a change to it can be
+// told from a retry of the same thing.
+func modelFingerprint(m catalog.Model) string {
+	b, _ := json.Marshal(m)
+	sum := sha256.Sum256(b)
+	return hex.EncodeToString(sum[:8])
+}
+
+// placementLogEvery is how often an unchanged placement failure is repeated.
+const placementLogEvery = 10 * time.Minute
+
+// notePlacementFailure logs why a model could not be placed, when the reason
+// is new or has not been said for a while. Said on every pass it was 250
+// identical lines in two days for one model, burying everything else.
+func (r *Reconciler) notePlacementFailure(model, why string) {
+	r.mu.Lock()
+	last := r.placementLogged[model]
+	say := last.text != why || r.now().Sub(last.at) >= placementLogEvery
+	if say {
+		r.placementLogged[model] = placementNote{text: why, at: r.now()}
+	}
+	r.mu.Unlock()
+	if say {
+		r.log.Warn("no placement for model", "model", model, "reasons", why)
+	}
+}
+
+// Retry clears a parked model and its backoff, so it is placed on the next
+// pass.
+func (r *Reconciler) Retry(model string) bool {
+	r.mu.Lock()
+	_, was := r.parked[model]
+	delete(r.parked, model)
+	r.attempts[model] = 0
+	delete(r.nextAttempt, model)
+	r.mu.Unlock()
+	r.Notify()
+	return was
+}
+
+// ModelHealth is how one model stands, for status and metrics.
+type ModelHealth struct {
+	Model string
+	// Desired is how many replicas it should have now; 0 for a stopped model
+	// or an on-demand one nobody is asking for.
+	Desired int
+	Ready   int
+	// Failures counts replicas that failed in a row.
+	Failures int
+	// Parked is set when Failures reached ParkAfter; Reason is the last error.
+	Parked      bool
+	ParkedSince time.Time
+	Reason      string
+	// Unplaceable is set when it is short of replicas and the last attempt to
+	// place one found nowhere; Placement says why.
+	Unplaceable bool
+	Placement   string
+	// Class is the capacity class it is treated as. Preempted counts the
+	// replicas it has lost to models of a higher class since the control
+	// plane started; HeldFor names the model it is giving room to now.
+	Class     string
+	Preempted int
+	HeldFor   string
+}
+
+// ModelHealth reports every model in the catalog.
+func (r *Reconciler) ModelHealth() []ModelHealth {
+	models := r.cat.Load().Models()
+	desired := map[string]int{}
+	for _, m := range models {
+		desired[m.Name] = r.desiredFor(m)
+	}
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	ready, live := map[string]int{}, map[string]int{}
+	for _, rep := range r.replicas {
+		if rep.State == Ready {
+			ready[rep.Model]++
+		}
+		if rep.State.Counts() {
+			live[rep.Model]++
+		}
+	}
+	out := make([]ModelHealth, 0, len(models))
+	for _, m := range models {
+		h := ModelHealth{Model: m.Name, Desired: desired[m.Name], Ready: ready[m.Name],
+			Failures: r.attempts[m.Name], Class: m.EffectiveClass(), Preempted: r.preempted[m.Name]}
+		if hd, ok := r.held[m.Name]; ok && r.now().Before(hd.Until) {
+			h.HeldFor = hd.For
+		}
+		if p, ok := r.parked[m.Name]; ok {
+			h.Parked, h.ParkedSince, h.Reason = true, p.Since, p.Reason
+		}
+		if res, ok := r.lastExplain[m.Name]; ok && res.Placement == nil && live[m.Name] < h.Desired {
+			h.Unplaceable, h.Placement = true, res.Explain()
+		}
+		out = append(out, h)
+	}
+	return out
+}
+
+// Unservable says why a model cannot be served at all, so a request for it is
+// told now rather than asked to retry something that will never come up.
+func (r *Reconciler) Unservable(model string) (string, bool) {
+	m, ok := r.cat.Load().Resolve(model)
+	if !ok {
+		return "", false
+	}
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	if p, ok := r.parked[m.Name]; ok {
+		return fmt.Sprintf("%s failed to start %d times in a row and is not being retried: %s",
+			m.Name, p.Failures, p.Reason), true
+	}
+	for _, rep := range r.replicas {
+		if rep.Model == m.Name && rep.State.Counts() {
+			return "", false // something is on its way
+		}
+	}
+	if res, ok := r.lastExplain[m.Name]; ok && res.Placement == nil {
+		return m.Name + " cannot be placed on any node: " + firstLine(res.Explain()), true
+	}
+	return "", false
+}
+
+func firstLine(s string) string {
+	s = strings.TrimSpace(s)
+	if i := strings.IndexByte(s, '\n'); i >= 0 {
+		return strings.TrimSpace(s[:i]) + " (and others; see llmgw ctl explain-placement)"
+	}
+	return s
+}
@@ -0,0 +1,108 @@
+package reconciler
+
+import (
+	"sort"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/placement"
+)
+
+// Measured is what one replica of a model on a profile was measured to serve
+// while meeting its SLO, by `llmgw ctl bench --save`.
+type Measured struct {
+	// MaxConcurrency is the most requests at once that still met the SLO.
+	MaxConcurrency int
+	// TokPerSec is the throughput at that concurrency.
+	TokPerSec float64
+}
+
+// SetMeasurements supplies benchmark results, keyed model/profile, read
+// afresh each time it is called. Without them scaling goes by queue alone and
+// profiles are chosen by priority alone.
+func (r *Reconciler) SetMeasurements(f func() map[string]Measured) {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	r.measurements = f
+}
+
+// measured is a model's measurement on one profile.
+func (r *Reconciler) measured(model, profile string) (Measured, bool) {
+	r.mu.Lock()
+	f := r.measurements
+	r.mu.Unlock()
+	if f == nil {
+		return Measured{}, false
+	}
+	m, ok := f()[model+"/"+profile]
+	return m, ok && m.MaxConcurrency > 0
+}
+
+// orderedProfiles is a model's profiles as placement should try them: by
+// priority, and among equal priorities the one measured to serve the most per
+// GPU — or per core, for a profile with none — first. Unmeasured profiles keep
+// their catalog order behind measured ones of the same priority.
+//
+// Placement sorts by priority itself, so the order found here is written back
+// as priorities, descending, for it to keep.
+func (r *Reconciler) orderedProfiles(m catalog.Model) []placement.Profile {
+	ps := m.PlacementProfiles()
+	eff := make([]float64, len(ps))
+	for i, p := range ps {
+		meas, ok := r.measured(m.Name, p.Name)
+		if !ok {
+			continue
+		}
+		units := float64(max(p.GPUsPerNode, 0) * max(p.Nodes, 1))
+		if units == 0 {
+			units = float64(max(p.CPUCores, 1))
+		}
+		eff[i] = meas.TokPerSec / units
+	}
+	idx := make([]int, len(ps))
+	for i := range idx {
+		idx[i] = i
+	}
+	sort.SliceStable(idx, func(a, b int) bool {
+		pa, pb := ps[idx[a]], ps[idx[b]]
+		if pa.Priority != pb.Priority {
+			return pa.Priority > pb.Priority
+		}
+		return eff[idx[a]] > eff[idx[b]]
+	})
+	out := make([]placement.Profile, len(ps))
+	for rank, i := range idx {
+		out[rank] = ps[i]
+		out[rank].Priority = len(ps) - rank
+	}
+	return out
+}
+
+// MeasuredFor is the measurement that sizes a model: that of the profile its
+// replicas run on, or, with none running, of the first profile placement
+// would try that was measured.
+func (r *Reconciler) MeasuredFor(model string) (Measured, string, bool) {
+	r.mu.Lock()
+	profile := ""
+	for _, rep := range r.replicas {
+		if rep.Model == model && rep.State.Counts() {
+			profile = rep.Profile
+			break
+		}
+	}
+	r.mu.Unlock()
+	if profile != "" {
+		if m, ok := r.measured(model, profile); ok {
+			return m, profile, true
+		}
+	}
+	cm, ok := r.cat.Load().Resolve(model)
+	if !ok {
+		return Measured{}, "", false
+	}
+	for _, p := range r.orderedProfiles(cm) {
+		if m, ok := r.measured(cm.Name, p.Name); ok {
+			return m, p.Name, true
+		}
+	}
+	return Measured{}, "", false
+}
@@ -0,0 +1,96 @@
+package reconciler
+
+import (
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+)
+
+func measure(h *harness, byKey map[string]Measured) {
+	h.r.SetMeasurements(func() map[string]Measured { return byKey })
+}
+
+func TestMoreRequestsThanWereMeasuredToMeetTheSLOIsBusy(t *testing.T) {
+	m := testModel(1)
+	m.Scaling.Max = 2
+	h := newHarness(t, m, gpuNode("1", 8), gpuNode("2", 8))
+	h.r.Reconcile(t.Context())
+	h.markReady()
+	// Six in flight, nothing waiting at the engine: by the queue it is fine.
+	load := map[string]Load{"llama": {Replicas: 1, Running: 6}}
+	hold := func() {
+		h.r.RecordLoad(load)
+		h.advance(61 * time.Second)
+		h.r.RecordLoad(load)
+		h.r.Reconcile(t.Context())
+	}
+	hold()
+	if n := liveCount(h); n != 1 {
+		t.Fatalf("scaled up with no measurement and an empty queue: %d", n)
+	}
+	// Measured to meet its SLO at four at once: six is past it.
+	measure(h, map[string]Measured{"llama/tp4": {MaxConcurrency: 4, TokPerSec: 200}})
+	hold()
+	if n := liveCount(h); n != 2 {
+		t.Errorf("%d replicas with 6 in flight against a measured 4", n)
+	}
+}
+
+func TestAReplicaIsTakenBackOnlyIfTheRestWouldHaveRoomToSpare(t *testing.T) {
+	m := testModel(1)
+	m.Scaling.Max = 2
+	h := newHarness(t, m, gpuNode("1", 8), gpuNode("2", 8))
+	measure(h, map[string]Measured{"llama/tp4": {MaxConcurrency: 4, TokPerSec: 200}})
+	h.r.Reconcile(t.Context())
+	h.markReady()
+	h.r.mu.Lock()
+	h.r.extra["llama"] = 1
+	h.r.mu.Unlock()
+	h.r.Reconcile(t.Context())
+	h.markReady()
+	if liveCount(h) != 2 {
+		t.Fatal("setup: not at two replicas")
+	}
+	quietFor := func(l Load) {
+		for range 3 {
+			h.r.RecordLoad(map[string]Load{"llama": l})
+			h.advance(DefaultConfig().ScaleDownAfter + time.Second)
+			h.r.RecordLoad(map[string]Load{"llama": l})
+			h.r.Reconcile(t.Context())
+		}
+	}
+	// Nothing waiting, but three running: one replica left would be at 3 of
+	// its 4, past 70%.
+	quietFor(Load{Replicas: 2, Running: 3})
+	if n := liveCount(h); n != 2 {
+		t.Errorf("scaled down to the edge of what one replica holds: %d", n)
+	}
+	quietFor(Load{Replicas: 2, Running: 2})
+	if n := liveCount(h); n != 1 {
+		t.Errorf("two running fits one replica's 70%% of 4; still %d", n)
+	}
+}
+
+func TestTheProfileMeasuredToServeMostPerGPUIsTriedFirst(t *testing.T) {
+	m := catalog.Model{Name: "llama", Source: "/models/llama", Scaling: catalog.Scaling{Min: 1, Max: 1},
+		Profiles: []catalog.Profile{
+			{Name: "tp4", GPUClass: "h100-80g", TP: 4},
+			{Name: "tp2", GPUClass: "h100-80g", TP: 2},
+			{Name: "tp8", GPUClass: "h100-80g", TP: 8, Priority: 10},
+		}}
+	h := newHarness(t, m, gpuNode("1", 8))
+	// tp4: 400/4 = 100 per GPU; tp2: 300/2 = 150. tp8 is unmeasured but has
+	// the higher priority, which an operator set and a benchmark does not
+	// overrule.
+	measure(h, map[string]Measured{"llama/tp4": {MaxConcurrency: 8, TokPerSec: 400},
+		"llama/tp2": {MaxConcurrency: 6, TokPerSec: 300}})
+	cm, _ := h.r.cat.Load().Resolve("llama")
+	var order []string
+	for _, p := range h.r.orderedProfiles(cm) {
+		order = append(order, p.Name)
+	}
+	if len(order) != 3 || order[0] != "tp8" || order[1] != "tp2" || order[2] != "tp4" {
+		t.Errorf("order %v, expected tp8 (priority), then tp2 (150/GPU), then tp4 (100/GPU)", order)
+	}
+}
@@ -0,0 +1,324 @@
+package reconciler
+
+import (
+	"context"
+	"slices"
+	"sort"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/placement"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
+)
+
+// hold keeps a model that gave up room from taking it back.
+type hold struct {
+	Until time.Time
+	For   string
+}
+
+// maxVictims bounds how many replicas one placement may stop. Past this the
+// model is better reported as not fitting than paid for with half a cluster.
+const maxVictims = 3
+
+// victim is a replica that could be stopped for another model.
+type victim struct {
+	ID       string
+	Model    string
+	Class    string
+	AboveMin bool
+	Tier     int
+	Idle     time.Duration
+	// Size is what stopping it frees, for choosing the smaller of two
+	// otherwise equal victims.
+	Size quota.Amount
+}
+
+// preemptFor stops replicas of lower classes when a model cannot otherwise be
+// placed, and says whether it did.
+//
+// What may be taken from whom:
+//
+//   - A lower class gives way to a higher one: opportunistic to standard,
+//     standard to critical.
+//   - Within a class, only a replica above its model's minimum, and only for
+//     a model still below its own. Room a model grew into under load is not a
+//     promise; its minimum is.
+//   - A critical model's minimum is never taken.
+//
+// Victims only ever come from a lower rank, or from above a minimum for a
+// model below one, so two models cannot take turns stopping each other.
+//
+// The set is tried on a copy of the inventory first, and only stopped if it
+// would let the model be placed: stopping replicas that do not make room is
+// the one outcome worse than not preempting.
+func (r *Reconciler) preemptFor(ctx context.Context, m catalog.Model, avoid []string) bool {
+	cands := r.victims(m)
+	if len(cands) == 0 {
+		return false
+	}
+	// Candidates in order until the model fits; then every one whose
+	// stopping turns out not to be needed is kept after all. Without that
+	// second step an idle replica on the wrong node, tried first, would be
+	// stopped along with the one that actually made the room.
+	var chosen []victim
+	for _, v := range cands {
+		if len(chosen) == maxVictims {
+			break
+		}
+		chosen = append(chosen, v)
+		if !r.fitsWithout(m, avoid, chosen) {
+			continue
+		}
+		for i := len(chosen) - 1; i >= 0 && len(chosen) > 1; i-- {
+			fewer := append(append([]victim(nil), chosen[:i]...), chosen[i+1:]...)
+			if r.fitsWithout(m, avoid, fewer) {
+				chosen = fewer
+			}
+		}
+		r.preempt(ctx, m, chosen)
+		return true
+	}
+	return false
+}
+
+// fitsWithout reports whether m could be placed with these replicas gone,
+// tried on a copy of the inventory.
+func (r *Reconciler) fitsWithout(m catalog.Model, avoid []string, gone []victim) bool {
+	trial := r.inv.Clone()
+	for _, v := range gone {
+		trial.ReleaseReplica(v.ID)
+	}
+	sched := placement.New(trial)
+	res := sched.Place(r.orderedProfiles(m), avoid)
+	if res.Placement == nil && len(avoid) > 0 {
+		res = sched.Place(r.orderedProfiles(m), nil)
+	}
+	return res.Placement != nil
+}
+
+// preempt drains the chosen replicas and holds their models back.
+func (r *Reconciler) preempt(ctx context.Context, m catalog.Model, chosen []victim) {
+	now := r.now()
+	var ids []string
+	r.mu.Lock()
+	for _, v := range chosen {
+		if rep, ok := r.replicas[v.ID]; ok {
+			rep.PreemptedFor = m.Name
+		}
+		ids = append(ids, v.ID)
+		r.held[v.Model] = hold{Until: now.Add(r.cfg.PreemptHold), For: m.Name}
+		r.preempted[v.Model]++
+	}
+	r.preempting[m.Name] = ids
+	r.mu.Unlock()
+
+	for _, v := range chosen {
+		r.log.Warn("preempting a replica to make room",
+			"replica", v.ID, "model", v.Model, "class", v.Class, "above_min", v.AboveMin,
+			"for", m.Name, "for_class", m.EffectiveClass())
+	}
+	r.scaleDown(ctx, ids)
+}
+
+// awaitingPreemption reports whether replicas stopped for a model are still
+// on their way out, so it waits for them rather than stopping more.
+func (r *Reconciler) awaitingPreemption(model string) bool {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	ids := r.preempting[model]
+	ids = slices.DeleteFunc(slices.Clone(ids), func(id string) bool {
+		_, ok := r.replicas[id]
+		return !ok
+	})
+	if len(ids) == 0 {
+		delete(r.preempting, model)
+		return false
+	}
+	r.preempting[model] = ids
+	return true
+}
+
+// isHeld reports whether a model gave up room and may not take it back yet.
+// The hold ends early once the model it was given to has what it asked for.
+//
+// Today a preempted replica is only removed by its own model's drain loop,
+// which runs after that model has decided not to scale up, so the room it
+// frees reaches the model it was freed for first anyway. The hold is what
+// keeps that true for any other way a replica can go — a stop reported
+// between passes, a reap at the start of one — rather than it resting on the
+// order of lines in reconcileModel.
+func (r *Reconciler) isHeld(model string) bool {
+	r.mu.Lock()
+	h, ok := r.held[model]
+	r.mu.Unlock()
+	if !ok {
+		return false
+	}
+	if r.now().After(h.Until) {
+		r.mu.Lock()
+		delete(r.held, model)
+		r.mu.Unlock()
+		return false
+	}
+	if to, ok := r.cat.Load().Resolve(h.For); ok && r.liveCount(h.For) >= r.desiredFor(to) {
+		r.mu.Lock()
+		delete(r.held, model)
+		r.mu.Unlock()
+		return false
+	}
+	return true
+}
+
+// liveCount is how many replicas of a model hold capacity.
+func (r *Reconciler) liveCount(model string) int {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	n := 0
+	for _, rep := range r.replicas {
+		if rep.Model == model && rep.State.Counts() && rep.RetiringAt.IsZero() {
+			n++
+		}
+	}
+	return n
+}
+
+// victims lists the replicas that may be stopped for m, in the order they
+// should be: idle on-demand ones first, then what other tenants borrowed past
+// their guarantees when m's owner is within its own, then what lower classes
+// grew into under load, then the same within m's class, then lower classes'
+// minimums; within each, the longest unused first.
+//
+// A replica within its owner's guarantee is taken only for a model of the
+// same owner: a guarantee that another tenant's higher class could take is
+// not one.
+func (r *Reconciler) victims(m catalog.Model) []victim {
+	cat := r.cat.Load()
+	now := r.now()
+	myRank := catalog.ClassRank(m.EffectiveClass())
+	belowMin := r.liveCount(m.Name) < m.Scaling.Min
+
+	pol := r.policy()
+	var borrowed map[string]bool
+	reclaim := false
+	if pol != nil {
+		hs := r.Holdings()
+		borrowed = pol.Borrowed(hs)
+		reclaim = pol.Guaranteed(m.Owner, pol.Usage(hs), profileNeed(m))
+	}
+
+	r.mu.Lock()
+	defer r.mu.Unlock()
+
+	// Per model, its live replicas newest first: the newest are the ones
+	// above its minimum.
+	byModel := map[string][]*Replica{}
+	for _, rep := range r.replicas {
+		if rep.Model == m.Name || !rep.State.Counts() || !rep.RetiringAt.IsZero() || rep.PreemptedFor != "" {
+			continue
+		}
+		byModel[rep.Model] = append(byModel[rep.Model], rep)
+	}
+
+	var out []victim
+	for model, reps := range byModel {
+		other, ok := cat.Resolve(model)
+		if !ok {
+			continue
+		}
+		class := other.EffectiveClass()
+		rank := catalog.ClassRank(class)
+		sort.Slice(reps, func(i, j int) bool { return reps[i].CreatedAt.After(reps[j].CreatedAt) })
+		above := max(len(reps)-other.Scaling.Min, 0)
+		for i, rep := range reps {
+			aboveMin := i < above
+			tier := -1
+			switch {
+			case rank < myRank && other.Scaling.OnDemand() && idleFor(rep, now) >= r.cfg.EvictionGrace:
+				tier = 0
+			case rank < myRank && aboveMin:
+				tier = 2
+			case rank == myRank && aboveMin && belowMin:
+				tier = 3
+			case rank < myRank && class != catalog.ClassCritical:
+				tier = 4
+			}
+			if pol != nil && other.Owner != m.Owner {
+				switch {
+				case borrowed[rep.ID] && reclaim && (class != catalog.ClassCritical || aboveMin):
+					// Past its owner's guarantee, wanted by an owner within
+					// its own: taken back whatever the classes.
+					if tier != 0 {
+						tier = 1
+					}
+				case !borrowed[rep.ID] && guaranteed(pol, other.Owner):
+					continue // held within its owner's guarantee
+				}
+			}
+			if tier < 0 {
+				continue
+			}
+			// Just started: stopping it throws away a model load that has
+			// only now been paid for.
+			if !rep.ReadyAt.IsZero() && now.Sub(rep.ReadyAt) < r.cfg.MinLifetime {
+				continue
+			}
+			if len(nodesOf(rep)) > 1 {
+				continue // see makeRoomFor: a multi-node replica is its own decision
+			}
+			prof, _ := other.ProfileByName(rep.Profile)
+			var size quota.Amount
+			for _, sh := range rep.Shards {
+				size = size.Plus(quota.Amount{GPUs: len(sh.GPUIndices), CPUCores: cpuCores(prof), MemoryGB: prof.MinMemoryGB})
+			}
+			out = append(out, victim{ID: rep.ID, Model: model, Class: class,
+				AboveMin: aboveMin, Tier: tier, Idle: idleFor(rep, now), Size: size})
+		}
+	}
+	sort.SliceStable(out, func(i, j int) bool {
+		a, b := out[i], out[j]
+		if a.Tier != b.Tier {
+			return a.Tier < b.Tier
+		}
+		if a.Idle != b.Idle {
+			return a.Idle > b.Idle
+		}
+		// The smaller first: the least served is lost if it is enough, and
+		// the pruning after keeps a larger one only when it is needed.
+		if a.Size.GPUs != b.Size.GPUs {
+			return a.Size.GPUs < b.Size.GPUs
+		}
+		if a.Size.MemoryGB != b.Size.MemoryGB {
+			return a.Size.MemoryGB < b.Size.MemoryGB
+		}
+		return a.ID < b.ID
+	})
+	return out
+}
+
+// idleFor is how long a replica has gone unused; zero when nobody has said,
+// which is not the same as idle.
+func idleFor(rep *Replica, now time.Time) time.Duration {
+	if rep.LastUsedAt.IsZero() {
+		return 0
+	}
+	return now.Sub(rep.LastUsedAt)
+}
+
+func nodesOf(rep *Replica) map[string]struct{} {
+	out := map[string]struct{}{}
+	for _, sh := range rep.Shards {
+		out[sh.NodeID] = struct{}{}
+	}
+	return out
+}
+
+// guaranteed reports whether an owner has a guarantee anywhere up its tree.
+func guaranteed(pol *quota.Policy, owner string) bool {
+	for _, q := range pol.Governing(owner) {
+		if q.Guarantee.Any() {
+			return true
+		}
+	}
+	return false
+}
@@ -0,0 +1,418 @@
+package reconciler
+
+import (
+	"log/slog"
+	"strings"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
+)
+
+// newFleetHarness is newHarness with a catalog of several models.
+func newFleetHarness(t *testing.T, models []catalog.Model, nodes ...inventory.Node) *harness {
+	t.Helper()
+	cat, err := catalog.New(models)
+	if err != nil {
+		t.Fatal(err)
+	}
+	inv := inventory.NewStore()
+	c := newClock()
+	for _, n := range nodes {
+		inv.Upsert(n)
+	}
+	disp := &fakeDispatcher{}
+	cfg := DefaultConfig()
+	cfg.UnreachableAfter = 15 * time.Second
+	cfg.LostAfter = 60 * time.Second
+	r := New(cfg, catalog.NewLive(cat), inv, disp, slog.New(slog.DiscardHandler))
+	r.SetClock(c.now)
+	c.advance(cfg.SettleFor)
+	for _, n := range nodes {
+		touchAt(inv, n.ID, c.now())
+	}
+	return &harness{r: r, inv: inv, disp: disp, clock: c}
+}
+
+// gpuModel runs on one whole 8-GPU node.
+func gpuModel(name, class string, min int) catalog.Model {
+	return catalog.Model{Name: name, Source: "/models/" + name, Class: class,
+		Scaling:  catalog.Scaling{Min: min, Max: min},
+		Profiles: []catalog.Profile{{Name: "tp8", Engine: "vllm", GPUClass: "h100-80g", TP: 8}}}
+}
+
+// settle runs passes until nothing changes, marking what starts ready and
+// letting preempted replicas run out their grace.
+func (h *harness) settle(t *testing.T, passes int) {
+	t.Helper()
+	for range passes {
+		h.r.Reconcile(t.Context())
+		h.markReady()
+		h.advance(h.r.cfg.PreemptGrace + time.Second)
+	}
+}
+
+func replicasOf(h *harness, model string) (live int) {
+	for _, rep := range h.r.Replicas() {
+		if rep.Model == model && rep.State.Counts() {
+			live++
+		}
+	}
+	return live
+}
+
+// fill places the standard model, then ages it past MinLifetime so it may be
+// taken from, and only then adds the model that wants its node.
+func fillThenWant(t *testing.T, want catalog.Model, have catalog.Model, nodes ...inventory.Node) *harness {
+	t.Helper()
+	h := newFleetHarness(t, []catalog.Model{have, withMin(want, 0)}, nodes...)
+	h.settle(t, 3)
+	if got := replicasOf(h, have.Name); got != have.Scaling.Min {
+		t.Fatalf("%s has %d replicas before anything wanted its room", have.Name, got)
+	}
+	h.advance(h.r.cfg.MinLifetime + time.Minute)
+	cat, err := catalog.New([]catalog.Model{have, want})
+	if err != nil {
+		t.Fatal(err)
+	}
+	h.r.cat.Store(cat)
+	return h
+}
+
+func withMin(m catalog.Model, n int) catalog.Model {
+	m.Scaling = catalog.Scaling{Min: n, Max: n}
+	return m
+}
+
+func TestACriticalModelTakesAStandardModelsNode(t *testing.T) {
+	h := fillThenWant(t, gpuModel("vital", catalog.ClassCritical, 1), gpuModel("chat", "", 1), gpuNode("1", 8))
+
+	h.r.Reconcile(t.Context())
+	var draining int
+	for _, rep := range h.r.Replicas() {
+		if rep.Model == "chat" && rep.State == Draining && rep.PreemptedFor == "vital" {
+			draining++
+		}
+	}
+	if draining != 1 {
+		t.Fatalf("the standard replica was not drained for the critical one: %+v", h.r.Replicas())
+	}
+	if hs := health(h, "chat"); hs.HeldFor != "vital" || hs.Preempted != 1 {
+		t.Errorf("the victim's health does not say it gave way: %+v", hs)
+	}
+
+	h.settle(t, 4)
+	if replicasOf(h, "vital") != 1 {
+		t.Fatalf("the critical model was not placed once room was made: %+v", h.r.Replicas())
+	}
+	// Held back until the critical model had its replica, then short again
+	// because there is no room: it must not have taken the node back — not
+	// even for a moment, which would load a model only to preempt it again.
+	if replicasOf(h, "chat") != 0 {
+		t.Error("the standard model took the node back")
+	}
+	started := 0
+	for _, cmd := range h.disp.startedShards() {
+		if strings.HasPrefix(cmd.ReplicaID, "chat-") {
+			started++
+		}
+	}
+	if started != 1 {
+		t.Errorf("the standard model started %d replicas; it should not have tried again", started)
+	}
+}
+
+func TestAStandardModelDoesNotTakeAnothersMinimum(t *testing.T) {
+	h := fillThenWant(t, gpuModel("second", "", 1), gpuModel("first", "", 1), gpuNode("1", 8))
+	h.settle(t, 3)
+	if replicasOf(h, "first") != 1 || replicasOf(h, "second") != 0 {
+		t.Errorf("a model of the same class took another's minimum: %+v", h.r.Replicas())
+	}
+	if hs := health(h, "second"); !hs.Unplaceable {
+		t.Error("the model left without room should say it cannot be placed")
+	}
+}
+
+func TestWithinAClassOnlyWhatWasGrownIntoIsTaken(t *testing.T) {
+	grown := gpuModel("busy", "", 1)
+	grown.Scaling = catalog.Scaling{Min: 1, Max: 2}
+	h := newFleetHarness(t, []catalog.Model{grown, withMin(gpuModel("new", "", 1), 0)}, gpuNode("1", 8), gpuNode("2", 8))
+	h.settle(t, 2)
+	// Load grows it to its max: the second replica is above its minimum.
+	h.r.mu.Lock()
+	h.r.extra["busy"] = 1
+	h.r.mu.Unlock()
+	h.settle(t, 3)
+	if replicasOf(h, "busy") != 2 {
+		t.Fatalf("setup: busy did not grow to 2: %+v", h.r.Replicas())
+	}
+	h.advance(h.r.cfg.MinLifetime + time.Minute)
+	cat, _ := catalog.New([]catalog.Model{grown, gpuModel("new", "", 1)})
+	h.r.cat.Store(cat)
+
+	h.settle(t, 5)
+	if replicasOf(h, "new") != 1 || replicasOf(h, "busy") != 1 {
+		t.Errorf("the grown replica should give way to a minimum: busy %d, new %d",
+			replicasOf(h, "busy"), replicasOf(h, "new"))
+	}
+}
+
+func TestNothingIsStoppedThatWouldNotMakeRoom(t *testing.T) {
+	// Two nodes, each half used by an opportunistic model; the critical one
+	// needs a whole node. Stopping one frees half a node, which is no use —
+	// and both together are on two nodes, which is no use either.
+	half := catalog.Model{Name: "half", Source: "/m", Class: catalog.ClassOpportunistic,
+		Scaling:  catalog.Scaling{Min: 2, Max: 2},
+		Profiles: []catalog.Profile{{Name: "tp4", Engine: "vllm", GPUClass: "h100-80g", TP: 4}}}
+	five := func(id string) inventory.Node { return gpuNode(id, 5) }
+	h := fillThenWant(t, gpuModel("vital", catalog.ClassCritical, 1), half, five("1"), five("2"))
+	h.r.Reconcile(t.Context())
+	for _, rep := range h.r.Replicas() {
+		if rep.PreemptedFor != "" || rep.State == Draining {
+			t.Fatalf("a replica was stopped without making room: %+v", rep)
+		}
+	}
+}
+
+func TestACriticalMinimumIsNeverTaken(t *testing.T) {
+	h := fillThenWant(t, gpuModel("also-vital", catalog.ClassCritical, 1), gpuModel("vital", catalog.ClassCritical, 1), gpuNode("1", 8))
+	h.settle(t, 3)
+	if replicasOf(h, "vital") != 1 {
+		t.Error("a critical model's minimum was taken")
+	}
+}
+
+func TestAReplicaThatJustStartedIsLeftAlone(t *testing.T) {
+	h := newFleetHarness(t, []catalog.Model{gpuModel("chat", "", 1), withMin(gpuModel("vital", catalog.ClassCritical, 1), 0)}, gpuNode("1", 8))
+	h.settle(t, 2)
+	cat, _ := catalog.New([]catalog.Model{gpuModel("chat", "", 1), gpuModel("vital", catalog.ClassCritical, 1)})
+	h.r.cat.Store(cat)
+	h.r.Reconcile(t.Context())
+	for _, rep := range h.r.Replicas() {
+		if rep.State == Draining {
+			t.Fatal("a replica that had only just loaded was preempted")
+		}
+	}
+}
+
+func health(h *harness, model string) ModelHealth {
+	for _, x := range h.r.ModelHealth() {
+		if x.Model == model {
+			return x
+		}
+	}
+	return ModelHealth{}
+}
+
+func TestAHoldLastsUntilTheRoomIsUsedOrItExpires(t *testing.T) {
+	h := fillThenWant(t, gpuModel("vital", catalog.ClassCritical, 1), gpuModel("chat", "", 1), gpuNode("1", 8))
+	h.r.Reconcile(t.Context())
+	if !h.r.isHeld("chat") {
+		t.Fatal("the model that gave way is not held")
+	}
+	if h.r.isHeld("vital") {
+		t.Error("the model room was made for is held")
+	}
+	// Released as soon as the model it was given to has what it asked for.
+	h.settle(t, 3)
+	if replicasOf(h, "vital") != 1 {
+		t.Fatalf("setup: vital was not placed: %+v", h.r.Replicas())
+	}
+	if h.r.isHeld("chat") {
+		t.Error("the hold outlasted its purpose")
+	}
+
+	// And, whatever happens, after PreemptHold.
+	h.r.mu.Lock()
+	h.r.held["other"] = hold{Until: h.r.now().Add(time.Minute), For: "never-placed"}
+	h.r.mu.Unlock()
+	if !h.r.isHeld("other") {
+		t.Fatal("a fresh hold is not in force")
+	}
+	h.advance(2 * time.Minute)
+	if h.r.isHeld("other") {
+		t.Error("a hold outlived its time")
+	}
+}
+
+func TestOnlyTheReplicasThatMakeTheRoomAreStopped(t *testing.T) {
+	// Two half-used nodes. "idle" on node 1 is on demand and long unused, so
+	// it is first in line; but the critical model needs node 2's other half,
+	// which only stopping "chat" frees.
+	idle := catalog.Model{Name: "idle", Source: "/m", Scaling: catalog.Scaling{Min: 0, Max: 1},
+		Profiles: []catalog.Profile{{Name: "tp4", Engine: "vllm", GPUClass: "h100-80g", TP: 4}}}
+	chat := catalog.Model{Name: "chat", Source: "/m", Scaling: catalog.Scaling{Min: 1, Max: 1},
+		Profiles: []catalog.Profile{{Name: "tp6", Engine: "vllm", GPUClass: "h100-80g", TP: 6}}}
+	vital := catalog.Model{Name: "vital", Source: "/m", Class: catalog.ClassCritical,
+		Scaling:  catalog.Scaling{Min: 1, Max: 1},
+		Profiles: []catalog.Profile{{Name: "tp8", Engine: "vllm", GPUClass: "h100-80g", TP: 8}}}
+	h := newFleetHarness(t, []catalog.Model{idle, chat}, gpuNode("1", 6), gpuNode("2", 8))
+	h.r.Want("idle")
+	h.settle(t, 4)
+	var idleNode, chatNode string
+	for _, rep := range h.r.Replicas() {
+		switch rep.Model {
+		case "idle":
+			idleNode = rep.Shards[0].NodeID
+		case "chat":
+			chatNode = rep.Shards[0].NodeID
+		}
+	}
+	if idleNode == "" || chatNode != "2" {
+		t.Fatalf("setup: idle on %q, chat on %q; chat must hold node 2: %+v", idleNode, chatNode, h.r.Replicas())
+	}
+	h.r.mu.Lock()
+	for _, rep := range h.r.replicas {
+		rep.LastUsedAt = h.r.now().Add(-time.Hour)
+	}
+	h.r.mu.Unlock()
+	h.advance(h.r.cfg.MinLifetime + time.Minute)
+	cat, _ := catalog.New([]catalog.Model{idle, chat, vital})
+	h.r.cat.Store(cat)
+
+	h.r.Reconcile(t.Context())
+	var stopped []string
+	for _, rep := range h.r.Replicas() {
+		if rep.PreemptedFor == "vital" {
+			stopped = append(stopped, rep.Model)
+		}
+	}
+	if len(stopped) != 1 || stopped[0] != "chat" {
+		t.Errorf("stopped %v; only chat's replica makes room for a whole node", stopped)
+	}
+}
+
+func TestOfTwoThatWouldEachMakeRoomTheSmallerIsStopped(t *testing.T) {
+	cpu := inventory.Node{ID: "a", MgmtAddr: "10.0.0.1", State: inventory.Ready, MemoryTotalMB: 31 * 1024, CPUCount: 8}
+	big := catalog.Model{Name: "big", Source: "/m", Scaling: catalog.Scaling{Min: 1, Max: 1},
+		Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 12, CPUCores: 4}}}
+	small := catalog.Model{Name: "small", Source: "/m", Scaling: catalog.Scaling{Min: 1, Max: 1},
+		Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 2, CPUCores: 1}}}
+	vital := catalog.Model{Name: "vital", Source: "/m", Class: catalog.ClassCritical,
+		Scaling:  catalog.Scaling{Min: 1, Max: 1},
+		Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 15, CPUCores: 4}}}
+	// 8 cores: big 4 + small 1 leaves 3, and vital needs 4. Stopping either
+	// is enough; stopping big loses far more.
+	for range 5 { // the choice must not depend on map order
+		h := newFleetHarness(t, []catalog.Model{big, small}, cpu)
+		h.settle(t, 3)
+		h.advance(h.r.cfg.MinLifetime + time.Minute)
+		cat, _ := catalog.New([]catalog.Model{big, small, vital})
+		h.r.cat.Store(cat)
+		h.r.Reconcile(t.Context())
+		var stopped []string
+		for _, rep := range h.r.Replicas() {
+			if rep.PreemptedFor == "vital" {
+				stopped = append(stopped, rep.Model)
+			}
+		}
+		if len(stopped) != 1 || stopped[0] != "small" {
+			t.Fatalf("stopped %v; the small replica alone makes room", stopped)
+		}
+	}
+}
+
+func TestHeadroomCountsWhatAHigherClassCouldTake(t *testing.T) {
+	h := fillThenWant(t, gpuModel("vital", catalog.ClassCritical, 1), gpuModel("chat", "", 1), gpuNode("1", 8))
+	got, _ := h.r.Headroom("vital", 3)
+	if got.More != 0 || got.ByPreempting != 1 || len(got.Preempts) != 1 || got.Preempts[0] != "chat" {
+		t.Errorf("a critical model should see one more by stopping chat: %+v", got)
+	}
+	// Tried on a copy: nothing was stopped.
+	if replicasOf(h, "chat") != 1 {
+		t.Fatal("asking about room stopped a replica")
+	}
+	// A model of chat's own class may not take its minimum, so has no such room.
+	peer := newFleetHarness(t, []catalog.Model{gpuModel("chat", "", 1), withMin(gpuModel("peer", "", 1), 0)}, gpuNode("1", 8))
+	peer.settle(t, 3)
+	peer.advance(peer.r.cfg.MinLifetime + time.Minute)
+	if got, _ := peer.r.Headroom("peer", 3); got.ByPreempting != 0 {
+		t.Errorf("a standard model counted room in another's minimum: %+v", got)
+	}
+}
+
+func TestARestartBorrowsALowerClassesRoomAndGivesItBack(t *testing.T) {
+	// chat fills node 1; the only other node holds an opportunistic replica.
+	chat := gpuModel("chat", "", 1)
+	extra := gpuModel("extra", catalog.ClassOpportunistic, 1)
+	h := newFleetHarness(t, []catalog.Model{chat, extra}, gpuNode("1", 8), gpuNode("2", 8))
+	h.settle(t, 3)
+	h.advance(h.r.cfg.MinLifetime + time.Minute)
+	var old string
+	for _, rep := range h.r.Replicas() {
+		if rep.Model == "chat" {
+			old = rep.ID
+		}
+	}
+	if old == "" || replicasOf(h, "extra") != 1 {
+		t.Fatalf("setup: %+v", h.r.Replicas())
+	}
+
+	if err := h.r.Restart(old); err != nil {
+		t.Fatal(err)
+	}
+	// Replacement first, beside the old one, in the room extra gives up;
+	// the old one keeps serving until then.
+	servedThroughout := true
+	for range 8 {
+		h.r.Reconcile(t.Context())
+		h.markReady()
+		ready := 0
+		for _, rep := range h.r.Replicas() {
+			if rep.Model == "chat" && rep.State == Ready {
+				ready++
+			}
+		}
+		if ready == 0 {
+			servedThroughout = false
+		}
+		h.advance(h.r.cfg.PreemptGrace + time.Second)
+	}
+	if !servedThroughout {
+		t.Error("chat had no ready replica during its restart")
+	}
+	var newChat bool
+	for _, rep := range h.r.Replicas() {
+		if rep.Model == "chat" && rep.ID != old && rep.State == Ready {
+			newChat = true
+		}
+		if rep.ID == old {
+			t.Errorf("the old replica is still there: %+v", rep)
+		}
+	}
+	if !newChat {
+		t.Fatalf("no new chat replica: %+v", h.r.Replicas())
+	}
+	// And extra has its replica back, in the room the old chat left.
+	h.settle(t, 4)
+	if replicasOf(h, "extra") != 1 {
+		t.Errorf("the opportunistic model did not get its room back: %+v", h.r.Replicas())
+	}
+}
+
+func TestHeadroomNamesOnlyTheReplicasThatWouldMakeRoom(t *testing.T) {
+	// vital needs a whole node and chat is on the only one; a second
+	// critical can never fit. Asking for 3: one by stopping chat, then no
+	// candidates left — nothing else may be named.
+	h := fillThenWant(t, gpuModel("vital", catalog.ClassCritical, 1), gpuModel("chat", "", 1), gpuNode("1", 8))
+	got, _ := h.r.Headroom("vital", 3)
+	if got.ByPreempting != 1 || len(got.Preempts) != 1 {
+		t.Errorf("%+v", got)
+	}
+	// Free room for one, and candidates that cannot make room for a second:
+	// none of them may be listed.
+	small := catalog.Model{Name: "small", Source: "/m", Class: catalog.ClassOpportunistic,
+		Scaling:  catalog.Scaling{Min: 1, Max: 1},
+		Profiles: []catalog.Profile{{Name: "tp2", Engine: "vllm", GPUClass: "h100-80g", TP: 2}}}
+	h2 := fillThenWant(t, gpuModel("big", "", 1), small, gpuNode("1", 8), gpuNode("2", 8))
+	got2, _ := h2.r.Headroom("big", 3)
+	if got2.More != 1 || got2.ByPreempting != 1 || len(got2.Preempts) != 1 {
+		t.Errorf("one free, one by stopping small: %+v", got2)
+	}
+	h3 := fillThenWant(t, gpuModel("big", "", 1), small, gpuNode("1", 8), gpuNode("2", 6))
+	got3, _ := h3.r.Headroom("big", 3)
+	if got3.ByPreempting != 0 || len(got3.Preempts) != 0 {
+		t.Errorf("stopping small frees a 6-GPU node, too small for big; nobody may be named: %+v", got3)
+	}
+}
@@ -0,0 +1,105 @@
+package reconciler
+
+import (
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/placement"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// SetQuotas supplies the tenancy tree's capacity quotas, read afresh each
+// time it is called. Without it nothing is guaranteed or limited, and
+// preemption goes by class alone.
+func (r *Reconciler) SetQuotas(f func() *quota.Policy) {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	r.quotas = f
+}
+
+func (r *Reconciler) policy() *quota.Policy {
+	r.mu.Lock()
+	f := r.quotas
+	r.mu.Unlock()
+	if f == nil {
+		return nil
+	}
+	return f()
+}
+
+// Holdings is what every replica holds, and for whom.
+func (r *Reconciler) Holdings() []quota.Holding {
+	cat := r.cat.Load()
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	out := make([]quota.Holding, 0, len(r.replicas))
+	for _, rep := range r.replicas {
+		m, ok := cat.Resolve(rep.Model)
+		if !ok {
+			continue
+		}
+		prof, _ := m.ProfileByName(rep.Profile)
+		var a quota.Amount
+		for _, sh := range rep.Shards {
+			a = a.Plus(quota.Amount{GPUs: len(sh.GPUIndices), CPUCores: cpuCores(prof), MemoryGB: prof.MinMemoryGB})
+		}
+		out = append(out, quota.Holding{Replica: rep.ID, Owner: m.Owner, Amount: a, Since: rep.CreatedAt})
+	}
+	return out
+}
+
+// placementNeed is what a placement would hold.
+func placementNeed(m catalog.Model, pl *placement.Placement) quota.Amount {
+	prof, _ := m.ProfileByName(pl.Profile.Name)
+	var a quota.Amount
+	for _, n := range pl.Nodes {
+		a = a.Plus(quota.Amount{GPUs: len(n.GPUIndices), CPUCores: cpuCores(prof), MemoryGB: prof.MinMemoryGB})
+	}
+	return a
+}
+
+// profileNeed is what a replica of a model's first profile would hold, for
+// deciding before there is a placement.
+func profileNeed(m catalog.Model) quota.Amount {
+	if len(m.Profiles) == 0 {
+		return quota.Amount{}
+	}
+	p := m.Profiles[0]
+	nodes := max(p.Nodes, 1)
+	gpus := p.GPUsPerNode
+	if gpus == 0 && !p.CPUOnly() {
+		gpus = max(p.TP, 1)
+	}
+	return quota.Amount{GPUs: gpus * nodes, CPUCores: cpuCores(p) * nodes, MemoryGB: p.MinMemoryGB * nodes}
+}
+
+// overLimit says why a placement would take its model's owner past a limit.
+func (r *Reconciler) overLimit(m catalog.Model, pl *placement.Placement) string {
+	pol := r.policy()
+	if pol == nil || m.Owner == "" {
+		return ""
+	}
+	return pol.AtLimit(m.Owner, pol.Usage(r.Holdings()), placementNeed(m, pl))
+}
+
+// QuotaUse is one scope's quota beside what it holds.
+type QuotaUse struct {
+	Scope     string
+	Guarantee store.Resources
+	Limit     store.Resources
+	Holds     quota.Amount
+}
+
+// QuotaUse lists every quota with what its scope holds now.
+func (r *Reconciler) QuotaUse() []QuotaUse {
+	pol := r.policy()
+	if pol == nil {
+		return nil
+	}
+	usage := pol.Usage(r.Holdings())
+	var out []QuotaUse
+	for _, q := range pol.Quotas() {
+		key := quota.Key(q.Scope, q.ScopeID)
+		out = append(out, QuotaUse{Scope: key, Guarantee: q.Guarantee, Limit: q.Limit, Holds: usage[key]})
+	}
+	return out
+}
@@ -0,0 +1,81 @@
+package reconciler
+
+import (
+	"strings"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+func owned(m catalog.Model, owner string) catalog.Model { m.Owner = owner; return m }
+
+func gpus(n int) store.Resources { return store.Resources{GPUs: &n} }
+
+// Two teams in one org; each 8-GPU node holds one tp8 replica.
+func withQuotas(h *harness, qs ...store.CapacityQuota) {
+	pol := quota.New(qs, map[string][]string{"team:ml": {"org:acme"}, "team:web": {"org:acme"}})
+	h.r.SetQuotas(func() *quota.Policy { return pol })
+}
+
+func TestATenantWithinItsGuaranteeTakesBackWhatAnotherBorrowed(t *testing.T) {
+	// web is guaranteed one node and borrowed the second; ml is guaranteed
+	// one node and wants it. Same class, and web's replica is at its minimum
+	// — by class alone nothing could be taken.
+	web := owned(gpuModel("web-chat", "", 2), "team:web")
+	ml := owned(gpuModel("ml-train", "", 1), "team:ml")
+	h := newFleetHarness(t, []catalog.Model{web, withMin(ml, 0)}, gpuNode("1", 8), gpuNode("2", 8))
+	withQuotas(h,
+		store.CapacityQuota{Scope: store.ScopeTeam, ScopeID: "web", Guarantee: gpus(8)},
+		store.CapacityQuota{Scope: store.ScopeTeam, ScopeID: "ml", Guarantee: gpus(8)})
+	h.settle(t, 4)
+	if replicasOf(h, "web-chat") != 2 {
+		t.Fatalf("setup: web did not borrow the second node: %+v", h.r.Replicas())
+	}
+	h.advance(h.r.cfg.MinLifetime + time.Minute)
+	cat, _ := catalog.New([]catalog.Model{web, ml})
+	h.r.cat.Store(cat)
+
+	h.settle(t, 5)
+	if replicasOf(h, "ml-train") != 1 || replicasOf(h, "web-chat") != 1 {
+		t.Errorf("the borrowed node was not taken back: web %d, ml %d",
+			replicasOf(h, "web-chat"), replicasOf(h, "ml-train"))
+	}
+}
+
+func TestAReplicaWithinItsGuaranteeIsNotTakenByAHigherClassElsewhere(t *testing.T) {
+	web := owned(gpuModel("web-chat", "", 1), "team:web")
+	vital := owned(gpuModel("vital", catalog.ClassCritical, 1), "team:ml")
+	h := fillThenWant(t, vital, web, gpuNode("1", 8))
+	withQuotas(h, store.CapacityQuota{Scope: store.ScopeTeam, ScopeID: "web", Guarantee: gpus(8)})
+	h.settle(t, 3)
+	if replicasOf(h, "web-chat") != 1 || replicasOf(h, "vital") != 0 {
+		t.Errorf("a guaranteed replica was preempted by another tenant's class: %+v", h.r.Replicas())
+	}
+	// Without the guarantee, critical takes it: the guarantee was the reason.
+	h2 := fillThenWant(t, vital, web, gpuNode("1", 8))
+	h2.settle(t, 3)
+	if replicasOf(h2, "vital") != 1 {
+		t.Error("setup: without a guarantee the critical model should have taken the node")
+	}
+}
+
+func TestALimitStopsAModelBeforeItStarts(t *testing.T) {
+	web := owned(gpuModel("web-chat", "", 2), "team:web")
+	h := newFleetHarness(t, []catalog.Model{web}, gpuNode("1", 8), gpuNode("2", 8))
+	withQuotas(h, store.CapacityQuota{Scope: store.ScopeOrg, ScopeID: "acme", Limit: gpus(8)})
+	h.settle(t, 4)
+	if replicasOf(h, "web-chat") != 1 {
+		t.Errorf("a second replica went past the org's 8-GPU limit: %+v", h.r.Replicas())
+	}
+	hs := health(h, "web-chat")
+	if !hs.Unplaceable || !strings.Contains(hs.Placement, "org:acme is at its capacity limit (16 GPUs of 8)") {
+		t.Errorf("the refusal should name the limit: %+v", hs)
+	}
+	room, _ := h.r.Headroom("web-chat", 5)
+	if room.More != 0 || len(room.Blocked) != 1 || !strings.Contains(room.Blocked[0], "capacity limit") {
+		t.Errorf("headroom should stop at the limit, not the empty node: %+v", room)
+	}
+}
@@ -3,15 +3,10 @@ package reconciler
 import (
 	"context"
 	crypterand "crypto/rand"
-	"crypto/sha256"
-	"encoding/hex"
-	"encoding/json"
-	"errors"
 	"fmt"
 	"log/slog"
 	"math/rand/v2"
 	"slices"
-	"strings"
 	"sync"
 	"time"

@@ -20,6 +15,7 @@ import (
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/placement"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
 )

 // ShardCommand is everything an agent needs to launch one shard.
@@ -70,6 +66,19 @@ type Config struct {
 	// Without a bound, draining a node whose model cannot be placed anywhere
 	// else would never finish.
 	DrainGrace time.Duration
+	// PreemptGrace is how long a replica stopped to make room for another
+	// is given to finish what it is serving: a request, not a replacement.
+	PreemptGrace time.Duration
+	// PreemptHold is how long a model that gave up room may not take it back,
+	// so the replica it was given up for is placed first.
+	PreemptHold time.Duration
+	// ScaleUpWaiting is the average number of requests waiting per ready
+	// replica, held for ScaleUpAfter, that adds a replica — up to the model's
+	// max. ScaleDownAfter is how long with nothing waiting before one of the
+	// replicas added that way is taken away again, down to its min.
+	ScaleUpWaiting float64
+	ScaleUpAfter   time.Duration
+	ScaleDownAfter time.Duration
 	// ParkAfter is how many failures in a row stop a model being retried.
 	//
 	// Backoff alone never gives up: it grows to MaxBackoff and keeps trying,
@@ -139,7 +148,12 @@ func DefaultConfig() Config {
 		EvictionGrace:    2 * time.Minute,
 		MinLifetime:      5 * time.Minute,
 		DrainGrace:       5 * time.Minute,
+		PreemptGrace:     30 * time.Second,
+		PreemptHold:      5 * time.Minute,
 		ParkAfter:        5,
+		ScaleUpWaiting:   1,
+		ScaleUpAfter:     time.Minute,
+		ScaleDownAfter:   10 * time.Minute,
 		PortRange:        [2]int{20000, 20999},
 		MgmtInterface:    "eth0",
 	}
@@ -153,6 +167,12 @@ func (c *Config) normalize() {
 	if c.BaseBackoff <= 0 {
 		c.BaseBackoff = d.BaseBackoff
 	}
+	if c.PreemptGrace <= 0 {
+		c.PreemptGrace = d.PreemptGrace
+	}
+	if c.PreemptHold <= 0 {
+		c.PreemptHold = d.PreemptHold
+	}
 	if c.MaxBackoff < c.BaseBackoff {
 		c.MaxBackoff = max(c.BaseBackoff, d.MaxBackoff)
 	}
@@ -182,6 +202,15 @@ func (c *Config) normalize() {
 	if c.ParkAfter <= 0 {
 		c.ParkAfter = d.ParkAfter
 	}
+	if c.ScaleUpWaiting <= 0 {
+		c.ScaleUpWaiting = d.ScaleUpWaiting
+	}
+	if c.ScaleUpAfter <= 0 {
+		c.ScaleUpAfter = d.ScaleUpAfter
+	}
+	if c.ScaleDownAfter <= 0 {
+		c.ScaleDownAfter = d.ScaleDownAfter
+	}
 	if c.PortRange[1] <= c.PortRange[0] {
 		c.PortRange = d.PortRange
 	}
@@ -228,9 +257,24 @@ type Reconciler struct {
 	restartQueue map[string][]string
 	// parked holds models that failed ParkAfter times in a row.
 	parked map[string]parkRecord
+	// extra is how many replicas load has added above a model's base, and
+	// busySince/quietSince when its load last crossed the thresholds.
+	extra      map[string]int
+	busySince  map[string]time.Time
+	quietSince map[string]time.Time
 	// placementLogged remembers the last "no placement" said per model, so it
 	// is said when it changes rather than on every five-second pass.
 	placementLogged map[string]placementNote
+	// preempting is, per model, the replicas stopped to make room for it and
+	// not gone yet; held is, per model that gave them up, until when it may
+	// not take the room back; preempted counts, per model, replicas it lost.
+	preempting map[string][]string
+	held       map[string]hold
+	preempted  map[string]int
+	// quotas is the tenancy tree's capacity quotas as last read; nil is none.
+	quotas func() *quota.Policy
+	// measurements are benchmark results by model/profile; nil is none.
+	measurements func() map[string]Measured

 	wake chan struct{}
 	stop chan struct{}
@@ -261,8 +305,14 @@ func New(cfg Config, cat *catalog.Live, inv *inventory.Store, disp Dispatcher, l
 		wanted:          map[string]time.Time{},
 		restartQueue:    map[string][]string{},
 		parked:          map[string]parkRecord{},
+		extra:           map[string]int{},
+		busySince:       map[string]time.Time{},
+		quietSince:      map[string]time.Time{},
 		placementLogged: map[string]placementNote{},
 		lastExplain:     map[string]placement.Result{},
+		preempting:      map[string][]string{},
+		held:            map[string]hold{},
+		preempted:       map[string]int{},
 		wake:            make(chan struct{}, 1),
 		stop:            make(chan struct{}),
 		done:            make(chan struct{}),
@@ -433,127 +483,6 @@ func (r *Reconciler) fleetHasSettled() bool {
 	return settled
 }

-// RecordUsage stamps replicas with when each was last sent a request.
-//
-// Replicas absent from the map are left alone rather than zeroed: a router that
-// has just restarted knows nothing about what happened before it, and treating
-// that as "never used" would make every replica look abandoned at once —
-// exactly when a fleet is least able to afford acting on that.
-func (r *Reconciler) RecordUsage(lastUsed map[string]time.Time) {
-	if len(lastUsed) == 0 {
-		return
-	}
-	r.mu.Lock()
-	defer r.mu.Unlock()
-	for id, at := range lastUsed {
-		rep, ok := r.replicas[id]
-		if !ok {
-			continue
-		}
-		// Only forwards. Two routers reporting, or a stale snapshot arriving
-		// late, must not move a replica's last use backwards.
-		if at.After(rep.LastUsedAt) {
-			rep.LastUsedAt = at
-		}
-	}
-	r.noteUsage(lastUsed)
-}
-
-// IdleFor reports how long a replica has gone without a request, and whether
-// that is known at all.
-//
-// The second return is not a detail. "Idle for an hour" and "nobody has told me
-// yet" are different states, and anything that acts on idleness — eviction,
-// most obviously — has to refuse to act on the second.
-func (r *Reconciler) IdleFor(id string) (time.Duration, bool) {
-	r.mu.Lock()
-	defer r.mu.Unlock()
-	rep, ok := r.replicas[id]
-	if !ok || rep.LastUsedAt.IsZero() {
-		return 0, false
-	}
-	return r.now().Sub(rep.LastUsedAt), true
-}
-
-// Want marks an on-demand model as asked for, so the next pass places it.
-//
-// Called by whatever fields the request — the router, in practice. It returns
-// whether the model is one this applies to, so a caller can tell "being loaded"
-// from "there is genuinely nothing serving this", which are different answers
-// to give a client.
-func (r *Reconciler) Want(model string) bool {
-	m, ok := r.cat.Load().Resolve(model)
-	if !ok || !m.Scaling.OnDemand() {
-		return false
-	}
-	r.mu.Lock()
-	r.wanted[m.Name] = r.now()
-	// A placement that failed before this request says nothing about now:
-	// forgotten, so the pass this triggers decides afresh and Unservable
-	// does not answer from a stale result.
-	if res, ok := r.lastExplain[m.Name]; ok && res.Placement == nil {
-		delete(r.lastExplain, m.Name)
-	}
-	r.mu.Unlock()
-	// Immediately, not on the next tick. Someone is waiting on this request,
-	// and five seconds of ticker is five seconds they spend staring at nothing.
-	r.Notify()
-	return true
-}
-
-// desiredFor is how many replicas a model should have right now.
-//
-// For an ordinary model that is its configured minimum. For an on-demand one it
-// is one while somebody is asking and none once they stop, which is the whole
-// feature: a cluster with more models than GPUs can hold every model in its
-// catalog and only the wanted ones in its cards.
-func (r *Reconciler) desiredFor(m catalog.Model) int {
-	if !m.Scaling.OnDemand() {
-		return m.Scaling.Min
-	}
-	r.mu.Lock()
-	last, asked := r.wanted[m.Name]
-	r.mu.Unlock()
-	if !asked {
-		return 0
-	}
-	if r.now().Sub(last) > r.idleTimeout(m) {
-		return 0
-	}
-	return 1
-}
-
-// idleTimeout is how long an on-demand model stays up unasked-for.
-func (r *Reconciler) idleTimeout(m catalog.Model) time.Duration {
-	if m.Scaling.IdleTimeout > 0 {
-		return m.Scaling.IdleTimeout
-	}
-	return r.cfg.IdleTimeout
-}
-
-// noteUsage keeps an on-demand model alive while it is being used.
-//
-// Without this a model would be evicted exactly IdleTimeout after the request
-// that loaded it, however busy it had been since — the timer would measure
-// time since the first request rather than the last.
-func (r *Reconciler) noteUsage(lastUsed map[string]time.Time) {
-	byModel := map[string]time.Time{}
-	for id, at := range lastUsed {
-		rep, ok := r.replicas[id]
-		if !ok {
-			continue
-		}
-		if at.After(byModel[rep.Model]) {
-			byModel[rep.Model] = at
-		}
-	}
-	for model, at := range byModel {
-		if cur, ok := r.wanted[model]; ok && at.After(cur) {
-			r.wanted[model] = at
-		}
-	}
-}
-
 // expireStuckReplicas fails replicas that never finished coming up.
 //
 // Every other path out of Starting depends on the agent reporting something. A
@@ -803,6 +732,9 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
 	switch {
 	case len(liveIDs) < desired && isParked:
 		// Not placed. Its failure is on every status the operator reads.
+	case len(liveIDs) < desired && r.isHeld(m.Name):
+		// It gave up room to a model of a higher class, which is placed
+		// first; taking it back now would undo the preemption.
 	case len(liveIDs) < desired:
 		if r.now().Before(blockedUntil) {
 			// Still in backoff. Rebuilding immediately would hot-loop on a
@@ -823,8 +755,12 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
 		r.mu.Lock()
 		rep, ok := r.replicas[id]
 		var expired bool
+		grace, preemptedFor := r.cfg.DrainGrace, ""
 		if ok {
-			expired = !rep.DrainingAt.IsZero() && r.now().Sub(rep.DrainingAt) > r.cfg.DrainGrace
+			if rep.PreemptedFor != "" {
+				grace, preemptedFor = r.cfg.PreemptGrace, rep.PreemptedFor
+			}
+			expired = !rep.DrainingAt.IsZero() && r.now().Sub(rep.DrainingAt) > grace
 		}
 		r.mu.Unlock()

@@ -832,11 +768,15 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
 		case !ok:
 		case replaced:
 			r.stopReplica(ctx, id)
+		case expired && preemptedFor != "":
+			r.log.Info("stopping a preempted replica", "replica", id,
+				"for", preemptedFor, "after", grace)
+			r.stopReplica(ctx, id)
 		case expired:
 			// The replacement never arrived — often because nowhere else can
 			// host it. Draining forever would leave the node undrainable.
 			r.log.Warn("stopping a drained replica without a replacement",
-				"replica", id, "after", r.cfg.DrainGrace)
+				"replica", id, "after", grace)
 			r.stopReplica(ctx, id)
 		}
 	}
@@ -844,122 +784,19 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
 	r.retire(ctx, retiringIDs, readyLive, len(liveIDs), desired)
 }

-// retire finishes restarting replicas.
-//
-// A retiring replica keeps serving until its replacement is ready, and is then
-// drained like any other. When no replacement could be placed at all —
-// nowhere else has the memory, which on a small CPU fleet is the usual case —
-// it is stopped after DrainGrace so its replacement can take its place: a
-// restart in place, with the model unserved while it loads if it had no other
-// replica. Said in the log, since that is the one case with an outage.
-func (r *Reconciler) retire(ctx context.Context, ids []string, readyLive, live, desired int) {
-	for _, id := range ids {
-		r.mu.Lock()
-		rep, ok := r.replicas[id]
-		var since time.Time
-		if ok {
-			since = rep.RetiringAt
-		}
-		r.mu.Unlock()
-		if !ok {
-			continue
-		}
-		switch {
-		case readyLive >= desired:
-			r.drainForRestart(ctx, id)
-		case live < desired && r.now().Sub(since) > r.cfg.DrainGrace:
-			r.log.Warn("restarting a replica in place: no room for its replacement beside it",
-				"replica", id, "after", r.cfg.DrainGrace)
-			r.drainForRestart(ctx, id)
-			r.stopReplica(ctx, id)
-		}
-	}
-}
-
-func (r *Reconciler) drainForRestart(ctx context.Context, id string) {
-	r.mu.Lock()
-	rep, ok := r.replicas[id]
-	var shards []Shard
-	if ok {
-		rep.State, rep.DrainingAt, rep.RetiringAt = Draining, r.now(), time.Time{}
-		shards = slices.Clone(rep.Shards)
-	}
-	r.mu.Unlock()
-	for _, s := range shards {
-		if err := r.disp.DrainShard(ctx, s.NodeID, s.ID); err != nil {
-			r.log.Warn("draining a shard failed", "shard", s.ID, "err", err)
-		}
-	}
-}
-
-// nextRestartLocked pops the next queued replica of a model that is still
-// ready to be restarted. Called with mu held.
-func (r *Reconciler) nextRestartLocked(model string) (string, bool) {
-	q := r.restartQueue[model]
-	for len(q) > 0 {
-		id := q[0]
-		q = q[1:]
-		if rep, ok := r.replicas[id]; ok && rep.State == Ready && rep.RetiringAt.IsZero() {
-			r.restartQueue[model] = q
-			return id, true
-		}
-	}
-	delete(r.restartQueue, model)
-	return "", false
-}
-
-// ErrNoSuchReplica means no ready replica has that id.
-var ErrNoSuchReplica = errors.New("no ready replica with that id")
-
-// Restart queues replicas to be replaced, each by a new one started beside it
-// before it is stopped — to pick up an edited profile, an engine key, a new
-// engine version. Replicas of one model are replaced one at a time.
-func (r *Reconciler) Restart(ids ...string) error {
-	r.mu.Lock()
-	for _, id := range ids {
-		rep, ok := r.replicas[id]
-		if !ok || rep.State != Ready {
-			r.mu.Unlock()
-			return fmt.Errorf("%w: %s", ErrNoSuchReplica, id)
-		}
-		if !slices.Contains(r.restartQueue[rep.Model], id) && rep.RetiringAt.IsZero() {
-			r.restartQueue[rep.Model] = append(r.restartQueue[rep.Model], id)
-		}
-	}
-	r.mu.Unlock()
-	r.Notify()
-	return nil
-}
-
-// RestartState says where a replica's restart stands: "queued", "replacing",
-// or "" when none was asked for.
-func (r *Reconciler) RestartState(id string) string {
-	r.mu.Lock()
-	defer r.mu.Unlock()
-	rep, ok := r.replicas[id]
-	switch {
-	case !ok:
-		return ""
-	case !rep.RetiringAt.IsZero():
-		return "replacing"
-	case slices.Contains(r.restartQueue[rep.Model], id):
-		return "queued"
-	}
-	return ""
-}
-
 // scaleUp places and starts one replica per pass.
 //
 // One at a time on purpose: placement decisions depend on what is already
 // allocated, and starting several at once from the same inventory snapshot
 // would hand them the same GPUs.
 func (r *Reconciler) scaleUp(ctx context.Context, m catalog.Model, avoid []string) {
-	res := r.sched.Place(m.PlacementProfiles(), avoid)
+	profiles := r.orderedProfiles(m)
+	res := r.sched.Place(profiles, avoid)
 	if res.Placement == nil && len(avoid) > 0 {
 		// Spreading across failure domains is a preference, not a rule. If it
 		// is the only thing standing in the way, take the placement anyway:
 		// a second replica on one node still survives an engine crash.
-		res = r.sched.Place(m.PlacementProfiles(), nil)
+		res = r.sched.Place(profiles, nil)
 	}

 	r.mu.Lock()
@@ -978,9 +815,22 @@ func (r *Reconciler) scaleUp(ctx context.Context, m catalog.Model, avoid []strin
 		if m.Scaling.OnDemand() && r.makeRoomFor(ctx, m) {
 			return
 		}
+		if r.awaitingPreemption(m.Name) || r.preemptFor(ctx, m, avoid) {
+			return
+		}
 		r.notePlacementFailure(m.Name, res.Explain())
 		return
 	}
+	if why := r.overLimit(m, res.Placement); why != "" {
+		// Recorded as the placement's answer, so the model reads as not
+		// placeable, and says why, wherever that is shown.
+		r.mu.Lock()
+		r.lastExplain[m.Name] = placement.Result{Rejections: []placement.Rejection{
+			{Profile: res.Placement.Profile.Name, Reason: why}}}
+		r.mu.Unlock()
+		r.notePlacementFailure(m.Name, why)
+		return
+	}
 	if err := r.startReplica(ctx, m, res.Placement); err != nil {
 		r.log.Error("starting replica failed", "model", m.Name, "err", err)
 	}
@@ -1028,7 +878,7 @@ func (r *Reconciler) startReplica(ctx context.Context, m catalog.Model, pl *plac
 		})
 		r.inv.Allocate(a.NodeID, inventory.Allocation{
 			ShardID: shardID, ReplicaID: id, GPUIndices: a.GPUIndices,
-			MemoryGB: prof.MinMemoryGB,
+			MemoryGB: prof.MinMemoryGB, CPUCores: cpuCores(prof),
 		})
 	}
 	rep.Endpoint = fmt.Sprintf("http://%s:%d", addrOf(leaderNode), basePort)
@@ -1072,6 +922,7 @@ func (r *Reconciler) startReplica(ctx context.Context, m catalog.Model, pl *plac
 				HTTPPort:      s.HTTPPort,
 				Profile:       espec,
 				APIKey:        r.engineKey(id),
+				Threads:       cpuCores(prof),
 			},
 			StartupTimeout: prof.StartupTimeout,
 			Pool:           pl.Pool,
@@ -1189,253 +1040,6 @@ func (r *Reconciler) stopReplica(ctx context.Context, id string) {
 	r.mu.Unlock()
 }

-// Adopt rebuilds replica state from shards an agent reports it is already
-// running (plan §6.5).
-//
-// This is what makes a control plane restart survivable. Without it the
-// reconciler sees an empty cluster, places duplicates, and the engines already
-// running keep holding GPUs the inventory believes are free — so the duplicates
-// fail to allocate, with an error naming the new process rather than the old.
-//
-// Shards whose model or profile is no longer in the catalog are returned for
-// the caller to stop: nothing will ever claim them.
-func (r *Reconciler) Adopt(nodeID string, shards []AdoptedShard) (unclaimed []AdoptedShard) {
-	defer r.forgetShardsTheAgentDoesNotHave(nodeID, shards)
-
-	byReplica := map[string][]AdoptedShard{}
-	for _, s := range shards {
-		byReplica[s.ReplicaID] = append(byReplica[s.ReplicaID], s)
-	}
-
-	r.mu.Lock()
-	defer r.mu.Unlock()
-
-	for replicaID, group := range byReplica {
-		if existing, known := r.replicas[replicaID]; known {
-			// The other half of a replica we are still assembling. Merge rather
-			// than choose: these shards belong to the same replica as the ones
-			// already adopted, they are simply on a different node whose agent
-			// registered a moment later. Treating them as a competing claim is
-			// what tore multi-node replicas apart on every control plane
-			// restart — one half adopted and failed for being short, the other
-			// stopped as unclaimed when it finally arrived.
-			if !existing.AdoptedAt.IsZero() && len(existing.Shards) < existing.WantShards {
-				r.mergeAdopted(existing, group)
-				continue
-			}
-			// Usually this is an agent reconnecting rather than a control plane
-			// restart, and its status reports keep the state current.
-			//
-			// Not always. Replica ids are derived from the model name and a
-			// counter, so a control plane that restarts and places before the
-			// agents reconnect produces a *different* replica under the same
-			// id, on a freshly allocated port. Skipping the adoption then
-			// leaves the cluster pointing at a port nothing listens on, while
-			// the engine that is actually serving is treated as a stray.
-			//
-			// The fleet is authoritative about what is running (plan §6.5), so
-			// what we merely placed loses to what an agent reports — unless our
-			// copy has a shard that has actually come up, in which case two
-			// things really are running and the one we did not ask for is the
-			// stray.
-			if existing.hasLiveShard() {
-				// Only what we do not recognise. A reconnecting agent reports
-				// the shards we already know, and disowning those would tell it
-				// to stop the very replica that is serving.
-				known := map[string]struct{}{}
-				for _, sh := range existing.Shards {
-					known[sh.ID] = struct{}{}
-				}
-				var strays []AdoptedShard
-				for _, s := range group {
-					if _, ok := known[s.ShardID]; !ok {
-						strays = append(strays, s)
-					}
-				}
-				unclaimed = append(unclaimed, strays...)
-				r.reservePorts(strays)
-				continue
-			}
-			r.log.Warn("a running replica reported by the fleet replaces one we placed",
-				"replica", replicaID, "placed_endpoint", existing.Endpoint)
-			for _, s := range existing.Shards {
-				r.inv.Release(s.ID)
-			}
-			delete(r.replicas, replicaID)
-		}
-
-		first := group[0]
-		m, ok := r.cat.Load().Resolve(first.Model)
-		if !ok {
-			unclaimed = append(unclaimed, group...)
-			r.reservePorts(group)
-			continue
-		}
-		prof, ok := m.ProfileByName(first.Profile)
-		if !ok {
-			unclaimed = append(unclaimed, group...)
-			r.reservePorts(group)
-			continue
-		}
-
-		rep := &Replica{
-			ID: replicaID, Model: m.Name, Profile: prof.Name,
-			Fabric: first.Fabric, Pool: first.Pool,
-			// From the agent, which was told it when the shard started. Empty
-			// from an agent too old to report it, which reads as unknown.
-			ProfileFP: first.ProfileFP,
-			Weight:    prof.Weight, CreatedAt: r.now(),
-			Endpoint: first.Endpoint,
-			// From the profile, not from what was reported: the point is to
-			// notice when what was reported is short.
-			WantShards: max(prof.Nodes, 1),
-			AdoptedAt:  r.now(),
-		}
-		r.mergeAdopted(rep, group)
-		r.replicas[replicaID] = rep
-
-	}
-	return unclaimed
-}
-
-// mergeAdopted folds shards an agent reports into a replica being reassembled.
-//
-// Called both for the first group, which creates the replica, and for every
-// later group belonging to it, so a replica spanning N nodes is built up over N
-// registrations. Shard ids already present are skipped: an agent that
-// reconnects repeats what it is running, and counting those again would carry a
-// replica past its shard count without a second node ever appearing.
-func (r *Reconciler) mergeAdopted(rep *Replica, group []AdoptedShard) {
-	have := make(map[string]struct{}, len(rep.Shards))
-	for _, sh := range rep.Shards {
-		have[sh.ID] = struct{}{}
-	}
-	for _, s := range group {
-		if _, dup := have[s.ShardID]; dup {
-			continue
-		}
-		have[s.ShardID] = struct{}{}
-		rep.Shards = append(rep.Shards, Shard{
-			ID: s.ShardID, NodeID: s.NodeID, Rank: s.Rank,
-			GPUIndices: s.GPUIndices, HCAs: s.HCAs,
-			HTTPPort: s.HTTPPort, State: s.State,
-		})
-		// Reclaim the GPUs and the memory, or the next placement hands them
-		// out twice. The memory comes from the catalog rather than the agent:
-		// what a profile asked for is not something a running process reports.
-		r.inv.Allocate(s.NodeID, inventory.Allocation{
-			ShardID: s.ShardID, ReplicaID: rep.ID, GPUIndices: s.GPUIndices,
-			MemoryGB: r.profileMemory(rep.Model, rep.Profile),
-		})
-		// The endpoint comes from the leader, which may be in any of the
-		// groups — the rank 0 node is not necessarily the first to register.
-		if s.Rank == 0 && s.Endpoint != "" {
-			rep.Endpoint = s.Endpoint
-		}
-	}
-	slices.SortFunc(rep.Shards, func(a, b Shard) int { return a.Rank - b.Rank })
-	rep.deriveState(r.now())
-
-	r.log.Info("adopted replica from the fleet",
-		"replica", rep.ID, "model", rep.Model,
-		"shards", len(rep.Shards), "want", rep.WantShards, "state", rep.State)
-}
-
-// forgetShardsTheAgentDoesNotHave fails shards we believe are on a node that the
-// agent, having just told us everything it is running, did not mention.
-//
-// The mirror image of adoption, and the half that was missing. Adoption handles
-// "the agent has something we do not know about"; this handles "we think the
-// agent has something it does not". Both follow from the same rule: the fleet is
-// authoritative about what is running (plan §6.5).
-//
-// Without it, an engine that dies while its agent survives leaves the control
-// plane believing a replica is Ready forever. Observed on a real cluster: the
-// engine was killed, the agent restarted and registered with nothing running,
-// and the replica stayed Ready pointing at a dead port for as long as anyone
-// watched. Traffic survived — the breaker took the endpoint out — but the
-// cluster ran a third short with no sign of it.
-//
-// Safe against the obvious race. A registration follows a fresh connection, so
-// anything we dispatched over a previous one is not running by definition;
-// there is no window where a shard is legitimately started but unreported.
-func (r *Reconciler) forgetShardsTheAgentDoesNotHave(nodeID string, running []AdoptedShard) {
-	if nodeID == "" {
-		return
-	}
-	reported := make(map[string]struct{}, len(running))
-	for _, s := range running {
-		reported[s.ShardID] = struct{}{}
-	}
-
-	var lost []string
-	r.mu.Lock()
-	for _, rep := range r.replicas {
-		if rep.State == Stopped {
-			continue
-		}
-		changed := false
-		for i := range rep.Shards {
-			sh := &rep.Shards[i]
-			if sh.NodeID != nodeID || sh.State.Terminal() {
-				continue
-			}
-			if _, ok := reported[sh.ID]; ok {
-				continue
-			}
-			sh.State = supervisor.Failed
-			sh.Err = "the agent reports it is not running"
-			r.inv.Release(sh.ID)
-			changed = true
-			lost = append(lost, sh.ID)
-		}
-		if changed {
-			rep.deriveState(r.now())
-		}
-	}
-	r.mu.Unlock()
-
-	for _, id := range lost {
-		r.log.Warn("a shard we believed was running is not; the agent does not have it",
-			"node", nodeID, "shard", id)
-	}
-}
-
-// reservePorts records ports held by shards nothing owns.
-//
-// They are being stopped, but stopping is not instant and the engine keeps its
-// socket until it exits. Handing the port to a new shard in the meantime makes
-// that shard fail to bind, which fails its replica, which places another one on
-// the same port — a loop that does not end on its own.
-//
-// Called with the lock held.
-func (r *Reconciler) reservePorts(shards []AdoptedShard) {
-	for _, s := range shards {
-		if s.HTTPPort > 0 {
-			r.reservedPorts[s.HTTPPort] = struct{}{}
-		}
-	}
-}
-
-// AdoptedShard is one running shard an agent reported at registration.
-type AdoptedShard struct {
-	NodeID     string
-	ReplicaID  string
-	ShardID    string
-	Rank       int
-	GPUIndices []int
-	HCAs       []string
-	HTTPPort   int
-	State      supervisor.State
-	Model      string
-	Profile    string
-	Pool       string
-	Fabric     string
-	ProfileFP  string
-	// Endpoint is the leader's base URL, set on the rank 0 shard.
-	Endpoint string
-}
-
 // OnFetchProgress records how far a shard's download has got.
 //
 // Separate from OnShardStatus because it is not a transition: the shard is
@@ -1558,16 +1162,31 @@ func addrOf(n inventory.Node) string {
 // claim on restart would place a second large model onto a node already full of
 // the first.
 func (r *Reconciler) profileMemory(model, profile string) int {
+	p, _ := r.profileOf(model, profile)
+	return p.MinMemoryGB
+}
+
+// profileOf is a replica's profile as the catalog holds it now.
+func (r *Reconciler) profileOf(model, profile string) (catalog.Profile, bool) {
 	m, ok := r.cat.Load().Resolve(model)
 	if !ok {
-		return 0
+		return catalog.Profile{}, false
 	}
 	for _, p := range m.Profiles {
 		if p.Name == profile {
-			return p.MinMemoryGB
+			return p, true
 		}
 	}
-	return 0
+	return catalog.Profile{}, false
+}
+
+// cpuCores is what a profile reserves of a node's CPUs: its cores when it
+// computes on them, nothing when a GPU does the work.
+func cpuCores(p catalog.Profile) int {
+	if !p.CPUOnly() {
+		return 0
+	}
+	return p.CPUCores
 }

 func (r *Reconciler) engineKey(replicaID string) string {
@@ -1576,142 +1195,3 @@ func (r *Reconciler) engineKey(replicaID string) string {
 	}
 	return r.cfg.EngineKey(replicaID)
 }
-
-// parkRecord is why a model stopped being retried.
-type parkRecord struct {
-	Since    time.Time
-	Reason   string
-	Failures int
-	// fp is the model's definition when it was parked; a different one
-	// unparks it.
-	fp string
-}
-
-type placementNote struct {
-	text string
-	at   time.Time
-}
-
-// modelFingerprint identifies a model's definition, so a change to it can be
-// told from a retry of the same thing.
-func modelFingerprint(m catalog.Model) string {
-	b, _ := json.Marshal(m)
-	sum := sha256.Sum256(b)
-	return hex.EncodeToString(sum[:8])
-}
-
-// placementLogEvery is how often an unchanged placement failure is repeated.
-const placementLogEvery = 10 * time.Minute
-
-// notePlacementFailure logs why a model could not be placed, when the reason
-// is new or has not been said for a while. Said on every pass it was 250
-// identical lines in two days for one model, burying everything else.
-func (r *Reconciler) notePlacementFailure(model, why string) {
-	r.mu.Lock()
-	last := r.placementLogged[model]
-	say := last.text != why || r.now().Sub(last.at) >= placementLogEvery
-	if say {
-		r.placementLogged[model] = placementNote{text: why, at: r.now()}
-	}
-	r.mu.Unlock()
-	if say {
-		r.log.Warn("no placement for model", "model", model, "reasons", why)
-	}
-}
-
-// Retry clears a parked model and its backoff, so it is placed on the next
-// pass.
-func (r *Reconciler) Retry(model string) bool {
-	r.mu.Lock()
-	_, was := r.parked[model]
-	delete(r.parked, model)
-	r.attempts[model] = 0
-	delete(r.nextAttempt, model)
-	r.mu.Unlock()
-	r.Notify()
-	return was
-}
-
-// ModelHealth is how one model stands, for status and metrics.
-type ModelHealth struct {
-	Model string
-	// Desired is how many replicas it should have now; 0 for a stopped model
-	// or an on-demand one nobody is asking for.
-	Desired int
-	Ready   int
-	// Failures counts replicas that failed in a row.
-	Failures int
-	// Parked is set when Failures reached ParkAfter; Reason is the last error.
-	Parked      bool
-	ParkedSince time.Time
-	Reason      string
-	// Unplaceable is set when it is short of replicas and the last attempt to
-	// place one found nowhere; Placement says why.
-	Unplaceable bool
-	Placement   string
-}
-
-// ModelHealth reports every model in the catalog.
-func (r *Reconciler) ModelHealth() []ModelHealth {
-	models := r.cat.Load().Models()
-	desired := map[string]int{}
-	for _, m := range models {
-		desired[m.Name] = r.desiredFor(m)
-	}
-	r.mu.Lock()
-	defer r.mu.Unlock()
-	ready, live := map[string]int{}, map[string]int{}
-	for _, rep := range r.replicas {
-		if rep.State == Ready {
-			ready[rep.Model]++
-		}
-		if rep.State.Counts() {
-			live[rep.Model]++
-		}
-	}
-	out := make([]ModelHealth, 0, len(models))
-	for _, m := range models {
-		h := ModelHealth{Model: m.Name, Desired: desired[m.Name], Ready: ready[m.Name],
-			Failures: r.attempts[m.Name]}
-		if p, ok := r.parked[m.Name]; ok {
-			h.Parked, h.ParkedSince, h.Reason = true, p.Since, p.Reason
-		}
-		if res, ok := r.lastExplain[m.Name]; ok && res.Placement == nil && live[m.Name] < h.Desired {
-			h.Unplaceable, h.Placement = true, res.Explain()
-		}
-		out = append(out, h)
-	}
-	return out
-}
-
-// Unservable says why a model cannot be served at all, so a request for it is
-// told now rather than asked to retry something that will never come up.
-func (r *Reconciler) Unservable(model string) (string, bool) {
-	m, ok := r.cat.Load().Resolve(model)
-	if !ok {
-		return "", false
-	}
-	r.mu.Lock()
-	defer r.mu.Unlock()
-	if p, ok := r.parked[m.Name]; ok {
-		return fmt.Sprintf("%s failed to start %d times in a row and is not being retried: %s",
-			m.Name, p.Failures, p.Reason), true
-	}
-	for _, rep := range r.replicas {
-		if rep.Model == m.Name && rep.State.Counts() {
-			return "", false // something is on its way
-		}
-	}
-	if res, ok := r.lastExplain[m.Name]; ok && res.Placement == nil {
-		return m.Name + " cannot be placed on any node: " + firstLine(res.Explain()), true
-	}
-	return "", false
-}
-
-func firstLine(s string) string {
-	s = strings.TrimSpace(s)
-	if i := strings.IndexByte(s, '\n'); i >= 0 {
-		return strings.TrimSpace(s[:i]) + " (and others; see llmgw ctl explain-placement)"
-	}
-	return s
-}
@@ -1932,3 +1932,74 @@ func TestAPlacementFailureIsLoggedWhenItChangesNotEveryPass(t *testing.T) {
 		t.Errorf("health %+v", mh)
 	}
 }
+
+func liveCount(h *harness) int {
+	n := 0
+	for _, r := range h.r.Replicas() {
+		if r.State.Counts() {
+			n++
+		}
+	}
+	return n
+}
+
+func TestLoadAddsReplicasUpToMaxAndQuietTakesThemBack(t *testing.T) {
+	m := testModel(1)
+	m.Scaling.Max = 3
+	h := newHarness(t, m, gpuNode("1", 8), gpuNode("2", 8), gpuNode("3", 8))
+	h.r.Reconcile(t.Context())
+	h.markReady()
+	busy := map[string]Load{"llama": {Replicas: 1, Running: 4, Waiting: 3}}
+
+	// A spike shorter than ScaleUpAfter adds nothing.
+	h.r.RecordLoad(busy)
+	h.advance(30 * time.Second)
+	h.r.RecordLoad(busy)
+	h.r.Reconcile(t.Context())
+	if n := liveCount(h); n != 1 {
+		t.Fatalf("%d replicas after 30s of load", n)
+	}
+
+	// Held past it, one more.
+	h.advance(31 * time.Second)
+	h.r.RecordLoad(busy)
+	h.r.Reconcile(t.Context())
+	if n := liveCount(h); n != 2 {
+		t.Fatalf("%d replicas after a minute of load; expected 2", n)
+	}
+
+	// Not another while that one is still starting.
+	h.advance(2 * time.Minute)
+	h.r.RecordLoad(busy)
+	h.r.Reconcile(t.Context())
+	if n := liveCount(h); n != 2 {
+		t.Fatalf("stacked a replica on one still starting: %d", n)
+	}
+
+	// Once it is up and load holds, the third — and never a fourth.
+	h.markReady()
+	for i := 0; i < 5; i++ {
+		h.r.RecordLoad(busy)
+		h.advance(61 * time.Second)
+		h.r.RecordLoad(busy)
+		h.r.Reconcile(t.Context())
+		h.markReady()
+	}
+	if n := liveCount(h); n != 3 {
+		t.Fatalf("%d replicas under sustained load; max is 3", n)
+	}
+
+	// Quiet: back down one step per ScaleDownAfter, to min and no further.
+	quiet := map[string]Load{"llama": {Replicas: 3}}
+	for i := 0; i < 6; i++ {
+		h.r.RecordLoad(quiet)
+		h.advance(DefaultConfig().ScaleDownAfter + time.Second)
+		h.r.RecordLoad(quiet)
+		for j := 0; j < 3; j++ {
+			h.r.Reconcile(t.Context())
+		}
+	}
+	if n := liveCount(h); n != 1 {
+		t.Errorf("%d replicas after a long quiet spell; min is 1", n)
+	}
+}
@@ -156,6 +156,8 @@ type Replica struct {
 	// DrainingAt is when the replica stopped taking new work, so the wait for a
 	// replacement can be bounded.
 	DrainingAt time.Time
+	// PreemptedFor is the model this replica was stopped to make room for.
+	PreemptedFor string
 	// RetiringAt is set while the replica is being restarted: it keeps serving
 	// but no longer counts toward the model's replicas, so a replacement is
 	// placed beside it, and it is drained only once that replacement is ready.
@@ -0,0 +1,113 @@
+package reconciler
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"slices"
+	"time"
+)
+
+// retire finishes restarting replicas.
+//
+// A retiring replica keeps serving until its replacement is ready, and is then
+// drained like any other. When no replacement could be placed at all —
+// nowhere else has the memory, which on a small CPU fleet is the usual case —
+// it is stopped after DrainGrace so its replacement can take its place: a
+// restart in place, with the model unserved while it loads if it had no other
+// replica. Said in the log, since that is the one case with an outage.
+func (r *Reconciler) retire(ctx context.Context, ids []string, readyLive, live, desired int) {
+	for _, id := range ids {
+		r.mu.Lock()
+		rep, ok := r.replicas[id]
+		var since time.Time
+		if ok {
+			since = rep.RetiringAt
+		}
+		r.mu.Unlock()
+		if !ok {
+			continue
+		}
+		switch {
+		case readyLive >= desired:
+			r.drainForRestart(ctx, id)
+		case live < desired && r.now().Sub(since) > r.cfg.DrainGrace:
+			r.log.Warn("restarting a replica in place: no room for its replacement beside it",
+				"replica", id, "after", r.cfg.DrainGrace)
+			r.drainForRestart(ctx, id)
+			r.stopReplica(ctx, id)
+		}
+	}
+}
+
+func (r *Reconciler) drainForRestart(ctx context.Context, id string) {
+	r.mu.Lock()
+	rep, ok := r.replicas[id]
+	var shards []Shard
+	if ok {
+		rep.State, rep.DrainingAt, rep.RetiringAt = Draining, r.now(), time.Time{}
+		shards = slices.Clone(rep.Shards)
+	}
+	r.mu.Unlock()
+	for _, s := range shards {
+		if err := r.disp.DrainShard(ctx, s.NodeID, s.ID); err != nil {
+			r.log.Warn("draining a shard failed", "shard", s.ID, "err", err)
+		}
+	}
+}
+
+// nextRestartLocked pops the next queued replica of a model that is still
+// ready to be restarted. Called with mu held.
+func (r *Reconciler) nextRestartLocked(model string) (string, bool) {
+	q := r.restartQueue[model]
+	for len(q) > 0 {
+		id := q[0]
+		q = q[1:]
+		if rep, ok := r.replicas[id]; ok && rep.State == Ready && rep.RetiringAt.IsZero() {
+			r.restartQueue[model] = q
+			return id, true
+		}
+	}
+	delete(r.restartQueue, model)
+	return "", false
+}
+
+// ErrNoSuchReplica means no ready replica has that id.
+var ErrNoSuchReplica = errors.New("no ready replica with that id")
+
+// Restart queues replicas to be replaced, each by a new one started beside it
+// before it is stopped — to pick up an edited profile, an engine key, a new
+// engine version. Replicas of one model are replaced one at a time.
+func (r *Reconciler) Restart(ids ...string) error {
+	r.mu.Lock()
+	for _, id := range ids {
+		rep, ok := r.replicas[id]
+		if !ok || rep.State != Ready {
+			r.mu.Unlock()
+			return fmt.Errorf("%w: %s", ErrNoSuchReplica, id)
+		}
+		if !slices.Contains(r.restartQueue[rep.Model], id) && rep.RetiringAt.IsZero() {
+			r.restartQueue[rep.Model] = append(r.restartQueue[rep.Model], id)
+		}
+	}
+	r.mu.Unlock()
+	r.Notify()
+	return nil
+}
+
+// RestartState says where a replica's restart stands: "queued", "replacing",
+// or "" when none was asked for.
+func (r *Reconciler) RestartState(id string) string {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	rep, ok := r.replicas[id]
+	switch {
+	case !ok:
+		return ""
+	case !rep.RetiringAt.IsZero():
+		return "replacing"
+	case slices.Contains(r.restartQueue[rep.Model], id):
+		return "queued"
+	}
+	return ""
+}
@@ -14,7 +14,7 @@ import (
 // a new configuration file is a failure here rather than a silent omission.
 var Sources = []string{
 	"internal/config/config.go",
-	"internal/agent/agent.go",
+	"internal/agent/config.go",
 	"internal/agent/discovery/discover.go",
 	"internal/agent/engines/engine.go",
 	"internal/agent/engines/llamacpp.go",
@@ -25,6 +25,10 @@ type Header struct {
 	Version      uint32
 	Architecture string
 	Name         string
+	// Type is general.type: "model", or "adapter" for a LoRA adapter, which
+	// is applied to a model rather than run on its own. Empty in files made
+	// before the key existed, which are models.
+	Type string

 	// The shape the KV cache is sized from, from the architecture's own keys
 	// (llama.block_count, qwen2.attention.head_count_kv, ...). Zero when the
@@ -235,6 +239,8 @@ func Read(r io.Reader) (Header, error) {
 			h.Architecture = v
 		case "general.name":
 			h.Name = v
+		case "general.type":
+			h.Type = v
 		}
 		if h.Architecture != "" {
 			switch strings.TrimPrefix(key, h.Architecture+".") {
@@ -290,6 +296,41 @@ func FromURL(ctx context.Context, client *http.Client, url string) (Header, erro
 	return Read(io.LimitReader(resp.Body, headerBytes))
 }

+// IsAdapter reports whether the file is a LoRA adapter rather than a model.
+func (h Header) IsAdapter() bool { return h.Type == "adapter" }
+
+// adapterProbe is how much of a file is read to tell an adapter from a model:
+// general.type comes before the tokenizer, which is what makes a model's
+// header megabytes long.
+const adapterProbe = 256 << 10
+
+// KindFromURL reads just enough of a GGUF file to say what it is — its type
+// and architecture — over HTTP. A header cut short by the probe is fine: the
+// keys asked for come first.
+func KindFromURL(ctx context.Context, client *http.Client, url string) (Header, error) {
+	if client == nil {
+		client = &http.Client{Timeout: 30 * time.Second}
+	}
+	req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
+	if err != nil {
+		return Header{}, err
+	}
+	req.Header.Set("Range", fmt.Sprintf("bytes=0-%d", adapterProbe-1))
+	resp, err := client.Do(req)
+	if err != nil {
+		return Header{}, err
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode != http.StatusOK && resp.StatusCode != http.StatusPartialContent {
+		return Header{}, fmt.Errorf("%s answered %s", url, resp.Status)
+	}
+	h, err := Read(io.LimitReader(resp.Body, adapterProbe))
+	if h.Architecture != "" {
+		return h, nil // what was asked for was read, whatever came after
+	}
+	return h, err
+}
+
 // IsGGUF reports whether a source names a GGUF file, or a directory of them.
 func IsGGUF(source string) bool {
 	s := strings.ToLower(strings.TrimSuffix(source, "/"))
@@ -0,0 +1,31 @@
+package gguf
+
+import (
+	"os"
+	"testing"
+)
+
+// Run by hand against the real adapter this was written for:
+//
+//	LLMGW_LIVE_GGUF=1 go test ./internal/gguf -run LiveAdapter -v
+func TestLiveAdapterIsAnAdapter(t *testing.T) {
+	if os.Getenv("LLMGW_LIVE_GGUF") == "" {
+		t.Skip("set LLMGW_LIVE_GGUF to read a real adapter from Hugging Face")
+	}
+	h, err := KindFromURL(t.Context(), nil,
+		"https://huggingface.co/ggml-org/LoRA-Qwen2.5-Coder-7B-Instruct-F16-GGUF/resolve/main/LoRA-Qwen2.5-Coder-7B-Instruct-f16.gguf")
+	if err != nil {
+		t.Fatal(err)
+	}
+	if !h.IsAdapter() || h.Architecture != "qwen2" {
+		t.Errorf("%+v", h)
+	}
+	m, err := KindFromURL(t.Context(), nil,
+		"https://huggingface.co/bartowski/Qwen2.5-0.5B-Instruct-GGUF/resolve/main/Qwen2.5-0.5B-Instruct-Q4_K_M.gguf")
+	if err != nil {
+		t.Fatal(err)
+	}
+	if m.IsAdapter() || m.Architecture != "qwen2" {
+		t.Errorf("a model read as %+v", m)
+	}
+}
@@ -0,0 +1,58 @@
+package gguf
+
+import (
+	"bytes"
+	"encoding/binary"
+	"net/http"
+	"net/http/httptest"
+	"testing"
+)
+
+// adapterHeader is the start of a LoRA adapter as llama.cpp's converter
+// writes it, followed by padding that stands in for its tensors.
+func adapterHeader(kind string, pad int) []byte {
+	var b bytes.Buffer
+	w := func(v any) { _ = binary.Write(&b, binary.LittleEndian, v) }
+	str := func(s string) { w(uint64(len(s))); b.WriteString(s) }
+	b.WriteString("GGUF")
+	w(uint32(3))
+	w(uint64(0))
+	w(uint64(3))
+	str("general.architecture")
+	w(uint32(tString))
+	str("qwen2")
+	str("general.type")
+	w(uint32(tString))
+	str(kind)
+	str("adapter.type")
+	w(uint32(tString))
+	str("lora")
+	b.Write(make([]byte, pad))
+	return b.Bytes()
+}
+
+func TestAnAdapterIsToldFromAModelByItsFirstBytes(t *testing.T) {
+	var asked string
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		asked = r.Header.Get("Range")
+		w.WriteHeader(http.StatusPartialContent)
+		if r.URL.Path == "/adapter.gguf" {
+			_, _ = w.Write(adapterHeader("adapter", 1<<20))
+			return
+		}
+		_, _ = w.Write(header("qwen2"))
+	}))
+	defer srv.Close()
+
+	h, err := KindFromURL(t.Context(), srv.Client(), srv.URL+"/adapter.gguf")
+	if err != nil || !h.IsAdapter() || h.Architecture != "qwen2" {
+		t.Errorf("an adapter read as %+v, %v", h, err)
+	}
+	if asked != "bytes=0-262143" {
+		t.Errorf("asked for %q: the probe should read the start only", asked)
+	}
+	m, err := KindFromURL(t.Context(), srv.Client(), srv.URL+"/model.gguf")
+	if err != nil || m.IsAdapter() || m.Architecture != "qwen2" {
+		t.Errorf("a model read as %+v, %v", m, err)
+	}
+}
@@ -156,6 +156,10 @@ type Query struct {
 	// applies it, so a page of small models is a full page.
 	MaxParams int64
 	Limit     int
+	// Filters are further registry tags every result must carry, such as
+	// "lora", or "base_model:adapter:Qwen/Qwen2.5-7B-Instruct" for the
+	// adapters made for one model.
+	Filters []string
 }

 // exploreFields is everything a listing shows. Asking for any field with
@@ -188,7 +192,10 @@ func (c *Client) Explore(ctx context.Context, q Query) ([]Model, error) {
 		v.Set("pipeline_tag", q.Task)
 	}
 	if q.GGUF {
-		v.Set("filter", "gguf")
+		v.Add("filter", "gguf")
+	}
+	for _, f := range q.Filters {
+		v.Add("filter", f)
 	}
 	if q.MaxParams > 0 {
 		v.Set("num_parameters", fmt.Sprintf("min:0,max:%d", q.MaxParams))
@@ -558,3 +565,16 @@ func Group(files []File) []FileSet {
 	}
 	return out
 }
+
+// ModelName is the catalog name a repository suggests: its last element,
+// lowercased, with the -GGUF marker dropped. The owner is not part of what
+// anybody calls the model, and "-GGUF" describes the file format.
+func ModelName(repoID string) string {
+	name := repoID
+	if i := strings.LastIndex(name, "/"); i >= 0 {
+		name = name[i+1:]
+	}
+	name = strings.TrimSuffix(name, "-GGUF")
+	name = strings.TrimSuffix(name, "-gguf")
+	return strings.ToLower(name)
+}
@@ -0,0 +1,72 @@
+package obs
+
+import (
+	"context"
+	"fmt"
+	"net/url"
+	"os"
+	"time"
+
+	"go.opentelemetry.io/otel"
+	"go.opentelemetry.io/otel/attribute"
+	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
+	"go.opentelemetry.io/otel/propagation"
+	"go.opentelemetry.io/otel/sdk/resource"
+	sdktrace "go.opentelemetry.io/otel/sdk/trace"
+)
+
+// TracingOptions says where traces go.
+type TracingOptions struct {
+	Endpoint    string  // http(s)://host:4318; empty reads OTEL_EXPORTER_OTLP_ENDPOINT
+	SampleRatio float64 // of traces the caller did not already decide; zero takes 1
+	Service     string
+	Version     string
+}
+
+// SetupTracing installs the global tracer provider and the W3C propagator,
+// and returns what flushes it on the way out. With no endpoint it installs
+// nothing, and every span is the no-op one: tracing that is off costs a
+// function call per span.
+func SetupTracing(ctx context.Context, o TracingOptions) (func(context.Context) error, error) {
+	noop := func(context.Context) error { return nil }
+	endpoint := o.Endpoint
+	if endpoint == "" {
+		endpoint = os.Getenv("OTEL_EXPORTER_OTLP_ENDPOINT")
+	}
+	if endpoint == "" {
+		return noop, nil
+	}
+	u, err := url.Parse(endpoint)
+	if err != nil || (u.Scheme != "http" && u.Scheme != "https") || u.Host == "" {
+		return noop, fmt.Errorf("tracing endpoint %q is not an http(s) address", endpoint)
+	}
+	opts := []otlptracehttp.Option{otlptracehttp.WithEndpoint(u.Host)}
+	if u.Scheme == "http" {
+		opts = append(opts, otlptracehttp.WithInsecure())
+	}
+	if u.Path != "" && u.Path != "/" {
+		opts = append(opts, otlptracehttp.WithURLPath(u.Path))
+	}
+	exp, err := otlptracehttp.New(ctx, opts...)
+	if err != nil {
+		return noop, err
+	}
+	ratio := o.SampleRatio
+	if ratio <= 0 || ratio > 1 {
+		ratio = 1
+	}
+	res := resource.NewSchemaless(
+		attribute.String("service.name", o.Service),
+		attribute.String("service.version", o.Version),
+	)
+	tp := sdktrace.NewTracerProvider(
+		sdktrace.WithBatcher(exp, sdktrace.WithBatchTimeout(5*time.Second)),
+		sdktrace.WithResource(res),
+		// A caller that sent a sampled traceparent has already decided; the
+		// ratio is for requests that arrive without one.
+		sdktrace.WithSampler(sdktrace.ParentBased(sdktrace.TraceIDRatioBased(ratio))),
+	)
+	otel.SetTracerProvider(tp)
+	otel.SetTextMapPropagator(propagation.TraceContext{})
+	return tp.Shutdown, nil
+}
@@ -0,0 +1,19 @@
+package obs
+
+import (
+	"context"
+	"testing"
+)
+
+func TestTracingIsOffUntilGivenSomewhereToSend(t *testing.T) {
+	t.Setenv("OTEL_EXPORTER_OTLP_ENDPOINT", "")
+	stop, err := SetupTracing(context.Background(), TracingOptions{Service: "t"})
+	if err != nil || stop(context.Background()) != nil {
+		t.Fatal(err)
+	}
+	for _, bad := range []string{"otel:4318", "ftp://otel:4318", "http://"} {
+		if _, err := SetupTracing(context.Background(), TracingOptions{Endpoint: bad}); err == nil {
+			t.Errorf("%q accepted", bad)
+		}
+	}
+}
@@ -498,14 +498,17 @@ func (r *Runner) certificate(ctx context.Context, res *Result) error {
 			return err
 		}
 	}
-	// Trusting the root too, so other tools on the machine do not treat the
-	// cluster's certificates as strangers. Best effort: either family's way.
-	_, _ = r.Remote.Run(ctx, `if [ -d /usr/local/share/ca-certificates ]; then
-  sudo install -m 0644 /etc/llmgw/ca.pem /usr/local/share/ca-certificates/llmgw-ca.crt && sudo update-ca-certificates >/dev/null 2>&1
-elif [ -d /etc/pki/ca-trust/source/anchors ]; then
-  sudo install -m 0644 /etc/llmgw/ca.pem /etc/pki/ca-trust/source/anchors/llmgw-ca.pem && sudo update-ca-trust >/dev/null 2>&1
+	// The agent trusts the cluster's root through its own configuration only.
+	// Not system-wide: that root signs whatever the control plane asks it to,
+	// and every TLS client on the machine would then take it for any site.
+	// A node onboarded by an older version had it added; take it back out.
+	_, _ = r.Remote.Run(ctx, `if [ -f /usr/local/share/ca-certificates/llmgw-ca.crt ]; then
+  sudo rm -f /usr/local/share/ca-certificates/llmgw-ca.crt && sudo update-ca-certificates --fresh >/dev/null 2>&1
+fi
+if [ -f /etc/pki/ca-trust/source/anchors/llmgw-ca.pem ]; then
+  sudo rm -f /etc/pki/ca-trust/source/anchors/llmgw-ca.pem && sudo update-ca-trust >/dev/null 2>&1
 fi; true`, nil)
-	r.emit(StepCert, Done, "certificate for "+res.Node+" installed, the cluster's root trusted", "")
+	r.emit(StepCert, Done, "certificate for "+res.Node+" installed", "")
 	return nil
 }

@@ -9,6 +9,7 @@ package release

 import (
 	"bufio"
+	"bytes"
 	"context"
 	"crypto/sha256"
 	"encoding/hex"
@@ -51,7 +52,7 @@ func Install(ctx context.Context, client *http.Client, base, version, goos, goar
 	asset := AssetName(version, goos, goarch)
 	root := strings.TrimSuffix(base, "/") + "/" + version + "/"

-	want, err := checksum(ctx, client, root+"checksums.txt", asset)
+	want, err := checksum(ctx, client, root, asset)
 	if err != nil {
 		return "", err
 	}
@@ -85,13 +86,35 @@ func Install(ctx context.Context, client *http.Client, base, version, goos, goar
 	return backup, nil
 }

-func checksum(ctx context.Context, client *http.Client, url, asset string) (string, error) {
-	body, err := get(ctx, client, url)
+// checksum reads the release's checksum for asset — after checking the
+// checksums' signature, when this build trusts a signing key. A release with
+// no signature is then refused rather than taken on its checksums: a missing
+// signature is exactly what replacing a release would look like.
+func checksum(ctx context.Context, client *http.Client, root, asset string) (string, error) {
+	body, err := get(ctx, client, root+"checksums.txt")
 	if err != nil {
 		return "", err
 	}
 	defer body.Close()
-	sc := bufio.NewScanner(io.LimitReader(body, 1<<20))
+	sums, err := io.ReadAll(io.LimitReader(body, 1<<20))
+	if err != nil {
+		return "", err
+	}
+	if key := TrustedKey(); key != "" {
+		sigBody, err := get(ctx, client, root+SignatureName)
+		if err != nil {
+			return "", fmt.Errorf("this build only installs signed releases, and %s could not be read: %w", SignatureName, err)
+		}
+		sig, err := io.ReadAll(io.LimitReader(sigBody, 4096))
+		sigBody.Close()
+		if err != nil {
+			return "", err
+		}
+		if err := Verify(key, sums, string(sig)); err != nil {
+			return "", err
+		}
+	}
+	sc := bufio.NewScanner(bytes.NewReader(sums))
 	for sc.Scan() {
 		f := strings.Fields(sc.Text())
 		if len(f) == 2 && f[1] == asset && len(f[0]) == 64 {
@@ -0,0 +1,72 @@
+package release
+
+import (
+	"crypto/ed25519"
+	"crypto/rand"
+	_ "embed"
+	"encoding/base64"
+	"errors"
+	"fmt"
+	"os"
+	"strings"
+)
+
+// trustedPub is the public key releases are signed with, base64, built into
+// every binary from the repository. Empty — as it is until a key is made —
+// trusts a release on its checksums alone, as every release before this did.
+//
+// Built in rather than fetched, because the point of a signature is that the
+// server the release comes from cannot vouch for it: a machine that already
+// runs llmgw checks the next release against the key the release it runs was
+// built with, so replacing both the binaries and the checksums on the server
+// is no longer enough.
+//
+//go:embed trusted.pub
+var trustedPub string
+
+// TrustedKey is the key releases are verified with: LLMGW_RELEASE_KEY when set
+// (for rotating a key, or testing), the built-in one otherwise; empty is none.
+func TrustedKey() string {
+	if k := strings.TrimSpace(os.Getenv("LLMGW_RELEASE_KEY")); k != "" {
+		return k
+	}
+	return strings.TrimSpace(trustedPub)
+}
+
+// SignatureName is the file a release's checksums are signed in.
+const SignatureName = "checksums.txt.sig"
+
+// GenerateKey makes a signing key pair, both base64: the private key is the
+// 32-byte seed, which is what goes into the CI secret.
+func GenerateKey() (public, private string, err error) {
+	pub, priv, err := ed25519.GenerateKey(rand.Reader)
+	if err != nil {
+		return "", "", err
+	}
+	return base64.StdEncoding.EncodeToString(pub), base64.StdEncoding.EncodeToString(priv.Seed()), nil
+}
+
+// Sign signs data with a base64 seed, returning the signature in base64.
+func Sign(privateSeed string, data []byte) (string, error) {
+	seed, err := base64.StdEncoding.DecodeString(strings.TrimSpace(privateSeed))
+	if err != nil || len(seed) != ed25519.SeedSize {
+		return "", errors.New("the signing key is not a base64 ed25519 seed")
+	}
+	return base64.StdEncoding.EncodeToString(ed25519.Sign(ed25519.NewKeyFromSeed(seed), data)), nil
+}
+
+// Verify checks a base64 signature of data against a base64 public key.
+func Verify(public string, data []byte, signature string) error {
+	pub, err := base64.StdEncoding.DecodeString(strings.TrimSpace(public))
+	if err != nil || len(pub) != ed25519.PublicKeySize {
+		return errors.New("the trusted release key is not a base64 ed25519 public key")
+	}
+	sig, err := base64.StdEncoding.DecodeString(strings.TrimSpace(signature))
+	if err != nil {
+		return fmt.Errorf("the release signature is not base64: %w", err)
+	}
+	if !ed25519.Verify(pub, data, sig) {
+		return errors.New("the release's checksums are not signed by the trusted key")
+	}
+	return nil
+}
@@ -0,0 +1,99 @@
+package release
+
+import (
+	"crypto/sha256"
+	"encoding/hex"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+)
+
+// signedServer publishes a release whose checksums are signed with seed,
+// or not signed at all when seed is empty; tamper rewrites the checksums
+// after signing.
+func signedServer(t *testing.T, seed string, body []byte, tamper bool) (*httptest.Server, string) {
+	t.Helper()
+	asset := AssetName("v2.0.0", "linux", "amd64")
+	sum := sha256.Sum256(body)
+	sums := hex.EncodeToString(sum[:]) + "  " + asset + "\n"
+	sig := ""
+	if seed != "" {
+		var err error
+		if sig, err = Sign(seed, []byte(sums)); err != nil {
+			t.Fatal(err)
+		}
+	}
+	if tamper {
+		sums += strings.Repeat("0", 64) + "  extra-file\n"
+	}
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		switch r.URL.Path {
+		case "/v2.0.0/checksums.txt":
+			_, _ = w.Write([]byte(sums))
+		case "/v2.0.0/" + SignatureName:
+			if sig == "" {
+				http.NotFound(w, r)
+				return
+			}
+			_, _ = w.Write([]byte(sig + "\n"))
+		case "/v2.0.0/" + asset:
+			_, _ = w.Write(body)
+		default:
+			http.NotFound(w, r)
+		}
+	}))
+	t.Cleanup(srv.Close)
+	return srv, asset
+}
+
+func TestABuildThatTrustsAKeyInstallsOnlyReleasesSignedWithIt(t *testing.T) {
+	pub, priv, err := GenerateKey()
+	if err != nil {
+		t.Fatal(err)
+	}
+	_, otherPriv, _ := GenerateKey()
+	t.Setenv("LLMGW_RELEASE_KEY", pub)
+	install := func(srv *httptest.Server) error {
+		target := filepath.Join(t.TempDir(), "llmgw")
+		_ = os.WriteFile(target, []byte("old"), 0o755)
+		_, err := Install(t.Context(), srv.Client(), srv.URL, "v2.0.0", "linux", "amd64", target)
+		return err
+	}
+	body := []byte("new binary")
+
+	good, _ := signedServer(t, priv, body, false)
+	if err := install(good); err != nil {
+		t.Fatalf("a release signed with the trusted key was refused: %v", err)
+	}
+	for name, srv := range map[string]*httptest.Server{
+		"unsigned":            func() *httptest.Server { s, _ := signedServer(t, "", body, false); return s }(),
+		"signed by another":   func() *httptest.Server { s, _ := signedServer(t, otherPriv, body, false); return s }(),
+		"checksums rewritten": func() *httptest.Server { s, _ := signedServer(t, priv, body, true); return s }(),
+	} {
+		if err := install(srv); err == nil {
+			t.Errorf("%s: installed", name)
+		}
+	}
+
+	// A build that trusts no key installs on checksums, as before.
+	t.Setenv("LLMGW_RELEASE_KEY", "")
+	if TrustedKey() != "" {
+		t.Skip("this build embeds a trusted key; the unsigned path cannot be tested here")
+	}
+	unsigned, _ := signedServer(t, "", body, false)
+	if err := install(unsigned); err != nil {
+		t.Errorf("without a trusted key an unsigned release was refused: %v", err)
+	}
+}
+
+func TestAKeyThatIsNotOneIsSaid(t *testing.T) {
+	if _, err := Sign("not-a-key", []byte("x")); err == nil {
+		t.Error("signed with a malformed key")
+	}
+	if err := Verify("bm9wZQ==", []byte("x"), "c2ln"); err == nil || !strings.Contains(err.Error(), "public key") {
+		t.Errorf("a malformed trusted key: %v", err)
+	}
+}
@@ -0,0 +1,98 @@
+package router
+
+import (
+	"encoding/json"
+	"net/http"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/auth"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+func adapterModel() catalog.Model {
+	m := catModel("coder", nil)
+	m.Profiles = []catalog.Profile{{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 4}}
+	m.Adapters = []catalog.Adapter{
+		{Name: "coder-sql", Source: "http://mirror/sql.gguf"},
+		{Name: "coder-docs", Source: "http://mirror/docs.gguf", Scale: 0.5},
+	}
+	return m
+}
+
+func TestAnAdapterIsAskedForByNameAndAppliedOnItsModel(t *testing.T) {
+	e := newFakeEngine(t, "coder")
+	cfg := baseConfig(config.Endpoint{ID: "a", URL: e.srv.URL, Models: []string{"coder"}})
+	cfg.Models = []catalog.Model{adapterModel()}
+	rt := newTestRouter(t, cfg)
+
+	sent := func(model string) map[string]any {
+		t.Helper()
+		rec := post(t, rt.Handler(), map[string]any{"model": model, "messages": []any{}}, nil)
+		if rec.Code != http.StatusOK {
+			t.Fatalf("%s: status %d: %s", model, rec.Code, rec.Body.String())
+		}
+		var body map[string]any
+		_ = json.Unmarshal(e.last.Load().([]byte), &body)
+		return body
+	}
+	body := sent("coder-docs")
+	if body["model"] != "coder" {
+		t.Errorf("the engine was asked for %v, not the model the adapter runs on", body["model"])
+	}
+	lora, _ := json.Marshal(body["lora"])
+	if string(lora) != `[{"id":1,"scale":0.5}]` {
+		t.Errorf("lora %s, expected the second adapter at its scale", lora)
+	}
+	if body := sent("coder-sql"); func() string { b, _ := json.Marshal(body["lora"]); return string(b) }() != `[{"id":0,"scale":1}]` {
+		t.Errorf("the first adapter, at the default scale: %v", body["lora"])
+	}
+	if body := sent("coder"); body["lora"] != nil {
+		t.Errorf("the base model was sent an adapter: %v", body["lora"])
+	}
+}
+
+func TestAKeyAllowedAnAdapterReachesThatAdapterOnly(t *testing.T) {
+	cfg := baseConfig()
+	cfg.Models = []catalog.Model{adapterModel()}
+	rt := newTestRouter(t, cfg)
+	key := func(allowed ...string) auth.Principal {
+		return auth.Principal{Limits: store.Effective{AllowedModels: allowed}}
+	}
+	for _, c := range []struct {
+		allowed   []string
+		requested string
+		want      bool
+	}{
+		{[]string{"coder-sql"}, "coder-sql", true},
+		{[]string{"coder-sql"}, "coder", false},
+		{[]string{"coder-sql"}, "coder-docs", false},
+		{[]string{"coder"}, "coder-sql", true}, // the model allows its adapters
+		{[]string{"coder"}, "coder", true},
+	} {
+		if got := rt.permits(key(c.allowed...), c.requested); got != c.want {
+			t.Errorf("allowed %v, asked for %s: %v", c.allowed, c.requested, got)
+		}
+	}
+}
+
+func TestAdaptersShareTheNamespaceOfModels(t *testing.T) {
+	a := adapterModel()
+	b := catModel("coder-sql", nil)
+	if _, err := catalog.New([]catalog.Model{a, b}); err == nil {
+		t.Error("an adapter named like another model was accepted")
+	}
+	bad := adapterModel()
+	bad.Profiles = []catalog.Profile{{Name: "gpu", Engine: "vllm", GPUClass: "h100-80g", TP: 1}}
+	if _, err := catalog.New([]catalog.Model{bad}); err == nil {
+		t.Error("adapters on a vLLM profile were accepted")
+	}
+	c, err := catalog.New([]catalog.Model{adapterModel()})
+	if err != nil {
+		t.Fatal(err)
+	}
+	if m, ad, i, ok := c.ResolveAdapter("coder-docs"); !ok || m.Name != "coder" || i != 1 || ad.EffectiveScale() != 0.5 {
+		t.Errorf("resolve: %v %+v %d %v", m.Name, ad, i, ok)
+	}
+}
@@ -34,6 +34,12 @@ type Config struct {
 	// QueueCapacityPerBand bounds each band's queue. Rejecting immediately
 	// beats accumulating silently.
 	QueueCapacityPerBand int `koanf:"queue_capacity_per_band"`
+	// QueueCapacityTotal bounds everything queued, across bands. When it is
+	// reached, a request is not refused while a lower band has requests
+	// queued: the newest of the lowest band is turned away instead
+	// (displacement), since the oldest have waited longest and are nearest
+	// their TTL. Zero takes half of every band's capacity together.
+	QueueCapacityTotal int `koanf:"queue_capacity_total"`
 	// Interval is how often the dispatcher re-checks saturation. Releases also
 	// wake it, so this is only the backstop for saturation falling on its own.
 	Interval time.Duration `koanf:"interval"`
@@ -94,6 +100,9 @@ func (c *Config) normalize() error {
 	if c.QueueCapacityPerBand <= 0 {
 		c.QueueCapacityPerBand = 2000
 	}
+	if c.QueueCapacityTotal <= 0 {
+		c.QueueCapacityTotal = max(c.QueueCapacityPerBand*len(c.Bands)/2, 1)
+	}
 	if c.Interval <= 0 {
 		c.Interval = 10 * time.Millisecond
 	}
@@ -123,6 +132,10 @@ var (
 	ErrQueueTimeout = errors.New("admission queue timeout")
 	// ErrClosed means the controller is shutting down.
 	ErrClosed = errors.New("admission controller closed")
+	// ErrDisplaced means a queued request was turned away to make room for
+	// one of a higher band. Worth retrying: the queue is full now, not for
+	// good.
+	ErrDisplaced = errors.New("displaced from the admission queue by higher-priority requests")
 )

 // Stats is a snapshot of one band's queue, for metrics.
@@ -259,6 +272,12 @@ func (c *Controller) Acquire(ctx context.Context, r Request) (*Lease, error) {
 		c.mu.Unlock()
 		return nil, fmt.Errorf("%w: band %q", ErrQueueFull, band.Name)
 	}
+	if c.queuedLocked() >= c.cfg.QueueCapacityTotal && !c.displaceBelowLocked(idx) {
+		c.rejected[band.Name]++
+		c.mu.Unlock()
+		return nil, fmt.Errorf("%w: %d queued in all, and nothing of a lower band to make room",
+			ErrQueueFull, c.cfg.QueueCapacityTotal)
+	}

 	w := &waiter{
 		tenant:   tenantKey(r),
@@ -279,6 +298,10 @@ func (c *Controller) Acquire(ctx context.Context, r Request) (*Lease, error) {
 			c.mu.Unlock()
 			return nil, ErrClosed
 		}
+		if w.err != nil {
+			c.mu.Unlock()
+			return nil, w.err
+		}
 		c.waitTotal[band.Name] += time.Since(w.enqueued)
 		c.mu.Unlock()
 		return &Lease{c: c, band: band.Name, model: r.Model}, nil
@@ -297,8 +320,9 @@ func (c *Controller) Acquire(ctx context.Context, r Request) (*Lease, error) {
 func (c *Controller) remove(idx int, w *waiter, reason string) {
 	c.mu.Lock()
 	defer c.mu.Unlock()
-	if w.admitted {
-		// It won the race; the caller will see ready closed next time round.
+	if w.admitted || w.err != nil {
+		// It won the race, or was displaced; the caller will see ready
+		// closed next time round.
 		return
 	}
 	c.queues[idx].remove(w)
@@ -410,3 +434,31 @@ func tenantKey(r Request) string {
 	}
 	return "anonymous"
 }
+
+// queuedLocked is how many requests wait in every band. Called with mu held.
+func (c *Controller) queuedLocked() int {
+	n := 0
+	for _, q := range c.queues {
+		n += q.len()
+	}
+	return n
+}
+
+// displaceBelowLocked turns away the newest request of the lowest band below
+// idx that has any queued, and says whether there was one. Called with mu held.
+func (c *Controller) displaceBelowLocked(idx int) bool {
+	// Bands are sorted highest priority first, so lower bands have larger
+	// indices, and the lowest is last.
+	for j := len(c.queues) - 1; j > idx; j-- {
+		w := c.queues[j].newest()
+		if w == nil {
+			continue
+		}
+		c.queues[j].remove(w)
+		w.err = fmt.Errorf("%w (band %q)", ErrDisplaced, c.cfg.Bands[j].Name)
+		close(w.ready)
+		c.rejected[c.cfg.Bands[j].Name+"/displaced"]++
+		return true
+	}
+	return false
+}
@@ -0,0 +1,89 @@
+package admission
+
+import (
+	"context"
+	"errors"
+	"testing"
+	"time"
+)
+
+// queueUntil starts an Acquire in the background and waits until the band
+// holds depth requests, returning where its result will arrive.
+func queueUntil(t *testing.T, c *Controller, band string, bandIdx, depth int) <-chan error {
+	t.Helper()
+	out := make(chan error, 1)
+	go func() {
+		l, err := c.Acquire(context.Background(), Request{Model: "m", Band: band})
+		if err == nil {
+			l.Release()
+		}
+		out <- err
+	}()
+	deadline := time.Now().Add(2 * time.Second)
+	for c.Depths()[bandIdx].Depth < depth && time.Now().Before(deadline) {
+		time.Sleep(time.Millisecond)
+	}
+	if got := c.Depths()[bandIdx].Depth; got != depth {
+		t.Fatalf("band %s holds %d, expected %d", band, got, depth)
+	}
+	return out
+}
+
+func TestAHigherBandDisplacesTheNewestOfTheLowest(t *testing.T) {
+	s := &fixedSat{}
+	s.set(5) // past every ceiling: everything queues
+	cfg := testConfig()
+	cfg.QueueCapacityTotal = 3
+	c := newController(t, cfg, s.fn)
+	// Bands, highest first: platinum 0, gold 1, silver 2, bronze 3.
+	oldest := queueUntil(t, c, "bronze", 3, 1)
+	time.Sleep(2 * time.Millisecond)
+	silver := queueUntil(t, c, "silver", 2, 1)
+	time.Sleep(2 * time.Millisecond)
+	newest := queueUntil(t, c, "bronze", 3, 2)
+
+	// Full at three. Gold arrives: the newest bronze goes, not the oldest,
+	// and not silver, which outranks bronze.
+	gold := queueUntil(t, c, "gold", 1, 1)
+	select {
+	case err := <-newest:
+		if !errors.Is(err, ErrDisplaced) {
+			t.Fatalf("the newest bronze got %v", err)
+		}
+	case <-time.After(time.Second):
+		t.Fatal("nothing was displaced")
+	}
+	if d := c.Depths(); d[3].Depth != 1 || d[2].Depth != 1 || d[1].Depth != 1 {
+		t.Errorf("depths %+v: one bronze, the silver and the gold should remain", d)
+	}
+
+	// Bronze cannot displace anything: nothing is below it.
+	_, err := c.Acquire(t.Context(), Request{Model: "m", Band: "bronze"})
+	if !errors.Is(err, ErrQueueFull) {
+		t.Errorf("a bronze request at a full queue got %v", err)
+	}
+	// Another gold takes the remaining bronze, then silver.
+	queueUntil(t, c, "gold", 1, 2)
+	if err := <-oldest; !errors.Is(err, ErrDisplaced) {
+		t.Errorf("the last bronze got %v", err)
+	}
+	queueUntil(t, c, "gold", 1, 3)
+	if err := <-silver; !errors.Is(err, ErrDisplaced) {
+		t.Errorf("silver got %v", err)
+	}
+	// All gold now: a fourth has nobody lower to displace.
+	if _, err := c.Acquire(t.Context(), Request{Model: "m", Band: "gold"}); !errors.Is(err, ErrQueueFull) {
+		t.Errorf("a gold request with only gold queued got %v", err)
+	}
+
+	// And they are served once there is room.
+	s.set(0)
+	select {
+	case err := <-gold:
+		if err != nil {
+			t.Errorf("the first gold was not served: %v", err)
+		}
+	case <-time.After(2 * time.Second):
+		t.Error("the queue did not drain")
+	}
+}
@@ -12,6 +12,9 @@ type waiter struct {
 	// admitted guards the race between the dispatcher admitting a waiter and
 	// the waiter giving up on its own.
 	admitted bool
+	// err is why the waiter was turned away while queued — displaced by a
+	// higher band — set before ready is closed.
+	err error
 }

 // bandQueue holds one band's waiters, grouped by tenant.
@@ -166,3 +169,17 @@ func (q *bandQueue) dropExpired(now time.Time, onDrop func()) {
 		q.byTenant[tenant] = kept
 	}
 }
+
+// newest is the most recently queued waiter, across every tenant: the one
+// displacement takes, since it has invested the least waiting.
+func (q *bandQueue) newest() *waiter {
+	var out *waiter
+	for _, ws := range q.byTenant {
+		for _, w := range ws {
+			if out == nil || w.enqueued.After(out.enqueued) {
+				out = w
+			}
+		}
+	}
+	return out
+}
@@ -37,8 +37,16 @@ type Info struct {
 	Weight float64
 	// Priority expresses pool preference; higher pools are tried first.
 	Priority int
+	// UpstreamModel is what the endpoint calls the model, when that differs
+	// from the catalog's name: an external provider's. Requests are rewritten
+	// to it. Empty sends the name as the client did.
+	UpstreamModel string
 }

+// EngineExternal marks an endpoint that is a provider outside the cluster.
+// It publishes no metrics, so it is never scraped and never reads as stale.
+const EngineExternal = "external"
+
 // Serves reports whether the replica serves the given model.
 func (i Info) Serves(model string) bool {
 	return slices.Contains(i.Models, model)
@@ -242,7 +250,7 @@ func (r *Registry) Snapshots(model string) []Snapshot {
 			continue
 		}
 		var age time.Duration
-		stale := true
+		stale := l.info.Engine != EngineExternal
 		if !l.scrapedAt.IsZero() {
 			age = now.Sub(l.scrapedAt)
 			stale = age > r.cfg.StaleAfter
@@ -0,0 +1,89 @@
+package router
+
+import (
+	"encoding/json"
+	"fmt"
+	"io"
+	"net/http"
+	"net/http/httptest"
+	"sync/atomic"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint"
+)
+
+func TestALocalModelWithNoCapacityFallsBackToAnExternalProvider(t *testing.T) {
+	local := newFakeEngine(t, "local")
+	local.status.Store(http.StatusServiceUnavailable)
+
+	var calls atomic.Int64
+	var gotModel, gotAuth atomic.Value
+	provider := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		if r.URL.Path == "/metrics" {
+			t.Error("the provider was scraped for metrics")
+		}
+		calls.Add(1)
+		gotAuth.Store(r.Header.Get("Authorization"))
+		var body map[string]any
+		b, _ := io.ReadAll(r.Body)
+		_ = json.Unmarshal(b, &body)
+		gotModel.Store(fmt.Sprint(body["model"]))
+		w.Header().Set("Content-Type", "application/json")
+		fmt.Fprint(w, `{"model":"gpt-4o-mini","choices":[{"message":{"content":"from outside"}}],"usage":{"prompt_tokens":3,"completion_tokens":2}}`)
+	}))
+	t.Cleanup(provider.Close)
+
+	cfg := baseConfig(config.Endpoint{ID: "l", URL: local.srv.URL, Models: []string{"local"}})
+	cloud := catalog.Model{Name: "cloud", Source: "external",
+		External: &catalog.External{URL: provider.URL, Model: "gpt-4o-mini"}}
+	cfg.Models = []catalog.Model{catModel("local", nil, "cloud"), cloud}
+	rt := newTestRouter(t, cfg)
+	rt.SetEndpoints(t.Context(), []endpoint.Info{
+		{ID: "l", URL: local.srv.URL, Models: []string{"local"}},
+		{ID: "external:cloud", URL: provider.URL, Models: []string{"cloud"}, APIKey: "provider-key",
+			Engine: endpoint.EngineExternal, Pool: "external", Weight: 1, UpstreamModel: "gpt-4o-mini"},
+	})
+
+	rec := post(t, rt.Handler(), map[string]any{"model": "local", "messages": []any{}}, nil)
+	if rec.Code != http.StatusOK {
+		t.Fatalf("status %d: %s", rec.Code, rec.Body.String())
+	}
+	if calls.Load() != 1 {
+		t.Fatalf("the provider was called %d times", calls.Load())
+	}
+	if gotModel.Load() != "gpt-4o-mini" {
+		t.Errorf("the provider was asked for %v, not its own name for the model", gotModel.Load())
+	}
+	if gotAuth.Load() != "Bearer provider-key" {
+		t.Errorf("the provider got %v, not its key", gotAuth.Load())
+	}
+	if got := rec.Header().Get("x-llmgw-served-model"); got != "cloud" {
+		t.Errorf("the client should be told it was served by cloud, got %q", got)
+	}
+	// Never scraped, and never stale for it.
+	for _, s := range rt.reg.Snapshots("cloud") {
+		if s.Stale {
+			t.Error("an external endpoint reads as stale")
+		}
+	}
+}
+
+func TestAnExternalModelIsNotAModelWithProfiles(t *testing.T) {
+	good := catalog.Model{Name: "cloud", Source: "external",
+		External: &catalog.External{URL: "https://api.example.com", Model: "gpt-4o-mini"}}
+	if _, err := catalog.New([]catalog.Model{good}); err != nil {
+		t.Fatalf("a valid external model was refused: %v", err)
+	}
+	for name, m := range map[string]catalog.Model{
+		"profiles": {Name: "x", Source: "external", External: &catalog.External{URL: "https://a.b", Model: "m"},
+			Profiles: []catalog.Profile{{Name: "p", Engine: "llamacpp"}}},
+		"no url":   {Name: "x", Source: "external", External: &catalog.External{URL: "api.example.com", Model: "m"}},
+		"no model": {Name: "x", Source: "external", External: &catalog.External{URL: "https://a.b"}},
+	} {
+		if _, err := catalog.New([]catalog.Model{m}); err == nil {
+			t.Errorf("%s: accepted", name)
+		}
+	}
+}
@@ -0,0 +1,122 @@
+package router
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"net/http"
+	"net/http/httptest"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/accounting"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/router/admission"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/router/proxy"
+)
+
+// What a client is told for each way a dispatch can end. Each status is
+// something a client acts on: 503 with Retry-After is "try again", 503 alone
+// is "not here", 502 is "the engine failed", and a stream already under way
+// cannot be given a status at all.
+func TestEachWayADispatchEndsIsToldToTheClientAsItShouldBe(t *testing.T) {
+	rt := newTestRouter(t, baseConfig())
+	for _, c := range []struct {
+		name       string
+		res        dispatchResult
+		status     int
+		body       bool // an error body was written
+		retryAfter bool
+	}{
+		{"served", dispatchResult{outcome: proxy.Outcome{Status: 200, Wrote: true}}, 200, false, false},
+		{"engine refused, relayed", dispatchResult{outcome: proxy.Outcome{Status: 400, Wrote: true}}, 400, false, false},
+		{"client went away", dispatchResult{err: proxy.ErrClientGone}, 499, false, false},
+		{"stream cut short", dispatchResult{outcome: proxy.Outcome{Status: 200, Wrote: true}, err: errors.New("eof")}, 200, false, false},
+		{"stream cut before a status", dispatchResult{outcome: proxy.Outcome{Wrote: true}, err: errors.New("eof")}, 200, false, false},
+		{"still loading", dispatchResult{err: fmt.Errorf("%w: retry shortly", errLoading)}, 503, true, true},
+		{"nothing serves it", dispatchResult{err: fmt.Errorf("%w: no replica", errNoCapacity)}, 503, true, false},
+		{"engine failed", dispatchResult{err: errors.New("connection reset")}, 502, true, false},
+	} {
+		t.Run(c.name, func(t *testing.T) {
+			w := httptest.NewRecorder()
+			got := rt.finish(w, &parsedRequest{Model: "m"}, c.res, time.Now())
+			if got != c.status {
+				t.Errorf("status %d, expected %d", got, c.status)
+			}
+			if c.body {
+				if w.Code != c.status {
+					t.Errorf("written %d, expected %d", w.Code, c.status)
+				}
+				var e struct {
+					Error struct {
+						Message string `json:"message"`
+					} `json:"error"`
+				}
+				if err := json.Unmarshal(w.Body.Bytes(), &e); err != nil || e.Error.Message == "" {
+					t.Errorf("error body %q", w.Body.String())
+				}
+			} else if w.Body.Len() != 0 {
+				t.Errorf("wrote %q over a response that was not ours to write", w.Body.String())
+			}
+			if (w.Header().Get("Retry-After") != "") != c.retryAfter {
+				t.Errorf("Retry-After %q", w.Header().Get("Retry-After"))
+			}
+		})
+	}
+}
+
+func TestAnUpstreamStatusIsKeptWhenTheStreamBreaks(t *testing.T) {
+	rt := newTestRouter(t, baseConfig())
+	w := httptest.NewRecorder()
+	res := dispatchResult{outcome: proxy.Outcome{Status: http.StatusTooManyRequests, Wrote: true}, err: errors.New("eof")}
+	if got := rt.finish(w, &parsedRequest{Model: "m"}, res, time.Now()); got != 429 {
+		t.Errorf("status %d", got)
+	}
+}
+
+func TestARefusalBeforeDispatchSaysWhetherWaitingHelps(t *testing.T) {
+	rt := newTestRouter(t, baseConfig())
+	for _, c := range []struct {
+		name       string
+		refuse     func(http.ResponseWriter)
+		status     int
+		retryAfter string
+	}{
+		{"queue full", func(w http.ResponseWriter) {
+			rt.rejectAdmission(w, &parsedRequest{Model: "m"}, admission.ErrQueueFull, time.Now())
+		}, 503, ""},
+		{"queued too long", func(w http.ResponseWriter) {
+			rt.rejectAdmission(w, &parsedRequest{Model: "m"}, admission.ErrQueueTimeout, time.Now())
+		}, 504, ""},
+		{"displaced for a higher band", func(w http.ResponseWriter) {
+			rt.rejectAdmission(w, &parsedRequest{Model: "m"}, admission.ErrDisplaced, time.Now())
+		}, 503, "5"},
+		{"shutting down", func(w http.ResponseWriter) {
+			rt.rejectAdmission(w, &parsedRequest{Model: "m"}, admission.ErrClosed, time.Now())
+		}, 503, ""},
+		{"client left the queue", func(w http.ResponseWriter) {
+			rt.rejectAdmission(w, &parsedRequest{Model: "m"}, context.Canceled, time.Now())
+		}, 200, ""}, // nothing written: nobody is listening
+		{"rate limited", func(w http.ResponseWriter) {
+			rt.rejectLimit(w, &parsedRequest{Model: "m"},
+				&accounting.LimitError{Err: accounting.ErrRateLimited, RetryAfter: 3 * time.Second}, time.Now())
+		}, 429, "3"},
+		{"budget spent", func(w http.ResponseWriter) {
+			rt.rejectLimit(w, &parsedRequest{Model: "m"}, &accounting.LimitError{Err: accounting.ErrBudgetSpent}, time.Now())
+		}, 429, ""},
+		{"not a limit at all", func(w http.ResponseWriter) {
+			rt.rejectLimit(w, &parsedRequest{Model: "m"}, errors.New("store down"), time.Now())
+		}, 500, ""},
+	} {
+		t.Run(c.name, func(t *testing.T) {
+			w := httptest.NewRecorder()
+			c.refuse(w)
+			if w.Code != c.status {
+				t.Errorf("status %d, expected %d", w.Code, c.status)
+			}
+			if got := w.Header().Get("Retry-After"); got != c.retryAfter {
+				t.Errorf("Retry-After %q, expected %q", got, c.retryAfter)
+			}
+		})
+	}
+}
@@ -15,6 +15,9 @@ import (
 	"strings"
 	"time"

+	"go.opentelemetry.io/otel/attribute"
+	"go.opentelemetry.io/otel/codes"
+
 	"gitea.mynodes.xyz/ilker/llmgw/internal/accounting"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/auth"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/router/admission"
@@ -57,9 +60,15 @@ func (r *Router) rejectAdmission(w http.ResponseWriter, req *parsedRequest, err
 		return
 	case errors.Is(err, admission.ErrQueueTimeout):
 		status = http.StatusGatewayTimeout
+	case errors.Is(err, admission.ErrDisplaced):
+		// Turned away for higher-priority work, not for good: said so the
+		// client backs off and comes back rather than giving up.
+		w.Header().Set("Retry-After", "5")
 	}
 	reason := "unavailable"
 	switch {
+	case errors.Is(err, admission.ErrDisplaced):
+		reason = "displaced"
 	case errors.Is(err, admission.ErrQueueFull):
 		reason = "queue_full"
 	case errors.Is(err, admission.ErrQueueTimeout):
@@ -182,6 +191,11 @@ func (r *Router) Handler() http.Handler {
 			pages.RegisterAdminUsers(mux, r.store)
 			// Quotas are written on the same tree the accounts page builds, so
 			// they are mounted together.
+			// Capacity quotas are the control plane's, reached through the
+			// same admin API the CLI uses.
+			if c, ok := r.clusterView.(web.CapacityAdmin); ok {
+				pages.SetCapacity(c)
+			}
 			pages.RegisterAdminQuotas(mux, r.store, r.acct)
 			// And the report of what that tree actually used.
 			pages.RegisterAdminUsage(mux, r.store, r.acct)
@@ -288,9 +302,13 @@ func (r *Router) handleInference(path string) http.HandlerFunc {
 		// somebody asks what happened to it.
 		rid := requestID(req.Header.Get("X-Request-Id"))
 		w.Header().Set("X-Request-Id", rid)
+		ctx, span := startInference(req, path, rid)
+		defer span.End()
+		req = req.WithContext(ctx)

 		parsed, err := r.parse(req, path)
 		if err == nil {
+			span.SetAttributes(attribute.String("llmgw.model", parsed.Model))
 			parsed.RequestID = rid
 			if parsed.Header == nil {
 				parsed.Header = http.Header{}
@@ -355,7 +373,9 @@ func (r *Router) handleInference(path string) http.HandlerFunc {
 		}

 		// Step 5 of the lifecycle: "now?" before "where?" (plan §7.1).
+		_, aspan := tracer().Start(req.Context(), "admission")
 		lease, err := r.admit(req.Context(), parsed)
+		endSpan(aspan, err)
 		if err != nil {
 			r.rejectAdmission(w, parsed, err, start)
 			return
@@ -381,7 +401,13 @@ func (r *Router) handleInference(path string) http.HandlerFunc {
 		}

 		res := r.dispatch(req.Context(), w, parsed)
-		r.finish(w, parsed, res, start)
+		status := r.finish(w, parsed, res, start)
+		span.SetAttributes(attribute.Int("http.response.status_code", status),
+			attribute.String("llmgw.served_model", res.model),
+			attribute.String("llmgw.endpoint", res.endpointID))
+		if status >= 500 {
+			span.SetStatus(codes.Error, http.StatusText(status))
+		}
 		// After the client has been served. A conversation that failed to
 		// record is a worse outcome than a slow reply, but only just, and the
 		// caller has their answer either way.
@@ -400,6 +426,12 @@ func (r *Router) handleInference(path string) http.HandlerFunc {
 // around the allowlist; matching only the canonical name would silently ignore
 // an allowlist written in the names clients actually use.
 func (r *Router) permits(p auth.Principal, requested string) bool {
+	// An adapter named in the allowlist allows that adapter: not its model,
+	// and not the model's other adapters, which the checks below would let
+	// through if the adapter's own name were not asked about first.
+	if _, ok := r.adapter(requested); ok && p.Allows(requested) {
+		return true
+	}
 	canonical, m := r.resolve(requested)
 	if p.Allows(canonical) {
 		return true
@@ -413,7 +445,8 @@ func (r *Router) permits(p auth.Principal, requested string) bool {
 }

 // finish records metrics and, when nothing was written, produces the error body.
-func (r *Router) finish(w http.ResponseWriter, req *parsedRequest, res dispatchResult, start time.Time) {
+// It returns the status the request ended with.
+func (r *Router) finish(w http.ResponseWriter, req *parsedRequest, res dispatchResult, start time.Time) int {
 	status := res.outcome.Status
 	switch {
 	case res.err == nil:
@@ -447,6 +480,7 @@ func (r *Router) finish(w http.ResponseWriter, req *parsedRequest, res dispatchR
 	r.m.observeRequest(r.metricModel(req.Model), res.pool, status, time.Since(start), res.outcome.Usage, res.outcome.TTFT)
 	r.account(req, res, status, time.Since(start))
 	r.accessLog(req, res, status, time.Since(start))
+	return status
 }

 // account settles a finished request against its key.
@@ -956,3 +990,44 @@ func (r *Router) rateLimitHeaders(w http.ResponseWriter, p auth.Principal) {
 		w.Header().Set("x-ratelimit-reset-tokens", reset)
 	}
 }
+
+// withModel is body with its model set to name, for an endpoint that knows the
+// model by another name — an external provider. The body unchanged when it is
+// not a JSON object, which the endpoint then refuses as it would have anyway.
+func withModel(body []byte, name string) []byte {
+	var m map[string]json.RawMessage
+	if err := json.Unmarshal(body, &m); err != nil {
+		return body
+	}
+	encoded, err := json.Marshal(name)
+	if err != nil {
+		return body
+	}
+	m["model"] = encoded
+	out, err := json.Marshal(m)
+	if err != nil {
+		return body
+	}
+	return out
+}
+
+// withAdapter is body asking for model with a LoRA adapter applied, in
+// llama.cpp's form: "lora": [{"id": index, "scale": scale}]. Adapters not
+// listed stay at zero, the scale the engine loads each one at.
+func withAdapter(body []byte, model string, index int, scale float64) []byte {
+	var m map[string]json.RawMessage
+	if err := json.Unmarshal(body, &m); err != nil {
+		return body
+	}
+	name, err1 := json.Marshal(model)
+	lora, err2 := json.Marshal([]map[string]any{{"id": index, "scale": scale}})
+	if err1 != nil || err2 != nil {
+		return body
+	}
+	m["model"], m["lora"] = name, lora
+	out, err := json.Marshal(m)
+	if err != nil {
+		return body
+	}
+	return out
+}
@@ -11,19 +11,24 @@ import (
 	"fmt"
 	"log/slog"
 	"math"
+	"net"
 	"net/http"
+	"net/netip"
 	"slices"
 	"strings"
 	"sync/atomic"
 	"time"

 	"github.com/prometheus/client_golang/prometheus"
+	"go.opentelemetry.io/otel/attribute"
+	"go.opentelemetry.io/otel/trace"

 	"gitea.mynodes.xyz/ilker/llmgw/internal/account"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/accounting"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/auth"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/router/admission"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/router/breaker"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint"
@@ -56,6 +61,8 @@ type Router struct {
 	// router's half of not having to. A request takes the map once and works
 	// from it, so a swap mid-request is invisible rather than a race.
 	models atomic.Pointer[map[string]catalog.Model]
+	// adapters are the LoRA adapters by the name clients ask for them by.
+	adapters atomic.Pointer[map[string]adapterRef]

 	// extraMetrics are collectors other components publish on /metrics.
 	extraMetrics []prometheus.Collector
@@ -230,6 +237,11 @@ func New(cfg config.Server, log *slog.Logger) (*Router, error) {
 		Timeout:  cfg.Router.ScrapeTimeout,
 	}, log)

+	if !cfg.Auth.Enabled && !loopbackOnly(cfg.Listen) {
+		log.Warn("auth is off and the API listens beyond this machine: anyone who can reach "+
+			cfg.Listen+" can use every model; set auth.enabled, or listen on 127.0.0.1",
+			"listen", cfg.Listen)
+	}
 	if cfg.Auth.Enabled {
 		st, err := storepkg.Open(context.Background(), cfg.Auth.Store)
 		if err != nil {
@@ -305,14 +317,7 @@ func New(cfg config.Server, log *slog.Logger) (*Router, error) {
 	// all. A control plane pushes over the top of this at startup and on every
 	// change after.
 	if len(cfg.Models) > 0 {
-		seed := map[string]catalog.Model{}
-		for _, m := range cfg.Models {
-			seed[m.Name] = m
-			for _, a := range m.Aliases {
-				seed[a] = m
-			}
-		}
-		r.models.Store(&seed)
+		r.storeModels(cfg.Models)
 	}
 	return r, nil
 }
@@ -337,6 +342,7 @@ func admissionConfig(rc config.Router) admission.Config {
 		},
 		QueueTTL:             rc.Admission.QueueTTL,
 		QueueCapacityPerBand: rc.Admission.QueueCapacityPerBand,
+		QueueCapacityTotal:   rc.Admission.QueueCapacityTotal,
 		Interval:             rc.Admission.Interval,
 	}
 }
@@ -494,15 +500,44 @@ func (r *Router) modelMap() map[string]catalog.Model {
 // The catalog's counterpart to SetEndpoints, and pushed for the same reason:
 // the control plane owns what should exist, the router needs to know, and
 // neither has to be in the other's process for that to work.
-func (r *Router) SetCatalog(c *catalog.Catalog) {
+func (r *Router) SetCatalog(c *catalog.Catalog) { r.storeModels(c.Models()) }
+
+// storeModels makes models what names resolve to: each model by its name, its
+// aliases and its adapters. One place for it, whether the models came from
+// configuration at start or from the control plane since.
+func (r *Router) storeModels(models []catalog.Model) {
 	m := map[string]catalog.Model{}
-	for _, entry := range c.Models() {
+	ads := map[string]adapterRef{}
+	for _, entry := range models {
 		m[entry.Name] = entry
 		for _, a := range entry.Aliases {
 			m[a] = entry
 		}
+		// An adapter resolves to the model it runs on, as an alias does, and
+		// is applied to the request on the way there.
+		for i, a := range entry.Adapters {
+			m[a.Name] = entry
+			ads[a.Name] = adapterRef{base: entry.Name, index: i, scale: a.EffectiveScale()}
+		}
 	}
 	r.models.Store(&m)
+	r.adapters.Store(&ads)
+}
+
+// adapterRef is where an adapter runs and how it is applied.
+type adapterRef struct {
+	base  string
+	index int
+	scale float64
+}
+
+// adapter finds one by the name a client asked for.
+func (r *Router) adapter(name string) (adapterRef, bool) {
+	if p := r.adapters.Load(); p != nil {
+		a, ok := (*p)[name]
+		return a, ok
+	}
+	return adapterRef{}, false
 }

 // LastUsed reports when each endpoint was last sent a request, satisfying
@@ -510,6 +545,24 @@ func (r *Router) SetCatalog(c *catalog.Catalog) {
 // holding its cards out of habit.
 func (r *Router) LastUsed() map[string]time.Time { return r.reg.LastUsed() }

+// ModelLoad sums what each model's replicas report: how many there are, and
+// requests running and waiting at their engines. Keyed by the catalog name,
+// which the control plane publishes first among each endpoint's names.
+func (r *Router) ModelLoad() map[string]reconciler.Load {
+	out := map[string]reconciler.Load{}
+	for _, s := range r.reg.Snapshots("") {
+		if len(s.Models) == 0 {
+			continue
+		}
+		l := out[s.Models[0]]
+		l.Replicas++
+		l.Running += s.Stats.Running
+		l.Waiting += s.Stats.Waiting
+		out[s.Models[0]] = l
+	}
+	return out
+}
+
 func (r *Router) SetEndpoints(ctx context.Context, infos []endpoint.Info) {
 	if r.prefix != nil {
 		// A replica that has gone takes its cache with it. Leaving its entries
@@ -663,7 +716,13 @@ func (r *Router) dispatchOne(ctx context.Context, w http.ResponseWriter, req *pa
 		if len(candidates) == 0 {
 			break
 		}
+		_, sspan := tracer().Start(ctx, "schedule", trace.WithAttributes(
+			attribute.String("llmgw.model", model), attribute.Int("llmgw.candidates", len(candidates))))
 		dec, err := profile.Schedule(sctx, candidates)
+		if err == nil {
+			sspan.SetAttributes(attribute.String("llmgw.endpoint", dec.Endpoint.ID))
+		}
+		endSpan(sspan, err)
 		if err != nil {
 			last = dispatchResult{err: fmt.Errorf("%w: %v", errNoCapacity, err)}
 			break
@@ -677,15 +736,30 @@ func (r *Router) dispatchOne(ctx context.Context, w http.ResponseWriter, req *pa
 			continue
 		}

-		out, perr := r.px.Do(ctx, w, proxy.Request{
+		uctx, uspan := tracer().Start(ctx, "upstream", trace.WithSpanKind(trace.SpanKindClient),
+			trace.WithAttributes(attribute.String("llmgw.endpoint", dec.Endpoint.ID),
+				attribute.String("url.full", dec.Endpoint.URL+req.Path)))
+		body := req.UpstreamBody
+		if dec.Endpoint.UpstreamModel != "" {
+			body = withModel(body, dec.Endpoint.UpstreamModel)
+		}
+		// An adapter asked for by name, served by its model's replicas: the
+		// engine is asked for the model, with the adapter applied. Not on a
+		// fallback to another model, which does not have it.
+		if a, ok := r.adapter(req.Model); ok && a.base == model {
+			body = withAdapter(body, model, a.index, a.scale)
+		}
+		out, perr := r.px.Do(uctx, w, proxy.Request{
 			URL:                dec.Endpoint.URL + req.Path,
 			APIKey:             dec.Endpoint.APIKey,
-			Body:               req.UpstreamBody,
-			Header:             req.Header,
+			Body:               body,
+			Header:             withTraceparent(uctx, req.Header),
 			Stream:             req.Stream,
 			SuppressUsageChunk: req.SuppressUsageChunk,
 			CaptureReply:       req.captureBytes(),
 		})
+		uspan.SetAttributes(attribute.Int("http.response.status_code", out.Status))
+		endSpan(uspan, perr)
 		// A retryable failure is the endpoint's fault; a client disconnect is not.
 		release(perr == nil || errors.Is(perr, proxy.ErrClientGone))

@@ -745,3 +819,17 @@ func personalLimits(p config.Personal) *storepkg.Limits {
 	}
 	return &l
 }
+
+// loopbackOnly reports whether a listen address is reachable from this
+// machine alone. An empty host (":8080") is every interface.
+func loopbackOnly(listen string) bool {
+	host, _, err := net.SplitHostPort(listen)
+	if err != nil {
+		return false
+	}
+	if host == "localhost" {
+		return true
+	}
+	a, err := netip.ParseAddr(host)
+	return err == nil && a.IsLoopback()
+}