Describe v0.8.2

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

.gitea/workflows/release.yml

@@ -150,3 +150,31 @@ jobs:
               || { echo "upload of $(basename "$f") was refused" >&2; exit 1; }
           done
           echo "attached $(ls dist | wc -l) files to $tag"
+
+  # The release page says what changed, copied from CHANGELOG.md. Gitea makes
+  # an empty release for every tag and the job above only attaches files, so
+  # every release until this existed was a list of binaries and nothing else.
+  #
+  # After the build, because that is what guarantees the release exists. Run by
+  # hand from a branch it syncs every version in the changelog, which is how
+  # the releases made before this got their notes.
+  notes:
+    needs: build
+    runs-on: ubuntu-latest
+    permissions:
+      contents: write
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Copy the changelog onto the release
+        env:
+          GITEA_TOKEN: ${{ secrets.GITEA_TOKEN }}
+          GITEA_INSTANCE: ${{ github.server_url }}
+          GITEA_REPO: ${{ github.repository }}
+        run: |
+          set -euo pipefail
+          if [ "${GITHUB_REF_TYPE:-}" = "tag" ]; then
+            python3 scripts/gitea/release_notes.py --tag "${GITHUB_REF_NAME}"
+          else
+            python3 scripts/gitea/release_notes.py
+          fi

CHANGELOG.md

@@ -12,11 +12,176 @@ is only released once its **exit criterion** is met, not when its code lands.
 | v0.5.0 | 3 | Control plane: catalog, placement, reconciler | released |
 | v0.6.0 | 4 | Multi-node replicas and fabric awareness | released |
 | v0.7.0 | — | Conversations, agent memory, disk management | released |
+| v0.8.0 | — | PostgreSQL, leader election, agent failover | released |
 | v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
 
 Until v1.0.0 the minor version is the breaking one: configuration formats and Go
 APIs change between minors without a deprecation period.
 
+## v0.8.2 — 2026-09-28
+
+A node no longer runs whatever engine somebody once put on it. It installs its
+own.
+
+### Engines are installed by the node
+
+`engines:` in server.yaml names the engines every node has and their versions,
+defaulting to the llama.cpp build this release was checked against (b10991).
+Each agent converges on it: it picks the build for its hardware — CUDA with its
+runtime when an NVIDIA driver is loaded, CPU otherwise — installs the OS
+packages that build needs with the machine's own package manager, downloads the
+archive and checks it against the sha256 GitHub publishes, unpacks it refusing
+anything that would land outside its directory, and runs it. `ldd` names a
+missing library or a system too old for the build ("libstdc++.so.6 is too old
+(needs GLIBCXX_3.4.30)"); `--version` proves the rest. Installs are checked
+again every ten minutes.
+
+An engine is reported only once installed and seen to run, or when the agent's
+configuration names it on purpose. A node that has installed nothing yet is not
+a candidate for any model, where before it was a candidate for all of them.
+`ctl nodes`, the cluster page and placement's refusals say where each
+installation stands, and warn about a CPU without AVX2.
+
+Not every system is refused for its name: the build runs on Ubuntu 22.04+,
+Debian 12+ and RHEL-family 10, and `ldd` decides. vLLM and SGLang are not yet
+installed this way and must still be named in the agent's configuration.
+
+### Onboarding
+
+Onboarding installs llmgw from the CI release for the node's architecture,
+checked against the release's checksums, and writes an agent configuration of
+only the control plane's addresses and the TLS files.
+
+### Fixes
+
+- A race on the weights fingerprint, which two heartbeat loops touched during a
+  failover.
+
+## v0.8.1 — 2026-09-28
+
+Nothing here changes a configuration format: every file that worked with
+v0.8.0 still works. What changes is how much of it needs writing, and how much
+of running the cluster can be done from a page rather than a shell.
+
+### A node describes itself
+
+An agent file used to name its node, its address, the path to llama-server and
+its weight directories — different on every node and kept true by hand. The
+agent now finds these out: its id is the hostname, its address is the one on
+the route to the control plane, llama-server is looked for on PATH and in
+`/opt/llama.cpp`, a node without nvidia-smi simply has no NVIDIA cards, and
+fetched weights land in `/var/lib/llmgw-weights`, which is always scanned.
+Memory is re-read on every heartbeat and installed engines once a minute, so a
+VM given more RAM or a node given an engine becomes a candidate without an
+agent restart; the control plane logs both. The agent file shrinks to the
+control plane's addresses and the TLS files, the same on every node.
+
+### Models from the mirror, without writing YAML
+
+Search the registry from `/admin/mirror`, fetch a sharded model in one action,
+and serve a mirrored version from its row — name, source, engine and an
+estimated memory requirement filled in. A model can say what it is for, and
+`/v1/models` shows it. Placement no longer hands the same memory to two models
+or puts an engine on a node that lacks it, and says when a replica runs an
+edited profile.
+
+### Tenancy, quotas and usage on pages
+
+The tenancy tree, quotas and budgets are managed from the admin pages; an
+organization's budget bounds its teams, a group's quota counts against the
+group, and a budget that cannot be spent against says so. `/admin/usage`
+charts requests, tokens and spend over time by organization, team, project or
+person, and a request is priced by the model that served it.
+
+### Watching the cluster
+
+The cluster page has its own charts, backed by a year of Prometheus history,
+with alerts that were made to fire before being trusted — including for the
+disk the metrics share with the models.
+
+### Chat
+
+The picker offers each model once rather than once per alias, never asks for
+more tokens than the model allows, and offers a model that loads on demand —
+sending it a message is what loads it.
+
+## v0.8.0 — 2026-09-27
+
+The store stops being a file on one machine's disk. PostgreSQL, a way to move
+an existing installation onto it without reissuing every key, and the two
+things a second control plane needs so that running one is safe rather than
+clever.
+
+### A second control plane, and a way to fail over to it
+
+Leadership is a lease in the store, so two control planes against one database
+cannot both place work — which they otherwise would, both seeing a model
+wanting replicas and none running, leaving twice the replicas each holding GPUs
+the other believes are free.
+
+A standby also refuses agent streams. Commands travel down the stream the agent
+opened, so a standby holding one is a node the leader cannot command, and the
+leader's placements would fail with "no agent connected" for a node that is
+plainly connected. Agents therefore take a list of addresses rather than one,
+trying each in turn: a standby refusing, a stopped control plane refusing the
+connection and a crashed one never answering are the same thing from a node.
+
+Measured on the cluster. Leadership moved three seconds after the leader was
+stopped — it released the lease on the way out rather than letting it lapse —
+and all five agents had registered with the standby six seconds later. The
+standby adopted every replica rather than restarting any: six replicas, all
+ready, engines that never stopped serving.
+
+Everything is unset-means-leader, so a single control plane and every
+deployment that existed before this behave exactly as they did.
+
+What this does not do is move client traffic. The standby served inference
+correctly on its own port while the public address returned 502, because
+nothing in front of them moved — that needs a load balancer or a shared
+address, and it is the remaining piece.
+
+### PostgreSQL
+
+The store runs on PostgreSQL, chosen by what the address looks like: a path is
+still SQLite, a `postgres://` URL is not, and `@` followed by a path reads the
+address out of a file so a DSN carrying a password stays out of a configuration
+that gets copied and pasted into issues. Every existing installation keeps
+working untouched.
+
+One set of queries rather than one per database, because the SQL here is almost
+entirely portable and two copies would drift one query at a time. The rewrite
+of placeholders is a wrapper around the handle rather than a change at sixty
+call sites — which found six places reaching past it, including all three
+manual transactions and two prepared statements, every one of which worked on
+SQLite and would have failed on PostgreSQL.
+
+Exactly one query is not the same sentence in both: the note search, MATCH
+against `@@`. SQLite gets full-text search from an FTS5 table kept in step by
+three triggers; PostgreSQL derives a tsvector from the row, so there is nothing
+to keep in step and no trigger to forget. It uses the 'simple' configuration
+rather than English — these are an agent's own notes, in whatever language it
+chose, and stemming English rules over Turkish would quietly stop finding
+things.
+
+`llmgw keys move-store` copies an existing store rather than starting a fresh
+one, because keys are stored hashed: a migration that drops them cannot
+recreate them, and every person and every script would need a new one.
+
+Testing it against a real database found three things nothing else would have.
+`schema_migrations` is created in Go rather than in a migration file, so it
+escaped the port and still said INTEGER — 64 bits in SQLite, 32 in PostgreSQL,
+and a millisecond timestamp does not fit. The copy order followed the order the
+migrations create tables in, and `users` is created after `api_keys` even
+though `api_keys` references it. And `key_claims.sealed` is emptied the moment
+a claim is used, so a claimed row holds an empty blob in a NOT NULL column — an
+empty blob scans as a typed nil slice and a real NULL as an untyped one, a
+difference pgx cannot see, so it arrived as NULL.
+
+The driver is pgx, which is pure Go: the whole binary is built with
+CGO_ENABLED=0 so that it runs on a node whose libc is older than the builder's,
+and a cgo driver would take that away from every deployment, including the ones
+staying on SQLite.
+
 ## v0.7.0 — 2026-09-25
 
 What a cluster needs to be lived with rather than demonstrated: somewhere to

README.md

@@ -31,6 +31,7 @@ Releases track the roadmap phases; see [CHANGELOG.md](CHANGELOG.md).
 | v0.5.0 | 3 | Control plane: catalog, placement, reconciler, ctl | released |
 | v0.6.0 | 4 | Multi-node placement and fabric awareness | released |
 | v0.7.0 | — | Conversations, agent memory, disk management | released |
+| v0.8.0 | — | PostgreSQL, leader election, agent failover | released |
 | v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
 
 As of v0.6.0 a cluster runs unattended: agents report their hardware, the
@@ -51,6 +52,25 @@ without a fabric. The InfiniBand, NVML and NCCL paths are written against the
 formats and failure modes those tools actually have, but they have not yet run
 on a machine that has them.
 
+## Documentation
+
+- [docs/install.md](docs/install.md) — installing from nothing: control plane,
+  nodes, engines, weights, first request
+- [docs/quickstart.md](docs/quickstart.md) — the short path
+- [docs/choosing-models.md](docs/choosing-models.md) — which models to run,
+  how many of each, and where the weights should live
+- [docs/data.md](docs/data.md) — what llmgw stores, what it does not, and
+  what leaves the machine; written for a security review
+- [docs/reference.md](docs/reference.md) — every configuration key, generated
+  from the structs and checked by a test, so it cannot describe a setting that
+  does not exist or miss one that does
+- [deploy/mirror/](deploy/mirror/) — a model mirror, which is the cheapest
+  and most useful thing to build around llmgw
+- [deploy/ha/](deploy/ha/) — running two control planes, and the one health
+  check that has to be right
+- [docs/node-onboarding.md](docs/node-onboarding.md),
+  [docs/onboarding.md](docs/onboarding.md) — adding machines, and adding people
+
 ## Building
 
 ```sh

api/proto/llmgw/v1/agent.proto

@@ -34,6 +34,7 @@ message ControlMessage {
     DrainShard drain = 3;
     Ack ack = 4;
     RemoveWeights remove_weights = 5;
+    Provision provision = 6;
   }
 }
 
@@ -101,6 +102,34 @@ message Register {
   // Empty means the node was not asked to look, exactly as for weights, and
   // constrains nothing.
   repeated Storage storage = 12;
+
+  // host is what the machine is, so the control plane can have the mirror
+  // fetch software built for it rather than assuming every node matches the
+  // first.
+  Host host = 13;
+
+  // engines_probed says engine_versions is an answer. Without it an empty map
+  // meant "an agent too old to say" and constrained nothing — so a freshly
+  // built node with no engine installed yet was a candidate for every model.
+  bool engines_probed = 14;
+}
+
+// Host describes the operating system and processor of a node.
+message Host {
+  // From /etc/os-release: "ubuntu", "24.04", "noble".
+  string os_id = 1;
+  string os_version = 2;
+  string os_codename = 3;
+  // Debian's spelling, which is what package indexes are keyed by: "amd64",
+  // "arm64".
+  string arch = 4;
+  // The vector extensions inference cares about, of those the CPU has. A VM
+  // given QEMU's default CPU model has none of them, and runs, and is five
+  // times slower, and nothing says why.
+  repeated string cpu_flags = 5;
+  // nvidia_driver is the driver version, empty when there is none. A card
+  // without a driver is not a GPU anything can use.
+  string nvidia_driver = 6;
 }
 
 // Storage is one filesystem a node keeps weights on.
@@ -145,6 +174,13 @@ message RunningShard {
 
   string leader_addr = 13;
   int32 leader_port = 14;
+
+  // profile_fp is the fingerprint of the profile this shard was started from,
+  // echoed back from StartShard. The control plane keeps no record of a running
+  // replica across its own restart, so without this a shard adopted after a
+  // restart could not be compared with the catalog — which is exactly when the
+  // catalog is most likely to have been edited.
+  string profile_fp = 15;
 }
 
 message GPU {
@@ -215,6 +251,55 @@ message Heartbeat {
   // report", and the first would quietly empty a node's inventory.
   bool weights_changed = 5;
   repeated LocalWeights weights = 6;
+
+  // memory_total_mb for the same reason again: RAM is added to a virtual
+  // machine while it runs, or across a reboot the agent does not notice, and a
+  // figure read once at registration kept a node that had grown to 32 GB
+  // looking like the 7 GB machine it used to be. 0 means not read.
+  int64 memory_total_mb = 7;
+
+  // engine_versions only when they changed, like weights. An engine installed
+  // after the agent started is otherwise invisible to placement until
+  // somebody restarts the agent — which is a step nobody knows to take.
+  bool engines_changed = 8;
+  map<string, string> engine_versions = 9;
+
+  // provision is where installing the node's software stands, sent when it
+  // changed. The flag makes an empty list mean "nothing to install" rather
+  // than "nothing to say", as for weights.
+  bool provision_changed = 10;
+  repeated ProvisionStatus provision = 11;
+}
+
+// Provision is the software a node should have: which engines, at which
+// versions. Nothing more.
+//
+// Everything else is the node's to work out, because only the node knows it:
+// which build fits its hardware, where that build is published, what its
+// operating system is missing, and whether the result actually runs. The agent
+// converges on this — installs what is absent, checks what is there — and
+// reports each engine through ProvisionStatus. Nothing is taken on trust from
+// what happens to be on the machine: an engine is offered to placement only
+// once the agent has installed it and seen it run.
+message Provision {
+  repeated EngineWant engines = 1;
+}
+
+// EngineWant names one engine at one version.
+message EngineWant {
+  string engine = 1;  // "llamacpp"
+  string version = 2; // "b10991"
+}
+
+// ProvisionStatus is where one engine's installation stands on a node.
+message ProvisionStatus {
+  string engine = 1;
+  string version = 2;
+  string variant = 3;
+  // "installing", "ready", "failed", or "unsupported" when no build fits.
+  string state = 4;
+  // detail says what is happening or what went wrong, in words for a person.
+  string detail = 5;
 }
 
 // ShardStatus reports a shard's lifecycle transition.
@@ -229,6 +314,15 @@ message ShardStatus {
   // fails to load this is the only diagnosis available.
   repeated string last_output = 7;
 
+  // fetch is how far a download has got, when the shard is waiting on one.
+  //
+  // The agent logs this already, on the node, where nobody is looking. From
+  // the control plane a replica pulling a hundred gigabytes and one that is
+  // wedged are the same thing — a shard that has been starting for ten
+  // minutes — and telling them apart is the difference between waiting and
+  // going to look.
+  FetchProgress fetch = 8;
+
   enum State {
     STATE_UNSPECIFIED = 0;
     STATE_PENDING = 1;
@@ -241,6 +335,16 @@ message ShardStatus {
   }
 }
 
+// FetchProgress is a download in flight.
+message FetchProgress {
+  string source = 1;
+  int64 bytes_done = 2;
+  // bytes_total is 0 when the server did not say how large the file is, which
+  // is a real answer: a percentage would then be invented, and an invented
+  // percentage that stops moving is worse than a byte count that keeps rising.
+  int64 bytes_total = 3;
+}
+
 // StartShard launches one shard. It is idempotent by shard_id: the control
 // plane retries, and a repeat must not produce a second process.
 message StartShard {
@@ -267,6 +371,10 @@ message StartShard {
   // are sent so the agent can echo them back in RunningShard after a restart.
   string pool = 13;
   string fabric = 14;
+
+  // profile_fp identifies the version of the profile, so the agent can say
+  // which one a running shard was started from. The agent never interprets it.
+  string profile_fp = 15;
 }
 
 message EngineProfile {

deploy/bootstrap/add-node.sh

@@ -108,6 +108,5 @@ echo "== 2/2  onboarding $NAME"
 echo
 echo "Check it with: llmgw ctl nodes"
 echo
-echo "A node is not finished until the engines its profiles name are installed"
-echo "and the weights are on disk. Onboarding checks that a configured engine"
-echo "runs, but it installs neither — see docs/node-onboarding.md."
+echo "The agent installs its engines itself; the node takes no model until they"
+echo "are installed and seen to run. 'llmgw ctl nodes' shows where that stands."

deploy/bootstrap/onboard-node.sh

@@ -29,10 +29,12 @@ usage: $0 --node ID --addr HOST --server URL [options]
                     --fabric NAME:hca1,hca2 . Omit for a node with no fabric,
                     which can host single-node replicas and will never be given
                     a multi-node one.
-  --mgmt-addr ADDR  address engines are reachable on (default: --addr)
-  --mgmt-if NAME    interface NCCL bootstraps over (default eth0)
-  --llamacpp-bin P  path to llama-server
-  --vllm-bin P      path to the vllm entry point
+  --mgmt-addr ADDR  address engines are reachable on (default: the node's
+                    address on the route to --server, found by the agent)
+  --mgmt-if NAME    interface NCCL bootstraps over (default: the one holding
+                    that address)
+  --llamacpp-bin P  use this llama-server instead of the one the agent installs
+  --vllm-bin P      path to the vllm entry point (not yet installed by the agent)
   --sglang-python P python interpreter of the SGLang virtualenv
   --nvidia-smi PATH where nvidia-smi lives, when it is not on PATH. Containerized
                     driver installs routinely put it somewhere else.
@@ -42,16 +44,19 @@ usage: $0 --node ID --addr HOST --server URL [options]
                     Releases carry static linux/amd64 and linux/arm64 builds;
                     downloading one is better than copying whatever happens to
                     be on the machine you are running this from.
+  --releases URL    where release binaries are downloaded from (default the
+                    project's Gitea releases)
   --ssh-user NAME   (default ilker)
 USAGE
   exit 2
 }
 
-NODE= ADDR= SERVER= MGMT= MGMT_IF=eth0
+NODE= ADDR= SERVER= MGMT_OVERRIDE= MGMT_IF_OVERRIDE=
 NVIDIA_SMI= IB_ROOT=/sys/class/infiniband
 LLAMACPP_BIN= VLLM_BIN= SGLANG_PYTHON=
 SSH_WAIT=180
 PKI=/etc/llmgw/pki BINARY= SSH_USER=ilker
+RELEASES=https://gitea.mynodes.xyz/ilker/llmgw/releases/download
 FABRIC_NAME= FABRIC_HCAS="mlx5_0,mlx5_1"
 
 while [ $# -gt 0 ]; do
@@ -60,8 +65,8 @@ while [ $# -gt 0 ]; do
     --addr) ADDR="$2"; shift 2;;
     --server) SERVER="$2"; shift 2;;
     --fabric) FABRIC_NAME="${2%%:*}"; [ "${2#*:}" != "$2" ] && FABRIC_HCAS="${2#*:}"; shift 2;;
-    --mgmt-addr) MGMT="$2"; shift 2;;
-    --mgmt-if) MGMT_IF="$2"; shift 2;;
+    --mgmt-addr) MGMT_OVERRIDE="$2"; shift 2;;
+    --mgmt-if) MGMT_IF_OVERRIDE="$2"; shift 2;;
     --llamacpp-bin) LLAMACPP_BIN="$2"; shift 2;;
     --vllm-bin) VLLM_BIN="$2"; shift 2;;
     --sglang-python) SGLANG_PYTHON="$2"; shift 2;;
@@ -69,12 +74,12 @@ while [ $# -gt 0 ]; do
     --ib-root) IB_ROOT="$2"; shift 2;;
     --pki) PKI="$2"; shift 2;;
     --binary) BINARY="$2"; shift 2;;
+    --releases) RELEASES="$2"; shift 2;;
     --ssh-user) SSH_USER="$2"; shift 2;;
     *) usage;;
   esac
 done
 [ -n "$NODE" ] && [ -n "$ADDR" ] && [ -n "$SERVER" ] || usage
-MGMT="${MGMT:-$ADDR}"
 BINARY="${BINARY:-$(command -v llmgw)}"
 [ -x "$BINARY" ] || { echo "no llmgw binary; pass --binary" >&2; exit 1; }
 [ -f "$PKI/ca-key.pem" ] || { echo "no authority in $PKI; run 'llmgw ctl cert init-ca' first" >&2; exit 1; }
@@ -130,68 +135,73 @@ $SSH "set -e
   rm -f /tmp/ca.pem /tmp/$NODE.pem /tmp/$NODE-key.pem"
 
 # 4. The binary, the configuration and the unit.
-say "installing the binary"
-scp -q $SSHOPTS "$BINARY" "$SSH_USER@$ADDR:/tmp/llmgw"
-$SSH 'sudo install -m 0755 /tmp/llmgw /usr/local/bin/llmgw && rm -f /tmp/llmgw'
+# The node fetches the release CI built for this version, for its own
+# architecture, and checks it against the release's checksums — rather than
+# being handed whatever binary is on the machine running this script. A build
+# that is not a release (a developer's, with commits past the tag) has no such
+# file, and is copied as before.
+TAG="$("$BINARY" --version 2>/dev/null | awk '{print $NF}')"
+if printf '%s' "$TAG" | grep -Eq '^v[0-9]+\.[0-9]+\.[0-9]+$'; then
+  say "installing llmgw $TAG from the release"
+  $SSH "set -e
+    arch=\$(dpkg --print-architecture)
+    name=llmgw-$TAG-linux-\$arch
+    cd /tmp && rm -f \$name checksums.txt
+    curl -fsSL -o \$name '$RELEASES/$TAG/'\$name
+    curl -fsSL -o checksums.txt '$RELEASES/$TAG/checksums.txt'
+    grep \" \$name\\\$\" checksums.txt | sha256sum -c - >/dev/null
+    sudo install -m 0755 \$name /usr/local/bin/llmgw
+    rm -f \$name checksums.txt"
+else
+  say "installing the binary (${TAG:-unversioned}, not a release)"
+  scp -q $SSHOPTS "$BINARY" "$SSH_USER@$ADDR:/tmp/llmgw"
+  $SSH 'sudo install -m 0755 /tmp/llmgw /usr/local/bin/llmgw && rm -f /tmp/llmgw'
+fi
 
 say "writing the configuration"
-FABRICS="fabrics: []"
-if [ -n "$FABRIC_NAME" ]; then
-  FABRICS="fabrics:
-  - name: $FABRIC_NAME
-    hcas: [$(echo "$FABRIC_HCAS" | sed 's/,/, /g')]"
-fi
-cat > "$TMP/agent.yaml" <<YAML
-# $NODE
-#
-# Written by onboard-node.sh. Fabric membership is configuration and there is no
-# way around it: OpenSM's default subnet prefix is the same on every fabric, so
-# a matching prefix proves nothing and a wrong guess produces a replica that
-# starts, passes health checks and then hangs on its first all-reduce.
-node_id: $NODE
-server: "$SERVER"
-
-tls:
-  ca: /etc/llmgw/ca.pem
-  cert: /etc/llmgw/agent.pem
-  key: /etc/llmgw/agent-key.pem
-
-mgmt_addr: "$MGMT"
-mgmt_interface: $MGMT_IF
-
-heartbeat_interval: 5s
-reconnect_base: 1s
-reconnect_max: 30s
-
-$FABRICS
-
-discovery:
-  # Empty means look on PATH. Worth setting where the driver lives outside the
-  # default paths, which containerized installs routinely do.
-  nvidia_smi: "$NVIDIA_SMI"
-  ib_root: $IB_ROOT
-  prefer_nvml: false
-  skip_gpus: false
-
-# Where the engines live. Empty means "on PATH", which is rarely true: a node
-# that cannot find its engine registers healthy and then fails every shard
-# placed on it with "executable file not found".
-engines:
-  llamacpp:
-    bin: "$LLAMACPP_BIN"
-    parallel: 4
-  vllm:
-    bin: "$VLLM_BIN"
-    launch_mode: ray
-  sglang:
-    python: "$SGLANG_PYTHON"
-
-supervisor:
-  work_dir: /var/lib/llmgw-agent
-  log_lines: 512
-  stop_timeout: 60s
-  health_interval: 2s
-YAML
+NODE_IS_HOSTNAME=no
+[ "$($SSH hostname 2>/dev/null)" = "$NODE" ] && NODE_IS_HOSTNAME=yes
+# Only what the node cannot find out for itself. Its name is its hostname, its
+# address is the one on the route to the control plane, llama-server is looked
+# for on PATH and where release archives are unpacked, GPUs are there when
+# nvidia-smi is, and memory, disk and engines are re-read while it runs. Every
+# setting written here is one that has to be kept true by hand — so an option
+# below is written only when it was given, as an override.
+{
+  echo "# $NODE, written by onboard-node.sh."
+  echo "server: \"$SERVER\""
+  echo
+  echo "tls:"
+  echo "  ca: /etc/llmgw/ca.pem"
+  echo "  cert: /etc/llmgw/agent.pem"
+  echo "  key: /etc/llmgw/agent-key.pem"
+  # The certificate's common name is --node, and the control plane refuses a
+  # stream whose node id differs from it. Said explicitly only when the
+  # hostname would say something else.
+  [ "$NODE_IS_HOSTNAME" = yes ] || echo "node_id: $NODE"
+  [ -n "$MGMT_OVERRIDE" ] && echo "mgmt_addr: \"$MGMT_OVERRIDE\""
+  [ -n "$MGMT_IF_OVERRIDE" ] && echo "mgmt_interface: $MGMT_IF_OVERRIDE"
+  if [ -n "$FABRIC_NAME" ]; then
+    # Fabric membership is the one thing that cannot be found: OpenSM's default
+    # subnet prefix is the same on every fabric, so a matching prefix proves
+    # nothing and a wrong guess hangs the first all-reduce.
+    echo "fabrics:"
+    echo "  - name: $FABRIC_NAME"
+    echo "    hcas: [$(echo "$FABRIC_HCAS" | sed 's/,/, /g')]"
+  fi
+  if [ -n "$NVIDIA_SMI" ] || [ "$IB_ROOT" != /sys/class/infiniband ]; then
+    echo "discovery:"
+    [ -n "$NVIDIA_SMI" ] && echo "  nvidia_smi: \"$NVIDIA_SMI\""
+    [ "$IB_ROOT" != /sys/class/infiniband ] && echo "  ib_root: $IB_ROOT"
+  fi
+  if [ -n "$LLAMACPP_BIN$VLLM_BIN$SGLANG_PYTHON" ]; then
+    echo "engines:"
+    [ -n "$LLAMACPP_BIN" ] && printf '  llamacpp:\n    bin: "%s"\n' "$LLAMACPP_BIN"
+    [ -n "$VLLM_BIN" ] && printf '  vllm:\n    bin: "%s"\n' "$VLLM_BIN"
+    [ -n "$SGLANG_PYTHON" ] && printf '  sglang:\n    python: "%s"\n' "$SGLANG_PYTHON"
+  fi
+  true
+} > "$TMP/agent.yaml"
 scp -q "$TMP/agent.yaml" "$SSH_USER@$ADDR:/tmp/agent.yaml"
 
 cat > "$TMP/llmgw-agent.service" <<'UNIT'
@@ -255,8 +265,9 @@ $SSH 'set -e
 
 echo
 if [ -z "$LLAMACPP_BIN$VLLM_BIN$SGLANG_PYTHON" ]; then
-  say "no engine paths given; this node can only run the 'sim' engine until"
-  say "one is installed and the configuration points at it"
+  say "the agent now installs the engines the control plane asks for: the"
+  say "build for this machine, checked against its publisher's checksum and run"
+  say "before the node is offered any model. 'llmgw ctl nodes' shows progress."
 fi
 if [ -n "$ENGINE_PROBLEMS" ]; then
   echo "$NODE is onboarded but an engine will not run:$ENGINE_PROBLEMS"

deploy/examples/agent.yaml

@@ -4,12 +4,19 @@
 # keep serving if the agent restarts: the router talks to them directly, and
 # orphan adoption picks them up again on the next start.
 
-node_id: gpu-a-01          # defaults to the hostname
+# Everything a node can find out about itself, it does: its id is its hostname,
+# its address is the one on the route to the control plane, llama-server is
+# looked for on PATH and in /opt/llama.cpp, GPUs are there when nvidia-smi is,
+# weights are fetched into /var/lib/llmgw-weights and looked for there, and
+# memory, disk and installed engines are re-read while the agent runs. So a
+# plain CPU node needs only the two blocks below, and the same file on every
+# node. What follows them are overrides, for when the defaults are wrong.
+
 server: "http://llmgw-server.internal:7443"
 
 # The agent's half of mutual TLS. The certificate's common name must be this
-# node's id, because the control plane compares the two and refuses the stream
-# when they disagree:
+# node's id — its hostname, unless node_id says otherwise — because the control
+# plane compares the two and refuses the stream when they disagree:
 #
 #   llmgw ctl cert issue-client gpu-a-01 --dir /etc/llmgw/pki
 #
@@ -22,13 +29,19 @@ server: "http://llmgw-server.internal:7443"
 #   cert: /etc/llmgw/agent.pem
 #   key: /etc/llmgw/agent-key.pem
 
+# ---- overrides ---------------------------------------------------------------
+
+# node_id: gpu-a-01          # when it is not the hostname
+
 # The address engines are reachable on, and the address NCCL bootstraps over.
 # Never a fabric address: NCCL rendezvouses over TCP before moving to RDMA.
-mgmt_addr: "10.0.0.11"
-mgmt_interface: eth0
+# mgmt_addr: "10.0.0.11"
+# mgmt_interface: eth0
 
-labels:
-  rack: r12
+# Free-form, for a profile's node_selector. Nothing needs them; do not restate
+# what is measured (memory, GPUs, engines), because a label goes stale.
+# labels:
+#   rack: r12
 
 heartbeat_interval: 5s
 reconnect_base: 1s
@@ -55,12 +68,9 @@ discovery:
   # tensor-parallel group off the PCIe path inside a node. It needs a binary
   # built with `-tags nvml`; otherwise this silently falls back to nvidia-smi.
   prefer_nvml: true
-  skip_gpus: false
 
-# Where this node keeps model weights.
-#
-# Unset by default, and unset means the node is not asked to look — which is
-# not the same as having nothing, and constrains nothing. Storage differs at
+# Where this node keeps model weights besides /var/lib/llmgw-weights, which is
+# where fetched weights land and is always looked in. Storage differs at
 # every installation and llmgw deliberately does not own it: a Lustre mount, an
 # image exported over the fabric with SRP or NVMe-oF, local NVMe, or a copy
 # made by hand all look identical from here.
@@ -74,6 +84,15 @@ discovery:
 #   - /var/lib/llmgw-models
 #   - /mnt/lustre/models
 
+# The agent installs the engines the control plane names (see `engines:` in
+# server.yaml) under /opt/llmgw/engines: the build for this machine — CUDA when
+# an NVIDIA driver is loaded — checked against its publisher's checksum, with
+# the OS packages it needs, and run before the node takes any model. llama.cpp
+# is installed this way today. Naming a binary below opts that engine out and
+# uses yours instead; vLLM and SGLang still have to be named until the agent
+# installs them too.
+# engines_dir: /opt/llmgw/engines
+
 engines:
   vllm:
     bin: /opt/venvs/vllm/bin/vllm

deploy/examples/server.yaml

@@ -197,6 +197,13 @@ router:
     session_load_factor: 1.25
 
 # How aggressively the control plane reconciles.
+# The engines every node installs, and at which version. Unset keeps the
+# versions this release was checked against (llama.cpp b10991). Each agent
+# picks the build for its own hardware and installs it itself; changing a
+# version here reaches every node on its next connect.
+# engines:
+#   llamacpp: b10991
+
 control:
   interval: 5s
   base_backoff: 10s

deploy/ha/README.md

@@ -0,0 +1,87 @@
+# Two control planes
+
+llmgw handles the hard half of this itself: only one control plane places work,
+and agents find whichever one is leading. What it does not do is move client
+traffic, because that happens in front of both and llmgw is not in front of
+anything.
+
+This directory is the missing half. It is a pattern rather than a product:
+substitute your own load balancer if you have one.
+
+## What llmgw does on its own
+
+**One leader.** Leadership is a lease in the shared store. A second control
+plane starts, sees the lease, and stands by — it places nothing, evicts
+nothing and drains nothing. Start one by accident and your cluster is
+unharmed.
+
+**Agents follow.** A standby refuses agent streams, so agents move to the
+leader by themselves. Give each agent both addresses:
+
+```yaml
+servers:
+  - "https://ctl-1.internal:7443"
+  - "https://ctl-2.internal:7443"
+```
+
+Measured on a cluster: leadership moved three seconds after the leader was
+stopped, every agent had arrived six seconds later, and every replica was
+adopted rather than restarted — the engines never stopped serving.
+
+## What it does not do
+
+Client traffic. During that same failover the standby served inference
+correctly on its own port while the public address returned 502, because
+nothing in front of the two had moved.
+
+So something has to. The options, in the order most people should consider
+them:
+
+**A load balancer you already run.** Health-check `/ready` on each control
+plane and send traffic to whichever answers. This is the right answer if you
+have one, and it needs nothing else from this directory.
+
+**A floating address.** `keepalived.conf.example` here moves one IP between
+the two machines. Fewer moving parts than a load balancer and no extra hop,
+at the cost of one machine carrying all the traffic.
+
+**DNS.** Works, eventually. A record with a short TTL is a failover measured
+in whatever your resolvers actually honour rather than what you set, which on
+a bad day is minutes. Fine for a cluster people use interactively, wrong for
+one an agent is driving.
+
+## Check `/ready`, not `/health`
+
+This is the one thing to get right, and the obvious choice is the wrong one.
+
+`/health` says the process is up. It says nothing about whether it can serve,
+and a standby is exactly that case: alive, answering, and holding no endpoints
+at all — its reconciler placed nothing and it has no agent streams to adopt
+from. A load balancer checking `/health` would send it half the traffic, and
+half the traffic would be a 503.
+
+`/ready` says this process has somewhere to send a request. It is 503 on a
+standby, 503 on a leader whose replicas have all failed, and 200 when there is
+something to serve — which is the question a load balancer is actually asking.
+Both are open, because a load balancer has no key.
+
+During a failover neither passes for about the length of the lease handover:
+three seconds when the leader is stopped cleanly, up to the lease TTL when it
+is not. That gap is real and there is no way to have it both ways — traffic
+cannot go to a control plane that has nothing to route to.
+
+Not the admin port, for a different reason: it serves pprof, and pprof can
+dump the process heap, where API keys sit in plaintext.
+
+## Both control planes need the same store
+
+Leadership lives in it, and so do keys, budgets and conversations. Two control
+planes on two SQLite files are two clusters that happen to share a name.
+
+```yaml
+auth:
+  store: "@/etc/llmgw/secrets/store-dsn"
+```
+
+See [docs/install.md](../../docs/install.md) for moving an existing
+installation onto PostgreSQL without reissuing every key.

deploy/ha/keepalived.conf.example

@@ -0,0 +1,65 @@
+# A floating address for two control planes.
+#
+# One IP that lives on whichever machine is up. Clients and the reverse proxy
+# in front of llmgw point at the address, not at either host, so a failover is
+# invisible to them.
+#
+# This is not a load balancer: all traffic goes to the holder. That is usually
+# the right trade for a control plane, which serves far less than the nodes
+# it schedules.
+#
+# Copy to /etc/keepalived/keepalived.conf on both machines, changing state,
+# priority and interface.
+
+global_defs {
+    enable_script_security
+    script_user nobody
+}
+
+# The address moves only while llmgw can actually serve, not merely while the
+# machine is powered on.
+#
+# /ready, not /health. /health is 200 on a standby, which is up, answering and
+# holding no endpoints at all — so the address would sit on a host that refuses
+# every request. /ready is 200 only when there is somewhere to send one.
+#
+# Not the admin port either: it serves pprof, which can dump the heap that
+# holds every API key.
+vrrp_script check_llmgw {
+    script "/usr/bin/curl -fsS --max-time 2 -o /dev/null http://127.0.0.1:8080/ready"
+    interval 2
+    timeout 3
+    # Two failures before giving up the address, so one slow moment does not
+    # move it. Two successes before taking it back, so a flapping gateway does
+    # not drag the address with it.
+    fall 2
+    rise 2
+    weight -40
+}
+
+vrrp_instance llmgw {
+    # MASTER on the machine you would rather serve; BACKUP on the other.
+    state MASTER
+    interface eth0
+    # Any number both machines agree on and nothing else on the segment uses.
+    virtual_router_id 51
+    # The higher one wins while both are healthy. The weight above is enough
+    # to invert this when the preferred machine's gateway is not answering.
+    priority 100
+    advert_int 1
+
+    authentication {
+        auth_type PASS
+        # Change this. It stops another host on the segment claiming the
+        # address, which is otherwise a single unicast packet away.
+        auth_pass CHANGEME
+    }
+
+    virtual_ipaddress {
+        10.0.0.150/24
+    }
+
+    track_script {
+        check_llmgw
+    }
+}

deploy/mirror/README.md

@@ -0,0 +1,262 @@
+# A model mirror
+
+One machine that holds the model files, so nodes fetch over your own network
+instead of over the internet.
+
+This is the highest-value piece of infrastructure around llmgw and the cheapest.
+Eight nodes each pulling forty gigabytes from Hugging Face is a day's work, and
+on plenty of corporate networks it is not possible at all. From a mirror on the
+same switch it is minutes.
+
+llmgw needs nothing special for this: a model's `source` is a URL, and the URL
+can be yours.
+
+## What it is
+
+An HTTP server serving a directory. That is the whole design.
+
+```
+/srv/models/
+  qwen2.5-14b-instruct/
+    2026-03/
+      q4_k_m.gguf
+  bge-m3/
+    2026-01/
+      model.safetensors
+```
+
+The date in the path is the point of the layout, not decoration. A model
+repository that changes upstream changes what you are running, and the first
+sign is a prompt that worked last week answering differently this week with
+nobody able to say why. Pin it here and that cannot happen.
+
+## Running it
+
+`llmgw` is the mirror. Same binary as the control plane and the agents, one
+subcommand:
+
+```
+sudo mkdir -p /srv/models && sudo chown llmgw:llmgw /srv/models
+llmgw mirror serve --dir /srv/models --base http://models.internal:8081
+```
+
+There is a unit in `deploy/systemd/llmgw-mirror.service`. Set `--base` to the
+address nodes reach this machine on — not `localhost`, and not a name only this
+host knows itself by. It is the string that ends up in catalog entries, and a
+wrong one makes `mirror unused` report files as unreferenced when they are in
+use. The next step after that list is deletion.
+
+### Two listeners
+
+`--listen :8081` serves the weights. Read-only, unauthenticated, Range requests
+supported so an interrupted fetch on a node resumes rather than restarts. This
+is the path eight nodes pull forty gigabytes from.
+
+`--api-listen :8082` is the management API. Separate on purpose: putting a
+client certificate in front of the byte path would mean re-certifying every
+agent's fetcher in order to change how models are administered.
+
+The API is unauthenticated unless you give it the fleet's CA, and it says so at
+startup rather than leaving you to find out. Anyone who can reach that port can
+start downloads and delete models, so either keep it on a trusted network or
+hand it certificates:
+
+```
+llmgw mirror serve --dir /srv/models --base http://models.internal:8081 \
+  --tls-ca /etc/llmgw/ca.pem \
+  --tls-cert /etc/llmgw/mirror.pem \
+  --tls-key /etc/llmgw/mirror-key.pem
+```
+
+That is the same CA the agents already use, so the certificates come from the
+authority you have:
+
+```
+llmgw ctl cert issue-server mirror --host models.internal --host 10.0.0.146
+llmgw ctl cert issue-client control-plane
+```
+
+`--host` has to cover every address the API is called on, because that is what
+a client verifies against. Both sides are checked — the mirror will not accept a
+caller without a certificate from this CA, the same rule the agent stream
+follows.
+
+### nginx instead
+
+If you already run nginx, it serves a directory perfectly well and you do not
+need `mirror serve` for the bytes:
+
+```nginx
+server {
+    listen 80;
+    server_name models.internal;
+
+    root /srv/models;
+
+    # Read-only. This is deliberate: a mirror that can be written to over HTTP
+    # is a way to replace a model with another one, and the nodes would fetch
+    # it without complaint.
+    location / {
+        autoindex on;
+        # Large files, and the client is a machine with a resumable fetcher.
+        # Sendfile and no buffering keep memory flat regardless of size.
+        sendfile on;
+        tcp_nopush on;
+    }
+}
+```
+
+`autoindex` is there so a person can browse what is available; llmgw does not
+need it.
+
+## Filling it
+
+From the mirror itself:
+
+```
+llmgw mirror add \
+  --model qwen2.5-14b-instruct --version 2026-03 \
+  "https://huggingface.co/<repo>/resolve/<revision>/q4_k_m.gguf"
+```
+
+Use a revision rather than `main` in the source URL. `main` is what moves.
+
+It downloads to a temporary name and renames when complete, so an interrupted
+download never leaves something a catalog could be pointed at, and it prints the
+sha256 when it lands. Compare that against what upstream publishes — Hugging
+Face gives it as `lfs.oid` from
+`https://huggingface.co/api/models/<repo>/tree/<revision>`. A mirror is the one
+place in this system where a corrupt file is copied to every node instead of
+being caught once.
+
+It refuses to overwrite a version that is already there. Add a new one beside
+it: that refusal is what makes the old weights still available when the new ones
+turn out to be worse.
+
+## Driving it from elsewhere
+
+The same fetch over the API, so the control plane — or anything else — can ask
+for a model without anyone logging in:
+
+```
+curl -X POST http://models.internal:8082/v1/models -d '{
+  "source": "https://huggingface.co/<repo>/resolve/<revision>/q4_k_m.gguf",
+  "model": "qwen2.5-14b-instruct",
+  "version": "2026-03"
+}'
+```
+
+The mirror does the downloading. A control plane that pulled the bytes itself
+and pushed them here would carry every model twice over the network, through the
+one machine least able to spare the bandwidth.
+
+It answers `202` with the URL a catalog entry should use, and the bytes are not
+here yet. Poll for them:
+
+| | |
+|---|---|
+| `GET /v1/models` | everything held or arriving, and the base to build URLs from |
+| `POST /v1/models` | fetch this source into model/version |
+| `GET /v1/models/{model}/{version}` | `fetching` with a byte count, `present`, or `failed` with the reason |
+| `DELETE /v1/models/{model}/{version}` | remove a version; refuses while it is downloading |
+| `GET /v1/health` | up *and* able to read its own directory |
+
+A failed fetch stays `failed` with what went wrong, rather than disappearing:
+otherwise the thing waiting on that model waits forever with nothing to say why.
+Asking twice for the same download joins the one already running instead of
+starting a rival writer on the same file.
+
+`DELETE` takes the whole version, never one file of it. A model split across
+several files is useless with one missing, and half a version on disk looks
+present and will not load.
+
+## Retiring a model
+
+```
+llmgw mirror unused     # what no catalog entry refers to
+llmgw mirror remove --model <m> --version <v> --file <f>
+```
+
+`unused` compares against every `source` in the running catalog, so the answer
+is what can go rather than what looks old. It refuses to answer when it cannot
+reach the control plane: an empty catalog and an unreachable one produce the
+same list — everything — and the obvious next step after that list is deletion.
+
+Retire the catalog entry first, let the fleet settle, then remove. A node that
+has to refetch a file you deleted cannot, and the failure arrives later, on the
+node, looking like something else.
+
+## From the control plane's admin page
+
+Configure the mirror on the control plane and the same operations appear at
+`/admin/mirror`, behind the admin session the other admin pages use:
+
+```yaml
+mirror:
+  api: https://models.internal:8082
+  tls:
+    ca: /etc/llmgw/ca.crt
+    cert: /etc/llmgw/control-plane.crt
+    key: /etc/llmgw/control-plane.key
+```
+
+Leave the block out and the page is not mounted at all, which is the right page
+for a deployment with no mirror — not an empty one pointing at nothing.
+
+The page lists what is held, what is arriving with its byte count, and what
+failed with the reason; it starts a fetch, retries a failed one, and removes a
+version. It reloads itself only while something is downloading, because a reload
+at any other moment takes with it whatever is half-typed in the form.
+
+Two things it deliberately will not do. It cannot point the control plane at a
+different mirror — that is configuration, and changing it needs access to the
+host rather than a session. And it does not write the catalog for you: it shows
+the URL an entry should use, and putting it there is still a deliberate act,
+because that is the moment a model starts being served.
+
+The address is a mistake worth naming: `api` is the **management** port, not the
+one nodes fetch bytes from. They are two listeners so this one can be locked
+down without re-certifying every agent's fetcher.
+
+## Pointing llmgw at it
+
+```yaml
+- name: qwen2.5-14b-instruct
+  source: http://models.internal/qwen2.5-14b-instruct/2026-03/q4_k_m.gguf
+  scaling: { min: 2, max: 2 }
+  profiles:
+    - name: gpu
+      engine: vllm
+      gpu_class: h100-80g
+      tp: 1
+```
+
+Each node fetches once, in the background, before its engine starts — and
+`llmgw ctl replicas` shows how far along it is while it does. A killed download
+resumes rather than restarting, and a partial file is never visible under the
+model's name, so an interrupted fetch cannot be handed to an engine.
+
+## What this is not
+
+It is not a cache. Nothing populates it on its own: a model is there because
+somebody asked for it, by hand or over the API, and one nobody asked for is not
+there. That is on purpose. A mirror that fetched on demand would reach the
+internet at the least convenient moment, and in a network where that is blocked
+it would fail in a way that looks like llmgw being broken.
+
+It is also not a replacement for a shared filesystem. If you already run one
+that every node mounts, put the models on it and use a path as the `source` —
+nothing is fetched at all, and llmgw notices the filesystem is shared and will
+not count the same free space once per node.
+
+## When this stops being enough
+
+Somewhere above ten or twenty nodes, one mirror serving a forty gigabyte file
+to all of them at once becomes the bottleneck instead of the fix. At that point
+the options are a filer, a second mirror per rack, or distribution between the
+nodes themselves.
+
+llmgw knows which nodes already hold which weights — it is in `ctl status` as
+"N models on X/Y nodes" — so distributing between them is a feature it is
+positioned to add. It has not been built, because a mirror is enough for
+everyone who has needed it so far.

deploy/monitoring/README.md

@@ -0,0 +1,170 @@
+# Watching a cluster
+
+Prometheus, Alertmanager and Grafana, configured for what llmgw actually
+publishes. Every alert here is written against a series this system emits;
+nothing is aimed at a metric that might exist.
+
+## What it is
+
+One scrape target for the whole fleet.
+
+Agents publish no metrics of their own. The control plane gathers what they
+report — node state, memory, disk, engine queue depth, KV occupancy — and
+republishes all of it, so a single endpoint describes every machine. That is
+worth knowing before going looking for an exporter on each node.
+
+## The port it scrapes
+
+`metrics_listen`, not `admin_listen`.
+
+The admin listener is bound to loopback for one reason: it also serves pprof,
+and pprof can dump the process heap, where API keys sit in plaintext. Metrics
+are the one thing on that listener something else genuinely needs to reach, so
+they get a port of their own instead of becoming an argument for opening the
+other one.
+
+```yaml
+# /etc/llmgw/server.yaml
+metrics_listen: ":9091"
+```
+
+Confirm it carries nothing else:
+
+```
+curl -s -o /dev/null -w '%{http_code}\n' http://<control-plane>:9091/debug/pprof/   # 404
+curl -s -o /dev/null -w '%{http_code}\n' http://<control-plane>:9091/api/v1/nodes   # 404
+```
+
+## Installing
+
+On Ubuntu 24.04 the first two come from the archive; Grafana needs its own
+repository.
+
+```
+apt-get install prometheus prometheus-alertmanager
+
+install -m 0644 prometheus.yml            /etc/prometheus/prometheus.yml
+install -m 0644 llmgw-rules.yml           /etc/prometheus/rules/llmgw.yml
+install -m 0644 alertmanager.yml          /etc/prometheus/alertmanager.yml
+promtool check config /etc/prometheus/prometheus.yml
+systemctl restart prometheus prometheus-alertmanager
+
+install -m 0644 grafana/datasource.yaml   /etc/grafana/provisioning/datasources/llmgw.yaml
+install -m 0644 grafana/dashboards.yaml   /etc/grafana/provisioning/dashboards/llmgw.yaml
+install -m 0644 -o grafana -g grafana grafana/llmgw-overview.json \
+    /var/lib/grafana/dashboards/llmgw.json
+systemctl restart grafana-server
+```
+
+The dashboard is provisioned from a file rather than clicked together, so it
+survives a reinstall and can be reviewed in a diff.
+
+## Where to run it
+
+Not on the control plane. Monitoring that dies with the thing it monitors tells
+you nothing at the moment you need it, and the control plane is also the busiest
+machine in a small cluster.
+
+Anywhere else on the management network will do. The mirror is a reasonable
+host: it has the disk, and serving weights is bursty rather than constant.
+
+## Nothing is sent anywhere
+
+Alertmanager groups and holds alerts and has no receiver.
+
+Where an alert should go — an address, a Slack workspace, a webhook — is a
+decision with credentials attached, and a monitoring system that quietly starts
+mailing people is worse than one that is visibly not wired up. Alerts are
+visible at `:9093` and in Prometheus; add a receiver to `alertmanager.yml` to
+make them travel. `critical` already routes separately so it can be pointed
+somewhere else without touching the rest.
+
+## Check that it fires
+
+An alerting system nobody has seen fire is an assumption.
+
+```
+llmgw ctl cordon <node>     # work already there keeps running
+# ~2 minutes later
+curl -s http://<host>:9093/api/v2/alerts | jq '.[].labels'
+llmgw ctl uncordon <node>   # the alert resolves on its own
+```
+
+Cordoning is the cheapest real test: it changes a metric the rules read, stops
+nothing, and reverses in one command.
+
+## What the alerts assume
+
+**A stopped model publishes no series.** `llmgw_model_replicas_ready` exists
+only for models with a replica placed, so `== 0` cannot fire for one that is
+deliberately off. That is what makes an on-demand catalogue quiet here.
+
+**Saturation of 1 is the healthy limit, not failure.** It is the worse of queue
+depth and KV occupancy, measured by `ctl bench` rather than guessed. The alert
+allows five minutes above it before saying anything.
+
+**Latencies are in seconds.** These thresholds are written for CPU inference. A
+p99 first token of thirty seconds is the alerting point here and would be
+absurd on a GPU; change it when the hardware changes.
+
+**Disk is only reported by nodes that hold weights.** A node with no model
+directory reports nothing rather than zero, and the rule divides free by total
+so a node that reports neither never fires.
+
+## Where the metrics live
+
+On the machine that scrapes them, in Prometheus's own store, and that is the
+right answer at this size. Measured on this cluster: 2,359 series at 41 samples
+a second, which compresses to roughly 7 MB a day — about 3 GB for a year.
+
+```
+# /etc/default/prometheus
+ARGS="--storage.tsdb.retention.time=365d --storage.tsdb.retention.size=20GB"
+```
+
+A year because the questions worth asking of this data are seasonal: what did
+capacity look like last quarter, was it always this slow, how much did the
+fleet grow. Fifteen days answers none of them.
+
+Both limits, not one. The time limit is the intent; the size cap is what stops
+Prometheus becoming the thing that fills the disk if the series count grows
+tenfold after somebody adds a fleet.
+
+No remote write, no object store, no second tier. At two thousand series that
+is infrastructure for a problem this cluster does not have, and every one of
+those has its own failure modes to learn.
+
+### The part that is not about capacity
+
+The disk is not the constraint — 900 GB free against 3 GB a year. What matters
+is what shares it.
+
+On this cluster the model mirror and the metrics database are on the same
+filesystem. A mirror that fills it stops Prometheus writing, so monitoring goes
+quiet at exactly the moment the disk is the problem, and the alert that would
+have said so is the one that died.
+
+Two ways out, in order of how much they cost:
+
+- Watch it, which is now done. `prometheus-node-exporter` arrives with the
+  Prometheus package and was running, unscraped, on this machine — so the disk
+  holding both was watched by nothing. It is a scrape target now, and
+  `MonitoringDiskFilling` fires at fifteen percent rather than five, because
+  the thing being protected is the ability to be told.
+- Separate them: a volume of its own for `/var/lib/prometheus`, or monitoring
+  on its own machine. Worth doing before the mirror holds a few hundred
+  gigabytes; not worth doing today at seven percent used.
+
+## What is not here
+
+No exporter on the llmgw nodes, so no host CPU, load average or network for
+them. The control plane publishes memory, disk and liveness for every node,
+which is what placement decides on; the rest is a `node_exporter` away if
+somebody wants it. The monitoring host itself is scraped, for the reason above.
+
+No probe of the mirror. Its `/v1/health` is JSON rather than Prometheus text,
+and checking it properly means `blackbox_exporter`.
+
+No recording rules. At this size every query in the dashboard runs fast enough
+raw, and a recording rule is a second definition of a number to keep in step
+with the first.

deploy/monitoring/alertmanager.yml

@@ -0,0 +1,41 @@
+# Alertmanager for the llmgw cluster.
+#
+# Nothing is sent anywhere yet, on purpose. Where an alert should go — an
+# address, a Slack workspace, a webhook — is a decision with credentials
+# attached, and a monitoring system that quietly starts mailing people is worse
+# than one that is visibly not wired up. Alerts group and fire here and are
+# visible at :9093 and in Prometheus; add a receiver below to make them travel.
+global:
+  resolve_timeout: 5m
+
+route:
+  # By alert and by what it is about, so five nodes going quiet in the same
+  # network blip arrive as one notification rather than five.
+  group_by: ['alertname', 'node', 'model']
+  group_wait: 30s
+  group_interval: 5m
+  # Long, because a repeat is a reminder and not news. A cluster with one thing
+  # wrong should not produce a message an hour.
+  repeat_interval: 12h
+  receiver: none
+
+  routes:
+    # Critical gets its own branch so it can be pointed somewhere else later
+    # without touching the rest: the difference between "a node is draining"
+    # and "nothing is serving" is who should be woken.
+    - matchers: [severity="critical"]
+      receiver: none
+      group_wait: 10s
+      repeat_interval: 4h
+
+receivers:
+  - name: none
+
+inhibit_rules:
+  # A node that is down explains every replica on it. Reporting both says the
+  # same thing twice and buries whichever is the actual cause.
+  - source_matchers: [alertname="NodeNotReady"]
+    target_matchers: [alertname="EndpointMetricsStale"]
+    equal: ['node']
+  - source_matchers: [alertname="ControlPlaneUnreachable"]
+    target_matchers: [severity="warning"]

deploy/monitoring/grafana/dashboards.yaml

@@ -0,0 +1,11 @@
+apiVersion: 1
+providers:
+  - name: llmgw
+    orgId: 1
+    folder: llmgw
+    type: file
+    disableDeletion: false
+    updateIntervalSeconds: 30
+    allowUiUpdates: true
+    options:
+      path: /var/lib/grafana/dashboards

deploy/monitoring/grafana/datasource.yaml

@@ -0,0 +1,11 @@
+# Provisioned rather than clicked in, so the dashboard below can reference it by
+# a fixed uid and the whole setup survives a reinstall of the machine.
+apiVersion: 1
+datasources:
+  - name: Prometheus
+    uid: llmgw-prom
+    type: prometheus
+    access: proxy
+    url: http://localhost:9090
+    isDefault: true
+    editable: false

deploy/monitoring/grafana/llmgw-overview.json

@@ -0,0 +1,492 @@
+{
+  "uid": "llmgw-overview",
+  "title": "llmgw \u2014 cluster",
+  "tags": [
+    "llmgw"
+  ],
+  "timezone": "browser",
+  "schemaVersion": 39,
+  "version": 1,
+  "refresh": "30s",
+  "time": {
+    "from": "now-6h",
+    "to": "now"
+  },
+  "panels": [
+    {
+      "id": 1,
+      "title": "Nodes ready",
+      "type": "stat",
+      "description": "Nodes taking work. A node that is known but draining or cordoned is not counted.",
+      "gridPos": {
+        "x": 0,
+        "y": 0,
+        "w": 4,
+        "h": 4
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "count(llmgw_node_up == 1)",
+          "legendFormat": "ready",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {},
+        "overrides": []
+      },
+      "options": {
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        },
+        "textMode": "auto",
+        "colorMode": "value"
+      }
+    },
+    {
+      "id": 2,
+      "title": "Replicas serving",
+      "type": "stat",
+      "description": "Across every model. A model deliberately stopped contributes nothing and is not a fault.",
+      "gridPos": {
+        "x": 4,
+        "y": 0,
+        "w": 4,
+        "h": 4
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "sum(llmgw_model_replicas_ready)",
+          "legendFormat": "ready",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {},
+        "overrides": []
+      },
+      "options": {
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        },
+        "textMode": "auto",
+        "colorMode": "value"
+      }
+    },
+    {
+      "id": 3,
+      "title": "Models with something serving",
+      "type": "stat",
+      "description": "",
+      "gridPos": {
+        "x": 8,
+        "y": 0,
+        "w": 4,
+        "h": 4
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "count(llmgw_model_replicas_ready > 0)",
+          "legendFormat": "models",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {},
+        "overrides": []
+      },
+      "options": {
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        },
+        "textMode": "auto",
+        "colorMode": "value"
+      }
+    },
+    {
+      "id": 4,
+      "title": "Requests per second",
+      "type": "timeseries",
+      "description": "Failing is counted separately rather than as a percentage, so a quiet cluster reads as quiet instead of as 0% of nothing.",
+      "gridPos": {
+        "x": 12,
+        "y": 0,
+        "w": 12,
+        "h": 4
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "sum(rate(llmgw_requests_total[5m]))",
+          "legendFormat": "all",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        },
+        {
+          "refId": "B",
+          "expr": "sum(rate(llmgw_requests_total{status=~\"5..\"}[5m]))",
+          "legendFormat": "failing",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "reqps"
+        },
+        "overrides": []
+      }
+    },
+    {
+      "id": 5,
+      "title": "Replicas ready, by model",
+      "type": "timeseries",
+      "description": "Only models with a replica placed appear. One that is stopped on purpose publishes no series.",
+      "gridPos": {
+        "x": 0,
+        "y": 4,
+        "w": 12,
+        "h": 7
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "llmgw_model_replicas_ready",
+          "legendFormat": "{{model}}",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {},
+        "overrides": []
+      }
+    },
+    {
+      "id": 6,
+      "title": "Pool saturation",
+      "type": "timeseries",
+      "description": "The worse of queue depth and KV occupancy. 1 is the limit measured as healthy, not the point of failure.",
+      "gridPos": {
+        "x": 12,
+        "y": 4,
+        "w": 12,
+        "h": 7
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "llmgw_pool_saturation",
+          "legendFormat": "{{model}}",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "max": 2,
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "red",
+                "value": 1
+              }
+            ]
+          }
+        },
+        "overrides": []
+      }
+    },
+    {
+      "id": 7,
+      "title": "Time to first token",
+      "type": "timeseries",
+      "description": "As the router sees it. On CPU these are seconds, not milliseconds; compare against what ctl bench measured.",
+      "gridPos": {
+        "x": 0,
+        "y": 11,
+        "w": 12,
+        "h": 7
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "histogram_quantile(0.5, sum by (model, le) (rate(llmgw_ttft_seconds_bucket[5m])))",
+          "legendFormat": "{{model}} p50",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        },
+        {
+          "refId": "B",
+          "expr": "histogram_quantile(0.99, sum by (model, le) (rate(llmgw_ttft_seconds_bucket[5m])))",
+          "legendFormat": "{{model}} p99",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "s"
+        },
+        "overrides": []
+      }
+    },
+    {
+      "id": 8,
+      "title": "Tokens per second",
+      "type": "timeseries",
+      "description": "",
+      "gridPos": {
+        "x": 12,
+        "y": 11,
+        "w": 12,
+        "h": 7
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "sum by (direction) (rate(llmgw_tokens_total[5m]))",
+          "legendFormat": "{{direction}}",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "short"
+        },
+        "overrides": []
+      }
+    },
+    {
+      "id": 9,
+      "title": "Engine queue depth",
+      "type": "timeseries",
+      "description": "Requests the engine has accepted and not started. Rising here is the first sign of saturation.",
+      "gridPos": {
+        "x": 0,
+        "y": 18,
+        "w": 8,
+        "h": 6
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "llmgw_endpoint_waiting",
+          "legendFormat": "{{replica}}",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {},
+        "overrides": []
+      }
+    },
+    {
+      "id": 10,
+      "title": "KV cache occupancy",
+      "type": "timeseries",
+      "description": "",
+      "gridPos": {
+        "x": 8,
+        "y": 18,
+        "w": 8,
+        "h": 6
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "llmgw_endpoint_kv_cache_usage",
+          "legendFormat": "{{replica}}",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "percentunit"
+        },
+        "overrides": []
+      }
+    },
+    {
+      "id": 11,
+      "title": "Model disk free",
+      "type": "timeseries",
+      "description": "The filesystems holding weights. A node that cannot hold the next model is skipped by placement rather than failing.",
+      "gridPos": {
+        "x": 16,
+        "y": 18,
+        "w": 8,
+        "h": 6
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "llmgw_node_disk_free_bytes",
+          "legendFormat": "{{node}}",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "bytes"
+        },
+        "overrides": []
+      }
+    },
+    {
+      "id": 12,
+      "title": "Seconds since each node last reported",
+      "type": "timeseries",
+      "description": "Heartbeats are every 5s. A line that climbs is an agent going quiet; the control plane stops placing there at 15s.",
+      "gridPos": {
+        "x": 0,
+        "y": 24,
+        "w": 12,
+        "h": 6
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "llmgw_node_last_seen_seconds",
+          "legendFormat": "{{node}}",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "s"
+        },
+        "overrides": []
+      }
+    },
+    {
+      "id": 13,
+      "title": "Replicas by state",
+      "type": "timeseries",
+      "description": "",
+      "gridPos": {
+        "x": 12,
+        "y": 24,
+        "w": 12,
+        "h": 6
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "sum by (state) (llmgw_replicas)",
+          "legendFormat": "{{state}}",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {},
+        "overrides": []
+      }
+    }
+  ]
+}

deploy/monitoring/llmgw-rules.yml

@@ -0,0 +1,155 @@
+# Alerts for llmgw. Every expression here is against a series this cluster
+# actually publishes; nothing is written against a metric that might exist.
+groups:
+  - name: llmgw-fleet
+    rules:
+      - alert: ControlPlaneUnreachable
+        expr: up{job="llmgw"} == 0
+        for: 1m
+        labels: { severity: critical }
+        annotations:
+          summary: "The control plane is not answering"
+          description: >-
+            Nothing has scraped 10.0.0.140:9091 for a minute. Engines already
+            running keep serving — the router is in the same process, so if
+            this is down, so is the gateway.
+
+      - alert: NodeNotReady
+        expr: llmgw_node_up == 0
+        for: 2m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.node }} is {{ $labels.state }}"
+          description: >-
+            The node is known but not taking work. Two minutes, because a node
+            restarting its agent is briefly not ready and is not a problem.
+
+      - alert: AgentGoneQuiet
+        expr: llmgw_node_last_seen_seconds > 60
+        for: 1m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.node }} has not reported for {{ $value | humanizeDuration }}"
+          description: >-
+            Heartbeats are every 5s. A minute of silence is the network or the
+            agent, and the control plane stages expiry at 15s unreachable and
+            60s lost — so this fires around the point it stops placing there.
+
+      - alert: ModelHasNothingServing
+        expr: llmgw_model_replicas_ready == 0
+        for: 10m
+        labels: { severity: critical }
+        annotations:
+          summary: "{{ $labels.model }} has no replica ready"
+          description: >-
+            The model is placed and nothing is serving it. Ten minutes, because
+            a large model on CPU legitimately takes minutes to load. A model
+            that is deliberately stopped publishes no series at all, so this
+            cannot fire for one nobody asked for.
+
+      - alert: PoolSaturated
+        expr: llmgw_pool_saturation > 1
+        for: 5m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.model }} is over its healthy limit"
+          description: >-
+            Saturation is the worse of queue depth and KV occupancy; 1 is the
+            limit measured as healthy, not the point of failure. Sustained above
+            it means requests are queueing — add a replica or lower the load.
+
+      - alert: EndpointBreakerOpen
+        expr: llmgw_endpoint_breaker_open == 1
+        for: 2m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.replica }} has been taken out of rotation"
+          description: >-
+            The router stopped sending to this replica after repeated failures.
+            Traffic is going elsewhere; if it is the only replica, the model is
+            down.
+
+      - alert: EndpointMetricsStale
+        expr: llmgw_endpoint_metrics_stale == 1
+        for: 5m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.replica }} is being routed to blind"
+          description: >-
+            Its queue depth and KV usage are old, so the scheduler is choosing
+            with stale numbers. The replica may be serving perfectly well; the
+            decisions about it are the problem.
+
+      - alert: NodeDiskFilling
+        expr: llmgw_node_disk_free_bytes / llmgw_node_disk_total_bytes < 0.10
+        for: 15m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.node }} has {{ $value | humanizePercentage }} of its model disk free"
+          description: >-
+            Weights are tens of gigabytes. A node that cannot hold the next one
+            is skipped by placement rather than failing, so this is quiet until
+            somebody wonders why a model will not land there.
+
+  - name: monitoring-host
+    rules:
+      - alert: MonitoringDiskFilling
+        expr: >-
+          node_filesystem_avail_bytes{job="monitoring-host",fstype!~"tmpfs|overlay|squashfs"}
+          / node_filesystem_size_bytes{job="monitoring-host",fstype!~"tmpfs|overlay|squashfs"} < 0.15
+        for: 10m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.mountpoint }} on the monitoring host is {{ $value | humanizePercentage }} free"
+          description: >-
+            This filesystem holds the model mirror and the metrics database at
+            once. Filling it stops new weights arriving and stops Prometheus
+            writing — so monitoring goes quiet at the moment it is most needed,
+            which is why this fires at fifteen percent rather than five.
+
+      - alert: MonitoringDiskAlmostFull
+        expr: >-
+          node_filesystem_avail_bytes{job="monitoring-host",fstype!~"tmpfs|overlay|squashfs"}
+          / node_filesystem_size_bytes{job="monitoring-host",fstype!~"tmpfs|overlay|squashfs"} < 0.05
+        for: 5m
+        labels: { severity: critical }
+        annotations:
+          summary: "{{ $labels.mountpoint }} on the monitoring host has {{ $value | humanizePercentage }} left"
+
+  - name: llmgw-traffic
+    rules:
+      # These need traffic through the gateway to have any series at all. A
+      # counter with no observations publishes nothing, so a quiet cluster
+      # simply has no data here rather than a rule that is always false.
+      - alert: HighErrorRate
+        expr: >-
+          sum by (model) (rate(llmgw_requests_total{status=~"5.."}[5m]))
+          / sum by (model) (rate(llmgw_requests_total[5m])) > 0.05
+        for: 5m
+        labels: { severity: critical }
+        annotations:
+          summary: "{{ $labels.model }} is failing {{ $value | humanizePercentage }} of requests"
+
+      - alert: SlowFirstToken
+        expr: >-
+          histogram_quantile(0.99,
+            sum by (model, le) (rate(llmgw_ttft_seconds_bucket[10m]))) > 30
+        for: 10m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.model }} takes {{ $value | humanizeDuration }} to first token at p99"
+          description: >-
+            On CPU this is expected under load and is why the threshold is
+            thirty seconds rather than one. Compare with what ctl bench measured
+            before treating it as a fault.
+
+      - alert: RequestsBeingRejected
+        expr: sum by (band) (rate(llmgw_admission_rejected_total[5m])) > 0
+        for: 5m
+        labels: { severity: warning }
+        annotations:
+          summary: "Admission is turning away {{ $labels.band }} traffic"
+          description: >-
+            Working as designed when the cluster is full: the lowest band is
+            held first. Worth knowing about because the callers being refused
+            do not know why.

deploy/monitoring/prometheus.yml

@@ -1,45 +1,42 @@
-# Scraping llmgw.
+# Prometheus for the llmgw cluster.
 #
-# Point this at metrics_listen, not admin_listen. They are different ports for
-# a reason: the admin listener also carries pprof, which can dump the process
-# heap, and API keys are in that heap in plaintext.
-#
-# In the control plane's configuration:
-#
-#   admin_listen:   "127.0.0.1:9090"   # dashboard, management API, pprof
-#   metrics_listen: "0.0.0.0:9091"     # metrics only, safe to reach
-#
-# One target for the whole fleet. Node and GPU figures are exported by the
-# control plane, which already holds them — agents report hardware at
-# registration and state on every heartbeat — so there is nothing to scrape on
-# the nodes and no Prometheus configuration per machine.
+# One scrape target for the whole fleet. Agents publish nothing of their own —
+# the control plane gathers what they report and republishes it, so node health,
+# disk and endpoint state all arrive here from 10.0.0.140.
 global:
   scrape_interval: 15s
+  evaluation_interval: 15s
+  external_labels:
+    cluster: mynodes
+
+alerting:
+  alertmanagers:
+    - static_configs:
+        - targets: ['localhost:9093']
+
+rule_files:
+  - /etc/prometheus/rules/llmgw.yml
 
 scrape_configs:
+  # The control plane's dedicated metrics port. Not its admin listener: that one
+  # is loopback because it also serves pprof, and pprof can dump the heap where
+  # API keys sit in plaintext.
   - job_name: llmgw
     static_configs:
-      - targets: ["llmgw-ctl.internal:9091"]
+      - targets: ['10.0.0.140:9091']
         labels:
-          cluster: primary
+          instance: control-plane
 
-# Worth alerting on, in rough order of how often it matters.
-#
-#   A node that has stopped reporting:
-#     llmgw_node_last_seen_seconds > 60
-#
-#   A model that should be serving and is not. Covers a replica that will never
-#   start as well as one that has died, which look the same from a distance:
-#     llmgw_model_replicas_ready == 0
-#
-#   A card kept out of placement by a fault, which is otherwise invisible until
-#   a model will not fit:
-#     llmgw_gpu_healthy == 0
-#
-#   A fleet holding models nobody is asking for. Memory used says a model is
-#   loaded; utilisation says whether it is doing anything:
-#     avg_over_time(llmgw_gpu_utilization_percent[1h]) < 5
-#
-#   Requests queueing because the cluster is full, which is the pair of figures
-#   that needed two systems to correlate before these were on one endpoint:
-#     llmgw_admission_queue_depth > 0
+  - job_name: prometheus
+    static_configs:
+      - targets: ['localhost:9090']
+
+  # The machine this runs on. It came with the Prometheus package and was not
+  # being scraped, which left the disk holding both the model mirror and the
+  # metrics themselves watched by nothing — so a mirror that filled it would
+  # take monitoring down and the alert that would have warned did not exist.
+  - job_name: monitoring-host
+    static_configs:
+      - targets: ['localhost:9100']
+        labels:
+          instance: monitoring

deploy/systemd/llmgw-mirror.service

@@ -0,0 +1,29 @@
+[Unit]
+Description=llmgw model mirror
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+# --base is what a catalog entry will say, so it has to be the address nodes
+# reach this machine on — not localhost, and not the name only this host knows
+# itself by. Getting it wrong makes `mirror unused` report files as unreferenced
+# when they are in use, and the next step after that list is deletion.
+ExecStart=/usr/local/bin/llmgw mirror serve \
+    --dir /srv/models \
+    --listen :8081 \
+    --api-listen :8082 \
+    --base http://models.internal:8081
+User=llmgw
+Group=llmgw
+Restart=on-failure
+RestartSec=2s
+NoNewPrivileges=true
+ProtectSystem=strict
+ProtectHome=true
+PrivateTmp=true
+# The one writable path. The mirror serves reads to the whole fleet and writes
+# only what its own API was asked to fetch, so nothing else needs to be.
+ReadWritePaths=/srv/models
+
+[Install]
+WantedBy=multi-user.target

docs/choosing-models.md

@@ -0,0 +1,149 @@
+# Choosing what to run
+
+llmgw serves whatever you give it and has no opinion about which models those
+are. This page has an opinion, because "we have GPUs, now what" is the question
+that stalls most of these projects — and the usual first answer, one large
+model, is the wrong shape.
+
+Model releases move faster than documentation. Treat the families named here as
+a starting point and check what is current before committing; the reasoning is
+what should outlive the names.
+
+## Three jobs, not one model
+
+The requests a company actually makes fall into three groups with different
+needs, and no single model serves all three well.
+
+**Interactive** — chat, autocomplete, short questions. Most of your traffic.
+Here latency beats quality: a person waits two seconds and does not wait
+twenty. A 7–14B class model, always resident.
+
+**Heavy** — long document analysis, code generation, anything that needs
+reasoning. Quality beats latency, and a queue is acceptable. A 30–70B class
+model, loaded on demand:
+
+```yaml
+scaling: { min: 0, max: 1, idle_timeout: 30m }
+```
+
+llmgw starts it when somebody asks and stops it when nobody has, so it costs
+nothing while it is not in use. The first request waits for the load, and the
+client is told that rather than refused.
+
+**Embeddings** — and this is the one that gets forgotten. Most "let us chat
+with our documents" projects are retrieval projects, and retrieval runs on an
+embedding model. It is small, it is cheap, and it carries the highest request
+volume of the three: every document chunk once, every query forever. llmgw
+routes `/v1/embeddings` like any other endpoint.
+
+A deployment with the first two and not the third can answer questions from
+what the model already knew. It cannot answer questions about your company.
+
+## Swedish and English
+
+The first thing to know is that this combination is harder than either alone.
+
+Open models are strongest in English by a wide margin. Swedish is served —
+better than most languages its size, because the Nordic languages appear in
+multilingual training sets — but the gap is real, and it widens as models get
+smaller. **A 7B model that feels fine in English can feel noticeably worse in
+Swedish**, which means the interactive tier may need to be larger than an
+English-only deployment would choose. Test before deciding; do not assume the
+English benchmark transfers.
+
+Two things matter more than the model's headline multilingual score:
+
+**Cross-lingual retrieval.** A Swedish company writes internal documents in
+Swedish and reads technical documentation in English. Somebody will ask a
+question in Swedish whose answer is in an English document. An embedding model
+that handles both languages separately may still fail to connect them, and it
+fails silently — the model gives a confident answer from the wrong source, and
+nobody notices because the answer reads well. Test this explicitly: ask in one
+language, check that documents in the other are retrieved.
+
+**Swedish compounds.** *Kundtjänstmedarbetare* is one token-hostile word where
+English would use three. This hurts keyword search more than dense retrieval,
+so if you run hybrid search, expect the lexical half to underperform on Swedish
+and weight accordingly.
+
+## How to actually decide
+
+Benchmarks will not settle this. Take twenty real documents and twenty real
+questions from the company — in both languages, including at least a few where
+the question and the answer are in different languages — and compare candidates
+on those. An afternoon of this is worth more than a week of reading leaderboards,
+and it is the only way to find the failure modes that matter to you.
+
+Do the same for the embedding model, separately. A good generation model on top
+of bad retrieval produces fluent, confident, wrong answers, which is the worst
+failure this stack has: it does not look like a failure.
+
+## How many replicas
+
+Not "how many employees". **Concurrency**, which is far lower than headcount
+because people spend most of their time reading rather than waiting on a model.
+Two hundred people generate perhaps five to fifteen concurrent requests at
+peak, for interactive use.
+
+vLLM batches continuously, so one replica on a capable card serves many
+concurrent requests. Which means the replica count is usually decided by
+**redundancy rather than throughput**:
+
+| Tier | Replicas | Why |
+|---|---|---|
+| Embeddings | 2 | Not for capacity. If retrieval stops, every document feature stops. |
+| Interactive | 2 | So a node can be rebooted during working hours. |
+| Heavy | 0–1, on demand | Loaded when asked for, stopped when idle. |
+
+Then measure rather than guess:
+
+```
+llmgw ctl bench <model>
+```
+
+It climbs concurrency until the SLO breaks and reports the last level that met
+it. Peak throughput arrives long after latency has become unusable, so
+throughput alone overstates what the hardware can serve.
+
+## Where the weights live
+
+The single highest-value piece of infrastructure here is **a local mirror**: one
+internal HTTP server holding the model files. [deploy/mirror/](../deploy/mirror/)
+has the nginx config and the directory layout. Hugging Face is slow, rate
+limited, and blocked outright on plenty of corporate networks. Eight nodes each
+pulling forty gigabytes from the internet is a day; from a mirror on the same
+switch it is minutes. Point a model's `source` at it and llmgw fetches from
+there.
+
+**Pin the version.** Put it in the path — `/models/qwen2.5-14b/2026-03/q4_k_m.gguf`
+— so a repository that changes upstream cannot change what you are running.
+Without this, a prompt that worked last week gives a different answer this week
+and nobody can explain why.
+
+**Quantisation.** For llama.cpp, `q4_k_m` is the practical default: the quality
+difference from `q8_0` is smaller than most people expect and the memory
+difference is a factor of two. For vLLM on modern cards, FP8 or AWQ. Run
+`llmgw ctl plan <model>` to see whether a given precision fits before
+downloading anything.
+
+**Disk.** Budget three to five times what you first estimate. Several
+quantisations, several versions, and partial downloads all accumulate.
+`llmgw ctl nodes` shows free space per node once `model_dirs` is configured,
+and `llmgw ctl weights` reclaims files llmgw downloaded and no model needs —
+never files you placed yourself.
+
+Under ten nodes, let each keep its own copy; it is simpler and the disk is
+cheap. Above that, consider a shared filesystem — llmgw notices one and will
+not count the same free terabyte once per node.
+
+## Before anyone starts using it
+
+Set budgets and limits on day one, not later:
+
+```
+llmgw keys limits --scope project --id <project> --rpm 60 --budget 50000000
+```
+
+The reason is not technical. A resource that appears free and unlimited becomes,
+three months later, a cost nobody can account for — and introducing limits at
+that point is a much harder conversation than having had them from the start.

docs/data.md

@@ -0,0 +1,133 @@
+# What llmgw stores, and what leaves the machine
+
+Written for the review that happens before a gateway is allowed near anyone's
+documents. It describes what the software does. Mapping that onto an
+obligation — GDPR, a sector rule, an internal policy — is work for whoever
+holds the obligation; this page exists so that work starts from facts rather
+than from a vendor's summary.
+
+Everything below is checkable in the source, and the places to look are named.
+
+## The short version
+
+By default llmgw stores **no prompt or completion text anywhere**. It stores
+who made a request, when, how many tokens, and what it cost.
+
+Two features change that, and both are off until somebody turns them on for a
+named project, and neither can be turned on without also saying for how long.
+
+Nothing on the serving path leaves the cluster.
+
+## What is stored, exactly
+
+**Usage records**, one per request: the key, the project, the model asked for,
+the model that answered, prompt and completion token counts, cost, HTTP status
+and latency. No message content. The row is in `internal/store/store.go` as
+`Usage` and there is no field for text in it.
+
+**API keys**, as SHA-256 hashes. The key itself is shown once, at creation, and
+never stored — so a leaked database hands over no working credential, and
+neither llmgw nor an administrator can recover a key somebody lost. A prefix is
+kept so a key can be identified in a listing without revealing it.
+
+**People and tenancy**: accounts, the organisation/team/project tree, and the
+limits attached to it. Passwords are hashed.
+
+**Sessions** for the browser pages, as tokens.
+
+**Conversations and agent notes** — only for a project that asked. See below.
+
+## What is not stored
+
+Prompts. Completions. Anything the model was shown or produced, unless the
+project turned on conversations or notes.
+
+The chat page in llmgw's own web interface keeps the conversation in a
+JavaScript variable in the browser. Refresh it and the conversation is gone,
+because it was never anywhere else.
+
+## Conversations and notes, when they are turned on
+
+Two features keep text, and both work the same way:
+
+- **Conversations** (`x-llmgw-thread`) let a client send only its new turn while
+  llmgw replays the history — including to a different model than the one the
+  conversation started on.
+- **Agent notes** (`/v1/memory`, and MCP tools) let an agent keep something for
+  itself across conversations.
+
+Both are **off for every project** until an operator runs `llmgw keys threads`
+or `llmgw keys memory` for that project by name. A request carrying a thread id
+against a project that has not is refused with an explanation, not quietly
+served without its history.
+
+Both **require a retention window**. There is no way to turn either on without
+saying how many hours, because a store with no expiry is a transcript of
+everything a team ever typed, kept forever because nobody chose a number. The
+window is measured from the last turn, so an active conversation does not
+expire underneath it, and a background sweep removes what has passed it.
+
+Both are **scoped to a project**, and the scope is part of every lookup rather
+than a check afterwards. Reading a conversation that belongs to another project
+returns the same answer as reading one that does not exist, so a leaked id
+cannot be used to discover what exists elsewhere.
+
+`DELETE /v1/threads/{id}` removes a conversation and its messages immediately.
+It is a client endpoint, not an administrative one: the person who said the
+thing can unsay it without asking somebody else to read it first.
+
+Turning retention off stops new storage, and existing conversations go at the
+next hourly sweep.
+
+## What leaves the machine
+
+**Nothing on the serving path.** A request arrives, is routed to an engine on
+your own hardware, and the answer comes back. The router talks to engines, the
+engines talk to nothing, the agents talk to the control plane. All of it is
+inside the cluster.
+
+Two things can reach outside, both started by an operator rather than by a
+request:
+
+**Fetching weights.** When a model's `source` is a URL, the agent downloads it.
+Point that at an internal mirror and nothing leaves at all — which is also the
+faster arrangement, because eight nodes pulling forty gigabytes from the
+internet is a day's work and from a local mirror it is minutes. A `source` that
+is a path on disk is never fetched.
+
+**`llmgw ctl plan`.** Reads a model's architecture from huggingface.co to work
+out whether it fits your hardware. An operator command run at planning time,
+not part of serving, and not run by anything automatically.
+
+There is no telemetry, no licence check, and no call home. The binary is
+static, built with `CGO_ENABLED=0`, and runs in a network with no route to the
+internet as long as the weights are already on disk.
+
+## What the software cannot do for you
+
+**The admin port serves pprof**, and pprof can dump the process heap — where
+API keys sit in plaintext while they are being checked. Bind it to loopback and
+reach it over SSH. The default does this; a deployment that changes it should
+know why.
+
+**The store holds everything.** With conversations turned on it holds text as
+well as accounting, so where it lives, who can read it and how it is backed up
+are decisions with the same weight as the ones above. The PostgreSQL DSN
+belongs in a file readable only by the user llmgw runs as, which is why the
+configuration takes `@/path/to/file` rather than the address itself.
+
+**Model weights are yours.** llmgw reclaims only files it downloaded, and it
+identifies those by a name it generated. Anything you put on a node it will not
+touch, whatever the catalog says.
+
+## Where to look
+
+| Claim | Where |
+|---|---|
+| Usage rows carry no text | `internal/store/store.go`, type `Usage` |
+| Keys are stored hashed | `internal/store/keys.go` |
+| Conversations are off by default | `internal/store/threads.go`, `ThreadRetention` |
+| Retention is required | `llmgw keys threads --help` |
+| Scope is part of the lookup | `internal/store/threads.go`, `Thread` |
+| Deletion is a client endpoint | `internal/router/threads.go`, `handleDeleteThread` |
+| Weight fetching is the only serving-path egress | `internal/agent/weights/fetch.go` |

docs/install.md

@@ -13,7 +13,7 @@ this page says otherwise, and where it says otherwise it says so plainly.
 Read this first, because it is the difference between an afternoon and a week.
 
 llmgw runs engines and routes requests. It does **not** install GPU drivers,
-CUDA, inference engines, or model weights. A node missing any of them still
+CUDA, or the inference engines themselves. A node missing any of them still
 registers, still reports `ready`, and still fails every replica placed on it —
 so the useful order is to make a node able to serve *before* onboarding it, not
 after.
@@ -23,12 +23,15 @@ after.
 | NVIDIA driver and `nvidia-smi` | GPU discovery, classes, health |
 | InfiniBand stack (`rdma-core`/OFED) | Fabric discovery and the invariant that a replica never spans one |
 | vLLM, SGLang or llama.cpp | Launching them, the right flags, supervision |
-| Model weights on disk | Placement, routing, accounting |
+| Model weights, from somewhere | Fetching them, placement, routing, accounting |
 
 The reason is boring and unavoidable: drivers and engines are platform decisions
-with kernel modules and version matrices attached, and distributing weights is
-its own problem — hundreds of gigabytes that want a shared filesystem, not a
-copy from somebody's laptop. It is a known gap, not an oversight.
+with kernel modules and version matrices attached.
+
+Weights are the one item on that list llmgw does handle. It includes a mirror —
+step 4 — which holds the files on your own network and hands them to the nodes,
+so getting weights onto every machine is not a problem you solve before
+starting. What llmgw will not do is choose the models for you.
 
 ## 0. Before you begin
 
@@ -49,7 +52,10 @@ them, so nodes need no inbound access except your own SSH.
 | --- | --- | --- |
 | 8080 | clients, the web UI | your users, usually via a reverse proxy |
 | 7443 | node agents | the node network only |
-| 9090 | `llmgw ctl`, metrics, pprof | loopback only — it serves pprof, which can dump the process heap, and API keys live in that heap |
+| 9090 | `llmgw ctl`, the management API, pprof | loopback only — it serves pprof, which can dump the process heap, and API keys live in that heap |
+| 9091 | Prometheus, if you run it | your monitoring host; metrics only, nothing else is served here |
+| 8081 | node agents fetching weights | the node network only (the mirror machine) |
+| 8082 | the control plane administering the mirror | the control plane only — this port can delete models and start downloads |
 
 Decide now which address nodes will use to reach the control plane. It goes into
 a certificate and changing it later means reissuing one.
@@ -59,7 +65,7 @@ a certificate and changing it later means reissuing one.
 One static binary, no runtime dependencies, the same on every machine.
 
 ```
-VERSION=v0.7.0    # check the releases page for the current tag
+VERSION=v0.8.0    # check the releases page for the current tag
 curl -fsSLO https://gitea.mynodes.xyz/ilker/llmgw/releases/download/$VERSION/llmgw-$VERSION-linux-amd64
 curl -fsSL  https://gitea.mynodes.xyz/ilker/llmgw/releases/download/$VERSION/checksums.txt | sha256sum -c --ignore-missing
 sudo install -m0755 llmgw-$VERSION-linux-amd64 /usr/local/bin/llmgw
@@ -148,11 +154,22 @@ auth:
   # that is the bill or the outage.
   default_rpm: 60
   default_tpm: 200000
+  # What a request costs, which is what makes a budget mean anything. Without
+  # a price every request costs zero, so a budget is displayed, counted against
+  # nothing, and never binds — and nobody finds out until they expected it to
+  # stop something. Self-hosted models have no per-token price, so the unit is
+  # yours to choose: GPU-seconds, a notional rate, or 1000 everywhere, which
+  # turns a budget into a plain token quota.
+  default_price:
+    prompt_per_1k: 1000
+    completion_per_1k: 1000
+  # pricing:
+  #   qwen2.5-coder-7b-instruct: { prompt_per_1k: 500, completion_per_1k: 1500 }
 
 landing:
   contact: "platform@example.com"
 
-# Start with none. Models are added while the cluster runs — see step 7 —
+# Start with none. Models are added while the cluster runs — see step 8 —
 # and anything listed here is declarative: it is in the catalog on every start
 # and cannot be removed without editing this file.
 models: []
@@ -200,7 +217,103 @@ sudo -u llmgw llmgw ctl status
 been added yet. It is worth running now so its output is familiar before you
 need it.
 
-## 4. Preparing a node
+## 4. The mirror
+
+Do this before the nodes, not after. Every node fetches its own copy of a
+model's weights, and where they fetch from decides whether bringing up a cluster
+takes minutes or a day. Eight nodes pulling forty gigabytes each from Hugging
+Face is a day's work, and on a good number of corporate networks it is not
+possible at all — the nodes have no route out, which is usually the point.
+
+The mirror is one machine holding the files on your own network. It is the same
+binary again, and it is the highest-value piece of infrastructure around llmgw
+for the least work.
+
+### Running it
+
+```
+sudo install -d -o llmgw -g llmgw /srv/models
+llmgw mirror serve --dir /srv/models --base http://models.internal:8081
+```
+
+In production, the unit:
+
+```
+sudo cp deploy/systemd/llmgw-mirror.service /etc/systemd/system/
+sudo systemctl daemon-reload && sudo systemctl enable --now llmgw-mirror
+```
+
+**Two listeners, on purpose.** `:8081` serves bytes and nothing else — that is
+what every agent talks to, and it is plain HTTP because a weights fetch is a
+public file transfer that is checksummed anyway. `:8082` is the management API:
+it can start downloads and delete models, and only the control plane needs it.
+The split is what lets you lock the second one down without re-certifying every
+agent that fetches from the first.
+
+Give the management port the cluster's own CA and it takes client certificates:
+
+```
+llmgw mirror serve --dir /srv/models --base http://models.internal:8081 \
+  --tls-ca /etc/llmgw/ca.pem \
+  --tls-cert /etc/llmgw/mirror.pem --tls-key /etc/llmgw/mirror-key.pem
+```
+
+Without those three flags the management port is open to anyone who can reach
+it, and the mirror says so in its own log at every start rather than letting you
+find out later.
+
+### The layout
+
+```
+/srv/models/
+  qwen2.5-14b-instruct/
+    2026-03/
+      q4_k_m.gguf
+```
+
+Model, then version, then files. The version directory is the point of the
+layout rather than decoration: a repository that changes upstream changes what
+you are running, and the first sign is a prompt that worked last week answering
+differently this week with nobody able to say why. Pin it here and that cannot
+happen.
+
+A version holding several files is a sharded model, and is served as a set: the
+agent asks the directory what it consists of and fetches all of it into one
+place, marking it complete only when the last file lands. A catalogue entry for
+one names the directory, trailing slash included.
+
+### Telling the control plane about it
+
+```yaml
+# /etc/llmgw/server.yaml
+mirror:
+  api: https://models.internal:8082
+  tls:
+    ca: /etc/llmgw/ca.pem
+    cert: /etc/llmgw/server.pem
+    key: /etc/llmgw/server-key.pem
+```
+
+With this set, `/admin/mirror` appears: what the mirror holds, what it is
+downloading and how far along, a search of Hugging Face, and a form that puts a
+mirrored model into service without anyone writing YAML. Without it that page
+stays unmounted and the mirror is still perfectly usable — a model's `source` is
+just a URL, and the URL can be yours.
+
+If the control plane cannot reach the internet, `mirror.registry` points the
+search at an internal registry mirror instead.
+
+### Filling it
+
+```
+llmgw mirror fetch --model qwen2.5-14b-instruct --version 2026-03 \
+  hf:Qwen/Qwen2.5-14B-Instruct-GGUF/qwen2.5-14b-instruct-q4_k_m.gguf
+```
+
+or from the admin page, or by copying files into the directory by hand — the
+mirror is a directory, and anything that puts files in it works.
+
+## 5. Preparing a node
 
 **Do this before onboarding.** A node missing its driver or engine registers
 healthy and fails on first placement, and the reason is three logs away.
@@ -305,10 +418,13 @@ llmgw ctl engine-check vllm --bin /opt/venvs/vllm/bin/vllm
 
 ### Weights
 
+Nothing to do here if you built the mirror in step 4 — the agent fetches what a
+model's `source` names, and that source is your mirror. This section is for the
+arrangements where the files are already on the machine.
+
 Put them where the node can read them. `scp`, a shared filesystem, or the
 engine's own downloader are all valid, and which you want depends on how many
-nodes you have. If you would rather not choose, skip to the next section and
-let llmgw fetch them.
+nodes you have.
 
 ```
 sudo install -d -o root -g root /var/lib/llmgw-models
@@ -325,11 +441,18 @@ model's `source` at a file to download and the agent fetches it once, before the
 engine is launched:
 
 ```yaml
+# your mirror, which is what step 4 was for
+source: http://models.internal:8081/qwen2.5-coder-3b-instruct/2026-03/q4_k_m.gguf
+# a whole version, for a model of several files — the trailing slash is the signal
+source: http://models.internal:8081/llama-3.1-70b-instruct/2026-03/
+# or straight from Hugging Face, where the nodes can reach it
 source: hf:Qwen/Qwen2.5-Coder-3B-Instruct-GGUF/qwen2.5-coder-3b-instruct-q4_k_m.gguf
-# or an ordinary URL
-source: https://models.internal/qwen2.5-coder-3b-q4_k_m.gguf
 ```
 
+A fetch reports its progress to the control plane, so `llmgw ctl replicas` and
+the dashboard both show how far along a node is. A replica pulling a hundred
+gigabytes and one that is wedged do not look alike.
+
 Eight shards of one replica starting together produce one download, not eight.
 An interrupted fetch resumes rather than starting over, and a partial file is
 never visible under the model's name — so a half-downloaded model cannot be
@@ -375,7 +498,7 @@ How much disk a node needs follows from the models rather than a rule of thumb.
 `llmgw ctl model plan <repo>` prints the sizes; as a starting point, a 70B at
 bf16 is 141 GB and the same model at q4 is 43 GB.
 
-## 5. Adding the node
+## 6. Adding the node
 
 Two halves, because they need different access: building the machine needs the
 hypervisor, and issuing its identity needs the certificate authority. Where one
@@ -432,7 +555,7 @@ Repeat for each node. Nodes in the same fabric must be given the same `--fabric`
 name; nodes with no InfiniBand take no `--fabric` at all and are each their own
 island, which is fine for single-node models.
 
-## 6. Setting the node's CPU model, if it is a VM
+## 7. Setting the node's CPU model, if it is a VM
 
 Proxmox defaults a VM to `kvm64`, which hides AVX, AVX2, FMA and even SSE4.2
 from the guest. Inference then falls back to scalar code and is not slightly
@@ -449,7 +572,7 @@ only change being `--cpu host`:
 probably did not, and nothing about it looks broken — every number is simply
 small. Check with `grep -qw avx2 /proc/cpuinfo` on the node.
 
-## 7. Adding a model
+## 8. Adding a model
 
 Write the definition. This is the same shape a model takes in `server.yaml`, so
 an entry moves between the two by being copied:
@@ -458,7 +581,7 @@ an entry moves between the two by being copied:
 # qwen-coder-3b.yaml
 name: qwen2.5-coder-3b-instruct
 aliases: [coder]
-source: /var/lib/llmgw-models/qwen2.5-coder-3b-instruct-q4_k_m.gguf
+source: http://models.internal:8081/qwen2.5-coder-3b-instruct/2026-03/q4_k_m.gguf
 routing_profile: chat
 # What a client may use in one request. Worth stating: llama.cpp divides
 # --ctx-size across its parallel slots, so a pool of 65536 over two slots is
@@ -481,6 +604,17 @@ llmgw ctl model add -f qwen-coder-3b.yaml
 llmgw ctl models
 ```
 
+Adding a model checks that the source is actually there before accepting it, so
+a typo or a mirror that is down is a sentence now rather than a replica failing
+on a node in five minutes with an error naming the node. `--no-source-check`
+skips it, which is right when the nodes reach the mirror by an address this
+machine does not.
+
+With a mirror configured you can skip the file entirely: `/admin/mirror` lists
+what the mirror holds and offers a form that writes the same definition, with
+the engine, the source and the memory figure already filled in from the file
+itself.
+
 ```
 MODEL                      SERVING  REPLICAS  STATUS   ALIASES
 qwen2.5-coder-3b-instruct  yes      1/1       ready    coder
@@ -545,7 +679,42 @@ llmgw ctl explain-placement qwen2.5-coder-3b-instruct
 which names every candidate and why each was rejected, rather than leaving you
 to infer it.
 
-## 8. The first request
+### How many replicas, and how much each one carries
+
+Replica count is changed while the cluster runs, without editing the
+definition:
+
+```
+llmgw ctl scale qwen2.5-coder-3b-instruct --min 3
+```
+
+Models added at runtime only. One declared in `server.yaml` is read again on
+every start, so a number set here would hold until the next restart and then
+revert — change it in the file instead.
+
+How much traffic each replica is given is the profile's `weight`, and guessing
+it is how a cluster ends up with one saturated replica beside two idle ones.
+Measure it instead:
+
+```
+llmgw ctl bench qwen2.5-coder-3b-instruct --save
+```
+
+Concurrency climbs until the latency target stops being met, and the last level
+that held it is the capacity — peak throughput arrives long after latency has
+become unusable, so throughput alone would overstate what the hardware serves.
+It prints the `weight` to put in the profile, and `--save` records the figure
+against the model so the dashboard can show what the cluster is measured to
+carry rather than what somebody typed.
+
+This drives real traffic and competes with real requests. Run it against a
+cordoned replica, or in a window where that is acceptable.
+
+An engine that publishes no metrics endpoint is measured from the outside only.
+The run says so, and queue depth and KV occupancy are then not part of the
+answer.
+
+## 9. The first request
 
 ```
 curl http://<control-plane>:8080/v1/chat/completions \
@@ -586,8 +755,10 @@ One scrape target for the whole fleet. The node and GPU figures come from the
 control plane, which already holds them, so there is nothing to install on the
 nodes and no Prometheus configuration per machine.
 
-A starting configuration, with the alerts worth having, is in
-[deploy/monitoring/prometheus.yml](../deploy/monitoring/prometheus.yml).
+A starting configuration, with the alerts worth having and a provisioned
+dashboard, is in [deploy/monitoring/](../deploy/monitoring/) — including where
+to run it, which is not the control plane: monitoring that dies with the thing
+it monitors tells you nothing at the moment you need it.
 
 Two figures repay knowing about. `llmgw_gpu_utilization_percent` is absent
 rather than zero when the driver does not report it — a flat zero across a
@@ -595,6 +766,52 @@ fleet means the measurement is missing, not that the fleet is idle. And
 `llmgw_gpu_nvlink_peers` at zero everywhere means the topology is unknown,
 which is what a binary built without the nvml tag reports.
 
+## PostgreSQL, when SQLite is no longer the right shape
+
+SQLite is the default and stays the right answer for most installations: one
+file, no server, and a request path that barely touches it — usage is written
+in batches, budgets are counted in memory, authentication is cached.
+
+Two things change that. Conversations and agent notes read and write on every
+turn, which is the first per-request load this store has had; and SQLite takes
+one writer at a time, so those turns serialise. And a second control plane is
+impossible while the store is a file on one machine's disk.
+
+The setting is the same one, and the address decides:
+
+```yaml
+auth:
+  store: "@/etc/llmgw/secrets/store-dsn"
+```
+
+A path is SQLite, a `postgres://` URL is PostgreSQL, and `@` followed by a path
+reads the address out of that file — which is how a DSN carrying a password
+stays out of a configuration that gets copied, diffed and pasted into issues.
+Keep that file at `0600` and owned by the user llmgw runs as.
+
+Moving an existing installation is a copy, not a fresh start. Keys are stored
+hashed, so starting fresh means reissuing every one of them and being unable
+to say which was which:
+
+```
+sudo systemctl stop llmgw-server
+llmgw keys move-store --from /var/lib/llmgw/llmgw.db --to "@/etc/llmgw/secrets/store-dsn"
+# point auth.store at the new address, then
+sudo systemctl start llmgw-server
+```
+
+Stop the gateway first. The copy makes no attempt to follow a store being
+written to: rows landing after the table they belong to has been passed would
+simply be absent, and nothing would say so. The destination must be empty, and
+the source is left untouched — so the way back is to point the configuration at
+the file again.
+
+On a machine that also runs the control plane, PostgreSQL's defaults assume it
+owns the box. A gateway swapped out to give the database cache is a worse
+outcome than a smaller cache, so cap it: `shared_buffers = 96MB` and
+`max_connections = 40` are enough for a store whose working set is measured in
+megabytes.
+
 ## Conversations across models
 
 Off everywhere until a project asks. llmgw otherwise stores no message content
@@ -672,6 +889,103 @@ Without authentication, `threads.store` and `threads.retain_hours` in
 `server.yaml` keep both conversations and notes in one shared pool. That is for
 one trusted machine only; see [quickstart.md](quickstart.md).
 
+## Two control planes
+
+llmgw handles the hard half itself: only one places work, and agents find
+whichever is leading. What it does not do is move client traffic, which
+happens in front of both.
+
+[deploy/ha/](../deploy/ha/) has the pattern and a keepalived example. The one
+thing to get right is the health check: point it at `/ready`, not `/health`.
+`/health` is 200 on a standby, which is alive and holds no endpoints at all,
+so a load balancer using it would send half the traffic to something that
+refuses every request.
+
+## Upgrading
+
+One binary on every machine, so an upgrade is a copy and a restart. The part
+worth care is the copy, not the order.
+
+**Nothing stops serving.** Engines are started with `KillMode=process`, so they
+keep running when the agent restarts, and the agent reports them back on its
+next connection. The control plane holds no replica state across its own
+restart either — it rebuilds the fleet from what the agents report. A rolling
+upgrade of the whole cluster interrupts no request that is not in flight at the
+moment of the restart.
+
+**A mixed fleet is fine while you roll.** The agent protocol only ever gains
+fields, and both sides treat an absent one as unknown rather than as a default.
+Order does not matter; finish the roll the same day anyway, because
+`ctl nodes` reports each agent's version and a fleet that disagrees with itself
+is one more thing to hold in your head.
+
+```
+VERSION=v0.9.0
+# copy to every machine first, install nowhere yet
+for h in 10.0.0.140 10.0.0.141 10.0.0.142 10.0.0.146; do
+  ssh $h 'cat > /tmp/llmgw-new; stat -c %s /tmp/llmgw-new' < llmgw-$VERSION-linux-amd64
+done
+```
+
+**Check every size before installing anything.** This is not defensive writing:
+on a host whose login shell prints anything, `scp` fails outright, and the
+obvious workaround — `ssh -n host 'cat > file'` — points stdin at `/dev/null`
+and writes a *zero-byte file* while reporting success. Installing that on a
+fleet takes the fleet down. Compare the number on each machine with the local
+one, then:
+
+```
+sudo install -m0755 /tmp/llmgw-new /usr/local/bin/llmgw
+sudo systemctl restart llmgw-agent     # or llmgw-server, or llmgw-mirror
+llmgw --version
+```
+
+Then confirm what you have, rather than assuming:
+
+```
+llmgw ctl nodes       # every agent back, LAST SEEN in seconds
+llmgw ctl replicas    # same replicas, adopted rather than replaced
+```
+
+A replica may come back marked **`ready (stale profile)`**. That means it is
+running a profile that has since been edited — the reconciler leaves a serving
+replica alone on purpose, because churning one costs more than the difference.
+Restart it when you want the new profile to take effect; the flag is there so
+the choice is yours rather than silent.
+
+## Backing up
+
+Three things, in descending order of how badly you want them.
+
+**The certificate authority.** `ca-key.pem`, wherever you put it in step 2.
+Losing it does not stop the cluster, and it means every node identity has to be
+reissued the next time you touch one. Nothing in llmgw can reconstruct it.
+
+**The store.** Users, API keys, organisations, teams, projects, every limit and
+budget, all usage history, models added at runtime, and measured capacity.
+
+```
+# SQLite
+sqlite3 /var/lib/llmgw/llmgw.db ".backup '/backup/llmgw-$(date +%F).db'"
+
+# PostgreSQL
+pg_dump --format=custom llmgw > /backup/llmgw-$(date +%F).dump
+```
+
+`.backup` and `pg_dump` are both safe against a running gateway. Copying the
+SQLite file with `cp` while it is being written is not.
+
+**Keys are stored hashed and cannot be recovered from a backup or from
+anywhere else.** Restoring an old store restores which keys exist and who owns
+them, not their secrets. A key created after the backup is gone and has to be
+reissued — which is why the store is worth backing up on a schedule rather than
+before upgrades.
+
+**The mirror's directory** is the one thing you can rebuild, by downloading it
+all again. Worth remembering that a pinned version may no longer be upstream to
+download: that is exactly what pinning it protected you from, and a backup is
+what makes the protection outlive the disk.
+
 ## Reclaiming disk
 
 `llmgw ctl nodes` has a DISK column: free space against the size it is free
@@ -714,7 +1028,7 @@ output, which is the actual diagnosis.
 node and the reason: wrong GPU class, not enough free cards, too little memory,
 no fabric, a label it does not carry.
 
-**Everything is slow.** On a VM, check AVX2 (step 6). On CPU, check the
+**Everything is slow.** On a VM, check AVX2 (step 7). On CPU, check the
 quantisation — the difference between `q8_0` and `q4_k_m` is roughly a factor of
 two. `llmgw ctl status` shows how full the cluster is.
 
@@ -728,6 +1042,17 @@ requests. On a slow model, set `timeouts.generation` higher.
 
 ## What to read next
 
+- `docs/choosing-models.md` — which models to run and how many, if the answer
+  to "we have GPUs, now what" is not already obvious
+- `docs/data.md` — what is stored and what leaves the machine, for the review
+  that happens before this is allowed near anyone's documents
+- `docs/reference.md` — every configuration key, generated from the source, so
+  it describes the version you are running rather than the one somebody last
+  wrote about
+- `deploy/mirror/README.md` — the mirror in more detail, including the layout
+  and what belongs on which port
+- `deploy/monitoring/README.md` — Prometheus, Grafana and the alerts, with what
+  each one assumes about this system
 - `docs/node-onboarding.md` — adding nodes in more detail, and what onboarding
   deliberately does not do
 - `docs/onboarding.md` — giving people accounts and keys
@@ -735,17 +1060,20 @@ requests. On a slow model, set `timeouts.generation` higher.
 
 ## Known gaps, so you find them here rather than at 2am
 
-- **A fetch is silent while it runs.** llmgw will fetch weights for you, but
-  it reports no progress, so a first start on a large model looks like a hang.
-  `ls -l` in the model directory is the only progress bar there is.
 - **Every node fetches its own copy.** There is no seeding between nodes, so
-  bringing up eight nodes on a new model is eight downloads from the same
-  origin. On a fabric where the nodes can see each other this is wasteful, and
-  it is the obvious next thing to build.
+  bringing up eight nodes on a new model is eight downloads. The mirror makes
+  those eight local, which is most of the problem; on a fabric where the nodes
+  can see each other, seeding between them is the obvious next thing to build.
+- **A budget without a price never binds.** Set `default_price` or `pricing`
+  (step 3) or every request costs zero, and a budget is something the dashboard
+  displays rather than something that stops anything. The quotas page says so
+  where it can, which is not the same as being told before you rely on it.
 - **Engines are yours to install.** Onboarding verifies one runs; it installs
   nothing.
-- **One control plane.** No HA yet; it is a single process with a SQLite file.
-  Back up `/var/lib/llmgw/llmgw.db`.
+- **Two control planes have not been run for long.** The leader election and
+  agent failover work and there is a pattern in `deploy/ha/`, but every cluster
+  this has been run on in anger has had one. A single control plane is the
+  well-trodden path.
 - **Multi-node replicas have not been run on real InfiniBand.** The scheduling,
   the fabric invariant and the rendezvous are exercised against simulated
   hardware only. Single-node placement on real GPUs is the well-trodden path;

go.mod

@@ -6,6 +6,7 @@ require (
 	connectrpc.com/connect v1.21.0
 	github.com/NVIDIA/go-nvml v0.13.4-0
 	github.com/cespare/xxhash/v2 v2.3.0
+	github.com/jackc/pgx/v5 v5.11.0
 	github.com/knadh/koanf/parsers/yaml v1.1.1
 	github.com/knadh/koanf/providers/env/v2 v2.0.1
 	github.com/knadh/koanf/providers/file v1.2.1
@@ -14,6 +15,7 @@ require (
 	github.com/prometheus/client_model v0.6.3
 	github.com/prometheus/common v0.71.0
 	github.com/spf13/cobra v1.10.2
+	golang.org/x/crypto v0.57.0
 	golang.org/x/net v0.59.0
 	golang.org/x/sync v0.23.0
 	google.golang.org/protobuf v1.36.12
@@ -27,6 +29,9 @@ require (
 	github.com/go-viper/mapstructure/v2 v2.4.0 // indirect
 	github.com/google/uuid v1.6.0 // indirect
 	github.com/inconshreveable/mousetrap v1.1.0 // indirect
+	github.com/jackc/pgpassfile v1.0.0 // indirect
+	github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
+	github.com/jackc/puddle/v2 v2.2.2 // indirect
 	github.com/knadh/koanf/maps v0.1.2 // indirect
 	github.com/mattn/go-isatty v0.0.24 // indirect
 	github.com/mitchellh/copystructure v1.2.0 // indirect
@@ -37,7 +42,6 @@ require (
 	github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
 	github.com/spf13/pflag v1.0.9 // indirect
 	go.yaml.in/yaml/v3 v3.0.5 // indirect
-	golang.org/x/crypto v0.57.0 // indirect
 	golang.org/x/sys v0.48.0 // indirect
 	golang.org/x/text v0.42.0 // indirect
 	modernc.org/libc v1.75.7 // indirect

go.sum

@@ -7,6 +7,7 @@ github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6r
 github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
 github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
 github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
+github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
 github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
 github.com/fsnotify/fsnotify v1.9.0 h1:2Ml+OJNzbYCTzsxtv8vKSFD9PbJjmhYF14k/jKC7S9k=
@@ -23,6 +24,14 @@ github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs
 github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
 github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
 github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
+github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM=
+github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg=
+github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 h1:iCEnooe7UlwOQYpKFhBabPMi4aNAfoODPEFNiAnClxo=
+github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761/go.mod h1:5TJZWKEWniPve33vlWYSoGYefn3gLQRzjfDlhSJ9ZKM=
+github.com/jackc/pgx/v5 v5.11.0 h1:IzBBtyK9AHqf98cctWFifYSci2hgQR/cd56wB4p+ogg=
+github.com/jackc/pgx/v5 v5.11.0/go.mod h1:mal1tBGAFfLHvZzaYh77YS/eC6IX9OWbRV1QIIM0Jn4=
+github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
+github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
 github.com/klauspost/compress v1.19.1 h1:VsB4HPswih7mmZ8WleSFQ75c/Ui1M4trX5oAsJnhSlk=
 github.com/klauspost/compress v1.19.1/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
 github.com/knadh/koanf/maps v0.1.2 h1:RBfmAW5CnZT+PJ1CVc1QSJKf4Xu9kxfQgYVQSu8hpbo=
@@ -47,6 +56,7 @@ github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq
 github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
 github.com/ncruces/go-strftime v1.0.0 h1:HMFp8mLCTPp341M/ZnA4qaf7ZlsbTc+miZjCLOFAw7w=
 github.com/ncruces/go-strftime v1.0.0/go.mod h1:Fwc5htZGVVkseilnfgOVb9mKy6w1naJmn9CehxcKcls=
+github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
 github.com/prometheus/client_golang v1.24.1 h1:JnJkREXzWxUdCuPFpIWZiPispT9xVV59uiuyR2bPlnU=
 github.com/prometheus/client_golang v1.24.1/go.mod h1:F+oSRECHg4sse5ucfYpYDeIv/hu68Zo0uoHKetWnzcE=
 github.com/prometheus/client_model v0.6.3 h1:O0jaTVAYNxTHYInEPFJt5I3+sN8zqBtVMPTB1qyxiEo=
@@ -62,6 +72,9 @@ github.com/spf13/cobra v1.10.2 h1:DMTTonx5m65Ic0GOoRY2c16WCbHxOOw6xxezuLaBpcU=
 github.com/spf13/cobra v1.10.2/go.mod h1:7C1pvHqHw5A4vrJfjNwvOdzYu0Gml16OCs2GRiTUUS4=
 github.com/spf13/pflag v1.0.9 h1:9exaQaMOCwffKiiiYk6/BndUBv+iRViNW+4lEMi0PvY=
 github.com/spf13/pflag v1.0.9/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
+github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
+github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
+github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
 github.com/stretchr/testify v1.12.1 h1:EuwCh5fleGS7H32xRwO3wRGT7DxrDhLAT6FF8MpWDWE=
 github.com/stretchr/testify v1.12.1/go.mod h1:MDEgiDPPsNp5cuIrHPPCyornHKgEVbtFUmoNlxoYthg=
 go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
@@ -88,6 +101,7 @@ golang.org/x/tools v0.49.0/go.mod h1:SJNXV9DBKT0UbdttsQjbfJlAE/q+y36++zo3uL3N0Oo
 google.golang.org/protobuf v1.36.12 h1:pJOKDDOyeXErUroCihFAd5LQuwXBSpVnKGrj5o/fwxc=
 google.golang.org/protobuf v1.36.12/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
 modernc.org/cc/v4 v4.29.2 h1:h6+9ciCnPKutf4I03CvheAvDLX7+IHlqR6Iy6J+cgd8=
 modernc.org/cc/v4 v4.29.2/go.mod h1:OnovgIhbbMXMu1aISnJ0wvVD1KnW+cAUJkIrAWh+kVI=
 modernc.org/ccgo/v4 v4.35.0 h1:F+TUsmw09QxLzmi3aeYYGxjAXarmZaKgj3mKQHNaA8w=

internal/account/account_test.go

@@ -12,7 +12,7 @@ import (
 	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
 )
 
-func newStore(t *testing.T) *store.SQLite {
+func newStore(t *testing.T) *store.DB {
 	t.Helper()
 	s, err := store.OpenSQLite(context.Background(), filepath.Join(t.TempDir(), "a.db"))
 	if err != nil {
@@ -33,7 +33,7 @@ func newStore(t *testing.T) *store.SQLite {
 	return s
 }
 
-func newService(t *testing.T) (*Service, *store.SQLite) {
+func newService(t *testing.T) (*Service, *store.DB) {
 	t.Helper()
 	s := newStore(t)
 	return New(s, DefaultConfig(), slog.New(slog.DiscardHandler)), s
@@ -113,7 +113,7 @@ func TestApprovalNeedsAProject(t *testing.T) {
 	}
 }
 
-func newStoreAndUser(t *testing.T) (*Service, *store.SQLite) {
+func newStoreAndUser(t *testing.T) (*Service, *store.DB) {
 	t.Helper()
 	svc, st := newService(t)
 	hash, _ := HashPassword(goodPassword)

internal/accounting/accounting.go

@@ -70,19 +70,54 @@ func (a *Accountant) RestoreBudgets(ctx context.Context, s store.Store, period s
 	if start.IsZero() {
 		return nil
 	}
-	spend, err := s.SpendSince(ctx, start)
+	rows, err := s.SpendSinceByScope(ctx, start)
 	if err != nil {
 		return fmt.Errorf("restoring budgets: %w", err)
 	}
+	// Every scope a key sits under, not just the key. A budget owned by a team
+	// is one counter fed by every key beneath it, and restoring only per-key
+	// totals would bring the control plane back up believing the team had spent
+	// nothing — which is a budget that resets on every restart, for exactly the
+	// scopes where it matters most.
+	spend := make(map[string]int64, len(rows)*4)
+	for _, r := range rows {
+		for _, o := range []store.LimitOwner{
+			{Scope: store.ScopeKey, ID: r.KeyID},
+			{Scope: store.ScopeProject, ID: r.ProjectID},
+			{Scope: store.ScopeTeam, ID: r.TeamID},
+			{Scope: store.ScopeOrg, ID: r.OrgID},
+		} {
+			if k := o.CounterKey(); k != "" {
+				spend[k] += r.Micros
+			}
+		}
+	}
 	a.budgets.Restore(period, spend)
-	a.log.Info("restored budgets from the store", "keys", len(spend), "since", start)
+	a.log.Info("restored budgets from the store",
+		"keys", len(rows), "counters", len(spend), "since", start)
 	return nil
 }
 
 // Reservation is what Check returns: the estimate it charged, so Record can
 // settle the difference against the real figure.
+// pool is one level's counter and the period its budget runs over.
+type pool struct {
+	Counter string
+	Period  store.Period
+}
+
 type Reservation struct {
-	KeyID           string
+	KeyID string
+	// Budgets and Tokens are every counter the check went through, carried so
+	// the settlement lands on the same ones. Every, not the tightest: an
+	// organization's pool only fills up if each team's spend is added to it
+	// as well as to the team's own.
+	//
+	// Derived again at settlement would mean re-resolving the limits, and a
+	// budget checked against the team and settled against the key is a budget
+	// that never fills up.
+	Budgets         []pool
+	Tokens          []string
 	EstimatedTokens int64
 	Price           Price
 	// Period is the key's budget period, carried from the check so the record
@@ -99,14 +134,28 @@ type Reservation struct {
 // figure only exists once the request has run; Record settles the difference.
 func (a *Accountant) Check(p auth.Principal, model string, estimatedTokens int64) (Reservation, error) {
 	price := a.pricing.For(model)
+	lim := p.Limits
+	// Every level that set a limit, each against its own pool. A quota written
+	// on a team is one pool the team shares, and the organization above it is a
+	// second pool that the team's spending also fills — both have to hold, or
+	// an organization given 1000 with four teams given 500 spends 2000.
+	chain := levels(lim, p.KeyID)
+
 	res := Reservation{
 		KeyID: p.KeyID, EstimatedTokens: estimatedTokens,
-		Price: price, Period: p.Limits.BudgetPeriod,
+		Price: price, Period: lim.BudgetPeriod,
+	}
+	for _, l := range chain {
+		if l.Budget != nil {
+			res.Budgets = append(res.Budgets, pool{l.Counter, l.Period})
+		}
+		if l.TPM != nil {
+			res.Tokens = append(res.Tokens, l.Counter)
+		}
 	}
 	if p.KeyID == "" {
 		return res, nil
 	}
-	lim := p.Limits
 
 	// Budget first. It is the limit whose refusal cannot be retried away, so
 	// telling the client about it first is more useful than reporting a rate
@@ -114,15 +163,29 @@ func (a *Accountant) Check(p auth.Principal, model string, estimatedTokens int64
 	//
 	// A model that costs nothing is exempt, which is what lets a key with no
 	// budget left still be served a cheap fallback (plan §7.7).
-	if lim.Budget != nil && !price.Zero() &&
-		a.budgets.Exhausted(p.KeyID, lim.BudgetPeriod, *lim.Budget) {
-		return res, &LimitError{Err: ErrBudgetSpent}
+	// Budgets first, and all of them. Exhausted only reads, so the order costs
+	// nothing and a refusal here charges no counter.
+	if !price.Zero() {
+		for _, l := range chain {
+			if l.Budget != nil && a.budgets.Exhausted(l.Counter, l.Period, *l.Budget) {
+				return res, &LimitError{Err: ErrBudgetSpent}
+			}
+		}
 	}
 
-	if lim.RPM != nil && !a.requests.Allow(p.KeyID, *lim.RPM, 1) {
+	// Rate limits charge as they check — deliberately, so two concurrent
+	// requests cannot both see room. With several levels that means a refusal
+	// at one has already taken from the levels before it, so those are handed
+	// back. Without the refund a team at its ceiling would quietly eat its
+	// organization's rate limit with requests that were never served.
+	if err := a.allowAll(chain, func(l level) (string, int, int64, bool) {
+		return l.Counter, valInt(l.RPM), 1, l.RPM != nil
+	}, a.requests); err != nil {
 		return res, &LimitError{Err: ErrRateLimited, RetryAfter: time.Second}
 	}
-	if lim.TPM != nil && !a.tokens.Allow(p.KeyID, *lim.TPM, estimatedTokens) {
+	if err := a.allowAll(chain, func(l level) (string, int, int64, bool) {
+		return l.Counter, valInt(l.TPM), estimatedTokens, l.TPM != nil
+	}, a.tokens); err != nil {
 		// The request was not charged against the request-rate limit in vain:
 		// it was, and that is correct — it did arrive.
 		return res, &LimitError{Err: ErrTokenLimited, RetryAfter: time.Second}
@@ -130,6 +193,88 @@ func (a *Accountant) Check(p auth.Principal, model string, estimatedTokens int64
 	return res, nil
 }
 
+// level is one scope's limits with the counter they are enforced against.
+type level struct {
+	Counter string
+	Period  store.Period
+	RPM     *int
+	TPM     *int
+	Budget  *int64
+}
+
+// levels turns a resolved chain into the pools a request must pass.
+//
+// Falls back to the collapsed values when the chain is empty, which is what a
+// principal assembled by hand looks like — in tests, and anywhere that builds
+// limits without walking the tree.
+func levels(e store.Effective, keyID string) []level {
+	if len(e.Chain) == 0 {
+		var out []level
+		if e.RPM != nil || e.TPM != nil || e.Budget != nil {
+			out = append(out, level{
+				Counter: counterFor(e.BudgetOwner, keyID),
+				Period:  e.BudgetPeriod, Budget: e.Budget,
+			})
+			out[0].RPM, out[0].TPM = e.RPM, e.TPM
+			// One entry, so a hand-built principal with different owners per
+			// limit still counts each where it said to.
+			if r, b := counterFor(e.RPMOwner, keyID), out[0].Counter; r != b && e.RPM != nil {
+				out[0].RPM = nil
+				out = append(out, level{Counter: r, RPM: e.RPM})
+			}
+			if tp, b := counterFor(e.TPMOwner, keyID), out[0].Counter; tp != b && e.TPM != nil {
+				out[0].TPM = nil
+				out = append(out, level{Counter: tp, TPM: e.TPM})
+			}
+		}
+		return out
+	}
+	out := make([]level, 0, len(e.Chain))
+	for _, l := range e.Chain {
+		out = append(out, level{
+			Counter: store.LimitOwner{Scope: l.Scope, ID: l.ScopeID}.CounterKey(),
+			Period:  l.BudgetPeriod, RPM: l.RPM, TPM: l.TPM, Budget: l.Budget,
+		})
+	}
+	return out
+}
+
+// allowAll charges every level that has this limit, refunding on a refusal.
+func (a *Accountant) allowAll(chain []level,
+	pick func(level) (key string, limit int, n int64, has bool), lim *Limiter,
+) error {
+	var charged []struct {
+		key string
+		n   int64
+	}
+	for _, l := range chain {
+		key, limit, n, has := pick(l)
+		if !has {
+			continue
+		}
+		if !lim.Allow(key, limit, n) {
+			for _, c := range charged {
+				lim.Charge(c.key, -c.n)
+			}
+			return errRefused
+		}
+		charged = append(charged, struct {
+			key string
+			n   int64
+		}{key, n})
+	}
+	return nil
+}
+
+var errRefused = errors.New("refused")
+
+func valInt(v *int) int {
+	if v == nil {
+		return 0
+	}
+	return *v
+}
+
 // Record settles a finished request: it reconciles the token estimate, adds to
 // the budget, and queues the usage row.
 func (a *Accountant) Record(res Reservation, u store.Usage) {
@@ -141,10 +286,16 @@ func (a *Accountant) Record(res Reservation, u store.Usage) {
 	// The estimate was charged up front; charge the difference now. A caller
 	// whose requests are consistently larger than estimated is still bounded,
 	// one window later than with a perfect estimate.
-	a.tokens.Charge(res.KeyID, actual-res.EstimatedTokens)
+	// Every level the check went through, so an organization's pool fills up
+	// from its teams' spending rather than only from requests that named it.
+	for _, c := range res.Tokens {
+		a.tokens.Charge(c, actual-res.EstimatedTokens)
+	}
 
 	u.CostMicros = res.Price.Cost(u.PromptTokens, u.CompletionTokens)
-	a.budgets.Add(res.KeyID, res.Period, u.CostMicros)
+	for _, b := range res.Budgets {
+		a.budgets.Add(b.Counter, b.Period, u.CostMicros)
+	}
 
 	if a.recorder != nil {
 		a.recorder.Record(u)
@@ -156,6 +307,65 @@ func (a *Accountant) Status(p auth.Principal) (spent int64, rpm, tpm float64) {
 	if p.KeyID == "" {
 		return 0, 0, 0
 	}
-	return a.budgets.Spent(p.KeyID, p.Limits.BudgetPeriod),
-		a.requests.Rate(p.KeyID), a.tokens.Rate(p.KeyID)
+	// The pool that will refuse them, which is not always the one with the
+	// smallest number on it. A team given 2000 inside an organization given the
+	// same is bounded by whichever is nearer its ceiling, and that is the
+	// organization as soon as a second team spends anything. A header that
+	// reported the team's would disagree with the refusal that follows it.
+	for _, l := range levels(p.Limits, p.KeyID) {
+		if l.Budget != nil {
+			if v := a.budgets.Spent(l.Counter, l.Period); v > spent {
+				spent = v
+			}
+		}
+		if l.RPM != nil {
+			if v := a.requests.Rate(l.Counter); v > rpm {
+				rpm = v
+			}
+		}
+		if l.TPM != nil {
+			if v := a.tokens.Rate(l.Counter); v > tpm {
+				tpm = v
+			}
+		}
+	}
+	return spent, rpm, tpm
 }
+
+// counterFor is the counter a limit owned by this scope should use.
+//
+// Falls back to the key's own counter when no scope owns the limit, which
+// happens when nothing in the chain set one.
+//
+// Namespaced even then. A bare key id here and a "key:"-prefixed one in the
+// restore path would be two different counters for the same key, so a budget
+// would check against an empty one and survive every restart — which is how
+// this was wrong when it was first written.
+func counterFor(o store.LimitOwner, keyID string) string {
+	if k := o.CounterKey(); k != "" {
+		return k
+	}
+	return keyCounter(keyID)
+}
+
+// keyCounter is a key's own counter, in the same namespace as every other.
+func or(v, fallback string) string {
+	if v == "" {
+		return fallback
+	}
+	return v
+}
+
+func keyCounter(keyID string) string {
+	if keyID == "" {
+		return ""
+	}
+	return store.LimitOwner{Scope: store.ScopeKey, ID: keyID}.CounterKey()
+}
+
+// AnythingPriced reports whether a budget can ever be spent against.
+//
+// On the accountant rather than making callers reach for the price table: what
+// they are asking is whether the budgets they are about to display mean
+// anything, and the accountant is what would enforce them.
+func (a *Accountant) AnythingPriced() bool { return a.pricing.AnythingPriced() }

internal/accounting/accounting_test.go

@@ -288,3 +288,345 @@ func must(t *testing.T, err error) {
 		t.Fatal(err)
 	}
 }
+
+// teamMember is a key whose budget and rate limit were set on its team, which
+// is the arrangement an enterprise actually asks for: a group gets a quota and
+// the people in it share it.
+func teamMember(keyID, teamID string, tune func(*store.Effective)) auth.Principal {
+	e := store.Effective{KeyID: keyID, ProjectID: "proj", TeamID: teamID}
+	owner := store.LimitOwner{Scope: store.ScopeTeam, ID: teamID}
+	e.BudgetOwner, e.RPMOwner, e.TPMOwner = owner, owner, owner
+	if tune != nil {
+		tune(&e)
+	}
+	return auth.Principal{KeyID: keyID, ProjectID: "proj", Limits: e}
+}
+
+func TestATeamBudgetIsOnePoolRatherThanOnePerMember(t *testing.T) {
+	a := New(pricing(t), nil, slog.New(slog.DiscardHandler))
+	budget := func(e *store.Effective) {
+		e.Budget = i64p(3000)
+		e.BudgetPeriod = store.PeriodMonth
+	}
+	first := teamMember("k1", "t1", budget)
+	second := teamMember("k2", "t1", budget)
+
+	// One member spends the lot.
+	res, err := a.Check(first, "paid", 1000)
+	if err != nil {
+		t.Fatal(err)
+	}
+	a.Record(res, store.Usage{KeyID: "k1", PromptTokens: 1000, CompletionTokens: 1000})
+
+	// The second must now be refused. Counted per key it would have its own
+	// untouched 3000 — a team budget of 3000 with ten keys would permit thirty
+	// thousand, which is the bug this change exists for.
+	if _, err := a.Check(second, "paid", 10); !errors.Is(err, ErrBudgetSpent) {
+		t.Errorf("the team's budget should be shared; the second member got %v", err)
+	}
+}
+
+func TestATeamRateLimitIsSharedToo(t *testing.T) {
+	a := New(pricing(t), nil, slog.New(slog.DiscardHandler))
+	rpm := func(e *store.Effective) { e.RPM = intp(1) }
+
+	if _, err := a.Check(teamMember("k1", "t1", rpm), "free", 1); err != nil {
+		t.Fatal(err)
+	}
+	if _, err := a.Check(teamMember("k2", "t1", rpm), "free", 1); !errors.Is(err, ErrRateLimited) {
+		t.Error("a rate limit set on the team should be shared by its keys")
+	}
+}
+
+func TestTwoTeamsDoNotShareACounter(t *testing.T) {
+	a := New(pricing(t), nil, slog.New(slog.DiscardHandler))
+	rpm := func(e *store.Effective) { e.RPM = intp(1) }
+
+	if _, err := a.Check(teamMember("k1", "t1", rpm), "free", 1); err != nil {
+		t.Fatal(err)
+	}
+	// Pooling is the point; pooling the wrong things is the danger. One team
+	// exhausting another's quota would be worse than the bug being fixed.
+	if _, err := a.Check(teamMember("k2", "t2", rpm), "free", 1); err != nil {
+		t.Errorf("a second team should have its own counter, got %v", err)
+	}
+}
+
+func TestAKeyLimitStaysPerKey(t *testing.T) {
+	a := New(pricing(t), nil, slog.New(slog.DiscardHandler))
+	perKey := func(id string) auth.Principal {
+		return principal(id, func(e *store.Effective) {
+			e.RPM = intp(1)
+			e.RPMOwner = store.LimitOwner{Scope: store.ScopeKey, ID: id}
+		})
+	}
+
+	if _, err := a.Check(perKey("k1"), "free", 1); err != nil {
+		t.Fatal(err)
+	}
+	// The tree still has to be able to say "each of you may do this much".
+	// Sharing a group pool must not become the only thing expressible.
+	if _, err := a.Check(perKey("k2"), "free", 1); err != nil {
+		t.Errorf("a limit owned by the key should not be shared, got %v", err)
+	}
+}
+
+func TestTheDifferentLimitsCanBeOwnedByDifferentScopes(t *testing.T) {
+	a := New(pricing(t), nil, slog.New(slog.DiscardHandler))
+	// A team budget with a per-key rate limit is the ordinary arrangement: the
+	// group's money, the individual's throughput.
+	mk := func(id string) auth.Principal {
+		e := store.Effective{KeyID: id, TeamID: "t1"}
+		e.Budget, e.BudgetPeriod = i64p(3000), store.PeriodMonth
+		e.BudgetOwner = store.LimitOwner{Scope: store.ScopeTeam, ID: "t1"}
+		e.RPM = intp(1)
+		e.RPMOwner = store.LimitOwner{Scope: store.ScopeKey, ID: id}
+		return auth.Principal{KeyID: id, Limits: e}
+	}
+
+	res, err := a.Check(mk("k1"), "paid", 1000)
+	if err != nil {
+		t.Fatal(err)
+	}
+	a.Record(res, store.Usage{KeyID: "k1", PromptTokens: 1000, CompletionTokens: 1000})
+
+	// k2's rate limit is its own and untouched; the budget is the team's and
+	// gone. The budget is checked first, so that is what comes back.
+	if _, err := a.Check(mk("k2"), "paid", 10); !errors.Is(err, ErrBudgetSpent) {
+		t.Errorf("expected the shared budget to bite, got %v", err)
+	}
+}
+
+func TestStatusReportsTheCounterThatWillRefuseYou(t *testing.T) {
+	a := New(pricing(t), nil, slog.New(slog.DiscardHandler))
+	budget := func(e *store.Effective) {
+		e.Budget = i64p(9000)
+		e.BudgetPeriod = store.PeriodMonth
+	}
+	res, err := a.Check(teamMember("k1", "t1", budget), "paid", 1000)
+	if err != nil {
+		t.Fatal(err)
+	}
+	a.Record(res, store.Usage{KeyID: "k1", PromptTokens: 1000, CompletionTokens: 1000})
+
+	// A member who has spent nothing themselves still sees the team's spend,
+	// because that is the number that will stop them. A header disagreeing with
+	// the refusal it precedes is worse than no header.
+	spent, _, _ := a.Status(teamMember("k2", "t1", budget))
+	if spent == 0 {
+		t.Error("a member should see the pool's spend, not their own")
+	}
+}
+
+func TestATeamBudgetSurvivesARestart(t *testing.T) {
+	// The per-key restore could not do this: it rebuilt a counter named after
+	// the key, while the check reads one named after the team. The control
+	// plane came back believing the team had spent nothing — a budget that
+	// resets on every restart, for exactly the scope where it matters most.
+	ctx := context.Background()
+	s, err := store.OpenSQLite(ctx, filepath.Join(t.TempDir(), "team.db"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer s.Close()
+
+	must(t, s.PutOrg(ctx, store.Org{ID: "o", Name: "o"}))
+	must(t, s.PutTeam(ctx, store.Team{ID: "t1", OrgID: "o", Name: "t"}))
+	must(t, s.PutProject(ctx, store.Project{ID: "proj", TeamID: "t1", Name: "p"}))
+	for _, id := range []string{"k1", "k2"} {
+		_, k, _ := store.NewKey(id, "n", "proj")
+		k.CreatedAt = time.Now()
+		must(t, s.CreateKey(ctx, k))
+	}
+
+	// Two members of one team, 2500 each: neither alone exceeds a 4000 budget,
+	// together they do.
+	must(t, s.WriteUsage(ctx, []store.Usage{
+		{At: time.Now(), KeyID: "k1", ProjectID: "proj", Model: "paid",
+			ServedModel: "paid", CostMicros: 2500, Status: 200},
+		{At: time.Now(), KeyID: "k2", ProjectID: "proj", Model: "paid",
+			ServedModel: "paid", CostMicros: 2500, Status: 200},
+	}))
+
+	a := New(pricing(t), nil, slog.New(slog.DiscardHandler))
+	must(t, a.RestoreBudgets(ctx, s, store.PeriodMonth))
+
+	p := teamMember("k1", "t1", func(e *store.Effective) {
+		e.Budget = i64p(4000)
+		e.BudgetPeriod = store.PeriodMonth
+	})
+	if _, err := a.Check(p, "paid", 10); !errors.Is(err, ErrBudgetSpent) {
+		t.Errorf("the team's spend should have been restored and pooled: %v", err)
+	}
+}
+
+// inTree is a key under org/team/project, with whatever each level sets.
+func inTree(keyID string, chain ...store.Limits) auth.Principal {
+	e := store.Effective{KeyID: keyID, Chain: chain}
+	// The collapsed values still exist for headers and the profile page; they
+	// are not what enforcement reads any more.
+	for _, l := range chain {
+		if l.Budget != nil && (e.Budget == nil || *l.Budget < *e.Budget) {
+			e.Budget, e.BudgetPeriod = l.Budget, l.BudgetPeriod
+			e.BudgetOwner = store.LimitOwner{Scope: l.Scope, ID: l.ScopeID}
+		}
+	}
+	return auth.Principal{KeyID: keyID, Limits: e}
+}
+
+func orgBudget(id string, v int64) store.Limits {
+	return store.Limits{Scope: store.ScopeOrg, ScopeID: id,
+		Budget: i64p(v), BudgetPeriod: store.PeriodMonth}
+}
+
+func teamBudget(id string, v int64) store.Limits {
+	return store.Limits{Scope: store.ScopeTeam, ScopeID: id,
+		Budget: i64p(v), BudgetPeriod: store.PeriodMonth}
+}
+
+func TestTheSumOfTeamsCannotExceedTheOrganization(t *testing.T) {
+	a := New(pricing(t), nil, slog.New(slog.DiscardHandler))
+
+	// Each request costs 800. The organization allows 2000 and each of its four
+	// teams allows 2000, so no team comes close to its own ceiling — but the
+	// organization's fills after three. Taking only the most restrictive value,
+	// the organization's number was never looked at and all four went through.
+	//
+	// A budget stops spending once it is reached rather than refusing the
+	// request that would cross it, so the third is served and the fourth is not.
+	spend := func(team string) error {
+		p := inTree(team+"-k", orgBudget("o", 2000), teamBudget(team, 2000))
+		res, err := a.Check(p, "paid", 1000)
+		if err != nil {
+			return err
+		}
+		a.Record(res, store.Usage{KeyID: team + "-k", PromptTokens: 200, CompletionTokens: 200})
+		return nil
+	}
+
+	total := 0
+	for _, team := range []string{"t1", "t2", "t3", "t4"} {
+		if err := spend(team); err == nil {
+			total++
+		}
+	}
+	if total != 3 {
+		t.Errorf("expected the organization to stop the fourth team, %d got through", total)
+	}
+}
+
+func TestATeamStillStopsBeforeItsOrganizationDoes(t *testing.T) {
+	a := New(pricing(t), nil, slog.New(slog.DiscardHandler))
+	p := func() auth.Principal {
+		return inTree("k1", orgBudget("o", 100_000), teamBudget("t1", 800))
+	}
+
+	res, err := a.Check(p(), "paid", 1000)
+	if err != nil {
+		t.Fatal(err)
+	}
+	a.Record(res, store.Usage{KeyID: "k1", PromptTokens: 200, CompletionTokens: 200})
+
+	// The narrower pool binds first, which is what it is for. Enforcing every
+	// level must not stop the tighter one working.
+	if _, err := a.Check(p(), "paid", 10); !errors.Is(err, ErrBudgetSpent) {
+		t.Errorf("the team's own budget should still bite, got %v", err)
+	}
+}
+
+func TestSpendingFillsEveryPoolAboveIt(t *testing.T) {
+	a := New(pricing(t), nil, slog.New(slog.DiscardHandler))
+	p := inTree("k1", orgBudget("o", 5000), teamBudget("t1", 5000))
+
+	res, err := a.Check(p, "paid", 1000)
+	if err != nil {
+		t.Fatal(err)
+	}
+	a.Record(res, store.Usage{KeyID: "k1", PromptTokens: 200, CompletionTokens: 200})
+
+	// Settled onto both, or the organization's pool would only ever fill from
+	// requests that named it directly — which is none of them.
+	if got := a.budgets.Spent("org:o", store.PeriodMonth); got == 0 {
+		t.Error("the organization's pool did not fill")
+	}
+	if got := a.budgets.Spent("team:t1", store.PeriodMonth); got == 0 {
+		t.Error("the team's pool did not fill")
+	}
+}
+
+func TestARefusalHandsBackWhatTheNarrowerLevelsTook(t *testing.T) {
+	a := New(pricing(t), nil, slog.New(slog.DiscardHandler))
+	// The team allows plenty; the organization allows one request a minute.
+	chain := []store.Limits{
+		{Scope: store.ScopeOrg, ScopeID: "o", RPM: intp(1)},
+		{Scope: store.ScopeTeam, ScopeID: "t1", RPM: intp(100)},
+	}
+	if _, err := a.Check(inTree("k1", chain...), "free", 1); err != nil {
+		t.Fatal(err)
+	}
+	before := a.requests.Rate("team:t1")
+
+	if _, err := a.Check(inTree("k2", chain...), "free", 1); !errors.Is(err, ErrRateLimited) {
+		t.Fatalf("the organization should have refused the second, got %v", err)
+	}
+
+	// A rate limiter charges as it checks, so a refusal further along has
+	// already taken from the levels before it. Left there, a team at its
+	// organization's ceiling would quietly eat its own rate limit with requests
+	// that were never served.
+	if after := a.requests.Rate("team:t1"); after != before {
+		t.Errorf("the refused request stayed charged to the team: %v -> %v", before, after)
+	}
+}
+
+func TestAChainWithNoLimitsAnywhereAllowsEverything(t *testing.T) {
+	a := New(pricing(t), nil, slog.New(slog.DiscardHandler))
+	for i := 0; i < 5; i++ {
+		if _, err := a.Check(inTree("k1"), "paid", 1000); err != nil {
+			t.Fatalf("nothing is limited, but request %d was refused: %v", i, err)
+		}
+	}
+}
+
+func TestStatusReportsTheFullestPoolNotTheSmallestNumber(t *testing.T) {
+	a := New(pricing(t), nil, slog.New(slog.DiscardHandler))
+	chain := func(team string) auth.Principal {
+		return inTree(team+"-k", orgBudget("o", 9000), teamBudget(team, 9000))
+	}
+
+	// Another team spends; this one has spent nothing of its own.
+	res, err := a.Check(chain("t1"), "paid", 1000)
+	if err != nil {
+		t.Fatal(err)
+	}
+	a.Record(res, store.Usage{KeyID: "t1-k", PromptTokens: 200, CompletionTokens: 200})
+
+	spent, _, _ := a.Status(chain("t2"))
+	// The organization's pool is the one nearer its ceiling, so it is the one
+	// that will refuse them. Reporting the team's nil would disagree with the
+	// refusal that eventually follows it.
+	if spent == 0 {
+		t.Error("a member should see the fullest pool that bounds them")
+	}
+}
+
+func TestAnEmptyPriceTableMeansNoBudgetCanEverBind(t *testing.T) {
+	// The ordinary case for self-hosted models, and the reason a budget on a
+	// team can read as a limit while binding on nothing.
+	if NewPricing(nil, Price{}).AnythingPriced() {
+		t.Error("nothing priced should say so")
+	}
+	if NewPricing(map[string]Price{"m": {}}, Price{}).AnythingPriced() {
+		t.Error("a model priced at zero is not priced")
+	}
+	if !NewPricing(map[string]Price{"m": {PromptPer1k: 1}}, Price{}).AnythingPriced() {
+		t.Error("one priced model is enough for a budget to move")
+	}
+	if !NewPricing(nil, Price{CompletionPer1k: 1}).AnythingPriced() {
+		t.Error("a fallback price prices everything")
+	}
+	var nilPricing *Pricing
+	if nilPricing.AnythingPriced() {
+		t.Error("no price table at all is not priced")
+	}
+}

internal/accounting/cost.go

@@ -68,3 +68,30 @@ func (p *Pricing) For(model string) Price {
 	}
 	return p.fallback
 }
+
+// AnythingPriced reports whether any model costs anything.
+//
+// A budget is spend per period, and spend is computed from a price table. With
+// no prices every request costs nothing, so a budget written on a team is a
+// number that can never move — displayed as a limit, reported as zero percent
+// used, and binding on nothing.
+//
+// Self-hosted models have no per-token price, so an empty table is the ordinary
+// case rather than a misconfiguration. What is not ordinary is showing somebody
+// a budget without saying so.
+func (p *Pricing) AnythingPriced() bool {
+	if p == nil {
+		return false
+	}
+	p.mu.RLock()
+	defer p.mu.RUnlock()
+	if !p.fallback.Zero() {
+		return true
+	}
+	for _, price := range p.byModel {
+		if !price.Zero() {
+			return true
+		}
+	}
+	return false
+}

internal/agent/agent.go

@@ -10,11 +10,14 @@ import (
 	"fmt"
 	"io"
 	"log/slog"
+	"maps"
 	"math/rand/v2"
 	"net"
 	"net/http"
+	"net/url"
 	"os"
 	"path/filepath"
+	"slices"
 	"strings"
 	"sync"
 	"time"
@@ -25,6 +28,7 @@ import (
 	"gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1/llmgwv1connect"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/supervisor"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/weights"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/agentsvc"
@@ -34,13 +38,33 @@ import (
 
 // Config configures the agent.
 type Config struct {
-	NodeID   string            `koanf:"node_id"`
-	Server   string            `koanf:"server"`
-	Hostname string            `koanf:"hostname"`
-	MgmtAddr string            `koanf:"mgmt_addr"`
-	Labels   map[string]string `koanf:"labels"`
+	// NodeID defaults to the hostname. It must equal the common name of the
+	// agent's certificate, so set it only where the two differ.
+	NodeID string `koanf:"node_id"`
+	// Server is the control plane, and Servers is the same thing when there is
+	// more than one. Both are accepted: every configuration written before
+	// there could be a second control plane names one address, and none of
+	// them should have to change.
+	//
+	// The order is tried in order, and a refusal moves to the next. Only the
+	// leader accepts an agent, so a standby is skipped in the time it takes to
+	// be told so — which is what makes failover a reconnect rather than an
+	// edit to every node's configuration.
+	Server  string   `koanf:"server"`
+	Servers []string `koanf:"servers"`
+	// Hostname defaults to the machine's.
+	Hostname string `koanf:"hostname"`
+	// MgmtAddr is the address engines are reached on. It defaults to this
+	// machine's address on the route to the control plane, which is the one
+	// the control plane and the router can reach.
+	MgmtAddr string `koanf:"mgmt_addr"`
+	// Labels are free-form facts about a node for a profile's node_selector.
+	// Nothing needs them: memory, GPUs, engines and disk are measured, and a
+	// label restating one of those goes stale the day the machine changes.
+	Labels map[string]string `koanf:"labels"`
 
-	// MgmtInterface is the ethernet interface NCCL bootstraps over.
+	// MgmtInterface is the ethernet interface NCCL bootstraps over. It
+	// defaults to the interface holding MgmtAddr, then eth0.
 	MgmtInterface string `koanf:"mgmt_interface"`
 	// Fabrics labels this node's HCAs. It is explicit on purpose: OpenSM's
 	// default subnet prefix is the same on every fabric, so nothing can be
@@ -63,19 +87,22 @@ type Config struct {
 	// keeps the plaintext behaviour, which only belongs on a trusted network.
 	TLS tlsconf.Config `koanf:"tls"`
 
-	// ModelDirs are the directories this node keeps model weights in.
-	//
-	// Empty by default, and empty means the node is not asked to look — which
-	// is not the same as having nothing. Storage differs at every installation,
+	// ModelDirs are the directories this node keeps model weights in, beyond
+	// WeightsCache, which is always looked in. Storage differs at every installation,
 	// so llmgw observes it rather than owning it: a Lustre mount, an image
 	// exported over the fabric, local NVMe and a copy made by hand all look
 	// identical from here, and placement only needs to know whether the file
 	// is present.
 	ModelDirs []string `koanf:"model_dirs"`
 
+	// EnginesDir is where the agent installs engines, one directory per
+	// build. Defaults to /opt/llmgw/engines.
+	EnginesDir string `koanf:"engines_dir"`
+
 	// WeightsCache is where a remote source is downloaded to. Defaults to the
 	// first entry of ModelDirs, because a node that says where its models live
-	// has already answered this question.
+	// has already answered this question, and otherwise to
+	// /var/lib/llmgw-weights.
 	//
 	// Fetching is the simple arrangement: every node keeps its own copy. The
 	// exotic ones need no code — weights on a shared filer or an image exported
@@ -105,21 +132,47 @@ func DefaultConfig() Config {
 
 func (c *Config) normalize() error {
 	d := DefaultConfig()
-	if c.NodeID == "" {
-		return errors.New("node_id is required")
-	}
-	if c.Server == "" {
-		return errors.New("server is required")
-	}
 	if err := c.TLS.Validate(); err != nil {
 		return err
 	}
 	if err := c.normalizeServerURL(); err != nil {
 		return err
 	}
+	// Everything below is found rather than written. A file that has to say
+	// which machine it is on, and at what address, is a file that differs on
+	// every node — and a copy taken from one node to the next registers the
+	// second as the first.
+	host, _ := os.Hostname()
+	if c.NodeID == "" {
+		c.NodeID = host
+	}
+	if c.NodeID == "" {
+		return errors.New("node_id is not set and the hostname could not be read")
+	}
+	if c.Hostname == "" {
+		c.Hostname = host
+	}
+	if c.MgmtAddr == "" {
+		c.MgmtAddr = addrToward(c.Server)
+	}
+	if c.MgmtInterface == "" {
+		c.MgmtInterface = interfaceOf(c.MgmtAddr)
+	}
 	if c.MgmtInterface == "" {
 		c.MgmtInterface = d.MgmtInterface
 	}
+	// The directory downloads land in is always one that is looked in. When
+	// they were two settings, a node could fill its disk with fetched weights
+	// that placement never saw — which is what every node here was doing.
+	if c.WeightsCache == "" && len(c.ModelDirs) > 0 {
+		c.WeightsCache = c.ModelDirs[0]
+	}
+	if c.WeightsCache == "" {
+		c.WeightsCache = DefaultWeightsCache
+	}
+	if !slices.Contains(c.ModelDirs, c.WeightsCache) {
+		c.ModelDirs = append(c.ModelDirs, c.WeightsCache)
+	}
 	if c.HeartbeatInterval <= 0 {
 		c.HeartbeatInterval = d.HeartbeatInterval
 	}
@@ -140,26 +193,114 @@ func (c *Config) normalize() error {
 // that says plaintext while TLS is configured. Silently connecting anyway would
 // mean a cluster believing it has mutual authentication when it has none.
 func (c *Config) normalizeServerURL() error {
-	secure := c.TLS.Enabled()
-	switch {
-	case strings.HasPrefix(c.Server, "https://"):
-		if !secure {
-			return errors.New("server is an https:// address but no tls certificates are configured")
-		}
-	case strings.HasPrefix(c.Server, "http://"):
-		if secure {
-			return errors.New("tls is configured but server is an http:// address; use https://")
-		}
-	case strings.Contains(c.Server, "://"):
-		return fmt.Errorf("server %q must be an http:// or https:// address", c.Server)
-	case secure:
-		c.Server = "https://" + c.Server
-	default:
-		c.Server = "http://" + c.Server
+	// One list, however it was written. Server first when both are given,
+	// because an operator adding a standby to an existing file expects the
+	// one already there to stay preferred.
+	all := make([]string, 0, 1+len(c.Servers))
+	if c.Server != "" {
+		all = append(all, c.Server)
 	}
+	all = append(all, c.Servers...)
+
+	seen := map[string]struct{}{}
+	out := make([]string, 0, len(all))
+	for _, s := range all {
+		fixed, err := normalizeOne(s, c.TLS.Enabled())
+		if err != nil {
+			return err
+		}
+		// A duplicate would be tried twice in a row on failover, which reads
+		// as the rotation not working.
+		if _, dup := seen[fixed]; dup {
+			continue
+		}
+		seen[fixed] = struct{}{}
+		out = append(out, fixed)
+	}
+	if len(out) == 0 {
+		return errors.New("server is required")
+	}
+	c.Servers = out
+	c.Server = out[0]
 	return nil
 }
 
+// DefaultWeightsCache is where fetched weights go when nothing says otherwise.
+const DefaultWeightsCache = "/var/lib/llmgw-weights"
+
+// addrToward is this machine's address on the route to the control plane.
+//
+// The address engines are reached on is whichever one the control plane can
+// reach, and the kernel already knows which that is: a UDP "connection" picks
+// the source address for the route without sending a packet. Asking the
+// interfaces instead would have to guess among docker bridges, VPNs and
+// link-local addresses.
+func addrToward(server string) string {
+	u, err := url.Parse(server)
+	if err != nil || u.Host == "" {
+		return ""
+	}
+	host := u.Hostname()
+	port := u.Port()
+	if port == "" {
+		port = "443"
+	}
+	conn, err := net.Dial("udp", net.JoinHostPort(host, port))
+	if err != nil {
+		return ""
+	}
+	defer conn.Close()
+	if a, ok := conn.LocalAddr().(*net.UDPAddr); ok {
+		return a.IP.String()
+	}
+	return ""
+}
+
+// interfaceOf names the interface that holds an address, or "" when none does.
+func interfaceOf(addr string) string {
+	ip := net.ParseIP(addr)
+	if ip == nil {
+		return ""
+	}
+	ifaces, err := net.Interfaces()
+	if err != nil {
+		return ""
+	}
+	for _, ifc := range ifaces {
+		addrs, err := ifc.Addrs()
+		if err != nil {
+			continue
+		}
+		for _, a := range addrs {
+			if n, ok := a.(*net.IPNet); ok && n.IP.Equal(ip) {
+				return ifc.Name
+			}
+		}
+	}
+	return ""
+}
+
+// normalizeOne applies the scheme rules to a single address.
+func normalizeOne(addr string, secure bool) (string, error) {
+	switch {
+	case strings.HasPrefix(addr, "https://"):
+		if !secure {
+			return "", fmt.Errorf("%s is an https:// address but no tls certificates are configured", addr)
+		}
+	case strings.HasPrefix(addr, "http://"):
+		if secure {
+			return "", fmt.Errorf("tls is configured but %s is an http:// address; use https://", addr)
+		}
+	case strings.Contains(addr, "://"):
+		return "", fmt.Errorf("server %q must be an http:// or https:// address", addr)
+	case secure:
+		return "https://" + addr, nil
+	default:
+		return "http://" + addr, nil
+	}
+	return addr, nil
+}
+
 // Agent runs on a GPU node.
 type Agent struct {
 	cfg  Config
@@ -168,6 +309,35 @@ type Agent struct {
 	reg  *engines.Registry
 	http connectHTTPClient
 
+	// engineOnce probes the configured engines once before registering.
+	//
+	// Once, because probing SGLang imports it in a Python interpreter — seconds
+	// on a cold cache — and a reconnect is not a reason to pay that again.
+	// Later changes are found by the slower re-probe the heartbeat starts.
+	engineOnce sync.Once
+	// configuredVers are the engines the configuration names and that answer.
+	// Guarded by mu once the first probe has run.
+	configuredVers map[string]string
+	// installedVers are the engines this agent installed and saw run.
+	// Guarded by mu.
+	installedVers map[string]string
+
+	// prov installs the engines the control plane asks for. provWants is what
+	// it asked for, nil until it has; provStatus is where each stands and
+	// provSent the fingerprint of what the control plane last heard. All
+	// guarded by mu. provKick starts a pass without waiting for the timer.
+	prov       *provision.Installer
+	provWants  []provision.Want
+	provStatus map[string]provision.Status
+	provSent   string
+	provKick   chan struct{}
+	// enginesSent is the fingerprint of the engines the control plane last
+	// heard about, from registration or a heartbeat. Guarded by mu.
+	enginesSent string
+	// lastEngineProbe and probingEngines pace the re-probe. Guarded by mu.
+	lastEngineProbe time.Time
+	probingEngines  bool
+
 	mu  sync.Mutex
 	inv discovery.Inventory
 	// shardReplica remembers which replica a shard belongs to, so a status
@@ -176,9 +346,16 @@ type Agent struct {
 	fetcher      *weights.Fetcher
 
 	// lastWeights is the fingerprint of the file list most recently sent, so a
-	// heartbeat carries the list only when it changed. Touched only by the
-	// heartbeat goroutine, which is the only thing that reads or writes it.
+	// heartbeat carries the list only when it changed. Guarded by mu: during a
+	// failover the old stream's heartbeat loop can still be running when the
+	// new one's starts.
 	lastWeights string
+
+	// fetchWatchers are the shards waiting on each source, so progress on one
+	// download reaches every shard that asked for it. A fetch is single-flight
+	// — eight shards starting together produce one download — and all eight
+	// are waiting, so all eight should say so.
+	fetchWatchers map[string][]func(weights.Progress)
 }
 
 // connectHTTPClient is the HTTP client connect uses; an interface so tests can
@@ -204,12 +381,16 @@ func New(cfg Config, httpClient connectHTTPClient, log *slog.Logger) (*Agent, er
 	if err != nil {
 		return nil, err
 	}
-	return &Agent{
+	a := &Agent{
 		cfg: cfg, log: log, sup: sup,
 		reg:          engines.NewRegistry(cfg.Engines),
 		http:         httpClient,
 		shardReplica: map[string]string{},
-	}, nil
+		provStatus:   map[string]provision.Status{},
+		provKick:     make(chan struct{}, 1),
+	}
+	a.prov = &provision.Installer{Root: cfg.EnginesDir, Log: log, OnStatus: a.provisionStatus}
+	return a, nil
 }
 
 // Supervisor exposes the process supervisor, for tests and `ctl logs`.
@@ -228,17 +409,33 @@ func (a *Agent) Run(ctx context.Context) error {
 		a.log.Error("hardware discovery failed", "err", err)
 	}
 	a.recoverOrphans()
+	go a.provisionLoop(ctx)
 
 	attempt := 0
+	next := 0
 	for ctx.Err() == nil {
-		err := a.session(ctx)
+		server := a.cfg.Servers[next%len(a.cfg.Servers)]
+		err := a.session(ctx, server)
 		if ctx.Err() != nil {
 			break
 		}
+		// On to the next address whatever went wrong. A standby refuses, a
+		// stopped control plane refuses the connection, and a crashed one
+		// never answers — from here they are the same thing, and trying
+		// somewhere else is the answer to all three.
+		next++
 		attempt++
 		delay := a.reconnectDelay(attempt)
+		// No wait when there is somewhere else to try: the delay exists to
+		// stop hammering one control plane, and the next address has not been
+		// hammered. With a single address this is the old behaviour exactly.
+		if len(a.cfg.Servers) > 1 && next%len(a.cfg.Servers) != 0 {
+			delay = 0
+		}
 		a.log.Warn("disconnected from the control plane, retrying",
-			"err", err, "in", delay.Round(time.Millisecond), "attempt", attempt)
+			"err", err, "server", server,
+			"next", a.cfg.Servers[next%len(a.cfg.Servers)],
+			"in", delay.Round(time.Millisecond), "attempt", attempt)
 
 		select {
 		case <-ctx.Done():
@@ -401,12 +598,20 @@ func (a *Agent) weights() *weights.Fetcher {
 		CacheDir: cache,
 		Log:      a.log,
 		OnProgress: func(p weights.Progress) {
-			// Logged rather than silent: a replica that is downloading a
-			// hundred gigabytes and one that is wedged look identical from the
-			// control plane, and the difference matters to whoever is waiting.
 			a.log.Info("fetching weights",
 				"source", p.Source, "percent", p.Percent(),
 				"bytes", p.BytesDone, "total", p.BytesTotal)
+			// And upward, to whoever is watching from the control plane. The
+			// log line above is on the node, which is where nobody is looking
+			// when a replica has been starting for ten minutes.
+			a.mu.Lock()
+			watchers := slices.Clone(a.fetchWatchers[p.Source])
+			a.mu.Unlock()
+			for _, w := range watchers {
+				if w != nil {
+					w(p)
+				}
+			}
 		},
 	}
 	return a.fetcher
@@ -437,6 +642,7 @@ func (a *Agent) runningShards() []*llmgwv1.RunningShard {
 			Profile:    spec.Profile.Name,
 			Pool:       spec.Pool,
 			Fabric:     spec.Fabric,
+			ProfileFp:  spec.ProfileFP,
 			LeaderAddr: a.cfg.MgmtAddr,
 			LeaderPort: int32(spec.LeaderPort),
 		})
@@ -462,15 +668,19 @@ func (a *Agent) scan(ctx context.Context) error {
 	opts := a.cfg.Discovery
 	opts.ModelDirs = a.cfg.ModelDirs
 	inv, err := discovery.Discover(ctx, opts, a.log)
+	host := discovery.DiscoverHost()
 	a.mu.Lock()
 	a.inv = inv
+	a.prov.Host = host
 	a.mu.Unlock()
+	a.log.Info("this node", "os", host.OSID+" "+host.OSVersion, "arch", host.Arch,
+		"cpu_flags", host.CPUFlags, "nvidia_driver", host.NVIDIADriver)
 	return err
 }
 
 // session runs one connection until it breaks.
-func (a *Agent) session(ctx context.Context) error {
-	client := llmgwv1connect.NewAgentServiceClient(a.http, a.cfg.Server)
+func (a *Agent) session(ctx context.Context, server string) error {
+	client := llmgwv1connect.NewAgentServiceClient(a.http, server)
 
 	// parent is kept because the derived context is cancelled from inside this
 	// function; asking the derived one whether we are shutting down would
@@ -497,7 +707,23 @@ func (a *Agent) session(ctx context.Context) error {
 	// Every registration reports what is already running, not just the first
 	// one: a reconnect may be a control plane restart, and it has to recover
 	// the same way (plan §6.5).
+	// What this node can actually run, so the control plane stops placing
+	// engines that are not here. The registry holds an adapter for every
+	// engine whether or not its binary exists, so without this a node with no
+	// vLLM accepted a vLLM shard and failed it minutes later.
+	reg.EngineVersions = a.engines(parent)
+	reg.EnginesProbed = true
 	reg.RunningShards = a.runningShards()
+	a.mu.Lock()
+	h := a.prov.Host
+	// A new stream is a control plane that knows nothing of where installs
+	// stand, so the next heartbeat tells it again.
+	a.provSent = ""
+	a.mu.Unlock()
+	reg.Host = &llmgwv1.Host{
+		OsId: h.OSID, OsVersion: h.OSVersion, OsCodename: h.OSCodename,
+		Arch: h.Arch, CpuFlags: h.CPUFlags, NvidiaDriver: h.NVIDIADriver,
+	}
 
 	if err := stream.Send(&llmgwv1.AgentMessage{
 		Msg: &llmgwv1.AgentMessage_Register{Register: reg},
@@ -584,6 +810,33 @@ func (a *Agent) heartbeat() *llmgwv1.AgentMessage {
 	a.mu.Unlock()
 
 	hb := &llmgwv1.Heartbeat{NodeId: a.cfg.NodeID}
+
+	// Read every time; it is one small file. Memory is added to a running VM,
+	// and a node that grew from 7 GB to 32 GB has to be seen to have grown
+	// without anybody restarting anything.
+	hb.MemoryTotalMb = int64(discovery.MemoryTotalMB())
+
+	a.reprobeEngines()
+	a.mu.Lock()
+	if vers := a.engineReportLocked(); a.configuredVers != nil &&
+		enginesFingerprint(vers) != a.enginesSent {
+		a.enginesSent = enginesFingerprint(vers)
+		hb.EnginesChanged = true
+		hb.EngineVersions = vers
+	}
+	if fp := provisionFingerprint(a.provStatus); fp != a.provSent {
+		a.provSent = fp
+		hb.ProvisionChanged = true
+		for _, e := range slices.Sorted(maps.Keys(a.provStatus)) {
+			st := a.provStatus[e]
+			hb.Provision = append(hb.Provision, &llmgwv1.ProvisionStatus{
+				Engine: st.Engine, Version: st.Version, Variant: st.Variant,
+				State: st.State, Detail: st.Detail,
+			})
+		}
+	}
+	a.mu.Unlock()
+
 	for _, g := range inv.GPUs {
 		hb.GpuMemoryUsedMb = append(hb.GpuMemoryUsedMb, int32(g.MemoryUsedMB))
 	}
@@ -604,8 +857,12 @@ func (a *Agent) heartbeat() *llmgwv1.AgentMessage {
 	// rather than "nothing to report" — without it, the first heartbeat after
 	// a model was deleted would be indistinguishable from silence.
 	weights := discovery.ScanWeights(a.cfg.ModelDirs)
-	if fp := weightsFingerprint(weights); fp != a.lastWeights {
-		a.lastWeights = fp
+	fp := weightsFingerprint(weights)
+	a.mu.Lock()
+	changed := fp != a.lastWeights
+	a.lastWeights = fp
+	a.mu.Unlock()
+	if changed {
 		hb.WeightsChanged = true
 		for _, w := range weights {
 			hb.Weights = append(hb.Weights, &llmgwv1.LocalWeights{
@@ -696,6 +953,8 @@ func (a *Agent) handle(ctx context.Context, msg *llmgwv1.ControlMessage, out cha
 		}
 	case *llmgwv1.ControlMessage_RemoveWeights:
 		a.removeWeights(m.RemoveWeights.GetPaths())
+	case *llmgwv1.ControlMessage_Provision:
+		a.setProvision(m.Provision)
 	case *llmgwv1.ControlMessage_Ack:
 		if !m.Ack.GetOk() {
 			a.log.Warn("control plane rejected a message",
@@ -708,6 +967,42 @@ func (a *Agent) handle(ctx context.Context, msg *llmgwv1.ControlMessage, out cha
 //
 // It runs in its own goroutine because model loading takes minutes, and the
 // stream has to keep carrying heartbeats and other commands meanwhile.
+// watchFetch registers interest in a source's download progress and returns
+// the function that stops watching.
+//
+// Keyed by source rather than by shard because a fetch is single-flight: eight
+// shards starting together produce one download, and every one of them is
+// waiting on it.
+func (a *Agent) watchFetch(source string, fn func(weights.Progress)) (stop func()) {
+	a.mu.Lock()
+	if a.fetchWatchers == nil {
+		a.fetchWatchers = map[string][]func(weights.Progress){}
+	}
+	a.fetchWatchers[source] = append(a.fetchWatchers[source], fn)
+	i := len(a.fetchWatchers[source]) - 1
+	a.mu.Unlock()
+
+	var once sync.Once
+	return func() {
+		once.Do(func() {
+			a.mu.Lock()
+			defer a.mu.Unlock()
+			w := a.fetchWatchers[source]
+			if i < len(w) {
+				// Cleared rather than removed, so the indexes other watchers
+				// hold stay valid. The slice is dropped when the last one goes.
+				w[i] = nil
+			}
+			for _, f := range w {
+				if f != nil {
+					return
+				}
+			}
+			delete(a.fetchWatchers, source)
+		})
+	}
+}
+
 func (a *Agent) startShard(ctx context.Context, m *llmgwv1.StartShard, out chan<- *llmgwv1.AgentMessage) {
 	shardID, replicaID := m.GetShardId(), m.GetReplicaId()
 
@@ -715,16 +1010,18 @@ func (a *Agent) startShard(ctx context.Context, m *llmgwv1.StartShard, out chan<
 	a.shardReplica[shardID] = replicaID
 	a.mu.Unlock()
 
-	report := func(st supervisor.State, errMsg string, tail []string) {
-		msg := &llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_Shard{Shard: &llmgwv1.ShardStatus{
-			NodeId: a.cfg.NodeID, ReplicaId: replicaID, ShardId: shardID,
-			State: agentsvc.StateToWire(st), Error: errMsg, LastOutput: tail,
-		}}}
+	send := func(s *llmgwv1.ShardStatus) {
+		s.NodeId, s.ReplicaId, s.ShardId = a.cfg.NodeID, replicaID, shardID
 		select {
-		case out <- msg:
+		case out <- &llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_Shard{Shard: s}}:
 		case <-ctx.Done():
 		}
 	}
+	report := func(st supervisor.State, errMsg string, tail []string) {
+		send(&llmgwv1.ShardStatus{
+			State: agentsvc.StateToWire(st), Error: errMsg, LastOutput: tail,
+		})
+	}
 
 	spec, timeout := agentsvc.ShardSpecFromStart(m)
 
@@ -753,10 +1050,23 @@ func (a *Agent) startShard(ctx context.Context, m *llmgwv1.StartShard, out chan<
 	// arrives minutes into a load, in its own words, on a node nobody is
 	// watching — and because eight shards starting together would otherwise
 	// each fetch the same file.
+	//
+	// Progress is reported upward while it runs, so the control plane can tell
+	// a download from a hang.
+	stop := a.watchFetch(spec.Profile.ModelPath, func(p weights.Progress) {
+		send(&llmgwv1.ShardStatus{
+			State: agentsvc.StateToWire(supervisor.Starting),
+			Fetch: &llmgwv1.FetchProgress{
+				Source: p.Source, BytesDone: p.BytesDone, BytesTotal: p.BytesTotal,
+			},
+		})
+	})
 	if path, err := a.weights().Ensure(ctx, spec.Profile.ModelPath); err != nil {
+		stop()
 		report(supervisor.Failed, err.Error(), nil)
 		return
 	} else {
+		stop()
 		spec.Profile.ModelPath = path
 	}
 
@@ -897,3 +1207,172 @@ func Client(cfg tlsconf.Config, timeout time.Duration) (*http.Client, error) {
 	}
 	return &http.Client{Transport: &http2.Transport{TLSClientConfig: tc}}, nil
 }
+
+// engines is what this node can run: what the agent installed and saw run,
+// and what the configuration names and answers. Nothing merely found on the
+// machine — a llama-server on PATH is somebody's, not necessarily one that
+// works, and placing on it is how a replica fails on a node nobody watches.
+func (a *Agent) engines(ctx context.Context) map[string]string {
+	a.engineOnce.Do(func() {
+		vers := a.reg.ProbeConfigured(ctx)
+		a.mu.Lock()
+		a.configuredVers, a.lastEngineProbe = vers, time.Now()
+		a.mu.Unlock()
+		if len(vers) > 0 {
+			a.log.Info("engines named in the configuration",
+				"engines", engines.ProbedNames(vers))
+		}
+	})
+	a.mu.Lock()
+	defer a.mu.Unlock()
+	vers := a.engineReportLocked()
+	// Whatever registration sends is what the control plane now knows, so a
+	// heartbeat repeats it only once it differs.
+	a.enginesSent = enginesFingerprint(vers)
+	return vers
+}
+
+// engineReportLocked merges the two sources. Installed wins: it is the one
+// that was seen to run. Called with mu held.
+func (a *Agent) engineReportLocked() map[string]string {
+	out := maps.Clone(a.configuredVers)
+	if out == nil {
+		out = map[string]string{}
+	}
+	maps.Copy(out, a.installedVers)
+	return out
+}
+
+// provisionEvery is how often installed engines are checked again. A library
+// removed by an unrelated upgrade is found here rather than by the next
+// replica to start.
+const provisionEvery = 10 * time.Minute
+
+// setProvision takes the control plane's list of wanted engines.
+func (a *Agent) setProvision(p *llmgwv1.Provision) {
+	wants := []provision.Want{}
+	for _, e := range p.GetEngines() {
+		wants = append(wants, provision.Want{Engine: e.GetEngine(), Version: e.GetVersion()})
+	}
+	a.mu.Lock()
+	a.provWants = wants
+	a.mu.Unlock()
+	select {
+	case a.provKick <- struct{}{}:
+	default:
+	}
+}
+
+// provisionLoop installs and re-checks the wanted engines, one pass at a time.
+func (a *Agent) provisionLoop(ctx context.Context) {
+	t := time.NewTicker(provisionEvery)
+	defer t.Stop()
+	for {
+		select {
+		case <-ctx.Done():
+			return
+		case <-a.provKick:
+		case <-t.C:
+		}
+		a.mu.Lock()
+		wants := slices.Clone(a.provWants)
+		managed := a.provWants != nil
+		a.mu.Unlock()
+		if !managed {
+			continue
+		}
+
+		statuses := a.prov.Converge(ctx, wants)
+		installed := map[string]string{}
+		keep := map[string]bool{}
+		for _, st := range statuses {
+			keep[st.Engine] = true
+			if st.State == provision.Ready {
+				a.reg.UseInstalled(st.Engine, st.Binary)
+				installed[st.Engine] = st.Version
+			}
+		}
+		a.mu.Lock()
+		if enginesFingerprint(installed) != enginesFingerprint(a.installedVers) {
+			a.log.Info("engines installed and running", "engines", engines.ProbedNames(installed))
+		}
+		a.installedVers = installed
+		// An engine no longer asked for is no longer reported on.
+		for e := range a.provStatus {
+			if !keep[e] {
+				delete(a.provStatus, e)
+			}
+		}
+		a.mu.Unlock()
+	}
+}
+
+// provisionStatus records a change in one engine's installation.
+func (a *Agent) provisionStatus(st provision.Status) {
+	a.mu.Lock()
+	prev, had := a.provStatus[st.Engine]
+	a.provStatus[st.Engine] = st
+	a.mu.Unlock()
+	if !had || prev.State != st.State || prev.Detail != st.Detail {
+		lvl := slog.LevelInfo
+		if st.State == provision.Failed || st.State == provision.Unsupported {
+			lvl = slog.LevelWarn
+		}
+		a.log.Log(context.Background(), lvl, "engine installation",
+			"engine", st.Engine, "version", st.Version, "variant", st.Variant,
+			"state", st.State, "detail", st.Detail)
+	}
+}
+
+func provisionFingerprint(m map[string]provision.Status) string {
+	var b strings.Builder
+	for _, k := range slices.Sorted(maps.Keys(m)) {
+		st := m[k]
+		fmt.Fprintf(&b, "%s|%s|%s|%s|%s\n", st.Engine, st.Version, st.Variant, st.State, st.Detail)
+	}
+	return b.String()
+}
+
+// engineReprobe is how often the installed engines are looked for again.
+//
+// Minutes, not seconds: an engine is installed by somebody at a keyboard, so a
+// minute's delay in noticing is invisible, while SGLang's probe starts a
+// Python interpreter that imports it.
+const engineReprobe = time.Minute
+
+// reprobeEngines looks for installed engines again when it is time, off the
+// heartbeat goroutine so a slow probe never delays a heartbeat.
+func (a *Agent) reprobeEngines() {
+	a.mu.Lock()
+	due := !a.probingEngines && !a.lastEngineProbe.IsZero() &&
+		time.Since(a.lastEngineProbe) >= engineReprobe
+	if due {
+		a.probingEngines = true
+	}
+	a.mu.Unlock()
+	if !due {
+		return
+	}
+	go func() {
+		ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
+		defer cancel()
+		vers := a.reg.ProbeConfigured(ctx)
+
+		a.mu.Lock()
+		changed := enginesFingerprint(vers) != enginesFingerprint(a.configuredVers)
+		a.configuredVers, a.lastEngineProbe, a.probingEngines = vers, time.Now(), false
+		a.mu.Unlock()
+		if changed {
+			a.log.Info("engines named in the configuration changed",
+				"engines", engines.ProbedNames(vers))
+		}
+	}()
+}
+
+func enginesFingerprint(vers map[string]string) string {
+	var b strings.Builder
+	for _, k := range slices.Sorted(maps.Keys(vers)) {
+		fmt.Fprintf(&b, "%s=%s\n", k, vers[k])
+	}
+	return b.String()
+}

internal/agent/agent_test.go

@@ -38,6 +38,7 @@ type statusRecorder struct {
 	calls   []adoptCall
 	adopted []reconciler.AdoptedShard
 	unclaim []string // shard ids to report back as unclaimed
+	fetches []fetchReport
 }
 
 type status struct {
@@ -55,6 +56,25 @@ func (r *statusRecorder) OnShardStatus(replicaID, shardID string, st supervisor.
 	}
 }
 
+// fetches records download progress, so a test can assert that a fetch is
+// visible from the control plane rather than only in the node's log.
+func (r *statusRecorder) OnFetchProgress(replicaID, shardID, source string, done, total int64) {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	r.fetches = append(r.fetches, fetchReport{replicaID, shardID, source, done, total})
+}
+
+func (r *statusRecorder) fetchReports() []fetchReport {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	return append([]fetchReport(nil), r.fetches...)
+}
+
+type fetchReport struct {
+	replicaID, shardID, source string
+	done, total                int64
+}
+
 func (r *statusRecorder) Adopt(nodeID string, shards []reconciler.AdoptedShard) []reconciler.AdoptedShard {
 	r.mu.Lock()
 	defer r.mu.Unlock()

internal/agent/discovery/discover.go

@@ -24,7 +24,8 @@ type Options struct {
 	// PreferNVML uses the NVML backend when this binary has one. It is the only
 	// source of NVLink topology.
 	PreferNVML bool `koanf:"prefer_nvml"`
-	// SkipGPUs omits GPU discovery, for nodes that have none.
+	// SkipGPUs omits GPU discovery. A node without nvidia-smi is already
+	// found to have none, so this is for one whose driver is broken.
 	SkipGPUs bool `koanf:"skip_gpus"`
 	// ModelDirs are scanned for weights. Empty means do not look.
 	ModelDirs []string `koanf:"-"`

internal/agent/discovery/gpu_smi.go

@@ -44,6 +44,13 @@ const mandatorySMIFields = 10
 // NVML build adds.
 func DiscoverGPUsSMI(ctx context.Context, binary string) ([]GPU, error) {
 	if binary == "" {
+		// Not configured and not installed is a node without an NVIDIA card,
+		// not a failed discovery. Reported as an error it also cut discovery
+		// short before InfiniBand was looked at, on every CPU node that had
+		// not been told by hand to skip GPUs.
+		if _, err := exec.LookPath("nvidia-smi"); err != nil {
+			return nil, nil
+		}
 		binary = "nvidia-smi"
 	}
 	out, err := exec.CommandContext(ctx, binary,

internal/agent/discovery/host.go

@@ -0,0 +1,154 @@
+package discovery
+
+import (
+	"bufio"
+	"os"
+	"runtime"
+	"strings"
+)
+
+// Host is what a node is, as far as installing software on it is concerned.
+type Host struct {
+	// OSID, OSVersion and OSCodename come from /etc/os-release: "ubuntu",
+	// "24.04", "noble".
+	OSID       string `json:"os_id,omitempty"`
+	OSVersion  string `json:"os_version,omitempty"`
+	OSCodename string `json:"os_codename,omitempty"`
+	// OSLike is ID_LIKE: the families this system says it follows, "debian"
+	// for Ubuntu, "rhel centos fedora" for Rocky. It is what decides the
+	// package manager, which the name alone does not for every derivative.
+	OSLike []string `json:"os_like,omitempty"`
+	// Arch is Debian's spelling, "amd64" or "arm64", because that is what
+	// package names and release archives are keyed by.
+	Arch string `json:"arch,omitempty"`
+	// CPUFlags are the vector extensions inference cares about, of those this
+	// processor has.
+	CPUFlags []string `json:"cpu_flags,omitempty"`
+	// NVIDIADriver is the loaded driver's version, empty when there is none.
+	NVIDIADriver string `json:"nvidia_driver,omitempty"`
+}
+
+// Family is the packaging family: "deb", "rpm", or "" when neither is known.
+func (h Host) Family() string {
+	for _, id := range append([]string{h.OSID}, h.OSLike...) {
+		switch id {
+		case "debian", "ubuntu":
+			return "deb"
+		case "rhel", "fedora", "centos", "rocky", "almalinux", "suse", "opensuse":
+			return "rpm"
+		}
+	}
+	return ""
+}
+
+// HasCPUFlag reports whether the processor has an extension.
+func (h Host) HasCPUFlag(f string) bool {
+	for _, have := range h.CPUFlags {
+		if have == f {
+			return true
+		}
+	}
+	return false
+}
+
+// interestingFlags are the extensions that decide which build of an engine
+// runs well. A VM on QEMU's default CPU model has none of them — and still
+// runs, five times slower, with nothing saying why.
+var interestingFlags = []string{"avx", "avx2", "fma", "f16c", "avx512f", "avx512_bf16", "amx_tile", "asimd", "sve"}
+
+// DiscoverHost reads what the machine is.
+func DiscoverHost() Host {
+	h := hostFromOSRelease("/etc/os-release")
+	h.Arch = debianArch(runtime.GOARCH)
+	h.CPUFlags = cpuFlagsFrom("/proc/cpuinfo")
+	h.NVIDIADriver = nvidiaDriverFrom("/proc/driver/nvidia/version")
+	return h
+}
+
+func hostFromOSRelease(path string) Host {
+	var h Host
+	f, err := os.Open(path) //nolint:gosec // a fixed system path
+	if err != nil {
+		return h
+	}
+	defer f.Close()
+	sc := bufio.NewScanner(f)
+	for sc.Scan() {
+		k, v, ok := strings.Cut(sc.Text(), "=")
+		if !ok {
+			continue
+		}
+		v = strings.Trim(v, `"'`)
+		switch k {
+		case "ID":
+			h.OSID = v
+		case "VERSION_ID":
+			h.OSVersion = v
+		case "VERSION_CODENAME":
+			h.OSCodename = v
+		case "ID_LIKE":
+			h.OSLike = strings.Fields(v)
+		}
+	}
+	return h
+}
+
+func debianArch(goarch string) string {
+	switch goarch {
+	case "amd64", "arm64":
+		return goarch
+	case "386":
+		return "i386"
+	case "arm":
+		return "armhf"
+	}
+	return goarch
+}
+
+func cpuFlagsFrom(path string) []string {
+	b, err := os.ReadFile(path) //nolint:gosec // a fixed system path
+	if err != nil {
+		return nil
+	}
+	have := map[string]bool{}
+	for _, line := range strings.Split(string(b), "\n") {
+		k, v, ok := strings.Cut(line, ":")
+		if !ok {
+			continue
+		}
+		// "flags" on x86, "Features" on ARM. The first processor is enough:
+		// a machine does not mix instruction sets across its cores.
+		if k = strings.TrimSpace(k); k != "flags" && k != "Features" {
+			continue
+		}
+		for _, f := range strings.Fields(v) {
+			have[f] = true
+		}
+		break
+	}
+	var out []string
+	for _, f := range interestingFlags {
+		if have[f] {
+			out = append(out, f)
+		}
+	}
+	return out
+}
+
+// nvidiaDriverFrom reads the loaded kernel module's version, which exists
+// exactly when a driver is loaded — unlike nvidia-smi, which can be installed
+// on a machine whose module failed to load.
+func nvidiaDriverFrom(path string) string {
+	b, err := os.ReadFile(path) //nolint:gosec // a fixed system path
+	if err != nil {
+		return ""
+	}
+	// "NVRM version: NVIDIA UNIX x86_64 Kernel Module  570.86.15  Thu ..."
+	fields := strings.Fields(string(b))
+	for i, f := range fields {
+		if f == "Module" && i+1 < len(fields) {
+			return fields[i+1]
+		}
+	}
+	return ""
+}

internal/agent/engines/engine.go

@@ -72,6 +72,11 @@ type ShardSpec struct {
 	// not used when building a command.
 	Pool   string
 	Fabric string
+	// ProfileFP identifies the version of the profile this shard was started
+	// from. Carried for the same reason as Pool and Fabric, and never read: it
+	// lets the control plane tell, after its own restart, whether a shard that
+	// is still running matches the profile the catalog holds now.
+	ProfileFP string
 }
 
 // Leader reports whether this shard serves HTTP for the replica.
@@ -146,6 +151,16 @@ func NewRegistry(cfg Config) *Registry {
 	return r
 }
 
+// UseInstalled points an engine at a build the agent installed, and reports
+// whether the engine takes one.
+func (r *Registry) UseInstalled(name, path string) bool {
+	u, ok := r.engines[name].(interface{ UseInstalled(string) })
+	if ok {
+		u.UseInstalled(path)
+	}
+	return ok
+}
+
 // Add registers an engine, replacing any engine of the same name.
 func (r *Registry) Add(e Engine) { r.engines[e.Name()] = e }
 

internal/agent/engines/llamacpp.go

@@ -2,14 +2,20 @@ package engines
 
 import (
 	"fmt"
+	"os"
+	"path/filepath"
+	"sort"
 	"strconv"
 	"strings"
+	"sync/atomic"
 )
 
 // LlamaCppConfig locates the llama.cpp installation.
 type LlamaCppConfig struct {
 	// Bin is the llama-server binary. It is the server, not the CLI: the
 	// gateway talks OpenAI over HTTP and needs the one that serves it.
+	// Empty means the agent installs llama.cpp itself, at the version the
+	// control plane asks for. Setting it opts out of that.
 	Bin string `koanf:"bin"`
 	// Parallel is how many requests a server handles at once, when a profile
 	// does not say. llama.cpp defaults to one, which would make every second
@@ -23,7 +29,10 @@ type LlamaCppConfig struct {
 // small model on CPU at a speed that is fine for testing the whole path, and it
 // serves the same OpenAI API as the others, so nothing downstream can tell the
 // difference.
-type LlamaCpp struct{ cfg LlamaCppConfig }
+type LlamaCpp struct {
+	cfg       LlamaCppConfig
+	installed atomic.Pointer[string]
+}
 
 // Name identifies the engine in the catalog.
 func (*LlamaCpp) Name() string { return "llamacpp" }
@@ -65,14 +74,15 @@ func (*LlamaCpp) Validate(p Profile, gpus []GPUInfo) error {
 
 // Commands returns the processes for one shard, which is always one process.
 func (l *LlamaCpp) Commands(s ShardSpec) ([]Command, error) {
-	bin := l.cfg.Bin
-	if bin == "" {
-		bin = "llama-server"
-	}
+	bin := l.bin()
 	p := s.Profile
 
+	model, err := ggufIn(p.ModelPath)
+	if err != nil {
+		return nil, err
+	}
 	args := []string{
-		"--model", p.ModelPath,
+		"--model", model,
 		"--host", "0.0.0.0",
 		"--port", strconv.Itoa(s.HTTPPort),
 		// Without this there is no /metrics, and an endpoint the router cannot
@@ -127,3 +137,61 @@ func hasFlag(args []string, names ...string) bool {
 	}
 	return false
 }
+
+// ggufIn resolves a model path that may be a directory.
+//
+// A multi-file version arrives as the directory it was downloaded into, because
+// that is the only shape a set can take. llama.cpp wants a file: given the
+// first part of a split GGUF it finds the rest beside it, so the first part is
+// what it gets.
+//
+// A path that is already a file is returned untouched, which is every
+// single-file model and every hand-staged one.
+func ggufIn(path string) (string, error) {
+	st, err := os.Stat(path)
+	if err != nil || !st.IsDir() {
+		// Not a directory, or not there at all — the second is the engine's to
+		// report, with the message it would have given anyway.
+		return path, nil //nolint:nilerr // a missing path is the engine's error to give
+	}
+	entries, err := os.ReadDir(path)
+	if err != nil {
+		return "", err
+	}
+	var names []string
+	for _, e := range entries {
+		if !e.IsDir() && strings.HasSuffix(strings.ToLower(e.Name()), ".gguf") {
+			names = append(names, e.Name())
+		}
+	}
+	if len(names) == 0 {
+		return "", fmt.Errorf("%s holds no .gguf file", path)
+	}
+	// Sorted, so a split model's first part comes first: the parts are named
+	// "-00001-of-00004" and lexical order is shard order.
+	sort.Strings(names)
+	return filepath.Join(path, names[0]), nil
+}
+
+// UseInstalled points the engine at a build the agent installed and checked.
+// It takes precedence over everything else, because it is the only one that
+// was seen to run.
+func (l *LlamaCpp) UseInstalled(path string) { l.installed.Store(&path) }
+
+// Configured reports whether the agent's configuration names a binary. A
+// named binary is the operator's deliberate choice; one merely found on PATH
+// is not, and is not reported as installed.
+func (l *LlamaCpp) Configured() bool { return l.cfg.Bin != "" }
+
+// bin is the llama-server to run: the installed build, else the configured
+// one, else whatever PATH has — which is only ever reached by a node not
+// provisioned by the agent and not told where its engine is.
+func (l *LlamaCpp) bin() string {
+	if p := l.installed.Load(); p != nil {
+		return *p
+	}
+	if l.cfg.Bin != "" {
+		return l.cfg.Bin
+	}
+	return "llama-server"
+}

internal/agent/engines/llamacpp_test.go

@@ -1,6 +1,8 @@
 package engines
 
 import (
+	"os"
+	"path/filepath"
 	"slices"
 	"strings"
 	"testing"
@@ -187,3 +189,59 @@ func TestLlamaCppLetsAProfileTurnTheTemplateOff(t *testing.T) {
 		t.Errorf("the profile said --no-jinja and got both: %v", cmds[0].Args)
 	}
 }
+
+func TestASetDirectoryResolvesToItsFirstPart(t *testing.T) {
+	dir := t.TempDir()
+	// Deliberately out of order on disk, because readdir order is not shard
+	// order and the first part is the one llama.cpp must be given.
+	for _, n := range []string{
+		"qwen-q4_k_m-00003-of-00003.gguf",
+		"qwen-q4_k_m-00001-of-00003.gguf",
+		"qwen-q4_k_m-00002-of-00003.gguf",
+	} {
+		if err := os.WriteFile(filepath.Join(dir, n), []byte("x"), 0o644); err != nil {
+			t.Fatal(err)
+		}
+	}
+
+	got, err := ggufIn(dir)
+	if err != nil {
+		t.Fatal(err)
+	}
+	// Given the first part llama.cpp finds the rest beside it; given the third
+	// it loads a third of a model.
+	if filepath.Base(got) != "qwen-q4_k_m-00001-of-00003.gguf" {
+		t.Errorf("expected the first shard, got %s", filepath.Base(got))
+	}
+}
+
+func TestAPlainFilePathIsUntouched(t *testing.T) {
+	// Every single-file model and every hand-staged one takes this path.
+	p := filepath.Join(t.TempDir(), "model.gguf")
+	if err := os.WriteFile(p, []byte("x"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	if got, _ := ggufIn(p); got != p {
+		t.Errorf("expected the path unchanged, got %s", got)
+	}
+}
+
+func TestADirectoryWithNoWeightsSaysSo(t *testing.T) {
+	dir := t.TempDir()
+	if err := os.WriteFile(filepath.Join(dir, "config.json"), []byte("{}"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	if _, err := ggufIn(dir); err == nil {
+		t.Error("a directory with no .gguf in it is not a model llama.cpp can load")
+	}
+}
+
+func TestAMissingPathIsLeftToTheEngine(t *testing.T) {
+	// The engine's own "no such file" names the path and the flag it came
+	// from, which is more use than anything this could say.
+	p := "/nonexistent/model.gguf"
+	got, err := ggufIn(p)
+	if err != nil || got != p {
+		t.Errorf("expected it passed through, got %q %v", got, err)
+	}
+}

internal/agent/engines/probe.go

@@ -0,0 +1,224 @@
+package engines
+
+import (
+	"context"
+	"errors"
+	"os/exec"
+	"regexp"
+	"sort"
+	"strings"
+	"time"
+)
+
+// Unknown is the version reported for an engine that is installed and would
+// not say which version it is.
+//
+// A value, not an absent key, because the two mean different things and
+// placement reads the keys: absent is "this node cannot run it", Unknown is
+// "it is here and the probe could not get a number out of it". Collapsing them
+// would make a slow engine look like a missing one and take a node out of
+// scheduling for being slow to answer --version.
+const Unknown = "unknown"
+
+// Prober is an engine that can say whether it is installed on this machine.
+//
+// Optional, and asserted for rather than part of Engine: an adapter that has no
+// way to check should report nothing rather than be forced to invent an answer.
+type Prober interface {
+	// Probe returns the engine's version, or an error when it is not installed.
+	Probe(ctx context.Context) (string, error)
+}
+
+// Probe asks every registered engine whether it is here.
+//
+// The result is what the node reports at registration, and its key set is what
+// placement filters on. Until this existed nobody knew: the registry registers
+// every adapter whether or not its binary exists, so a node with no vLLM
+// accepted a vLLM shard, registered healthy, and failed the shard minutes later
+// with "executable file not found" — in the engine's own words, on a machine
+// nobody was watching.
+//
+// Bounded, because it runs on the path to registering and an agent that cannot
+// reach the control plane is worse than one that reports an incomplete list.
+func (r *Registry) Probe(ctx context.Context) map[string]string {
+	out := map[string]string{}
+	for name, e := range r.engines {
+		p, ok := e.(Prober)
+		if !ok {
+			continue
+		}
+		v, err := p.Probe(ctx)
+		if err != nil {
+			continue
+		}
+		if v == "" {
+			v = Unknown
+		}
+		out[name] = v
+	}
+	return out
+}
+
+// Names of the engines in a probe result, sorted.
+func ProbedNames(m map[string]string) []string {
+	out := make([]string, 0, len(m))
+	for n := range m {
+		out = append(out, n)
+	}
+	sort.Strings(out)
+	return out
+}
+
+// probeTimeout bounds one engine's version check.
+//
+// Generous, because vLLM's --version imports torch and that is seconds on a
+// cold page cache, and mean, because this is startup: an engine that has not
+// answered in this long is reported as present-with-unknown-version rather
+// than waited for.
+const probeTimeout = 8 * time.Second
+
+// run executes a version command, saying whether the program was even there.
+//
+// The two failures are different answers and the caller has to tell them
+// apart. A binary that is absent is an engine this node cannot run. A binary
+// that is present and exits non-zero is an engine it probably can — unless the
+// command was a test of whether a module imports, where a non-zero exit is the
+// whole answer.
+func run(ctx context.Context, bin string, args ...string) (out string, found bool, ok bool) {
+	path, err := exec.LookPath(bin)
+	if err != nil {
+		return "", false, false
+	}
+	ctx, cancel := context.WithTimeout(ctx, probeTimeout)
+	defer cancel()
+
+	// Combined, not just stdout: engines print their version to either, and one
+	// that chose stderr would otherwise read as having no version at all.
+	b, err := exec.CommandContext(ctx, path, args...).CombinedOutput()
+	return string(b), true, err == nil
+}
+
+// installedBinary reports an engine that is its own executable.
+//
+// It resolves the path and runs nothing. The first version of this ran the
+// engine with --version, which is fine for an engine and not fine for the
+// wrapper scripts people put in front of one: a script that shifts past flags
+// it does not recognise and then execs the server will start an inference
+// server, at agent startup, on every node. The agent's own test suite installs
+// exactly such a script, and caught it.
+//
+// So the version is not read here. Presence is what placement needs, and what
+// the version is for — telling whether an engine still accepts the flags llmgw
+// builds — is `llmgw ctl engine-check`, which is run deliberately by somebody
+// who meant to run the engine.
+func installedBinary(bin string) (string, error) {
+	if _, err := exec.LookPath(bin); err != nil {
+		return "", errNotInstalled
+	}
+	return Unknown, nil
+}
+
+// versionOfModule probes an engine that is a module inside an interpreter.
+//
+// This one does execute, and safely: the command is llmgw's own one-line
+// import, not the engine's launcher, so there is no wrapper to shift past
+// flags and start a server. There is also no alternative — a module has no
+// path to look up.
+//
+// A non-zero exit is the answer: python3 exists on almost every machine, so
+// presence of the interpreter says nothing. Treating a failed import as
+// "installed" reported SGLang on every node in a fleet that had none, which
+// would take nodes into scheduling for an engine they cannot run — the exact
+// failure this whole change exists to stop.
+func versionOfModule(ctx context.Context, python string, args ...string) (string, error) {
+	out, found, ok := run(ctx, python, args...)
+	if !found || !ok {
+		return "", errNotInstalled
+	}
+	return firstVersion(out), nil
+}
+
+var errNotInstalled = errors.New("not installed on this node")
+
+var versionPattern = regexp.MustCompile(`\b\d+\.\d+(\.\d+)?\S*`)
+
+// firstVersion is the first thing in the output that looks like a version.
+//
+// Pattern rather than parsing each engine's format: they print "vllm 0.6.3",
+// "version: b4123 (…)", and a paragraph with the number in the middle, and all
+// of those change between releases. What this is for is telling one build from
+// another, which the first number does.
+func firstVersion(s string) string {
+	for _, line := range strings.Split(s, "\n") {
+		if m := versionPattern.FindString(line); m != "" {
+			return strings.TrimRight(m, ",;)")
+		}
+	}
+	return Unknown
+}
+
+// Configured is an engine whose location the agent's configuration names.
+type Configured interface {
+	Configured() bool
+}
+
+// ProbeConfigured probes only the engines the configuration names.
+//
+// Probe looks everywhere — PATH, the default interpreter — and reports what
+// it finds, which is exactly the trust in whatever happens to be on a machine
+// that installing engines replaces. A node the agent provisions reports what
+// it installed, plus what the operator named on purpose; nothing else.
+func (r *Registry) ProbeConfigured(ctx context.Context) map[string]string {
+	out := map[string]string{}
+	for name, e := range r.engines {
+		c, ok := e.(Configured)
+		if !ok || !c.Configured() {
+			continue
+		}
+		p, ok := e.(Prober)
+		if !ok {
+			continue
+		}
+		if v, err := p.Probe(ctx); err == nil {
+			if v == "" {
+				v = Unknown
+			}
+			out[name] = v
+		}
+	}
+	return out
+}
+
+// Probe reports whether the llama.cpp server is installed.
+func (l *LlamaCpp) Probe(context.Context) (string, error) {
+	return installedBinary(l.bin())
+}
+
+// Configured reports whether the configuration names vLLM.
+func (v *VLLM) Configured() bool { return v.cfg.Bin != "" }
+
+// Configured reports whether the configuration names SGLang's interpreter.
+func (g *SGLang) Configured() bool { return g.cfg.Python != "" }
+
+// Probe reports whether vLLM is installed.
+func (v *VLLM) Probe(context.Context) (string, error) {
+	bin := v.cfg.Bin
+	if bin == "" {
+		bin = "vllm"
+	}
+	return installedBinary(bin)
+}
+
+// Probe reports the SGLang version.
+//
+// Through the interpreter, because SGLang is a module rather than a binary:
+// asking the shell for "sglang" finds nothing on a machine that can run it
+// perfectly well.
+func (g *SGLang) Probe(ctx context.Context) (string, error) {
+	python := g.cfg.Python
+	if python == "" {
+		python = "python3"
+	}
+	return versionOfModule(ctx, python, "-c",
+		"import sglang, sys; sys.stdout.write(getattr(sglang, '__version__', ''))")
+}

internal/agent/engines/probe_test.go

@@ -0,0 +1,107 @@
+package engines
+
+import (
+	"context"
+	"os"
+	"path/filepath"
+	"testing"
+)
+
+// script writes an executable that stands in for an engine.
+func script(t *testing.T, name, body string) string {
+	t.Helper()
+	dir := t.TempDir()
+	p := filepath.Join(dir, name)
+	if err := os.WriteFile(p, []byte("#!/bin/sh\n"+body+"\n"), 0o755); err != nil {
+		t.Fatal(err)
+	}
+	return p
+}
+
+func TestAnEngineThatIsNotInstalledIsAbsentFromTheReport(t *testing.T) {
+	r := NewRegistry(Config{})
+	r.engines = map[string]Engine{
+		"vllm": &VLLM{cfg: VLLMConfig{Bin: "/nonexistent/vllm"}},
+	}
+	// Absent, not present-with-unknown-version. Placement reads the keys, and
+	// a node listed for an engine it cannot run is the failure being fixed.
+	if got := r.Probe(t.Context()); len(got) != 0 {
+		t.Errorf("expected nothing, got %v", got)
+	}
+}
+
+func TestProbingNeverRunsTheEngine(t *testing.T) {
+	// A wrapper that shifts past flags it does not recognise and then starts
+	// the server. People put these in front of engines, and the agent's own
+	// tests install one. Probing with --version would launch an inference
+	// server at agent startup, on every node.
+	marker := filepath.Join(t.TempDir(), "ran")
+	bin := script(t, "fake-vllm", "touch "+marker+"; sleep 60")
+
+	r := NewRegistry(Config{})
+	r.engines = map[string]Engine{"vllm": &VLLM{cfg: VLLMConfig{Bin: bin}}}
+
+	got := r.Probe(t.Context())
+	if _, ok := got["vllm"]; !ok {
+		t.Error("a binary that is there should be reported present")
+	}
+	if _, err := os.Stat(marker); err == nil {
+		t.Fatal("the engine was executed by a probe")
+	}
+}
+
+func TestAModuleThatWillNotImportIsNotInstalled(t *testing.T) {
+	// python3 is on almost every machine, so the interpreter being there says
+	// nothing about SGLang. Reporting it present put every node in a fleet
+	// forward for an engine none of them had.
+	bin := script(t, "python3", "exit 1")
+	r := NewRegistry(Config{})
+	r.engines = map[string]Engine{"sglang": &SGLang{cfg: SGLangConfig{Python: bin}}}
+
+	if got := r.Probe(t.Context()); len(got) != 0 {
+		t.Errorf("a failed import means not installed, got %v", got)
+	}
+}
+
+func TestAModuleThatImportsIsReportedWithItsVersion(t *testing.T) {
+	bin := script(t, "python3", `echo "0.4.1"`)
+	r := NewRegistry(Config{})
+	r.engines = map[string]Engine{"sglang": &SGLang{cfg: SGLangConfig{Python: bin}}}
+
+	got := r.Probe(t.Context())
+	if got["sglang"] != "0.4.1" {
+		t.Errorf("expected the version, got %v", got)
+	}
+}
+
+func TestAVersionIsPickedOutOfWhateverIsPrinted(t *testing.T) {
+	// Only the module probe reaches this now, and a Python package version is
+	// dotted. A bare build number is deliberately not matched: it would turn a
+	// stray number in a line of prose into a version.
+	for in, want := range map[string]string{
+		"0.4.1":                 "0.4.1",
+		"sglang 0.4.post1":      "0.4.post1",
+		"warning: noise\n0.4.1": "0.4.1",
+		"":                      Unknown,
+		"no numbers here":       Unknown,
+		"just 4123":             Unknown,
+	} {
+		if got := firstVersion(in); got != want {
+			t.Errorf("firstVersion(%q) = %q, want %q", in, got, want)
+		}
+	}
+}
+
+func TestProbingIsBoundedByTheContext(t *testing.T) {
+	ctx, cancel := context.WithCancel(context.Background())
+	cancel()
+	bin := script(t, "python3", "sleep 60")
+	r := NewRegistry(Config{})
+	r.engines = map[string]Engine{"sglang": &SGLang{cfg: SGLangConfig{Python: bin}}}
+
+	// Registration waits on this. An agent that cannot reach the control plane
+	// is worse than one reporting an incomplete list.
+	if got := r.Probe(ctx); len(got) != 0 {
+		t.Errorf("a cancelled probe should report nothing, got %v", got)
+	}
+}

internal/agent/failover_test.go

@@ -0,0 +1,107 @@
+package agent
+
+import (
+	"context"
+	"log/slog"
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"testing"
+	"time"
+
+	"golang.org/x/net/http2"
+	"golang.org/x/net/http2/h2c"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/agentsvc"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
+)
+
+type standby bool
+
+func (s standby) IsLeader() bool { return !bool(s) }
+
+// startPlain runs an agent service over h2c, which is enough to test who
+// accepts a connection and who refuses it.
+//
+// A nil leader means none was configured, which is every deployment that
+// existed before leadership did and has to keep accepting agents.
+func startPlain(t *testing.T, leader agentsvc.Leader) (*inventory.Store, string) {
+	t.Helper()
+	inv := inventory.NewStore()
+	svc := agentsvc.New(inv, newRecorder(), false, slog.New(slog.DiscardHandler))
+	if leader != nil {
+		svc.SetLeadership(leader)
+	}
+
+	path, handler := svc.Handler()
+	mux := http.NewServeMux()
+	mux.Handle(path, handler)
+	// h2c, because the agent speaks HTTP/2 with prior knowledge and a plain
+	// server negotiates HTTP/1.1 — which turns a clear refusal into a frame
+	// parsing error that says nothing about why.
+	srv := httptest.NewServer(h2c.NewHandler(mux, &http2.Server{}))
+	t.Cleanup(srv.Close)
+	return inv, srv.URL
+}
+
+func TestAStandbyRefusesAnAgent(t *testing.T) {
+	// Commands travel down the stream the agent opened, so whichever control
+	// plane holds it is the only one that can start anything on that node. A
+	// standby holding it means the leader's placements fail with "no agent
+	// connected" for a node that is plainly connected.
+	_, addr := startPlain(t, standby(true))
+
+	ag := runAgent(t, Config{NodeID: "n1", Server: addr, MgmtAddr: "127.0.0.1"})
+	err := ag.session(context.Background(), ag.cfg.Server)
+	if err == nil {
+		t.Fatal("a standby accepted an agent")
+	}
+	if !strings.Contains(err.Error(), "standby") {
+		t.Errorf("the refusal does not say why: %v", err)
+	}
+}
+
+func TestAnAgentReachesTheLeaderPastAStandby(t *testing.T) {
+	// The point of the list. The standby is first, so the agent only registers
+	// if it moved on rather than sitting on the refusal.
+	_, standbyAddr := startPlain(t, standby(true))
+	leaderInv, leaderAddr := startPlain(t, standby(false))
+
+	ag := runAgent(t, Config{
+		NodeID: "n1", MgmtAddr: "127.0.0.1",
+		Servers: []string{standbyAddr, leaderAddr},
+	})
+
+	ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
+	defer cancel()
+	go ag.Run(ctx)
+
+	deadline := time.Now().Add(8 * time.Second)
+	for time.Now().Before(deadline) {
+		if _, ok := leaderInv.Get("n1"); ok {
+			return
+		}
+		time.Sleep(50 * time.Millisecond)
+	}
+	t.Fatal("the agent never reached the leader past the standby")
+}
+
+func TestALeaderStillAcceptsWithNoLeadershipConfigured(t *testing.T) {
+	// Every deployment before this had no leadership at all, and must keep
+	// accepting agents with nothing configured.
+	inv, addr := startPlain(t, nil)
+
+	ag := runAgent(t, Config{NodeID: "n1", Server: addr, MgmtAddr: "127.0.0.1"})
+	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+	defer cancel()
+	go ag.Run(ctx)
+
+	deadline := time.Now().Add(4 * time.Second)
+	for time.Now().Before(deadline) {
+		if _, ok := inv.Get("n1"); ok {
+			return
+		}
+		time.Sleep(50 * time.Millisecond)
+	}
+	t.Fatal("a service with no leadership configured refused an agent")
+}

internal/agent/mtls_test.go

@@ -255,7 +255,7 @@ func TestRefusedAgentLearnsWhy(t *testing.T) {
 		t.Fatal(err)
 	}
 
-	err = ag.session(context.Background())
+	err = ag.session(context.Background(), ag.cfg.Server)
 	if err == nil {
 		t.Fatal("a refused session reported success")
 	}

internal/agent/provision/install.go

@@ -0,0 +1,614 @@
+package provision
+
+import (
+	"archive/tar"
+	"compress/gzip"
+	"context"
+	"crypto/sha256"
+	"encoding/hex"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"io/fs"
+	"log/slog"
+	"net/http"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"strings"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
+)
+
+// Want is one engine at one version.
+type Want struct {
+	Engine  string
+	Version string
+}
+
+// The states an engine's installation can be in.
+const (
+	Installing  = "installing"
+	Ready       = "ready"
+	Failed      = "failed"
+	Unsupported = "unsupported"
+)
+
+// Status is where one engine's installation stands.
+type Status struct {
+	Engine  string
+	Version string
+	Variant string
+	State   string
+	// Detail is for a person: what is happening, or what went wrong.
+	Detail string
+	// Binary is the executable, once ready.
+	Binary string
+}
+
+// Installer installs engines under one directory.
+type Installer struct {
+	// Root is where engines are installed, one directory per build:
+	// <root>/<engine>/<version>-<variant>.
+	Root string
+	Host discovery.Host
+	Log  *slog.Logger
+	HTTP *http.Client
+	// OnStatus hears every change, so a download that takes minutes is seen
+	// to be under way rather than silent.
+	OnStatus func(Status)
+
+	// Run executes a command; a seam for tests. Nil runs it for real.
+	Run func(ctx context.Context, env []string, name string, args ...string) ([]byte, error)
+	// APIBase and DownloadBase default to GitHub; tests point them at a fake.
+	APIBase      string
+	DownloadBase string
+}
+
+// DefaultRoot is where engines are installed when nothing says otherwise.
+const DefaultRoot = "/opt/llmgw/engines"
+
+// marker is written into a build's directory once it has been installed and
+// seen to run. Its absence means the directory is not to be trusted.
+const marker = ".llmgw-installed"
+
+type markerFile struct {
+	Engine    string            `json:"engine"`
+	Version   string            `json:"version"`
+	Variant   string            `json:"variant"`
+	Binary    string            `json:"binary"`
+	SHA256    map[string]string `json:"sha256"`
+	Installed time.Time         `json:"installed"`
+}
+
+// Converge brings every wanted engine to ready, or says why it cannot.
+//
+// An engine already installed is checked again rather than believed: its
+// directory can be damaged, or a library it needs removed by somebody's
+// upgrade, and that is found here rather than by the next replica to fail.
+func (in *Installer) Converge(ctx context.Context, wants []Want) []Status {
+	out := make([]Status, 0, len(wants))
+	for _, w := range wants {
+		st := in.one(ctx, w)
+		in.report(st)
+		out = append(out, st)
+	}
+	return out
+}
+
+func (in *Installer) one(ctx context.Context, w Want) Status {
+	st := Status{Engine: w.Engine, Version: w.Version}
+	recipe, ok := Recipes[w.Engine]
+	if !ok {
+		st.State = Unsupported
+		st.Detail = "this agent does not know how to install " + w.Engine +
+			"; install it by hand and name it in the agent's configuration"
+		return st
+	}
+	build, err := recipe.Choose(w.Version, in.Host)
+	if err != nil {
+		st.State, st.Detail = Unsupported, err.Error()
+		return st
+	}
+	st.Variant = build.Variant
+	dir := filepath.Join(in.root(), w.Engine, w.Version+"-"+build.Variant)
+
+	// Installed before: check it still runs, and trust nothing else about it.
+	if _, err := os.Stat(filepath.Join(dir, marker)); err == nil {
+		// The packages are checked too. Removing one is exactly the kind of
+		// change an unrelated upgrade makes, and ldd below would catch it only
+		// as a missing library with no hint of which package held it.
+		if err := in.packages(ctx, build.Packages, &st); err != nil {
+			st.State, st.Detail = Failed, err.Error()
+			return st
+		}
+		bin, version, err := in.verify(ctx, dir, build)
+		if err == nil {
+			return in.ready(st, build, bin, version)
+		}
+		in.log().Warn("an installed engine no longer runs; installing it again",
+			"engine", w.Engine, "version", w.Version, "err", err)
+	}
+
+	if err := in.install(ctx, recipe, w, build, dir, &st); err != nil {
+		st.State, st.Detail, st.Binary = Failed, err.Error(), ""
+		return st
+	}
+	bin, version, err := in.verify(ctx, dir, build)
+	if err != nil {
+		st.State, st.Detail = Failed, err.Error()
+		return st
+	}
+	return in.ready(st, build, bin, version)
+}
+
+func (in *Installer) ready(st Status, b Build, bin, version string) Status {
+	st.State, st.Binary = Ready, bin
+	st.Detail = strings.TrimSpace(filepath.Base(bin) + " " + version)
+	var missing []string
+	for _, f := range b.CPUFlags {
+		if !in.Host.HasCPUFlag(f) {
+			missing = append(missing, f)
+		}
+	}
+	if len(missing) > 0 {
+		// Said on every report, not once in a log: it is the difference between
+		// five tokens a second and forty, and a VM given QEMU's default CPU
+		// model looks entirely healthy otherwise.
+		st.Detail += "; this CPU lacks " + strings.Join(missing, ", ") +
+			", so it will run several times slower (a VM needs CPU type \"host\")"
+	}
+	return st
+}
+
+// install fetches, checks and unpacks a build into dir.
+//
+// Into a scratch directory renamed into place at the end, so a failure halfway
+// leaves nothing that looks installed.
+func (in *Installer) install(ctx context.Context, r Recipe, w Want, b Build, dir string, st *Status) error {
+	if err := in.packages(ctx, b.Packages, st); err != nil {
+		return err
+	}
+
+	in.progress(st, "asking GitHub for the checksums of %s %s", w.Engine, w.Version)
+	sums, err := in.checksums(ctx, r.Repo, w.Version)
+	if err != nil {
+		return err
+	}
+
+	tmp := dir + ".partial"
+	_ = os.RemoveAll(tmp)
+	if err := os.MkdirAll(tmp, 0o755); err != nil {
+		return err
+	}
+	defer os.RemoveAll(tmp) //nolint:errcheck // gone either way once renamed
+
+	got := map[string]string{}
+	for _, asset := range b.Assets {
+		want, ok := sums[asset]
+		if !ok {
+			// Refused rather than installed unchecked: this is a binary that
+			// will run as root on every node.
+			return fmt.Errorf("%s %s publishes no checksum for %s, so it cannot be verified",
+				w.Engine, w.Version, asset)
+		}
+		archive := filepath.Join(tmp, asset+".download")
+		if err := in.download(ctx, r.Repo, w.Version, asset, archive, want, st); err != nil {
+			return err
+		}
+		in.progress(st, "unpacking %s", asset)
+		if err := untarGz(archive, tmp); err != nil {
+			return fmt.Errorf("unpacking %s: %w", asset, err)
+		}
+		_ = os.Remove(archive)
+		got[asset] = want
+	}
+
+	in.progress(st, "checking that %s runs", b.Binary)
+	bin, _, err := in.verify(ctx, tmp, b)
+	if err != nil {
+		return err
+	}
+	rel, _ := filepath.Rel(tmp, bin)
+	m, _ := json.MarshalIndent(markerFile{
+		Engine: w.Engine, Version: w.Version, Variant: b.Variant,
+		Binary: rel, SHA256: got, Installed: time.Now().UTC(),
+	}, "", "  ")
+	if err := os.WriteFile(filepath.Join(tmp, marker), m, 0o644); err != nil { //nolint:gosec // not secret
+		return err
+	}
+	_ = os.RemoveAll(dir)
+	if err := os.Rename(tmp, dir); err != nil {
+		return err
+	}
+	in.log().Info("installed an engine", "engine", w.Engine, "version", w.Version,
+		"variant", b.Variant, "dir", dir)
+	return nil
+}
+
+// verify finds the binary and runs it.
+//
+// Two checks, because they fail differently. ldd names a missing library
+// outright, where running the binary would only say it could not start. And
+// running it is the only proof that the rest — the right instruction set, the
+// right driver — is in place.
+func (in *Installer) verify(ctx context.Context, dir string, b Build) (bin, version string, err error) {
+	bin, err = findFile(dir, b.Binary)
+	if err != nil {
+		return "", "", err
+	}
+	if out, err := in.run(ctx, nil, "ldd", bin); err == nil {
+		if missing := lddMissing(string(out)); len(missing) > 0 {
+			return "", "", fmt.Errorf("%s cannot run on this system: %s", b.Binary, strings.Join(missing, "; "))
+		}
+	}
+	tctx, cancel := context.WithTimeout(ctx, 60*time.Second)
+	defer cancel()
+	out, err := in.run(tctx, nil, bin, "--version")
+	if err != nil {
+		return "", "", fmt.Errorf("%s --version failed: %v: %s", b.Binary, err, lastLine(out))
+	}
+	return bin, firstVersionLine(out), nil
+}
+
+// packages installs the operating system packages a build needs.
+//
+// Only those missing, and from the node's own package sources: a package the
+// system already has is not touched, so an engine never upgrades something
+// the rest of the machine depends on.
+func (in *Installer) packages(ctx context.Context, byFamily map[string][]string, st *Status) error {
+	family := in.Host.Family()
+	pkgs := byFamily[family]
+	if len(pkgs) == 0 {
+		// A system of no family this knows is not refused: its packages may
+		// well be there, and if they are not, ldd names the missing library.
+		return nil
+	}
+	pm := packageManagers[family]
+	var missing []string
+	for _, p := range pkgs {
+		if _, err := in.run(ctx, nil, pm.query[0], append(pm.query[1:], p)...); err != nil {
+			missing = append(missing, p)
+		}
+	}
+	if len(missing) == 0 {
+		return nil
+	}
+	tool, cmds := pm.install(in, missing)
+	if tool == "" {
+		return fmt.Errorf("%s is missing and no package manager was found to install it",
+			strings.Join(missing, ", "))
+	}
+	in.progress(st, "installing %s with %s", strings.Join(missing, ", "), tool)
+	for _, c := range cmds {
+		if out, err := in.run(ctx, pm.env, c[0], c[1:]...); err != nil {
+			return fmt.Errorf("%s: %v: %s", strings.Join(c, " "), err, lastLine(out))
+		}
+	}
+	return nil
+}
+
+// packageManager is how one packaging family asks and installs.
+type packageManager struct {
+	// query exits zero when the package named after it is installed.
+	query []string
+	env   []string
+	// install picks the tool present on this machine and the commands to run.
+	install func(in *Installer, pkgs []string) (tool string, cmds [][]string)
+}
+
+var packageManagers = map[string]packageManager{
+	"deb": {
+		// dpkg -s exits zero for a package merely known about, so the status
+		// is asked for and checked by the command itself: ${db:Status-Abbrev}
+		// is "ii " only when installed.
+		query: []string{"sh", "-c", `test "$(dpkg-query -W -f='${db:Status-Abbrev}' "$0" 2>/dev/null)" = "ii "`},
+		env:   []string{"DEBIAN_FRONTEND=noninteractive"},
+		install: func(_ *Installer, pkgs []string) (string, [][]string) {
+			// A lock timeout rather than failing on the lock: a machine a
+			// minute old is running cloud-init's and unattended-upgrades' own
+			// apt, and the first attempt on every new node would fail.
+			lock := "-o=DPkg::Lock::Timeout=600"
+			return "apt-get", [][]string{
+				{"apt-get", lock, "update"},
+				append([]string{"apt-get", lock, "install", "-y", "--no-install-recommends"}, pkgs...),
+			}
+		},
+	},
+	"rpm": {
+		query: []string{"rpm", "-q"},
+		install: func(in *Installer, pkgs []string) (string, [][]string) {
+			for _, tool := range []string{"dnf", "yum"} {
+				if in.has(tool) {
+					return tool, [][]string{append([]string{tool, "install", "-y"}, pkgs...)}
+				}
+			}
+			if in.has("zypper") {
+				return "zypper", [][]string{append([]string{"zypper", "--non-interactive", "install"}, pkgs...)}
+			}
+			return "", nil
+		},
+	},
+}
+
+// has reports whether a command is on PATH. Asked through the same seam as
+// every other command, so a test can say which tools a machine has.
+func (in *Installer) has(tool string) bool {
+	_, err := in.run(context.Background(), nil, "sh", "-c", `command -v "$0"`, tool)
+	return err == nil
+}
+
+// checksums reads the sha256 GitHub publishes for each asset of a release.
+func (in *Installer) checksums(ctx context.Context, repo, tag string) (map[string]string, error) {
+	url := strings.TrimSuffix(in.apiBase(), "/") + "/repos/" + repo + "/releases/tags/" + tag
+	req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
+	if err != nil {
+		return nil, err
+	}
+	req.Header.Set("Accept", "application/vnd.github+json")
+	resp, err := in.client().Do(req)
+	if err != nil {
+		return nil, fmt.Errorf("reaching GitHub for %s %s: %w", repo, tag, err)
+	}
+	defer resp.Body.Close()
+	switch {
+	case resp.StatusCode == http.StatusNotFound:
+		return nil, fmt.Errorf("%s has no release %s", repo, tag)
+	case resp.StatusCode == http.StatusForbidden || resp.StatusCode == http.StatusTooManyRequests:
+		// Unauthenticated callers get sixty requests an hour per address, and
+		// a whole fleet behind one NAT shares them.
+		return nil, fmt.Errorf("GitHub is rate-limiting this address (%s); it retries later", resp.Status)
+	case resp.StatusCode != http.StatusOK:
+		return nil, fmt.Errorf("GitHub answered %s for %s %s", resp.Status, repo, tag)
+	}
+	var rel struct {
+		Assets []struct {
+			Name   string `json:"name"`
+			Digest string `json:"digest"`
+		} `json:"assets"`
+	}
+	if err := json.NewDecoder(resp.Body).Decode(&rel); err != nil {
+		return nil, fmt.Errorf("reading %s %s from GitHub: %w", repo, tag, err)
+	}
+	out := map[string]string{}
+	for _, a := range rel.Assets {
+		if sum, ok := strings.CutPrefix(a.Digest, "sha256:"); ok && len(sum) == 64 {
+			out[a.Name] = sum
+		}
+	}
+	return out, nil
+}
+
+func (in *Installer) download(ctx context.Context, repo, tag, asset, dst, want string, st *Status) error {
+	url := strings.TrimSuffix(in.downloadBase(), "/") + "/" + repo + "/releases/download/" + tag + "/" + asset
+	req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
+	if err != nil {
+		return err
+	}
+	resp, err := in.client().Do(req)
+	if err != nil {
+		return fmt.Errorf("downloading %s: %w", asset, err)
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode != http.StatusOK {
+		return fmt.Errorf("downloading %s: %s", asset, resp.Status)
+	}
+	if resp.ContentLength > 0 {
+		in.progress(st, "downloading %s (%d MB)", asset, resp.ContentLength>>20)
+	} else {
+		in.progress(st, "downloading %s", asset)
+	}
+	f, err := os.Create(dst) //nolint:gosec // inside the install root
+	if err != nil {
+		return err
+	}
+	h := sha256.New()
+	_, err = io.Copy(io.MultiWriter(f, h), resp.Body)
+	if cerr := f.Close(); err == nil {
+		err = cerr
+	}
+	if err != nil {
+		return fmt.Errorf("downloading %s: %w", asset, err)
+	}
+	if got := hex.EncodeToString(h.Sum(nil)); got != want {
+		return fmt.Errorf("%s does not match the checksum its publisher gave (got %s, expected %s)",
+			asset, got[:12], want[:12])
+	}
+	return nil
+}
+
+// untarGz unpacks an archive under dir, refusing anything that would land
+// outside it. A release archive is somebody else's file, and a path of
+// "../../etc/cron.d/x" in it would otherwise be written as root.
+func untarGz(archive, dir string) error {
+	f, err := os.Open(archive) //nolint:gosec // inside the install root
+	if err != nil {
+		return err
+	}
+	defer f.Close()
+	gz, err := gzip.NewReader(f)
+	if err != nil {
+		return err
+	}
+	tr := tar.NewReader(gz)
+	for {
+		h, err := tr.Next()
+		if errors.Is(err, io.EOF) {
+			return nil
+		}
+		if err != nil {
+			return err
+		}
+		target, err := within(dir, h.Name)
+		if err != nil {
+			return err
+		}
+		switch h.Typeflag {
+		case tar.TypeDir:
+			if err := os.MkdirAll(target, 0o755); err != nil {
+				return err
+			}
+		case tar.TypeReg:
+			if err := os.MkdirAll(filepath.Dir(target), 0o755); err != nil {
+				return err
+			}
+			out, err := os.OpenFile(target, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, fs.FileMode(h.Mode)&0o755) //nolint:gosec // checked above
+			if err != nil {
+				return err
+			}
+			if _, err := io.Copy(out, tr); err != nil { //nolint:gosec // a release archive, size bounded by its checksum
+				out.Close()
+				return err
+			}
+			if err := out.Close(); err != nil {
+				return err
+			}
+		case tar.TypeSymlink:
+			// Libraries are versioned through symlinks, and those are
+			// relative. One pointing anywhere else is refused.
+			if filepath.IsAbs(h.Linkname) {
+				return fmt.Errorf("%s links outside the archive", h.Name)
+			}
+			if _, err := within(dir, filepath.Join(filepath.Dir(h.Name), h.Linkname)); err != nil {
+				return fmt.Errorf("%s links outside the archive", h.Name)
+			}
+			if err := os.MkdirAll(filepath.Dir(target), 0o755); err != nil {
+				return err
+			}
+			_ = os.Remove(target)
+			if err := os.Symlink(h.Linkname, target); err != nil {
+				return err
+			}
+		default:
+			// Devices, FIFOs and hard links have no business in a release.
+		}
+	}
+}
+
+func within(dir, name string) (string, error) {
+	target := filepath.Join(dir, name) //nolint:gosec // checked on the next line
+	rel, err := filepath.Rel(dir, target)
+	if err != nil || rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) {
+		return "", fmt.Errorf("%s would unpack outside the install directory", name)
+	}
+	return target, nil
+}
+
+// lddMissing reads what ldd could not satisfy, in two shapes:
+//
+//	libgomp.so.1 => not found
+//	/lib64/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by ...)
+//
+// The second is a system too old for the build — RHEL 9 against a build that
+// wants GCC 12's libstdc++ — and is said as such, rather than as a library
+// that is plainly there being "not found".
+func lddMissing(out string) []string {
+	seen := map[string]bool{}
+	var missing []string
+	for _, line := range strings.Split(out, "\n") {
+		line = strings.TrimSpace(line)
+		if !strings.Contains(line, "not found") {
+			continue
+		}
+		var what string
+		if lib, rest, ok := strings.Cut(line, ": version `"); ok {
+			sym, _, _ := strings.Cut(rest, "'")
+			what = filepath.Base(lib) + " is too old (needs " + sym + ")"
+		} else {
+			what = strings.TrimSpace(strings.SplitN(line, "=>", 2)[0]) + " is not installed"
+		}
+		if !seen[what] {
+			seen[what] = true
+			missing = append(missing, what)
+		}
+	}
+	return missing
+}
+
+func findFile(dir, name string) (string, error) {
+	var found string
+	_ = filepath.WalkDir(dir, func(p string, d fs.DirEntry, err error) error {
+		if err != nil || found != "" {
+			return nil //nolint:nilerr // an unreadable subtree is not the binary
+		}
+		if !d.IsDir() && d.Name() == name {
+			found = p
+		}
+		return nil
+	})
+	if found == "" {
+		return "", fmt.Errorf("the build has no %s in it", name)
+	}
+	return found, nil
+}
+
+func (in *Installer) progress(st *Status, format string, args ...any) {
+	st.State, st.Detail = Installing, fmt.Sprintf(format, args...)
+	in.report(*st)
+}
+
+func (in *Installer) report(st Status) {
+	if in.OnStatus != nil {
+		in.OnStatus(st)
+	}
+}
+
+func (in *Installer) run(ctx context.Context, env []string, name string, args ...string) ([]byte, error) {
+	if in.Run != nil {
+		return in.Run(ctx, env, name, args...)
+	}
+	cmd := exec.CommandContext(ctx, name, args...) //nolint:gosec // fixed commands and paths we installed
+	cmd.Env = append(os.Environ(), env...)
+	return cmd.CombinedOutput()
+}
+
+func (in *Installer) root() string {
+	if in.Root != "" {
+		return in.Root
+	}
+	return DefaultRoot
+}
+
+func (in *Installer) log() *slog.Logger {
+	if in.Log != nil {
+		return in.Log
+	}
+	return slog.Default()
+}
+
+func (in *Installer) client() *http.Client {
+	if in.HTTP != nil {
+		return in.HTTP
+	}
+	return &http.Client{Timeout: 30 * time.Minute}
+}
+
+func (in *Installer) apiBase() string {
+	if in.APIBase != "" {
+		return in.APIBase
+	}
+	return "https://api.github.com"
+}
+
+func (in *Installer) downloadBase() string {
+	if in.DownloadBase != "" {
+		return in.DownloadBase
+	}
+	return "https://github.com"
+}
+
+func lastLine(b []byte) string {
+	lines := strings.Split(strings.TrimSpace(string(b)), "\n")
+	return strings.TrimSpace(lines[len(lines)-1])
+}
+
+func firstVersionLine(b []byte) string {
+	for _, l := range strings.Split(string(b), "\n") {
+		if strings.Contains(strings.ToLower(l), "version") {
+			return strings.TrimSpace(l)
+		}
+	}
+	return ""
+}

internal/agent/provision/install_test.go

@@ -0,0 +1,302 @@
+package provision
+
+import (
+	"archive/tar"
+	"bytes"
+	"compress/gzip"
+	"context"
+	"crypto/sha256"
+	"encoding/hex"
+	"encoding/json"
+	"errors"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"strings"
+	"sync"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
+)
+
+var ubuntu = discovery.Host{OSID: "ubuntu", OSVersion: "24.04", Arch: "amd64", CPUFlags: []string{"avx2"}}
+
+// release is a fake GitHub: one release, its assets and their digests.
+type release struct {
+	assets  map[string][]byte
+	digests map[string]string
+	gets    int
+	mu      sync.Mutex
+}
+
+func tarGz(t *testing.T, files map[string]string, links map[string]string) []byte {
+	t.Helper()
+	var buf bytes.Buffer
+	gz := gzip.NewWriter(&buf)
+	tw := tar.NewWriter(gz)
+	for name, body := range files {
+		_ = tw.WriteHeader(&tar.Header{Name: name, Mode: 0o755, Size: int64(len(body)), Typeflag: tar.TypeReg})
+		_, _ = tw.Write([]byte(body))
+	}
+	for name, target := range links {
+		_ = tw.WriteHeader(&tar.Header{Name: name, Linkname: target, Typeflag: tar.TypeSymlink})
+	}
+	_ = tw.Close()
+	_ = gz.Close()
+	return buf.Bytes()
+}
+
+func (r *release) serve(t *testing.T) *httptest.Server {
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
+		r.mu.Lock()
+		r.gets++
+		r.mu.Unlock()
+		if strings.HasPrefix(req.URL.Path, "/repos/") {
+			var out struct {
+				Assets []map[string]string `json:"assets"`
+			}
+			for name := range r.assets {
+				out.Assets = append(out.Assets, map[string]string{"name": name, "digest": r.digests[name]})
+			}
+			_ = json.NewEncoder(w).Encode(out)
+			return
+		}
+		body, ok := r.assets[filepath.Base(req.URL.Path)]
+		if !ok {
+			http.NotFound(w, req)
+			return
+		}
+		_, _ = w.Write(body)
+	}))
+	t.Cleanup(srv.Close)
+	return srv
+}
+
+func newRelease(assets map[string][]byte) *release {
+	r := &release{assets: assets, digests: map[string]string{}}
+	for name, b := range assets {
+		sum := sha256.Sum256(b)
+		r.digests[name] = "sha256:" + hex.EncodeToString(sum[:])
+	}
+	return r
+}
+
+// fakeSystem answers the package tools, ldd and the engine itself.
+type fakeSystem struct {
+	mu        sync.Mutex
+	installed map[string]bool
+	tools     map[string]bool
+	lddOut    string
+	calls     []string
+}
+
+var errExit = errors.New("exit 1")
+
+func (f *fakeSystem) run(_ context.Context, _ []string, name string, args ...string) ([]byte, error) {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	f.calls = append(f.calls, name+" "+strings.Join(args, " "))
+	switch {
+	case name == "sh" && strings.Contains(args[1], "dpkg-query"),
+		name == "rpm" && args[0] == "-q":
+		if f.installed[args[len(args)-1]] {
+			return nil, nil
+		}
+		return nil, errExit
+	case name == "sh" && strings.Contains(args[1], "command -v"):
+		if f.tools[args[2]] {
+			return nil, nil
+		}
+		return nil, errExit
+	case name == "apt-get" || name == "dnf":
+		for i, a := range args {
+			if a == "install" {
+				for _, p := range args[i+1:] {
+					if !strings.HasPrefix(p, "-") {
+						f.installed[p] = true
+					}
+				}
+			}
+		}
+		return nil, nil
+	case name == "ldd":
+		return []byte(f.lddOut), nil
+	case strings.HasSuffix(name, "llama-server"):
+		return []byte("version: 10991 (abc)\n"), nil
+	}
+	return nil, errors.New("unexpected command " + name)
+}
+
+func installer(t *testing.T, srv *httptest.Server, sys *fakeSystem, host discovery.Host) (*Installer, *[]Status) {
+	var seen []Status
+	var mu sync.Mutex
+	return &Installer{
+		Root: t.TempDir(), Host: host, Run: sys.run,
+		APIBase: srv.URL, DownloadBase: srv.URL, HTTP: srv.Client(),
+		OnStatus: func(s Status) { mu.Lock(); seen = append(seen, s); mu.Unlock() },
+	}, &seen
+}
+
+func llamaRelease(t *testing.T) *release {
+	return newRelease(map[string][]byte{
+		"llama-b1-bin-ubuntu-x64.tar.gz": tarGz(t,
+			map[string]string{"llama-b1/llama-server": "#!/bin/sh\n", "llama-b1/libllama.so.0": "x"},
+			map[string]string{"llama-b1/libllama.so": "libllama.so.0"}),
+	})
+}
+
+func TestAnEngineIsInstalledFromNothing(t *testing.T) {
+	// A node rebuilt from a bare image: no engine, no OpenMP runtime. What it
+	// ends with has to have come from the publisher, checked, and been run.
+	rel := llamaRelease(t)
+	sys := &fakeSystem{installed: map[string]bool{}}
+	in, seen := installer(t, rel.serve(t), sys, ubuntu)
+
+	got := in.Converge(t.Context(), []Want{{Engine: "llamacpp", Version: "b1"}})
+	if len(got) != 1 || got[0].State != Ready {
+		t.Fatalf("not ready: %+v", got)
+	}
+	if !sys.installed["libgomp1"] {
+		t.Error("the OpenMP runtime the build links against was not installed")
+	}
+	if !strings.HasPrefix(got[0].Binary, in.Root) {
+		t.Errorf("binary %s is not the one installed under %s", got[0].Binary, in.Root)
+	}
+	if _, err := os.Lstat(filepath.Join(filepath.Dir(got[0].Binary), "libllama.so")); err != nil {
+		t.Errorf("the library symlink was not unpacked: %v", err)
+	}
+	var states []string
+	for _, s := range *seen {
+		states = append(states, s.State+": "+s.Detail)
+	}
+	if !strings.Contains(strings.Join(states, "\n"), "downloading") {
+		t.Errorf("a download was not reported while it ran:\n%s", strings.Join(states, "\n"))
+	}
+}
+
+func TestAnInstalledEngineIsCheckedNotFetchedAgain(t *testing.T) {
+	rel := llamaRelease(t)
+	sys := &fakeSystem{installed: map[string]bool{}}
+	in, _ := installer(t, rel.serve(t), sys, ubuntu)
+	w := []Want{{Engine: "llamacpp", Version: "b1"}}
+	in.Converge(t.Context(), w)
+	before := rel.gets
+
+	got := in.Converge(t.Context(), w)
+	if got[0].State != Ready {
+		t.Fatalf("second pass: %+v", got[0])
+	}
+	if rel.gets != before {
+		t.Errorf("a ready engine was downloaded again (%d requests)", rel.gets-before)
+	}
+}
+
+func TestAnEngineThatCannotLoadALibraryIsNotReady(t *testing.T) {
+	// Found on a node before: llama-server present, executable, and unable to
+	// start for want of a library — while the node reported ready.
+	rel := llamaRelease(t)
+	sys := &fakeSystem{installed: map[string]bool{"libgomp1": true},
+		lddOut: "\tlibcuda.so.1 => not found\n"}
+	in, _ := installer(t, rel.serve(t), sys, ubuntu)
+
+	got := in.Converge(t.Context(), []Want{{Engine: "llamacpp", Version: "b1"}})
+	if got[0].State != Failed || !strings.Contains(got[0].Detail, "libcuda.so.1") {
+		t.Fatalf("expected a failure naming the library: %+v", got[0])
+	}
+	if _, err := os.Stat(filepath.Join(in.Root, "llamacpp", "b1-cpu", marker)); err == nil {
+		t.Error("a build that does not run was marked installed")
+	}
+}
+
+func TestADownloadThatDoesNotMatchItsChecksumIsRefused(t *testing.T) {
+	rel := llamaRelease(t)
+	for name := range rel.digests {
+		rel.digests[name] = "sha256:" + strings.Repeat("0", 64)
+	}
+	sys := &fakeSystem{installed: map[string]bool{"libgomp1": true}}
+	in, _ := installer(t, rel.serve(t), sys, ubuntu)
+
+	got := in.Converge(t.Context(), []Want{{Engine: "llamacpp", Version: "b1"}})
+	if got[0].State != Failed || !strings.Contains(got[0].Detail, "checksum") {
+		t.Fatalf("a tampered download was not refused: %+v", got[0])
+	}
+}
+
+func TestAnArchiveCannotWriteOutsideItsDirectory(t *testing.T) {
+	rel := newRelease(map[string][]byte{
+		"llama-b1-bin-ubuntu-x64.tar.gz": tarGz(t,
+			map[string]string{"../../escaped": "x", "llama-server": ""}, nil),
+	})
+	sys := &fakeSystem{installed: map[string]bool{"libgomp1": true}}
+	in, _ := installer(t, rel.serve(t), sys, ubuntu)
+
+	got := in.Converge(t.Context(), []Want{{Engine: "llamacpp", Version: "b1"}})
+	if got[0].State != Failed {
+		t.Fatalf("an archive escaping its directory was installed: %+v", got[0])
+	}
+}
+
+func TestTheBuildFollowsTheHardware(t *testing.T) {
+	gpu := ubuntu
+	gpu.NVIDIADriver = "570.86.15"
+	b, err := chooseLlamaCpp("b1", gpu)
+	if err != nil || !strings.HasPrefix(b.Variant, "cuda") || len(b.Assets) != 2 {
+		t.Errorf("an NVIDIA node got %+v, %v; expected the CUDA build and its runtime", b, err)
+	}
+	if _, err := chooseLlamaCpp("b1", discovery.Host{OSID: "ubuntu", Arch: "riscv64"}); err == nil {
+		t.Error("a build was chosen for an architecture llama.cpp does not publish for")
+	}
+
+	slow := ubuntu
+	slow.CPUFlags = nil
+	st := (&Installer{Host: slow}).ready(Status{}, Build{CPUFlags: []string{"avx2"}}, "/x/llama-server", "")
+	if !strings.Contains(st.Detail, "avx2") {
+		t.Errorf("a CPU without AVX2 was not called out: %q", st.Detail)
+	}
+}
+
+func TestAnEngineTheAgentCannotInstallSaysSo(t *testing.T) {
+	in := &Installer{Root: t.TempDir(), Host: ubuntu}
+	got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.29.0"}})
+	if got[0].State != Unsupported {
+		t.Errorf("got %+v", got[0])
+	}
+}
+
+func TestARedHatFamilyNodeInstallsWithItsOwnPackageManager(t *testing.T) {
+	// The build is not Ubuntu's in any way that matters; the package that
+	// carries OpenMP's runtime is named differently, and installed by dnf.
+	rocky := discovery.Host{OSID: "rocky", OSVersion: "10.0", OSLike: []string{"rhel", "centos", "fedora"},
+		Arch: "amd64", CPUFlags: []string{"avx2"}}
+	rel := llamaRelease(t)
+	sys := &fakeSystem{installed: map[string]bool{}, tools: map[string]bool{"dnf": true}}
+	in, _ := installer(t, rel.serve(t), sys, rocky)
+
+	got := in.Converge(t.Context(), []Want{{Engine: "llamacpp", Version: "b1"}})
+	if got[0].State != Ready {
+		t.Fatalf("not ready: %+v", got[0])
+	}
+	if !sys.installed["libgomp"] || sys.installed["libgomp1"] {
+		t.Errorf("installed %v; expected the rpm name, libgomp", sys.installed)
+	}
+}
+
+func TestASystemTooOldForTheBuildSaysWhatItLacks(t *testing.T) {
+	// RHEL 9: glibc is new enough, libstdc++ is one GCC release short. ldd
+	// finds the library and reports the symbol, and "libstdc++ not found"
+	// would send somebody looking for a package that is installed.
+	rel := llamaRelease(t)
+	sys := &fakeSystem{installed: map[string]bool{"libgomp1": true},
+		lddOut: "\t/lib64/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /x/libllama.so)\n" +
+			"\t/lib64/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /x/llama-server)\n"}
+	in, _ := installer(t, rel.serve(t), sys, ubuntu)
+
+	got := in.Converge(t.Context(), []Want{{Engine: "llamacpp", Version: "b1"}})
+	if got[0].State != Failed || !strings.Contains(got[0].Detail, "libstdc++.so.6 is too old (needs GLIBCXX_3.4.30)") {
+		t.Fatalf("detail %q", got[0].Detail)
+	}
+	if strings.Count(got[0].Detail, "GLIBCXX") != 1 {
+		t.Errorf("the same shortfall was listed more than once: %q", got[0].Detail)
+	}
+}

internal/agent/provision/recipe.go

@@ -0,0 +1,109 @@
+// Package provision installs the engines a node is asked to have.
+//
+// The control plane says which engines at which versions; everything else is
+// worked out here, on the node, because only the node knows it: which build
+// fits its processor and driver, what its operating system is missing, and
+// whether the result actually runs. Nothing already on the machine is taken
+// on trust. An engine counts as installed when this package installed it,
+// checked every file against the checksum its publisher gave, and watched it
+// start.
+package provision
+
+import (
+	"fmt"
+	"strings"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
+)
+
+// Build is one published build of an engine, chosen for one machine.
+type Build struct {
+	// Variant names the build: "cpu", "cuda-12.8".
+	Variant string
+	// Assets are the release files to fetch, unpacked in order into one
+	// directory.
+	Assets []string
+	// Packages are operating system packages the build links against and a
+	// minimal install may not carry, by packaging family ("deb", "rpm"): the
+	// same library is libgomp1 on one and libgomp on the other.
+	Packages map[string][]string
+	// Binary is the executable's file name, found anywhere in the unpacked
+	// tree.
+	Binary string
+	// CPUFlags are extensions the build is slow without. Missing ones are
+	// reported, not refused: slow is still running.
+	CPUFlags []string
+}
+
+// Recipe knows how to install one engine.
+type Recipe struct {
+	// Repo is the GitHub repository whose releases carry the builds.
+	Repo string
+	// Choose picks the build for a machine, or says why none fits.
+	Choose func(version string, h discovery.Host) (Build, error)
+}
+
+// Recipes are the engines this agent can install.
+//
+// vLLM and SGLang are not here yet: they are Python environments of several
+// gigabytes rather than an archive, and are installed by hand and named in the
+// agent's configuration until they are.
+var Recipes = map[string]Recipe{
+	"llamacpp": {Repo: "ggml-org/llama.cpp", Choose: chooseLlamaCpp},
+}
+
+// cudaForLlamaCpp is the CUDA build taken on an NVIDIA node. The release also
+// carries a matching cudart archive, so the node needs a driver and nothing
+// else — no toolkit.
+const cudaForLlamaCpp = "12.8"
+
+func chooseLlamaCpp(version string, h discovery.Host) (Build, error) {
+	// The published Linux builds are built on Ubuntu, but nothing about them
+	// is Ubuntu's: they need glibc 2.34 and the libstdc++ of GCC 12, which
+	// Debian 12 and RHEL 10 have and RHEL 9 does not. So the distribution's
+	// name decides nothing here. The build is tried everywhere, and ldd says
+	// whether this system can load it — in the words of the missing symbol.
+	var arch string
+	switch h.Arch {
+	case "amd64":
+		arch = "x64"
+	case "arm64":
+		arch = "arm64"
+	default:
+		return Build{}, fmt.Errorf("llama.cpp publishes no Linux build for %s", orUnknown(h.Arch))
+	}
+
+	b := Build{
+		Variant: "cpu",
+		Assets:  []string{fmt.Sprintf("llama-%s-bin-ubuntu-%s.tar.gz", version, arch)},
+		// Everything else the build links is in the archive or in any current
+		// glibc system; OpenMP's runtime is the one a cloud image lacks, and
+		// the server does not start without it.
+		Packages: map[string][]string{"deb": {"libgomp1"}, "rpm": {"libgomp"}},
+		Binary:   "llama-server",
+	}
+	if arch == "x64" {
+		// The archive carries a backend per CPU generation and loads the best
+		// one, down to SSE4.2. Below AVX2 it runs, several times slower.
+		b.CPUFlags = []string{"avx2"}
+	}
+	// The CUDA build is published for x64 at this version; on arm64 it is a
+	// newer CUDA that the driver may not support, so an arm64 GPU node runs
+	// the CPU build until someone decides otherwise.
+	if h.NVIDIADriver != "" && arch == "x64" {
+		name := fmt.Sprintf("bin-ubuntu-cuda-%s-x64.tar.gz", cudaForLlamaCpp)
+		b.Variant = "cuda-" + cudaForLlamaCpp
+		b.Assets = []string{
+			fmt.Sprintf("llama-%s-%s", version, name),
+			fmt.Sprintf("cudart-llama-%s-%s", version, name),
+		}
+	}
+	return b, nil
+}
+
+func orUnknown(s string) string {
+	if strings.TrimSpace(s) == "" {
+		return "an unrecognised system"
+	}
+	return s
+}

internal/agent/selfdescribe_test.go

@@ -0,0 +1,119 @@
+package agent
+
+import (
+	"context"
+	"os"
+	"slices"
+	"testing"
+	"time"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+)
+
+func TestAnAgentConfigNeedsOnlyWhereTheControlPlaneIs(t *testing.T) {
+	// Every node carried its own name, its own address and the same two
+	// directories written out by hand, so no two agent files were alike and a
+	// copy taken from one node registered the next as the first.
+	cfg := Config{Server: "127.0.0.1:7443"}
+	if err := cfg.normalize(); err != nil {
+		t.Fatal(err)
+	}
+	host, _ := os.Hostname()
+	if cfg.NodeID != host || cfg.Hostname != host {
+		t.Errorf("node_id %q, hostname %q; expected the machine's name %q",
+			cfg.NodeID, cfg.Hostname, host)
+	}
+	if cfg.MgmtAddr != "127.0.0.1" {
+		t.Errorf("mgmt_addr %q, expected the address on the route to the control plane",
+			cfg.MgmtAddr)
+	}
+	if cfg.WeightsCache != DefaultWeightsCache {
+		t.Errorf("weights land in %q", cfg.WeightsCache)
+	}
+	if !slices.Contains(cfg.ModelDirs, cfg.WeightsCache) {
+		t.Errorf("the download directory %q is not looked in: %v",
+			cfg.WeightsCache, cfg.ModelDirs)
+	}
+}
+
+func TestWhatIsWrittenStillWins(t *testing.T) {
+	cfg := Config{
+		Server: "127.0.0.1:7443", NodeID: "gpu-a-01", MgmtAddr: "10.9.9.9",
+		ModelDirs: []string{"/lustre/models"}, WeightsCache: "/scratch/w",
+	}
+	if err := cfg.normalize(); err != nil {
+		t.Fatal(err)
+	}
+	if cfg.NodeID != "gpu-a-01" || cfg.MgmtAddr != "10.9.9.9" {
+		t.Errorf("configured identity overridden: %q %q", cfg.NodeID, cfg.MgmtAddr)
+	}
+	if !slices.Equal(cfg.ModelDirs, []string{"/lustre/models", "/scratch/w"}) {
+		t.Errorf("model dirs %v", cfg.ModelDirs)
+	}
+}
+
+func TestAHeartbeatSaysWhenTheEnginesChanged(t *testing.T) {
+	cfg := Config{Server: "127.0.0.1:7443", NodeID: "n1", WeightsCache: t.TempDir()}
+	cfg.Superv.WorkDir = t.TempDir()
+	a, err := New(cfg, nil, nil)
+	if err != nil {
+		t.Fatal(err)
+	}
+	engines := func(m *llmgwv1.AgentMessage) *llmgwv1.Heartbeat { return m.GetHeartbeat() }
+
+	a.mu.Lock()
+	a.configuredVers = map[string]string{"llamacpp": "unknown"}
+	a.enginesSent = enginesFingerprint(a.configuredVers)
+	a.mu.Unlock()
+	if hb := engines(a.heartbeat()); hb.GetEnginesChanged() {
+		t.Errorf("unchanged engines were resent")
+	}
+
+	a.mu.Lock()
+	a.configuredVers = map[string]string{"llamacpp": "unknown", "vllm": "unknown"}
+	a.mu.Unlock()
+	hb := engines(a.heartbeat())
+	if !hb.GetEnginesChanged() || len(hb.GetEngineVersions()) != 2 {
+		t.Errorf("an installed engine was not reported: %v", hb.GetEngineVersions())
+	}
+	if engines(a.heartbeat()).GetEnginesChanged() {
+		t.Errorf("reported again on the next heartbeat")
+	}
+}
+
+func TestWhatTheControlPlaneAsksToInstallIsReportedBack(t *testing.T) {
+	cfg := Config{Server: "127.0.0.1:7443", NodeID: "n1", WeightsCache: t.TempDir(),
+		EnginesDir: t.TempDir()}
+	cfg.Superv.WorkDir = t.TempDir()
+	a, err := New(cfg, nil, nil)
+	if err != nil {
+		t.Fatal(err)
+	}
+	ctx, cancel := context.WithCancel(t.Context())
+	defer cancel()
+	go a.provisionLoop(ctx)
+
+	// An engine this agent cannot install: no network is needed to learn so,
+	// and the answer has to reach the control plane rather than a log.
+	a.setProvision(&llmgwv1.Provision{Engines: []*llmgwv1.EngineWant{{Engine: "vllm", Version: "0.29.0"}}})
+
+	deadline := time.Now().Add(3 * time.Second)
+	for time.Now().Before(deadline) {
+		hb := a.heartbeat().GetHeartbeat()
+		if hb.GetProvisionChanged() {
+			p := hb.GetProvision()
+			if len(p) != 1 || p[0].GetState() != "unsupported" {
+				t.Fatalf("reported %v", p)
+			}
+			a.mu.Lock()
+			_, reported := a.engineReportLocked()["vllm"]
+			a.mu.Unlock()
+			if reported {
+				t.Error("an engine that was not installed is reported as present")
+			}
+			return
+		}
+		time.Sleep(20 * time.Millisecond)
+	}
+	t.Fatal("the installation's outcome never reached a heartbeat")
+}

internal/agent/servers_test.go

@@ -0,0 +1,90 @@
+package agent
+
+import (
+	"slices"
+	"strings"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
+)
+
+func normalized(t *testing.T, c Config) Config {
+	t.Helper()
+	if err := c.normalizeServerURL(); err != nil {
+		t.Fatalf("normalising %+v: %v", c, err)
+	}
+	return c
+}
+
+func TestOneServerStillWorksExactlyAsBefore(t *testing.T) {
+	// Every agent configuration written before there could be a second control
+	// plane names one address, and none of them should have to change.
+	c := normalized(t, Config{Server: "10.0.0.140:7443"})
+	if !slices.Equal(c.Servers, []string{"http://10.0.0.140:7443"}) {
+		t.Errorf("servers = %v", c.Servers)
+	}
+	if c.Server != "http://10.0.0.140:7443" {
+		t.Errorf("server = %q", c.Server)
+	}
+}
+
+func TestTheAddressAlreadyThereStaysPreferred(t *testing.T) {
+	// An operator adding a standby to an existing file expects the control
+	// plane that was already working to keep being tried first.
+	c := normalized(t, Config{
+		Server:  "first:7443",
+		Servers: []string{"second:7443", "third:7443"},
+	})
+	want := []string{"http://first:7443", "http://second:7443", "http://third:7443"}
+	if !slices.Equal(c.Servers, want) {
+		t.Errorf("servers = %v, want %v", c.Servers, want)
+	}
+}
+
+func TestADuplicateAddressIsNotTriedTwiceInARow(t *testing.T) {
+	// Which would look exactly like the rotation not working.
+	c := normalized(t, Config{
+		Server:  "a:7443",
+		Servers: []string{"http://a:7443", "b:7443"},
+	})
+	want := []string{"http://a:7443", "http://b:7443"}
+	if !slices.Equal(c.Servers, want) {
+		t.Errorf("servers = %v, want %v", c.Servers, want)
+	}
+}
+
+func TestEveryAddressIsCheckedAgainstTheTLSSetting(t *testing.T) {
+	// The check existed for one address and has to hold for all of them: a
+	// standby reached over plaintext while the leader is authenticated would
+	// be a way around mutual TLS, not a fallback.
+	c := Config{
+		Servers: []string{"https://a:7443", "http://b:7443"},
+		TLS:     tlsconf.Config{CA: "ca.pem", Cert: "c.pem", Key: "k.pem"},
+	}
+	err := c.normalizeServerURL()
+	if err == nil {
+		t.Fatal("a plaintext standby was accepted alongside TLS")
+	}
+	if !strings.Contains(err.Error(), "b:7443") {
+		t.Errorf("the error does not say which address: %v", err)
+	}
+}
+
+func TestNoServerAtAllIsStillAnError(t *testing.T) {
+	c := Config{}
+	if err := c.normalizeServerURL(); err == nil {
+		t.Error("an agent with no control plane configured was accepted")
+	}
+}
+
+func TestSchemeIsAddedPerAddress(t *testing.T) {
+	c := normalized(t, Config{
+		Servers: []string{"a:7443", "b:7443"},
+		TLS:     tlsconf.Config{CA: "ca.pem", Cert: "c.pem", Key: "k.pem"},
+	})
+	for _, s := range c.Servers {
+		if !strings.HasPrefix(s, "https://") {
+			t.Errorf("%q did not get the secure scheme", s)
+		}
+	}
+}

internal/agent/supervisor/supervisor_test.go

@@ -8,6 +8,7 @@ import (
 	"net"
 	"os"
 	"path/filepath"
+	"slices"
 	"strings"
 	"sync/atomic"
 	"syscall"
@@ -536,9 +537,9 @@ func TestCorruptStateFileIsCleanedUp(t *testing.T) {
 
 // --- multi-process shards (plan §11, phase 4) ---
 
-func TestSidecarRunsBeforeEngine(t *testing.T) {
-	// The Ray shape: a supporting process starts first and keeps running, then
-	// the engine starts against it.
+func TestASidecarAndAnEngineBothRunUnderOneShard(t *testing.T) {
+	// The Ray shape: a supporting process runs alongside the engine for the
+	// shard's lifetime, but the engine is the one whose exit ends the shard.
 	s, _ := newSupervisor(t)
 	dir := t.TempDir()
 	order := filepath.Join(dir, "order")
@@ -571,13 +572,20 @@ func TestSidecarRunsBeforeEngine(t *testing.T) {
 		}
 	}
 	// Asserted rather than broken out of: the original fell through on timeout,
-	// so a slow machine turned the ordering check into a no-op and the failure
-	// surfaced later as something else.
+	// so a slow machine turned this into a no-op and the failure surfaced later
+	// as something else.
 	if len(started) != 2 {
 		t.Fatalf("only %d of 2 processes started: %v", len(started), started)
 	}
-	if started[0] != "sidecar" || started[1] != "engine" {
-		t.Fatalf("start order was %v", started)
+	// Which of the two appended first is deliberately not asserted. The
+	// supervisor spawns the sidecar first, but spawning is not sequencing:
+	// nothing here makes the engine wait, so two shells racing to append give
+	// either order and a test that demanded one would fail for the scheduler's
+	// reasons rather than the supervisor's. The wait that is a real guarantee
+	// is WaitForLeaderPort, and TestWaitForLeaderPortBlocksUntilListening is
+	// where it is checked.
+	if !slices.Contains(started, "sidecar") || !slices.Contains(started, "engine") {
+		t.Fatalf("expected both processes to have run, got %v", started)
 	}
 
 	// The shard's pid is the engine's, not the sidecar's: the engine's exit is

internal/agent/weights/fetch.go

@@ -84,6 +84,11 @@ func (f *Fetcher) Ensure(ctx context.Context, source string) (string, error) {
 	if source == "" {
 		return "", fmt.Errorf("no source given")
 	}
+	// A version rather than a file. Everything below assumes one URL is one
+	// file, and a set has to be handled before any of it.
+	if IsSet(source) {
+		return f.ensureSet(ctx, source)
+	}
 	if !isRemote(source) {
 		if !isPath(source) {
 			// A bare model id — "Qwen/Qwen2.5-7B-Instruct" — is not a file and
@@ -134,6 +139,15 @@ func (f *Fetcher) Ensure(ctx context.Context, source string) (string, error) {
 
 // fetch downloads a source to a target, resuming and verifying.
 func (f *Fetcher) fetch(ctx context.Context, source, target string) (string, error) {
+	return f.fetchAs(ctx, source, source, target)
+}
+
+// fetchAs downloads one URL but reports progress under another name.
+//
+// The two differ for a set: the shards waiting on it asked for the version, not
+// for the fourth file in it, and progress reported under the file's own URL
+// reaches nobody. What they see is one number climbing across the whole set.
+func (f *Fetcher) fetchAs(ctx context.Context, source, reportAs, target string) (string, error) {
 	url, err := resolveURL(source)
 	if err != nil {
 		return "", err
@@ -192,8 +206,8 @@ func (f *Fetcher) fetch(ctx context.Context, source, target string) (string, err
 	}
 
 	total := existing + resp.ContentLength
-	f.report(Progress{Source: source, BytesDone: existing, BytesTotal: total})
-	written, err := f.copyWithProgress(ctx, out, resp.Body, source, existing, total)
+	f.report(Progress{Source: reportAs, BytesDone: existing, BytesTotal: total})
+	written, err := f.copyWithProgress(ctx, out, resp.Body, reportAs, existing, total)
 	closeErr := out.Close()
 	if err != nil {
 		// The part file is kept on purpose, so the next attempt resumes.

internal/agent/weights/set.go

@@ -0,0 +1,129 @@
+package weights
+
+import (
+	"context"
+	"encoding/json"
+	"fmt"
+	"io"
+	"net/http"
+	"os"
+	"path/filepath"
+	"strings"
+)
+
+// IsSet reports whether a source names a set of files rather than one.
+//
+// By the trailing slash, which is the whole signal: "…/v1/model.gguf" is a file
+// and "…/v1/" is the version it belongs to. A flag beside the source would be a
+// second place for the two to disagree, and the URL already distinguishes a
+// directory from a file everywhere else on the web.
+func IsSet(source string) bool {
+	return isRemote(source) && strings.HasSuffix(source, "/")
+}
+
+// manifest is what the mirror answers for a version.
+type manifest struct {
+	Model   string `json:"model"`
+	Version string `json:"version"`
+	Files   []struct {
+		Name  string `json:"name"`
+		Bytes int64  `json:"bytes"`
+	} `json:"files"`
+	Bytes int64 `json:"bytes"`
+}
+
+// ensureSet downloads every file of a version into one directory.
+//
+// A directory, because that is what a multi-file model is: vLLM is handed the
+// folder and reads the shards and the config out of it, and llama.cpp is handed
+// the first part and finds the rest beside it. Downloading them into one place
+// is what makes either possible, and a catalog entry naming one file could
+// express neither.
+//
+// The whole set or nothing. A directory with three of four shards in it loads
+// as a broken model rather than failing to load, so a partial set is left
+// unfinished under its temporary name and the next attempt starts again.
+func (f *Fetcher) ensureSet(ctx context.Context, source string) (string, error) {
+	dir, err := f.cachePath(strings.TrimSuffix(source, "/"))
+	if err != nil {
+		return "", err
+	}
+	// A complete set is marked by a file written last. Checking for the
+	// directory alone would treat an interrupted download as finished, which is
+	// exactly the failure the marker exists to prevent.
+	done := filepath.Join(dir, ".complete")
+	if _, err := os.Stat(done); err == nil {
+		return dir, nil
+	}
+
+	m, err := f.manifestFor(ctx, source)
+	if err != nil {
+		return "", err
+	}
+	if len(m.Files) == 0 {
+		return "", fmt.Errorf("%s holds no files", source)
+	}
+
+	if err := os.MkdirAll(dir, 0o755); err != nil {
+		return "", err
+	}
+	for _, file := range m.Files {
+		// Rejected rather than joined: a name with a separator in it would
+		// write outside the directory the mirror said it was in, and the
+		// mirror is a place an operator copies files into by hand.
+		if strings.ContainsAny(file.Name, `/\`) || file.Name == ".." {
+			return "", fmt.Errorf("%s: the mirror offered a file named %q", source, file.Name)
+		}
+		target := filepath.Join(dir, file.Name)
+		if st, err := os.Stat(target); err == nil && st.Size() == file.Bytes {
+			continue
+		}
+		// Reported under the set's own address, because that is what the
+		// shards are waiting on. Reusing fetch rather than writing a second
+		// downloader keeps resume, the .part rename and the progress callback
+		// in one place.
+		if _, err := f.fetchAs(ctx, source+file.Name, source, target); err != nil {
+			return "", err
+		}
+	}
+
+	if err := os.WriteFile(done, []byte(source+"\n"), 0o644); err != nil {
+		return "", err
+	}
+	return dir, nil
+}
+
+// manifestFor asks the mirror what a version consists of.
+func (f *Fetcher) manifestFor(ctx context.Context, source string) (manifest, error) {
+	req, err := http.NewRequestWithContext(ctx, http.MethodGet, source, nil)
+	if err != nil {
+		return manifest{}, err
+	}
+	client := f.Client
+	if client == nil {
+		client = &http.Client{}
+	}
+	resp, err := client.Do(req)
+	if err != nil {
+		return manifest{}, fmt.Errorf("asking %s what it holds: %w", source, err)
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode != http.StatusOK {
+		return manifest{}, fmt.Errorf("%s answered %d", source, resp.StatusCode)
+	}
+
+	var m manifest
+	body, err := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
+	if err != nil {
+		return manifest{}, err
+	}
+	if err := json.Unmarshal(body, &m); err != nil {
+		// The likely cause is a plain file server with a directory index, which
+		// answers HTML. Said plainly, because "invalid character '<'" sends
+		// somebody looking at the wrong thing.
+		return manifest{}, fmt.Errorf(
+			"%s did not answer with a manifest; a source ending in / has to be a "+
+				"mirror version, not a directory listing", source)
+	}
+	return m, nil
+}

internal/agent/weights/set_test.go

@@ -0,0 +1,167 @@
+package weights
+
+import (
+	"fmt"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+)
+
+// mirrorOf serves a manifest at "/m/v/" and the files beneath it.
+func mirrorOf(t *testing.T, files map[string]string) (*Fetcher, *httptest.Server, *int) {
+	t.Helper()
+	hits := 0
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		if strings.HasSuffix(r.URL.Path, "/") {
+			var b strings.Builder
+			b.WriteString(`{"model":"m","version":"v","files":[`)
+			first := true
+			total := 0
+			for name, body := range files {
+				if !first {
+					b.WriteString(",")
+				}
+				first = false
+				fmt.Fprintf(&b, `{"name":%q,"bytes":%d}`, name, len(body))
+				total += len(body)
+			}
+			fmt.Fprintf(&b, `],"bytes":%d}`, total)
+			_, _ = w.Write([]byte(b.String()))
+			return
+		}
+		body, ok := files[strings.TrimPrefix(r.URL.Path, "/m/v/")]
+		if !ok {
+			http.NotFound(w, r)
+			return
+		}
+		hits++
+		_, _ = w.Write([]byte(body))
+	}))
+	t.Cleanup(srv.Close)
+	return &Fetcher{CacheDir: t.TempDir()}, srv, &hits
+}
+
+func TestASetIsFetchedIntoOneDirectory(t *testing.T) {
+	f, srv, _ := mirrorOf(t, map[string]string{
+		"model-00001-of-00002.safetensors": "aaaa",
+		"model-00002-of-00002.safetensors": "bb",
+		"config.json":                      "{}",
+	})
+
+	got, err := f.Ensure(t.Context(), srv.URL+"/m/v/")
+	if err != nil {
+		t.Fatal(err)
+	}
+	st, err := os.Stat(got)
+	if err != nil || !st.IsDir() {
+		t.Fatalf("expected a directory, got %q: %v", got, err)
+	}
+	// A directory, because that is what a multi-file model is: vLLM reads the
+	// shards and the config out of the folder, and a catalog entry naming one
+	// file could express neither.
+	for _, name := range []string{"model-00001-of-00002.safetensors", "config.json"} {
+		if _, err := os.Stat(filepath.Join(got, name)); err != nil {
+			t.Errorf("%s did not arrive: %v", name, err)
+		}
+	}
+}
+
+func TestAFinishedSetIsNotFetchedAgain(t *testing.T) {
+	f, srv, hits := mirrorOf(t, map[string]string{"a.gguf": "aaaa", "b.gguf": "bb"})
+
+	if _, err := f.Ensure(t.Context(), srv.URL+"/m/v/"); err != nil {
+		t.Fatal(err)
+	}
+	after := *hits
+	if _, err := f.Ensure(t.Context(), srv.URL+"/m/v/"); err != nil {
+		t.Fatal(err)
+	}
+	if *hits != after {
+		t.Errorf("the second call downloaded again: %d then %d", after, *hits)
+	}
+}
+
+func TestAHalfFetchedSetIsNotTreatedAsFinished(t *testing.T) {
+	f, srv, _ := mirrorOf(t, map[string]string{"a.gguf": "aaaa", "b.gguf": "bb"})
+	dir, err := f.cachePath(srv.URL + "/m/v")
+	if err != nil {
+		t.Fatal(err)
+	}
+	// One file present, no completion marker — an interrupted download.
+	if err := os.MkdirAll(dir, 0o755); err != nil {
+		t.Fatal(err)
+	}
+	if err := os.WriteFile(filepath.Join(dir, "a.gguf"), []byte("aaaa"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+
+	got, err := f.Ensure(t.Context(), srv.URL+"/m/v/")
+	if err != nil {
+		t.Fatal(err)
+	}
+	// A directory with three of four shards loads as a broken model rather
+	// than failing to load, so the marker is what says the set is whole.
+	if _, err := os.Stat(filepath.Join(got, "b.gguf")); err != nil {
+		t.Errorf("the missing file was not fetched: %v", err)
+	}
+}
+
+func TestAFileAlreadyThereAtTheRightSizeIsNotRefetched(t *testing.T) {
+	f, srv, hits := mirrorOf(t, map[string]string{"a.gguf": "aaaa", "b.gguf": "bb"})
+	dir, _ := f.cachePath(srv.URL + "/m/v")
+	_ = os.MkdirAll(dir, 0o755)
+	_ = os.WriteFile(filepath.Join(dir, "a.gguf"), []byte("aaaa"), 0o644)
+
+	if _, err := f.Ensure(t.Context(), srv.URL+"/m/v/"); err != nil {
+		t.Fatal(err)
+	}
+	// Resuming a set means not paying for the shards that already landed.
+	if *hits != 1 {
+		t.Errorf("expected only the missing file to be fetched, got %d downloads", *hits)
+	}
+}
+
+func TestAFilenameThatEscapesTheDirectoryIsRefused(t *testing.T) {
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+		_, _ = w.Write([]byte(`{"files":[{"name":"../../etc/passwd","bytes":1}]}`))
+	}))
+	defer srv.Close()
+	f := &Fetcher{CacheDir: t.TempDir()}
+
+	_, err := f.Ensure(t.Context(), srv.URL+"/m/v/")
+	// The mirror is a place an operator copies files into by hand, so what it
+	// reports is not automatically safe to join onto a path.
+	if err == nil || !strings.Contains(err.Error(), "offered a file named") {
+		t.Errorf("expected the name to be refused, got %v", err)
+	}
+}
+
+func TestADirectoryListingIsNotAManifest(t *testing.T) {
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+		_, _ = w.Write([]byte("<pre><a href=\"a.gguf\">a.gguf</a></pre>"))
+	}))
+	defer srv.Close()
+	f := &Fetcher{CacheDir: t.TempDir()}
+
+	_, err := f.Ensure(t.Context(), srv.URL+"/m/v/")
+	// "invalid character '<'" sends somebody looking at the wrong thing.
+	if err == nil || !strings.Contains(err.Error(), "has to be a mirror version") {
+		t.Errorf("the reason should name the cause, got %v", err)
+	}
+}
+
+func TestOnlyATrailingSlashMeansASet(t *testing.T) {
+	for src, want := range map[string]bool{
+		"http://mirror/m/v/":           true,
+		"http://mirror/m/v/model.gguf": false,
+		"/var/lib/models/":             false, // a local directory is not fetched
+		"Qwen/Qwen2.5-7B-Instruct":     false, // a repo id, resolved by the engine
+	} {
+		if got := IsSet(src); got != want {
+			t.Errorf("IsSet(%q) = %v, want %v", src, got, want)
+		}
+	}
+}

internal/auth/auth_test.go

@@ -16,7 +16,7 @@ import (
 )
 
 // fixture builds a store with one org/team/project and one key.
-func fixture(t *testing.T) (*store.SQLite, string, string) {
+func fixture(t *testing.T) (*store.DB, string, string) {
 	t.Helper()
 	ctx := context.Background()
 	s, err := store.OpenSQLite(ctx, filepath.Join(t.TempDir(), "t.db"))

internal/bench/capacity.go

@@ -0,0 +1,77 @@
+package bench
+
+import "time"
+
+// Capacity is what one replica was measured to serve, kept so the measurement
+// outlives the terminal it was printed in.
+//
+// The conclusion of a run, not the run. The ramp itself — every concurrency
+// level, its latencies, its errors — is worth reading once, while deciding
+// whether the run was sound; it is not worth carrying around afterwards, and
+// storing it would grow without bound as a model is re-benchmarked. What a
+// later reader needs is the number and enough provenance to judge whether it is
+// still true.
+type Capacity struct {
+	Model   string `json:"model"`
+	Profile string `json:"profile"`
+
+	// MaxConcurrencyAtSLO is the capacity: the highest concurrency that still
+	// met the target. Zero means even one request did not, which is a real
+	// result and not a missing one — it says this hardware cannot serve this
+	// model at this SLO at all.
+	MaxConcurrencyAtSLO int `json:"max_concurrency_at_slo"`
+	// ThroughputTokPerSec and the latencies are taken at that level, not at the
+	// peak. Peak throughput arrives long after latency has become unusable, so
+	// reporting it would overstate what the hardware can serve.
+	ThroughputTokPerSec float64       `json:"throughput_tok_per_sec"`
+	TTFTp99             time.Duration `json:"ttft_p99"`
+	TPOTp99             time.Duration `json:"tpot_p99"`
+
+	// SuggestedTargetWaiting and SuggestedKVCeiling are the queue depth and
+	// cache pressure seen while the SLO was still being met, which is what
+	// admission should treat as healthy for this model. Admission currently
+	// applies one global default to every model; these are the per-model
+	// numbers it should be using instead.
+	SuggestedTargetWaiting int     `json:"suggested_target_waiting"`
+	SuggestedKVCeiling     float64 `json:"suggested_kv_ceiling"`
+
+	// SLO is recorded because capacity is meaningless without it. The same
+	// replica has a different capacity against a 200ms first-token target than
+	// against a 2s one, and a number stored without the target it was measured
+	// against cannot be compared with anything.
+	SLO SLO `json:"slo"`
+
+	// Target is the endpoint the load actually went to, kept so a profile
+	// measured against the gateway is never mistaken for one measured against
+	// a replica. Only the second says anything about the hardware.
+	Target     string    `json:"target"`
+	MeasuredAt time.Time `json:"measured_at"`
+	// Stale is not a field. Whether a measurement still holds depends on
+	// whether the hardware and the engine build have changed since, which this
+	// package cannot know; MeasuredAt is given so the reader can decide.
+}
+
+// Capacity summarizes a completed run for storage.
+func (r Result) Capacity(profile string) Capacity {
+	return Capacity{
+		Model:                  r.Model,
+		Profile:                profile,
+		MaxConcurrencyAtSLO:    r.MaxConcurrencyAtSLO,
+		ThroughputTokPerSec:    r.ThroughputTokPerSec,
+		TTFTp99:                r.TTFTp99,
+		TPOTp99:                r.TPOTp99,
+		SuggestedTargetWaiting: r.SuggestedTargetWaiting,
+		SuggestedKVCeiling:     r.SuggestedKVCeiling,
+		SLO:                    r.SLO,
+		Target:                 r.Target,
+		MeasuredAt:             r.RanAt,
+	}
+}
+
+// Usable reports whether this measurement says anything.
+//
+// A run that never met the SLO, or one stopped before it finished a level,
+// concludes at zero. That is worth recording — it is the answer to "can this
+// run here at all" — but a caller sizing a model against it would compute a
+// replica count of infinity, so it has to be asked about rather than used.
+func (c Capacity) Usable() bool { return c.MaxConcurrencyAtSLO > 0 }

internal/cli/bench.go

@@ -3,14 +3,16 @@ package cli
 import (
 	"encoding/json"
 	"fmt"
+	"strings"
 	"time"
 
 	"github.com/spf13/cobra"
 
 	"gitea.mynodes.xyz/ilker/llmgw/internal/bench"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
 )
 
-func newBenchCmd(asJSON *bool) *cobra.Command {
+func newBenchCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 	cfg := bench.DefaultConfig()
 	var (
 		steps   []int
@@ -18,6 +20,7 @@ func newBenchCmd(asJSON *bool) *cobra.Command {
 		metrics string
 		engine  string
 		key     string
+		save    bool
 	)
 
 	cmd := &cobra.Command{
@@ -78,6 +81,15 @@ func newBenchCmd(asJSON *bool) *cobra.Command {
 			if err != nil {
 				return err
 			}
+			if save {
+				if err := saveCapacity(cmd, client, res, target); err != nil {
+					// Reported, not fatal. The run has already cost the GPU
+					// time; losing its conclusion to a failed write would mean
+					// running it again, and the numbers are on screen either
+					// way.
+					cmd.PrintErrf("the measurement was not saved: %v\n", err)
+				}
+			}
 			if *asJSON {
 				return json.NewEncoder(cmd.OutOrStdout()).Encode(res)
 			}
@@ -99,9 +111,63 @@ func newBenchCmd(asJSON *bool) *cobra.Command {
 		"send every request the same prompt, measuring the cached case instead of the hardware")
 	f.DurationVar(&cfg.SLO.TTFTp99, "ttft-p99", cfg.SLO.TTFTp99, "time-to-first-token target")
 	f.DurationVar(&cfg.SLO.TPOTp99, "tpot-p99", cfg.SLO.TPOTp99, "per-output-token target")
+	f.BoolVar(&save, "save", false,
+		"record the capacity against the model and profile, so the number outlives this terminal")
 	return cmd
 }
 
+// saveCapacity records a run's conclusion against the profile it was measured
+// on.
+//
+// The profile is looked up from the replica rather than asked for. Capacity
+// belongs to the hardware and the engine arguments as much as to the weights,
+// so a measurement filed under the wrong profile is worse than none — and the
+// control plane already knows which profile is answering on which endpoint, so
+// nobody needs to be trusted to type it.
+func saveCapacity(cmd *cobra.Command, client func() *ctlClient, res bench.Result, target string) error {
+	var reps []adminapi.ReplicaView
+	if err := client().get(cmd.Context(), "/api/v1/replicas", &reps); err != nil {
+		return err
+	}
+	profile := ""
+	for _, rep := range reps {
+		if rep.Model == res.Model && sameEndpoint(rep.Endpoint, target) {
+			profile = rep.Profile
+			break
+		}
+	}
+	if profile == "" {
+		// Almost always because --target was the gateway. Benchmarking the
+		// whole path is a useful thing to do and says nothing about what one
+		// replica can serve, so there is nothing here worth filing as capacity.
+		return fmt.Errorf("%s is not a replica of %s, so this run measures the path rather than the hardware; "+
+			"point --target at a replica endpoint from `llmgw ctl replicas`", target, res.Model)
+	}
+
+	body, err := json.Marshal(res.Capacity(profile))
+	if err != nil {
+		return err
+	}
+	var saved bench.Capacity
+	if err := client().put(cmd.Context(),
+		"/api/v1/models/"+res.Model+"/capacity", body, &saved); err != nil {
+		return err
+	}
+	cmd.PrintErrf("saved: %s on profile %s, %d concurrent at the SLO\n",
+		saved.Model, saved.Profile, saved.MaxConcurrencyAtSLO)
+	return nil
+}
+
+// sameEndpoint compares two base URLs, ignoring a trailing slash.
+//
+// Written out because the two come from different places: one is what the
+// control plane recorded, the other is what somebody typed or pasted, and a
+// trailing slash is the difference that would silently make every run look like
+// it was aimed at the gateway.
+func sameEndpoint(a, b string) bool {
+	return strings.TrimSuffix(a, "/") == strings.TrimSuffix(b, "/")
+}
+
 func printConclusion(cmd *cobra.Command, res bench.Result, cfg bench.Config) {
 	out := cmd.OutOrStdout()
 	fmt.Fprintln(out)

internal/cli/capacity.go

@@ -0,0 +1,77 @@
+package cli
+
+import (
+	"encoding/json"
+	"fmt"
+	"text/tabwriter"
+	"time"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+)
+
+// newCapacityCmd shows what benchmarks have found out.
+//
+// `ctl bench` has always been able to measure this; until it could be saved,
+// the number lived in whoever ran it's scrollback. This is the other half:
+// somewhere to read it back, so sizing a model is looking something up rather
+// than re-running an hour of load.
+func newCapacityCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
+	return &cobra.Command{
+		Use:   "capacity",
+		Short: "What a replica of each model was measured to serve",
+		Long: "Recorded by `llmgw ctl bench --save`, per model and profile — the same\n" +
+			"model on CPU and on a GPU are different numbers.\n\n" +
+			"Age matters. Hardware is replaced and engine builds change, so a\n" +
+			"measurement is an observation with a date on it rather than a fact about\n" +
+			"the model. The column is there to be read.",
+		Args: cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			var rows []adminapi.CapacityView
+			if err := client().get(cmd.Context(), "/api/v1/capacity", &rows); err != nil {
+				return err
+			}
+			if *asJSON {
+				return json.NewEncoder(cmd.OutOrStdout()).Encode(rows)
+			}
+			if len(rows) == 0 {
+				fmt.Fprintln(cmd.OutOrStdout(),
+					"Nothing has been measured. Run `llmgw ctl bench <model> --target <replica> --save`.")
+				return nil
+			}
+
+			w := tabwriter.NewWriter(cmd.OutOrStdout(), 0, 0, 2, ' ', 0)
+			fmt.Fprintln(w, "MODEL\tPROFILE\tCONCURRENT\tTOK/S\tTTFT p99\tTPOT p99\tWAIT\tKV\tMEASURED\tBY")
+			for _, c := range rows {
+				concurrent := fmt.Sprint(c.MaxConcurrencyAtSLO)
+				if !c.Usable() {
+					// Zero is a real result — this hardware cannot serve this
+					// model at this SLO — and printing it as a number invites
+					// somebody to divide by it.
+					concurrent = "none at SLO"
+				}
+				fmt.Fprintf(w, "%s\t%s\t%s\t%.0f\t%s\t%s\t%d\t%.0f%%\t%s\t%s\n",
+					c.Model, c.Profile, concurrent, c.ThroughputTokPerSec,
+					ms(c.TTFTp99), ms(c.TPOTp99),
+					c.SuggestedTargetWaiting, c.SuggestedKVCeiling*100,
+					age(c.MeasuredAt), or(c.MeasuredBy, "—"))
+			}
+			return w.Flush()
+		},
+	}
+}
+
+func age(t time.Time) string {
+	if t.IsZero() {
+		return "unknown"
+	}
+	return time.Since(t).Round(time.Minute).String() + " ago"
+}
+
+func or(v, fallback string) string {
+	if v == "" {
+		return fallback
+	}
+	return v
+}

internal/cli/ctl.go

@@ -43,6 +43,12 @@ func (c *ctlClient) post(ctx context.Context, path string, body []byte, out any)
 	return c.do(ctx, http.MethodPost, path, json.RawMessage(body), out)
 }
 
+// put replaces a resource. Used where a second call is a better answer to the
+// same question rather than another one: re-measuring a model's capacity.
+func (c *ctlClient) put(ctx context.Context, path string, body []byte, out any) error {
+	return c.do(ctx, http.MethodPut, path, json.RawMessage(body), out)
+}
+
 func (c *ctlClient) delete(ctx context.Context, path string, out any) error {
 	return c.do(ctx, http.MethodDelete, path, nil, out)
 }
@@ -121,57 +127,100 @@ func newCtlCmd() *cobra.Command {
 			"Move work off a node, one replica at a time"),
 		// cert works on files rather than the admin API, so --server does not
 		// apply to it; it lives here because this is where operators look.
-		newBenchCmd(&asJSON),
+		newBenchCmd(client, &asJSON),
 		newStatusCmd(client, &asJSON),
 		newModelCmd(client),
+		newScaleCmd(client),
 		newEngineCheckCmd(),
 		newToolCheckCmd(),
 		newWeightsCmd(client, &asJSON),
+		newCapacityCmd(client, &asJSON),
+		newUsageCmd(client, &asJSON),
 		newCertCmd(),
 	)
 	return cmd
 }
 
 func newNodesCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
-	return &cobra.Command{
+	var f nodeFilter
+	c := &cobra.Command{
 		Use:   "nodes",
 		Short: "List nodes and their capacity",
+		Long: "On a fleet too large to read a whole table of, the filters are how a\n" +
+			"question gets asked: which nodes on this fabric, which of this GPU class,\n" +
+			"which are not ready. They combine, and they apply to --json too.",
 		RunE: func(cmd *cobra.Command, _ []string) error {
 			var nodes []adminapi.NodeView
 			if err := client().get(cmd.Context(), "/api/v1/nodes", &nodes); err != nil {
 				return err
 			}
+			total := len(nodes)
+			nodes, err := f.apply(nodes)
+			if err != nil {
+				return err
+			}
 			if *asJSON {
 				return emitJSON(nodes)
 			}
 			if len(nodes) == 0 {
+				// A filter that matched nothing and an empty fleet call for
+				// opposite next steps: one is a cluster to look at, the other
+				// is a word to correct.
+				if !f.empty() && total > 0 {
+					fmt.Println(noMatches("nodes", f.describe(), total))
+					return nil
+				}
 				fmt.Println("No nodes have registered.")
 				return nil
 			}
 
 			w := tab()
-			fmt.Fprintln(w, "NODE\tSTATE\tGPUS\tMEM\tDISK\tCLASSES\tFABRICS\tREPLICAS\tLAST SEEN")
+			fmt.Fprintln(w, "NODE\tSTATE\tGPUS\tMEM\tDISK\tENGINES\tCLASSES\tFABRICS\tREPLICAS\tLAST SEEN")
 			for _, n := range nodes {
-				fmt.Fprintf(w, "%s\t%s\t%d/%d free\t%s\t%s\t%s\t%s\t%d\t%s\n",
+				fmt.Fprintf(w, "%s\t%s\t%d/%d free\t%s\t%s\t%s\t%s\t%s\t%d\t%s\n",
 					n.ID, n.State, n.GPUFree, n.GPUTotal, memColumn(n.MemoryGB),
 					diskColumn(n.DiskFreeGB, n.DiskTotalGB),
+					// What the node can run, which is the first thing to look
+					// at when a model will not place and the hardware looks
+					// fine.
+					join(n.Engines),
 					join(n.Classes), join(n.Fabrics), n.Replicas, n.LastSeenAgo)
 			}
 			_ = w.Flush()
 
 			// Unhealthy GPUs are usually the answer when a node looks idle but
-			// nothing schedules onto it.
+			// nothing schedules onto it. An engine still installing, or one
+			// that failed to, is the other usual answer.
+			said := false
 			for _, n := range nodes {
 				for _, p := range n.Unhealthy {
 					fmt.Printf("\n%s: %s (kept out of placement)", n.ID, p)
+					said = true
+				}
+				for _, p := range n.Provision {
+					if p.State == "ready" {
+						continue
+					}
+					fmt.Printf("\n%s: %s %s %s: %s", n.ID, p.Engine, p.Version, p.State, p.Detail)
+					said = true
+				}
+				for _, w := range n.Warnings {
+					fmt.Printf("\n%s: %s", n.ID, w)
+					said = true
 				}
 			}
-			if hasUnhealthy(nodes) {
+			if said {
 				fmt.Println()
 			}
 			return nil
 		},
 	}
+	c.Flags().StringVar(&f.fabric, "fabric", "", "only nodes attached to this fabric")
+	c.Flags().StringVar(&f.class, "class", "", "only nodes with this GPU class")
+	c.Flags().StringVar(&f.state, "state", "", "only nodes in this state (prefix match)")
+	c.Flags().StringArrayVar(&f.labels, "label", nil,
+		"only nodes carrying key=value; repeatable, and key alone matches any value")
+	return c
 }
 
 // modelStatus says what is happening to a model, in the words that decide what
@@ -275,6 +324,7 @@ func newModelsCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 
 func newReplicasCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 	var model string
+	var f replicaFilter
 	c := &cobra.Command{
 		Use:   "replicas",
 		Short: "List replicas and the shards they are made of",
@@ -287,10 +337,16 @@ func newReplicasCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 			if err := client().get(cmd.Context(), path, &reps); err != nil {
 				return err
 			}
+			total := len(reps)
+			reps = f.apply(reps)
 			if *asJSON {
 				return emitJSON(reps)
 			}
 			if len(reps) == 0 {
+				if !f.empty() && total > 0 {
+					fmt.Println(noMatches("replicas", f.describe(), total))
+					return nil
+				}
 				fmt.Println("No replicas are running.")
 				return nil
 			}
@@ -302,10 +358,27 @@ func newReplicasCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 				for _, s := range r.Shards {
 					nodes = append(nodes, fmt.Sprintf("%s:%v", s.Node, s.GPUs))
 				}
+				// The state says stale rather than a column of its own: most
+				// replicas are not, a column would be empty for all of them,
+				// and when one is it is the first thing worth knowing about it.
+				state := r.State
+				if r.Stale {
+					state += " (stale profile)"
+				}
 				fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n",
-					r.ID, r.Model, r.State, r.Pool, join(nodes),
+					r.ID, r.Model, state, r.Pool, join(nodes),
 					(time.Duration(r.AgeSec) * time.Second).String(),
 					idleColumn(r.IdleSec), r.Endpoint)
+				// A line rather than a column, because most replicas are not
+				// downloading anything and a column would be empty for all of
+				// them. When one is, it is the only thing worth reading.
+				for _, s := range r.Shards {
+					if s.Fetch == nil {
+						continue
+					}
+					fmt.Fprintf(w, "  %s: fetching %s\t\t\t\t\t\t\t\n",
+						s.Node, fetchColumn(s.Fetch))
+				}
 			}
 			_ = w.Flush()
 
@@ -323,6 +396,9 @@ func newReplicasCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 		},
 	}
 	c.Flags().StringVar(&model, "model", "", "only this model's replicas")
+	c.Flags().StringVar(&f.node, "node", "", "only replicas with a shard on this node")
+	c.Flags().StringVar(&f.state, "state", "", "only replicas in this state (prefix match)")
+	c.Flags().StringVar(&f.pool, "pool", "", "only this pool; matches either half of fabric/class")
 	return c
 }
 
@@ -416,12 +492,3 @@ func dash(s string) string {
 	}
 	return s
 }
-
-func hasUnhealthy(nodes []adminapi.NodeView) bool {
-	for _, n := range nodes {
-		if len(n.Unhealthy) > 0 {
-			return true
-		}
-	}
-	return false
-}

internal/cli/ctlweights.go

@@ -3,6 +3,7 @@ package cli
 import (
 	"encoding/json"
 	"fmt"
+	"strings"
 
 	"github.com/spf13/cobra"
 
@@ -151,3 +152,23 @@ func fileSize(b int64) string {
 		return "size unknown"
 	}
 }
+
+// fetchColumn renders a download's progress.
+//
+// A percentage when the size is known and a byte count when it is not, because
+// a server that does not send Content-Length is common enough — and an
+// invented percentage that stops moving is worse than a number that rises.
+func fetchColumn(f *adminapi.FetchView) string {
+	if f.Percent != nil {
+		return fmt.Sprintf("%s  %d%% of %s", shortSource(f.Source), *f.Percent, fileSize(f.BytesTotal))
+	}
+	return fmt.Sprintf("%s  %s so far", shortSource(f.Source), fileSize(f.BytesDone))
+}
+
+// shortSource keeps the end of a source, which is the part that identifies it.
+func shortSource(s string) string {
+	if i := strings.LastIndex(s, "/"); i >= 0 && i < len(s)-1 {
+		return s[i+1:]
+	}
+	return s
+}

internal/cli/filter.go

@@ -0,0 +1,170 @@
+package cli
+
+import (
+	"fmt"
+	"slices"
+	"strings"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+)
+
+// nodeFilter narrows a node listing.
+//
+// Client-side on the whole list rather than as query parameters, because the
+// list is small even on a large fleet — a few hundred rows — and doing it here
+// means --json is filtered the same way the table is. A server-side filter
+// would be two implementations of the same predicate, and the JSON one is the
+// one nobody would notice had drifted.
+type nodeFilter struct {
+	fabric string
+	class  string
+	state  string
+	labels []string
+}
+
+// empty reports whether this filter would narrow anything.
+//
+// The difference matters: an empty fleet and a filter that matched nothing are
+// different situations with different next steps, and "no nodes" for both
+// makes a mistyped fabric look like an outage.
+func (f nodeFilter) empty() bool {
+	return f.fabric == "" && f.class == "" && f.state == "" && len(f.labels) == 0
+}
+
+func (f nodeFilter) describe() string {
+	var parts []string
+	if f.fabric != "" {
+		parts = append(parts, "fabric "+f.fabric)
+	}
+	if f.class != "" {
+		parts = append(parts, "class "+f.class)
+	}
+	if f.state != "" {
+		parts = append(parts, "state "+f.state)
+	}
+	for _, l := range f.labels {
+		parts = append(parts, "label "+l)
+	}
+	return strings.Join(parts, ", ")
+}
+
+func (f nodeFilter) apply(in []adminapi.NodeView) ([]adminapi.NodeView, error) {
+	want := map[string]string{}
+	for _, l := range f.labels {
+		// A bare key is allowed and means "carries this at all", which is how
+		// an operator asks for every node with a role without caring which.
+		// An empty key is not: "=value" is a typo, and matching everything
+		// would look like the filter having worked.
+		k, v, _ := strings.Cut(l, "=")
+		if k == "" {
+			return nil, fmt.Errorf("--label needs a key: %q has none", l)
+		}
+		want[k] = v
+	}
+
+	out := make([]adminapi.NodeView, 0, len(in))
+	for _, n := range in {
+		if f.fabric != "" && !slices.Contains(n.Fabrics, f.fabric) {
+			continue
+		}
+		if f.class != "" && !slices.Contains(n.Classes, f.class) {
+			continue
+		}
+		// Prefix rather than equality, so --state ready matches and --state re
+		// does too. Node states are few and distinct, and an operator scanning
+		// a fleet is typing quickly.
+		if f.state != "" && !strings.HasPrefix(n.State, strings.ToLower(f.state)) {
+			continue
+		}
+		if !hasLabels(n.Labels, want) {
+			continue
+		}
+		out = append(out, n)
+	}
+	return out, nil
+}
+
+// hasLabels reports whether a node carries every wanted label.
+//
+// An empty value means "has this key at all", which is how an operator asks
+// for every node that carries a role without caring which.
+func hasLabels(have map[string]string, want map[string]string) bool {
+	for k, v := range want {
+		got, ok := have[k]
+		if !ok {
+			return false
+		}
+		if v != "" && got != v {
+			return false
+		}
+	}
+	return true
+}
+
+// replicaFilter narrows a replica listing.
+type replicaFilter struct {
+	node  string
+	state string
+	pool  string
+}
+
+func (f replicaFilter) empty() bool {
+	return f.node == "" && f.state == "" && f.pool == ""
+}
+
+func (f replicaFilter) describe() string {
+	var parts []string
+	if f.node != "" {
+		parts = append(parts, "node "+f.node)
+	}
+	if f.state != "" {
+		parts = append(parts, "state "+f.state)
+	}
+	if f.pool != "" {
+		parts = append(parts, "pool "+f.pool)
+	}
+	return strings.Join(parts, ", ")
+}
+
+func (f replicaFilter) apply(in []adminapi.ReplicaView) []adminapi.ReplicaView {
+	out := make([]adminapi.ReplicaView, 0, len(in))
+	for _, r := range in {
+		// Through the shards, because that is where the nodes are: a replica
+		// spanning two machines is on both, and asking "what is on node05"
+		// has to find it from either side.
+		if f.node != "" && !onNode(r, f.node) {
+			continue
+		}
+		if f.state != "" && !strings.HasPrefix(strings.ToLower(r.State), strings.ToLower(f.state)) {
+			continue
+		}
+		// Substring, because a pool is "fabric/class" and both halves are
+		// things an operator narrows by: --pool ib-a and --pool h100-80g are
+		// both questions worth asking.
+		if f.pool != "" && !strings.Contains(r.Pool, f.pool) {
+			continue
+		}
+		out = append(out, r)
+	}
+	return out
+}
+
+// onNode reports whether any shard of a replica runs on a node.
+func onNode(r adminapi.ReplicaView, node string) bool {
+	for _, sh := range r.Shards {
+		if sh.Node == node {
+			return true
+		}
+	}
+	return false
+}
+
+// noMatches is what to print when a filter excluded everything.
+//
+// Distinct from "there are none", because they call for opposite next steps:
+// one is a cluster to look at and the other is a word to correct. A mistyped
+// fabric name otherwise reads as an outage.
+func noMatches(kind, what string, total int) string {
+	return fmt.Sprintf("No %s match %s. %d %s in total; run without the filter to see them.",
+		kind, what, total, kind)
+}

internal/cli/filter_test.go

@@ -0,0 +1,155 @@
+package cli
+
+import (
+	"strings"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+)
+
+func fleet() []adminapi.NodeView {
+	return []adminapi.NodeView{
+		{ID: "n1", State: "ready", Fabrics: []string{"ib-a"}, Classes: []string{"h100-80g"},
+			Labels: map[string]string{"role": "cpu-large", "rack": "r1"}},
+		{ID: "n2", State: "ready", Fabrics: []string{"ib-b"}, Classes: []string{"h100-80g"},
+			Labels: map[string]string{"role": "cpu-large"}},
+		{ID: "n3", State: "cordoned", Fabrics: []string{"ib-a"}, Classes: []string{"l40s-45g"}},
+		{ID: "n4", State: "ready"},
+	}
+}
+
+func ids(in []adminapi.NodeView) string {
+	var out []string
+	for _, n := range in {
+		out = append(out, n.ID)
+	}
+	return strings.Join(out, ",")
+}
+
+func mustApply(t *testing.T, f nodeFilter) []adminapi.NodeView {
+	t.Helper()
+	got, err := f.apply(fleet())
+	if err != nil {
+		t.Fatal(err)
+	}
+	return got
+}
+
+func TestFilteringByFabricClassAndState(t *testing.T) {
+	if got := ids(mustApply(t, nodeFilter{fabric: "ib-a"})); got != "n1,n3" {
+		t.Errorf("--fabric ib-a = %s", got)
+	}
+	if got := ids(mustApply(t, nodeFilter{class: "l40s-45g"})); got != "n3" {
+		t.Errorf("--class l40s-45g = %s", got)
+	}
+	if got := ids(mustApply(t, nodeFilter{state: "cordoned"})); got != "n3" {
+		t.Errorf("--state cordoned = %s", got)
+	}
+}
+
+func TestFiltersCombine(t *testing.T) {
+	// Which is the question worth asking on a large fleet: not "what is on
+	// this fabric" but "what on this fabric is not ready".
+	if got := ids(mustApply(t, nodeFilter{fabric: "ib-a", state: "ready"})); got != "n1" {
+		t.Errorf("combined = %s", got)
+	}
+}
+
+func TestAStatePrefixIsEnough(t *testing.T) {
+	// Node states are few and distinct, and somebody scanning a fleet types
+	// quickly.
+	if got := ids(mustApply(t, nodeFilter{state: "cord"})); got != "n3" {
+		t.Errorf("--state cord = %s", got)
+	}
+}
+
+func TestALabelKeyAloneMatchesAnyValue(t *testing.T) {
+	// How an operator asks for every node carrying a role without caring which.
+	if got := ids(mustApply(t, nodeFilter{labels: []string{"role"}})); got != "n1,n2" {
+		t.Errorf("--label role = %s", got)
+	}
+	if got := ids(mustApply(t, nodeFilter{labels: []string{"role=cpu-large", "rack=r1"}})); got != "n1" {
+		t.Errorf("two labels = %s", got)
+	}
+	if got := ids(mustApply(t, nodeFilter{labels: []string{"rack=r9"}})); got != "" {
+		t.Errorf("a label nothing carries matched %s", got)
+	}
+}
+
+func TestAMalformedLabelIsRefusedRatherThanIgnored(t *testing.T) {
+	// Silently ignoring it would show the whole fleet, which reads as the
+	// filter having matched everything rather than as a typo.
+	_, err := nodeFilter{labels: []string{"=value"}}.apply(fleet())
+	if err == nil {
+		t.Error("a label with no key was accepted")
+	}
+}
+
+func TestNoFilterIsRecognisedAsNone(t *testing.T) {
+	// It decides which of two messages an empty result gets.
+	if !(nodeFilter{}).empty() {
+		t.Error("an unset filter did not report itself as empty")
+	}
+	if (nodeFilter{state: "ready"}).empty() {
+		t.Error("a set filter reported itself as empty")
+	}
+}
+
+func TestTheEmptyMessageSaysWhichCaseItIs(t *testing.T) {
+	// A mistyped fabric name otherwise reads as an outage.
+	msg := noMatches("nodes", "fabric ib-c", 12)
+	for _, want := range []string{"fabric ib-c", "12"} {
+		if !strings.Contains(msg, want) {
+			t.Errorf("the message does not mention %q: %s", want, msg)
+		}
+	}
+}
+
+func replicas() []adminapi.ReplicaView {
+	return []adminapi.ReplicaView{
+		{ID: "r1", Model: "big", State: "ready", Pool: "ib-a/h100-80g",
+			Shards: []adminapi.ShardView{{Node: "n1"}, {Node: "n2"}}},
+		{ID: "r2", Model: "small", State: "starting", Pool: "none/cpu",
+			Shards: []adminapi.ShardView{{Node: "n5"}}},
+		{ID: "r3", Model: "small", State: "ready", Pool: "ib-b/h100-80g",
+			Shards: []adminapi.ShardView{{Node: "n2"}}},
+	}
+}
+
+func repIDs(in []adminapi.ReplicaView) string {
+	var out []string
+	for _, r := range in {
+		out = append(out, r.ID)
+	}
+	return strings.Join(out, ",")
+}
+
+func TestAReplicaIsFoundFromEitherOfItsNodes(t *testing.T) {
+	// A replica spanning two machines is on both, and "what is running on n2"
+	// has to find it from either side.
+	if got := repIDs(replicaFilter{node: "n1"}.apply(replicas())); got != "r1" {
+		t.Errorf("--node n1 = %s", got)
+	}
+	if got := repIDs(replicaFilter{node: "n2"}.apply(replicas())); got != "r1,r3" {
+		t.Errorf("--node n2 = %s", got)
+	}
+}
+
+func TestAPoolMatchesEitherHalf(t *testing.T) {
+	// A pool is fabric/class and both are things worth narrowing by.
+	if got := repIDs(replicaFilter{pool: "ib-a"}.apply(replicas())); got != "r1" {
+		t.Errorf("--pool ib-a = %s", got)
+	}
+	if got := repIDs(replicaFilter{pool: "h100-80g"}.apply(replicas())); got != "r1,r3" {
+		t.Errorf("--pool h100-80g = %s", got)
+	}
+}
+
+func TestReplicaStateFilters(t *testing.T) {
+	if got := repIDs(replicaFilter{state: "start"}.apply(replicas())); got != "r2" {
+		t.Errorf("--state start = %s", got)
+	}
+	if got := repIDs(replicaFilter{node: "n2", state: "ready"}.apply(replicas())); got != "r1,r3" {
+		t.Errorf("combined = %s", got)
+	}
+}

internal/cli/keys.go

@@ -16,7 +16,7 @@ import (
 )
 
 func newTenancyCmd() *cobra.Command {
-	var dbPath string
+	var sf storeFlags
 
 	cmd := &cobra.Command{
 		Use:   "keys",
@@ -25,9 +25,11 @@ func newTenancyCmd() *cobra.Command {
 			"can be issued before the server is running and without exposing an endpoint\n" +
 			"that mints credentials.",
 	}
-	cmd.PersistentFlags().StringVar(&dbPath, "store", "/var/lib/llmgw/llmgw.db", "path to the store")
+	sf.bind(cmd)
 
-	open := func(ctx context.Context) (*store.SQLite, error) { return store.OpenSQLite(ctx, dbPath) }
+	// Takes a context rather than the command, because some of these run
+	// before the command tree is entered.
+	open := func(ctx context.Context) (*store.DB, error) { return sf.openCtx(ctx) }
 
 	cmd.AddCommand(
 		newKeyInitCmd(open),
@@ -37,11 +39,12 @@ func newTenancyCmd() *cobra.Command {
 		newLimitsCmd(open),
 		newThreadsCmd(open),
 		newMemoryCmd(open),
+		newMoveStoreCmd(),
 	)
 	return cmd
 }
 
-type opener func(context.Context) (*store.SQLite, error)
+type opener func(context.Context) (*store.DB, error)
 
 func newKeyInitCmd(open opener) *cobra.Command {
 	var org, team, project string

internal/cli/mirror.go

@@ -0,0 +1,400 @@
+package cli
+
+import (
+	"encoding/json"
+	"fmt"
+	"log/slog"
+	"net/http"
+	"os"
+	"strings"
+	"time"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
+)
+
+// newMirrorCmd manages a directory of model weights served to the fleet.
+//
+// Operator tooling, run on the mirror host, the way `llmgw keys` is run on the
+// machine holding the store. The gateway never writes here: a mirror that can
+// be written to by the thing reading from it is a way to replace a model with
+// another one that every node would then fetch without complaint.
+func newMirrorCmd() *cobra.Command {
+	var dir string
+
+	c := &cobra.Command{
+		Use:   "mirror",
+		Short: "Hold model weights for the fleet to fetch from",
+		Long: "One machine holding the model files, so nodes fetch over your own\n" +
+			"network instead of the internet. Eight nodes each pulling forty gigabytes\n" +
+			"from outside is a day's work, and on many corporate networks it is not\n" +
+			"possible at all.\n\n" +
+			"Files live at <model>/<version>/<file>, and a version is never\n" +
+			"overwritten. That is what makes rolling back possible: changing the\n" +
+			"catalog line back works because the old weights are still there.",
+	}
+	c.PersistentFlags().StringVar(&dir, "dir", "/srv/models", "the mirror's root directory")
+
+	c.AddCommand(
+		newMirrorAddCmd(&dir),
+		newMirrorListCmd(&dir),
+		newMirrorUnusedCmd(&dir),
+		newMirrorRemoveCmd(&dir),
+		newMirrorServeCmd(&dir),
+	)
+	return c
+}
+
+func newMirrorAddCmd(dir *string) *cobra.Command {
+	var model, version, as string
+
+	c := &cobra.Command{
+		Use:   "add <url>",
+		Short: "Download a model into the mirror",
+		Long: "Downloads to a temporary name and renames when complete, so an\n" +
+			"interrupted download never leaves something a catalog could be pointed\n" +
+			"at. Reports the checksum: a mirror is the one place in this system where\n" +
+			"a corrupted file is copied to every node rather than caught once.\n\n" +
+			"Refuses to replace a version that is already here. Add a new one beside\n" +
+			"it instead — that is what makes the old one still available when the new\n" +
+			"one turns out to be worse.",
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			source := args[0]
+			file := as
+			if file == "" {
+				file = mirror.LastPathElement(source)
+			}
+			if model == "" || version == "" {
+				return fmt.Errorf("--model and --version are required: they are the layout")
+			}
+
+			e := mirror.Entry{Model: model, Version: version, File: file}
+			out := cmd.OutOrStdout()
+			fmt.Fprintf(out, "fetching %s\n", source)
+
+			added, err := mirror.Add(cmd.Context(), *dir, source, e, func(done, total int64) {
+				if total > 0 {
+					fmt.Fprintf(out, "  %s of %s (%d%%)\n",
+						fileSize(done), fileSize(total), done*100/total)
+					return
+				}
+				fmt.Fprintf(out, "  %s so far\n", fileSize(done))
+			})
+			if err != nil {
+				return err
+			}
+			fmt.Fprintf(out, "\n%s/%s/%s  %s\n  sha256 %s\n",
+				added.Entry.Model, added.Entry.Version, added.Entry.File,
+				fileSize(added.Entry.Bytes), added.SHA256)
+			fmt.Fprintf(out, "\nPoint a model at it with:\n  source: %s\n",
+				added.Entry.URL("http://<this-host>:8081"))
+			return nil
+		},
+	}
+	c.Flags().StringVar(&model, "model", "", "the model's name, one directory level")
+	c.Flags().StringVar(&version, "version", "", "the version, one directory level; a date works well")
+	c.Flags().StringVar(&as, "as", "", "the filename to save as; taken from the URL when omitted")
+	return c
+}
+
+func newMirrorListCmd(dir *string) *cobra.Command {
+	return &cobra.Command{
+		Use:   "list",
+		Short: "Show what the mirror holds",
+		Args:  cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			entries, err := mirror.List(*dir)
+			if err != nil {
+				return err
+			}
+			if len(entries) == 0 {
+				fmt.Fprintf(cmd.OutOrStdout(), "The mirror at %s is empty.\n", *dir)
+				return nil
+			}
+			w := tab()
+			fmt.Fprintln(w, "MODEL\tVERSION\tFILE\tSIZE")
+			var total int64
+			for _, e := range entries {
+				fmt.Fprintf(w, "%s\t%s\t%s\t%s\n", e.Model, e.Version, e.File, fileSize(e.Bytes))
+				total += e.Bytes
+			}
+			_ = w.Flush()
+			fmt.Fprintf(cmd.OutOrStdout(), "\n%s in %d files.\n", fileSize(total), len(entries))
+			return nil
+		},
+	}
+}
+
+func newMirrorUnusedCmd(dir *string) *cobra.Command {
+	var server, base string
+
+	c := &cobra.Command{
+		Use:   "unused",
+		Short: "Show files no model in the catalog refers to",
+		Long: "Compares what is here against every source in the catalog, so the\n" +
+			"answer is what could be deleted rather than what looks old.\n\n" +
+			"It refuses to answer when it cannot read the catalog. An empty catalog\n" +
+			"and an unreachable one produce the same list — everything — and the\n" +
+			"obvious next step after that list is deletion.",
+		Args: cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			entries, err := mirror.List(*dir)
+			if err != nil {
+				return err
+			}
+			sources, err := catalogSources(cmd, server)
+			if err != nil {
+				// Named as the reason rather than folded into a generic
+				// failure: an operator who sees "no catalog" will fix the
+				// address, and one who sees a list will start deleting.
+				return fmt.Errorf("cannot read the catalog, so nothing can be called unused: %w", err)
+			}
+
+			unused := mirror.Unused(entries, sources, base)
+			out := cmd.OutOrStdout()
+			if len(unused) == 0 {
+				fmt.Fprintf(out, "Every file here is referred to by a model. (%d files, %d catalog sources.)\n",
+					len(entries), len(sources))
+				return nil
+			}
+			w := tab()
+			fmt.Fprintln(w, "MODEL\tVERSION\tFILE\tSIZE")
+			var total int64
+			for _, e := range unused {
+				fmt.Fprintf(w, "%s\t%s\t%s\t%s\n", e.Model, e.Version, e.File, fileSize(e.Bytes))
+				total += e.Bytes
+			}
+			_ = w.Flush()
+			fmt.Fprintf(out, "\n%s in %d files that no model refers to.\n", fileSize(total), len(unused))
+			fmt.Fprintln(out, "\nBefore deleting: a node that has to refetch one of these will not be\n"+
+				"able to. Retire the catalog entry first, let the fleet settle, then remove.")
+			return nil
+		},
+	}
+	c.Flags().StringVar(&server, "server", "127.0.0.1:9090", "admin API address to read the catalog from")
+	c.Flags().StringVar(&base, "base", "http://localhost:8081",
+		"the address nodes use to reach this mirror, as it appears in catalog sources")
+	return c
+}
+
+func newMirrorRemoveCmd(dir *string) *cobra.Command {
+	var model, version, file, server, base string
+	var force bool
+
+	c := &cobra.Command{
+		Use:   "remove",
+		Short: "Delete one version from the mirror",
+		Long: "Checks the catalog first, and refuses if a model still refers to it —\n" +
+			"a node that has to refetch a deleted file cannot, and the failure arrives\n" +
+			"later, on the node, looking like something else.\n\n" +
+			"Retire the catalog entry first, let the fleet settle, then remove.",
+		Args: cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			if model == "" || version == "" || file == "" {
+				return fmt.Errorf("--model, --version and --file are all required")
+			}
+			e := mirror.Entry{Model: model, Version: version, File: file}
+
+			if !force {
+				sources, err := catalogSources(cmd, server)
+				if err != nil {
+					return fmt.Errorf("cannot read the catalog to check whether this is still in use: %w\n"+
+						"  Pass --force to remove it anyway", err)
+				}
+				if len(mirror.Unused([]mirror.Entry{e}, sources, base)) == 0 {
+					return fmt.Errorf("a model still refers to %s/%s/%s; retire the catalog entry first",
+						model, version, file)
+				}
+			}
+			if err := mirror.Remove(*dir, e); err != nil {
+				return err
+			}
+			fmt.Fprintf(cmd.OutOrStdout(), "removed %s/%s/%s\n", model, version, file)
+			return nil
+		},
+	}
+	c.Flags().StringVar(&model, "model", "", "the model directory")
+	c.Flags().StringVar(&version, "version", "", "the version directory")
+	c.Flags().StringVar(&file, "file", "", "the filename")
+	c.Flags().StringVar(&server, "server", "127.0.0.1:9090", "admin API address to read the catalog from")
+	c.Flags().StringVar(&base, "base", "http://localhost:8081", "the address nodes use to reach this mirror")
+	c.Flags().BoolVar(&force, "force", false, "remove without checking the catalog")
+	return c
+}
+
+func newMirrorServeCmd(dir *string) *cobra.Command {
+	var listen, apiListen, base string
+	var tlsc tlsconf.Config
+
+	c := &cobra.Command{
+		Use:   "serve",
+		Short: "Serve the mirror, and an API to manage it",
+		Long: "Two listeners, on purpose.\n\n" +
+			"The first serves the weights themselves, read-only and unauthenticated,\n" +
+			"because that is what eight nodes pull forty gigabytes from and putting a\n" +
+			"client certificate in front of it would mean re-certifying every agent's\n" +
+			"fetcher to change how models are administered. Range requests work, which\n" +
+			"is what makes an interrupted download on a node resume rather than restart.\n\n" +
+			"The second is the management API the control plane calls: ask the mirror to\n" +
+			"fetch a model, watch it arrive, remove one that is retired. It can be given\n" +
+			"the fleet's CA and then only holders of a certificate from it get in.\n\n" +
+			"The mirror does its own downloading. A control plane that had to pull the\n" +
+			"bytes and push them here would carry every model twice over the network,\n" +
+			"through the one machine least able to spare the bandwidth.",
+		Args: cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			root, err := os.Stat(*dir)
+			if err != nil || !root.IsDir() {
+				return fmt.Errorf("%s is not a directory to serve", *dir)
+			}
+			if err := tlsc.Validate(); err != nil {
+				return err
+			}
+
+			out := cmd.OutOrStdout()
+			log := slog.Default()
+			mgr := mirror.NewManager(*dir, log)
+			api := mirror.NewAPI(mgr, strings.TrimSuffix(base, "/"))
+
+			// http.Dir refuses paths that escape the root, which is the whole
+			// traversal question answered by the standard library rather than
+			// by me.
+			files := &http.Server{
+				Addr:              listen,
+				Handler:           readOnly(withManifest(*dir, http.FileServer(http.Dir(*dir)))),
+				ReadHeaderTimeout: 10 * time.Second,
+			}
+			admin := &http.Server{
+				Addr:              apiListen,
+				Handler:           api.Handler(),
+				ReadHeaderTimeout: 10 * time.Second,
+			}
+
+			errs := make(chan error, 2)
+			go func() { errs <- fmt.Errorf("serving weights: %w", files.ListenAndServe()) }()
+			fmt.Fprintf(out, "serving %s on %s\n", *dir, listen)
+
+			go func() { errs <- fmt.Errorf("serving the API: %w", serveAPI(admin, tlsc)) }()
+			if tlsc.Enabled() {
+				fmt.Fprintf(out, "management API on %s, mutual TLS against %s\n", apiListen, tlsc.CA)
+			} else {
+				// Said plainly rather than left to be discovered. An API that
+				// can delete models and start downloads, open to anyone who can
+				// reach the port, is a choice the operator should have made
+				// rather than inherited from a default.
+				fmt.Fprintf(out, "management API on %s, UNAUTHENTICATED — keep it on a trusted network\n", apiListen)
+			}
+			fmt.Fprintf(out, "catalog sources should point at %s\n", base)
+
+			// The first failure ends it. A mirror serving bytes but refusing
+			// management, or the reverse, is half a service pretending to be
+			// whole, and the half that works keeps the alert quiet.
+			return <-errs
+		},
+	}
+	c.Flags().StringVar(&listen, "listen", ":8081", "address to serve the weights on")
+	c.Flags().StringVar(&apiListen, "api-listen", ":8082", "address to serve the management API on")
+	c.Flags().StringVar(&base, "base", "http://localhost:8081",
+		"the address nodes use to reach this mirror, as it should appear in catalog sources")
+	c.Flags().StringVar(&tlsc.CA, "tls-ca", "", "CA bundle that management clients must present a certificate from")
+	c.Flags().StringVar(&tlsc.Cert, "tls-cert", "", "this mirror's certificate")
+	c.Flags().StringVar(&tlsc.Key, "tls-key", "", "this mirror's private key")
+	return c
+}
+
+// serveAPI starts the management listener, with mutual TLS when it is
+// configured.
+//
+// NextProtos is cleared from what tlsconf builds. That configuration is for the
+// agent stream, which needs HTTP/2 for bidirectional streams and so pins "h2";
+// this is an ordinary REST API, and pinning h2 would fail the handshake for
+// every HTTP/1.1 client — curl included — with an error that says nothing about
+// ALPN. Empty lets net/http offer both.
+func serveAPI(srv *http.Server, c tlsconf.Config) error {
+	if !c.Enabled() {
+		return srv.ListenAndServe()
+	}
+	cfg, err := c.ServerConfig()
+	if err != nil {
+		return err
+	}
+	cfg.NextProtos = nil
+	srv.TLSConfig = cfg
+	return srv.ListenAndServeTLS("", "")
+}
+
+// readOnly refuses anything but a read.
+//
+// http.FileServer already ignores other methods, but saying so here means the
+// property is stated rather than inherited — and a handler added later cannot
+// quietly make the mirror writable.
+func readOnly(h http.Handler) http.Handler {
+	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		if r.Method != http.MethodGet && r.Method != http.MethodHead {
+			http.Error(w, "the mirror is read-only", http.StatusMethodNotAllowed)
+			return
+		}
+		h.ServeHTTP(w, r)
+	})
+}
+
+// catalogSources reads every model's source from a running control plane.
+func catalogSources(cmd *cobra.Command, server string) ([]string, error) {
+	var models []adminapi.ModelView
+	if err := newCtlClient(server).get(cmd.Context(), "/api/v1/models", &models); err != nil {
+		return nil, err
+	}
+	out := make([]string, 0, len(models))
+	for _, m := range models {
+		out = append(out, m.Source)
+	}
+	return out, nil
+}
+
+// withManifest answers a request for a version with the list of files in it.
+//
+// A directory otherwise gets FileServer's HTML index, and something that has to
+// know what a version consists of would be left parsing it. A model is
+// routinely several files — sharded safetensors, or a GGUF split across parts —
+// and a catalog entry names one source, so without a way to ask "what is in
+// this version" a sharded model can be mirrored and never served.
+//
+// On the byte listener rather than the management API on purpose. Fetching
+// weights is the one thing every node does, and making it depend on the
+// administrative port would undo the split that keeps that port closeable.
+func withManifest(root string, next http.Handler) http.Handler {
+	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		model, version, ok := versionPath(r.URL.Path)
+		if !ok {
+			next.ServeHTTP(w, r)
+			return
+		}
+		m, err := mirror.ManifestFor(root, model, version)
+		if err != nil {
+			http.NotFound(w, r)
+			return
+		}
+		w.Header().Set("Content-Type", "application/json")
+		_ = json.NewEncoder(w).Encode(m)
+	})
+}
+
+// versionPath matches "/model/version/" and nothing else.
+//
+// The trailing slash is the whole signal: "/m/v/file.gguf" is a file and
+// "/m/v/" is the set it belongs to. Requiring it means a source URL says which
+// of the two it means without a flag beside it.
+func versionPath(p string) (model, version string, ok bool) {
+	if !strings.HasSuffix(p, "/") {
+		return "", "", false
+	}
+	parts := strings.Split(strings.Trim(p, "/"), "/")
+	if len(parts) != 2 || parts[0] == "" || parts[1] == "" {
+		return "", "", false
+	}
+	return parts[0], parts[1], true
+}

internal/cli/mirror_test.go

@@ -0,0 +1,74 @@
+package cli
+
+import (
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
+)
+
+func TestTheMirrorRefusesEverythingButReads(t *testing.T) {
+	// Stated here rather than inherited from http.FileServer, so a handler
+	// added later cannot quietly make the mirror writable — which would turn
+	// it into a way to replace a model with another one that every node then
+	// fetches without complaint.
+	h := readOnly(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+		w.WriteHeader(http.StatusOK)
+	}))
+
+	for _, m := range []string{http.MethodGet, http.MethodHead} {
+		rec := httptest.NewRecorder()
+		h.ServeHTTP(rec, httptest.NewRequest(m, "/x.gguf", nil))
+		if rec.Code != http.StatusOK {
+			t.Errorf("%s was refused: %d", m, rec.Code)
+		}
+	}
+	for _, m := range []string{http.MethodPut, http.MethodPost, http.MethodDelete, http.MethodPatch} {
+		rec := httptest.NewRecorder()
+		h.ServeHTTP(rec, httptest.NewRequest(m, "/x.gguf", nil))
+		if rec.Code != http.StatusMethodNotAllowed {
+			t.Errorf("%s was allowed: %d", m, rec.Code)
+		}
+	}
+}
+
+func TestTheFilenameIsTakenFromTheURLWhenNotGiven(t *testing.T) {
+	for _, c := range []struct{ url, want string }{
+		{"https://h/a/b/model.gguf", "model.gguf"},
+		{"https://h/model.gguf?download=true", "model.gguf"},
+		{"model.gguf", "model.gguf"},
+	} {
+		if got := mirror.LastPathElement(c.url); got != c.want {
+			t.Errorf("mirror.LastPathElement(%q) = %q, want %q", c.url, got, c.want)
+		}
+	}
+}
+
+func TestAVersionAnswersHeadAsWellAsGet(t *testing.T) {
+	// A sharded model's catalog source is the version directory, and the serve
+	// path checks a source with HEAD before accepting it. If the manifest
+	// answered only GET, every multi-file model would be refused as missing.
+	root := t.TempDir()
+	if err := os.MkdirAll(filepath.Join(root, "llama", "v1"), 0o755); err != nil {
+		t.Fatal(err)
+	}
+	for _, name := range []string{"model-00001-of-00002.safetensors", "model-00002-of-00002.safetensors"} {
+		if err := os.WriteFile(filepath.Join(root, "llama", "v1", name), []byte("x"), 0o644); err != nil {
+			t.Fatal(err)
+		}
+	}
+
+	h := readOnly(withManifest(root, http.FileServer(http.Dir(root))))
+	w := httptest.NewRecorder()
+	h.ServeHTTP(w, httptest.NewRequest(http.MethodHead, "/llama/v1/", nil))
+
+	if w.Code != http.StatusOK {
+		t.Fatalf("a version should answer HEAD; got %d", w.Code)
+	}
+	if got := w.Header().Get("Content-Type"); got != "application/json" {
+		t.Errorf("expected a manifest, got %q", got)
+	}
+}

internal/cli/model.go

@@ -4,13 +4,16 @@ import (
 	"encoding/json"
 	"fmt"
 	"os/user"
+	"time"
 
 	"github.com/knadh/koanf/parsers/yaml"
 	"github.com/knadh/koanf/providers/file"
 	"github.com/knadh/koanf/v2"
 	"github.com/spf13/cobra"
 
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/sourcecheck"
 )
 
 // newModelCmd adds and removes models without a restart.
@@ -31,6 +34,7 @@ func newModelCmd(client func() *ctlClient) *cobra.Command {
 
 func newModelAddCmd(client func() *ctlClient) *cobra.Command {
 	var path string
+	var skipCheck bool
 	c := &cobra.Command{
 		Use:   "add -f <file.yaml>",
 		Short: "Add a model from a definition file",
@@ -61,6 +65,23 @@ func newModelAddCmd(client func() *ctlClient) *cobra.Command {
 			if m.Name == "" {
 				return fmt.Errorf("%s: the definition has no name", path)
 			}
+
+			// Before the catalog is touched, because a source that does not
+			// resolve is otherwise discovered by a replica failing to start
+			// minutes later on a node — with an error naming the node rather
+			// than the typo.
+			if !skipCheck {
+				size, err := sourcecheck.Check(cmd.Context(), m.Source)
+				if err != nil {
+					return fmt.Errorf("%w\n  Pass --no-source-check to add it anyway, "+
+						"which is right when the mirror is not up yet or the nodes "+
+						"reach it by an address this machine does not.", err)
+				}
+				if _, isURL := sourcecheck.HTTPSource(m.Source); isURL {
+					fmt.Fprintln(cmd.OutOrStdout(), sourceNote(m.Source, size))
+				}
+			}
+
 			body, err := json.Marshal(m)
 			if err != nil {
 				return err
@@ -77,6 +98,8 @@ func newModelAddCmd(client func() *ctlClient) *cobra.Command {
 		},
 	}
 	c.Flags().StringVarP(&path, "file", "f", "", "YAML file holding the model definition")
+	c.Flags().BoolVar(&skipCheck, "no-source-check", false,
+		"add it without asking whether the weights are actually at that URL")
 	return c
 }
 
@@ -108,3 +131,58 @@ func actor() string {
 	}
 	return "unknown"
 }
+
+func newScaleCmd(client func() *ctlClient) *cobra.Command {
+	var min, max int
+	var idle time.Duration
+	c := &cobra.Command{
+		Use:   "scale <model> --min N [--max M]",
+		Short: "Change how many replicas a model runs",
+		Long: "The replica count is --min. --max is a ceiling, not a target: a model\n" +
+			"with a minimum runs exactly that many, and nothing scales it up under\n" +
+			"load. Leaving --max out makes it follow --min.\n\n" +
+			"--min 0 with a --max above it is the on-demand model: nothing runs until\n" +
+			"somebody asks, and it stops again after the idle timeout. That loads one\n" +
+			"replica however high --max is. --min 0 --max 0 is a model kept in the\n" +
+			"catalog and deliberately not running.\n\n" +
+			"Only models added at runtime. One declared in server.yaml is read again\n" +
+			"on every start, so a number set here would hold until the next restart\n" +
+			"and then revert — change it in the file instead.\n\n" +
+			"It takes effect on the reconciler's next pass, and survives a restart.",
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			if !cmd.Flags().Changed("min") {
+				return fmt.Errorf("--min is required: it is the replica count")
+			}
+			body, err := json.Marshal(adminapi.ScaleRequest{Min: min, Max: max, IdleTimeout: idle})
+			if err != nil {
+				return err
+			}
+			var res adminapi.ScaleResponse
+			if err := client().post(cmd.Context(),
+				"/api/v1/models/"+args[0]+"/scale", body, &res); err != nil {
+				return err
+			}
+			// What is in force, not what was asked for: the catalog fills in a
+			// max that was left out, and printing the request back would show a
+			// number the cluster is not using.
+			out := cmd.OutOrStdout()
+			switch {
+			case res.OnDemand:
+				fmt.Fprintf(out, "%s is on-demand: loaded when asked for, up to %d, stopped when idle.\n",
+					res.Model, res.Max)
+			case res.Min == 0:
+				fmt.Fprintf(out, "%s will not be placed; it stays in the catalog.\n", res.Model)
+			default:
+				fmt.Fprintf(out, "%s: %d replicas (max %d). It takes effect on the next pass; watch it with:\n"+
+					"  llmgw ctl models\n", res.Model, res.Min, res.Max)
+			}
+			return nil
+		},
+	}
+	c.Flags().IntVar(&min, "min", 0, "replicas to keep running; this is the count")
+	c.Flags().IntVar(&max, "max", 0, "ceiling for an on-demand model; follows --min when left out")
+	c.Flags().DurationVar(&idle, "idle-timeout", 0,
+		"how long an on-demand model stays up unasked-for; zero takes the cluster default")
+	return c
+}

internal/cli/movestore.go

@@ -0,0 +1,71 @@
+package cli
+
+import (
+	"fmt"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// newMoveStoreCmd copies a store into another one.
+//
+// Written because keys are stored hashed. Moving to PostgreSQL without this
+// means every person and every script needs a new key issued, and the old ones
+// cannot even be looked up to say which they were.
+//
+// It reads and writes, and nothing else runs while it does: run it with the
+// gateway stopped. A copy taken while requests are landing would miss whatever
+// arrived after the table it had already passed.
+func newMoveStoreCmd() *cobra.Command {
+	var from, to string
+
+	cmd := &cobra.Command{
+		Use:   "move-store",
+		Short: "Copy a store into another one, for moving SQLite to PostgreSQL",
+		Long: "Copies every table, parents first, into an empty destination.\n\n" +
+			"Stop the gateway first. This makes no attempt to copy a store that is\n" +
+			"being written to: rows landing after the table they belong to has been\n" +
+			"passed would simply be absent, and nothing would say so.\n\n" +
+			"The destination must be empty. Merging two stores means deciding which\n" +
+			"of two projects with the same id was meant, and there is no answer to\n" +
+			"that which is not a guess.",
+		Args: cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			if from == "" || to == "" {
+				return fmt.Errorf("--from and --to are both required")
+			}
+			src, err := store.Open(cmd.Context(), from)
+			if err != nil {
+				return fmt.Errorf("opening the source: %w", err)
+			}
+			defer src.Close()
+
+			dst, err := store.Open(cmd.Context(), to)
+			if err != nil {
+				return fmt.Errorf("opening the destination: %w", err)
+			}
+			defer dst.Close()
+
+			res, err := src.CopyTo(cmd.Context(), dst)
+			// Printed even on failure: a partial copy is the one case where
+			// knowing how far it got decides what to do next.
+			out := cmd.OutOrStdout()
+			var total int64
+			for _, r := range res {
+				if r.Rows > 0 {
+					fmt.Fprintf(out, "  %-16s %d\n", r.Table, r.Rows)
+				}
+				total += r.Rows
+			}
+			if err != nil {
+				return err
+			}
+			fmt.Fprintf(out, "\n%d rows copied. Point the gateway at the new store and start it.\n", total)
+			return nil
+		},
+	}
+	cmd.Flags().StringVar(&from, "from", "", "source: a SQLite path, a postgres:// URL, or @file holding one")
+	cmd.Flags().StringVar(&to, "to", "", "destination, same forms")
+	return cmd
+}

internal/cli/root.go

@@ -42,6 +42,7 @@ func Root() *cobra.Command {
 		newCtlCmd(),
 		newTenancyCmd(),
 		newUsersCmd(),
+		newMirrorCmd(),
 		newSimEngineCmd(),
 	)
 	return root

internal/cli/server.go

@@ -27,7 +27,11 @@ import (
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/fleetmetrics"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelsvc"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/promq"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/router"
+	storepkg "gitea.mynodes.xyz/ilker/llmgw/internal/store"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/version"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/web"
 )
@@ -156,6 +160,16 @@ func newServerCmd() *cobra.Command {
 						admin.ReplaceCatalog(ctx, built)
 					}
 					api.SetModels(svc)
+					// And the mirror page, so a model that has just been
+					// fetched can be served without writing YAML by hand.
+					rt.SetCatalogAdmin(svc)
+					// Benchmark results live here too. A measurement costs real
+					// GPU time; without somewhere to put it the number lives in
+					// whoever ran it's terminal and is gone by morning.
+					api.SetCapacity(st)
+					// And what each group of the tenancy tree spent, which is
+					// the other half of a quota being reviewable.
+					api.SetUsage(st)
 				}
 				api.Register(adminMux)
 
@@ -168,8 +182,26 @@ func newServerCmd() *cobra.Command {
 				if st := rt.Store(); st != nil {
 					usage = st
 				}
+				// The dashboard's operational charts. llmgw publishes these
+				// metrics; what it cannot do is remember them, so the history
+				// is read back from wherever they were scraped to.
+				//
+				// Built before either copy of the page is registered, and given
+				// to both. The first version built it afterwards and handed it
+				// only to the one on the client port, so the copy an operator
+				// reaches by SSH tunnel — the one on this very listener — had
+				// no charts at all and said nothing about why.
+				var history web.History
+				if h := promq.New(cfg.Monitoring.Prometheus, cfg.Monitoring.Timeout); h != nil {
+					history = h
+					log.Info("dashboard history configured",
+						"prometheus", cfg.Monitoring.Prometheus)
+				}
+				rt.SetHistory(history, cfg.Monitoring.Window)
+
 				adminMux.HandleFunc("GET /", web.Dashboard(
-					api, usage, version.Version, 5*time.Second, true, log))
+					api, usage, version.Version, 5*time.Second, true,
+					web.NavExtras{History: history, Window: cfg.Monitoring.Window}, log))
 				adminMux.Handle("GET /_static/", web.Static())
 
 				// The same page on the client-facing port at /admin, behind a
@@ -178,6 +210,24 @@ func newServerCmd() *cobra.Command {
 				rt.SetClusterView(api, usage)
 			}
 
+			// The model mirror, when one is configured. Built here and not at
+			// first use so a mistyped address stops the control plane starting,
+			// rather than turning into a broken admin page hours later when
+			// somebody finally opens it.
+			if cfg.Mirror.API != "" {
+				mc, err := mirror.NewClient(cfg.Mirror.API, cfg.Mirror.TLS, cfg.Mirror.Timeout)
+				if err != nil {
+					return fmt.Errorf("the model mirror: %w", err)
+				}
+				rt.SetMirror(mc)
+				// Browsing the registry is what turns "paste a resolve URL"
+				// into "find the model". It needs a way out to the internet
+				// from this machine, which the mirror's own fetching does not.
+				rt.SetRegistry(hf.New(cfg.Mirror.Registry, cfg.Mirror.Timeout))
+				log.Info("model mirror configured",
+					"api", cfg.Mirror.API, "mtls", cfg.Mirror.TLS.Enabled())
+			}
+
 			apiSrv := &http.Server{
 				Handler:           rt.Handler(),
 				ReadHeaderTimeout: 10 * time.Second,
@@ -335,7 +385,31 @@ func startControlPlane(ctx context.Context, cfg config.Server, cat *catalog.Live
 	if u, ok := sink.(control.UsageSource); ok {
 		ctl.SetUsageSource(u)
 	}
+	// With a shared store, two control planes can see each other, so this one
+	// competes rather than assuming it is alone. Without one there is nobody
+	// to compete with and it places unconditionally, exactly as before.
+	// The same shape as the usage assertion above, and for the same reason:
+	// the sink is the router when the router is serving, and nothing when a
+	// control plane runs on its own.
+	var lead *control.Leadership
+	if h, ok := sink.(interface{ Store() *storepkg.DB }); ok {
+		if st := h.Store(); st != nil {
+			lead = control.NewLeadership(st, cfg.Control.LeaseTTL, log)
+			ctl.SetLeadership(lead)
+			log.Info("competing to be the control plane that places work",
+				"owner", lead.Owner())
+		}
+	}
 	svc := agentsvc.New(ctl.Inventory(), ctl.Reconciler(), requireIdentity, log)
+	svc.SetEngines(cfg.WantedEngines())
+	log.Info("nodes install these engines", "engines", cfg.WantedEngines())
+	// A standby must refuse agents as well as refusing to place. Commands
+	// travel down the stream the agent opened, so a standby holding one is a
+	// node the leader cannot command — and the leader's placements would fail
+	// with "no agent connected" for a node that is plainly connected.
+	if lead != nil {
+		svc.SetLeadership(lead)
+	}
 	fwd.set(svc)
 
 	ctl.Start(ctx)

internal/cli/sourcenote.go

@@ -0,0 +1,17 @@
+package cli
+
+import "fmt"
+
+// sourceNote is what to print about a checked source.
+//
+// It says where the answer came from, because the control plane's network is
+// not the nodes' network: a mirror this machine can reach may be unreachable
+// from a node behind a different firewall, and the reverse. The check catches
+// typos and a mirror that is down, which is most of what goes wrong, and it
+// should not be read as proof that the fetch will work.
+func sourceNote(source string, size int64) string {
+	if size > 0 {
+		return fmt.Sprintf("  source reachable from here: %s (%s)", source, fileSize(size))
+	}
+	return fmt.Sprintf("  source reachable from here: %s (size not advertised)", source)
+}

internal/cli/sourcenote_test.go

@@ -0,0 +1,16 @@
+package cli
+
+import (
+	"strings"
+	"testing"
+)
+
+func TestTheNoteSaysWhereTheAnswerCameFrom(t *testing.T) {
+	// The control plane's network is not the nodes' network. The check catches
+	// typos and a mirror that is down; it is not proof the fetch will work,
+	// and the wording must not suggest it is.
+	note := sourceNote("http://models.internal/x.gguf", 1<<30)
+	if !strings.Contains(note, "from here") {
+		t.Errorf("the note overstates what was checked: %s", note)
+	}
+}

internal/cli/storeflag.go

@@ -0,0 +1,76 @@
+package cli
+
+import (
+	"context"
+	"fmt"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// defaultStorePath is where a single-machine deployment keeps its store.
+const defaultStorePath = "/var/lib/llmgw/llmgw.db"
+
+// defaultConfigPath is where the control plane's configuration usually lives.
+//
+// Consulted rather than required. The commands that work on the store run on
+// the same machine as the server far more often than not, and the server
+// already knows where the store is — asking it is better than making the
+// operator repeat an answer that is written down three lines away.
+const defaultConfigPath = "/etc/llmgw/server.yaml"
+
+// storeFlags is how a command that works directly on the store finds it.
+type storeFlags struct {
+	address string
+	config  string
+}
+
+// bind adds --store and --config to a command.
+func (f *storeFlags) bind(cmd *cobra.Command) {
+	cmd.PersistentFlags().StringVar(&f.address, "store", "",
+		"store address: a SQLite path, a postgres:// DSN, or @file holding one "+
+			"(default: auth.store from --config, else "+defaultStorePath+")")
+	cmd.PersistentFlags().StringVar(&f.config, "config", defaultConfigPath,
+		"server configuration to read the store address from")
+}
+
+// open resolves the store and opens it.
+//
+// Explicit --store wins; otherwise the server's own configuration is asked,
+// and only then does the single-machine default apply. That order exists
+// because the default was a SQLite path and nothing else: on a cluster whose
+// store is PostgreSQL, every one of these commands failed with "unable to open
+// database file" while the right answer sat in the config file beside it. The
+// operator's conclusion was that the tool was broken, and they were not far off.
+func (f *storeFlags) open(cmd *cobra.Command) (*store.DB, error) {
+	return f.openCtx(cmd.Context())
+}
+
+// openCtx is open for a caller that has a context rather than a command.
+func (f *storeFlags) openCtx(ctx context.Context) (*store.DB, error) {
+	address, from := f.resolve()
+	s, err := store.Open(ctx, address)
+	if err != nil {
+		return nil, fmt.Errorf("opening the store at %s (%s): %w", address, from, err)
+	}
+	return s, nil
+}
+
+// resolve returns the address and where it came from, for the error message.
+//
+// Where it came from is half the message. "unable to open database file" says
+// nothing about which of three sources chose that path, and the whole problem
+// is that the wrong one did.
+func (f *storeFlags) resolve() (address, from string) {
+	if f.address != "" {
+		return f.address, "--store"
+	}
+	if f.config != "" {
+		if cfg, err := config.Load(f.config); err == nil && cfg.Auth.Store != "" {
+			return cfg.Auth.Store, "auth.store in " + f.config
+		}
+	}
+	return defaultStorePath, "the default; no --store and nothing usable in " + f.config
+}

internal/cli/storeflag_test.go

@@ -0,0 +1,82 @@
+package cli
+
+import (
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+)
+
+func writeConfig(t *testing.T, body string) string {
+	t.Helper()
+	p := filepath.Join(t.TempDir(), "server.yaml")
+	if err := os.WriteFile(p, []byte(body), 0o600); err != nil {
+		t.Fatal(err)
+	}
+	return p
+}
+
+func TestAnExplicitStoreWins(t *testing.T) {
+	cfg := writeConfig(t, "listen: \":8080\"\nauth:\n  enabled: true\n  store: postgres://from-config/db\n")
+	f := storeFlags{address: "postgres://explicit/db", config: cfg}
+
+	got, from := f.resolve()
+	if got != "postgres://explicit/db" || !strings.Contains(from, "--store") {
+		t.Errorf("an explicit --store should win; got %q from %q", got, from)
+	}
+}
+
+func TestTheServerConfigIsAskedBeforeTheDefault(t *testing.T) {
+	cfg := writeConfig(t, "listen: \":8080\"\nauth:\n  enabled: true\n  store: postgres://from-config/db\n")
+	f := storeFlags{config: cfg}
+
+	got, from := f.resolve()
+	// The whole defect: on a cluster whose store is PostgreSQL, these commands
+	// reached for a SQLite path and failed with "unable to open database file"
+	// while the right answer sat in the config beside them.
+	if got != "postgres://from-config/db" {
+		t.Errorf("expected the address from the config, got %q", got)
+	}
+	if !strings.Contains(from, "auth.store") {
+		t.Errorf("the source should be named for the error message; got %q", from)
+	}
+}
+
+func TestTheSingleMachineDefaultStillApplies(t *testing.T) {
+	f := storeFlags{config: filepath.Join(t.TempDir(), "absent.yaml")}
+
+	got, from := f.resolve()
+	// A deployment with no config file is the single-machine one, and it worked
+	// before this existed. It has to keep working with no flags at all.
+	if got != defaultStorePath {
+		t.Errorf("expected the default path, got %q", got)
+	}
+	if !strings.Contains(from, "default") {
+		t.Errorf("the source should say it fell back; got %q", from)
+	}
+}
+
+func TestAConfigWithNoStoreFallsThrough(t *testing.T) {
+	// Authentication off means no store at all. Reading an empty value as the
+	// answer would open a database named "".
+	cfg := writeConfig(t, "listen: \":8080\"\n")
+	f := storeFlags{config: cfg}
+
+	if got, _ := f.resolve(); got != defaultStorePath {
+		t.Errorf("expected the default when the config names no store, got %q", got)
+	}
+}
+
+func TestTheErrorSaysWhichSourceChoseThePath(t *testing.T) {
+	f := storeFlags{address: filepath.Join(t.TempDir(), "nope", "x.db")}
+
+	_, err := f.openCtx(t.Context())
+	if err == nil {
+		t.Fatal("expected opening a store in a missing directory to fail")
+	}
+	// "unable to open database file" says nothing about which of three sources
+	// picked that path, and the whole problem is that the wrong one did.
+	if !strings.Contains(err.Error(), "--store") {
+		t.Errorf("the error should name where the address came from; got %q", err)
+	}
+}

internal/cli/usage.go

@@ -0,0 +1,74 @@
+package cli
+
+import (
+	"encoding/json"
+	"fmt"
+	"text/tabwriter"
+	"time"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+)
+
+// newUsageCmd shows what each group of the tenancy tree spent.
+//
+// Quotas could be written on an organization, a team or a project long before
+// anything could show what any of them had used: usage came back per key, per
+// user and per model only. A quota whose consumption cannot be seen is one
+// nobody reviews until the day it refuses somebody.
+func newUsageCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
+	var by string
+	var since time.Duration
+
+	c := &cobra.Command{
+		Use:   "usage",
+		Short: "What each organization, team or project has spent",
+		Long: "Grouped by a level of the tenancy tree, which is the level a quota is\n" +
+			"written on. Spend is summed across every key beneath it, so two people\n" +
+			"sharing a team budget appear as one number — the number that will refuse\n" +
+			"them.\n\n" +
+			"The budget column is what that level sets, and is blank when it sets none.\n" +
+			"A team bounded only by its organization has no budget of its own, and\n" +
+			"printing the organization's beside each of its teams would read as though\n" +
+			"each had that much to spend.",
+		Args: cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			var rows []adminapi.ScopeUsageView
+			path := fmt.Sprintf("/api/v1/usage?by=%s&since=%s", by, since)
+			if err := client().get(cmd.Context(), path, &rows); err != nil {
+				return err
+			}
+			if *asJSON {
+				return json.NewEncoder(cmd.OutOrStdout()).Encode(rows)
+			}
+			if len(rows) == 0 {
+				fmt.Fprintf(cmd.OutOrStdout(), "No %s used anything in the last %s.\n", by, since)
+				return nil
+			}
+
+			w := tabwriter.NewWriter(cmd.OutOrStdout(), 0, 0, 2, ' ', 0)
+			fmt.Fprintln(w, "NAME\tID\tKEYS\tREQUESTS\tERRORS\tTOKENS\tSPEND\tBUDGET\tUSED\tLAST")
+			for _, u := range rows {
+				budget, used := "—", "—"
+				if u.Budget != nil {
+					budget = fmt.Sprintf("%.2f", float64(*u.Budget)/1e6)
+					if u.BudgetPeriod != "" {
+						budget += "/" + u.BudgetPeriod
+					}
+				}
+				if u.Fraction != nil {
+					used = fmt.Sprintf("%.0f%%", *u.Fraction*100)
+				}
+				fmt.Fprintf(w, "%s\t%s\t%d\t%d\t%d\t%d\t%.2f\t%s\t%s\t%s\n",
+					u.Name, u.ID, u.Keys, u.Requests, u.Errors,
+					u.Prompt+u.Completion, float64(u.CostMicros)/1e6,
+					budget, used, age(u.LastAt))
+			}
+			return w.Flush()
+		},
+	}
+	c.Flags().StringVar(&by, "by", "team", "level to group by: org, team or project")
+	c.Flags().DurationVar(&since, "since", 24*time.Hour, "how far back to total")
+	return c
+}

internal/cli/users.go

@@ -12,7 +12,7 @@ import (
 )
 
 func newUsersCmd() *cobra.Command {
-	var dbPath string
+	var sf storeFlags
 
 	cmd := &cobra.Command{
 		Use:   "users",
@@ -22,9 +22,9 @@ func newUsersCmd() *cobra.Command {
 			"the step that decides who gets to spend the cluster, and it needs a shell\n" +
 			"on this machine rather than a session cookie.",
 	}
-	cmd.PersistentFlags().StringVar(&dbPath, "store", "/var/lib/llmgw/llmgw.db", "path to the store")
-	open := func(cmd *cobra.Command) (*store.SQLite, error) {
-		return store.OpenSQLite(cmd.Context(), dbPath)
+	sf.bind(cmd)
+	open := func(cmd *cobra.Command) (*store.DB, error) {
+		return sf.open(cmd)
 	}
 
 	cmd.AddCommand(newUserListCmd(open), newUserApproveCmd(open),
@@ -38,7 +38,7 @@ func newUsersCmd() *cobra.Command {
 	return cmd
 }
 
-type userOpener func(*cobra.Command) (*store.SQLite, error)
+type userOpener func(*cobra.Command) (*store.DB, error)
 
 func newUserListCmd(open userOpener) *cobra.Command {
 	var pending bool

internal/config/config.go

@@ -65,6 +65,22 @@ type Server struct {
 	// trusted certificate, while this one is mutual and private.
 	AgentTLS tlsconf.Config `koanf:"agent_tls"`
 
+	// Monitoring is where the dashboard reads its history from.
+	Monitoring Monitoring `koanf:"monitoring"`
+
+	// Mirror points the control plane at a model mirror it can administer.
+	// Empty means there is none, and /admin/mirror stays unmounted.
+	Mirror Mirror `koanf:"mirror"`
+
+	// Engines are the engines every node installs, and at which version. The
+	// agents install them themselves — the right build for their hardware,
+	// checked against its publisher's checksum and seen to run — so a node is
+	// never trusted to already have what somebody once put there.
+	//
+	// Unset keeps DefaultEngines. A version replaces the default; an empty
+	// one leaves that engine to be installed by hand.
+	Engines map[string]string `koanf:"engines"`
+
 	Router  Router          `koanf:"router"`
 	Control Control         `koanf:"control"`
 	Models  []catalog.Model `koanf:"models"`
@@ -75,6 +91,57 @@ type Server struct {
 	Endpoints []Endpoint `koanf:"endpoints"`
 }
 
+// Monitoring points the dashboard at a Prometheus holding this cluster's
+// history.
+//
+// Read back rather than kept twice. llmgw publishes these metrics already; what
+// a gauge cannot do is remember, and a ring buffer in this process would be a
+// rival copy with its own retention, lost on every restart and disagreeing with
+// the one an operator already has on screen.
+type Monitoring struct {
+	// Prometheus is the server's address, for example
+	// http://monitoring.internal:9090. Empty leaves the charts off the
+	// dashboard rather than showing an error where one should be.
+	Prometheus string `koanf:"prometheus"`
+	// Window is how far back the charts look. Zero takes an hour.
+	Window time.Duration `koanf:"window"`
+	// Timeout bounds one query. It runs while somebody waits for a page, so a
+	// monitoring system that is slow today is skipped rather than waited for.
+	Timeout time.Duration `koanf:"timeout"`
+}
+
+// Mirror is a model mirror this control plane administers over its API.
+//
+// The mirror is a separate machine with a large disk, not part of the gateway,
+// so what is configured here is an address and how to authenticate to it —
+// never a directory. A control plane that could write to the mirror's disk
+// directly would be a way to replace a model with another one that every node
+// would then fetch without complaint.
+type Mirror struct {
+	// API is the mirror's management address, scheme included, for example
+	// https://models.internal:8082. Empty means no mirror is configured.
+	//
+	// The management port, not the one nodes fetch bytes from. Those are two
+	// listeners on the mirror precisely so this one can be locked down without
+	// re-certifying every agent's fetcher.
+	API string `koanf:"api"`
+	// TLS authenticates to the mirror's management API, with the same CA the
+	// agents use. Empty leaves the connection in plain HTTP, which is only
+	// acceptable on a network where nothing untrusted can reach that port —
+	// the mirror says as much in its own log when it starts that way.
+	TLS tlsconf.Config `koanf:"tls"`
+	// Registry is the model registry the mirror page browses, empty for the
+	// public Hugging Face. Set it to an internal mirror of the registry where
+	// the control plane cannot reach the internet — or leave the whole mirror
+	// block out, which turns the explorer off rather than giving it a search
+	// box that can only time out.
+	Registry string `koanf:"registry"`
+	// Timeout bounds a management call. It does not bound a download: the
+	// mirror answers a fetch as soon as it has started one, and the bytes keep
+	// arriving long after this deadline has passed.
+	Timeout time.Duration `koanf:"timeout"`
+}
+
 // Landing configures the public page at "/".
 //
 // It exists for one thing the gateway cannot know: who to ask for a key. A page
@@ -177,6 +244,27 @@ type Control struct {
 	PortRange []int `koanf:"port_range"`
 	// MgmtInterface is the ethernet interface NCCL bootstraps over.
 	MgmtInterface string `koanf:"mgmt_interface"`
+
+	// Enabled runs the control plane even when configuration declares no
+	// models.
+	//
+	// Without it, a deployment whose catalog lives entirely in the database
+	// cannot start: control plane mode is inferred from the config file, so an
+	// empty models block means "router only", the process refuses to start for
+	// want of anything to serve, and the models it would have loaded are never
+	// read. That is a cluster that cannot be managed at runtime because it is
+	// managed at runtime. AgentListen cannot carry this signal — it has a
+	// default, so every deployment would become a control plane.
+	Enabled bool `koanf:"enabled"`
+
+	// LeaseTTL is how long this control plane's claim to be the one placing
+	// work stands without being renewed.
+	//
+	// Only meaningful with a shared store, where a second control plane can
+	// see the claim. Long enough to survive a slow query or a brief network
+	// stall, short enough that a standby takes over before anyone has finished
+	// reading the alert. Zero takes the default.
+	LeaseTTL time.Duration `koanf:"lease_ttl"`
 }
 
 // Router holds routing behaviour.
@@ -453,6 +541,9 @@ func (s Server) Catalog() (*catalog.Catalog, error) {
 // plus a static endpoint list is the router-only mode used for testing against
 // sim-engines.
 func (s Server) ControlPlaneMode() bool {
+	if s.Control.Enabled {
+		return true
+	}
 	for _, m := range s.Models {
 		if len(m.Profiles) > 0 {
 			return true
@@ -460,3 +551,25 @@ func (s Server) ControlPlaneMode() bool {
 	}
 	return false
 }
+
+// DefaultEngines are the engine versions this release was checked against:
+// `llmgw ctl engine-check` accepted every flag llmgw builds for them.
+var DefaultEngines = map[string]string{
+	"llamacpp": "b10991",
+}
+
+// WantedEngines is DefaultEngines with the configuration laid over it.
+func (s Server) WantedEngines() map[string]string {
+	out := map[string]string{}
+	for k, v := range DefaultEngines {
+		out[k] = v
+	}
+	for k, v := range s.Engines {
+		if v == "" {
+			delete(out, k)
+			continue
+		}
+		out[k] = v
+	}
+	return out
+}

internal/config/config_test.go

@@ -154,3 +154,48 @@ func TestValidateAcceptsDistinctAliases(t *testing.T) {
 		t.Errorf("valid aliases rejected: %v", err)
 	}
 }
+
+func TestAControlPlaneCanBeAskedForWithNoModelsInTheFile(t *testing.T) {
+	// Without this, a deployment whose catalog lives entirely in the database
+	// cannot start: the mode is inferred from the file, an empty models block
+	// reads as "router only", and the process refuses to start for want of
+	// anything to serve — so the stored models it would have loaded are never
+	// read at all.
+	var s Server
+	if s.ControlPlaneMode() {
+		t.Fatal("an empty configuration is not a control plane")
+	}
+	s.Control.Enabled = true
+	if !s.ControlPlaneMode() {
+		t.Error("control.enabled should run the control plane with no models declared")
+	}
+}
+
+func TestAModelWithProfilesStillMakesAControlPlane(t *testing.T) {
+	// The inference stays, so no existing configuration has to add a flag.
+	var s Server
+	s.Models = []catalog.Model{{
+		Name:     "m",
+		Profiles: []catalog.Profile{{Name: "p", Engine: "llamacpp"}},
+	}}
+	if !s.ControlPlaneMode() {
+		t.Error("a declared model with profiles is still a control plane")
+	}
+}
+
+func TestEnginesDefaultToTheCheckedVersions(t *testing.T) {
+	var s Server
+	if got := s.WantedEngines()["llamacpp"]; got != DefaultEngines["llamacpp"] {
+		t.Errorf("unset config wants llamacpp %q", got)
+	}
+	s.Engines = map[string]string{"llamacpp": "b11000", "vllm": "0.29.0"}
+	got := s.WantedEngines()
+	if got["llamacpp"] != "b11000" || got["vllm"] != "0.29.0" {
+		t.Errorf("configured versions not taken: %v", got)
+	}
+	// Empty leaves an engine to be installed by hand.
+	s.Engines = map[string]string{"llamacpp": ""}
+	if _, ok := s.WantedEngines()["llamacpp"]; ok {
+		t.Error("an emptied engine is still wanted")
+	}
+}

internal/config/example_test.go

@@ -39,8 +39,10 @@ func TestTheDocumentedAgentExampleLoads(t *testing.T) {
 	if f.RateGbps == 0 {
 		t.Error("the fabric's rate did not bind")
 	}
-	if cfg.NodeID == "" || cfg.Server == "" {
-		t.Errorf("node_id=%q server=%q", cfg.NodeID, cfg.Server)
+	// node_id is left to the hostname, so the example need not say it; where
+	// the control plane is, nothing can find out.
+	if cfg.Server == "" {
+		t.Errorf("server=%q", cfg.Server)
 	}
 }
 

internal/control/adminapi/api.go

@@ -12,18 +12,24 @@ import (
 	"fmt"
 	"io"
 	"net/http"
+	"slices"
+	"strings"
 	"time"
 
+	"gitea.mynodes.xyz/ilker/llmgw/internal/bench"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
 )
 
 // API serves the management endpoints.
 type API struct {
-	models Models
-	ctl    *control.Controller
+	models   Models
+	capacity Capacity
+	usage    Usage
+	ctl      *control.Controller
 }
 
 // New builds the API over a controller.
@@ -37,6 +43,10 @@ func (a *API) Register(mux *http.ServeMux) {
 	// it does: reaching this port already means holding the control plane.
 	mux.HandleFunc("POST /api/v1/models", a.addModel)
 	mux.HandleFunc("DELETE /api/v1/models/{name}", a.removeModel)
+	mux.HandleFunc("POST /api/v1/models/{name}/scale", a.scaleModel)
+	mux.HandleFunc("PUT /api/v1/models/{name}/capacity", a.putCapacity)
+	mux.HandleFunc("GET /api/v1/capacity", a.listCapacity)
+	mux.HandleFunc("GET /api/v1/usage", a.usageByScope)
 	mux.HandleFunc("GET /api/v1/nodes", a.listNodes)
 	mux.HandleFunc("GET /api/v1/models", a.listModels)
 	mux.HandleFunc("GET /api/v1/replicas", a.listReplicas)
@@ -64,6 +74,21 @@ type NodeView struct {
 	// what decides whether a CPU profile can be placed here, so it belongs
 	// where someone looks when a model will not schedule.
 	MemoryGB int `json:"memory_gb,omitempty"`
+	// Engines are the inference engines installed here, as the agent found
+	// them. Absent means the agent is older than the probe and said nothing,
+	// which is not the same as having none — placement treats it that way too,
+	// or a control plane upgrade would empty a fleet.
+	Engines []string `json:"engines,omitempty"`
+	// Provision is where installing each wanted engine stands. An engine is in
+	// Engines only once its installation here is ready.
+	Provision []inventory.EngineInstall `json:"provision,omitempty"`
+	// OS is the operating system and architecture, as the agent read them.
+	OS string `json:"os,omitempty"`
+	// Warnings are facts about the machine that make it slow or unusable
+	// without making it unhealthy: a VM with no vector instructions runs every
+	// model, at a fifth of the speed.
+	Warnings []string `json:"warnings,omitempty"`
+
 	// Models are the weights this node can see. Absent means it was not asked
 	// to look, which is not the same as holding none.
 	Models []string `json:"models_on_disk,omitempty"`
@@ -93,22 +118,31 @@ type NodeView struct {
 
 // ModelView is a model as `ctl models` shows it.
 type ModelView struct {
-	Name      string   `json:"name"`
-	Aliases   []string `json:"aliases,omitempty"`
-	Source    string   `json:"source"`
-	Min       int      `json:"min_replicas"`
-	Max       int      `json:"max_replicas"`
-	Ready     int      `json:"ready_replicas"`
-	Total     int      `json:"total_replicas"`
-	Fallbacks []string `json:"fallbacks,omitempty"`
-	Profiles  []string `json:"profiles"`
+	Name string `json:"name"`
+	// Description is what the model is for, when somebody said.
+	Description string   `json:"description,omitempty"`
+	Aliases     []string `json:"aliases,omitempty"`
+	Source      string   `json:"source"`
+	Min         int      `json:"min_replicas"`
+	Max         int      `json:"max_replicas"`
+	Ready       int      `json:"ready_replicas"`
+	Total       int      `json:"total_replicas"`
+	Fallbacks   []string `json:"fallbacks,omitempty"`
+	Profiles    []string `json:"profiles"`
 }
 
 // ReplicaView is a replica as `ctl replicas` shows it.
 type ReplicaView struct {
-	ID       string `json:"id"`
-	Model    string `json:"model"`
-	Profile  string `json:"profile"`
+	ID      string `json:"id"`
+	Model   string `json:"model"`
+	Profile string `json:"profile"`
+	// Stale is true when the profile has been edited since this replica was
+	// placed, so it is running a definition the catalog no longer holds.
+	//
+	// Reported rather than acted on. Moving a replica that is serving costs
+	// more than the imperfection, and the operator who made the edit is the
+	// one who should decide whether it is worth a restart.
+	Stale    bool   `json:"stale,omitempty"`
 	State    string `json:"state"`
 	Pool     string `json:"pool,omitempty"`
 	Fabric   string `json:"fabric,omitempty"`
@@ -131,6 +165,22 @@ type ShardView struct {
 	GPUs  []int  `json:"gpus"`
 	State string `json:"state"`
 	Err   string `json:"error,omitempty"`
+	// Fetch is the download this shard is waiting on, absent when it is not
+	// waiting on one. It is what tells a replica pulling a hundred gigabytes
+	// apart from one that is wedged — which look identical without it.
+	Fetch *FetchView `json:"fetch,omitempty"`
+}
+
+// FetchView is a download in flight.
+type FetchView struct {
+	Source    string `json:"source"`
+	BytesDone int64  `json:"bytes_done"`
+	// BytesTotal is 0 when the server did not say how large the file is.
+	BytesTotal int64 `json:"bytes_total,omitempty"`
+	// Percent is absent rather than zero when the total is unknown: an
+	// invented percentage that stops moving is worse than a byte count that
+	// keeps rising.
+	Percent *int `json:"percent,omitempty"`
 }
 
 // ExplainView is why a model could not be placed.
@@ -186,6 +236,18 @@ func (a *API) Nodes() []NodeView {
 			v.DiskFreeGB = int(free / (1 << 30))
 			v.DiskTotalGB = int(total / (1 << 30))
 		}
+		for name := range n.EngineVersions {
+			v.Engines = append(v.Engines, name)
+		}
+		slices.Sort(v.Engines)
+		v.Provision = n.Provision
+		if h := n.Host; h.OSID != "" {
+			v.OS = strings.TrimSpace(h.OSID + " " + h.OSVersion + " " + h.Arch)
+			if h.Arch == "amd64" && !h.HasCPUFlag("avx2") {
+				v.Warnings = append(v.Warnings, "no AVX2: inference runs several times slower "+
+					"(a VM needs CPU type \"host\")")
+			}
+		}
 		for _, w := range n.Weights {
 			v.Models = append(v.Models, w.Path)
 		}
@@ -225,7 +287,7 @@ func (a *API) Models() []ModelView {
 	out := make([]ModelView, 0)
 	for _, m := range a.ctl.Catalog().Models() {
 		v := ModelView{
-			Name: m.Name, Aliases: m.Aliases, Source: m.Source,
+			Name: m.Name, Description: m.Description, Aliases: m.Aliases, Source: m.Source,
 			Min: m.Scaling.Min, Max: m.Scaling.Max,
 			Ready: ready[m.Name], Total: total[m.Name],
 			Fallbacks: m.Fallbacks,
@@ -253,6 +315,7 @@ func (a *API) Replicas(model string) []ReplicaView {
 			ID: rep.ID, Model: rep.Model, Profile: rep.Profile,
 			State: rep.State.String(), Pool: rep.Pool, Fabric: rep.Fabric,
 			Endpoint: rep.Endpoint, Err: rep.Err,
+			Stale:   a.profileChanged(rep.Model, rep.Profile, rep.ProfileFP),
 			AgeSec:  int64(time.Since(rep.CreatedAt).Seconds()),
 			IdleSec: -1,
 		}
@@ -262,10 +325,19 @@ func (a *API) Replicas(model string) []ReplicaView {
 			v.IdleSec = int64(time.Since(rep.LastUsedAt).Seconds())
 		}
 		for _, s := range rep.Shards {
-			v.Shards = append(v.Shards, ShardView{
+			sv := ShardView{
 				ID: s.ID, Node: s.NodeID, Rank: s.Rank,
 				GPUs: s.GPUIndices, State: s.State.String(), Err: s.Err,
-			})
+			}
+			if f := s.Fetch; f != nil {
+				sv.Fetch = &FetchView{
+					Source: f.Source, BytesDone: f.BytesDone, BytesTotal: f.BytesTotal,
+				}
+				if p, ok := f.Percent(); ok {
+					sv.Fetch.Percent = &p
+				}
+			}
+			v.Shards = append(v.Shards, sv)
 		}
 		out = append(out, v)
 	}
@@ -279,6 +351,173 @@ func (a *API) Replicas(model string) []ReplicaView {
 type Models interface {
 	Add(ctx context.Context, m catalog.Model, by string) error
 	Remove(ctx context.Context, name string) error
+	Scale(ctx context.Context, name string, s catalog.Scaling, by string) (catalog.Model, error)
+}
+
+// Usage reports what each group of the tenancy tree spent. The store
+// implements it.
+type Usage interface {
+	UsageByScope(ctx context.Context, scope store.Scope, since time.Time) ([]store.ScopeUsage, error)
+}
+
+// SetUsage supplies the usage reporting source.
+func (a *API) SetUsage(u Usage) { a.usage = u }
+
+// ScopeUsageView is one group's spend, with its quota when it has one.
+type ScopeUsageView struct {
+	Scope string `json:"scope"`
+	ID    string `json:"id"`
+	Name  string `json:"name"`
+
+	Keys       int64 `json:"keys"`
+	Requests   int64 `json:"requests"`
+	Prompt     int64 `json:"prompt_tokens"`
+	Completion int64 `json:"completion_tokens"`
+	CostMicros int64 `json:"cost_micros"`
+	Errors     int64 `json:"errors"`
+
+	LastAt time.Time `json:"last_at,omitzero"`
+	// Budget is what this level sets, absent when it sets none. A group bounded
+	// only by its parent has no budget of its own, and reporting the parent's
+	// against each child would read as though each had that much.
+	Budget       *int64 `json:"budget_micros,omitempty"`
+	BudgetPeriod string `json:"budget_period,omitempty"`
+	// Fraction is spend over budget, absent for the same reason.
+	Fraction *float64 `json:"fraction,omitempty"`
+}
+
+func (a *API) usageByScope(w http.ResponseWriter, r *http.Request) {
+	if a.usage == nil {
+		writeError(w, http.StatusNotImplemented, "this control plane has no store, so nothing is recorded")
+		return
+	}
+	scope := store.Scope(r.URL.Query().Get("by"))
+	if scope == "" {
+		scope = store.ScopeTeam
+	}
+	since := 24 * time.Hour
+	if v := r.URL.Query().Get("since"); v != "" {
+		d, err := time.ParseDuration(v)
+		if err != nil || d <= 0 {
+			writeError(w, http.StatusBadRequest, "since must be a positive duration, like 24h")
+			return
+		}
+		since = d
+	}
+
+	rows, err := a.usage.UsageByScope(r.Context(), scope, time.Now().Add(-since))
+	if err != nil {
+		// The only failure a caller can act on is naming a level that is not
+		// one, which is what this mostly is.
+		writeError(w, http.StatusBadRequest, err.Error())
+		return
+	}
+	out := make([]ScopeUsageView, 0, len(rows))
+	for _, u := range rows {
+		v := ScopeUsageView{
+			Scope: string(u.Scope), ID: u.ID, Name: u.Name,
+			Keys: u.Keys, Requests: u.Requests, Prompt: u.Prompt,
+			Completion: u.Completion, CostMicros: u.CostMicros, Errors: u.Errors,
+			LastAt: u.LastAt, Budget: u.Budget, BudgetPeriod: string(u.BudgetPeriod),
+		}
+		if f, ok := u.Spent(); ok {
+			v.Fraction = &f
+		}
+		out = append(out, v)
+	}
+	writeJSON(w, http.StatusOK, out)
+}
+
+// Capacity is where measured capacity is kept. The store implements it.
+type Capacity interface {
+	PutCapacity(ctx context.Context, c store.StoredCapacity) error
+	Capacities(ctx context.Context) ([]store.StoredCapacity, error)
+}
+
+// SetCapacity supplies somewhere to keep benchmark results.
+func (a *API) SetCapacity(c Capacity) { a.capacity = c }
+
+// putCapacity records what a replica of a model was measured to serve.
+//
+// A PUT rather than a POST: re-running a benchmark replaces what was known
+// about that model on that profile. It is a better answer to the same question,
+// not another data point.
+func (a *API) putCapacity(w http.ResponseWriter, r *http.Request) {
+	if a.capacity == nil {
+		writeError(w, http.StatusNotImplemented,
+			"this control plane has no store, so a measurement recorded here would not survive a restart")
+		return
+	}
+	var c bench.Capacity
+	if err := json.NewDecoder(io.LimitReader(r.Body, 1<<20)).Decode(&c); err != nil {
+		writeError(w, http.StatusBadRequest, "the measurement is not valid JSON: "+err.Error())
+		return
+	}
+	c.Model = r.PathValue("name")
+	if c.Profile == "" {
+		writeError(w, http.StatusBadRequest,
+			"a measurement needs the profile it was taken on: the same model on CPU and on a GPU are different numbers")
+		return
+	}
+
+	// The model has to exist now, which catches a typo in the name — a
+	// measurement filed under a misspelling is worse than none, because it
+	// looks like the model was never benchmarked. It is not deleted when the
+	// model is, though: a benchmark costs real GPU time and outliving the
+	// catalog entry is the point.
+	m, ok := a.ctl.Catalog().Resolve(c.Model)
+	if !ok {
+		writeError(w, http.StatusNotFound, "no model named "+c.Model+" is in the catalog")
+		return
+	}
+	if !slices.ContainsFunc(m.Profiles, func(p catalog.Profile) bool { return p.Name == c.Profile }) {
+		writeError(w, http.StatusBadRequest,
+			"model "+c.Model+" has no profile named "+c.Profile)
+		return
+	}
+
+	body, err := json.Marshal(c)
+	if err != nil {
+		writeError(w, http.StatusInternalServerError, err.Error())
+		return
+	}
+	if err := a.capacity.PutCapacity(r.Context(), store.StoredCapacity{
+		Model: c.Model, Profile: c.Profile, Measurement: body,
+		MeasuredAt: c.MeasuredAt, MeasuredBy: r.Header.Get("X-Llmgw-Actor"),
+	}); err != nil {
+		writeError(w, http.StatusInternalServerError, err.Error())
+		return
+	}
+	writeJSON(w, http.StatusOK, c)
+}
+
+// CapacityView is one stored measurement, with who recorded it.
+type CapacityView struct {
+	bench.Capacity
+	MeasuredBy string `json:"measured_by,omitempty"`
+}
+
+func (a *API) listCapacity(w http.ResponseWriter, r *http.Request) {
+	if a.capacity == nil {
+		writeJSON(w, http.StatusOK, []CapacityView{})
+		return
+	}
+	rows, err := a.capacity.Capacities(r.Context())
+	if err != nil {
+		writeError(w, http.StatusInternalServerError, err.Error())
+		return
+	}
+	out := make([]CapacityView, 0, len(rows))
+	for _, row := range rows {
+		var c bench.Capacity
+		if err := json.Unmarshal(row.Measurement, &c); err != nil {
+			// Skipped rather than failing the listing: one row written by an
+			// older shape must not hide every other measurement.
+			continue
+		}
+		out = append(out, CapacityView{Capacity: c, MeasuredBy: row.MeasuredBy})
+	}
+	writeJSON(w, http.StatusOK, out)
 }
 
 // SetModels supplies the catalog-changing service.
@@ -305,6 +544,59 @@ func (a *API) addModel(w http.ResponseWriter, r *http.Request) {
 	writeJSON(w, http.StatusOK, map[string]string{"added": m.Name})
 }
 
+// ScaleRequest is how many replicas a model should run.
+type ScaleRequest struct {
+	Min int `json:"min"`
+	// Max omitted follows Min, which is what the catalog does for a model that
+	// declares only a minimum.
+	Max         int           `json:"max,omitempty"`
+	IdleTimeout time.Duration `json:"idle_timeout,omitempty"`
+}
+
+// ScaleResponse reports the scaling that is now in force.
+//
+// What is in force rather than what was asked for: the catalog fills in a max
+// that was left out, and a caller that echoed its own request would print a
+// number the cluster is not using.
+type ScaleResponse struct {
+	Model       string        `json:"model"`
+	Min         int           `json:"min"`
+	Max         int           `json:"max"`
+	IdleTimeout time.Duration `json:"idle_timeout,omitempty"`
+	OnDemand    bool          `json:"on_demand"`
+}
+
+func (a *API) scaleModel(w http.ResponseWriter, r *http.Request) {
+	if a.models == nil {
+		writeError(w, http.StatusNotImplemented, "this control plane has no store")
+		return
+	}
+	var req ScaleRequest
+	if err := json.NewDecoder(io.LimitReader(r.Body, 1<<16)).Decode(&req); err != nil {
+		writeError(w, http.StatusBadRequest, "the scaling request is not valid JSON: "+err.Error())
+		return
+	}
+	if req.Min < 0 || req.Max < 0 {
+		writeError(w, http.StatusBadRequest, "a replica count cannot be negative")
+		return
+	}
+	name := r.PathValue("name")
+	m, err := a.models.Scale(r.Context(), name,
+		catalog.Scaling{Min: req.Min, Max: req.Max, IdleTimeout: req.IdleTimeout},
+		r.Header.Get("X-Llmgw-Actor"))
+	if err != nil {
+		// Like addModel: every refusal here is the request being wrong — a
+		// model declared in configuration, one that was never added, a max
+		// below a min. None are server faults.
+		writeError(w, http.StatusBadRequest, err.Error())
+		return
+	}
+	writeJSON(w, http.StatusOK, ScaleResponse{
+		Model: m.Name, Min: m.Scaling.Min, Max: m.Scaling.Max,
+		IdleTimeout: m.Scaling.IdleTimeout, OnDemand: m.Scaling.OnDemand(),
+	})
+}
+
 func (a *API) removeModel(w http.ResponseWriter, r *http.Request) {
 	if a.models == nil {
 		writeError(w, http.StatusNotImplemented, "this control plane has no store")
@@ -409,3 +701,27 @@ func writeJSON(w http.ResponseWriter, code int, v any) {
 func writeError(w http.ResponseWriter, code int, msg string) {
 	writeJSON(w, code, map[string]string{"error": msg})
 }
+
+// profileChanged reports whether a replica is running a profile the catalog no
+// longer holds.
+//
+// A replica with no recorded fingerprint is not stale: that is every replica
+// adopted from an agent older than the field, and calling those stale would
+// mark a whole fleet as wrong on the first upgrade. Nor is one whose profile
+// has been deleted outright — the model or the profile going away is a
+// different thing, and the replica's own state already says it.
+func (a *API) profileChanged(model, profile, was string) bool {
+	if was == "" {
+		return false
+	}
+	m, ok := a.ctl.Catalog().Resolve(model)
+	if !ok {
+		return false
+	}
+	for _, p := range m.Profiles {
+		if p.Name == profile {
+			return p.Fingerprint() != was
+		}
+	}
+	return false
+}

internal/control/adminapi/api_test.go

@@ -9,6 +9,7 @@ import (
 	"net/http/httptest"
 	"strings"
 	"testing"
+	"time"
 
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control"
@@ -16,6 +17,7 @@ import (
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
 )
 
 // nullSink discards published endpoints; these tests are about the API.
@@ -252,9 +254,24 @@ type fakeModels struct {
 	added   []catalog.Model
 	actors  []string
 	removed []string
+	scaled  []catalog.Scaling
 	err     error
 }
 
+func (f *fakeModels) Scale(_ context.Context, name string, s catalog.Scaling, by string) (catalog.Model, error) {
+	if f.err != nil {
+		return catalog.Model{}, f.err
+	}
+	f.scaled = append(f.scaled, s)
+	f.actors = append(f.actors, by)
+	// Max following min is what the catalog does; the fake has to do it too or
+	// the handler test would pass while the real answer reported a zero.
+	if s.Max == 0 && s.Min > 0 {
+		s.Max = s.Min
+	}
+	return catalog.Model{Name: name, Scaling: s}, nil
+}
+
 func (f *fakeModels) Add(_ context.Context, m catalog.Model, by string) error {
 	if f.err != nil {
 		return f.err
@@ -353,3 +370,300 @@ func TestRemovingAModelOverTheAPI(t *testing.T) {
 		t.Errorf("removed = %v", fm.removed)
 	}
 }
+
+func TestScalingAModelOverTheAPI(t *testing.T) {
+	a, _ := setup(t, testModels())
+	fm := &fakeModels{}
+	a.SetModels(fm)
+	mux := http.NewServeMux()
+	a.Register(mux)
+
+	req := httptest.NewRequest(http.MethodPost, "/api/v1/models/m1/scale",
+		strings.NewReader(`{"min":3}`))
+	req.Header.Set("X-Llmgw-Actor", "ilker")
+	rec := httptest.NewRecorder()
+	mux.ServeHTTP(rec, req)
+
+	if rec.Code != http.StatusOK {
+		t.Fatalf("expected 200, got %d: %s", rec.Code, rec.Body)
+	}
+	var res ScaleResponse
+	if err := json.Unmarshal(rec.Body.Bytes(), &res); err != nil {
+		t.Fatal(err)
+	}
+	// The scaling in force, not the request echoed back: max follows min when
+	// it is left out, and a caller printing its own request would show 0.
+	if res.Min != 3 || res.Max != 3 {
+		t.Errorf("expected 3/3 in force, got %d/%d", res.Min, res.Max)
+	}
+	if len(fm.actors) != 1 || fm.actors[0] != "ilker" {
+		t.Errorf("the change should be attributed to whoever made it; got %v", fm.actors)
+	}
+}
+
+func TestANegativeReplicaCountIsRefused(t *testing.T) {
+	a, _ := setup(t, testModels())
+	fm := &fakeModels{}
+	a.SetModels(fm)
+	mux := http.NewServeMux()
+	a.Register(mux)
+
+	req := httptest.NewRequest(http.MethodPost, "/api/v1/models/m1/scale",
+		strings.NewReader(`{"min":-1}`))
+	rec := httptest.NewRecorder()
+	mux.ServeHTTP(rec, req)
+
+	if rec.Code != http.StatusBadRequest {
+		t.Errorf("expected 400, got %d", rec.Code)
+	}
+	if len(fm.scaled) != 0 {
+		t.Error("nothing should have reached the service")
+	}
+}
+
+type memCapacity struct{ rows []store.StoredCapacity }
+
+func (m *memCapacity) PutCapacity(_ context.Context, c store.StoredCapacity) error {
+	m.rows = append(m.rows, c)
+	return nil
+}
+
+func (m *memCapacity) Capacities(context.Context) ([]store.StoredCapacity, error) {
+	return m.rows, nil
+}
+
+func capacityAPI(t *testing.T) (*API, *memCapacity, *http.ServeMux) {
+	t.Helper()
+	a, _ := setup(t, testModels())
+	cap := &memCapacity{}
+	a.SetCapacity(cap)
+	mux := http.NewServeMux()
+	a.Register(mux)
+	return a, cap, mux
+}
+
+func TestAMeasurementIsFiledAgainstItsModelAndProfile(t *testing.T) {
+	_, cap, mux := capacityAPI(t)
+
+	body := `{"profile":"tp8","max_concurrency_at_slo":8,"suggested_target_waiting":3}`
+	req := httptest.NewRequest(http.MethodPut, "/api/v1/models/llama/capacity", strings.NewReader(body))
+	req.Header.Set("X-Llmgw-Actor", "ilker")
+	rec := httptest.NewRecorder()
+	mux.ServeHTTP(rec, req)
+
+	if rec.Code != http.StatusOK {
+		t.Fatalf("expected 200, got %d: %s", rec.Code, rec.Body)
+	}
+	if len(cap.rows) != 1 {
+		t.Fatalf("expected one row, got %d", len(cap.rows))
+	}
+	// The model comes from the path, not the body: a measurement filed under a
+	// name the caller made up would be worse than none.
+	if cap.rows[0].Model != "llama" || cap.rows[0].Profile != "tp8" {
+		t.Errorf("wrong key: %s/%s", cap.rows[0].Model, cap.rows[0].Profile)
+	}
+	if cap.rows[0].MeasuredBy != "ilker" {
+		t.Errorf("the measurement should say who took it; got %q", cap.rows[0].MeasuredBy)
+	}
+}
+
+func TestAMeasurementForAModelThatIsNotThereIsRefused(t *testing.T) {
+	_, cap, mux := capacityAPI(t)
+
+	req := httptest.NewRequest(http.MethodPut, "/api/v1/models/typo/capacity",
+		strings.NewReader(`{"profile":"tp8"}`))
+	rec := httptest.NewRecorder()
+	mux.ServeHTTP(rec, req)
+
+	// A typo in the name files the measurement somewhere nobody will look, and
+	// the model then looks as though it was never benchmarked.
+	if rec.Code != http.StatusNotFound {
+		t.Errorf("expected 404 for an unknown model, got %d", rec.Code)
+	}
+	if len(cap.rows) != 0 {
+		t.Error("nothing should have been stored")
+	}
+}
+
+func TestAMeasurementNeedsTheProfileItWasTakenOn(t *testing.T) {
+	_, cap, mux := capacityAPI(t)
+
+	req := httptest.NewRequest(http.MethodPut, "/api/v1/models/llama/capacity",
+		strings.NewReader(`{"max_concurrency_at_slo":8}`))
+	rec := httptest.NewRecorder()
+	mux.ServeHTTP(rec, req)
+
+	// Capacity belongs to the hardware as much as the weights. Without a
+	// profile there is nothing to compare a later run against.
+	if rec.Code != http.StatusBadRequest {
+		t.Errorf("expected 400 without a profile, got %d", rec.Code)
+	}
+	if len(cap.rows) != 0 {
+		t.Error("nothing should have been stored")
+	}
+}
+
+func TestAProfileTheModelDoesNotHaveIsRefused(t *testing.T) {
+	_, _, mux := capacityAPI(t)
+
+	req := httptest.NewRequest(http.MethodPut, "/api/v1/models/llama/capacity",
+		strings.NewReader(`{"profile":"not-a-profile"}`))
+	rec := httptest.NewRecorder()
+	mux.ServeHTTP(rec, req)
+
+	if rec.Code != http.StatusBadRequest {
+		t.Errorf("expected 400 for a profile the model does not have, got %d", rec.Code)
+	}
+}
+
+func TestCapacityWithoutAStoreIsAnEmptyListRatherThanAnError(t *testing.T) {
+	a, _ := setup(t, testModels())
+	mux := http.NewServeMux()
+	a.Register(mux)
+
+	rec := httptest.NewRecorder()
+	mux.ServeHTTP(rec, httptest.NewRequest(http.MethodGet, "/api/v1/capacity", nil))
+
+	// Nothing measured and no store look the same to a reader, and both mean
+	// "there is nothing here" — which is not a server fault.
+	if rec.Code != http.StatusOK {
+		t.Errorf("expected 200, got %d", rec.Code)
+	}
+	var rows []CapacityView
+	if err := json.Unmarshal(rec.Body.Bytes(), &rows); err != nil || len(rows) != 0 {
+		t.Errorf("expected an empty list, got %s", rec.Body)
+	}
+}
+
+type memUsage struct {
+	rows  []store.ScopeUsage
+	scope store.Scope
+	since time.Time
+	err   error
+}
+
+func (m *memUsage) UsageByScope(_ context.Context, s store.Scope, since time.Time) ([]store.ScopeUsage, error) {
+	m.scope, m.since = s, since
+	return m.rows, m.err
+}
+
+func usageAPI(t *testing.T, u *memUsage) *http.ServeMux {
+	t.Helper()
+	a, _ := setup(t, testModels())
+	a.SetUsage(u)
+	mux := http.NewServeMux()
+	a.Register(mux)
+	return mux
+}
+
+func TestUsageDefaultsToTeamsOverTheLastDay(t *testing.T) {
+	u := &memUsage{}
+	rec := httptest.NewRecorder()
+	usageAPI(t, u).ServeHTTP(rec, httptest.NewRequest(http.MethodGet, "/api/v1/usage", nil))
+
+	if rec.Code != http.StatusOK {
+		t.Fatalf("expected 200, got %d: %s", rec.Code, rec.Body)
+	}
+	// The team is the level a quota is usually written on, so it is the one
+	// somebody asking for "usage" with no argument means.
+	if u.scope != store.ScopeTeam {
+		t.Errorf("expected teams by default, got %q", u.scope)
+	}
+	if d := time.Since(u.since); d < 23*time.Hour || d > 25*time.Hour {
+		t.Errorf("expected roughly a day, got %v", d)
+	}
+}
+
+func TestUsageReportsTheFractionOnlyWhenThereIsABudget(t *testing.T) {
+	budget := int64(2000)
+	u := &memUsage{rows: []store.ScopeUsage{
+		{Scope: store.ScopeTeam, ID: "t1", Name: "platform", CostMicros: 1000,
+			Budget: &budget, BudgetPeriod: store.PeriodMonth},
+		{Scope: store.ScopeTeam, ID: "t2", Name: "research", CostMicros: 100},
+	}}
+	rec := httptest.NewRecorder()
+	usageAPI(t, u).ServeHTTP(rec, httptest.NewRequest(http.MethodGet, "/api/v1/usage", nil))
+
+	var got []ScopeUsageView
+	if err := json.Unmarshal(rec.Body.Bytes(), &got); err != nil {
+		t.Fatal(err)
+	}
+	if got[0].Fraction == nil || *got[0].Fraction != 0.5 {
+		t.Errorf("expected half the budget used, got %v", got[0].Fraction)
+	}
+	// A team bounded only by its organization has no budget of its own; a
+	// fraction of nothing would have to invent a denominator.
+	if got[1].Fraction != nil || got[1].Budget != nil {
+		t.Errorf("a group with no budget should report neither, got %+v", got[1])
+	}
+}
+
+func TestAnUnknownGroupingIsTheCallersMistake(t *testing.T) {
+	u := &memUsage{err: errors.New(`usage cannot be grouped by "model"`)}
+	rec := httptest.NewRecorder()
+	usageAPI(t, u).ServeHTTP(rec,
+		httptest.NewRequest(http.MethodGet, "/api/v1/usage?by=model", nil))
+
+	if rec.Code != http.StatusBadRequest {
+		t.Errorf("expected 400, got %d", rec.Code)
+	}
+}
+
+func TestABadSinceIsRefusedBeforeTheQuery(t *testing.T) {
+	u := &memUsage{}
+	rec := httptest.NewRecorder()
+	usageAPI(t, u).ServeHTTP(rec,
+		httptest.NewRequest(http.MethodGet, "/api/v1/usage?since=lastweek", nil))
+
+	if rec.Code != http.StatusBadRequest {
+		t.Errorf("expected 400, got %d", rec.Code)
+	}
+	if !u.since.IsZero() {
+		t.Error("nothing should have reached the store")
+	}
+}
+
+func TestAReplicaRunningAnEditedProfileIsMarkedStale(t *testing.T) {
+	models := testModels()
+	a, ctl := setup(t, models)
+
+	// Placed from the profile as it was, then the profile is edited and keeps
+	// its name — which is what makes the two indistinguishable without a
+	// fingerprint.
+	was := models[0].Profiles[0].Fingerprint()
+	edited := models[0]
+	edited.Profiles[0].MinMemoryGB = 64
+	built, err := catalog.New([]catalog.Model{edited})
+	if err != nil {
+		t.Fatal(err)
+	}
+	ctl.ReplaceCatalog(t.Context(), built)
+
+	if !a.profileChanged("llama", "tp8", was) {
+		t.Error("an edited profile should read as changed")
+	}
+	if a.profileChanged("llama", "tp8", edited.Profiles[0].Fingerprint()) {
+		t.Error("the profile in force is not a change")
+	}
+}
+
+func TestAReplicaWithNoRecordedProfileIsNotStale(t *testing.T) {
+	a, _ := setup(t, testModels())
+	// Every replica adopted from an agent older than the field. Marking those
+	// stale would call a whole fleet wrong on the first upgrade.
+	if a.profileChanged("llama", "tp8", "") {
+		t.Error("unknown is not stale")
+	}
+}
+
+func TestAProfileThatIsGoneIsNotReportedAsStale(t *testing.T) {
+	a, _ := setup(t, testModels())
+	// The model or the profile disappearing is a different thing, and the
+	// replica's own state already says it.
+	if a.profileChanged("llama", "no-such-profile", "abc") {
+		t.Error("a missing profile is not a changed one")
+	}
+	if a.profileChanged("no-such-model", "tp8", "abc") {
+		t.Error("a missing model is not a changed profile")
+	}
+}

internal/control/agentsvc/convert.go

@@ -23,8 +23,15 @@ func NodeFromRegister(r *llmgwv1.Register) inventory.Node {
 		State:          inventory.Ready,
 		AgentVersion:   r.GetAgentVersion(),
 		EngineVersions: r.GetEngineVersions(),
+		EnginesProbed:  r.GetEnginesProbed(),
 		MemoryTotalMB:  int(r.GetMemoryTotalMb()),
 	}
+	if h := r.GetHost(); h != nil {
+		n.Host = discovery.Host{
+			OSID: h.GetOsId(), OSVersion: h.GetOsVersion(), OSCodename: h.GetOsCodename(),
+			Arch: h.GetArch(), CPUFlags: h.GetCpuFlags(), NVIDIADriver: h.GetNvidiaDriver(),
+		}
+	}
 	for _, w := range r.GetWeights() {
 		n.Weights = append(n.Weights, discovery.LocalWeights{
 			Path: w.GetPath(), SizeBytes: w.GetSizeBytes(), Shared: w.GetShared(),
@@ -136,6 +143,7 @@ func AdoptedFromRegister(nodeID string, shards []*llmgwv1.RunningShard) []reconc
 			Profile:    s.GetProfile(),
 			Pool:       s.GetPool(),
 			Fabric:     s.GetFabric(),
+			ProfileFP:  s.GetProfileFp(),
 		}
 		if a.Rank == 0 && s.GetLeaderAddr() != "" {
 			a.Endpoint = fmt.Sprintf("http://%s:%d", s.GetLeaderAddr(), s.GetHttpPort())
@@ -169,6 +177,7 @@ func StartShardFromCommand(cmd reconciler.ShardCommand) *llmgwv1.StartShard {
 		StartupTimeoutMs: cmd.StartupTimeout.Milliseconds(),
 		Pool:             cmd.Pool,
 		Fabric:           cmd.Fabric,
+		ProfileFp:        cmd.ProfileFP,
 	}
 }
 
@@ -188,6 +197,7 @@ func ShardSpecFromStart(m *llmgwv1.StartShard) (engines.ShardSpec, time.Duration
 		HTTPPort:      int(m.GetHttpPort()),
 		Pool:          m.GetPool(),
 		Fabric:        m.GetFabric(),
+		ProfileFP:     m.GetProfileFp(),
 		Profile: engines.Profile{
 			Name: p.GetName(), Engine: p.GetEngine(), GPUClass: p.GetGpuClass(),
 			TP: int(p.GetTp()), PP: int(p.GetPp()),

internal/control/agentsvc/heartbeat_test.go

@@ -1,6 +1,7 @@
 package agentsvc
 
 import (
+	"log/slog"
 	"testing"
 
 	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
@@ -11,7 +12,8 @@ import (
 func nodeWith(s *Service, n inventory.Node) { s.inv.Upsert(n) }
 
 func newTestService() *Service {
-	return &Service{inv: inventory.NewStore(), conns: map[string]*conn{}}
+	return &Service{inv: inventory.NewStore(), conns: map[string]*conn{},
+		log: slog.New(slog.DiscardHandler)}
 }
 
 func TestDiskIsRefreshedWhileTheAgentRuns(t *testing.T) {
@@ -112,3 +114,44 @@ func TestAHeartbeatAboutNothingChangesNothing(t *testing.T) {
 		t.Errorf("disk changed: %d", free)
 	}
 }
+
+func TestAHeartbeatCarriesMemoryAddedToARunningNode(t *testing.T) {
+	// Read once at registration, a node that was given 32 GB went on looking
+	// like the 7 GB machine it had been, and every model that needed the
+	// difference stayed unplaced until somebody restarted its agent.
+	s := newTestService()
+	nodeWith(s, inventory.Node{ID: "n1", State: inventory.Ready, MemoryTotalMB: 7 * 1024})
+
+	s.applyHeartbeat("n1", &llmgwv1.Heartbeat{NodeId: "n1", MemoryTotalMb: 32 * 1024})
+
+	if n, _ := s.inv.Get("n1"); n.MemoryTotalMB != 32*1024 {
+		t.Errorf("memory is %d MB, expected the 32 GB the node now has", n.MemoryTotalMB)
+	}
+
+	// Zero is an agent that could not read it, not a node that lost its RAM.
+	s.applyHeartbeat("n1", &llmgwv1.Heartbeat{NodeId: "n1"})
+	if n, _ := s.inv.Get("n1"); n.MemoryTotalMB != 32*1024 {
+		t.Errorf("a heartbeat without memory erased it: %d", n.MemoryTotalMB)
+	}
+}
+
+func TestAHeartbeatCarriesAnEngineInstalledAfterRegistration(t *testing.T) {
+	s := newTestService()
+	nodeWith(s, inventory.Node{ID: "n1", State: inventory.Ready,
+		EngineVersions: map[string]string{"llamacpp": "unknown"}})
+
+	// Unchanged engines are not resent, and silence must not clear them.
+	s.applyHeartbeat("n1", &llmgwv1.Heartbeat{NodeId: "n1", MemoryTotalMb: 1024})
+	if n, _ := s.inv.Get("n1"); len(n.EngineVersions) != 1 {
+		t.Fatalf("engines changed without being reported: %v", n.EngineVersions)
+	}
+
+	s.applyHeartbeat("n1", &llmgwv1.Heartbeat{
+		NodeId: "n1", EnginesChanged: true,
+		EngineVersions: map[string]string{"llamacpp": "unknown", "vllm": "unknown"},
+	})
+	n, _ := s.inv.Get("n1")
+	if _, ok := n.EngineVersions["vllm"]; !ok {
+		t.Errorf("the newly installed engine is not known: %v", n.EngineVersions)
+	}
+}

internal/control/agentsvc/service.go

@@ -6,6 +6,7 @@ import (
 	"fmt"
 	"io"
 	"log/slog"
+	"maps"
 	"net/http"
 	"slices"
 	"sync"
@@ -24,6 +25,10 @@ import (
 // agents report they are already running. The reconciler implements it.
 type StatusSink interface {
 	OnShardStatus(replicaID, shardID string, state supervisor.State, errMsg string)
+	// OnFetchProgress is separate because it is not a transition: the shard is
+	// still starting, and treating each report as a state change would reset
+	// the timers that decide when a slow start has become a failed one.
+	OnFetchProgress(replicaID, shardID, source string, done, total int64)
 	Adopt(nodeID string, shards []reconciler.AdoptedShard) (unclaimed []reconciler.AdoptedShard)
 }
 
@@ -31,6 +36,9 @@ type StatusSink interface {
 // time the reconciler's Dispatcher: it holds the open streams, so it is the
 // only thing that can deliver a command to a node.
 type Service struct {
+	// leader gates whether this process accepts agent streams at all.
+	leader Leader
+
 	inv    *inventory.Store
 	status StatusSink
 	log    *slog.Logger
@@ -41,8 +49,14 @@ type Service struct {
 
 	mu    sync.RWMutex
 	conns map[string]*conn
+
+	// engines is what every node is told to install, set once at start.
+	engines map[string]string
 }
 
+// SetEngines sets the engines every node installs, by name and version.
+func (s *Service) SetEngines(e map[string]string) { s.engines = maps.Clone(e) }
+
 // conn is one agent's open stream.
 type conn struct {
 	nodeID string
@@ -86,6 +100,19 @@ func (s *Service) Connect(
 	ctx context.Context,
 	stream *connect.BidiStream[llmgwv1.AgentMessage, llmgwv1.ControlMessage],
 ) error {
+	// A standby must not hold an agent's stream. Commands travel down the
+	// stream the agent opened, so whichever control plane holds it is the only
+	// one that can start or stop anything on that node — and if that is not
+	// the leader, the leader's placements fail with "no agent connected" for a
+	// node that is plainly connected.
+	//
+	// Refused before the opening message is read, so the agent spends nothing
+	// on a connection that cannot serve it.
+	if s.leader != nil && !s.leader.IsLeader() {
+		return connect.NewError(connect.CodeUnavailable,
+			errors.New("this control plane is a standby; connect to the leader"))
+	}
+
 	first, err := stream.Receive()
 	if err != nil {
 		return connect.NewError(connect.CodeInvalidArgument, fmt.Errorf("no opening message: %w", err))
@@ -136,6 +163,21 @@ func (s *Service) Connect(
 		return err
 	}
 
+	// What this node should have installed. Sent on every registration, so a
+	// node rebuilt from nothing is told the same as one that has been here a
+	// month, and a changed version reaches every node on its next connect.
+	if s.engines != nil {
+		p := &llmgwv1.Provision{}
+		for _, e := range slices.Sorted(maps.Keys(s.engines)) {
+			p.Engines = append(p.Engines, &llmgwv1.EngineWant{Engine: e, Version: s.engines[e]})
+		}
+		if err := stream.Send(&llmgwv1.ControlMessage{
+			Msg: &llmgwv1.ControlMessage_Provision{Provision: p},
+		}); err != nil {
+			return err
+		}
+	}
+
 	// Recover replica state from what this node is already running, before any
 	// reconciliation pass can conclude the cluster is empty and place
 	// duplicates (plan §6.5).
@@ -217,6 +259,14 @@ func (s *Service) handle(nodeID string, msg *llmgwv1.AgentMessage) {
 		s.applyHeartbeat(nodeID, m.Heartbeat)
 
 	case *llmgwv1.AgentMessage_Shard:
+		// Progress rather than a transition. Falling through to OnShardStatus
+		// would re-mark the shard as starting on every chunk, which is how a
+		// download that takes an hour would keep a start timer at zero.
+		if f := m.Shard.GetFetch(); f != nil {
+			s.status.OnFetchProgress(m.Shard.GetReplicaId(), m.Shard.GetShardId(),
+				f.GetSource(), f.GetBytesDone(), f.GetBytesTotal())
+			return
+		}
 		st := StateFromWire(m.Shard.GetState())
 		s.status.OnShardStatus(m.Shard.GetReplicaId(), m.Shard.GetShardId(), st, m.Shard.GetError())
 		if m.Shard.GetState() == llmgwv1.ShardStatus_STATE_FAILED {
@@ -241,7 +291,10 @@ func (s *Service) applyHeartbeat(nodeID string, hb *llmgwv1.Heartbeat) {
 	used := hb.GetGpuMemoryUsedMb()
 	problems := hb.GetNewProblems()
 	storage := hb.GetStorage()
-	if len(used) == 0 && len(problems) == 0 && len(storage) == 0 && !hb.GetWeightsChanged() {
+	memMB := int(hb.GetMemoryTotalMb())
+	if len(used) == 0 && len(problems) == 0 && len(storage) == 0 &&
+		!hb.GetWeightsChanged() && memMB == 0 && !hb.GetEnginesChanged() &&
+		!hb.GetProvisionChanged() {
 		return
 	}
 
@@ -288,6 +341,50 @@ func (s *Service) applyHeartbeat(nodeID string, hb *llmgwv1.Heartbeat) {
 			})
 		}
 	}
+
+	// RAM and engines change under a running agent too: memory added to a VM,
+	// an engine installed this afternoon. Said in the log when they do, because
+	// "the node now qualifies" is the answer to why a model placed an hour after
+	// it was asked for, and "the node no longer does" is the answer to why one
+	// stopped placing.
+	if memMB > 0 && memMB != n.MemoryTotalMB {
+		s.log.Info("node memory changed", "node", nodeID,
+			"from_mb", n.MemoryTotalMB, "to_mb", memMB)
+		n.MemoryTotalMB = memMB
+	}
+	if hb.GetEnginesChanged() {
+		vers := maps.Clone(hb.GetEngineVersions())
+		if !maps.Equal(vers, n.EngineVersions) {
+			s.log.Info("node engines changed", "node", nodeID,
+				"from", slices.Sorted(maps.Keys(n.EngineVersions)),
+				"to", slices.Sorted(maps.Keys(vers)))
+		}
+		n.EngineVersions = vers
+	}
+	if hb.GetProvisionChanged() {
+		prev := map[string]inventory.EngineInstall{}
+		for _, p := range n.Provision {
+			prev[p.Engine] = p
+		}
+		n.Provision = n.Provision[:0]
+		for _, p := range hb.GetProvision() {
+			e := inventory.EngineInstall{
+				Engine: p.GetEngine(), Version: p.GetVersion(), Variant: p.GetVariant(),
+				State: p.GetState(), Detail: p.GetDetail(),
+			}
+			// A state change is worth a line; a download's progress is not.
+			if old, ok := prev[e.Engine]; !ok || old.State != e.State {
+				lvl := slog.LevelInfo
+				if e.State == "failed" || e.State == "unsupported" {
+					lvl = slog.LevelWarn
+				}
+				s.log.Log(context.Background(), lvl, "engine installation on a node",
+					"node", nodeID, "engine", e.Engine, "version", e.Version,
+					"variant", e.Variant, "state", e.State, "detail", e.Detail)
+			}
+			n.Provision = append(n.Provision, e)
+		}
+	}
 	s.inv.Upsert(n)
 }
 
@@ -345,6 +442,16 @@ func (s *Service) ConnectedNodes() []string {
 
 // --- Dispatcher ---
 
+// Leader reports whether this process is the one that should be holding agent
+// streams.
+type Leader interface{ IsLeader() bool }
+
+// SetLeadership makes this service refuse agents while it is a standby.
+//
+// Nil accepts them unconditionally, which is what a single control plane wants
+// and what every deployment before leadership had.
+func (s *Service) SetLeadership(l Leader) { s.leader = l }
+
 // ErrNotConnected means no agent stream is open for that node.
 var ErrNotConnected = errors.New("no agent connected")
 

internal/control/catalog/catalog.go

@@ -3,6 +3,9 @@
 package catalog
 
 import (
+	"crypto/sha256"
+	"encoding/hex"
+	"encoding/json"
 	"fmt"
 	"slices"
 	"strings"
@@ -123,6 +126,17 @@ func cpuCapable(engine string) bool { return engine == "llamacpp" }
 type Model struct {
 	Name    string   `json:"name,omitempty" koanf:"name"`
 	Aliases []string `json:"aliases,omitempty" koanf:"aliases"`
+	// Description is what this model is for, in a line.
+	//
+	// Written by whoever added it, because nobody else can say it. A registry
+	// offers a pipeline tag and a wall of README, and neither answers the
+	// question a person picking from a list is asking: which of these nine
+	// names should I send this request to.
+	//
+	// It reaches clients through /v1/models, which is the only view most of
+	// them ever see. A gateway serving nine models by name alone makes that
+	// choice by guesswork.
+	Description string `json:"description,omitempty" koanf:"description"`
 	// Source is the weights path or a HuggingFace id.
 	Source    string    `json:"source,omitempty" koanf:"source"`
 	Profiles  []Profile `json:"profiles,omitempty" koanf:"profiles"`
@@ -433,3 +447,24 @@ func (m Model) Simulated() bool {
 	}
 	return true
 }
+
+// Fingerprint identifies a profile's contents.
+//
+// The name is not enough. A profile can be edited — a memory requirement
+// raised, a node label added, an engine argument changed — and keep its name,
+// so a replica running the old definition is indistinguishable from one
+// running the new. The reconciler does not move a running replica when its
+// profile changes, which is right: churning a replica that is serving costs
+// more than the imperfection. What was missing is any way to see that it had
+// happened, and the answer to "why is this one on the wrong node" was in
+// somebody's memory of what they edited an hour ago.
+func (p Profile) Fingerprint() string {
+	b, err := json.Marshal(p)
+	if err != nil {
+		// A profile that will not marshal cannot be compared, and saying so by
+		// never matching is better than saying two differing ones are the same.
+		return "unmarshalable"
+	}
+	sum := sha256.Sum256(b)
+	return hex.EncodeToString(sum[:8])
+}

internal/control/control.go

@@ -75,6 +75,7 @@ type Controller struct {
 	inv   *inventory.Store
 	cat   *catalog.Live
 	rec   *reconciler.Reconciler
+	lead  *Leadership
 	disp  reconciler.Dispatcher
 	sink  EndpointSink
 	usage UsageSource
@@ -172,6 +173,22 @@ func (c *Controller) Reconciler() *reconciler.Reconciler { return c.rec }
 // Catalog exposes the model catalog.
 func (c *Controller) Catalog() *catalog.Catalog { return c.cat.Load() }
 
+// SetLeadership makes this control plane compete for the right to place work,
+// instead of assuming it is the only one.
+//
+// Wired after construction because deciding leadership needs a store, and the
+// store is opened by the router. Left unset, this process places
+// unconditionally — which is what a single control plane wants, and what every
+// deployment that existed before this had.
+func (c *Controller) SetLeadership(l *Leadership) {
+	c.lead = l
+	c.rec.SetLeadership(l)
+}
+
+// IsLeader reports whether this process is placing work. True when leadership
+// was never configured, because then there is nobody to compete with.
+func (c *Controller) IsLeader() bool { return c.lead == nil || c.lead.IsLeader() }
+
 // WeightRemover can ask a node to delete cached model files.
 //
 // Separate from Dispatcher because it is not part of scheduling and not every
@@ -189,6 +206,9 @@ func (c *Controller) WeightRemover() WeightRemover {
 
 // Start runs the reconciler and begins publishing endpoints.
 func (c *Controller) Start(ctx context.Context) {
+	if c.lead != nil {
+		go c.lead.Run(ctx)
+	}
 	c.rec.Start(ctx)
 
 	go func() {

internal/control/inventory/inventory.go

@@ -102,6 +102,24 @@ type Node struct {
 	// version decides which metric names and CLI flags apply (plan §14).
 	AgentVersion   string            `json:"agent_version,omitempty"`
 	EngineVersions map[string]string `json:"engine_versions,omitempty"`
+	// EnginesProbed says EngineVersions is an answer, so an empty map means
+	// "has none" rather than "an agent too old to say".
+	EnginesProbed bool `json:"engines_probed,omitempty"`
+
+	// Host is the operating system and processor, as the agent reads them.
+	Host discovery.Host `json:"host"`
+	// Provision is where installing each wanted engine stands on this node.
+	Provision []EngineInstall `json:"provision,omitempty"`
+}
+
+// EngineInstall is where one engine's installation stands on a node.
+type EngineInstall struct {
+	Engine  string `json:"engine"`
+	Version string `json:"version"`
+	Variant string `json:"variant,omitempty"`
+	// State is "installing", "ready", "failed" or "unsupported".
+	State  string `json:"state"`
+	Detail string `json:"detail,omitempty"`
 }
 
 // HasFabric reports whether the node is attached to a fabric.
@@ -147,6 +165,17 @@ type Allocation struct {
 	ShardID    string
 	ReplicaID  string
 	GPUIndices []int
+	// MemoryGB is the host RAM this shard's profile asked for.
+	//
+	// Recorded for the same reason the GPU indices are: it was claimed, and
+	// placement has to stop handing it out twice. Without it a node's memory
+	// was compared against its total on every decision, so two models each
+	// declaring most of a machine were both placed on it — and the second one
+	// is what the kernel kills.
+	//
+	// Zero for a profile that declared nothing, which constrains nothing and
+	// is counted as nothing.
+	MemoryGB int
 }
 
 // Store holds the cluster inventory. It is safe for concurrent use.

internal/control/leadership.go

@@ -0,0 +1,182 @@
+package control
+
+import (
+	"context"
+	"crypto/rand"
+	"encoding/hex"
+	"errors"
+	"log/slog"
+	"os"
+	"sync/atomic"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// LeaseStore is the part of the store leadership needs.
+//
+// An interface so the controller does not take a dependency on a concrete
+// database for one feature, and so a deployment with no store at all — a
+// router-only run, or a single control plane on a static configuration — keeps
+// working with nothing to configure.
+type LeaseStore interface {
+	AcquireLeadership(ctx context.Context, role, owner string, ttl time.Duration) (store.Lease, error)
+	ReleaseLeadership(ctx context.Context, role, owner string) error
+}
+
+// controlRole is the one thing there is to be leader of: placing work.
+const controlRole = "control-plane"
+
+// Leadership decides whether this process is the one that places work.
+//
+// Two control planes against one store both reconcile: both see a model
+// wanting two replicas and none running, and both place. The result is twice
+// the replicas, each holding GPUs the other believes are free, and nothing
+// crashes to say so.
+//
+// A standby is not idle — it serves its own routing and keeps its connection
+// to the store — it simply does not place, evict or drain. That makes running
+// a second one safe rather than clever, which is the property worth having
+// first: the common failure is not a planned failover, it is somebody starting
+// a second control plane by accident.
+type Leadership struct {
+	store LeaseStore
+	log   *slog.Logger
+	// owner identifies this process. Random rather than the hostname, so a
+	// restart on the same machine cannot be fooled by its own stale lease.
+	owner string
+	ttl   time.Duration
+	// renew is how often the holder pushes the expiry out. Well under ttl, so
+	// one slow query does not cost the leadership.
+	renew time.Duration
+
+	leader atomic.Bool
+}
+
+// NewLeadership prepares leadership over a store.
+//
+// A nil store means there is nobody to coordinate with, and this process is
+// the leader by definition — which is every single-control-plane deployment,
+// and must stay the case without anyone opting in.
+func NewLeadership(st LeaseStore, ttl time.Duration, log *slog.Logger) *Leadership {
+	if log == nil {
+		log = slog.Default()
+	}
+	if ttl <= 0 {
+		// Long enough to survive a slow database or a brief network stall,
+		// short enough that a standby takes over before anyone has finished
+		// reading the alert.
+		ttl = 15 * time.Second
+	}
+	l := &Leadership{
+		store: st,
+		log:   log,
+		owner: newOwnerID(),
+		ttl:   ttl,
+		renew: ttl / 3,
+	}
+	if st == nil {
+		l.leader.Store(true)
+	}
+	return l
+}
+
+// IsLeader reports whether this process should be placing work.
+func (l *Leadership) IsLeader() bool { return l.leader.Load() }
+
+// Owner is this process's id, for logs and for ctl.
+func (l *Leadership) Owner() string { return l.owner }
+
+// Run keeps the lease for as long as it can hold it, and keeps trying when it
+// cannot.
+//
+// A standby polls at the same interval the leader renews at, so a lapsed lease
+// is picked up within about a third of its life rather than at the end of it.
+func (l *Leadership) Run(ctx context.Context) {
+	if l.store == nil {
+		return
+	}
+	t := time.NewTicker(l.renew)
+	defer t.Stop()
+
+	l.attempt(ctx)
+	for {
+		select {
+		case <-ctx.Done():
+			l.step(ctx)
+			return
+		case <-t.C:
+			l.attempt(ctx)
+		}
+	}
+}
+
+// attempt takes or renews the lease, logging only transitions.
+//
+// Only transitions, because this runs every few seconds for the life of the
+// process: a line per attempt would bury everything else, and the thing an
+// operator needs to see is the moment leadership moved.
+func (l *Leadership) attempt(ctx context.Context) {
+	lease, err := l.store.AcquireLeadership(ctx, controlRole, l.owner, l.ttl)
+	switch {
+	case err == nil:
+		if l.leader.CompareAndSwap(false, true) {
+			l.log.Info("this control plane is now placing work", "owner", l.owner)
+		}
+	case errors.Is(err, store.ErrNotLeader):
+		if l.leader.CompareAndSwap(true, false) {
+			// Losing a lease that was held is worth a warning: it means this
+			// process was too slow to renew, and whatever made it slow is
+			// still true.
+			l.log.Warn("lost leadership; standing by",
+				"owner", l.owner, "now", lease.Owner)
+		} else if l.leader.Load() == false && lease.Owner != "" {
+			l.log.Debug("standing by", "leader", lease.Owner,
+				"until", lease.ExpiresAt.Format(time.RFC3339))
+		}
+	default:
+		// A store that cannot be reached is not evidence that somebody else
+		// is leading. Keeping the claim is the safer of two bad options: two
+		// leaders is worse than one, but no leader at all means nothing is
+		// placed and nothing is evicted, and a brief database blip should not
+		// stop a cluster from healing itself.
+		l.log.Warn("could not reach the store to renew leadership; keeping it for now",
+			"err", err)
+	}
+}
+
+// step gives the lease up on the way out, so a standby takes over in seconds
+// rather than waiting out the clock.
+func (l *Leadership) step(ctx context.Context) {
+	if !l.leader.Load() {
+		return
+	}
+	// A fresh context: the one that just ended is the reason this is running.
+	ctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 3*time.Second)
+	defer cancel()
+	if err := l.store.ReleaseLeadership(ctx, controlRole, l.owner); err != nil {
+		l.log.Warn("could not release leadership; a standby will wait it out", "err", err)
+		return
+	}
+	l.log.Info("released leadership", "owner", l.owner)
+}
+
+// newOwnerID is random, not the hostname.
+//
+// A hostname would let a restarted process match its own stale lease and
+// believe it had been leading all along — including through whatever gap made
+// it restart, during which another process may have taken over and started
+// placing.
+func newOwnerID() string {
+	var b [8]byte
+	if _, err := rand.Read(b[:]); err != nil {
+		// Only reached if the system has no entropy, where a pid and the time
+		// is still better than a fixed string.
+		return "ctl-" + hex.EncodeToString([]byte(time.Now().Format(time.RFC3339Nano)))[:16]
+	}
+	host, _ := os.Hostname()
+	if host == "" {
+		host = "ctl"
+	}
+	return host + "-" + hex.EncodeToString(b[:])
+}

internal/control/modelsvc/modelsvc.go

@@ -164,6 +164,73 @@ func (s *Service) Remove(ctx context.Context, name string) error {
 	return nil
 }
 
+// Scale changes how many replicas a model runs, without a restart.
+//
+// Only the scaling block moves. Everything else about the model — its source,
+// its profiles, the engine arguments somebody tuned — is read back from what
+// was stored and written out again untouched, so changing a number cannot
+// quietly become a redefinition.
+//
+// A model declared in configuration is refused, for the same reason Remove
+// refuses one: the YAML is read on every start, so the new number would hold
+// until the next restart and then silently revert. A cluster that disagrees
+// with its own config file is worse than an error message saying where to edit.
+func (s *Service) Scale(ctx context.Context, name string, sc catalog.Scaling, by string) (catalog.Model, error) {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+
+	if s.Declared(name) {
+		return catalog.Model{}, fmt.Errorf(
+			"model %q is declared in configuration; change its scaling there, or it reverts on the next restart", name)
+	}
+
+	stored, err := s.storedModels(ctx)
+	if err != nil {
+		return catalog.Model{}, err
+	}
+
+	next := append([]catalog.Model{}, s.declared...)
+	found := false
+	for _, e := range stored {
+		if e.Name == name {
+			e.Scaling = sc
+			found = true
+		}
+		next = append(next, e)
+	}
+	if !found {
+		return catalog.Model{}, fmt.Errorf("model %q was not added at runtime", name)
+	}
+
+	// Built before anything is written, like Add: catalog.New is what applies
+	// the defaults (max follows min when it is left out) and what refuses a
+	// max below a min, so the model that comes back is the one that will run
+	// rather than the one that was asked for.
+	built, err := catalog.New(next)
+	if err != nil {
+		return catalog.Model{}, fmt.Errorf("scaling %q that way would make the catalog invalid: %w", name, err)
+	}
+	final, ok := built.Resolve(name)
+	if !ok {
+		return catalog.Model{}, fmt.Errorf("model %q vanished from the catalog it was just built into", name)
+	}
+
+	def, err := json.Marshal(final)
+	if err != nil {
+		return catalog.Model{}, err
+	}
+	if err := s.store.PutModel(ctx, store.StoredModel{
+		Name: name, Definition: def, CreatedAt: time.Now(), CreatedBy: by,
+	}); err != nil {
+		return catalog.Model{}, err
+	}
+
+	s.apply.ReplaceCatalog(ctx, built)
+	s.log.Info("model scaled", "model", name,
+		"min", final.Scaling.Min, "max", final.Scaling.Max, "by", by)
+	return final, nil
+}
+
 // Load builds the catalog at startup: configuration plus what was stored.
 //
 // A stored model that no longer validates — its engine left the build, a

internal/control/modelsvc/modelsvc_test.go

@@ -194,3 +194,146 @@ func TestAStoredModelThatNoLongerValidatesIsSkippedAlone(t *testing.T) {
 		t.Error("a valid stored model was dropped because another was invalid")
 	}
 }
+
+func TestScalingChangesTheCountAndSurvivesARestart(t *testing.T) {
+	s, st, rec := newService(t)
+	if err := s.Add(t.Context(), model("added"), "ilker"); err != nil {
+		t.Fatal(err)
+	}
+
+	got, err := s.Scale(t.Context(), "added", catalog.Scaling{Min: 3}, "ilker")
+	if err != nil {
+		t.Fatal(err)
+	}
+	// Max follows min when it is left out, and the answer reports what the
+	// catalog settled on rather than what was asked for — a caller echoing its
+	// own request would print a max of zero.
+	if got.Scaling.Min != 3 || got.Scaling.Max != 3 {
+		t.Errorf("expected 3/3 in force, got %d/%d", got.Scaling.Min, got.Scaling.Max)
+	}
+	if m, _ := rec.last.Resolve("added"); m.Scaling.Min != 3 {
+		t.Errorf("the applied catalog still says %d", m.Scaling.Min)
+	}
+
+	s2, _, _ := newService(t)
+	s2.store = st
+	built, err := s2.Load(t.Context())
+	if err != nil {
+		t.Fatal(err)
+	}
+	if m, _ := built.Resolve("added"); m.Scaling.Min != 3 {
+		t.Errorf("the new count did not survive a restart; got %d", m.Scaling.Min)
+	}
+}
+
+func TestScalingChangesNothingButTheCount(t *testing.T) {
+	s, _, rec := newService(t)
+	m := model("added")
+	m.Aliases = []string{"a"}
+	m.Profiles[0].Args = []string{"--ctx-size", "16384"}
+	if err := s.Add(t.Context(), m, "ilker"); err != nil {
+		t.Fatal(err)
+	}
+
+	if _, err := s.Scale(t.Context(), "added", catalog.Scaling{Min: 2}, "ilker"); err != nil {
+		t.Fatal(err)
+	}
+
+	// Changing a number must not quietly become a redefinition: the source and
+	// the engine arguments somebody tuned are read back and written out again.
+	got, _ := rec.last.Resolve("added")
+	if got.Source != "/models/added" {
+		t.Errorf("the source changed to %q", got.Source)
+	}
+	if len(got.Profiles) != 1 || strings.Join(got.Profiles[0].Args, " ") != "--ctx-size 16384" {
+		t.Errorf("the profile arguments were lost: %+v", got.Profiles)
+	}
+	if len(got.Aliases) != 1 || got.Aliases[0] != "a" {
+		t.Errorf("the aliases were lost: %v", got.Aliases)
+	}
+}
+
+func TestADeclaredModelCannotBeScaledHere(t *testing.T) {
+	s, _, _ := newService(t, model("declared"))
+
+	_, err := s.Scale(t.Context(), "declared", catalog.Scaling{Min: 5}, "ilker")
+
+	// The YAML is read on every start, so the number would hold until the next
+	// restart and then silently revert. Refusing and saying where to edit is
+	// the smaller surprise — the same call Remove makes.
+	if err == nil {
+		t.Fatal("expected scaling a declared model to be refused")
+	}
+	if !strings.Contains(err.Error(), "configuration") {
+		t.Errorf("the error should say where to change it; got %q", err)
+	}
+}
+
+func TestScalingAModelThatWasNeverAddedIsAnError(t *testing.T) {
+	s, _, _ := newService(t)
+	if _, err := s.Scale(t.Context(), "ghost", catalog.Scaling{Min: 1}, "ilker"); err == nil {
+		t.Error("expected an error for a model that is not there")
+	}
+}
+
+func TestAMaxBelowAMinIsRefusedBeforeAnythingIsWritten(t *testing.T) {
+	s, st, _ := newService(t)
+	if err := s.Add(t.Context(), model("added"), "ilker"); err != nil {
+		t.Fatal(err)
+	}
+
+	if _, err := s.Scale(t.Context(), "added", catalog.Scaling{Min: 3, Max: 1}, "ilker"); err == nil {
+		t.Fatal("expected a max below a min to be refused")
+	}
+
+	// Refused before the write, so the stored definition is untouched: a
+	// rejected change that had already been persisted would come back on the
+	// next restart as a catalog that does not build.
+	var stored catalog.Model
+	if err := json.Unmarshal(st.rows["added"].Definition, &stored); err != nil {
+		t.Fatal(err)
+	}
+	if stored.Scaling.Min != 1 {
+		t.Errorf("the stored definition was changed to %d despite the refusal", stored.Scaling.Min)
+	}
+}
+
+func TestScalingToZeroKeepsTheModelInTheCatalog(t *testing.T) {
+	s, _, rec := newService(t)
+	if err := s.Add(t.Context(), model("added"), "ilker"); err != nil {
+		t.Fatal(err)
+	}
+
+	got, err := s.Scale(t.Context(), "added", catalog.Scaling{Min: 0, Max: 0}, "ilker")
+	if err != nil {
+		t.Fatal(err)
+	}
+	// Configured and deliberately not running is a different and equally
+	// useful thing from removed: the entry stays so it can be brought back
+	// with one number.
+	if got.Scaling.OnDemand() {
+		t.Error("min 0 max 0 is not on-demand; it is off")
+	}
+	if _, ok := rec.last.Resolve("added"); !ok {
+		t.Error("scaling to zero must not remove the model")
+	}
+}
+
+func TestMinZeroWithAMaxIsOnDemand(t *testing.T) {
+	s, _, _ := newService(t)
+	if err := s.Add(t.Context(), model("added"), "ilker"); err != nil {
+		t.Fatal(err)
+	}
+
+	got, err := s.Scale(t.Context(), "added",
+		catalog.Scaling{Min: 0, Max: 2, IdleTimeout: 10 * time.Minute}, "ilker")
+	if err != nil {
+		t.Fatal(err)
+	}
+	if !got.Scaling.OnDemand() {
+		t.Error("min 0 with a max above it should be on-demand")
+	}
+	if got.Scaling.IdleTimeout != 10*time.Minute {
+		t.Errorf("the idle timeout was lost; got %v", got.Scaling.IdleTimeout)
+	}
+}

internal/control/placement/placement.go

@@ -394,13 +394,42 @@ func (s *Scheduler) eligibleNodes(p Profile, fabric string, avoid []string, reje
 			reject("does not carry " + miss)
 			continue
 		}
+		// The engine has to be installed here. The field has been on this
+		// struct since placement was written and was never read, so a node
+		// with no vLLM was as good a candidate as one with it: the replica was
+		// placed, the node registered healthy, and the shard failed minutes
+		// later with "executable file not found" in the engine's own words.
+		//
+		// A node that reports no engines at all is not excluded. That is what
+		// every agent older than this reports, and emptying a fleet on a
+		// control plane upgrade is a worse failure than the one being fixed.
+		//
+		// An agent that says it probed is believed when it reports nothing:
+		// that is a node still installing, or one where installing failed,
+		// and either way not a place to start an engine. sim is built into
+		// the agent and never installed.
+		if p.Engine != "" && p.Engine != "sim" && (n.EnginesProbed || len(n.EngineVersions) > 0) {
+			if _, ok := n.EngineVersions[p.Engine]; !ok {
+				reject("does not have " + p.Engine + " installed" + installing(n, p.Engine))
+				continue
+			}
+		}
 		// A node that does not report its memory is not excluded: an agent
 		// older than that field would otherwise have its node quietly emptied
 		// by a control plane upgrade, which is a worse failure than the one
 		// this prevents.
+		//
+		// What is left, not what the machine has. Comparing against the total
+		// meant a node with 31 GB accepted two models that each declared 15,
+		// because each was asked about on its own — and the second one is what
+		// the kernel kills. GPUs were never handed out twice this way; memory
+		// was, because nothing recorded that it had been claimed.
 		if p.MinMemoryGB > 0 && n.MemoryTotalMB > 0 {
-			if have := n.MemoryTotalMB / 1024; have < p.MinMemoryGB {
-				reject(fmt.Sprintf("has %d GB of memory, needs %d", have, p.MinMemoryGB))
+			total := int(n.MemoryTotalMB / 1024)
+			free := total - s.claimedMemory(n.ID)
+			if free < p.MinMemoryGB {
+				reject(fmt.Sprintf("has %d GB of memory with %d free, needs %d",
+					total, free, p.MinMemoryGB))
 				continue
 			}
 		}
@@ -487,3 +516,28 @@ func (s *Scheduler) pickGPUs(n inventory.Node, free []discovery.GPU, want int) [
 	slices.Sort(out)
 	return out
 }
+
+// claimedMemory is what the shards already on a node asked for.
+//
+// Declared, not measured. A profile that says it needs 15 GB is treated as
+// holding 15 whether the process has touched that much yet or not, which is the
+// only figure available when the decision is made — the alternative is placing
+// against what a model has grown into so far and being wrong every time one is
+// still loading.
+func (s *Scheduler) claimedMemory(nodeID string) int {
+	total := 0
+	for _, a := range s.inv.Allocations(nodeID) {
+		total += a.MemoryGB
+	}
+	return total
+}
+
+// installing says why an engine is missing, when the node has said.
+func installing(n inventory.Node, engine string) string {
+	for _, p := range n.Provision {
+		if p.Engine == engine && p.Detail != "" {
+			return " (" + p.State + ": " + p.Detail + ")"
+		}
+	}
+	return ""
+}

internal/control/placement/placement_test.go

@@ -679,3 +679,151 @@ func TestAProfileWithNoSourceIsUnaffected(t *testing.T) {
 		t.Fatalf("nothing placed:\n%s", res.Explain())
 	}
 }
+
+// withEngines is a node that reports which engines it has.
+func withEngines(id string, gpus int, names ...string) inventory.Node {
+	n := node(id, "h100-80g", gpus)
+	n.EngineVersions = map[string]string{}
+	for _, name := range names {
+		n.EngineVersions[name] = "unknown"
+	}
+	return n
+}
+
+func TestAReplicaIsNotPlacedOnANodeWithoutItsEngine(t *testing.T) {
+	// The engine has been on the Profile struct since placement was written
+	// and was never read, so a node with no vLLM was as good a candidate as
+	// one with it: the replica was placed, the node registered healthy, and
+	// the shard failed minutes later with "executable file not found".
+	s := New(store(withEngines("no-vllm", 8, "llamacpp"), withEngines("has-vllm", 8, "vllm")))
+
+	res := s.Place([]Profile{{
+		Name: "tp1", Engine: "vllm", GPUClass: "h100-80g", GPUsPerNode: 1, Nodes: 1,
+	}}, nil)
+	if res.Placement == nil {
+		t.Fatalf("nothing was placed:\n%s", res.Explain())
+	}
+	if got := res.Placement.Nodes[0].NodeID; got != "has-vllm" {
+		t.Errorf("placed on %q, which does not have vllm", got)
+	}
+}
+
+func TestAModelWithNoNodeRunningItsEngineSaysSo(t *testing.T) {
+	s := New(store(withEngines("cpu-only", 8, "llamacpp")))
+
+	res := s.Place([]Profile{{
+		Name: "tp1", Engine: "vllm", GPUClass: "h100-80g", GPUsPerNode: 1, Nodes: 1,
+	}}, nil)
+	if res.Placement != nil {
+		t.Fatal("a model whose engine is installed nowhere should not be placed")
+	}
+	// The reason belongs in explain-placement, or the answer to "why is this
+	// not running" is a retry loop in the log.
+	if !strings.Contains(res.Explain(), "does not have vllm installed") {
+		t.Errorf("the rejection should name the engine:\n%s", res.Explain())
+	}
+}
+
+func TestANodeThatReportsNoEnginesIsStillEligible(t *testing.T) {
+	// Every agent older than the probe reports nothing. Excluding those would
+	// empty a fleet on a control plane upgrade, which is a worse failure than
+	// the one being fixed.
+	old := node("older-agent", "h100-80g", 8)
+
+	s := New(store(old))
+	res := s.Place([]Profile{{
+		Name: "tp1", Engine: "vllm", GPUClass: "h100-80g", GPUsPerNode: 1, Nodes: 1,
+	}}, nil)
+	if res.Placement == nil {
+		t.Errorf("an agent that says nothing must not be excluded:\n%s", res.Explain())
+	}
+}
+
+func TestASecondLargeModelDoesNotLandOnAFullNode(t *testing.T) {
+	// 31 GB with 20 already claimed. Compared against the total it was placed
+	// anyway: each model was asked about on its own and the machine was asked
+	// nothing at all, so the second one is what the kernel kills.
+	big := inventory.Node{ID: "big", State: inventory.Ready, MemoryTotalMB: 31 * 1024}
+	st := store(big)
+	st.Allocate("big", inventory.Allocation{
+		ShardID: "first-r0", ReplicaID: "first", MemoryGB: 20,
+	})
+
+	s := New(st)
+	res := s.Place([]Profile{{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 15, Nodes: 1}}, nil)
+	if res.Placement != nil {
+		t.Fatalf("placed on %s with 11 GB left of 31 and 15 needed",
+			res.Placement.Nodes[0].NodeID)
+	}
+	// Both numbers, or "needs 15" beside a 31 GB machine reads as a bug in the
+	// scheduler rather than a full node.
+	if !strings.Contains(res.Explain(), "31 GB of memory with 11 free") {
+		t.Errorf("the rejection should show what is left:\n%s", res.Explain())
+	}
+}
+
+func TestOneLargeModelStillFits(t *testing.T) {
+	big := inventory.Node{ID: "big", State: inventory.Ready, MemoryTotalMB: 31 * 1024}
+	s := New(store(big))
+	res := s.Place([]Profile{{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 15, Nodes: 1}}, nil)
+	if res.Placement == nil {
+		t.Errorf("an empty node should take the first one:\n%s", res.Explain())
+	}
+}
+
+func TestTwoSmallModelsStillShareANode(t *testing.T) {
+	// Accounting for what is claimed must not turn into one model per machine.
+	big := inventory.Node{ID: "big", State: inventory.Ready, MemoryTotalMB: 31 * 1024}
+	st := store(big)
+	st.Allocate("big", inventory.Allocation{ShardID: "a-r0", ReplicaID: "a", MemoryGB: 3})
+
+	s := New(st)
+	res := s.Place([]Profile{{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 3, Nodes: 1}}, nil)
+	if res.Placement == nil {
+		t.Errorf("3 GB beside 3 GB on a 31 GB node should fit:\n%s", res.Explain())
+	}
+}
+
+func TestAProfileThatDeclaresNothingClaimsNothing(t *testing.T) {
+	// Zero is "nobody said", and counting it as a claim would let one
+	// undeclared model make a node look full.
+	big := inventory.Node{ID: "big", State: inventory.Ready, MemoryTotalMB: 31 * 1024}
+	st := store(big)
+	st.Allocate("big", inventory.Allocation{ShardID: "a-r0", ReplicaID: "a"})
+
+	s := New(st)
+	res := s.Place([]Profile{{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 30, Nodes: 1}}, nil)
+	if res.Placement == nil {
+		t.Errorf("an undeclared neighbour should not fill the node:\n%s", res.Explain())
+	}
+}
+
+func TestANodeStillInstallingItsEngineIsNotACandidate(t *testing.T) {
+	// A node rebuilt from nothing registers before its agent has installed
+	// anything, and reports no engines. Treated as "too old to say", it was a
+	// candidate for every model at the moment it could run none of them.
+	fresh := node("fresh", "h100-80g", 8)
+	fresh.EnginesProbed = true
+	fresh.Provision = []inventory.EngineInstall{{
+		Engine: "vllm", Version: "0.29.0", State: "installing", Detail: "downloading",
+	}}
+
+	res := New(store(fresh)).Place([]Profile{{
+		Name: "tp1", Engine: "vllm", GPUClass: "h100-80g", GPUsPerNode: 1, Nodes: 1,
+	}}, nil)
+	if res.Placement != nil {
+		t.Fatal("placed on a node whose engine is not installed yet")
+	}
+	if !strings.Contains(res.Explain(), "installing: downloading") {
+		t.Errorf("the rejection should say the engine is on its way:\n%s", res.Explain())
+	}
+
+	// The simulated engine is part of the agent, so it is never installed and
+	// never missing.
+	res = New(store(fresh)).Place([]Profile{{
+		Name: "sim", Engine: "sim", GPUClass: "h100-80g", GPUsPerNode: 1, Nodes: 1,
+	}}, nil)
+	if res.Placement == nil {
+		t.Errorf("sim was refused:\n%s", res.Explain())
+	}
+}

internal/control/reconciler/fetch_test.go

@@ -0,0 +1,70 @@
+package reconciler
+
+import (
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/supervisor"
+)
+
+func TestProgressIsVisibleWithoutBeingATransition(t *testing.T) {
+	// The failure this replaces: a replica pulling a hundred gigabytes and one
+	// that is wedged look identical from the control plane — both are a shard
+	// that has been starting for ten minutes.
+	//
+	// And progress must not count as a state change. Treating each chunk as
+	// one would reset the timer that decides when a slow start has become a
+	// failed one, so a download would keep a replica alive forever.
+	h := newHarness(t, testModel(1), gpuNode("n1", 8))
+	h.r.Reconcile(t.Context())
+
+	reps := h.r.Replicas()
+	if len(reps) != 1 || len(reps[0].Shards) == 0 {
+		t.Fatalf("expected a placed replica with shards, got %+v", reps)
+	}
+	rep, shard := reps[0].ID, reps[0].Shards[0].ID
+
+	h.r.OnFetchProgress(rep, shard, "hf:org/repo/model.gguf", 3<<30, 12<<30)
+
+	got := h.r.Replicas()[0].Shards[0]
+	if got.Fetch == nil {
+		t.Fatal("the download is not visible on the shard")
+	}
+	if got.Fetch.BytesDone != 3<<30 || got.Fetch.BytesTotal != 12<<30 {
+		t.Errorf("progress came back as %+v", got.Fetch)
+	}
+	if p, ok := got.Fetch.Percent(); !ok || p != 25 {
+		t.Errorf("percent = %d (known %v), want 25", p, ok)
+	}
+}
+
+func TestAnUnknownTotalHasNoPercentage(t *testing.T) {
+	// A server that sends no Content-Length is common. An invented percentage
+	// that stops moving is worse than a byte count that keeps rising.
+	f := FetchProgress{Source: "https://x/y.gguf", BytesDone: 500}
+	if _, ok := f.Percent(); ok {
+		t.Error("a percentage was invented from an unknown total")
+	}
+}
+
+func TestATransitionClearsTheDownload(t *testing.T) {
+	// Left in place it would sit in the listing at ninety-nine percent under
+	// a shard that has been serving for an hour.
+	h := newHarness(t, testModel(1), gpuNode("n1", 8))
+	h.r.Reconcile(t.Context())
+	reps := h.r.Replicas()
+	rep, shard := reps[0].ID, reps[0].Shards[0].ID
+
+	h.r.OnFetchProgress(rep, shard, "hf:org/repo/model.gguf", 1, 2)
+	h.r.OnShardStatus(rep, shard, supervisor.Ready, "")
+
+	if got := h.r.Replicas()[0].Shards[0]; got.Fetch != nil {
+		t.Errorf("a finished shard still shows a download: %+v", got.Fetch)
+	}
+}
+
+func TestProgressForAReplicaThatIsGoneIsIgnored(t *testing.T) {
+	// A download that finishes after its replica was torn down should not
+	// resurrect anything or panic.
+	h := newHarness(t, testModel(1), gpuNode("n1", 8))
+	h.r.OnFetchProgress("no-such-replica", "no-such-shard", "x", 1, 2)
+}

internal/control/reconciler/leadership_test.go

@@ -0,0 +1,48 @@
+package reconciler
+
+import (
+	"testing"
+)
+
+type fixedLeader bool
+
+func (f fixedLeader) IsLeader() bool { return bool(f) }
+
+func TestAStandbyPlacesNothing(t *testing.T) {
+	// The failure this prevents: two control planes against one store both see
+	// a model wanting replicas and none running, and both place. The model
+	// ends up with twice what it asked for, each replica holding GPUs the
+	// other's inventory believes are free, and nothing crashes to say so.
+	h := newHarness(t, testModel(1), gpuNode("n1", 8))
+	h.r.SetLeadership(fixedLeader(false))
+
+	h.r.Reconcile(t.Context())
+
+	if got := len(h.r.Replicas()); got != 0 {
+		t.Errorf("a standby placed %d replicas", got)
+	}
+}
+
+func TestALeaderPlacesAsBefore(t *testing.T) {
+	h := newHarness(t, testModel(1), gpuNode("n1", 8))
+	h.r.SetLeadership(fixedLeader(true))
+
+	h.r.Reconcile(t.Context())
+
+	if got := len(h.r.Replicas()); got == 0 {
+		t.Error("a leader placed nothing")
+	}
+}
+
+func TestWithoutLeadershipConfiguredNothingChanges(t *testing.T) {
+	// Every single-control-plane deployment, and every test written before
+	// leadership existed. Unset must mean "act", not "wait for permission
+	// from a thing that was never wired up".
+	h := newHarness(t, testModel(1), gpuNode("n1", 8))
+
+	h.r.Reconcile(t.Context())
+
+	if got := len(h.r.Replicas()); got == 0 {
+		t.Error("a reconciler with no leadership configured placed nothing")
+	}
+}

internal/control/reconciler/reconciler.go

@@ -30,6 +30,9 @@ type ShardCommand struct {
 	// control plane restart (plan §6.5).
 	Pool   string
 	Fabric string
+	// ProfileFP travels for the same reason, and answers a different question:
+	// whether the profile has been edited since this shard was started.
+	ProfileFP string
 }
 
 // Dispatcher sends commands to agents.
@@ -166,6 +169,9 @@ func (c *Config) normalize() {
 
 // Reconciler compares the catalog against what is running and closes the gap.
 type Reconciler struct {
+	// leader gates every action; nil means act unconditionally.
+	leader Leader
+
 	cfg   Config
 	cat   *catalog.Live
 	inv   *inventory.Store
@@ -235,6 +241,17 @@ func (r *Reconciler) SetClock(now func() time.Time) {
 	r.startedAt = now()
 }
 
+// Leader reports whether this process should be acting on the fleet.
+type Leader interface{ IsLeader() bool }
+
+// SetLeadership gates every action on holding leadership.
+//
+// A setter rather than a constructor argument because the thing that decides
+// leadership needs a store, and the store is wired after the controller. Nil
+// leaves the reconciler acting unconditionally, which is what a single control
+// plane wants and what every test expects.
+func (r *Reconciler) SetLeadership(l Leader) { r.leader = l }
+
 // Start runs the loop until Close.
 func (r *Reconciler) Start(ctx context.Context) {
 	go func() {
@@ -278,6 +295,16 @@ func (r *Reconciler) Notify() {
 // Reconcile performs one pass. It is exported so tests can drive it directly
 // instead of racing a ticker.
 func (r *Reconciler) Reconcile(ctx context.Context) {
+	// A standby does nothing at all here. Everything below issues commands to
+	// agents — placing, evicting, draining, reaping — and two control planes
+	// doing that against one fleet is how a model ends up with twice the
+	// replicas it asked for, each holding GPUs the other believes are free.
+	//
+	// Unset means leader, so a single control plane and every test behave
+	// exactly as they did before leadership existed.
+	if r.leader != nil && !r.leader.IsLeader() {
+		return
+	}
 	r.expireNodes()
 	r.expireStuckReplicas()
 	r.drainCordonedNodes(ctx)
@@ -788,7 +815,7 @@ func (r *Reconciler) startReplica(ctx context.Context, m catalog.Model, pl *plac
 	r.mu.Unlock()
 
 	rep := &Replica{
-		ID: id, Model: m.Name, Profile: prof.Name,
+		ID: id, Model: m.Name, Profile: prof.Name, ProfileFP: prof.Fingerprint(),
 		Fabric: pl.Fabric, Pool: pl.Pool, State: Placing,
 		Weight: prof.Weight, CreatedAt: r.now(), Attempt: attempt,
 		WantShards: len(pl.Nodes),
@@ -813,6 +840,7 @@ func (r *Reconciler) startReplica(ctx context.Context, m catalog.Model, pl *plac
 		})
 		r.inv.Allocate(a.NodeID, inventory.Allocation{
 			ShardID: shardID, ReplicaID: id, GPUIndices: a.GPUIndices,
+			MemoryGB: prof.MinMemoryGB,
 		})
 	}
 	rep.Endpoint = fmt.Sprintf("http://%s:%d", addrOf(leaderNode), basePort)
@@ -859,6 +887,7 @@ func (r *Reconciler) startReplica(ctx context.Context, m catalog.Model, pl *plac
 			StartupTimeout: prof.StartupTimeout,
 			Pool:           pl.Pool,
 			Fabric:         pl.Fabric,
+			ProfileFP:      prof.Fingerprint(),
 		}
 		if err := r.disp.StartShard(ctx, cmd); err != nil {
 			r.markShard(id, s.ID, supervisor.Failed, err.Error())
@@ -1063,7 +1092,10 @@ func (r *Reconciler) Adopt(nodeID string, shards []AdoptedShard) (unclaimed []Ad
 		rep := &Replica{
 			ID: replicaID, Model: m.Name, Profile: prof.Name,
 			Fabric: first.Fabric, Pool: first.Pool,
-			Weight: prof.Weight, CreatedAt: r.now(),
+			// From the agent, which was told it when the shard started. Empty
+			// from an agent too old to report it, which reads as unknown.
+			ProfileFP: first.ProfileFP,
+			Weight:    prof.Weight, CreatedAt: r.now(),
 			Endpoint: first.Endpoint,
 			// From the profile, not from what was reported: the point is to
 			// notice when what was reported is short.
@@ -1099,9 +1131,12 @@ func (r *Reconciler) mergeAdopted(rep *Replica, group []AdoptedShard) {
 			GPUIndices: s.GPUIndices, HCAs: s.HCAs,
 			HTTPPort: s.HTTPPort, State: s.State,
 		})
-		// Reclaim the GPUs, or the next placement will hand them out twice.
+		// Reclaim the GPUs and the memory, or the next placement hands them
+		// out twice. The memory comes from the catalog rather than the agent:
+		// what a profile asked for is not something a running process reports.
 		r.inv.Allocate(s.NodeID, inventory.Allocation{
 			ShardID: s.ShardID, ReplicaID: rep.ID, GPUIndices: s.GPUIndices,
+			MemoryGB: r.profileMemory(rep.Model, rep.Profile),
 		})
 		// The endpoint comes from the leader, which may be in any of the
 		// groups — the rank 0 node is not necessarily the first to register.
@@ -1207,10 +1242,34 @@ type AdoptedShard struct {
 	Profile    string
 	Pool       string
 	Fabric     string
+	ProfileFP  string
 	// Endpoint is the leader's base URL, set on the rank 0 shard.
 	Endpoint string
 }
 
+// OnFetchProgress records how far a shard's download has got.
+//
+// Separate from OnShardStatus because it is not a transition: the shard is
+// still starting, and treating each progress report as a state change would
+// reset the timers that decide when a slow start becomes a failed one.
+func (r *Reconciler) OnFetchProgress(replicaID, shardID, source string, done, total int64) {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+
+	rep, ok := r.replicas[replicaID]
+	if !ok {
+		return
+	}
+	for i := range rep.Shards {
+		if rep.Shards[i].ID != shardID {
+			continue
+		}
+		rep.Shards[i].Fetch = &FetchProgress{
+			Source: source, BytesDone: done, BytesTotal: total,
+		}
+	}
+}
+
 // OnShardStatus records a status report from an agent.
 func (r *Reconciler) OnShardStatus(replicaID, shardID string, st supervisor.State, errMsg string) {
 	r.markShard(replicaID, shardID, st, errMsg)
@@ -1233,6 +1292,10 @@ func (r *Reconciler) markShard(replicaID, shardID string, st supervisor.State, e
 		if errMsg != "" {
 			rep.Shards[i].Err = errMsg
 		}
+		// A real transition means the download is over, one way or the other.
+		// Left in place it would sit in the listing at ninety-nine percent
+		// under a shard that has been serving for an hour.
+		rep.Shards[i].Fetch = nil
 		break
 	}
 
@@ -1298,3 +1361,22 @@ func addrOf(n inventory.Node) string {
 	}
 	return n.ID
 }
+
+// profileMemory is what a model's profile declared it needs, or zero.
+//
+// Looked up when a replica is adopted after a restart, because the allocation
+// has to carry the same claim it did before — a control plane that forgot every
+// claim on restart would place a second large model onto a node already full of
+// the first.
+func (r *Reconciler) profileMemory(model, profile string) int {
+	m, ok := r.cat.Load().Resolve(model)
+	if !ok {
+		return 0
+	}
+	for _, p := range m.Profiles {
+		if p.Name == profile {
+			return p.MinMemoryGB
+		}
+	}
+	return 0
+}

internal/control/reconciler/reconciler_test.go

@@ -1665,3 +1665,59 @@ func TestAFreshlyStartedReplicaIsNotEvicted(t *testing.T) {
 		t.Error("a long-running idle replica was never offered for eviction")
 	}
 }
+
+func TestAStartedShardIsToldWhichProfileVersionItIs(t *testing.T) {
+	// The fingerprint has to leave the control plane, or it cannot come back
+	// after a restart — and a restart is exactly when the configuration is
+	// likely to have been edited.
+	h := newHarness(t, testModel(1), gpuNode("1", 8))
+	h.r.Reconcile(t.Context())
+
+	started := h.disp.startedShards()
+	if len(started) == 0 {
+		t.Fatal("nothing was started")
+	}
+	// From the catalog rather than the literal, because the catalog fills in
+	// defaults and the fingerprint is of the profile as it is actually run.
+	m, ok := h.r.cat.Load().Resolve("llama")
+	if !ok {
+		t.Fatal("the model is not in the catalog")
+	}
+	want := m.Profiles[0].Fingerprint()
+	for _, cmd := range started {
+		if cmd.ProfileFP != want {
+			t.Errorf("shard %s carries %q, expected %q", cmd.ShardID, cmd.ProfileFP, want)
+		}
+	}
+}
+
+func TestAnAdoptedReplicaKeepsTheProfileVersionItWasStartedWith(t *testing.T) {
+	// Adoption used to leave this empty, which made every replica read as
+	// unknown after a control plane restart — so an edited profile could never
+	// be noticed on the machines that were already running.
+	h := newHarness(t, testModel(1), gpuNode("1", 8))
+
+	shard := adopted("llama-99", "llama-99-r0", "1", 0, []int{0, 1, 2, 3})
+	shard.ProfileFP = "1122334455667788"
+	h.r.Adopt("1", []AdoptedShard{shard})
+
+	reps := h.r.Replicas()
+	if len(reps) != 1 {
+		t.Fatalf("expected 1 adopted replica, got %d", len(reps))
+	}
+	if reps[0].ProfileFP != "1122334455667788" {
+		t.Errorf("the agent's answer was dropped: %q", reps[0].ProfileFP)
+	}
+}
+
+func TestAnAgentTooOldToReportTheProfileVersionIsNotGuessedAt(t *testing.T) {
+	h := newHarness(t, testModel(1), gpuNode("1", 8))
+	h.r.Adopt("1", []AdoptedShard{adopted("llama-99", "llama-99-r0", "1", 0, []int{0, 1, 2, 3})})
+
+	// Empty, not filled in from the catalog. Copying the current profile here
+	// would make the replica claim to match whatever the catalog says next,
+	// which is the opposite of what the field is for.
+	if fp := h.r.Replicas()[0].ProfileFP; fp != "" {
+		t.Errorf("an unreported fingerprint should stay empty, got %q", fp)
+	}
+}

internal/control/reconciler/replica.go

@@ -65,6 +65,35 @@ type Shard struct {
 	HTTPPort   int
 	State      supervisor.State
 	Err        string
+
+	// Fetch is the download this shard is waiting on, if any. Cleared when the
+	// shard reaches any other state, so a finished download does not sit in
+	// the listing looking like one still running.
+	Fetch *FetchProgress
+}
+
+// FetchProgress is how far a shard's weights have got.
+//
+// Carried so an operator can tell a replica pulling a hundred gigabytes from
+// one that is wedged. From outside they are the same thing — a shard that has
+// been starting for ten minutes — and the difference is whether to wait or to
+// go and look.
+type FetchProgress struct {
+	Source    string
+	BytesDone int64
+	// BytesTotal is 0 when the server did not say. A percentage would then be
+	// invented, and an invented one that stops moving is worse than a byte
+	// count that keeps rising.
+	BytesTotal int64
+}
+
+// Percent returns progress as a percentage, and false when the total is
+// unknown.
+func (f FetchProgress) Percent() (int, bool) {
+	if f.BytesTotal <= 0 {
+		return 0, false
+	}
+	return int(f.BytesDone * 100 / f.BytesTotal), true
 }
 
 // Leader reports whether this shard serves HTTP for the replica.
@@ -75,6 +104,18 @@ type Replica struct {
 	ID      string
 	Model   string
 	Profile string
+	// ProfileFP is the profile's contents when this replica was placed.
+	//
+	// Compared with the catalog's to tell a replica running the definition in
+	// force from one running whatever it was edited from. A running replica is
+	// deliberately not moved when its profile changes — churn costs more than
+	// the imperfection — so without this there is nothing that says the two
+	// have diverged, and "why is this one on the wrong node" is answered from
+	// memory of what somebody edited an hour ago.
+	//
+	// Empty on a replica adopted from an agent that predates it, which reads
+	// as unknown rather than as stale.
+	ProfileFP string
 	// Fabric is set only for multi-node replicas.
 	Fabric string
 	Pool   string

internal/docgen/docgen.go

@@ -0,0 +1,158 @@
+// Package docgen turns the configuration structs into reference documentation.
+//
+// Generated rather than written, because a hand-written reference is a promise
+// that someone will remember. This project added six settings in a fortnight —
+// store, servers, lease_ttl, thread and memory retention, min_disk_gb — and
+// every one of them reached an operator only through a narrative document that
+// happened to mention it. A reference nobody maintains is worse than none: it
+// is the same list, quietly missing the setting you came to look up.
+//
+// Derived from the source, so the only way for it to be wrong is for the code
+// to be wrong. A test regenerates it and fails when it has drifted, which is
+// what keeps that true after everyone has forgotten this exists.
+package docgen
+
+//go:generate go run ./gen
+
+import (
+	"fmt"
+	"go/ast"
+	"go/parser"
+	"go/token"
+	"reflect"
+	"sort"
+	"strings"
+)
+
+// Field is one configuration key.
+type Field struct {
+	Key  string
+	Type string
+	Doc  string
+	// Nested names the struct this key expands into, empty for a leaf. It is
+	// what lets the reference say "see Auth" rather than printing the same
+	// twenty keys under four parents.
+	Nested string
+}
+
+// Block is a struct that appears in configuration.
+type Block struct {
+	Name   string
+	Doc    string
+	Fields []Field
+}
+
+// Collect reads koanf-tagged structs out of Go source files.
+func Collect(files []string) ([]Block, error) {
+	fset := token.NewFileSet()
+	var blocks []Block
+	seen := map[string]bool{}
+
+	for _, path := range files {
+		f, err := parser.ParseFile(fset, path, nil, parser.ParseComments)
+		if err != nil {
+			return nil, fmt.Errorf("parsing %s: %w", path, err)
+		}
+		ast.Inspect(f, func(n ast.Node) bool {
+			ts, ok := n.(*ast.TypeSpec)
+			if !ok {
+				return true
+			}
+			st, ok := ts.Type.(*ast.StructType)
+			if !ok {
+				return true
+			}
+			b, ok := blockOf(ts, st)
+			if !ok || seen[b.Name] {
+				return true
+			}
+			seen[b.Name] = true
+			blocks = append(blocks, b)
+			return true
+		})
+	}
+	sort.Slice(blocks, func(i, j int) bool { return blocks[i].Name < blocks[j].Name })
+	return blocks, nil
+}
+
+// blockOf builds a Block, or reports that this struct carries no settings.
+func blockOf(ts *ast.TypeSpec, st *ast.StructType) (Block, bool) {
+	b := Block{Name: ts.Name.Name, Doc: docText(ts.Doc)}
+	for _, f := range st.Fields.List {
+		if f.Tag == nil {
+			continue
+		}
+		tag := reflect.StructTag(strings.Trim(f.Tag.Value, "`"))
+		key, ok := tag.Lookup("koanf")
+		// "-" is a field carried in the struct but never read from a file;
+		// listing it would send someone looking for a setting that does not
+		// exist.
+		if !ok || key == "" || key == "-" {
+			continue
+		}
+		typ, nested := typeOf(f.Type)
+		b.Fields = append(b.Fields, Field{
+			Key: key, Type: typ, Doc: docText(f.Doc), Nested: nested,
+		})
+	}
+	return b, len(b.Fields) > 0
+}
+
+// typeOf renders a field's type, and names the struct it expands into.
+func typeOf(e ast.Expr) (rendered, nested string) {
+	switch t := e.(type) {
+	case *ast.Ident:
+		if isNamedStruct(t.Name) {
+			return t.Name, t.Name
+		}
+		return t.Name, ""
+	case *ast.SelectorExpr:
+		// A type from another package, such as tlsconf.Config.
+		name := t.Sel.Name
+		if pkg, ok := t.X.(*ast.Ident); ok {
+			return pkg.Name + "." + name, name
+		}
+		return name, name
+	case *ast.ArrayType:
+		inner, _ := typeOf(t.Elt)
+		return "[]" + inner, ""
+	case *ast.MapType:
+		k, _ := typeOf(t.Key)
+		v, _ := typeOf(t.Value)
+		return "map[" + k + "]" + v, ""
+	case *ast.StarExpr:
+		inner, n := typeOf(t.X)
+		return "*" + inner, n
+	default:
+		return "value", ""
+	}
+}
+
+// isNamedStruct guesses whether an identifier names a configuration block.
+//
+// By capitalisation, because the alternative is resolving types across
+// packages for a document. A wrong guess costs a "see also" that points at
+// nothing, which the renderer drops.
+func isNamedStruct(name string) bool {
+	if name == "" {
+		return false
+	}
+	switch name {
+	case "string", "bool", "int", "int64", "float64", "any", "byte", "rune":
+		return false
+	}
+	return name[0] >= 'A' && name[0] <= 'Z'
+}
+
+// docText flattens a comment group into one paragraph per blank line.
+func docText(g *ast.CommentGroup) string {
+	if g == nil {
+		return ""
+	}
+	var lines []string
+	for _, c := range g.List {
+		t := strings.TrimPrefix(c.Text, "//")
+		lines = append(lines, strings.TrimPrefix(t, " "))
+	}
+	return strings.TrimSpace(strings.Join(lines, "\n"))
+}

internal/docgen/docgen_test.go

@@ -0,0 +1,138 @@
+package docgen
+
+import (
+	"os"
+	"path/filepath"
+	"runtime"
+	"slices"
+	"strings"
+	"testing"
+)
+
+// repoRoot is where the source paths in Sources are relative to.
+func repoRoot(t *testing.T) string {
+	t.Helper()
+	_, file, _, ok := runtime.Caller(0)
+	if !ok {
+		t.Fatal("cannot locate this test")
+	}
+	return filepath.Join(filepath.Dir(file), "..", "..")
+}
+
+func sourcePaths(t *testing.T) []string {
+	t.Helper()
+	root := repoRoot(t)
+	out := make([]string, len(Sources))
+	for i, s := range Sources {
+		out[i] = filepath.Join(root, s)
+	}
+	return out
+}
+
+func TestTheReferenceIsNotStale(t *testing.T) {
+	// The whole point. A generated document that nobody regenerates is the
+	// same lie as a hand-written one, arrived at more slowly: it describes the
+	// configuration of some earlier week, and the setting somebody came to
+	// look up is the one that is missing.
+	blocks, err := Collect(sourcePaths(t))
+	if err != nil {
+		t.Fatal(err)
+	}
+	want := Render(blocks)
+
+	path := filepath.Join(repoRoot(t), "docs", "reference.md")
+	got, err := os.ReadFile(path)
+	if err != nil {
+		t.Fatalf("docs/reference.md is missing; run go generate ./...: %v", err)
+	}
+	if string(got) != want {
+		t.Error("docs/reference.md no longer matches the configuration structs.\n" +
+			"Run: go generate ./...")
+	}
+}
+
+func TestEveryConfigurationFileIsListed(t *testing.T) {
+	// Sources is a list, and a list is a thing to forget. A new file full of
+	// settings would otherwise be absent from the reference with nothing
+	// saying so — which is exactly the failure the reference exists to fix.
+	root := repoRoot(t)
+	var tagged []string
+	err := filepath.Walk(filepath.Join(root, "internal"), func(p string, info os.FileInfo, err error) error {
+		if err != nil || info.IsDir() || !strings.HasSuffix(p, ".go") || strings.HasSuffix(p, "_test.go") {
+			return err
+		}
+		b, err := os.ReadFile(p)
+		if err != nil {
+			return err
+		}
+		if !strings.Contains(string(b), `koanf:"`) {
+			return nil
+		}
+		rel, err := filepath.Rel(root, p)
+		if err != nil {
+			return err
+		}
+		tagged = append(tagged, rel)
+		return nil
+	})
+	if err != nil {
+		t.Fatal(err)
+	}
+	if len(tagged) == 0 {
+		t.Fatal("found no configuration files at all, so this test proves nothing")
+	}
+
+	for _, f := range tagged {
+		if slices.Contains(Sources, f) {
+			continue
+		}
+		// Deliberately out: these hold koanf tags for structs an operator
+		// never writes — inventory and catalog entries come from the API or
+		// from a model file, not from the configuration.
+		if slices.Contains(notReference, f) {
+			continue
+		}
+		t.Errorf("%s declares configuration and is not in Sources or notReference", f)
+	}
+}
+
+func TestADocumentedKeyKeepsItsComment(t *testing.T) {
+	// The descriptions are the field comments, which is what makes the
+	// reference worth reading rather than a list of names. A field whose
+	// comment was the only explanation anywhere must not render blank.
+	blocks, err := Collect(sourcePaths(t))
+	if err != nil {
+		t.Fatal(err)
+	}
+	var lease Field
+	for _, b := range blocks {
+		for _, f := range b.Fields {
+			if f.Key == "lease_ttl" {
+				lease = f
+			}
+		}
+	}
+	if lease.Key == "" {
+		t.Fatal("lease_ttl is not in the reference")
+	}
+	if !strings.Contains(lease.Doc, "shared store") {
+		t.Errorf("lease_ttl lost its explanation: %q", lease.Doc)
+	}
+}
+
+func TestAFieldMarkedNotConfigurableIsLeftOut(t *testing.T) {
+	// koanf:"-" means the field is carried in the struct and never read from
+	// a file. Listing it would send somebody looking for a setting that does
+	// not exist.
+	blocks, err := Collect(sourcePaths(t))
+	if err != nil {
+		t.Fatal(err)
+	}
+	for _, b := range blocks {
+		for _, f := range b.Fields {
+			if f.Key == "-" || f.Key == "" {
+				t.Errorf("%s lists a non-configurable field: %+v", b.Name, f)
+			}
+		}
+	}
+}

internal/docgen/gen/main.go

@@ -0,0 +1,42 @@
+// Command gen writes docs/reference.md from the configuration structs.
+//
+// Run by `go generate ./...`, and checked by a test, so the reference cannot
+// quietly stop describing the code.
+package main
+
+import (
+	"fmt"
+	"os"
+	"path/filepath"
+	"runtime"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/docgen"
+)
+
+func main() {
+	// The repository root, found from this file rather than from the working
+	// directory, so `go generate ./...` produces the same document wherever
+	// it is run from.
+	_, self, _, ok := runtime.Caller(0)
+	if !ok {
+		fmt.Fprintln(os.Stderr, "error: cannot locate the generator")
+		os.Exit(1)
+	}
+	root := filepath.Join(filepath.Dir(self), "..", "..", "..")
+
+	paths := make([]string, len(docgen.Sources))
+	for i, s := range docgen.Sources {
+		paths[i] = filepath.Join(root, s)
+	}
+
+	blocks, err := docgen.Collect(paths)
+	if err != nil {
+		fmt.Fprintln(os.Stderr, "error:", err)
+		os.Exit(1)
+	}
+	if err := os.WriteFile(filepath.Join(root, "docs", "reference.md"), []byte(docgen.Render(blocks)), 0o644); err != nil {
+		fmt.Fprintln(os.Stderr, "error:", err)
+		os.Exit(1)
+	}
+	fmt.Printf("docs/reference.md: %d blocks\n", len(blocks))
+}

internal/docgen/render.go

@@ -0,0 +1,99 @@
+package docgen
+
+import (
+	"fmt"
+	"strings"
+)
+
+// Sources are the files configuration is declared in.
+//
+// Listed rather than discovered, because "every file with a koanf tag" would
+// sweep in structs that are configuration in the Go sense and not in the
+// operator's — a scoring weight built at wiring time is not something anyone
+// writes in a file. A test checks the list against what is actually tagged, so
+// a new configuration file is a failure here rather than a silent omission.
+var Sources = []string{
+	"internal/config/config.go",
+	"internal/agent/agent.go",
+	"internal/agent/discovery/discover.go",
+	"internal/agent/engines/engine.go",
+	"internal/agent/engines/llamacpp.go",
+	"internal/agent/engines/vllm.go",
+	"internal/agent/engines/sglang.go",
+	"internal/agent/engines/sim.go",
+	"internal/agent/supervisor/supervisor.go",
+	"internal/tlsconf/tlsconf.go",
+	"internal/control/catalog/catalog.go",
+	"internal/router/admission/controller.go",
+	"internal/accounting/cost.go",
+}
+
+// notReference are files with koanf tags that an operator never writes.
+//
+// Inventory and catalog entries carry them so the same structs can be decoded
+// from an API payload or a model file; putting them in a configuration
+// reference would invite somebody to write them into server.yaml.
+var notReference = []string{
+	"internal/control/inventory/inventory.go",
+}
+
+// Render turns collected blocks into the reference document.
+func Render(blocks []Block) string {
+	var b strings.Builder
+	b.WriteString(`# Configuration reference
+
+Every setting llmgw reads from a file, with the type it takes and what it is
+for.
+
+**Generated from the source.** The descriptions are the comments on the fields
+themselves, so this cannot describe a setting that does not exist or miss one
+that does — and a test fails when it has drifted. Do not edit it; edit the
+comment and run ` + "`go generate ./...`" + `.
+
+Where a setting takes another block, the block is listed under its own heading
+rather than repeated.
+
+`)
+
+	// An index first: the reason to open a reference is usually to find one
+	// key, and scanning thirteen headings for it is the thing this replaces.
+	b.WriteString("## Every key\n\n| Key | Type | In |\n|---|---|---|\n")
+	for _, bl := range blocks {
+		for _, f := range bl.Fields {
+			b.WriteString(fmt.Sprintf("| `%s` | %s | [%s](#%s) |\n",
+				f.Key, code(f.Type), bl.Name, anchor(bl.Name)))
+		}
+	}
+	b.WriteString("\n")
+
+	for _, bl := range blocks {
+		fmt.Fprintf(&b, "## %s\n\n", bl.Name)
+		if bl.Doc != "" {
+			b.WriteString(bl.Doc + "\n\n")
+		}
+		for _, f := range bl.Fields {
+			fmt.Fprintf(&b, "### `%s`\n\n%s\n\n", f.Key, code(f.Type))
+			if f.Doc != "" {
+				b.WriteString(f.Doc + "\n\n")
+			}
+			if f.Nested != "" && hasBlock(blocks, f.Nested) {
+				fmt.Fprintf(&b, "Takes a [%s](#%s) block.\n\n", f.Nested, anchor(f.Nested))
+			}
+		}
+	}
+	return b.String()
+}
+
+func hasBlock(blocks []Block, name string) bool {
+	for _, b := range blocks {
+		if b.Name == name {
+			return true
+		}
+	}
+	return false
+}
+
+func code(s string) string { return "`" + s + "`" }
+
+// anchor is how GitHub and Gitea slug a heading.
+func anchor(s string) string { return strings.ToLower(s) }

internal/hf/hf.go

@@ -0,0 +1,323 @@
+// Package hf reads the Hugging Face model registry.
+//
+// Enough of it to answer two questions an operator has before mirroring a
+// model: what is there, and which file do I want. Nothing here downloads
+// anything — the mirror does that, over its own network, and the point of the
+// split is that the control plane never carries a model's bytes.
+//
+// Read-only and anonymous. A gated or private repository answers 401 to this
+// client, which is reported rather than worked around: a token belongs to the
+// machine that fetches the weights, not to the one browsing a catalogue.
+package hf
+
+import (
+	"context"
+	"encoding/json"
+	"fmt"
+	"io"
+	"net/http"
+	"net/url"
+	"regexp"
+	"strings"
+	"time"
+)
+
+// DefaultEndpoint is the public registry.
+const DefaultEndpoint = "https://huggingface.co"
+
+// Client reads the registry.
+type Client struct {
+	base string
+	hc   *http.Client
+}
+
+// New builds a client against an endpoint, or the public one when empty.
+func New(endpoint string, timeout time.Duration) *Client {
+	if endpoint == "" {
+		endpoint = DefaultEndpoint
+	}
+	if timeout <= 0 {
+		// Short. This runs while somebody is looking at a page, and a registry
+		// that is slow today is better reported than waited for.
+		timeout = 10 * time.Second
+	}
+	return &Client{base: strings.TrimSuffix(endpoint, "/"),
+		hc: &http.Client{Timeout: timeout}}
+}
+
+// Model is a repository as a search result shows it.
+type Model struct {
+	ID           string    `json:"id"`
+	Downloads    int64     `json:"downloads"`
+	Likes        int64     `json:"likes"`
+	PipelineTag  string    `json:"pipeline_tag"`
+	Tags         []string  `json:"tags"`
+	LastModified time.Time `json:"lastModified"`
+	// Gated is true when the repository asks for terms to be accepted. It
+	// decodes from either a bool or a string ("auto", "manual"), which is why
+	// it is not a plain bool here.
+	Gated   Gated `json:"gated"`
+	Private bool  `json:"private"`
+}
+
+// Gated is Hugging Face's gating flag, which is false or a word.
+type Gated struct{ Reason string }
+
+// UnmarshalJSON accepts false, true, or a string.
+//
+// The field is documented as a boolean and is a string in practice for most
+// gated repositories. Decoding it as a bool fails the whole response, which
+// would make one gated model in a page of results hide the other nineteen.
+func (g *Gated) UnmarshalJSON(b []byte) error {
+	var s string
+	if err := json.Unmarshal(b, &s); err == nil {
+		g.Reason = s
+		return nil
+	}
+	var v bool
+	if err := json.Unmarshal(b, &v); err != nil {
+		return nil // unknown shape: treat as not gated rather than fail the page
+	}
+	if v {
+		g.Reason = "yes"
+	}
+	return nil
+}
+
+// Is reports whether the repository is gated at all.
+func (g Gated) Is() bool { return g.Reason != "" && g.Reason != "false" }
+
+// Search finds repositories by name.
+func (c *Client) Search(ctx context.Context, query string, limit int) ([]Model, error) {
+	if strings.TrimSpace(query) == "" {
+		return nil, fmt.Errorf("search for something")
+	}
+	if limit <= 0 || limit > 50 {
+		limit = 20
+	}
+	q := url.Values{
+		"search": {query},
+		"limit":  {fmt.Sprint(limit)},
+		// By downloads, because the thing somebody means by a short name is
+		// almost always the one everybody else meant too.
+		"sort":      {"downloads"},
+		"direction": {"-1"},
+	}
+	var out []Model
+	err := c.get(ctx, "/api/models?"+q.Encode(), &out)
+	return out, err
+}
+
+// File is one file in a repository.
+type File struct {
+	Path string
+	// Bytes is the real size. For a file stored in LFS — every weights file —
+	// the plain size field is the size of the pointer, a few hundred bytes, so
+	// reading that one would report a 5 GB model as 133 bytes.
+	Bytes int64
+	// SHA256 is what the registry publishes for an LFS file, so it can be
+	// compared with what the mirror computes after fetching. Empty for a small
+	// file stored in git, where the oid is a git blob hash and not a checksum
+	// of the content.
+	SHA256 string
+}
+
+// IsWeights reports whether this file is a model rather than documentation.
+//
+// By extension, which is crude and right: the registry does not label them, and
+// an operator filling a mirror wants the four files out of thirty that are
+// worth gigabytes.
+func (f File) IsWeights() bool {
+	for _, ext := range []string{".gguf", ".safetensors", ".bin", ".pt", ".pth"} {
+		if strings.HasSuffix(strings.ToLower(f.Path), ext) {
+			return true
+		}
+	}
+	return false
+}
+
+// Repo is a repository's current revision and what is in it.
+type Repo struct {
+	ID string
+	// Revision is the commit the listing came from.
+	//
+	// Carried so a mirrored copy can name it. Fetching from "main" records
+	// nothing about what was fetched, and a repository that is re-uploaded
+	// under the same branch leaves two different sets of weights with the same
+	// provenance and no way to tell them apart.
+	Revision string
+	Gated    Gated
+	Files    []File
+}
+
+// ShortRevision is the revision as a person would write it.
+func (r Repo) ShortRevision() string {
+	if len(r.Revision) > 8 {
+		return r.Revision[:8]
+	}
+	return r.Revision
+}
+
+// Repo reads a repository's revision and file list.
+func (c *Client) Repo(ctx context.Context, id string) (Repo, error) {
+	id = strings.Trim(strings.TrimSpace(id), "/")
+	if !strings.Contains(id, "/") {
+		return Repo{}, fmt.Errorf("%q is not a repository; it looks like owner/name", id)
+	}
+
+	var meta struct {
+		ID    string `json:"id"`
+		SHA   string `json:"sha"`
+		Gated Gated  `json:"gated"`
+	}
+	if err := c.get(ctx, "/api/models/"+id, &meta); err != nil {
+		return Repo{}, err
+	}
+
+	var tree []struct {
+		Type string `json:"type"`
+		Path string `json:"path"`
+		Size int64  `json:"size"`
+		LFS  *struct {
+			OID  string `json:"oid"`
+			Size int64  `json:"size"`
+		} `json:"lfs"`
+	}
+	// By revision rather than by branch, so the listing and the URLs built from
+	// it describe the same commit even if the branch moves in between.
+	rev := meta.SHA
+	if rev == "" {
+		rev = "main"
+	}
+	if err := c.get(ctx, "/api/models/"+id+"/tree/"+rev, &tree); err != nil {
+		return Repo{}, err
+	}
+
+	r := Repo{ID: meta.ID, Revision: meta.SHA, Gated: meta.Gated}
+	if r.ID == "" {
+		r.ID = id
+	}
+	for _, e := range tree {
+		if e.Type != "file" {
+			continue
+		}
+		f := File{Path: e.Path, Bytes: e.Size}
+		if e.LFS != nil {
+			f.Bytes, f.SHA256 = e.LFS.Size, e.LFS.OID
+		}
+		r.Files = append(r.Files, f)
+	}
+	return r, nil
+}
+
+// DownloadURL is where the mirror should fetch a file from.
+//
+// Pinned to the revision the listing came from. "main" would mean the mirror
+// fetches whatever is there when it gets round to it, which is not necessarily
+// what was on screen when somebody pressed the button.
+func (r Repo) DownloadURL(base, path string) string {
+	if base == "" {
+		base = DefaultEndpoint
+	}
+	rev := r.Revision
+	if rev == "" {
+		rev = "main"
+	}
+	return strings.TrimSuffix(base, "/") + "/" + r.ID + "/resolve/" + rev + "/" + path
+}
+
+func (c *Client) get(ctx context.Context, path string, out any) error {
+	req, err := http.NewRequestWithContext(ctx, http.MethodGet, c.base+path, nil)
+	if err != nil {
+		return err
+	}
+	resp, err := c.hc.Do(req)
+	if err != nil {
+		return fmt.Errorf("reaching the model registry: %w", err)
+	}
+	defer resp.Body.Close()
+
+	// Capped: this is a third party, and a page that renders what it sends
+	// should not be handed a gigabyte of it.
+	body, err := io.ReadAll(io.LimitReader(resp.Body, 4<<20))
+	if err != nil {
+		return err
+	}
+	switch {
+	case resp.StatusCode == http.StatusNotFound:
+		return fmt.Errorf("the registry has nothing at %s", path)
+	case resp.StatusCode == http.StatusUnauthorized, resp.StatusCode == http.StatusForbidden:
+		// Said plainly, because the next thing somebody does is check their
+		// network. It is the repository, not the connection.
+		return fmt.Errorf("the registry refused this anonymously; the repository is gated or private")
+	case resp.StatusCode >= 300:
+		return fmt.Errorf("the registry answered %d", resp.StatusCode)
+	}
+	if err := json.Unmarshal(body, out); err != nil {
+		return fmt.Errorf("the registry's answer was not the JSON expected: %w", err)
+	}
+	return nil
+}
+
+// shardPattern matches the "-00002-of-00004" a split file carries before its
+// extension. Five digits in practice, but not required: the count is what
+// matters and a repository that used three would otherwise list every part as
+// a separate model.
+var shardPattern = regexp.MustCompile(`^(.*)-(\d+)-of-(\d+)(\.[^.]+)$`)
+
+// FileSet is one thing worth fetching: a file, or every shard of one.
+//
+// The distinction the page turns on. Two GGUF files in a repository are usually
+// alternatives — q4_k_m or q8_0, you want one — while two safetensors files are
+// halves of the same weights and useless apart. Fetching "all the weights" is
+// right for the second and wasteful for the first, so neither is offered:
+// a set is offered, and a set is whole.
+type FileSet struct {
+	// Name is what to call it. For shards it keeps the -of- notation, because
+	// a repository can hold both the split and the joined copy of the same
+	// quantisation and they must not read as one entry.
+	Name   string
+	Files  []File
+	Bytes  int64
+	Shards int
+}
+
+// SHA256 is the checksum when there is one to give.
+//
+// Empty for a shard set: several files have several checksums and no single
+// value for the set, and showing one of them invites comparing it against the
+// wrong thing.
+func (s FileSet) SHA256() string {
+	if len(s.Files) == 1 {
+		return s.Files[0].SHA256
+	}
+	return ""
+}
+
+// Group collects shards into sets, leaving every other file on its own.
+//
+// Order is preserved from the listing, so a repository that sorts its files
+// sensibly still reads sensibly here.
+func Group(files []File) []FileSet {
+	var out []FileSet
+	at := map[string]int{}
+	for _, f := range files {
+		m := shardPattern.FindStringSubmatch(f.Path)
+		if m == nil {
+			out = append(out, FileSet{Name: f.Path, Files: []File{f}, Bytes: f.Bytes})
+			continue
+		}
+		prefix, total, ext := m[1], m[3], m[4]
+		name := prefix + "-*-of-" + total + ext
+		i, ok := at[name]
+		if !ok {
+			at[name] = len(out)
+			out = append(out, FileSet{Name: name})
+			i = at[name]
+		}
+		out[i].Files = append(out[i].Files, f)
+		out[i].Bytes += f.Bytes
+		out[i].Shards = len(out[i].Files)
+	}
+	return out
+}

internal/hf/hf_test.go

@@ -0,0 +1,189 @@
+package hf
+
+import (
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"testing"
+	"time"
+)
+
+// registry stands in for Hugging Face, answering the two endpoints used.
+func registry(t *testing.T, h http.HandlerFunc) *Client {
+	t.Helper()
+	srv := httptest.NewServer(h)
+	t.Cleanup(srv.Close)
+	return New(srv.URL, time.Second)
+}
+
+func TestTheRealSizeOfAWeightsFileIsTheLFSOne(t *testing.T) {
+	c := registry(t, func(w http.ResponseWriter, r *http.Request) {
+		if strings.HasSuffix(r.URL.Path, "/tree/abc123") {
+			_, _ = w.Write([]byte(`[
+			  {"type":"file","path":"README.md","size":900,"oid":"gitblob"},
+			  {"type":"file","path":"model.gguf","size":133,
+			   "lfs":{"oid":"deadbeef","size":1117320736}}]`))
+			return
+		}
+		_, _ = w.Write([]byte(`{"id":"Qwen/X","sha":"abc123"}`))
+	})
+
+	repo, err := c.Repo(t.Context(), "Qwen/X")
+	if err != nil {
+		t.Fatal(err)
+	}
+	var gguf File
+	for _, f := range repo.Files {
+		if f.Path == "model.gguf" {
+			gguf = f
+		}
+	}
+	// The plain size of an LFS file is the size of the pointer. Reading that
+	// one reports a gigabyte model as 133 bytes, which is the number somebody
+	// would then plan disk against.
+	if gguf.Bytes != 1117320736 {
+		t.Errorf("expected the LFS size, got %d", gguf.Bytes)
+	}
+	if gguf.SHA256 != "deadbeef" {
+		t.Errorf("expected the published checksum, got %q", gguf.SHA256)
+	}
+}
+
+func TestASmallFileHasNoChecksum(t *testing.T) {
+	c := registry(t, func(w http.ResponseWriter, r *http.Request) {
+		if strings.Contains(r.URL.Path, "/tree/") {
+			_, _ = w.Write([]byte(`[{"type":"file","path":"README.md","size":900,"oid":"gitblob"}]`))
+			return
+		}
+		_, _ = w.Write([]byte(`{"id":"Qwen/X","sha":"abc123"}`))
+	})
+
+	repo, _ := c.Repo(t.Context(), "Qwen/X")
+	// A git blob hash is not a checksum of the content, so reporting it as one
+	// would invite somebody to compare it with what the mirror computes and
+	// conclude the download was corrupt.
+	if repo.Files[0].SHA256 != "" {
+		t.Errorf("a git-stored file should publish no checksum, got %q", repo.Files[0].SHA256)
+	}
+}
+
+func TestTheDownloadURLIsPinnedToTheRevisionThatWasListed(t *testing.T) {
+	r := Repo{ID: "Qwen/X", Revision: "abc123def456"}
+	got := r.DownloadURL("https://hf.example", "model.gguf")
+
+	// "main" would mean the mirror fetches whatever is there when it gets round
+	// to it, which is not necessarily what was on screen when the button was
+	// pressed.
+	if !strings.Contains(got, "/resolve/abc123def456/") {
+		t.Errorf("the URL should pin the revision; got %s", got)
+	}
+}
+
+func TestAGatedRepositorySaysSoRatherThanBlamingTheNetwork(t *testing.T) {
+	c := registry(t, func(w http.ResponseWriter, _ *http.Request) {
+		w.WriteHeader(http.StatusUnauthorized)
+	})
+
+	_, err := c.Repo(t.Context(), "meta/gated")
+	if err == nil || !strings.Contains(err.Error(), "gated or private") {
+		t.Errorf("expected the reason to be named; got %v", err)
+	}
+}
+
+func TestOneGatedResultDoesNotHideThePage(t *testing.T) {
+	c := registry(t, func(w http.ResponseWriter, _ *http.Request) {
+		// The field is documented as a boolean and is a word in practice.
+		// Decoding it strictly fails the whole response, so one gated model in
+		// a page of twenty would hide the other nineteen.
+		_, _ = w.Write([]byte(`[{"id":"a","gated":false},{"id":"b","gated":"manual"}]`))
+	})
+
+	got, err := c.Search(t.Context(), "qwen", 20)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if len(got) != 2 {
+		t.Fatalf("expected both results, got %d", len(got))
+	}
+	if got[0].Gated.Is() || !got[1].Gated.Is() {
+		t.Errorf("gating did not decode: %+v", got)
+	}
+}
+
+func TestSomethingThatIsNotARepositoryIsRefusedBeforeTheCall(t *testing.T) {
+	called := false
+	c := registry(t, func(http.ResponseWriter, *http.Request) { called = true })
+
+	if _, err := c.Repo(t.Context(), "qwen"); err == nil {
+		t.Error("a bare name is not owner/name and should be refused")
+	}
+	if called {
+		t.Error("nothing should have been requested")
+	}
+}
+
+func TestWeightsAreToldApartFromDocumentation(t *testing.T) {
+	// Crude and right: the registry does not label them, and an operator
+	// filling a mirror wants the four files out of thirty worth gigabytes.
+	for _, f := range []string{"model.gguf", "model-00001-of-00002.safetensors", "pytorch_model.bin"} {
+		if !(File{Path: f}).IsWeights() {
+			t.Errorf("%s should count as weights", f)
+		}
+	}
+	for _, f := range []string{"README.md", "config.json", ".gitattributes", "tokenizer.model"} {
+		if (File{Path: f}).IsWeights() {
+			t.Errorf("%s should not count as weights", f)
+		}
+	}
+}
+
+func TestShardsBecomeOneSetAndAlternativesStaySeparate(t *testing.T) {
+	// Straight from a real repository, which holds both the split and the
+	// joined copy of the same quantisation.
+	sets := Group([]File{
+		{Path: "qwen-fp16-00001-of-00004.gguf", Bytes: 1},
+		{Path: "qwen-fp16-00002-of-00004.gguf", Bytes: 1},
+		{Path: "qwen-fp16-00003-of-00004.gguf", Bytes: 1},
+		{Path: "qwen-fp16-00004-of-00004.gguf", Bytes: 1},
+		{Path: "qwen-fp16.gguf", Bytes: 4},
+		{Path: "qwen-q4_k_m.gguf", Bytes: 2},
+	})
+
+	if len(sets) != 3 {
+		t.Fatalf("expected the shards as one and the two whole files apart, got %d: %+v",
+			len(sets), sets)
+	}
+	if sets[0].Shards != 4 || sets[0].Bytes != 4 {
+		t.Errorf("the shard set should total its parts, got %+v", sets[0])
+	}
+	// Two GGUFs in a repository are alternatives: fetching both is wasting
+	// gigabytes on a copy nobody asked for.
+	if sets[1].Shards != 0 || sets[2].Shards != 0 {
+		t.Errorf("whole files are not shard sets: %+v %+v", sets[1], sets[2])
+	}
+}
+
+func TestASplitAndAJoinedCopyDoNotReadAsOneEntry(t *testing.T) {
+	sets := Group([]File{
+		{Path: "m-q4-00001-of-00002.gguf"}, {Path: "m-q4-00002-of-00002.gguf"},
+		{Path: "m-q4.gguf"},
+	})
+	if sets[0].Name == sets[1].Name {
+		t.Errorf("the split and the joined copy collided on %q", sets[0].Name)
+	}
+	if !strings.Contains(sets[0].Name, "-of-00002") {
+		t.Errorf("a shard set should keep the notation; got %q", sets[0].Name)
+	}
+}
+
+func TestAShardSetPublishesNoSingleChecksum(t *testing.T) {
+	sets := Group([]File{
+		{Path: "m-00001-of-00002.safetensors", SHA256: "aaa"},
+		{Path: "m-00002-of-00002.safetensors", SHA256: "bbb"},
+	})
+	// Several files have several checksums and no single value for the set;
+	// showing one invites comparing it against the wrong thing.
+	if got := sets[0].SHA256(); got != "" {
+		t.Errorf("expected no checksum for a set, got %q", got)
+	}
+}

internal/mirror/add.go

@@ -0,0 +1,158 @@
+package mirror
+
+import (
+	"context"
+	"fmt"
+	"io"
+	"net/http"
+	"os"
+	"path/filepath"
+	"strings"
+	"time"
+)
+
+// Added is what a download produced.
+type Added struct {
+	Entry  Entry
+	SHA256 string
+}
+
+// Add downloads a source into the mirror at model/version/file.
+//
+// Written to a temporary name and renamed when complete, for the same reason
+// the agent's fetcher does it: a mirror is copied to every node, so a partial
+// file here is a partial file everywhere, and a rename is the only step that
+// is atomic. An interrupted download leaves a `.part` that List ignores rather
+// than something a catalog could be pointed at.
+//
+// It refuses to replace an existing version. That refusal is the whole point of
+// the layout — a version that can be overwritten takes away the ability to
+// change the catalog back and still have the old weights.
+func Add(ctx context.Context, root, source string, e Entry, progress func(done, total int64)) (Added, error) {
+	if e.Model == "" || e.Version == "" || e.File == "" {
+		return Added{}, fmt.Errorf("a model, a version and a filename are all required")
+	}
+	if strings.ContainsAny(e.Model+e.Version+e.File, `/\`) {
+		// Each is one path element. A slash would put the file somewhere the
+		// layout does not describe, and List would not find it again.
+		return Added{}, fmt.Errorf("model, version and filename are single names, not paths")
+	}
+
+	final := e.Path(root)
+	if _, err := os.Stat(final); err == nil {
+		return Added{}, fmt.Errorf("%s/%s/%s: %w", e.Model, e.Version, e.File, ErrExists)
+	}
+	if err := os.MkdirAll(filepath.Dir(final), 0o755); err != nil {
+		return Added{}, err
+	}
+
+	req, err := http.NewRequestWithContext(ctx, http.MethodGet, source, nil)
+	if err != nil {
+		return Added{}, err
+	}
+	resp, err := http.DefaultClient.Do(req)
+	if err != nil {
+		return Added{}, fmt.Errorf("fetching %s: %w", source, err)
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode != http.StatusOK {
+		return Added{}, fmt.Errorf("%s answered %s", source, resp.Status)
+	}
+
+	part := final + partSuffix
+	f, err := os.OpenFile(part, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0o644) //nolint:gosec // inside the mirror root
+	if err != nil {
+		return Added{}, err
+	}
+	written, err := copyWithProgress(f, resp.Body, resp.ContentLength, progress)
+	closeErr := f.Close()
+	if err != nil {
+		_ = os.Remove(part)
+		return Added{}, err
+	}
+	if closeErr != nil {
+		_ = os.Remove(part)
+		return Added{}, closeErr
+	}
+	// A truncated download that the server ended cleanly would otherwise be
+	// renamed into place and copied to every node.
+	if resp.ContentLength > 0 && written != resp.ContentLength {
+		_ = os.Remove(part)
+		return Added{}, fmt.Errorf("%s: got %d bytes of %d", source, written, resp.ContentLength)
+	}
+
+	sum, err := sha256File(part)
+	if err != nil {
+		_ = os.Remove(part)
+		return Added{}, err
+	}
+	if err := os.Rename(part, final); err != nil {
+		return Added{}, err
+	}
+	e.Bytes = written
+	return Added{Entry: e, SHA256: sum}, nil
+}
+
+// copyWithProgress copies and reports occasionally.
+//
+// Occasionally rather than continuously: a forty gigabyte download is a
+// hundred thousand chunks, and a line each would bury whatever else the
+// operator is watching.
+func copyWithProgress(dst io.Writer, src io.Reader, total int64, report func(done, total int64)) (int64, error) {
+	buf := make([]byte, 1<<20)
+	var done int64
+	last := time.Now()
+
+	for {
+		n, err := src.Read(buf)
+		if n > 0 {
+			w, werr := dst.Write(buf[:n])
+			done += int64(w)
+			if werr != nil {
+				return done, werr
+			}
+			if report != nil && time.Since(last) > 2*time.Second {
+				report(done, total)
+				last = time.Now()
+			}
+		}
+		if err == io.EOF {
+			if report != nil {
+				report(done, total)
+			}
+			return done, nil
+		}
+		if err != nil {
+			return done, err
+		}
+	}
+}
+
+// Remove deletes one version, and the model directory if that was the last of
+// it.
+//
+// Leaving an empty directory behind would make `list` show a model that is not
+// there, which is the kind of small lie that costs somebody an afternoon.
+func Remove(root string, e Entry) error {
+	path := e.Path(root)
+	if err := os.Remove(path); err != nil {
+		return err
+	}
+	versionDir := filepath.Dir(path)
+	if empty, _ := isEmpty(versionDir); empty {
+		_ = os.Remove(versionDir)
+	}
+	modelDir := filepath.Dir(versionDir)
+	if empty, _ := isEmpty(modelDir); empty {
+		_ = os.Remove(modelDir)
+	}
+	return nil
+}
+
+func isEmpty(dir string) (bool, error) {
+	entries, err := os.ReadDir(dir)
+	if err != nil {
+		return false, err
+	}
+	return len(entries) == 0, nil
+}