Describe v0.8.3

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

.gitea/workflows/release.yml

@@ -150,3 +150,31 @@ jobs:
               || { echo "upload of $(basename "$f") was refused" >&2; exit 1; }
           done
           echo "attached $(ls dist | wc -l) files to $tag"
+
+  # The release page says what changed, copied from CHANGELOG.md. Gitea makes
+  # an empty release for every tag and the job above only attaches files, so
+  # every release until this existed was a list of binaries and nothing else.
+  #
+  # After the build, because that is what guarantees the release exists. Run by
+  # hand from a branch it syncs every version in the changelog, which is how
+  # the releases made before this got their notes.
+  notes:
+    needs: build
+    runs-on: ubuntu-latest
+    permissions:
+      contents: write
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Copy the changelog onto the release
+        env:
+          GITEA_TOKEN: ${{ secrets.GITEA_TOKEN }}
+          GITEA_INSTANCE: ${{ github.server_url }}
+          GITEA_REPO: ${{ github.repository }}
+        run: |
+          set -euo pipefail
+          if [ "${GITHUB_REF_TYPE:-}" = "tag" ]; then
+            python3 scripts/gitea/release_notes.py --tag "${GITHUB_REF_NAME}"
+          else
+            python3 scripts/gitea/release_notes.py
+          fi

CHANGELOG.md

@@ -11,11 +11,293 @@ is only released once its **exit criterion** is met, not when its code lands.
 | v0.4.0 | 2 | Agent: GPU/IB discovery, single-node engines | released as part of v0.5.0 |
 | v0.5.0 | 3 | Control plane: catalog, placement, reconciler | released |
 | v0.6.0 | 4 | Multi-node replicas and fabric awareness | released |
+| v0.7.0 | — | Conversations, agent memory, disk management | released |
+| v0.8.0 | — | PostgreSQL, leader election, agent failover | released |
 | v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
 
 Until v1.0.0 the minor version is the breaking one: configuration formats and Go
 APIs change between minors without a deprecation period.
 
+## v0.8.3 — 2026-09-28
+
+Found by building a node from nothing with v0.8.2.
+
+### Fixes
+
+- **A new node's first engine install failed on apt's lock.** cloud-init and
+  unattended-upgrades run apt in a machine's first minutes, and
+  `DPkg::Lock::Timeout` does not cover the lock `apt-get update` takes. A
+  package command held up by a lock now waits for up to ten minutes, and says
+  so; a failed install is retried after a minute rather than ten.
+- **A new node reported no disk** until its first download created the weights
+  directory. It is created when the agent starts.
+- **Onboarding took one control plane address.** It takes several, comma
+  separated, so a new node fails over like the rest.
+
+## v0.8.2 — 2026-09-28
+
+A node no longer runs whatever engine somebody once put on it. It installs its
+own.
+
+### Engines are installed by the node
+
+`engines:` in server.yaml names the engines every node has and their versions,
+defaulting to the llama.cpp build this release was checked against (b10991).
+Each agent converges on it: it picks the build for its hardware — CUDA with its
+runtime when an NVIDIA driver is loaded, CPU otherwise — installs the OS
+packages that build needs with the machine's own package manager, downloads the
+archive and checks it against the sha256 GitHub publishes, unpacks it refusing
+anything that would land outside its directory, and runs it. `ldd` names a
+missing library or a system too old for the build ("libstdc++.so.6 is too old
+(needs GLIBCXX_3.4.30)"); `--version` proves the rest. Installs are checked
+again every ten minutes.
+
+An engine is reported only once installed and seen to run, or when the agent's
+configuration names it on purpose. A node that has installed nothing yet is not
+a candidate for any model, where before it was a candidate for all of them.
+`ctl nodes`, the cluster page and placement's refusals say where each
+installation stands, and warn about a CPU without AVX2.
+
+Not every system is refused for its name: the build runs on Ubuntu 22.04+,
+Debian 12+ and RHEL-family 10, and `ldd` decides. vLLM and SGLang are not yet
+installed this way and must still be named in the agent's configuration.
+
+### Onboarding
+
+Onboarding installs llmgw from the CI release for the node's architecture,
+checked against the release's checksums, and writes an agent configuration of
+only the control plane's addresses and the TLS files.
+
+### Fixes
+
+- A race on the weights fingerprint, which two heartbeat loops touched during a
+  failover.
+
+## v0.8.1 — 2026-09-28
+
+Nothing here changes a configuration format: every file that worked with
+v0.8.0 still works. What changes is how much of it needs writing, and how much
+of running the cluster can be done from a page rather than a shell.
+
+### A node describes itself
+
+An agent file used to name its node, its address, the path to llama-server and
+its weight directories — different on every node and kept true by hand. The
+agent now finds these out: its id is the hostname, its address is the one on
+the route to the control plane, llama-server is looked for on PATH and in
+`/opt/llama.cpp`, a node without nvidia-smi simply has no NVIDIA cards, and
+fetched weights land in `/var/lib/llmgw-weights`, which is always scanned.
+Memory is re-read on every heartbeat and installed engines once a minute, so a
+VM given more RAM or a node given an engine becomes a candidate without an
+agent restart; the control plane logs both. The agent file shrinks to the
+control plane's addresses and the TLS files, the same on every node.
+
+### Models from the mirror, without writing YAML
+
+Search the registry from `/admin/mirror`, fetch a sharded model in one action,
+and serve a mirrored version from its row — name, source, engine and an
+estimated memory requirement filled in. A model can say what it is for, and
+`/v1/models` shows it. Placement no longer hands the same memory to two models
+or puts an engine on a node that lacks it, and says when a replica runs an
+edited profile.
+
+### Tenancy, quotas and usage on pages
+
+The tenancy tree, quotas and budgets are managed from the admin pages; an
+organization's budget bounds its teams, a group's quota counts against the
+group, and a budget that cannot be spent against says so. `/admin/usage`
+charts requests, tokens and spend over time by organization, team, project or
+person, and a request is priced by the model that served it.
+
+### Watching the cluster
+
+The cluster page has its own charts, backed by a year of Prometheus history,
+with alerts that were made to fire before being trusted — including for the
+disk the metrics share with the models.
+
+### Chat
+
+The picker offers each model once rather than once per alias, never asks for
+more tokens than the model allows, and offers a model that loads on demand —
+sending it a message is what loads it.
+
+## v0.8.0 — 2026-09-27
+
+The store stops being a file on one machine's disk. PostgreSQL, a way to move
+an existing installation onto it without reissuing every key, and the two
+things a second control plane needs so that running one is safe rather than
+clever.
+
+### A second control plane, and a way to fail over to it
+
+Leadership is a lease in the store, so two control planes against one database
+cannot both place work — which they otherwise would, both seeing a model
+wanting replicas and none running, leaving twice the replicas each holding GPUs
+the other believes are free.
+
+A standby also refuses agent streams. Commands travel down the stream the agent
+opened, so a standby holding one is a node the leader cannot command, and the
+leader's placements would fail with "no agent connected" for a node that is
+plainly connected. Agents therefore take a list of addresses rather than one,
+trying each in turn: a standby refusing, a stopped control plane refusing the
+connection and a crashed one never answering are the same thing from a node.
+
+Measured on the cluster. Leadership moved three seconds after the leader was
+stopped — it released the lease on the way out rather than letting it lapse —
+and all five agents had registered with the standby six seconds later. The
+standby adopted every replica rather than restarting any: six replicas, all
+ready, engines that never stopped serving.
+
+Everything is unset-means-leader, so a single control plane and every
+deployment that existed before this behave exactly as they did.
+
+What this does not do is move client traffic. The standby served inference
+correctly on its own port while the public address returned 502, because
+nothing in front of them moved — that needs a load balancer or a shared
+address, and it is the remaining piece.
+
+### PostgreSQL
+
+The store runs on PostgreSQL, chosen by what the address looks like: a path is
+still SQLite, a `postgres://` URL is not, and `@` followed by a path reads the
+address out of a file so a DSN carrying a password stays out of a configuration
+that gets copied and pasted into issues. Every existing installation keeps
+working untouched.
+
+One set of queries rather than one per database, because the SQL here is almost
+entirely portable and two copies would drift one query at a time. The rewrite
+of placeholders is a wrapper around the handle rather than a change at sixty
+call sites — which found six places reaching past it, including all three
+manual transactions and two prepared statements, every one of which worked on
+SQLite and would have failed on PostgreSQL.
+
+Exactly one query is not the same sentence in both: the note search, MATCH
+against `@@`. SQLite gets full-text search from an FTS5 table kept in step by
+three triggers; PostgreSQL derives a tsvector from the row, so there is nothing
+to keep in step and no trigger to forget. It uses the 'simple' configuration
+rather than English — these are an agent's own notes, in whatever language it
+chose, and stemming English rules over Turkish would quietly stop finding
+things.
+
+`llmgw keys move-store` copies an existing store rather than starting a fresh
+one, because keys are stored hashed: a migration that drops them cannot
+recreate them, and every person and every script would need a new one.
+
+Testing it against a real database found three things nothing else would have.
+`schema_migrations` is created in Go rather than in a migration file, so it
+escaped the port and still said INTEGER — 64 bits in SQLite, 32 in PostgreSQL,
+and a millisecond timestamp does not fit. The copy order followed the order the
+migrations create tables in, and `users` is created after `api_keys` even
+though `api_keys` references it. And `key_claims.sealed` is emptied the moment
+a claim is used, so a claimed row holds an empty blob in a NOT NULL column — an
+empty blob scans as a typed nil slice and a real NULL as an untyped one, a
+difference pgx cannot see, so it arrived as NULL.
+
+The driver is pgx, which is pure Go: the whole binary is built with
+CGO_ENABLED=0 so that it runs on a node whose libc is older than the builder's,
+and a cgo driver would take that away from every deployment, including the ones
+staying on SQLite.
+
+## v0.7.0 — 2026-09-25
+
+What a cluster needs to be lived with rather than demonstrated: somewhere to
+keep a conversation, somewhere to keep a note, and an honest answer about the
+disk underneath both.
+
+### Memory notes for agents, and threads without API keys
+
+Notes an agent writes for itself and reads back in later conversations, offered
+as REST (`/v1/memory`) and as MCP tools at `/mcp` (`remember`, `recall`,
+`forget`). A note is a key, text and tags, searched by words or tag, private to
+a project and to the agent id in `x-llmgw-agent`. llmgw does not inject the
+tools into chat requests: an agent that is not given the server never sees
+memory.
+
+Off for every project until someone runs `llmgw keys memory --retain-hours N`,
+and the retention is required, as it is for threads.
+
+On a gateway with no authentication, a `threads:` block (`store`,
+`retain_hours`) now keeps conversations and notes in one shared pool. Before,
+`x-llmgw-thread` was silently ignored there; now a gateway configured to keep
+nothing refuses it with an explanation instead. The shared pool is for one
+trusted machine: anyone who can reach the port and knows an id can read it.
+
+### Conversations that can move between models
+
+A thread a client names with `x-llmgw-thread`, carrying only its new turn.
+llmgw replays the stored history to whichever model is serving, which is what
+lets a conversation begun on a large model continue on a small one — including
+when a spent budget moved it there and the caller never chose.
+
+Off for every project until someone runs `llmgw keys threads`. Until then llmgw
+stores no message content anywhere, as it never has, and that default is worth
+more than the feature. The retention is required rather than optional, because
+a conversation store with no expiry is a transcript of everything a team ever
+typed kept forever by omission, and it is measured from the last turn so a
+conversation in progress does not expire underneath it.
+
+History that does not fit the serving model is trimmed oldest-first and the
+reply says how many turns went, in `x-llmgw-thread-trimmed`. System messages
+are never dropped — losing one silently changes what the model is for while it
+carries on answering — and neither is the turn the caller just sent.
+
+Threads are scoped to a project, and reading, writing or deleting one that
+belongs to another answers exactly as one that never existed: two answers would
+let a caller map which ids are real in projects they cannot read.
+
+Capturing the reply is opt-in and bounded, so a request without a thread
+accumulates nothing on the hottest path in the system, and one with a thread
+stops at a cap rather than growing to it.
+
+`DELETE /v1/threads/{id}` forgets a conversation, as a client rather than as an
+operator: the person who said the thing should not have to ask someone to read
+it before it can go.
+
+### Disk, and taking back only what llmgw put there
+
+Disk was the one resource nothing reported. `statfs` was already being called
+on every model directory, but only to ask whether the filesystem was a network
+one — the free space in the same struct was discarded, so the only way to find
+out why a node was filling up was to ssh to it.
+
+Nodes now report it, per filesystem rather than per configured directory: free
+space belongs to the disk, and two model directories on one disk are one pool
+of bytes. Shared filesystems are kept out of a node's own figure, because space
+on a filer belongs to everyone mounting it and twenty nodes would otherwise
+each report the same free terabyte. It shows in `ctl nodes`, the dashboard and
+Prometheus, and a node that reported nothing reads as unknown everywhere — an
+older agent must not look like a full disk.
+
+A profile may declare `min_disk_gb`, and a node without the room is no longer a
+candidate, with a reason `explain-placement` prints. Skipped on a node that
+already holds the weights: it needs no room for them, and without that the node
+best placed to run a model is the one most likely to be rejected.
+
+`ctl weights` lists model files llmgw downloaded for models no longer in the
+catalog, and `--collect` deletes them. The design is mostly about what it will
+not do. Only files llmgw fetched are candidates, and that is a calculation
+rather than a heuristic — the cache name is a hash of the source, so the wanted
+set is computable exactly. Anything an operator put on a node is left alone
+however unreferenced it looks. Each node checks every path again before
+unlinking it, refusing anything outside its cache directory or without the
+cache name, so a control plane that is wrong cannot talk a node into losing a
+model that took a week to copy.
+
+### Fixes
+
+- **A capability was lost at an indirection.** The controller is handed a
+  forwarding dispatcher, and reaches past its interface by type assertion to
+  find what lies beyond scheduling. Deleting weights was implemented on the
+  agent service and not on the forwarder, so it was simply absent at runtime —
+  with the whole suite green, because the dispatchers the tests use do not
+  implement it either and its absence there is the expected case.
+- **`ctl tool-check` blamed the engine for what the model did.** It had one
+  verdict for a failed attempt and told operators not to try another model.
+  Measured: the engine renders the tool prompt correctly, the coder models do
+  not use the `<tool_call>` wrapper it asks for, and `qwen2.5-3b-instruct` on
+  the same engine returns a populated `tool_calls` array. A failed attempt is
+  now two verdicts, told apart by whether the reply used the wrapper.
+
 ## v0.6.0 — 2026-09-18
 
 Phase 4: a replica can be bigger than one machine, and the fabric it spans

README.md

@@ -30,6 +30,8 @@ Releases track the roadmap phases; see [CHANGELOG.md](CHANGELOG.md).
 | v0.4.0 | 2 | Agent: GPU/IB discovery, engine supervision | released |
 | v0.5.0 | 3 | Control plane: catalog, placement, reconciler, ctl | released |
 | v0.6.0 | 4 | Multi-node placement and fabric awareness | released |
+| v0.7.0 | — | Conversations, agent memory, disk management | released |
+| v0.8.0 | — | PostgreSQL, leader election, agent failover | released |
 | v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
 
 As of v0.6.0 a cluster runs unattended: agents report their hardware, the
@@ -37,13 +39,38 @@ control plane places replicas — including ones larger than a single machine
 models load when someone asks for them and stop when nobody does, and keys,
 budgets and rate limits decide who may use any of it.
 
-Before v1.0.0: routing profiles selectable per request, spillover to a fallback
-pool, tracing, and reference documentation. One thing is worth saying plainly:
+Before v1.0.0: tracing, reference documentation, and filtering on `ctl nodes`
+and `ctl replicas` for fleets too large to read a whole table of. Two items
+that were on this list are not gaps: a model falls through to its fallbacks
+when there is no capacity as well as when a budget is spent, and routing
+profiles are chosen by the key rather than by a request header on purpose —
+a header would let any client promote itself into the top priority band.
+
+One thing is worth saying plainly:
 multi-node replicas are proven on a simulated cluster and on real hardware
 without a fabric. The InfiniBand, NVML and NCCL paths are written against the
 formats and failure modes those tools actually have, but they have not yet run
 on a machine that has them.
 
+## Documentation
+
+- [docs/install.md](docs/install.md) — installing from nothing: control plane,
+  nodes, engines, weights, first request
+- [docs/quickstart.md](docs/quickstart.md) — the short path
+- [docs/choosing-models.md](docs/choosing-models.md) — which models to run,
+  how many of each, and where the weights should live
+- [docs/data.md](docs/data.md) — what llmgw stores, what it does not, and
+  what leaves the machine; written for a security review
+- [docs/reference.md](docs/reference.md) — every configuration key, generated
+  from the structs and checked by a test, so it cannot describe a setting that
+  does not exist or miss one that does
+- [deploy/mirror/](deploy/mirror/) — a model mirror, which is the cheapest
+  and most useful thing to build around llmgw
+- [deploy/ha/](deploy/ha/) — running two control planes, and the one health
+  check that has to be right
+- [docs/node-onboarding.md](docs/node-onboarding.md),
+  [docs/onboarding.md](docs/onboarding.md) — adding machines, and adding people
+
 ## Building
 
 ```sh

api/proto/llmgw/v1/agent.proto

@@ -33,6 +33,8 @@ message ControlMessage {
     StopShard stop = 2;
     DrainShard drain = 3;
     Ack ack = 4;
+    RemoveWeights remove_weights = 5;
+    Provision provision = 6;
   }
 }
 
@@ -90,6 +92,61 @@ message Register {
   // nothing — an agent with no model directories configured behaves exactly as
   // every agent did before this existed, and constrains nothing.
   repeated LocalWeights weights = 11;
+
+  // storage is the disk behind those weights, one entry per filesystem.
+  //
+  // Reported so an operator can see a node filling up before a fetch dies at
+  // ninety percent, and so placement can refuse a node that cannot hold a
+  // model rather than discovering it during the download.
+  //
+  // Empty means the node was not asked to look, exactly as for weights, and
+  // constrains nothing.
+  repeated Storage storage = 12;
+
+  // host is what the machine is, so the control plane can have the mirror
+  // fetch software built for it rather than assuming every node matches the
+  // first.
+  Host host = 13;
+
+  // engines_probed says engine_versions is an answer. Without it an empty map
+  // meant "an agent too old to say" and constrained nothing — so a freshly
+  // built node with no engine installed yet was a candidate for every model.
+  bool engines_probed = 14;
+}
+
+// Host describes the operating system and processor of a node.
+message Host {
+  // From /etc/os-release: "ubuntu", "24.04", "noble".
+  string os_id = 1;
+  string os_version = 2;
+  string os_codename = 3;
+  // Debian's spelling, which is what package indexes are keyed by: "amd64",
+  // "arm64".
+  string arch = 4;
+  // The vector extensions inference cares about, of those the CPU has. A VM
+  // given QEMU's default CPU model has none of them, and runs, and is five
+  // times slower, and nothing says why.
+  repeated string cpu_flags = 5;
+  // nvidia_driver is the driver version, empty when there is none. A card
+  // without a driver is not a GPU anything can use.
+  string nvidia_driver = 6;
+}
+
+// Storage is one filesystem a node keeps weights on.
+//
+// Per filesystem rather than per configured directory: free space belongs to
+// the disk, and two model directories on one disk are one pool of bytes.
+message Storage {
+  // path is one directory on this filesystem, so the operator recognises it.
+  string path = 1;
+  // free_bytes is what a non-root process can write — not the root reserve,
+  // which is space the agent cannot have.
+  int64 free_bytes = 2;
+  int64 total_bytes = 3;
+  // shared marks a filesystem the node did not fill. Free space on one is
+  // shared with every other node using it, so two nodes each reporting 500 GB
+  // free on the same filer have 500 GB between them and not a terabyte.
+  bool shared = 4;
 }
 
 // RunningShard describes an engine already running on a node.
@@ -117,6 +174,13 @@ message RunningShard {
 
   string leader_addr = 13;
   int32 leader_port = 14;
+
+  // profile_fp is the fingerprint of the profile this shard was started from,
+  // echoed back from StartShard. The control plane keeps no record of a running
+  // replica across its own restart, so without this a shard adopted after a
+  // restart could not be compared with the catalog — which is exactly when the
+  // catalog is most likely to have been edited.
+  string profile_fp = 15;
 }
 
 message GPU {
@@ -171,6 +235,71 @@ message Heartbeat {
   // a card after registration.
   repeated int32 gpu_memory_used_mb = 2;
   repeated string new_problems = 3;
+
+  // storage for the same reason the GPU memory is here: it changes while the
+  // agent runs, and a figure sampled once at registration is a figure that is
+  // wrong on exactly the node worth watching — the one busy filling up.
+  //
+  // Always sent, because it is a few numbers per filesystem.
+  repeated Storage storage = 4;
+
+  // weights only when they changed, because the list is long enough that
+  // sending it every few seconds would be paying for nothing most of the time.
+  //
+  // weights_changed is what says the list is meant: without it an empty list
+  // would be ambiguous between "this node now holds nothing" and "nothing to
+  // report", and the first would quietly empty a node's inventory.
+  bool weights_changed = 5;
+  repeated LocalWeights weights = 6;
+
+  // memory_total_mb for the same reason again: RAM is added to a virtual
+  // machine while it runs, or across a reboot the agent does not notice, and a
+  // figure read once at registration kept a node that had grown to 32 GB
+  // looking like the 7 GB machine it used to be. 0 means not read.
+  int64 memory_total_mb = 7;
+
+  // engine_versions only when they changed, like weights. An engine installed
+  // after the agent started is otherwise invisible to placement until
+  // somebody restarts the agent — which is a step nobody knows to take.
+  bool engines_changed = 8;
+  map<string, string> engine_versions = 9;
+
+  // provision is where installing the node's software stands, sent when it
+  // changed. The flag makes an empty list mean "nothing to install" rather
+  // than "nothing to say", as for weights.
+  bool provision_changed = 10;
+  repeated ProvisionStatus provision = 11;
+}
+
+// Provision is the software a node should have: which engines, at which
+// versions. Nothing more.
+//
+// Everything else is the node's to work out, because only the node knows it:
+// which build fits its hardware, where that build is published, what its
+// operating system is missing, and whether the result actually runs. The agent
+// converges on this — installs what is absent, checks what is there — and
+// reports each engine through ProvisionStatus. Nothing is taken on trust from
+// what happens to be on the machine: an engine is offered to placement only
+// once the agent has installed it and seen it run.
+message Provision {
+  repeated EngineWant engines = 1;
+}
+
+// EngineWant names one engine at one version.
+message EngineWant {
+  string engine = 1;  // "llamacpp"
+  string version = 2; // "b10991"
+}
+
+// ProvisionStatus is where one engine's installation stands on a node.
+message ProvisionStatus {
+  string engine = 1;
+  string version = 2;
+  string variant = 3;
+  // "installing", "ready", "failed", or "unsupported" when no build fits.
+  string state = 4;
+  // detail says what is happening or what went wrong, in words for a person.
+  string detail = 5;
 }
 
 // ShardStatus reports a shard's lifecycle transition.
@@ -185,6 +314,15 @@ message ShardStatus {
   // fails to load this is the only diagnosis available.
   repeated string last_output = 7;
 
+  // fetch is how far a download has got, when the shard is waiting on one.
+  //
+  // The agent logs this already, on the node, where nobody is looking. From
+  // the control plane a replica pulling a hundred gigabytes and one that is
+  // wedged are the same thing — a shard that has been starting for ten
+  // minutes — and telling them apart is the difference between waiting and
+  // going to look.
+  FetchProgress fetch = 8;
+
   enum State {
     STATE_UNSPECIFIED = 0;
     STATE_PENDING = 1;
@@ -197,6 +335,16 @@ message ShardStatus {
   }
 }
 
+// FetchProgress is a download in flight.
+message FetchProgress {
+  string source = 1;
+  int64 bytes_done = 2;
+  // bytes_total is 0 when the server did not say how large the file is, which
+  // is a real answer: a percentage would then be invented, and an invented
+  // percentage that stops moving is worse than a byte count that keeps rising.
+  int64 bytes_total = 3;
+}
+
 // StartShard launches one shard. It is idempotent by shard_id: the control
 // plane retries, and a repeat must not produce a second process.
 message StartShard {
@@ -223,6 +371,10 @@ message StartShard {
   // are sent so the agent can echo them back in RunningShard after a restart.
   string pool = 13;
   string fabric = 14;
+
+  // profile_fp identifies the version of the profile, so the agent can say
+  // which one a running shard was started from. The agent never interprets it.
+  string profile_fp = 15;
 }
 
 message EngineProfile {
@@ -239,6 +391,17 @@ message EngineProfile {
   string served_model_name = 11;
 }
 
+// RemoveWeights deletes cached model files a node no longer needs.
+//
+// The paths are computed by the control plane from the catalog, but the agent
+// checks them again before unlinking anything: it deletes only files inside its
+// own cache directory whose names this system generated. A control plane that
+// is wrong, compromised or simply newer than the agent must not be able to
+// talk it into deleting a model an operator spent a week copying there.
+message RemoveWeights {
+  repeated string paths = 1;
+}
+
 // StopShard terminates a shard and frees its GPUs.
 message StopShard {
   string shard_id = 1;

deploy/bootstrap/add-node.sh

@@ -108,6 +108,5 @@ echo "== 2/2  onboarding $NAME"
 echo
 echo "Check it with: llmgw ctl nodes"
 echo
-echo "A node is not finished until the engines its profiles name are installed"
-echo "and the weights are on disk. Onboarding checks that a configured engine"
-echo "runs, but it installs neither — see docs/node-onboarding.md."
+echo "The agent installs its engines itself; the node takes no model until they"
+echo "are installed and seen to run. 'llmgw ctl nodes' shows where that stands."

deploy/bootstrap/onboard-node.sh

@@ -24,15 +24,18 @@ usage: $0 --node ID --addr HOST --server URL [options]
 
   --node ID         node id, hostname and certificate common name
   --addr HOST       how to reach the machine over ssh
-  --server URL      the control plane's agent endpoint, e.g. https://10.0.0.140:7443
+  --server URL      the control plane's agent endpoint, e.g. https://10.0.0.140:7443;
+                    several, comma separated, for a node that fails over
   --fabric NAME     InfiniBand fabric this node is attached to; repeatable as
                     --fabric NAME:hca1,hca2 . Omit for a node with no fabric,
                     which can host single-node replicas and will never be given
                     a multi-node one.
-  --mgmt-addr ADDR  address engines are reachable on (default: --addr)
-  --mgmt-if NAME    interface NCCL bootstraps over (default eth0)
-  --llamacpp-bin P  path to llama-server
-  --vllm-bin P      path to the vllm entry point
+  --mgmt-addr ADDR  address engines are reachable on (default: the node's
+                    address on the route to --server, found by the agent)
+  --mgmt-if NAME    interface NCCL bootstraps over (default: the one holding
+                    that address)
+  --llamacpp-bin P  use this llama-server instead of the one the agent installs
+  --vllm-bin P      path to the vllm entry point (not yet installed by the agent)
   --sglang-python P python interpreter of the SGLang virtualenv
   --nvidia-smi PATH where nvidia-smi lives, when it is not on PATH. Containerized
                     driver installs routinely put it somewhere else.
@@ -42,16 +45,19 @@ usage: $0 --node ID --addr HOST --server URL [options]
                     Releases carry static linux/amd64 and linux/arm64 builds;
                     downloading one is better than copying whatever happens to
                     be on the machine you are running this from.
+  --releases URL    where release binaries are downloaded from (default the
+                    project's Gitea releases)
   --ssh-user NAME   (default ilker)
 USAGE
   exit 2
 }
 
-NODE= ADDR= SERVER= MGMT= MGMT_IF=eth0
+NODE= ADDR= SERVER= MGMT_OVERRIDE= MGMT_IF_OVERRIDE=
 NVIDIA_SMI= IB_ROOT=/sys/class/infiniband
 LLAMACPP_BIN= VLLM_BIN= SGLANG_PYTHON=
 SSH_WAIT=180
 PKI=/etc/llmgw/pki BINARY= SSH_USER=ilker
+RELEASES=https://gitea.mynodes.xyz/ilker/llmgw/releases/download
 FABRIC_NAME= FABRIC_HCAS="mlx5_0,mlx5_1"
 
 while [ $# -gt 0 ]; do
@@ -60,8 +66,8 @@ while [ $# -gt 0 ]; do
     --addr) ADDR="$2"; shift 2;;
     --server) SERVER="$2"; shift 2;;
     --fabric) FABRIC_NAME="${2%%:*}"; [ "${2#*:}" != "$2" ] && FABRIC_HCAS="${2#*:}"; shift 2;;
-    --mgmt-addr) MGMT="$2"; shift 2;;
-    --mgmt-if) MGMT_IF="$2"; shift 2;;
+    --mgmt-addr) MGMT_OVERRIDE="$2"; shift 2;;
+    --mgmt-if) MGMT_IF_OVERRIDE="$2"; shift 2;;
     --llamacpp-bin) LLAMACPP_BIN="$2"; shift 2;;
     --vllm-bin) VLLM_BIN="$2"; shift 2;;
     --sglang-python) SGLANG_PYTHON="$2"; shift 2;;
@@ -69,12 +75,12 @@ while [ $# -gt 0 ]; do
     --ib-root) IB_ROOT="$2"; shift 2;;
     --pki) PKI="$2"; shift 2;;
     --binary) BINARY="$2"; shift 2;;
+    --releases) RELEASES="$2"; shift 2;;
     --ssh-user) SSH_USER="$2"; shift 2;;
     *) usage;;
   esac
 done
 [ -n "$NODE" ] && [ -n "$ADDR" ] && [ -n "$SERVER" ] || usage
-MGMT="${MGMT:-$ADDR}"
 BINARY="${BINARY:-$(command -v llmgw)}"
 [ -x "$BINARY" ] || { echo "no llmgw binary; pass --binary" >&2; exit 1; }
 [ -f "$PKI/ca-key.pem" ] || { echo "no authority in $PKI; run 'llmgw ctl cert init-ca' first" >&2; exit 1; }
@@ -130,68 +136,80 @@ $SSH "set -e
   rm -f /tmp/ca.pem /tmp/$NODE.pem /tmp/$NODE-key.pem"
 
 # 4. The binary, the configuration and the unit.
-say "installing the binary"
-scp -q $SSHOPTS "$BINARY" "$SSH_USER@$ADDR:/tmp/llmgw"
-$SSH 'sudo install -m 0755 /tmp/llmgw /usr/local/bin/llmgw && rm -f /tmp/llmgw'
+# The node fetches the release CI built for this version, for its own
+# architecture, and checks it against the release's checksums — rather than
+# being handed whatever binary is on the machine running this script. A build
+# that is not a release (a developer's, with commits past the tag) has no such
+# file, and is copied as before.
+TAG="$("$BINARY" --version 2>/dev/null | awk '{print $NF}')"
+if printf '%s' "$TAG" | grep -Eq '^v[0-9]+\.[0-9]+\.[0-9]+$'; then
+  say "installing llmgw $TAG from the release"
+  $SSH "set -e
+    arch=\$(dpkg --print-architecture)
+    name=llmgw-$TAG-linux-\$arch
+    cd /tmp && rm -f \$name checksums.txt
+    curl -fsSL -o \$name '$RELEASES/$TAG/'\$name
+    curl -fsSL -o checksums.txt '$RELEASES/$TAG/checksums.txt'
+    grep \" \$name\\\$\" checksums.txt | sha256sum -c - >/dev/null
+    sudo install -m 0755 \$name /usr/local/bin/llmgw
+    rm -f \$name checksums.txt"
+else
+  say "installing the binary (${TAG:-unversioned}, not a release)"
+  scp -q $SSHOPTS "$BINARY" "$SSH_USER@$ADDR:/tmp/llmgw"
+  $SSH 'sudo install -m 0755 /tmp/llmgw /usr/local/bin/llmgw && rm -f /tmp/llmgw'
+fi
 
 say "writing the configuration"
-FABRICS="fabrics: []"
-if [ -n "$FABRIC_NAME" ]; then
-  FABRICS="fabrics:
-  - name: $FABRIC_NAME
-    hcas: [$(echo "$FABRIC_HCAS" | sed 's/,/, /g')]"
-fi
-cat > "$TMP/agent.yaml" <<YAML
-# $NODE
-#
-# Written by onboard-node.sh. Fabric membership is configuration and there is no
-# way around it: OpenSM's default subnet prefix is the same on every fabric, so
-# a matching prefix proves nothing and a wrong guess produces a replica that
-# starts, passes health checks and then hangs on its first all-reduce.
-node_id: $NODE
-server: "$SERVER"
-
-tls:
-  ca: /etc/llmgw/ca.pem
-  cert: /etc/llmgw/agent.pem
-  key: /etc/llmgw/agent-key.pem
-
-mgmt_addr: "$MGMT"
-mgmt_interface: $MGMT_IF
-
-heartbeat_interval: 5s
-reconnect_base: 1s
-reconnect_max: 30s
-
-$FABRICS
-
-discovery:
-  # Empty means look on PATH. Worth setting where the driver lives outside the
-  # default paths, which containerized installs routinely do.
-  nvidia_smi: "$NVIDIA_SMI"
-  ib_root: $IB_ROOT
-  prefer_nvml: false
-  skip_gpus: false
-
-# Where the engines live. Empty means "on PATH", which is rarely true: a node
-# that cannot find its engine registers healthy and then fails every shard
-# placed on it with "executable file not found".
-engines:
-  llamacpp:
-    bin: "$LLAMACPP_BIN"
-    parallel: 4
-  vllm:
-    bin: "$VLLM_BIN"
-    launch_mode: ray
-  sglang:
-    python: "$SGLANG_PYTHON"
-
-supervisor:
-  work_dir: /var/lib/llmgw-agent
-  log_lines: 512
-  stop_timeout: 60s
-  health_interval: 2s
-YAML
+NODE_IS_HOSTNAME=no
+[ "$($SSH hostname 2>/dev/null)" = "$NODE" ] && NODE_IS_HOSTNAME=yes
+# Only what the node cannot find out for itself. Its name is its hostname, its
+# address is the one on the route to the control plane, llama-server is looked
+# for on PATH and where release archives are unpacked, GPUs are there when
+# nvidia-smi is, and memory, disk and engines are re-read while it runs. Every
+# setting written here is one that has to be kept true by hand — so an option
+# below is written only when it was given, as an override.
+{
+  echo "# $NODE, written by onboard-node.sh."
+  # Several addresses, comma separated, are every control plane the node may
+  # fail over to; a standby refuses it and it moves on to the next.
+  if [ "${SERVER#*,}" != "$SERVER" ]; then
+    echo "servers:"
+    printf '%s\n' "$SERVER" | tr ',' '\n' | sed 's/^ *//; s/^/  - "/; s/$/"/'
+  else
+    echo "server: \"$SERVER\""
+  fi
+  echo
+  echo "tls:"
+  echo "  ca: /etc/llmgw/ca.pem"
+  echo "  cert: /etc/llmgw/agent.pem"
+  echo "  key: /etc/llmgw/agent-key.pem"
+  # The certificate's common name is --node, and the control plane refuses a
+  # stream whose node id differs from it. Said explicitly only when the
+  # hostname would say something else.
+  [ "$NODE_IS_HOSTNAME" = yes ] || echo "node_id: $NODE"
+  [ -n "$MGMT_OVERRIDE" ] && echo "mgmt_addr: \"$MGMT_OVERRIDE\""
+  [ -n "$MGMT_IF_OVERRIDE" ] && echo "mgmt_interface: $MGMT_IF_OVERRIDE"
+  if [ -n "$FABRIC_NAME" ]; then
+    # Fabric membership is the one thing that cannot be found: OpenSM's default
+    # subnet prefix is the same on every fabric, so a matching prefix proves
+    # nothing and a wrong guess hangs the first all-reduce.
+    echo "fabrics:"
+    echo "  - name: $FABRIC_NAME"
+    echo "    hcas: [$(echo "$FABRIC_HCAS" | sed 's/,/, /g')]"
+  fi
+  if [ -n "$NVIDIA_SMI" ] || [ "$IB_ROOT" != /sys/class/infiniband ]; then
+    echo "discovery:"
+    [ -n "$NVIDIA_SMI" ] && echo "  nvidia_smi: \"$NVIDIA_SMI\""
+    [ "$IB_ROOT" != /sys/class/infiniband ] && echo "  ib_root: $IB_ROOT"
+  fi
+  if [ -n "$LLAMACPP_BIN$VLLM_BIN$SGLANG_PYTHON" ]; then
+    echo "engines:"
+    [ -n "$LLAMACPP_BIN" ] && printf '  llamacpp:\n    bin: "%s"\n' "$LLAMACPP_BIN"
+    [ -n "$VLLM_BIN" ] && printf '  vllm:\n    bin: "%s"\n' "$VLLM_BIN"
+    [ -n "$SGLANG_PYTHON" ] && printf '  sglang:\n    python: "%s"\n' "$SGLANG_PYTHON"
+  fi
+  true
+} > "$TMP/agent.yaml"
 scp -q "$TMP/agent.yaml" "$SSH_USER@$ADDR:/tmp/agent.yaml"
 
 cat > "$TMP/llmgw-agent.service" <<'UNIT'
@@ -255,8 +273,9 @@ $SSH 'set -e
 
 echo
 if [ -z "$LLAMACPP_BIN$VLLM_BIN$SGLANG_PYTHON" ]; then
-  say "no engine paths given; this node can only run the 'sim' engine until"
-  say "one is installed and the configuration points at it"
+  say "the agent now installs the engines the control plane asks for: the"
+  say "build for this machine, checked against its publisher's checksum and run"
+  say "before the node is offered any model. 'llmgw ctl nodes' shows progress."
 fi
 if [ -n "$ENGINE_PROBLEMS" ]; then
   echo "$NODE is onboarded but an engine will not run:$ENGINE_PROBLEMS"

deploy/examples/agent.yaml

@@ -4,12 +4,19 @@
 # keep serving if the agent restarts: the router talks to them directly, and
 # orphan adoption picks them up again on the next start.
 
-node_id: gpu-a-01          # defaults to the hostname
+# Everything a node can find out about itself, it does: its id is its hostname,
+# its address is the one on the route to the control plane, llama-server is
+# looked for on PATH and in /opt/llama.cpp, GPUs are there when nvidia-smi is,
+# weights are fetched into /var/lib/llmgw-weights and looked for there, and
+# memory, disk and installed engines are re-read while the agent runs. So a
+# plain CPU node needs only the two blocks below, and the same file on every
+# node. What follows them are overrides, for when the defaults are wrong.
+
 server: "http://llmgw-server.internal:7443"
 
 # The agent's half of mutual TLS. The certificate's common name must be this
-# node's id, because the control plane compares the two and refuses the stream
-# when they disagree:
+# node's id — its hostname, unless node_id says otherwise — because the control
+# plane compares the two and refuses the stream when they disagree:
 #
 #   llmgw ctl cert issue-client gpu-a-01 --dir /etc/llmgw/pki
 #
@@ -22,13 +29,19 @@ server: "http://llmgw-server.internal:7443"
 #   cert: /etc/llmgw/agent.pem
 #   key: /etc/llmgw/agent-key.pem
 
+# ---- overrides ---------------------------------------------------------------
+
+# node_id: gpu-a-01          # when it is not the hostname
+
 # The address engines are reachable on, and the address NCCL bootstraps over.
 # Never a fabric address: NCCL rendezvouses over TCP before moving to RDMA.
-mgmt_addr: "10.0.0.11"
-mgmt_interface: eth0
+# mgmt_addr: "10.0.0.11"
+# mgmt_interface: eth0
 
-labels:
-  rack: r12
+# Free-form, for a profile's node_selector. Nothing needs them; do not restate
+# what is measured (memory, GPUs, engines), because a label goes stale.
+# labels:
+#   rack: r12
 
 heartbeat_interval: 5s
 reconnect_base: 1s
@@ -55,12 +68,9 @@ discovery:
   # tensor-parallel group off the PCIe path inside a node. It needs a binary
   # built with `-tags nvml`; otherwise this silently falls back to nvidia-smi.
   prefer_nvml: true
-  skip_gpus: false
 
-# Where this node keeps model weights.
-#
-# Unset by default, and unset means the node is not asked to look — which is
-# not the same as having nothing, and constrains nothing. Storage differs at
+# Where this node keeps model weights besides /var/lib/llmgw-weights, which is
+# where fetched weights land and is always looked in. Storage differs at
 # every installation and llmgw deliberately does not own it: a Lustre mount, an
 # image exported over the fabric with SRP or NVMe-oF, local NVMe, or a copy
 # made by hand all look identical from here.
@@ -74,6 +84,15 @@ discovery:
 #   - /var/lib/llmgw-models
 #   - /mnt/lustre/models
 
+# The agent installs the engines the control plane names (see `engines:` in
+# server.yaml) under /opt/llmgw/engines: the build for this machine — CUDA when
+# an NVIDIA driver is loaded — checked against its publisher's checksum, with
+# the OS packages it needs, and run before the node takes any model. llama.cpp
+# is installed this way today. Naming a binary below opts that engine out and
+# uses yours instead; vLLM and SGLang still have to be named until the agent
+# installs them too.
+# engines_dir: /opt/llmgw/engines
+
 engines:
   vllm:
     bin: /opt/venvs/vllm/bin/vllm

deploy/examples/server.yaml

@@ -197,6 +197,13 @@ router:
     session_load_factor: 1.25
 
 # How aggressively the control plane reconciles.
+# The engines every node installs, and at which version. Unset keeps the
+# versions this release was checked against (llama.cpp b10991). Each agent
+# picks the build for its own hardware and installs it itself; changing a
+# version here reaches every node on its next connect.
+# engines:
+#   llamacpp: b10991
+
 control:
   interval: 5s
   base_backoff: 10s

deploy/ha/README.md

@@ -0,0 +1,87 @@
+# Two control planes
+
+llmgw handles the hard half of this itself: only one control plane places work,
+and agents find whichever one is leading. What it does not do is move client
+traffic, because that happens in front of both and llmgw is not in front of
+anything.
+
+This directory is the missing half. It is a pattern rather than a product:
+substitute your own load balancer if you have one.
+
+## What llmgw does on its own
+
+**One leader.** Leadership is a lease in the shared store. A second control
+plane starts, sees the lease, and stands by — it places nothing, evicts
+nothing and drains nothing. Start one by accident and your cluster is
+unharmed.
+
+**Agents follow.** A standby refuses agent streams, so agents move to the
+leader by themselves. Give each agent both addresses:
+
+```yaml
+servers:
+  - "https://ctl-1.internal:7443"
+  - "https://ctl-2.internal:7443"
+```
+
+Measured on a cluster: leadership moved three seconds after the leader was
+stopped, every agent had arrived six seconds later, and every replica was
+adopted rather than restarted — the engines never stopped serving.
+
+## What it does not do
+
+Client traffic. During that same failover the standby served inference
+correctly on its own port while the public address returned 502, because
+nothing in front of the two had moved.
+
+So something has to. The options, in the order most people should consider
+them:
+
+**A load balancer you already run.** Health-check `/ready` on each control
+plane and send traffic to whichever answers. This is the right answer if you
+have one, and it needs nothing else from this directory.
+
+**A floating address.** `keepalived.conf.example` here moves one IP between
+the two machines. Fewer moving parts than a load balancer and no extra hop,
+at the cost of one machine carrying all the traffic.
+
+**DNS.** Works, eventually. A record with a short TTL is a failover measured
+in whatever your resolvers actually honour rather than what you set, which on
+a bad day is minutes. Fine for a cluster people use interactively, wrong for
+one an agent is driving.
+
+## Check `/ready`, not `/health`
+
+This is the one thing to get right, and the obvious choice is the wrong one.
+
+`/health` says the process is up. It says nothing about whether it can serve,
+and a standby is exactly that case: alive, answering, and holding no endpoints
+at all — its reconciler placed nothing and it has no agent streams to adopt
+from. A load balancer checking `/health` would send it half the traffic, and
+half the traffic would be a 503.
+
+`/ready` says this process has somewhere to send a request. It is 503 on a
+standby, 503 on a leader whose replicas have all failed, and 200 when there is
+something to serve — which is the question a load balancer is actually asking.
+Both are open, because a load balancer has no key.
+
+During a failover neither passes for about the length of the lease handover:
+three seconds when the leader is stopped cleanly, up to the lease TTL when it
+is not. That gap is real and there is no way to have it both ways — traffic
+cannot go to a control plane that has nothing to route to.
+
+Not the admin port, for a different reason: it serves pprof, and pprof can
+dump the process heap, where API keys sit in plaintext.
+
+## Both control planes need the same store
+
+Leadership lives in it, and so do keys, budgets and conversations. Two control
+planes on two SQLite files are two clusters that happen to share a name.
+
+```yaml
+auth:
+  store: "@/etc/llmgw/secrets/store-dsn"
+```
+
+See [docs/install.md](../../docs/install.md) for moving an existing
+installation onto PostgreSQL without reissuing every key.

deploy/ha/keepalived.conf.example

@@ -0,0 +1,65 @@
+# A floating address for two control planes.
+#
+# One IP that lives on whichever machine is up. Clients and the reverse proxy
+# in front of llmgw point at the address, not at either host, so a failover is
+# invisible to them.
+#
+# This is not a load balancer: all traffic goes to the holder. That is usually
+# the right trade for a control plane, which serves far less than the nodes
+# it schedules.
+#
+# Copy to /etc/keepalived/keepalived.conf on both machines, changing state,
+# priority and interface.
+
+global_defs {
+    enable_script_security
+    script_user nobody
+}
+
+# The address moves only while llmgw can actually serve, not merely while the
+# machine is powered on.
+#
+# /ready, not /health. /health is 200 on a standby, which is up, answering and
+# holding no endpoints at all — so the address would sit on a host that refuses
+# every request. /ready is 200 only when there is somewhere to send one.
+#
+# Not the admin port either: it serves pprof, which can dump the heap that
+# holds every API key.
+vrrp_script check_llmgw {
+    script "/usr/bin/curl -fsS --max-time 2 -o /dev/null http://127.0.0.1:8080/ready"
+    interval 2
+    timeout 3
+    # Two failures before giving up the address, so one slow moment does not
+    # move it. Two successes before taking it back, so a flapping gateway does
+    # not drag the address with it.
+    fall 2
+    rise 2
+    weight -40
+}
+
+vrrp_instance llmgw {
+    # MASTER on the machine you would rather serve; BACKUP on the other.
+    state MASTER
+    interface eth0
+    # Any number both machines agree on and nothing else on the segment uses.
+    virtual_router_id 51
+    # The higher one wins while both are healthy. The weight above is enough
+    # to invert this when the preferred machine's gateway is not answering.
+    priority 100
+    advert_int 1
+
+    authentication {
+        auth_type PASS
+        # Change this. It stops another host on the segment claiming the
+        # address, which is otherwise a single unicast packet away.
+        auth_pass CHANGEME
+    }
+
+    virtual_ipaddress {
+        10.0.0.150/24
+    }
+
+    track_script {
+        check_llmgw
+    }
+}

deploy/mirror/README.md

@@ -0,0 +1,262 @@
+# A model mirror
+
+One machine that holds the model files, so nodes fetch over your own network
+instead of over the internet.
+
+This is the highest-value piece of infrastructure around llmgw and the cheapest.
+Eight nodes each pulling forty gigabytes from Hugging Face is a day's work, and
+on plenty of corporate networks it is not possible at all. From a mirror on the
+same switch it is minutes.
+
+llmgw needs nothing special for this: a model's `source` is a URL, and the URL
+can be yours.
+
+## What it is
+
+An HTTP server serving a directory. That is the whole design.
+
+```
+/srv/models/
+  qwen2.5-14b-instruct/
+    2026-03/
+      q4_k_m.gguf
+  bge-m3/
+    2026-01/
+      model.safetensors
+```
+
+The date in the path is the point of the layout, not decoration. A model
+repository that changes upstream changes what you are running, and the first
+sign is a prompt that worked last week answering differently this week with
+nobody able to say why. Pin it here and that cannot happen.
+
+## Running it
+
+`llmgw` is the mirror. Same binary as the control plane and the agents, one
+subcommand:
+
+```
+sudo mkdir -p /srv/models && sudo chown llmgw:llmgw /srv/models
+llmgw mirror serve --dir /srv/models --base http://models.internal:8081
+```
+
+There is a unit in `deploy/systemd/llmgw-mirror.service`. Set `--base` to the
+address nodes reach this machine on — not `localhost`, and not a name only this
+host knows itself by. It is the string that ends up in catalog entries, and a
+wrong one makes `mirror unused` report files as unreferenced when they are in
+use. The next step after that list is deletion.
+
+### Two listeners
+
+`--listen :8081` serves the weights. Read-only, unauthenticated, Range requests
+supported so an interrupted fetch on a node resumes rather than restarts. This
+is the path eight nodes pull forty gigabytes from.
+
+`--api-listen :8082` is the management API. Separate on purpose: putting a
+client certificate in front of the byte path would mean re-certifying every
+agent's fetcher in order to change how models are administered.
+
+The API is unauthenticated unless you give it the fleet's CA, and it says so at
+startup rather than leaving you to find out. Anyone who can reach that port can
+start downloads and delete models, so either keep it on a trusted network or
+hand it certificates:
+
+```
+llmgw mirror serve --dir /srv/models --base http://models.internal:8081 \
+  --tls-ca /etc/llmgw/ca.pem \
+  --tls-cert /etc/llmgw/mirror.pem \
+  --tls-key /etc/llmgw/mirror-key.pem
+```
+
+That is the same CA the agents already use, so the certificates come from the
+authority you have:
+
+```
+llmgw ctl cert issue-server mirror --host models.internal --host 10.0.0.146
+llmgw ctl cert issue-client control-plane
+```
+
+`--host` has to cover every address the API is called on, because that is what
+a client verifies against. Both sides are checked — the mirror will not accept a
+caller without a certificate from this CA, the same rule the agent stream
+follows.
+
+### nginx instead
+
+If you already run nginx, it serves a directory perfectly well and you do not
+need `mirror serve` for the bytes:
+
+```nginx
+server {
+    listen 80;
+    server_name models.internal;
+
+    root /srv/models;
+
+    # Read-only. This is deliberate: a mirror that can be written to over HTTP
+    # is a way to replace a model with another one, and the nodes would fetch
+    # it without complaint.
+    location / {
+        autoindex on;
+        # Large files, and the client is a machine with a resumable fetcher.
+        # Sendfile and no buffering keep memory flat regardless of size.
+        sendfile on;
+        tcp_nopush on;
+    }
+}
+```
+
+`autoindex` is there so a person can browse what is available; llmgw does not
+need it.
+
+## Filling it
+
+From the mirror itself:
+
+```
+llmgw mirror add \
+  --model qwen2.5-14b-instruct --version 2026-03 \
+  "https://huggingface.co/<repo>/resolve/<revision>/q4_k_m.gguf"
+```
+
+Use a revision rather than `main` in the source URL. `main` is what moves.
+
+It downloads to a temporary name and renames when complete, so an interrupted
+download never leaves something a catalog could be pointed at, and it prints the
+sha256 when it lands. Compare that against what upstream publishes — Hugging
+Face gives it as `lfs.oid` from
+`https://huggingface.co/api/models/<repo>/tree/<revision>`. A mirror is the one
+place in this system where a corrupt file is copied to every node instead of
+being caught once.
+
+It refuses to overwrite a version that is already there. Add a new one beside
+it: that refusal is what makes the old weights still available when the new ones
+turn out to be worse.
+
+## Driving it from elsewhere
+
+The same fetch over the API, so the control plane — or anything else — can ask
+for a model without anyone logging in:
+
+```
+curl -X POST http://models.internal:8082/v1/models -d '{
+  "source": "https://huggingface.co/<repo>/resolve/<revision>/q4_k_m.gguf",
+  "model": "qwen2.5-14b-instruct",
+  "version": "2026-03"
+}'
+```
+
+The mirror does the downloading. A control plane that pulled the bytes itself
+and pushed them here would carry every model twice over the network, through the
+one machine least able to spare the bandwidth.
+
+It answers `202` with the URL a catalog entry should use, and the bytes are not
+here yet. Poll for them:
+
+| | |
+|---|---|
+| `GET /v1/models` | everything held or arriving, and the base to build URLs from |
+| `POST /v1/models` | fetch this source into model/version |
+| `GET /v1/models/{model}/{version}` | `fetching` with a byte count, `present`, or `failed` with the reason |
+| `DELETE /v1/models/{model}/{version}` | remove a version; refuses while it is downloading |
+| `GET /v1/health` | up *and* able to read its own directory |
+
+A failed fetch stays `failed` with what went wrong, rather than disappearing:
+otherwise the thing waiting on that model waits forever with nothing to say why.
+Asking twice for the same download joins the one already running instead of
+starting a rival writer on the same file.
+
+`DELETE` takes the whole version, never one file of it. A model split across
+several files is useless with one missing, and half a version on disk looks
+present and will not load.
+
+## Retiring a model
+
+```
+llmgw mirror unused     # what no catalog entry refers to
+llmgw mirror remove --model <m> --version <v> --file <f>
+```
+
+`unused` compares against every `source` in the running catalog, so the answer
+is what can go rather than what looks old. It refuses to answer when it cannot
+reach the control plane: an empty catalog and an unreachable one produce the
+same list — everything — and the obvious next step after that list is deletion.
+
+Retire the catalog entry first, let the fleet settle, then remove. A node that
+has to refetch a file you deleted cannot, and the failure arrives later, on the
+node, looking like something else.
+
+## From the control plane's admin page
+
+Configure the mirror on the control plane and the same operations appear at
+`/admin/mirror`, behind the admin session the other admin pages use:
+
+```yaml
+mirror:
+  api: https://models.internal:8082
+  tls:
+    ca: /etc/llmgw/ca.crt
+    cert: /etc/llmgw/control-plane.crt
+    key: /etc/llmgw/control-plane.key
+```
+
+Leave the block out and the page is not mounted at all, which is the right page
+for a deployment with no mirror — not an empty one pointing at nothing.
+
+The page lists what is held, what is arriving with its byte count, and what
+failed with the reason; it starts a fetch, retries a failed one, and removes a
+version. It reloads itself only while something is downloading, because a reload
+at any other moment takes with it whatever is half-typed in the form.
+
+Two things it deliberately will not do. It cannot point the control plane at a
+different mirror — that is configuration, and changing it needs access to the
+host rather than a session. And it does not write the catalog for you: it shows
+the URL an entry should use, and putting it there is still a deliberate act,
+because that is the moment a model starts being served.
+
+The address is a mistake worth naming: `api` is the **management** port, not the
+one nodes fetch bytes from. They are two listeners so this one can be locked
+down without re-certifying every agent's fetcher.
+
+## Pointing llmgw at it
+
+```yaml
+- name: qwen2.5-14b-instruct
+  source: http://models.internal/qwen2.5-14b-instruct/2026-03/q4_k_m.gguf
+  scaling: { min: 2, max: 2 }
+  profiles:
+    - name: gpu
+      engine: vllm
+      gpu_class: h100-80g
+      tp: 1
+```
+
+Each node fetches once, in the background, before its engine starts — and
+`llmgw ctl replicas` shows how far along it is while it does. A killed download
+resumes rather than restarting, and a partial file is never visible under the
+model's name, so an interrupted fetch cannot be handed to an engine.
+
+## What this is not
+
+It is not a cache. Nothing populates it on its own: a model is there because
+somebody asked for it, by hand or over the API, and one nobody asked for is not
+there. That is on purpose. A mirror that fetched on demand would reach the
+internet at the least convenient moment, and in a network where that is blocked
+it would fail in a way that looks like llmgw being broken.
+
+It is also not a replacement for a shared filesystem. If you already run one
+that every node mounts, put the models on it and use a path as the `source` —
+nothing is fetched at all, and llmgw notices the filesystem is shared and will
+not count the same free space once per node.
+
+## When this stops being enough
+
+Somewhere above ten or twenty nodes, one mirror serving a forty gigabyte file
+to all of them at once becomes the bottleneck instead of the fix. At that point
+the options are a filer, a second mirror per rack, or distribution between the
+nodes themselves.
+
+llmgw knows which nodes already hold which weights — it is in `ctl status` as
+"N models on X/Y nodes" — so distributing between them is a feature it is
+positioned to add. It has not been built, because a mirror is enough for
+everyone who has needed it so far.

deploy/monitoring/README.md

@@ -0,0 +1,170 @@
+# Watching a cluster
+
+Prometheus, Alertmanager and Grafana, configured for what llmgw actually
+publishes. Every alert here is written against a series this system emits;
+nothing is aimed at a metric that might exist.
+
+## What it is
+
+One scrape target for the whole fleet.
+
+Agents publish no metrics of their own. The control plane gathers what they
+report — node state, memory, disk, engine queue depth, KV occupancy — and
+republishes all of it, so a single endpoint describes every machine. That is
+worth knowing before going looking for an exporter on each node.
+
+## The port it scrapes
+
+`metrics_listen`, not `admin_listen`.
+
+The admin listener is bound to loopback for one reason: it also serves pprof,
+and pprof can dump the process heap, where API keys sit in plaintext. Metrics
+are the one thing on that listener something else genuinely needs to reach, so
+they get a port of their own instead of becoming an argument for opening the
+other one.
+
+```yaml
+# /etc/llmgw/server.yaml
+metrics_listen: ":9091"
+```
+
+Confirm it carries nothing else:
+
+```
+curl -s -o /dev/null -w '%{http_code}\n' http://<control-plane>:9091/debug/pprof/   # 404
+curl -s -o /dev/null -w '%{http_code}\n' http://<control-plane>:9091/api/v1/nodes   # 404
+```
+
+## Installing
+
+On Ubuntu 24.04 the first two come from the archive; Grafana needs its own
+repository.
+
+```
+apt-get install prometheus prometheus-alertmanager
+
+install -m 0644 prometheus.yml            /etc/prometheus/prometheus.yml
+install -m 0644 llmgw-rules.yml           /etc/prometheus/rules/llmgw.yml
+install -m 0644 alertmanager.yml          /etc/prometheus/alertmanager.yml
+promtool check config /etc/prometheus/prometheus.yml
+systemctl restart prometheus prometheus-alertmanager
+
+install -m 0644 grafana/datasource.yaml   /etc/grafana/provisioning/datasources/llmgw.yaml
+install -m 0644 grafana/dashboards.yaml   /etc/grafana/provisioning/dashboards/llmgw.yaml
+install -m 0644 -o grafana -g grafana grafana/llmgw-overview.json \
+    /var/lib/grafana/dashboards/llmgw.json
+systemctl restart grafana-server
+```
+
+The dashboard is provisioned from a file rather than clicked together, so it
+survives a reinstall and can be reviewed in a diff.
+
+## Where to run it
+
+Not on the control plane. Monitoring that dies with the thing it monitors tells
+you nothing at the moment you need it, and the control plane is also the busiest
+machine in a small cluster.
+
+Anywhere else on the management network will do. The mirror is a reasonable
+host: it has the disk, and serving weights is bursty rather than constant.
+
+## Nothing is sent anywhere
+
+Alertmanager groups and holds alerts and has no receiver.
+
+Where an alert should go — an address, a Slack workspace, a webhook — is a
+decision with credentials attached, and a monitoring system that quietly starts
+mailing people is worse than one that is visibly not wired up. Alerts are
+visible at `:9093` and in Prometheus; add a receiver to `alertmanager.yml` to
+make them travel. `critical` already routes separately so it can be pointed
+somewhere else without touching the rest.
+
+## Check that it fires
+
+An alerting system nobody has seen fire is an assumption.
+
+```
+llmgw ctl cordon <node>     # work already there keeps running
+# ~2 minutes later
+curl -s http://<host>:9093/api/v2/alerts | jq '.[].labels'
+llmgw ctl uncordon <node>   # the alert resolves on its own
+```
+
+Cordoning is the cheapest real test: it changes a metric the rules read, stops
+nothing, and reverses in one command.
+
+## What the alerts assume
+
+**A stopped model publishes no series.** `llmgw_model_replicas_ready` exists
+only for models with a replica placed, so `== 0` cannot fire for one that is
+deliberately off. That is what makes an on-demand catalogue quiet here.
+
+**Saturation of 1 is the healthy limit, not failure.** It is the worse of queue
+depth and KV occupancy, measured by `ctl bench` rather than guessed. The alert
+allows five minutes above it before saying anything.
+
+**Latencies are in seconds.** These thresholds are written for CPU inference. A
+p99 first token of thirty seconds is the alerting point here and would be
+absurd on a GPU; change it when the hardware changes.
+
+**Disk is only reported by nodes that hold weights.** A node with no model
+directory reports nothing rather than zero, and the rule divides free by total
+so a node that reports neither never fires.
+
+## Where the metrics live
+
+On the machine that scrapes them, in Prometheus's own store, and that is the
+right answer at this size. Measured on this cluster: 2,359 series at 41 samples
+a second, which compresses to roughly 7 MB a day — about 3 GB for a year.
+
+```
+# /etc/default/prometheus
+ARGS="--storage.tsdb.retention.time=365d --storage.tsdb.retention.size=20GB"
+```
+
+A year because the questions worth asking of this data are seasonal: what did
+capacity look like last quarter, was it always this slow, how much did the
+fleet grow. Fifteen days answers none of them.
+
+Both limits, not one. The time limit is the intent; the size cap is what stops
+Prometheus becoming the thing that fills the disk if the series count grows
+tenfold after somebody adds a fleet.
+
+No remote write, no object store, no second tier. At two thousand series that
+is infrastructure for a problem this cluster does not have, and every one of
+those has its own failure modes to learn.
+
+### The part that is not about capacity
+
+The disk is not the constraint — 900 GB free against 3 GB a year. What matters
+is what shares it.
+
+On this cluster the model mirror and the metrics database are on the same
+filesystem. A mirror that fills it stops Prometheus writing, so monitoring goes
+quiet at exactly the moment the disk is the problem, and the alert that would
+have said so is the one that died.
+
+Two ways out, in order of how much they cost:
+
+- Watch it, which is now done. `prometheus-node-exporter` arrives with the
+  Prometheus package and was running, unscraped, on this machine — so the disk
+  holding both was watched by nothing. It is a scrape target now, and
+  `MonitoringDiskFilling` fires at fifteen percent rather than five, because
+  the thing being protected is the ability to be told.
+- Separate them: a volume of its own for `/var/lib/prometheus`, or monitoring
+  on its own machine. Worth doing before the mirror holds a few hundred
+  gigabytes; not worth doing today at seven percent used.
+
+## What is not here
+
+No exporter on the llmgw nodes, so no host CPU, load average or network for
+them. The control plane publishes memory, disk and liveness for every node,
+which is what placement decides on; the rest is a `node_exporter` away if
+somebody wants it. The monitoring host itself is scraped, for the reason above.
+
+No probe of the mirror. Its `/v1/health` is JSON rather than Prometheus text,
+and checking it properly means `blackbox_exporter`.
+
+No recording rules. At this size every query in the dashboard runs fast enough
+raw, and a recording rule is a second definition of a number to keep in step
+with the first.

deploy/monitoring/alertmanager.yml

@@ -0,0 +1,41 @@
+# Alertmanager for the llmgw cluster.
+#
+# Nothing is sent anywhere yet, on purpose. Where an alert should go — an
+# address, a Slack workspace, a webhook — is a decision with credentials
+# attached, and a monitoring system that quietly starts mailing people is worse
+# than one that is visibly not wired up. Alerts group and fire here and are
+# visible at :9093 and in Prometheus; add a receiver below to make them travel.
+global:
+  resolve_timeout: 5m
+
+route:
+  # By alert and by what it is about, so five nodes going quiet in the same
+  # network blip arrive as one notification rather than five.
+  group_by: ['alertname', 'node', 'model']
+  group_wait: 30s
+  group_interval: 5m
+  # Long, because a repeat is a reminder and not news. A cluster with one thing
+  # wrong should not produce a message an hour.
+  repeat_interval: 12h
+  receiver: none
+
+  routes:
+    # Critical gets its own branch so it can be pointed somewhere else later
+    # without touching the rest: the difference between "a node is draining"
+    # and "nothing is serving" is who should be woken.
+    - matchers: [severity="critical"]
+      receiver: none
+      group_wait: 10s
+      repeat_interval: 4h
+
+receivers:
+  - name: none
+
+inhibit_rules:
+  # A node that is down explains every replica on it. Reporting both says the
+  # same thing twice and buries whichever is the actual cause.
+  - source_matchers: [alertname="NodeNotReady"]
+    target_matchers: [alertname="EndpointMetricsStale"]
+    equal: ['node']
+  - source_matchers: [alertname="ControlPlaneUnreachable"]
+    target_matchers: [severity="warning"]

deploy/monitoring/grafana/dashboards.yaml

@@ -0,0 +1,11 @@
+apiVersion: 1
+providers:
+  - name: llmgw
+    orgId: 1
+    folder: llmgw
+    type: file
+    disableDeletion: false
+    updateIntervalSeconds: 30
+    allowUiUpdates: true
+    options:
+      path: /var/lib/grafana/dashboards

deploy/monitoring/grafana/datasource.yaml

@@ -0,0 +1,11 @@
+# Provisioned rather than clicked in, so the dashboard below can reference it by
+# a fixed uid and the whole setup survives a reinstall of the machine.
+apiVersion: 1
+datasources:
+  - name: Prometheus
+    uid: llmgw-prom
+    type: prometheus
+    access: proxy
+    url: http://localhost:9090
+    isDefault: true
+    editable: false

deploy/monitoring/grafana/llmgw-overview.json

@@ -0,0 +1,492 @@
+{
+  "uid": "llmgw-overview",
+  "title": "llmgw \u2014 cluster",
+  "tags": [
+    "llmgw"
+  ],
+  "timezone": "browser",
+  "schemaVersion": 39,
+  "version": 1,
+  "refresh": "30s",
+  "time": {
+    "from": "now-6h",
+    "to": "now"
+  },
+  "panels": [
+    {
+      "id": 1,
+      "title": "Nodes ready",
+      "type": "stat",
+      "description": "Nodes taking work. A node that is known but draining or cordoned is not counted.",
+      "gridPos": {
+        "x": 0,
+        "y": 0,
+        "w": 4,
+        "h": 4
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "count(llmgw_node_up == 1)",
+          "legendFormat": "ready",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {},
+        "overrides": []
+      },
+      "options": {
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        },
+        "textMode": "auto",
+        "colorMode": "value"
+      }
+    },
+    {
+      "id": 2,
+      "title": "Replicas serving",
+      "type": "stat",
+      "description": "Across every model. A model deliberately stopped contributes nothing and is not a fault.",
+      "gridPos": {
+        "x": 4,
+        "y": 0,
+        "w": 4,
+        "h": 4
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "sum(llmgw_model_replicas_ready)",
+          "legendFormat": "ready",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {},
+        "overrides": []
+      },
+      "options": {
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        },
+        "textMode": "auto",
+        "colorMode": "value"
+      }
+    },
+    {
+      "id": 3,
+      "title": "Models with something serving",
+      "type": "stat",
+      "description": "",
+      "gridPos": {
+        "x": 8,
+        "y": 0,
+        "w": 4,
+        "h": 4
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "count(llmgw_model_replicas_ready > 0)",
+          "legendFormat": "models",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {},
+        "overrides": []
+      },
+      "options": {
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        },
+        "textMode": "auto",
+        "colorMode": "value"
+      }
+    },
+    {
+      "id": 4,
+      "title": "Requests per second",
+      "type": "timeseries",
+      "description": "Failing is counted separately rather than as a percentage, so a quiet cluster reads as quiet instead of as 0% of nothing.",
+      "gridPos": {
+        "x": 12,
+        "y": 0,
+        "w": 12,
+        "h": 4
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "sum(rate(llmgw_requests_total[5m]))",
+          "legendFormat": "all",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        },
+        {
+          "refId": "B",
+          "expr": "sum(rate(llmgw_requests_total{status=~\"5..\"}[5m]))",
+          "legendFormat": "failing",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "reqps"
+        },
+        "overrides": []
+      }
+    },
+    {
+      "id": 5,
+      "title": "Replicas ready, by model",
+      "type": "timeseries",
+      "description": "Only models with a replica placed appear. One that is stopped on purpose publishes no series.",
+      "gridPos": {
+        "x": 0,
+        "y": 4,
+        "w": 12,
+        "h": 7
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "llmgw_model_replicas_ready",
+          "legendFormat": "{{model}}",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {},
+        "overrides": []
+      }
+    },
+    {
+      "id": 6,
+      "title": "Pool saturation",
+      "type": "timeseries",
+      "description": "The worse of queue depth and KV occupancy. 1 is the limit measured as healthy, not the point of failure.",
+      "gridPos": {
+        "x": 12,
+        "y": 4,
+        "w": 12,
+        "h": 7
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "llmgw_pool_saturation",
+          "legendFormat": "{{model}}",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "max": 2,
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "red",
+                "value": 1
+              }
+            ]
+          }
+        },
+        "overrides": []
+      }
+    },
+    {
+      "id": 7,
+      "title": "Time to first token",
+      "type": "timeseries",
+      "description": "As the router sees it. On CPU these are seconds, not milliseconds; compare against what ctl bench measured.",
+      "gridPos": {
+        "x": 0,
+        "y": 11,
+        "w": 12,
+        "h": 7
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "histogram_quantile(0.5, sum by (model, le) (rate(llmgw_ttft_seconds_bucket[5m])))",
+          "legendFormat": "{{model}} p50",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        },
+        {
+          "refId": "B",
+          "expr": "histogram_quantile(0.99, sum by (model, le) (rate(llmgw_ttft_seconds_bucket[5m])))",
+          "legendFormat": "{{model}} p99",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "s"
+        },
+        "overrides": []
+      }
+    },
+    {
+      "id": 8,
+      "title": "Tokens per second",
+      "type": "timeseries",
+      "description": "",
+      "gridPos": {
+        "x": 12,
+        "y": 11,
+        "w": 12,
+        "h": 7
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "sum by (direction) (rate(llmgw_tokens_total[5m]))",
+          "legendFormat": "{{direction}}",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "short"
+        },
+        "overrides": []
+      }
+    },
+    {
+      "id": 9,
+      "title": "Engine queue depth",
+      "type": "timeseries",
+      "description": "Requests the engine has accepted and not started. Rising here is the first sign of saturation.",
+      "gridPos": {
+        "x": 0,
+        "y": 18,
+        "w": 8,
+        "h": 6
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "llmgw_endpoint_waiting",
+          "legendFormat": "{{replica}}",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {},
+        "overrides": []
+      }
+    },
+    {
+      "id": 10,
+      "title": "KV cache occupancy",
+      "type": "timeseries",
+      "description": "",
+      "gridPos": {
+        "x": 8,
+        "y": 18,
+        "w": 8,
+        "h": 6
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "llmgw_endpoint_kv_cache_usage",
+          "legendFormat": "{{replica}}",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "percentunit"
+        },
+        "overrides": []
+      }
+    },
+    {
+      "id": 11,
+      "title": "Model disk free",
+      "type": "timeseries",
+      "description": "The filesystems holding weights. A node that cannot hold the next model is skipped by placement rather than failing.",
+      "gridPos": {
+        "x": 16,
+        "y": 18,
+        "w": 8,
+        "h": 6
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "llmgw_node_disk_free_bytes",
+          "legendFormat": "{{node}}",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "bytes"
+        },
+        "overrides": []
+      }
+    },
+    {
+      "id": 12,
+      "title": "Seconds since each node last reported",
+      "type": "timeseries",
+      "description": "Heartbeats are every 5s. A line that climbs is an agent going quiet; the control plane stops placing there at 15s.",
+      "gridPos": {
+        "x": 0,
+        "y": 24,
+        "w": 12,
+        "h": 6
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "llmgw_node_last_seen_seconds",
+          "legendFormat": "{{node}}",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "s"
+        },
+        "overrides": []
+      }
+    },
+    {
+      "id": 13,
+      "title": "Replicas by state",
+      "type": "timeseries",
+      "description": "",
+      "gridPos": {
+        "x": 12,
+        "y": 24,
+        "w": 12,
+        "h": 6
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "sum by (state) (llmgw_replicas)",
+          "legendFormat": "{{state}}",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {},
+        "overrides": []
+      }
+    }
+  ]
+}

deploy/monitoring/llmgw-rules.yml

@@ -0,0 +1,155 @@
+# Alerts for llmgw. Every expression here is against a series this cluster
+# actually publishes; nothing is written against a metric that might exist.
+groups:
+  - name: llmgw-fleet
+    rules:
+      - alert: ControlPlaneUnreachable
+        expr: up{job="llmgw"} == 0
+        for: 1m
+        labels: { severity: critical }
+        annotations:
+          summary: "The control plane is not answering"
+          description: >-
+            Nothing has scraped 10.0.0.140:9091 for a minute. Engines already
+            running keep serving — the router is in the same process, so if
+            this is down, so is the gateway.
+
+      - alert: NodeNotReady
+        expr: llmgw_node_up == 0
+        for: 2m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.node }} is {{ $labels.state }}"
+          description: >-
+            The node is known but not taking work. Two minutes, because a node
+            restarting its agent is briefly not ready and is not a problem.
+
+      - alert: AgentGoneQuiet
+        expr: llmgw_node_last_seen_seconds > 60
+        for: 1m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.node }} has not reported for {{ $value | humanizeDuration }}"
+          description: >-
+            Heartbeats are every 5s. A minute of silence is the network or the
+            agent, and the control plane stages expiry at 15s unreachable and
+            60s lost — so this fires around the point it stops placing there.
+
+      - alert: ModelHasNothingServing
+        expr: llmgw_model_replicas_ready == 0
+        for: 10m
+        labels: { severity: critical }
+        annotations:
+          summary: "{{ $labels.model }} has no replica ready"
+          description: >-
+            The model is placed and nothing is serving it. Ten minutes, because
+            a large model on CPU legitimately takes minutes to load. A model
+            that is deliberately stopped publishes no series at all, so this
+            cannot fire for one nobody asked for.
+
+      - alert: PoolSaturated
+        expr: llmgw_pool_saturation > 1
+        for: 5m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.model }} is over its healthy limit"
+          description: >-
+            Saturation is the worse of queue depth and KV occupancy; 1 is the
+            limit measured as healthy, not the point of failure. Sustained above
+            it means requests are queueing — add a replica or lower the load.
+
+      - alert: EndpointBreakerOpen
+        expr: llmgw_endpoint_breaker_open == 1
+        for: 2m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.replica }} has been taken out of rotation"
+          description: >-
+            The router stopped sending to this replica after repeated failures.
+            Traffic is going elsewhere; if it is the only replica, the model is
+            down.
+
+      - alert: EndpointMetricsStale
+        expr: llmgw_endpoint_metrics_stale == 1
+        for: 5m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.replica }} is being routed to blind"
+          description: >-
+            Its queue depth and KV usage are old, so the scheduler is choosing
+            with stale numbers. The replica may be serving perfectly well; the
+            decisions about it are the problem.
+
+      - alert: NodeDiskFilling
+        expr: llmgw_node_disk_free_bytes / llmgw_node_disk_total_bytes < 0.10
+        for: 15m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.node }} has {{ $value | humanizePercentage }} of its model disk free"
+          description: >-
+            Weights are tens of gigabytes. A node that cannot hold the next one
+            is skipped by placement rather than failing, so this is quiet until
+            somebody wonders why a model will not land there.
+
+  - name: monitoring-host
+    rules:
+      - alert: MonitoringDiskFilling
+        expr: >-
+          node_filesystem_avail_bytes{job="monitoring-host",fstype!~"tmpfs|overlay|squashfs"}
+          / node_filesystem_size_bytes{job="monitoring-host",fstype!~"tmpfs|overlay|squashfs"} < 0.15
+        for: 10m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.mountpoint }} on the monitoring host is {{ $value | humanizePercentage }} free"
+          description: >-
+            This filesystem holds the model mirror and the metrics database at
+            once. Filling it stops new weights arriving and stops Prometheus
+            writing — so monitoring goes quiet at the moment it is most needed,
+            which is why this fires at fifteen percent rather than five.
+
+      - alert: MonitoringDiskAlmostFull
+        expr: >-
+          node_filesystem_avail_bytes{job="monitoring-host",fstype!~"tmpfs|overlay|squashfs"}
+          / node_filesystem_size_bytes{job="monitoring-host",fstype!~"tmpfs|overlay|squashfs"} < 0.05
+        for: 5m
+        labels: { severity: critical }
+        annotations:
+          summary: "{{ $labels.mountpoint }} on the monitoring host has {{ $value | humanizePercentage }} left"
+
+  - name: llmgw-traffic
+    rules:
+      # These need traffic through the gateway to have any series at all. A
+      # counter with no observations publishes nothing, so a quiet cluster
+      # simply has no data here rather than a rule that is always false.
+      - alert: HighErrorRate
+        expr: >-
+          sum by (model) (rate(llmgw_requests_total{status=~"5.."}[5m]))
+          / sum by (model) (rate(llmgw_requests_total[5m])) > 0.05
+        for: 5m
+        labels: { severity: critical }
+        annotations:
+          summary: "{{ $labels.model }} is failing {{ $value | humanizePercentage }} of requests"
+
+      - alert: SlowFirstToken
+        expr: >-
+          histogram_quantile(0.99,
+            sum by (model, le) (rate(llmgw_ttft_seconds_bucket[10m]))) > 30
+        for: 10m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.model }} takes {{ $value | humanizeDuration }} to first token at p99"
+          description: >-
+            On CPU this is expected under load and is why the threshold is
+            thirty seconds rather than one. Compare with what ctl bench measured
+            before treating it as a fault.
+
+      - alert: RequestsBeingRejected
+        expr: sum by (band) (rate(llmgw_admission_rejected_total[5m])) > 0
+        for: 5m
+        labels: { severity: warning }
+        annotations:
+          summary: "Admission is turning away {{ $labels.band }} traffic"
+          description: >-
+            Working as designed when the cluster is full: the lowest band is
+            held first. Worth knowing about because the callers being refused
+            do not know why.

deploy/monitoring/prometheus.yml

@@ -1,45 +1,42 @@
-# Scraping llmgw.
+# Prometheus for the llmgw cluster.
 #
-# Point this at metrics_listen, not admin_listen. They are different ports for
-# a reason: the admin listener also carries pprof, which can dump the process
-# heap, and API keys are in that heap in plaintext.
-#
-# In the control plane's configuration:
-#
-#   admin_listen:   "127.0.0.1:9090"   # dashboard, management API, pprof
-#   metrics_listen: "0.0.0.0:9091"     # metrics only, safe to reach
-#
-# One target for the whole fleet. Node and GPU figures are exported by the
-# control plane, which already holds them — agents report hardware at
-# registration and state on every heartbeat — so there is nothing to scrape on
-# the nodes and no Prometheus configuration per machine.
+# One scrape target for the whole fleet. Agents publish nothing of their own —
+# the control plane gathers what they report and republishes it, so node health,
+# disk and endpoint state all arrive here from 10.0.0.140.
 global:
   scrape_interval: 15s
+  evaluation_interval: 15s
+  external_labels:
+    cluster: mynodes
+
+alerting:
+  alertmanagers:
+    - static_configs:
+        - targets: ['localhost:9093']
+
+rule_files:
+  - /etc/prometheus/rules/llmgw.yml
 
 scrape_configs:
+  # The control plane's dedicated metrics port. Not its admin listener: that one
+  # is loopback because it also serves pprof, and pprof can dump the heap where
+  # API keys sit in plaintext.
   - job_name: llmgw
     static_configs:
-      - targets: ["llmgw-ctl.internal:9091"]
+      - targets: ['10.0.0.140:9091']
         labels:
-          cluster: primary
+          instance: control-plane
 
-# Worth alerting on, in rough order of how often it matters.
-#
-#   A node that has stopped reporting:
-#     llmgw_node_last_seen_seconds > 60
-#
-#   A model that should be serving and is not. Covers a replica that will never
-#   start as well as one that has died, which look the same from a distance:
-#     llmgw_model_replicas_ready == 0
-#
-#   A card kept out of placement by a fault, which is otherwise invisible until
-#   a model will not fit:
-#     llmgw_gpu_healthy == 0
-#
-#   A fleet holding models nobody is asking for. Memory used says a model is
-#   loaded; utilisation says whether it is doing anything:
-#     avg_over_time(llmgw_gpu_utilization_percent[1h]) < 5
-#
-#   Requests queueing because the cluster is full, which is the pair of figures
-#   that needed two systems to correlate before these were on one endpoint:
-#     llmgw_admission_queue_depth > 0
+  - job_name: prometheus
+    static_configs:
+      - targets: ['localhost:9090']
+
+  # The machine this runs on. It came with the Prometheus package and was not
+  # being scraped, which left the disk holding both the model mirror and the
+  # metrics themselves watched by nothing — so a mirror that filled it would
+  # take monitoring down and the alert that would have warned did not exist.
+  - job_name: monitoring-host
+    static_configs:
+      - targets: ['localhost:9100']
+        labels:
+          instance: monitoring

deploy/systemd/llmgw-mirror.service

@@ -0,0 +1,29 @@
+[Unit]
+Description=llmgw model mirror
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+# --base is what a catalog entry will say, so it has to be the address nodes
+# reach this machine on — not localhost, and not the name only this host knows
+# itself by. Getting it wrong makes `mirror unused` report files as unreferenced
+# when they are in use, and the next step after that list is deletion.
+ExecStart=/usr/local/bin/llmgw mirror serve \
+    --dir /srv/models \
+    --listen :8081 \
+    --api-listen :8082 \
+    --base http://models.internal:8081
+User=llmgw
+Group=llmgw
+Restart=on-failure
+RestartSec=2s
+NoNewPrivileges=true
+ProtectSystem=strict
+ProtectHome=true
+PrivateTmp=true
+# The one writable path. The mirror serves reads to the whole fleet and writes
+# only what its own API was asked to fetch, so nothing else needs to be.
+ReadWritePaths=/srv/models
+
+[Install]
+WantedBy=multi-user.target

docs/choosing-models.md

@@ -0,0 +1,149 @@
+# Choosing what to run
+
+llmgw serves whatever you give it and has no opinion about which models those
+are. This page has an opinion, because "we have GPUs, now what" is the question
+that stalls most of these projects — and the usual first answer, one large
+model, is the wrong shape.
+
+Model releases move faster than documentation. Treat the families named here as
+a starting point and check what is current before committing; the reasoning is
+what should outlive the names.
+
+## Three jobs, not one model
+
+The requests a company actually makes fall into three groups with different
+needs, and no single model serves all three well.
+
+**Interactive** — chat, autocomplete, short questions. Most of your traffic.
+Here latency beats quality: a person waits two seconds and does not wait
+twenty. A 7–14B class model, always resident.
+
+**Heavy** — long document analysis, code generation, anything that needs
+reasoning. Quality beats latency, and a queue is acceptable. A 30–70B class
+model, loaded on demand:
+
+```yaml
+scaling: { min: 0, max: 1, idle_timeout: 30m }
+```
+
+llmgw starts it when somebody asks and stops it when nobody has, so it costs
+nothing while it is not in use. The first request waits for the load, and the
+client is told that rather than refused.
+
+**Embeddings** — and this is the one that gets forgotten. Most "let us chat
+with our documents" projects are retrieval projects, and retrieval runs on an
+embedding model. It is small, it is cheap, and it carries the highest request
+volume of the three: every document chunk once, every query forever. llmgw
+routes `/v1/embeddings` like any other endpoint.
+
+A deployment with the first two and not the third can answer questions from
+what the model already knew. It cannot answer questions about your company.
+
+## Swedish and English
+
+The first thing to know is that this combination is harder than either alone.
+
+Open models are strongest in English by a wide margin. Swedish is served —
+better than most languages its size, because the Nordic languages appear in
+multilingual training sets — but the gap is real, and it widens as models get
+smaller. **A 7B model that feels fine in English can feel noticeably worse in
+Swedish**, which means the interactive tier may need to be larger than an
+English-only deployment would choose. Test before deciding; do not assume the
+English benchmark transfers.
+
+Two things matter more than the model's headline multilingual score:
+
+**Cross-lingual retrieval.** A Swedish company writes internal documents in
+Swedish and reads technical documentation in English. Somebody will ask a
+question in Swedish whose answer is in an English document. An embedding model
+that handles both languages separately may still fail to connect them, and it
+fails silently — the model gives a confident answer from the wrong source, and
+nobody notices because the answer reads well. Test this explicitly: ask in one
+language, check that documents in the other are retrieved.
+
+**Swedish compounds.** *Kundtjänstmedarbetare* is one token-hostile word where
+English would use three. This hurts keyword search more than dense retrieval,
+so if you run hybrid search, expect the lexical half to underperform on Swedish
+and weight accordingly.
+
+## How to actually decide
+
+Benchmarks will not settle this. Take twenty real documents and twenty real
+questions from the company — in both languages, including at least a few where
+the question and the answer are in different languages — and compare candidates
+on those. An afternoon of this is worth more than a week of reading leaderboards,
+and it is the only way to find the failure modes that matter to you.
+
+Do the same for the embedding model, separately. A good generation model on top
+of bad retrieval produces fluent, confident, wrong answers, which is the worst
+failure this stack has: it does not look like a failure.
+
+## How many replicas
+
+Not "how many employees". **Concurrency**, which is far lower than headcount
+because people spend most of their time reading rather than waiting on a model.
+Two hundred people generate perhaps five to fifteen concurrent requests at
+peak, for interactive use.
+
+vLLM batches continuously, so one replica on a capable card serves many
+concurrent requests. Which means the replica count is usually decided by
+**redundancy rather than throughput**:
+
+| Tier | Replicas | Why |
+|---|---|---|
+| Embeddings | 2 | Not for capacity. If retrieval stops, every document feature stops. |
+| Interactive | 2 | So a node can be rebooted during working hours. |
+| Heavy | 0–1, on demand | Loaded when asked for, stopped when idle. |
+
+Then measure rather than guess:
+
+```
+llmgw ctl bench <model>
+```
+
+It climbs concurrency until the SLO breaks and reports the last level that met
+it. Peak throughput arrives long after latency has become unusable, so
+throughput alone overstates what the hardware can serve.
+
+## Where the weights live
+
+The single highest-value piece of infrastructure here is **a local mirror**: one
+internal HTTP server holding the model files. [deploy/mirror/](../deploy/mirror/)
+has the nginx config and the directory layout. Hugging Face is slow, rate
+limited, and blocked outright on plenty of corporate networks. Eight nodes each
+pulling forty gigabytes from the internet is a day; from a mirror on the same
+switch it is minutes. Point a model's `source` at it and llmgw fetches from
+there.
+
+**Pin the version.** Put it in the path — `/models/qwen2.5-14b/2026-03/q4_k_m.gguf`
+— so a repository that changes upstream cannot change what you are running.
+Without this, a prompt that worked last week gives a different answer this week
+and nobody can explain why.
+
+**Quantisation.** For llama.cpp, `q4_k_m` is the practical default: the quality
+difference from `q8_0` is smaller than most people expect and the memory
+difference is a factor of two. For vLLM on modern cards, FP8 or AWQ. Run
+`llmgw ctl plan <model>` to see whether a given precision fits before
+downloading anything.
+
+**Disk.** Budget three to five times what you first estimate. Several
+quantisations, several versions, and partial downloads all accumulate.
+`llmgw ctl nodes` shows free space per node once `model_dirs` is configured,
+and `llmgw ctl weights` reclaims files llmgw downloaded and no model needs —
+never files you placed yourself.
+
+Under ten nodes, let each keep its own copy; it is simpler and the disk is
+cheap. Above that, consider a shared filesystem — llmgw notices one and will
+not count the same free terabyte once per node.
+
+## Before anyone starts using it
+
+Set budgets and limits on day one, not later:
+
+```
+llmgw keys limits --scope project --id <project> --rpm 60 --budget 50000000
+```
+
+The reason is not technical. A resource that appears free and unlimited becomes,
+three months later, a cost nobody can account for — and introducing limits at
+that point is a much harder conversation than having had them from the start.

docs/data.md

@@ -0,0 +1,133 @@
+# What llmgw stores, and what leaves the machine
+
+Written for the review that happens before a gateway is allowed near anyone's
+documents. It describes what the software does. Mapping that onto an
+obligation — GDPR, a sector rule, an internal policy — is work for whoever
+holds the obligation; this page exists so that work starts from facts rather
+than from a vendor's summary.
+
+Everything below is checkable in the source, and the places to look are named.
+
+## The short version
+
+By default llmgw stores **no prompt or completion text anywhere**. It stores
+who made a request, when, how many tokens, and what it cost.
+
+Two features change that, and both are off until somebody turns them on for a
+named project, and neither can be turned on without also saying for how long.
+
+Nothing on the serving path leaves the cluster.
+
+## What is stored, exactly
+
+**Usage records**, one per request: the key, the project, the model asked for,
+the model that answered, prompt and completion token counts, cost, HTTP status
+and latency. No message content. The row is in `internal/store/store.go` as
+`Usage` and there is no field for text in it.
+
+**API keys**, as SHA-256 hashes. The key itself is shown once, at creation, and
+never stored — so a leaked database hands over no working credential, and
+neither llmgw nor an administrator can recover a key somebody lost. A prefix is
+kept so a key can be identified in a listing without revealing it.
+
+**People and tenancy**: accounts, the organisation/team/project tree, and the
+limits attached to it. Passwords are hashed.
+
+**Sessions** for the browser pages, as tokens.
+
+**Conversations and agent notes** — only for a project that asked. See below.
+
+## What is not stored
+
+Prompts. Completions. Anything the model was shown or produced, unless the
+project turned on conversations or notes.
+
+The chat page in llmgw's own web interface keeps the conversation in a
+JavaScript variable in the browser. Refresh it and the conversation is gone,
+because it was never anywhere else.
+
+## Conversations and notes, when they are turned on
+
+Two features keep text, and both work the same way:
+
+- **Conversations** (`x-llmgw-thread`) let a client send only its new turn while
+  llmgw replays the history — including to a different model than the one the
+  conversation started on.
+- **Agent notes** (`/v1/memory`, and MCP tools) let an agent keep something for
+  itself across conversations.
+
+Both are **off for every project** until an operator runs `llmgw keys threads`
+or `llmgw keys memory` for that project by name. A request carrying a thread id
+against a project that has not is refused with an explanation, not quietly
+served without its history.
+
+Both **require a retention window**. There is no way to turn either on without
+saying how many hours, because a store with no expiry is a transcript of
+everything a team ever typed, kept forever because nobody chose a number. The
+window is measured from the last turn, so an active conversation does not
+expire underneath it, and a background sweep removes what has passed it.
+
+Both are **scoped to a project**, and the scope is part of every lookup rather
+than a check afterwards. Reading a conversation that belongs to another project
+returns the same answer as reading one that does not exist, so a leaked id
+cannot be used to discover what exists elsewhere.
+
+`DELETE /v1/threads/{id}` removes a conversation and its messages immediately.
+It is a client endpoint, not an administrative one: the person who said the
+thing can unsay it without asking somebody else to read it first.
+
+Turning retention off stops new storage, and existing conversations go at the
+next hourly sweep.
+
+## What leaves the machine
+
+**Nothing on the serving path.** A request arrives, is routed to an engine on
+your own hardware, and the answer comes back. The router talks to engines, the
+engines talk to nothing, the agents talk to the control plane. All of it is
+inside the cluster.
+
+Two things can reach outside, both started by an operator rather than by a
+request:
+
+**Fetching weights.** When a model's `source` is a URL, the agent downloads it.
+Point that at an internal mirror and nothing leaves at all — which is also the
+faster arrangement, because eight nodes pulling forty gigabytes from the
+internet is a day's work and from a local mirror it is minutes. A `source` that
+is a path on disk is never fetched.
+
+**`llmgw ctl plan`.** Reads a model's architecture from huggingface.co to work
+out whether it fits your hardware. An operator command run at planning time,
+not part of serving, and not run by anything automatically.
+
+There is no telemetry, no licence check, and no call home. The binary is
+static, built with `CGO_ENABLED=0`, and runs in a network with no route to the
+internet as long as the weights are already on disk.
+
+## What the software cannot do for you
+
+**The admin port serves pprof**, and pprof can dump the process heap — where
+API keys sit in plaintext while they are being checked. Bind it to loopback and
+reach it over SSH. The default does this; a deployment that changes it should
+know why.
+
+**The store holds everything.** With conversations turned on it holds text as
+well as accounting, so where it lives, who can read it and how it is backed up
+are decisions with the same weight as the ones above. The PostgreSQL DSN
+belongs in a file readable only by the user llmgw runs as, which is why the
+configuration takes `@/path/to/file` rather than the address itself.
+
+**Model weights are yours.** llmgw reclaims only files it downloaded, and it
+identifies those by a name it generated. Anything you put on a node it will not
+touch, whatever the catalog says.
+
+## Where to look
+
+| Claim | Where |
+|---|---|
+| Usage rows carry no text | `internal/store/store.go`, type `Usage` |
+| Keys are stored hashed | `internal/store/keys.go` |
+| Conversations are off by default | `internal/store/threads.go`, `ThreadRetention` |
+| Retention is required | `llmgw keys threads --help` |
+| Scope is part of the lookup | `internal/store/threads.go`, `Thread` |
+| Deletion is a client endpoint | `internal/router/threads.go`, `handleDeleteThread` |
+| Weight fetching is the only serving-path egress | `internal/agent/weights/fetch.go` |

docs/install.md

@@ -13,7 +13,7 @@ this page says otherwise, and where it says otherwise it says so plainly.
 Read this first, because it is the difference between an afternoon and a week.
 
 llmgw runs engines and routes requests. It does **not** install GPU drivers,
-CUDA, inference engines, or model weights. A node missing any of them still
+CUDA, or the inference engines themselves. A node missing any of them still
 registers, still reports `ready`, and still fails every replica placed on it —
 so the useful order is to make a node able to serve *before* onboarding it, not
 after.
@@ -23,12 +23,15 @@ after.
 | NVIDIA driver and `nvidia-smi` | GPU discovery, classes, health |
 | InfiniBand stack (`rdma-core`/OFED) | Fabric discovery and the invariant that a replica never spans one |
 | vLLM, SGLang or llama.cpp | Launching them, the right flags, supervision |
-| Model weights on disk | Placement, routing, accounting |
+| Model weights, from somewhere | Fetching them, placement, routing, accounting |
 
 The reason is boring and unavoidable: drivers and engines are platform decisions
-with kernel modules and version matrices attached, and distributing weights is
-its own problem — hundreds of gigabytes that want a shared filesystem, not a
-copy from somebody's laptop. It is a known gap, not an oversight.
+with kernel modules and version matrices attached.
+
+Weights are the one item on that list llmgw does handle. It includes a mirror —
+step 4 — which holds the files on your own network and hands them to the nodes,
+so getting weights onto every machine is not a problem you solve before
+starting. What llmgw will not do is choose the models for you.
 
 ## 0. Before you begin
 
@@ -49,7 +52,10 @@ them, so nodes need no inbound access except your own SSH.
 | --- | --- | --- |
 | 8080 | clients, the web UI | your users, usually via a reverse proxy |
 | 7443 | node agents | the node network only |
-| 9090 | `llmgw ctl`, metrics, pprof | loopback only — it serves pprof, which can dump the process heap, and API keys live in that heap |
+| 9090 | `llmgw ctl`, the management API, pprof | loopback only — it serves pprof, which can dump the process heap, and API keys live in that heap |
+| 9091 | Prometheus, if you run it | your monitoring host; metrics only, nothing else is served here |
+| 8081 | node agents fetching weights | the node network only (the mirror machine) |
+| 8082 | the control plane administering the mirror | the control plane only — this port can delete models and start downloads |
 
 Decide now which address nodes will use to reach the control plane. It goes into
 a certificate and changing it later means reissuing one.
@@ -59,7 +65,7 @@ a certificate and changing it later means reissuing one.
 One static binary, no runtime dependencies, the same on every machine.
 
 ```
-VERSION=v0.6.0    # check the releases page for the current tag
+VERSION=v0.8.0    # check the releases page for the current tag
 curl -fsSLO https://gitea.mynodes.xyz/ilker/llmgw/releases/download/$VERSION/llmgw-$VERSION-linux-amd64
 curl -fsSL  https://gitea.mynodes.xyz/ilker/llmgw/releases/download/$VERSION/checksums.txt | sha256sum -c --ignore-missing
 sudo install -m0755 llmgw-$VERSION-linux-amd64 /usr/local/bin/llmgw
@@ -148,11 +154,22 @@ auth:
   # that is the bill or the outage.
   default_rpm: 60
   default_tpm: 200000
+  # What a request costs, which is what makes a budget mean anything. Without
+  # a price every request costs zero, so a budget is displayed, counted against
+  # nothing, and never binds — and nobody finds out until they expected it to
+  # stop something. Self-hosted models have no per-token price, so the unit is
+  # yours to choose: GPU-seconds, a notional rate, or 1000 everywhere, which
+  # turns a budget into a plain token quota.
+  default_price:
+    prompt_per_1k: 1000
+    completion_per_1k: 1000
+  # pricing:
+  #   qwen2.5-coder-7b-instruct: { prompt_per_1k: 500, completion_per_1k: 1500 }
 
 landing:
   contact: "platform@example.com"
 
-# Start with none. Models are added while the cluster runs — see step 7 —
+# Start with none. Models are added while the cluster runs — see step 8 —
 # and anything listed here is declarative: it is in the catalog on every start
 # and cannot be removed without editing this file.
 models: []
@@ -200,7 +217,103 @@ sudo -u llmgw llmgw ctl status
 been added yet. It is worth running now so its output is familiar before you
 need it.
 
-## 4. Preparing a node
+## 4. The mirror
+
+Do this before the nodes, not after. Every node fetches its own copy of a
+model's weights, and where they fetch from decides whether bringing up a cluster
+takes minutes or a day. Eight nodes pulling forty gigabytes each from Hugging
+Face is a day's work, and on a good number of corporate networks it is not
+possible at all — the nodes have no route out, which is usually the point.
+
+The mirror is one machine holding the files on your own network. It is the same
+binary again, and it is the highest-value piece of infrastructure around llmgw
+for the least work.
+
+### Running it
+
+```
+sudo install -d -o llmgw -g llmgw /srv/models
+llmgw mirror serve --dir /srv/models --base http://models.internal:8081
+```
+
+In production, the unit:
+
+```
+sudo cp deploy/systemd/llmgw-mirror.service /etc/systemd/system/
+sudo systemctl daemon-reload && sudo systemctl enable --now llmgw-mirror
+```
+
+**Two listeners, on purpose.** `:8081` serves bytes and nothing else — that is
+what every agent talks to, and it is plain HTTP because a weights fetch is a
+public file transfer that is checksummed anyway. `:8082` is the management API:
+it can start downloads and delete models, and only the control plane needs it.
+The split is what lets you lock the second one down without re-certifying every
+agent that fetches from the first.
+
+Give the management port the cluster's own CA and it takes client certificates:
+
+```
+llmgw mirror serve --dir /srv/models --base http://models.internal:8081 \
+  --tls-ca /etc/llmgw/ca.pem \
+  --tls-cert /etc/llmgw/mirror.pem --tls-key /etc/llmgw/mirror-key.pem
+```
+
+Without those three flags the management port is open to anyone who can reach
+it, and the mirror says so in its own log at every start rather than letting you
+find out later.
+
+### The layout
+
+```
+/srv/models/
+  qwen2.5-14b-instruct/
+    2026-03/
+      q4_k_m.gguf
+```
+
+Model, then version, then files. The version directory is the point of the
+layout rather than decoration: a repository that changes upstream changes what
+you are running, and the first sign is a prompt that worked last week answering
+differently this week with nobody able to say why. Pin it here and that cannot
+happen.
+
+A version holding several files is a sharded model, and is served as a set: the
+agent asks the directory what it consists of and fetches all of it into one
+place, marking it complete only when the last file lands. A catalogue entry for
+one names the directory, trailing slash included.
+
+### Telling the control plane about it
+
+```yaml
+# /etc/llmgw/server.yaml
+mirror:
+  api: https://models.internal:8082
+  tls:
+    ca: /etc/llmgw/ca.pem
+    cert: /etc/llmgw/server.pem
+    key: /etc/llmgw/server-key.pem
+```
+
+With this set, `/admin/mirror` appears: what the mirror holds, what it is
+downloading and how far along, a search of Hugging Face, and a form that puts a
+mirrored model into service without anyone writing YAML. Without it that page
+stays unmounted and the mirror is still perfectly usable — a model's `source` is
+just a URL, and the URL can be yours.
+
+If the control plane cannot reach the internet, `mirror.registry` points the
+search at an internal registry mirror instead.
+
+### Filling it
+
+```
+llmgw mirror fetch --model qwen2.5-14b-instruct --version 2026-03 \
+  hf:Qwen/Qwen2.5-14B-Instruct-GGUF/qwen2.5-14b-instruct-q4_k_m.gguf
+```
+
+or from the admin page, or by copying files into the directory by hand — the
+mirror is a directory, and anything that puts files in it works.
+
+## 5. Preparing a node
 
 **Do this before onboarding.** A node missing its driver or engine registers
 healthy and fails on first placement, and the reason is three logs away.
@@ -305,9 +418,13 @@ llmgw ctl engine-check vllm --bin /opt/venvs/vllm/bin/vllm
 
 ### Weights
 
-Put them where the node can read them. There is no distribution mechanism yet;
-`scp`, a shared filesystem, or the engine's own downloader are all valid, and
-which you want depends on how many nodes you have.
+Nothing to do here if you built the mirror in step 4 — the agent fetches what a
+model's `source` names, and that source is your mirror. This section is for the
+arrangements where the files are already on the machine.
+
+Put them where the node can read them. `scp`, a shared filesystem, or the
+engine's own downloader are all valid, and which you want depends on how many
+nodes you have.
 
 ```
 sudo install -d -o root -g root /var/lib/llmgw-models
@@ -324,11 +441,18 @@ model's `source` at a file to download and the agent fetches it once, before the
 engine is launched:
 
 ```yaml
+# your mirror, which is what step 4 was for
+source: http://models.internal:8081/qwen2.5-coder-3b-instruct/2026-03/q4_k_m.gguf
+# a whole version, for a model of several files — the trailing slash is the signal
+source: http://models.internal:8081/llama-3.1-70b-instruct/2026-03/
+# or straight from Hugging Face, where the nodes can reach it
 source: hf:Qwen/Qwen2.5-Coder-3B-Instruct-GGUF/qwen2.5-coder-3b-instruct-q4_k_m.gguf
-# or an ordinary URL
-source: https://models.internal/qwen2.5-coder-3b-q4_k_m.gguf
 ```
 
+A fetch reports its progress to the control plane, so `llmgw ctl replicas` and
+the dashboard both show how far along a node is. A replica pulling a hundred
+gigabytes and one that is wedged do not look alike.
+
 Eight shards of one replica starting together produce one download, not eight.
 An interrupted fetch resumes rather than starting over, and a partial file is
 never visible under the model's name — so a half-downloaded model cannot be
@@ -374,7 +498,7 @@ How much disk a node needs follows from the models rather than a rule of thumb.
 `llmgw ctl model plan <repo>` prints the sizes; as a starting point, a 70B at
 bf16 is 141 GB and the same model at q4 is 43 GB.
 
-## 5. Adding the node
+## 6. Adding the node
 
 Two halves, because they need different access: building the machine needs the
 hypervisor, and issuing its identity needs the certificate authority. Where one
@@ -431,7 +555,7 @@ Repeat for each node. Nodes in the same fabric must be given the same `--fabric`
 name; nodes with no InfiniBand take no `--fabric` at all and are each their own
 island, which is fine for single-node models.
 
-## 6. Setting the node's CPU model, if it is a VM
+## 7. Setting the node's CPU model, if it is a VM
 
 Proxmox defaults a VM to `kvm64`, which hides AVX, AVX2, FMA and even SSE4.2
 from the guest. Inference then falls back to scalar code and is not slightly
@@ -448,7 +572,7 @@ only change being `--cpu host`:
 probably did not, and nothing about it looks broken — every number is simply
 small. Check with `grep -qw avx2 /proc/cpuinfo` on the node.
 
-## 7. Adding a model
+## 8. Adding a model
 
 Write the definition. This is the same shape a model takes in `server.yaml`, so
 an entry moves between the two by being copied:
@@ -457,7 +581,7 @@ an entry moves between the two by being copied:
 # qwen-coder-3b.yaml
 name: qwen2.5-coder-3b-instruct
 aliases: [coder]
-source: /var/lib/llmgw-models/qwen2.5-coder-3b-instruct-q4_k_m.gguf
+source: http://models.internal:8081/qwen2.5-coder-3b-instruct/2026-03/q4_k_m.gguf
 routing_profile: chat
 # What a client may use in one request. Worth stating: llama.cpp divides
 # --ctx-size across its parallel slots, so a pool of 65536 over two slots is
@@ -480,6 +604,17 @@ llmgw ctl model add -f qwen-coder-3b.yaml
 llmgw ctl models
 ```
 
+Adding a model checks that the source is actually there before accepting it, so
+a typo or a mirror that is down is a sentence now rather than a replica failing
+on a node in five minutes with an error naming the node. `--no-source-check`
+skips it, which is right when the nodes reach the mirror by an address this
+machine does not.
+
+With a mirror configured you can skip the file entirely: `/admin/mirror` lists
+what the mirror holds and offers a form that writes the same definition, with
+the engine, the source and the memory figure already filled in from the file
+itself.
+
 ```
 MODEL                      SERVING  REPLICAS  STATUS   ALIASES
 qwen2.5-coder-3b-instruct  yes      1/1       ready    coder
@@ -501,8 +636,8 @@ scaling: { min: 0, max: 1, idle_timeout: 15m }
 model starts it and waits briefly; if it is not serving by then the client gets
 a 503 with `Retry-After` and an honest "the model is loading", rather than being
 told nothing serves it — a client told to retry succeeds where one told the
-model does not exist gives up. Once nobody has asked for `idle_timeout`, the
-replicas are stopped and the cards go back to the pool.
+model does not exist gives up. Once nobody has asked for it in `idle_timeout`,
+the replicas are stopped and the cards go back to the pool.
 
 `min: 0, max: 0` is different and equally useful: configured, and deliberately
 not running. A model with no `scaling` stanza at all gets that, because making
@@ -544,7 +679,42 @@ llmgw ctl explain-placement qwen2.5-coder-3b-instruct
 which names every candidate and why each was rejected, rather than leaving you
 to infer it.
 
-## 8. The first request
+### How many replicas, and how much each one carries
+
+Replica count is changed while the cluster runs, without editing the
+definition:
+
+```
+llmgw ctl scale qwen2.5-coder-3b-instruct --min 3
+```
+
+Models added at runtime only. One declared in `server.yaml` is read again on
+every start, so a number set here would hold until the next restart and then
+revert — change it in the file instead.
+
+How much traffic each replica is given is the profile's `weight`, and guessing
+it is how a cluster ends up with one saturated replica beside two idle ones.
+Measure it instead:
+
+```
+llmgw ctl bench qwen2.5-coder-3b-instruct --save
+```
+
+Concurrency climbs until the latency target stops being met, and the last level
+that held it is the capacity — peak throughput arrives long after latency has
+become unusable, so throughput alone would overstate what the hardware serves.
+It prints the `weight` to put in the profile, and `--save` records the figure
+against the model so the dashboard can show what the cluster is measured to
+carry rather than what somebody typed.
+
+This drives real traffic and competes with real requests. Run it against a
+cordoned replica, or in a window where that is acceptable.
+
+An engine that publishes no metrics endpoint is measured from the outside only.
+The run says so, and queue depth and KV occupancy are then not part of the
+answer.
+
+## 9. The first request
 
 ```
 curl http://<control-plane>:8080/v1/chat/completions \
@@ -566,9 +736,9 @@ temperature, ECC faults and utilisation.
 
 **The dashboard is the answer for most clusters.** `/admin` shows capacity per
 GPU class, free whole nodes per fabric, which replicas are holding hardware
-nothing is asking for, and the same node and model tables `ctl` prints — with
-no monitoring stack to stand up beside it. One binary, no dependencies, which
-is the point.
+nothing is asking for, who is using the cluster and what they are running, and
+the same node and model tables `ctl` prints — with no monitoring stack to stand
+up beside it. One binary, no dependencies, which is the point.
 
 Prometheus is worth adding when you want history, alerts, or one view across
 several clusters. The dashboard shows what is true now; it keeps nothing.
@@ -585,8 +755,10 @@ One scrape target for the whole fleet. The node and GPU figures come from the
 control plane, which already holds them, so there is nothing to install on the
 nodes and no Prometheus configuration per machine.
 
-A starting configuration, with the alerts worth having, is in
-[deploy/monitoring/prometheus.yml](../deploy/monitoring/prometheus.yml).
+A starting configuration, with the alerts worth having and a provisioned
+dashboard, is in [deploy/monitoring/](../deploy/monitoring/) — including where
+to run it, which is not the control plane: monitoring that dies with the thing
+it monitors tells you nothing at the moment you need it.
 
 Two figures repay knowing about. `llmgw_gpu_utilization_percent` is absent
 rather than zero when the driver does not report it — a flat zero across a
@@ -594,6 +766,258 @@ fleet means the measurement is missing, not that the fleet is idle. And
 `llmgw_gpu_nvlink_peers` at zero everywhere means the topology is unknown,
 which is what a binary built without the nvml tag reports.
 
+## PostgreSQL, when SQLite is no longer the right shape
+
+SQLite is the default and stays the right answer for most installations: one
+file, no server, and a request path that barely touches it — usage is written
+in batches, budgets are counted in memory, authentication is cached.
+
+Two things change that. Conversations and agent notes read and write on every
+turn, which is the first per-request load this store has had; and SQLite takes
+one writer at a time, so those turns serialise. And a second control plane is
+impossible while the store is a file on one machine's disk.
+
+The setting is the same one, and the address decides:
+
+```yaml
+auth:
+  store: "@/etc/llmgw/secrets/store-dsn"
+```
+
+A path is SQLite, a `postgres://` URL is PostgreSQL, and `@` followed by a path
+reads the address out of that file — which is how a DSN carrying a password
+stays out of a configuration that gets copied, diffed and pasted into issues.
+Keep that file at `0600` and owned by the user llmgw runs as.
+
+Moving an existing installation is a copy, not a fresh start. Keys are stored
+hashed, so starting fresh means reissuing every one of them and being unable
+to say which was which:
+
+```
+sudo systemctl stop llmgw-server
+llmgw keys move-store --from /var/lib/llmgw/llmgw.db --to "@/etc/llmgw/secrets/store-dsn"
+# point auth.store at the new address, then
+sudo systemctl start llmgw-server
+```
+
+Stop the gateway first. The copy makes no attempt to follow a store being
+written to: rows landing after the table they belong to has been passed would
+simply be absent, and nothing would say so. The destination must be empty, and
+the source is left untouched — so the way back is to point the configuration at
+the file again.
+
+On a machine that also runs the control plane, PostgreSQL's defaults assume it
+owns the box. A gateway swapped out to give the database cache is a worse
+outcome than a smaller cache, so cap it: `shared_buffers = 96MB` and
+`max_connections = 40` are enough for a store whose working set is measured in
+megabytes.
+
+## Conversations across models
+
+Off everywhere until a project asks. llmgw otherwise stores no message content
+at all — usage rows carry token counts, costs and latencies, never text, and
+even its own chat page keeps the conversation in the browser. A gateway that
+keeps nothing anyone typed is a much easier thing to have in front of a team,
+so keeping it is a decision someone makes rather than a default they discover.
+
+```
+llmgw keys threads --project default --retain-hours 24
+```
+
+The retention is required. A conversation store with no expiry is a transcript
+of everything a team ever typed, kept forever because nobody chose a number.
+The window is measured from the last turn, so a conversation still in progress
+does not expire underneath it.
+
+A client then sends a thread id and only its new turn:
+
+```
+curl $BASE/v1/chat/completions \
+  -H "Authorization: Bearer $LLMGW_API_KEY" \
+  -H "x-llmgw-thread: my-conversation" \
+  -d '{"model": "<model-name>", "messages": [{"role":"user","content":"and then?"}]}'
+```
+
+llmgw puts the stored history in front of that turn and sends it to whichever
+model is serving — **including a different one from the one the conversation
+started on**. That is the point: a thread begun on a large model can continue
+on a small one, and a caller dropped to a cheaper model by a spent budget keeps
+their conversation.
+
+When the history does not fit the model now serving, the oldest turns are
+dropped until it does, and the reply carries `x-llmgw-thread-trimmed` with how
+many. System messages are never dropped, and neither is the turn the caller
+just sent.
+
+`DELETE /v1/threads/{id}` forgets one. It is a client endpoint on purpose: the
+person who said the thing should be able to unsay it without asking an
+operator to read it first. A thread belonging to another project answers the
+same as one that never existed.
+
+Turning it off again stops new storage, and existing conversations go at the
+next hourly prune:
+
+```
+llmgw keys threads --project default --off
+```
+
+## Memory notes for agents
+
+Also off until a project asks, for the same reason. Where conversations are what
+was said, notes are what an agent chose to keep, and they persist across
+conversations:
+
+```
+llmgw keys memory --project default --retain-hours 720
+```
+
+The retention is required and is measured from the last write to a note. An agent
+then saves, searches and deletes notes through `/v1/memory` or, as tools, through
+the MCP server at `/mcp` (`remember`, `recall`, `forget`). Both need the API key
+of the project, and notes are private to the project and to the agent id sent in
+`x-llmgw-agent`. A note in another project or agent answers the same as one that
+does not exist. llmgw does not add these tools to requests itself.
+
+Turning it off stops new storage, and existing notes go at the next hourly
+prune:
+
+```
+llmgw keys memory --project default --off
+```
+
+Without authentication, `threads.store` and `threads.retain_hours` in
+`server.yaml` keep both conversations and notes in one shared pool. That is for
+one trusted machine only; see [quickstart.md](quickstart.md).
+
+## Two control planes
+
+llmgw handles the hard half itself: only one places work, and agents find
+whichever is leading. What it does not do is move client traffic, which
+happens in front of both.
+
+[deploy/ha/](../deploy/ha/) has the pattern and a keepalived example. The one
+thing to get right is the health check: point it at `/ready`, not `/health`.
+`/health` is 200 on a standby, which is alive and holds no endpoints at all,
+so a load balancer using it would send half the traffic to something that
+refuses every request.
+
+## Upgrading
+
+One binary on every machine, so an upgrade is a copy and a restart. The part
+worth care is the copy, not the order.
+
+**Nothing stops serving.** Engines are started with `KillMode=process`, so they
+keep running when the agent restarts, and the agent reports them back on its
+next connection. The control plane holds no replica state across its own
+restart either — it rebuilds the fleet from what the agents report. A rolling
+upgrade of the whole cluster interrupts no request that is not in flight at the
+moment of the restart.
+
+**A mixed fleet is fine while you roll.** The agent protocol only ever gains
+fields, and both sides treat an absent one as unknown rather than as a default.
+Order does not matter; finish the roll the same day anyway, because
+`ctl nodes` reports each agent's version and a fleet that disagrees with itself
+is one more thing to hold in your head.
+
+```
+VERSION=v0.9.0
+# copy to every machine first, install nowhere yet
+for h in 10.0.0.140 10.0.0.141 10.0.0.142 10.0.0.146; do
+  ssh $h 'cat > /tmp/llmgw-new; stat -c %s /tmp/llmgw-new' < llmgw-$VERSION-linux-amd64
+done
+```
+
+**Check every size before installing anything.** This is not defensive writing:
+on a host whose login shell prints anything, `scp` fails outright, and the
+obvious workaround — `ssh -n host 'cat > file'` — points stdin at `/dev/null`
+and writes a *zero-byte file* while reporting success. Installing that on a
+fleet takes the fleet down. Compare the number on each machine with the local
+one, then:
+
+```
+sudo install -m0755 /tmp/llmgw-new /usr/local/bin/llmgw
+sudo systemctl restart llmgw-agent     # or llmgw-server, or llmgw-mirror
+llmgw --version
+```
+
+Then confirm what you have, rather than assuming:
+
+```
+llmgw ctl nodes       # every agent back, LAST SEEN in seconds
+llmgw ctl replicas    # same replicas, adopted rather than replaced
+```
+
+A replica may come back marked **`ready (stale profile)`**. That means it is
+running a profile that has since been edited — the reconciler leaves a serving
+replica alone on purpose, because churning one costs more than the difference.
+Restart it when you want the new profile to take effect; the flag is there so
+the choice is yours rather than silent.
+
+## Backing up
+
+Three things, in descending order of how badly you want them.
+
+**The certificate authority.** `ca-key.pem`, wherever you put it in step 2.
+Losing it does not stop the cluster, and it means every node identity has to be
+reissued the next time you touch one. Nothing in llmgw can reconstruct it.
+
+**The store.** Users, API keys, organisations, teams, projects, every limit and
+budget, all usage history, models added at runtime, and measured capacity.
+
+```
+# SQLite
+sqlite3 /var/lib/llmgw/llmgw.db ".backup '/backup/llmgw-$(date +%F).db'"
+
+# PostgreSQL
+pg_dump --format=custom llmgw > /backup/llmgw-$(date +%F).dump
+```
+
+`.backup` and `pg_dump` are both safe against a running gateway. Copying the
+SQLite file with `cp` while it is being written is not.
+
+**Keys are stored hashed and cannot be recovered from a backup or from
+anywhere else.** Restoring an old store restores which keys exist and who owns
+them, not their secrets. A key created after the backup is gone and has to be
+reissued — which is why the store is worth backing up on a schedule rather than
+before upgrades.
+
+**The mirror's directory** is the one thing you can rebuild, by downloading it
+all again. Worth remembering that a pinned version may no longer be upstream to
+download: that is exactly what pinning it protected you from, and a backup is
+what makes the protection outlive the disk.
+
+## Reclaiming disk
+
+`llmgw ctl nodes` has a DISK column: free space against the size it is free
+out of, for the filesystems holding model weights. A dash means the node was
+not asked — an agent with no `model_dirs`, or one older than this — and not
+that it has no space. Shared filesystems are excluded from a node's figure,
+because space on a filer belongs to everyone mounting it.
+
+Weights llmgw downloaded for models that are no longer in the catalog can be
+listed and removed:
+
+```
+llmgw ctl weights             # what could be freed, per node
+llmgw ctl weights --collect   # delete it
+```
+
+**Only files llmgw fetched itself are ever candidates.** They are recognisable
+because llmgw names them after a hash of their source, so the set still wanted
+is computed from the catalog exactly rather than guessed. Anything you put on
+a node — by scp, by a shared mount, by hand — is left alone however
+unreferenced it looks, and no llmgw command will remove it. The nodes check
+every path again before deleting it, so a control plane that is wrong cannot
+talk one into losing a model that took a week to copy.
+
+Which also means llmgw will not reclaim the storage most likely to be filling
+your disk. Weights you placed yourself are yours to remove.
+
+A profile may declare `min_disk_gb`, and a node without that much free is then
+not a candidate for it — checked only on nodes that do not already hold the
+weights, since those need no room for them. Without the field, disk is not
+considered, which is how every profile written before it keeps behaving.
+
 ## When something does not work
 
 **The node is `ready` but every replica fails.** Almost always a missing engine
@@ -604,7 +1028,7 @@ output, which is the actual diagnosis.
 node and the reason: wrong GPU class, not enough free cards, too little memory,
 no fabric, a label it does not carry.
 
-**Everything is slow.** On a VM, check AVX2 (step 6). On CPU, check the
+**Everything is slow.** On a VM, check AVX2 (step 7). On CPU, check the
 quantisation — the difference between `q8_0` and `q4_k_m` is roughly a factor of
 two. `llmgw ctl status` shows how full the cluster is.
 
@@ -618,6 +1042,17 @@ requests. On a slow model, set `timeouts.generation` higher.
 
 ## What to read next
 
+- `docs/choosing-models.md` — which models to run and how many, if the answer
+  to "we have GPUs, now what" is not already obvious
+- `docs/data.md` — what is stored and what leaves the machine, for the review
+  that happens before this is allowed near anyone's documents
+- `docs/reference.md` — every configuration key, generated from the source, so
+  it describes the version you are running rather than the one somebody last
+  wrote about
+- `deploy/mirror/README.md` — the mirror in more detail, including the layout
+  and what belongs on which port
+- `deploy/monitoring/README.md` — Prometheus, Grafana and the alerts, with what
+  each one assumes about this system
 - `docs/node-onboarding.md` — adding nodes in more detail, and what onboarding
   deliberately does not do
 - `docs/onboarding.md` — giving people accounts and keys
@@ -625,11 +1060,20 @@ requests. On a slow model, set `timeouts.generation` higher.
 
 ## Known gaps, so you find them here rather than at 2am
 
-- **Weights are yours to distribute.** No cache, no fetch, no progress.
+- **Every node fetches its own copy.** There is no seeding between nodes, so
+  bringing up eight nodes on a new model is eight downloads. The mirror makes
+  those eight local, which is most of the problem; on a fabric where the nodes
+  can see each other, seeding between them is the obvious next thing to build.
+- **A budget without a price never binds.** Set `default_price` or `pricing`
+  (step 3) or every request costs zero, and a budget is something the dashboard
+  displays rather than something that stops anything. The quotas page says so
+  where it can, which is not the same as being told before you rely on it.
 - **Engines are yours to install.** Onboarding verifies one runs; it installs
   nothing.
-- **One control plane.** No HA yet; it is a single process with a SQLite file.
-  Back up `/var/lib/llmgw/llmgw.db`.
+- **Two control planes have not been run for long.** The leader election and
+  agent failover work and there is a pattern in `deploy/ha/`, but every cluster
+  this has been run on in anger has had one. A single control plane is the
+  well-trodden path.
 - **Multi-node replicas have not been run on real InfiniBand.** The scheduling,
   the fabric invariant and the rendezvous are exercised against simulated
   hardware only. Single-node placement on real GPUs is the well-trodden path;

docs/node-onboarding.md

@@ -139,7 +139,9 @@ So the node is not finished until:
 Both are deliberately outside this script. Engines are a platform decision with
 versions and drivers attached, and distributing weights is its own problem —
 hundreds of gigabytes that want a shared filesystem or a cache, not an scp from
-a laptop. llmgw has no answer for it yet; that is a known gap, not an oversight.
+a laptop. llmgw will fetch weights for a model whose `source` is a URL, one
+copy per node, which covers the simple case; anything larger stays the
+operator's arrangement, and llmgw is content to find the files already there.
 
 The `sim` engine needs neither, which is what makes it useful for exercising
 everything else.

docs/quickstart.md

@@ -0,0 +1,375 @@
+# Quickstart: one machine, one model, ten minutes
+
+Written for someone who wants to see llmgw answer a request on a single
+workstation before reading anything else. No systemd, no `sudo`. For a real
+cluster, see [install.md](install.md).
+
+What you end up with: an OpenAI-compatible endpoint on `localhost:8080` in front
+of a real model served by llama.cpp.
+
+## Two ways to run
+
+Steps 1 to 4 use the simple one, **router-only**: you start the engines yourself
+and tell the router where they are with `--endpoint`. No certificates, no node
+agent.
+
+Step 5 uses the **control plane and node agent**, the same architecture as a real
+cluster on one machine: the agent starts and supervises the engine, and the
+control plane places the model.
+
+## 1. Build
+
+```
+git clone https://gitea.mynodes.xyz/ilker/llmgw && cd llmgw
+make build
+make test
+./bin/llmgw --version
+```
+
+`make build` writes `./bin/llmgw`. Go 1.26 or later is the only requirement.
+
+## 2. Check the path with a fake engine
+
+Before involving a GPU, prove the router works. `sim-engine` is an
+OpenAI-compatible fake that streams placeholder words.
+
+```
+./bin/llmgw sim-engine --listen :8001 --model llama &
+./bin/llmgw server --endpoint e1=http://127.0.0.1:8001=llama &
+
+curl -N localhost:8080/v1/chat/completions \
+  -H 'content-type: application/json' \
+  -H 'x-session-id: demo' \
+  -d '{"model":"llama","messages":[{"role":"user","content":"hello"}],"stream":true}'
+```
+
+Chunks of `alpha beta gamma ...` ending in `data: [DONE]` mean the router is
+fine. Stop both before continuing (see [Stopping](#stopping)).
+
+## 3. Serve a real model
+
+Two processes, two ports. Clients only ever talk to the router; the router
+knows where the engine is because you tell it with `--endpoint`:
+
+```
+curl :8080  ->  llmgw router :8080  ->  llama serve :8082  ->  model
+ (client)        (the gateway)           (the engine)
+```
+
+Requests go to 8080. Port 8082 is the engine's own, and you only touch it to
+check that the engine is up.
+
+llmgw does not install engines or weights. You need:
+
+- a `llama-server` build from llama.cpp (CUDA build if you want the GPU), and
+- a `.gguf` file.
+
+Check the card first with `nvidia-smi`. An 8 GB card holds a 1B model at bf16
+(2.4 GB) or a 9B model at `q4_k_m` (5.4 GB).
+
+Start the engine on its own port. The model name given to `--alias` is the name
+clients will use:
+
+```
+llama serve \
+  -m /path/to/Llama-3.2-1B-Instruct-BF16.gguf \
+  --host 0.0.0.0 --port 8082 -ngl 99 --alias llama
+```
+
+`-ngl 99` offloads every layer to the GPU. Confirm it is up and knows the model:
+
+```
+curl -s localhost:8082/v1/models
+```
+
+Then point the router at it. The format is `id=URL=model`:
+
+```
+./bin/llmgw server --endpoint e1=http://127.0.0.1:8082=llama
+```
+
+## 4. Ask it something
+
+Streaming:
+
+```
+curl -N localhost:8080/v1/chat/completions \
+  -H 'content-type: application/json' \
+  -H 'x-session-id: demo' \
+  -d '{"model":"llama","messages":[{"role":"user","content":"hello"}],"stream":true}'
+```
+
+One reply at once:
+
+```
+curl -s localhost:8080/v1/chat/completions \
+  -H 'content-type: application/json' \
+  -d '{"model":"llama","messages":[{"role":"user","content":"hello"}],"max_tokens":100}'
+```
+
+Also useful: `curl -s localhost:8080/v1/models` and
+`curl -s localhost:8080/health`. `llmgw ctl` does not work in this mode: it
+talks to the control plane's admin API, which router-only mode does not have,
+so `ctl status` answers `404`.
+
+On the machine this was written on, a 1B bf16 model on an RTX 4070 laptop GPU
+answered at roughly 85 tokens per second.
+
+## Several replicas of one model
+
+Give the router more than one `--endpoint` with the same model name and it
+spreads requests across them. `x-session-id` keeps a conversation on the same
+replica so its cache is reused:
+
+```
+./bin/llmgw server \
+  --endpoint e1=http://127.0.0.1:8082=llama \
+  --endpoint e2=http://127.0.0.1:8083=llama
+```
+
+## Stopping
+
+```
+pkill -f "llmgw server"
+pkill -f "llmgw sim-engine"
+```
+
+Stop the engine the way you started it (Ctrl-C in its terminal).
+
+## What goes wrong
+
+**`listening on :8080: bind: address already in use`.** An earlier router is
+still running. `ss -ltnp | grep 8080` names the process; stop it first. This is
+the usual result of moving from step 2 to step 3 without stopping the fake
+engine's router.
+
+**`/v1/models` on the engine returns an empty list.** The engine was started
+without `-m`, so it is a router with nothing to load. Restart it with a model.
+
+**`error: nothing to serve`.** `llmgw server` with no `--endpoint` and no model
+carrying `profiles` has nothing to do. In router-only mode, always pass at least
+one `--endpoint`.
+
+**Requests fail with a model-not-found error.** The name in the request must
+match the name after the last `=` in `--endpoint`, and it must be a name the
+engine actually serves (its `--alias`).
+
+**Answers are poor.** A 1B model is a plumbing test, not a chat assistant. Swap
+in a larger `.gguf`; nothing else changes.
+
+## 5. Let llmgw run the engine (control plane and node agent)
+
+Steps 2 and 3 start the engine by hand. Here a **node agent** starts and
+supervises llama.cpp for you, and a control plane decides where the model runs.
+Still one machine, still no `sudo`. Stop everything from steps 2 and 3 first:
+the GPU has to be free and port 8080 unused.
+
+```
+curl :8080  ->  llmgw server (control plane + router)  <-  llmgw agent (this node)
+                                                              |
+                                                       starts llama-server :20000
+```
+
+The agent dials the control plane over mutual TLS, so this needs a small private
+certificate authority. Everything goes in a scratch directory, `devcluster/`,
+which is in `.gitignore`.
+
+### Certificates
+
+```
+mkdir -p devcluster/pki
+./bin/llmgw ctl cert init-ca --dir devcluster/pki
+./bin/llmgw ctl cert issue-server llmgw-ctl --host 127.0.0.1 --host localhost \
+  --dir devcluster/pki --out devcluster
+./bin/llmgw ctl cert issue-client llmgw-node01 --dir devcluster/pki --out devcluster
+```
+
+The name given to `issue-client` becomes the certificate's common name and must
+equal `node_id` in the agent config. The control plane refuses an agent whose
+claimed id differs from its certificate.
+
+### The engine wrapper
+
+The agent starts the engine as `<bin> --model ... --port ...`. A `llama` binary
+that wants `llama serve ...` refuses every flag, so put a wrapper in front of it:
+
+```
+cat > devcluster/llama-server <<'EOF'
+#!/bin/sh
+exec /path/to/llama serve "$@"
+EOF
+chmod +x devcluster/llama-server
+
+./bin/llmgw ctl engine-check llamacpp --bin devcluster/llama-server
+```
+
+`engine-check` should end with `every flag llmgw builds is accepted`. If your
+build already ships a `llama-server` binary, point `bin` at that and skip the
+wrapper.
+
+### Control plane config: `devcluster/server.yaml`
+
+```yaml
+listen: "127.0.0.1:8080"
+admin_listen: "127.0.0.1:9090"
+agent_listen: "127.0.0.1:7443"
+
+agent_tls:
+  ca: devcluster/pki/ca.pem
+  cert: devcluster/server.pem
+  key: devcluster/server-key.pem
+
+models:
+  - name: llama
+    source: /path/to/Llama-3.2-1B-Instruct-BF16.gguf
+    routing_profile: chat
+    context_length: 8192
+    scaling: { min: 1, max: 1 }
+    profiles:
+      - name: gpu
+        engine: llamacpp
+        gpu_class: 4070-8g
+        weight: 1
+        priority: 100
+        startup_timeout: 5m
+```
+
+Three things in there are easy to get wrong:
+
+- **The model must have at least one profile.** That is what switches the control
+  plane on. With `models: []` the server starts as a plain router and exits with
+  `nothing to serve`.
+- **`gpu_class` decides CPU or GPU.** Without it the replica is placed in the CPU
+  pool and runs on the CPU. Get the class name from `./bin/llmgw agent inventory`
+  (the `CLASS` column).
+- **There is no `auth:` block.** Leave it out for a local trial; with auth on,
+  requests need an API key.
+
+### Agent config: `devcluster/agent.yaml`
+
+```yaml
+node_id: llmgw-node01
+server: "https://127.0.0.1:7443"
+mgmt_addr: "127.0.0.1"
+
+tls:
+  ca: devcluster/pki/ca.pem
+  cert: devcluster/llmgw-node01.pem
+  key: devcluster/llmgw-node01-key.pem
+
+engines:
+  llamacpp:
+    bin: /absolute/path/to/devcluster/llama-server
+
+supervisor:
+  work_dir: devcluster/agent-work
+```
+
+### Run it
+
+Two terminals:
+
+```
+./bin/llmgw server --config devcluster/server.yaml
+./bin/llmgw agent  --config devcluster/agent.yaml
+```
+
+The model is not placed at once. The control plane waits about 35 seconds for the
+fleet to settle after it starts, then places the replica, and the agent then
+loads the weights. Watch it with:
+
+```
+./bin/llmgw ctl nodes
+./bin/llmgw ctl models
+./bin/llmgw ctl replicas
+./bin/llmgw ctl explain-placement llama
+```
+
+`STATUS ready` in `ctl models` means it is serving. `nvidia-smi` should now show
+the model in GPU memory. The same curl from step 4 works against port 8080.
+
+`ctl` works here, unlike in router-only mode, because there is a control plane
+for it to talk to.
+
+Stop it with Ctrl-C in each terminal. The agent leaves the engine running when
+the agent itself stops, so if you restart the agent and the GPU stays full,
+`pgrep -af "llama serve"` will show the engine still up.
+
+## 6. Memory: conversations and notes
+
+Off unless you ask. llmgw otherwise stores nothing anyone typed. On a single
+machine with no API keys, one block in the server config turns on two things:
+
+```yaml
+threads:
+  store: devcluster/data/threads.db
+  retain_hours: 24
+```
+
+Restart the server after adding it. `retain_hours` is required: everything kept
+expires that long after it was last written.
+
+Everything kept this way sits in one shared pool with no owner, and anyone who
+can reach the port and knows an id can read it. That is fine on one machine and
+wrong on a shared network. With auth on, threads and notes are per project
+instead and the block is ignored; see [install.md](install.md).
+
+### Conversations (threads)
+
+Send the same `x-llmgw-thread` id and only the new turn. llmgw puts the stored
+history in front of it:
+
+```
+curl -s localhost:8080/v1/chat/completions \
+  -H 'content-type: application/json' -H 'x-llmgw-thread: demo' \
+  -d '{"model":"llama","messages":[{"role":"user","content":"My name is Deniz."}]}'
+
+curl -s localhost:8080/v1/chat/completions \
+  -H 'content-type: application/json' -H 'x-llmgw-thread: demo' \
+  -d '{"model":"llama","messages":[{"role":"user","content":"What is my name?"}]}'
+
+curl -X DELETE localhost:8080/v1/threads/demo     # forget it
+```
+
+Without the block, a request carrying `x-llmgw-thread` is refused with
+`this gateway keeps no conversations` rather than answered without its history.
+
+### Notes an agent keeps for itself
+
+Notes outlive any one conversation. Each is a key, some text and optional tags,
+kept per agent; name the agent with `x-llmgw-agent` (default `default`), so two
+agents using the same key do not read each other's notes.
+
+```
+curl -X PUT localhost:8080/v1/memory/favorite-color \
+  -H 'content-type: application/json' \
+  -d '{"text":"The user likes dark green.","tags":["preference"]}'
+
+curl 'localhost:8080/v1/memory?q=green'          # search by words
+curl 'localhost:8080/v1/memory?tag=preference'   # or by tag
+curl localhost:8080/v1/memory/favorite-color
+curl -X DELETE localhost:8080/v1/memory/favorite-color
+```
+
+The same notes are offered to an agent as tools over MCP at
+`http://localhost:8080/mcp`: `remember`, `recall` and `forget`. Add that URL to
+your agent's MCP server list (HTTP transport; how you set request headers such
+as `x-llmgw-agent` depends on the client). llmgw does not put these tools into
+chat requests on its own, so an agent that is not given the server never sees
+memory.
+
+Limits: a key is up to 200 bytes, a note's text up to 16 KiB, at most 16 tags,
+1000 notes per agent. Tags are lowercase letters, digits, `-` and `_`.
+
+## Where this stops
+
+Router-only mode (steps 2 to 4) has no authentication, no placement and no model
+management: you start and restart engines yourself. Step 5 adds placement, model
+management and a node agent, and can add authentication as `install.md`
+describes. For a real cluster, with several nodes, systemd and a persistent
+certificate authority, follow [install.md](install.md).
+
+One caveat found while writing this page: `install.md` recommends starting with
+an empty `models: []` and adding models later with `ctl model add`. The control
+plane only turns on when the config declares a model with `profiles`, so that
+first start does not produce a control plane.

go.mod

@@ -6,6 +6,7 @@ require (
 	connectrpc.com/connect v1.21.0
 	github.com/NVIDIA/go-nvml v0.13.4-0
 	github.com/cespare/xxhash/v2 v2.3.0
+	github.com/jackc/pgx/v5 v5.11.0
 	github.com/knadh/koanf/parsers/yaml v1.1.1
 	github.com/knadh/koanf/providers/env/v2 v2.0.1
 	github.com/knadh/koanf/providers/file v1.2.1
@@ -14,6 +15,7 @@ require (
 	github.com/prometheus/client_model v0.6.3
 	github.com/prometheus/common v0.71.0
 	github.com/spf13/cobra v1.10.2
+	golang.org/x/crypto v0.57.0
 	golang.org/x/net v0.59.0
 	golang.org/x/sync v0.23.0
 	google.golang.org/protobuf v1.36.12
@@ -27,6 +29,9 @@ require (
 	github.com/go-viper/mapstructure/v2 v2.4.0 // indirect
 	github.com/google/uuid v1.6.0 // indirect
 	github.com/inconshreveable/mousetrap v1.1.0 // indirect
+	github.com/jackc/pgpassfile v1.0.0 // indirect
+	github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
+	github.com/jackc/puddle/v2 v2.2.2 // indirect
 	github.com/knadh/koanf/maps v0.1.2 // indirect
 	github.com/mattn/go-isatty v0.0.24 // indirect
 	github.com/mitchellh/copystructure v1.2.0 // indirect
@@ -37,7 +42,6 @@ require (
 	github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
 	github.com/spf13/pflag v1.0.9 // indirect
 	go.yaml.in/yaml/v3 v3.0.5 // indirect
-	golang.org/x/crypto v0.57.0 // indirect
 	golang.org/x/sys v0.48.0 // indirect
 	golang.org/x/text v0.42.0 // indirect
 	modernc.org/libc v1.75.7 // indirect

go.sum

@@ -7,6 +7,7 @@ github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6r
 github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
 github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
 github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
+github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
 github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
 github.com/fsnotify/fsnotify v1.9.0 h1:2Ml+OJNzbYCTzsxtv8vKSFD9PbJjmhYF14k/jKC7S9k=
@@ -23,6 +24,14 @@ github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs
 github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
 github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
 github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
+github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM=
+github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg=
+github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 h1:iCEnooe7UlwOQYpKFhBabPMi4aNAfoODPEFNiAnClxo=
+github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761/go.mod h1:5TJZWKEWniPve33vlWYSoGYefn3gLQRzjfDlhSJ9ZKM=
+github.com/jackc/pgx/v5 v5.11.0 h1:IzBBtyK9AHqf98cctWFifYSci2hgQR/cd56wB4p+ogg=
+github.com/jackc/pgx/v5 v5.11.0/go.mod h1:mal1tBGAFfLHvZzaYh77YS/eC6IX9OWbRV1QIIM0Jn4=
+github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
+github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
 github.com/klauspost/compress v1.19.1 h1:VsB4HPswih7mmZ8WleSFQ75c/Ui1M4trX5oAsJnhSlk=
 github.com/klauspost/compress v1.19.1/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
 github.com/knadh/koanf/maps v0.1.2 h1:RBfmAW5CnZT+PJ1CVc1QSJKf4Xu9kxfQgYVQSu8hpbo=
@@ -47,6 +56,7 @@ github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq
 github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
 github.com/ncruces/go-strftime v1.0.0 h1:HMFp8mLCTPp341M/ZnA4qaf7ZlsbTc+miZjCLOFAw7w=
 github.com/ncruces/go-strftime v1.0.0/go.mod h1:Fwc5htZGVVkseilnfgOVb9mKy6w1naJmn9CehxcKcls=
+github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
 github.com/prometheus/client_golang v1.24.1 h1:JnJkREXzWxUdCuPFpIWZiPispT9xVV59uiuyR2bPlnU=
 github.com/prometheus/client_golang v1.24.1/go.mod h1:F+oSRECHg4sse5ucfYpYDeIv/hu68Zo0uoHKetWnzcE=
 github.com/prometheus/client_model v0.6.3 h1:O0jaTVAYNxTHYInEPFJt5I3+sN8zqBtVMPTB1qyxiEo=
@@ -62,6 +72,9 @@ github.com/spf13/cobra v1.10.2 h1:DMTTonx5m65Ic0GOoRY2c16WCbHxOOw6xxezuLaBpcU=
 github.com/spf13/cobra v1.10.2/go.mod h1:7C1pvHqHw5A4vrJfjNwvOdzYu0Gml16OCs2GRiTUUS4=
 github.com/spf13/pflag v1.0.9 h1:9exaQaMOCwffKiiiYk6/BndUBv+iRViNW+4lEMi0PvY=
 github.com/spf13/pflag v1.0.9/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
+github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
+github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
+github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
 github.com/stretchr/testify v1.12.1 h1:EuwCh5fleGS7H32xRwO3wRGT7DxrDhLAT6FF8MpWDWE=
 github.com/stretchr/testify v1.12.1/go.mod h1:MDEgiDPPsNp5cuIrHPPCyornHKgEVbtFUmoNlxoYthg=
 go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
@@ -88,6 +101,7 @@ golang.org/x/tools v0.49.0/go.mod h1:SJNXV9DBKT0UbdttsQjbfJlAE/q+y36++zo3uL3N0Oo
 google.golang.org/protobuf v1.36.12 h1:pJOKDDOyeXErUroCihFAd5LQuwXBSpVnKGrj5o/fwxc=
 google.golang.org/protobuf v1.36.12/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
 modernc.org/cc/v4 v4.29.2 h1:h6+9ciCnPKutf4I03CvheAvDLX7+IHlqR6Iy6J+cgd8=
 modernc.org/cc/v4 v4.29.2/go.mod h1:OnovgIhbbMXMu1aISnJ0wvVD1KnW+cAUJkIrAWh+kVI=
 modernc.org/ccgo/v4 v4.35.0 h1:F+TUsmw09QxLzmi3aeYYGxjAXarmZaKgj3mKQHNaA8w=

internal/account/account_test.go

@@ -12,7 +12,7 @@ import (
 	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
 )
 
-func newStore(t *testing.T) *store.SQLite {
+func newStore(t *testing.T) *store.DB {
 	t.Helper()
 	s, err := store.OpenSQLite(context.Background(), filepath.Join(t.TempDir(), "a.db"))
 	if err != nil {
@@ -33,7 +33,7 @@ func newStore(t *testing.T) *store.SQLite {
 	return s
 }
 
-func newService(t *testing.T) (*Service, *store.SQLite) {
+func newService(t *testing.T) (*Service, *store.DB) {
 	t.Helper()
 	s := newStore(t)
 	return New(s, DefaultConfig(), slog.New(slog.DiscardHandler)), s
@@ -113,7 +113,7 @@ func TestApprovalNeedsAProject(t *testing.T) {
 	}
 }
 
-func newStoreAndUser(t *testing.T) (*Service, *store.SQLite) {
+func newStoreAndUser(t *testing.T) (*Service, *store.DB) {
 	t.Helper()
 	svc, st := newService(t)
 	hash, _ := HashPassword(goodPassword)

internal/accounting/accounting.go

@@ -70,19 +70,54 @@ func (a *Accountant) RestoreBudgets(ctx context.Context, s store.Store, period s
 	if start.IsZero() {
 		return nil
 	}
-	spend, err := s.SpendSince(ctx, start)
+	rows, err := s.SpendSinceByScope(ctx, start)
 	if err != nil {
 		return fmt.Errorf("restoring budgets: %w", err)
 	}
+	// Every scope a key sits under, not just the key. A budget owned by a team
+	// is one counter fed by every key beneath it, and restoring only per-key
+	// totals would bring the control plane back up believing the team had spent
+	// nothing — which is a budget that resets on every restart, for exactly the
+	// scopes where it matters most.
+	spend := make(map[string]int64, len(rows)*4)
+	for _, r := range rows {
+		for _, o := range []store.LimitOwner{
+			{Scope: store.ScopeKey, ID: r.KeyID},
+			{Scope: store.ScopeProject, ID: r.ProjectID},
+			{Scope: store.ScopeTeam, ID: r.TeamID},
+			{Scope: store.ScopeOrg, ID: r.OrgID},
+		} {
+			if k := o.CounterKey(); k != "" {
+				spend[k] += r.Micros
+			}
+		}
+	}
 	a.budgets.Restore(period, spend)
-	a.log.Info("restored budgets from the store", "keys", len(spend), "since", start)
+	a.log.Info("restored budgets from the store",
+		"keys", len(rows), "counters", len(spend), "since", start)
 	return nil
 }
 
 // Reservation is what Check returns: the estimate it charged, so Record can
 // settle the difference against the real figure.
+// pool is one level's counter and the period its budget runs over.
+type pool struct {
+	Counter string
+	Period  store.Period
+}
+
 type Reservation struct {
-	KeyID           string
+	KeyID string
+	// Budgets and Tokens are every counter the check went through, carried so
+	// the settlement lands on the same ones. Every, not the tightest: an
+	// organization's pool only fills up if each team's spend is added to it
+	// as well as to the team's own.
+	//
+	// Derived again at settlement would mean re-resolving the limits, and a
+	// budget checked against the team and settled against the key is a budget
+	// that never fills up.
+	Budgets         []pool
+	Tokens          []string
 	EstimatedTokens int64
 	Price           Price
 	// Period is the key's budget period, carried from the check so the record
@@ -99,14 +134,28 @@ type Reservation struct {
 // figure only exists once the request has run; Record settles the difference.
 func (a *Accountant) Check(p auth.Principal, model string, estimatedTokens int64) (Reservation, error) {
 	price := a.pricing.For(model)
+	lim := p.Limits
+	// Every level that set a limit, each against its own pool. A quota written
+	// on a team is one pool the team shares, and the organization above it is a
+	// second pool that the team's spending also fills — both have to hold, or
+	// an organization given 1000 with four teams given 500 spends 2000.
+	chain := levels(lim, p.KeyID)
+
 	res := Reservation{
 		KeyID: p.KeyID, EstimatedTokens: estimatedTokens,
-		Price: price, Period: p.Limits.BudgetPeriod,
+		Price: price, Period: lim.BudgetPeriod,
+	}
+	for _, l := range chain {
+		if l.Budget != nil {
+			res.Budgets = append(res.Budgets, pool{l.Counter, l.Period})
+		}
+		if l.TPM != nil {
+			res.Tokens = append(res.Tokens, l.Counter)
+		}
 	}
 	if p.KeyID == "" {
 		return res, nil
 	}
-	lim := p.Limits
 
 	// Budget first. It is the limit whose refusal cannot be retried away, so
 	// telling the client about it first is more useful than reporting a rate
@@ -114,15 +163,29 @@ func (a *Accountant) Check(p auth.Principal, model string, estimatedTokens int64
 	//
 	// A model that costs nothing is exempt, which is what lets a key with no
 	// budget left still be served a cheap fallback (plan §7.7).
-	if lim.Budget != nil && !price.Zero() &&
-		a.budgets.Exhausted(p.KeyID, lim.BudgetPeriod, *lim.Budget) {
-		return res, &LimitError{Err: ErrBudgetSpent}
+	// Budgets first, and all of them. Exhausted only reads, so the order costs
+	// nothing and a refusal here charges no counter.
+	if !price.Zero() {
+		for _, l := range chain {
+			if l.Budget != nil && a.budgets.Exhausted(l.Counter, l.Period, *l.Budget) {
+				return res, &LimitError{Err: ErrBudgetSpent}
+			}
+		}
 	}
 
-	if lim.RPM != nil && !a.requests.Allow(p.KeyID, *lim.RPM, 1) {
+	// Rate limits charge as they check — deliberately, so two concurrent
+	// requests cannot both see room. With several levels that means a refusal
+	// at one has already taken from the levels before it, so those are handed
+	// back. Without the refund a team at its ceiling would quietly eat its
+	// organization's rate limit with requests that were never served.
+	if err := a.allowAll(chain, func(l level) (string, int, int64, bool) {
+		return l.Counter, valInt(l.RPM), 1, l.RPM != nil
+	}, a.requests); err != nil {
 		return res, &LimitError{Err: ErrRateLimited, RetryAfter: time.Second}
 	}
-	if lim.TPM != nil && !a.tokens.Allow(p.KeyID, *lim.TPM, estimatedTokens) {
+	if err := a.allowAll(chain, func(l level) (string, int, int64, bool) {
+		return l.Counter, valInt(l.TPM), estimatedTokens, l.TPM != nil
+	}, a.tokens); err != nil {
 		// The request was not charged against the request-rate limit in vain:
 		// it was, and that is correct — it did arrive.
 		return res, &LimitError{Err: ErrTokenLimited, RetryAfter: time.Second}
@@ -130,6 +193,88 @@ func (a *Accountant) Check(p auth.Principal, model string, estimatedTokens int64
 	return res, nil
 }
 
+// level is one scope's limits with the counter they are enforced against.
+type level struct {
+	Counter string
+	Period  store.Period
+	RPM     *int
+	TPM     *int
+	Budget  *int64
+}
+
+// levels turns a resolved chain into the pools a request must pass.
+//
+// Falls back to the collapsed values when the chain is empty, which is what a
+// principal assembled by hand looks like — in tests, and anywhere that builds
+// limits without walking the tree.
+func levels(e store.Effective, keyID string) []level {
+	if len(e.Chain) == 0 {
+		var out []level
+		if e.RPM != nil || e.TPM != nil || e.Budget != nil {
+			out = append(out, level{
+				Counter: counterFor(e.BudgetOwner, keyID),
+				Period:  e.BudgetPeriod, Budget: e.Budget,
+			})
+			out[0].RPM, out[0].TPM = e.RPM, e.TPM
+			// One entry, so a hand-built principal with different owners per
+			// limit still counts each where it said to.
+			if r, b := counterFor(e.RPMOwner, keyID), out[0].Counter; r != b && e.RPM != nil {
+				out[0].RPM = nil
+				out = append(out, level{Counter: r, RPM: e.RPM})
+			}
+			if tp, b := counterFor(e.TPMOwner, keyID), out[0].Counter; tp != b && e.TPM != nil {
+				out[0].TPM = nil
+				out = append(out, level{Counter: tp, TPM: e.TPM})
+			}
+		}
+		return out
+	}
+	out := make([]level, 0, len(e.Chain))
+	for _, l := range e.Chain {
+		out = append(out, level{
+			Counter: store.LimitOwner{Scope: l.Scope, ID: l.ScopeID}.CounterKey(),
+			Period:  l.BudgetPeriod, RPM: l.RPM, TPM: l.TPM, Budget: l.Budget,
+		})
+	}
+	return out
+}
+
+// allowAll charges every level that has this limit, refunding on a refusal.
+func (a *Accountant) allowAll(chain []level,
+	pick func(level) (key string, limit int, n int64, has bool), lim *Limiter,
+) error {
+	var charged []struct {
+		key string
+		n   int64
+	}
+	for _, l := range chain {
+		key, limit, n, has := pick(l)
+		if !has {
+			continue
+		}
+		if !lim.Allow(key, limit, n) {
+			for _, c := range charged {
+				lim.Charge(c.key, -c.n)
+			}
+			return errRefused
+		}
+		charged = append(charged, struct {
+			key string
+			n   int64
+		}{key, n})
+	}
+	return nil
+}
+
+var errRefused = errors.New("refused")
+
+func valInt(v *int) int {
+	if v == nil {
+		return 0
+	}
+	return *v
+}
+
 // Record settles a finished request: it reconciles the token estimate, adds to
 // the budget, and queues the usage row.
 func (a *Accountant) Record(res Reservation, u store.Usage) {
@@ -141,10 +286,16 @@ func (a *Accountant) Record(res Reservation, u store.Usage) {
 	// The estimate was charged up front; charge the difference now. A caller
 	// whose requests are consistently larger than estimated is still bounded,
 	// one window later than with a perfect estimate.
-	a.tokens.Charge(res.KeyID, actual-res.EstimatedTokens)
+	// Every level the check went through, so an organization's pool fills up
+	// from its teams' spending rather than only from requests that named it.
+	for _, c := range res.Tokens {
+		a.tokens.Charge(c, actual-res.EstimatedTokens)
+	}
 
 	u.CostMicros = res.Price.Cost(u.PromptTokens, u.CompletionTokens)
-	a.budgets.Add(res.KeyID, res.Period, u.CostMicros)
+	for _, b := range res.Budgets {
+		a.budgets.Add(b.Counter, b.Period, u.CostMicros)
+	}
 
 	if a.recorder != nil {
 		a.recorder.Record(u)
@@ -156,6 +307,65 @@ func (a *Accountant) Status(p auth.Principal) (spent int64, rpm, tpm float64) {
 	if p.KeyID == "" {
 		return 0, 0, 0
 	}
-	return a.budgets.Spent(p.KeyID, p.Limits.BudgetPeriod),
-		a.requests.Rate(p.KeyID), a.tokens.Rate(p.KeyID)
+	// The pool that will refuse them, which is not always the one with the
+	// smallest number on it. A team given 2000 inside an organization given the
+	// same is bounded by whichever is nearer its ceiling, and that is the
+	// organization as soon as a second team spends anything. A header that
+	// reported the team's would disagree with the refusal that follows it.
+	for _, l := range levels(p.Limits, p.KeyID) {
+		if l.Budget != nil {
+			if v := a.budgets.Spent(l.Counter, l.Period); v > spent {
+				spent = v
+			}
+		}
+		if l.RPM != nil {
+			if v := a.requests.Rate(l.Counter); v > rpm {
+				rpm = v
+			}
+		}
+		if l.TPM != nil {
+			if v := a.tokens.Rate(l.Counter); v > tpm {
+				tpm = v
+			}
+		}
+	}
+	return spent, rpm, tpm
 }
+
+// counterFor is the counter a limit owned by this scope should use.
+//
+// Falls back to the key's own counter when no scope owns the limit, which
+// happens when nothing in the chain set one.
+//
+// Namespaced even then. A bare key id here and a "key:"-prefixed one in the
+// restore path would be two different counters for the same key, so a budget
+// would check against an empty one and survive every restart — which is how
+// this was wrong when it was first written.
+func counterFor(o store.LimitOwner, keyID string) string {
+	if k := o.CounterKey(); k != "" {
+		return k
+	}
+	return keyCounter(keyID)
+}
+
+// keyCounter is a key's own counter, in the same namespace as every other.
+func or(v, fallback string) string {
+	if v == "" {
+		return fallback
+	}
+	return v
+}
+
+func keyCounter(keyID string) string {
+	if keyID == "" {
+		return ""
+	}
+	return store.LimitOwner{Scope: store.ScopeKey, ID: keyID}.CounterKey()
+}
+
+// AnythingPriced reports whether a budget can ever be spent against.
+//
+// On the accountant rather than making callers reach for the price table: what
+// they are asking is whether the budgets they are about to display mean
+// anything, and the accountant is what would enforce them.
+func (a *Accountant) AnythingPriced() bool { return a.pricing.AnythingPriced() }

internal/accounting/accounting_test.go

@@ -288,3 +288,345 @@ func must(t *testing.T, err error) {
 		t.Fatal(err)
 	}
 }
+
+// teamMember is a key whose budget and rate limit were set on its team, which
+// is the arrangement an enterprise actually asks for: a group gets a quota and
+// the people in it share it.
+func teamMember(keyID, teamID string, tune func(*store.Effective)) auth.Principal {
+	e := store.Effective{KeyID: keyID, ProjectID: "proj", TeamID: teamID}
+	owner := store.LimitOwner{Scope: store.ScopeTeam, ID: teamID}
+	e.BudgetOwner, e.RPMOwner, e.TPMOwner = owner, owner, owner
+	if tune != nil {
+		tune(&e)
+	}
+	return auth.Principal{KeyID: keyID, ProjectID: "proj", Limits: e}
+}
+
+func TestATeamBudgetIsOnePoolRatherThanOnePerMember(t *testing.T) {
+	a := New(pricing(t), nil, slog.New(slog.DiscardHandler))
+	budget := func(e *store.Effective) {
+		e.Budget = i64p(3000)
+		e.BudgetPeriod = store.PeriodMonth
+	}
+	first := teamMember("k1", "t1", budget)
+	second := teamMember("k2", "t1", budget)
+
+	// One member spends the lot.
+	res, err := a.Check(first, "paid", 1000)
+	if err != nil {
+		t.Fatal(err)
+	}
+	a.Record(res, store.Usage{KeyID: "k1", PromptTokens: 1000, CompletionTokens: 1000})
+
+	// The second must now be refused. Counted per key it would have its own
+	// untouched 3000 — a team budget of 3000 with ten keys would permit thirty
+	// thousand, which is the bug this change exists for.
+	if _, err := a.Check(second, "paid", 10); !errors.Is(err, ErrBudgetSpent) {
+		t.Errorf("the team's budget should be shared; the second member got %v", err)
+	}
+}
+
+func TestATeamRateLimitIsSharedToo(t *testing.T) {
+	a := New(pricing(t), nil, slog.New(slog.DiscardHandler))
+	rpm := func(e *store.Effective) { e.RPM = intp(1) }
+
+	if _, err := a.Check(teamMember("k1", "t1", rpm), "free", 1); err != nil {
+		t.Fatal(err)
+	}
+	if _, err := a.Check(teamMember("k2", "t1", rpm), "free", 1); !errors.Is(err, ErrRateLimited) {
+		t.Error("a rate limit set on the team should be shared by its keys")
+	}
+}
+
+func TestTwoTeamsDoNotShareACounter(t *testing.T) {
+	a := New(pricing(t), nil, slog.New(slog.DiscardHandler))
+	rpm := func(e *store.Effective) { e.RPM = intp(1) }
+
+	if _, err := a.Check(teamMember("k1", "t1", rpm), "free", 1); err != nil {
+		t.Fatal(err)
+	}
+	// Pooling is the point; pooling the wrong things is the danger. One team
+	// exhausting another's quota would be worse than the bug being fixed.
+	if _, err := a.Check(teamMember("k2", "t2", rpm), "free", 1); err != nil {
+		t.Errorf("a second team should have its own counter, got %v", err)
+	}
+}
+
+func TestAKeyLimitStaysPerKey(t *testing.T) {
+	a := New(pricing(t), nil, slog.New(slog.DiscardHandler))
+	perKey := func(id string) auth.Principal {
+		return principal(id, func(e *store.Effective) {
+			e.RPM = intp(1)
+			e.RPMOwner = store.LimitOwner{Scope: store.ScopeKey, ID: id}
+		})
+	}
+
+	if _, err := a.Check(perKey("k1"), "free", 1); err != nil {
+		t.Fatal(err)
+	}
+	// The tree still has to be able to say "each of you may do this much".
+	// Sharing a group pool must not become the only thing expressible.
+	if _, err := a.Check(perKey("k2"), "free", 1); err != nil {
+		t.Errorf("a limit owned by the key should not be shared, got %v", err)
+	}
+}
+
+func TestTheDifferentLimitsCanBeOwnedByDifferentScopes(t *testing.T) {
+	a := New(pricing(t), nil, slog.New(slog.DiscardHandler))
+	// A team budget with a per-key rate limit is the ordinary arrangement: the
+	// group's money, the individual's throughput.
+	mk := func(id string) auth.Principal {
+		e := store.Effective{KeyID: id, TeamID: "t1"}
+		e.Budget, e.BudgetPeriod = i64p(3000), store.PeriodMonth
+		e.BudgetOwner = store.LimitOwner{Scope: store.ScopeTeam, ID: "t1"}
+		e.RPM = intp(1)
+		e.RPMOwner = store.LimitOwner{Scope: store.ScopeKey, ID: id}
+		return auth.Principal{KeyID: id, Limits: e}
+	}
+
+	res, err := a.Check(mk("k1"), "paid", 1000)
+	if err != nil {
+		t.Fatal(err)
+	}
+	a.Record(res, store.Usage{KeyID: "k1", PromptTokens: 1000, CompletionTokens: 1000})
+
+	// k2's rate limit is its own and untouched; the budget is the team's and
+	// gone. The budget is checked first, so that is what comes back.
+	if _, err := a.Check(mk("k2"), "paid", 10); !errors.Is(err, ErrBudgetSpent) {
+		t.Errorf("expected the shared budget to bite, got %v", err)
+	}
+}
+
+func TestStatusReportsTheCounterThatWillRefuseYou(t *testing.T) {
+	a := New(pricing(t), nil, slog.New(slog.DiscardHandler))
+	budget := func(e *store.Effective) {
+		e.Budget = i64p(9000)
+		e.BudgetPeriod = store.PeriodMonth
+	}
+	res, err := a.Check(teamMember("k1", "t1", budget), "paid", 1000)
+	if err != nil {
+		t.Fatal(err)
+	}
+	a.Record(res, store.Usage{KeyID: "k1", PromptTokens: 1000, CompletionTokens: 1000})
+
+	// A member who has spent nothing themselves still sees the team's spend,
+	// because that is the number that will stop them. A header disagreeing with
+	// the refusal it precedes is worse than no header.
+	spent, _, _ := a.Status(teamMember("k2", "t1", budget))
+	if spent == 0 {
+		t.Error("a member should see the pool's spend, not their own")
+	}
+}
+
+func TestATeamBudgetSurvivesARestart(t *testing.T) {
+	// The per-key restore could not do this: it rebuilt a counter named after
+	// the key, while the check reads one named after the team. The control
+	// plane came back believing the team had spent nothing — a budget that
+	// resets on every restart, for exactly the scope where it matters most.
+	ctx := context.Background()
+	s, err := store.OpenSQLite(ctx, filepath.Join(t.TempDir(), "team.db"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer s.Close()
+
+	must(t, s.PutOrg(ctx, store.Org{ID: "o", Name: "o"}))
+	must(t, s.PutTeam(ctx, store.Team{ID: "t1", OrgID: "o", Name: "t"}))
+	must(t, s.PutProject(ctx, store.Project{ID: "proj", TeamID: "t1", Name: "p"}))
+	for _, id := range []string{"k1", "k2"} {
+		_, k, _ := store.NewKey(id, "n", "proj")
+		k.CreatedAt = time.Now()
+		must(t, s.CreateKey(ctx, k))
+	}
+
+	// Two members of one team, 2500 each: neither alone exceeds a 4000 budget,
+	// together they do.
+	must(t, s.WriteUsage(ctx, []store.Usage{
+		{At: time.Now(), KeyID: "k1", ProjectID: "proj", Model: "paid",
+			ServedModel: "paid", CostMicros: 2500, Status: 200},
+		{At: time.Now(), KeyID: "k2", ProjectID: "proj", Model: "paid",
+			ServedModel: "paid", CostMicros: 2500, Status: 200},
+	}))
+
+	a := New(pricing(t), nil, slog.New(slog.DiscardHandler))
+	must(t, a.RestoreBudgets(ctx, s, store.PeriodMonth))
+
+	p := teamMember("k1", "t1", func(e *store.Effective) {
+		e.Budget = i64p(4000)
+		e.BudgetPeriod = store.PeriodMonth
+	})
+	if _, err := a.Check(p, "paid", 10); !errors.Is(err, ErrBudgetSpent) {
+		t.Errorf("the team's spend should have been restored and pooled: %v", err)
+	}
+}
+
+// inTree is a key under org/team/project, with whatever each level sets.
+func inTree(keyID string, chain ...store.Limits) auth.Principal {
+	e := store.Effective{KeyID: keyID, Chain: chain}
+	// The collapsed values still exist for headers and the profile page; they
+	// are not what enforcement reads any more.
+	for _, l := range chain {
+		if l.Budget != nil && (e.Budget == nil || *l.Budget < *e.Budget) {
+			e.Budget, e.BudgetPeriod = l.Budget, l.BudgetPeriod
+			e.BudgetOwner = store.LimitOwner{Scope: l.Scope, ID: l.ScopeID}
+		}
+	}
+	return auth.Principal{KeyID: keyID, Limits: e}
+}
+
+func orgBudget(id string, v int64) store.Limits {
+	return store.Limits{Scope: store.ScopeOrg, ScopeID: id,
+		Budget: i64p(v), BudgetPeriod: store.PeriodMonth}
+}
+
+func teamBudget(id string, v int64) store.Limits {
+	return store.Limits{Scope: store.ScopeTeam, ScopeID: id,
+		Budget: i64p(v), BudgetPeriod: store.PeriodMonth}
+}
+
+func TestTheSumOfTeamsCannotExceedTheOrganization(t *testing.T) {
+	a := New(pricing(t), nil, slog.New(slog.DiscardHandler))
+
+	// Each request costs 800. The organization allows 2000 and each of its four
+	// teams allows 2000, so no team comes close to its own ceiling — but the
+	// organization's fills after three. Taking only the most restrictive value,
+	// the organization's number was never looked at and all four went through.
+	//
+	// A budget stops spending once it is reached rather than refusing the
+	// request that would cross it, so the third is served and the fourth is not.
+	spend := func(team string) error {
+		p := inTree(team+"-k", orgBudget("o", 2000), teamBudget(team, 2000))
+		res, err := a.Check(p, "paid", 1000)
+		if err != nil {
+			return err
+		}
+		a.Record(res, store.Usage{KeyID: team + "-k", PromptTokens: 200, CompletionTokens: 200})
+		return nil
+	}
+
+	total := 0
+	for _, team := range []string{"t1", "t2", "t3", "t4"} {
+		if err := spend(team); err == nil {
+			total++
+		}
+	}
+	if total != 3 {
+		t.Errorf("expected the organization to stop the fourth team, %d got through", total)
+	}
+}
+
+func TestATeamStillStopsBeforeItsOrganizationDoes(t *testing.T) {
+	a := New(pricing(t), nil, slog.New(slog.DiscardHandler))
+	p := func() auth.Principal {
+		return inTree("k1", orgBudget("o", 100_000), teamBudget("t1", 800))
+	}
+
+	res, err := a.Check(p(), "paid", 1000)
+	if err != nil {
+		t.Fatal(err)
+	}
+	a.Record(res, store.Usage{KeyID: "k1", PromptTokens: 200, CompletionTokens: 200})
+
+	// The narrower pool binds first, which is what it is for. Enforcing every
+	// level must not stop the tighter one working.
+	if _, err := a.Check(p(), "paid", 10); !errors.Is(err, ErrBudgetSpent) {
+		t.Errorf("the team's own budget should still bite, got %v", err)
+	}
+}
+
+func TestSpendingFillsEveryPoolAboveIt(t *testing.T) {
+	a := New(pricing(t), nil, slog.New(slog.DiscardHandler))
+	p := inTree("k1", orgBudget("o", 5000), teamBudget("t1", 5000))
+
+	res, err := a.Check(p, "paid", 1000)
+	if err != nil {
+		t.Fatal(err)
+	}
+	a.Record(res, store.Usage{KeyID: "k1", PromptTokens: 200, CompletionTokens: 200})
+
+	// Settled onto both, or the organization's pool would only ever fill from
+	// requests that named it directly — which is none of them.
+	if got := a.budgets.Spent("org:o", store.PeriodMonth); got == 0 {
+		t.Error("the organization's pool did not fill")
+	}
+	if got := a.budgets.Spent("team:t1", store.PeriodMonth); got == 0 {
+		t.Error("the team's pool did not fill")
+	}
+}
+
+func TestARefusalHandsBackWhatTheNarrowerLevelsTook(t *testing.T) {
+	a := New(pricing(t), nil, slog.New(slog.DiscardHandler))
+	// The team allows plenty; the organization allows one request a minute.
+	chain := []store.Limits{
+		{Scope: store.ScopeOrg, ScopeID: "o", RPM: intp(1)},
+		{Scope: store.ScopeTeam, ScopeID: "t1", RPM: intp(100)},
+	}
+	if _, err := a.Check(inTree("k1", chain...), "free", 1); err != nil {
+		t.Fatal(err)
+	}
+	before := a.requests.Rate("team:t1")
+
+	if _, err := a.Check(inTree("k2", chain...), "free", 1); !errors.Is(err, ErrRateLimited) {
+		t.Fatalf("the organization should have refused the second, got %v", err)
+	}
+
+	// A rate limiter charges as it checks, so a refusal further along has
+	// already taken from the levels before it. Left there, a team at its
+	// organization's ceiling would quietly eat its own rate limit with requests
+	// that were never served.
+	if after := a.requests.Rate("team:t1"); after != before {
+		t.Errorf("the refused request stayed charged to the team: %v -> %v", before, after)
+	}
+}
+
+func TestAChainWithNoLimitsAnywhereAllowsEverything(t *testing.T) {
+	a := New(pricing(t), nil, slog.New(slog.DiscardHandler))
+	for i := 0; i < 5; i++ {
+		if _, err := a.Check(inTree("k1"), "paid", 1000); err != nil {
+			t.Fatalf("nothing is limited, but request %d was refused: %v", i, err)
+		}
+	}
+}
+
+func TestStatusReportsTheFullestPoolNotTheSmallestNumber(t *testing.T) {
+	a := New(pricing(t), nil, slog.New(slog.DiscardHandler))
+	chain := func(team string) auth.Principal {
+		return inTree(team+"-k", orgBudget("o", 9000), teamBudget(team, 9000))
+	}
+
+	// Another team spends; this one has spent nothing of its own.
+	res, err := a.Check(chain("t1"), "paid", 1000)
+	if err != nil {
+		t.Fatal(err)
+	}
+	a.Record(res, store.Usage{KeyID: "t1-k", PromptTokens: 200, CompletionTokens: 200})
+
+	spent, _, _ := a.Status(chain("t2"))
+	// The organization's pool is the one nearer its ceiling, so it is the one
+	// that will refuse them. Reporting the team's nil would disagree with the
+	// refusal that eventually follows it.
+	if spent == 0 {
+		t.Error("a member should see the fullest pool that bounds them")
+	}
+}
+
+func TestAnEmptyPriceTableMeansNoBudgetCanEverBind(t *testing.T) {
+	// The ordinary case for self-hosted models, and the reason a budget on a
+	// team can read as a limit while binding on nothing.
+	if NewPricing(nil, Price{}).AnythingPriced() {
+		t.Error("nothing priced should say so")
+	}
+	if NewPricing(map[string]Price{"m": {}}, Price{}).AnythingPriced() {
+		t.Error("a model priced at zero is not priced")
+	}
+	if !NewPricing(map[string]Price{"m": {PromptPer1k: 1}}, Price{}).AnythingPriced() {
+		t.Error("one priced model is enough for a budget to move")
+	}
+	if !NewPricing(nil, Price{CompletionPer1k: 1}).AnythingPriced() {
+		t.Error("a fallback price prices everything")
+	}
+	var nilPricing *Pricing
+	if nilPricing.AnythingPriced() {
+		t.Error("no price table at all is not priced")
+	}
+}

internal/accounting/cost.go

@@ -68,3 +68,30 @@ func (p *Pricing) For(model string) Price {
 	}
 	return p.fallback
 }
+
+// AnythingPriced reports whether any model costs anything.
+//
+// A budget is spend per period, and spend is computed from a price table. With
+// no prices every request costs nothing, so a budget written on a team is a
+// number that can never move — displayed as a limit, reported as zero percent
+// used, and binding on nothing.
+//
+// Self-hosted models have no per-token price, so an empty table is the ordinary
+// case rather than a misconfiguration. What is not ordinary is showing somebody
+// a budget without saying so.
+func (p *Pricing) AnythingPriced() bool {
+	if p == nil {
+		return false
+	}
+	p.mu.RLock()
+	defer p.mu.RUnlock()
+	if !p.fallback.Zero() {
+		return true
+	}
+	for _, price := range p.byModel {
+		if !price.Zero() {
+			return true
+		}
+	}
+	return false
+}

internal/agent/agent.go

@@ -3,14 +3,21 @@ package agent
 
 import (
 	"context"
+	"crypto/sha256"
 	"crypto/tls"
+	"encoding/hex"
 	"errors"
 	"fmt"
 	"io"
 	"log/slog"
+	"maps"
 	"math/rand/v2"
 	"net"
 	"net/http"
+	"net/url"
+	"os"
+	"path/filepath"
+	"slices"
 	"strings"
 	"sync"
 	"time"
@@ -21,6 +28,7 @@ import (
 	"gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1/llmgwv1connect"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/supervisor"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/weights"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/agentsvc"
@@ -30,13 +38,33 @@ import (
 
 // Config configures the agent.
 type Config struct {
-	NodeID   string            `koanf:"node_id"`
-	Server   string            `koanf:"server"`
-	Hostname string            `koanf:"hostname"`
-	MgmtAddr string            `koanf:"mgmt_addr"`
-	Labels   map[string]string `koanf:"labels"`
+	// NodeID defaults to the hostname. It must equal the common name of the
+	// agent's certificate, so set it only where the two differ.
+	NodeID string `koanf:"node_id"`
+	// Server is the control plane, and Servers is the same thing when there is
+	// more than one. Both are accepted: every configuration written before
+	// there could be a second control plane names one address, and none of
+	// them should have to change.
+	//
+	// The order is tried in order, and a refusal moves to the next. Only the
+	// leader accepts an agent, so a standby is skipped in the time it takes to
+	// be told so — which is what makes failover a reconnect rather than an
+	// edit to every node's configuration.
+	Server  string   `koanf:"server"`
+	Servers []string `koanf:"servers"`
+	// Hostname defaults to the machine's.
+	Hostname string `koanf:"hostname"`
+	// MgmtAddr is the address engines are reached on. It defaults to this
+	// machine's address on the route to the control plane, which is the one
+	// the control plane and the router can reach.
+	MgmtAddr string `koanf:"mgmt_addr"`
+	// Labels are free-form facts about a node for a profile's node_selector.
+	// Nothing needs them: memory, GPUs, engines and disk are measured, and a
+	// label restating one of those goes stale the day the machine changes.
+	Labels map[string]string `koanf:"labels"`
 
-	// MgmtInterface is the ethernet interface NCCL bootstraps over.
+	// MgmtInterface is the ethernet interface NCCL bootstraps over. It
+	// defaults to the interface holding MgmtAddr, then eth0.
 	MgmtInterface string `koanf:"mgmt_interface"`
 	// Fabrics labels this node's HCAs. It is explicit on purpose: OpenSM's
 	// default subnet prefix is the same on every fabric, so nothing can be
@@ -59,19 +87,22 @@ type Config struct {
 	// keeps the plaintext behaviour, which only belongs on a trusted network.
 	TLS tlsconf.Config `koanf:"tls"`
 
-	// ModelDirs are the directories this node keeps model weights in.
-	//
-	// Empty by default, and empty means the node is not asked to look — which
-	// is not the same as having nothing. Storage differs at every installation,
+	// ModelDirs are the directories this node keeps model weights in, beyond
+	// WeightsCache, which is always looked in. Storage differs at every installation,
 	// so llmgw observes it rather than owning it: a Lustre mount, an image
 	// exported over the fabric, local NVMe and a copy made by hand all look
 	// identical from here, and placement only needs to know whether the file
 	// is present.
 	ModelDirs []string `koanf:"model_dirs"`
 
+	// EnginesDir is where the agent installs engines, one directory per
+	// build. Defaults to /opt/llmgw/engines.
+	EnginesDir string `koanf:"engines_dir"`
+
 	// WeightsCache is where a remote source is downloaded to. Defaults to the
 	// first entry of ModelDirs, because a node that says where its models live
-	// has already answered this question.
+	// has already answered this question, and otherwise to
+	// /var/lib/llmgw-weights.
 	//
 	// Fetching is the simple arrangement: every node keeps its own copy. The
 	// exotic ones need no code — weights on a shared filer or an image exported
@@ -101,21 +132,47 @@ func DefaultConfig() Config {
 
 func (c *Config) normalize() error {
 	d := DefaultConfig()
-	if c.NodeID == "" {
-		return errors.New("node_id is required")
-	}
-	if c.Server == "" {
-		return errors.New("server is required")
-	}
 	if err := c.TLS.Validate(); err != nil {
 		return err
 	}
 	if err := c.normalizeServerURL(); err != nil {
 		return err
 	}
+	// Everything below is found rather than written. A file that has to say
+	// which machine it is on, and at what address, is a file that differs on
+	// every node — and a copy taken from one node to the next registers the
+	// second as the first.
+	host, _ := os.Hostname()
+	if c.NodeID == "" {
+		c.NodeID = host
+	}
+	if c.NodeID == "" {
+		return errors.New("node_id is not set and the hostname could not be read")
+	}
+	if c.Hostname == "" {
+		c.Hostname = host
+	}
+	if c.MgmtAddr == "" {
+		c.MgmtAddr = addrToward(c.Server)
+	}
+	if c.MgmtInterface == "" {
+		c.MgmtInterface = interfaceOf(c.MgmtAddr)
+	}
 	if c.MgmtInterface == "" {
 		c.MgmtInterface = d.MgmtInterface
 	}
+	// The directory downloads land in is always one that is looked in. When
+	// they were two settings, a node could fill its disk with fetched weights
+	// that placement never saw — which is what every node here was doing.
+	if c.WeightsCache == "" && len(c.ModelDirs) > 0 {
+		c.WeightsCache = c.ModelDirs[0]
+	}
+	if c.WeightsCache == "" {
+		c.WeightsCache = DefaultWeightsCache
+	}
+	if !slices.Contains(c.ModelDirs, c.WeightsCache) {
+		c.ModelDirs = append(c.ModelDirs, c.WeightsCache)
+	}
 	if c.HeartbeatInterval <= 0 {
 		c.HeartbeatInterval = d.HeartbeatInterval
 	}
@@ -136,26 +193,114 @@ func (c *Config) normalize() error {
 // that says plaintext while TLS is configured. Silently connecting anyway would
 // mean a cluster believing it has mutual authentication when it has none.
 func (c *Config) normalizeServerURL() error {
-	secure := c.TLS.Enabled()
-	switch {
-	case strings.HasPrefix(c.Server, "https://"):
-		if !secure {
-			return errors.New("server is an https:// address but no tls certificates are configured")
-		}
-	case strings.HasPrefix(c.Server, "http://"):
-		if secure {
-			return errors.New("tls is configured but server is an http:// address; use https://")
-		}
-	case strings.Contains(c.Server, "://"):
-		return fmt.Errorf("server %q must be an http:// or https:// address", c.Server)
-	case secure:
-		c.Server = "https://" + c.Server
-	default:
-		c.Server = "http://" + c.Server
+	// One list, however it was written. Server first when both are given,
+	// because an operator adding a standby to an existing file expects the
+	// one already there to stay preferred.
+	all := make([]string, 0, 1+len(c.Servers))
+	if c.Server != "" {
+		all = append(all, c.Server)
 	}
+	all = append(all, c.Servers...)
+
+	seen := map[string]struct{}{}
+	out := make([]string, 0, len(all))
+	for _, s := range all {
+		fixed, err := normalizeOne(s, c.TLS.Enabled())
+		if err != nil {
+			return err
+		}
+		// A duplicate would be tried twice in a row on failover, which reads
+		// as the rotation not working.
+		if _, dup := seen[fixed]; dup {
+			continue
+		}
+		seen[fixed] = struct{}{}
+		out = append(out, fixed)
+	}
+	if len(out) == 0 {
+		return errors.New("server is required")
+	}
+	c.Servers = out
+	c.Server = out[0]
 	return nil
 }
 
+// DefaultWeightsCache is where fetched weights go when nothing says otherwise.
+const DefaultWeightsCache = "/var/lib/llmgw-weights"
+
+// addrToward is this machine's address on the route to the control plane.
+//
+// The address engines are reached on is whichever one the control plane can
+// reach, and the kernel already knows which that is: a UDP "connection" picks
+// the source address for the route without sending a packet. Asking the
+// interfaces instead would have to guess among docker bridges, VPNs and
+// link-local addresses.
+func addrToward(server string) string {
+	u, err := url.Parse(server)
+	if err != nil || u.Host == "" {
+		return ""
+	}
+	host := u.Hostname()
+	port := u.Port()
+	if port == "" {
+		port = "443"
+	}
+	conn, err := net.Dial("udp", net.JoinHostPort(host, port))
+	if err != nil {
+		return ""
+	}
+	defer conn.Close()
+	if a, ok := conn.LocalAddr().(*net.UDPAddr); ok {
+		return a.IP.String()
+	}
+	return ""
+}
+
+// interfaceOf names the interface that holds an address, or "" when none does.
+func interfaceOf(addr string) string {
+	ip := net.ParseIP(addr)
+	if ip == nil {
+		return ""
+	}
+	ifaces, err := net.Interfaces()
+	if err != nil {
+		return ""
+	}
+	for _, ifc := range ifaces {
+		addrs, err := ifc.Addrs()
+		if err != nil {
+			continue
+		}
+		for _, a := range addrs {
+			if n, ok := a.(*net.IPNet); ok && n.IP.Equal(ip) {
+				return ifc.Name
+			}
+		}
+	}
+	return ""
+}
+
+// normalizeOne applies the scheme rules to a single address.
+func normalizeOne(addr string, secure bool) (string, error) {
+	switch {
+	case strings.HasPrefix(addr, "https://"):
+		if !secure {
+			return "", fmt.Errorf("%s is an https:// address but no tls certificates are configured", addr)
+		}
+	case strings.HasPrefix(addr, "http://"):
+		if secure {
+			return "", fmt.Errorf("tls is configured but %s is an http:// address; use https://", addr)
+		}
+	case strings.Contains(addr, "://"):
+		return "", fmt.Errorf("server %q must be an http:// or https:// address", addr)
+	case secure:
+		return "https://" + addr, nil
+	default:
+		return "http://" + addr, nil
+	}
+	return addr, nil
+}
+
 // Agent runs on a GPU node.
 type Agent struct {
 	cfg  Config
@@ -164,12 +309,53 @@ type Agent struct {
 	reg  *engines.Registry
 	http connectHTTPClient
 
+	// engineOnce probes the configured engines once before registering.
+	//
+	// Once, because probing SGLang imports it in a Python interpreter — seconds
+	// on a cold cache — and a reconnect is not a reason to pay that again.
+	// Later changes are found by the slower re-probe the heartbeat starts.
+	engineOnce sync.Once
+	// configuredVers are the engines the configuration names and that answer.
+	// Guarded by mu once the first probe has run.
+	configuredVers map[string]string
+	// installedVers are the engines this agent installed and saw run.
+	// Guarded by mu.
+	installedVers map[string]string
+
+	// prov installs the engines the control plane asks for. provWants is what
+	// it asked for, nil until it has; provStatus is where each stands and
+	// provSent the fingerprint of what the control plane last heard. All
+	// guarded by mu. provKick starts a pass without waiting for the timer.
+	prov       *provision.Installer
+	provWants  []provision.Want
+	provStatus map[string]provision.Status
+	provSent   string
+	provKick   chan struct{}
+	// enginesSent is the fingerprint of the engines the control plane last
+	// heard about, from registration or a heartbeat. Guarded by mu.
+	enginesSent string
+	// lastEngineProbe and probingEngines pace the re-probe. Guarded by mu.
+	lastEngineProbe time.Time
+	probingEngines  bool
+
 	mu  sync.Mutex
 	inv discovery.Inventory
 	// shardReplica remembers which replica a shard belongs to, so a status
 	// report can name both. The control plane keys on the replica.
 	shardReplica map[string]string
 	fetcher      *weights.Fetcher
+
+	// lastWeights is the fingerprint of the file list most recently sent, so a
+	// heartbeat carries the list only when it changed. Guarded by mu: during a
+	// failover the old stream's heartbeat loop can still be running when the
+	// new one's starts.
+	lastWeights string
+
+	// fetchWatchers are the shards waiting on each source, so progress on one
+	// download reaches every shard that asked for it. A fetch is single-flight
+	// — eight shards starting together produce one download — and all eight
+	// are waiting, so all eight should say so.
+	fetchWatchers map[string][]func(weights.Progress)
 }
 
 // connectHTTPClient is the HTTP client connect uses; an interface so tests can
@@ -195,12 +381,16 @@ func New(cfg Config, httpClient connectHTTPClient, log *slog.Logger) (*Agent, er
 	if err != nil {
 		return nil, err
 	}
-	return &Agent{
+	a := &Agent{
 		cfg: cfg, log: log, sup: sup,
 		reg:          engines.NewRegistry(cfg.Engines),
 		http:         httpClient,
 		shardReplica: map[string]string{},
-	}, nil
+		provStatus:   map[string]provision.Status{},
+		provKick:     make(chan struct{}, 1),
+	}
+	a.prov = &provision.Installer{Root: cfg.EnginesDir, Log: log, OnStatus: a.provisionStatus}
+	return a, nil
 }
 
 // Supervisor exposes the process supervisor, for tests and `ctl logs`.
@@ -219,17 +409,38 @@ func (a *Agent) Run(ctx context.Context) error {
 		a.log.Error("hardware discovery failed", "err", err)
 	}
 	a.recoverOrphans()
+	// Made now rather than by the first download, so a new node reports its
+	// disk from its first heartbeat instead of only once it holds a model.
+	if err := os.MkdirAll(a.cfg.WeightsCache, 0o755); err != nil {
+		a.log.Warn("cannot create the weights directory", "dir", a.cfg.WeightsCache, "err", err)
+	}
+	go a.provisionLoop(ctx)
 
 	attempt := 0
+	next := 0
 	for ctx.Err() == nil {
-		err := a.session(ctx)
+		server := a.cfg.Servers[next%len(a.cfg.Servers)]
+		err := a.session(ctx, server)
 		if ctx.Err() != nil {
 			break
 		}
+		// On to the next address whatever went wrong. A standby refuses, a
+		// stopped control plane refuses the connection, and a crashed one
+		// never answers — from here they are the same thing, and trying
+		// somewhere else is the answer to all three.
+		next++
 		attempt++
 		delay := a.reconnectDelay(attempt)
+		// No wait when there is somewhere else to try: the delay exists to
+		// stop hammering one control plane, and the next address has not been
+		// hammered. With a single address this is the old behaviour exactly.
+		if len(a.cfg.Servers) > 1 && next%len(a.cfg.Servers) != 0 {
+			delay = 0
+		}
 		a.log.Warn("disconnected from the control plane, retrying",
-			"err", err, "in", delay.Round(time.Millisecond), "attempt", attempt)
+			"err", err, "server", server,
+			"next", a.cfg.Servers[next%len(a.cfg.Servers)],
+			"in", delay.Round(time.Millisecond), "attempt", attempt)
 
 		select {
 		case <-ctx.Done():
@@ -392,12 +603,20 @@ func (a *Agent) weights() *weights.Fetcher {
 		CacheDir: cache,
 		Log:      a.log,
 		OnProgress: func(p weights.Progress) {
-			// Logged rather than silent: a replica that is downloading a
-			// hundred gigabytes and one that is wedged look identical from the
-			// control plane, and the difference matters to whoever is waiting.
 			a.log.Info("fetching weights",
 				"source", p.Source, "percent", p.Percent(),
 				"bytes", p.BytesDone, "total", p.BytesTotal)
+			// And upward, to whoever is watching from the control plane. The
+			// log line above is on the node, which is where nobody is looking
+			// when a replica has been starting for ten minutes.
+			a.mu.Lock()
+			watchers := slices.Clone(a.fetchWatchers[p.Source])
+			a.mu.Unlock()
+			for _, w := range watchers {
+				if w != nil {
+					w(p)
+				}
+			}
 		},
 	}
 	return a.fetcher
@@ -428,6 +647,7 @@ func (a *Agent) runningShards() []*llmgwv1.RunningShard {
 			Profile:    spec.Profile.Name,
 			Pool:       spec.Pool,
 			Fabric:     spec.Fabric,
+			ProfileFp:  spec.ProfileFP,
 			LeaderAddr: a.cfg.MgmtAddr,
 			LeaderPort: int32(spec.LeaderPort),
 		})
@@ -453,15 +673,19 @@ func (a *Agent) scan(ctx context.Context) error {
 	opts := a.cfg.Discovery
 	opts.ModelDirs = a.cfg.ModelDirs
 	inv, err := discovery.Discover(ctx, opts, a.log)
+	host := discovery.DiscoverHost()
 	a.mu.Lock()
 	a.inv = inv
+	a.prov.Host = host
 	a.mu.Unlock()
+	a.log.Info("this node", "os", host.OSID+" "+host.OSVersion, "arch", host.Arch,
+		"cpu_flags", host.CPUFlags, "nvidia_driver", host.NVIDIADriver)
 	return err
 }
 
 // session runs one connection until it breaks.
-func (a *Agent) session(ctx context.Context) error {
-	client := llmgwv1connect.NewAgentServiceClient(a.http, a.cfg.Server)
+func (a *Agent) session(ctx context.Context, server string) error {
+	client := llmgwv1connect.NewAgentServiceClient(a.http, server)
 
 	// parent is kept because the derived context is cancelled from inside this
 	// function; asking the derived one whether we are shutting down would
@@ -488,7 +712,23 @@ func (a *Agent) session(ctx context.Context) error {
 	// Every registration reports what is already running, not just the first
 	// one: a reconnect may be a control plane restart, and it has to recover
 	// the same way (plan §6.5).
+	// What this node can actually run, so the control plane stops placing
+	// engines that are not here. The registry holds an adapter for every
+	// engine whether or not its binary exists, so without this a node with no
+	// vLLM accepted a vLLM shard and failed it minutes later.
+	reg.EngineVersions = a.engines(parent)
+	reg.EnginesProbed = true
 	reg.RunningShards = a.runningShards()
+	a.mu.Lock()
+	h := a.prov.Host
+	// A new stream is a control plane that knows nothing of where installs
+	// stand, so the next heartbeat tells it again.
+	a.provSent = ""
+	a.mu.Unlock()
+	reg.Host = &llmgwv1.Host{
+		OsId: h.OSID, OsVersion: h.OSVersion, OsCodename: h.OSCodename,
+		Arch: h.Arch, CpuFlags: h.CPUFlags, NvidiaDriver: h.NVIDIADriver,
+	}
 
 	if err := stream.Send(&llmgwv1.AgentMessage{
 		Msg: &llmgwv1.AgentMessage_Register{Register: reg},
@@ -575,12 +815,137 @@ func (a *Agent) heartbeat() *llmgwv1.AgentMessage {
 	a.mu.Unlock()
 
 	hb := &llmgwv1.Heartbeat{NodeId: a.cfg.NodeID}
+
+	// Read every time; it is one small file. Memory is added to a running VM,
+	// and a node that grew from 7 GB to 32 GB has to be seen to have grown
+	// without anybody restarting anything.
+	hb.MemoryTotalMb = int64(discovery.MemoryTotalMB())
+
+	a.reprobeEngines()
+	a.mu.Lock()
+	if vers := a.engineReportLocked(); a.configuredVers != nil &&
+		enginesFingerprint(vers) != a.enginesSent {
+		a.enginesSent = enginesFingerprint(vers)
+		hb.EnginesChanged = true
+		hb.EngineVersions = vers
+	}
+	if fp := provisionFingerprint(a.provStatus); fp != a.provSent {
+		a.provSent = fp
+		hb.ProvisionChanged = true
+		for _, e := range slices.Sorted(maps.Keys(a.provStatus)) {
+			st := a.provStatus[e]
+			hb.Provision = append(hb.Provision, &llmgwv1.ProvisionStatus{
+				Engine: st.Engine, Version: st.Version, Variant: st.Variant,
+				State: st.State, Detail: st.Detail,
+			})
+		}
+	}
+	a.mu.Unlock()
+
 	for _, g := range inv.GPUs {
 		hb.GpuMemoryUsedMb = append(hb.GpuMemoryUsedMb, int32(g.MemoryUsedMB))
 	}
+
+	// Rescanned here rather than reused from registration. Disk is not a fact
+	// about a machine, it is a fact about a moment: a node that had 200 GB when
+	// its agent started can be full an hour later, and it is precisely the node
+	// busy filling up whose figure matters.
+	for _, st := range discovery.ScanStorage(a.cfg.ModelDirs) {
+		hb.Storage = append(hb.Storage, &llmgwv1.Storage{
+			Path: st.Path, FreeBytes: st.FreeBytes,
+			TotalBytes: st.TotalBytes, Shared: st.Shared,
+		})
+	}
+
+	// The file list is longer and changes rarely, so it goes only when it has.
+	// The flag is what makes an empty list mean "this node now holds nothing"
+	// rather than "nothing to report" — without it, the first heartbeat after
+	// a model was deleted would be indistinguishable from silence.
+	weights := discovery.ScanWeights(a.cfg.ModelDirs)
+	fp := weightsFingerprint(weights)
+	a.mu.Lock()
+	changed := fp != a.lastWeights
+	a.lastWeights = fp
+	a.mu.Unlock()
+	if changed {
+		hb.WeightsChanged = true
+		for _, w := range weights {
+			hb.Weights = append(hb.Weights, &llmgwv1.LocalWeights{
+				Path: w.Path, SizeBytes: w.SizeBytes, Shared: w.Shared,
+			})
+		}
+	}
 	return &llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_Heartbeat{Heartbeat: hb}}
 }
 
+// weightsFingerprint is enough to notice a model appearing, disappearing or
+// finishing its download.
+//
+// Paths and sizes, because a fetch in progress has the same name as the file
+// it will become — under a different name until it is renamed, but a shared
+// filesystem can also grow a file in place.
+func weightsFingerprint(ws []discovery.LocalWeights) string {
+	h := sha256.New()
+	for _, w := range ws {
+		fmt.Fprintf(h, "%s:%d\n", w.Path, w.SizeBytes)
+	}
+	return hex.EncodeToString(h.Sum(nil))
+}
+
+// removeWeights deletes cached model files the control plane says are no
+// longer needed.
+//
+// Every path is checked here as well as there, and that duplication is the
+// point. The control plane computes these from the catalog, but it is the node
+// that holds the files, and a node must not be talkable into deleting a model
+// an operator spent a week copying onto it — not by a control plane that is
+// wrong, not by one that is newer than this agent, not by one that has been
+// tampered with. So this deletes only what it can prove llmgw itself wrote:
+// inside its own cache directory, under a name this system generates.
+//
+// Failures are logged and skipped rather than reported upward. A file that is
+// already gone is the expected outcome of a retry, and one that will not
+// delete is a disk to look at, not a reason to fail a command whose purpose
+// was to free space.
+func (a *Agent) removeWeights(paths []string) {
+	cache := a.cfg.WeightsCache
+	if cache == "" && len(a.cfg.ModelDirs) > 0 {
+		cache = a.cfg.ModelDirs[0]
+	}
+	if cache == "" {
+		a.log.Warn("asked to remove weights with no cache directory configured")
+		return
+	}
+	cacheAbs, err := filepath.Abs(cache)
+	if err != nil {
+		a.log.Warn("cannot resolve the cache directory", "dir", cache, "err", err)
+		return
+	}
+
+	for _, p := range paths {
+		abs, err := filepath.Abs(p)
+		if err != nil {
+			continue
+		}
+		// Directly inside the cache directory, not merely under it: a nested
+		// path would let ".." or a symlinked subdirectory reach elsewhere, and
+		// nothing this package writes is nested anyway.
+		if filepath.Dir(abs) != cacheAbs {
+			a.log.Warn("refused to remove a file outside the weights cache", "path", p)
+			continue
+		}
+		if !weights.Owned(abs) {
+			a.log.Warn("refused to remove a file llmgw did not fetch", "path", p)
+			continue
+		}
+		if err := os.Remove(abs); err != nil && !os.IsNotExist(err) {
+			a.log.Warn("could not remove cached weights", "path", p, "err", err)
+			continue
+		}
+		a.log.Info("removed cached weights no model asks for", "path", p)
+	}
+}
+
 func (a *Agent) handle(ctx context.Context, msg *llmgwv1.ControlMessage, out chan<- *llmgwv1.AgentMessage) {
 	switch m := msg.Msg.(type) {
 	case *llmgwv1.ControlMessage_Start:
@@ -591,6 +956,10 @@ func (a *Agent) handle(ctx context.Context, msg *llmgwv1.ControlMessage, out cha
 		if err := a.sup.Drain(m.Drain.GetShardId()); err != nil {
 			a.log.Warn("draining failed", "shard", m.Drain.GetShardId(), "err", err)
 		}
+	case *llmgwv1.ControlMessage_RemoveWeights:
+		a.removeWeights(m.RemoveWeights.GetPaths())
+	case *llmgwv1.ControlMessage_Provision:
+		a.setProvision(m.Provision)
 	case *llmgwv1.ControlMessage_Ack:
 		if !m.Ack.GetOk() {
 			a.log.Warn("control plane rejected a message",
@@ -603,6 +972,42 @@ func (a *Agent) handle(ctx context.Context, msg *llmgwv1.ControlMessage, out cha
 //
 // It runs in its own goroutine because model loading takes minutes, and the
 // stream has to keep carrying heartbeats and other commands meanwhile.
+// watchFetch registers interest in a source's download progress and returns
+// the function that stops watching.
+//
+// Keyed by source rather than by shard because a fetch is single-flight: eight
+// shards starting together produce one download, and every one of them is
+// waiting on it.
+func (a *Agent) watchFetch(source string, fn func(weights.Progress)) (stop func()) {
+	a.mu.Lock()
+	if a.fetchWatchers == nil {
+		a.fetchWatchers = map[string][]func(weights.Progress){}
+	}
+	a.fetchWatchers[source] = append(a.fetchWatchers[source], fn)
+	i := len(a.fetchWatchers[source]) - 1
+	a.mu.Unlock()
+
+	var once sync.Once
+	return func() {
+		once.Do(func() {
+			a.mu.Lock()
+			defer a.mu.Unlock()
+			w := a.fetchWatchers[source]
+			if i < len(w) {
+				// Cleared rather than removed, so the indexes other watchers
+				// hold stay valid. The slice is dropped when the last one goes.
+				w[i] = nil
+			}
+			for _, f := range w {
+				if f != nil {
+					return
+				}
+			}
+			delete(a.fetchWatchers, source)
+		})
+	}
+}
+
 func (a *Agent) startShard(ctx context.Context, m *llmgwv1.StartShard, out chan<- *llmgwv1.AgentMessage) {
 	shardID, replicaID := m.GetShardId(), m.GetReplicaId()
 
@@ -610,16 +1015,18 @@ func (a *Agent) startShard(ctx context.Context, m *llmgwv1.StartShard, out chan<
 	a.shardReplica[shardID] = replicaID
 	a.mu.Unlock()
 
-	report := func(st supervisor.State, errMsg string, tail []string) {
-		msg := &llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_Shard{Shard: &llmgwv1.ShardStatus{
-			NodeId: a.cfg.NodeID, ReplicaId: replicaID, ShardId: shardID,
-			State: agentsvc.StateToWire(st), Error: errMsg, LastOutput: tail,
-		}}}
+	send := func(s *llmgwv1.ShardStatus) {
+		s.NodeId, s.ReplicaId, s.ShardId = a.cfg.NodeID, replicaID, shardID
 		select {
-		case out <- msg:
+		case out <- &llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_Shard{Shard: s}}:
 		case <-ctx.Done():
 		}
 	}
+	report := func(st supervisor.State, errMsg string, tail []string) {
+		send(&llmgwv1.ShardStatus{
+			State: agentsvc.StateToWire(st), Error: errMsg, LastOutput: tail,
+		})
+	}
 
 	spec, timeout := agentsvc.ShardSpecFromStart(m)
 
@@ -648,10 +1055,23 @@ func (a *Agent) startShard(ctx context.Context, m *llmgwv1.StartShard, out chan<
 	// arrives minutes into a load, in its own words, on a node nobody is
 	// watching — and because eight shards starting together would otherwise
 	// each fetch the same file.
+	//
+	// Progress is reported upward while it runs, so the control plane can tell
+	// a download from a hang.
+	stop := a.watchFetch(spec.Profile.ModelPath, func(p weights.Progress) {
+		send(&llmgwv1.ShardStatus{
+			State: agentsvc.StateToWire(supervisor.Starting),
+			Fetch: &llmgwv1.FetchProgress{
+				Source: p.Source, BytesDone: p.BytesDone, BytesTotal: p.BytesTotal,
+			},
+		})
+	})
 	if path, err := a.weights().Ensure(ctx, spec.Profile.ModelPath); err != nil {
+		stop()
 		report(supervisor.Failed, err.Error(), nil)
 		return
 	} else {
+		stop()
 		spec.Profile.ModelPath = path
 	}
 
@@ -792,3 +1212,185 @@ func Client(cfg tlsconf.Config, timeout time.Duration) (*http.Client, error) {
 	}
 	return &http.Client{Transport: &http2.Transport{TLSClientConfig: tc}}, nil
 }
+
+// engines is what this node can run: what the agent installed and saw run,
+// and what the configuration names and answers. Nothing merely found on the
+// machine — a llama-server on PATH is somebody's, not necessarily one that
+// works, and placing on it is how a replica fails on a node nobody watches.
+func (a *Agent) engines(ctx context.Context) map[string]string {
+	a.engineOnce.Do(func() {
+		vers := a.reg.ProbeConfigured(ctx)
+		a.mu.Lock()
+		a.configuredVers, a.lastEngineProbe = vers, time.Now()
+		a.mu.Unlock()
+		if len(vers) > 0 {
+			a.log.Info("engines named in the configuration",
+				"engines", engines.ProbedNames(vers))
+		}
+	})
+	a.mu.Lock()
+	defer a.mu.Unlock()
+	vers := a.engineReportLocked()
+	// Whatever registration sends is what the control plane now knows, so a
+	// heartbeat repeats it only once it differs.
+	a.enginesSent = enginesFingerprint(vers)
+	return vers
+}
+
+// engineReportLocked merges the two sources. Installed wins: it is the one
+// that was seen to run. Called with mu held.
+func (a *Agent) engineReportLocked() map[string]string {
+	out := maps.Clone(a.configuredVers)
+	if out == nil {
+		out = map[string]string{}
+	}
+	maps.Copy(out, a.installedVers)
+	return out
+}
+
+// provisionEvery is how often installed engines are checked again. A library
+// removed by an unrelated upgrade is found here rather than by the next
+// replica to start.
+const provisionEvery = 10 * time.Minute
+
+// setProvision takes the control plane's list of wanted engines.
+func (a *Agent) setProvision(p *llmgwv1.Provision) {
+	wants := []provision.Want{}
+	for _, e := range p.GetEngines() {
+		wants = append(wants, provision.Want{Engine: e.GetEngine(), Version: e.GetVersion()})
+	}
+	a.mu.Lock()
+	a.provWants = wants
+	a.mu.Unlock()
+	select {
+	case a.provKick <- struct{}{}:
+	default:
+	}
+}
+
+// provisionLoop installs and re-checks the wanted engines, one pass at a time.
+func (a *Agent) provisionLoop(ctx context.Context) {
+	// After a failure the next pass comes sooner, and backs off: a lock held
+	// by a new machine's own updates clears in a minute or two, while ten
+	// minutes' wait left a node idle for no reason.
+	next := provisionEvery
+	retry := time.Minute
+	for {
+		select {
+		case <-ctx.Done():
+			return
+		case <-a.provKick:
+		case <-time.After(next):
+		}
+		a.mu.Lock()
+		wants := slices.Clone(a.provWants)
+		managed := a.provWants != nil
+		a.mu.Unlock()
+		if !managed {
+			continue
+		}
+
+		statuses := a.prov.Converge(ctx, wants)
+		next = provisionEvery
+		for _, st := range statuses {
+			if st.State == provision.Failed {
+				next, retry = retry, min(retry*2, provisionEvery)
+				break
+			}
+		}
+		if next == provisionEvery {
+			retry = time.Minute
+		}
+		installed := map[string]string{}
+		keep := map[string]bool{}
+		for _, st := range statuses {
+			keep[st.Engine] = true
+			if st.State == provision.Ready {
+				a.reg.UseInstalled(st.Engine, st.Binary)
+				installed[st.Engine] = st.Version
+			}
+		}
+		a.mu.Lock()
+		if enginesFingerprint(installed) != enginesFingerprint(a.installedVers) {
+			a.log.Info("engines installed and running", "engines", engines.ProbedNames(installed))
+		}
+		a.installedVers = installed
+		// An engine no longer asked for is no longer reported on.
+		for e := range a.provStatus {
+			if !keep[e] {
+				delete(a.provStatus, e)
+			}
+		}
+		a.mu.Unlock()
+	}
+}
+
+// provisionStatus records a change in one engine's installation.
+func (a *Agent) provisionStatus(st provision.Status) {
+	a.mu.Lock()
+	prev, had := a.provStatus[st.Engine]
+	a.provStatus[st.Engine] = st
+	a.mu.Unlock()
+	if !had || prev.State != st.State || prev.Detail != st.Detail {
+		lvl := slog.LevelInfo
+		if st.State == provision.Failed || st.State == provision.Unsupported {
+			lvl = slog.LevelWarn
+		}
+		a.log.Log(context.Background(), lvl, "engine installation",
+			"engine", st.Engine, "version", st.Version, "variant", st.Variant,
+			"state", st.State, "detail", st.Detail)
+	}
+}
+
+func provisionFingerprint(m map[string]provision.Status) string {
+	var b strings.Builder
+	for _, k := range slices.Sorted(maps.Keys(m)) {
+		st := m[k]
+		fmt.Fprintf(&b, "%s|%s|%s|%s|%s\n", st.Engine, st.Version, st.Variant, st.State, st.Detail)
+	}
+	return b.String()
+}
+
+// engineReprobe is how often the installed engines are looked for again.
+//
+// Minutes, not seconds: an engine is installed by somebody at a keyboard, so a
+// minute's delay in noticing is invisible, while SGLang's probe starts a
+// Python interpreter that imports it.
+const engineReprobe = time.Minute
+
+// reprobeEngines looks for installed engines again when it is time, off the
+// heartbeat goroutine so a slow probe never delays a heartbeat.
+func (a *Agent) reprobeEngines() {
+	a.mu.Lock()
+	due := !a.probingEngines && !a.lastEngineProbe.IsZero() &&
+		time.Since(a.lastEngineProbe) >= engineReprobe
+	if due {
+		a.probingEngines = true
+	}
+	a.mu.Unlock()
+	if !due {
+		return
+	}
+	go func() {
+		ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
+		defer cancel()
+		vers := a.reg.ProbeConfigured(ctx)
+
+		a.mu.Lock()
+		changed := enginesFingerprint(vers) != enginesFingerprint(a.configuredVers)
+		a.configuredVers, a.lastEngineProbe, a.probingEngines = vers, time.Now(), false
+		a.mu.Unlock()
+		if changed {
+			a.log.Info("engines named in the configuration changed",
+				"engines", engines.ProbedNames(vers))
+		}
+	}()
+}
+
+func enginesFingerprint(vers map[string]string) string {
+	var b strings.Builder
+	for _, k := range slices.Sorted(maps.Keys(vers)) {
+		fmt.Fprintf(&b, "%s=%s\n", k, vers[k])
+	}
+	return b.String()
+}

internal/agent/agent_test.go

@@ -38,6 +38,7 @@ type statusRecorder struct {
 	calls   []adoptCall
 	adopted []reconciler.AdoptedShard
 	unclaim []string // shard ids to report back as unclaimed
+	fetches []fetchReport
 }
 
 type status struct {
@@ -55,6 +56,25 @@ func (r *statusRecorder) OnShardStatus(replicaID, shardID string, st supervisor.
 	}
 }
 
+// fetches records download progress, so a test can assert that a fetch is
+// visible from the control plane rather than only in the node's log.
+func (r *statusRecorder) OnFetchProgress(replicaID, shardID, source string, done, total int64) {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	r.fetches = append(r.fetches, fetchReport{replicaID, shardID, source, done, total})
+}
+
+func (r *statusRecorder) fetchReports() []fetchReport {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	return append([]fetchReport(nil), r.fetches...)
+}
+
+type fetchReport struct {
+	replicaID, shardID, source string
+	done, total                int64
+}
+
 func (r *statusRecorder) Adopt(nodeID string, shards []reconciler.AdoptedShard) []reconciler.AdoptedShard {
 	r.mu.Lock()
 	defer r.mu.Unlock()

internal/agent/discovery/discover.go

@@ -24,7 +24,8 @@ type Options struct {
 	// PreferNVML uses the NVML backend when this binary has one. It is the only
 	// source of NVLink topology.
 	PreferNVML bool `koanf:"prefer_nvml"`
-	// SkipGPUs omits GPU discovery, for nodes that have none.
+	// SkipGPUs omits GPU discovery. A node without nvidia-smi is already
+	// found to have none, so this is for one whose driver is broken.
 	SkipGPUs bool `koanf:"skip_gpus"`
 	// ModelDirs are scanned for weights. Empty means do not look.
 	ModelDirs []string `koanf:"-"`
@@ -51,6 +52,9 @@ func Discover(ctx context.Context, opts Options, log *slog.Logger) (Inventory, e
 	// holds has nothing to do with its cards, and a node with none is exactly
 	// where a CPU profile will look for them.
 	inv.Weights = ScanWeights(opts.ModelDirs)
+	// And the disk they sit on. A node can hold a model today and be unable to
+	// take the next one, which the file list alone never shows.
+	inv.Storage = ScanStorage(opts.ModelDirs)
 
 	if !opts.SkipGPUs {
 		switch {

internal/agent/discovery/gpu.go

@@ -114,6 +114,9 @@ type Inventory struct {
 	// Weights are the model files this node can see. Empty means it was not
 	// asked to look.
 	Weights []LocalWeights `json:"weights,omitempty"`
+	// Storage is the disk behind those weights, one entry per filesystem.
+	// Empty means the same thing it means for Weights: not asked.
+	Storage []Storage `json:"storage,omitempty"`
 }
 
 // Classes returns the distinct GPU classes present, sorted.

internal/agent/discovery/gpu_smi.go

@@ -44,6 +44,13 @@ const mandatorySMIFields = 10
 // NVML build adds.
 func DiscoverGPUsSMI(ctx context.Context, binary string) ([]GPU, error) {
 	if binary == "" {
+		// Not configured and not installed is a node without an NVIDIA card,
+		// not a failed discovery. Reported as an error it also cut discovery
+		// short before InfiniBand was looked at, on every CPU node that had
+		// not been told by hand to skip GPUs.
+		if _, err := exec.LookPath("nvidia-smi"); err != nil {
+			return nil, nil
+		}
 		binary = "nvidia-smi"
 	}
 	out, err := exec.CommandContext(ctx, binary,

internal/agent/discovery/host.go

@@ -0,0 +1,154 @@
+package discovery
+
+import (
+	"bufio"
+	"os"
+	"runtime"
+	"strings"
+)
+
+// Host is what a node is, as far as installing software on it is concerned.
+type Host struct {
+	// OSID, OSVersion and OSCodename come from /etc/os-release: "ubuntu",
+	// "24.04", "noble".
+	OSID       string `json:"os_id,omitempty"`
+	OSVersion  string `json:"os_version,omitempty"`
+	OSCodename string `json:"os_codename,omitempty"`
+	// OSLike is ID_LIKE: the families this system says it follows, "debian"
+	// for Ubuntu, "rhel centos fedora" for Rocky. It is what decides the
+	// package manager, which the name alone does not for every derivative.
+	OSLike []string `json:"os_like,omitempty"`
+	// Arch is Debian's spelling, "amd64" or "arm64", because that is what
+	// package names and release archives are keyed by.
+	Arch string `json:"arch,omitempty"`
+	// CPUFlags are the vector extensions inference cares about, of those this
+	// processor has.
+	CPUFlags []string `json:"cpu_flags,omitempty"`
+	// NVIDIADriver is the loaded driver's version, empty when there is none.
+	NVIDIADriver string `json:"nvidia_driver,omitempty"`
+}
+
+// Family is the packaging family: "deb", "rpm", or "" when neither is known.
+func (h Host) Family() string {
+	for _, id := range append([]string{h.OSID}, h.OSLike...) {
+		switch id {
+		case "debian", "ubuntu":
+			return "deb"
+		case "rhel", "fedora", "centos", "rocky", "almalinux", "suse", "opensuse":
+			return "rpm"
+		}
+	}
+	return ""
+}
+
+// HasCPUFlag reports whether the processor has an extension.
+func (h Host) HasCPUFlag(f string) bool {
+	for _, have := range h.CPUFlags {
+		if have == f {
+			return true
+		}
+	}
+	return false
+}
+
+// interestingFlags are the extensions that decide which build of an engine
+// runs well. A VM on QEMU's default CPU model has none of them — and still
+// runs, five times slower, with nothing saying why.
+var interestingFlags = []string{"avx", "avx2", "fma", "f16c", "avx512f", "avx512_bf16", "amx_tile", "asimd", "sve"}
+
+// DiscoverHost reads what the machine is.
+func DiscoverHost() Host {
+	h := hostFromOSRelease("/etc/os-release")
+	h.Arch = debianArch(runtime.GOARCH)
+	h.CPUFlags = cpuFlagsFrom("/proc/cpuinfo")
+	h.NVIDIADriver = nvidiaDriverFrom("/proc/driver/nvidia/version")
+	return h
+}
+
+func hostFromOSRelease(path string) Host {
+	var h Host
+	f, err := os.Open(path) //nolint:gosec // a fixed system path
+	if err != nil {
+		return h
+	}
+	defer f.Close()
+	sc := bufio.NewScanner(f)
+	for sc.Scan() {
+		k, v, ok := strings.Cut(sc.Text(), "=")
+		if !ok {
+			continue
+		}
+		v = strings.Trim(v, `"'`)
+		switch k {
+		case "ID":
+			h.OSID = v
+		case "VERSION_ID":
+			h.OSVersion = v
+		case "VERSION_CODENAME":
+			h.OSCodename = v
+		case "ID_LIKE":
+			h.OSLike = strings.Fields(v)
+		}
+	}
+	return h
+}
+
+func debianArch(goarch string) string {
+	switch goarch {
+	case "amd64", "arm64":
+		return goarch
+	case "386":
+		return "i386"
+	case "arm":
+		return "armhf"
+	}
+	return goarch
+}
+
+func cpuFlagsFrom(path string) []string {
+	b, err := os.ReadFile(path) //nolint:gosec // a fixed system path
+	if err != nil {
+		return nil
+	}
+	have := map[string]bool{}
+	for _, line := range strings.Split(string(b), "\n") {
+		k, v, ok := strings.Cut(line, ":")
+		if !ok {
+			continue
+		}
+		// "flags" on x86, "Features" on ARM. The first processor is enough:
+		// a machine does not mix instruction sets across its cores.
+		if k = strings.TrimSpace(k); k != "flags" && k != "Features" {
+			continue
+		}
+		for _, f := range strings.Fields(v) {
+			have[f] = true
+		}
+		break
+	}
+	var out []string
+	for _, f := range interestingFlags {
+		if have[f] {
+			out = append(out, f)
+		}
+	}
+	return out
+}
+
+// nvidiaDriverFrom reads the loaded kernel module's version, which exists
+// exactly when a driver is loaded — unlike nvidia-smi, which can be installed
+// on a machine whose module failed to load.
+func nvidiaDriverFrom(path string) string {
+	b, err := os.ReadFile(path) //nolint:gosec // a fixed system path
+	if err != nil {
+		return ""
+	}
+	// "NVRM version: NVIDIA UNIX x86_64 Kernel Module  570.86.15  Thu ..."
+	fields := strings.Fields(string(b))
+	for i, f := range fields {
+		if f == "Module" && i+1 < len(fields) {
+			return fields[i+1]
+		}
+	}
+	return ""
+}

internal/agent/discovery/storage.go

@@ -0,0 +1,90 @@
+package discovery
+
+import (
+	"os"
+	"syscall"
+)
+
+// Storage is one filesystem a node keeps weights on.
+//
+// One entry per filesystem rather than per configured directory, because free
+// space belongs to the disk and not to the path. Two model directories on the
+// same disk are one pool of bytes, and reporting them separately would let the
+// control plane add 24 GB to 24 GB and place a 40 GB model on a node with 24.
+type Storage struct {
+	// Path is one directory on this filesystem — the first one configured that
+	// landed here, so the operator recognises it.
+	Path string `json:"path"`
+	// FreeBytes is what a non-root process can still write. Deliberately not
+	// the root reserve: the agent does not run as root, so the reserve is space
+	// it cannot have.
+	FreeBytes int64 `json:"free_bytes"`
+	// TotalBytes is the filesystem's size, so a reading of "8 GB free" can be
+	// told apart on a 10 TB filer and on a 10 GB root partition.
+	TotalBytes int64 `json:"total_bytes"`
+	// Shared marks a filesystem the node did not have to fill. Free space on
+	// one is shared with every other node using it, so two nodes reporting
+	// 500 GB free on the same filer have 500 GB between them, not a terabyte.
+	Shared bool `json:"shared,omitempty"`
+}
+
+// ScanStorage reports the filesystems behind the given model directories.
+//
+// Empty in, empty out, and a directory that cannot be read is skipped rather
+// than reported as full: a mount that has not come back yet looks exactly like
+// a missing directory, and claiming zero free bytes would take the node out of
+// placement for a reason that is not true. Absent is not zero here, as
+// everywhere else in this system.
+func ScanStorage(dirs []string) []Storage {
+	var out []Storage
+	seen := map[uint64]int{} // filesystem id -> index in out
+
+	for _, dir := range dirs {
+		if dir == "" {
+			continue
+		}
+		dev, ok := filesystemID(dir)
+		if !ok {
+			continue
+		}
+		if i, dup := seen[dev]; dup {
+			// Same disk, already counted. Keep the first path: it is the one
+			// the operator wrote first and so the one they will recognise.
+			_ = i
+			continue
+		}
+
+		var st syscall.Statfs_t
+		if err := syscall.Statfs(dir, &st); err != nil {
+			continue
+		}
+		// Bsize is int32 on some platforms and int64 on others, which is why
+		// every field here is widened explicitly rather than assumed.
+		block := int64(st.Bsize)
+		seen[dev] = len(out)
+		out = append(out, Storage{
+			Path:       dir,
+			FreeBytes:  int64(st.Bavail) * block, //nolint:gosec,unconvert // widened on purpose
+			TotalBytes: int64(st.Blocks) * block, //nolint:gosec,unconvert // widened on purpose
+			Shared:     isNetworkFS(dir),
+		})
+	}
+	return out
+}
+
+// filesystemID identifies the disk a path is on.
+//
+// By device number rather than by the statfs filesystem id, because the field
+// holding that id is spelled differently on Linux and on macOS and this has to
+// compile on both — the agent runs on one and is developed on the other.
+func filesystemID(path string) (uint64, bool) {
+	fi, err := os.Stat(path)
+	if err != nil {
+		return 0, false
+	}
+	st, ok := fi.Sys().(*syscall.Stat_t)
+	if !ok {
+		return 0, false
+	}
+	return uint64(st.Dev), true //nolint:gosec,unconvert // int32 on macOS, uint64 on Linux
+}

internal/agent/discovery/storage_test.go

@@ -0,0 +1,63 @@
+package discovery
+
+import (
+	"os"
+	"path/filepath"
+	"testing"
+)
+
+func TestTwoDirectoriesOnOneDiskAreOnePoolOfBytes(t *testing.T) {
+	// The reason this matters: free space belongs to the disk, not to the path.
+	// Counting two directories separately lets the control plane add 24 GB to
+	// 24 GB and place a 40 GB model on a node that has 24.
+	base := t.TempDir()
+	a := filepath.Join(base, "models-a")
+	b := filepath.Join(base, "models-b")
+	for _, d := range []string{a, b} {
+		if err := os.Mkdir(d, 0o755); err != nil {
+			t.Fatal(err)
+		}
+	}
+
+	got := ScanStorage([]string{a, b})
+	if len(got) != 1 {
+		t.Fatalf("two directories on one filesystem reported as %d pools: %+v", len(got), got)
+	}
+	if got[0].Path != a {
+		t.Errorf("kept %q rather than the first directory configured", got[0].Path)
+	}
+}
+
+func TestStorageReportsSomethingPlausible(t *testing.T) {
+	got := ScanStorage([]string{t.TempDir()})
+	if len(got) != 1 {
+		t.Fatalf("a real directory reported %d filesystems", len(got))
+	}
+	s := got[0]
+	if s.TotalBytes <= 0 {
+		t.Error("a filesystem with no size at all")
+	}
+	if s.FreeBytes < 0 || s.FreeBytes > s.TotalBytes {
+		t.Errorf("free %d of total %d", s.FreeBytes, s.TotalBytes)
+	}
+}
+
+func TestAMissingDirectoryIsNotAFullDisk(t *testing.T) {
+	// A mount that has not come back yet looks exactly like a missing
+	// directory. Reporting it as zero free bytes would take the node out of
+	// placement for a reason that is not true; reporting nothing says only
+	// that nothing is known, which is the honest answer.
+	got := ScanStorage([]string{filepath.Join(t.TempDir(), "never-mounted")})
+	if len(got) != 0 {
+		t.Errorf("a missing directory was reported as a filesystem: %+v", got)
+	}
+}
+
+func TestNoDirectoriesConstrainNothing(t *testing.T) {
+	if got := ScanStorage(nil); len(got) != 0 {
+		t.Errorf("scanning nothing found %+v", got)
+	}
+	if got := ScanStorage([]string{""}); len(got) != 0 {
+		t.Errorf("an empty path found %+v", got)
+	}
+}

internal/agent/engines/engine.go

@@ -72,6 +72,11 @@ type ShardSpec struct {
 	// not used when building a command.
 	Pool   string
 	Fabric string
+	// ProfileFP identifies the version of the profile this shard was started
+	// from. Carried for the same reason as Pool and Fabric, and never read: it
+	// lets the control plane tell, after its own restart, whether a shard that
+	// is still running matches the profile the catalog holds now.
+	ProfileFP string
 }
 
 // Leader reports whether this shard serves HTTP for the replica.
@@ -146,6 +151,16 @@ func NewRegistry(cfg Config) *Registry {
 	return r
 }
 
+// UseInstalled points an engine at a build the agent installed, and reports
+// whether the engine takes one.
+func (r *Registry) UseInstalled(name, path string) bool {
+	u, ok := r.engines[name].(interface{ UseInstalled(string) })
+	if ok {
+		u.UseInstalled(path)
+	}
+	return ok
+}
+
 // Add registers an engine, replacing any engine of the same name.
 func (r *Registry) Add(e Engine) { r.engines[e.Name()] = e }
 

internal/agent/engines/llamacpp.go

@@ -2,14 +2,20 @@ package engines
 
 import (
 	"fmt"
+	"os"
+	"path/filepath"
+	"sort"
 	"strconv"
 	"strings"
+	"sync/atomic"
 )
 
 // LlamaCppConfig locates the llama.cpp installation.
 type LlamaCppConfig struct {
 	// Bin is the llama-server binary. It is the server, not the CLI: the
 	// gateway talks OpenAI over HTTP and needs the one that serves it.
+	// Empty means the agent installs llama.cpp itself, at the version the
+	// control plane asks for. Setting it opts out of that.
 	Bin string `koanf:"bin"`
 	// Parallel is how many requests a server handles at once, when a profile
 	// does not say. llama.cpp defaults to one, which would make every second
@@ -23,7 +29,10 @@ type LlamaCppConfig struct {
 // small model on CPU at a speed that is fine for testing the whole path, and it
 // serves the same OpenAI API as the others, so nothing downstream can tell the
 // difference.
-type LlamaCpp struct{ cfg LlamaCppConfig }
+type LlamaCpp struct {
+	cfg       LlamaCppConfig
+	installed atomic.Pointer[string]
+}
 
 // Name identifies the engine in the catalog.
 func (*LlamaCpp) Name() string { return "llamacpp" }
@@ -65,14 +74,15 @@ func (*LlamaCpp) Validate(p Profile, gpus []GPUInfo) error {
 
 // Commands returns the processes for one shard, which is always one process.
 func (l *LlamaCpp) Commands(s ShardSpec) ([]Command, error) {
-	bin := l.cfg.Bin
-	if bin == "" {
-		bin = "llama-server"
-	}
+	bin := l.bin()
 	p := s.Profile
 
+	model, err := ggufIn(p.ModelPath)
+	if err != nil {
+		return nil, err
+	}
 	args := []string{
-		"--model", p.ModelPath,
+		"--model", model,
 		"--host", "0.0.0.0",
 		"--port", strconv.Itoa(s.HTTPPort),
 		// Without this there is no /metrics, and an endpoint the router cannot
@@ -127,3 +137,61 @@ func hasFlag(args []string, names ...string) bool {
 	}
 	return false
 }
+
+// ggufIn resolves a model path that may be a directory.
+//
+// A multi-file version arrives as the directory it was downloaded into, because
+// that is the only shape a set can take. llama.cpp wants a file: given the
+// first part of a split GGUF it finds the rest beside it, so the first part is
+// what it gets.
+//
+// A path that is already a file is returned untouched, which is every
+// single-file model and every hand-staged one.
+func ggufIn(path string) (string, error) {
+	st, err := os.Stat(path)
+	if err != nil || !st.IsDir() {
+		// Not a directory, or not there at all — the second is the engine's to
+		// report, with the message it would have given anyway.
+		return path, nil //nolint:nilerr // a missing path is the engine's error to give
+	}
+	entries, err := os.ReadDir(path)
+	if err != nil {
+		return "", err
+	}
+	var names []string
+	for _, e := range entries {
+		if !e.IsDir() && strings.HasSuffix(strings.ToLower(e.Name()), ".gguf") {
+			names = append(names, e.Name())
+		}
+	}
+	if len(names) == 0 {
+		return "", fmt.Errorf("%s holds no .gguf file", path)
+	}
+	// Sorted, so a split model's first part comes first: the parts are named
+	// "-00001-of-00004" and lexical order is shard order.
+	sort.Strings(names)
+	return filepath.Join(path, names[0]), nil
+}
+
+// UseInstalled points the engine at a build the agent installed and checked.
+// It takes precedence over everything else, because it is the only one that
+// was seen to run.
+func (l *LlamaCpp) UseInstalled(path string) { l.installed.Store(&path) }
+
+// Configured reports whether the agent's configuration names a binary. A
+// named binary is the operator's deliberate choice; one merely found on PATH
+// is not, and is not reported as installed.
+func (l *LlamaCpp) Configured() bool { return l.cfg.Bin != "" }
+
+// bin is the llama-server to run: the installed build, else the configured
+// one, else whatever PATH has — which is only ever reached by a node not
+// provisioned by the agent and not told where its engine is.
+func (l *LlamaCpp) bin() string {
+	if p := l.installed.Load(); p != nil {
+		return *p
+	}
+	if l.cfg.Bin != "" {
+		return l.cfg.Bin
+	}
+	return "llama-server"
+}

internal/agent/engines/llamacpp_test.go

@@ -1,6 +1,8 @@
 package engines
 
 import (
+	"os"
+	"path/filepath"
 	"slices"
 	"strings"
 	"testing"
@@ -187,3 +189,59 @@ func TestLlamaCppLetsAProfileTurnTheTemplateOff(t *testing.T) {
 		t.Errorf("the profile said --no-jinja and got both: %v", cmds[0].Args)
 	}
 }
+
+func TestASetDirectoryResolvesToItsFirstPart(t *testing.T) {
+	dir := t.TempDir()
+	// Deliberately out of order on disk, because readdir order is not shard
+	// order and the first part is the one llama.cpp must be given.
+	for _, n := range []string{
+		"qwen-q4_k_m-00003-of-00003.gguf",
+		"qwen-q4_k_m-00001-of-00003.gguf",
+		"qwen-q4_k_m-00002-of-00003.gguf",
+	} {
+		if err := os.WriteFile(filepath.Join(dir, n), []byte("x"), 0o644); err != nil {
+			t.Fatal(err)
+		}
+	}
+
+	got, err := ggufIn(dir)
+	if err != nil {
+		t.Fatal(err)
+	}
+	// Given the first part llama.cpp finds the rest beside it; given the third
+	// it loads a third of a model.
+	if filepath.Base(got) != "qwen-q4_k_m-00001-of-00003.gguf" {
+		t.Errorf("expected the first shard, got %s", filepath.Base(got))
+	}
+}
+
+func TestAPlainFilePathIsUntouched(t *testing.T) {
+	// Every single-file model and every hand-staged one takes this path.
+	p := filepath.Join(t.TempDir(), "model.gguf")
+	if err := os.WriteFile(p, []byte("x"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	if got, _ := ggufIn(p); got != p {
+		t.Errorf("expected the path unchanged, got %s", got)
+	}
+}
+
+func TestADirectoryWithNoWeightsSaysSo(t *testing.T) {
+	dir := t.TempDir()
+	if err := os.WriteFile(filepath.Join(dir, "config.json"), []byte("{}"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	if _, err := ggufIn(dir); err == nil {
+		t.Error("a directory with no .gguf in it is not a model llama.cpp can load")
+	}
+}
+
+func TestAMissingPathIsLeftToTheEngine(t *testing.T) {
+	// The engine's own "no such file" names the path and the flag it came
+	// from, which is more use than anything this could say.
+	p := "/nonexistent/model.gguf"
+	got, err := ggufIn(p)
+	if err != nil || got != p {
+		t.Errorf("expected it passed through, got %q %v", got, err)
+	}
+}

internal/agent/engines/probe.go

@@ -0,0 +1,224 @@
+package engines
+
+import (
+	"context"
+	"errors"
+	"os/exec"
+	"regexp"
+	"sort"
+	"strings"
+	"time"
+)
+
+// Unknown is the version reported for an engine that is installed and would
+// not say which version it is.
+//
+// A value, not an absent key, because the two mean different things and
+// placement reads the keys: absent is "this node cannot run it", Unknown is
+// "it is here and the probe could not get a number out of it". Collapsing them
+// would make a slow engine look like a missing one and take a node out of
+// scheduling for being slow to answer --version.
+const Unknown = "unknown"
+
+// Prober is an engine that can say whether it is installed on this machine.
+//
+// Optional, and asserted for rather than part of Engine: an adapter that has no
+// way to check should report nothing rather than be forced to invent an answer.
+type Prober interface {
+	// Probe returns the engine's version, or an error when it is not installed.
+	Probe(ctx context.Context) (string, error)
+}
+
+// Probe asks every registered engine whether it is here.
+//
+// The result is what the node reports at registration, and its key set is what
+// placement filters on. Until this existed nobody knew: the registry registers
+// every adapter whether or not its binary exists, so a node with no vLLM
+// accepted a vLLM shard, registered healthy, and failed the shard minutes later
+// with "executable file not found" — in the engine's own words, on a machine
+// nobody was watching.
+//
+// Bounded, because it runs on the path to registering and an agent that cannot
+// reach the control plane is worse than one that reports an incomplete list.
+func (r *Registry) Probe(ctx context.Context) map[string]string {
+	out := map[string]string{}
+	for name, e := range r.engines {
+		p, ok := e.(Prober)
+		if !ok {
+			continue
+		}
+		v, err := p.Probe(ctx)
+		if err != nil {
+			continue
+		}
+		if v == "" {
+			v = Unknown
+		}
+		out[name] = v
+	}
+	return out
+}
+
+// Names of the engines in a probe result, sorted.
+func ProbedNames(m map[string]string) []string {
+	out := make([]string, 0, len(m))
+	for n := range m {
+		out = append(out, n)
+	}
+	sort.Strings(out)
+	return out
+}
+
+// probeTimeout bounds one engine's version check.
+//
+// Generous, because vLLM's --version imports torch and that is seconds on a
+// cold page cache, and mean, because this is startup: an engine that has not
+// answered in this long is reported as present-with-unknown-version rather
+// than waited for.
+const probeTimeout = 8 * time.Second
+
+// run executes a version command, saying whether the program was even there.
+//
+// The two failures are different answers and the caller has to tell them
+// apart. A binary that is absent is an engine this node cannot run. A binary
+// that is present and exits non-zero is an engine it probably can — unless the
+// command was a test of whether a module imports, where a non-zero exit is the
+// whole answer.
+func run(ctx context.Context, bin string, args ...string) (out string, found bool, ok bool) {
+	path, err := exec.LookPath(bin)
+	if err != nil {
+		return "", false, false
+	}
+	ctx, cancel := context.WithTimeout(ctx, probeTimeout)
+	defer cancel()
+
+	// Combined, not just stdout: engines print their version to either, and one
+	// that chose stderr would otherwise read as having no version at all.
+	b, err := exec.CommandContext(ctx, path, args...).CombinedOutput()
+	return string(b), true, err == nil
+}
+
+// installedBinary reports an engine that is its own executable.
+//
+// It resolves the path and runs nothing. The first version of this ran the
+// engine with --version, which is fine for an engine and not fine for the
+// wrapper scripts people put in front of one: a script that shifts past flags
+// it does not recognise and then execs the server will start an inference
+// server, at agent startup, on every node. The agent's own test suite installs
+// exactly such a script, and caught it.
+//
+// So the version is not read here. Presence is what placement needs, and what
+// the version is for — telling whether an engine still accepts the flags llmgw
+// builds — is `llmgw ctl engine-check`, which is run deliberately by somebody
+// who meant to run the engine.
+func installedBinary(bin string) (string, error) {
+	if _, err := exec.LookPath(bin); err != nil {
+		return "", errNotInstalled
+	}
+	return Unknown, nil
+}
+
+// versionOfModule probes an engine that is a module inside an interpreter.
+//
+// This one does execute, and safely: the command is llmgw's own one-line
+// import, not the engine's launcher, so there is no wrapper to shift past
+// flags and start a server. There is also no alternative — a module has no
+// path to look up.
+//
+// A non-zero exit is the answer: python3 exists on almost every machine, so
+// presence of the interpreter says nothing. Treating a failed import as
+// "installed" reported SGLang on every node in a fleet that had none, which
+// would take nodes into scheduling for an engine they cannot run — the exact
+// failure this whole change exists to stop.
+func versionOfModule(ctx context.Context, python string, args ...string) (string, error) {
+	out, found, ok := run(ctx, python, args...)
+	if !found || !ok {
+		return "", errNotInstalled
+	}
+	return firstVersion(out), nil
+}
+
+var errNotInstalled = errors.New("not installed on this node")
+
+var versionPattern = regexp.MustCompile(`\b\d+\.\d+(\.\d+)?\S*`)
+
+// firstVersion is the first thing in the output that looks like a version.
+//
+// Pattern rather than parsing each engine's format: they print "vllm 0.6.3",
+// "version: b4123 (…)", and a paragraph with the number in the middle, and all
+// of those change between releases. What this is for is telling one build from
+// another, which the first number does.
+func firstVersion(s string) string {
+	for _, line := range strings.Split(s, "\n") {
+		if m := versionPattern.FindString(line); m != "" {
+			return strings.TrimRight(m, ",;)")
+		}
+	}
+	return Unknown
+}
+
+// Configured is an engine whose location the agent's configuration names.
+type Configured interface {
+	Configured() bool
+}
+
+// ProbeConfigured probes only the engines the configuration names.
+//
+// Probe looks everywhere — PATH, the default interpreter — and reports what
+// it finds, which is exactly the trust in whatever happens to be on a machine
+// that installing engines replaces. A node the agent provisions reports what
+// it installed, plus what the operator named on purpose; nothing else.
+func (r *Registry) ProbeConfigured(ctx context.Context) map[string]string {
+	out := map[string]string{}
+	for name, e := range r.engines {
+		c, ok := e.(Configured)
+		if !ok || !c.Configured() {
+			continue
+		}
+		p, ok := e.(Prober)
+		if !ok {
+			continue
+		}
+		if v, err := p.Probe(ctx); err == nil {
+			if v == "" {
+				v = Unknown
+			}
+			out[name] = v
+		}
+	}
+	return out
+}
+
+// Probe reports whether the llama.cpp server is installed.
+func (l *LlamaCpp) Probe(context.Context) (string, error) {
+	return installedBinary(l.bin())
+}
+
+// Configured reports whether the configuration names vLLM.
+func (v *VLLM) Configured() bool { return v.cfg.Bin != "" }
+
+// Configured reports whether the configuration names SGLang's interpreter.
+func (g *SGLang) Configured() bool { return g.cfg.Python != "" }
+
+// Probe reports whether vLLM is installed.
+func (v *VLLM) Probe(context.Context) (string, error) {
+	bin := v.cfg.Bin
+	if bin == "" {
+		bin = "vllm"
+	}
+	return installedBinary(bin)
+}
+
+// Probe reports the SGLang version.
+//
+// Through the interpreter, because SGLang is a module rather than a binary:
+// asking the shell for "sglang" finds nothing on a machine that can run it
+// perfectly well.
+func (g *SGLang) Probe(ctx context.Context) (string, error) {
+	python := g.cfg.Python
+	if python == "" {
+		python = "python3"
+	}
+	return versionOfModule(ctx, python, "-c",
+		"import sglang, sys; sys.stdout.write(getattr(sglang, '__version__', ''))")
+}

internal/agent/engines/probe_test.go

@@ -0,0 +1,107 @@
+package engines
+
+import (
+	"context"
+	"os"
+	"path/filepath"
+	"testing"
+)
+
+// script writes an executable that stands in for an engine.
+func script(t *testing.T, name, body string) string {
+	t.Helper()
+	dir := t.TempDir()
+	p := filepath.Join(dir, name)
+	if err := os.WriteFile(p, []byte("#!/bin/sh\n"+body+"\n"), 0o755); err != nil {
+		t.Fatal(err)
+	}
+	return p
+}
+
+func TestAnEngineThatIsNotInstalledIsAbsentFromTheReport(t *testing.T) {
+	r := NewRegistry(Config{})
+	r.engines = map[string]Engine{
+		"vllm": &VLLM{cfg: VLLMConfig{Bin: "/nonexistent/vllm"}},
+	}
+	// Absent, not present-with-unknown-version. Placement reads the keys, and
+	// a node listed for an engine it cannot run is the failure being fixed.
+	if got := r.Probe(t.Context()); len(got) != 0 {
+		t.Errorf("expected nothing, got %v", got)
+	}
+}
+
+func TestProbingNeverRunsTheEngine(t *testing.T) {
+	// A wrapper that shifts past flags it does not recognise and then starts
+	// the server. People put these in front of engines, and the agent's own
+	// tests install one. Probing with --version would launch an inference
+	// server at agent startup, on every node.
+	marker := filepath.Join(t.TempDir(), "ran")
+	bin := script(t, "fake-vllm", "touch "+marker+"; sleep 60")
+
+	r := NewRegistry(Config{})
+	r.engines = map[string]Engine{"vllm": &VLLM{cfg: VLLMConfig{Bin: bin}}}
+
+	got := r.Probe(t.Context())
+	if _, ok := got["vllm"]; !ok {
+		t.Error("a binary that is there should be reported present")
+	}
+	if _, err := os.Stat(marker); err == nil {
+		t.Fatal("the engine was executed by a probe")
+	}
+}
+
+func TestAModuleThatWillNotImportIsNotInstalled(t *testing.T) {
+	// python3 is on almost every machine, so the interpreter being there says
+	// nothing about SGLang. Reporting it present put every node in a fleet
+	// forward for an engine none of them had.
+	bin := script(t, "python3", "exit 1")
+	r := NewRegistry(Config{})
+	r.engines = map[string]Engine{"sglang": &SGLang{cfg: SGLangConfig{Python: bin}}}
+
+	if got := r.Probe(t.Context()); len(got) != 0 {
+		t.Errorf("a failed import means not installed, got %v", got)
+	}
+}
+
+func TestAModuleThatImportsIsReportedWithItsVersion(t *testing.T) {
+	bin := script(t, "python3", `echo "0.4.1"`)
+	r := NewRegistry(Config{})
+	r.engines = map[string]Engine{"sglang": &SGLang{cfg: SGLangConfig{Python: bin}}}
+
+	got := r.Probe(t.Context())
+	if got["sglang"] != "0.4.1" {
+		t.Errorf("expected the version, got %v", got)
+	}
+}
+
+func TestAVersionIsPickedOutOfWhateverIsPrinted(t *testing.T) {
+	// Only the module probe reaches this now, and a Python package version is
+	// dotted. A bare build number is deliberately not matched: it would turn a
+	// stray number in a line of prose into a version.
+	for in, want := range map[string]string{
+		"0.4.1":                 "0.4.1",
+		"sglang 0.4.post1":      "0.4.post1",
+		"warning: noise\n0.4.1": "0.4.1",
+		"":                      Unknown,
+		"no numbers here":       Unknown,
+		"just 4123":             Unknown,
+	} {
+		if got := firstVersion(in); got != want {
+			t.Errorf("firstVersion(%q) = %q, want %q", in, got, want)
+		}
+	}
+}
+
+func TestProbingIsBoundedByTheContext(t *testing.T) {
+	ctx, cancel := context.WithCancel(context.Background())
+	cancel()
+	bin := script(t, "python3", "sleep 60")
+	r := NewRegistry(Config{})
+	r.engines = map[string]Engine{"sglang": &SGLang{cfg: SGLangConfig{Python: bin}}}
+
+	// Registration waits on this. An agent that cannot reach the control plane
+	// is worse than one reporting an incomplete list.
+	if got := r.Probe(ctx); len(got) != 0 {
+		t.Errorf("a cancelled probe should report nothing, got %v", got)
+	}
+}

internal/agent/failover_test.go

@@ -0,0 +1,107 @@
+package agent
+
+import (
+	"context"
+	"log/slog"
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"testing"
+	"time"
+
+	"golang.org/x/net/http2"
+	"golang.org/x/net/http2/h2c"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/agentsvc"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
+)
+
+type standby bool
+
+func (s standby) IsLeader() bool { return !bool(s) }
+
+// startPlain runs an agent service over h2c, which is enough to test who
+// accepts a connection and who refuses it.
+//
+// A nil leader means none was configured, which is every deployment that
+// existed before leadership did and has to keep accepting agents.
+func startPlain(t *testing.T, leader agentsvc.Leader) (*inventory.Store, string) {
+	t.Helper()
+	inv := inventory.NewStore()
+	svc := agentsvc.New(inv, newRecorder(), false, slog.New(slog.DiscardHandler))
+	if leader != nil {
+		svc.SetLeadership(leader)
+	}
+
+	path, handler := svc.Handler()
+	mux := http.NewServeMux()
+	mux.Handle(path, handler)
+	// h2c, because the agent speaks HTTP/2 with prior knowledge and a plain
+	// server negotiates HTTP/1.1 — which turns a clear refusal into a frame
+	// parsing error that says nothing about why.
+	srv := httptest.NewServer(h2c.NewHandler(mux, &http2.Server{}))
+	t.Cleanup(srv.Close)
+	return inv, srv.URL
+}
+
+func TestAStandbyRefusesAnAgent(t *testing.T) {
+	// Commands travel down the stream the agent opened, so whichever control
+	// plane holds it is the only one that can start anything on that node. A
+	// standby holding it means the leader's placements fail with "no agent
+	// connected" for a node that is plainly connected.
+	_, addr := startPlain(t, standby(true))
+
+	ag := runAgent(t, Config{NodeID: "n1", Server: addr, MgmtAddr: "127.0.0.1"})
+	err := ag.session(context.Background(), ag.cfg.Server)
+	if err == nil {
+		t.Fatal("a standby accepted an agent")
+	}
+	if !strings.Contains(err.Error(), "standby") {
+		t.Errorf("the refusal does not say why: %v", err)
+	}
+}
+
+func TestAnAgentReachesTheLeaderPastAStandby(t *testing.T) {
+	// The point of the list. The standby is first, so the agent only registers
+	// if it moved on rather than sitting on the refusal.
+	_, standbyAddr := startPlain(t, standby(true))
+	leaderInv, leaderAddr := startPlain(t, standby(false))
+
+	ag := runAgent(t, Config{
+		NodeID: "n1", MgmtAddr: "127.0.0.1",
+		Servers: []string{standbyAddr, leaderAddr},
+	})
+
+	ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
+	defer cancel()
+	go ag.Run(ctx)
+
+	deadline := time.Now().Add(8 * time.Second)
+	for time.Now().Before(deadline) {
+		if _, ok := leaderInv.Get("n1"); ok {
+			return
+		}
+		time.Sleep(50 * time.Millisecond)
+	}
+	t.Fatal("the agent never reached the leader past the standby")
+}
+
+func TestALeaderStillAcceptsWithNoLeadershipConfigured(t *testing.T) {
+	// Every deployment before this had no leadership at all, and must keep
+	// accepting agents with nothing configured.
+	inv, addr := startPlain(t, nil)
+
+	ag := runAgent(t, Config{NodeID: "n1", Server: addr, MgmtAddr: "127.0.0.1"})
+	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+	defer cancel()
+	go ag.Run(ctx)
+
+	deadline := time.Now().Add(4 * time.Second)
+	for time.Now().Before(deadline) {
+		if _, ok := inv.Get("n1"); ok {
+			return
+		}
+		time.Sleep(50 * time.Millisecond)
+	}
+	t.Fatal("a service with no leadership configured refused an agent")
+}

internal/agent/mtls_test.go

@@ -255,7 +255,7 @@ func TestRefusedAgentLearnsWhy(t *testing.T) {
 		t.Fatal(err)
 	}
 
-	err = ag.session(context.Background())
+	err = ag.session(context.Background(), ag.cfg.Server)
 	if err == nil {
 		t.Fatal("a refused session reported success")
 	}

internal/agent/provision/install.go

@@ -0,0 +1,658 @@
+package provision
+
+import (
+	"archive/tar"
+	"compress/gzip"
+	"context"
+	"crypto/sha256"
+	"encoding/hex"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"io/fs"
+	"log/slog"
+	"net/http"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"strings"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
+)
+
+// Want is one engine at one version.
+type Want struct {
+	Engine  string
+	Version string
+}
+
+// The states an engine's installation can be in.
+const (
+	Installing  = "installing"
+	Ready       = "ready"
+	Failed      = "failed"
+	Unsupported = "unsupported"
+)
+
+// Status is where one engine's installation stands.
+type Status struct {
+	Engine  string
+	Version string
+	Variant string
+	State   string
+	// Detail is for a person: what is happening, or what went wrong.
+	Detail string
+	// Binary is the executable, once ready.
+	Binary string
+}
+
+// Installer installs engines under one directory.
+type Installer struct {
+	// Root is where engines are installed, one directory per build:
+	// <root>/<engine>/<version>-<variant>.
+	Root string
+	Host discovery.Host
+	Log  *slog.Logger
+	HTTP *http.Client
+	// OnStatus hears every change, so a download that takes minutes is seen
+	// to be under way rather than silent.
+	OnStatus func(Status)
+
+	// Run executes a command; a seam for tests. Nil runs it for real.
+	Run func(ctx context.Context, env []string, name string, args ...string) ([]byte, error)
+	// APIBase and DownloadBase default to GitHub; tests point them at a fake.
+	APIBase      string
+	DownloadBase string
+}
+
+// DefaultRoot is where engines are installed when nothing says otherwise.
+const DefaultRoot = "/opt/llmgw/engines"
+
+// marker is written into a build's directory once it has been installed and
+// seen to run. Its absence means the directory is not to be trusted.
+const marker = ".llmgw-installed"
+
+type markerFile struct {
+	Engine    string            `json:"engine"`
+	Version   string            `json:"version"`
+	Variant   string            `json:"variant"`
+	Binary    string            `json:"binary"`
+	SHA256    map[string]string `json:"sha256"`
+	Installed time.Time         `json:"installed"`
+}
+
+// Converge brings every wanted engine to ready, or says why it cannot.
+//
+// An engine already installed is checked again rather than believed: its
+// directory can be damaged, or a library it needs removed by somebody's
+// upgrade, and that is found here rather than by the next replica to fail.
+func (in *Installer) Converge(ctx context.Context, wants []Want) []Status {
+	out := make([]Status, 0, len(wants))
+	for _, w := range wants {
+		st := in.one(ctx, w)
+		in.report(st)
+		out = append(out, st)
+	}
+	return out
+}
+
+func (in *Installer) one(ctx context.Context, w Want) Status {
+	st := Status{Engine: w.Engine, Version: w.Version}
+	recipe, ok := Recipes[w.Engine]
+	if !ok {
+		st.State = Unsupported
+		st.Detail = "this agent does not know how to install " + w.Engine +
+			"; install it by hand and name it in the agent's configuration"
+		return st
+	}
+	build, err := recipe.Choose(w.Version, in.Host)
+	if err != nil {
+		st.State, st.Detail = Unsupported, err.Error()
+		return st
+	}
+	st.Variant = build.Variant
+	dir := filepath.Join(in.root(), w.Engine, w.Version+"-"+build.Variant)
+
+	// Installed before: check it still runs, and trust nothing else about it.
+	if _, err := os.Stat(filepath.Join(dir, marker)); err == nil {
+		// The packages are checked too. Removing one is exactly the kind of
+		// change an unrelated upgrade makes, and ldd below would catch it only
+		// as a missing library with no hint of which package held it.
+		if err := in.packages(ctx, build.Packages, &st); err != nil {
+			st.State, st.Detail = Failed, err.Error()
+			return st
+		}
+		bin, version, err := in.verify(ctx, dir, build)
+		if err == nil {
+			return in.ready(st, build, bin, version)
+		}
+		in.log().Warn("an installed engine no longer runs; installing it again",
+			"engine", w.Engine, "version", w.Version, "err", err)
+	}
+
+	if err := in.install(ctx, recipe, w, build, dir, &st); err != nil {
+		st.State, st.Detail, st.Binary = Failed, err.Error(), ""
+		return st
+	}
+	bin, version, err := in.verify(ctx, dir, build)
+	if err != nil {
+		st.State, st.Detail = Failed, err.Error()
+		return st
+	}
+	return in.ready(st, build, bin, version)
+}
+
+func (in *Installer) ready(st Status, b Build, bin, version string) Status {
+	st.State, st.Binary = Ready, bin
+	st.Detail = strings.TrimSpace(filepath.Base(bin) + " " + version)
+	var missing []string
+	for _, f := range b.CPUFlags {
+		if !in.Host.HasCPUFlag(f) {
+			missing = append(missing, f)
+		}
+	}
+	if len(missing) > 0 {
+		// Said on every report, not once in a log: it is the difference between
+		// five tokens a second and forty, and a VM given QEMU's default CPU
+		// model looks entirely healthy otherwise.
+		st.Detail += "; this CPU lacks " + strings.Join(missing, ", ") +
+			", so it will run several times slower (a VM needs CPU type \"host\")"
+	}
+	return st
+}
+
+// install fetches, checks and unpacks a build into dir.
+//
+// Into a scratch directory renamed into place at the end, so a failure halfway
+// leaves nothing that looks installed.
+func (in *Installer) install(ctx context.Context, r Recipe, w Want, b Build, dir string, st *Status) error {
+	if err := in.packages(ctx, b.Packages, st); err != nil {
+		return err
+	}
+
+	in.progress(st, "asking GitHub for the checksums of %s %s", w.Engine, w.Version)
+	sums, err := in.checksums(ctx, r.Repo, w.Version)
+	if err != nil {
+		return err
+	}
+
+	tmp := dir + ".partial"
+	_ = os.RemoveAll(tmp)
+	if err := os.MkdirAll(tmp, 0o755); err != nil {
+		return err
+	}
+	defer os.RemoveAll(tmp) //nolint:errcheck // gone either way once renamed
+
+	got := map[string]string{}
+	for _, asset := range b.Assets {
+		want, ok := sums[asset]
+		if !ok {
+			// Refused rather than installed unchecked: this is a binary that
+			// will run as root on every node.
+			return fmt.Errorf("%s %s publishes no checksum for %s, so it cannot be verified",
+				w.Engine, w.Version, asset)
+		}
+		archive := filepath.Join(tmp, asset+".download")
+		if err := in.download(ctx, r.Repo, w.Version, asset, archive, want, st); err != nil {
+			return err
+		}
+		in.progress(st, "unpacking %s", asset)
+		if err := untarGz(archive, tmp); err != nil {
+			return fmt.Errorf("unpacking %s: %w", asset, err)
+		}
+		_ = os.Remove(archive)
+		got[asset] = want
+	}
+
+	in.progress(st, "checking that %s runs", b.Binary)
+	bin, _, err := in.verify(ctx, tmp, b)
+	if err != nil {
+		return err
+	}
+	rel, _ := filepath.Rel(tmp, bin)
+	m, _ := json.MarshalIndent(markerFile{
+		Engine: w.Engine, Version: w.Version, Variant: b.Variant,
+		Binary: rel, SHA256: got, Installed: time.Now().UTC(),
+	}, "", "  ")
+	if err := os.WriteFile(filepath.Join(tmp, marker), m, 0o644); err != nil { //nolint:gosec // not secret
+		return err
+	}
+	_ = os.RemoveAll(dir)
+	if err := os.Rename(tmp, dir); err != nil {
+		return err
+	}
+	in.log().Info("installed an engine", "engine", w.Engine, "version", w.Version,
+		"variant", b.Variant, "dir", dir)
+	return nil
+}
+
+// verify finds the binary and runs it.
+//
+// Two checks, because they fail differently. ldd names a missing library
+// outright, where running the binary would only say it could not start. And
+// running it is the only proof that the rest — the right instruction set, the
+// right driver — is in place.
+func (in *Installer) verify(ctx context.Context, dir string, b Build) (bin, version string, err error) {
+	bin, err = findFile(dir, b.Binary)
+	if err != nil {
+		return "", "", err
+	}
+	if out, err := in.run(ctx, nil, "ldd", bin); err == nil {
+		if missing := lddMissing(string(out)); len(missing) > 0 {
+			return "", "", fmt.Errorf("%s cannot run on this system: %s", b.Binary, strings.Join(missing, "; "))
+		}
+	}
+	tctx, cancel := context.WithTimeout(ctx, 60*time.Second)
+	defer cancel()
+	out, err := in.run(tctx, nil, bin, "--version")
+	if err != nil {
+		return "", "", fmt.Errorf("%s --version failed: %v: %s", b.Binary, err, lastLine(out))
+	}
+	return bin, firstVersionLine(out), nil
+}
+
+// packages installs the operating system packages a build needs.
+//
+// Only those missing, and from the node's own package sources: a package the
+// system already has is not touched, so an engine never upgrades something
+// the rest of the machine depends on.
+func (in *Installer) packages(ctx context.Context, byFamily map[string][]string, st *Status) error {
+	family := in.Host.Family()
+	pkgs := byFamily[family]
+	if len(pkgs) == 0 {
+		// A system of no family this knows is not refused: its packages may
+		// well be there, and if they are not, ldd names the missing library.
+		return nil
+	}
+	pm := packageManagers[family]
+	var missing []string
+	for _, p := range pkgs {
+		if _, err := in.run(ctx, nil, pm.query[0], append(pm.query[1:], p)...); err != nil {
+			missing = append(missing, p)
+		}
+	}
+	if len(missing) == 0 {
+		return nil
+	}
+	tool, cmds := pm.install(in, missing)
+	if tool == "" {
+		return fmt.Errorf("%s is missing and no package manager was found to install it",
+			strings.Join(missing, ", "))
+	}
+	in.progress(st, "installing %s with %s", strings.Join(missing, ", "), tool)
+	for _, c := range cmds {
+		if err := in.runWaitingForLock(ctx, pm.env, c, st); err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+// lockWait bounds how long a package command waits for another to finish.
+const lockWait = 10 * time.Minute
+
+// lockRetry is how often a locked package manager is tried again.
+var lockRetry = 10 * time.Second
+
+// runWaitingForLock runs a package command, waiting while another holds the
+// package manager's lock.
+//
+// A machine a minute old is running cloud-init's and unattended-upgrades' own
+// apt. apt's DPkg::Lock::Timeout covers dpkg's lock and not the one `apt-get
+// update` takes on its lists, so the first install on a new node failed with
+// "Unable to lock directory /var/lib/apt/lists/" — found by building one.
+func (in *Installer) runWaitingForLock(ctx context.Context, env, c []string, st *Status) error {
+	deadline := time.Now().Add(lockWait)
+	for {
+		out, err := in.run(ctx, env, c[0], c[1:]...)
+		if err == nil {
+			return nil
+		}
+		if !lockHeld(string(out)) || time.Now().After(deadline) {
+			return fmt.Errorf("%s: %v: %s", strings.Join(c, " "), err, lastLine(out))
+		}
+		in.progress(st, "waiting for another %s to finish (a new machine runs its own updates first)", c[0])
+		select {
+		case <-ctx.Done():
+			return ctx.Err()
+		case <-time.After(lockRetry):
+		}
+	}
+}
+
+// lockHeld reports whether a package command failed only because another
+// held the lock.
+func lockHeld(out string) bool {
+	for _, s := range []string{"Unable to lock", "Could not get lock", "is locked by another process",
+		"waiting for process with pid", "System management is locked"} {
+		if strings.Contains(out, s) {
+			return true
+		}
+	}
+	return false
+}
+
+// packageManager is how one packaging family asks and installs.
+type packageManager struct {
+	// query exits zero when the package named after it is installed.
+	query []string
+	env   []string
+	// install picks the tool present on this machine and the commands to run.
+	install func(in *Installer, pkgs []string) (tool string, cmds [][]string)
+}
+
+var packageManagers = map[string]packageManager{
+	"deb": {
+		// dpkg -s exits zero for a package merely known about, so the status
+		// is asked for and checked by the command itself: ${db:Status-Abbrev}
+		// is "ii " only when installed.
+		query: []string{"sh", "-c", `test "$(dpkg-query -W -f='${db:Status-Abbrev}' "$0" 2>/dev/null)" = "ii "`},
+		env:   []string{"DEBIAN_FRONTEND=noninteractive"},
+		install: func(_ *Installer, pkgs []string) (string, [][]string) {
+			// A lock timeout rather than failing on the lock: a machine a
+			// minute old is running cloud-init's and unattended-upgrades' own
+			// apt, and the first attempt on every new node would fail.
+			lock := "-o=DPkg::Lock::Timeout=600"
+			return "apt-get", [][]string{
+				{"apt-get", lock, "update"},
+				append([]string{"apt-get", lock, "install", "-y", "--no-install-recommends"}, pkgs...),
+			}
+		},
+	},
+	"rpm": {
+		query: []string{"rpm", "-q"},
+		install: func(in *Installer, pkgs []string) (string, [][]string) {
+			for _, tool := range []string{"dnf", "yum"} {
+				if in.has(tool) {
+					return tool, [][]string{append([]string{tool, "install", "-y"}, pkgs...)}
+				}
+			}
+			if in.has("zypper") {
+				return "zypper", [][]string{append([]string{"zypper", "--non-interactive", "install"}, pkgs...)}
+			}
+			return "", nil
+		},
+	},
+}
+
+// has reports whether a command is on PATH. Asked through the same seam as
+// every other command, so a test can say which tools a machine has.
+func (in *Installer) has(tool string) bool {
+	_, err := in.run(context.Background(), nil, "sh", "-c", `command -v "$0"`, tool)
+	return err == nil
+}
+
+// checksums reads the sha256 GitHub publishes for each asset of a release.
+func (in *Installer) checksums(ctx context.Context, repo, tag string) (map[string]string, error) {
+	url := strings.TrimSuffix(in.apiBase(), "/") + "/repos/" + repo + "/releases/tags/" + tag
+	req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
+	if err != nil {
+		return nil, err
+	}
+	req.Header.Set("Accept", "application/vnd.github+json")
+	resp, err := in.client().Do(req)
+	if err != nil {
+		return nil, fmt.Errorf("reaching GitHub for %s %s: %w", repo, tag, err)
+	}
+	defer resp.Body.Close()
+	switch {
+	case resp.StatusCode == http.StatusNotFound:
+		return nil, fmt.Errorf("%s has no release %s", repo, tag)
+	case resp.StatusCode == http.StatusForbidden || resp.StatusCode == http.StatusTooManyRequests:
+		// Unauthenticated callers get sixty requests an hour per address, and
+		// a whole fleet behind one NAT shares them.
+		return nil, fmt.Errorf("GitHub is rate-limiting this address (%s); it retries later", resp.Status)
+	case resp.StatusCode != http.StatusOK:
+		return nil, fmt.Errorf("GitHub answered %s for %s %s", resp.Status, repo, tag)
+	}
+	var rel struct {
+		Assets []struct {
+			Name   string `json:"name"`
+			Digest string `json:"digest"`
+		} `json:"assets"`
+	}
+	if err := json.NewDecoder(resp.Body).Decode(&rel); err != nil {
+		return nil, fmt.Errorf("reading %s %s from GitHub: %w", repo, tag, err)
+	}
+	out := map[string]string{}
+	for _, a := range rel.Assets {
+		if sum, ok := strings.CutPrefix(a.Digest, "sha256:"); ok && len(sum) == 64 {
+			out[a.Name] = sum
+		}
+	}
+	return out, nil
+}
+
+func (in *Installer) download(ctx context.Context, repo, tag, asset, dst, want string, st *Status) error {
+	url := strings.TrimSuffix(in.downloadBase(), "/") + "/" + repo + "/releases/download/" + tag + "/" + asset
+	req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
+	if err != nil {
+		return err
+	}
+	resp, err := in.client().Do(req)
+	if err != nil {
+		return fmt.Errorf("downloading %s: %w", asset, err)
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode != http.StatusOK {
+		return fmt.Errorf("downloading %s: %s", asset, resp.Status)
+	}
+	if resp.ContentLength > 0 {
+		in.progress(st, "downloading %s (%d MB)", asset, resp.ContentLength>>20)
+	} else {
+		in.progress(st, "downloading %s", asset)
+	}
+	f, err := os.Create(dst) //nolint:gosec // inside the install root
+	if err != nil {
+		return err
+	}
+	h := sha256.New()
+	_, err = io.Copy(io.MultiWriter(f, h), resp.Body)
+	if cerr := f.Close(); err == nil {
+		err = cerr
+	}
+	if err != nil {
+		return fmt.Errorf("downloading %s: %w", asset, err)
+	}
+	if got := hex.EncodeToString(h.Sum(nil)); got != want {
+		return fmt.Errorf("%s does not match the checksum its publisher gave (got %s, expected %s)",
+			asset, got[:12], want[:12])
+	}
+	return nil
+}
+
+// untarGz unpacks an archive under dir, refusing anything that would land
+// outside it. A release archive is somebody else's file, and a path of
+// "../../etc/cron.d/x" in it would otherwise be written as root.
+func untarGz(archive, dir string) error {
+	f, err := os.Open(archive) //nolint:gosec // inside the install root
+	if err != nil {
+		return err
+	}
+	defer f.Close()
+	gz, err := gzip.NewReader(f)
+	if err != nil {
+		return err
+	}
+	tr := tar.NewReader(gz)
+	for {
+		h, err := tr.Next()
+		if errors.Is(err, io.EOF) {
+			return nil
+		}
+		if err != nil {
+			return err
+		}
+		target, err := within(dir, h.Name)
+		if err != nil {
+			return err
+		}
+		switch h.Typeflag {
+		case tar.TypeDir:
+			if err := os.MkdirAll(target, 0o755); err != nil {
+				return err
+			}
+		case tar.TypeReg:
+			if err := os.MkdirAll(filepath.Dir(target), 0o755); err != nil {
+				return err
+			}
+			out, err := os.OpenFile(target, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, fs.FileMode(h.Mode)&0o755) //nolint:gosec // checked above
+			if err != nil {
+				return err
+			}
+			if _, err := io.Copy(out, tr); err != nil { //nolint:gosec // a release archive, size bounded by its checksum
+				out.Close()
+				return err
+			}
+			if err := out.Close(); err != nil {
+				return err
+			}
+		case tar.TypeSymlink:
+			// Libraries are versioned through symlinks, and those are
+			// relative. One pointing anywhere else is refused.
+			if filepath.IsAbs(h.Linkname) {
+				return fmt.Errorf("%s links outside the archive", h.Name)
+			}
+			if _, err := within(dir, filepath.Join(filepath.Dir(h.Name), h.Linkname)); err != nil {
+				return fmt.Errorf("%s links outside the archive", h.Name)
+			}
+			if err := os.MkdirAll(filepath.Dir(target), 0o755); err != nil {
+				return err
+			}
+			_ = os.Remove(target)
+			if err := os.Symlink(h.Linkname, target); err != nil {
+				return err
+			}
+		default:
+			// Devices, FIFOs and hard links have no business in a release.
+		}
+	}
+}
+
+func within(dir, name string) (string, error) {
+	target := filepath.Join(dir, name) //nolint:gosec // checked on the next line
+	rel, err := filepath.Rel(dir, target)
+	if err != nil || rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) {
+		return "", fmt.Errorf("%s would unpack outside the install directory", name)
+	}
+	return target, nil
+}
+
+// lddMissing reads what ldd could not satisfy, in two shapes:
+//
+//	libgomp.so.1 => not found
+//	/lib64/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by ...)
+//
+// The second is a system too old for the build — RHEL 9 against a build that
+// wants GCC 12's libstdc++ — and is said as such, rather than as a library
+// that is plainly there being "not found".
+func lddMissing(out string) []string {
+	seen := map[string]bool{}
+	var missing []string
+	for _, line := range strings.Split(out, "\n") {
+		line = strings.TrimSpace(line)
+		if !strings.Contains(line, "not found") {
+			continue
+		}
+		var what string
+		if lib, rest, ok := strings.Cut(line, ": version `"); ok {
+			sym, _, _ := strings.Cut(rest, "'")
+			what = filepath.Base(lib) + " is too old (needs " + sym + ")"
+		} else {
+			what = strings.TrimSpace(strings.SplitN(line, "=>", 2)[0]) + " is not installed"
+		}
+		if !seen[what] {
+			seen[what] = true
+			missing = append(missing, what)
+		}
+	}
+	return missing
+}
+
+func findFile(dir, name string) (string, error) {
+	var found string
+	_ = filepath.WalkDir(dir, func(p string, d fs.DirEntry, err error) error {
+		if err != nil || found != "" {
+			return nil //nolint:nilerr // an unreadable subtree is not the binary
+		}
+		if !d.IsDir() && d.Name() == name {
+			found = p
+		}
+		return nil
+	})
+	if found == "" {
+		return "", fmt.Errorf("the build has no %s in it", name)
+	}
+	return found, nil
+}
+
+func (in *Installer) progress(st *Status, format string, args ...any) {
+	st.State, st.Detail = Installing, fmt.Sprintf(format, args...)
+	in.report(*st)
+}
+
+func (in *Installer) report(st Status) {
+	if in.OnStatus != nil {
+		in.OnStatus(st)
+	}
+}
+
+func (in *Installer) run(ctx context.Context, env []string, name string, args ...string) ([]byte, error) {
+	if in.Run != nil {
+		return in.Run(ctx, env, name, args...)
+	}
+	cmd := exec.CommandContext(ctx, name, args...) //nolint:gosec // fixed commands and paths we installed
+	cmd.Env = append(os.Environ(), env...)
+	return cmd.CombinedOutput()
+}
+
+func (in *Installer) root() string {
+	if in.Root != "" {
+		return in.Root
+	}
+	return DefaultRoot
+}
+
+func (in *Installer) log() *slog.Logger {
+	if in.Log != nil {
+		return in.Log
+	}
+	return slog.Default()
+}
+
+func (in *Installer) client() *http.Client {
+	if in.HTTP != nil {
+		return in.HTTP
+	}
+	return &http.Client{Timeout: 30 * time.Minute}
+}
+
+func (in *Installer) apiBase() string {
+	if in.APIBase != "" {
+		return in.APIBase
+	}
+	return "https://api.github.com"
+}
+
+func (in *Installer) downloadBase() string {
+	if in.DownloadBase != "" {
+		return in.DownloadBase
+	}
+	return "https://github.com"
+}
+
+func lastLine(b []byte) string {
+	lines := strings.Split(strings.TrimSpace(string(b)), "\n")
+	return strings.TrimSpace(lines[len(lines)-1])
+}
+
+func firstVersionLine(b []byte) string {
+	for _, l := range strings.Split(string(b), "\n") {
+		if strings.Contains(strings.ToLower(l), "version") {
+			return strings.TrimSpace(l)
+		}
+	}
+	return ""
+}

internal/agent/provision/install_test.go

@@ -0,0 +1,332 @@
+package provision
+
+import (
+	"archive/tar"
+	"bytes"
+	"compress/gzip"
+	"context"
+	"crypto/sha256"
+	"encoding/hex"
+	"encoding/json"
+	"errors"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"strings"
+	"sync"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
+)
+
+var ubuntu = discovery.Host{OSID: "ubuntu", OSVersion: "24.04", Arch: "amd64", CPUFlags: []string{"avx2"}}
+
+// release is a fake GitHub: one release, its assets and their digests.
+type release struct {
+	assets  map[string][]byte
+	digests map[string]string
+	gets    int
+	mu      sync.Mutex
+}
+
+func tarGz(t *testing.T, files map[string]string, links map[string]string) []byte {
+	t.Helper()
+	var buf bytes.Buffer
+	gz := gzip.NewWriter(&buf)
+	tw := tar.NewWriter(gz)
+	for name, body := range files {
+		_ = tw.WriteHeader(&tar.Header{Name: name, Mode: 0o755, Size: int64(len(body)), Typeflag: tar.TypeReg})
+		_, _ = tw.Write([]byte(body))
+	}
+	for name, target := range links {
+		_ = tw.WriteHeader(&tar.Header{Name: name, Linkname: target, Typeflag: tar.TypeSymlink})
+	}
+	_ = tw.Close()
+	_ = gz.Close()
+	return buf.Bytes()
+}
+
+func (r *release) serve(t *testing.T) *httptest.Server {
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
+		r.mu.Lock()
+		r.gets++
+		r.mu.Unlock()
+		if strings.HasPrefix(req.URL.Path, "/repos/") {
+			var out struct {
+				Assets []map[string]string `json:"assets"`
+			}
+			for name := range r.assets {
+				out.Assets = append(out.Assets, map[string]string{"name": name, "digest": r.digests[name]})
+			}
+			_ = json.NewEncoder(w).Encode(out)
+			return
+		}
+		body, ok := r.assets[filepath.Base(req.URL.Path)]
+		if !ok {
+			http.NotFound(w, req)
+			return
+		}
+		_, _ = w.Write(body)
+	}))
+	t.Cleanup(srv.Close)
+	return srv
+}
+
+func newRelease(assets map[string][]byte) *release {
+	r := &release{assets: assets, digests: map[string]string{}}
+	for name, b := range assets {
+		sum := sha256.Sum256(b)
+		r.digests[name] = "sha256:" + hex.EncodeToString(sum[:])
+	}
+	return r
+}
+
+// fakeSystem answers the package tools, ldd and the engine itself.
+type fakeSystem struct {
+	mu        sync.Mutex
+	installed map[string]bool
+	tools     map[string]bool
+	lddOut    string
+	calls     []string
+	// locked is how many more package commands find the lock held.
+	locked int
+}
+
+var errExit = errors.New("exit 1")
+
+func (f *fakeSystem) run(_ context.Context, _ []string, name string, args ...string) ([]byte, error) {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	f.calls = append(f.calls, name+" "+strings.Join(args, " "))
+	switch {
+	case name == "sh" && strings.Contains(args[1], "dpkg-query"),
+		name == "rpm" && args[0] == "-q":
+		if f.installed[args[len(args)-1]] {
+			return nil, nil
+		}
+		return nil, errExit
+	case name == "sh" && strings.Contains(args[1], "command -v"):
+		if f.tools[args[2]] {
+			return nil, nil
+		}
+		return nil, errExit
+	case name == "apt-get" || name == "dnf":
+		if f.locked > 0 {
+			f.locked--
+			return []byte("E: Unable to lock directory /var/lib/apt/lists/\n"), errExit
+		}
+		for i, a := range args {
+			if a == "install" {
+				for _, p := range args[i+1:] {
+					if !strings.HasPrefix(p, "-") {
+						f.installed[p] = true
+					}
+				}
+			}
+		}
+		return nil, nil
+	case name == "ldd":
+		return []byte(f.lddOut), nil
+	case strings.HasSuffix(name, "llama-server"):
+		return []byte("version: 10991 (abc)\n"), nil
+	}
+	return nil, errors.New("unexpected command " + name)
+}
+
+func installer(t *testing.T, srv *httptest.Server, sys *fakeSystem, host discovery.Host) (*Installer, *[]Status) {
+	var seen []Status
+	var mu sync.Mutex
+	return &Installer{
+		Root: t.TempDir(), Host: host, Run: sys.run,
+		APIBase: srv.URL, DownloadBase: srv.URL, HTTP: srv.Client(),
+		OnStatus: func(s Status) { mu.Lock(); seen = append(seen, s); mu.Unlock() },
+	}, &seen
+}
+
+func llamaRelease(t *testing.T) *release {
+	return newRelease(map[string][]byte{
+		"llama-b1-bin-ubuntu-x64.tar.gz": tarGz(t,
+			map[string]string{"llama-b1/llama-server": "#!/bin/sh\n", "llama-b1/libllama.so.0": "x"},
+			map[string]string{"llama-b1/libllama.so": "libllama.so.0"}),
+	})
+}
+
+func TestAnEngineIsInstalledFromNothing(t *testing.T) {
+	// A node rebuilt from a bare image: no engine, no OpenMP runtime. What it
+	// ends with has to have come from the publisher, checked, and been run.
+	rel := llamaRelease(t)
+	sys := &fakeSystem{installed: map[string]bool{}}
+	in, seen := installer(t, rel.serve(t), sys, ubuntu)
+
+	got := in.Converge(t.Context(), []Want{{Engine: "llamacpp", Version: "b1"}})
+	if len(got) != 1 || got[0].State != Ready {
+		t.Fatalf("not ready: %+v", got)
+	}
+	if !sys.installed["libgomp1"] {
+		t.Error("the OpenMP runtime the build links against was not installed")
+	}
+	if !strings.HasPrefix(got[0].Binary, in.Root) {
+		t.Errorf("binary %s is not the one installed under %s", got[0].Binary, in.Root)
+	}
+	if _, err := os.Lstat(filepath.Join(filepath.Dir(got[0].Binary), "libllama.so")); err != nil {
+		t.Errorf("the library symlink was not unpacked: %v", err)
+	}
+	var states []string
+	for _, s := range *seen {
+		states = append(states, s.State+": "+s.Detail)
+	}
+	if !strings.Contains(strings.Join(states, "\n"), "downloading") {
+		t.Errorf("a download was not reported while it ran:\n%s", strings.Join(states, "\n"))
+	}
+}
+
+func TestAnInstalledEngineIsCheckedNotFetchedAgain(t *testing.T) {
+	rel := llamaRelease(t)
+	sys := &fakeSystem{installed: map[string]bool{}}
+	in, _ := installer(t, rel.serve(t), sys, ubuntu)
+	w := []Want{{Engine: "llamacpp", Version: "b1"}}
+	in.Converge(t.Context(), w)
+	before := rel.gets
+
+	got := in.Converge(t.Context(), w)
+	if got[0].State != Ready {
+		t.Fatalf("second pass: %+v", got[0])
+	}
+	if rel.gets != before {
+		t.Errorf("a ready engine was downloaded again (%d requests)", rel.gets-before)
+	}
+}
+
+func TestAnEngineThatCannotLoadALibraryIsNotReady(t *testing.T) {
+	// Found on a node before: llama-server present, executable, and unable to
+	// start for want of a library — while the node reported ready.
+	rel := llamaRelease(t)
+	sys := &fakeSystem{installed: map[string]bool{"libgomp1": true},
+		lddOut: "\tlibcuda.so.1 => not found\n"}
+	in, _ := installer(t, rel.serve(t), sys, ubuntu)
+
+	got := in.Converge(t.Context(), []Want{{Engine: "llamacpp", Version: "b1"}})
+	if got[0].State != Failed || !strings.Contains(got[0].Detail, "libcuda.so.1") {
+		t.Fatalf("expected a failure naming the library: %+v", got[0])
+	}
+	if _, err := os.Stat(filepath.Join(in.Root, "llamacpp", "b1-cpu", marker)); err == nil {
+		t.Error("a build that does not run was marked installed")
+	}
+}
+
+func TestADownloadThatDoesNotMatchItsChecksumIsRefused(t *testing.T) {
+	rel := llamaRelease(t)
+	for name := range rel.digests {
+		rel.digests[name] = "sha256:" + strings.Repeat("0", 64)
+	}
+	sys := &fakeSystem{installed: map[string]bool{"libgomp1": true}}
+	in, _ := installer(t, rel.serve(t), sys, ubuntu)
+
+	got := in.Converge(t.Context(), []Want{{Engine: "llamacpp", Version: "b1"}})
+	if got[0].State != Failed || !strings.Contains(got[0].Detail, "checksum") {
+		t.Fatalf("a tampered download was not refused: %+v", got[0])
+	}
+}
+
+func TestAnArchiveCannotWriteOutsideItsDirectory(t *testing.T) {
+	rel := newRelease(map[string][]byte{
+		"llama-b1-bin-ubuntu-x64.tar.gz": tarGz(t,
+			map[string]string{"../../escaped": "x", "llama-server": ""}, nil),
+	})
+	sys := &fakeSystem{installed: map[string]bool{"libgomp1": true}}
+	in, _ := installer(t, rel.serve(t), sys, ubuntu)
+
+	got := in.Converge(t.Context(), []Want{{Engine: "llamacpp", Version: "b1"}})
+	if got[0].State != Failed {
+		t.Fatalf("an archive escaping its directory was installed: %+v", got[0])
+	}
+}
+
+func TestTheBuildFollowsTheHardware(t *testing.T) {
+	gpu := ubuntu
+	gpu.NVIDIADriver = "570.86.15"
+	b, err := chooseLlamaCpp("b1", gpu)
+	if err != nil || !strings.HasPrefix(b.Variant, "cuda") || len(b.Assets) != 2 {
+		t.Errorf("an NVIDIA node got %+v, %v; expected the CUDA build and its runtime", b, err)
+	}
+	if _, err := chooseLlamaCpp("b1", discovery.Host{OSID: "ubuntu", Arch: "riscv64"}); err == nil {
+		t.Error("a build was chosen for an architecture llama.cpp does not publish for")
+	}
+
+	slow := ubuntu
+	slow.CPUFlags = nil
+	st := (&Installer{Host: slow}).ready(Status{}, Build{CPUFlags: []string{"avx2"}}, "/x/llama-server", "")
+	if !strings.Contains(st.Detail, "avx2") {
+		t.Errorf("a CPU without AVX2 was not called out: %q", st.Detail)
+	}
+}
+
+func TestAnEngineTheAgentCannotInstallSaysSo(t *testing.T) {
+	in := &Installer{Root: t.TempDir(), Host: ubuntu}
+	got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.29.0"}})
+	if got[0].State != Unsupported {
+		t.Errorf("got %+v", got[0])
+	}
+}
+
+func TestARedHatFamilyNodeInstallsWithItsOwnPackageManager(t *testing.T) {
+	// The build is not Ubuntu's in any way that matters; the package that
+	// carries OpenMP's runtime is named differently, and installed by dnf.
+	rocky := discovery.Host{OSID: "rocky", OSVersion: "10.0", OSLike: []string{"rhel", "centos", "fedora"},
+		Arch: "amd64", CPUFlags: []string{"avx2"}}
+	rel := llamaRelease(t)
+	sys := &fakeSystem{installed: map[string]bool{}, tools: map[string]bool{"dnf": true}}
+	in, _ := installer(t, rel.serve(t), sys, rocky)
+
+	got := in.Converge(t.Context(), []Want{{Engine: "llamacpp", Version: "b1"}})
+	if got[0].State != Ready {
+		t.Fatalf("not ready: %+v", got[0])
+	}
+	if !sys.installed["libgomp"] || sys.installed["libgomp1"] {
+		t.Errorf("installed %v; expected the rpm name, libgomp", sys.installed)
+	}
+}
+
+func TestASystemTooOldForTheBuildSaysWhatItLacks(t *testing.T) {
+	// RHEL 9: glibc is new enough, libstdc++ is one GCC release short. ldd
+	// finds the library and reports the symbol, and "libstdc++ not found"
+	// would send somebody looking for a package that is installed.
+	rel := llamaRelease(t)
+	sys := &fakeSystem{installed: map[string]bool{"libgomp1": true},
+		lddOut: "\t/lib64/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /x/libllama.so)\n" +
+			"\t/lib64/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /x/llama-server)\n"}
+	in, _ := installer(t, rel.serve(t), sys, ubuntu)
+
+	got := in.Converge(t.Context(), []Want{{Engine: "llamacpp", Version: "b1"}})
+	if got[0].State != Failed || !strings.Contains(got[0].Detail, "libstdc++.so.6 is too old (needs GLIBCXX_3.4.30)") {
+		t.Fatalf("detail %q", got[0].Detail)
+	}
+	if strings.Count(got[0].Detail, "GLIBCXX") != 1 {
+		t.Errorf("the same shortfall was listed more than once: %q", got[0].Detail)
+	}
+}
+
+func TestAPackageManagerBusyWithItsOwnUpdatesIsWaitedFor(t *testing.T) {
+	// A new node's first minutes: cloud-init and unattended-upgrades hold
+	// apt's lists lock, which DPkg::Lock::Timeout does not cover.
+	lockRetry = time.Millisecond
+	t.Cleanup(func() { lockRetry = 10 * time.Second })
+
+	rel := llamaRelease(t)
+	sys := &fakeSystem{installed: map[string]bool{}, locked: 3}
+	in, seen := installer(t, rel.serve(t), sys, ubuntu)
+
+	got := in.Converge(t.Context(), []Want{{Engine: "llamacpp", Version: "b1"}})
+	if got[0].State != Ready {
+		t.Fatalf("gave up on a lock another apt held: %+v", got[0])
+	}
+	waited := false
+	for _, s := range *seen {
+		waited = waited || strings.Contains(s.Detail, "waiting for another apt-get")
+	}
+	if !waited {
+		t.Error("the wait was not reported")
+	}
+}

internal/agent/provision/recipe.go

@@ -0,0 +1,109 @@
+// Package provision installs the engines a node is asked to have.
+//
+// The control plane says which engines at which versions; everything else is
+// worked out here, on the node, because only the node knows it: which build
+// fits its processor and driver, what its operating system is missing, and
+// whether the result actually runs. Nothing already on the machine is taken
+// on trust. An engine counts as installed when this package installed it,
+// checked every file against the checksum its publisher gave, and watched it
+// start.
+package provision
+
+import (
+	"fmt"
+	"strings"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
+)
+
+// Build is one published build of an engine, chosen for one machine.
+type Build struct {
+	// Variant names the build: "cpu", "cuda-12.8".
+	Variant string
+	// Assets are the release files to fetch, unpacked in order into one
+	// directory.
+	Assets []string
+	// Packages are operating system packages the build links against and a
+	// minimal install may not carry, by packaging family ("deb", "rpm"): the
+	// same library is libgomp1 on one and libgomp on the other.
+	Packages map[string][]string
+	// Binary is the executable's file name, found anywhere in the unpacked
+	// tree.
+	Binary string
+	// CPUFlags are extensions the build is slow without. Missing ones are
+	// reported, not refused: slow is still running.
+	CPUFlags []string
+}
+
+// Recipe knows how to install one engine.
+type Recipe struct {
+	// Repo is the GitHub repository whose releases carry the builds.
+	Repo string
+	// Choose picks the build for a machine, or says why none fits.
+	Choose func(version string, h discovery.Host) (Build, error)
+}
+
+// Recipes are the engines this agent can install.
+//
+// vLLM and SGLang are not here yet: they are Python environments of several
+// gigabytes rather than an archive, and are installed by hand and named in the
+// agent's configuration until they are.
+var Recipes = map[string]Recipe{
+	"llamacpp": {Repo: "ggml-org/llama.cpp", Choose: chooseLlamaCpp},
+}
+
+// cudaForLlamaCpp is the CUDA build taken on an NVIDIA node. The release also
+// carries a matching cudart archive, so the node needs a driver and nothing
+// else — no toolkit.
+const cudaForLlamaCpp = "12.8"
+
+func chooseLlamaCpp(version string, h discovery.Host) (Build, error) {
+	// The published Linux builds are built on Ubuntu, but nothing about them
+	// is Ubuntu's: they need glibc 2.34 and the libstdc++ of GCC 12, which
+	// Debian 12 and RHEL 10 have and RHEL 9 does not. So the distribution's
+	// name decides nothing here. The build is tried everywhere, and ldd says
+	// whether this system can load it — in the words of the missing symbol.
+	var arch string
+	switch h.Arch {
+	case "amd64":
+		arch = "x64"
+	case "arm64":
+		arch = "arm64"
+	default:
+		return Build{}, fmt.Errorf("llama.cpp publishes no Linux build for %s", orUnknown(h.Arch))
+	}
+
+	b := Build{
+		Variant: "cpu",
+		Assets:  []string{fmt.Sprintf("llama-%s-bin-ubuntu-%s.tar.gz", version, arch)},
+		// Everything else the build links is in the archive or in any current
+		// glibc system; OpenMP's runtime is the one a cloud image lacks, and
+		// the server does not start without it.
+		Packages: map[string][]string{"deb": {"libgomp1"}, "rpm": {"libgomp"}},
+		Binary:   "llama-server",
+	}
+	if arch == "x64" {
+		// The archive carries a backend per CPU generation and loads the best
+		// one, down to SSE4.2. Below AVX2 it runs, several times slower.
+		b.CPUFlags = []string{"avx2"}
+	}
+	// The CUDA build is published for x64 at this version; on arm64 it is a
+	// newer CUDA that the driver may not support, so an arm64 GPU node runs
+	// the CPU build until someone decides otherwise.
+	if h.NVIDIADriver != "" && arch == "x64" {
+		name := fmt.Sprintf("bin-ubuntu-cuda-%s-x64.tar.gz", cudaForLlamaCpp)
+		b.Variant = "cuda-" + cudaForLlamaCpp
+		b.Assets = []string{
+			fmt.Sprintf("llama-%s-%s", version, name),
+			fmt.Sprintf("cudart-llama-%s-%s", version, name),
+		}
+	}
+	return b, nil
+}
+
+func orUnknown(s string) string {
+	if strings.TrimSpace(s) == "" {
+		return "an unrecognised system"
+	}
+	return s
+}

internal/agent/removeweights_test.go

@@ -0,0 +1,96 @@
+package agent
+
+import (
+	"log/slog"
+	"os"
+	"path/filepath"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/weights"
+)
+
+// agentWithCache is an agent that only knows where its weights live, which is
+// all removeWeights needs.
+func agentWithCache(t *testing.T, dir string) *Agent {
+	t.Helper()
+	return &Agent{
+		cfg: Config{ModelDirs: []string{dir}},
+		log: slog.New(slog.DiscardHandler),
+	}
+}
+
+func write(t *testing.T, path string) string {
+	t.Helper()
+	if err := os.WriteFile(path, []byte("weights"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	return path
+}
+
+func TestTheAgentRefusesToDeleteWhatItDidNotFetch(t *testing.T) {
+	// The control plane computes these paths, but the node holds the files. A
+	// control plane that is wrong, newer than this agent, or tampered with must
+	// not be able to talk it into deleting a model an operator spent a week
+	// copying here. So the agent checks again, on its own.
+	dir := t.TempDir()
+	a := agentWithCache(t, dir)
+
+	operators := write(t, filepath.Join(dir, "qwen2.5-coder-14b-instruct-q8_0.gguf"))
+	fetched := write(t, weights.CachePathFor(dir, "https://models.internal/gone.gguf"))
+
+	a.removeWeights([]string{operators, fetched})
+
+	if _, err := os.Stat(operators); err != nil {
+		t.Errorf("a file llmgw never fetched was deleted: %v", err)
+	}
+	if _, err := os.Stat(fetched); !os.IsNotExist(err) {
+		t.Errorf("the fetched file was not removed: %v", err)
+	}
+}
+
+func TestTheAgentRefusesPathsOutsideItsCache(t *testing.T) {
+	// Even with a name that looks like one of ours. The cache directory is the
+	// boundary, and traversal out of it is the attack this closes.
+	dir := t.TempDir()
+	elsewhere := t.TempDir()
+	a := agentWithCache(t, dir)
+
+	// A correctly named cache file, in the wrong directory.
+	outside := write(t, weights.CachePathFor(elsewhere, "https://models.internal/x.gguf"))
+	// And the same thing reached by traversal from inside the cache.
+	traversal := filepath.Join(dir, "..", filepath.Base(elsewhere),
+		filepath.Base(outside))
+
+	a.removeWeights([]string{outside, traversal})
+
+	if _, err := os.Stat(outside); err != nil {
+		t.Errorf("a file outside the cache was deleted: %v", err)
+	}
+}
+
+func TestRemovingAFileThatIsAlreadyGoneIsNotAFailure(t *testing.T) {
+	// The expected outcome of a retry. It must not stop the rest of the batch.
+	dir := t.TempDir()
+	a := agentWithCache(t, dir)
+
+	missing := weights.CachePathFor(dir, "https://models.internal/never.gguf")
+	present := write(t, weights.CachePathFor(dir, "https://models.internal/here.gguf"))
+
+	a.removeWeights([]string{missing, present})
+
+	if _, err := os.Stat(present); !os.IsNotExist(err) {
+		t.Errorf("a missing file stopped the batch: %v", err)
+	}
+}
+
+func TestWithNoCacheConfiguredNothingIsDeleted(t *testing.T) {
+	dir := t.TempDir()
+	a := &Agent{cfg: Config{}, log: slog.New(slog.DiscardHandler)}
+	kept := write(t, weights.CachePathFor(dir, "https://models.internal/x.gguf"))
+
+	a.removeWeights([]string{kept})
+
+	if _, err := os.Stat(kept); err != nil {
+		t.Errorf("an agent with no cache directory deleted something: %v", err)
+	}
+}

internal/agent/selfdescribe_test.go

@@ -0,0 +1,119 @@
+package agent
+
+import (
+	"context"
+	"os"
+	"slices"
+	"testing"
+	"time"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+)
+
+func TestAnAgentConfigNeedsOnlyWhereTheControlPlaneIs(t *testing.T) {
+	// Every node carried its own name, its own address and the same two
+	// directories written out by hand, so no two agent files were alike and a
+	// copy taken from one node registered the next as the first.
+	cfg := Config{Server: "127.0.0.1:7443"}
+	if err := cfg.normalize(); err != nil {
+		t.Fatal(err)
+	}
+	host, _ := os.Hostname()
+	if cfg.NodeID != host || cfg.Hostname != host {
+		t.Errorf("node_id %q, hostname %q; expected the machine's name %q",
+			cfg.NodeID, cfg.Hostname, host)
+	}
+	if cfg.MgmtAddr != "127.0.0.1" {
+		t.Errorf("mgmt_addr %q, expected the address on the route to the control plane",
+			cfg.MgmtAddr)
+	}
+	if cfg.WeightsCache != DefaultWeightsCache {
+		t.Errorf("weights land in %q", cfg.WeightsCache)
+	}
+	if !slices.Contains(cfg.ModelDirs, cfg.WeightsCache) {
+		t.Errorf("the download directory %q is not looked in: %v",
+			cfg.WeightsCache, cfg.ModelDirs)
+	}
+}
+
+func TestWhatIsWrittenStillWins(t *testing.T) {
+	cfg := Config{
+		Server: "127.0.0.1:7443", NodeID: "gpu-a-01", MgmtAddr: "10.9.9.9",
+		ModelDirs: []string{"/lustre/models"}, WeightsCache: "/scratch/w",
+	}
+	if err := cfg.normalize(); err != nil {
+		t.Fatal(err)
+	}
+	if cfg.NodeID != "gpu-a-01" || cfg.MgmtAddr != "10.9.9.9" {
+		t.Errorf("configured identity overridden: %q %q", cfg.NodeID, cfg.MgmtAddr)
+	}
+	if !slices.Equal(cfg.ModelDirs, []string{"/lustre/models", "/scratch/w"}) {
+		t.Errorf("model dirs %v", cfg.ModelDirs)
+	}
+}
+
+func TestAHeartbeatSaysWhenTheEnginesChanged(t *testing.T) {
+	cfg := Config{Server: "127.0.0.1:7443", NodeID: "n1", WeightsCache: t.TempDir()}
+	cfg.Superv.WorkDir = t.TempDir()
+	a, err := New(cfg, nil, nil)
+	if err != nil {
+		t.Fatal(err)
+	}
+	engines := func(m *llmgwv1.AgentMessage) *llmgwv1.Heartbeat { return m.GetHeartbeat() }
+
+	a.mu.Lock()
+	a.configuredVers = map[string]string{"llamacpp": "unknown"}
+	a.enginesSent = enginesFingerprint(a.configuredVers)
+	a.mu.Unlock()
+	if hb := engines(a.heartbeat()); hb.GetEnginesChanged() {
+		t.Errorf("unchanged engines were resent")
+	}
+
+	a.mu.Lock()
+	a.configuredVers = map[string]string{"llamacpp": "unknown", "vllm": "unknown"}
+	a.mu.Unlock()
+	hb := engines(a.heartbeat())
+	if !hb.GetEnginesChanged() || len(hb.GetEngineVersions()) != 2 {
+		t.Errorf("an installed engine was not reported: %v", hb.GetEngineVersions())
+	}
+	if engines(a.heartbeat()).GetEnginesChanged() {
+		t.Errorf("reported again on the next heartbeat")
+	}
+}
+
+func TestWhatTheControlPlaneAsksToInstallIsReportedBack(t *testing.T) {
+	cfg := Config{Server: "127.0.0.1:7443", NodeID: "n1", WeightsCache: t.TempDir(),
+		EnginesDir: t.TempDir()}
+	cfg.Superv.WorkDir = t.TempDir()
+	a, err := New(cfg, nil, nil)
+	if err != nil {
+		t.Fatal(err)
+	}
+	ctx, cancel := context.WithCancel(t.Context())
+	defer cancel()
+	go a.provisionLoop(ctx)
+
+	// An engine this agent cannot install: no network is needed to learn so,
+	// and the answer has to reach the control plane rather than a log.
+	a.setProvision(&llmgwv1.Provision{Engines: []*llmgwv1.EngineWant{{Engine: "vllm", Version: "0.29.0"}}})
+
+	deadline := time.Now().Add(3 * time.Second)
+	for time.Now().Before(deadline) {
+		hb := a.heartbeat().GetHeartbeat()
+		if hb.GetProvisionChanged() {
+			p := hb.GetProvision()
+			if len(p) != 1 || p[0].GetState() != "unsupported" {
+				t.Fatalf("reported %v", p)
+			}
+			a.mu.Lock()
+			_, reported := a.engineReportLocked()["vllm"]
+			a.mu.Unlock()
+			if reported {
+				t.Error("an engine that was not installed is reported as present")
+			}
+			return
+		}
+		time.Sleep(20 * time.Millisecond)
+	}
+	t.Fatal("the installation's outcome never reached a heartbeat")
+}

internal/agent/servers_test.go

@@ -0,0 +1,90 @@
+package agent
+
+import (
+	"slices"
+	"strings"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
+)
+
+func normalized(t *testing.T, c Config) Config {
+	t.Helper()
+	if err := c.normalizeServerURL(); err != nil {
+		t.Fatalf("normalising %+v: %v", c, err)
+	}
+	return c
+}
+
+func TestOneServerStillWorksExactlyAsBefore(t *testing.T) {
+	// Every agent configuration written before there could be a second control
+	// plane names one address, and none of them should have to change.
+	c := normalized(t, Config{Server: "10.0.0.140:7443"})
+	if !slices.Equal(c.Servers, []string{"http://10.0.0.140:7443"}) {
+		t.Errorf("servers = %v", c.Servers)
+	}
+	if c.Server != "http://10.0.0.140:7443" {
+		t.Errorf("server = %q", c.Server)
+	}
+}
+
+func TestTheAddressAlreadyThereStaysPreferred(t *testing.T) {
+	// An operator adding a standby to an existing file expects the control
+	// plane that was already working to keep being tried first.
+	c := normalized(t, Config{
+		Server:  "first:7443",
+		Servers: []string{"second:7443", "third:7443"},
+	})
+	want := []string{"http://first:7443", "http://second:7443", "http://third:7443"}
+	if !slices.Equal(c.Servers, want) {
+		t.Errorf("servers = %v, want %v", c.Servers, want)
+	}
+}
+
+func TestADuplicateAddressIsNotTriedTwiceInARow(t *testing.T) {
+	// Which would look exactly like the rotation not working.
+	c := normalized(t, Config{
+		Server:  "a:7443",
+		Servers: []string{"http://a:7443", "b:7443"},
+	})
+	want := []string{"http://a:7443", "http://b:7443"}
+	if !slices.Equal(c.Servers, want) {
+		t.Errorf("servers = %v, want %v", c.Servers, want)
+	}
+}
+
+func TestEveryAddressIsCheckedAgainstTheTLSSetting(t *testing.T) {
+	// The check existed for one address and has to hold for all of them: a
+	// standby reached over plaintext while the leader is authenticated would
+	// be a way around mutual TLS, not a fallback.
+	c := Config{
+		Servers: []string{"https://a:7443", "http://b:7443"},
+		TLS:     tlsconf.Config{CA: "ca.pem", Cert: "c.pem", Key: "k.pem"},
+	}
+	err := c.normalizeServerURL()
+	if err == nil {
+		t.Fatal("a plaintext standby was accepted alongside TLS")
+	}
+	if !strings.Contains(err.Error(), "b:7443") {
+		t.Errorf("the error does not say which address: %v", err)
+	}
+}
+
+func TestNoServerAtAllIsStillAnError(t *testing.T) {
+	c := Config{}
+	if err := c.normalizeServerURL(); err == nil {
+		t.Error("an agent with no control plane configured was accepted")
+	}
+}
+
+func TestSchemeIsAddedPerAddress(t *testing.T) {
+	c := normalized(t, Config{
+		Servers: []string{"a:7443", "b:7443"},
+		TLS:     tlsconf.Config{CA: "ca.pem", Cert: "c.pem", Key: "k.pem"},
+	})
+	for _, s := range c.Servers {
+		if !strings.HasPrefix(s, "https://") {
+			t.Errorf("%q did not get the secure scheme", s)
+		}
+	}
+}

internal/agent/supervisor/supervisor_test.go

@@ -8,6 +8,7 @@ import (
 	"net"
 	"os"
 	"path/filepath"
+	"slices"
 	"strings"
 	"sync/atomic"
 	"syscall"
@@ -536,9 +537,9 @@ func TestCorruptStateFileIsCleanedUp(t *testing.T) {
 
 // --- multi-process shards (plan §11, phase 4) ---
 
-func TestSidecarRunsBeforeEngine(t *testing.T) {
-	// The Ray shape: a supporting process starts first and keeps running, then
-	// the engine starts against it.
+func TestASidecarAndAnEngineBothRunUnderOneShard(t *testing.T) {
+	// The Ray shape: a supporting process runs alongside the engine for the
+	// shard's lifetime, but the engine is the one whose exit ends the shard.
 	s, _ := newSupervisor(t)
 	dir := t.TempDir()
 	order := filepath.Join(dir, "order")
@@ -571,13 +572,20 @@ func TestSidecarRunsBeforeEngine(t *testing.T) {
 		}
 	}
 	// Asserted rather than broken out of: the original fell through on timeout,
-	// so a slow machine turned the ordering check into a no-op and the failure
-	// surfaced later as something else.
+	// so a slow machine turned this into a no-op and the failure surfaced later
+	// as something else.
 	if len(started) != 2 {
 		t.Fatalf("only %d of 2 processes started: %v", len(started), started)
 	}
-	if started[0] != "sidecar" || started[1] != "engine" {
-		t.Fatalf("start order was %v", started)
+	// Which of the two appended first is deliberately not asserted. The
+	// supervisor spawns the sidecar first, but spawning is not sequencing:
+	// nothing here makes the engine wait, so two shells racing to append give
+	// either order and a test that demanded one would fail for the scheduler's
+	// reasons rather than the supervisor's. The wait that is a real guarantee
+	// is WaitForLeaderPort, and TestWaitForLeaderPortBlocksUntilListening is
+	// where it is checked.
+	if !slices.Contains(started, "sidecar") || !slices.Contains(started, "engine") {
+		t.Fatalf("expected both processes to have run, got %v", started)
 	}
 
 	// The shard's pid is the engine's, not the sidecar's: the engine's exit is

internal/agent/weights/fetch.go

@@ -15,8 +15,6 @@ package weights
 
 import (
 	"context"
-	"crypto/sha256"
-	"encoding/hex"
 	"errors"
 	"fmt"
 	"io"
@@ -86,6 +84,11 @@ func (f *Fetcher) Ensure(ctx context.Context, source string) (string, error) {
 	if source == "" {
 		return "", fmt.Errorf("no source given")
 	}
+	// A version rather than a file. Everything below assumes one URL is one
+	// file, and a set has to be handled before any of it.
+	if IsSet(source) {
+		return f.ensureSet(ctx, source)
+	}
 	if !isRemote(source) {
 		if !isPath(source) {
 			// A bare model id — "Qwen/Qwen2.5-7B-Instruct" — is not a file and
@@ -136,6 +139,15 @@ func (f *Fetcher) Ensure(ctx context.Context, source string) (string, error) {
 
 // fetch downloads a source to a target, resuming and verifying.
 func (f *Fetcher) fetch(ctx context.Context, source, target string) (string, error) {
+	return f.fetchAs(ctx, source, source, target)
+}
+
+// fetchAs downloads one URL but reports progress under another name.
+//
+// The two differ for a set: the shards waiting on it asked for the version, not
+// for the fourth file in it, and progress reported under the file's own URL
+// reaches nobody. What they see is one number climbing across the whole set.
+func (f *Fetcher) fetchAs(ctx context.Context, source, reportAs, target string) (string, error) {
 	url, err := resolveURL(source)
 	if err != nil {
 		return "", err
@@ -194,8 +206,8 @@ func (f *Fetcher) fetch(ctx context.Context, source, target string) (string, err
 	}
 
 	total := existing + resp.ContentLength
-	f.report(Progress{Source: source, BytesDone: existing, BytesTotal: total})
-	written, err := f.copyWithProgress(ctx, out, resp.Body, source, existing, total)
+	f.report(Progress{Source: reportAs, BytesDone: existing, BytesTotal: total})
+	written, err := f.copyWithProgress(ctx, out, resp.Body, reportAs, existing, total)
 	closeErr := out.Close()
 	if err != nil {
 		// The part file is kept on purpose, so the next attempt resumes.
@@ -262,12 +274,7 @@ func (f *Fetcher) cachePath(source string) (string, error) {
 	if f.CacheDir == "" {
 		return "", fmt.Errorf("no cache directory configured, so %s cannot be fetched", source)
 	}
-	sum := sha256.Sum256([]byte(source))
-	name := filepath.Base(source)
-	if name == "" || name == "." || name == "/" {
-		name = "weights"
-	}
-	return filepath.Join(f.CacheDir, hex.EncodeToString(sum[:8])+"-"+name), nil
+	return CachePathFor(f.CacheDir, source), nil
 }
 
 // isPath reports whether a source names a file on this machine.

internal/agent/weights/gc.go

@@ -0,0 +1,99 @@
+package weights
+
+import (
+	"crypto/sha256"
+	"encoding/hex"
+	"path/filepath"
+	"regexp"
+	"sort"
+	"strings"
+)
+
+// cacheNamePattern matches a filename this package created.
+//
+// The eight-byte hash of the source, hex encoded, then a dash, then the
+// original filename — exactly what cachePath writes. It is the only mark of
+// ownership there is, and it has to be exact: everything else in a model
+// directory was put there by the operator, and deleting one of those files
+// could cost a week of someone's bandwidth.
+var cacheNamePattern = regexp.MustCompile(`^[0-9a-f]{16}-`)
+
+// CachePathFor is where a source would be cached in the given directory.
+//
+// Exported so the control plane can work out which files in a node's reported
+// list are ones llmgw fetched for a model it still wants, without asking the
+// node anything. The agent and the control plane must agree on this name
+// exactly, which is why there is one function and not two.
+func CachePathFor(dir, source string) string {
+	sum := sha256.Sum256([]byte(source))
+	name := filepath.Base(source)
+	if name == "" || name == "." || name == "/" {
+		name = "weights"
+	}
+	return filepath.Join(dir, hex.EncodeToString(sum[:8])+"-"+name)
+}
+
+// Owned reports whether a path is one this package created.
+//
+// The dividing line of the whole feature. A file without the cache name was
+// put there by the operator — by scp, by a shared mount, by hand — and llmgw
+// has no business deleting it however unreferenced it looks. A file with the
+// cache name is one llmgw downloaded, and llmgw can take it back.
+func Owned(path string) bool {
+	return cacheNamePattern.MatchString(filepath.Base(path))
+}
+
+// Orphans lists cached files that no wanted source accounts for.
+//
+// `have` is what a node reports on disk and `wanted` is every source in the
+// catalog; both are passed in so this is a calculation rather than a scan, and
+// so it can be run on the control plane against a node's reported inventory.
+//
+// Only files this package created are ever returned. A source that is not
+// fetched — a plain path, or a model id an engine resolves itself — produces
+// no cache name and so protects nothing and deletes nothing, which is correct:
+// those files are not in the cache to begin with.
+func Orphans(have []string, wanted []string) []string {
+	keep := map[string]struct{}{}
+	for _, src := range wanted {
+		if !isRemote(src) {
+			// Nothing was fetched for it, so nothing in the cache belongs to
+			// it. Computing a name here would reserve a file that does not
+			// exist and could never be an orphan anyway.
+			continue
+		}
+		// By base name rather than full path: the control plane does not know
+		// each node's cache directory, and the name is what identifies the
+		// file wherever the node keeps it.
+		keep[filepath.Base(CachePathFor("", src))] = struct{}{}
+	}
+
+	var out []string
+	seen := map[string]struct{}{}
+	for _, path := range have {
+		base := filepath.Base(path)
+		if !Owned(base) {
+			continue
+		}
+		if _, wanted := keep[base]; wanted {
+			continue
+		}
+		if _, dup := seen[path]; dup {
+			continue
+		}
+		seen[path] = struct{}{}
+		out = append(out, path)
+	}
+	sort.Strings(out)
+	return out
+}
+
+// PartialName reports whether a path is the temporary file a fetch writes to.
+//
+// Those are not orphans in the same sense — a fetch may be running right now —
+// so they are named separately rather than swept up with the rest. A `.part`
+// left behind by a killed agent is real garbage, but only its own node can
+// tell whether it is stale, and nothing here can.
+func PartialName(path string) bool {
+	return strings.HasSuffix(path, ".part")
+}

internal/agent/weights/gc_test.go

@@ -0,0 +1,116 @@
+package weights
+
+import (
+	"path/filepath"
+	"slices"
+	"testing"
+)
+
+func TestOnlyFilesLLMGWFetchedAreEverOrphans(t *testing.T) {
+	// The invariant the whole feature rests on. Everything without the cache
+	// name was put there by the operator — scp, a shared mount, a copy by hand
+	// — and deleting one could cost a week of someone's bandwidth. It does not
+	// matter how unreferenced it looks.
+	const dir = "/var/lib/llmgw-models"
+	have := []string{
+		filepath.Join(dir, "qwen2.5-coder-14b-instruct-q8_0-00001-of-00002.gguf"),
+		filepath.Join(dir, "some-model.safetensors"),
+		filepath.Join(dir, "notes.txt"),
+		CachePathFor(dir, "https://models.internal/gone.gguf"),
+	}
+
+	got := Orphans(have, nil)
+	if len(got) != 1 {
+		t.Fatalf("expected only the fetched file, got %v", got)
+	}
+	if !Owned(got[0]) {
+		t.Errorf("returned a file llmgw did not fetch: %s", got[0])
+	}
+}
+
+func TestAFetchedFileStillInTheCatalogIsKept(t *testing.T) {
+	const dir = "/var/lib/llmgw-models"
+	const wanted = "hf:Qwen/Qwen2.5-Coder-3B-Instruct-GGUF/q4.gguf"
+	const gone = "https://models.internal/old.gguf"
+
+	have := []string{CachePathFor(dir, wanted), CachePathFor(dir, gone)}
+	got := Orphans(have, []string{wanted})
+
+	if slices.Contains(got, CachePathFor(dir, wanted)) {
+		t.Error("a file the catalog still points at was called an orphan")
+	}
+	if !slices.Contains(got, CachePathFor(dir, gone)) {
+		t.Errorf("a file nothing points at was not found: %v", got)
+	}
+}
+
+func TestTheCacheNameMatchesWhatAFetchActuallyWrites(t *testing.T) {
+	// The control plane proposes deletions by recomputing this name from the
+	// catalog. If it ever drifted from what a fetch writes, the proposal would
+	// name a file that is in use — so the two must be one function, and this
+	// is what says so.
+	f := &Fetcher{CacheDir: "/cache"}
+	const source = "https://models.internal/x.gguf"
+
+	got, err := f.cachePath(source)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if want := CachePathFor("/cache", source); got != want {
+		t.Errorf("a fetch writes %q but the catalog would look for %q", got, want)
+	}
+}
+
+func TestOneOfTwoLookalikeFilesIsNotMistakenForTheOther(t *testing.T) {
+	// That the two cache to different names is covered where the naming lives.
+	// What matters here is the consequence: deleting the unreferenced one must
+	// not take the referenced one with it, and both end in "model.gguf".
+	a := CachePathFor("/cache", "https://one.example/model.gguf")
+	b := CachePathFor("/cache", "https://two.example/model.gguf")
+
+	got := Orphans([]string{a, b}, []string{"https://one.example/model.gguf"})
+	if !slices.Equal(got, []string{b}) {
+		t.Errorf("orphans = %v, want only %s", got, b)
+	}
+}
+
+func TestASourceThatIsNeverFetchedProtectsNothingAndDeletesNothing(t *testing.T) {
+	// A plain path, or a model id an engine resolves itself, puts nothing in
+	// the cache. Computing a cache name for it would reserve a file that does
+	// not exist, and it must not stop a real orphan being found either.
+	const dir = "/models"
+	orphan := CachePathFor(dir, "https://models.internal/gone.gguf")
+
+	got := Orphans([]string{orphan, filepath.Join(dir, "local.gguf")},
+		[]string{"/models/local.gguf", "Qwen/Qwen2.5-7B-Instruct"})
+
+	if !slices.Equal(got, []string{orphan}) {
+		t.Errorf("orphans = %v, want only %s", got, orphan)
+	}
+}
+
+func TestAnEmptyCatalogDoesNotMakeOperatorFilesDisposable(t *testing.T) {
+	// The dangerous case: every model removed from the catalog at once. Files
+	// llmgw fetched become orphans, which is right. Files it did not must stay
+	// untouched, which is the whole point.
+	const dir = "/models"
+	have := []string{
+		filepath.Join(dir, "hand-placed.gguf"),
+		CachePathFor(dir, "https://models.internal/a.gguf"),
+	}
+	got := Orphans(have, nil)
+	if len(got) != 1 || got[0] != CachePathFor(dir, "https://models.internal/a.gguf") {
+		t.Errorf("orphans = %v", got)
+	}
+}
+
+func TestPartialsAreNotSweptUpWithTheRest(t *testing.T) {
+	// A fetch may be running right now, and only the node itself can tell a
+	// live download from one a killed agent left behind.
+	if !PartialName("/models/abc-model.gguf.part") {
+		t.Error("a .part file was not recognised")
+	}
+	if PartialName("/models/abc-model.gguf") {
+		t.Error("a finished file was called partial")
+	}
+}

internal/agent/weights/set.go

@@ -0,0 +1,129 @@
+package weights
+
+import (
+	"context"
+	"encoding/json"
+	"fmt"
+	"io"
+	"net/http"
+	"os"
+	"path/filepath"
+	"strings"
+)
+
+// IsSet reports whether a source names a set of files rather than one.
+//
+// By the trailing slash, which is the whole signal: "…/v1/model.gguf" is a file
+// and "…/v1/" is the version it belongs to. A flag beside the source would be a
+// second place for the two to disagree, and the URL already distinguishes a
+// directory from a file everywhere else on the web.
+func IsSet(source string) bool {
+	return isRemote(source) && strings.HasSuffix(source, "/")
+}
+
+// manifest is what the mirror answers for a version.
+type manifest struct {
+	Model   string `json:"model"`
+	Version string `json:"version"`
+	Files   []struct {
+		Name  string `json:"name"`
+		Bytes int64  `json:"bytes"`
+	} `json:"files"`
+	Bytes int64 `json:"bytes"`
+}
+
+// ensureSet downloads every file of a version into one directory.
+//
+// A directory, because that is what a multi-file model is: vLLM is handed the
+// folder and reads the shards and the config out of it, and llama.cpp is handed
+// the first part and finds the rest beside it. Downloading them into one place
+// is what makes either possible, and a catalog entry naming one file could
+// express neither.
+//
+// The whole set or nothing. A directory with three of four shards in it loads
+// as a broken model rather than failing to load, so a partial set is left
+// unfinished under its temporary name and the next attempt starts again.
+func (f *Fetcher) ensureSet(ctx context.Context, source string) (string, error) {
+	dir, err := f.cachePath(strings.TrimSuffix(source, "/"))
+	if err != nil {
+		return "", err
+	}
+	// A complete set is marked by a file written last. Checking for the
+	// directory alone would treat an interrupted download as finished, which is
+	// exactly the failure the marker exists to prevent.
+	done := filepath.Join(dir, ".complete")
+	if _, err := os.Stat(done); err == nil {
+		return dir, nil
+	}
+
+	m, err := f.manifestFor(ctx, source)
+	if err != nil {
+		return "", err
+	}
+	if len(m.Files) == 0 {
+		return "", fmt.Errorf("%s holds no files", source)
+	}
+
+	if err := os.MkdirAll(dir, 0o755); err != nil {
+		return "", err
+	}
+	for _, file := range m.Files {
+		// Rejected rather than joined: a name with a separator in it would
+		// write outside the directory the mirror said it was in, and the
+		// mirror is a place an operator copies files into by hand.
+		if strings.ContainsAny(file.Name, `/\`) || file.Name == ".." {
+			return "", fmt.Errorf("%s: the mirror offered a file named %q", source, file.Name)
+		}
+		target := filepath.Join(dir, file.Name)
+		if st, err := os.Stat(target); err == nil && st.Size() == file.Bytes {
+			continue
+		}
+		// Reported under the set's own address, because that is what the
+		// shards are waiting on. Reusing fetch rather than writing a second
+		// downloader keeps resume, the .part rename and the progress callback
+		// in one place.
+		if _, err := f.fetchAs(ctx, source+file.Name, source, target); err != nil {
+			return "", err
+		}
+	}
+
+	if err := os.WriteFile(done, []byte(source+"\n"), 0o644); err != nil {
+		return "", err
+	}
+	return dir, nil
+}
+
+// manifestFor asks the mirror what a version consists of.
+func (f *Fetcher) manifestFor(ctx context.Context, source string) (manifest, error) {
+	req, err := http.NewRequestWithContext(ctx, http.MethodGet, source, nil)
+	if err != nil {
+		return manifest{}, err
+	}
+	client := f.Client
+	if client == nil {
+		client = &http.Client{}
+	}
+	resp, err := client.Do(req)
+	if err != nil {
+		return manifest{}, fmt.Errorf("asking %s what it holds: %w", source, err)
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode != http.StatusOK {
+		return manifest{}, fmt.Errorf("%s answered %d", source, resp.StatusCode)
+	}
+
+	var m manifest
+	body, err := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
+	if err != nil {
+		return manifest{}, err
+	}
+	if err := json.Unmarshal(body, &m); err != nil {
+		// The likely cause is a plain file server with a directory index, which
+		// answers HTML. Said plainly, because "invalid character '<'" sends
+		// somebody looking at the wrong thing.
+		return manifest{}, fmt.Errorf(
+			"%s did not answer with a manifest; a source ending in / has to be a "+
+				"mirror version, not a directory listing", source)
+	}
+	return m, nil
+}

internal/agent/weights/set_test.go

@@ -0,0 +1,167 @@
+package weights
+
+import (
+	"fmt"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+)
+
+// mirrorOf serves a manifest at "/m/v/" and the files beneath it.
+func mirrorOf(t *testing.T, files map[string]string) (*Fetcher, *httptest.Server, *int) {
+	t.Helper()
+	hits := 0
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		if strings.HasSuffix(r.URL.Path, "/") {
+			var b strings.Builder
+			b.WriteString(`{"model":"m","version":"v","files":[`)
+			first := true
+			total := 0
+			for name, body := range files {
+				if !first {
+					b.WriteString(",")
+				}
+				first = false
+				fmt.Fprintf(&b, `{"name":%q,"bytes":%d}`, name, len(body))
+				total += len(body)
+			}
+			fmt.Fprintf(&b, `],"bytes":%d}`, total)
+			_, _ = w.Write([]byte(b.String()))
+			return
+		}
+		body, ok := files[strings.TrimPrefix(r.URL.Path, "/m/v/")]
+		if !ok {
+			http.NotFound(w, r)
+			return
+		}
+		hits++
+		_, _ = w.Write([]byte(body))
+	}))
+	t.Cleanup(srv.Close)
+	return &Fetcher{CacheDir: t.TempDir()}, srv, &hits
+}
+
+func TestASetIsFetchedIntoOneDirectory(t *testing.T) {
+	f, srv, _ := mirrorOf(t, map[string]string{
+		"model-00001-of-00002.safetensors": "aaaa",
+		"model-00002-of-00002.safetensors": "bb",
+		"config.json":                      "{}",
+	})
+
+	got, err := f.Ensure(t.Context(), srv.URL+"/m/v/")
+	if err != nil {
+		t.Fatal(err)
+	}
+	st, err := os.Stat(got)
+	if err != nil || !st.IsDir() {
+		t.Fatalf("expected a directory, got %q: %v", got, err)
+	}
+	// A directory, because that is what a multi-file model is: vLLM reads the
+	// shards and the config out of the folder, and a catalog entry naming one
+	// file could express neither.
+	for _, name := range []string{"model-00001-of-00002.safetensors", "config.json"} {
+		if _, err := os.Stat(filepath.Join(got, name)); err != nil {
+			t.Errorf("%s did not arrive: %v", name, err)
+		}
+	}
+}
+
+func TestAFinishedSetIsNotFetchedAgain(t *testing.T) {
+	f, srv, hits := mirrorOf(t, map[string]string{"a.gguf": "aaaa", "b.gguf": "bb"})
+
+	if _, err := f.Ensure(t.Context(), srv.URL+"/m/v/"); err != nil {
+		t.Fatal(err)
+	}
+	after := *hits
+	if _, err := f.Ensure(t.Context(), srv.URL+"/m/v/"); err != nil {
+		t.Fatal(err)
+	}
+	if *hits != after {
+		t.Errorf("the second call downloaded again: %d then %d", after, *hits)
+	}
+}
+
+func TestAHalfFetchedSetIsNotTreatedAsFinished(t *testing.T) {
+	f, srv, _ := mirrorOf(t, map[string]string{"a.gguf": "aaaa", "b.gguf": "bb"})
+	dir, err := f.cachePath(srv.URL + "/m/v")
+	if err != nil {
+		t.Fatal(err)
+	}
+	// One file present, no completion marker — an interrupted download.
+	if err := os.MkdirAll(dir, 0o755); err != nil {
+		t.Fatal(err)
+	}
+	if err := os.WriteFile(filepath.Join(dir, "a.gguf"), []byte("aaaa"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+
+	got, err := f.Ensure(t.Context(), srv.URL+"/m/v/")
+	if err != nil {
+		t.Fatal(err)
+	}
+	// A directory with three of four shards loads as a broken model rather
+	// than failing to load, so the marker is what says the set is whole.
+	if _, err := os.Stat(filepath.Join(got, "b.gguf")); err != nil {
+		t.Errorf("the missing file was not fetched: %v", err)
+	}
+}
+
+func TestAFileAlreadyThereAtTheRightSizeIsNotRefetched(t *testing.T) {
+	f, srv, hits := mirrorOf(t, map[string]string{"a.gguf": "aaaa", "b.gguf": "bb"})
+	dir, _ := f.cachePath(srv.URL + "/m/v")
+	_ = os.MkdirAll(dir, 0o755)
+	_ = os.WriteFile(filepath.Join(dir, "a.gguf"), []byte("aaaa"), 0o644)
+
+	if _, err := f.Ensure(t.Context(), srv.URL+"/m/v/"); err != nil {
+		t.Fatal(err)
+	}
+	// Resuming a set means not paying for the shards that already landed.
+	if *hits != 1 {
+		t.Errorf("expected only the missing file to be fetched, got %d downloads", *hits)
+	}
+}
+
+func TestAFilenameThatEscapesTheDirectoryIsRefused(t *testing.T) {
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+		_, _ = w.Write([]byte(`{"files":[{"name":"../../etc/passwd","bytes":1}]}`))
+	}))
+	defer srv.Close()
+	f := &Fetcher{CacheDir: t.TempDir()}
+
+	_, err := f.Ensure(t.Context(), srv.URL+"/m/v/")
+	// The mirror is a place an operator copies files into by hand, so what it
+	// reports is not automatically safe to join onto a path.
+	if err == nil || !strings.Contains(err.Error(), "offered a file named") {
+		t.Errorf("expected the name to be refused, got %v", err)
+	}
+}
+
+func TestADirectoryListingIsNotAManifest(t *testing.T) {
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+		_, _ = w.Write([]byte("<pre><a href=\"a.gguf\">a.gguf</a></pre>"))
+	}))
+	defer srv.Close()
+	f := &Fetcher{CacheDir: t.TempDir()}
+
+	_, err := f.Ensure(t.Context(), srv.URL+"/m/v/")
+	// "invalid character '<'" sends somebody looking at the wrong thing.
+	if err == nil || !strings.Contains(err.Error(), "has to be a mirror version") {
+		t.Errorf("the reason should name the cause, got %v", err)
+	}
+}
+
+func TestOnlyATrailingSlashMeansASet(t *testing.T) {
+	for src, want := range map[string]bool{
+		"http://mirror/m/v/":           true,
+		"http://mirror/m/v/model.gguf": false,
+		"/var/lib/models/":             false, // a local directory is not fetched
+		"Qwen/Qwen2.5-7B-Instruct":     false, // a repo id, resolved by the engine
+	} {
+		if got := IsSet(src); got != want {
+			t.Errorf("IsSet(%q) = %v, want %v", src, got, want)
+		}
+	}
+}

internal/auth/auth_test.go

@@ -16,7 +16,7 @@ import (
 )
 
 // fixture builds a store with one org/team/project and one key.
-func fixture(t *testing.T) (*store.SQLite, string, string) {
+func fixture(t *testing.T) (*store.DB, string, string) {
 	t.Helper()
 	ctx := context.Background()
 	s, err := store.OpenSQLite(ctx, filepath.Join(t.TempDir(), "t.db"))

internal/bench/capacity.go

@@ -0,0 +1,77 @@
+package bench
+
+import "time"
+
+// Capacity is what one replica was measured to serve, kept so the measurement
+// outlives the terminal it was printed in.
+//
+// The conclusion of a run, not the run. The ramp itself — every concurrency
+// level, its latencies, its errors — is worth reading once, while deciding
+// whether the run was sound; it is not worth carrying around afterwards, and
+// storing it would grow without bound as a model is re-benchmarked. What a
+// later reader needs is the number and enough provenance to judge whether it is
+// still true.
+type Capacity struct {
+	Model   string `json:"model"`
+	Profile string `json:"profile"`
+
+	// MaxConcurrencyAtSLO is the capacity: the highest concurrency that still
+	// met the target. Zero means even one request did not, which is a real
+	// result and not a missing one — it says this hardware cannot serve this
+	// model at this SLO at all.
+	MaxConcurrencyAtSLO int `json:"max_concurrency_at_slo"`
+	// ThroughputTokPerSec and the latencies are taken at that level, not at the
+	// peak. Peak throughput arrives long after latency has become unusable, so
+	// reporting it would overstate what the hardware can serve.
+	ThroughputTokPerSec float64       `json:"throughput_tok_per_sec"`
+	TTFTp99             time.Duration `json:"ttft_p99"`
+	TPOTp99             time.Duration `json:"tpot_p99"`
+
+	// SuggestedTargetWaiting and SuggestedKVCeiling are the queue depth and
+	// cache pressure seen while the SLO was still being met, which is what
+	// admission should treat as healthy for this model. Admission currently
+	// applies one global default to every model; these are the per-model
+	// numbers it should be using instead.
+	SuggestedTargetWaiting int     `json:"suggested_target_waiting"`
+	SuggestedKVCeiling     float64 `json:"suggested_kv_ceiling"`
+
+	// SLO is recorded because capacity is meaningless without it. The same
+	// replica has a different capacity against a 200ms first-token target than
+	// against a 2s one, and a number stored without the target it was measured
+	// against cannot be compared with anything.
+	SLO SLO `json:"slo"`
+
+	// Target is the endpoint the load actually went to, kept so a profile
+	// measured against the gateway is never mistaken for one measured against
+	// a replica. Only the second says anything about the hardware.
+	Target     string    `json:"target"`
+	MeasuredAt time.Time `json:"measured_at"`
+	// Stale is not a field. Whether a measurement still holds depends on
+	// whether the hardware and the engine build have changed since, which this
+	// package cannot know; MeasuredAt is given so the reader can decide.
+}
+
+// Capacity summarizes a completed run for storage.
+func (r Result) Capacity(profile string) Capacity {
+	return Capacity{
+		Model:                  r.Model,
+		Profile:                profile,
+		MaxConcurrencyAtSLO:    r.MaxConcurrencyAtSLO,
+		ThroughputTokPerSec:    r.ThroughputTokPerSec,
+		TTFTp99:                r.TTFTp99,
+		TPOTp99:                r.TPOTp99,
+		SuggestedTargetWaiting: r.SuggestedTargetWaiting,
+		SuggestedKVCeiling:     r.SuggestedKVCeiling,
+		SLO:                    r.SLO,
+		Target:                 r.Target,
+		MeasuredAt:             r.RanAt,
+	}
+}
+
+// Usable reports whether this measurement says anything.
+//
+// A run that never met the SLO, or one stopped before it finished a level,
+// concludes at zero. That is worth recording — it is the answer to "can this
+// run here at all" — but a caller sizing a model against it would compute a
+// replica count of infinity, so it has to be asked about rather than used.
+func (c Capacity) Usable() bool { return c.MaxConcurrencyAtSLO > 0 }

internal/cli/bench.go

@@ -3,14 +3,16 @@ package cli
 import (
 	"encoding/json"
 	"fmt"
+	"strings"
 	"time"
 
 	"github.com/spf13/cobra"
 
 	"gitea.mynodes.xyz/ilker/llmgw/internal/bench"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
 )
 
-func newBenchCmd(asJSON *bool) *cobra.Command {
+func newBenchCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 	cfg := bench.DefaultConfig()
 	var (
 		steps   []int
@@ -18,6 +20,7 @@ func newBenchCmd(asJSON *bool) *cobra.Command {
 		metrics string
 		engine  string
 		key     string
+		save    bool
 	)
 
 	cmd := &cobra.Command{
@@ -78,6 +81,15 @@ func newBenchCmd(asJSON *bool) *cobra.Command {
 			if err != nil {
 				return err
 			}
+			if save {
+				if err := saveCapacity(cmd, client, res, target); err != nil {
+					// Reported, not fatal. The run has already cost the GPU
+					// time; losing its conclusion to a failed write would mean
+					// running it again, and the numbers are on screen either
+					// way.
+					cmd.PrintErrf("the measurement was not saved: %v\n", err)
+				}
+			}
 			if *asJSON {
 				return json.NewEncoder(cmd.OutOrStdout()).Encode(res)
 			}
@@ -99,9 +111,63 @@ func newBenchCmd(asJSON *bool) *cobra.Command {
 		"send every request the same prompt, measuring the cached case instead of the hardware")
 	f.DurationVar(&cfg.SLO.TTFTp99, "ttft-p99", cfg.SLO.TTFTp99, "time-to-first-token target")
 	f.DurationVar(&cfg.SLO.TPOTp99, "tpot-p99", cfg.SLO.TPOTp99, "per-output-token target")
+	f.BoolVar(&save, "save", false,
+		"record the capacity against the model and profile, so the number outlives this terminal")
 	return cmd
 }
 
+// saveCapacity records a run's conclusion against the profile it was measured
+// on.
+//
+// The profile is looked up from the replica rather than asked for. Capacity
+// belongs to the hardware and the engine arguments as much as to the weights,
+// so a measurement filed under the wrong profile is worse than none — and the
+// control plane already knows which profile is answering on which endpoint, so
+// nobody needs to be trusted to type it.
+func saveCapacity(cmd *cobra.Command, client func() *ctlClient, res bench.Result, target string) error {
+	var reps []adminapi.ReplicaView
+	if err := client().get(cmd.Context(), "/api/v1/replicas", &reps); err != nil {
+		return err
+	}
+	profile := ""
+	for _, rep := range reps {
+		if rep.Model == res.Model && sameEndpoint(rep.Endpoint, target) {
+			profile = rep.Profile
+			break
+		}
+	}
+	if profile == "" {
+		// Almost always because --target was the gateway. Benchmarking the
+		// whole path is a useful thing to do and says nothing about what one
+		// replica can serve, so there is nothing here worth filing as capacity.
+		return fmt.Errorf("%s is not a replica of %s, so this run measures the path rather than the hardware; "+
+			"point --target at a replica endpoint from `llmgw ctl replicas`", target, res.Model)
+	}
+
+	body, err := json.Marshal(res.Capacity(profile))
+	if err != nil {
+		return err
+	}
+	var saved bench.Capacity
+	if err := client().put(cmd.Context(),
+		"/api/v1/models/"+res.Model+"/capacity", body, &saved); err != nil {
+		return err
+	}
+	cmd.PrintErrf("saved: %s on profile %s, %d concurrent at the SLO\n",
+		saved.Model, saved.Profile, saved.MaxConcurrencyAtSLO)
+	return nil
+}
+
+// sameEndpoint compares two base URLs, ignoring a trailing slash.
+//
+// Written out because the two come from different places: one is what the
+// control plane recorded, the other is what somebody typed or pasted, and a
+// trailing slash is the difference that would silently make every run look like
+// it was aimed at the gateway.
+func sameEndpoint(a, b string) bool {
+	return strings.TrimSuffix(a, "/") == strings.TrimSuffix(b, "/")
+}
+
 func printConclusion(cmd *cobra.Command, res bench.Result, cfg bench.Config) {
 	out := cmd.OutOrStdout()
 	fmt.Fprintln(out)

internal/cli/capacity.go

@@ -0,0 +1,77 @@
+package cli
+
+import (
+	"encoding/json"
+	"fmt"
+	"text/tabwriter"
+	"time"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+)
+
+// newCapacityCmd shows what benchmarks have found out.
+//
+// `ctl bench` has always been able to measure this; until it could be saved,
+// the number lived in whoever ran it's scrollback. This is the other half:
+// somewhere to read it back, so sizing a model is looking something up rather
+// than re-running an hour of load.
+func newCapacityCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
+	return &cobra.Command{
+		Use:   "capacity",
+		Short: "What a replica of each model was measured to serve",
+		Long: "Recorded by `llmgw ctl bench --save`, per model and profile — the same\n" +
+			"model on CPU and on a GPU are different numbers.\n\n" +
+			"Age matters. Hardware is replaced and engine builds change, so a\n" +
+			"measurement is an observation with a date on it rather than a fact about\n" +
+			"the model. The column is there to be read.",
+		Args: cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			var rows []adminapi.CapacityView
+			if err := client().get(cmd.Context(), "/api/v1/capacity", &rows); err != nil {
+				return err
+			}
+			if *asJSON {
+				return json.NewEncoder(cmd.OutOrStdout()).Encode(rows)
+			}
+			if len(rows) == 0 {
+				fmt.Fprintln(cmd.OutOrStdout(),
+					"Nothing has been measured. Run `llmgw ctl bench <model> --target <replica> --save`.")
+				return nil
+			}
+
+			w := tabwriter.NewWriter(cmd.OutOrStdout(), 0, 0, 2, ' ', 0)
+			fmt.Fprintln(w, "MODEL\tPROFILE\tCONCURRENT\tTOK/S\tTTFT p99\tTPOT p99\tWAIT\tKV\tMEASURED\tBY")
+			for _, c := range rows {
+				concurrent := fmt.Sprint(c.MaxConcurrencyAtSLO)
+				if !c.Usable() {
+					// Zero is a real result — this hardware cannot serve this
+					// model at this SLO — and printing it as a number invites
+					// somebody to divide by it.
+					concurrent = "none at SLO"
+				}
+				fmt.Fprintf(w, "%s\t%s\t%s\t%.0f\t%s\t%s\t%d\t%.0f%%\t%s\t%s\n",
+					c.Model, c.Profile, concurrent, c.ThroughputTokPerSec,
+					ms(c.TTFTp99), ms(c.TPOTp99),
+					c.SuggestedTargetWaiting, c.SuggestedKVCeiling*100,
+					age(c.MeasuredAt), or(c.MeasuredBy, "—"))
+			}
+			return w.Flush()
+		},
+	}
+}
+
+func age(t time.Time) string {
+	if t.IsZero() {
+		return "unknown"
+	}
+	return time.Since(t).Round(time.Minute).String() + " ago"
+}
+
+func or(v, fallback string) string {
+	if v == "" {
+		return fallback
+	}
+	return v
+}

internal/cli/ctl.go

@@ -43,6 +43,12 @@ func (c *ctlClient) post(ctx context.Context, path string, body []byte, out any)
 	return c.do(ctx, http.MethodPost, path, json.RawMessage(body), out)
 }
 
+// put replaces a resource. Used where a second call is a better answer to the
+// same question rather than another one: re-measuring a model's capacity.
+func (c *ctlClient) put(ctx context.Context, path string, body []byte, out any) error {
+	return c.do(ctx, http.MethodPut, path, json.RawMessage(body), out)
+}
+
 func (c *ctlClient) delete(ctx context.Context, path string, out any) error {
 	return c.do(ctx, http.MethodDelete, path, nil, out)
 }
@@ -121,55 +127,100 @@ func newCtlCmd() *cobra.Command {
 			"Move work off a node, one replica at a time"),
 		// cert works on files rather than the admin API, so --server does not
 		// apply to it; it lives here because this is where operators look.
-		newBenchCmd(&asJSON),
+		newBenchCmd(client, &asJSON),
 		newStatusCmd(client, &asJSON),
 		newModelCmd(client),
+		newScaleCmd(client),
 		newEngineCheckCmd(),
 		newToolCheckCmd(),
+		newWeightsCmd(client, &asJSON),
+		newCapacityCmd(client, &asJSON),
+		newUsageCmd(client, &asJSON),
 		newCertCmd(),
 	)
 	return cmd
 }
 
 func newNodesCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
-	return &cobra.Command{
+	var f nodeFilter
+	c := &cobra.Command{
 		Use:   "nodes",
 		Short: "List nodes and their capacity",
+		Long: "On a fleet too large to read a whole table of, the filters are how a\n" +
+			"question gets asked: which nodes on this fabric, which of this GPU class,\n" +
+			"which are not ready. They combine, and they apply to --json too.",
 		RunE: func(cmd *cobra.Command, _ []string) error {
 			var nodes []adminapi.NodeView
 			if err := client().get(cmd.Context(), "/api/v1/nodes", &nodes); err != nil {
 				return err
 			}
+			total := len(nodes)
+			nodes, err := f.apply(nodes)
+			if err != nil {
+				return err
+			}
 			if *asJSON {
 				return emitJSON(nodes)
 			}
 			if len(nodes) == 0 {
+				// A filter that matched nothing and an empty fleet call for
+				// opposite next steps: one is a cluster to look at, the other
+				// is a word to correct.
+				if !f.empty() && total > 0 {
+					fmt.Println(noMatches("nodes", f.describe(), total))
+					return nil
+				}
 				fmt.Println("No nodes have registered.")
 				return nil
 			}
 
 			w := tab()
-			fmt.Fprintln(w, "NODE\tSTATE\tGPUS\tMEM\tCLASSES\tFABRICS\tREPLICAS\tLAST SEEN")
+			fmt.Fprintln(w, "NODE\tSTATE\tGPUS\tMEM\tDISK\tENGINES\tCLASSES\tFABRICS\tREPLICAS\tLAST SEEN")
 			for _, n := range nodes {
-				fmt.Fprintf(w, "%s\t%s\t%d/%d free\t%s\t%s\t%s\t%d\t%s\n",
+				fmt.Fprintf(w, "%s\t%s\t%d/%d free\t%s\t%s\t%s\t%s\t%s\t%d\t%s\n",
 					n.ID, n.State, n.GPUFree, n.GPUTotal, memColumn(n.MemoryGB),
+					diskColumn(n.DiskFreeGB, n.DiskTotalGB),
+					// What the node can run, which is the first thing to look
+					// at when a model will not place and the hardware looks
+					// fine.
+					join(n.Engines),
 					join(n.Classes), join(n.Fabrics), n.Replicas, n.LastSeenAgo)
 			}
 			_ = w.Flush()
 
 			// Unhealthy GPUs are usually the answer when a node looks idle but
-			// nothing schedules onto it.
+			// nothing schedules onto it. An engine still installing, or one
+			// that failed to, is the other usual answer.
+			said := false
 			for _, n := range nodes {
 				for _, p := range n.Unhealthy {
 					fmt.Printf("\n%s: %s (kept out of placement)", n.ID, p)
+					said = true
+				}
+				for _, p := range n.Provision {
+					if p.State == "ready" {
+						continue
+					}
+					fmt.Printf("\n%s: %s %s %s: %s", n.ID, p.Engine, p.Version, p.State, p.Detail)
+					said = true
+				}
+				for _, w := range n.Warnings {
+					fmt.Printf("\n%s: %s", n.ID, w)
+					said = true
 				}
 			}
-			if hasUnhealthy(nodes) {
+			if said {
 				fmt.Println()
 			}
 			return nil
 		},
 	}
+	c.Flags().StringVar(&f.fabric, "fabric", "", "only nodes attached to this fabric")
+	c.Flags().StringVar(&f.class, "class", "", "only nodes with this GPU class")
+	c.Flags().StringVar(&f.state, "state", "", "only nodes in this state (prefix match)")
+	c.Flags().StringArrayVar(&f.labels, "label", nil,
+		"only nodes carrying key=value; repeatable, and key alone matches any value")
+	return c
 }
 
 // modelStatus says what is happening to a model, in the words that decide what
@@ -221,6 +272,23 @@ func memColumn(gb int) string {
 	return fmt.Sprintf("%dG", gb)
 }
 
+// diskColumn renders free space against the size it is free out of.
+//
+// Both, because they answer different questions and the first is useless
+// without the second: 8 GB free is comfortable on a 10 TB filer and nearly
+// full on a 10 GB root partition, and it is the second case that stops a
+// model from being placed.
+//
+// A dash for a node that reported nothing, which an older agent and a node
+// with no model directories both do. It has to read as "not known" rather
+// than as "no space", or every unconfigured node looks like it is full.
+func diskColumn(freeGB, totalGB int) string {
+	if totalGB <= 0 {
+		return "-"
+	}
+	return fmt.Sprintf("%dG/%dG", freeGB, totalGB)
+}
+
 func newModelsCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 	return &cobra.Command{
 		Use:   "models",
@@ -256,6 +324,7 @@ func newModelsCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 
 func newReplicasCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 	var model string
+	var f replicaFilter
 	c := &cobra.Command{
 		Use:   "replicas",
 		Short: "List replicas and the shards they are made of",
@@ -268,10 +337,16 @@ func newReplicasCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 			if err := client().get(cmd.Context(), path, &reps); err != nil {
 				return err
 			}
+			total := len(reps)
+			reps = f.apply(reps)
 			if *asJSON {
 				return emitJSON(reps)
 			}
 			if len(reps) == 0 {
+				if !f.empty() && total > 0 {
+					fmt.Println(noMatches("replicas", f.describe(), total))
+					return nil
+				}
 				fmt.Println("No replicas are running.")
 				return nil
 			}
@@ -283,10 +358,27 @@ func newReplicasCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 				for _, s := range r.Shards {
 					nodes = append(nodes, fmt.Sprintf("%s:%v", s.Node, s.GPUs))
 				}
+				// The state says stale rather than a column of its own: most
+				// replicas are not, a column would be empty for all of them,
+				// and when one is it is the first thing worth knowing about it.
+				state := r.State
+				if r.Stale {
+					state += " (stale profile)"
+				}
 				fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n",
-					r.ID, r.Model, r.State, r.Pool, join(nodes),
+					r.ID, r.Model, state, r.Pool, join(nodes),
 					(time.Duration(r.AgeSec) * time.Second).String(),
 					idleColumn(r.IdleSec), r.Endpoint)
+				// A line rather than a column, because most replicas are not
+				// downloading anything and a column would be empty for all of
+				// them. When one is, it is the only thing worth reading.
+				for _, s := range r.Shards {
+					if s.Fetch == nil {
+						continue
+					}
+					fmt.Fprintf(w, "  %s: fetching %s\t\t\t\t\t\t\t\n",
+						s.Node, fetchColumn(s.Fetch))
+				}
 			}
 			_ = w.Flush()
 
@@ -304,6 +396,9 @@ func newReplicasCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 		},
 	}
 	c.Flags().StringVar(&model, "model", "", "only this model's replicas")
+	c.Flags().StringVar(&f.node, "node", "", "only replicas with a shard on this node")
+	c.Flags().StringVar(&f.state, "state", "", "only replicas in this state (prefix match)")
+	c.Flags().StringVar(&f.pool, "pool", "", "only this pool; matches either half of fabric/class")
 	return c
 }
 
@@ -397,12 +492,3 @@ func dash(s string) string {
 	}
 	return s
 }
-
-func hasUnhealthy(nodes []adminapi.NodeView) bool {
-	for _, n := range nodes {
-		if len(n.Unhealthy) > 0 {
-			return true
-		}
-	}
-	return false
-}

internal/cli/ctlweights.go

@@ -0,0 +1,174 @@
+package cli
+
+import (
+	"encoding/json"
+	"fmt"
+	"strings"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+)
+
+// newWeightsCmd reports and reclaims model files nothing needs.
+//
+// The dangerous half of storage management, so its shape is defensive: listing
+// is the default and deleting takes a flag, only files llmgw fetched are ever
+// candidates, and the node checks every path again before unlinking it.
+//
+// What it will never touch is the more important half of the contract. Weights
+// an operator put on a node — by scp, by a shared mount, by hand — are none of
+// this system's business however unreferenced they look, and a model that took
+// a week to copy must not be removable by a command someone ran to tidy up.
+func newWeightsCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
+	cmd := &cobra.Command{
+		Use:   "weights",
+		Short: "Show cached model files no model in the catalog needs",
+		Long: "Lists weights llmgw downloaded for models that are no longer in the\n" +
+			"catalog, per node, with what deleting them would free.\n\n" +
+			"Only files llmgw fetched are considered. Anything an operator put on a\n" +
+			"node is left alone, whatever the catalog says — llmgw observes storage\n" +
+			"rather than owning it, and only takes back what it put there itself.\n\n" +
+			"Nothing is deleted without --collect.",
+		Args: cobra.NoArgs,
+	}
+	var collect bool
+	var node string
+
+	cmd.Flags().BoolVar(&collect, "collect", false, "delete them, rather than only listing")
+	cmd.Flags().StringVar(&node, "node", "", "limit to one node")
+
+	cmd.RunE = func(cmd *cobra.Command, _ []string) error {
+		if collect {
+			return collectOrphans(cmd, client(), node, *asJSON)
+		}
+		return listOrphanWeights(cmd, client(), node, *asJSON)
+	}
+	return cmd
+}
+
+func listOrphanWeights(cmd *cobra.Command, c *ctlClient, node string, asJSON bool) error {
+	var found []adminapi.OrphanView
+	if err := c.get(cmd.Context(), "/api/v1/weights/orphans", &found); err != nil {
+		return err
+	}
+	if node != "" {
+		found = onlyNode(found, node)
+	}
+	if asJSON {
+		return emitJSON(found)
+	}
+	if len(found) == 0 {
+		fmt.Fprintln(cmd.OutOrStdout(),
+			"Nothing to reclaim: every file llmgw fetched is still wanted by a model.")
+		return nil
+	}
+
+	w := tab()
+	fmt.Fprintln(w, "NODE\tFILES\tWOULD FREE")
+	var total int64
+	for _, o := range found {
+		fmt.Fprintf(w, "%s\t%d\t%s\n", o.Node, len(o.Paths), fileSize(o.Bytes))
+		total += o.Bytes
+	}
+	_ = w.Flush()
+
+	for _, o := range found {
+		for _, p := range o.Paths {
+			fmt.Fprintf(cmd.OutOrStdout(), "\n  %s: %s", o.Node, p)
+		}
+	}
+	fmt.Fprintf(cmd.OutOrStdout(), "\n\n%s in total. Delete with --collect.\n", fileSize(total))
+	return nil
+}
+
+func collectOrphans(cmd *cobra.Command, c *ctlClient, node string, asJSON bool) error {
+	body, err := json.Marshal(map[string]string{"node": node})
+	if err != nil {
+		return err
+	}
+	var results []struct {
+		Node  string `json:"node"`
+		Files int    `json:"files"`
+		Bytes int64  `json:"bytes"`
+		Error string `json:"error,omitempty"`
+	}
+	if err := c.post(cmd.Context(), "/api/v1/weights/collect", body, &results); err != nil {
+		return err
+	}
+	if asJSON {
+		return emitJSON(results)
+	}
+	if len(results) == 0 {
+		fmt.Fprintln(cmd.OutOrStdout(), "Nothing to reclaim.")
+		return nil
+	}
+	var freed int64
+	for _, r := range results {
+		if r.Error != "" {
+			// Reported rather than fatal: the other nodes' space is still worth
+			// freeing, and this one will be offered again next time.
+			fmt.Fprintf(cmd.OutOrStdout(), "%s: %s\n", r.Node, r.Error)
+			continue
+		}
+		fmt.Fprintf(cmd.OutOrStdout(), "%s: asked to remove %d files, %s\n",
+			r.Node, r.Files, fileSize(r.Bytes))
+		freed += r.Bytes
+	}
+	// "asked", not "removed": each node decides, and one that refuses a path it
+	// cannot prove it fetched is the feature working rather than failing.
+	fmt.Fprintf(cmd.OutOrStdout(), "\n%s requested. The nodes verify each path before deleting it.\n",
+		fileSize(freed))
+	return nil
+}
+
+func onlyNode(in []adminapi.OrphanView, node string) []adminapi.OrphanView {
+	out := make([]adminapi.OrphanView, 0, len(in))
+	for _, o := range in {
+		if o.Node == node {
+			out = append(out, o)
+		}
+	}
+	return out
+}
+
+// fileSize renders a size the way someone reads one when deciding whether a
+// file is worth deleting.
+//
+// Separate from the per-token cache size renderer in modelplan, which never
+// leaves KiB and MiB. Model files are tens of gigabytes, and "8601.6 MiB" is
+// not a number anyone weighs a decision against.
+func fileSize(b int64) string {
+	switch {
+	case b >= 1<<30:
+		return fmt.Sprintf("%.1f GB", float64(b)/(1<<30))
+	case b >= 1<<20:
+		return fmt.Sprintf("%.0f MB", float64(b)/(1<<20))
+	case b > 0:
+		return fmt.Sprintf("%d B", b)
+	default:
+		// The node reported no size. Saying "0 B" would suggest deleting it
+		// gains nothing, which is a different claim from not knowing.
+		return "size unknown"
+	}
+}
+
+// fetchColumn renders a download's progress.
+//
+// A percentage when the size is known and a byte count when it is not, because
+// a server that does not send Content-Length is common enough — and an
+// invented percentage that stops moving is worse than a number that rises.
+func fetchColumn(f *adminapi.FetchView) string {
+	if f.Percent != nil {
+		return fmt.Sprintf("%s  %d%% of %s", shortSource(f.Source), *f.Percent, fileSize(f.BytesTotal))
+	}
+	return fmt.Sprintf("%s  %s so far", shortSource(f.Source), fileSize(f.BytesDone))
+}
+
+// shortSource keeps the end of a source, which is the part that identifies it.
+func shortSource(s string) string {
+	if i := strings.LastIndex(s, "/"); i >= 0 && i < len(s)-1 {
+		return s[i+1:]
+	}
+	return s
+}

internal/cli/dispatcherref_test.go

@@ -0,0 +1,32 @@
+package cli
+
+import (
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/agentsvc"
+)
+
+func TestTheForwardingDispatcherCarriesEveryCapability(t *testing.T) {
+	// dispatcherRef exists to break a construction cycle, and it satisfies the
+	// dispatcher interface by forwarding each method by hand. That makes it a
+	// place where capabilities go missing silently: the controller reaches
+	// past the interface by type assertion, and an assertion against this type
+	// fails for anything nobody remembered to forward.
+	//
+	// It happened. `ctl weights --collect` returned "this control plane cannot
+	// ask nodes to delete files" against a real cluster, while every unit test
+	// passed — because the test dispatchers do not implement the capability
+	// either, so its absence looked exactly like the expected case.
+	var ref any = &dispatcherRef{}
+	if _, ok := ref.(control.WeightRemover); !ok {
+		t.Error("dispatcherRef drops WeightRemover, so the control plane cannot reclaim disk")
+	}
+
+	// And the thing it forwards to must have it, or the forwarding has nothing
+	// to reach.
+	var svc any = &agentsvc.Service{}
+	if _, ok := svc.(control.WeightRemover); !ok {
+		t.Error("the agent service does not implement WeightRemover")
+	}
+}

internal/cli/filter.go

@@ -0,0 +1,170 @@
+package cli
+
+import (
+	"fmt"
+	"slices"
+	"strings"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+)
+
+// nodeFilter narrows a node listing.
+//
+// Client-side on the whole list rather than as query parameters, because the
+// list is small even on a large fleet — a few hundred rows — and doing it here
+// means --json is filtered the same way the table is. A server-side filter
+// would be two implementations of the same predicate, and the JSON one is the
+// one nobody would notice had drifted.
+type nodeFilter struct {
+	fabric string
+	class  string
+	state  string
+	labels []string
+}
+
+// empty reports whether this filter would narrow anything.
+//
+// The difference matters: an empty fleet and a filter that matched nothing are
+// different situations with different next steps, and "no nodes" for both
+// makes a mistyped fabric look like an outage.
+func (f nodeFilter) empty() bool {
+	return f.fabric == "" && f.class == "" && f.state == "" && len(f.labels) == 0
+}
+
+func (f nodeFilter) describe() string {
+	var parts []string
+	if f.fabric != "" {
+		parts = append(parts, "fabric "+f.fabric)
+	}
+	if f.class != "" {
+		parts = append(parts, "class "+f.class)
+	}
+	if f.state != "" {
+		parts = append(parts, "state "+f.state)
+	}
+	for _, l := range f.labels {
+		parts = append(parts, "label "+l)
+	}
+	return strings.Join(parts, ", ")
+}
+
+func (f nodeFilter) apply(in []adminapi.NodeView) ([]adminapi.NodeView, error) {
+	want := map[string]string{}
+	for _, l := range f.labels {
+		// A bare key is allowed and means "carries this at all", which is how
+		// an operator asks for every node with a role without caring which.
+		// An empty key is not: "=value" is a typo, and matching everything
+		// would look like the filter having worked.
+		k, v, _ := strings.Cut(l, "=")
+		if k == "" {
+			return nil, fmt.Errorf("--label needs a key: %q has none", l)
+		}
+		want[k] = v
+	}
+
+	out := make([]adminapi.NodeView, 0, len(in))
+	for _, n := range in {
+		if f.fabric != "" && !slices.Contains(n.Fabrics, f.fabric) {
+			continue
+		}
+		if f.class != "" && !slices.Contains(n.Classes, f.class) {
+			continue
+		}
+		// Prefix rather than equality, so --state ready matches and --state re
+		// does too. Node states are few and distinct, and an operator scanning
+		// a fleet is typing quickly.
+		if f.state != "" && !strings.HasPrefix(n.State, strings.ToLower(f.state)) {
+			continue
+		}
+		if !hasLabels(n.Labels, want) {
+			continue
+		}
+		out = append(out, n)
+	}
+	return out, nil
+}
+
+// hasLabels reports whether a node carries every wanted label.
+//
+// An empty value means "has this key at all", which is how an operator asks
+// for every node that carries a role without caring which.
+func hasLabels(have map[string]string, want map[string]string) bool {
+	for k, v := range want {
+		got, ok := have[k]
+		if !ok {
+			return false
+		}
+		if v != "" && got != v {
+			return false
+		}
+	}
+	return true
+}
+
+// replicaFilter narrows a replica listing.
+type replicaFilter struct {
+	node  string
+	state string
+	pool  string
+}
+
+func (f replicaFilter) empty() bool {
+	return f.node == "" && f.state == "" && f.pool == ""
+}
+
+func (f replicaFilter) describe() string {
+	var parts []string
+	if f.node != "" {
+		parts = append(parts, "node "+f.node)
+	}
+	if f.state != "" {
+		parts = append(parts, "state "+f.state)
+	}
+	if f.pool != "" {
+		parts = append(parts, "pool "+f.pool)
+	}
+	return strings.Join(parts, ", ")
+}
+
+func (f replicaFilter) apply(in []adminapi.ReplicaView) []adminapi.ReplicaView {
+	out := make([]adminapi.ReplicaView, 0, len(in))
+	for _, r := range in {
+		// Through the shards, because that is where the nodes are: a replica
+		// spanning two machines is on both, and asking "what is on node05"
+		// has to find it from either side.
+		if f.node != "" && !onNode(r, f.node) {
+			continue
+		}
+		if f.state != "" && !strings.HasPrefix(strings.ToLower(r.State), strings.ToLower(f.state)) {
+			continue
+		}
+		// Substring, because a pool is "fabric/class" and both halves are
+		// things an operator narrows by: --pool ib-a and --pool h100-80g are
+		// both questions worth asking.
+		if f.pool != "" && !strings.Contains(r.Pool, f.pool) {
+			continue
+		}
+		out = append(out, r)
+	}
+	return out
+}
+
+// onNode reports whether any shard of a replica runs on a node.
+func onNode(r adminapi.ReplicaView, node string) bool {
+	for _, sh := range r.Shards {
+		if sh.Node == node {
+			return true
+		}
+	}
+	return false
+}
+
+// noMatches is what to print when a filter excluded everything.
+//
+// Distinct from "there are none", because they call for opposite next steps:
+// one is a cluster to look at and the other is a word to correct. A mistyped
+// fabric name otherwise reads as an outage.
+func noMatches(kind, what string, total int) string {
+	return fmt.Sprintf("No %s match %s. %d %s in total; run without the filter to see them.",
+		kind, what, total, kind)
+}

internal/cli/filter_test.go

@@ -0,0 +1,155 @@
+package cli
+
+import (
+	"strings"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+)
+
+func fleet() []adminapi.NodeView {
+	return []adminapi.NodeView{
+		{ID: "n1", State: "ready", Fabrics: []string{"ib-a"}, Classes: []string{"h100-80g"},
+			Labels: map[string]string{"role": "cpu-large", "rack": "r1"}},
+		{ID: "n2", State: "ready", Fabrics: []string{"ib-b"}, Classes: []string{"h100-80g"},
+			Labels: map[string]string{"role": "cpu-large"}},
+		{ID: "n3", State: "cordoned", Fabrics: []string{"ib-a"}, Classes: []string{"l40s-45g"}},
+		{ID: "n4", State: "ready"},
+	}
+}
+
+func ids(in []adminapi.NodeView) string {
+	var out []string
+	for _, n := range in {
+		out = append(out, n.ID)
+	}
+	return strings.Join(out, ",")
+}
+
+func mustApply(t *testing.T, f nodeFilter) []adminapi.NodeView {
+	t.Helper()
+	got, err := f.apply(fleet())
+	if err != nil {
+		t.Fatal(err)
+	}
+	return got
+}
+
+func TestFilteringByFabricClassAndState(t *testing.T) {
+	if got := ids(mustApply(t, nodeFilter{fabric: "ib-a"})); got != "n1,n3" {
+		t.Errorf("--fabric ib-a = %s", got)
+	}
+	if got := ids(mustApply(t, nodeFilter{class: "l40s-45g"})); got != "n3" {
+		t.Errorf("--class l40s-45g = %s", got)
+	}
+	if got := ids(mustApply(t, nodeFilter{state: "cordoned"})); got != "n3" {
+		t.Errorf("--state cordoned = %s", got)
+	}
+}
+
+func TestFiltersCombine(t *testing.T) {
+	// Which is the question worth asking on a large fleet: not "what is on
+	// this fabric" but "what on this fabric is not ready".
+	if got := ids(mustApply(t, nodeFilter{fabric: "ib-a", state: "ready"})); got != "n1" {
+		t.Errorf("combined = %s", got)
+	}
+}
+
+func TestAStatePrefixIsEnough(t *testing.T) {
+	// Node states are few and distinct, and somebody scanning a fleet types
+	// quickly.
+	if got := ids(mustApply(t, nodeFilter{state: "cord"})); got != "n3" {
+		t.Errorf("--state cord = %s", got)
+	}
+}
+
+func TestALabelKeyAloneMatchesAnyValue(t *testing.T) {
+	// How an operator asks for every node carrying a role without caring which.
+	if got := ids(mustApply(t, nodeFilter{labels: []string{"role"}})); got != "n1,n2" {
+		t.Errorf("--label role = %s", got)
+	}
+	if got := ids(mustApply(t, nodeFilter{labels: []string{"role=cpu-large", "rack=r1"}})); got != "n1" {
+		t.Errorf("two labels = %s", got)
+	}
+	if got := ids(mustApply(t, nodeFilter{labels: []string{"rack=r9"}})); got != "" {
+		t.Errorf("a label nothing carries matched %s", got)
+	}
+}
+
+func TestAMalformedLabelIsRefusedRatherThanIgnored(t *testing.T) {
+	// Silently ignoring it would show the whole fleet, which reads as the
+	// filter having matched everything rather than as a typo.
+	_, err := nodeFilter{labels: []string{"=value"}}.apply(fleet())
+	if err == nil {
+		t.Error("a label with no key was accepted")
+	}
+}
+
+func TestNoFilterIsRecognisedAsNone(t *testing.T) {
+	// It decides which of two messages an empty result gets.
+	if !(nodeFilter{}).empty() {
+		t.Error("an unset filter did not report itself as empty")
+	}
+	if (nodeFilter{state: "ready"}).empty() {
+		t.Error("a set filter reported itself as empty")
+	}
+}
+
+func TestTheEmptyMessageSaysWhichCaseItIs(t *testing.T) {
+	// A mistyped fabric name otherwise reads as an outage.
+	msg := noMatches("nodes", "fabric ib-c", 12)
+	for _, want := range []string{"fabric ib-c", "12"} {
+		if !strings.Contains(msg, want) {
+			t.Errorf("the message does not mention %q: %s", want, msg)
+		}
+	}
+}
+
+func replicas() []adminapi.ReplicaView {
+	return []adminapi.ReplicaView{
+		{ID: "r1", Model: "big", State: "ready", Pool: "ib-a/h100-80g",
+			Shards: []adminapi.ShardView{{Node: "n1"}, {Node: "n2"}}},
+		{ID: "r2", Model: "small", State: "starting", Pool: "none/cpu",
+			Shards: []adminapi.ShardView{{Node: "n5"}}},
+		{ID: "r3", Model: "small", State: "ready", Pool: "ib-b/h100-80g",
+			Shards: []adminapi.ShardView{{Node: "n2"}}},
+	}
+}
+
+func repIDs(in []adminapi.ReplicaView) string {
+	var out []string
+	for _, r := range in {
+		out = append(out, r.ID)
+	}
+	return strings.Join(out, ",")
+}
+
+func TestAReplicaIsFoundFromEitherOfItsNodes(t *testing.T) {
+	// A replica spanning two machines is on both, and "what is running on n2"
+	// has to find it from either side.
+	if got := repIDs(replicaFilter{node: "n1"}.apply(replicas())); got != "r1" {
+		t.Errorf("--node n1 = %s", got)
+	}
+	if got := repIDs(replicaFilter{node: "n2"}.apply(replicas())); got != "r1,r3" {
+		t.Errorf("--node n2 = %s", got)
+	}
+}
+
+func TestAPoolMatchesEitherHalf(t *testing.T) {
+	// A pool is fabric/class and both are things worth narrowing by.
+	if got := repIDs(replicaFilter{pool: "ib-a"}.apply(replicas())); got != "r1" {
+		t.Errorf("--pool ib-a = %s", got)
+	}
+	if got := repIDs(replicaFilter{pool: "h100-80g"}.apply(replicas())); got != "r1,r3" {
+		t.Errorf("--pool h100-80g = %s", got)
+	}
+}
+
+func TestReplicaStateFilters(t *testing.T) {
+	if got := repIDs(replicaFilter{state: "start"}.apply(replicas())); got != "r2" {
+		t.Errorf("--state start = %s", got)
+	}
+	if got := repIDs(replicaFilter{node: "n2", state: "ready"}.apply(replicas())); got != "r1,r3" {
+		t.Errorf("combined = %s", got)
+	}
+}

internal/cli/keys.go

@@ -16,7 +16,7 @@ import (
 )
 
 func newTenancyCmd() *cobra.Command {
-	var dbPath string
+	var sf storeFlags
 
 	cmd := &cobra.Command{
 		Use:   "keys",
@@ -25,9 +25,11 @@ func newTenancyCmd() *cobra.Command {
 			"can be issued before the server is running and without exposing an endpoint\n" +
 			"that mints credentials.",
 	}
-	cmd.PersistentFlags().StringVar(&dbPath, "store", "/var/lib/llmgw/llmgw.db", "path to the store")
+	sf.bind(cmd)
 
-	open := func(ctx context.Context) (*store.SQLite, error) { return store.OpenSQLite(ctx, dbPath) }
+	// Takes a context rather than the command, because some of these run
+	// before the command tree is entered.
+	open := func(ctx context.Context) (*store.DB, error) { return sf.openCtx(ctx) }
 
 	cmd.AddCommand(
 		newKeyInitCmd(open),
@@ -35,11 +37,14 @@ func newTenancyCmd() *cobra.Command {
 		newKeyListCmd(open),
 		newKeyRevokeCmd(open),
 		newLimitsCmd(open),
+		newThreadsCmd(open),
+		newMemoryCmd(open),
+		newMoveStoreCmd(),
 	)
 	return cmd
 }
 
-type opener func(context.Context) (*store.SQLite, error)
+type opener func(context.Context) (*store.DB, error)
 
 func newKeyInitCmd(open opener) *cobra.Command {
 	var org, team, project string

internal/cli/memory.go

@@ -0,0 +1,62 @@
+package cli
+
+import (
+	"fmt"
+
+	"github.com/spf13/cobra"
+)
+
+// newMemoryCmd turns note storage on for a project, or off.
+//
+// A command rather than a configuration key for the same reason threads are:
+// keeping what an agent decided to remember is a decision about data, not about
+// deployment, and it should look like an act rather than a default nobody read.
+func newMemoryCmd(open opener) *cobra.Command {
+	var (
+		project string
+		hours   int
+		off     bool
+	)
+
+	cmd := &cobra.Command{
+		Use:   "memory",
+		Short: "Keep notes an agent writes for itself, across conversations",
+		Long: "With this on, an agent can save and search notes through /v1/memory or the\n" +
+			"MCP tools at /mcp (remember, recall, forget), and find them again in later\n" +
+			"conversations.\n\n" +
+			"Off for every project until this is run. The retention is required: notes\n" +
+			"expire that many hours after they were last written, so a store nobody\n" +
+			"chose a number for does not keep everything forever.",
+		Args: cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			if !off && hours <= 0 {
+				return fmt.Errorf("--retain-hours is required: say how long notes are kept, or pass --off")
+			}
+			s, err := open(cmd.Context())
+			if err != nil {
+				return err
+			}
+			defer s.Close()
+
+			keep := hours
+			if off {
+				keep = 0
+			}
+			if err := s.SetMemoryRetention(cmd.Context(), project, keep); err != nil {
+				return err
+			}
+			if keep == 0 {
+				fmt.Fprintf(cmd.OutOrStdout(),
+					"%s keeps no notes. Existing ones are removed at the next prune.\n", project)
+				return nil
+			}
+			fmt.Fprintf(cmd.OutOrStdout(),
+				"%s keeps notes for %d hours after they were last written.\n", project, keep)
+			return nil
+		},
+	}
+	cmd.Flags().StringVar(&project, "project", "default", "project id")
+	cmd.Flags().IntVar(&hours, "retain-hours", 0, "how long a note is kept after it was last written")
+	cmd.Flags().BoolVar(&off, "off", false, "stop keeping notes for this project")
+	return cmd
+}

internal/cli/mirror.go

@@ -0,0 +1,400 @@
+package cli
+
+import (
+	"encoding/json"
+	"fmt"
+	"log/slog"
+	"net/http"
+	"os"
+	"strings"
+	"time"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
+)
+
+// newMirrorCmd manages a directory of model weights served to the fleet.
+//
+// Operator tooling, run on the mirror host, the way `llmgw keys` is run on the
+// machine holding the store. The gateway never writes here: a mirror that can
+// be written to by the thing reading from it is a way to replace a model with
+// another one that every node would then fetch without complaint.
+func newMirrorCmd() *cobra.Command {
+	var dir string
+
+	c := &cobra.Command{
+		Use:   "mirror",
+		Short: "Hold model weights for the fleet to fetch from",
+		Long: "One machine holding the model files, so nodes fetch over your own\n" +
+			"network instead of the internet. Eight nodes each pulling forty gigabytes\n" +
+			"from outside is a day's work, and on many corporate networks it is not\n" +
+			"possible at all.\n\n" +
+			"Files live at <model>/<version>/<file>, and a version is never\n" +
+			"overwritten. That is what makes rolling back possible: changing the\n" +
+			"catalog line back works because the old weights are still there.",
+	}
+	c.PersistentFlags().StringVar(&dir, "dir", "/srv/models", "the mirror's root directory")
+
+	c.AddCommand(
+		newMirrorAddCmd(&dir),
+		newMirrorListCmd(&dir),
+		newMirrorUnusedCmd(&dir),
+		newMirrorRemoveCmd(&dir),
+		newMirrorServeCmd(&dir),
+	)
+	return c
+}
+
+func newMirrorAddCmd(dir *string) *cobra.Command {
+	var model, version, as string
+
+	c := &cobra.Command{
+		Use:   "add <url>",
+		Short: "Download a model into the mirror",
+		Long: "Downloads to a temporary name and renames when complete, so an\n" +
+			"interrupted download never leaves something a catalog could be pointed\n" +
+			"at. Reports the checksum: a mirror is the one place in this system where\n" +
+			"a corrupted file is copied to every node rather than caught once.\n\n" +
+			"Refuses to replace a version that is already here. Add a new one beside\n" +
+			"it instead — that is what makes the old one still available when the new\n" +
+			"one turns out to be worse.",
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			source := args[0]
+			file := as
+			if file == "" {
+				file = mirror.LastPathElement(source)
+			}
+			if model == "" || version == "" {
+				return fmt.Errorf("--model and --version are required: they are the layout")
+			}
+
+			e := mirror.Entry{Model: model, Version: version, File: file}
+			out := cmd.OutOrStdout()
+			fmt.Fprintf(out, "fetching %s\n", source)
+
+			added, err := mirror.Add(cmd.Context(), *dir, source, e, func(done, total int64) {
+				if total > 0 {
+					fmt.Fprintf(out, "  %s of %s (%d%%)\n",
+						fileSize(done), fileSize(total), done*100/total)
+					return
+				}
+				fmt.Fprintf(out, "  %s so far\n", fileSize(done))
+			})
+			if err != nil {
+				return err
+			}
+			fmt.Fprintf(out, "\n%s/%s/%s  %s\n  sha256 %s\n",
+				added.Entry.Model, added.Entry.Version, added.Entry.File,
+				fileSize(added.Entry.Bytes), added.SHA256)
+			fmt.Fprintf(out, "\nPoint a model at it with:\n  source: %s\n",
+				added.Entry.URL("http://<this-host>:8081"))
+			return nil
+		},
+	}
+	c.Flags().StringVar(&model, "model", "", "the model's name, one directory level")
+	c.Flags().StringVar(&version, "version", "", "the version, one directory level; a date works well")
+	c.Flags().StringVar(&as, "as", "", "the filename to save as; taken from the URL when omitted")
+	return c
+}
+
+func newMirrorListCmd(dir *string) *cobra.Command {
+	return &cobra.Command{
+		Use:   "list",
+		Short: "Show what the mirror holds",
+		Args:  cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			entries, err := mirror.List(*dir)
+			if err != nil {
+				return err
+			}
+			if len(entries) == 0 {
+				fmt.Fprintf(cmd.OutOrStdout(), "The mirror at %s is empty.\n", *dir)
+				return nil
+			}
+			w := tab()
+			fmt.Fprintln(w, "MODEL\tVERSION\tFILE\tSIZE")
+			var total int64
+			for _, e := range entries {
+				fmt.Fprintf(w, "%s\t%s\t%s\t%s\n", e.Model, e.Version, e.File, fileSize(e.Bytes))
+				total += e.Bytes
+			}
+			_ = w.Flush()
+			fmt.Fprintf(cmd.OutOrStdout(), "\n%s in %d files.\n", fileSize(total), len(entries))
+			return nil
+		},
+	}
+}
+
+func newMirrorUnusedCmd(dir *string) *cobra.Command {
+	var server, base string
+
+	c := &cobra.Command{
+		Use:   "unused",
+		Short: "Show files no model in the catalog refers to",
+		Long: "Compares what is here against every source in the catalog, so the\n" +
+			"answer is what could be deleted rather than what looks old.\n\n" +
+			"It refuses to answer when it cannot read the catalog. An empty catalog\n" +
+			"and an unreachable one produce the same list — everything — and the\n" +
+			"obvious next step after that list is deletion.",
+		Args: cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			entries, err := mirror.List(*dir)
+			if err != nil {
+				return err
+			}
+			sources, err := catalogSources(cmd, server)
+			if err != nil {
+				// Named as the reason rather than folded into a generic
+				// failure: an operator who sees "no catalog" will fix the
+				// address, and one who sees a list will start deleting.
+				return fmt.Errorf("cannot read the catalog, so nothing can be called unused: %w", err)
+			}
+
+			unused := mirror.Unused(entries, sources, base)
+			out := cmd.OutOrStdout()
+			if len(unused) == 0 {
+				fmt.Fprintf(out, "Every file here is referred to by a model. (%d files, %d catalog sources.)\n",
+					len(entries), len(sources))
+				return nil
+			}
+			w := tab()
+			fmt.Fprintln(w, "MODEL\tVERSION\tFILE\tSIZE")
+			var total int64
+			for _, e := range unused {
+				fmt.Fprintf(w, "%s\t%s\t%s\t%s\n", e.Model, e.Version, e.File, fileSize(e.Bytes))
+				total += e.Bytes
+			}
+			_ = w.Flush()
+			fmt.Fprintf(out, "\n%s in %d files that no model refers to.\n", fileSize(total), len(unused))
+			fmt.Fprintln(out, "\nBefore deleting: a node that has to refetch one of these will not be\n"+
+				"able to. Retire the catalog entry first, let the fleet settle, then remove.")
+			return nil
+		},
+	}
+	c.Flags().StringVar(&server, "server", "127.0.0.1:9090", "admin API address to read the catalog from")
+	c.Flags().StringVar(&base, "base", "http://localhost:8081",
+		"the address nodes use to reach this mirror, as it appears in catalog sources")
+	return c
+}
+
+func newMirrorRemoveCmd(dir *string) *cobra.Command {
+	var model, version, file, server, base string
+	var force bool
+
+	c := &cobra.Command{
+		Use:   "remove",
+		Short: "Delete one version from the mirror",
+		Long: "Checks the catalog first, and refuses if a model still refers to it —\n" +
+			"a node that has to refetch a deleted file cannot, and the failure arrives\n" +
+			"later, on the node, looking like something else.\n\n" +
+			"Retire the catalog entry first, let the fleet settle, then remove.",
+		Args: cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			if model == "" || version == "" || file == "" {
+				return fmt.Errorf("--model, --version and --file are all required")
+			}
+			e := mirror.Entry{Model: model, Version: version, File: file}
+
+			if !force {
+				sources, err := catalogSources(cmd, server)
+				if err != nil {
+					return fmt.Errorf("cannot read the catalog to check whether this is still in use: %w\n"+
+						"  Pass --force to remove it anyway", err)
+				}
+				if len(mirror.Unused([]mirror.Entry{e}, sources, base)) == 0 {
+					return fmt.Errorf("a model still refers to %s/%s/%s; retire the catalog entry first",
+						model, version, file)
+				}
+			}
+			if err := mirror.Remove(*dir, e); err != nil {
+				return err
+			}
+			fmt.Fprintf(cmd.OutOrStdout(), "removed %s/%s/%s\n", model, version, file)
+			return nil
+		},
+	}
+	c.Flags().StringVar(&model, "model", "", "the model directory")
+	c.Flags().StringVar(&version, "version", "", "the version directory")
+	c.Flags().StringVar(&file, "file", "", "the filename")
+	c.Flags().StringVar(&server, "server", "127.0.0.1:9090", "admin API address to read the catalog from")
+	c.Flags().StringVar(&base, "base", "http://localhost:8081", "the address nodes use to reach this mirror")
+	c.Flags().BoolVar(&force, "force", false, "remove without checking the catalog")
+	return c
+}
+
+func newMirrorServeCmd(dir *string) *cobra.Command {
+	var listen, apiListen, base string
+	var tlsc tlsconf.Config
+
+	c := &cobra.Command{
+		Use:   "serve",
+		Short: "Serve the mirror, and an API to manage it",
+		Long: "Two listeners, on purpose.\n\n" +
+			"The first serves the weights themselves, read-only and unauthenticated,\n" +
+			"because that is what eight nodes pull forty gigabytes from and putting a\n" +
+			"client certificate in front of it would mean re-certifying every agent's\n" +
+			"fetcher to change how models are administered. Range requests work, which\n" +
+			"is what makes an interrupted download on a node resume rather than restart.\n\n" +
+			"The second is the management API the control plane calls: ask the mirror to\n" +
+			"fetch a model, watch it arrive, remove one that is retired. It can be given\n" +
+			"the fleet's CA and then only holders of a certificate from it get in.\n\n" +
+			"The mirror does its own downloading. A control plane that had to pull the\n" +
+			"bytes and push them here would carry every model twice over the network,\n" +
+			"through the one machine least able to spare the bandwidth.",
+		Args: cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			root, err := os.Stat(*dir)
+			if err != nil || !root.IsDir() {
+				return fmt.Errorf("%s is not a directory to serve", *dir)
+			}
+			if err := tlsc.Validate(); err != nil {
+				return err
+			}
+
+			out := cmd.OutOrStdout()
+			log := slog.Default()
+			mgr := mirror.NewManager(*dir, log)
+			api := mirror.NewAPI(mgr, strings.TrimSuffix(base, "/"))
+
+			// http.Dir refuses paths that escape the root, which is the whole
+			// traversal question answered by the standard library rather than
+			// by me.
+			files := &http.Server{
+				Addr:              listen,
+				Handler:           readOnly(withManifest(*dir, http.FileServer(http.Dir(*dir)))),
+				ReadHeaderTimeout: 10 * time.Second,
+			}
+			admin := &http.Server{
+				Addr:              apiListen,
+				Handler:           api.Handler(),
+				ReadHeaderTimeout: 10 * time.Second,
+			}
+
+			errs := make(chan error, 2)
+			go func() { errs <- fmt.Errorf("serving weights: %w", files.ListenAndServe()) }()
+			fmt.Fprintf(out, "serving %s on %s\n", *dir, listen)
+
+			go func() { errs <- fmt.Errorf("serving the API: %w", serveAPI(admin, tlsc)) }()
+			if tlsc.Enabled() {
+				fmt.Fprintf(out, "management API on %s, mutual TLS against %s\n", apiListen, tlsc.CA)
+			} else {
+				// Said plainly rather than left to be discovered. An API that
+				// can delete models and start downloads, open to anyone who can
+				// reach the port, is a choice the operator should have made
+				// rather than inherited from a default.
+				fmt.Fprintf(out, "management API on %s, UNAUTHENTICATED — keep it on a trusted network\n", apiListen)
+			}
+			fmt.Fprintf(out, "catalog sources should point at %s\n", base)
+
+			// The first failure ends it. A mirror serving bytes but refusing
+			// management, or the reverse, is half a service pretending to be
+			// whole, and the half that works keeps the alert quiet.
+			return <-errs
+		},
+	}
+	c.Flags().StringVar(&listen, "listen", ":8081", "address to serve the weights on")
+	c.Flags().StringVar(&apiListen, "api-listen", ":8082", "address to serve the management API on")
+	c.Flags().StringVar(&base, "base", "http://localhost:8081",
+		"the address nodes use to reach this mirror, as it should appear in catalog sources")
+	c.Flags().StringVar(&tlsc.CA, "tls-ca", "", "CA bundle that management clients must present a certificate from")
+	c.Flags().StringVar(&tlsc.Cert, "tls-cert", "", "this mirror's certificate")
+	c.Flags().StringVar(&tlsc.Key, "tls-key", "", "this mirror's private key")
+	return c
+}
+
+// serveAPI starts the management listener, with mutual TLS when it is
+// configured.
+//
+// NextProtos is cleared from what tlsconf builds. That configuration is for the
+// agent stream, which needs HTTP/2 for bidirectional streams and so pins "h2";
+// this is an ordinary REST API, and pinning h2 would fail the handshake for
+// every HTTP/1.1 client — curl included — with an error that says nothing about
+// ALPN. Empty lets net/http offer both.
+func serveAPI(srv *http.Server, c tlsconf.Config) error {
+	if !c.Enabled() {
+		return srv.ListenAndServe()
+	}
+	cfg, err := c.ServerConfig()
+	if err != nil {
+		return err
+	}
+	cfg.NextProtos = nil
+	srv.TLSConfig = cfg
+	return srv.ListenAndServeTLS("", "")
+}
+
+// readOnly refuses anything but a read.
+//
+// http.FileServer already ignores other methods, but saying so here means the
+// property is stated rather than inherited — and a handler added later cannot
+// quietly make the mirror writable.
+func readOnly(h http.Handler) http.Handler {
+	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		if r.Method != http.MethodGet && r.Method != http.MethodHead {
+			http.Error(w, "the mirror is read-only", http.StatusMethodNotAllowed)
+			return
+		}
+		h.ServeHTTP(w, r)
+	})
+}
+
+// catalogSources reads every model's source from a running control plane.
+func catalogSources(cmd *cobra.Command, server string) ([]string, error) {
+	var models []adminapi.ModelView
+	if err := newCtlClient(server).get(cmd.Context(), "/api/v1/models", &models); err != nil {
+		return nil, err
+	}
+	out := make([]string, 0, len(models))
+	for _, m := range models {
+		out = append(out, m.Source)
+	}
+	return out, nil
+}
+
+// withManifest answers a request for a version with the list of files in it.
+//
+// A directory otherwise gets FileServer's HTML index, and something that has to
+// know what a version consists of would be left parsing it. A model is
+// routinely several files — sharded safetensors, or a GGUF split across parts —
+// and a catalog entry names one source, so without a way to ask "what is in
+// this version" a sharded model can be mirrored and never served.
+//
+// On the byte listener rather than the management API on purpose. Fetching
+// weights is the one thing every node does, and making it depend on the
+// administrative port would undo the split that keeps that port closeable.
+func withManifest(root string, next http.Handler) http.Handler {
+	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		model, version, ok := versionPath(r.URL.Path)
+		if !ok {
+			next.ServeHTTP(w, r)
+			return
+		}
+		m, err := mirror.ManifestFor(root, model, version)
+		if err != nil {
+			http.NotFound(w, r)
+			return
+		}
+		w.Header().Set("Content-Type", "application/json")
+		_ = json.NewEncoder(w).Encode(m)
+	})
+}
+
+// versionPath matches "/model/version/" and nothing else.
+//
+// The trailing slash is the whole signal: "/m/v/file.gguf" is a file and
+// "/m/v/" is the set it belongs to. Requiring it means a source URL says which
+// of the two it means without a flag beside it.
+func versionPath(p string) (model, version string, ok bool) {
+	if !strings.HasSuffix(p, "/") {
+		return "", "", false
+	}
+	parts := strings.Split(strings.Trim(p, "/"), "/")
+	if len(parts) != 2 || parts[0] == "" || parts[1] == "" {
+		return "", "", false
+	}
+	return parts[0], parts[1], true
+}

internal/cli/mirror_test.go

@@ -0,0 +1,74 @@
+package cli
+
+import (
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
+)
+
+func TestTheMirrorRefusesEverythingButReads(t *testing.T) {
+	// Stated here rather than inherited from http.FileServer, so a handler
+	// added later cannot quietly make the mirror writable — which would turn
+	// it into a way to replace a model with another one that every node then
+	// fetches without complaint.
+	h := readOnly(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+		w.WriteHeader(http.StatusOK)
+	}))
+
+	for _, m := range []string{http.MethodGet, http.MethodHead} {
+		rec := httptest.NewRecorder()
+		h.ServeHTTP(rec, httptest.NewRequest(m, "/x.gguf", nil))
+		if rec.Code != http.StatusOK {
+			t.Errorf("%s was refused: %d", m, rec.Code)
+		}
+	}
+	for _, m := range []string{http.MethodPut, http.MethodPost, http.MethodDelete, http.MethodPatch} {
+		rec := httptest.NewRecorder()
+		h.ServeHTTP(rec, httptest.NewRequest(m, "/x.gguf", nil))
+		if rec.Code != http.StatusMethodNotAllowed {
+			t.Errorf("%s was allowed: %d", m, rec.Code)
+		}
+	}
+}
+
+func TestTheFilenameIsTakenFromTheURLWhenNotGiven(t *testing.T) {
+	for _, c := range []struct{ url, want string }{
+		{"https://h/a/b/model.gguf", "model.gguf"},
+		{"https://h/model.gguf?download=true", "model.gguf"},
+		{"model.gguf", "model.gguf"},
+	} {
+		if got := mirror.LastPathElement(c.url); got != c.want {
+			t.Errorf("mirror.LastPathElement(%q) = %q, want %q", c.url, got, c.want)
+		}
+	}
+}
+
+func TestAVersionAnswersHeadAsWellAsGet(t *testing.T) {
+	// A sharded model's catalog source is the version directory, and the serve
+	// path checks a source with HEAD before accepting it. If the manifest
+	// answered only GET, every multi-file model would be refused as missing.
+	root := t.TempDir()
+	if err := os.MkdirAll(filepath.Join(root, "llama", "v1"), 0o755); err != nil {
+		t.Fatal(err)
+	}
+	for _, name := range []string{"model-00001-of-00002.safetensors", "model-00002-of-00002.safetensors"} {
+		if err := os.WriteFile(filepath.Join(root, "llama", "v1", name), []byte("x"), 0o644); err != nil {
+			t.Fatal(err)
+		}
+	}
+
+	h := readOnly(withManifest(root, http.FileServer(http.Dir(root))))
+	w := httptest.NewRecorder()
+	h.ServeHTTP(w, httptest.NewRequest(http.MethodHead, "/llama/v1/", nil))
+
+	if w.Code != http.StatusOK {
+		t.Fatalf("a version should answer HEAD; got %d", w.Code)
+	}
+	if got := w.Header().Get("Content-Type"); got != "application/json" {
+		t.Errorf("expected a manifest, got %q", got)
+	}
+}

internal/cli/model.go

@@ -4,13 +4,16 @@ import (
 	"encoding/json"
 	"fmt"
 	"os/user"
+	"time"
 
 	"github.com/knadh/koanf/parsers/yaml"
 	"github.com/knadh/koanf/providers/file"
 	"github.com/knadh/koanf/v2"
 	"github.com/spf13/cobra"
 
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/sourcecheck"
 )
 
 // newModelCmd adds and removes models without a restart.
@@ -31,6 +34,7 @@ func newModelCmd(client func() *ctlClient) *cobra.Command {
 
 func newModelAddCmd(client func() *ctlClient) *cobra.Command {
 	var path string
+	var skipCheck bool
 	c := &cobra.Command{
 		Use:   "add -f <file.yaml>",
 		Short: "Add a model from a definition file",
@@ -61,6 +65,23 @@ func newModelAddCmd(client func() *ctlClient) *cobra.Command {
 			if m.Name == "" {
 				return fmt.Errorf("%s: the definition has no name", path)
 			}
+
+			// Before the catalog is touched, because a source that does not
+			// resolve is otherwise discovered by a replica failing to start
+			// minutes later on a node — with an error naming the node rather
+			// than the typo.
+			if !skipCheck {
+				size, err := sourcecheck.Check(cmd.Context(), m.Source)
+				if err != nil {
+					return fmt.Errorf("%w\n  Pass --no-source-check to add it anyway, "+
+						"which is right when the mirror is not up yet or the nodes "+
+						"reach it by an address this machine does not.", err)
+				}
+				if _, isURL := sourcecheck.HTTPSource(m.Source); isURL {
+					fmt.Fprintln(cmd.OutOrStdout(), sourceNote(m.Source, size))
+				}
+			}
+
 			body, err := json.Marshal(m)
 			if err != nil {
 				return err
@@ -77,6 +98,8 @@ func newModelAddCmd(client func() *ctlClient) *cobra.Command {
 		},
 	}
 	c.Flags().StringVarP(&path, "file", "f", "", "YAML file holding the model definition")
+	c.Flags().BoolVar(&skipCheck, "no-source-check", false,
+		"add it without asking whether the weights are actually at that URL")
 	return c
 }
 
@@ -108,3 +131,58 @@ func actor() string {
 	}
 	return "unknown"
 }
+
+func newScaleCmd(client func() *ctlClient) *cobra.Command {
+	var min, max int
+	var idle time.Duration
+	c := &cobra.Command{
+		Use:   "scale <model> --min N [--max M]",
+		Short: "Change how many replicas a model runs",
+		Long: "The replica count is --min. --max is a ceiling, not a target: a model\n" +
+			"with a minimum runs exactly that many, and nothing scales it up under\n" +
+			"load. Leaving --max out makes it follow --min.\n\n" +
+			"--min 0 with a --max above it is the on-demand model: nothing runs until\n" +
+			"somebody asks, and it stops again after the idle timeout. That loads one\n" +
+			"replica however high --max is. --min 0 --max 0 is a model kept in the\n" +
+			"catalog and deliberately not running.\n\n" +
+			"Only models added at runtime. One declared in server.yaml is read again\n" +
+			"on every start, so a number set here would hold until the next restart\n" +
+			"and then revert — change it in the file instead.\n\n" +
+			"It takes effect on the reconciler's next pass, and survives a restart.",
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			if !cmd.Flags().Changed("min") {
+				return fmt.Errorf("--min is required: it is the replica count")
+			}
+			body, err := json.Marshal(adminapi.ScaleRequest{Min: min, Max: max, IdleTimeout: idle})
+			if err != nil {
+				return err
+			}
+			var res adminapi.ScaleResponse
+			if err := client().post(cmd.Context(),
+				"/api/v1/models/"+args[0]+"/scale", body, &res); err != nil {
+				return err
+			}
+			// What is in force, not what was asked for: the catalog fills in a
+			// max that was left out, and printing the request back would show a
+			// number the cluster is not using.
+			out := cmd.OutOrStdout()
+			switch {
+			case res.OnDemand:
+				fmt.Fprintf(out, "%s is on-demand: loaded when asked for, up to %d, stopped when idle.\n",
+					res.Model, res.Max)
+			case res.Min == 0:
+				fmt.Fprintf(out, "%s will not be placed; it stays in the catalog.\n", res.Model)
+			default:
+				fmt.Fprintf(out, "%s: %d replicas (max %d). It takes effect on the next pass; watch it with:\n"+
+					"  llmgw ctl models\n", res.Model, res.Min, res.Max)
+			}
+			return nil
+		},
+	}
+	c.Flags().IntVar(&min, "min", 0, "replicas to keep running; this is the count")
+	c.Flags().IntVar(&max, "max", 0, "ceiling for an on-demand model; follows --min when left out")
+	c.Flags().DurationVar(&idle, "idle-timeout", 0,
+		"how long an on-demand model stays up unasked-for; zero takes the cluster default")
+	return c
+}

internal/cli/movestore.go

@@ -0,0 +1,71 @@
+package cli
+
+import (
+	"fmt"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// newMoveStoreCmd copies a store into another one.
+//
+// Written because keys are stored hashed. Moving to PostgreSQL without this
+// means every person and every script needs a new key issued, and the old ones
+// cannot even be looked up to say which they were.
+//
+// It reads and writes, and nothing else runs while it does: run it with the
+// gateway stopped. A copy taken while requests are landing would miss whatever
+// arrived after the table it had already passed.
+func newMoveStoreCmd() *cobra.Command {
+	var from, to string
+
+	cmd := &cobra.Command{
+		Use:   "move-store",
+		Short: "Copy a store into another one, for moving SQLite to PostgreSQL",
+		Long: "Copies every table, parents first, into an empty destination.\n\n" +
+			"Stop the gateway first. This makes no attempt to copy a store that is\n" +
+			"being written to: rows landing after the table they belong to has been\n" +
+			"passed would simply be absent, and nothing would say so.\n\n" +
+			"The destination must be empty. Merging two stores means deciding which\n" +
+			"of two projects with the same id was meant, and there is no answer to\n" +
+			"that which is not a guess.",
+		Args: cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			if from == "" || to == "" {
+				return fmt.Errorf("--from and --to are both required")
+			}
+			src, err := store.Open(cmd.Context(), from)
+			if err != nil {
+				return fmt.Errorf("opening the source: %w", err)
+			}
+			defer src.Close()
+
+			dst, err := store.Open(cmd.Context(), to)
+			if err != nil {
+				return fmt.Errorf("opening the destination: %w", err)
+			}
+			defer dst.Close()
+
+			res, err := src.CopyTo(cmd.Context(), dst)
+			// Printed even on failure: a partial copy is the one case where
+			// knowing how far it got decides what to do next.
+			out := cmd.OutOrStdout()
+			var total int64
+			for _, r := range res {
+				if r.Rows > 0 {
+					fmt.Fprintf(out, "  %-16s %d\n", r.Table, r.Rows)
+				}
+				total += r.Rows
+			}
+			if err != nil {
+				return err
+			}
+			fmt.Fprintf(out, "\n%d rows copied. Point the gateway at the new store and start it.\n", total)
+			return nil
+		},
+	}
+	cmd.Flags().StringVar(&from, "from", "", "source: a SQLite path, a postgres:// URL, or @file holding one")
+	cmd.Flags().StringVar(&to, "to", "", "destination, same forms")
+	return cmd
+}

internal/cli/root.go

@@ -42,6 +42,7 @@ func Root() *cobra.Command {
 		newCtlCmd(),
 		newTenancyCmd(),
 		newUsersCmd(),
+		newMirrorCmd(),
 		newSimEngineCmd(),
 	)
 	return root

internal/cli/server.go

@@ -27,7 +27,11 @@ import (
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/fleetmetrics"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelsvc"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/promq"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/router"
+	storepkg "gitea.mynodes.xyz/ilker/llmgw/internal/store"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/version"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/web"
 )
@@ -156,6 +160,16 @@ func newServerCmd() *cobra.Command {
 						admin.ReplaceCatalog(ctx, built)
 					}
 					api.SetModels(svc)
+					// And the mirror page, so a model that has just been
+					// fetched can be served without writing YAML by hand.
+					rt.SetCatalogAdmin(svc)
+					// Benchmark results live here too. A measurement costs real
+					// GPU time; without somewhere to put it the number lives in
+					// whoever ran it's terminal and is gone by morning.
+					api.SetCapacity(st)
+					// And what each group of the tenancy tree spent, which is
+					// the other half of a quota being reviewable.
+					api.SetUsage(st)
 				}
 				api.Register(adminMux)
 
@@ -168,8 +182,26 @@ func newServerCmd() *cobra.Command {
 				if st := rt.Store(); st != nil {
 					usage = st
 				}
+				// The dashboard's operational charts. llmgw publishes these
+				// metrics; what it cannot do is remember them, so the history
+				// is read back from wherever they were scraped to.
+				//
+				// Built before either copy of the page is registered, and given
+				// to both. The first version built it afterwards and handed it
+				// only to the one on the client port, so the copy an operator
+				// reaches by SSH tunnel — the one on this very listener — had
+				// no charts at all and said nothing about why.
+				var history web.History
+				if h := promq.New(cfg.Monitoring.Prometheus, cfg.Monitoring.Timeout); h != nil {
+					history = h
+					log.Info("dashboard history configured",
+						"prometheus", cfg.Monitoring.Prometheus)
+				}
+				rt.SetHistory(history, cfg.Monitoring.Window)
+
 				adminMux.HandleFunc("GET /", web.Dashboard(
-					api, usage, version.Version, 5*time.Second, true, log))
+					api, usage, version.Version, 5*time.Second, true,
+					web.NavExtras{History: history, Window: cfg.Monitoring.Window}, log))
 				adminMux.Handle("GET /_static/", web.Static())
 
 				// The same page on the client-facing port at /admin, behind a
@@ -178,6 +210,24 @@ func newServerCmd() *cobra.Command {
 				rt.SetClusterView(api, usage)
 			}
 
+			// The model mirror, when one is configured. Built here and not at
+			// first use so a mistyped address stops the control plane starting,
+			// rather than turning into a broken admin page hours later when
+			// somebody finally opens it.
+			if cfg.Mirror.API != "" {
+				mc, err := mirror.NewClient(cfg.Mirror.API, cfg.Mirror.TLS, cfg.Mirror.Timeout)
+				if err != nil {
+					return fmt.Errorf("the model mirror: %w", err)
+				}
+				rt.SetMirror(mc)
+				// Browsing the registry is what turns "paste a resolve URL"
+				// into "find the model". It needs a way out to the internet
+				// from this machine, which the mirror's own fetching does not.
+				rt.SetRegistry(hf.New(cfg.Mirror.Registry, cfg.Mirror.Timeout))
+				log.Info("model mirror configured",
+					"api", cfg.Mirror.API, "mtls", cfg.Mirror.TLS.Enabled())
+			}
+
 			apiSrv := &http.Server{
 				Handler:           rt.Handler(),
 				ReadHeaderTimeout: 10 * time.Second,
@@ -335,7 +385,31 @@ func startControlPlane(ctx context.Context, cfg config.Server, cat *catalog.Live
 	if u, ok := sink.(control.UsageSource); ok {
 		ctl.SetUsageSource(u)
 	}
+	// With a shared store, two control planes can see each other, so this one
+	// competes rather than assuming it is alone. Without one there is nobody
+	// to compete with and it places unconditionally, exactly as before.
+	// The same shape as the usage assertion above, and for the same reason:
+	// the sink is the router when the router is serving, and nothing when a
+	// control plane runs on its own.
+	var lead *control.Leadership
+	if h, ok := sink.(interface{ Store() *storepkg.DB }); ok {
+		if st := h.Store(); st != nil {
+			lead = control.NewLeadership(st, cfg.Control.LeaseTTL, log)
+			ctl.SetLeadership(lead)
+			log.Info("competing to be the control plane that places work",
+				"owner", lead.Owner())
+		}
+	}
 	svc := agentsvc.New(ctl.Inventory(), ctl.Reconciler(), requireIdentity, log)
+	svc.SetEngines(cfg.WantedEngines())
+	log.Info("nodes install these engines", "engines", cfg.WantedEngines())
+	// A standby must refuse agents as well as refusing to place. Commands
+	// travel down the stream the agent opened, so a standby holding one is a
+	// node the leader cannot command — and the leader's placements would fail
+	// with "no agent connected" for a node that is plainly connected.
+	if lead != nil {
+		svc.SetLeadership(lead)
+	}
 	fwd.set(svc)
 
 	ctl.Start(ctx)
@@ -372,6 +446,26 @@ func (r *dispatcherRef) get() (reconciler.Dispatcher, error) {
 	return r.d, nil
 }
 
+// RemoveWeights forwards a capability beyond scheduling.
+//
+// It has to be spelled out here rather than inherited, because this type
+// satisfies reconciler.Dispatcher by forwarding each method by hand — so
+// anything the controller reaches by type assertion stops at this boundary
+// unless it is forwarded too. A capability added to the service and not to
+// this type is simply absent at runtime, with nothing at compile time to say
+// so.
+func (r *dispatcherRef) RemoveWeights(ctx context.Context, nodeID string, paths []string) error {
+	d, err := r.get()
+	if err != nil {
+		return err
+	}
+	rm, ok := d.(control.WeightRemover)
+	if !ok {
+		return errors.New("the agent service cannot remove weights")
+	}
+	return rm.RemoveWeights(ctx, nodeID, paths)
+}
+
 func (r *dispatcherRef) StartShard(ctx context.Context, cmd reconciler.ShardCommand) error {
 	d, err := r.get()
 	if err != nil {

internal/cli/sourcenote.go

@@ -0,0 +1,17 @@
+package cli
+
+import "fmt"
+
+// sourceNote is what to print about a checked source.
+//
+// It says where the answer came from, because the control plane's network is
+// not the nodes' network: a mirror this machine can reach may be unreachable
+// from a node behind a different firewall, and the reverse. The check catches
+// typos and a mirror that is down, which is most of what goes wrong, and it
+// should not be read as proof that the fetch will work.
+func sourceNote(source string, size int64) string {
+	if size > 0 {
+		return fmt.Sprintf("  source reachable from here: %s (%s)", source, fileSize(size))
+	}
+	return fmt.Sprintf("  source reachable from here: %s (size not advertised)", source)
+}

internal/cli/sourcenote_test.go

@@ -0,0 +1,16 @@
+package cli
+
+import (
+	"strings"
+	"testing"
+)
+
+func TestTheNoteSaysWhereTheAnswerCameFrom(t *testing.T) {
+	// The control plane's network is not the nodes' network. The check catches
+	// typos and a mirror that is down; it is not proof the fetch will work,
+	// and the wording must not suggest it is.
+	note := sourceNote("http://models.internal/x.gguf", 1<<30)
+	if !strings.Contains(note, "from here") {
+		t.Errorf("the note overstates what was checked: %s", note)
+	}
+}

internal/cli/storeflag.go

@@ -0,0 +1,76 @@
+package cli
+
+import (
+	"context"
+	"fmt"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// defaultStorePath is where a single-machine deployment keeps its store.
+const defaultStorePath = "/var/lib/llmgw/llmgw.db"
+
+// defaultConfigPath is where the control plane's configuration usually lives.
+//
+// Consulted rather than required. The commands that work on the store run on
+// the same machine as the server far more often than not, and the server
+// already knows where the store is — asking it is better than making the
+// operator repeat an answer that is written down three lines away.
+const defaultConfigPath = "/etc/llmgw/server.yaml"
+
+// storeFlags is how a command that works directly on the store finds it.
+type storeFlags struct {
+	address string
+	config  string
+}
+
+// bind adds --store and --config to a command.
+func (f *storeFlags) bind(cmd *cobra.Command) {
+	cmd.PersistentFlags().StringVar(&f.address, "store", "",
+		"store address: a SQLite path, a postgres:// DSN, or @file holding one "+
+			"(default: auth.store from --config, else "+defaultStorePath+")")
+	cmd.PersistentFlags().StringVar(&f.config, "config", defaultConfigPath,
+		"server configuration to read the store address from")
+}
+
+// open resolves the store and opens it.
+//
+// Explicit --store wins; otherwise the server's own configuration is asked,
+// and only then does the single-machine default apply. That order exists
+// because the default was a SQLite path and nothing else: on a cluster whose
+// store is PostgreSQL, every one of these commands failed with "unable to open
+// database file" while the right answer sat in the config file beside it. The
+// operator's conclusion was that the tool was broken, and they were not far off.
+func (f *storeFlags) open(cmd *cobra.Command) (*store.DB, error) {
+	return f.openCtx(cmd.Context())
+}
+
+// openCtx is open for a caller that has a context rather than a command.
+func (f *storeFlags) openCtx(ctx context.Context) (*store.DB, error) {
+	address, from := f.resolve()
+	s, err := store.Open(ctx, address)
+	if err != nil {
+		return nil, fmt.Errorf("opening the store at %s (%s): %w", address, from, err)
+	}
+	return s, nil
+}
+
+// resolve returns the address and where it came from, for the error message.
+//
+// Where it came from is half the message. "unable to open database file" says
+// nothing about which of three sources chose that path, and the whole problem
+// is that the wrong one did.
+func (f *storeFlags) resolve() (address, from string) {
+	if f.address != "" {
+		return f.address, "--store"
+	}
+	if f.config != "" {
+		if cfg, err := config.Load(f.config); err == nil && cfg.Auth.Store != "" {
+			return cfg.Auth.Store, "auth.store in " + f.config
+		}
+	}
+	return defaultStorePath, "the default; no --store and nothing usable in " + f.config
+}

internal/cli/storeflag_test.go

@@ -0,0 +1,82 @@
+package cli
+
+import (
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+)
+
+func writeConfig(t *testing.T, body string) string {
+	t.Helper()
+	p := filepath.Join(t.TempDir(), "server.yaml")
+	if err := os.WriteFile(p, []byte(body), 0o600); err != nil {
+		t.Fatal(err)
+	}
+	return p
+}
+
+func TestAnExplicitStoreWins(t *testing.T) {
+	cfg := writeConfig(t, "listen: \":8080\"\nauth:\n  enabled: true\n  store: postgres://from-config/db\n")
+	f := storeFlags{address: "postgres://explicit/db", config: cfg}
+
+	got, from := f.resolve()
+	if got != "postgres://explicit/db" || !strings.Contains(from, "--store") {
+		t.Errorf("an explicit --store should win; got %q from %q", got, from)
+	}
+}
+
+func TestTheServerConfigIsAskedBeforeTheDefault(t *testing.T) {
+	cfg := writeConfig(t, "listen: \":8080\"\nauth:\n  enabled: true\n  store: postgres://from-config/db\n")
+	f := storeFlags{config: cfg}
+
+	got, from := f.resolve()
+	// The whole defect: on a cluster whose store is PostgreSQL, these commands
+	// reached for a SQLite path and failed with "unable to open database file"
+	// while the right answer sat in the config beside them.
+	if got != "postgres://from-config/db" {
+		t.Errorf("expected the address from the config, got %q", got)
+	}
+	if !strings.Contains(from, "auth.store") {
+		t.Errorf("the source should be named for the error message; got %q", from)
+	}
+}
+
+func TestTheSingleMachineDefaultStillApplies(t *testing.T) {
+	f := storeFlags{config: filepath.Join(t.TempDir(), "absent.yaml")}
+
+	got, from := f.resolve()
+	// A deployment with no config file is the single-machine one, and it worked
+	// before this existed. It has to keep working with no flags at all.
+	if got != defaultStorePath {
+		t.Errorf("expected the default path, got %q", got)
+	}
+	if !strings.Contains(from, "default") {
+		t.Errorf("the source should say it fell back; got %q", from)
+	}
+}
+
+func TestAConfigWithNoStoreFallsThrough(t *testing.T) {
+	// Authentication off means no store at all. Reading an empty value as the
+	// answer would open a database named "".
+	cfg := writeConfig(t, "listen: \":8080\"\n")
+	f := storeFlags{config: cfg}
+
+	if got, _ := f.resolve(); got != defaultStorePath {
+		t.Errorf("expected the default when the config names no store, got %q", got)
+	}
+}
+
+func TestTheErrorSaysWhichSourceChoseThePath(t *testing.T) {
+	f := storeFlags{address: filepath.Join(t.TempDir(), "nope", "x.db")}
+
+	_, err := f.openCtx(t.Context())
+	if err == nil {
+		t.Fatal("expected opening a store in a missing directory to fail")
+	}
+	// "unable to open database file" says nothing about which of three sources
+	// picked that path, and the whole problem is that the wrong one did.
+	if !strings.Contains(err.Error(), "--store") {
+		t.Errorf("the error should name where the address came from; got %q", err)
+	}
+}

internal/cli/threads.go

@@ -0,0 +1,70 @@
+package cli
+
+import (
+	"fmt"
+
+	"github.com/spf13/cobra"
+)
+
+// newThreadsCmd turns conversation storage on for a project, or off.
+//
+// A command rather than a configuration key because it is a decision about
+// data, not about deployment: someone has to choose to keep what a team types,
+// and choosing it should look like an act rather than like a default nobody
+// read.
+//
+// Until it is run for a project, llmgw stores no message content for that
+// project — usage rows carry token counts and costs, never text — and that is
+// how every project starts.
+func newThreadsCmd(open opener) *cobra.Command {
+	var (
+		project string
+		hours   int
+		off     bool
+	)
+
+	cmd := &cobra.Command{
+		Use:   "threads",
+		Short: "Keep conversations for a project, so a thread can move between models",
+		Long: "With this on, a client sends x-llmgw-thread and only its new turn, and\n" +
+			"llmgw replays the conversation to whichever model serves — including a\n" +
+			"different one from the one it started on.\n\n" +
+			"Off for every project until this is run. llmgw otherwise stores no\n" +
+			"message content anywhere, which is a much easier thing to have deployed\n" +
+			"in front of a team, so turning it on is deliberate.\n\n" +
+			"The retention is required rather than optional: a conversation store with\n" +
+			"no expiry is a transcript of everything a team ever typed, kept forever\n" +
+			"because nobody chose a number.",
+		Args: cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			if !off && hours <= 0 {
+				return fmt.Errorf("--retain-hours is required: say how long conversations are kept, or pass --off")
+			}
+			s, err := open(cmd.Context())
+			if err != nil {
+				return err
+			}
+			defer s.Close()
+
+			keep := hours
+			if off {
+				keep = 0
+			}
+			if err := s.SetThreadRetention(cmd.Context(), project, keep); err != nil {
+				return err
+			}
+			if keep == 0 {
+				fmt.Fprintf(cmd.OutOrStdout(),
+					"%s keeps no conversations. Existing ones are removed at the next prune.\n", project)
+				return nil
+			}
+			fmt.Fprintf(cmd.OutOrStdout(),
+				"%s keeps conversations for %d hours, measured from the last turn.\n", project, keep)
+			return nil
+		},
+	}
+	cmd.Flags().StringVar(&project, "project", "default", "project id")
+	cmd.Flags().IntVar(&hours, "retain-hours", 0, "how long a conversation is kept after its last turn")
+	cmd.Flags().BoolVar(&off, "off", false, "stop keeping conversations for this project")
+	return cmd
+}

internal/cli/toolcheck.go

@@ -24,8 +24,16 @@ import (
 // Found the hard way. qwen2.5-coder produced exactly the right call —
 // {"name": "list_files", "arguments": {"path": "."}} — as plain reply text,
 // with tool_calls null, and diagnosing that took an afternoon of eliminating
-// the gateway, the model size, the chat template and tool_choice in turn. The
-// answer was the engine build. This runs that elimination in one command.
+// the gateway, the model size, the chat template and tool_choice in turn.
+//
+// The answer was neither the engine nor the model alone, which is why this
+// command now distinguishes them by what the reply looks like rather than
+// asserting one. The engine rendered the tool prompt correctly, including the
+// <tool_call> wrapper its template asks the model to use; the model wrapped
+// the call in a markdown fence instead, and the parser — which looks for that
+// wrapper — left it in the content. A differently trained model fixes that. A
+// bigger one from the same family does not: qwen2.5-coder-7b invented
+// <function-call> and failed the same way.
 func newToolCheckCmd() *cobra.Command {
 	var (
 		endpoint string
@@ -81,8 +89,17 @@ type ToolProbe struct {
 	// separates "the model ignored its tools" from "the engine did not parse
 	// the call the model made" — the same symptom, different fix.
 	Attempted bool
-	Content   string
-	Raw       string
+	// Wrapped means the attempt used the <tool_call> tags that the Qwen and
+	// Hermes templates instruct the model to use.
+	//
+	// This is what separates the two halves of a failed attempt. Wrapped and
+	// unparsed is the engine's parser failing on output that was correct.
+	// Unwrapped is the model answering in a shape of its own — the engine had
+	// no way to recognise it, and the fix is a model trained for the format,
+	// not a newer engine.
+	Wrapped bool
+	Content string
+	Raw     string
 }
 
 func probeToolCall(ctx context.Context, endpoint, model string) (ToolProbe, error) {
@@ -162,6 +179,7 @@ func readToolProbe(raw []byte) (ToolProbe, error) {
 	// The model tried, and the engine handed the attempt back as text. The
 	// tell is the tool's name in a reply that was supposed to be a call.
 	p.Attempted = p.Structured || bytes.Contains([]byte(msg.Content), []byte("get_weather"))
+	p.Wrapped = bytes.Contains([]byte(msg.Content), []byte("<tool_call>"))
 	return p, nil
 }
 
@@ -170,12 +188,22 @@ func report(cmd *cobra.Command, p ToolProbe) {
 	switch {
 	case p.Structured:
 		fmt.Fprintln(out, "tool calls work: the engine returned a populated tool_calls array.")
-	case p.Attempted:
-		fmt.Fprintln(out, "the model called the tool and the engine did not parse it.")
+	case p.Attempted && p.Wrapped:
+		fmt.Fprintln(out, "the model called the tool correctly and the engine did not parse it.")
 		fmt.Fprintln(out, "\n  The call is in the reply text instead of tool_calls:")
 		fmt.Fprintf(out, "    %s\n", trim(p.Content, 200))
-		fmt.Fprintln(out, "\n  This is the engine, not the model. Choosing a bigger model will not")
-		fmt.Fprintln(out, "  help. Check the engine's build and its chat template support.")
+		fmt.Fprintln(out, "\n  It used the <tool_call> wrapper its template asks for, so the model")
+		fmt.Fprintln(out, "  did its part. This is the engine's parser. Check its build; another")
+		fmt.Fprintln(out, "  model will fail the same way.")
+	case p.Attempted:
+		fmt.Fprintln(out, "the model called the tool in a shape the engine does not recognise.")
+		fmt.Fprintln(out, "\n  The call is in the reply text instead of tool_calls:")
+		fmt.Fprintf(out, "    %s\n", trim(p.Content, 200))
+		fmt.Fprintln(out, "\n  The engine rendered the tools correctly — the model knew the tool")
+		fmt.Fprintln(out, "  existed and what to pass it — but did not use the <tool_call> wrapper")
+		fmt.Fprintln(out, "  its template asked for, so there was nothing for the parser to find.")
+		fmt.Fprintln(out, "\n  Fixed by a model trained for this format, not by a bigger one: the")
+		fmt.Fprintln(out, "  instruct models generally manage it where the coder variants do not.")
 	default:
 		fmt.Fprintln(out, "the model did not try to call the tool.")
 		fmt.Fprintf(out, "\n  It replied: %s\n", trim(p.Content, 200))

internal/cli/toolcheck_test.go

@@ -19,7 +19,7 @@ func TestAStructuredToolCallIsRecognised(t *testing.T) {
 	}
 }
 
-func TestACallReturnedAsTextIsBlamedOnTheEngine(t *testing.T) {
+func TestACallReturnedAsTextIsAnAttempt(t *testing.T) {
 	// The exact reply that cost an afternoon: the model made the right call and
 	// the engine handed it back as prose.
 	p, err := readToolProbe([]byte(`{"choices":[{"message":{"role":"assistant",
@@ -33,6 +33,50 @@ func TestACallReturnedAsTextIsBlamedOnTheEngine(t *testing.T) {
 	if !p.Attempted {
 		t.Error("the model plainly tried; that must be distinguished from ignoring the tools")
 	}
+	if p.Wrapped {
+		t.Error("bare JSON was read as having used the <tool_call> wrapper")
+	}
+}
+
+func TestAWrappedCallBlamesTheEngineAndABareOneDoesNot(t *testing.T) {
+	// Both look identical from outside — tool_calls null, the call sitting in
+	// the content — and they have opposite fixes. The whole point of the
+	// command is telling them apart, and the earlier version did not: it
+	// blamed the engine for both, which sends someone off rebuilding llama.cpp
+	// when the answer is a different model.
+	wrapped, err := readToolProbe([]byte(`{"choices":[{"message":{"role":"assistant",
+	  "content":"<tool_call>\n{\"name\": \"get_weather\", \"arguments\": {\"city\": \"Istanbul\"}}\n</tool_call>"}}]}`))
+	if err != nil {
+		t.Fatal(err)
+	}
+	if !wrapped.Attempted || !wrapped.Wrapped {
+		t.Errorf("a correctly wrapped call was not recognised as one: %+v", wrapped)
+	}
+
+	// What qwen2.5-coder-3b actually returns, measured: the right call in a
+	// markdown fence, which the parser has no way to find.
+	fenced, err := readToolProbe([]byte(`{"choices":[{"message":{"role":"assistant",
+	  "content":"` + "```" + `json\n{\"name\": \"get_weather\", \"arguments\": {\"city\": \"Istanbul\"}}\n` + "```" + `"}}]}`))
+	if err != nil {
+		t.Fatal(err)
+	}
+	if !fenced.Attempted {
+		t.Error("a fenced call is still an attempt")
+	}
+	if fenced.Wrapped {
+		t.Error("a markdown fence was mistaken for the <tool_call> wrapper")
+	}
+
+	// And what qwen2.5-coder-7b returns: a tag it invented, which is not the
+	// one the template asked for and is not parsed either.
+	invented, err := readToolProbe([]byte(`{"choices":[{"message":{"role":"assistant",
+	  "content":"<function-call>\n{\"name\": \"get_weather\", \"arguments\": {\"city\": \"Istanbul\"}}\n</function-call>"}}]}`))
+	if err != nil {
+		t.Fatal(err)
+	}
+	if !invented.Attempted || invented.Wrapped {
+		t.Errorf("an invented tag was read as the template's wrapper: %+v", invented)
+	}
 }
 
 func TestAModelThatIgnoresItsToolsIsBlamedOnTheModel(t *testing.T) {

internal/cli/usage.go

@@ -0,0 +1,74 @@
+package cli
+
+import (
+	"encoding/json"
+	"fmt"
+	"text/tabwriter"
+	"time"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+)
+
+// newUsageCmd shows what each group of the tenancy tree spent.
+//
+// Quotas could be written on an organization, a team or a project long before
+// anything could show what any of them had used: usage came back per key, per
+// user and per model only. A quota whose consumption cannot be seen is one
+// nobody reviews until the day it refuses somebody.
+func newUsageCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
+	var by string
+	var since time.Duration
+
+	c := &cobra.Command{
+		Use:   "usage",
+		Short: "What each organization, team or project has spent",
+		Long: "Grouped by a level of the tenancy tree, which is the level a quota is\n" +
+			"written on. Spend is summed across every key beneath it, so two people\n" +
+			"sharing a team budget appear as one number — the number that will refuse\n" +
+			"them.\n\n" +
+			"The budget column is what that level sets, and is blank when it sets none.\n" +
+			"A team bounded only by its organization has no budget of its own, and\n" +
+			"printing the organization's beside each of its teams would read as though\n" +
+			"each had that much to spend.",
+		Args: cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			var rows []adminapi.ScopeUsageView
+			path := fmt.Sprintf("/api/v1/usage?by=%s&since=%s", by, since)
+			if err := client().get(cmd.Context(), path, &rows); err != nil {
+				return err
+			}
+			if *asJSON {
+				return json.NewEncoder(cmd.OutOrStdout()).Encode(rows)
+			}
+			if len(rows) == 0 {
+				fmt.Fprintf(cmd.OutOrStdout(), "No %s used anything in the last %s.\n", by, since)
+				return nil
+			}
+
+			w := tabwriter.NewWriter(cmd.OutOrStdout(), 0, 0, 2, ' ', 0)
+			fmt.Fprintln(w, "NAME\tID\tKEYS\tREQUESTS\tERRORS\tTOKENS\tSPEND\tBUDGET\tUSED\tLAST")
+			for _, u := range rows {
+				budget, used := "—", "—"
+				if u.Budget != nil {
+					budget = fmt.Sprintf("%.2f", float64(*u.Budget)/1e6)
+					if u.BudgetPeriod != "" {
+						budget += "/" + u.BudgetPeriod
+					}
+				}
+				if u.Fraction != nil {
+					used = fmt.Sprintf("%.0f%%", *u.Fraction*100)
+				}
+				fmt.Fprintf(w, "%s\t%s\t%d\t%d\t%d\t%d\t%.2f\t%s\t%s\t%s\n",
+					u.Name, u.ID, u.Keys, u.Requests, u.Errors,
+					u.Prompt+u.Completion, float64(u.CostMicros)/1e6,
+					budget, used, age(u.LastAt))
+			}
+			return w.Flush()
+		},
+	}
+	c.Flags().StringVar(&by, "by", "team", "level to group by: org, team or project")
+	c.Flags().DurationVar(&since, "since", 24*time.Hour, "how far back to total")
+	return c
+}

internal/cli/users.go

@@ -12,7 +12,7 @@ import (
 )
 
 func newUsersCmd() *cobra.Command {
-	var dbPath string
+	var sf storeFlags
 
 	cmd := &cobra.Command{
 		Use:   "users",
@@ -22,9 +22,9 @@ func newUsersCmd() *cobra.Command {
 			"the step that decides who gets to spend the cluster, and it needs a shell\n" +
 			"on this machine rather than a session cookie.",
 	}
-	cmd.PersistentFlags().StringVar(&dbPath, "store", "/var/lib/llmgw/llmgw.db", "path to the store")
-	open := func(cmd *cobra.Command) (*store.SQLite, error) {
-		return store.OpenSQLite(cmd.Context(), dbPath)
+	sf.bind(cmd)
+	open := func(cmd *cobra.Command) (*store.DB, error) {
+		return sf.open(cmd)
 	}
 
 	cmd.AddCommand(newUserListCmd(open), newUserApproveCmd(open),
@@ -38,7 +38,7 @@ func newUsersCmd() *cobra.Command {
 	return cmd
 }
 
-type userOpener func(*cobra.Command) (*store.SQLite, error)
+type userOpener func(*cobra.Command) (*store.DB, error)
 
 func newUserListCmd(open userOpener) *cobra.Command {
 	var pending bool

internal/config/config.go

@@ -52,6 +52,11 @@ type Server struct {
 	// Auth turns on API keys on the client-facing port.
 	Auth Auth `koanf:"auth"`
 
+	// Threads keeps conversations and agent memory without API keys, for a
+	// gateway on one trusted machine. With auth on they live in the auth store,
+	// per project, and this block is ignored.
+	Threads Threads `koanf:"threads"`
+
 	// Landing configures the page served at "/".
 	Landing Landing `koanf:"landing"`
 
@@ -60,6 +65,22 @@ type Server struct {
 	// trusted certificate, while this one is mutual and private.
 	AgentTLS tlsconf.Config `koanf:"agent_tls"`
 
+	// Monitoring is where the dashboard reads its history from.
+	Monitoring Monitoring `koanf:"monitoring"`
+
+	// Mirror points the control plane at a model mirror it can administer.
+	// Empty means there is none, and /admin/mirror stays unmounted.
+	Mirror Mirror `koanf:"mirror"`
+
+	// Engines are the engines every node installs, and at which version. The
+	// agents install them themselves — the right build for their hardware,
+	// checked against its publisher's checksum and seen to run — so a node is
+	// never trusted to already have what somebody once put there.
+	//
+	// Unset keeps DefaultEngines. A version replaces the default; an empty
+	// one leaves that engine to be installed by hand.
+	Engines map[string]string `koanf:"engines"`
+
 	Router  Router          `koanf:"router"`
 	Control Control         `koanf:"control"`
 	Models  []catalog.Model `koanf:"models"`
@@ -70,6 +91,57 @@ type Server struct {
 	Endpoints []Endpoint `koanf:"endpoints"`
 }
 
+// Monitoring points the dashboard at a Prometheus holding this cluster's
+// history.
+//
+// Read back rather than kept twice. llmgw publishes these metrics already; what
+// a gauge cannot do is remember, and a ring buffer in this process would be a
+// rival copy with its own retention, lost on every restart and disagreeing with
+// the one an operator already has on screen.
+type Monitoring struct {
+	// Prometheus is the server's address, for example
+	// http://monitoring.internal:9090. Empty leaves the charts off the
+	// dashboard rather than showing an error where one should be.
+	Prometheus string `koanf:"prometheus"`
+	// Window is how far back the charts look. Zero takes an hour.
+	Window time.Duration `koanf:"window"`
+	// Timeout bounds one query. It runs while somebody waits for a page, so a
+	// monitoring system that is slow today is skipped rather than waited for.
+	Timeout time.Duration `koanf:"timeout"`
+}
+
+// Mirror is a model mirror this control plane administers over its API.
+//
+// The mirror is a separate machine with a large disk, not part of the gateway,
+// so what is configured here is an address and how to authenticate to it —
+// never a directory. A control plane that could write to the mirror's disk
+// directly would be a way to replace a model with another one that every node
+// would then fetch without complaint.
+type Mirror struct {
+	// API is the mirror's management address, scheme included, for example
+	// https://models.internal:8082. Empty means no mirror is configured.
+	//
+	// The management port, not the one nodes fetch bytes from. Those are two
+	// listeners on the mirror precisely so this one can be locked down without
+	// re-certifying every agent's fetcher.
+	API string `koanf:"api"`
+	// TLS authenticates to the mirror's management API, with the same CA the
+	// agents use. Empty leaves the connection in plain HTTP, which is only
+	// acceptable on a network where nothing untrusted can reach that port —
+	// the mirror says as much in its own log when it starts that way.
+	TLS tlsconf.Config `koanf:"tls"`
+	// Registry is the model registry the mirror page browses, empty for the
+	// public Hugging Face. Set it to an internal mirror of the registry where
+	// the control plane cannot reach the internet — or leave the whole mirror
+	// block out, which turns the explorer off rather than giving it a search
+	// box that can only time out.
+	Registry string `koanf:"registry"`
+	// Timeout bounds a management call. It does not bound a download: the
+	// mirror answers a fetch as soon as it has started one, and the bytes keep
+	// arriving long after this deadline has passed.
+	Timeout time.Duration `koanf:"timeout"`
+}
+
 // Landing configures the public page at "/".
 //
 // It exists for one thing the gateway cannot know: who to ask for a key. A page
@@ -84,6 +156,22 @@ type Landing struct {
 	Notice string `koanf:"notice"`
 }
 
+// Threads configures conversation and memory storage for a gateway without
+// authentication.
+//
+// Everything stored is in one pool with no owner, readable by anyone who can
+// reach the client port and knows an id. That is acceptable on one machine and
+// wrong on a shared network, which is why it takes a setting rather than being
+// the default.
+type Threads struct {
+	// Store is the SQLite file. Empty means nothing is kept.
+	Store string `koanf:"store"`
+	// RetainHours is how long idle conversations and notes are kept. Required
+	// with Store: a store with no expiry keeps everything anyone typed, by
+	// omission.
+	RetainHours int `koanf:"retain_hours"`
+}
+
 // Auth configures client authentication (plan §7.9).
 //
 // Off by default, because a gateway on a trusted network with no keys issued
@@ -156,6 +244,27 @@ type Control struct {
 	PortRange []int `koanf:"port_range"`
 	// MgmtInterface is the ethernet interface NCCL bootstraps over.
 	MgmtInterface string `koanf:"mgmt_interface"`
+
+	// Enabled runs the control plane even when configuration declares no
+	// models.
+	//
+	// Without it, a deployment whose catalog lives entirely in the database
+	// cannot start: control plane mode is inferred from the config file, so an
+	// empty models block means "router only", the process refuses to start for
+	// want of anything to serve, and the models it would have loaded are never
+	// read. That is a cluster that cannot be managed at runtime because it is
+	// managed at runtime. AgentListen cannot carry this signal — it has a
+	// default, so every deployment would become a control plane.
+	Enabled bool `koanf:"enabled"`
+
+	// LeaseTTL is how long this control plane's claim to be the one placing
+	// work stands without being renewed.
+	//
+	// Only meaningful with a shared store, where a second control plane can
+	// see the claim. Long enough to survive a slow query or a brief network
+	// stall, short enough that a standby takes over before anyone has finished
+	// reading the alert. Zero takes the default.
+	LeaseTTL time.Duration `koanf:"lease_ttl"`
 }
 
 // Router holds routing behaviour.
@@ -391,6 +500,9 @@ func (s Server) Validate() error {
 	if s.Auth.Enabled && s.Auth.Store == "" {
 		return fmt.Errorf("auth.store is required when auth is enabled")
 	}
+	if s.Threads.Store != "" && s.Threads.RetainHours <= 0 {
+		return fmt.Errorf("threads.retain_hours must be positive when threads.store is set")
+	}
 
 	seen := make(map[string]struct{}, len(s.Endpoints))
 	for i, e := range s.Endpoints {
@@ -429,6 +541,9 @@ func (s Server) Catalog() (*catalog.Catalog, error) {
 // plus a static endpoint list is the router-only mode used for testing against
 // sim-engines.
 func (s Server) ControlPlaneMode() bool {
+	if s.Control.Enabled {
+		return true
+	}
 	for _, m := range s.Models {
 		if len(m.Profiles) > 0 {
 			return true
@@ -436,3 +551,25 @@ func (s Server) ControlPlaneMode() bool {
 	}
 	return false
 }
+
+// DefaultEngines are the engine versions this release was checked against:
+// `llmgw ctl engine-check` accepted every flag llmgw builds for them.
+var DefaultEngines = map[string]string{
+	"llamacpp": "b10991",
+}
+
+// WantedEngines is DefaultEngines with the configuration laid over it.
+func (s Server) WantedEngines() map[string]string {
+	out := map[string]string{}
+	for k, v := range DefaultEngines {
+		out[k] = v
+	}
+	for k, v := range s.Engines {
+		if v == "" {
+			delete(out, k)
+			continue
+		}
+		out[k] = v
+	}
+	return out
+}

internal/config/config_test.go

@@ -154,3 +154,48 @@ func TestValidateAcceptsDistinctAliases(t *testing.T) {
 		t.Errorf("valid aliases rejected: %v", err)
 	}
 }
+
+func TestAControlPlaneCanBeAskedForWithNoModelsInTheFile(t *testing.T) {
+	// Without this, a deployment whose catalog lives entirely in the database
+	// cannot start: the mode is inferred from the file, an empty models block
+	// reads as "router only", and the process refuses to start for want of
+	// anything to serve — so the stored models it would have loaded are never
+	// read at all.
+	var s Server
+	if s.ControlPlaneMode() {
+		t.Fatal("an empty configuration is not a control plane")
+	}
+	s.Control.Enabled = true
+	if !s.ControlPlaneMode() {
+		t.Error("control.enabled should run the control plane with no models declared")
+	}
+}
+
+func TestAModelWithProfilesStillMakesAControlPlane(t *testing.T) {
+	// The inference stays, so no existing configuration has to add a flag.
+	var s Server
+	s.Models = []catalog.Model{{
+		Name:     "m",
+		Profiles: []catalog.Profile{{Name: "p", Engine: "llamacpp"}},
+	}}
+	if !s.ControlPlaneMode() {
+		t.Error("a declared model with profiles is still a control plane")
+	}
+}
+
+func TestEnginesDefaultToTheCheckedVersions(t *testing.T) {
+	var s Server
+	if got := s.WantedEngines()["llamacpp"]; got != DefaultEngines["llamacpp"] {
+		t.Errorf("unset config wants llamacpp %q", got)
+	}
+	s.Engines = map[string]string{"llamacpp": "b11000", "vllm": "0.29.0"}
+	got := s.WantedEngines()
+	if got["llamacpp"] != "b11000" || got["vllm"] != "0.29.0" {
+		t.Errorf("configured versions not taken: %v", got)
+	}
+	// Empty leaves an engine to be installed by hand.
+	s.Engines = map[string]string{"llamacpp": ""}
+	if _, ok := s.WantedEngines()["llamacpp"]; ok {
+		t.Error("an emptied engine is still wanted")
+	}
+}

internal/config/example_test.go

@@ -39,8 +39,10 @@ func TestTheDocumentedAgentExampleLoads(t *testing.T) {
 	if f.RateGbps == 0 {
 		t.Error("the fabric's rate did not bind")
 	}
-	if cfg.NodeID == "" || cfg.Server == "" {
-		t.Errorf("node_id=%q server=%q", cfg.NodeID, cfg.Server)
+	// node_id is left to the hostname, so the example need not say it; where
+	// the control plane is, nothing can find out.
+	if cfg.Server == "" {
+		t.Errorf("server=%q", cfg.Server)
 	}
 }
 

internal/control/adminapi/api.go

@@ -12,18 +12,24 @@ import (
 	"fmt"
 	"io"
 	"net/http"
+	"slices"
+	"strings"
 	"time"
 
+	"gitea.mynodes.xyz/ilker/llmgw/internal/bench"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
 )
 
 // API serves the management endpoints.
 type API struct {
-	models Models
-	ctl    *control.Controller
+	models   Models
+	capacity Capacity
+	usage    Usage
+	ctl      *control.Controller
 }
 
 // New builds the API over a controller.
@@ -37,9 +43,17 @@ func (a *API) Register(mux *http.ServeMux) {
 	// it does: reaching this port already means holding the control plane.
 	mux.HandleFunc("POST /api/v1/models", a.addModel)
 	mux.HandleFunc("DELETE /api/v1/models/{name}", a.removeModel)
+	mux.HandleFunc("POST /api/v1/models/{name}/scale", a.scaleModel)
+	mux.HandleFunc("PUT /api/v1/models/{name}/capacity", a.putCapacity)
+	mux.HandleFunc("GET /api/v1/capacity", a.listCapacity)
+	mux.HandleFunc("GET /api/v1/usage", a.usageByScope)
 	mux.HandleFunc("GET /api/v1/nodes", a.listNodes)
 	mux.HandleFunc("GET /api/v1/models", a.listModels)
 	mux.HandleFunc("GET /api/v1/replicas", a.listReplicas)
+	// Listing is a read and deleting is a separate verb, so a mistyped
+	// command cannot remove anything.
+	mux.HandleFunc("GET /api/v1/weights/orphans", a.listOrphans)
+	mux.HandleFunc("POST /api/v1/weights/collect", a.collectWeights)
 	mux.HandleFunc("GET /api/v1/explain-placement", a.explainPlacement)
 	mux.HandleFunc("POST /api/v1/nodes/{id}/state", a.setNodeState)
 }
@@ -60,11 +74,38 @@ type NodeView struct {
 	// what decides whether a CPU profile can be placed here, so it belongs
 	// where someone looks when a model will not schedule.
 	MemoryGB int `json:"memory_gb,omitempty"`
+	// Engines are the inference engines installed here, as the agent found
+	// them. Absent means the agent is older than the probe and said nothing,
+	// which is not the same as having none — placement treats it that way too,
+	// or a control plane upgrade would empty a fleet.
+	Engines []string `json:"engines,omitempty"`
+	// Provision is where installing each wanted engine stands. An engine is in
+	// Engines only once its installation here is ready.
+	Provision []inventory.EngineInstall `json:"provision,omitempty"`
+	// OS is the operating system and architecture, as the agent read them.
+	OS string `json:"os,omitempty"`
+	// Warnings are facts about the machine that make it slow or unusable
+	// without making it unhealthy: a VM with no vector instructions runs every
+	// model, at a fifth of the speed.
+	Warnings []string `json:"warnings,omitempty"`
+
 	// Models are the weights this node can see. Absent means it was not asked
 	// to look, which is not the same as holding none.
-	Models  []string `json:"models_on_disk,omitempty"`
-	Classes []string `json:"gpu_classes,omitempty"`
-	Fabrics []string `json:"fabrics,omitempty"`
+	Models []string `json:"models_on_disk,omitempty"`
+	// DiskFreeGB and DiskTotalGB are the filesystems holding those weights,
+	// summed. Zero means the agent did not report any, which an older agent
+	// and a node with no model directories both do — so it reads as unknown
+	// rather than as a full disk.
+	//
+	// Summed because an operator asking "can this node take another model"
+	// wants one number; the per-filesystem detail is in the node's own record
+	// for anyone who needs it. Shared filesystems are left out of the sum: the
+	// space on a filer is not this node's to spend, and adding it would let a
+	// fleet of twenty nodes each believe it has the same free terabyte.
+	DiskFreeGB  int      `json:"disk_free_gb,omitempty"`
+	DiskTotalGB int      `json:"disk_total_gb,omitempty"`
+	Classes     []string `json:"gpu_classes,omitempty"`
+	Fabrics     []string `json:"fabrics,omitempty"`
 
 	// Unhealthy lists GPUs kept out of placement and why, which is usually the
 	// answer when a node looks idle but nothing will schedule onto it.
@@ -77,22 +118,31 @@ type NodeView struct {
 
 // ModelView is a model as `ctl models` shows it.
 type ModelView struct {
-	Name      string   `json:"name"`
-	Aliases   []string `json:"aliases,omitempty"`
-	Source    string   `json:"source"`
-	Min       int      `json:"min_replicas"`
-	Max       int      `json:"max_replicas"`
-	Ready     int      `json:"ready_replicas"`
-	Total     int      `json:"total_replicas"`
-	Fallbacks []string `json:"fallbacks,omitempty"`
-	Profiles  []string `json:"profiles"`
+	Name string `json:"name"`
+	// Description is what the model is for, when somebody said.
+	Description string   `json:"description,omitempty"`
+	Aliases     []string `json:"aliases,omitempty"`
+	Source      string   `json:"source"`
+	Min         int      `json:"min_replicas"`
+	Max         int      `json:"max_replicas"`
+	Ready       int      `json:"ready_replicas"`
+	Total       int      `json:"total_replicas"`
+	Fallbacks   []string `json:"fallbacks,omitempty"`
+	Profiles    []string `json:"profiles"`
 }
 
 // ReplicaView is a replica as `ctl replicas` shows it.
 type ReplicaView struct {
-	ID       string `json:"id"`
-	Model    string `json:"model"`
-	Profile  string `json:"profile"`
+	ID      string `json:"id"`
+	Model   string `json:"model"`
+	Profile string `json:"profile"`
+	// Stale is true when the profile has been edited since this replica was
+	// placed, so it is running a definition the catalog no longer holds.
+	//
+	// Reported rather than acted on. Moving a replica that is serving costs
+	// more than the imperfection, and the operator who made the edit is the
+	// one who should decide whether it is worth a restart.
+	Stale    bool   `json:"stale,omitempty"`
 	State    string `json:"state"`
 	Pool     string `json:"pool,omitempty"`
 	Fabric   string `json:"fabric,omitempty"`
@@ -115,6 +165,22 @@ type ShardView struct {
 	GPUs  []int  `json:"gpus"`
 	State string `json:"state"`
 	Err   string `json:"error,omitempty"`
+	// Fetch is the download this shard is waiting on, absent when it is not
+	// waiting on one. It is what tells a replica pulling a hundred gigabytes
+	// apart from one that is wedged — which look identical without it.
+	Fetch *FetchView `json:"fetch,omitempty"`
+}
+
+// FetchView is a download in flight.
+type FetchView struct {
+	Source    string `json:"source"`
+	BytesDone int64  `json:"bytes_done"`
+	// BytesTotal is 0 when the server did not say how large the file is.
+	BytesTotal int64 `json:"bytes_total,omitempty"`
+	// Percent is absent rather than zero when the total is unknown: an
+	// invented percentage that stops moving is worse than a byte count that
+	// keeps rising.
+	Percent *int `json:"percent,omitempty"`
 }
 
 // ExplainView is why a model could not be placed.
@@ -166,6 +232,22 @@ func (a *API) Nodes() []NodeView {
 			Fabrics:  n.FabricNames(), AgentVersion: n.AgentVersion,
 			Replicas: replicasOn[n.ID],
 		}
+		if free, total := n.LocalDisk(); total > 0 {
+			v.DiskFreeGB = int(free / (1 << 30))
+			v.DiskTotalGB = int(total / (1 << 30))
+		}
+		for name := range n.EngineVersions {
+			v.Engines = append(v.Engines, name)
+		}
+		slices.Sort(v.Engines)
+		v.Provision = n.Provision
+		if h := n.Host; h.OSID != "" {
+			v.OS = strings.TrimSpace(h.OSID + " " + h.OSVersion + " " + h.Arch)
+			if h.Arch == "amd64" && !h.HasCPUFlag("avx2") {
+				v.Warnings = append(v.Warnings, "no AVX2: inference runs several times slower "+
+					"(a VM needs CPU type \"host\")")
+			}
+		}
 		for _, w := range n.Weights {
 			v.Models = append(v.Models, w.Path)
 		}
@@ -205,7 +287,7 @@ func (a *API) Models() []ModelView {
 	out := make([]ModelView, 0)
 	for _, m := range a.ctl.Catalog().Models() {
 		v := ModelView{
-			Name: m.Name, Aliases: m.Aliases, Source: m.Source,
+			Name: m.Name, Description: m.Description, Aliases: m.Aliases, Source: m.Source,
 			Min: m.Scaling.Min, Max: m.Scaling.Max,
 			Ready: ready[m.Name], Total: total[m.Name],
 			Fallbacks: m.Fallbacks,
@@ -233,6 +315,7 @@ func (a *API) Replicas(model string) []ReplicaView {
 			ID: rep.ID, Model: rep.Model, Profile: rep.Profile,
 			State: rep.State.String(), Pool: rep.Pool, Fabric: rep.Fabric,
 			Endpoint: rep.Endpoint, Err: rep.Err,
+			Stale:   a.profileChanged(rep.Model, rep.Profile, rep.ProfileFP),
 			AgeSec:  int64(time.Since(rep.CreatedAt).Seconds()),
 			IdleSec: -1,
 		}
@@ -242,10 +325,19 @@ func (a *API) Replicas(model string) []ReplicaView {
 			v.IdleSec = int64(time.Since(rep.LastUsedAt).Seconds())
 		}
 		for _, s := range rep.Shards {
-			v.Shards = append(v.Shards, ShardView{
+			sv := ShardView{
 				ID: s.ID, Node: s.NodeID, Rank: s.Rank,
 				GPUs: s.GPUIndices, State: s.State.String(), Err: s.Err,
-			})
+			}
+			if f := s.Fetch; f != nil {
+				sv.Fetch = &FetchView{
+					Source: f.Source, BytesDone: f.BytesDone, BytesTotal: f.BytesTotal,
+				}
+				if p, ok := f.Percent(); ok {
+					sv.Fetch.Percent = &p
+				}
+			}
+			v.Shards = append(v.Shards, sv)
 		}
 		out = append(out, v)
 	}
@@ -259,6 +351,173 @@ func (a *API) Replicas(model string) []ReplicaView {
 type Models interface {
 	Add(ctx context.Context, m catalog.Model, by string) error
 	Remove(ctx context.Context, name string) error
+	Scale(ctx context.Context, name string, s catalog.Scaling, by string) (catalog.Model, error)
+}
+
+// Usage reports what each group of the tenancy tree spent. The store
+// implements it.
+type Usage interface {
+	UsageByScope(ctx context.Context, scope store.Scope, since time.Time) ([]store.ScopeUsage, error)
+}
+
+// SetUsage supplies the usage reporting source.
+func (a *API) SetUsage(u Usage) { a.usage = u }
+
+// ScopeUsageView is one group's spend, with its quota when it has one.
+type ScopeUsageView struct {
+	Scope string `json:"scope"`
+	ID    string `json:"id"`
+	Name  string `json:"name"`
+
+	Keys       int64 `json:"keys"`
+	Requests   int64 `json:"requests"`
+	Prompt     int64 `json:"prompt_tokens"`
+	Completion int64 `json:"completion_tokens"`
+	CostMicros int64 `json:"cost_micros"`
+	Errors     int64 `json:"errors"`
+
+	LastAt time.Time `json:"last_at,omitzero"`
+	// Budget is what this level sets, absent when it sets none. A group bounded
+	// only by its parent has no budget of its own, and reporting the parent's
+	// against each child would read as though each had that much.
+	Budget       *int64 `json:"budget_micros,omitempty"`
+	BudgetPeriod string `json:"budget_period,omitempty"`
+	// Fraction is spend over budget, absent for the same reason.
+	Fraction *float64 `json:"fraction,omitempty"`
+}
+
+func (a *API) usageByScope(w http.ResponseWriter, r *http.Request) {
+	if a.usage == nil {
+		writeError(w, http.StatusNotImplemented, "this control plane has no store, so nothing is recorded")
+		return
+	}
+	scope := store.Scope(r.URL.Query().Get("by"))
+	if scope == "" {
+		scope = store.ScopeTeam
+	}
+	since := 24 * time.Hour
+	if v := r.URL.Query().Get("since"); v != "" {
+		d, err := time.ParseDuration(v)
+		if err != nil || d <= 0 {
+			writeError(w, http.StatusBadRequest, "since must be a positive duration, like 24h")
+			return
+		}
+		since = d
+	}
+
+	rows, err := a.usage.UsageByScope(r.Context(), scope, time.Now().Add(-since))
+	if err != nil {
+		// The only failure a caller can act on is naming a level that is not
+		// one, which is what this mostly is.
+		writeError(w, http.StatusBadRequest, err.Error())
+		return
+	}
+	out := make([]ScopeUsageView, 0, len(rows))
+	for _, u := range rows {
+		v := ScopeUsageView{
+			Scope: string(u.Scope), ID: u.ID, Name: u.Name,
+			Keys: u.Keys, Requests: u.Requests, Prompt: u.Prompt,
+			Completion: u.Completion, CostMicros: u.CostMicros, Errors: u.Errors,
+			LastAt: u.LastAt, Budget: u.Budget, BudgetPeriod: string(u.BudgetPeriod),
+		}
+		if f, ok := u.Spent(); ok {
+			v.Fraction = &f
+		}
+		out = append(out, v)
+	}
+	writeJSON(w, http.StatusOK, out)
+}
+
+// Capacity is where measured capacity is kept. The store implements it.
+type Capacity interface {
+	PutCapacity(ctx context.Context, c store.StoredCapacity) error
+	Capacities(ctx context.Context) ([]store.StoredCapacity, error)
+}
+
+// SetCapacity supplies somewhere to keep benchmark results.
+func (a *API) SetCapacity(c Capacity) { a.capacity = c }
+
+// putCapacity records what a replica of a model was measured to serve.
+//
+// A PUT rather than a POST: re-running a benchmark replaces what was known
+// about that model on that profile. It is a better answer to the same question,
+// not another data point.
+func (a *API) putCapacity(w http.ResponseWriter, r *http.Request) {
+	if a.capacity == nil {
+		writeError(w, http.StatusNotImplemented,
+			"this control plane has no store, so a measurement recorded here would not survive a restart")
+		return
+	}
+	var c bench.Capacity
+	if err := json.NewDecoder(io.LimitReader(r.Body, 1<<20)).Decode(&c); err != nil {
+		writeError(w, http.StatusBadRequest, "the measurement is not valid JSON: "+err.Error())
+		return
+	}
+	c.Model = r.PathValue("name")
+	if c.Profile == "" {
+		writeError(w, http.StatusBadRequest,
+			"a measurement needs the profile it was taken on: the same model on CPU and on a GPU are different numbers")
+		return
+	}
+
+	// The model has to exist now, which catches a typo in the name — a
+	// measurement filed under a misspelling is worse than none, because it
+	// looks like the model was never benchmarked. It is not deleted when the
+	// model is, though: a benchmark costs real GPU time and outliving the
+	// catalog entry is the point.
+	m, ok := a.ctl.Catalog().Resolve(c.Model)
+	if !ok {
+		writeError(w, http.StatusNotFound, "no model named "+c.Model+" is in the catalog")
+		return
+	}
+	if !slices.ContainsFunc(m.Profiles, func(p catalog.Profile) bool { return p.Name == c.Profile }) {
+		writeError(w, http.StatusBadRequest,
+			"model "+c.Model+" has no profile named "+c.Profile)
+		return
+	}
+
+	body, err := json.Marshal(c)
+	if err != nil {
+		writeError(w, http.StatusInternalServerError, err.Error())
+		return
+	}
+	if err := a.capacity.PutCapacity(r.Context(), store.StoredCapacity{
+		Model: c.Model, Profile: c.Profile, Measurement: body,
+		MeasuredAt: c.MeasuredAt, MeasuredBy: r.Header.Get("X-Llmgw-Actor"),
+	}); err != nil {
+		writeError(w, http.StatusInternalServerError, err.Error())
+		return
+	}
+	writeJSON(w, http.StatusOK, c)
+}
+
+// CapacityView is one stored measurement, with who recorded it.
+type CapacityView struct {
+	bench.Capacity
+	MeasuredBy string `json:"measured_by,omitempty"`
+}
+
+func (a *API) listCapacity(w http.ResponseWriter, r *http.Request) {
+	if a.capacity == nil {
+		writeJSON(w, http.StatusOK, []CapacityView{})
+		return
+	}
+	rows, err := a.capacity.Capacities(r.Context())
+	if err != nil {
+		writeError(w, http.StatusInternalServerError, err.Error())
+		return
+	}
+	out := make([]CapacityView, 0, len(rows))
+	for _, row := range rows {
+		var c bench.Capacity
+		if err := json.Unmarshal(row.Measurement, &c); err != nil {
+			// Skipped rather than failing the listing: one row written by an
+			// older shape must not hide every other measurement.
+			continue
+		}
+		out = append(out, CapacityView{Capacity: c, MeasuredBy: row.MeasuredBy})
+	}
+	writeJSON(w, http.StatusOK, out)
 }
 
 // SetModels supplies the catalog-changing service.
@@ -285,6 +544,59 @@ func (a *API) addModel(w http.ResponseWriter, r *http.Request) {
 	writeJSON(w, http.StatusOK, map[string]string{"added": m.Name})
 }
 
+// ScaleRequest is how many replicas a model should run.
+type ScaleRequest struct {
+	Min int `json:"min"`
+	// Max omitted follows Min, which is what the catalog does for a model that
+	// declares only a minimum.
+	Max         int           `json:"max,omitempty"`
+	IdleTimeout time.Duration `json:"idle_timeout,omitempty"`
+}
+
+// ScaleResponse reports the scaling that is now in force.
+//
+// What is in force rather than what was asked for: the catalog fills in a max
+// that was left out, and a caller that echoed its own request would print a
+// number the cluster is not using.
+type ScaleResponse struct {
+	Model       string        `json:"model"`
+	Min         int           `json:"min"`
+	Max         int           `json:"max"`
+	IdleTimeout time.Duration `json:"idle_timeout,omitempty"`
+	OnDemand    bool          `json:"on_demand"`
+}
+
+func (a *API) scaleModel(w http.ResponseWriter, r *http.Request) {
+	if a.models == nil {
+		writeError(w, http.StatusNotImplemented, "this control plane has no store")
+		return
+	}
+	var req ScaleRequest
+	if err := json.NewDecoder(io.LimitReader(r.Body, 1<<16)).Decode(&req); err != nil {
+		writeError(w, http.StatusBadRequest, "the scaling request is not valid JSON: "+err.Error())
+		return
+	}
+	if req.Min < 0 || req.Max < 0 {
+		writeError(w, http.StatusBadRequest, "a replica count cannot be negative")
+		return
+	}
+	name := r.PathValue("name")
+	m, err := a.models.Scale(r.Context(), name,
+		catalog.Scaling{Min: req.Min, Max: req.Max, IdleTimeout: req.IdleTimeout},
+		r.Header.Get("X-Llmgw-Actor"))
+	if err != nil {
+		// Like addModel: every refusal here is the request being wrong — a
+		// model declared in configuration, one that was never added, a max
+		// below a min. None are server faults.
+		writeError(w, http.StatusBadRequest, err.Error())
+		return
+	}
+	writeJSON(w, http.StatusOK, ScaleResponse{
+		Model: m.Name, Min: m.Scaling.Min, Max: m.Scaling.Max,
+		IdleTimeout: m.Scaling.IdleTimeout, OnDemand: m.Scaling.OnDemand(),
+	})
+}
+
 func (a *API) removeModel(w http.ResponseWriter, r *http.Request) {
 	if a.models == nil {
 		writeError(w, http.StatusNotImplemented, "this control plane has no store")
@@ -389,3 +701,27 @@ func writeJSON(w http.ResponseWriter, code int, v any) {
 func writeError(w http.ResponseWriter, code int, msg string) {
 	writeJSON(w, code, map[string]string{"error": msg})
 }
+
+// profileChanged reports whether a replica is running a profile the catalog no
+// longer holds.
+//
+// A replica with no recorded fingerprint is not stale: that is every replica
+// adopted from an agent older than the field, and calling those stale would
+// mark a whole fleet as wrong on the first upgrade. Nor is one whose profile
+// has been deleted outright — the model or the profile going away is a
+// different thing, and the replica's own state already says it.
+func (a *API) profileChanged(model, profile, was string) bool {
+	if was == "" {
+		return false
+	}
+	m, ok := a.ctl.Catalog().Resolve(model)
+	if !ok {
+		return false
+	}
+	for _, p := range m.Profiles {
+		if p.Name == profile {
+			return p.Fingerprint() != was
+		}
+	}
+	return false
+}

internal/control/adminapi/api_test.go

@@ -9,6 +9,7 @@ import (
 	"net/http/httptest"
 	"strings"
 	"testing"
+	"time"
 
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control"
@@ -16,6 +17,7 @@ import (
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
 )
 
 // nullSink discards published endpoints; these tests are about the API.
@@ -252,9 +254,24 @@ type fakeModels struct {
 	added   []catalog.Model
 	actors  []string
 	removed []string
+	scaled  []catalog.Scaling
 	err     error
 }
 
+func (f *fakeModels) Scale(_ context.Context, name string, s catalog.Scaling, by string) (catalog.Model, error) {
+	if f.err != nil {
+		return catalog.Model{}, f.err
+	}
+	f.scaled = append(f.scaled, s)
+	f.actors = append(f.actors, by)
+	// Max following min is what the catalog does; the fake has to do it too or
+	// the handler test would pass while the real answer reported a zero.
+	if s.Max == 0 && s.Min > 0 {
+		s.Max = s.Min
+	}
+	return catalog.Model{Name: name, Scaling: s}, nil
+}
+
 func (f *fakeModels) Add(_ context.Context, m catalog.Model, by string) error {
 	if f.err != nil {
 		return f.err
@@ -353,3 +370,300 @@ func TestRemovingAModelOverTheAPI(t *testing.T) {
 		t.Errorf("removed = %v", fm.removed)
 	}
 }
+
+func TestScalingAModelOverTheAPI(t *testing.T) {
+	a, _ := setup(t, testModels())
+	fm := &fakeModels{}
+	a.SetModels(fm)
+	mux := http.NewServeMux()
+	a.Register(mux)
+
+	req := httptest.NewRequest(http.MethodPost, "/api/v1/models/m1/scale",
+		strings.NewReader(`{"min":3}`))
+	req.Header.Set("X-Llmgw-Actor", "ilker")
+	rec := httptest.NewRecorder()
+	mux.ServeHTTP(rec, req)
+
+	if rec.Code != http.StatusOK {
+		t.Fatalf("expected 200, got %d: %s", rec.Code, rec.Body)
+	}
+	var res ScaleResponse
+	if err := json.Unmarshal(rec.Body.Bytes(), &res); err != nil {
+		t.Fatal(err)
+	}
+	// The scaling in force, not the request echoed back: max follows min when
+	// it is left out, and a caller printing its own request would show 0.
+	if res.Min != 3 || res.Max != 3 {
+		t.Errorf("expected 3/3 in force, got %d/%d", res.Min, res.Max)
+	}
+	if len(fm.actors) != 1 || fm.actors[0] != "ilker" {
+		t.Errorf("the change should be attributed to whoever made it; got %v", fm.actors)
+	}
+}
+
+func TestANegativeReplicaCountIsRefused(t *testing.T) {
+	a, _ := setup(t, testModels())
+	fm := &fakeModels{}
+	a.SetModels(fm)
+	mux := http.NewServeMux()
+	a.Register(mux)
+
+	req := httptest.NewRequest(http.MethodPost, "/api/v1/models/m1/scale",
+		strings.NewReader(`{"min":-1}`))
+	rec := httptest.NewRecorder()
+	mux.ServeHTTP(rec, req)
+
+	if rec.Code != http.StatusBadRequest {
+		t.Errorf("expected 400, got %d", rec.Code)
+	}
+	if len(fm.scaled) != 0 {
+		t.Error("nothing should have reached the service")
+	}
+}
+
+type memCapacity struct{ rows []store.StoredCapacity }
+
+func (m *memCapacity) PutCapacity(_ context.Context, c store.StoredCapacity) error {
+	m.rows = append(m.rows, c)
+	return nil
+}
+
+func (m *memCapacity) Capacities(context.Context) ([]store.StoredCapacity, error) {
+	return m.rows, nil
+}
+
+func capacityAPI(t *testing.T) (*API, *memCapacity, *http.ServeMux) {
+	t.Helper()
+	a, _ := setup(t, testModels())
+	cap := &memCapacity{}
+	a.SetCapacity(cap)
+	mux := http.NewServeMux()
+	a.Register(mux)
+	return a, cap, mux
+}
+
+func TestAMeasurementIsFiledAgainstItsModelAndProfile(t *testing.T) {
+	_, cap, mux := capacityAPI(t)
+
+	body := `{"profile":"tp8","max_concurrency_at_slo":8,"suggested_target_waiting":3}`
+	req := httptest.NewRequest(http.MethodPut, "/api/v1/models/llama/capacity", strings.NewReader(body))
+	req.Header.Set("X-Llmgw-Actor", "ilker")
+	rec := httptest.NewRecorder()
+	mux.ServeHTTP(rec, req)
+
+	if rec.Code != http.StatusOK {
+		t.Fatalf("expected 200, got %d: %s", rec.Code, rec.Body)
+	}
+	if len(cap.rows) != 1 {
+		t.Fatalf("expected one row, got %d", len(cap.rows))
+	}
+	// The model comes from the path, not the body: a measurement filed under a
+	// name the caller made up would be worse than none.
+	if cap.rows[0].Model != "llama" || cap.rows[0].Profile != "tp8" {
+		t.Errorf("wrong key: %s/%s", cap.rows[0].Model, cap.rows[0].Profile)
+	}
+	if cap.rows[0].MeasuredBy != "ilker" {
+		t.Errorf("the measurement should say who took it; got %q", cap.rows[0].MeasuredBy)
+	}
+}
+
+func TestAMeasurementForAModelThatIsNotThereIsRefused(t *testing.T) {
+	_, cap, mux := capacityAPI(t)
+
+	req := httptest.NewRequest(http.MethodPut, "/api/v1/models/typo/capacity",
+		strings.NewReader(`{"profile":"tp8"}`))
+	rec := httptest.NewRecorder()
+	mux.ServeHTTP(rec, req)
+
+	// A typo in the name files the measurement somewhere nobody will look, and
+	// the model then looks as though it was never benchmarked.
+	if rec.Code != http.StatusNotFound {
+		t.Errorf("expected 404 for an unknown model, got %d", rec.Code)
+	}
+	if len(cap.rows) != 0 {
+		t.Error("nothing should have been stored")
+	}
+}
+
+func TestAMeasurementNeedsTheProfileItWasTakenOn(t *testing.T) {
+	_, cap, mux := capacityAPI(t)
+
+	req := httptest.NewRequest(http.MethodPut, "/api/v1/models/llama/capacity",
+		strings.NewReader(`{"max_concurrency_at_slo":8}`))
+	rec := httptest.NewRecorder()
+	mux.ServeHTTP(rec, req)
+
+	// Capacity belongs to the hardware as much as the weights. Without a
+	// profile there is nothing to compare a later run against.
+	if rec.Code != http.StatusBadRequest {
+		t.Errorf("expected 400 without a profile, got %d", rec.Code)
+	}
+	if len(cap.rows) != 0 {
+		t.Error("nothing should have been stored")
+	}
+}
+
+func TestAProfileTheModelDoesNotHaveIsRefused(t *testing.T) {
+	_, _, mux := capacityAPI(t)
+
+	req := httptest.NewRequest(http.MethodPut, "/api/v1/models/llama/capacity",
+		strings.NewReader(`{"profile":"not-a-profile"}`))
+	rec := httptest.NewRecorder()
+	mux.ServeHTTP(rec, req)
+
+	if rec.Code != http.StatusBadRequest {
+		t.Errorf("expected 400 for a profile the model does not have, got %d", rec.Code)
+	}
+}
+
+func TestCapacityWithoutAStoreIsAnEmptyListRatherThanAnError(t *testing.T) {
+	a, _ := setup(t, testModels())
+	mux := http.NewServeMux()
+	a.Register(mux)
+
+	rec := httptest.NewRecorder()
+	mux.ServeHTTP(rec, httptest.NewRequest(http.MethodGet, "/api/v1/capacity", nil))
+
+	// Nothing measured and no store look the same to a reader, and both mean
+	// "there is nothing here" — which is not a server fault.
+	if rec.Code != http.StatusOK {
+		t.Errorf("expected 200, got %d", rec.Code)
+	}
+	var rows []CapacityView
+	if err := json.Unmarshal(rec.Body.Bytes(), &rows); err != nil || len(rows) != 0 {
+		t.Errorf("expected an empty list, got %s", rec.Body)
+	}
+}
+
+type memUsage struct {
+	rows  []store.ScopeUsage
+	scope store.Scope
+	since time.Time
+	err   error
+}
+
+func (m *memUsage) UsageByScope(_ context.Context, s store.Scope, since time.Time) ([]store.ScopeUsage, error) {
+	m.scope, m.since = s, since
+	return m.rows, m.err
+}
+
+func usageAPI(t *testing.T, u *memUsage) *http.ServeMux {
+	t.Helper()
+	a, _ := setup(t, testModels())
+	a.SetUsage(u)
+	mux := http.NewServeMux()
+	a.Register(mux)
+	return mux
+}
+
+func TestUsageDefaultsToTeamsOverTheLastDay(t *testing.T) {
+	u := &memUsage{}
+	rec := httptest.NewRecorder()
+	usageAPI(t, u).ServeHTTP(rec, httptest.NewRequest(http.MethodGet, "/api/v1/usage", nil))
+
+	if rec.Code != http.StatusOK {
+		t.Fatalf("expected 200, got %d: %s", rec.Code, rec.Body)
+	}
+	// The team is the level a quota is usually written on, so it is the one
+	// somebody asking for "usage" with no argument means.
+	if u.scope != store.ScopeTeam {
+		t.Errorf("expected teams by default, got %q", u.scope)
+	}
+	if d := time.Since(u.since); d < 23*time.Hour || d > 25*time.Hour {
+		t.Errorf("expected roughly a day, got %v", d)
+	}
+}
+
+func TestUsageReportsTheFractionOnlyWhenThereIsABudget(t *testing.T) {
+	budget := int64(2000)
+	u := &memUsage{rows: []store.ScopeUsage{
+		{Scope: store.ScopeTeam, ID: "t1", Name: "platform", CostMicros: 1000,
+			Budget: &budget, BudgetPeriod: store.PeriodMonth},
+		{Scope: store.ScopeTeam, ID: "t2", Name: "research", CostMicros: 100},
+	}}
+	rec := httptest.NewRecorder()
+	usageAPI(t, u).ServeHTTP(rec, httptest.NewRequest(http.MethodGet, "/api/v1/usage", nil))
+
+	var got []ScopeUsageView
+	if err := json.Unmarshal(rec.Body.Bytes(), &got); err != nil {
+		t.Fatal(err)
+	}
+	if got[0].Fraction == nil || *got[0].Fraction != 0.5 {
+		t.Errorf("expected half the budget used, got %v", got[0].Fraction)
+	}
+	// A team bounded only by its organization has no budget of its own; a
+	// fraction of nothing would have to invent a denominator.
+	if got[1].Fraction != nil || got[1].Budget != nil {
+		t.Errorf("a group with no budget should report neither, got %+v", got[1])
+	}
+}
+
+func TestAnUnknownGroupingIsTheCallersMistake(t *testing.T) {
+	u := &memUsage{err: errors.New(`usage cannot be grouped by "model"`)}
+	rec := httptest.NewRecorder()
+	usageAPI(t, u).ServeHTTP(rec,
+		httptest.NewRequest(http.MethodGet, "/api/v1/usage?by=model", nil))
+
+	if rec.Code != http.StatusBadRequest {
+		t.Errorf("expected 400, got %d", rec.Code)
+	}
+}
+
+func TestABadSinceIsRefusedBeforeTheQuery(t *testing.T) {
+	u := &memUsage{}
+	rec := httptest.NewRecorder()
+	usageAPI(t, u).ServeHTTP(rec,
+		httptest.NewRequest(http.MethodGet, "/api/v1/usage?since=lastweek", nil))
+
+	if rec.Code != http.StatusBadRequest {
+		t.Errorf("expected 400, got %d", rec.Code)
+	}
+	if !u.since.IsZero() {
+		t.Error("nothing should have reached the store")
+	}
+}
+
+func TestAReplicaRunningAnEditedProfileIsMarkedStale(t *testing.T) {
+	models := testModels()
+	a, ctl := setup(t, models)
+
+	// Placed from the profile as it was, then the profile is edited and keeps
+	// its name — which is what makes the two indistinguishable without a
+	// fingerprint.
+	was := models[0].Profiles[0].Fingerprint()
+	edited := models[0]
+	edited.Profiles[0].MinMemoryGB = 64
+	built, err := catalog.New([]catalog.Model{edited})
+	if err != nil {
+		t.Fatal(err)
+	}
+	ctl.ReplaceCatalog(t.Context(), built)
+
+	if !a.profileChanged("llama", "tp8", was) {
+		t.Error("an edited profile should read as changed")
+	}
+	if a.profileChanged("llama", "tp8", edited.Profiles[0].Fingerprint()) {
+		t.Error("the profile in force is not a change")
+	}
+}
+
+func TestAReplicaWithNoRecordedProfileIsNotStale(t *testing.T) {
+	a, _ := setup(t, testModels())
+	// Every replica adopted from an agent older than the field. Marking those
+	// stale would call a whole fleet wrong on the first upgrade.
+	if a.profileChanged("llama", "tp8", "") {
+		t.Error("unknown is not stale")
+	}
+}
+
+func TestAProfileThatIsGoneIsNotReportedAsStale(t *testing.T) {
+	a, _ := setup(t, testModels())
+	// The model or the profile disappearing is a different thing, and the
+	// replica's own state already says it.
+	if a.profileChanged("llama", "no-such-profile", "abc") {
+		t.Error("a missing profile is not a changed one")
+	}
+	if a.profileChanged("no-such-model", "tp8", "abc") {
+		t.Error("a missing model is not a changed profile")
+	}
+}

internal/control/adminapi/weights.go

@@ -0,0 +1,107 @@
+package adminapi
+
+import (
+	"encoding/json"
+	"net/http"
+	"sort"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/weights"
+)
+
+// OrphanView is cached weights on one node that no catalog model accounts for.
+type OrphanView struct {
+	Node  string   `json:"node"`
+	Paths []string `json:"paths"`
+	// Bytes is what deleting them would free, as far as the node reported. A
+	// file whose size was not reported counts as zero rather than being left
+	// out, so the list and the number always describe the same set.
+	Bytes int64 `json:"bytes"`
+}
+
+// Orphans finds cached weights nothing in the catalog points at.
+//
+// Computed here from what nodes already report, so listing costs no round trip
+// and a node that is offline simply contributes nothing — its files are not
+// deletable while it is gone anyway.
+//
+// Only files llmgw fetched are ever considered. Everything else in a model
+// directory was put there by the operator and is none of this system's
+// business, however unreferenced it looks.
+func (a *API) Orphans() []OrphanView {
+	var wanted []string
+	for _, m := range a.ctl.Catalog().Models() {
+		wanted = append(wanted, m.Source)
+	}
+
+	out := make([]OrphanView, 0)
+	for _, n := range a.ctl.Inventory().Nodes() {
+		size := map[string]int64{}
+		have := make([]string, 0, len(n.Weights))
+		for _, w := range n.Weights {
+			have = append(have, w.Path)
+			size[w.Path] = w.SizeBytes
+		}
+
+		paths := weights.Orphans(have, wanted)
+		if len(paths) == 0 {
+			continue
+		}
+		v := OrphanView{Node: n.ID, Paths: paths}
+		for _, p := range paths {
+			v.Bytes += size[p]
+		}
+		out = append(out, v)
+	}
+	sort.Slice(out, func(i, j int) bool { return out[i].Node < out[j].Node })
+	return out
+}
+
+func (a *API) listOrphans(w http.ResponseWriter, _ *http.Request) {
+	writeJSON(w, http.StatusOK, a.Orphans())
+}
+
+// collectWeights deletes them.
+//
+// A proposal, not an instruction: each node checks every path again before
+// unlinking it, and refuses anything it cannot prove it fetched. The two
+// checks are deliberate duplication — the control plane knows the catalog and
+// the node knows its disk, and only the node can be sure.
+func (a *API) collectWeights(w http.ResponseWriter, r *http.Request) {
+	rm := a.ctl.WeightRemover()
+	if rm == nil {
+		writeError(w, http.StatusNotImplemented,
+			"this control plane cannot ask nodes to delete files")
+		return
+	}
+
+	var body struct {
+		// Node limits it to one node. Empty means every node, which is the
+		// common case after removing a model from the catalog.
+		Node string `json:"node,omitempty"`
+	}
+	// An empty body is the "all nodes" case and must not be an error.
+	_ = json.NewDecoder(r.Body).Decode(&body)
+
+	type result struct {
+		Node  string `json:"node"`
+		Files int    `json:"files"`
+		Bytes int64  `json:"bytes"`
+		Error string `json:"error,omitempty"`
+	}
+	out := make([]result, 0)
+	for _, o := range a.Orphans() {
+		if body.Node != "" && o.Node != body.Node {
+			continue
+		}
+		res := result{Node: o.Node, Files: len(o.Paths), Bytes: o.Bytes}
+		// A node that has gone since the list was built is reported rather than
+		// failing the whole sweep: the other nodes' space is still worth
+		// freeing, and this one will be offered again next time.
+		if err := rm.RemoveWeights(r.Context(), o.Node, o.Paths); err != nil {
+			res.Error = err.Error()
+			res.Files, res.Bytes = 0, 0
+		}
+		out = append(out, res)
+	}
+	writeJSON(w, http.StatusOK, out)
+}

internal/control/agentsvc/convert.go

@@ -23,13 +23,28 @@ func NodeFromRegister(r *llmgwv1.Register) inventory.Node {
 		State:          inventory.Ready,
 		AgentVersion:   r.GetAgentVersion(),
 		EngineVersions: r.GetEngineVersions(),
+		EnginesProbed:  r.GetEnginesProbed(),
 		MemoryTotalMB:  int(r.GetMemoryTotalMb()),
 	}
+	if h := r.GetHost(); h != nil {
+		n.Host = discovery.Host{
+			OSID: h.GetOsId(), OSVersion: h.GetOsVersion(), OSCodename: h.GetOsCodename(),
+			Arch: h.GetArch(), CPUFlags: h.GetCpuFlags(), NVIDIADriver: h.GetNvidiaDriver(),
+		}
+	}
 	for _, w := range r.GetWeights() {
 		n.Weights = append(n.Weights, discovery.LocalWeights{
 			Path: w.GetPath(), SizeBytes: w.GetSizeBytes(), Shared: w.GetShared(),
 		})
 	}
+	for _, s := range r.GetStorage() {
+		n.Storage = append(n.Storage, discovery.Storage{
+			Path:       s.GetPath(),
+			FreeBytes:  s.GetFreeBytes(),
+			TotalBytes: s.GetTotalBytes(),
+			Shared:     s.GetShared(),
+		})
+	}
 	for _, g := range r.GetGpus() {
 		n.GPUs = append(n.GPUs, discovery.GPU{
 			Index:              int(g.GetIndex()),
@@ -75,6 +90,14 @@ func RegisterFromInventory(nodeID, hostname, mgmtAddr, agentVersion string,
 			Path: w.Path, SizeBytes: w.SizeBytes, Shared: w.Shared,
 		})
 	}
+	for _, s := range inv.Storage {
+		r.Storage = append(r.Storage, &llmgwv1.Storage{
+			Path:       s.Path,
+			FreeBytes:  s.FreeBytes,
+			TotalBytes: s.TotalBytes,
+			Shared:     s.Shared,
+		})
+	}
 	for _, g := range inv.GPUs {
 		r.Gpus = append(r.Gpus, &llmgwv1.GPU{
 			Index:              int32(g.Index),
@@ -120,6 +143,7 @@ func AdoptedFromRegister(nodeID string, shards []*llmgwv1.RunningShard) []reconc
 			Profile:    s.GetProfile(),
 			Pool:       s.GetPool(),
 			Fabric:     s.GetFabric(),
+			ProfileFP:  s.GetProfileFp(),
 		}
 		if a.Rank == 0 && s.GetLeaderAddr() != "" {
 			a.Endpoint = fmt.Sprintf("http://%s:%d", s.GetLeaderAddr(), s.GetHttpPort())
@@ -153,6 +177,7 @@ func StartShardFromCommand(cmd reconciler.ShardCommand) *llmgwv1.StartShard {
 		StartupTimeoutMs: cmd.StartupTimeout.Milliseconds(),
 		Pool:             cmd.Pool,
 		Fabric:           cmd.Fabric,
+		ProfileFp:        cmd.ProfileFP,
 	}
 }
 
@@ -172,6 +197,7 @@ func ShardSpecFromStart(m *llmgwv1.StartShard) (engines.ShardSpec, time.Duration
 		HTTPPort:      int(m.GetHttpPort()),
 		Pool:          m.GetPool(),
 		Fabric:        m.GetFabric(),
+		ProfileFP:     m.GetProfileFp(),
 		Profile: engines.Profile{
 			Name: p.GetName(), Engine: p.GetEngine(), GPUClass: p.GetGpuClass(),
 			TP: int(p.GetTp()), PP: int(p.GetPp()),

internal/control/agentsvc/heartbeat_test.go

@@ -0,0 +1,157 @@
+package agentsvc
+
+import (
+	"log/slog"
+	"testing"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
+)
+
+func nodeWith(s *Service, n inventory.Node) { s.inv.Upsert(n) }
+
+func newTestService() *Service {
+	return &Service{inv: inventory.NewStore(), conns: map[string]*conn{},
+		log: slog.New(slog.DiscardHandler)}
+}
+
+func TestDiskIsRefreshedWhileTheAgentRuns(t *testing.T) {
+	// It was sampled once, at registration, and never again. Found on the live
+	// cluster: an agent running for two days still reported the disk it had
+	// when it started, and a file deleted in between was still being offered
+	// for deletion. Disk is a fact about a moment, and the node whose figure
+	// matters most is the one busy filling up.
+	s := newTestService()
+	nodeWith(s, inventory.Node{
+		ID:    "n1",
+		State: inventory.Ready,
+		Storage: []discovery.Storage{
+			{Path: "/models", FreeBytes: 200 << 30, TotalBytes: 400 << 30},
+		},
+	})
+
+	s.applyHeartbeat("n1", &llmgwv1.Heartbeat{
+		NodeId: "n1",
+		Storage: []*llmgwv1.Storage{
+			{Path: "/models", FreeBytes: 5 << 30, TotalBytes: 400 << 30},
+		},
+	})
+
+	n, _ := s.inv.Get("n1")
+	free, total := n.LocalDisk()
+	if free != 5<<30 || total != 400<<30 {
+		t.Errorf("disk is %d of %d, want the figure the heartbeat carried", free, total)
+	}
+}
+
+func TestAnEmptyWeightListOnlyCountsWhenTheAgentMeansIt(t *testing.T) {
+	// Most heartbeats carry no file list because nothing changed. Treating
+	// that as "this node holds nothing" would empty a node's inventory five
+	// seconds after it registered, and placement would stop preferring the
+	// node that actually has the model.
+	s := newTestService()
+	nodeWith(s, inventory.Node{
+		ID:      "n1",
+		State:   inventory.Ready,
+		Weights: []discovery.LocalWeights{{Path: "/models/a.gguf", SizeBytes: 100}},
+	})
+
+	// Silence about weights: the list must survive.
+	s.applyHeartbeat("n1", &llmgwv1.Heartbeat{
+		NodeId:  "n1",
+		Storage: []*llmgwv1.Storage{{Path: "/models", FreeBytes: 1, TotalBytes: 2}},
+	})
+	if n, _ := s.inv.Get("n1"); len(n.Weights) != 1 {
+		t.Fatalf("a heartbeat with no weight list emptied the inventory: %+v", n.Weights)
+	}
+
+	// And when the agent says it means an empty list, it is taken.
+	s.applyHeartbeat("n1", &llmgwv1.Heartbeat{NodeId: "n1", WeightsChanged: true})
+	if n, _ := s.inv.Get("n1"); len(n.Weights) != 0 {
+		t.Errorf("a deliberate empty list was ignored: %+v", n.Weights)
+	}
+}
+
+func TestANewlyFetchedModelAppearsWithoutARestart(t *testing.T) {
+	// The other direction of the same bug: a model fetched after registration
+	// was invisible until the agent restarted, so placement could not prefer
+	// the node that had just spent ten minutes downloading it.
+	s := newTestService()
+	nodeWith(s, inventory.Node{ID: "n1", State: inventory.Ready})
+
+	s.applyHeartbeat("n1", &llmgwv1.Heartbeat{
+		NodeId:         "n1",
+		WeightsChanged: true,
+		Weights: []*llmgwv1.LocalWeights{
+			{Path: "/models/new.gguf", SizeBytes: 4 << 30},
+		},
+	})
+
+	n, _ := s.inv.Get("n1")
+	if !n.HasWeights("/models/new.gguf") {
+		t.Errorf("a model fetched after registration did not appear: %+v", n.Weights)
+	}
+}
+
+func TestAHeartbeatAboutNothingChangesNothing(t *testing.T) {
+	s := newTestService()
+	before := inventory.Node{
+		ID:      "n1",
+		State:   inventory.Ready,
+		Weights: []discovery.LocalWeights{{Path: "/models/a.gguf", SizeBytes: 100}},
+		Storage: []discovery.Storage{{Path: "/models", FreeBytes: 7, TotalBytes: 9}},
+	}
+	nodeWith(s, before)
+
+	s.applyHeartbeat("n1", &llmgwv1.Heartbeat{NodeId: "n1"})
+
+	n, _ := s.inv.Get("n1")
+	if len(n.Weights) != 1 {
+		t.Errorf("weights changed: %+v", n.Weights)
+	}
+	if free, _ := n.LocalDisk(); free != 7 {
+		t.Errorf("disk changed: %d", free)
+	}
+}
+
+func TestAHeartbeatCarriesMemoryAddedToARunningNode(t *testing.T) {
+	// Read once at registration, a node that was given 32 GB went on looking
+	// like the 7 GB machine it had been, and every model that needed the
+	// difference stayed unplaced until somebody restarted its agent.
+	s := newTestService()
+	nodeWith(s, inventory.Node{ID: "n1", State: inventory.Ready, MemoryTotalMB: 7 * 1024})
+
+	s.applyHeartbeat("n1", &llmgwv1.Heartbeat{NodeId: "n1", MemoryTotalMb: 32 * 1024})
+
+	if n, _ := s.inv.Get("n1"); n.MemoryTotalMB != 32*1024 {
+		t.Errorf("memory is %d MB, expected the 32 GB the node now has", n.MemoryTotalMB)
+	}
+
+	// Zero is an agent that could not read it, not a node that lost its RAM.
+	s.applyHeartbeat("n1", &llmgwv1.Heartbeat{NodeId: "n1"})
+	if n, _ := s.inv.Get("n1"); n.MemoryTotalMB != 32*1024 {
+		t.Errorf("a heartbeat without memory erased it: %d", n.MemoryTotalMB)
+	}
+}
+
+func TestAHeartbeatCarriesAnEngineInstalledAfterRegistration(t *testing.T) {
+	s := newTestService()
+	nodeWith(s, inventory.Node{ID: "n1", State: inventory.Ready,
+		EngineVersions: map[string]string{"llamacpp": "unknown"}})
+
+	// Unchanged engines are not resent, and silence must not clear them.
+	s.applyHeartbeat("n1", &llmgwv1.Heartbeat{NodeId: "n1", MemoryTotalMb: 1024})
+	if n, _ := s.inv.Get("n1"); len(n.EngineVersions) != 1 {
+		t.Fatalf("engines changed without being reported: %v", n.EngineVersions)
+	}
+
+	s.applyHeartbeat("n1", &llmgwv1.Heartbeat{
+		NodeId: "n1", EnginesChanged: true,
+		EngineVersions: map[string]string{"llamacpp": "unknown", "vllm": "unknown"},
+	})
+	n, _ := s.inv.Get("n1")
+	if _, ok := n.EngineVersions["vllm"]; !ok {
+		t.Errorf("the newly installed engine is not known: %v", n.EngineVersions)
+	}
+}

internal/control/agentsvc/service.go

@@ -6,6 +6,7 @@ import (
 	"fmt"
 	"io"
 	"log/slog"
+	"maps"
 	"net/http"
 	"slices"
 	"sync"
@@ -14,6 +15,7 @@ import (
 
 	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
 	"gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1/llmgwv1connect"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/supervisor"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
@@ -23,6 +25,10 @@ import (
 // agents report they are already running. The reconciler implements it.
 type StatusSink interface {
 	OnShardStatus(replicaID, shardID string, state supervisor.State, errMsg string)
+	// OnFetchProgress is separate because it is not a transition: the shard is
+	// still starting, and treating each report as a state change would reset
+	// the timers that decide when a slow start has become a failed one.
+	OnFetchProgress(replicaID, shardID, source string, done, total int64)
 	Adopt(nodeID string, shards []reconciler.AdoptedShard) (unclaimed []reconciler.AdoptedShard)
 }
 
@@ -30,6 +36,9 @@ type StatusSink interface {
 // time the reconciler's Dispatcher: it holds the open streams, so it is the
 // only thing that can deliver a command to a node.
 type Service struct {
+	// leader gates whether this process accepts agent streams at all.
+	leader Leader
+
 	inv    *inventory.Store
 	status StatusSink
 	log    *slog.Logger
@@ -40,8 +49,14 @@ type Service struct {
 
 	mu    sync.RWMutex
 	conns map[string]*conn
+
+	// engines is what every node is told to install, set once at start.
+	engines map[string]string
 }
 
+// SetEngines sets the engines every node installs, by name and version.
+func (s *Service) SetEngines(e map[string]string) { s.engines = maps.Clone(e) }
+
 // conn is one agent's open stream.
 type conn struct {
 	nodeID string
@@ -85,6 +100,19 @@ func (s *Service) Connect(
 	ctx context.Context,
 	stream *connect.BidiStream[llmgwv1.AgentMessage, llmgwv1.ControlMessage],
 ) error {
+	// A standby must not hold an agent's stream. Commands travel down the
+	// stream the agent opened, so whichever control plane holds it is the only
+	// one that can start or stop anything on that node — and if that is not
+	// the leader, the leader's placements fail with "no agent connected" for a
+	// node that is plainly connected.
+	//
+	// Refused before the opening message is read, so the agent spends nothing
+	// on a connection that cannot serve it.
+	if s.leader != nil && !s.leader.IsLeader() {
+		return connect.NewError(connect.CodeUnavailable,
+			errors.New("this control plane is a standby; connect to the leader"))
+	}
+
 	first, err := stream.Receive()
 	if err != nil {
 		return connect.NewError(connect.CodeInvalidArgument, fmt.Errorf("no opening message: %w", err))
@@ -135,6 +163,21 @@ func (s *Service) Connect(
 		return err
 	}
 
+	// What this node should have installed. Sent on every registration, so a
+	// node rebuilt from nothing is told the same as one that has been here a
+	// month, and a changed version reaches every node on its next connect.
+	if s.engines != nil {
+		p := &llmgwv1.Provision{}
+		for _, e := range slices.Sorted(maps.Keys(s.engines)) {
+			p.Engines = append(p.Engines, &llmgwv1.EngineWant{Engine: e, Version: s.engines[e]})
+		}
+		if err := stream.Send(&llmgwv1.ControlMessage{
+			Msg: &llmgwv1.ControlMessage_Provision{Provision: p},
+		}); err != nil {
+			return err
+		}
+	}
+
 	// Recover replica state from what this node is already running, before any
 	// reconciliation pass can conclude the cluster is empty and place
 	// duplicates (plan §6.5).
@@ -216,6 +259,14 @@ func (s *Service) handle(nodeID string, msg *llmgwv1.AgentMessage) {
 		s.applyHeartbeat(nodeID, m.Heartbeat)
 
 	case *llmgwv1.AgentMessage_Shard:
+		// Progress rather than a transition. Falling through to OnShardStatus
+		// would re-mark the shard as starting on every chunk, which is how a
+		// download that takes an hour would keep a start timer at zero.
+		if f := m.Shard.GetFetch(); f != nil {
+			s.status.OnFetchProgress(m.Shard.GetReplicaId(), m.Shard.GetShardId(),
+				f.GetSource(), f.GetBytesDone(), f.GetBytesTotal())
+			return
+		}
 		st := StateFromWire(m.Shard.GetState())
 		s.status.OnShardStatus(m.Shard.GetReplicaId(), m.Shard.GetShardId(), st, m.Shard.GetError())
 		if m.Shard.GetState() == llmgwv1.ShardStatus_STATE_FAILED {
@@ -239,7 +290,11 @@ func (s *Service) handle(nodeID string, msg *llmgwv1.AgentMessage) {
 func (s *Service) applyHeartbeat(nodeID string, hb *llmgwv1.Heartbeat) {
 	used := hb.GetGpuMemoryUsedMb()
 	problems := hb.GetNewProblems()
-	if len(used) == 0 && len(problems) == 0 {
+	storage := hb.GetStorage()
+	memMB := int(hb.GetMemoryTotalMb())
+	if len(used) == 0 && len(problems) == 0 && len(storage) == 0 &&
+		!hb.GetWeightsChanged() && memMB == 0 && !hb.GetEnginesChanged() &&
+		!hb.GetProvisionChanged() {
 		return
 	}
 
@@ -259,6 +314,77 @@ func (s *Service) applyHeartbeat(nodeID string, hb *llmgwv1.Heartbeat) {
 		// conservative direction.
 		n.GPUs[0].Health.Problems = append(n.GPUs[0].Health.Problems, problems...)
 	}
+
+	// Disk is a fact about a moment, not about a machine. Sampled only at
+	// registration it is wrong on exactly the node worth watching — the one
+	// busy filling up — and it stays wrong until that agent restarts.
+	if len(storage) > 0 {
+		n.Storage = n.Storage[:0]
+		for _, st := range storage {
+			n.Storage = append(n.Storage, discovery.Storage{
+				Path:       st.GetPath(),
+				FreeBytes:  st.GetFreeBytes(),
+				TotalBytes: st.GetTotalBytes(),
+				Shared:     st.GetShared(),
+			})
+		}
+	}
+
+	// Only when the agent says the list is meant. An empty list without the
+	// flag is silence, and treating it as "this node holds nothing" would
+	// empty a node's inventory on every heartbeat.
+	if hb.GetWeightsChanged() {
+		n.Weights = n.Weights[:0]
+		for _, w := range hb.GetWeights() {
+			n.Weights = append(n.Weights, discovery.LocalWeights{
+				Path: w.GetPath(), SizeBytes: w.GetSizeBytes(), Shared: w.GetShared(),
+			})
+		}
+	}
+
+	// RAM and engines change under a running agent too: memory added to a VM,
+	// an engine installed this afternoon. Said in the log when they do, because
+	// "the node now qualifies" is the answer to why a model placed an hour after
+	// it was asked for, and "the node no longer does" is the answer to why one
+	// stopped placing.
+	if memMB > 0 && memMB != n.MemoryTotalMB {
+		s.log.Info("node memory changed", "node", nodeID,
+			"from_mb", n.MemoryTotalMB, "to_mb", memMB)
+		n.MemoryTotalMB = memMB
+	}
+	if hb.GetEnginesChanged() {
+		vers := maps.Clone(hb.GetEngineVersions())
+		if !maps.Equal(vers, n.EngineVersions) {
+			s.log.Info("node engines changed", "node", nodeID,
+				"from", slices.Sorted(maps.Keys(n.EngineVersions)),
+				"to", slices.Sorted(maps.Keys(vers)))
+		}
+		n.EngineVersions = vers
+	}
+	if hb.GetProvisionChanged() {
+		prev := map[string]inventory.EngineInstall{}
+		for _, p := range n.Provision {
+			prev[p.Engine] = p
+		}
+		n.Provision = n.Provision[:0]
+		for _, p := range hb.GetProvision() {
+			e := inventory.EngineInstall{
+				Engine: p.GetEngine(), Version: p.GetVersion(), Variant: p.GetVariant(),
+				State: p.GetState(), Detail: p.GetDetail(),
+			}
+			// A state change is worth a line; a download's progress is not.
+			if old, ok := prev[e.Engine]; !ok || old.State != e.State {
+				lvl := slog.LevelInfo
+				if e.State == "failed" || e.State == "unsupported" {
+					lvl = slog.LevelWarn
+				}
+				s.log.Log(context.Background(), lvl, "engine installation on a node",
+					"node", nodeID, "engine", e.Engine, "version", e.Version,
+					"variant", e.Variant, "state", e.State, "detail", e.Detail)
+			}
+			n.Provision = append(n.Provision, e)
+		}
+	}
 	s.inv.Upsert(n)
 }
 
@@ -316,6 +442,16 @@ func (s *Service) ConnectedNodes() []string {
 
 // --- Dispatcher ---
 
+// Leader reports whether this process is the one that should be holding agent
+// streams.
+type Leader interface{ IsLeader() bool }
+
+// SetLeadership makes this service refuse agents while it is a standby.
+//
+// Nil accepts them unconditionally, which is what a single control plane wants
+// and what every deployment before leadership had.
+func (s *Service) SetLeadership(l Leader) { s.leader = l }
+
 // ErrNotConnected means no agent stream is open for that node.
 var ErrNotConnected = errors.New("no agent connected")
 
@@ -340,6 +476,22 @@ func (s *Service) DrainShard(_ context.Context, nodeID, shardID string) error {
 	})
 }
 
+// RemoveWeights asks a node to delete cached files no model needs.
+//
+// The node checks every path again before unlinking it, so this is a proposal
+// rather than an instruction — which is the right shape for a message that
+// deletes data the control plane cannot see.
+func (s *Service) RemoveWeights(_ context.Context, nodeID string, paths []string) error {
+	if len(paths) == 0 {
+		return nil
+	}
+	return s.send(nodeID, &llmgwv1.ControlMessage{
+		Msg: &llmgwv1.ControlMessage_RemoveWeights{
+			RemoveWeights: &llmgwv1.RemoveWeights{Paths: paths},
+		},
+	})
+}
+
 func (s *Service) send(nodeID string, msg *llmgwv1.ControlMessage) error {
 	s.mu.RLock()
 	c, ok := s.conns[nodeID]

internal/control/catalog/catalog.go

@@ -3,6 +3,9 @@
 package catalog
 
 import (
+	"crypto/sha256"
+	"encoding/hex"
+	"encoding/json"
 	"fmt"
 	"slices"
 	"strings"
@@ -71,6 +74,20 @@ type Profile struct {
 	// the control plane.
 	MinMemoryGB int `json:"min_memory_gb,omitempty" koanf:"min_memory_gb"`
 
+	// MinDiskGB is the free space a node needs before it can hold these
+	// weights.
+	//
+	// Only checked when the node does not already have them: a node holding
+	// the file needs no room for it, and requiring the space twice would keep
+	// a model off the one node best placed to run it.
+	//
+	// Declared rather than derived, because the size of a model that has never
+	// been fetched anywhere is not knowable from here — and a wrong guess
+	// fails late, halfway through a download, which is the failure this exists
+	// to prevent. Zero means unconstrained, and a node that reports no storage
+	// is not excluded: unknown is not full.
+	MinDiskGB int `json:"min_disk_gb,omitempty" koanf:"min_disk_gb"`
+
 	// NodeSelector restricts this profile to nodes carrying every one of these
 	// labels, with these values.
 	//
@@ -109,6 +126,17 @@ func cpuCapable(engine string) bool { return engine == "llamacpp" }
 type Model struct {
 	Name    string   `json:"name,omitempty" koanf:"name"`
 	Aliases []string `json:"aliases,omitempty" koanf:"aliases"`
+	// Description is what this model is for, in a line.
+	//
+	// Written by whoever added it, because nobody else can say it. A registry
+	// offers a pipeline tag and a wall of README, and neither answers the
+	// question a person picking from a list is asking: which of these nine
+	// names should I send this request to.
+	//
+	// It reaches clients through /v1/models, which is the only view most of
+	// them ever see. A gateway serving nine models by name alone makes that
+	// choice by guesswork.
+	Description string `json:"description,omitempty" koanf:"description"`
 	// Source is the weights path or a HuggingFace id.
 	Source    string    `json:"source,omitempty" koanf:"source"`
 	Profiles  []Profile `json:"profiles,omitempty" koanf:"profiles"`
@@ -344,6 +372,7 @@ func (m Model) PlacementProfiles() []placement.Profile {
 			Priority:     p.Priority,
 			NodeSelector: p.NodeSelector,
 			MinMemoryGB:  p.MinMemoryGB,
+			MinDiskGB:    p.MinDiskGB,
 			// The model's source, so placement can prefer nodes that already
 			// hold the weights rather than one that must first move them.
 			Source: m.Source,
@@ -418,3 +447,24 @@ func (m Model) Simulated() bool {
 	}
 	return true
 }
+
+// Fingerprint identifies a profile's contents.
+//
+// The name is not enough. A profile can be edited — a memory requirement
+// raised, a node label added, an engine argument changed — and keep its name,
+// so a replica running the old definition is indistinguishable from one
+// running the new. The reconciler does not move a running replica when its
+// profile changes, which is right: churning a replica that is serving costs
+// more than the imperfection. What was missing is any way to see that it had
+// happened, and the answer to "why is this one on the wrong node" was in
+// somebody's memory of what they edited an hour ago.
+func (p Profile) Fingerprint() string {
+	b, err := json.Marshal(p)
+	if err != nil {
+		// A profile that will not marshal cannot be compared, and saying so by
+		// never matching is better than saying two differing ones are the same.
+		return "unmarshalable"
+	}
+	sum := sha256.Sum256(b)
+	return hex.EncodeToString(sum[:8])
+}

internal/control/control.go

@@ -75,6 +75,8 @@ type Controller struct {
 	inv   *inventory.Store
 	cat   *catalog.Live
 	rec   *reconciler.Reconciler
+	lead  *Leadership
+	disp  reconciler.Dispatcher
 	sink  EndpointSink
 	usage UsageSource
 
@@ -101,6 +103,7 @@ func New(cfg Config, cat *catalog.Live, disp reconciler.Dispatcher, sink Endpoin
 		inv:  inv,
 		cat:  cat,
 		rec:  reconciler.New(cfg.Reconciler, cat, inv, disp, log),
+		disp: disp,
 		sink: sink,
 		stop: make(chan struct{}),
 		done: make(chan struct{}),
@@ -170,8 +173,42 @@ func (c *Controller) Reconciler() *reconciler.Reconciler { return c.rec }
 // Catalog exposes the model catalog.
 func (c *Controller) Catalog() *catalog.Catalog { return c.cat.Load() }
 
+// SetLeadership makes this control plane compete for the right to place work,
+// instead of assuming it is the only one.
+//
+// Wired after construction because deciding leadership needs a store, and the
+// store is opened by the router. Left unset, this process places
+// unconditionally — which is what a single control plane wants, and what every
+// deployment that existed before this had.
+func (c *Controller) SetLeadership(l *Leadership) {
+	c.lead = l
+	c.rec.SetLeadership(l)
+}
+
+// IsLeader reports whether this process is placing work. True when leadership
+// was never configured, because then there is nobody to compete with.
+func (c *Controller) IsLeader() bool { return c.lead == nil || c.lead.IsLeader() }
+
+// WeightRemover can ask a node to delete cached model files.
+//
+// Separate from Dispatcher because it is not part of scheduling and not every
+// dispatcher has it — the ones tests use do not. A caller gets nil rather than
+// a method that silently does nothing.
+type WeightRemover interface {
+	RemoveWeights(ctx context.Context, nodeID string, paths []string) error
+}
+
+// WeightRemover returns the dispatcher's deletion capability, or nil.
+func (c *Controller) WeightRemover() WeightRemover {
+	r, _ := c.disp.(WeightRemover)
+	return r
+}
+
 // Start runs the reconciler and begins publishing endpoints.
 func (c *Controller) Start(ctx context.Context) {
+	if c.lead != nil {
+		go c.lead.Run(ctx)
+	}
 	c.rec.Start(ctx)
 
 	go func() {

internal/control/fleetmetrics/fleetmetrics.go

@@ -45,6 +45,8 @@ type Collector struct {
 	nodeGPUsFree *prometheus.Desc
 	nodeMemory   *prometheus.Desc
 	nodeSeen     *prometheus.Desc
+	nodeDiskFree *prometheus.Desc
+	nodeDiskSize *prometheus.Desc
 
 	gpuMemoryTotal *prometheus.Desc
 	gpuMemoryUsed  *prometheus.Desc
@@ -76,6 +78,15 @@ func New(src Source) *Collector {
 			"A node's total RAM, 0 when it does not report one.", node, nil),
 		nodeSeen: prometheus.NewDesc("llmgw_node_last_seen_seconds",
 			"Seconds since a node last reported. Rising means an agent has gone quiet.", node, nil),
+		// Only the disk a node fills itself: space on a shared filer is not
+		// this node's to spend, and summing it per node would have a fleet of
+		// twenty each report the same free terabyte. Absent rather than zero
+		// when nothing was reported, so an alert on "free is low" does not
+		// fire for every node that was never asked about its disk.
+		nodeDiskFree: prometheus.NewDesc("llmgw_node_disk_free_bytes",
+			"Free space on the local filesystems holding model weights. Absent when the node reports no storage.", node, nil),
+		nodeDiskSize: prometheus.NewDesc("llmgw_node_disk_total_bytes",
+			"Size of those filesystems, so free space can be read as a fraction.", node, nil),
 
 		gpuMemoryTotal: prometheus.NewDesc("llmgw_gpu_memory_total_bytes",
 			"A GPU's framebuffer size.", gpu, nil),
@@ -111,6 +122,7 @@ func New(src Source) *Collector {
 func (c *Collector) Describe(ch chan<- *prometheus.Desc) {
 	for _, d := range []*prometheus.Desc{
 		c.nodeUp, c.nodeGPUs, c.nodeGPUsFree, c.nodeMemory, c.nodeSeen,
+		c.nodeDiskFree, c.nodeDiskSize,
 		c.gpuMemoryTotal, c.gpuMemoryUsed, c.gpuUtil, c.gpuTemp, c.gpuHealthy,
 		c.gpuNVLink, c.gpuECC,
 		c.replicaState, c.replicaReady, c.replicaIdle,
@@ -143,6 +155,15 @@ func (c *Collector) Collect(ch chan<- prometheus.Metric) {
 			float64(n.MemoryTotalMB)*1024*1024, n.ID)
 		ch <- prometheus.MustNewConstMetric(c.nodeSeen, prometheus.GaugeValue,
 			now.Sub(n.LastSeen).Seconds(), n.ID)
+		// Nothing reported means nothing is known, so no sample at all. A zero
+		// here would read as a full disk and page someone about a node that
+		// was simply never asked.
+		if free, total := n.LocalDisk(); total > 0 {
+			ch <- prometheus.MustNewConstMetric(c.nodeDiskFree, prometheus.GaugeValue,
+				float64(free), n.ID)
+			ch <- prometheus.MustNewConstMetric(c.nodeDiskSize, prometheus.GaugeValue,
+				float64(total), n.ID)
+		}
 
 		byClass := map[string]int{}
 		for _, g := range n.GPUs {

internal/control/inventory/inventory.go

@@ -84,6 +84,12 @@ type Node struct {
 	// Empty means the node was not asked to look, which is not the same as
 	// holding none — so it constrains nothing.
 	Weights []discovery.LocalWeights `json:"weights,omitempty"`
+	// Storage is the disk those weights sit on, one entry per filesystem.
+	//
+	// Empty means the node was not asked, not that it has no disk — so an
+	// agent too old to report it, or one with no model directories, is left
+	// unconstrained rather than treated as full.
+	Storage []discovery.Storage `json:"storage,omitempty"`
 
 	GPUs []discovery.GPU `json:"gpus"`
 	// Fabrics is empty on a node without IB. Such a node is its own
@@ -96,6 +102,24 @@ type Node struct {
 	// version decides which metric names and CLI flags apply (plan §14).
 	AgentVersion   string            `json:"agent_version,omitempty"`
 	EngineVersions map[string]string `json:"engine_versions,omitempty"`
+	// EnginesProbed says EngineVersions is an answer, so an empty map means
+	// "has none" rather than "an agent too old to say".
+	EnginesProbed bool `json:"engines_probed,omitempty"`
+
+	// Host is the operating system and processor, as the agent reads them.
+	Host discovery.Host `json:"host"`
+	// Provision is where installing each wanted engine stands on this node.
+	Provision []EngineInstall `json:"provision,omitempty"`
+}
+
+// EngineInstall is where one engine's installation stands on a node.
+type EngineInstall struct {
+	Engine  string `json:"engine"`
+	Version string `json:"version"`
+	Variant string `json:"variant,omitempty"`
+	// State is "installing", "ready", "failed" or "unsupported".
+	State  string `json:"state"`
+	Detail string `json:"detail,omitempty"`
 }
 
 // HasFabric reports whether the node is attached to a fabric.
@@ -141,6 +165,17 @@ type Allocation struct {
 	ShardID    string
 	ReplicaID  string
 	GPUIndices []int
+	// MemoryGB is the host RAM this shard's profile asked for.
+	//
+	// Recorded for the same reason the GPU indices are: it was claimed, and
+	// placement has to stop handing it out twice. Without it a node's memory
+	// was compared against its total on every decision, so two models each
+	// declaring most of a machine were both placed on it — and the second one
+	// is what the kernel kills.
+	//
+	// Zero for a profile that declared nothing, which constrains nothing and
+	// is counted as nothing.
+	MemoryGB int
 }
 
 // Store holds the cluster inventory. It is safe for concurrent use.
@@ -310,6 +345,28 @@ func (s *Store) FreeGPUs(nodeID string) []int {
 	return out
 }
 
+// LocalDisk sums the space on filesystems this node fills itself.
+//
+// Shared filesystems are left out on purpose. Space on a filer belongs to
+// everyone mounting it, and counting it per node would let twenty nodes each
+// believe they have the same free terabyte — then place twenty models into it.
+// A node whose weights are all on a shared mount therefore reports no local
+// disk, which is correct: it has none to spend.
+//
+// Both zero means nothing is known — an agent too old to report storage, or a
+// node with no model directories — and callers must read that as unknown
+// rather than as a full disk.
+func (n Node) LocalDisk() (free, total int64) {
+	for _, s := range n.Storage {
+		if s.Shared {
+			continue
+		}
+		free += s.FreeBytes
+		total += s.TotalBytes
+	}
+	return free, total
+}
+
 // HasWeights reports whether this node can see the given source.
 //
 // An exact path match, deliberately. A source that is not a path — a model id

internal/control/inventory/inventory_test.go

@@ -248,3 +248,42 @@ func TestConcurrentAccess(t *testing.T) {
 	}
 	wg.Wait()
 }
+
+func TestSharedFilesystemsAreNotThisNodesSpaceToSpend(t *testing.T) {
+	// Twenty nodes mounting one filer have 500 GB between them, not ten
+	// terabytes. Counting a shared mount per node is how a scheduler places
+	// twenty models into space for one.
+	n := Node{Storage: []discovery.Storage{
+		{Path: "/mnt/filer", FreeBytes: 500 << 30, TotalBytes: 900 << 30, Shared: true},
+		{Path: "/var/lib/llmgw-models", FreeBytes: 20 << 30, TotalBytes: 60 << 30},
+	}}
+	free, total := n.LocalDisk()
+	if free != 20<<30 || total != 60<<30 {
+		t.Errorf("shared space was counted as local: free=%d total=%d", free, total)
+	}
+}
+
+func TestANodeThatReportedNoStorageIsUnknownNotFull(t *testing.T) {
+	// An agent too old to report storage, and a node with no model directories,
+	// both send nothing. Reading that as a full disk would take every such node
+	// out of placement for a reason that is not true.
+	free, total := Node{}.LocalDisk()
+	if free != 0 || total != 0 {
+		t.Errorf("a node with no storage reported %d/%d", free, total)
+	}
+	// The contract callers rely on: total of zero is the signal for "unknown",
+	// which is why nothing may read free alone.
+	if total != 0 {
+		t.Error("callers cannot distinguish unknown from full")
+	}
+}
+
+func TestLocalDiskSumsSeveralDisks(t *testing.T) {
+	n := Node{Storage: []discovery.Storage{
+		{Path: "/a", FreeBytes: 10, TotalBytes: 100},
+		{Path: "/b", FreeBytes: 5, TotalBytes: 50},
+	}}
+	if free, total := n.LocalDisk(); free != 15 || total != 150 {
+		t.Errorf("free=%d total=%d, want 15/150", free, total)
+	}
+}

internal/control/leadership.go

@@ -0,0 +1,182 @@
+package control
+
+import (
+	"context"
+	"crypto/rand"
+	"encoding/hex"
+	"errors"
+	"log/slog"
+	"os"
+	"sync/atomic"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// LeaseStore is the part of the store leadership needs.
+//
+// An interface so the controller does not take a dependency on a concrete
+// database for one feature, and so a deployment with no store at all — a
+// router-only run, or a single control plane on a static configuration — keeps
+// working with nothing to configure.
+type LeaseStore interface {
+	AcquireLeadership(ctx context.Context, role, owner string, ttl time.Duration) (store.Lease, error)
+	ReleaseLeadership(ctx context.Context, role, owner string) error
+}
+
+// controlRole is the one thing there is to be leader of: placing work.
+const controlRole = "control-plane"
+
+// Leadership decides whether this process is the one that places work.
+//
+// Two control planes against one store both reconcile: both see a model
+// wanting two replicas and none running, and both place. The result is twice
+// the replicas, each holding GPUs the other believes are free, and nothing
+// crashes to say so.
+//
+// A standby is not idle — it serves its own routing and keeps its connection
+// to the store — it simply does not place, evict or drain. That makes running
+// a second one safe rather than clever, which is the property worth having
+// first: the common failure is not a planned failover, it is somebody starting
+// a second control plane by accident.
+type Leadership struct {
+	store LeaseStore
+	log   *slog.Logger
+	// owner identifies this process. Random rather than the hostname, so a
+	// restart on the same machine cannot be fooled by its own stale lease.
+	owner string
+	ttl   time.Duration
+	// renew is how often the holder pushes the expiry out. Well under ttl, so
+	// one slow query does not cost the leadership.
+	renew time.Duration
+
+	leader atomic.Bool
+}
+
+// NewLeadership prepares leadership over a store.
+//
+// A nil store means there is nobody to coordinate with, and this process is
+// the leader by definition — which is every single-control-plane deployment,
+// and must stay the case without anyone opting in.
+func NewLeadership(st LeaseStore, ttl time.Duration, log *slog.Logger) *Leadership {
+	if log == nil {
+		log = slog.Default()
+	}
+	if ttl <= 0 {
+		// Long enough to survive a slow database or a brief network stall,
+		// short enough that a standby takes over before anyone has finished
+		// reading the alert.
+		ttl = 15 * time.Second
+	}
+	l := &Leadership{
+		store: st,
+		log:   log,
+		owner: newOwnerID(),
+		ttl:   ttl,
+		renew: ttl / 3,
+	}
+	if st == nil {
+		l.leader.Store(true)
+	}
+	return l
+}
+
+// IsLeader reports whether this process should be placing work.
+func (l *Leadership) IsLeader() bool { return l.leader.Load() }
+
+// Owner is this process's id, for logs and for ctl.
+func (l *Leadership) Owner() string { return l.owner }
+
+// Run keeps the lease for as long as it can hold it, and keeps trying when it
+// cannot.
+//
+// A standby polls at the same interval the leader renews at, so a lapsed lease
+// is picked up within about a third of its life rather than at the end of it.
+func (l *Leadership) Run(ctx context.Context) {
+	if l.store == nil {
+		return
+	}
+	t := time.NewTicker(l.renew)
+	defer t.Stop()
+
+	l.attempt(ctx)
+	for {
+		select {
+		case <-ctx.Done():
+			l.step(ctx)
+			return
+		case <-t.C:
+			l.attempt(ctx)
+		}
+	}
+}
+
+// attempt takes or renews the lease, logging only transitions.
+//
+// Only transitions, because this runs every few seconds for the life of the
+// process: a line per attempt would bury everything else, and the thing an
+// operator needs to see is the moment leadership moved.
+func (l *Leadership) attempt(ctx context.Context) {
+	lease, err := l.store.AcquireLeadership(ctx, controlRole, l.owner, l.ttl)
+	switch {
+	case err == nil:
+		if l.leader.CompareAndSwap(false, true) {
+			l.log.Info("this control plane is now placing work", "owner", l.owner)
+		}
+	case errors.Is(err, store.ErrNotLeader):
+		if l.leader.CompareAndSwap(true, false) {
+			// Losing a lease that was held is worth a warning: it means this
+			// process was too slow to renew, and whatever made it slow is
+			// still true.
+			l.log.Warn("lost leadership; standing by",
+				"owner", l.owner, "now", lease.Owner)
+		} else if l.leader.Load() == false && lease.Owner != "" {
+			l.log.Debug("standing by", "leader", lease.Owner,
+				"until", lease.ExpiresAt.Format(time.RFC3339))
+		}
+	default:
+		// A store that cannot be reached is not evidence that somebody else
+		// is leading. Keeping the claim is the safer of two bad options: two
+		// leaders is worse than one, but no leader at all means nothing is
+		// placed and nothing is evicted, and a brief database blip should not
+		// stop a cluster from healing itself.
+		l.log.Warn("could not reach the store to renew leadership; keeping it for now",
+			"err", err)
+	}
+}
+
+// step gives the lease up on the way out, so a standby takes over in seconds
+// rather than waiting out the clock.
+func (l *Leadership) step(ctx context.Context) {
+	if !l.leader.Load() {
+		return
+	}
+	// A fresh context: the one that just ended is the reason this is running.
+	ctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 3*time.Second)
+	defer cancel()
+	if err := l.store.ReleaseLeadership(ctx, controlRole, l.owner); err != nil {
+		l.log.Warn("could not release leadership; a standby will wait it out", "err", err)
+		return
+	}
+	l.log.Info("released leadership", "owner", l.owner)
+}
+
+// newOwnerID is random, not the hostname.
+//
+// A hostname would let a restarted process match its own stale lease and
+// believe it had been leading all along — including through whatever gap made
+// it restart, during which another process may have taken over and started
+// placing.
+func newOwnerID() string {
+	var b [8]byte
+	if _, err := rand.Read(b[:]); err != nil {
+		// Only reached if the system has no entropy, where a pid and the time
+		// is still better than a fixed string.
+		return "ctl-" + hex.EncodeToString([]byte(time.Now().Format(time.RFC3339Nano)))[:16]
+	}
+	host, _ := os.Hostname()
+	if host == "" {
+		host = "ctl"
+	}
+	return host + "-" + hex.EncodeToString(b[:])
+}