Find LoRA adapters on Hugging Face and attach them from the mirror page

An adapters view in the explorer, filtered by the model a held model was
made from; GGUF files told apart as adapter or model by their first bytes;
held adapter files marked; and a model's card attaches and detaches held
adapters of its architecture, refusing models and mismatched architectures.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

.gitea/workflows/release.yml

@@ -55,6 +55,13 @@ jobs:
             (cd dist && sha256sum "${name}" >> checksums.txt)
           done
 
+          # The reverse proxy configurations, so a release page carries
+          # everything install.md asks for and the repository is not needed.
+          for f in deploy/proxy/Caddyfile deploy/proxy/nginx.conf; do
+            cp "$f" dist/
+            (cd dist && sha256sum "$(basename "$f")" >> checksums.txt)
+          done
+
           # An NVML build for nodes with NVLink, which the portable binary
           # cannot serve.
           #
@@ -86,6 +93,24 @@ jobs:
           ./dist/llmgw-${version}-linux-amd64 --version
           cat dist/checksums.txt
 
+      # Signed when the repository has a signing key, so a machine that trusts
+      # it installs nothing the server alone could have put there. Without the
+      # secret nothing is signed and releases are checked on their checksums,
+      # as before; a build that trusts a key refuses those, so the secret has
+      # to be in place before trusted.pub is filled in.
+      - name: Sign the checksums
+        env:
+          RELEASE_SIGNING_KEY: ${{ secrets.RELEASE_SIGNING_KEY }}
+        run: |
+          set -euo pipefail
+          version="${GITHUB_REF_NAME:-$(git describe --tags --always)}"
+          if [ -z "${RELEASE_SIGNING_KEY:-}" ]; then
+            echo "no RELEASE_SIGNING_KEY secret; releases are not signed"
+            exit 0
+          fi
+          ./dist/llmgw-${version}-linux-amd64 release sign dist/checksums.txt
+          echo "signed: dist/checksums.txt.sig"
+
       - name: Attach to the release
         env:
           TOKEN: ${{ secrets.GITEA_TOKEN }}
@@ -150,3 +175,31 @@ jobs:
               || { echo "upload of $(basename "$f") was refused" >&2; exit 1; }
           done
           echo "attached $(ls dist | wc -l) files to $tag"
+
+  # The release page says what changed, copied from CHANGELOG.md. Gitea makes
+  # an empty release for every tag and the job above only attaches files, so
+  # every release until this existed was a list of binaries and nothing else.
+  #
+  # After the build, because that is what guarantees the release exists. Run by
+  # hand from a branch it syncs every version in the changelog, which is how
+  # the releases made before this got their notes.
+  notes:
+    needs: build
+    runs-on: ubuntu-latest
+    permissions:
+      contents: write
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Copy the changelog onto the release
+        env:
+          GITEA_TOKEN: ${{ secrets.GITEA_TOKEN }}
+          GITEA_INSTANCE: ${{ github.server_url }}
+          GITEA_REPO: ${{ github.repository }}
+        run: |
+          set -euo pipefail
+          if [ "${GITHUB_REF_TYPE:-}" = "tag" ]; then
+            python3 scripts/gitea/release_notes.py --tag "${GITHUB_REF_NAME}"
+          else
+            python3 scripts/gitea/release_notes.py
+          fi

README.md

@@ -1,7 +1,7 @@
 # llmgw
 
-A single-binary LLM gateway for clusters with multiple InfiniBand fabrics and
-heterogeneous GPUs.
+A single-binary LLM gateway for clusters with multiple InfiniBand fabrics,
+NVLink domains and heterogeneous GPUs.
 
 It serves LLMs running on your own GPU servers (vLLM / SGLang) behind one
 OpenAI-compatible endpoint. Design document: [llmgw-plan.md](llmgw-plan.md).
@@ -32,6 +32,7 @@ Releases track the roadmap phases; see [CHANGELOG.md](CHANGELOG.md).
 | v0.6.0 | 4 | Multi-node placement and fabric awareness | released |
 | v0.7.0 | — | Conversations, agent memory, disk management | released |
 | v0.8.0 | — | PostgreSQL, leader election, agent failover | released |
+| v0.9.0 | — | Managing people and the tenancy tree | released |
 | v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
 
 As of v0.6.0 a cluster runs unattended: agents report their hardware, the
@@ -48,9 +49,38 @@ a header would let any client promote itself into the top priority band.
 
 One thing is worth saying plainly:
 multi-node replicas are proven on a simulated cluster and on real hardware
-without a fabric. The InfiniBand, NVML and NCCL paths are written against the
-formats and failure modes those tools actually have, but they have not yet run
-on a machine that has them.
+without a fabric. The InfiniBand, NVLink and NCCL paths — telling fabrics apart
+by their subnet managers, mapping them, finding NVLink domains, launching across
+nodes — are written against the formats and failure modes those tools actually
+have and tested against them, but they have not yet run on a machine that has
+them. On such a machine, run `llmgw doctor --json` first and keep the report;
+[docs/fabrics.md](docs/fabrics.md) says what to look at after that.
+
+## Documentation
+
+- [docs/quick-install.md](docs/quick-install.md) — installing on servers, in
+  six steps: start here
+- [docs/install.md](docs/install.md) — the same from nothing, with every option
+  explained: control plane, nodes, engines, weights, first request
+- [docs/quickstart.md](docs/quickstart.md) — one workstation, from source
+- [docs/fabrics.md](docs/fabrics.md) — InfiniBand fabrics and NVLink: what
+  llmgw needs of them, checking a node with `llmgw doctor`, and what to look at
+  when a replica spanning nodes will not run
+- [docs/choosing-models.md](docs/choosing-models.md) — which models to run,
+  how many of each, and where the weights should live
+- [docs/data.md](docs/data.md) — what llmgw stores, what it does not, and
+  what leaves the machine; written for a security review
+- [docs/reference.md](docs/reference.md) — every configuration key, generated
+  from the structs and checked by a test, so it cannot describe a setting that
+  does not exist or miss one that does
+- [deploy/mirror/](deploy/mirror/) — a model mirror, which is the cheapest
+  and most useful thing to build around llmgw
+- [deploy/proxy/](deploy/proxy/) — Caddy and nginx in front of the client
+  port, with streaming and the forwarded headers right
+- [deploy/ha/](deploy/ha/) — running two control planes, and the one health
+  check that has to be right
+- [docs/node-onboarding.md](docs/node-onboarding.md),
+  [docs/onboarding.md](docs/onboarding.md) — adding machines, and adding people
 
 ## Building
 
@@ -69,6 +99,7 @@ llmgw ctl replicas --model llama     # replicas and their shards
 llmgw ctl explain-placement llama    # why a model has not been placed
 llmgw ctl drain gpu-a-01             # move work off a node, one replica at a time
 llmgw ctl cordon gpu-a-01            # stop new placement, leave what runs
+sudo llmgw doctor                    # on a node: what its GPUs, NVLink and fabric need
 ```
 
 `ctl` talks to the admin port (`:9090` by default), not the client-facing one.

api/proto/llmgw/v1/agent.proto

@@ -23,6 +23,10 @@ message AgentMessage {
     Register register = 1;
     Heartbeat heartbeat = 2;
     ShardStatus shard = 3;
+    LogsReply logs = 4;
+    FabricScanReply fabric_scan = 5;
+    FabricTestReply fabric_test = 6;
+    UpgradeReply upgrade = 7;
   }
 }
 
@@ -34,6 +38,11 @@ message ControlMessage {
     DrainShard drain = 3;
     Ack ack = 4;
     RemoveWeights remove_weights = 5;
+    Provision provision = 6;
+    LogsRequest logs = 7;
+    FabricScanRequest fabric_scan = 8;
+    FabricTestRequest fabric_test = 9;
+    UpgradeAgent upgrade = 10;
   }
 }
 
@@ -101,6 +110,53 @@ message Register {
   // Empty means the node was not asked to look, exactly as for weights, and
   // constrains nothing.
   repeated Storage storage = 12;
+
+  // host is what the machine is, so the control plane can have the mirror
+  // fetch software built for it rather than assuming every node matches the
+  // first.
+  Host host = 13;
+
+  // engines_probed says engine_versions is an answer. Without it an empty map
+  // meant "an agent too old to say" and constrained nothing — so a freshly
+  // built node with no engine installed yet was a candidate for every model.
+  bool engines_probed = 14;
+  // cpu_count is the CPUs the agent may run on. Placement reserves cores
+  // for CPU replicas out of it, the way it reserves memory out of
+  // memory_total_mb; zero, from an older agent, reserves nothing.
+  int32 cpu_count = 18;
+  // imex is the service that lets a multi-node NVLink domain share memory.
+  IMEX imex = 15;
+
+  // installable_engines are the engines this agent knows how to install
+  // itself. The control plane asks it for no others. Empty is an agent older
+  // than the field, which could install llama.cpp and nothing else: asked for
+  // vLLM, it answered "unsupported", and every node of a cluster half way
+  // through an upgrade showed as failing.
+  repeated string installable_engines = 16;
+
+  // features are the requests this agent answers beyond the ones every agent
+  // does: "fabric-test". An agent that does not know a request ignores it, so
+  // without this the control plane would wait minutes for an answer that is
+  // never coming.
+  repeated string features = 17;
+}
+
+// Host describes the operating system and processor of a node.
+message Host {
+  // From /etc/os-release: "ubuntu", "24.04", "noble".
+  string os_id = 1;
+  string os_version = 2;
+  string os_codename = 3;
+  // Debian's spelling, which is what package indexes are keyed by: "amd64",
+  // "arm64".
+  string arch = 4;
+  // The vector extensions inference cares about, of those the CPU has. A VM
+  // given QEMU's default CPU model has none of them, and runs, and is five
+  // times slower, and nothing says why.
+  repeated string cpu_flags = 5;
+  // nvidia_driver is the driver version, empty when there is none. A card
+  // without a driver is not a GPU anything can use.
+  string nvidia_driver = 6;
 }
 
 // Storage is one filesystem a node keeps weights on.
@@ -145,6 +201,13 @@ message RunningShard {
 
   string leader_addr = 13;
   int32 leader_port = 14;
+
+  // profile_fp is the fingerprint of the profile this shard was started from,
+  // echoed back from StartShard. The control plane keeps no record of a running
+  // replica across its own restart, so without this a shard adopted after a
+  // restart could not be compared with the catalog — which is exactly when the
+  // catalog is most likely to have been edited.
+  string profile_fp = 15;
 }
 
 message GPU {
@@ -169,6 +232,17 @@ message GPU {
   // Zero when the driver is too old to report it, which is not the same as
   // idle: the metric is simply absent rather than wrong.
   int32 utilization_percent = 15;
+  // nic_paths is how far each network adapter is from this card, by HCA
+  // name: PIX, PXB, PHB, NODE or SYS, from nvidia-smi topo.
+  map<string, string> nic_paths = 16;
+  // The card's multi-node NVLink membership, from nvidia-smi -q: state is
+  // Completed once the fabric manager has joined it; cluster_uuid and
+  // clique_id name the domain it joined.
+  string fabric_state = 17;
+  string fabric_status = 18;
+  string cluster_uuid = 19;
+  string clique_id = 20;
+  string fabric_bandwidth = 21;
 }
 
 message FabricAttachment {
@@ -193,6 +267,121 @@ message LocalWeights {
   bool shared = 3;
 }
 
+// NodeLoad is a node's use of its own CPU and memory, and each engine's share.
+//
+// Carried on the heartbeat rather than scraped from the node: the control
+// plane already exports the fleet from one place, and a node exporter per
+// machine would be a scrape target, a port and a firewall rule each.
+message NodeLoad {
+  // cpu_percent is the whole machine, 0 to 100 however many cores it has.
+  double cpu_percent = 1;
+  int32 cpu_count = 2;
+  // The kernel's run-queue averages over 1, 5 and 15 minutes.
+  double load1 = 3;
+  double load5 = 4;
+  double load15 = 5;
+  // memory_used_bytes is total less available: what the machine could not
+  // hand to a new process without evicting something.
+  int64 memory_used_bytes = 6;
+  int64 memory_available_bytes = 7;
+  repeated ShardLoad shards = 8;
+  // What the machine's own sensors say; zero is not measured. cpu_temp_c is
+  // the hottest CPU sensor, cpu_power_w the CPU packages' draw from RAPL,
+  // inlet_temp_c the board's inlet or ambient sensor where it has one.
+  double cpu_temp_c = 9;
+  double cpu_power_w = 10;
+  double inlet_temp_c = 11;
+  repeated GPUSensor gpus = 12;
+}
+
+// GPUSensor is one card's heat, power, and what is holding it back.
+message GPUSensor {
+  int32 index = 1;
+  double temp_c = 2;
+  double memory_temp_c = 3;
+  double power_w = 4;
+  double power_limit_w = 5;
+  // throttle names what is holding the clocks down now: sw_power_cap,
+  // hw_slowdown, sw_thermal, hw_thermal, power_brake.
+  repeated string throttle = 6;
+}
+
+// ShardLoad is one engine process group's use.
+message ShardLoad {
+  string shard_id = 1;
+  // cpu_cores is in cores, the way top counts a process: 2.5 is two and a
+  // half cores busy, which a percentage of the machine would hide on a large
+  // one.
+  double cpu_cores = 2;
+  int64 memory_rss_bytes = 3;
+}
+
+// IBPort is one InfiniBand or RoCE port's state and counters.
+message IBPort {
+  string hca = 1;
+  int32 port = 2;
+  // state is ACTIVE, INIT, DOWN...; phys_state tells a cable unplugged
+  // (Polling, Disabled) from a link up with no subnet manager (LinkUp and INIT).
+  string state = 3;
+  string phys_state = 4;
+  int32 rate_gbps = 5;
+  // link_layer is InfiniBand or Ethernet (RoCE).
+  string link_layer = 6;
+  // counters are cumulative, by sysfs name; data counters in bytes.
+  map<string, uint64> counters = 7;
+
+  // Identity: the adapter's node GUID and the port's own, as the fabric
+  // knows them.
+  string node_guid = 8;
+  string port_guid = 9;
+  // The subnet manager this port sees as master, on InfiniBand. Every port on
+  // one physical fabric sees the same one, which is what tells separate
+  // fabrics apart when their names are only what somebody typed.
+  string sm_guid = 10;
+  int32 sm_lid = 11;
+  int32 sm_priority = 12;
+  string sm_state = 13;
+  // sm_error says why sm_guid is empty on an active InfiniBand port:
+  // infiniband-diags missing, or no subnet manager answering.
+  string sm_error = 14;
+  // network is, on RoCE, the IP network of the RoCE v2 address: RoCE has no
+  // subnet manager, and the network is what its fabric is.
+  string network = 15;
+  string gid_prefix = 16;
+  // subnet_managers is every subnet manager the fabric's subnet
+  // administrator lists — master and standbys — asked through one port per
+  // fabric, so it is on that port only. It is what keeps a fabric one fabric
+  // through a failover: the master changes, the list does not.
+  repeated SubnetManager subnet_managers = 17;
+  string subnet_managers_error = 18;
+}
+
+// SubnetManager is one subnet manager on a fabric.
+message SubnetManager {
+  string guid = 1;
+  int32 lid = 2;
+  int32 priority = 3;
+  // state is master, standby, discovering or not active.
+  string state = 4;
+}
+
+// IMEX is the nvidia-imex service and its channels.
+message IMEX {
+  // state is systemd's word for it, empty where it is not installed.
+  string state = 1;
+  int32 channels = 2;
+}
+
+// GPUFabricState is one GPU's NVLink domain membership.
+message GPUFabricState {
+  int32 index = 1;
+  string state = 2;
+  string status = 3;
+  string cluster_uuid = 4;
+  string clique_id = 5;
+  string bandwidth = 6;
+}
+
 message Heartbeat {
   string node_id = 1;
   // gpu_memory_used_mb, indexed by GPU, catches a foreign process appearing on
@@ -215,6 +404,78 @@ message Heartbeat {
   // report", and the first would quietly empty a node's inventory.
   bool weights_changed = 5;
   repeated LocalWeights weights = 6;
+
+  // memory_total_mb for the same reason again: RAM is added to a virtual
+  // machine while it runs, or across a reboot the agent does not notice, and a
+  // figure read once at registration kept a node that had grown to 32 GB
+  // looking like the 7 GB machine it used to be. 0 means not read.
+  int64 memory_total_mb = 7;
+
+  // engine_versions only when they changed, like weights. An engine installed
+  // after the agent started is otherwise invisible to placement until
+  // somebody restarts the agent — which is a step nobody knows to take.
+  bool engines_changed = 8;
+  map<string, string> engine_versions = 9;
+
+  // provision is where installing the node's software stands, sent when it
+  // changed. The flag makes an empty list mean "nothing to install" rather
+  // than "nothing to say", as for weights.
+  bool provision_changed = 10;
+  repeated ProvisionStatus provision = 11;
+
+  // load is how busy the machine is, sampled over the interval since the
+  // previous heartbeat. Absent on the first one, which has nothing to take a
+  // difference from, and from an agent on a system without /proc.
+  NodeLoad load = 12;
+
+  // ib_ports is every InfiniBand or RoCE port as it stands now, re-read on
+  // each heartbeat: a port that goes down after the agent started used to go
+  // unnoticed until the agent restarted.
+  repeated IBPort ib_ports = 13;
+  // fabrics is the fabrics this node can now carry traffic on, sent only when
+  // that changed since registration or the last heartbeat that said so — a
+  // port going down takes its fabric away, and one coming back returns it.
+  bool fabrics_changed = 14;
+  repeated FabricAttachment fabrics = 15;
+
+  // gpu_fabric is each GPU's NVLink domain membership as it stands now, sent
+  // once a minute: a fabric manager restarting drops every card out of its
+  // domain, and the node has to stop being offered work that spans it.
+  repeated GPUFabricState gpu_fabric = 16;
+  IMEX imex = 17;
+  // cpu_count as in Register, for a VM given more CPUs while it runs.
+  int32 cpu_count = 18;
+}
+
+// Provision is the software a node should have: which engines, at which
+// versions. Nothing more.
+//
+// Everything else is the node's to work out, because only the node knows it:
+// which build fits its hardware, where that build is published, what its
+// operating system is missing, and whether the result actually runs. The agent
+// converges on this — installs what is absent, checks what is there — and
+// reports each engine through ProvisionStatus. Nothing is taken on trust from
+// what happens to be on the machine: an engine is offered to placement only
+// once the agent has installed it and seen it run.
+message Provision {
+  repeated EngineWant engines = 1;
+}
+
+// EngineWant names one engine at one version.
+message EngineWant {
+  string engine = 1;  // "llamacpp"
+  string version = 2; // "b10991"
+}
+
+// ProvisionStatus is where one engine's installation stands on a node.
+message ProvisionStatus {
+  string engine = 1;
+  string version = 2;
+  string variant = 3;
+  // "installing", "ready", "failed", or "unsupported" when no build fits.
+  string state = 4;
+  // detail says what is happening or what went wrong, in words for a person.
+  string detail = 5;
 }
 
 // ShardStatus reports a shard's lifecycle transition.
@@ -229,6 +490,15 @@ message ShardStatus {
   // fails to load this is the only diagnosis available.
   repeated string last_output = 7;
 
+  // fetch is how far a download has got, when the shard is waiting on one.
+  //
+  // The agent logs this already, on the node, where nobody is looking. From
+  // the control plane a replica pulling a hundred gigabytes and one that is
+  // wedged are the same thing — a shard that has been starting for ten
+  // minutes — and telling them apart is the difference between waiting and
+  // going to look.
+  FetchProgress fetch = 8;
+
   enum State {
     STATE_UNSPECIFIED = 0;
     STATE_PENDING = 1;
@@ -241,6 +511,16 @@ message ShardStatus {
   }
 }
 
+// FetchProgress is a download in flight.
+message FetchProgress {
+  string source = 1;
+  int64 bytes_done = 2;
+  // bytes_total is 0 when the server did not say how large the file is, which
+  // is a real answer: a percentage would then be invented, and an invented
+  // percentage that stops moving is worse than a byte count that keeps rising.
+  int64 bytes_total = 3;
+}
+
 // StartShard launches one shard. It is idempotent by shard_id: the control
 // plane retries, and a repeat must not produce a second process.
 message StartShard {
@@ -267,6 +547,17 @@ message StartShard {
   // are sent so the agent can echo them back in RunningShard after a restart.
   string pool = 13;
   string fabric = 14;
+
+  // profile_fp identifies the version of the profile, so the agent can say
+  // which one a running shard was started from. The agent never interprets it.
+  string profile_fp = 15;
+  // api_key is what the engine demands of every request, derived by the
+  // control plane from the replica id. The router sends it; nothing else
+  // that can reach the node's ports has it.
+  string api_key = 16;
+  // cpu_threads is how many CPU threads a shard with no GPU computes on:
+  // the cores placement reserved for it. Zero leaves it to the agent.
+  int32 cpu_threads = 17;
 }
 
 message EngineProfile {
@@ -281,6 +572,9 @@ message EngineProfile {
   map<string, string> env = 9;
   string model_path = 10;
   string served_model_name = 11;
+  // adapters are LoRA adapters loaded beside the model, in order: the index
+  // is the id a request applies one by.
+  repeated LoraAdapter adapters = 12;
 }
 
 // RemoveWeights deletes cached model files a node no longer needs.
@@ -310,3 +604,113 @@ message Ack {
   bool ok = 2;
   string error = 3;
 }
+
+// LogsRequest asks a node for the recent lines of one of its logs.
+//
+// Over the stream the agent already holds, because that is the one way into a
+// node every deployment has: the admin page otherwise means ssh through a jump
+// host to a machine whose address nobody remembers, to read a journal only
+// root may open.
+message LogsRequest {
+  // request_id pairs the reply with its request; several may be in flight.
+  string request_id = 1;
+  // source is "agent" for the agent's own log, or a shard id for that
+  // engine's output.
+  string source = 2;
+  int32 lines = 3;
+}
+
+// LogsReply answers a LogsRequest.
+message LogsReply {
+  string request_id = 1;
+  repeated string lines = 2;
+  // sources are every log this node can show, so a page can offer them
+  // without knowing what runs there.
+  repeated string sources = 3;
+  // error says why the lines are absent: an unknown source, say.
+  string error = 4;
+}
+
+// FabricScanRequest asks a node to map the fabric one of its ports is on,
+// with ibnetdiscover: every switch, every adapter and every link between
+// them. One node per fabric is asked, every few minutes; a fabric looks the
+// same from all of its ports.
+message FabricScanRequest {
+  string request_id = 1;
+  string hca = 2;
+  int32 port = 3;
+}
+
+// FabricScanReply is ibnetdiscover's output as it printed it. Parsed by the
+// control plane, so a format the parser does not yet know is a control plane
+// upgrade rather than one on every node.
+message FabricScanReply {
+  string request_id = 1;
+  string output = 2;
+  string error = 3;
+  int64 took_ms = 4;
+}
+
+// FabricTestRequest asks a node for one side of an RDMA bandwidth test
+// between two nodes: ib_write_bw, from perftest, over one adapter. The
+// server side is asked first and answers when the test is over; the client
+// dials it. What it proves is what a replica spanning nodes needs and nothing
+// on one node can show: that these two adapters reach each other through the
+// fabric, and how fast.
+message FabricTestRequest {
+  string request_id = 1;
+  // role is "server" or "client".
+  string role = 2;
+  string hca = 3;
+  int32 port = 4;
+  // peer is the server's management address, for the client. perftest
+  // exchanges its connection details over TCP before any RDMA.
+  string peer = 5;
+  // tcp_port is that exchange's port, the same on both sides.
+  int32 tcp_port = 6;
+  int32 seconds = 7;
+}
+
+// FabricTestReply is what one side measured.
+message FabricTestReply {
+  string request_id = 1;
+  // gbps is the average bandwidth, in gigabits a second.
+  double gbps = 2;
+  // output is the end of the tool's output, for when the number is not
+  // enough.
+  string output = 3;
+  string error = 4;
+  int64 took_ms = 5;
+}
+
+// UpgradeAgent tells an agent to replace its own binary with a release and
+// restart.
+//
+// The agent fetches the release for its own platform and checks it against
+// the release's checksums before replacing anything, keeps the old binary
+// beside the new, then exits so systemd starts the new one. Engines keep
+// running throughout, as for any agent restart, and are adopted on the way
+// back. Over the stream the agent already holds, so upgrading a fleet needs
+// no ssh into it.
+message UpgradeAgent {
+  string request_id = 1;
+  // version must be a release: v0.10.4 or v0.10.4-rc.2.
+  string version = 2;
+  // releases is the base URL release files are downloaded from.
+  string releases = 3;
+}
+
+// UpgradeReply says whether the new binary is in place. Sent before the
+// agent exits; an empty error means it will be back on the new version.
+message UpgradeReply {
+  string request_id = 1;
+  string version = 2;
+  string error = 3;
+}
+
+// LoraAdapter is one adapter: the name clients ask for it by, and its source,
+// which the agent fetches as it does the model's weights.
+message LoraAdapter {
+  string name = 1;
+  string source = 2;
+}

deploy/bootstrap/add-node.sh

@@ -108,6 +108,5 @@ echo "== 2/2  onboarding $NAME"
 echo
 echo "Check it with: llmgw ctl nodes"
 echo
-echo "A node is not finished until the engines its profiles name are installed"
-echo "and the weights are on disk. Onboarding checks that a configured engine"
-echo "runs, but it installs neither — see docs/node-onboarding.md."
+echo "The agent installs its engines itself; the node takes no model until they"
+echo "are installed and seen to run. 'llmgw ctl nodes' shows where that stands."

deploy/bootstrap/onboard-node.sh

@@ -24,16 +24,20 @@ usage: $0 --node ID --addr HOST --server URL [options]
 
   --node ID         node id, hostname and certificate common name
   --addr HOST       how to reach the machine over ssh
-  --server URL      the control plane's agent endpoint, e.g. https://10.0.0.140:7443
+  --server URL      the control plane's agent endpoint, e.g. https://10.0.0.140:7443;
+                    several, comma separated, for a node that fails over
   --fabric NAME     InfiniBand fabric this node is attached to; repeatable as
                     --fabric NAME:hca1,hca2 . Omit for a node with no fabric,
                     which can host single-node replicas and will never be given
                     a multi-node one.
-  --mgmt-addr ADDR  address engines are reachable on (default: --addr)
-  --mgmt-if NAME    interface NCCL bootstraps over (default eth0)
-  --llamacpp-bin P  path to llama-server
-  --vllm-bin P      path to the vllm entry point
-  --sglang-python P python interpreter of the SGLang virtualenv
+  --mgmt-addr ADDR  address engines are reachable on (default: the node's
+                    address on the route to --server, found by the agent)
+  --mgmt-if NAME    interface NCCL bootstraps over (default: the one holding
+                    that address)
+  --llamacpp-bin P  use this llama-server instead of the one the agent installs
+  --vllm-bin P      use this vllm instead of the one the agent installs on a
+                    GPU node (also set engines.vllm: "" in server.yaml)
+  --sglang-python P python interpreter of an SGLang virtualenv you installed
   --nvidia-smi PATH where nvidia-smi lives, when it is not on PATH. Containerized
                     driver installs routinely put it somewhere else.
   --ib-root DIR     the sysfs InfiniBand directory (default /sys/class/infiniband)
@@ -42,16 +46,19 @@ usage: $0 --node ID --addr HOST --server URL [options]
                     Releases carry static linux/amd64 and linux/arm64 builds;
                     downloading one is better than copying whatever happens to
                     be on the machine you are running this from.
+  --releases URL    where release binaries are downloaded from (default the
+                    project's Gitea releases)
   --ssh-user NAME   (default ilker)
 USAGE
   exit 2
 }
 
-NODE= ADDR= SERVER= MGMT= MGMT_IF=eth0
+NODE= ADDR= SERVER= MGMT_OVERRIDE= MGMT_IF_OVERRIDE=
 NVIDIA_SMI= IB_ROOT=/sys/class/infiniband
 LLAMACPP_BIN= VLLM_BIN= SGLANG_PYTHON=
 SSH_WAIT=180
 PKI=/etc/llmgw/pki BINARY= SSH_USER=ilker
+RELEASES=https://gitea.mynodes.xyz/ilker/llmgw/releases/download
 FABRIC_NAME= FABRIC_HCAS="mlx5_0,mlx5_1"
 
 while [ $# -gt 0 ]; do
@@ -60,8 +67,8 @@ while [ $# -gt 0 ]; do
     --addr) ADDR="$2"; shift 2;;
     --server) SERVER="$2"; shift 2;;
     --fabric) FABRIC_NAME="${2%%:*}"; [ "${2#*:}" != "$2" ] && FABRIC_HCAS="${2#*:}"; shift 2;;
-    --mgmt-addr) MGMT="$2"; shift 2;;
-    --mgmt-if) MGMT_IF="$2"; shift 2;;
+    --mgmt-addr) MGMT_OVERRIDE="$2"; shift 2;;
+    --mgmt-if) MGMT_IF_OVERRIDE="$2"; shift 2;;
     --llamacpp-bin) LLAMACPP_BIN="$2"; shift 2;;
     --vllm-bin) VLLM_BIN="$2"; shift 2;;
     --sglang-python) SGLANG_PYTHON="$2"; shift 2;;
@@ -69,12 +76,12 @@ while [ $# -gt 0 ]; do
     --ib-root) IB_ROOT="$2"; shift 2;;
     --pki) PKI="$2"; shift 2;;
     --binary) BINARY="$2"; shift 2;;
+    --releases) RELEASES="$2"; shift 2;;
     --ssh-user) SSH_USER="$2"; shift 2;;
     *) usage;;
   esac
 done
 [ -n "$NODE" ] && [ -n "$ADDR" ] && [ -n "$SERVER" ] || usage
-MGMT="${MGMT:-$ADDR}"
 BINARY="${BINARY:-$(command -v llmgw)}"
 [ -x "$BINARY" ] || { echo "no llmgw binary; pass --binary" >&2; exit 1; }
 [ -f "$PKI/ca-key.pem" ] || { echo "no authority in $PKI; run 'llmgw ctl cert init-ca' first" >&2; exit 1; }
@@ -115,9 +122,10 @@ TMP="$(mktemp -d)"
 trap 'rm -rf "$TMP"' EXIT
 "$BINARY" ctl cert issue-client "$NODE" --dir "$PKI" --out "$TMP" >/dev/null
 
-# 2 and 3. Identity, and trusting the root so future tooling on that machine
-#    does not treat our certificates as strangers.
-say "installing the certificate and trusting the root"
+# 2 and 3. Identity. The root is trusted by the agent's configuration only,
+#    never system-wide: it signs whatever the control plane asks, and every
+#    TLS client on the machine would take it for any site.
+say "installing the certificate"
 scp -q $SSHOPTS "$PKI/ca.pem" "$TMP/$NODE.pem" "$TMP/$NODE-key.pem" "$SSH_USER@$ADDR:/tmp/"
 $SSH "set -e
   sudo install -d -m 0755 /etc/llmgw
@@ -125,73 +133,87 @@ $SSH "set -e
   sudo install -m 0644 /tmp/$NODE.pem /etc/llmgw/agent.pem
   sudo install -m 0600 /tmp/$NODE-key.pem /etc/llmgw/agent-key.pem
   sudo chown root:root /etc/llmgw/*
-  sudo install -m 0644 /tmp/ca.pem /usr/local/share/ca-certificates/llmgw-ca.crt
-  sudo update-ca-certificates >/dev/null 2>&1 || true
+  if [ -f /usr/local/share/ca-certificates/llmgw-ca.crt ]; then
+    sudo rm -f /usr/local/share/ca-certificates/llmgw-ca.crt
+    sudo update-ca-certificates --fresh >/dev/null 2>&1 || true
+  fi
   rm -f /tmp/ca.pem /tmp/$NODE.pem /tmp/$NODE-key.pem"
 
 # 4. The binary, the configuration and the unit.
-say "installing the binary"
-scp -q $SSHOPTS "$BINARY" "$SSH_USER@$ADDR:/tmp/llmgw"
-$SSH 'sudo install -m 0755 /tmp/llmgw /usr/local/bin/llmgw && rm -f /tmp/llmgw'
+# The node fetches the release CI built for this version, for its own
+# architecture, and checks it against the release's checksums — rather than
+# being handed whatever binary is on the machine running this script. A build
+# that is not a release (a developer's, with commits past the tag) has no such
+# file, and is copied as before.
+TAG="$("$BINARY" --version 2>/dev/null | awk '{print $NF}')"
+if printf '%s' "$TAG" | grep -Eq '^v[0-9]+\.[0-9]+\.[0-9]+(-rc\.[0-9]+)?$'; then
+  say "installing llmgw $TAG from the release"
+  $SSH "set -e
+    arch=\$(dpkg --print-architecture)
+    name=llmgw-$TAG-linux-\$arch
+    cd /tmp && rm -f \$name checksums.txt
+    curl -fsSL -o \$name '$RELEASES/$TAG/'\$name
+    curl -fsSL -o checksums.txt '$RELEASES/$TAG/checksums.txt'
+    grep \" \$name\\\$\" checksums.txt | sha256sum -c - >/dev/null
+    sudo install -m 0755 \$name /usr/local/bin/llmgw
+    rm -f \$name checksums.txt"
+else
+  say "installing the binary (${TAG:-unversioned}, not a release)"
+  scp -q $SSHOPTS "$BINARY" "$SSH_USER@$ADDR:/tmp/llmgw"
+  $SSH 'sudo install -m 0755 /tmp/llmgw /usr/local/bin/llmgw && rm -f /tmp/llmgw'
+fi
 
 say "writing the configuration"
-FABRICS="fabrics: []"
-if [ -n "$FABRIC_NAME" ]; then
-  FABRICS="fabrics:
-  - name: $FABRIC_NAME
-    hcas: [$(echo "$FABRIC_HCAS" | sed 's/,/, /g')]"
-fi
-cat > "$TMP/agent.yaml" <<YAML
-# $NODE
-#
-# Written by onboard-node.sh. Fabric membership is configuration and there is no
-# way around it: OpenSM's default subnet prefix is the same on every fabric, so
-# a matching prefix proves nothing and a wrong guess produces a replica that
-# starts, passes health checks and then hangs on its first all-reduce.
-node_id: $NODE
-server: "$SERVER"
-
-tls:
-  ca: /etc/llmgw/ca.pem
-  cert: /etc/llmgw/agent.pem
-  key: /etc/llmgw/agent-key.pem
-
-mgmt_addr: "$MGMT"
-mgmt_interface: $MGMT_IF
-
-heartbeat_interval: 5s
-reconnect_base: 1s
-reconnect_max: 30s
-
-$FABRICS
-
-discovery:
-  # Empty means look on PATH. Worth setting where the driver lives outside the
-  # default paths, which containerized installs routinely do.
-  nvidia_smi: "$NVIDIA_SMI"
-  ib_root: $IB_ROOT
-  prefer_nvml: false
-  skip_gpus: false
-
-# Where the engines live. Empty means "on PATH", which is rarely true: a node
-# that cannot find its engine registers healthy and then fails every shard
-# placed on it with "executable file not found".
-engines:
-  llamacpp:
-    bin: "$LLAMACPP_BIN"
-    parallel: 4
-  vllm:
-    bin: "$VLLM_BIN"
-    launch_mode: ray
-  sglang:
-    python: "$SGLANG_PYTHON"
-
-supervisor:
-  work_dir: /var/lib/llmgw-agent
-  log_lines: 512
-  stop_timeout: 60s
-  health_interval: 2s
-YAML
+NODE_IS_HOSTNAME=no
+[ "$($SSH hostname 2>/dev/null)" = "$NODE" ] && NODE_IS_HOSTNAME=yes
+# Only what the node cannot find out for itself. Its name is its hostname, its
+# address is the one on the route to the control plane, llama-server is looked
+# for on PATH and where release archives are unpacked, GPUs are there when
+# nvidia-smi is, and memory, disk and engines are re-read while it runs. Every
+# setting written here is one that has to be kept true by hand — so an option
+# below is written only when it was given, as an override.
+{
+  echo "# $NODE, written by onboard-node.sh."
+  # Several addresses, comma separated, are every control plane the node may
+  # fail over to; a standby refuses it and it moves on to the next.
+  if [ "${SERVER#*,}" != "$SERVER" ]; then
+    echo "servers:"
+    printf '%s\n' "$SERVER" | tr ',' '\n' | sed 's/^ *//; s/^/  - "/; s/$/"/'
+  else
+    echo "server: \"$SERVER\""
+  fi
+  echo
+  echo "tls:"
+  echo "  ca: /etc/llmgw/ca.pem"
+  echo "  cert: /etc/llmgw/agent.pem"
+  echo "  key: /etc/llmgw/agent-key.pem"
+  # The certificate's common name is --node, and the control plane refuses a
+  # stream whose node id differs from it. Said explicitly only when the
+  # hostname would say something else.
+  [ "$NODE_IS_HOSTNAME" = yes ] || echo "node_id: $NODE"
+  [ -n "$MGMT_OVERRIDE" ] && echo "mgmt_addr: \"$MGMT_OVERRIDE\""
+  [ -n "$MGMT_IF_OVERRIDE" ] && echo "mgmt_interface: $MGMT_IF_OVERRIDE"
+  if [ -n "$FABRIC_NAME" ]; then
+    # Fabric membership is the one thing that cannot be found: OpenSM's default
+    # subnet prefix is the same on every fabric, so a matching prefix proves
+    # nothing and a wrong guess hangs the first all-reduce.
+    echo "fabrics:"
+    echo "  - name: $FABRIC_NAME"
+    echo "    hcas: [$(echo "$FABRIC_HCAS" | sed 's/,/, /g')]"
+  fi
+  if [ -n "$NVIDIA_SMI" ] || [ "$IB_ROOT" != /sys/class/infiniband ]; then
+    echo "discovery:"
+    [ -n "$NVIDIA_SMI" ] && echo "  nvidia_smi: \"$NVIDIA_SMI\""
+    [ "$IB_ROOT" != /sys/class/infiniband ] && echo "  ib_root: $IB_ROOT"
+  fi
+  if [ -n "$LLAMACPP_BIN$VLLM_BIN$SGLANG_PYTHON" ]; then
+    echo "engines:"
+    [ -n "$LLAMACPP_BIN" ] && printf '  llamacpp:\n    bin: "%s"\n' "$LLAMACPP_BIN"
+    [ -n "$VLLM_BIN" ] && printf '  vllm:\n    bin: "%s"\n' "$VLLM_BIN"
+    [ -n "$SGLANG_PYTHON" ] && printf '  sglang:\n    python: "%s"\n' "$SGLANG_PYTHON"
+  fi
+  true
+} > "$TMP/agent.yaml"
 scp -q "$TMP/agent.yaml" "$SSH_USER@$ADDR:/tmp/agent.yaml"
 
 cat > "$TMP/llmgw-agent.service" <<'UNIT'
@@ -208,6 +230,10 @@ ExecStart=/usr/local/bin/llmgw agent --config /etc/llmgw/agent.yaml
 KillMode=process
 Restart=on-failure
 RestartSec=2s
+# RDMA pins the buffers it registers, and NCCL registers a great deal: with
+# systemd's default 8 MB lock limit, an engine that drops the capability that
+# lets root ignore it fails its first collective over InfiniBand.
+LimitMEMLOCK=infinity
 StateDirectory=llmgw-agent
 
 [Install]
@@ -255,8 +281,9 @@ $SSH 'set -e
 
 echo
 if [ -z "$LLAMACPP_BIN$VLLM_BIN$SGLANG_PYTHON" ]; then
-  say "no engine paths given; this node can only run the 'sim' engine until"
-  say "one is installed and the configuration points at it"
+  say "the agent now installs the engines the control plane asks for: the"
+  say "build for this machine, checked against its publisher's checksum and run"
+  say "before the node is offered any model. 'llmgw ctl nodes' shows progress."
 fi
 if [ -n "$ENGINE_PROBLEMS" ]; then
   echo "$NODE is onboarded but an engine will not run:$ENGINE_PROBLEMS"

deploy/examples/agent.yaml

@@ -4,12 +4,19 @@
 # keep serving if the agent restarts: the router talks to them directly, and
 # orphan adoption picks them up again on the next start.
 
-node_id: gpu-a-01          # defaults to the hostname
+# Everything a node can find out about itself, it does: its id is its hostname,
+# its address is the one on the route to the control plane, llama-server is
+# looked for on PATH and in /opt/llama.cpp, GPUs are there when nvidia-smi is,
+# weights are fetched into /var/lib/llmgw-weights and looked for there, and
+# memory, disk and installed engines are re-read while the agent runs. So a
+# plain CPU node needs only the two blocks below, and the same file on every
+# node. What follows them are overrides, for when the defaults are wrong.
+
 server: "http://llmgw-server.internal:7443"
 
 # The agent's half of mutual TLS. The certificate's common name must be this
-# node's id, because the control plane compares the two and refuses the stream
-# when they disagree:
+# node's id — its hostname, unless node_id says otherwise — because the control
+# plane compares the two and refuses the stream when they disagree:
 #
 #   llmgw ctl cert issue-client gpu-a-01 --dir /etc/llmgw/pki
 #
@@ -22,45 +29,46 @@ server: "http://llmgw-server.internal:7443"
 #   cert: /etc/llmgw/agent.pem
 #   key: /etc/llmgw/agent-key.pem
 
+# ---- overrides ---------------------------------------------------------------
+
+# node_id: gpu-a-01          # when it is not the hostname
+
 # The address engines are reachable on, and the address NCCL bootstraps over.
 # Never a fabric address: NCCL rendezvouses over TCP before moving to RDMA.
-mgmt_addr: "10.0.0.11"
-mgmt_interface: eth0
+# mgmt_addr: "10.0.0.11"
+# mgmt_interface: eth0
 
-labels:
-  rack: r12
+# Free-form, for a profile's node_selector. Nothing needs them; do not restate
+# what is measured (memory, GPUs, engines), because a label goes stale.
+# labels:
+#   rack: r12
 
 heartbeat_interval: 5s
 reconnect_base: 1s
 reconnect_max: 30s
 
-# Fabric labelling is explicit and there is no way around it. OpenSM's default
-# subnet prefix is fe80:: on every fabric, so a matching prefix proves nothing
-# and a wrong guess produces a replica that starts, passes health checks and
-# then hangs on its first all-reduce (plan §5.2).
+# Each fabric this node's adapters are on, by name. The name is checked against
+# what the adapters report — the subnet managers every port on one fabric sees —
+# so two independent fabrics given one name are placed as two and the cluster
+# page says so; give each its own. The rate is measured, not read from here.
 #
 # A node with no fabrics is its own single-node island: it can host single-node
-# replicas and will never be given a multi-node one.
+# replicas and will never be given a multi-node one. docs/fabrics.md has more.
 fabrics:
   - name: ib-a
     hcas: [mlx5_0, mlx5_1, mlx5_2, mlx5_3]
-    rate_gbps: 200
 
 discovery:
   # Worth setting where the driver lives outside the default paths, which
   # containerized driver installs often do.
   nvidia_smi: ""
   ib_root: /sys/class/infiniband
-  # NVML is the only source of NVLink topology, which is what keeps a
-  # tensor-parallel group off the PCIe path inside a node. It needs a binary
-  # built with `-tags nvml`; otherwise this silently falls back to nvidia-smi.
+  # NVLink topology comes from `nvidia-smi topo -m` on every build; a binary
+  # built with `-tags nvml` reads it from NVML instead, when this is set.
   prefer_nvml: true
-  skip_gpus: false
 
-# Where this node keeps model weights.
-#
-# Unset by default, and unset means the node is not asked to look — which is
-# not the same as having nothing, and constrains nothing. Storage differs at
+# Where this node keeps model weights besides /var/lib/llmgw-weights, which is
+# where fetched weights land and is always looked in. Storage differs at
 # every installation and llmgw deliberately does not own it: a Lustre mount, an
 # image exported over the fabric with SRP or NVMe-oF, local NVMe, or a copy
 # made by hand all look identical from here.
@@ -74,13 +82,23 @@ discovery:
 #   - /var/lib/llmgw-models
 #   - /mnt/lustre/models
 
+# The agent installs the engines the control plane names (see `engines:` in
+# server.yaml) under /opt/llmgw/engines: the build for this machine — CUDA when
+# an NVIDIA driver is loaded — checked against its publisher's checksum, with
+# the OS packages it needs, and run before the node takes any model. llama.cpp
+# is installed this way today. Naming a binary below opts that engine out and
+# uses yours instead; vLLM and SGLang still have to be named until the agent
+# installs them too.
+# engines_dir: /opt/llmgw/engines
+
 engines:
   vllm:
     bin: /opt/venvs/vllm/bin/vllm
-    # Which multi-node mechanism works depends on the vLLM version, and the
-    # flags have changed more than once. Check the version you deploy.
-    launch_mode: ray      # "ray" or "mp"
-    ray_bin: /opt/venvs/vllm/bin/ray
+    # How a replica spanning nodes is launched. mp, the default, is what vLLM
+    # documents today and needs nothing but vLLM; ray is for a vLLM too old
+    # for mp's multi-node flags, and needs Ray on every node.
+    launch_mode: mp       # "mp" or "ray"
+    # ray_bin: /opt/venvs/vllm/bin/ray
   sglang:
     # SGLang is launched as a module, which is its documented entry point.
     python: /opt/venvs/sglang/bin/python

deploy/examples/server.yaml

@@ -197,6 +197,13 @@ router:
     session_load_factor: 1.25
 
 # How aggressively the control plane reconciles.
+# The engines every node installs, and at which version. Unset keeps the
+# versions this release was checked against (llama.cpp b10991). Each agent
+# picks the build for its own hardware and installs it itself; changing a
+# version here reaches every node on its next connect.
+# engines:
+#   llamacpp: b10991
+
 control:
   interval: 5s
   base_backoff: 10s

deploy/ha/README.md

@@ -0,0 +1,87 @@
+# Two control planes
+
+llmgw handles the hard half of this itself: only one control plane places work,
+and agents find whichever one is leading. What it does not do is move client
+traffic, because that happens in front of both and llmgw is not in front of
+anything.
+
+This directory is the missing half. It is a pattern rather than a product:
+substitute your own load balancer if you have one.
+
+## What llmgw does on its own
+
+**One leader.** Leadership is a lease in the shared store. A second control
+plane starts, sees the lease, and stands by — it places nothing, evicts
+nothing and drains nothing. Start one by accident and your cluster is
+unharmed.
+
+**Agents follow.** A standby refuses agent streams, so agents move to the
+leader by themselves. Give each agent both addresses:
+
+```yaml
+servers:
+  - "https://ctl-1.internal:7443"
+  - "https://ctl-2.internal:7443"
+```
+
+Measured on a cluster: leadership moved three seconds after the leader was
+stopped, every agent had arrived six seconds later, and every replica was
+adopted rather than restarted — the engines never stopped serving.
+
+## What it does not do
+
+Client traffic. During that same failover the standby served inference
+correctly on its own port while the public address returned 502, because
+nothing in front of the two had moved.
+
+So something has to. The options, in the order most people should consider
+them:
+
+**A load balancer you already run.** Health-check `/ready` on each control
+plane and send traffic to whichever answers. This is the right answer if you
+have one, and it needs nothing else from this directory.
+
+**A floating address.** `keepalived.conf.example` here moves one IP between
+the two machines. Fewer moving parts than a load balancer and no extra hop,
+at the cost of one machine carrying all the traffic.
+
+**DNS.** Works, eventually. A record with a short TTL is a failover measured
+in whatever your resolvers actually honour rather than what you set, which on
+a bad day is minutes. Fine for a cluster people use interactively, wrong for
+one an agent is driving.
+
+## Check `/ready`, not `/health`
+
+This is the one thing to get right, and the obvious choice is the wrong one.
+
+`/health` says the process is up. It says nothing about whether it can serve,
+and a standby is exactly that case: alive, answering, and holding no endpoints
+at all — its reconciler placed nothing and it has no agent streams to adopt
+from. A load balancer checking `/health` would send it half the traffic, and
+half the traffic would be a 503.
+
+`/ready` says this process has somewhere to send a request. It is 503 on a
+standby, 503 on a leader whose replicas have all failed, and 200 when there is
+something to serve — which is the question a load balancer is actually asking.
+Both are open, because a load balancer has no key.
+
+During a failover neither passes for about the length of the lease handover:
+three seconds when the leader is stopped cleanly, up to the lease TTL when it
+is not. That gap is real and there is no way to have it both ways — traffic
+cannot go to a control plane that has nothing to route to.
+
+Not the admin port, for a different reason: it serves pprof, and pprof can
+dump the process heap, where API keys sit in plaintext.
+
+## Both control planes need the same store
+
+Leadership lives in it, and so do keys, budgets and conversations. Two control
+planes on two SQLite files are two clusters that happen to share a name.
+
+```yaml
+auth:
+  store: "@/etc/llmgw/secrets/store-dsn"
+```
+
+See [docs/install.md](../../docs/install.md) for moving an existing
+installation onto PostgreSQL without reissuing every key.

deploy/ha/keepalived.conf.example

@@ -0,0 +1,65 @@
+# A floating address for two control planes.
+#
+# One IP that lives on whichever machine is up. Clients and the reverse proxy
+# in front of llmgw point at the address, not at either host, so a failover is
+# invisible to them.
+#
+# This is not a load balancer: all traffic goes to the holder. That is usually
+# the right trade for a control plane, which serves far less than the nodes
+# it schedules.
+#
+# Copy to /etc/keepalived/keepalived.conf on both machines, changing state,
+# priority and interface.
+
+global_defs {
+    enable_script_security
+    script_user nobody
+}
+
+# The address moves only while llmgw can actually serve, not merely while the
+# machine is powered on.
+#
+# /ready, not /health. /health is 200 on a standby, which is up, answering and
+# holding no endpoints at all — so the address would sit on a host that refuses
+# every request. /ready is 200 only when there is somewhere to send one.
+#
+# Not the admin port either: it serves pprof, which can dump the heap that
+# holds every API key.
+vrrp_script check_llmgw {
+    script "/usr/bin/curl -fsS --max-time 2 -o /dev/null http://127.0.0.1:8080/ready"
+    interval 2
+    timeout 3
+    # Two failures before giving up the address, so one slow moment does not
+    # move it. Two successes before taking it back, so a flapping gateway does
+    # not drag the address with it.
+    fall 2
+    rise 2
+    weight -40
+}
+
+vrrp_instance llmgw {
+    # MASTER on the machine you would rather serve; BACKUP on the other.
+    state MASTER
+    interface eth0
+    # Any number both machines agree on and nothing else on the segment uses.
+    virtual_router_id 51
+    # The higher one wins while both are healthy. The weight above is enough
+    # to invert this when the preferred machine's gateway is not answering.
+    priority 100
+    advert_int 1
+
+    authentication {
+        auth_type PASS
+        # Change this. It stops another host on the segment claiming the
+        # address, which is otherwise a single unicast packet away.
+        auth_pass CHANGEME
+    }
+
+    virtual_ipaddress {
+        10.0.0.150/24
+    }
+
+    track_script {
+        check_llmgw
+    }
+}

deploy/mirror/README.md

@@ -0,0 +1,262 @@
+# A model mirror
+
+One machine that holds the model files, so nodes fetch over your own network
+instead of over the internet.
+
+This is the highest-value piece of infrastructure around llmgw and the cheapest.
+Eight nodes each pulling forty gigabytes from Hugging Face is a day's work, and
+on plenty of corporate networks it is not possible at all. From a mirror on the
+same switch it is minutes.
+
+llmgw needs nothing special for this: a model's `source` is a URL, and the URL
+can be yours.
+
+## What it is
+
+An HTTP server serving a directory. That is the whole design.
+
+```
+/srv/models/
+  qwen2.5-14b-instruct/
+    2026-03/
+      q4_k_m.gguf
+  bge-m3/
+    2026-01/
+      model.safetensors
+```
+
+The date in the path is the point of the layout, not decoration. A model
+repository that changes upstream changes what you are running, and the first
+sign is a prompt that worked last week answering differently this week with
+nobody able to say why. Pin it here and that cannot happen.
+
+## Running it
+
+`llmgw` is the mirror. Same binary as the control plane and the agents, one
+subcommand:
+
+```
+sudo mkdir -p /srv/models && sudo chown llmgw:llmgw /srv/models
+llmgw mirror serve --dir /srv/models --base http://models.internal:8081
+```
+
+There is a unit in `deploy/systemd/llmgw-mirror.service`. Set `--base` to the
+address nodes reach this machine on — not `localhost`, and not a name only this
+host knows itself by. It is the string that ends up in catalog entries, and a
+wrong one makes `mirror unused` report files as unreferenced when they are in
+use. The next step after that list is deletion.
+
+### Two listeners
+
+`--listen :8081` serves the weights. Read-only, unauthenticated, Range requests
+supported so an interrupted fetch on a node resumes rather than restarts. This
+is the path eight nodes pull forty gigabytes from.
+
+`--api-listen :8082` is the management API. Separate on purpose: putting a
+client certificate in front of the byte path would mean re-certifying every
+agent's fetcher in order to change how models are administered.
+
+The API is unauthenticated unless you give it the fleet's CA, and it says so at
+startup rather than leaving you to find out. Anyone who can reach that port can
+start downloads and delete models, so either keep it on a trusted network or
+hand it certificates:
+
+```
+llmgw mirror serve --dir /srv/models --base http://models.internal:8081 \
+  --tls-ca /etc/llmgw/ca.pem \
+  --tls-cert /etc/llmgw/mirror.pem \
+  --tls-key /etc/llmgw/mirror-key.pem
+```
+
+That is the same CA the agents already use, so the certificates come from the
+authority you have:
+
+```
+llmgw ctl cert issue-server mirror --host models.internal --host 10.0.0.146
+llmgw ctl cert issue-client control-plane
+```
+
+`--host` has to cover every address the API is called on, because that is what
+a client verifies against. Both sides are checked — the mirror will not accept a
+caller without a certificate from this CA, the same rule the agent stream
+follows.
+
+### nginx instead
+
+If you already run nginx, it serves a directory perfectly well and you do not
+need `mirror serve` for the bytes:
+
+```nginx
+server {
+    listen 80;
+    server_name models.internal;
+
+    root /srv/models;
+
+    # Read-only. This is deliberate: a mirror that can be written to over HTTP
+    # is a way to replace a model with another one, and the nodes would fetch
+    # it without complaint.
+    location / {
+        autoindex on;
+        # Large files, and the client is a machine with a resumable fetcher.
+        # Sendfile and no buffering keep memory flat regardless of size.
+        sendfile on;
+        tcp_nopush on;
+    }
+}
+```
+
+`autoindex` is there so a person can browse what is available; llmgw does not
+need it.
+
+## Filling it
+
+From the mirror itself:
+
+```
+llmgw mirror add \
+  --model qwen2.5-14b-instruct --version 2026-03 \
+  "https://huggingface.co/<repo>/resolve/<revision>/q4_k_m.gguf"
+```
+
+Use a revision rather than `main` in the source URL. `main` is what moves.
+
+It downloads to a temporary name and renames when complete, so an interrupted
+download never leaves something a catalog could be pointed at, and it prints the
+sha256 when it lands. Compare that against what upstream publishes — Hugging
+Face gives it as `lfs.oid` from
+`https://huggingface.co/api/models/<repo>/tree/<revision>`. A mirror is the one
+place in this system where a corrupt file is copied to every node instead of
+being caught once.
+
+It refuses to overwrite a version that is already there. Add a new one beside
+it: that refusal is what makes the old weights still available when the new ones
+turn out to be worse.
+
+## Driving it from elsewhere
+
+The same fetch over the API, so the control plane — or anything else — can ask
+for a model without anyone logging in:
+
+```
+curl -X POST http://models.internal:8082/v1/models -d '{
+  "source": "https://huggingface.co/<repo>/resolve/<revision>/q4_k_m.gguf",
+  "model": "qwen2.5-14b-instruct",
+  "version": "2026-03"
+}'
+```
+
+The mirror does the downloading. A control plane that pulled the bytes itself
+and pushed them here would carry every model twice over the network, through the
+one machine least able to spare the bandwidth.
+
+It answers `202` with the URL a catalog entry should use, and the bytes are not
+here yet. Poll for them:
+
+| | |
+|---|---|
+| `GET /v1/models` | everything held or arriving, and the base to build URLs from |
+| `POST /v1/models` | fetch this source into model/version |
+| `GET /v1/models/{model}/{version}` | `fetching` with a byte count, `present`, or `failed` with the reason |
+| `DELETE /v1/models/{model}/{version}` | remove a version; refuses while it is downloading |
+| `GET /v1/health` | up *and* able to read its own directory |
+
+A failed fetch stays `failed` with what went wrong, rather than disappearing:
+otherwise the thing waiting on that model waits forever with nothing to say why.
+Asking twice for the same download joins the one already running instead of
+starting a rival writer on the same file.
+
+`DELETE` takes the whole version, never one file of it. A model split across
+several files is useless with one missing, and half a version on disk looks
+present and will not load.
+
+## Retiring a model
+
+```
+llmgw mirror unused     # what no catalog entry refers to
+llmgw mirror remove --model <m> --version <v> --file <f>
+```
+
+`unused` compares against every `source` in the running catalog, so the answer
+is what can go rather than what looks old. It refuses to answer when it cannot
+reach the control plane: an empty catalog and an unreachable one produce the
+same list — everything — and the obvious next step after that list is deletion.
+
+Retire the catalog entry first, let the fleet settle, then remove. A node that
+has to refetch a file you deleted cannot, and the failure arrives later, on the
+node, looking like something else.
+
+## From the control plane's admin page
+
+Configure the mirror on the control plane and the same operations appear at
+`/admin/mirror`, behind the admin session the other admin pages use:
+
+```yaml
+mirror:
+  api: https://models.internal:8082
+  tls:
+    ca: /etc/llmgw/ca.crt
+    cert: /etc/llmgw/control-plane.crt
+    key: /etc/llmgw/control-plane.key
+```
+
+Leave the block out and the page is not mounted at all, which is the right page
+for a deployment with no mirror — not an empty one pointing at nothing.
+
+The page lists what is held, what is arriving with its byte count, and what
+failed with the reason; it starts a fetch, retries a failed one, and removes a
+version. It reloads itself only while something is downloading, because a reload
+at any other moment takes with it whatever is half-typed in the form.
+
+Two things it deliberately will not do. It cannot point the control plane at a
+different mirror — that is configuration, and changing it needs access to the
+host rather than a session. And it does not write the catalog for you: it shows
+the URL an entry should use, and putting it there is still a deliberate act,
+because that is the moment a model starts being served.
+
+The address is a mistake worth naming: `api` is the **management** port, not the
+one nodes fetch bytes from. They are two listeners so this one can be locked
+down without re-certifying every agent's fetcher.
+
+## Pointing llmgw at it
+
+```yaml
+- name: qwen2.5-14b-instruct
+  source: http://models.internal/qwen2.5-14b-instruct/2026-03/q4_k_m.gguf
+  scaling: { min: 2, max: 2 }
+  profiles:
+    - name: gpu
+      engine: vllm
+      gpu_class: h100-80g
+      tp: 1
+```
+
+Each node fetches once, in the background, before its engine starts — and
+`llmgw ctl replicas` shows how far along it is while it does. A killed download
+resumes rather than restarting, and a partial file is never visible under the
+model's name, so an interrupted fetch cannot be handed to an engine.
+
+## What this is not
+
+It is not a cache. Nothing populates it on its own: a model is there because
+somebody asked for it, by hand or over the API, and one nobody asked for is not
+there. That is on purpose. A mirror that fetched on demand would reach the
+internet at the least convenient moment, and in a network where that is blocked
+it would fail in a way that looks like llmgw being broken.
+
+It is also not a replacement for a shared filesystem. If you already run one
+that every node mounts, put the models on it and use a path as the `source` —
+nothing is fetched at all, and llmgw notices the filesystem is shared and will
+not count the same free space once per node.
+
+## When this stops being enough
+
+Somewhere above ten or twenty nodes, one mirror serving a forty gigabyte file
+to all of them at once becomes the bottleneck instead of the fix. At that point
+the options are a filer, a second mirror per rack, or distribution between the
+nodes themselves.
+
+llmgw knows which nodes already hold which weights — it is in `ctl status` as
+"N models on X/Y nodes" — so distributing between them is a feature it is
+positioned to add. It has not been built, because a mirror is enough for
+everyone who has needed it so far.

deploy/monitoring/README.md

@@ -0,0 +1,184 @@
+# Watching a cluster
+
+Prometheus, Alertmanager and Grafana, configured for what llmgw actually
+publishes. Every alert here is written against a series this system emits;
+nothing is aimed at a metric that might exist.
+
+## What it is
+
+One scrape target for the whole fleet.
+
+Agents publish no metrics of their own. The control plane gathers what they
+report — node state, memory, disk, engine queue depth, KV occupancy — and
+republishes all of it, so a single endpoint describes every machine. That is
+worth knowing before going looking for an exporter on each node.
+
+## The port it scrapes
+
+`metrics_listen`, not `admin_listen`.
+
+The admin listener is bound to loopback for one reason: it also serves pprof,
+and pprof can dump the process heap, where API keys sit in plaintext. Metrics
+are the one thing on that listener something else genuinely needs to reach, so
+they get a port of their own instead of becoming an argument for opening the
+other one.
+
+```yaml
+# /etc/llmgw/server.yaml
+metrics_listen: ":9091"
+```
+
+Confirm it carries nothing else:
+
+```
+curl -s -o /dev/null -w '%{http_code}\n' http://<control-plane>:9091/debug/pprof/   # 404
+curl -s -o /dev/null -w '%{http_code}\n' http://<control-plane>:9091/api/v1/nodes   # 404
+```
+
+## Installing
+
+On Ubuntu 24.04 the first two come from the archive; Grafana needs its own
+repository.
+
+```
+apt-get install prometheus prometheus-alertmanager
+
+install -m 0644 prometheus.yml            /etc/prometheus/prometheus.yml
+install -m 0644 llmgw-rules.yml           /etc/prometheus/rules/llmgw.yml
+install -m 0644 alertmanager.yml          /etc/prometheus/alertmanager.yml
+# Alertmanager answers on loopback only. Its API takes silences from anyone who
+# can reach it, with no login, and silencing BackupTooOld or ModelParked is how
+# something that should be noticed is not. Prometheus beside it is its only
+# client; one Alertmanager needs no cluster port.
+echo 'ARGS="--web.listen-address=127.0.0.1:9093 --cluster.listen-address="' \
+    > /etc/default/prometheus-alertmanager
+# Prometheus itself is read-only as installed: its admin and lifecycle APIs
+# stay off (no --web.enable-admin-api, no --web.enable-lifecycle). It still
+# answers any query from anyone who reaches :9090, which says which models and
+# nodes there are and how busy each is. Its readers are the control plane
+# (monitoring.prometheus) and Grafana beside it; to keep it to those, allow
+# the control plane and drop the rest, e.g. with nftables:
+#   nft add rule inet filter input tcp dport 9090 ip saddr != { 127.0.0.1, CONTROL_PLANE_IP } drop
+promtool check config /etc/prometheus/prometheus.yml
+systemctl restart prometheus prometheus-alertmanager
+
+install -m 0644 grafana/datasource.yaml   /etc/grafana/provisioning/datasources/llmgw.yaml
+install -m 0644 grafana/dashboards.yaml   /etc/grafana/provisioning/dashboards/llmgw.yaml
+install -m 0644 -o grafana -g grafana grafana/llmgw-overview.json \
+    /var/lib/grafana/dashboards/llmgw.json
+systemctl restart grafana-server
+```
+
+The dashboard is provisioned from a file rather than clicked together, so it
+survives a reinstall and can be reviewed in a diff.
+
+## Where to run it
+
+Not on the control plane. Monitoring that dies with the thing it monitors tells
+you nothing at the moment you need it, and the control plane is also the busiest
+machine in a small cluster.
+
+Anywhere else on the management network will do. The mirror is a reasonable
+host: it has the disk, and serving weights is bursty rather than constant.
+
+## Nothing is sent anywhere
+
+Alertmanager groups and holds alerts and has no receiver.
+
+Where an alert should go — an address, a Slack workspace, a webhook — is a
+decision with credentials attached, and a monitoring system that quietly starts
+mailing people is worse than one that is visibly not wired up. Alerts are
+visible in Prometheus, and at `:9093` on the monitoring host itself (`ssh -L
+9093:127.0.0.1:9093 <host>` to open its page); add a receiver to `alertmanager.yml` to
+make them travel. `critical` already routes separately so it can be pointed
+somewhere else without touching the rest.
+
+## Check that it fires
+
+An alerting system nobody has seen fire is an assumption.
+
+```
+llmgw ctl cordon <node>     # work already there keeps running
+# ~2 minutes later
+ssh <host> curl -s http://127.0.0.1:9093/api/v2/alerts | jq '.[].labels'
+llmgw ctl uncordon <node>   # the alert resolves on its own
+```
+
+Cordoning is the cheapest real test: it changes a metric the rules read, stops
+nothing, and reverses in one command.
+
+## What the alerts assume
+
+**A stopped model publishes no series.** `llmgw_model_replicas_ready` exists
+only for models with a replica placed, so `== 0` cannot fire for one that is
+deliberately off. That is what makes an on-demand catalogue quiet here.
+
+**Saturation of 1 is the healthy limit, not failure.** It is the worse of queue
+depth and KV occupancy, measured by `ctl bench` rather than guessed. The alert
+allows five minutes above it before saying anything.
+
+**Latencies are in seconds.** These thresholds are written for CPU inference. A
+p99 first token of thirty seconds is the alerting point here and would be
+absurd on a GPU; change it when the hardware changes.
+
+**Disk is only reported by nodes that hold weights.** A node with no model
+directory reports nothing rather than zero, and the rule divides free by total
+so a node that reports neither never fires.
+
+## Where the metrics live
+
+On the machine that scrapes them, in Prometheus's own store, and that is the
+right answer at this size. Measured on this cluster: 2,359 series at 41 samples
+a second, which compresses to roughly 7 MB a day — about 3 GB for a year.
+
+```
+# /etc/default/prometheus
+ARGS="--storage.tsdb.retention.time=365d --storage.tsdb.retention.size=20GB"
+```
+
+A year because the questions worth asking of this data are seasonal: what did
+capacity look like last quarter, was it always this slow, how much did the
+fleet grow. Fifteen days answers none of them.
+
+Both limits, not one. The time limit is the intent; the size cap is what stops
+Prometheus becoming the thing that fills the disk if the series count grows
+tenfold after somebody adds a fleet.
+
+No remote write, no object store, no second tier. At two thousand series that
+is infrastructure for a problem this cluster does not have, and every one of
+those has its own failure modes to learn.
+
+### The part that is not about capacity
+
+The disk is not the constraint — 900 GB free against 3 GB a year. What matters
+is what shares it.
+
+On this cluster the model mirror and the metrics database are on the same
+filesystem. A mirror that fills it stops Prometheus writing, so monitoring goes
+quiet at exactly the moment the disk is the problem, and the alert that would
+have said so is the one that died.
+
+Two ways out, in order of how much they cost:
+
+- Watch it, which is now done. `prometheus-node-exporter` arrives with the
+  Prometheus package and was running, unscraped, on this machine — so the disk
+  holding both was watched by nothing. It is a scrape target now, and
+  `MonitoringDiskFilling` fires at fifteen percent rather than five, because
+  the thing being protected is the ability to be told.
+- Separate them: a volume of its own for `/var/lib/prometheus`, or monitoring
+  on its own machine. Worth doing before the mirror holds a few hundred
+  gigabytes; not worth doing today at seven percent used.
+
+## What is not here
+
+No exporter on the llmgw nodes, so no host CPU, load average or network for
+them. The control plane publishes memory, disk and liveness for every node,
+which is what placement decides on; the rest is a `node_exporter` away if
+somebody wants it. The monitoring host itself is scraped, for the reason above.
+
+No probe of the mirror. Its `/v1/health` is JSON rather than Prometheus text,
+and checking it properly means `blackbox_exporter`.
+
+No recording rules. At this size every query in the dashboard runs fast enough
+raw, and a recording rule is a second definition of a number to keep in step
+with the first.

deploy/monitoring/alertmanager.yml

@@ -0,0 +1,41 @@
+# Alertmanager for the llmgw cluster.
+#
+# Nothing is sent anywhere yet, on purpose. Where an alert should go — an
+# address, a Slack workspace, a webhook — is a decision with credentials
+# attached, and a monitoring system that quietly starts mailing people is worse
+# than one that is visibly not wired up. Alerts group and fire here and are
+# visible at :9093 and in Prometheus; add a receiver below to make them travel.
+global:
+  resolve_timeout: 5m
+
+route:
+  # By alert and by what it is about, so five nodes going quiet in the same
+  # network blip arrive as one notification rather than five.
+  group_by: ['alertname', 'node', 'model']
+  group_wait: 30s
+  group_interval: 5m
+  # Long, because a repeat is a reminder and not news. A cluster with one thing
+  # wrong should not produce a message an hour.
+  repeat_interval: 12h
+  receiver: none
+
+  routes:
+    # Critical gets its own branch so it can be pointed somewhere else later
+    # without touching the rest: the difference between "a node is draining"
+    # and "nothing is serving" is who should be woken.
+    - matchers: [severity="critical"]
+      receiver: none
+      group_wait: 10s
+      repeat_interval: 4h
+
+receivers:
+  - name: none
+
+inhibit_rules:
+  # A node that is down explains every replica on it. Reporting both says the
+  # same thing twice and buries whichever is the actual cause.
+  - source_matchers: [alertname="NodeNotReady"]
+    target_matchers: [alertname="EndpointMetricsStale"]
+    equal: ['node']
+  - source_matchers: [alertname="ControlPlaneUnreachable"]
+    target_matchers: [severity="warning"]

deploy/monitoring/grafana/dashboards.yaml

@@ -0,0 +1,11 @@
+apiVersion: 1
+providers:
+  - name: llmgw
+    orgId: 1
+    folder: llmgw
+    type: file
+    disableDeletion: false
+    updateIntervalSeconds: 30
+    allowUiUpdates: true
+    options:
+      path: /var/lib/grafana/dashboards

deploy/monitoring/grafana/datasource.yaml

@@ -0,0 +1,11 @@
+# Provisioned rather than clicked in, so the dashboard below can reference it by
+# a fixed uid and the whole setup survives a reinstall of the machine.
+apiVersion: 1
+datasources:
+  - name: Prometheus
+    uid: llmgw-prom
+    type: prometheus
+    access: proxy
+    url: http://localhost:9090
+    isDefault: true
+    editable: false

deploy/monitoring/grafana/llmgw-overview.json

@@ -0,0 +1,492 @@
+{
+  "uid": "llmgw-overview",
+  "title": "llmgw \u2014 cluster",
+  "tags": [
+    "llmgw"
+  ],
+  "timezone": "browser",
+  "schemaVersion": 39,
+  "version": 1,
+  "refresh": "30s",
+  "time": {
+    "from": "now-6h",
+    "to": "now"
+  },
+  "panels": [
+    {
+      "id": 1,
+      "title": "Nodes ready",
+      "type": "stat",
+      "description": "Nodes taking work. A node that is known but draining or cordoned is not counted.",
+      "gridPos": {
+        "x": 0,
+        "y": 0,
+        "w": 4,
+        "h": 4
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "count(llmgw_node_up == 1)",
+          "legendFormat": "ready",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {},
+        "overrides": []
+      },
+      "options": {
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        },
+        "textMode": "auto",
+        "colorMode": "value"
+      }
+    },
+    {
+      "id": 2,
+      "title": "Replicas serving",
+      "type": "stat",
+      "description": "Across every model. A model deliberately stopped contributes nothing and is not a fault.",
+      "gridPos": {
+        "x": 4,
+        "y": 0,
+        "w": 4,
+        "h": 4
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "sum(llmgw_model_replicas_ready)",
+          "legendFormat": "ready",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {},
+        "overrides": []
+      },
+      "options": {
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        },
+        "textMode": "auto",
+        "colorMode": "value"
+      }
+    },
+    {
+      "id": 3,
+      "title": "Models with something serving",
+      "type": "stat",
+      "description": "",
+      "gridPos": {
+        "x": 8,
+        "y": 0,
+        "w": 4,
+        "h": 4
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "count(llmgw_model_replicas_ready > 0)",
+          "legendFormat": "models",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {},
+        "overrides": []
+      },
+      "options": {
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        },
+        "textMode": "auto",
+        "colorMode": "value"
+      }
+    },
+    {
+      "id": 4,
+      "title": "Requests per second",
+      "type": "timeseries",
+      "description": "Failing is counted separately rather than as a percentage, so a quiet cluster reads as quiet instead of as 0% of nothing.",
+      "gridPos": {
+        "x": 12,
+        "y": 0,
+        "w": 12,
+        "h": 4
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "sum(rate(llmgw_requests_total[5m]))",
+          "legendFormat": "all",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        },
+        {
+          "refId": "B",
+          "expr": "sum(rate(llmgw_requests_total{status=~\"5..\"}[5m]))",
+          "legendFormat": "failing",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "reqps"
+        },
+        "overrides": []
+      }
+    },
+    {
+      "id": 5,
+      "title": "Replicas ready, by model",
+      "type": "timeseries",
+      "description": "Only models with a replica placed appear. One that is stopped on purpose publishes no series.",
+      "gridPos": {
+        "x": 0,
+        "y": 4,
+        "w": 12,
+        "h": 7
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "llmgw_model_replicas_ready",
+          "legendFormat": "{{model}}",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {},
+        "overrides": []
+      }
+    },
+    {
+      "id": 6,
+      "title": "Pool saturation",
+      "type": "timeseries",
+      "description": "The worse of queue depth and KV occupancy. 1 is the limit measured as healthy, not the point of failure.",
+      "gridPos": {
+        "x": 12,
+        "y": 4,
+        "w": 12,
+        "h": 7
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "llmgw_pool_saturation",
+          "legendFormat": "{{model}}",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "max": 2,
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "red",
+                "value": 1
+              }
+            ]
+          }
+        },
+        "overrides": []
+      }
+    },
+    {
+      "id": 7,
+      "title": "Time to first token",
+      "type": "timeseries",
+      "description": "As the router sees it. On CPU these are seconds, not milliseconds; compare against what ctl bench measured.",
+      "gridPos": {
+        "x": 0,
+        "y": 11,
+        "w": 12,
+        "h": 7
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "histogram_quantile(0.5, sum by (model, le) (rate(llmgw_ttft_seconds_bucket[5m])))",
+          "legendFormat": "{{model}} p50",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        },
+        {
+          "refId": "B",
+          "expr": "histogram_quantile(0.99, sum by (model, le) (rate(llmgw_ttft_seconds_bucket[5m])))",
+          "legendFormat": "{{model}} p99",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "s"
+        },
+        "overrides": []
+      }
+    },
+    {
+      "id": 8,
+      "title": "Tokens per second",
+      "type": "timeseries",
+      "description": "",
+      "gridPos": {
+        "x": 12,
+        "y": 11,
+        "w": 12,
+        "h": 7
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "sum by (direction) (rate(llmgw_tokens_total[5m]))",
+          "legendFormat": "{{direction}}",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "short"
+        },
+        "overrides": []
+      }
+    },
+    {
+      "id": 9,
+      "title": "Engine queue depth",
+      "type": "timeseries",
+      "description": "Requests the engine has accepted and not started. Rising here is the first sign of saturation.",
+      "gridPos": {
+        "x": 0,
+        "y": 18,
+        "w": 8,
+        "h": 6
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "llmgw_endpoint_waiting",
+          "legendFormat": "{{replica}}",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {},
+        "overrides": []
+      }
+    },
+    {
+      "id": 10,
+      "title": "KV cache occupancy",
+      "type": "timeseries",
+      "description": "",
+      "gridPos": {
+        "x": 8,
+        "y": 18,
+        "w": 8,
+        "h": 6
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "llmgw_endpoint_kv_cache_usage",
+          "legendFormat": "{{replica}}",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "percentunit"
+        },
+        "overrides": []
+      }
+    },
+    {
+      "id": 11,
+      "title": "Model disk free",
+      "type": "timeseries",
+      "description": "The filesystems holding weights. A node that cannot hold the next model is skipped by placement rather than failing.",
+      "gridPos": {
+        "x": 16,
+        "y": 18,
+        "w": 8,
+        "h": 6
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "llmgw_node_disk_free_bytes",
+          "legendFormat": "{{node}}",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "bytes"
+        },
+        "overrides": []
+      }
+    },
+    {
+      "id": 12,
+      "title": "Seconds since each node last reported",
+      "type": "timeseries",
+      "description": "Heartbeats are every 5s. A line that climbs is an agent going quiet; the control plane stops placing there at 15s.",
+      "gridPos": {
+        "x": 0,
+        "y": 24,
+        "w": 12,
+        "h": 6
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "llmgw_node_last_seen_seconds",
+          "legendFormat": "{{node}}",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "s"
+        },
+        "overrides": []
+      }
+    },
+    {
+      "id": 13,
+      "title": "Replicas by state",
+      "type": "timeseries",
+      "description": "",
+      "gridPos": {
+        "x": 12,
+        "y": 24,
+        "w": 12,
+        "h": 6
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "sum by (state) (llmgw_replicas)",
+          "legendFormat": "{{state}}",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "llmgw-prom"
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {},
+        "overrides": []
+      }
+    }
+  ]
+}

deploy/monitoring/llmgw-rules.yml

@@ -0,0 +1,264 @@
+# Alerts for llmgw. Every expression here is against a series this cluster
+# actually publishes; nothing is written against a metric that might exist.
+groups:
+  - name: llmgw-fleet
+    rules:
+      - alert: ControlPlaneUnreachable
+        expr: up{job="llmgw"} == 0
+        for: 1m
+        labels: { severity: critical }
+        annotations:
+          summary: "The control plane is not answering"
+          description: >-
+            Nothing has scraped 10.0.0.140:9091 for a minute. Engines already
+            running keep serving — the router is in the same process, so if
+            this is down, so is the gateway.
+
+      - alert: NodeNotReady
+        expr: llmgw_node_up == 0
+        for: 2m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.node }} is {{ $labels.state }}"
+          description: >-
+            The node is known but not taking work. Two minutes, because a node
+            restarting its agent is briefly not ready and is not a problem.
+
+      - alert: AgentGoneQuiet
+        expr: llmgw_node_last_seen_seconds > 60
+        for: 1m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.node }} has not reported for {{ $value | humanizeDuration }}"
+          description: >-
+            Heartbeats are every 5s. A minute of silence is the network or the
+            agent, and the control plane stages expiry at 15s unreachable and
+            60s lost — so this fires around the point it stops placing there.
+
+      - alert: ModelHasNothingServing
+        expr: llmgw_model_replicas_ready == 0
+        for: 10m
+        labels: { severity: critical }
+        annotations:
+          summary: "{{ $labels.model }} has no replica ready"
+          description: >-
+            The model should be running and nothing is serving it. Ten minutes,
+            because a large model on CPU legitimately takes minutes to load. The
+            series is published at 0 for every model that should be running —
+            before v0.10.4 it was published only while a replica was ready, so
+            this could not fire at all. A model deliberately stopped, or an
+            on-demand one nobody is asking for, publishes nothing.
+
+      - alert: ModelParked
+        expr: llmgw_model_parked == 1
+        labels: { severity: critical }
+        annotations:
+          summary: "{{ $labels.model }} failed every time it started and is no longer retried"
+          description: >-
+            Five replicas in a row failed at start. Before this existed a model
+            whose weights its engine could not load was started 155 times over
+            eight hours. `llmgw ctl status` has the last error; once it is dealt
+            with, `llmgw ctl retry <model>`. Changing the model's definition
+            also retries it.
+
+      - alert: ModelFailingToStart
+        expr: llmgw_model_start_failures >= 2
+        for: 5m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.model }} has failed to start {{ $value }} times in a row"
+          description: >-
+            On its way to being parked. The engine's own words are in
+            `llmgw ctl logs <node> --source <shard>` for the next attempt, or in
+            the control plane's log for the last one.
+
+      - alert: ModelCannotBePlaced
+        expr: llmgw_model_unplaceable == 1
+        for: 10m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.model }} is wanted and fits on no node"
+          description: >-
+            Short of replicas, and the last attempt found nowhere with the
+            memory, cards or engine it needs. `llmgw ctl explain-placement
+            <model>` says what each node lacks. For an on-demand model, requests
+            for it are told this rather than asked to retry.
+
+      - alert: ModelPreempted
+        expr: increase(llmgw_model_preempted_total[15m]) > 0
+        labels: { severity: info }
+        annotations:
+          summary: "{{ $labels.model }} ({{ $labels.class }}) gave up replicas to a higher class"
+          description: >-
+            Capacity ran short and a model of a higher class took this one's
+            room. Expected now and then; if it keeps firing for the same model,
+            the cluster is too small for what is asked of it at these classes.
+            `llmgw ctl models` shows who it gave way to.
+
+      - alert: CapacityAtLimit
+        expr: llmgw_capacity_holds >= on(scope, resource) llmgw_capacity_limit
+        for: 15m
+        labels: { severity: info }
+        annotations:
+          summary: "{{ $labels.scope }} holds all the {{ $labels.resource }} it may"
+          description: >-
+            Its models get no further replicas, under load or not, until
+            something of theirs stops or the limit is raised: `llmgw ctl quota
+            list`, `llmgw ctl quota set`.
+
+      - alert: EngineInstallFailed
+        expr: llmgw_node_engine_install{state="failed"} == 1
+        for: 15m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.engine }} {{ $labels.version }} will not install on {{ $labels.node }}"
+          description: >-
+            The node takes no model needing it until it does. The agent retries
+            on its own; `llmgw ctl nodes` and the node's logs on the cluster
+            page say why it failed.
+
+      - alert: ControlPlaneLeadership
+        expr: sum(llmgw_control_plane_leader) != 1
+        for: 2m
+        labels: { severity: critical }
+        annotations:
+          summary: "{{ $value }} control planes believe they are placing work"
+          description: >-
+            None means nothing is placed or replaced; two means both are, each
+            into GPUs the other believes free. The lease is in the store: check
+            it is reachable from every control plane.
+
+      - alert: CertificateExpiresSoon
+        expr: (llmgw_certificate_expiry_timestamp_seconds - time()) < 30 * 86400
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.cert }} ({{ $labels.subject }}) expires in {{ $value | humanizeDuration }}"
+          description: >-
+            Certificates last a year and nothing renews them. When the agent
+            server's runs out every agent refuses to connect at once. Reissue
+            with `llmgw ctl cert` and restart the control plane.
+
+      - alert: CertificateExpiresVerySoon
+        expr: (llmgw_certificate_expiry_timestamp_seconds - time()) < 7 * 86400
+        labels: { severity: critical }
+        annotations:
+          summary: "{{ $labels.cert }} ({{ $labels.subject }}) expires in {{ $value | humanizeDuration }}"
+          description: See CertificateExpiresSoon; a week is not long.
+
+      - alert: BackupTooOld
+        expr: time() - llmgw_backup_last_success_timestamp_seconds > 26 * 3600
+        labels: { severity: warning }
+        annotations:
+          summary: "The newest llmgw backup is {{ $value | humanizeDuration }} old"
+          description: >-
+            llmgw-backup.timer runs nightly. Also fires when there has never
+            been a backup (the series is 0 then). `journalctl -u
+            llmgw-backup.service` on the control plane says why the last one
+            failed; docs/install.md, "Backing up", says how to set it up.
+
+      - alert: PoolSaturated
+        expr: llmgw_pool_saturation > 1
+        for: 5m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.model }} is over its healthy limit"
+          description: >-
+            Saturation is the worse of queue depth and KV occupancy; 1 is the
+            limit measured as healthy, not the point of failure. Sustained above
+            it means requests are queueing — add a replica or lower the load.
+
+      - alert: EndpointBreakerOpen
+        expr: llmgw_endpoint_breaker_open == 1
+        for: 2m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.replica }} has been taken out of rotation"
+          description: >-
+            The router stopped sending to this replica after repeated failures.
+            Traffic is going elsewhere; if it is the only replica, the model is
+            down.
+
+      - alert: EndpointMetricsStale
+        expr: llmgw_endpoint_metrics_stale == 1
+        for: 5m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.replica }} is being routed to blind"
+          description: >-
+            Its queue depth and KV usage are old, so the scheduler is choosing
+            with stale numbers. The replica may be serving perfectly well; the
+            decisions about it are the problem.
+
+      - alert: NodeDiskFilling
+        expr: llmgw_node_disk_free_bytes / llmgw_node_disk_total_bytes < 0.10
+        for: 15m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.node }} has {{ $value | humanizePercentage }} of its model disk free"
+          description: >-
+            Weights are tens of gigabytes. A node that cannot hold the next one
+            is skipped by placement rather than failing, so this is quiet until
+            somebody wonders why a model will not land there.
+
+  - name: monitoring-host
+    rules:
+      - alert: MonitoringDiskFilling
+        expr: >-
+          node_filesystem_avail_bytes{job="monitoring-host",fstype!~"tmpfs|overlay|squashfs"}
+          / node_filesystem_size_bytes{job="monitoring-host",fstype!~"tmpfs|overlay|squashfs"} < 0.15
+        for: 10m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.mountpoint }} on the monitoring host is {{ $value | humanizePercentage }} free"
+          description: >-
+            This filesystem holds the model mirror and the metrics database at
+            once. Filling it stops new weights arriving and stops Prometheus
+            writing — so monitoring goes quiet at the moment it is most needed,
+            which is why this fires at fifteen percent rather than five.
+
+      - alert: MonitoringDiskAlmostFull
+        expr: >-
+          node_filesystem_avail_bytes{job="monitoring-host",fstype!~"tmpfs|overlay|squashfs"}
+          / node_filesystem_size_bytes{job="monitoring-host",fstype!~"tmpfs|overlay|squashfs"} < 0.05
+        for: 5m
+        labels: { severity: critical }
+        annotations:
+          summary: "{{ $labels.mountpoint }} on the monitoring host has {{ $value | humanizePercentage }} left"
+
+  - name: llmgw-traffic
+    rules:
+      # These need traffic through the gateway to have any series at all. A
+      # counter with no observations publishes nothing, so a quiet cluster
+      # simply has no data here rather than a rule that is always false.
+      - alert: HighErrorRate
+        expr: >-
+          sum by (model) (rate(llmgw_requests_total{status=~"5.."}[5m]))
+          / sum by (model) (rate(llmgw_requests_total[5m])) > 0.05
+        for: 5m
+        labels: { severity: critical }
+        annotations:
+          summary: "{{ $labels.model }} is failing {{ $value | humanizePercentage }} of requests"
+
+      - alert: SlowFirstToken
+        expr: >-
+          histogram_quantile(0.99,
+            sum by (model, le) (rate(llmgw_ttft_seconds_bucket[10m]))) > 30
+        for: 10m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.model }} takes {{ $value | humanizeDuration }} to first token at p99"
+          description: >-
+            On CPU this is expected under load and is why the threshold is
+            thirty seconds rather than one. Compare with what ctl bench measured
+            before treating it as a fault.
+
+      - alert: RequestsBeingRejected
+        expr: sum by (band) (rate(llmgw_admission_rejected_total[5m])) > 0
+        for: 5m
+        labels: { severity: warning }
+        annotations:
+          summary: "Admission is turning away {{ $labels.band }} traffic"
+          description: >-
+            Working as designed when the cluster is full: the lowest band is
+            held first. Worth knowing about because the callers being refused
+            do not know why.

deploy/monitoring/prometheus.yml

@@ -1,45 +1,42 @@
-# Scraping llmgw.
+# Prometheus for the llmgw cluster.
 #
-# Point this at metrics_listen, not admin_listen. They are different ports for
-# a reason: the admin listener also carries pprof, which can dump the process
-# heap, and API keys are in that heap in plaintext.
-#
-# In the control plane's configuration:
-#
-#   admin_listen:   "127.0.0.1:9090"   # dashboard, management API, pprof
-#   metrics_listen: "0.0.0.0:9091"     # metrics only, safe to reach
-#
-# One target for the whole fleet. Node and GPU figures are exported by the
-# control plane, which already holds them — agents report hardware at
-# registration and state on every heartbeat — so there is nothing to scrape on
-# the nodes and no Prometheus configuration per machine.
+# One scrape target for the whole fleet. Agents publish nothing of their own —
+# the control plane gathers what they report and republishes it, so node health,
+# disk and endpoint state all arrive here from 10.0.0.140.
 global:
   scrape_interval: 15s
+  evaluation_interval: 15s
+  external_labels:
+    cluster: mynodes
+
+alerting:
+  alertmanagers:
+    - static_configs:
+        - targets: ['localhost:9093']
+
+rule_files:
+  - /etc/prometheus/rules/llmgw.yml
 
 scrape_configs:
+  # The control plane's dedicated metrics port. Not its admin listener: that one
+  # is loopback because it also serves pprof, and pprof can dump the heap where
+  # API keys sit in plaintext.
   - job_name: llmgw
     static_configs:
-      - targets: ["llmgw-ctl.internal:9091"]
+      - targets: ['10.0.0.140:9091']
         labels:
-          cluster: primary
+          instance: control-plane
 
-# Worth alerting on, in rough order of how often it matters.
-#
-#   A node that has stopped reporting:
-#     llmgw_node_last_seen_seconds > 60
-#
-#   A model that should be serving and is not. Covers a replica that will never
-#   start as well as one that has died, which look the same from a distance:
-#     llmgw_model_replicas_ready == 0
-#
-#   A card kept out of placement by a fault, which is otherwise invisible until
-#   a model will not fit:
-#     llmgw_gpu_healthy == 0
-#
-#   A fleet holding models nobody is asking for. Memory used says a model is
-#   loaded; utilisation says whether it is doing anything:
-#     avg_over_time(llmgw_gpu_utilization_percent[1h]) < 5
-#
-#   Requests queueing because the cluster is full, which is the pair of figures
-#   that needed two systems to correlate before these were on one endpoint:
-#     llmgw_admission_queue_depth > 0
+  - job_name: prometheus
+    static_configs:
+      - targets: ['localhost:9090']
+
+  # The machine this runs on. It came with the Prometheus package and was not
+  # being scraped, which left the disk holding both the model mirror and the
+  # metrics themselves watched by nothing — so a mirror that filled it would
+  # take monitoring down and the alert that would have warned did not exist.
+  - job_name: monitoring-host
+    static_configs:
+      - targets: ['localhost:9100']
+        labels:
+          instance: monitoring

deploy/proxy/Caddyfile

@@ -0,0 +1,41 @@
+# llmgw behind Caddy: the client-facing port (8080) on a public name, with a
+# certificate Caddy obtains and renews by itself.
+#
+# Replace llm.example.com, point its DNS at this machine, open 80 and 443, and
+# run: caddy run --config /etc/caddy/Caddyfile
+#
+# Only 8080 goes through here. 7443 (agents) is mutual TLS with the cluster's
+# own authority and must reach llmgw directly; 9090 and 9091 stay private.
+
+llm.example.com {
+	# The router reads at most 32 MiB of body (router.max_body_bytes). Refusing
+	# more here gives the client a 413 before the upload, not after it.
+	request_body {
+		max_size 32MB
+	}
+
+	reverse_proxy 127.0.0.1:8080 {
+		# Tokens are written the moment they are generated. Without this a
+		# streamed reply can arrive in bursts, and a client measuring time to
+		# first token measures the proxy instead.
+		flush_interval -1
+
+		# Caddy sets X-Forwarded-Proto and X-Forwarded-Host itself and, with
+		# no trusted_proxies, discards whatever the client sent. llmgw relies
+		# on that: the proto decides whether session cookies are Secure, and
+		# the host is the base URL the landing page tells people to use.
+		# Behind a load balancer of your own, name it in trusted_proxies.
+
+		transport http {
+			# A long generation can be silent for a while before the first
+			# token. llmgw has its own limits (timeouts.*); this must not be
+			# the shorter one.
+			response_header_timeout 15m
+		}
+	}
+
+	# No `encode` on purpose: compressing a stream buffers it.
+
+	# Access log to stderr, which under systemd is the journal.
+	log
+}

deploy/proxy/README.md

@@ -0,0 +1,39 @@
+# A reverse proxy in front of llmgw
+
+The client-facing port (8080) is plain HTTP. On anything but a trusted network,
+put it behind a proxy that holds a publicly trusted certificate.
+
+| File | For |
+| --- | --- |
+| [Caddyfile](Caddyfile) | Caddy — obtains and renews its own certificate; the least work |
+| [nginx.conf](nginx.conf) | nginx, with a certificate from certbot |
+
+Both are also attached to every release.
+
+The `/about` page shows both, from copies in `internal/web/proxy`; a test fails
+when those differ from these, so change them here and copy them over.
+
+Four things both files get right, and any other proxy has to as well:
+
+- **Streaming is not buffered.** Replies are server-sent events; a buffering
+  proxy delivers them in one burst at the end.
+- **The read timeout is longer than a model's silence.** nginx's default of 60
+  seconds turns a slow first token into a 504.
+- **`X-Forwarded-Proto` and `X-Forwarded-Host` are set by the proxy**, replacing
+  whatever the client sent. llmgw uses the first to mark session cookies
+  `Secure` and the second for the base URL on its landing page.
+- **The body limit matches llmgw's** (32 MiB, `router.max_body_bytes`), so an
+  oversized request is refused before it is uploaded.
+
+Only port 8080 is proxied. Agents reach 7443 directly — it is mutual TLS with
+the cluster's own authority, and a proxy terminating it would take away the
+node identity it carries. 9090 (admin, pprof) stays on loopback; 9091
+(metrics) goes to your monitoring host only.
+
+With a proxy on the same machine, bind llmgw to loopback so nothing reaches it
+around the proxy:
+
+```yaml
+# /etc/llmgw/server.yaml
+listen: "127.0.0.1:8080"
+```

deploy/proxy/nginx.conf

@@ -0,0 +1,63 @@
+# llmgw behind nginx: the client-facing port (8080) on a public name.
+#
+# Put this in /etc/nginx/sites-available/llmgw (or conf.d/llmgw.conf), replace
+# llm.example.com, and get a certificate — certbot --nginx -d llm.example.com
+# fills in the ssl_ lines below. Then: nginx -t && systemctl reload nginx
+#
+# Only 8080 goes through here. 7443 (agents) is mutual TLS with the cluster's
+# own authority and must reach llmgw directly; 9090 and 9091 stay private.
+
+upstream llmgw {
+    server 127.0.0.1:8080;
+    keepalive 32;
+}
+
+server {
+    listen 80;
+    listen [::]:80;
+    server_name llm.example.com;
+    return 301 https://$host$request_uri;
+}
+
+server {
+    # `listen ... http2` rather than `http2 on;`: the latter needs nginx
+    # 1.25.1, and Ubuntu 22.04 and Debian 12 ship older ones. Newer nginx
+    # accepts this form with a deprecation warning.
+    listen 443 ssl http2;
+    listen [::]:443 ssl http2;
+    server_name llm.example.com;
+
+    ssl_certificate     /etc/letsencrypt/live/llm.example.com/fullchain.pem;
+    ssl_certificate_key /etc/letsencrypt/live/llm.example.com/privkey.pem;
+
+    # The router reads at most 32 MiB of body (router.max_body_bytes).
+    client_max_body_size 32m;
+
+    location / {
+        proxy_pass http://llmgw;
+        proxy_http_version 1.1;
+        proxy_set_header Connection "";
+
+        # Set, never appended: llmgw decides from X-Forwarded-Proto whether
+        # session cookies are Secure, and builds the base URL the landing page
+        # shows from X-Forwarded-Host. A value passed through from the client
+        # would be the client's choice.
+        proxy_set_header Host              $host;
+        proxy_set_header X-Forwarded-Host  $host;
+        proxy_set_header X-Forwarded-Proto $scheme;
+        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
+
+        # Streamed replies: tokens go out as they are generated. Buffering
+        # turns a stream into one late burst, and gzip buffers too.
+        proxy_buffering off;
+        proxy_cache off;
+        gzip off;
+
+        # nginx's default is 60 seconds between two reads from upstream. A
+        # large model can think longer than that before its first token, and
+        # the client then sees a 504 for a request that was about to succeed.
+        # llmgw has its own limits (timeouts.*); this must not be the shorter.
+        proxy_read_timeout 15m;
+        proxy_send_timeout 15m;
+    }
+}

deploy/systemd/llmgw-agent.service

@@ -11,6 +11,10 @@ ExecStart=/usr/local/bin/llmgw agent --config /etc/llmgw/agent.yaml
 KillMode=process
 Restart=on-failure
 RestartSec=2s
+# RDMA pins the buffers it registers, and NCCL registers a great deal: with
+# systemd's default 8 MB lock limit, an engine that drops the capability that
+# lets root ignore it fails its first collective over InfiniBand.
+LimitMEMLOCK=infinity
 StateDirectory=llmgw-agent
 
 [Install]

deploy/systemd/llmgw-backup.service

@@ -0,0 +1,14 @@
+[Unit]
+Description=llmgw backup: the store and /etc/llmgw, to /var/backups/llmgw-server
+After=llmgw-server.service postgresql.service
+
+[Service]
+Type=oneshot
+# Nothing it writes is for anyone else: the archive holds the CA's key and a
+# copy of the store.
+UMask=0077
+# Root, because the archive includes /etc/llmgw/secrets and the certificate
+# authority's key, which the llmgw user deliberately cannot read all of.
+ExecStart=/usr/local/bin/llmgw server backup --config /etc/llmgw/server.yaml --keep 14
+Nice=10
+IOSchedulingClass=idle

deploy/systemd/llmgw-backup.timer

@@ -0,0 +1,12 @@
+[Unit]
+Description=Back up llmgw every night
+
+[Timer]
+OnCalendar=*-*-* 03:15:00
+# A machine that was off at 03:15 backs up when it next starts, rather than
+# waiting another day with a gap nobody notices until the alert.
+Persistent=true
+RandomizedDelaySec=10m
+
+[Install]
+WantedBy=timers.target

deploy/systemd/llmgw-mirror.service

@@ -0,0 +1,29 @@
+[Unit]
+Description=llmgw model mirror
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+# --base is what a catalog entry will say, so it has to be the address nodes
+# reach this machine on — not localhost, and not the name only this host knows
+# itself by. Getting it wrong makes `mirror unused` report files as unreferenced
+# when they are in use, and the next step after that list is deletion.
+ExecStart=/usr/local/bin/llmgw mirror serve \
+    --dir /srv/models \
+    --listen :8081 \
+    --api-listen :8082 \
+    --base http://models.internal:8081
+User=llmgw
+Group=llmgw
+Restart=on-failure
+RestartSec=2s
+NoNewPrivileges=true
+ProtectSystem=strict
+ProtectHome=true
+PrivateTmp=true
+# The one writable path. The mirror serves reads to the whole fleet and writes
+# only what its own API was asked to fetch, so nothing else needs to be.
+ReadWritePaths=/srv/models
+
+[Install]
+WantedBy=multi-user.target

docs/choosing-models.md

@@ -0,0 +1,149 @@
+# Choosing what to run
+
+llmgw serves whatever you give it and has no opinion about which models those
+are. This page has an opinion, because "we have GPUs, now what" is the question
+that stalls most of these projects — and the usual first answer, one large
+model, is the wrong shape.
+
+Model releases move faster than documentation. Treat the families named here as
+a starting point and check what is current before committing; the reasoning is
+what should outlive the names.
+
+## Three jobs, not one model
+
+The requests a company actually makes fall into three groups with different
+needs, and no single model serves all three well.
+
+**Interactive** — chat, autocomplete, short questions. Most of your traffic.
+Here latency beats quality: a person waits two seconds and does not wait
+twenty. A 7–14B class model, always resident.
+
+**Heavy** — long document analysis, code generation, anything that needs
+reasoning. Quality beats latency, and a queue is acceptable. A 30–70B class
+model, loaded on demand:
+
+```yaml
+scaling: { min: 0, max: 1, idle_timeout: 30m }
+```
+
+llmgw starts it when somebody asks and stops it when nobody has, so it costs
+nothing while it is not in use. The first request waits for the load, and the
+client is told that rather than refused.
+
+**Embeddings** — and this is the one that gets forgotten. Most "let us chat
+with our documents" projects are retrieval projects, and retrieval runs on an
+embedding model. It is small, it is cheap, and it carries the highest request
+volume of the three: every document chunk once, every query forever. llmgw
+routes `/v1/embeddings` like any other endpoint.
+
+A deployment with the first two and not the third can answer questions from
+what the model already knew. It cannot answer questions about your company.
+
+## Swedish and English
+
+The first thing to know is that this combination is harder than either alone.
+
+Open models are strongest in English by a wide margin. Swedish is served —
+better than most languages its size, because the Nordic languages appear in
+multilingual training sets — but the gap is real, and it widens as models get
+smaller. **A 7B model that feels fine in English can feel noticeably worse in
+Swedish**, which means the interactive tier may need to be larger than an
+English-only deployment would choose. Test before deciding; do not assume the
+English benchmark transfers.
+
+Two things matter more than the model's headline multilingual score:
+
+**Cross-lingual retrieval.** A Swedish company writes internal documents in
+Swedish and reads technical documentation in English. Somebody will ask a
+question in Swedish whose answer is in an English document. An embedding model
+that handles both languages separately may still fail to connect them, and it
+fails silently — the model gives a confident answer from the wrong source, and
+nobody notices because the answer reads well. Test this explicitly: ask in one
+language, check that documents in the other are retrieved.
+
+**Swedish compounds.** *Kundtjänstmedarbetare* is one token-hostile word where
+English would use three. This hurts keyword search more than dense retrieval,
+so if you run hybrid search, expect the lexical half to underperform on Swedish
+and weight accordingly.
+
+## How to actually decide
+
+Benchmarks will not settle this. Take twenty real documents and twenty real
+questions from the company — in both languages, including at least a few where
+the question and the answer are in different languages — and compare candidates
+on those. An afternoon of this is worth more than a week of reading leaderboards,
+and it is the only way to find the failure modes that matter to you.
+
+Do the same for the embedding model, separately. A good generation model on top
+of bad retrieval produces fluent, confident, wrong answers, which is the worst
+failure this stack has: it does not look like a failure.
+
+## How many replicas
+
+Not "how many employees". **Concurrency**, which is far lower than headcount
+because people spend most of their time reading rather than waiting on a model.
+Two hundred people generate perhaps five to fifteen concurrent requests at
+peak, for interactive use.
+
+vLLM batches continuously, so one replica on a capable card serves many
+concurrent requests. Which means the replica count is usually decided by
+**redundancy rather than throughput**:
+
+| Tier | Replicas | Why |
+|---|---|---|
+| Embeddings | 2 | Not for capacity. If retrieval stops, every document feature stops. |
+| Interactive | 2 | So a node can be rebooted during working hours. |
+| Heavy | 0–1, on demand | Loaded when asked for, stopped when idle. |
+
+Then measure rather than guess:
+
+```
+llmgw ctl bench <model>
+```
+
+It climbs concurrency until the SLO breaks and reports the last level that met
+it. Peak throughput arrives long after latency has become unusable, so
+throughput alone overstates what the hardware can serve.
+
+## Where the weights live
+
+The single highest-value piece of infrastructure here is **a local mirror**: one
+internal HTTP server holding the model files. [deploy/mirror/](../deploy/mirror/)
+has the nginx config and the directory layout. Hugging Face is slow, rate
+limited, and blocked outright on plenty of corporate networks. Eight nodes each
+pulling forty gigabytes from the internet is a day; from a mirror on the same
+switch it is minutes. Point a model's `source` at it and llmgw fetches from
+there.
+
+**Pin the version.** Put it in the path — `/models/qwen2.5-14b/2026-03/q4_k_m.gguf`
+— so a repository that changes upstream cannot change what you are running.
+Without this, a prompt that worked last week gives a different answer this week
+and nobody can explain why.
+
+**Quantisation.** For llama.cpp, `q4_k_m` is the practical default: the quality
+difference from `q8_0` is smaller than most people expect and the memory
+difference is a factor of two. For vLLM on modern cards, FP8 or AWQ. Run
+`llmgw ctl plan <model>` to see whether a given precision fits before
+downloading anything.
+
+**Disk.** Budget three to five times what you first estimate. Several
+quantisations, several versions, and partial downloads all accumulate.
+`llmgw ctl nodes` shows free space per node once `model_dirs` is configured,
+and `llmgw ctl weights` reclaims files llmgw downloaded and no model needs —
+never files you placed yourself.
+
+Under ten nodes, let each keep its own copy; it is simpler and the disk is
+cheap. Above that, consider a shared filesystem — llmgw notices one and will
+not count the same free terabyte once per node.
+
+## Before anyone starts using it
+
+Set budgets and limits on day one, not later:
+
+```
+llmgw keys limits --scope project --id <project> --rpm 60 --budget 50
+```
+
+The reason is not technical. A resource that appears free and unlimited becomes,
+three months later, a cost nobody can account for — and introducing limits at
+that point is a much harder conversation than having had them from the start.

docs/data.md

@@ -0,0 +1,133 @@
+# What llmgw stores, and what leaves the machine
+
+Written for the review that happens before a gateway is allowed near anyone's
+documents. It describes what the software does. Mapping that onto an
+obligation — GDPR, a sector rule, an internal policy — is work for whoever
+holds the obligation; this page exists so that work starts from facts rather
+than from a vendor's summary.
+
+Everything below is checkable in the source, and the places to look are named.
+
+## The short version
+
+By default llmgw stores **no prompt or completion text anywhere**. It stores
+who made a request, when, how many tokens, and what it cost.
+
+Two features change that, and both are off until somebody turns them on for a
+named project, and neither can be turned on without also saying for how long.
+
+Nothing on the serving path leaves the cluster.
+
+## What is stored, exactly
+
+**Usage records**, one per request: the key, the project, the model asked for,
+the model that answered, prompt and completion token counts, cost, HTTP status
+and latency. No message content. The row is in `internal/store/store.go` as
+`Usage` and there is no field for text in it.
+
+**API keys**, as SHA-256 hashes. The key itself is shown once, at creation, and
+never stored — so a leaked database hands over no working credential, and
+neither llmgw nor an administrator can recover a key somebody lost. A prefix is
+kept so a key can be identified in a listing without revealing it.
+
+**People and tenancy**: accounts, the organisation/team/project tree, and the
+limits attached to it. Passwords are hashed.
+
+**Sessions** for the browser pages, as tokens.
+
+**Conversations and agent notes** — only for a project that asked. See below.
+
+## What is not stored
+
+Prompts. Completions. Anything the model was shown or produced, unless the
+project turned on conversations or notes.
+
+The chat page in llmgw's own web interface keeps the conversation in a
+JavaScript variable in the browser. Refresh it and the conversation is gone,
+because it was never anywhere else.
+
+## Conversations and notes, when they are turned on
+
+Two features keep text, and both work the same way:
+
+- **Conversations** (`x-llmgw-thread`) let a client send only its new turn while
+  llmgw replays the history — including to a different model than the one the
+  conversation started on.
+- **Agent notes** (`/v1/memory`, and MCP tools) let an agent keep something for
+  itself across conversations.
+
+Both are **off for every project** until an operator runs `llmgw keys threads`
+or `llmgw keys memory` for that project by name. A request carrying a thread id
+against a project that has not is refused with an explanation, not quietly
+served without its history.
+
+Both **require a retention window**. There is no way to turn either on without
+saying how many hours, because a store with no expiry is a transcript of
+everything a team ever typed, kept forever because nobody chose a number. The
+window is measured from the last turn, so an active conversation does not
+expire underneath it, and a background sweep removes what has passed it.
+
+Both are **scoped to a project**, and the scope is part of every lookup rather
+than a check afterwards. Reading a conversation that belongs to another project
+returns the same answer as reading one that does not exist, so a leaked id
+cannot be used to discover what exists elsewhere.
+
+`DELETE /v1/threads/{id}` removes a conversation and its messages immediately.
+It is a client endpoint, not an administrative one: the person who said the
+thing can unsay it without asking somebody else to read it first.
+
+Turning retention off stops new storage, and existing conversations go at the
+next hourly sweep.
+
+## What leaves the machine
+
+**Nothing on the serving path.** A request arrives, is routed to an engine on
+your own hardware, and the answer comes back. The router talks to engines, the
+engines talk to nothing, the agents talk to the control plane. All of it is
+inside the cluster.
+
+Two things can reach outside, both started by an operator rather than by a
+request:
+
+**Fetching weights.** When a model's `source` is a URL, the agent downloads it.
+Point that at an internal mirror and nothing leaves at all — which is also the
+faster arrangement, because eight nodes pulling forty gigabytes from the
+internet is a day's work and from a local mirror it is minutes. A `source` that
+is a path on disk is never fetched.
+
+**`llmgw ctl plan`.** Reads a model's architecture from huggingface.co to work
+out whether it fits your hardware. An operator command run at planning time,
+not part of serving, and not run by anything automatically.
+
+There is no telemetry, no licence check, and no call home. The binary is
+static, built with `CGO_ENABLED=0`, and runs in a network with no route to the
+internet as long as the weights are already on disk.
+
+## What the software cannot do for you
+
+**The admin port serves pprof**, and pprof can dump the process heap — where
+API keys sit in plaintext while they are being checked. Bind it to loopback and
+reach it over SSH. The default does this; a deployment that changes it should
+know why.
+
+**The store holds everything.** With conversations turned on it holds text as
+well as accounting, so where it lives, who can read it and how it is backed up
+are decisions with the same weight as the ones above. The PostgreSQL DSN
+belongs in a file readable only by the user llmgw runs as, which is why the
+configuration takes `@/path/to/file` rather than the address itself.
+
+**Model weights are yours.** llmgw reclaims only files it downloaded, and it
+identifies those by a name it generated. Anything you put on a node it will not
+touch, whatever the catalog says.
+
+## Where to look
+
+| Claim | Where |
+|---|---|
+| Usage rows carry no text | `internal/store/store.go`, type `Usage` |
+| Keys are stored hashed | `internal/store/keys.go` |
+| Conversations are off by default | `internal/store/threads.go`, `ThreadRetention` |
+| Retention is required | `llmgw keys threads --help` |
+| Scope is part of the lookup | `internal/store/threads.go`, `Thread` |
+| Deletion is a client endpoint | `internal/router/threads.go`, `handleDeleteThread` |
+| Weight fetching is the only serving-path egress | `internal/agent/weights/fetch.go` |

docs/fabrics.md

@@ -0,0 +1,210 @@
+# InfiniBand fabrics and NVLink
+
+For a site whose GPU nodes are joined by InfiniBand or RoCE, by NVLink, or both
+— several independent fabrics, NVL72 racks, eight-GPU HGX boards. It says what
+llmgw does with them, what it needs from you before it can, how to check a node
+in one command, and where to look when a replica spanning nodes will not run.
+
+Read [install.md](install.md) first; this is the part of it that only matters
+once there is a fabric.
+
+> **Where this stands.** Every piece here was written against the real formats
+> of `sminfo`, `saquery`, `ibnetdiscover`, `nvidia-smi topo -m` and
+> `nvidia-smi -q`, and tested against them — but none of it has yet run on a
+> site with the hardware. Run `llmgw doctor --json` on two or three nodes first
+> and keep the reports: if a parser meets a format it does not know, that report
+> is what fixes it, and the fix is a control plane upgrade, not one per node.
+
+## What llmgw does with them
+
+**Two kinds of interconnect.** InfiniBand or RoCE joins nodes (scale-out).
+NVLink joins GPUs — inside one machine on an HGX board, or across the nodes of a
+rack on an NVL72 or an NVLink Switch system (scale-up). A replica spanning nodes
+is placed, in order of preference:
+
+1. inside one **multi-node NVLink domain**, if any has enough usable nodes:
+   tensor parallelism between its nodes then runs over NVLink;
+2. on one **physical InfiniBand or RoCE fabric**, preferring nodes under one
+   leaf switch and putting nodes with a degraded cable last.
+
+It is never split across two fabrics or two NVLink domains. Within a node, the
+GPUs a shard gets are the set with the most NVLink pairs, then the nearest to
+the fabric's adapters, then on one NUMA node.
+
+**A fabric has a name and an identity.** The name is what `agent.yaml` says; the
+identity is what the adapters report — on InfiniBand the subnet managers every
+port on one fabric sees, on RoCE the IP network of its RoCE v2 address. The name
+is how you address a fabric; the identity is how llmgw catches the name being
+wrong: two independent fabrics given one name are placed as two, and the
+cluster page says so.
+
+## Before a node joins
+
+On every GPU node:
+
+| What | Why | Check |
+|---|---|---|
+| NVIDIA driver | everything | `nvidia-smi` |
+| `nvidia-fabricmanager`, on an NVSwitch machine | without it CUDA does not start on those GPUs | `systemctl is-active nvidia-fabricmanager` |
+| `rdma-core`, `ib_uverbs` loaded | user-space RDMA at all | `ls /sys/class/infiniband`, `lsmod \| grep ib_uverbs` |
+| `nvidia_peermem` loaded | GPUDirect RDMA; without it NCCL copies every transfer through host memory | `lsmod \| grep nvidia_peermem` |
+| `infiniband-diags` | tells llmgw which fabric a port is on; the agent installs it if missing | `which sminfo` |
+
+On an NVL72 or NVLink Switch system, also:
+
+| What | Why | Check |
+|---|---|---|
+| `nvidia-imex` running, every node of the domain in `/etc/nvidia-imex/nodes_config.cfg` | GPUs in different nodes share memory through it | `systemctl is-active nvidia-imex` |
+| an IMEX channel (`options nvidia NVreg_CreateImexChannel0=1`) | an engine opens one | `ls /dev/nvidia-caps-imex-channels` |
+| every GPU joined to its domain | the fabric manager does this | `nvidia-smi -q \| grep -A6 Fabric` shows `State: Completed` |
+
+On each InfiniBand fabric:
+
+- **A subnet manager, and a standby.** Without one, ports come up to `INIT` and
+  stay there; without a standby, the fabric keeps its routes if the master
+  stops but cannot recover from any change after. llmgw watches this but does
+  not run one.
+- **Its own name** in `agent.yaml` on every node attached to it. Optionally, a
+  distinct `subnet_prefix` in each subnet manager's configuration, which makes
+  GIDs unique across fabrics; llmgw does not need it.
+
+## Declaring the fabrics
+
+In `/etc/llmgw/agent.yaml`, the adapters on each fabric, by name:
+
+```yaml
+fabrics:
+  - name: ib-compute
+    hcas: [mlx5_0, mlx5_1, mlx5_2, mlx5_3, mlx5_6, mlx5_7, mlx5_8, mlx5_9]
+  - name: ib-storage
+    hcas: [mlx5_4]
+```
+
+or `--fabric ib-compute:mlx5_0,mlx5_1,…` when onboarding ([node-onboarding.md](node-onboarding.md)).
+An adapter belongs to one fabric; a node may be on several. A replica on
+`ib-compute` is given only `ib-compute`'s adapters (`NCCL_IB_HCA`).
+
+The rate is measured, not taken from the file: a fabric's rate on a node is the
+slowest of its active adapters, because a collective runs at the speed of its
+slowest link.
+
+The agent checks the declaration against the adapters on every heartbeat. An
+adapter that is missing, or whose port is down, takes that fabric off the node
+within seconds — the node stops being offered multi-node work on it — and
+returns it when the port comes back.
+
+## Checking a node: `llmgw doctor`
+
+```
+sudo llmgw doctor                 # uses /etc/llmgw/agent.yaml when present
+sudo llmgw doctor --json > $(hostname)-doctor.json
+```
+
+It reads the node and says what is wrong and how to fix it, most serious first,
+and exits non-zero when anything fails:
+
+```
+FAIL  nvlink  this machine has NVSwitch but nvidia-fabricmanager is inactive: CUDA will not initialise on its GPUs
+              → systemctl enable --now nvidia-fabricmanager (its version must match the driver's)
+WARN  fabric  nvidia_peermem is not loaded: GPUDirect RDMA is off, and NCCL copies every transfer through host memory
+              → modprobe nvidia_peermem
+OK    gpu     8 GPUs: 8× NVIDIA H100 80GB HBM3
+OK    nvlink  NVLink groups: [0 1 2 3 4 5 6 7]
+OK    fabric  mlx5_0/1 ACTIVE at 400 Gb/s (InfiniBand), subnet manager 0x248a07030030f0c0 (LID 1, SMINFO_MASTER)
+OK    fabric  subnet managers of 0x248a07030030f0c0's fabric: 0x248a07030030f0c0 (master, priority 15), 0x248a07030030f1d4 (standby, priority 14)
+OK    agent   fabric ib-compute: mlx5_0,mlx5_1,… usable at 400 Gb/s
+OK    agent   NCCL bootstraps over ens3 (10.1.0.21)
+```
+
+followed by each GPU's NVLink peers and nearest adapter, each port's state,
+rate and errors, and the NCCL environment a shard on each fabric would be given.
+`--json` writes every fact it read as well as what it concluded — send that
+file when the report is not enough.
+
+## Checking two nodes: `llmgw ctl fabric-test`
+
+Every check on one node can pass while two nodes still cannot reach each other
+— a cable into the wrong switch, a partition key, a RoCE network with no route
+— and a replica spanning them then hangs on its first all-reduce. Before
+placing one, on the control plane:
+
+```
+llmgw ctl fabric-test gpu01 gpu02
+```
+
+```
+gpu01 → gpu02 on ib-compute, 41s
+
+RAIL              LINK      MEASURED    VERDICT  DETAIL
+mlx5_0 → mlx5_0   400 Gb/s  388.2 Gb/s  ok       97% of 400 Gb/s
+mlx5_1 → mlx5_1   400 Gb/s  241.0 Gb/s  slow     60% of 400 Gb/s: check the cable, and the card's PCIe width and NUMA node
+mlx5_2 → mlx5_2   400 Gb/s  0.0 Gb/s    fail     gpu01: ib_write_bw: Couldn't connect to 10.1.0.22:18517
+```
+
+It runs `ib_write_bw` (perftest, installed on a node the first time) from each
+of the first node's adapters on the fabric to the same adapter on the second —
+mlx5_0 to mlx5_0, the way NCCL pairs them — one rail at a time, through the
+agents, so no SSH between nodes is needed. Each rail is graded against its
+link's rate: **ok** at 80% or more, **slow** at 50%, **fail** below that or when
+nothing got through. `--fabric` picks one when the nodes share several,
+`--seconds` the length of each rail's run. The traffic is real: run it when the
+fabric is quiet.
+
+It measures RDMA between adapters, not GPUDirect or NCCL itself: a pass says
+the fabric carries traffic between these two nodes at its rate, which is what
+a first all-reduce needs and what nothing on one node can show.
+
+## Watching them
+
+- **Cluster page, Fabrics.** Each declared fabric and what it physically is;
+  a table of subnet managers per fabric — master, priority, standbys, the last
+  change of master — and every disagreement: one name on two fabrics, two names
+  on one, an adapter up but undeclared, ports up with no subnet manager, a
+  fabric with no standby, a recent failover.
+- **`/admin/fabrics`.** Each fabric's map, from `ibnetdiscover` run by one of
+  its nodes every five minutes: switches in their tiers, adapters under their
+  leaves, cables coloured by rate, degraded ones in red, rail alignment, and
+  the cluster's own adapters live with their traffic. Click anything for its
+  detail.
+- **`/admin/nvlink`.** Each multi-node NVLink domain as a rack of trays: which
+  GPUs are joined, which node's IMEX is down, which link runs degraded.
+- **A node's page.** Its ports with state, rate, errors and what they are
+  physically on; its GPU-by-GPU NVLink matrix; its load.
+- **Metrics**, for alerts:
+
+| Metric | Alert when |
+|---|---|
+| `llmgw_ib_port_active` | a declared port is 0 |
+| `llmgw_ib_port_counter_total{counter="symbol_error"}` and `link_downed` | rising |
+| `llmgw_ib_port_rate_gbps` | below its usual rate |
+| `llmgw_fabric_subnet_managers{state="standby"}` | absent or 0 |
+| `llmgw_fabric_master_sm` | its `sm_guid` changes (a failover) |
+| `llmgw_fabric_problems{level="fail"}` | above 0 |
+| `llmgw_gpu_nvlink_joined` | 0 |
+| `llmgw_imex_up` | 0 |
+
+## What an engine is given
+
+For a shard on an InfiniBand or RoCE fabric:
+
+| Variable | Value |
+|---|---|
+| `NCCL_IB_HCA` | that fabric's adapters on this node, and no others |
+| `NCCL_IB_DISABLE` | `0` (`1` for a single-node shard) |
+| `NCCL_SOCKET_IFNAME`, `GLOO_SOCKET_IFNAME` | the node's management interface: `mgmt_interface`, or the one holding its management address |
+| `NCCL_IB_GID_INDEX` | on RoCE only: the RoCE v2 entry with an IPv4 address |
+| `NCCL_MNNVL_ENABLE`, `NCCL_CUMEM_ENABLE` | `1`, for a replica inside a multi-node NVLink domain |
+
+A profile's `env` overrides any of them. vLLM launches a multi-node replica
+with its own multiprocessing (`engines.vllm.launch_mode: mp`, the default):
+every node runs `vllm serve` with its rank and the leader's address, the
+workers headless. `launch_mode: ray` is there for a vLLM too old for that.
+
+## When a replica spanning nodes will not run
+
+| It | Look at |
+|---|---|
+| is not placed | `llmgw ctl explain-placement <model>`: a fabric split by identity, a node whose IMEX is down, a node refused because its adapters are cabled to two fabrics |
+| starts and hangs on its first collective | `llmgw ctl fabric-test` between two of its nodes; the cluster page's fabric problems; `doctor` on each node — a port down, the wrong interface, RoCE without a v2 GID |
+| runs, slowly | `nvidia_peermem`; the map for a degraded cable; the node page's NVLink matrix for a group that crossed PCIe |
+| worked yesterday | the subnet manager table for a failover; `link_downed` on the node's ports |

docs/facility-aware-placement.md

@@ -0,0 +1,303 @@
+# Facility-aware placement: rack power, temperature and cooling
+
+**Status: step 0, reading the nodes' own sensors, is built; steps 1–5 are
+deferred** until the rack, PDU and power information of a real site is at
+hand — without it they would be designed against guesses.
+This is the design for step 5 of resource allocation, after reserved CPU cores
+and memory (1), capacity classes and preemption (2), tenant capacity quotas (3)
+and sizing by measured throughput (4), all of which shipped in v0.10.5. Of
+what follows, only the node sensors (rollout step 0) exist in the code: they
+are read, published and shown, and change no decision.
+
+## Why
+
+In a large installation the scarce thing is often not a GPU but the room it
+sits in. Three failures are invisible to placement today:
+
+- **A rack goes over its power budget.** Every node in it reports healthy;
+  placement sees free cards and fills them; the breaker, or the facility's
+  power cap, decides the rest. A rack with A and B feeds is worse: it runs
+  fine until one feed fails and the other cannot carry the load.
+- **Hot hardware is slow hardware.** GPUs throttle long before they fail. A
+  replica placed in a rack whose inlet air is too warm serves fewer tokens per
+  second than its measured capacity says, and nothing connects the two.
+- **Correlated failure.** Two replicas of a model on two nodes in the same rack
+  or on the same PDU are one failure away from zero. Placement spreads replicas
+  across nodes, not across racks.
+
+The goal is for placement to treat facility limits the way it already treats
+memory and cores: reserve against a budget, refuse what does not fit, say why,
+and never let missing data stop the cluster.
+
+## What it builds on
+
+| Already in place | Used for |
+|---|---|
+| Node labels, reported by agents, matched by `node_selector` | Where a node is: its rack, row, PDU |
+| `inventory.Allocation` carries GPUs, memory and cores per shard | Adding watts, summed per rack and PDU |
+| Placement rejections (`explain-placement`, headroom's "why not") | Naming the rack that is full or hot |
+| Headroom: the scheduler dry-run on a copy of the inventory | Room that respects power budgets, for free |
+| Preemption with classes, minimums and a dry run (step 2) | Shedding load from a hot rack in the right order |
+| Measured throughput per profile (step 4) | Tokens per watt as a placement preference |
+| Node sensors, read by every agent (step 0, built) | Heat and power with no facility integration: see below |
+| `internal/promq`, a Prometheus client | Reading facility metrics that DCIM exporters publish |
+
+## Concepts
+
+**Location.** Each node has a place in a tree: site, room, row, rack. Power has
+its own tree beside it: a rack draws from one or more PDUs, and each PDU from a
+feed. Both are read from node labels with well-known keys:
+
+```yaml
+# agent.yaml
+labels:
+  llmgw.io/site: sto1
+  llmgw.io/room: hall-b
+  llmgw.io/row: "12"
+  llmgw.io/rack: r12-07
+  llmgw.io/pdu: r12-07-a,r12-07-b   # both feeds, for a dual-fed rack
+```
+
+A node without these labels is its own rack of one: placement behaves as it
+does today.
+
+**Domain.** A rack, a row, a PDU, a room: anything with a budget or a
+condition. Domains are declared in a facility file (below) or discovered from
+node labels; a label naming a domain the file does not declare creates it with
+no budget and no condition.
+
+**Budget.** A domain may have a power cap in watts. Placement reserves each
+replica's estimated draw against every domain its node belongs to and refuses a
+placement that would take any of them past its cap less a safety margin. For a
+dual-fed rack the cap that matters is what one feed can carry alone, since that
+is what the rack must survive.
+
+**Condition.** A domain has a thermal and cooling condition: `ok`, `warm`, `hot`
+or `unknown`, derived from readings against thresholds with hysteresis.
+Conditions change what placement prefers and allows; they are not amounts to
+fill.
+
+**Reading.** A measured value with a time: inlet temperature, power draw, a
+cooling unit's alarm state. Readings older than their staleness limit are
+treated as absent.
+
+## Estimating a replica's draw
+
+What placement reserves has to be known before the replica runs, so it is an
+estimate first and a measurement later:
+
+1. **Declared.** A profile may say `power_watts`. The operator's figure wins.
+2. **Derived.** Otherwise GPU profiles take each card's power limit from NVML
+   (the agent already reads the cards; the power limit is one more field) plus
+   a per-node share for CPU, memory and fans; CPU profiles take their reserved
+   cores times a per-core figure for the node's CPU model.
+3. **Measured.** Once replicas run, the agent reports their cards' actual draw
+   (NVML) and, where the BMC allows, the node's (Redfish or IPMI DCMI). A
+   domain's headroom is computed from the larger of what is reserved and what
+   is measured, so an estimate that was too low is corrected by the room it
+   stops offering, not by an outage.
+
+## Decisions
+
+### Power
+
+- **Hard limit.** A placement that would take any domain past `cap × (1 −
+  margin)` is rejected, with the domain and the figures: *rack r12-07: 14.2 of
+  16.0 kW reserved, replica needs ~2.8, margin 10%*.
+- **Feeds.** For a domain with two feeds the cap is the smaller feed's rating
+  times its derating (typically 80%), not the sum.
+- **Already over.** A domain found over its cap, because the estimate was low
+  or the cap was lowered, gets no new placements, and an alert. It is not
+  emptied automatically; shedding power is the facility's decision, and
+  stopping inference to do it is ours only if asked (see shedding below).
+
+### Temperature and cooling
+
+Thresholds are per domain with defaults, and each condition needs to hold for a
+while before it changes, in both directions, so a single reading neither
+starts nor ends anything:
+
+| Condition | Enters when | Placement |
+|---|---|---|
+| `ok` | below the warm threshold for 5 min | normal |
+| `warm` | inlet above the warm threshold for 2 min, GPU temperatures near throttling, or any GPU thermally throttled | allowed, scored down: other domains first |
+| `hot` | inlet above the hot threshold for 2 min, a cooling unit in alarm, or a sustained share of the domain's GPUs thermally throttled | no new placements, as a cordon would, but automatic and reversible |
+| `unknown` | no fresh reading | as `ok`, and reported as unknown |
+
+**Shedding** (optional, off by default): a domain `hot` for longer than its
+shed delay has load moved off it using step 2's machinery. The lowest class goes
+first, and replicas above their model's minimum go before any at it. A critical
+minimum is never touched, and moves are rate limited to one replica per domain
+per interval. The replica is placed elsewhere first where there is room, so
+shedding moves work rather than dropping it. GPU power capping (`nvidia-smi
+-pl`) is a gentler alternative worth measuring against shedding: less
+throughput, no restart.
+
+### Spreading
+
+Replicas of one model are spread across racks, and across PDUs, the way they
+are spread across nodes today. This is a preference, dropped when it is the only
+thing standing in the way, as node spreading already is.
+
+### Scoring
+
+Placement today filters, then takes the first candidate in a fixed order
+(fewest free GPUs for GPU profiles, fewest replicas for CPU ones, nodes holding
+the weights first). Facility signals are better as a score than as more
+filters. Candidates that pass the hard limits are ordered by:
+
+1. condition: `ok` before `warm`;
+2. power headroom left in the rack after this replica, more first, so load
+   levels across racks;
+3. the existing order.
+
+Once measurements include power, step 4's profile preference can use tokens per
+watt instead of tokens per GPU where both are known.
+
+## Where the data comes from
+
+A `FacilitySource` interface, implemented several ways, chosen in
+`server.yaml`:
+
+```go
+type FacilitySource interface {
+    // Domains is what the source knows: budgets, thresholds, members.
+    Domains(ctx) ([]Domain, error)
+    // Readings is the latest value of each signal, per domain.
+    Readings(ctx) ([]Reading, error)
+}
+```
+
+- **Static file.** Domains, caps and thresholds, written by hand or generated
+  from a DCIM export. Enough for power budgets with no live data at all.
+- **Prometheus.** DCIM, PDU (SNMP exporter) and BMC (Redfish exporter)
+  readings usually end up in Prometheus. The source runs configured instant
+  queries and maps their labels onto domains:
+
+  ```yaml
+  facility:
+    source: prometheus
+    prometheus: http://monitoring:9090
+    signals:
+      inlet_temp_c:
+        query: max by (rack) (dcim_rack_inlet_temperature_celsius)
+        domain_label: rack
+        stale_after: 2m
+      power_watts:
+        query: sum by (pdu) (pdu_outlet_power_watts)
+        domain_label: pdu
+      cooling_alarm:
+        query: max by (row) (crac_alarm)
+        domain_label: row
+  ```
+- **Agents (built, step 0).** What each node reads about itself, every ten
+  seconds at most, sent with its heartbeat:
+  - **GPUs, from nvidia-smi:** temperature, memory (HBM) temperature where the
+    driver reports it, power draw, the enforced power limit, and what is
+    holding the clocks down now — `sw_power_cap`, `hw_slowdown`, `sw_thermal`,
+    `hw_thermal`, `power_brake`.
+  - **CPUs, from hwmon** (what lm_sensors reads, without it installed): the
+    package temperature, or the hottest core; and the board's inlet or ambient
+    sensor where it has one (`SYSTIN`, `Inlet`, `Ambient`).
+  - **CPU power, from RAPL:** the packages' energy counters over time,
+    subdomains not counted twice, counter wrap handled.
+
+  A machine without a sensor reports nothing for it, never a zero; a VM has
+  none of them. Published as `llmgw_gpu_temperature_celsius`,
+  `llmgw_gpu_memory_temperature_celsius`, `llmgw_gpu_power_watts`,
+  `llmgw_gpu_power_limit_watts`, `llmgw_gpu_throttled{reason}`,
+  `llmgw_node_cpu_temperature_celsius`, `llmgw_node_cpu_power_watts` and
+  `llmgw_node_inlet_temperature_celsius`, and on each node's page.
+
+  What they can and cannot say:
+  - **Throttling is the strongest signal there is.** A temperature is a proxy;
+    a card holding its clocks down is the fact that invalidates its measured
+    capacity, and it needs no threshold to be chosen.
+  - **Measured power** is GPU draw plus CPU packages; memory, disks, fans and
+    power-supply losses are not in it, so a node's draw is that plus an
+    estimated remainder. A rack's is its llmgw nodes' plus a declared figure
+    for everything else in it: switches and storage report nothing here.
+  - **CPU and GPU temperatures follow the load, not the room.** A hot rack is
+    inferred from many nodes running hot at comparable load, or from
+    throttling, not from one node's CPU. Only a board inlet sensor reads the
+    air, and most boards have none.
+  - **No budget comes from a sensor.** A reading is a draw; the cap it is
+    measured against still comes from the facility.
+- **Simulated.** Readings from a script, for tests and for rehearsing a
+  thermal event on a cluster without facility data.
+
+## Safety
+
+- **Missing data never blocks.** A source that is down, a reading that is stale,
+  a node without location labels: placement behaves as it does today, and says
+  the data is missing (a metric, a line on the cluster page). Facility
+  awareness can make placement more careful, never impossible.
+- **Hysteresis and hold times** on every condition, so a noisy sensor does not
+  flap a rack between allowed and forbidden.
+- **Observe-only first.** A mode where every decision is computed, logged and
+  shown ("would have refused: rack r12-07 hot") but nothing is enforced, run
+  for long enough to trust the thresholds before turning enforcement on.
+- **Manual override.** `llmgw ctl facility hold RACK` and `release RACK`, for the
+  facility team to mark a domain unavailable, or to clear a condition they know
+  to be a sensor fault.
+- **Shedding is opt-in**, per domain or cluster-wide, and rate limited.
+
+## What people see
+
+- `explain-placement` and headroom name the domain: *rack r12-07 is hot (inlet
+  31°C for 6 min)*, *pdu r12-07-a: 7.1 of 7.4 kW*.
+- `llmgw ctl facility` lists domains with budget, reserved, measured and
+  condition. The cluster page gets a rack summary beside the nodes table.
+- Metrics: `llmgw_domain_power_budget_watts`, `_reserved_watts`,
+  `_measured_watts`, `llmgw_domain_condition{condition}`,
+  `llmgw_domain_reading_age_seconds`; alerts for a domain near its cap, a
+  domain hot, a domain over its cap, and facility data gone stale.
+
+## Rollout
+
+Each step ships on its own and is useful without the next:
+
+0. **Node sensors, observe-only (built).** GPU heat, power and throttling,
+   CPU heat and power, inlet air where the board has a sensor; published and
+   shown, no decision changed.
+1. **Location and spreading.** Well-known labels, domains from labels, replicas
+   spread across racks and PDUs, locations on the node views. No facility data
+   needed.
+2. **Power budgets from a static file.** Caps and margins, estimated draw per
+   replica (declared or derived), hard limit in placement and headroom.
+3. **Readings, observe-only.** The Prometheus source beside the agents',
+   conditions with hysteresis (throttling first among them), measured power
+   folded into headroom; decisions shown, not enforced.
+4. **Thermal enforcement.** `warm` scored down, `hot` refused, manual hold and
+   release.
+5. **Shedding and tokens per watt.** Opt-in shedding through step 2's
+   preemption; measured power in profile preference.
+
+## Testing
+
+- The simulated source drives unit tests of every condition transition, the
+  hysteresis, staleness, and the placement outcomes, with mutation checks as
+  for steps 1–4.
+- Property tests: whatever the readings, placement never exceeds a declared
+  cap with fresh data, and never refuses everything because data is missing.
+- On the PoC cluster, with no facility data: step 1 with labelled VMs, step 2
+  with a static file of made-up caps, steps 3–4 with the simulated source
+  replaying a thermal event.
+- Before enforcement anywhere real: a period in observe-only mode against the
+  site's actual data.
+
+## Open questions
+
+- **What the sites have.** Which DCIM, which PDUs, which BMCs, and whether any
+  of it already reaches a Prometheus. The source design assumes Prometheus is
+  the common ground; a site without it needs another source.
+- **Who owns the numbers.** Caps, deratings and thresholds belong to the
+  facility team. Whether they edit llmgw's file, or llmgw reads theirs.
+- **Phase balance.** Three-phase PDUs have per-phase limits; whether placement
+  should see phases or only the PDU total.
+- **Cooling granularity.** Cooling is often per row or per aisle, not per rack;
+  the domain tree allows either, but the defaults should match what sites
+  actually measure.
+- **Power capping instead of shedding.** Whether to lower GPU power limits in a
+  warm rack rather than move replicas, and how that interacts with measured
+  capacity from step 4, which would no longer hold at the lower limit.

docs/node-onboarding.md

@@ -4,24 +4,16 @@ Written for whoever runs the cluster.
 
 ## The short version
 
-Where one person holds both the hypervisor and the certificate authority —
-which is the common case — one command does the whole thing in about
-45 seconds:
+On the control plane:
 
 ```
-llmgw-add-node \
-  --name llmgw-node05 --ip 10.0.0.145/24 \
-  --server https://10.0.0.140:7443 \
-  --cores 8 --memory 32768 --disk 60 \
-  --llamacpp-bin /opt/llama.cpp/llama-server
+sudo llmgw node add ubuntu@10.0.0.145
 ```
 
-The VMID defaults to the last octet of the address, so `qm list` and
-`llmgw ctl nodes` stay in the same order. `--provision-only` and
-`--onboard-only` split it back into the two halves below when the two kinds
-of access belong to different people.
-
-The rest of this page is what that command does, and what it still does not.
+That is all a machine that already exists needs — bare metal, or a VM you
+built. [install.md](install.md#5-adding-the-node) says what it does. On
+Proxmox, the machine itself can be built first by the script below, on the
+hypervisor.
 
 ## The two halves
 
@@ -54,41 +46,31 @@ their working tree.
 
 ## 2. The identity
 
-On the host that holds the certificate authority — this is the part that cannot
-be delegated, because it issues the credential by which the control plane
-believes the node is who it says it is:
+On the control plane, which holds the certificate authority — this is the
+part that cannot be delegated, because it issues the credential by which the
+control plane believes the node is who it says it is:
 
 ```
-llmgw-onboard-node \
-  --node llmgw-node04 \
-  --addr 10.0.0.144 \
-  --server https://10.0.0.140:7443 \
-  --fabric ib-b
+sudo llmgw node add ilker@10.0.0.144 --fabric ib-b:mlx5_0,mlx5_1
 ```
 
-It issues a client certificate whose common name is the node id, installs it
-along with the CA root, trusts that root system-wide so other tooling on the
-machine stops treating llmgw's certificates as strangers, installs the binary
-and the unit, and starts the agent. The node appears in `llmgw ctl nodes` within
-a few seconds.
+It checks the machine with `llmgw doctor`, issues a client certificate whose
+common name is the node id (the hostname), installs it along with the CA root,
+trusts that root system-wide, installs the binary and the unit, starts the
+agent, and waits for the node to join.
 
-If it does not, the control plane's log says why. A certificate whose common
-name does not match the claimed node id is refused by design — that check is
-what makes a node's identity something it proves rather than something it picks.
+If it does not join, the agent's log on the node says why. A certificate whose
+common name does not match the claimed node id is refused by design — that
+check is what makes a node's identity something it proves rather than
+something it picks.
 
-**It needs ssh from the CA host to the node.** Provisioning copies that host's
-`authorized_keys` into the new machine, so a node built by the script above
-already satisfies it; an older machine may need the key added.
+**It needs SSH from the control plane to the node.** Provisioning copies the
+hypervisor's `authorized_keys` into the new machine; with `ssh -A` to the
+control plane and `sudo --preserve-env=SSH_AUTH_SOCK`, the same key that
+reaches the hypervisor reaches the node.
 
-It waits for ssh rather than failing on it, and learns the host key of a machine
-it has not seen (`accept-new`, which still refuses a *changed* key — the risk
-being guarded against is a substituted machine, not a new one). Both are needed
-to run the halves back to back: for the first minute of its life a new machine
-is still regenerating host keys and restarting sshd, so it refuses connections
-it accepted a moment earlier.
-
-It also checks that a configured engine actually runs, which is cheap here and
-expensive later — see below.
+`deploy/bootstrap/onboard-node.sh` is the script this replaced, for a machine
+holding an authority but not a current llmgw.
 
 ## The CPU model matters, and its default is wrong
 
@@ -116,32 +98,23 @@ The trade is live migration: a VM pinned to the host CPU cannot move to a
 machine with a different one. For a node whose job is arithmetic that is the
 right way round, and `--cpu` overrides it where it is not.
 
-## What onboarding does not do
+## Engines
 
-It gives a machine an identity and an agent. It does **not** install inference
-engines or model weights, and a node missing either looks healthy in
-`llmgw ctl nodes` right up until a replica is placed on it and the shard dies
-with "executable file not found".
+The script installs none, and does not need to: once the agent is connected,
+the control plane tells it which engines to have and it installs them itself
+— llama.cpp on every node, vLLM on every node with an NVIDIA GPU it can run
+on, SGLang where the control plane names a version. Each is checked to run
+before the node offers it, so a node never advertises an engine that cannot
+start. [install.md](install.md#engines) says what vLLM needs of the node.
 
-Onboarding does check that an engine you point it at can be executed, and says
-so loudly when it cannot. That check exists because the failure is invisible
-otherwise: `llama-server` on node05 was present, executable, and could not start
-for want of `libgomp1`, and the node reported `ready` in `llmgw ctl nodes` the
-whole time. Being told at onboarding beats finding out when a replica is placed
-and the reason is three logs away.
+`--llamacpp-bin`, `--vllm-bin` and `--sglang-python` are for an engine you
+installed yourself. For the agent to use yours rather than install its own,
+also set that engine's version to `""` under `engines` in `server.yaml`.
 
-So the node is not finished until:
-
-- the engines its catalog profiles name are installed (`vllm`, `sglang`,
-  `llama-server`), at the paths the agent configuration gives
-- the weights are on disk, at the `source` each model declares
-
-Both are deliberately outside this script. Engines are a platform decision with
-versions and drivers attached, and distributing weights is its own problem —
-hundreds of gigabytes that want a shared filesystem or a cache, not an scp from
-a laptop. llmgw will fetch weights for a model whose `source` is a URL, one
-copy per node, which covers the simple case; anything larger stays the
-operator's arrangement, and llmgw is content to find the files already there.
+Weights are the other thing a node needs. llmgw fetches them for a model whose
+`source` is a URL, one copy per node, which covers the simple case; anything
+larger — a shared filesystem, a cache — stays the operator's arrangement, and
+llmgw is content to find the files already there.
 
 The `sim` engine needs neither, which is what makes it useful for exercising
 everything else.
@@ -178,18 +151,24 @@ was cut short.
 
 ## Fabrics
 
-`--fabric` is how a node says which InfiniBand fabric it is attached to, and
-there is no way around stating it. OpenSM's default subnet prefix is `fe80::` on
-every fabric, so a matching prefix proves nothing and a wrong guess produces a
-replica that starts, passes health checks, and then hangs on its first
-all-reduce.
+`--fabric` is how a node says which InfiniBand fabric it is attached to. The
+name is checked against what the adapters report — every port on one fabric
+sees the same subnet managers — so two independent fabrics given one name are
+placed as two, and the cluster page says so; give each fabric its own name.
+The agent installs `infiniband-diags`, which that check needs, if it is missing.
+
+Before a node with a fabric joins, run `sudo llmgw doctor` on it: it checks the
+subnet manager, GPUDirect RDMA, the fabric manager and IMEX on NVLink machines,
+and the declared fabrics against the adapters. [fabrics.md](fabrics.md) has the
+rest.
 
 Omit it for a node with no fabric. Such a node is its own island: it can host
 single-node replicas and will never be given a multi-node one.
 
 Two adapters per fabric is the default; `--fabric ib-b:mlx5_2,mlx5_3` names them
 explicitly. An adapter must never appear under two fabrics — the agent refuses
-that configuration, because the consequence is the hang above.
+that configuration: a replica given adapters on two fabrics starts, passes its
+health checks, and hangs on its first all-reduce.
 
 ## Where the driver is
 

docs/onboarding.md

@@ -39,16 +39,40 @@ The landing page then links to it, and the rest is:
 
 ```
 llmgw users list --pending --store …
-ID                    EMAIL              NAME   STATE    PROJECT  SIGNED UP
-usr-k4mq…             ayse@example.com   Ayşe   pending  -        4m12s ago
+ID          EMAIL              STATE    ADMIN  MEMBER OF     SIGNED UP
+usr-k4mq…   ayse@example.com   pending  -      on their own  4m12s ago
 
-llmgw users approve usr-k4mq… --project default --by ilker --store …
+llmgw users approve ayse@example.com --by ilker --store …
+llmgw users approve ayse@example.com --project Acme/ml/training --store …   # or straight into a project
 ```
 
 That is the whole operator loop. From then on they sign in at `/login`, and
-`/profile` is where they create keys, see when each was last used, and revoke the
-ones they are done with. A key is shown once on the page that creates it and
-never again.
+their account page (`/profile`) is where they create keys, see when each was
+last used, revoke the ones they are done with, and see their own use over the
+last thirty days against every budget that applies to them. A key is shown once
+on the page that creates it and never again.
+
+### Connecting an editor
+
+The account page's **Connect your editor** makes a key for Continue, Codex or
+Cline and shows what the tool needs with that key, this gateway's address and a
+model already in it:
+
+- **Continue** — an assistant file of its own, `~/.continue/assistants/llmgw.yaml`,
+  so their `config.yaml` is not touched.
+- **Codex** — a provider and a profile in `~/.codex/config.toml`, run with
+  `codex --profile llmgw`; their default model stays as it was. Codex speaks the
+  Responses API, which the gateway translates at `/v1/responses`.
+- **Cline** — four settings to paste into its own screen, since it keeps them
+  in VS Code's secret storage where no file reaches.
+
+For Continue and Codex there is a one-line command,
+`curl -fsSL <gateway>/setup/<tool>?model=… | sh`. Its script is served without
+signing in and holds no secret — the key comes from the environment — and it
+keeps a copy of anything it replaces. Revoking the key disconnects the tool.
+
+How well an editor's agent mode works depends on the model: it needs one that
+calls tools well, which the coder models do and small general ones often do not.
 
 **This gateway sends no email.** There is no "your account was approved" message
 and no password reset, so tell people what to expect: they sign up, they tell
@@ -88,11 +112,90 @@ means anything.
 If you have no structure yet, one of each is fine:
 
 ```
-llmgw keys init --store … --org mynodes --team default --project default
+llmgw org add org mynodes
+llmgw org add team default --in mynodes
+llmgw org add project default --in mynodes/default
+llmgw org tree --people
 ```
 
-It is idempotent: running it again renames rather than duplicating, so it can
-live in whatever provisions the machine.
+Everything that takes an organization, team or project — here, in `users` and
+in `keys` — accepts its id, its path (`mynodes/default/default`) or its name
+when the name is unique, so an id never has to be copied off a page. When there
+is only one project, `--project` can be left out.
+
+The tree is not fixed once made:
+
+```
+llmgw org rename project Acme/platform/gateway api-gateway
+llmgw org move project Acme/platform/gateway --to Scania/ml
+llmgw org remove team Acme/old                 # only when nothing is in it
+llmgw users delete ayse@example.com --yes      # signs out, revokes their keys
+llmgw keys list --all                          # every key, where it spends, and whose
+```
+
+A project that ever had a key cannot be removed, because usage history refers
+to it — rename it instead.
+
+### Where people belong
+
+A person can be a member of any number of organizations, teams and projects, or
+of none:
+
+```
+llmgw users join ayse@example.com --team Acme/ml          # every project in ml
+llmgw users join ayse@example.com --project Scania/research/vision
+llmgw users leave ayse@example.com --project Acme/ml/training
+```
+
+A member of a team or organization may use every project in it. Leaving revokes
+the keys the person made in projects they can no longer reach; a project they
+still reach through another membership keeps them.
+
+Someone who is a member of nothing still uses the gateway, on their own:
+personal keys, and the chat page, within their personal limits. When they make
+a key on `/profile` they choose what it spends against — one of their projects,
+or "Personal". An operator can do the same:
+
+```
+llmgw keys create laptop --for ayse@example.com                      # personal
+llmgw keys create ci --for ayse@example.com --project Acme/ml/training
+```
+
+Personal use can be limited by default, or turned off so people use the gateway
+only through their projects:
+
+```yaml
+auth:
+  personal:
+    rpm: 20
+    tpm: 50000
+    budget: 10        # whole units
+    period: month
+    # disabled: true
+```
+
+A personal key has no project, so it keeps no conversations or agent notes;
+those are kept per project.
+
+### Organization managers
+
+A member of an organization can be made its manager:
+
+```
+llmgw users join boss@acme.com --org Acme --role manager
+```
+
+or with the star on their organization membership on `/admin/users`. A manager
+gets an **Organization** link and a page at `/org`: usage over time by team,
+project or person, the teams and projects with their limits and spend, and every
+member. Only what was spent in that organization's projects is on it; what its
+people do elsewhere is not. The page is read-only — a manager is not an
+administrator of the cluster, and limits and memberships stay with those who
+are.
+
+`llmgw keys init` still works for provisioning scripts and is idempotent, but it
+will no longer move a team or project that already exists under another parent;
+it says so and stops.
 
 Split it when you want separate limits or separate reporting, not before.
 
@@ -156,18 +259,34 @@ Allowlists compose by intersection: a key cannot add a model its team withheld.
 A model's canonical name and any of its aliases name the same thing, so either
 form works.
 
-Budgets are counted in micro-units, and what a unit means is yours to decide.
-Self-hosted models have no per-token price, so it can be GPU-seconds, a notional
-rate, or 1 000 000 everywhere — which turns a budget into a plain token quota.
+Budgets are given in whole units, as the quotas page shows them, and what a unit
+means is yours to decide. Self-hosted models have no per-token price, so it can
+be GPU-seconds, a notional rate, or one unit per thousand tokens everywhere —
+a price of 1 000 000 in `auth.pricing`, which is in micro-units — and a budget
+becomes a plain quota of thousands of tokens.
 Models are priced in the server's `auth.pricing` section; a model priced at zero
 is exempt from budget checks, which is what lets a key with nothing left still be
 served a cheap fallback.
 
 ```
-llmgw keys limits --store … --scope project --id default \
-  --budget 50000000 --period month        # 50 units a month
+llmgw keys limits --store … --scope project --id Acme/platform/gateway \
+  --budget 50 --period month              # 50 units a month
 ```
 
+Two more kinds of limit follow people rather than the tree:
+
+```
+# A person, everywhere: every key they hold, every project, the chat page
+llmgw keys limits --scope user --id ayse@example.com --budget 25
+
+# Each member of a project separately — a fair share, not one shared pool
+llmgw keys limits --scope project --id Acme/ml/training --per-member --rpm 20
+```
+
+A request is held to all of them at once: the key's own, its project's, team's
+and organization's, each per-member limit on those for its owner, and the
+owner's own. The strictest wins, and each is counted in its own pool.
+
 ## The landing page
 
 Someone who reaches the gateway without a key sees the landing page. Tell it who
@@ -286,9 +405,14 @@ page exists.
 
 ## Managing accounts from the browser
 
-`/admin/users` lists who is waiting and who is active. Approving (into a project)
-and suspending are there, because that is the work an operator does constantly
-and it should not need a terminal.
+`/admin/users` is the tree: organizations, teams and projects, each project
+opening onto its members and its keys. From there an admin approves people
+(into a project or on their own), adds them to and takes them out of
+organizations, teams and projects, suspends and removes them, revokes keys, and
+adds, renames, moves and removes organizations, teams and projects. `/admin/quotas`
+sets limits on the tree, per member of any part of it, and per person. A searchable list of every account is at the
+bottom. The page works as plain forms; with scripts on, a change updates it in
+place and the projects you had open stay open.
 
 Three things are deliberately not there:
 

docs/quick-install.md

@@ -0,0 +1,202 @@
+# Quick install
+
+From empty servers to a model answering requests, in six steps. This page
+has only the commands and what to check after each; [install.md](install.md)
+explains every choice.
+
+## What you need
+
+| | |
+|---|---|
+| **One control plane machine** | Linux, x86-64 or ARM64. Debian 12, Ubuntu 22.04+ or RHEL 10. No GPU needed; 2 GB RAM, 20 GB disk. |
+| **GPU nodes** (optional to start) | The same Linux, with the NVIDIA driver installed. vLLM needs driver 580 or newer and a Turing GPU or newer (not V100). |
+| **root** | on the control plane, and `sudo` without a password on every node. |
+| **SSH** | from the control plane to each node, as a user with passwordless `sudo` there. |
+| **Network** | nodes reach the control plane on 7443 and 8081; users reach it on 8080. The machines need internet access to download engines and models. |
+
+You do not install llama.cpp, vLLM, Python or CUDA. The nodes do that
+themselves once they join.
+
+## 1. Put llmgw on the control plane
+
+```
+VERSION=$(curl -fsSL https://gitea.mynodes.xyz/api/v1/repos/ilker/llmgw/releases/latest \
+  | grep -o '"tag_name":"[^"]*"' | cut -d'"' -f4)
+U=https://gitea.mynodes.xyz/ilker/llmgw/releases/download/$VERSION
+curl -fsSLO $U/llmgw-$VERSION-linux-amd64          # -linux-arm64 on ARM
+curl -fsSL $U/checksums.txt | sha256sum -c --ignore-missing
+sudo install -m0755 llmgw-$VERSION-linux-amd64 /usr/local/bin/llmgw
+llmgw --version
+```
+
+`sha256sum` must print `OK`. If it does not, the download is damaged; do not
+install it.
+
+## 2. Set up the control plane
+
+```
+sudo llmgw server init
+```
+
+It asks for your email and a few choices, then sets up everything on this
+machine, in a few minutes. In a script, give them as flags instead:
+`sudo llmgw server init --admin you@example.com`.
+
+- the service, its certificates and its configuration;
+- your administrator account. **The password is printed once. Copy it now.**
+- a model mirror, which downloads model files once for the whole cluster;
+- this machine as a first node, and a small model (Qwen2.5 0.5B) on it.
+
+If nodes will reach this machine by a name or address it does not know itself
+(a DNS name, a NAT address), add `--host` for each:
+
+```
+sudo llmgw server init --admin you@example.com --host llm.example.com
+```
+
+It is safe to run again. It keeps what exists and finishes what is missing.
+
+**The mirror on its own machine**, when it wants a larger disk than the
+control plane has: first on that machine, with llmgw installed as in step 1,
+
+```
+sudo llmgw mirror init --control-plane <control plane address>
+```
+
+then on the control plane `sudo llmgw server init --mirror http://<mirror
+address>:8082`. The mirror serves model files to every node, and takes
+orders only from the control plane. Untick "This machine as a node" in the
+form (or add `--no-node`) to keep models off the control plane; the starter
+model then starts on the first node you add.
+
+**Check:** open `http://<control-plane>:8080/chat`, sign in with the printed
+password, and ask something. The small model answers. Change the password on
+the account page.
+
+## 3. Add the GPU nodes
+
+On the control plane:
+
+```
+sudo llmgw node add ubuntu@10.0.0.21
+```
+
+Several at once: `sudo llmgw node add ubuntu@10.0.0.21 ubuntu@10.0.0.22 ubuntu@10.0.0.23`.
+
+Replace `ubuntu` with the user you log in to the node as, and `10.0.0.21` with
+its address. That user needs `sudo` without a password on the node.
+
+It logs in over SSH and, while you watch:
+
+1. installs llmgw on the node, the release for its processor, checksum checked;
+2. runs `llmgw doctor` there, and stops if the node has a problem. Each problem
+   comes with the command that fixes it. Fix it and run the same command again;
+3. gives the node its certificate and configuration, and starts its agent;
+4. waits for the node to join and install llama.cpp.
+
+The node's name in the cluster is its hostname. vLLM installs by itself after
+that, in a few more minutes on a GPU node (about 8 GB). The cluster page
+(`/admin`) shows its progress, or why the node cannot run it.
+
+**InfiniBand.** When the node's adapters are up, `node add` shows them grouped
+by the fabric they are on and asks for each fabric's name. Give every node on
+the same fabric the same name. In a script, give it instead:
+
+```
+sudo llmgw node add ubuntu@10.0.0.21 --fabric ib-a:mlx5_0,mlx5_1,mlx5_2,mlx5_3
+```
+
+Once two nodes on a fabric are in, check they reach each other at the link's
+rate before placing anything across them:
+
+```
+llmgw ctl fabric-test gpu01 gpu02
+```
+
+**If it cannot log in.** `sudo` runs SSH as root, with root's keys. To use your
+own keys instead, log in to the control plane with `ssh -A` and keep your
+agent through sudo:
+
+```
+sudo --preserve-env=SSH_AUTH_SOCK llmgw node add ubuntu@10.0.0.21
+```
+
+**Check:** `llmgw ctl nodes` lists the node. Running `node add` again on the
+same node is safe: it renews its certificate and configuration, and keeps what
+runs there.
+
+A machine retired for good is removed with `llmgw node remove <name>` once its
+agent is stopped.
+
+## 4. Add a model
+
+Open `/admin/mirror` and choose **Explore models**. It lists what is popular on
+Hugging Face, showing only what your nodes can run.
+
+1. Click a model to read its card.
+2. Pick one of its files, usually a `q4_k_m` for a first try, and click
+   **Fetch**. It is downloaded into the mirror once, for every node.
+3. When the download finishes, click **serve** on its row in the table above.
+4. In the form, choose how it runs: **always on**, or **on demand**, which
+   loads it on the first request and stops it after 15 minutes unused. Click
+   **Add to the catalog**.
+
+The model is placed on a node that fits it, and its status on `/admin` turns
+`ready`.
+
+**Check:** the model appears in the list on `/chat`. Ask it something.
+
+## 5. Give people access
+
+**A key for a program:** each person makes their own on the account page
+(`/profile`). The key is shown once.
+
+```
+curl http://<control-plane>:8080/v1/chat/completions \
+  -H "Authorization: Bearer $LLMGW_API_KEY" \
+  -H "Content-Type: application/json" \
+  -d '{"model": "<model-name>", "messages": [{"role": "user", "content": "hello"}]}'
+```
+
+Any OpenAI client works. Only the base URL changes:
+`http://<control-plane>:8080/v1`.
+
+**An editor:** the account page connects Continue, Codex or Cline. Pick the
+tool and a model, and it gives a one-line command that writes the
+configuration.
+
+**Other people's accounts:** they sign up at `/signup`, and you approve them on
+the people page (`/admin/users`).
+
+## 6. Put it behind HTTPS
+
+Port 8080 is plain HTTP. Before users outside a trusted network use it, put a
+reverse proxy in front with a real certificate. With Caddy, which gets the
+certificate by itself:
+
+```
+sudo apt-get install -y caddy          # Debian and Ubuntu
+curl -fsSL $U/Caddyfile | sed 's/llm.example.com/YOUR.DOMAIN/' | sudo tee /etc/caddy/Caddyfile
+sudo systemctl reload caddy
+```
+
+Keep 7443 (nodes) and 8081 (mirror) on the internal network. They never go
+through the proxy. There is an nginx configuration too:
+[deploy/proxy/](../deploy/proxy/).
+
+## When something does not work
+
+| What you see | What to do |
+|---|---|
+| `node add` stops at `join` | `sudo journalctl -u llmgw-agent -n 50` on the node. Usually it cannot reach port 7443 on the control plane. |
+| `node add` says the address is not in the certificate | Run `sudo llmgw server init --host <that address>`, then `node add` again. |
+| A node says it cannot run vLLM | The reason is on `/admin`: the driver is older than 580, the GPU is older than Turing, or there is less than 25 GB free. |
+| A model shows `not placed` | `llmgw ctl explain-placement <model>` names each node and why it was not used. |
+| A reply stops after 60 seconds behind nginx | Use the nginx configuration from the release. Its timeouts are set for slow models. |
+| Anything else about a node | `sudo llmgw doctor` on it. `--json` writes everything it read, to send to whoever helps you. |
+
+## Next
+
+- [install.md](install.md): the same steps with every option explained.
+- [fabrics.md](fabrics.md): InfiniBand and NVLink.
+- [onboarding.md](onboarding.md): accounts, projects, budgets and keys.

docs/quickstart.md

@@ -1,8 +1,8 @@
 # Quickstart: one machine, one model, ten minutes
 
-Written for someone who wants to see llmgw answer a request on a single
-workstation before reading anything else. No systemd, no `sudo`. For a real
-cluster, see [install.md](install.md).
+Written for a developer who wants to see llmgw answer a request on a single
+workstation, built from source, with no systemd and no `sudo`. **To install
+llmgw on servers, read [quick-install.md](quick-install.md) instead.**
 
 What you end up with: an OpenAI-compatible endpoint on `localhost:8080` in front
 of a real model served by llama.cpp.

go.mod

@@ -6,15 +6,23 @@ require (
 	connectrpc.com/connect v1.21.0
 	github.com/NVIDIA/go-nvml v0.13.4-0
 	github.com/cespare/xxhash/v2 v2.3.0
+	github.com/charmbracelet/bubbletea v1.3.10
+	github.com/charmbracelet/lipgloss v1.1.0
 	github.com/jackc/pgx/v5 v5.11.0
 	github.com/knadh/koanf/parsers/yaml v1.1.1
 	github.com/knadh/koanf/providers/env/v2 v2.0.1
 	github.com/knadh/koanf/providers/file v1.2.1
 	github.com/knadh/koanf/v2 v2.3.6
+	github.com/mattn/go-isatty v0.0.24
 	github.com/prometheus/client_golang v1.24.1
 	github.com/prometheus/client_model v0.6.3
 	github.com/prometheus/common v0.71.0
 	github.com/spf13/cobra v1.10.2
+	github.com/yuin/goldmark v1.8.6
+	go.opentelemetry.io/otel v1.46.0
+	go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.46.0
+	go.opentelemetry.io/otel/sdk v1.46.0
+	go.opentelemetry.io/otel/trace v1.46.0
 	golang.org/x/crypto v0.57.0
 	golang.org/x/net v0.59.0
 	golang.org/x/sync v0.23.0
@@ -23,27 +31,54 @@ require (
 )
 
 require (
+	github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
 	github.com/beorn7/perks v1.0.1 // indirect
+	github.com/cenkalti/backoff/v5 v5.0.3 // indirect
+	github.com/charmbracelet/colorprofile v0.4.1 // indirect
+	github.com/charmbracelet/x/ansi v0.11.6 // indirect
+	github.com/charmbracelet/x/cellbuf v0.0.15 // indirect
+	github.com/charmbracelet/x/term v0.2.2 // indirect
+	github.com/clipperhouse/displaywidth v0.9.0 // indirect
+	github.com/clipperhouse/stringish v0.1.1 // indirect
+	github.com/clipperhouse/uax29/v2 v2.5.0 // indirect
 	github.com/dustin/go-humanize v1.0.1 // indirect
+	github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f // indirect
 	github.com/fsnotify/fsnotify v1.9.0 // indirect
-	github.com/go-viper/mapstructure/v2 v2.4.0 // indirect
+	github.com/go-logr/logr v1.4.4 // indirect
+	github.com/go-logr/stdr v1.2.2 // indirect
+	github.com/go-viper/mapstructure/v2 v2.5.0 // indirect
 	github.com/google/uuid v1.6.0 // indirect
+	github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0 // indirect
 	github.com/inconshreveable/mousetrap v1.1.0 // indirect
 	github.com/jackc/pgpassfile v1.0.0 // indirect
 	github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
 	github.com/jackc/puddle/v2 v2.2.2 // indirect
 	github.com/knadh/koanf/maps v0.1.2 // indirect
-	github.com/mattn/go-isatty v0.0.24 // indirect
+	github.com/lucasb-eyer/go-colorful v1.3.0 // indirect
+	github.com/mattn/go-localereader v0.0.1 // indirect
+	github.com/mattn/go-runewidth v0.0.19 // indirect
 	github.com/mitchellh/copystructure v1.2.0 // indirect
 	github.com/mitchellh/reflectwalk v1.0.2 // indirect
+	github.com/muesli/ansi v0.0.0-20230316100256-276c6243b2f6 // indirect
+	github.com/muesli/cancelreader v0.2.2 // indirect
+	github.com/muesli/termenv v0.16.0 // indirect
 	github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
 	github.com/ncruces/go-strftime v1.0.0 // indirect
 	github.com/prometheus/procfs v0.21.1 // indirect
 	github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
+	github.com/rivo/uniseg v0.4.7 // indirect
 	github.com/spf13/pflag v1.0.9 // indirect
+	github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e // indirect
+	go.opentelemetry.io/auto/sdk v1.2.1 // indirect
+	go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.46.0 // indirect
+	go.opentelemetry.io/otel/metric v1.46.0 // indirect
+	go.opentelemetry.io/proto/otlp v1.11.0 // indirect
 	go.yaml.in/yaml/v3 v3.0.5 // indirect
 	golang.org/x/sys v0.48.0 // indirect
 	golang.org/x/text v0.42.0 // indirect
+	google.golang.org/genproto/googleapis/api v0.0.0-20260819154853-08b0e4226688 // indirect
+	google.golang.org/genproto/googleapis/rpc v0.0.0-20260819154853-08b0e4226688 // indirect
+	google.golang.org/grpc v1.83.1 // indirect
 	modernc.org/libc v1.75.7 // indirect
 	modernc.org/mathutil v1.7.1 // indirect
 	modernc.org/memory v1.12.1 // indirect

go.sum

@@ -2,24 +2,57 @@ connectrpc.com/connect v1.21.0 h1:LhqSJt7jHf5NJBo9Jq/t/9FjcYAideif0mg+qe2jCUs=
 connectrpc.com/connect v1.21.0/go.mod h1:A2ygJrukXwWy32vkCAAHNVguZrqZ+jeZ9rGRnGR4dN4=
 github.com/NVIDIA/go-nvml v0.13.4-0 h1:o3jp9u2x1R9ShFE3v+Aesp55XOSIQFMJz/VGNUcJaNE=
 github.com/NVIDIA/go-nvml v0.13.4-0/go.mod h1:id63qwpoDWpFXwnwM6psDCSqW4BmNu6mWpr4YeQtPGo=
+github.com/aymanbagabas/go-osc52/v2 v2.0.1 h1:HwpRHbFMcZLEVr42D4p7XBqjyuxQH5SMiErDT4WkJ2k=
+github.com/aymanbagabas/go-osc52/v2 v2.0.1/go.mod h1:uYgXzlJ7ZpABp8OJ+exZzJJhRNQ2ASbcXHWsFqH8hp8=
 github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
 github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
+github.com/cenkalti/backoff/v5 v5.0.3 h1:ZN+IMa753KfX5hd8vVaMixjnqRZ3y8CuJKRKj1xcsSM=
+github.com/cenkalti/backoff/v5 v5.0.3/go.mod h1:rkhZdG3JZukswDf7f0cwqPNk4K0sa+F97BxZthm/crw=
 github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
 github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
+github.com/charmbracelet/bubbletea v1.3.10 h1:otUDHWMMzQSB0Pkc87rm691KZ3SWa4KUlvF9nRvCICw=
+github.com/charmbracelet/bubbletea v1.3.10/go.mod h1:ORQfo0fk8U+po9VaNvnV95UPWA1BitP1E0N6xJPlHr4=
+github.com/charmbracelet/colorprofile v0.4.1 h1:a1lO03qTrSIRaK8c3JRxJDZOvhvIeSco3ej+ngLk1kk=
+github.com/charmbracelet/colorprofile v0.4.1/go.mod h1:U1d9Dljmdf9DLegaJ0nGZNJvoXAhayhmidOdcBwAvKk=
+github.com/charmbracelet/lipgloss v1.1.0 h1:vYXsiLHVkK7fp74RkV7b2kq9+zDLoEU4MZoFqR/noCY=
+github.com/charmbracelet/lipgloss v1.1.0/go.mod h1:/6Q8FR2o+kj8rz4Dq0zQc3vYf7X+B0binUUBwA0aL30=
+github.com/charmbracelet/x/ansi v0.11.6 h1:GhV21SiDz/45W9AnV2R61xZMRri5NlLnl6CVF7ihZW8=
+github.com/charmbracelet/x/ansi v0.11.6/go.mod h1:2JNYLgQUsyqaiLovhU2Rv/pb8r6ydXKS3NIttu3VGZQ=
+github.com/charmbracelet/x/cellbuf v0.0.15 h1:ur3pZy0o6z/R7EylET877CBxaiE1Sp1GMxoFPAIztPI=
+github.com/charmbracelet/x/cellbuf v0.0.15/go.mod h1:J1YVbR7MUuEGIFPCaaZ96KDl5NoS0DAWkskup+mOY+Q=
+github.com/charmbracelet/x/term v0.2.2 h1:xVRT/S2ZcKdhhOuSP4t5cLi5o+JxklsoEObBSgfgZRk=
+github.com/charmbracelet/x/term v0.2.2/go.mod h1:kF8CY5RddLWrsgVwpw4kAa6TESp6EB5y3uxGLeCqzAI=
+github.com/clipperhouse/displaywidth v0.9.0 h1:Qb4KOhYwRiN3viMv1v/3cTBlz3AcAZX3+y9OLhMtAtA=
+github.com/clipperhouse/displaywidth v0.9.0/go.mod h1:aCAAqTlh4GIVkhQnJpbL0T/WfcrJXHcj8C0yjYcjOZA=
+github.com/clipperhouse/stringish v0.1.1 h1:+NSqMOr3GR6k1FdRhhnXrLfztGzuG+VuFDfatpWHKCs=
+github.com/clipperhouse/stringish v0.1.1/go.mod h1:v/WhFtE1q0ovMta2+m+UbpZ+2/HEXNWYXQgCt4hdOzA=
+github.com/clipperhouse/uax29/v2 v2.5.0 h1:x7T0T4eTHDONxFJsL94uKNKPHrclyFI0lm7+w94cO8U=
+github.com/clipperhouse/uax29/v2 v2.5.0/go.mod h1:Wn1g7MK6OoeDT0vL+Q0SQLDz/KpfsVRgg6W7ihQeh4g=
 github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
 github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
 github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
+github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f h1:Y/CXytFA4m6baUTXGLOoWe4PQhGxaX0KpnayAqC48p4=
+github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f/go.mod h1:vw97MGsxSvLiUE2X8qFplwetxpGLQrlU1Q9AUEIzCaM=
 github.com/fsnotify/fsnotify v1.9.0 h1:2Ml+OJNzbYCTzsxtv8vKSFD9PbJjmhYF14k/jKC7S9k=
 github.com/fsnotify/fsnotify v1.9.0/go.mod h1:8jBTzvmWwFyi3Pb8djgCCO5IBqzKJ/Jwo8TRcHyHii0=
-github.com/go-viper/mapstructure/v2 v2.4.0 h1:EBsztssimR/CONLSZZ04E8qAkxNYq4Qp9LvH92wZUgs=
-github.com/go-viper/mapstructure/v2 v2.4.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM=
+github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
+github.com/go-logr/logr v1.4.4 h1:tG4xh9yMsRCAiodLVTxyrkzSZ9+o0L1Kg/+cPVcbP/8=
+github.com/go-logr/logr v1.4.4/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
+github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
+github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
+github.com/go-viper/mapstructure/v2 v2.5.0 h1:vM5IJoUAy3d7zRSVtIwQgBj7BiWtMPfmPEgAXnvj1Ro=
+github.com/go-viper/mapstructure/v2 v2.5.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM=
+github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek=
+github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps=
 github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
 github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
 github.com/google/pprof v0.0.0-20260802141513-ef3492d7dac3 h1:LMLX+LgTNWpfvCBdFebv6EsYotImrt/Ppc5cXIriCSo=
 github.com/google/pprof v0.0.0-20260802141513-ef3492d7dac3/go.mod h1:jl5iWTm0/hd5PjEYEOuwAJ57L/CibdZfrqZ5XA5GrCk=
 github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
 github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
+github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0 h1:/Tnpcb2E0Pz/tN9s3bfEY2Q8ePCEX9iuS+cneUwncnw=
+github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0/go.mod h1:zOBXOsUaBSjKgmH4OGzV1esUpR3oUSCPYVd2cUBjKYY=
 github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
 github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
 github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
@@ -46,12 +79,24 @@ github.com/knadh/koanf/v2 v2.3.6 h1:JoQPSJmvS4aP0xNc8xMDr5tcrkSEInL23/Il7pITAKo=
 github.com/knadh/koanf/v2 v2.3.6/go.mod h1:gRb40VRAbd4iJMYYD5IxZ6hfuopFcXBpc9bbQpZwo28=
 github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc=
 github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw=
+github.com/lucasb-eyer/go-colorful v1.3.0 h1:2/yBRLdWBZKrf7gB40FoiKfAWYQ0lqNcbuQwVHXptag=
+github.com/lucasb-eyer/go-colorful v1.3.0/go.mod h1:R4dSotOR9KMtayYi1e77YzuveK+i7ruzyGqttikkLy0=
 github.com/mattn/go-isatty v0.0.24 h1:tGZZoVgT/KiqK1c8ocVLeDS8BSWMRd47J3Lbz7vsReI=
 github.com/mattn/go-isatty v0.0.24/go.mod h1:nMCL3Zebbrt45jsMDgnfIwz6ydEQApk5oEI3HqDio6A=
+github.com/mattn/go-localereader v0.0.1 h1:ygSAOl7ZXTx4RdPYinUpg6W99U8jWvWi9Ye2JC/oIi4=
+github.com/mattn/go-localereader v0.0.1/go.mod h1:8fBrzywKY7BI3czFoHkuzRoWE9C+EiG4R1k4Cjx5p88=
+github.com/mattn/go-runewidth v0.0.19 h1:v++JhqYnZuu5jSKrk9RbgF5v4CGUjqRfBm05byFGLdw=
+github.com/mattn/go-runewidth v0.0.19/go.mod h1:XBkDxAl56ILZc9knddidhrOlY5R/pDhgLpndooCuJAs=
 github.com/mitchellh/copystructure v1.2.0 h1:vpKXTN4ewci03Vljg/q9QvCGUDttBOGBIa15WveJJGw=
 github.com/mitchellh/copystructure v1.2.0/go.mod h1:qLl+cE2AmVv+CoeAwDPye/v+N2HKCj9FbZEVFJRxO9s=
 github.com/mitchellh/reflectwalk v1.0.2 h1:G2LzWKi524PWgd3mLHV8Y5k7s6XUvT0Gef6zxSIeXaQ=
 github.com/mitchellh/reflectwalk v1.0.2/go.mod h1:mSTlrgnPZtwu0c4WaC2kGObEpuNDbx0jmZXqmk4esnw=
+github.com/muesli/ansi v0.0.0-20230316100256-276c6243b2f6 h1:ZK8zHtRHOkbHy6Mmr5D264iyp3TiX5OmNcI5cIARiQI=
+github.com/muesli/ansi v0.0.0-20230316100256-276c6243b2f6/go.mod h1:CJlz5H+gyd6CUWT45Oy4q24RdLyn7Md9Vj2/ldJBSIo=
+github.com/muesli/cancelreader v0.2.2 h1:3I4Kt4BQjOR54NavqnDogx/MIoWBFa0StPA8ELUXHmA=
+github.com/muesli/cancelreader v0.2.2/go.mod h1:3XuTXfFS2VjM+HTLZY9Ak0l6eUKfijIfMUZ4EgX0QYo=
+github.com/muesli/termenv v0.16.0 h1:S5AlUN9dENB57rsbnkPyfdGuWIlkmzJjbFf0Tf5FWUc=
+github.com/muesli/termenv v0.16.0/go.mod h1:ZRfOIKPFDYQoDFF4Olj7/QJbW60Ol/kL1pU3VfY/Cnk=
 github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
 github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
 github.com/ncruces/go-strftime v1.0.0 h1:HMFp8mLCTPp341M/ZnA4qaf7ZlsbTc+miZjCLOFAw7w=
@@ -67,6 +112,8 @@ github.com/prometheus/procfs v0.21.1 h1:GljZCt+zSTS+NZq88cyQ1LjZ+RCHp3uVuabBWA5+
 github.com/prometheus/procfs v0.21.1/go.mod h1:aB55Cww9pdSJVHk0hUf0inxWyyjPogFIjmHKYgMKmtY=
 github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE=
 github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo=
+github.com/rivo/uniseg v0.4.7 h1:WUdvkW8uEhrYfLC4ZzdpI2ztxP1I582+49Oc5Mq64VQ=
+github.com/rivo/uniseg v0.4.7/go.mod h1:FN3SvrM+Zdj16jyLfmOkMNblXMcoc8DfTHruCPUcx88=
 github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
 github.com/spf13/cobra v1.10.2 h1:DMTTonx5m65Ic0GOoRY2c16WCbHxOOw6xxezuLaBpcU=
 github.com/spf13/cobra v1.10.2/go.mod h1:7C1pvHqHw5A4vrJfjNwvOdzYu0Gml16OCs2GRiTUUS4=
@@ -77,6 +124,28 @@ github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UV
 github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
 github.com/stretchr/testify v1.12.1 h1:EuwCh5fleGS7H32xRwO3wRGT7DxrDhLAT6FF8MpWDWE=
 github.com/stretchr/testify v1.12.1/go.mod h1:MDEgiDPPsNp5cuIrHPPCyornHKgEVbtFUmoNlxoYthg=
+github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e h1:JVG44RsyaB9T2KIHavMF/ppJZNG9ZpyihvCd0w101no=
+github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e/go.mod h1:RbqR21r5mrJuqunuUZ/Dhy/avygyECGrLceyNeo4LiM=
+github.com/yuin/goldmark v1.8.6 h1:d0VcaP1sx9GkFVkoW+KtggpGi2KZ965i14b0+bDQST4=
+github.com/yuin/goldmark v1.8.6/go.mod h1:ip/1k0VRfGynBgxOz0yCqHrbZXhcjxyuS66Brc7iBKg=
+go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64=
+go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y=
+go.opentelemetry.io/otel v1.46.0 h1:FHt5/CDyVxi/8IM1CH7VE/rRgq3kLHa2mSTVMO8AWyc=
+go.opentelemetry.io/otel v1.46.0/go.mod h1:Gj3SEScelsNC45tp4nSxRYlS+f5iez7W8XPMCt905kE=
+go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.46.0 h1:OFnwLJr+pF3iHrlGSzbxyuo6/6HyBlnlN1CWEJmBVcw=
+go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.46.0/go.mod h1:716wFneO0ov19A2beH5hjfh9AK5z/VWNAtDijp1Y0/g=
+go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.46.0 h1:KrC1YrQeSt46ITMWAbgQx1M1eV1/1TKzttrBzymPmss=
+go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.46.0/go.mod h1:zDSEzoEqsOrgBeGvH66KRgxh90VonFyJqBHA0Pk3+rM=
+go.opentelemetry.io/otel/metric v1.46.0 h1:yBnkXvgV7AXFILZc5K6IZe/CBFF3OS7BJ8ov6/lj0K8=
+go.opentelemetry.io/otel/metric v1.46.0/go.mod h1:iPmdWqifKUdzziPkvvzIJXITl56fQx2mGM/DHLB3/2o=
+go.opentelemetry.io/otel/sdk v1.46.0 h1:h5CNQQjEbuQXY/JfZtgt3i7HVFV3aHPO2OAwO2eTYPI=
+go.opentelemetry.io/otel/sdk v1.46.0/go.mod h1:GAERFXFt5SYCEB+YiKUbMBeza6UaDH7GmGOZEfh2gSM=
+go.opentelemetry.io/otel/sdk/metric v1.46.0 h1:0piZ26EG4RBfebb2jhDH6ERCYHoVWduc3kLgPCwSnSE=
+go.opentelemetry.io/otel/sdk/metric v1.46.0/go.mod h1:I1PbKrdVc8Qu8HYVDNtqVIwLwjNrhsV/uFuxfwg8mO4=
+go.opentelemetry.io/otel/trace v1.46.0 h1:OULy7ccdJnZtJ0UDYFOIGaCmiWzJ8Vi2G/Rsu60qs1c=
+go.opentelemetry.io/otel/trace v1.46.0/go.mod h1:J7GAXweO77XSFkB/rmAqk9D6ihszhFjLU+d9WuUxDLI=
+go.opentelemetry.io/proto/otlp v1.11.0 h1:5rrYs0Ykyj50sdU/JU0x8etU+LubXWb+gED6TbEdMIk=
+go.opentelemetry.io/proto/otlp v1.11.0/go.mod h1:SmVizdCOAm3XBtG1g1NnOdhW6jtddT72hLMhv8VwA8E=
 go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
 go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
 go.yaml.in/yaml/v2 v2.4.4 h1:tuyd0P+2Ont/d6e2rl3be67goVK4R6deVxCUX5vyPaQ=
@@ -86,18 +155,29 @@ go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw=
 go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg=
 golang.org/x/crypto v0.57.0 h1:3ZVCjf8Ggz7zneR/EHRVx68Ctf+2pmIMP2UFhh9cC6M=
 golang.org/x/crypto v0.57.0/go.mod h1:Fdz0i5U6CoizGwLda9DttjSk6qlZo25zYNtR+ycvuZA=
+golang.org/x/exp v0.0.0-20231006140011-7918f672742d h1:jtJma62tbqLibJ5sFQz8bKtEM8rJBtfilJ2qTU199MI=
+golang.org/x/exp v0.0.0-20231006140011-7918f672742d/go.mod h1:ldy0pHrwJyGW56pPQzzkH36rKxoZW1tw7ZJpeKx+hdo=
 golang.org/x/mod v0.41.0 h1:qJmnOUb4YB+FsEuM3HcWucdZASCPGhsX6uljO6pog0c=
 golang.org/x/mod v0.41.0/go.mod h1:Ek9pY8RKWXwsWvd3rQiHYtMqkjSUV+s1Rj7j4H5Ur6o=
 golang.org/x/net v0.59.0 h1:5zfYln+w5XCxwrnMMJPufRgNoXEaGxl0wo5GqPXyues=
 golang.org/x/net v0.59.0/go.mod h1:2DA/G1UfVbCpQPeWTmMPGY7Cs2PkBkwu743bVX5PIVg=
 golang.org/x/sync v0.23.0 h1:KameEIfc1IkluZyXWLn39Wd4tURc6GbCiISGiZm2bQk=
 golang.org/x/sync v0.23.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0=
+golang.org/x/sys v0.0.0-20210809222454-d867a43fc93e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo=
 golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
 golang.org/x/text v0.42.0 h1:JbOZXgfeCPU9gacVtYliJqOhD+zhrEqK4LfdpmlUZqI=
 golang.org/x/text v0.42.0/go.mod h1:ojzP1Z+2QtioaF8DTtO8K5q7JWVVYwZKenzujK0Zd0E=
 golang.org/x/tools v0.49.0 h1:3NI7VXzL9+1WZD52Dx2ttoPwD5DWrFGpl9mFZDlmisI=
 golang.org/x/tools v0.49.0/go.mod h1:SJNXV9DBKT0UbdttsQjbfJlAE/q+y36++zo3uL3N0Oo=
+gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4=
+gonum.org/v1/gonum v0.17.0/go.mod h1:El3tOrEuMpv2UdMrbNlKEh9vd86bmQ6vqIcDwxEOc1E=
+google.golang.org/genproto/googleapis/api v0.0.0-20260819154853-08b0e4226688 h1:ax2KzoSRIZU/M0cIxri3pKxy99vniH1PVxWC6si/eZI=
+google.golang.org/genproto/googleapis/api v0.0.0-20260819154853-08b0e4226688/go.mod h1:1RJ9BQGyNdZwkGc1eTqkErfRZ6RJyYPHZo73BZ1vQqI=
+google.golang.org/genproto/googleapis/rpc v0.0.0-20260819154853-08b0e4226688 h1:cYNAzI2sUwhmCcoj9TxvihSrqsxt6uIkj3rDRhSDmW4=
+google.golang.org/genproto/googleapis/rpc v0.0.0-20260819154853-08b0e4226688/go.mod h1:DjtHYE8FKJLivXcBEjGwndXfIC23G0VpXiXKqG179uA=
+google.golang.org/grpc v1.83.1 h1:HIO0+BEtBP6soyqvqC8sNUjZ7bTs+0hFQuFF+RAy++Y=
+google.golang.org/grpc v1.83.1/go.mod h1:kDyl6SKsiHKt0uylY5gtn5cEjkrIOhQOGDgIc4JGwzQ=
 google.golang.org/protobuf v1.36.12 h1:pJOKDDOyeXErUroCihFAd5LQuwXBSpVnKGrj5o/fwxc=
 google.golang.org/protobuf v1.36.12/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=

internal/account/account_test.go

@@ -12,7 +12,7 @@ import (
 	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
 )
 
-func newStore(t *testing.T) *store.SQLite {
+func newStore(t *testing.T) *store.DB {
 	t.Helper()
 	s, err := store.OpenSQLite(context.Background(), filepath.Join(t.TempDir(), "a.db"))
 	if err != nil {
@@ -33,7 +33,7 @@ func newStore(t *testing.T) *store.SQLite {
 	return s
 }
 
-func newService(t *testing.T) (*Service, *store.SQLite) {
+func newService(t *testing.T) (*Service, *store.DB) {
 	t.Helper()
 	s := newStore(t)
 	return New(s, DefaultConfig(), slog.New(slog.DiscardHandler)), s
@@ -99,21 +99,25 @@ func TestApprovalIsWhatMakesAnAccountWork(t *testing.T) {
 	if err != nil {
 		t.Fatal(err)
 	}
-	if got.ProjectID != "proj" {
-		t.Errorf("the approved account landed in project %q", got.ProjectID)
+	if ok, err := st.CanUseProject(ctx, got.ID, "proj"); err != nil || !ok {
+		t.Errorf("the approved account is not a member of the project it was approved into: %v", err)
 	}
 }
 
-func TestApprovalNeedsAProject(t *testing.T) {
-	// An approved account with nowhere to put its keys passes every check and
-	// then fails at the first key it tries to make.
-	_, st := newStoreAndUser(t)
-	if err := st.ApproveUser(context.Background(), "usr-1", "", "ilker", time.Now()); err == nil {
-		t.Error("an account was approved into no project")
+func TestApprovalNeedNotNameAProject(t *testing.T) {
+	// A person need not belong to anything to use the gateway: approved into
+	// nothing, they work on their own within their personal limits.
+	svc, st := newStoreAndUser(t)
+	ctx := context.Background()
+	if err := st.ApproveUser(ctx, "usr-1", "", "ilker", time.Now()); err != nil {
+		t.Fatalf("approving with no project: %v", err)
+	}
+	if _, _, err := svc.SignIn(ctx, "ayse@example.com", goodPassword); err != nil {
+		t.Errorf("an account approved into nothing cannot sign in: %v", err)
 	}
 }
 
-func newStoreAndUser(t *testing.T) (*Service, *store.SQLite) {
+func newStoreAndUser(t *testing.T) (*Service, *store.DB) {
 	t.Helper()
 	svc, st := newService(t)
 	hash, _ := HashPassword(goodPassword)

internal/account/password.go

@@ -15,6 +15,7 @@ import (
 	"encoding/base64"
 	"errors"
 	"fmt"
+	"runtime"
 	"strings"
 
 	"golang.org/x/crypto/argon2"
@@ -56,7 +57,7 @@ func HashPassword(password string) (string, error) {
 	if _, err := rand.Read(salt); err != nil {
 		return "", err
 	}
-	sum := argon2.IDKey([]byte(password), salt, argonTime, argonMemory, argonThreads, argonKeyLen)
+	sum := idKey([]byte(password), salt, argonTime, argonMemory, argonThreads, argonKeyLen)
 
 	b64 := base64.RawStdEncoding.EncodeToString
 	return fmt.Sprintf("$argon2id$v=%d$m=%d,t=%d,p=%d$%s$%s",
@@ -93,7 +94,7 @@ func VerifyPassword(password, encoded string) (bool, error) {
 		return false, ErrBadHash
 	}
 
-	got := argon2.IDKey([]byte(password), salt, time, memory, threads, uint32(len(want)))
+	got := idKey([]byte(password), salt, time, memory, threads, uint32(len(want)))
 	return subtle.ConstantTimeCompare(got, want) == 1, nil
 }
 
@@ -110,3 +111,16 @@ func init() {
 	}
 	dummyHash = h
 }
+
+// argonSlots bounds how many hashes run at once. Each takes 64 MiB, and the
+// login and signup forms are on the client-facing port: without a bound, a
+// few hundred parallel attempts at random addresses asked for gigabytes at
+// once, and the per-address throttle does not see them — each address is new.
+// Beyond the bound, attempts wait their turn instead.
+var argonSlots = make(chan struct{}, max(2, runtime.NumCPU()/2))
+
+func idKey(password, salt []byte, time, memory uint32, threads uint8, keyLen uint32) []byte {
+	argonSlots <- struct{}{}
+	defer func() { <-argonSlots }()
+	return argon2.IDKey(password, salt, time, memory, threads, keyLen)
+}

internal/account/service.go

@@ -230,6 +230,22 @@ func (s *Service) allowAttempt(email string) bool {
 	defer s.mu.Unlock()
 
 	w, ok := s.attempts[email]
+	if !ok && len(s.attempts) >= maxTrackedAddresses {
+		// Full of addresses nobody has signed in as for a while: forget the
+		// ones whose window has passed. Only expiring, never evicting early —
+		// dropping a live window is how an attacker spraying random addresses
+		// would buy unlimited guesses at the one they care about.
+		for k, v := range s.attempts {
+			if now.Sub(v.since) > s.cfg.AttemptWindow {
+				delete(s.attempts, k)
+			}
+		}
+		if len(s.attempts) >= maxTrackedAddresses {
+			// Still full, so somebody is spraying addresses faster than they
+			// expire. New ones wait; those already tracked are unaffected.
+			return false
+		}
+	}
 	if !ok || now.Sub(w.since) > s.cfg.AttemptWindow {
 		s.attempts[email] = &attemptWindow{count: 1, since: now}
 		return true
@@ -238,6 +254,10 @@ func (s *Service) allowAttempt(email string) bool {
 	return w.count <= s.cfg.MaxAttempts
 }
 
+// maxTrackedAddresses bounds the throttle's memory. Unbounded, it grew by one
+// entry per address tried and shrank only on a successful sign-in.
+const maxTrackedAddresses = 50000
+
 func (s *Service) clearAttempts(email string) {
 	s.mu.Lock()
 	delete(s.attempts, email)

internal/account/throttle_bound_test.go

@@ -0,0 +1,33 @@
+package account
+
+import (
+	"fmt"
+	"testing"
+	"time"
+)
+
+func TestSprayingAddressesCannotGrowTheThrottleOrEscapeIt(t *testing.T) {
+	now := time.Unix(0, 0)
+	s := &Service{cfg: DefaultConfig(), attempts: map[string]*attemptWindow{}, now: func() time.Time { return now }}
+
+	// The target is being guessed at.
+	for i := 0; i < s.cfg.MaxAttempts; i++ {
+		s.allowAttempt("victim@example.com")
+	}
+	// Then the table is flooded with made-up addresses.
+	for i := 0; i < maxTrackedAddresses+100; i++ {
+		s.allowAttempt(fmt.Sprintf("x%d@example.com", i))
+	}
+	if n := len(s.attempts); n > maxTrackedAddresses {
+		t.Errorf("the table grew to %d", n)
+	}
+	if s.allowAttempt("victim@example.com") {
+		t.Error("flooding the table reset the target's throttle")
+	}
+
+	// Once the window passes, everything expires and sign-ins work again.
+	now = now.Add(s.cfg.AttemptWindow + time.Second)
+	if !s.allowAttempt("new@example.com") {
+		t.Error("a new address was refused after the flood expired")
+	}
+}

internal/accounting/accounting.go

@@ -5,6 +5,7 @@ import (
 	"errors"
 	"fmt"
 	"log/slog"
+	"strings"
 	"time"
 
 	"gitea.mynodes.xyz/ilker/llmgw/internal/auth"
@@ -70,19 +71,67 @@ func (a *Accountant) RestoreBudgets(ctx context.Context, s store.Store, period s
 	if start.IsZero() {
 		return nil
 	}
-	spend, err := s.SpendSince(ctx, start)
+	rows, err := s.SpendSinceByScope(ctx, start)
 	if err != nil {
 		return fmt.Errorf("restoring budgets: %w", err)
 	}
+	// Every scope a key sits under, not just the key. A budget owned by a team
+	// is one counter fed by every key beneath it, and restoring only per-key
+	// totals would bring the control plane back up believing the team had spent
+	// nothing — which is a budget that resets on every restart, for exactly the
+	// scopes where it matters most.
+	// And every pool a person is in: their own, and each per-member limit of
+	// the project, team and organization they spent it in.
+	spend := make(map[string]int64, len(rows)*8)
+	for _, r := range rows {
+		owners := []store.LimitOwner{
+			{Scope: store.ScopeKey, ID: r.KeyID},
+			{Scope: store.ScopeProject, ID: r.ProjectID},
+			{Scope: store.ScopeTeam, ID: r.TeamID},
+			{Scope: store.ScopeOrg, ID: r.OrgID},
+			{Scope: store.ScopeUser, ID: r.UserID},
+		}
+		if r.UserID != "" && r.ProjectID != "" {
+			owners = append(owners,
+				store.MemberOwner(store.ScopeProjectMember, r.ProjectID, r.UserID),
+				store.MemberOwner(store.ScopeTeamMember, r.TeamID, r.UserID),
+				store.MemberOwner(store.ScopeOrgMember, r.OrgID, r.UserID))
+		}
+		for _, o := range owners {
+			if o.ID == "" || strings.HasPrefix(o.ID, "/") {
+				continue
+			}
+			if k := o.CounterKey(); k != "" {
+				spend[k] += r.Micros
+			}
+		}
+	}
 	a.budgets.Restore(period, spend)
-	a.log.Info("restored budgets from the store", "keys", len(spend), "since", start)
+	a.log.Info("restored budgets from the store",
+		"keys", len(rows), "counters", len(spend), "since", start)
 	return nil
 }
 
 // Reservation is what Check returns: the estimate it charged, so Record can
 // settle the difference against the real figure.
+// pool is one level's counter and the period its budget runs over.
+type pool struct {
+	Counter string
+	Period  store.Period
+}
+
 type Reservation struct {
-	KeyID           string
+	KeyID string
+	// Budgets and Tokens are every counter the check went through, carried so
+	// the settlement lands on the same ones. Every, not the tightest: an
+	// organization's pool only fills up if each team's spend is added to it
+	// as well as to the team's own.
+	//
+	// Derived again at settlement would mean re-resolving the limits, and a
+	// budget checked against the team and settled against the key is a budget
+	// that never fills up.
+	Budgets         []pool
+	Tokens          []string
 	EstimatedTokens int64
 	Price           Price
 	// Period is the key's budget period, carried from the check so the record
@@ -99,14 +148,28 @@ type Reservation struct {
 // figure only exists once the request has run; Record settles the difference.
 func (a *Accountant) Check(p auth.Principal, model string, estimatedTokens int64) (Reservation, error) {
 	price := a.pricing.For(model)
+	lim := p.Limits
+	// Every level that set a limit, each against its own pool. A quota written
+	// on a team is one pool the team shares, and the organization above it is a
+	// second pool that the team's spending also fills — both have to hold, or
+	// an organization given 1000 with four teams given 500 spends 2000.
+	chain := levels(lim, p.KeyID)
+
 	res := Reservation{
 		KeyID: p.KeyID, EstimatedTokens: estimatedTokens,
-		Price: price, Period: p.Limits.BudgetPeriod,
+		Price: price, Period: lim.BudgetPeriod,
+	}
+	for _, l := range chain {
+		if l.Budget != nil {
+			res.Budgets = append(res.Budgets, pool{l.Counter, l.Period})
+		}
+		if l.TPM != nil {
+			res.Tokens = append(res.Tokens, l.Counter)
+		}
 	}
 	if p.KeyID == "" {
 		return res, nil
 	}
-	lim := p.Limits
 
 	// Budget first. It is the limit whose refusal cannot be retried away, so
 	// telling the client about it first is more useful than reporting a rate
@@ -114,15 +177,29 @@ func (a *Accountant) Check(p auth.Principal, model string, estimatedTokens int64
 	//
 	// A model that costs nothing is exempt, which is what lets a key with no
 	// budget left still be served a cheap fallback (plan §7.7).
-	if lim.Budget != nil && !price.Zero() &&
-		a.budgets.Exhausted(p.KeyID, lim.BudgetPeriod, *lim.Budget) {
-		return res, &LimitError{Err: ErrBudgetSpent}
+	// Budgets first, and all of them. Exhausted only reads, so the order costs
+	// nothing and a refusal here charges no counter.
+	if !price.Zero() {
+		for _, l := range chain {
+			if l.Budget != nil && a.budgets.Exhausted(l.Counter, l.Period, *l.Budget) {
+				return res, &LimitError{Err: ErrBudgetSpent}
+			}
+		}
 	}
 
-	if lim.RPM != nil && !a.requests.Allow(p.KeyID, *lim.RPM, 1) {
+	// Rate limits charge as they check — deliberately, so two concurrent
+	// requests cannot both see room. With several levels that means a refusal
+	// at one has already taken from the levels before it, so those are handed
+	// back. Without the refund a team at its ceiling would quietly eat its
+	// organization's rate limit with requests that were never served.
+	if err := a.allowAll(chain, func(l level) (string, int, int64, bool) {
+		return l.Counter, valInt(l.RPM), 1, l.RPM != nil
+	}, a.requests); err != nil {
 		return res, &LimitError{Err: ErrRateLimited, RetryAfter: time.Second}
 	}
-	if lim.TPM != nil && !a.tokens.Allow(p.KeyID, *lim.TPM, estimatedTokens) {
+	if err := a.allowAll(chain, func(l level) (string, int, int64, bool) {
+		return l.Counter, valInt(l.TPM), estimatedTokens, l.TPM != nil
+	}, a.tokens); err != nil {
 		// The request was not charged against the request-rate limit in vain:
 		// it was, and that is correct — it did arrive.
 		return res, &LimitError{Err: ErrTokenLimited, RetryAfter: time.Second}
@@ -130,6 +207,88 @@ func (a *Accountant) Check(p auth.Principal, model string, estimatedTokens int64
 	return res, nil
 }
 
+// level is one scope's limits with the counter they are enforced against.
+type level struct {
+	Counter string
+	Period  store.Period
+	RPM     *int
+	TPM     *int
+	Budget  *int64
+}
+
+// levels turns a resolved chain into the pools a request must pass.
+//
+// Falls back to the collapsed values when the chain is empty, which is what a
+// principal assembled by hand looks like — in tests, and anywhere that builds
+// limits without walking the tree.
+func levels(e store.Effective, keyID string) []level {
+	if len(e.Chain) == 0 {
+		var out []level
+		if e.RPM != nil || e.TPM != nil || e.Budget != nil {
+			out = append(out, level{
+				Counter: counterFor(e.BudgetOwner, keyID),
+				Period:  e.BudgetPeriod, Budget: e.Budget,
+			})
+			out[0].RPM, out[0].TPM = e.RPM, e.TPM
+			// One entry, so a hand-built principal with different owners per
+			// limit still counts each where it said to.
+			if r, b := counterFor(e.RPMOwner, keyID), out[0].Counter; r != b && e.RPM != nil {
+				out[0].RPM = nil
+				out = append(out, level{Counter: r, RPM: e.RPM})
+			}
+			if tp, b := counterFor(e.TPMOwner, keyID), out[0].Counter; tp != b && e.TPM != nil {
+				out[0].TPM = nil
+				out = append(out, level{Counter: tp, TPM: e.TPM})
+			}
+		}
+		return out
+	}
+	out := make([]level, 0, len(e.Chain))
+	for _, l := range e.Chain {
+		out = append(out, level{
+			Counter: store.LimitOwner{Scope: l.Scope, ID: l.ScopeID}.CounterKey(),
+			Period:  l.BudgetPeriod, RPM: l.RPM, TPM: l.TPM, Budget: l.Budget,
+		})
+	}
+	return out
+}
+
+// allowAll charges every level that has this limit, refunding on a refusal.
+func (a *Accountant) allowAll(chain []level,
+	pick func(level) (key string, limit int, n int64, has bool), lim *Limiter,
+) error {
+	var charged []struct {
+		key string
+		n   int64
+	}
+	for _, l := range chain {
+		key, limit, n, has := pick(l)
+		if !has {
+			continue
+		}
+		if !lim.Allow(key, limit, n) {
+			for _, c := range charged {
+				lim.Charge(c.key, -c.n)
+			}
+			return errRefused
+		}
+		charged = append(charged, struct {
+			key string
+			n   int64
+		}{key, n})
+	}
+	return nil
+}
+
+var errRefused = errors.New("refused")
+
+func valInt(v *int) int {
+	if v == nil {
+		return 0
+	}
+	return *v
+}
+
 // Record settles a finished request: it reconciles the token estimate, adds to
 // the budget, and queues the usage row.
 func (a *Accountant) Record(res Reservation, u store.Usage) {
@@ -141,10 +300,16 @@ func (a *Accountant) Record(res Reservation, u store.Usage) {
 	// The estimate was charged up front; charge the difference now. A caller
 	// whose requests are consistently larger than estimated is still bounded,
 	// one window later than with a perfect estimate.
-	a.tokens.Charge(res.KeyID, actual-res.EstimatedTokens)
+	// Every level the check went through, so an organization's pool fills up
+	// from its teams' spending rather than only from requests that named it.
+	for _, c := range res.Tokens {
+		a.tokens.Charge(c, actual-res.EstimatedTokens)
+	}
 
 	u.CostMicros = res.Price.Cost(u.PromptTokens, u.CompletionTokens)
-	a.budgets.Add(res.KeyID, res.Period, u.CostMicros)
+	for _, b := range res.Budgets {
+		a.budgets.Add(b.Counter, b.Period, u.CostMicros)
+	}
 
 	if a.recorder != nil {
 		a.recorder.Record(u)
@@ -156,6 +321,101 @@ func (a *Accountant) Status(p auth.Principal) (spent int64, rpm, tpm float64) {
 	if p.KeyID == "" {
 		return 0, 0, 0
 	}
-	return a.budgets.Spent(p.KeyID, p.Limits.BudgetPeriod),
-		a.requests.Rate(p.KeyID), a.tokens.Rate(p.KeyID)
+	// The pool that will refuse them, which is not always the one with the
+	// smallest number on it. A team given 2000 inside an organization given the
+	// same is bounded by whichever is nearer its ceiling, and that is the
+	// organization as soon as a second team spends anything. A header that
+	// reported the team's would disagree with the refusal that follows it.
+	for _, l := range levels(p.Limits, p.KeyID) {
+		if l.Budget != nil {
+			if v := a.budgets.Spent(l.Counter, l.Period); v > spent {
+				spent = v
+			}
+		}
+		if l.RPM != nil {
+			if v := a.requests.Rate(l.Counter); v > rpm {
+				rpm = v
+			}
+		}
+		if l.TPM != nil {
+			if v := a.tokens.Rate(l.Counter); v > tpm {
+				tpm = v
+			}
+		}
+	}
+	return spent, rpm, tpm
+}
+
+// counterFor is the counter a limit owned by this scope should use.
+//
+// Falls back to the key's own counter when no scope owns the limit, which
+// happens when nothing in the chain set one.
+//
+// Namespaced even then. A bare key id here and a "key:"-prefixed one in the
+// restore path would be two different counters for the same key, so a budget
+// would check against an empty one and survive every restart — which is how
+// this was wrong when it was first written.
+func counterFor(o store.LimitOwner, keyID string) string {
+	if k := o.CounterKey(); k != "" {
+		return k
+	}
+	return keyCounter(keyID)
+}
+
+// keyCounter is a key's own counter, in the same namespace as every other.
+func or(v, fallback string) string {
+	if v == "" {
+		return fallback
+	}
+	return v
+}
+
+func keyCounter(keyID string) string {
+	if keyID == "" {
+		return ""
+	}
+	return store.LimitOwner{Scope: store.ScopeKey, ID: keyID}.CounterKey()
+}
+
+// AnythingPriced reports whether a budget can ever be spent against.
+//
+// On the accountant rather than making callers reach for the price table: what
+// they are asking is whether the budgets they are about to display mean
+// anything, and the accountant is what would enforce them.
+func (a *Accountant) AnythingPriced() bool { return a.pricing.AnythingPriced() }
+
+// Headroom is the tightest of a principal's request and token limits: how
+// much is left of each, and when the minute it is counted over rolls on.
+type Headroom struct {
+	RequestLimit, RequestsLeft int
+	TokenLimit, TokensLeft     int
+	Reset                      time.Duration
+}
+
+// Headroom reports what a principal has left, after whatever Check just
+// charged. Of several levels the one with least left is the one that will
+// refuse first, so that is the one reported. A limit of 0 means none is set.
+func (a *Accountant) Headroom(p auth.Principal) Headroom {
+	var h Headroom
+	firstReq, firstTok := true, true
+	for _, l := range levels(p.Limits, p.KeyID) {
+		if l.RPM != nil && *l.RPM > 0 {
+			left := max(0, *l.RPM-int(a.requests.Rate(l.Counter)))
+			if firstReq || left < h.RequestsLeft {
+				h.RequestLimit, h.RequestsLeft, firstReq = *l.RPM, left, false
+			}
+		}
+		if l.TPM != nil && *l.TPM > 0 {
+			left := max(0, *l.TPM-int(a.tokens.Rate(l.Counter)))
+			if firstTok || left < h.TokensLeft {
+				h.TokenLimit, h.TokensLeft, firstTok = *l.TPM, left, false
+			}
+		}
+	}
+	// The windows slide, so room comes back continuously; the next minute
+	// boundary is when a whole window's worth has moved on, and is the figure
+	// a client pacing itself can act on.
+	now := time.Now()
+	h.Reset = now.Truncate(windowLen).Add(windowLen).Sub(now)
+	return h
 }

internal/accounting/accounting_test.go

@@ -288,3 +288,373 @@ func must(t *testing.T, err error) {
 		t.Fatal(err)
 	}
 }
+
+// teamMember is a key whose budget and rate limit were set on its team, which
+// is the arrangement an enterprise actually asks for: a group gets a quota and
+// the people in it share it.
+func teamMember(keyID, teamID string, tune func(*store.Effective)) auth.Principal {
+	e := store.Effective{KeyID: keyID, ProjectID: "proj", TeamID: teamID}
+	owner := store.LimitOwner{Scope: store.ScopeTeam, ID: teamID}
+	e.BudgetOwner, e.RPMOwner, e.TPMOwner = owner, owner, owner
+	if tune != nil {
+		tune(&e)
+	}
+	return auth.Principal{KeyID: keyID, ProjectID: "proj", Limits: e}
+}
+
+func TestATeamBudgetIsOnePoolRatherThanOnePerMember(t *testing.T) {
+	a := New(pricing(t), nil, slog.New(slog.DiscardHandler))
+	budget := func(e *store.Effective) {
+		e.Budget = i64p(3000)
+		e.BudgetPeriod = store.PeriodMonth
+	}
+	first := teamMember("k1", "t1", budget)
+	second := teamMember("k2", "t1", budget)
+
+	// One member spends the lot.
+	res, err := a.Check(first, "paid", 1000)
+	if err != nil {
+		t.Fatal(err)
+	}
+	a.Record(res, store.Usage{KeyID: "k1", PromptTokens: 1000, CompletionTokens: 1000})
+
+	// The second must now be refused. Counted per key it would have its own
+	// untouched 3000 — a team budget of 3000 with ten keys would permit thirty
+	// thousand, which is the bug this change exists for.
+	if _, err := a.Check(second, "paid", 10); !errors.Is(err, ErrBudgetSpent) {
+		t.Errorf("the team's budget should be shared; the second member got %v", err)
+	}
+}
+
+func TestATeamRateLimitIsSharedToo(t *testing.T) {
+	a := New(pricing(t), nil, slog.New(slog.DiscardHandler))
+	rpm := func(e *store.Effective) { e.RPM = intp(1) }
+
+	if _, err := a.Check(teamMember("k1", "t1", rpm), "free", 1); err != nil {
+		t.Fatal(err)
+	}
+	if _, err := a.Check(teamMember("k2", "t1", rpm), "free", 1); !errors.Is(err, ErrRateLimited) {
+		t.Error("a rate limit set on the team should be shared by its keys")
+	}
+}
+
+func TestTwoTeamsDoNotShareACounter(t *testing.T) {
+	a := New(pricing(t), nil, slog.New(slog.DiscardHandler))
+	rpm := func(e *store.Effective) { e.RPM = intp(1) }
+
+	if _, err := a.Check(teamMember("k1", "t1", rpm), "free", 1); err != nil {
+		t.Fatal(err)
+	}
+	// Pooling is the point; pooling the wrong things is the danger. One team
+	// exhausting another's quota would be worse than the bug being fixed.
+	if _, err := a.Check(teamMember("k2", "t2", rpm), "free", 1); err != nil {
+		t.Errorf("a second team should have its own counter, got %v", err)
+	}
+}
+
+func TestAKeyLimitStaysPerKey(t *testing.T) {
+	a := New(pricing(t), nil, slog.New(slog.DiscardHandler))
+	perKey := func(id string) auth.Principal {
+		return principal(id, func(e *store.Effective) {
+			e.RPM = intp(1)
+			e.RPMOwner = store.LimitOwner{Scope: store.ScopeKey, ID: id}
+		})
+	}
+
+	if _, err := a.Check(perKey("k1"), "free", 1); err != nil {
+		t.Fatal(err)
+	}
+	// The tree still has to be able to say "each of you may do this much".
+	// Sharing a group pool must not become the only thing expressible.
+	if _, err := a.Check(perKey("k2"), "free", 1); err != nil {
+		t.Errorf("a limit owned by the key should not be shared, got %v", err)
+	}
+}
+
+func TestTheDifferentLimitsCanBeOwnedByDifferentScopes(t *testing.T) {
+	a := New(pricing(t), nil, slog.New(slog.DiscardHandler))
+	// A team budget with a per-key rate limit is the ordinary arrangement: the
+	// group's money, the individual's throughput.
+	mk := func(id string) auth.Principal {
+		e := store.Effective{KeyID: id, TeamID: "t1"}
+		e.Budget, e.BudgetPeriod = i64p(3000), store.PeriodMonth
+		e.BudgetOwner = store.LimitOwner{Scope: store.ScopeTeam, ID: "t1"}
+		e.RPM = intp(1)
+		e.RPMOwner = store.LimitOwner{Scope: store.ScopeKey, ID: id}
+		return auth.Principal{KeyID: id, Limits: e}
+	}
+
+	res, err := a.Check(mk("k1"), "paid", 1000)
+	if err != nil {
+		t.Fatal(err)
+	}
+	a.Record(res, store.Usage{KeyID: "k1", PromptTokens: 1000, CompletionTokens: 1000})
+
+	// k2's rate limit is its own and untouched; the budget is the team's and
+	// gone. The budget is checked first, so that is what comes back.
+	if _, err := a.Check(mk("k2"), "paid", 10); !errors.Is(err, ErrBudgetSpent) {
+		t.Errorf("expected the shared budget to bite, got %v", err)
+	}
+}
+
+func TestStatusReportsTheCounterThatWillRefuseYou(t *testing.T) {
+	a := New(pricing(t), nil, slog.New(slog.DiscardHandler))
+	budget := func(e *store.Effective) {
+		e.Budget = i64p(9000)
+		e.BudgetPeriod = store.PeriodMonth
+	}
+	res, err := a.Check(teamMember("k1", "t1", budget), "paid", 1000)
+	if err != nil {
+		t.Fatal(err)
+	}
+	a.Record(res, store.Usage{KeyID: "k1", PromptTokens: 1000, CompletionTokens: 1000})
+
+	// A member who has spent nothing themselves still sees the team's spend,
+	// because that is the number that will stop them. A header disagreeing with
+	// the refusal it precedes is worse than no header.
+	spent, _, _ := a.Status(teamMember("k2", "t1", budget))
+	if spent == 0 {
+		t.Error("a member should see the pool's spend, not their own")
+	}
+}
+
+func TestATeamBudgetSurvivesARestart(t *testing.T) {
+	// The per-key restore could not do this: it rebuilt a counter named after
+	// the key, while the check reads one named after the team. The control
+	// plane came back believing the team had spent nothing — a budget that
+	// resets on every restart, for exactly the scope where it matters most.
+	ctx := context.Background()
+	s, err := store.OpenSQLite(ctx, filepath.Join(t.TempDir(), "team.db"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer s.Close()
+
+	must(t, s.PutOrg(ctx, store.Org{ID: "o", Name: "o"}))
+	must(t, s.PutTeam(ctx, store.Team{ID: "t1", OrgID: "o", Name: "t"}))
+	must(t, s.PutProject(ctx, store.Project{ID: "proj", TeamID: "t1", Name: "p"}))
+	for _, id := range []string{"k1", "k2"} {
+		_, k, _ := store.NewKey(id, "n", "proj")
+		k.CreatedAt = time.Now()
+		must(t, s.CreateKey(ctx, k))
+	}
+
+	// Two members of one team, 2500 each: neither alone exceeds a 4000 budget,
+	// together they do.
+	must(t, s.WriteUsage(ctx, []store.Usage{
+		{At: time.Now(), KeyID: "k1", ProjectID: "proj", Model: "paid",
+			ServedModel: "paid", CostMicros: 2500, Status: 200},
+		{At: time.Now(), KeyID: "k2", ProjectID: "proj", Model: "paid",
+			ServedModel: "paid", CostMicros: 2500, Status: 200},
+	}))
+
+	a := New(pricing(t), nil, slog.New(slog.DiscardHandler))
+	must(t, a.RestoreBudgets(ctx, s, store.PeriodMonth))
+
+	p := teamMember("k1", "t1", func(e *store.Effective) {
+		e.Budget = i64p(4000)
+		e.BudgetPeriod = store.PeriodMonth
+	})
+	if _, err := a.Check(p, "paid", 10); !errors.Is(err, ErrBudgetSpent) {
+		t.Errorf("the team's spend should have been restored and pooled: %v", err)
+	}
+}
+
+// inTree is a key under org/team/project, with whatever each level sets.
+func inTree(keyID string, chain ...store.Limits) auth.Principal {
+	e := store.Effective{KeyID: keyID, Chain: chain}
+	// The collapsed values still exist for headers and the profile page; they
+	// are not what enforcement reads any more.
+	for _, l := range chain {
+		if l.Budget != nil && (e.Budget == nil || *l.Budget < *e.Budget) {
+			e.Budget, e.BudgetPeriod = l.Budget, l.BudgetPeriod
+			e.BudgetOwner = store.LimitOwner{Scope: l.Scope, ID: l.ScopeID}
+		}
+	}
+	return auth.Principal{KeyID: keyID, Limits: e}
+}
+
+func orgBudget(id string, v int64) store.Limits {
+	return store.Limits{Scope: store.ScopeOrg, ScopeID: id,
+		Budget: i64p(v), BudgetPeriod: store.PeriodMonth}
+}
+
+func teamBudget(id string, v int64) store.Limits {
+	return store.Limits{Scope: store.ScopeTeam, ScopeID: id,
+		Budget: i64p(v), BudgetPeriod: store.PeriodMonth}
+}
+
+func TestTheSumOfTeamsCannotExceedTheOrganization(t *testing.T) {
+	a := New(pricing(t), nil, slog.New(slog.DiscardHandler))
+
+	// Each request costs 800. The organization allows 2000 and each of its four
+	// teams allows 2000, so no team comes close to its own ceiling — but the
+	// organization's fills after three. Taking only the most restrictive value,
+	// the organization's number was never looked at and all four went through.
+	//
+	// A budget stops spending once it is reached rather than refusing the
+	// request that would cross it, so the third is served and the fourth is not.
+	spend := func(team string) error {
+		p := inTree(team+"-k", orgBudget("o", 2000), teamBudget(team, 2000))
+		res, err := a.Check(p, "paid", 1000)
+		if err != nil {
+			return err
+		}
+		a.Record(res, store.Usage{KeyID: team + "-k", PromptTokens: 200, CompletionTokens: 200})
+		return nil
+	}
+
+	total := 0
+	for _, team := range []string{"t1", "t2", "t3", "t4"} {
+		if err := spend(team); err == nil {
+			total++
+		}
+	}
+	if total != 3 {
+		t.Errorf("expected the organization to stop the fourth team, %d got through", total)
+	}
+}
+
+func TestATeamStillStopsBeforeItsOrganizationDoes(t *testing.T) {
+	a := New(pricing(t), nil, slog.New(slog.DiscardHandler))
+	p := func() auth.Principal {
+		return inTree("k1", orgBudget("o", 100_000), teamBudget("t1", 800))
+	}
+
+	res, err := a.Check(p(), "paid", 1000)
+	if err != nil {
+		t.Fatal(err)
+	}
+	a.Record(res, store.Usage{KeyID: "k1", PromptTokens: 200, CompletionTokens: 200})
+
+	// The narrower pool binds first, which is what it is for. Enforcing every
+	// level must not stop the tighter one working.
+	if _, err := a.Check(p(), "paid", 10); !errors.Is(err, ErrBudgetSpent) {
+		t.Errorf("the team's own budget should still bite, got %v", err)
+	}
+}
+
+func TestSpendingFillsEveryPoolAboveIt(t *testing.T) {
+	a := New(pricing(t), nil, slog.New(slog.DiscardHandler))
+	p := inTree("k1", orgBudget("o", 5000), teamBudget("t1", 5000))
+
+	res, err := a.Check(p, "paid", 1000)
+	if err != nil {
+		t.Fatal(err)
+	}
+	a.Record(res, store.Usage{KeyID: "k1", PromptTokens: 200, CompletionTokens: 200})
+
+	// Settled onto both, or the organization's pool would only ever fill from
+	// requests that named it directly — which is none of them.
+	if got := a.budgets.Spent("org:o", store.PeriodMonth); got == 0 {
+		t.Error("the organization's pool did not fill")
+	}
+	if got := a.budgets.Spent("team:t1", store.PeriodMonth); got == 0 {
+		t.Error("the team's pool did not fill")
+	}
+}
+
+func TestARefusalHandsBackWhatTheNarrowerLevelsTook(t *testing.T) {
+	a := New(pricing(t), nil, slog.New(slog.DiscardHandler))
+	// The team allows plenty; the organization allows one request a minute.
+	chain := []store.Limits{
+		{Scope: store.ScopeOrg, ScopeID: "o", RPM: intp(1)},
+		{Scope: store.ScopeTeam, ScopeID: "t1", RPM: intp(100)},
+	}
+	if _, err := a.Check(inTree("k1", chain...), "free", 1); err != nil {
+		t.Fatal(err)
+	}
+	before := a.requests.Rate("team:t1")
+
+	if _, err := a.Check(inTree("k2", chain...), "free", 1); !errors.Is(err, ErrRateLimited) {
+		t.Fatalf("the organization should have refused the second, got %v", err)
+	}
+
+	// A rate limiter charges as it checks, so a refusal further along has
+	// already taken from the levels before it. Left there, a team at its
+	// organization's ceiling would quietly eat its own rate limit with requests
+	// that were never served.
+	if after := a.requests.Rate("team:t1"); after != before {
+		t.Errorf("the refused request stayed charged to the team: %v -> %v", before, after)
+	}
+}
+
+func TestAChainWithNoLimitsAnywhereAllowsEverything(t *testing.T) {
+	a := New(pricing(t), nil, slog.New(slog.DiscardHandler))
+	for i := 0; i < 5; i++ {
+		if _, err := a.Check(inTree("k1"), "paid", 1000); err != nil {
+			t.Fatalf("nothing is limited, but request %d was refused: %v", i, err)
+		}
+	}
+}
+
+func TestStatusReportsTheFullestPoolNotTheSmallestNumber(t *testing.T) {
+	a := New(pricing(t), nil, slog.New(slog.DiscardHandler))
+	chain := func(team string) auth.Principal {
+		return inTree(team+"-k", orgBudget("o", 9000), teamBudget(team, 9000))
+	}
+
+	// Another team spends; this one has spent nothing of its own.
+	res, err := a.Check(chain("t1"), "paid", 1000)
+	if err != nil {
+		t.Fatal(err)
+	}
+	a.Record(res, store.Usage{KeyID: "t1-k", PromptTokens: 200, CompletionTokens: 200})
+
+	spent, _, _ := a.Status(chain("t2"))
+	// The organization's pool is the one nearer its ceiling, so it is the one
+	// that will refuse them. Reporting the team's nil would disagree with the
+	// refusal that eventually follows it.
+	if spent == 0 {
+		t.Error("a member should see the fullest pool that bounds them")
+	}
+}
+
+func TestAnEmptyPriceTableMeansNoBudgetCanEverBind(t *testing.T) {
+	// The ordinary case for self-hosted models, and the reason a budget on a
+	// team can read as a limit while binding on nothing.
+	if NewPricing(nil, Price{}).AnythingPriced() {
+		t.Error("nothing priced should say so")
+	}
+	if NewPricing(map[string]Price{"m": {}}, Price{}).AnythingPriced() {
+		t.Error("a model priced at zero is not priced")
+	}
+	if !NewPricing(map[string]Price{"m": {PromptPer1k: 1}}, Price{}).AnythingPriced() {
+		t.Error("one priced model is enough for a budget to move")
+	}
+	if !NewPricing(nil, Price{CompletionPer1k: 1}).AnythingPriced() {
+		t.Error("a fallback price prices everything")
+	}
+	var nilPricing *Pricing
+	if nilPricing.AnythingPriced() {
+		t.Error("no price table at all is not priced")
+	}
+}
+
+func TestHeadroomIsTheTightestLimitAfterThisRequest(t *testing.T) {
+	// A project allowing 100 a minute under a team allowing 3: the team is
+	// what will refuse, so the team is what a client is told about.
+	a := New(pricing(t), nil, slog.New(slog.DiscardHandler))
+	p := principal("k1", func(e *store.Effective) {
+		e.Chain = []store.Limits{
+			{Scope: store.ScopeProject, ScopeID: "proj", RPM: intp(100), TPM: intp(50000)},
+			{Scope: store.ScopeTeam, ScopeID: "team", RPM: intp(3)},
+		}
+	})
+	if _, err := a.Check(p, "m", 10); err != nil {
+		t.Fatal(err)
+	}
+	h := a.Headroom(p)
+	if h.RequestLimit != 3 || h.RequestsLeft != 2 {
+		t.Errorf("requests %d left of %d; expected 2 of the team's 3", h.RequestsLeft, h.RequestLimit)
+	}
+	if h.TokenLimit != 50000 || h.TokensLeft != 50000-10 {
+		t.Errorf("tokens %d left of %d", h.TokensLeft, h.TokenLimit)
+	}
+	if h.Reset <= 0 || h.Reset > time.Minute {
+		t.Errorf("reset %v", h.Reset)
+	}
+	if none := a.Headroom(principal("k2", nil)); none.RequestLimit != 0 || none.TokenLimit != 0 {
+		t.Errorf("a key with no limits advertised some: %+v", none)
+	}
+}

internal/accounting/cost.go

@@ -68,3 +68,30 @@ func (p *Pricing) For(model string) Price {
 	}
 	return p.fallback
 }
+
+// AnythingPriced reports whether any model costs anything.
+//
+// A budget is spend per period, and spend is computed from a price table. With
+// no prices every request costs nothing, so a budget written on a team is a
+// number that can never move — displayed as a limit, reported as zero percent
+// used, and binding on nothing.
+//
+// Self-hosted models have no per-token price, so an empty table is the ordinary
+// case rather than a misconfiguration. What is not ordinary is showing somebody
+// a budget without saying so.
+func (p *Pricing) AnythingPriced() bool {
+	if p == nil {
+		return false
+	}
+	p.mu.RLock()
+	defer p.mu.RUnlock()
+	if !p.fallback.Zero() {
+		return true
+	}
+	for _, price := range p.byModel {
+		if !price.Zero() {
+			return true
+		}
+	}
+	return false
+}

internal/accounting/people_test.go

@@ -0,0 +1,126 @@
+package accounting
+
+import (
+	"context"
+	"errors"
+	"log/slog"
+	"path/filepath"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/auth"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// peopleStore has Acme/ml/training with two members, ayse and deniz.
+func peopleStore(t *testing.T) (*store.DB, string) {
+	t.Helper()
+	ctx := context.Background()
+	s, err := store.OpenSQLite(ctx, filepath.Join(t.TempDir(), "p.db"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	t.Cleanup(func() { _ = s.Close() })
+	org, _ := s.CreateOrg(ctx, "Acme")
+	team, _ := s.CreateTeam(ctx, org, "ml")
+	proj, _ := s.CreateProject(ctx, team, "training")
+	for _, id := range []string{"ayse", "deniz"} {
+		if err := s.CreateUser(ctx, store.User{ID: id, Email: id + "@x", Name: id,
+			PasswordHash: "h", State: store.UserPending, CreatedAt: time.Now()}); err != nil {
+			t.Fatal(err)
+		}
+		if err := s.ApproveUser(ctx, id, proj, "test", time.Now()); err != nil {
+			t.Fatal(err)
+		}
+	}
+	return s, proj
+}
+
+// keyFor makes a key for a person and resolves it the way authentication does.
+func keyFor(t *testing.T, s *store.DB, id, user, project string) auth.Principal {
+	t.Helper()
+	ctx := context.Background()
+	_, k, _ := store.NewKey(id, id, project)
+	if err := s.CreateUserKey(ctx, user, k); err != nil {
+		t.Fatal(err)
+	}
+	e, err := s.Resolve(ctx, id)
+	if err != nil {
+		t.Fatalf("resolving %s: %v", id, err)
+	}
+	return auth.Principal{KeyID: id, UserID: e.UserID, ProjectID: e.ProjectID, Limits: e}
+}
+
+func TestAPersonsLimitIsOnePoolAcrossTheirKeys(t *testing.T) {
+	s, proj := peopleStore(t)
+	if err := s.PutLimits(context.Background(),
+		store.Limits{Scope: store.ScopeUser, ScopeID: "ayse", RPM: intp(2)}); err != nil {
+		t.Fatal(err)
+	}
+	a := New(pricing(t), nil, slog.New(slog.DiscardHandler))
+	personal := keyFor(t, s, "k-personal", "ayse", "")
+	inProject := keyFor(t, s, "k-project", "ayse", proj)
+
+	// Two requests, one on each key, use up the person's two a minute.
+	for _, p := range []auth.Principal{personal, inProject} {
+		if _, err := a.Check(p, "free", 1); err != nil {
+			t.Fatalf("%s: %v", p.KeyID, err)
+		}
+	}
+	if _, err := a.Check(personal, "free", 1); !errors.Is(err, ErrRateLimited) {
+		t.Errorf("a third request across her keys was allowed: %v", err)
+	}
+	// Deniz is somebody else.
+	if _, err := a.Check(keyFor(t, s, "k-deniz", "deniz", proj), "free", 1); err != nil {
+		t.Errorf("another person was limited by hers: %v", err)
+	}
+}
+
+func TestAPerMemberLimitCountsEachMemberApart(t *testing.T) {
+	s, proj := peopleStore(t)
+	if err := s.PutLimits(context.Background(),
+		store.Limits{Scope: store.ScopeProjectMember, ScopeID: proj, RPM: intp(1)}); err != nil {
+		t.Fatal(err)
+	}
+	a := New(pricing(t), nil, slog.New(slog.DiscardHandler))
+	ayse := keyFor(t, s, "k-a", "ayse", proj)
+	deniz := keyFor(t, s, "k-d", "deniz", proj)
+
+	if _, err := a.Check(ayse, "free", 1); err != nil {
+		t.Fatal(err)
+	}
+	if _, err := a.Check(ayse, "free", 1); !errors.Is(err, ErrRateLimited) {
+		t.Errorf("her second request passed a one-a-minute per-member limit: %v", err)
+	}
+	// Her share is hers: Deniz still has his.
+	if _, err := a.Check(deniz, "free", 1); err != nil {
+		t.Errorf("one member's use counted against another: %v", err)
+	}
+	// And outside the project it does not apply at all.
+	if _, err := a.Check(keyFor(t, s, "k-a-own", "ayse", ""), "free", 1); err != nil {
+		t.Errorf("a project's per-member limit held her personal key: %v", err)
+	}
+}
+
+func TestAPersonsBudgetSurvivesARestart(t *testing.T) {
+	s, _ := peopleStore(t)
+	ctx := context.Background()
+	if err := s.PutLimits(ctx, store.Limits{Scope: store.ScopeUser, ScopeID: "ayse",
+		Budget: i64p(1000), BudgetPeriod: store.PeriodMonth}); err != nil {
+		t.Fatal(err)
+	}
+	// Spent before the restart, personally and on the chat page.
+	if err := s.WriteUsage(ctx, []store.Usage{
+		{At: time.Now(), KeyID: "k-old", UserID: "ayse", CostMicros: 600, Model: "paid", ServedModel: "paid"},
+		{At: time.Now(), KeyID: "session:ayse", UserID: "ayse", CostMicros: 500, Model: "paid", ServedModel: "paid"},
+	}); err != nil {
+		t.Fatal(err)
+	}
+	a := New(pricing(t), nil, slog.New(slog.DiscardHandler))
+	if err := a.RestoreBudgets(ctx, s, store.PeriodMonth); err != nil {
+		t.Fatal(err)
+	}
+	if _, err := a.Check(keyFor(t, s, "k-new", "ayse", ""), "paid", 1); !errors.Is(err, ErrBudgetSpent) {
+		t.Errorf("a restart forgot what she had spent: %v", err)
+	}
+}

internal/agent/agent.go

@@ -10,12 +10,14 @@ import (
 	"fmt"
 	"io"
 	"log/slog"
+	"maps"
 	"math/rand/v2"
 	"net"
 	"net/http"
 	"os"
 	"path/filepath"
-	"strings"
+	"runtime"
+	"slices"
 	"sync"
 	"time"
 
@@ -25,181 +27,13 @@ import (
 	"gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1/llmgwv1connect"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/supervisor"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/weights"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/agentsvc"
-	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
 )
 
-// Config configures the agent.
-type Config struct {
-	NodeID string `koanf:"node_id"`
-	// Server is the control plane, and Servers is the same thing when there is
-	// more than one. Both are accepted: every configuration written before
-	// there could be a second control plane names one address, and none of
-	// them should have to change.
-	//
-	// The order is tried in order, and a refusal moves to the next. Only the
-	// leader accepts an agent, so a standby is skipped in the time it takes to
-	// be told so — which is what makes failover a reconnect rather than an
-	// edit to every node's configuration.
-	Server   string            `koanf:"server"`
-	Servers  []string          `koanf:"servers"`
-	Hostname string            `koanf:"hostname"`
-	MgmtAddr string            `koanf:"mgmt_addr"`
-	Labels   map[string]string `koanf:"labels"`
-
-	// MgmtInterface is the ethernet interface NCCL bootstraps over.
-	MgmtInterface string `koanf:"mgmt_interface"`
-	// Fabrics labels this node's HCAs. It is explicit on purpose: OpenSM's
-	// default subnet prefix is the same on every fabric, so nothing can be
-	// inferred from the wire (plan §5.2).
-	Fabrics []inventory.FabricAttachment `koanf:"fabrics"`
-
-	HeartbeatInterval time.Duration `koanf:"heartbeat_interval"`
-	// ReconnectBase is the first reconnect delay; it backs off from there.
-	ReconnectBase time.Duration `koanf:"reconnect_base"`
-	ReconnectMax  time.Duration `koanf:"reconnect_max"`
-
-	// AdoptReadyTimeout bounds how long an adopted engine is given to answer a
-	// health check before the agent stops waiting on it.
-	//
-	// Shorter than a startup timeout on purpose: this engine is already
-	// running, so it is not loading a model, it is either serving or it is not.
-	AdoptReadyTimeout time.Duration `koanf:"adopt_ready_timeout"`
-
-	// TLS authenticates both ends of the control-plane stream. Leaving it unset
-	// keeps the plaintext behaviour, which only belongs on a trusted network.
-	TLS tlsconf.Config `koanf:"tls"`
-
-	// ModelDirs are the directories this node keeps model weights in.
-	//
-	// Empty by default, and empty means the node is not asked to look — which
-	// is not the same as having nothing. Storage differs at every installation,
-	// so llmgw observes it rather than owning it: a Lustre mount, an image
-	// exported over the fabric, local NVMe and a copy made by hand all look
-	// identical from here, and placement only needs to know whether the file
-	// is present.
-	ModelDirs []string `koanf:"model_dirs"`
-
-	// WeightsCache is where a remote source is downloaded to. Defaults to the
-	// first entry of ModelDirs, because a node that says where its models live
-	// has already answered this question.
-	//
-	// Fetching is the simple arrangement: every node keeps its own copy. The
-	// exotic ones need no code — weights on a shared filer or an image exported
-	// over the fabric are simply present, and present means nothing is fetched.
-	WeightsCache string `koanf:"weights_cache"`
-
-	Discovery discovery.Options `koanf:"discovery"`
-	Engines   engines.Config    `koanf:"engines"`
-	Superv    supervisor.Config `koanf:"supervisor"`
-
-	Version string
-}
-
-// DefaultConfig returns sensible defaults.
-func DefaultConfig() Config {
-	return Config{
-		MgmtInterface:     "eth0",
-		HeartbeatInterval: 5 * time.Second,
-		ReconnectBase:     time.Second,
-		ReconnectMax:      30 * time.Second,
-		AdoptReadyTimeout: 60 * time.Second,
-		Superv:            supervisor.DefaultConfig(),
-		// NVML gives NVLink topology, so prefer it when this build has it.
-		Discovery: discovery.Options{PreferNVML: true},
-	}
-}
-
-func (c *Config) normalize() error {
-	d := DefaultConfig()
-	if c.NodeID == "" {
-		return errors.New("node_id is required")
-	}
-	if err := c.TLS.Validate(); err != nil {
-		return err
-	}
-	if err := c.normalizeServerURL(); err != nil {
-		return err
-	}
-	if c.MgmtInterface == "" {
-		c.MgmtInterface = d.MgmtInterface
-	}
-	if c.HeartbeatInterval <= 0 {
-		c.HeartbeatInterval = d.HeartbeatInterval
-	}
-	if c.ReconnectBase <= 0 {
-		c.ReconnectBase = d.ReconnectBase
-	}
-	if c.AdoptReadyTimeout <= 0 {
-		c.AdoptReadyTimeout = d.AdoptReadyTimeout
-	}
-	if c.ReconnectMax < c.ReconnectBase {
-		c.ReconnectMax = max(c.ReconnectBase, d.ReconnectMax)
-	}
-	return nil
-}
-
-// normalizeServerURL makes the configured address a URL connect can use, and
-// refuses the combination that would quietly undo the certificates: a scheme
-// that says plaintext while TLS is configured. Silently connecting anyway would
-// mean a cluster believing it has mutual authentication when it has none.
-func (c *Config) normalizeServerURL() error {
-	// One list, however it was written. Server first when both are given,
-	// because an operator adding a standby to an existing file expects the
-	// one already there to stay preferred.
-	all := make([]string, 0, 1+len(c.Servers))
-	if c.Server != "" {
-		all = append(all, c.Server)
-	}
-	all = append(all, c.Servers...)
-
-	seen := map[string]struct{}{}
-	out := make([]string, 0, len(all))
-	for _, s := range all {
-		fixed, err := normalizeOne(s, c.TLS.Enabled())
-		if err != nil {
-			return err
-		}
-		// A duplicate would be tried twice in a row on failover, which reads
-		// as the rotation not working.
-		if _, dup := seen[fixed]; dup {
-			continue
-		}
-		seen[fixed] = struct{}{}
-		out = append(out, fixed)
-	}
-	if len(out) == 0 {
-		return errors.New("server is required")
-	}
-	c.Servers = out
-	c.Server = out[0]
-	return nil
-}
-
-// normalizeOne applies the scheme rules to a single address.
-func normalizeOne(addr string, secure bool) (string, error) {
-	switch {
-	case strings.HasPrefix(addr, "https://"):
-		if !secure {
-			return "", fmt.Errorf("%s is an https:// address but no tls certificates are configured", addr)
-		}
-	case strings.HasPrefix(addr, "http://"):
-		if secure {
-			return "", fmt.Errorf("tls is configured but %s is an http:// address; use https://", addr)
-		}
-	case strings.Contains(addr, "://"):
-		return "", fmt.Errorf("server %q must be an http:// or https:// address", addr)
-	case secure:
-		return "https://" + addr, nil
-	default:
-		return "http://" + addr, nil
-	}
-	return addr, nil
-}
-
 // Agent runs on a GPU node.
 type Agent struct {
 	cfg  Config
@@ -208,6 +42,37 @@ type Agent struct {
 	reg  *engines.Registry
 	http connectHTTPClient
 
+	// engineOnce probes the configured engines once before registering.
+	//
+	// Once, because probing SGLang imports it in a Python interpreter — seconds
+	// on a cold cache — and a reconnect is not a reason to pay that again.
+	// Later changes are found by the slower re-probe the heartbeat starts.
+	engineOnce sync.Once
+	// configuredVers are the engines the configuration names and that answer.
+	// Guarded by mu once the first probe has run.
+	configuredVers map[string]string
+	// installedVers are the engines this agent installed and saw run.
+	// Guarded by mu.
+	installedVers map[string]string
+
+	// prov installs the engines the control plane asks for. provWants is what
+	// it asked for, nil until it has; provStatus is where each stands and
+	// provSent the fingerprint of what the control plane last heard. All
+	// guarded by mu. provKick starts a pass without waiting for the timer.
+	logRing *logRing
+
+	prov       *provision.Installer
+	provWants  []provision.Want
+	provStatus map[string]provision.Status
+	provSent   string
+	provKick   chan struct{}
+	// enginesSent is the fingerprint of the engines the control plane last
+	// heard about, from registration or a heartbeat. Guarded by mu.
+	enginesSent string
+	// lastEngineProbe and probingEngines pace the re-probe. Guarded by mu.
+	lastEngineProbe time.Time
+	probingEngines  bool
+
 	mu  sync.Mutex
 	inv discovery.Inventory
 	// shardReplica remembers which replica a shard belongs to, so a status
@@ -216,9 +81,35 @@ type Agent struct {
 	fetcher      *weights.Fetcher
 
 	// lastWeights is the fingerprint of the file list most recently sent, so a
-	// heartbeat carries the list only when it changed. Touched only by the
-	// heartbeat goroutine, which is the only thing that reads or writes it.
+	// heartbeat carries the list only when it changed. Guarded by mu: during a
+	// failover the old stream's heartbeat loop can still be running when the
+	// new one's starts.
 	lastWeights string
+	// fabricsSent is what the control plane was last told about fabrics.
+	fabricsSent string
+	// sm caches which subnet manager each InfiniBand port sees, asked once
+	// a minute: a management datagram per port per heartbeat is more than
+	// an answer that changes on a failover is worth.
+	sm      map[string]*discovery.SMInfo
+	smErr   map[string]string
+	smAsked time.Time
+	// nvlAsked is when the GPUs' NVLink fabric state was last read.
+	nvlAsked time.Time
+	// sms is every subnet manager on a fabric, asked through one port per
+	// master, keyed by that port; smsErr says why a port could not ask.
+	sms    map[string][]discovery.SMRecord
+	smsErr map[string]string
+	// load measures the machine between heartbeats.
+	load *discovery.LoadSampler
+	// sensors reads heat and power: CPU and board from hwmon and RAPL, GPUs
+	// from nvidia-smi.
+	sensors *discovery.SensorSampler
+
+	// fetchWatchers are the shards waiting on each source, so progress on one
+	// download reaches every shard that asked for it. A fetch is single-flight
+	// — eight shards starting together produce one download — and all eight
+	// are waiting, so all eight should say so.
+	fetchWatchers map[string][]func(weights.Progress)
 }
 
 // connectHTTPClient is the HTTP client connect uses; an interface so tests can
@@ -235,6 +126,11 @@ func New(cfg Config, httpClient connectHTTPClient, log *slog.Logger) (*Agent, er
 	if log == nil {
 		log = slog.Default()
 	}
+	// Everything the agent logs also goes to a ring the admin page can read,
+	// so seeing what a node did does not mean ssh and a journal only root may
+	// open.
+	ring := newLogRing(agentLogLines)
+	log = slog.New(teeHandler{log.Handler(), slog.NewTextHandler(ring, nil)})
 	if httpClient == nil {
 		httpClient = http.DefaultClient
 	}
@@ -244,12 +140,19 @@ func New(cfg Config, httpClient connectHTTPClient, log *slog.Logger) (*Agent, er
 	if err != nil {
 		return nil, err
 	}
-	return &Agent{
+	a := &Agent{
 		cfg: cfg, log: log, sup: sup,
 		reg:          engines.NewRegistry(cfg.Engines),
 		http:         httpClient,
 		shardReplica: map[string]string{},
-	}, nil
+		provStatus:   map[string]provision.Status{},
+		logRing:      ring,
+		provKick:     make(chan struct{}, 1),
+		load:         discovery.NewLoadSampler(),
+		sensors:      discovery.NewSensorSampler(),
+	}
+	a.prov = &provision.Installer{Root: cfg.EnginesDir, Log: log, OnStatus: a.provisionStatus}
+	return a, nil
 }
 
 // Supervisor exposes the process supervisor, for tests and `ctl logs`.
@@ -268,6 +171,13 @@ func (a *Agent) Run(ctx context.Context) error {
 		a.log.Error("hardware discovery failed", "err", err)
 	}
 	a.recoverOrphans()
+	// Made now rather than by the first download, so a new node reports its
+	// disk from its first heartbeat instead of only once it holds a model.
+	if err := os.MkdirAll(a.cfg.WeightsCache, 0o755); err != nil {
+		a.log.Warn("cannot create the weights directory", "dir", a.cfg.WeightsCache, "err", err)
+	}
+	go a.provisionLoop(ctx)
+	go a.ensureDiags(ctx)
 
 	attempt := 0
 	next := 0
@@ -408,38 +318,6 @@ func (a *Agent) settleAdopted(sh *supervisor.Shard, o supervisor.Orphan) {
 		"shard", o.ShardID, "replica", o.ReplicaID)
 }
 
-// usableFabrics reports the fabrics this node can actually carry traffic on.
-//
-// Loud about what it drops. A fabric silently missing from a registration looks
-// identical to one never configured, and the operator who typed the adapter
-// name is the only person who can fix it — so the reason is logged here, on the
-// machine where both halves are visible, rather than inferred later from a
-// replica that will not converge.
-func (a *Agent) usableFabrics(inv discovery.Inventory) []inventory.FabricAttachment {
-	declared := map[string][]string{}
-	for _, f := range a.cfg.Fabrics {
-		declared[f.Fabric] = f.HCAs
-	}
-
-	var out []inventory.FabricAttachment
-	for _, c := range discovery.CheckFabrics(declared, inv.HCAs) {
-		for _, p := range c.Problems {
-			a.log.Error("a declared fabric adapter cannot carry traffic",
-				"fabric", c.Fabric, "problem", p)
-		}
-		if !c.OK() {
-			a.log.Error("fabric not advertised; no declared adapter on it can carry traffic",
-				"fabric", c.Fabric,
-				"consequence", "no multi-node replica will be placed here")
-			continue
-		}
-		out = append(out, inventory.FabricAttachment{
-			Fabric: c.Fabric, HCAs: c.Usable, RateGbps: c.RateGbps,
-		})
-	}
-	return out
-}
-
 // weights builds the fetcher lazily, so an agent that never starts a shard
 // never needs a cache directory.
 func (a *Agent) weights() *weights.Fetcher {
@@ -456,12 +334,20 @@ func (a *Agent) weights() *weights.Fetcher {
 		CacheDir: cache,
 		Log:      a.log,
 		OnProgress: func(p weights.Progress) {
-			// Logged rather than silent: a replica that is downloading a
-			// hundred gigabytes and one that is wedged look identical from the
-			// control plane, and the difference matters to whoever is waiting.
 			a.log.Info("fetching weights",
 				"source", p.Source, "percent", p.Percent(),
 				"bytes", p.BytesDone, "total", p.BytesTotal)
+			// And upward, to whoever is watching from the control plane. The
+			// log line above is on the node, which is where nobody is looking
+			// when a replica has been starting for ten minutes.
+			a.mu.Lock()
+			watchers := slices.Clone(a.fetchWatchers[p.Source])
+			a.mu.Unlock()
+			for _, w := range watchers {
+				if w != nil {
+					w(p)
+				}
+			}
 		},
 	}
 	return a.fetcher
@@ -492,6 +378,7 @@ func (a *Agent) runningShards() []*llmgwv1.RunningShard {
 			Profile:    spec.Profile.Name,
 			Pool:       spec.Pool,
 			Fabric:     spec.Fabric,
+			ProfileFp:  spec.ProfileFP,
 			LeaderAddr: a.cfg.MgmtAddr,
 			LeaderPort: int32(spec.LeaderPort),
 		})
@@ -517,9 +404,14 @@ func (a *Agent) scan(ctx context.Context) error {
 	opts := a.cfg.Discovery
 	opts.ModelDirs = a.cfg.ModelDirs
 	inv, err := discovery.Discover(ctx, opts, a.log)
+	host := discovery.DiscoverHost()
+	host.MinComputeCapability = discovery.MinComputeCapability(inv.GPUs)
 	a.mu.Lock()
 	a.inv = inv
+	a.prov.Host = host
 	a.mu.Unlock()
+	a.log.Info("this node", "os", host.OSID+" "+host.OSVersion, "arch", host.Arch,
+		"cpu_flags", host.CPUFlags, "nvidia_driver", host.NVIDIADriver)
 	return err
 }
 
@@ -552,7 +444,25 @@ func (a *Agent) session(ctx context.Context, server string) error {
 	// Every registration reports what is already running, not just the first
 	// one: a reconnect may be a control plane restart, and it has to recover
 	// the same way (plan §6.5).
+	// What this node can actually run, so the control plane stops placing
+	// engines that are not here. The registry holds an adapter for every
+	// engine whether or not its binary exists, so without this a node with no
+	// vLLM accepted a vLLM shard and failed it minutes later.
+	reg.EngineVersions = a.engines(parent)
+	reg.EnginesProbed = true
+	reg.InstallableEngines = provision.Installable()
+	reg.Features = []string{FeatureFabricTest}
 	reg.RunningShards = a.runningShards()
+	a.mu.Lock()
+	h := a.prov.Host
+	// A new stream is a control plane that knows nothing of where installs
+	// stand, so the next heartbeat tells it again.
+	a.provSent = ""
+	a.mu.Unlock()
+	reg.Host = &llmgwv1.Host{
+		OsId: h.OSID, OsVersion: h.OSVersion, OsCodename: h.OSCodename,
+		Arch: h.Arch, CpuFlags: h.CPUFlags, NvidiaDriver: h.NVIDIADriver,
+	}
 
 	if err := stream.Send(&llmgwv1.AgentMessage{
 		Msg: &llmgwv1.AgentMessage_Register{Register: reg},
@@ -639,6 +549,34 @@ func (a *Agent) heartbeat() *llmgwv1.AgentMessage {
 	a.mu.Unlock()
 
 	hb := &llmgwv1.Heartbeat{NodeId: a.cfg.NodeID}
+
+	// Read every time; it is one small file. Memory is added to a running VM,
+	// and a node that grew from 7 GB to 32 GB has to be seen to have grown
+	// without anybody restarting anything.
+	hb.MemoryTotalMb = int64(discovery.MemoryTotalMB())
+	hb.CpuCount = int32(runtime.NumCPU())
+
+	a.reprobeEngines()
+	a.mu.Lock()
+	if vers := a.engineReportLocked(); a.configuredVers != nil &&
+		enginesFingerprint(vers) != a.enginesSent {
+		a.enginesSent = enginesFingerprint(vers)
+		hb.EnginesChanged = true
+		hb.EngineVersions = vers
+	}
+	if fp := provisionFingerprint(a.provStatus); fp != a.provSent {
+		a.provSent = fp
+		hb.ProvisionChanged = true
+		for _, e := range slices.Sorted(maps.Keys(a.provStatus)) {
+			st := a.provStatus[e]
+			hb.Provision = append(hb.Provision, &llmgwv1.ProvisionStatus{
+				Engine: st.Engine, Version: st.Version, Variant: st.Variant,
+				State: st.State, Detail: st.Detail,
+			})
+		}
+	}
+	a.mu.Unlock()
+
 	for _, g := range inv.GPUs {
 		hb.GpuMemoryUsedMb = append(hb.GpuMemoryUsedMb, int32(g.MemoryUsedMB))
 	}
@@ -659,8 +597,12 @@ func (a *Agent) heartbeat() *llmgwv1.AgentMessage {
 	// rather than "nothing to report" — without it, the first heartbeat after
 	// a model was deleted would be indistinguishable from silence.
 	weights := discovery.ScanWeights(a.cfg.ModelDirs)
-	if fp := weightsFingerprint(weights); fp != a.lastWeights {
-		a.lastWeights = fp
+	fp := weightsFingerprint(weights)
+	a.mu.Lock()
+	changed := fp != a.lastWeights
+	a.lastWeights = fp
+	a.mu.Unlock()
+	if changed {
 		hb.WeightsChanged = true
 		for _, w := range weights {
 			hb.Weights = append(hb.Weights, &llmgwv1.LocalWeights{
@@ -668,9 +610,51 @@ func (a *Agent) heartbeat() *llmgwv1.AgentMessage {
 			})
 		}
 	}
+	hb.Load = a.sampleLoad()
+	a.sampleIB(hb)
+	a.sampleNVLink(hb)
 	return &llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_Heartbeat{Heartbeat: hb}}
 }
 
+// sampleLoad measures the machine and each running engine since the last
+// heartbeat. Nil on the first, which has nothing to compare with.
+func (a *Agent) sampleLoad() *llmgwv1.NodeLoad {
+	if a.load == nil {
+		return nil
+	}
+	pids := map[string]int{}
+	for _, st := range a.sup.Statuses() {
+		if st.PID > 0 {
+			pids[st.ID] = st.PID
+		}
+	}
+	l, ok := a.load.Sample(pids)
+	if !ok {
+		return nil
+	}
+	out := &llmgwv1.NodeLoad{
+		CpuPercent: l.CPUPercent, CpuCount: int32(l.CPUCount),
+		Load1: l.Load1, Load5: l.Load5, Load15: l.Load15,
+		MemoryUsedBytes: l.MemUsed, MemoryAvailableBytes: l.MemAvail,
+	}
+	for _, sh := range l.Shards {
+		out.Shards = append(out.Shards, &llmgwv1.ShardLoad{
+			ShardId: sh.ID, CpuCores: sh.CPUCores, MemoryRssBytes: sh.RSSBytes,
+		})
+	}
+	if a.sensors != nil {
+		se := a.sensors.Sample(context.Background())
+		out.CpuTempC, out.CpuPowerW, out.InletTempC = se.CPUTempC, se.CPUPowerW, se.InletTempC
+		for _, g := range se.GPUs {
+			out.Gpus = append(out.Gpus, &llmgwv1.GPUSensor{
+				Index: int32(g.Index), TempC: g.TempC, MemoryTempC: g.MemoryTempC,
+				PowerW: g.PowerW, PowerLimitW: g.PowerLimitW, Throttle: g.Throttle,
+			})
+		}
+	}
+	return out
+}
+
 // weightsFingerprint is enough to notice a model appearing, disappearing or
 // finishing its download.
 //
@@ -751,6 +735,47 @@ func (a *Agent) handle(ctx context.Context, msg *llmgwv1.ControlMessage, out cha
 		}
 	case *llmgwv1.ControlMessage_RemoveWeights:
 		a.removeWeights(m.RemoveWeights.GetPaths())
+	case *llmgwv1.ControlMessage_Provision:
+		a.setProvision(m.Provision)
+	case *llmgwv1.ControlMessage_Upgrade:
+		go a.upgrade(ctx, m.Upgrade, out)
+	case *llmgwv1.ControlMessage_Logs:
+		reply := a.logsReply(m.Logs)
+		// Off the receive loop: a full send queue must not stop commands being
+		// read.
+		go func() {
+			select {
+			case out <- &llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_Logs{Logs: reply}}:
+			case <-ctx.Done():
+			}
+		}()
+	case *llmgwv1.ControlMessage_FabricScan:
+		// A scan takes seconds to minutes: off the receive loop entirely.
+		go func(req *llmgwv1.FabricScanRequest) {
+			start := time.Now()
+			reply := &llmgwv1.FabricScanReply{RequestId: req.GetRequestId()}
+			text, err := scanFabric(ctx, req.GetHca(), int(req.GetPort()))
+			if err != nil {
+				reply.Error = err.Error()
+			} else {
+				reply.Output = text
+			}
+			reply.TookMs = time.Since(start).Milliseconds()
+			select {
+			case out <- &llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_FabricScan{FabricScan: reply}}:
+			case <-ctx.Done():
+			}
+		}(m.FabricScan)
+	case *llmgwv1.ControlMessage_FabricTest:
+		// Seconds of traffic, and a package install the first time: off the
+		// receive loop too.
+		go func(req *llmgwv1.FabricTestRequest) {
+			reply := a.fabricTest(ctx, req)
+			select {
+			case out <- &llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_FabricTest{FabricTest: reply}}:
+			case <-ctx.Done():
+			}
+		}(m.FabricTest)
 	case *llmgwv1.ControlMessage_Ack:
 		if !m.Ack.GetOk() {
 			a.log.Warn("control plane rejected a message",
@@ -763,6 +788,42 @@ func (a *Agent) handle(ctx context.Context, msg *llmgwv1.ControlMessage, out cha
 //
 // It runs in its own goroutine because model loading takes minutes, and the
 // stream has to keep carrying heartbeats and other commands meanwhile.
+// watchFetch registers interest in a source's download progress and returns
+// the function that stops watching.
+//
+// Keyed by source rather than by shard because a fetch is single-flight: eight
+// shards starting together produce one download, and every one of them is
+// waiting on it.
+func (a *Agent) watchFetch(source string, fn func(weights.Progress)) (stop func()) {
+	a.mu.Lock()
+	if a.fetchWatchers == nil {
+		a.fetchWatchers = map[string][]func(weights.Progress){}
+	}
+	a.fetchWatchers[source] = append(a.fetchWatchers[source], fn)
+	i := len(a.fetchWatchers[source]) - 1
+	a.mu.Unlock()
+
+	var once sync.Once
+	return func() {
+		once.Do(func() {
+			a.mu.Lock()
+			defer a.mu.Unlock()
+			w := a.fetchWatchers[source]
+			if i < len(w) {
+				// Cleared rather than removed, so the indexes other watchers
+				// hold stay valid. The slice is dropped when the last one goes.
+				w[i] = nil
+			}
+			for _, f := range w {
+				if f != nil {
+					return
+				}
+			}
+			delete(a.fetchWatchers, source)
+		})
+	}
+}
+
 func (a *Agent) startShard(ctx context.Context, m *llmgwv1.StartShard, out chan<- *llmgwv1.AgentMessage) {
 	shardID, replicaID := m.GetShardId(), m.GetReplicaId()
 
@@ -770,18 +831,21 @@ func (a *Agent) startShard(ctx context.Context, m *llmgwv1.StartShard, out chan<
 	a.shardReplica[shardID] = replicaID
 	a.mu.Unlock()
 
-	report := func(st supervisor.State, errMsg string, tail []string) {
-		msg := &llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_Shard{Shard: &llmgwv1.ShardStatus{
-			NodeId: a.cfg.NodeID, ReplicaId: replicaID, ShardId: shardID,
-			State: agentsvc.StateToWire(st), Error: errMsg, LastOutput: tail,
-		}}}
+	send := func(s *llmgwv1.ShardStatus) {
+		s.NodeId, s.ReplicaId, s.ShardId = a.cfg.NodeID, replicaID, shardID
 		select {
-		case out <- msg:
+		case out <- &llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_Shard{Shard: s}}:
 		case <-ctx.Done():
 		}
 	}
+	report := func(st supervisor.State, errMsg string, tail []string) {
+		send(&llmgwv1.ShardStatus{
+			State: agentsvc.StateToWire(st), Error: errMsg, LastOutput: tail,
+		})
+	}
 
 	spec, timeout := agentsvc.ShardSpecFromStart(m)
+	a.localizeSpec(&spec)
 
 	eng, err := a.reg.Get(spec.Profile.Engine)
 	if err != nil {
@@ -808,12 +872,36 @@ func (a *Agent) startShard(ctx context.Context, m *llmgwv1.StartShard, out chan<
 	// arrives minutes into a load, in its own words, on a node nobody is
 	// watching — and because eight shards starting together would otherwise
 	// each fetch the same file.
+	//
+	// Progress is reported upward while it runs, so the control plane can tell
+	// a download from a hang.
+	stop := a.watchFetch(spec.Profile.ModelPath, func(p weights.Progress) {
+		send(&llmgwv1.ShardStatus{
+			State: agentsvc.StateToWire(supervisor.Starting),
+			Fetch: &llmgwv1.FetchProgress{
+				Source: p.Source, BytesDone: p.BytesDone, BytesTotal: p.BytesTotal,
+			},
+		})
+	})
 	if path, err := a.weights().Ensure(ctx, spec.Profile.ModelPath); err != nil {
+		stop()
 		report(supervisor.Failed, err.Error(), nil)
 		return
 	} else {
+		stop()
 		spec.Profile.ModelPath = path
 	}
+	// Adapters are fetched like the model, and before the engine starts: it
+	// loads them at start, and one missing is a replica that answers for the
+	// base model under the adapter's name.
+	for i, ad := range spec.Profile.Adapters {
+		path, err := a.weights().Ensure(ctx, ad.Path)
+		if err != nil {
+			report(supervisor.Failed, fmt.Sprintf("adapter %s: %v", ad.Name, err), nil)
+			return
+		}
+		spec.Profile.Adapters[i].Path = path
+	}
 
 	cmds, err := eng.Commands(spec)
 	if err != nil {
@@ -952,3 +1040,54 @@ func Client(cfg tlsconf.Config, timeout time.Duration) (*http.Client, error) {
 	}
 	return &http.Client{Transport: &http2.Transport{TLSClientConfig: tc}}, nil
 }
+
+// localizeSpec fills in what only this node knows.
+//
+// The interface NCCL bootstraps over is this node's, not the cluster's: the
+// control plane sends one name for every node, and a node whose management
+// network is on ens3 rather than eth0 would have NCCL bind nothing. And on
+// RoCE, which GID entry routes is a property of these adapters.
+func (a *Agent) localizeSpec(spec *engines.ShardSpec) {
+	if a.cfg.MgmtInterface != "" {
+		spec.MgmtInterface = a.cfg.MgmtInterface
+	}
+	if len(spec.HCAs) == 0 {
+		return
+	}
+	a.mu.Lock()
+	hcas := a.inv.HCAs
+	a.mu.Unlock()
+	want := map[string]bool{}
+	for _, h := range spec.HCAs {
+		want[h] = true
+	}
+	for _, h := range hcas {
+		if !want[h.Name] {
+			continue
+		}
+		for _, p := range h.Ports {
+			if p.Active() && p.RoCEv2GID >= 0 {
+				idx := p.RoCEv2GID
+				spec.GIDIndex = &idx
+				return
+			}
+		}
+	}
+}
+
+// ResolveManagement is the address and interface this node's agent would use,
+// found the way normalize finds them, for a tool that reports them.
+func ResolveManagement(c Config) (addr, iface string) {
+	addr, iface = c.MgmtAddr, c.MgmtInterface
+	server := c.Server
+	if server == "" && len(c.Servers) > 0 {
+		server = c.Servers[0]
+	}
+	if addr == "" {
+		addr = addrToward(server)
+	}
+	if iface == "" {
+		iface = interfaceOf(addr)
+	}
+	return addr, iface
+}

internal/agent/agent_test.go

@@ -4,6 +4,7 @@ import (
 	"context"
 	"crypto/tls"
 	"errors"
+	"fmt"
 	"log/slog"
 	"net"
 	"net/http"
@@ -22,6 +23,8 @@ import (
 	"golang.org/x/net/http2"
 	"golang.org/x/net/http2/h2c"
 
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/supervisor"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/agentsvc"
@@ -38,6 +41,7 @@ type statusRecorder struct {
 	calls   []adoptCall
 	adopted []reconciler.AdoptedShard
 	unclaim []string // shard ids to report back as unclaimed
+	fetches []fetchReport
 }
 
 type status struct {
@@ -55,6 +59,25 @@ func (r *statusRecorder) OnShardStatus(replicaID, shardID string, st supervisor.
 	}
 }
 
+// fetches records download progress, so a test can assert that a fetch is
+// visible from the control plane rather than only in the node's log.
+func (r *statusRecorder) OnFetchProgress(replicaID, shardID, source string, done, total int64) {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	r.fetches = append(r.fetches, fetchReport{replicaID, shardID, source, done, total})
+}
+
+func (r *statusRecorder) fetchReports() []fetchReport {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	return append([]fetchReport(nil), r.fetches...)
+}
+
+type fetchReport struct {
+	replicaID, shardID, source string
+	done, total                int64
+}
+
 func (r *statusRecorder) Adopt(nodeID string, shards []reconciler.AdoptedShard) []reconciler.AdoptedShard {
 	r.mu.Lock()
 	defer r.mu.Unlock()
@@ -377,11 +400,16 @@ func TestHeartbeatKeepsNodeAlive(t *testing.T) {
 	n, _ := h.inv.Get("n1")
 	first := n.LastSeen
 
-	time.Sleep(200 * time.Millisecond)
-	n, _ = h.inv.Get("n1")
-	if !n.LastSeen.After(first) {
-		t.Error("heartbeats are not refreshing LastSeen; the node would be declared lost")
+	// Waited for rather than slept through: four heartbeat intervals was not
+	// always enough on a loaded runner for one to have landed.
+	deadline := time.Now().Add(3 * time.Second)
+	for time.Now().Before(deadline) {
+		if n, _ = h.inv.Get("n1"); n.LastSeen.After(first) {
+			return
+		}
+		time.Sleep(20 * time.Millisecond)
 	}
+	t.Error("heartbeats are not refreshing LastSeen; the node would be declared lost")
 }
 
 func TestStartAndStopShardEndToEnd(t *testing.T) {
@@ -736,3 +764,157 @@ func TestAFabricRateIsTheSlowestActiveLink(t *testing.T) {
 		}
 	}
 }
+
+func TestAPortGoingDownIsToldToTheControlPlaneOnce(t *testing.T) {
+	// Checked only at registration, a port going down went unnoticed until
+	// the agent restarted, and the node went on being offered multi-node
+	// work that would hang on its first collective.
+	root := ibTreeWithActivePorts(t, "mlx5_0")
+	h := start(t, func(c *Config) {
+		c.Discovery.IBRoot = root
+		c.Fabrics = []inventory.FabricAttachment{{Fabric: "ib-a", HCAs: []string{"mlx5_0"}}}
+	})
+	if n, _ := h.inv.Get("n1"); !n.HasFabric("ib-a") {
+		t.Fatalf("the fabric should be advertised at registration: %+v", n.Fabrics)
+	}
+
+	hb := h.ag.heartbeat().GetHeartbeat()
+	if hb.GetFabricsChanged() || len(hb.GetIbPorts()) != 1 || hb.GetIbPorts()[0].GetState() != "ACTIVE" {
+		t.Fatalf("a quiet heartbeat should carry the port and no change: %+v", hb)
+	}
+
+	if err := os.WriteFile(filepath.Join(root, "mlx5_0", "ports", "1", "state"), []byte("1: DOWN\n"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	hb = h.ag.heartbeat().GetHeartbeat()
+	if !hb.GetFabricsChanged() || len(hb.GetFabrics()) != 0 {
+		t.Errorf("the port going down should withdraw the fabric: changed=%v fabrics=%v",
+			hb.GetFabricsChanged(), hb.GetFabrics())
+	}
+	if hb = h.ag.heartbeat().GetHeartbeat(); hb.GetFabricsChanged() {
+		t.Error("the change should be sent once, not on every heartbeat")
+	}
+}
+
+func TestAShardUsesThisNodesInterfaceAndRoCEIndex(t *testing.T) {
+	// The control plane sends one interface name for every node; a node whose
+	// management network is on another had NCCL bind nothing. And which RoCE
+	// GID entry routes is a property of this node's adapters.
+	a := &Agent{cfg: Config{MgmtInterface: "ens3"}}
+	a.inv.HCAs = []discovery.HCA{{Name: "mlx5_0", Ports: []discovery.Port{
+		{Num: 1, State: "ACTIVE", LinkLayer: discovery.LinkEthernet, RoCEv2GID: 3}}}}
+	spec := engines.ShardSpec{MgmtInterface: "eth0", HCAs: []string{"mlx5_0"}}
+	a.localizeSpec(&spec)
+	if spec.MgmtInterface != "ens3" {
+		t.Errorf("interface = %q", spec.MgmtInterface)
+	}
+	if spec.GIDIndex == nil || *spec.GIDIndex != 3 {
+		t.Errorf("GID index = %v", spec.GIDIndex)
+	}
+
+	ib := engines.ShardSpec{HCAs: []string{"mlx5_1"}}
+	a.inv.HCAs = append(a.inv.HCAs, discovery.HCA{Name: "mlx5_1", Ports: []discovery.Port{
+		{Num: 1, State: "ACTIVE", LinkLayer: discovery.LinkInfiniBand, RoCEv2GID: -1}}})
+	a.localizeSpec(&ib)
+	if ib.GIDIndex != nil {
+		t.Error("an InfiniBand shard was given a GID index")
+	}
+}
+
+func TestTheManagementInterfaceIsFoundNotAssumed(t *testing.T) {
+	// It was defaulted to eth0 before it could be looked for, so it never
+	// was: a node on ens3 or bond0 had NCCL bootstrap over nothing.
+	cfg := DefaultConfig()
+	cfg.Server = "http://127.0.0.1:7443"
+	cfg.NodeID = "n1"
+	cfg.MgmtAddr = "127.0.0.1"
+	if err := cfg.normalize(); err != nil {
+		t.Fatal(err)
+	}
+	if cfg.MgmtInterface == "" || cfg.MgmtInterface == "eth0" {
+		t.Errorf("interface = %q, want the loopback interface holding 127.0.0.1", cfg.MgmtInterface)
+	}
+}
+
+func TestEachFabricsSubnetManagersAreAskedOnce(t *testing.T) {
+	// Two adapters on one fabric see one master. Its managers are listed
+	// through one of them — the same list twice is a query too many — and
+	// every port says which master it sees.
+	oldSM, oldSMs := querySM, querySMs
+	t.Cleanup(func() { querySM, querySMs = oldSM, oldSMs })
+	querySM = func(string, int) (*discovery.SMInfo, error) {
+		return &discovery.SMInfo{GUID: "0x01", LID: 1, State: "SMINFO_MASTER"}, nil
+	}
+	lists := 0
+	querySMs = func(string, int) ([]discovery.SMRecord, error) {
+		lists++
+		return []discovery.SMRecord{{GUID: "0x01", State: "master"}, {GUID: "0x02", State: "standby"}}, nil
+	}
+	h := start(t, func(c *Config) {
+		c.Discovery.IBRoot = ibTreeWithActivePorts(t, "mlx5_0", "mlx5_1")
+		c.Fabrics = []inventory.FabricAttachment{{Fabric: "ib-a", HCAs: []string{"mlx5_0", "mlx5_1"}}}
+	})
+	hb := h.ag.heartbeat().GetHeartbeat()
+	withList := 0
+	for _, p := range hb.GetIbPorts() {
+		if p.GetSmGuid() != "0x01" {
+			t.Errorf("%s sees master %q", p.GetHca(), p.GetSmGuid())
+		}
+		if len(p.GetSubnetManagers()) > 0 {
+			withList++
+		}
+	}
+	if lists != 1 || withList != 1 {
+		t.Errorf("listed %d times, carried on %d ports; want once, on one", lists, withList)
+	}
+}
+
+func TestAFabricScanIsAskedForAndAnswered(t *testing.T) {
+	// One node per fabric maps it; the control plane asks and parses.
+	old := scanFabric
+	t.Cleanup(func() { scanFabric = old })
+	var asked string
+	scanFabric = func(_ context.Context, hca string, port int) (string, error) {
+		asked = fmt.Sprintf("%s/%d", hca, port)
+		return "Switch\t36 \"S-00000000000000a1\"\t\t# \"MF0;leaf-a:QM8700/U1\" enhanced port 0 lid 1 lmc 0\n", nil
+	}
+	h := start(t)
+	r, err := h.svc.ScanFabric(t.Context(), "n1", "mlx5_0", 1)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if asked != "mlx5_0/1" || !strings.Contains(r.GetOutput(), "leaf-a") || r.GetError() != "" {
+		t.Errorf("asked %q, got %+v", asked, r)
+	}
+}
+
+func TestABandwidthTestIsRunOnTheAdapterAskedFor(t *testing.T) {
+	oldRun, oldHave := runWriteBW, haveWriteBW
+	t.Cleanup(func() { runWriteBW, haveWriteBW = oldRun, oldHave })
+	haveWriteBW = func() bool { return true }
+	var got discovery.WriteBW
+	runWriteBW = func(_ context.Context, w discovery.WriteBW) (float64, string, error) {
+		got = w
+		return 388.2, " #bytes  #iterations  BW peak[Gb/sec]  BW average[Gb/sec]\n", nil
+	}
+	h := start(t, func(c *Config) { c.Discovery.IBRoot = ibTreeWithActivePorts(t, "mlx5_0") })
+
+	r, err := h.svc.FabricTest(t.Context(), "n1", &llmgwv1.FabricTestRequest{
+		Role: "client", Hca: "mlx5_0", Port: 1, Peer: "10.0.0.2", TcpPort: 18515, Seconds: 3})
+	if err != nil {
+		t.Fatal(err)
+	}
+	if r.GetGbps() != 388.2 || r.GetError() != "" {
+		t.Errorf("reply %+v", r)
+	}
+	if got.HCA != "mlx5_0" || got.Peer != "10.0.0.2" || got.TCPPort != 18515 || got.GIDIndex != -1 {
+		t.Errorf("ran %+v", got)
+	}
+
+	// An adapter the node does not have is said so, without running anything.
+	got = discovery.WriteBW{}
+	r, _ = h.svc.FabricTest(t.Context(), "n1", &llmgwv1.FabricTestRequest{Role: "server", Hca: "mlx5_9", Port: 1, Seconds: 1})
+	if !strings.Contains(r.GetError(), "no port mlx5_9/1") || got.HCA != "" {
+		t.Errorf("reply %+v, ran %+v", r, got)
+	}
+}

internal/agent/config.go

@@ -0,0 +1,286 @@
+package agent
+
+import (
+	"errors"
+	"fmt"
+	"net"
+	"net/url"
+	"os"
+	"slices"
+	"strings"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/supervisor"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
+)
+
+// Config configures the agent.
+type Config struct {
+	// NodeID defaults to the hostname. It must equal the common name of the
+	// agent's certificate, so set it only where the two differ.
+	NodeID string `koanf:"node_id"`
+	// Server is the control plane, and Servers is the same thing when there is
+	// more than one. Both are accepted: every configuration written before
+	// there could be a second control plane names one address, and none of
+	// them should have to change.
+	//
+	// The order is tried in order, and a refusal moves to the next. Only the
+	// leader accepts an agent, so a standby is skipped in the time it takes to
+	// be told so — which is what makes failover a reconnect rather than an
+	// edit to every node's configuration.
+	Server  string   `koanf:"server"`
+	Servers []string `koanf:"servers"`
+	// Hostname defaults to the machine's.
+	Hostname string `koanf:"hostname"`
+	// MgmtAddr is the address engines are reached on. It defaults to this
+	// machine's address on the route to the control plane, which is the one
+	// the control plane and the router can reach.
+	MgmtAddr string `koanf:"mgmt_addr"`
+	// Labels are free-form facts about a node for a profile's node_selector.
+	// Nothing needs them: memory, GPUs, engines and disk are measured, and a
+	// label restating one of those goes stale the day the machine changes.
+	Labels map[string]string `koanf:"labels"`
+
+	// MgmtInterface is the ethernet interface NCCL bootstraps over. It
+	// defaults to the interface holding MgmtAddr, then eth0.
+	MgmtInterface string `koanf:"mgmt_interface"`
+	// Fabrics labels this node's HCAs. It is explicit on purpose: OpenSM's
+	// default subnet prefix is the same on every fabric, so nothing can be
+	// inferred from the wire (plan §5.2).
+	Fabrics []inventory.FabricAttachment `koanf:"fabrics"`
+
+	HeartbeatInterval time.Duration `koanf:"heartbeat_interval"`
+	// ReconnectBase is the first reconnect delay; it backs off from there.
+	ReconnectBase time.Duration `koanf:"reconnect_base"`
+	ReconnectMax  time.Duration `koanf:"reconnect_max"`
+
+	// AdoptReadyTimeout bounds how long an adopted engine is given to answer a
+	// health check before the agent stops waiting on it.
+	//
+	// Shorter than a startup timeout on purpose: this engine is already
+	// running, so it is not loading a model, it is either serving or it is not.
+	AdoptReadyTimeout time.Duration `koanf:"adopt_ready_timeout"`
+
+	// TLS authenticates both ends of the control-plane stream. Leaving it unset
+	// keeps the plaintext behaviour, which only belongs on a trusted network.
+	TLS tlsconf.Config `koanf:"tls"`
+
+	// ModelDirs are the directories this node keeps model weights in, beyond
+	// WeightsCache, which is always looked in. Storage differs at every installation,
+	// so llmgw observes it rather than owning it: a Lustre mount, an image
+	// exported over the fabric, local NVMe and a copy made by hand all look
+	// identical from here, and placement only needs to know whether the file
+	// is present.
+	ModelDirs []string `koanf:"model_dirs"`
+
+	// EnginesDir is where the agent installs engines, one directory per
+	// build. Defaults to /opt/llmgw/engines.
+	EnginesDir string `koanf:"engines_dir"`
+
+	// WeightsCache is where a remote source is downloaded to. Defaults to the
+	// first entry of ModelDirs, because a node that says where its models live
+	// has already answered this question, and otherwise to
+	// /var/lib/llmgw-weights.
+	//
+	// Fetching is the simple arrangement: every node keeps its own copy. The
+	// exotic ones need no code — weights on a shared filer or an image exported
+	// over the fabric are simply present, and present means nothing is fetched.
+	WeightsCache string `koanf:"weights_cache"`
+
+	Discovery discovery.Options `koanf:"discovery"`
+	Engines   engines.Config    `koanf:"engines"`
+	Superv    supervisor.Config `koanf:"supervisor"`
+
+	Version string
+}
+
+// DefaultConfig returns sensible defaults.
+func DefaultConfig() Config {
+	// No MgmtInterface: normalize finds the one holding the management
+	// address, and falls back to eth0 only when it cannot. Defaulted here, it
+	// was never found — every node said eth0 unless told otherwise, and a node
+	// on ens3 or bond0 had NCCL bootstrap over an interface it does not have.
+	return Config{
+		HeartbeatInterval: 5 * time.Second,
+		ReconnectBase:     time.Second,
+		ReconnectMax:      30 * time.Second,
+		AdoptReadyTimeout: 60 * time.Second,
+		Superv:            supervisor.DefaultConfig(),
+		// NVML gives NVLink topology, so prefer it when this build has it.
+		Discovery: discovery.Options{PreferNVML: true},
+	}
+}
+
+func (c *Config) normalize() error {
+	d := DefaultConfig()
+	if err := c.TLS.Validate(); err != nil {
+		return err
+	}
+	if err := c.normalizeServerURL(); err != nil {
+		return err
+	}
+	// Everything below is found rather than written. A file that has to say
+	// which machine it is on, and at what address, is a file that differs on
+	// every node — and a copy taken from one node to the next registers the
+	// second as the first.
+	host, _ := os.Hostname()
+	if c.NodeID == "" {
+		c.NodeID = host
+	}
+	if c.NodeID == "" {
+		return errors.New("node_id is not set and the hostname could not be read")
+	}
+	if c.Hostname == "" {
+		c.Hostname = host
+	}
+	if c.MgmtAddr == "" {
+		c.MgmtAddr = addrToward(c.Server)
+	}
+	if c.MgmtInterface == "" {
+		c.MgmtInterface = interfaceOf(c.MgmtAddr)
+	}
+	if c.MgmtInterface == "" {
+		c.MgmtInterface = "eth0"
+	}
+	// The directory downloads land in is always one that is looked in. When
+	// they were two settings, a node could fill its disk with fetched weights
+	// that placement never saw — which is what every node here was doing.
+	if c.WeightsCache == "" && len(c.ModelDirs) > 0 {
+		c.WeightsCache = c.ModelDirs[0]
+	}
+	if c.WeightsCache == "" {
+		c.WeightsCache = DefaultWeightsCache
+	}
+	if !slices.Contains(c.ModelDirs, c.WeightsCache) {
+		c.ModelDirs = append(c.ModelDirs, c.WeightsCache)
+	}
+	if c.HeartbeatInterval <= 0 {
+		c.HeartbeatInterval = d.HeartbeatInterval
+	}
+	if c.ReconnectBase <= 0 {
+		c.ReconnectBase = d.ReconnectBase
+	}
+	if c.AdoptReadyTimeout <= 0 {
+		c.AdoptReadyTimeout = d.AdoptReadyTimeout
+	}
+	if c.ReconnectMax < c.ReconnectBase {
+		c.ReconnectMax = max(c.ReconnectBase, d.ReconnectMax)
+	}
+	return nil
+}
+
+// normalizeServerURL makes the configured address a URL connect can use, and
+// refuses the combination that would quietly undo the certificates: a scheme
+// that says plaintext while TLS is configured. Silently connecting anyway would
+// mean a cluster believing it has mutual authentication when it has none.
+func (c *Config) normalizeServerURL() error {
+	// One list, however it was written. Server first when both are given,
+	// because an operator adding a standby to an existing file expects the
+	// one already there to stay preferred.
+	all := make([]string, 0, 1+len(c.Servers))
+	if c.Server != "" {
+		all = append(all, c.Server)
+	}
+	all = append(all, c.Servers...)
+
+	seen := map[string]struct{}{}
+	out := make([]string, 0, len(all))
+	for _, s := range all {
+		fixed, err := normalizeOne(s, c.TLS.Enabled())
+		if err != nil {
+			return err
+		}
+		// A duplicate would be tried twice in a row on failover, which reads
+		// as the rotation not working.
+		if _, dup := seen[fixed]; dup {
+			continue
+		}
+		seen[fixed] = struct{}{}
+		out = append(out, fixed)
+	}
+	if len(out) == 0 {
+		return errors.New("server is required")
+	}
+	c.Servers = out
+	c.Server = out[0]
+	return nil
+}
+
+// DefaultWeightsCache is where fetched weights go when nothing says otherwise.
+const DefaultWeightsCache = "/var/lib/llmgw-weights"
+
+// addrToward is this machine's address on the route to the control plane.
+//
+// The address engines are reached on is whichever one the control plane can
+// reach, and the kernel already knows which that is: a UDP "connection" picks
+// the source address for the route without sending a packet. Asking the
+// interfaces instead would have to guess among docker bridges, VPNs and
+// link-local addresses.
+func addrToward(server string) string {
+	u, err := url.Parse(server)
+	if err != nil || u.Host == "" {
+		return ""
+	}
+	host := u.Hostname()
+	port := u.Port()
+	if port == "" {
+		port = "443"
+	}
+	conn, err := net.Dial("udp", net.JoinHostPort(host, port))
+	if err != nil {
+		return ""
+	}
+	defer conn.Close()
+	if a, ok := conn.LocalAddr().(*net.UDPAddr); ok {
+		return a.IP.String()
+	}
+	return ""
+}
+
+// interfaceOf names the interface that holds an address, or "" when none does.
+func interfaceOf(addr string) string {
+	ip := net.ParseIP(addr)
+	if ip == nil {
+		return ""
+	}
+	ifaces, err := net.Interfaces()
+	if err != nil {
+		return ""
+	}
+	for _, ifc := range ifaces {
+		addrs, err := ifc.Addrs()
+		if err != nil {
+			continue
+		}
+		for _, a := range addrs {
+			if n, ok := a.(*net.IPNet); ok && n.IP.Equal(ip) {
+				return ifc.Name
+			}
+		}
+	}
+	return ""
+}
+
+// normalizeOne applies the scheme rules to a single address.
+func normalizeOne(addr string, secure bool) (string, error) {
+	switch {
+	case strings.HasPrefix(addr, "https://"):
+		if !secure {
+			return "", fmt.Errorf("%s is an https:// address but no tls certificates are configured", addr)
+		}
+	case strings.HasPrefix(addr, "http://"):
+		if secure {
+			return "", fmt.Errorf("tls is configured but %s is an http:// address; use https://", addr)
+		}
+	case strings.Contains(addr, "://"):
+		return "", fmt.Errorf("server %q must be an http:// or https:// address", addr)
+	case secure:
+		return "https://" + addr, nil
+	default:
+		return "http://" + addr, nil
+	}
+	return addr, nil
+}

internal/agent/discovery/discover.go

@@ -21,10 +21,12 @@ type Options struct {
 	NvidiaSMI string `koanf:"nvidia_smi"`
 	// IBRoot is the sysfs InfiniBand directory.
 	IBRoot string `koanf:"ib_root"`
-	// PreferNVML uses the NVML backend when this binary has one. It is the only
-	// source of NVLink topology.
+	// PreferNVML uses the NVML backend when this binary has one. NVLink
+	// topology is read from nvidia-smi topo -m on every build; NVML, where it
+	// is used, is the more exact source of the peers.
 	PreferNVML bool `koanf:"prefer_nvml"`
-	// SkipGPUs omits GPU discovery, for nodes that have none.
+	// SkipGPUs omits GPU discovery. A node without nvidia-smi is already
+	// found to have none, so this is for one whose driver is broken.
 	SkipGPUs bool `koanf:"skip_gpus"`
 	// ModelDirs are scanned for weights. Empty means do not look.
 	ModelDirs []string `koanf:"-"`
@@ -63,8 +65,7 @@ func Discover(ctx context.Context, opts Options, log *slog.Logger) (Inventory, e
 				inv.GPUs, inv.Source = gpus, "nvml"
 				break
 			}
-			log.Warn("NVML discovery failed, falling back to nvidia-smi; "+
-				"NVLink topology will be unavailable", "err", err)
+			log.Warn("NVML discovery failed, falling back to nvidia-smi", "err", err)
 			fallthrough
 		default:
 			gpus, err := DiscoverGPUsSMI(ctx, opts.NvidiaSMI)
@@ -73,6 +74,26 @@ func Discover(ctx context.Context, opts Options, log *slog.Logger) (Inventory, e
 			}
 			inv.GPUs, inv.Source = gpus, "nvidia-smi"
 		}
+		// NVLink and which adapter sits nearest each card, for either path:
+		// NVML knows NVLink but not the adapters, nvidia-smi's query knows
+		// neither. A failure costs the topology and nothing else — the cards
+		// are still cards.
+		if len(inv.GPUs) > 0 {
+			topo, err := DiscoverTopology(ctx, opts.NvidiaSMI)
+			if err != nil {
+				log.Warn("could not read the GPU topology; tensor-parallel groups "+
+					"will be chosen without NVLink", "err", err)
+			} else {
+				topo.Apply(inv.GPUs)
+			}
+			// Multi-node NVLink: which domain each card has joined.
+			if fab, err := DiscoverGPUFabric(ctx, opts.NvidiaSMI); err != nil {
+				log.Warn("could not read the GPUs' NVLink fabric state", "err", err)
+			} else {
+				ApplyFabric(inv.GPUs, fab)
+			}
+			inv.IMEX = DiscoverIMEX(ctx)
+		}
 	}
 
 	hcas, err := DiscoverIB(opts.IBRoot)

internal/agent/discovery/fabrics.go

@@ -10,6 +10,8 @@ import (
 // has.
 type FabricCheck struct {
 	Fabric string
+	// Declared are the adapters the configuration names for it.
+	Declared []string
 	// Usable are the declared adapters that exist and have an active port.
 	Usable []string
 	// RateGbps is the slowest active port among them, because a collective
@@ -49,7 +51,7 @@ func CheckFabrics(declared map[string][]string, discovered []HCA) []FabricCheck
 
 	out := make([]FabricCheck, 0, len(names))
 	for _, fabric := range names {
-		c := FabricCheck{Fabric: fabric}
+		c := FabricCheck{Fabric: fabric, Declared: declared[fabric]}
 		for _, want := range declared[fabric] {
 			hca, found := byName[want]
 			if !found {

internal/agent/discovery/gpu.go

@@ -38,6 +38,13 @@ type GPU struct {
 	// NVLinkPeers are the indices of GPUs reachable over NVLink. Within a node
 	// a TP group should stay inside one NVLink domain.
 	NVLinkPeers []int `json:"nvlink_peers,omitempty"`
+	// NICPaths is how far each network adapter is from this GPU, by HCA name:
+	// PIX, PXB, PHB, NODE or SYS, as nvidia-smi topo reports it. What decides
+	// which adapter a GPU's share of a multi-node job should go through.
+	NICPaths map[string]string `json:"nic_paths,omitempty"`
+	// Fabric is the GPU's multi-node NVLink membership, nil where the
+	// driver reports none.
+	Fabric *GPUFabric `json:"fabric,omitempty"`
 	// Health carries anything that should keep the GPU out of placement.
 	Health GPUHealth `json:"health"`
 }
@@ -117,6 +124,9 @@ type Inventory struct {
 	// Storage is the disk behind those weights, one entry per filesystem.
 	// Empty means the same thing it means for Weights: not asked.
 	Storage []Storage `json:"storage,omitempty"`
+	// IMEX is the service a multi-node NVLink domain needs, read where the
+	// node has GPUs.
+	IMEX IMEX `json:"imex"`
 }
 
 // Classes returns the distinct GPU classes present, sorted.

internal/agent/discovery/gpu_fabric.go

@@ -0,0 +1,160 @@
+package discovery
+
+import (
+	"context"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"strings"
+	"time"
+)
+
+// GPUFabric is a GPU's place in a multi-node NVLink domain, as the driver
+// reports it: the NVL72 rack, or the NVLink Switch system, it belongs to.
+type GPUFabric struct {
+	// State is Completed once the fabric manager has joined the GPU to its
+	// domain; In Progress, Not Started or Not Supported otherwise.
+	State string `json:"state"`
+	// Status is Success, or the reason the join failed.
+	Status string `json:"status,omitempty"`
+	// ClusterUUID and CliqueID name the domain: GPUs sharing both can reach
+	// each other over NVLink whatever node they are in.
+	ClusterUUID string `json:"cluster_uuid,omitempty"`
+	CliqueID    string `json:"clique_id,omitempty"`
+	// Bandwidth is Full, or how the domain's NVLink is degraded.
+	Bandwidth string `json:"bandwidth,omitempty"`
+}
+
+// Domain is the NVLink domain the GPU is joined to, empty when it is in none
+// that crosses nodes: a card with no fabric, one not joined, or one whose
+// cluster is all zeros, which is what a single machine's NVSwitch reports.
+func (f *GPUFabric) Domain() string {
+	if f == nil || !strings.EqualFold(f.State, "Completed") {
+		return ""
+	}
+	u := strings.ToLower(strings.TrimSpace(f.ClusterUUID))
+	if u == "" || strings.Trim(u, "0-") == "" {
+		return ""
+	}
+	return u + "/" + f.CliqueID
+}
+
+// ParseGPUFabric reads the Fabric sections of `nvidia-smi -q`, by the PCI
+// address each GPU's section starts with.
+func ParseGPUFabric(out string) map[string]GPUFabric {
+	res := map[string]GPUFabric{}
+	var bus string
+	fabricIndent := -1
+	var cur *GPUFabric
+	flush := func() {
+		if bus != "" && cur != nil && cur.State != "" {
+			res[bus] = *cur
+		}
+		cur, fabricIndent = nil, -1
+	}
+	for _, line := range strings.Split(out, "\n") {
+		trim := strings.TrimSpace(line)
+		indent := len(line) - len(strings.TrimLeft(line, " \t"))
+		if strings.HasPrefix(line, "GPU ") && indent == 0 {
+			flush()
+			bus = strings.ToLower(strings.TrimSpace(strings.TrimPrefix(line, "GPU ")))
+			continue
+		}
+		if trim == "Fabric" {
+			flush()
+			cur, fabricIndent = &GPUFabric{}, indent
+			continue
+		}
+		if cur == nil {
+			continue
+		}
+		if trim != "" && indent <= fabricIndent {
+			flush()
+			continue
+		}
+		k, v, ok := strings.Cut(trim, ":")
+		if !ok {
+			continue
+		}
+		k, v = strings.TrimSpace(k), strings.TrimSpace(v)
+		if strings.EqualFold(v, "N/A") {
+			v = ""
+		}
+		switch k {
+		case "State":
+			cur.State = v
+		case "Status":
+			cur.Status = v
+		case "ClusterUUID", "Cluster UUID":
+			cur.ClusterUUID = v
+		case "CliqueId", "Clique ID":
+			cur.CliqueID = v
+		case "Bandwidth":
+			cur.Bandwidth = v
+		}
+	}
+	flush()
+	return res
+}
+
+// DiscoverGPUFabric runs `nvidia-smi -q` for the Fabric sections. Nothing
+// is reported on a driver or a card without them.
+func DiscoverGPUFabric(ctx context.Context, binary string) (map[string]GPUFabric, error) {
+	if binary == "" {
+		if _, err := exec.LookPath("nvidia-smi"); err != nil {
+			return nil, nil
+		}
+		binary = "nvidia-smi"
+	}
+	ctx, cancel := context.WithTimeout(ctx, 20*time.Second)
+	defer cancel()
+	out, err := exec.CommandContext(ctx, binary, "-q").Output()
+	if err != nil {
+		return nil, err
+	}
+	return ParseGPUFabric(string(out)), nil
+}
+
+// ApplyFabric puts each GPU's fabric on it, matched by PCI address.
+func ApplyFabric(gpus []GPU, fab map[string]GPUFabric) {
+	for i := range gpus {
+		if f, ok := fab[strings.ToLower(gpus[i].PCIBusID)]; ok {
+			f := f
+			gpus[i].Fabric = &f
+		}
+	}
+}
+
+// IMEX is the service that lets GPUs in different nodes of one NVLink
+// domain share memory: without it, multi-node NVLink carries nothing.
+type IMEX struct {
+	// State is systemd's word for nvidia-imex: active, inactive, or empty
+	// where it is not installed.
+	State string `json:"state,omitempty"`
+	// Channels is how many IMEX channels exist; a process needs one.
+	Channels int `json:"channels"`
+}
+
+// DiscoverIMEX reads the IMEX service and its channels.
+func DiscoverIMEX(ctx context.Context) IMEX {
+	var m IMEX
+	if entries, err := os.ReadDir("/dev/nvidia-caps-imex-channels"); err == nil {
+		m.Channels = len(entries)
+	}
+	if _, err := exec.LookPath("systemctl"); err == nil {
+		ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
+		defer cancel()
+		out, _ := exec.CommandContext(ctx, "systemctl", "is-active", "nvidia-imex").Output()
+		m.State = strings.TrimSpace(string(out))
+		if m.State == "inactive" {
+			// "inactive" is also what systemd says of a unit that does not
+			// exist; the binary says which.
+			if _, err := exec.LookPath("nvidia-imex"); err != nil {
+				if _, err := os.Stat(filepath.Join("/usr/bin", "nvidia-imex")); err != nil {
+					m.State = ""
+				}
+			}
+		}
+	}
+	return m
+}

internal/agent/discovery/gpu_fabric_test.go

@@ -0,0 +1,77 @@
+package discovery
+
+import "testing"
+
+// smiQ is an excerpt of `nvidia-smi -q` on a GB200 tray: two GPUs joined to
+// one NVL72 domain, then a PCIe card that has no fabric at all.
+const smiQ = `
+==============NVSMI LOG==============
+
+Timestamp                                 : Mon Sep 29 10:00:00 2026
+Driver Version                            : 570.86.15
+Attached GPUs                             : 3
+GPU 00000008:01:00.0
+    Product Name                          : NVIDIA GB200
+    Minor Number                          : 0
+    Fabric
+        State                             : Completed
+        Status                            : Success
+        CliqueId                          : 32766
+        ClusterUUID                       : 76B1AD76-7c24-4b6b-9f6c-2a1e0d8c11aa
+        Health
+            Bandwidth                     : Full
+    Processes                             : None
+
+GPU 00000009:01:00.0
+    Product Name                          : NVIDIA GB200
+    Fabric
+        State                             : In Progress
+        Status                            : N/A
+        CliqueId                          : N/A
+        ClusterUUID                       : N/A
+    Processes                             : None
+
+GPU 0000000A:01:00.0
+    Product Name                          : NVIDIA L40S
+    Fabric
+        State                             : N/A
+        Status                            : N/A
+`
+
+func TestAGPUsNVLinkDomainIsRead(t *testing.T) {
+	fab := ParseGPUFabric(smiQ)
+	a, ok := fab["00000008:01:00.0"]
+	if !ok {
+		t.Fatalf("fabric = %+v", fab)
+	}
+	if a.State != "Completed" || a.CliqueID != "32766" || a.Bandwidth != "Full" {
+		t.Errorf("GB200 = %+v", a)
+	}
+	if got := a.Domain(); got != "76b1ad76-7c24-4b6b-9f6c-2a1e0d8c11aa/32766" {
+		t.Errorf("domain = %q", got)
+	}
+	// Still joining: no domain yet, and said as such.
+	b := fab["00000009:01:00.0"]
+	if b.State != "In Progress" || b.Domain() != "" {
+		t.Errorf("joining GPU = %+v", b)
+	}
+	// A card with no fabric reports nothing rather than an empty one.
+	if _, ok := fab["0000000a:01:00.0"]; ok {
+		t.Error("a card without a fabric was given one")
+	}
+
+	gpus := []GPU{{Index: 0, PCIBusID: "00000008:01:00.0"}, {Index: 1, PCIBusID: "0000000A:01:00.0"}}
+	ApplyFabric(gpus, fab)
+	if gpus[0].Fabric == nil || gpus[1].Fabric != nil {
+		t.Errorf("applied = %+v %+v", gpus[0].Fabric, gpus[1].Fabric)
+	}
+}
+
+func TestASingleMachinesNVSwitchIsNoDomain(t *testing.T) {
+	// An eight-GPU HGX reports its fabric joined, with a cluster of zeros: its
+	// NVLink ends at the chassis.
+	f := &GPUFabric{State: "Completed", ClusterUUID: "00000000-0000-0000-0000-000000000000", CliqueID: "0"}
+	if f.Domain() != "" {
+		t.Errorf("a single machine read as a domain: %q", f.Domain())
+	}
+}

internal/agent/discovery/gpu_smi.go

@@ -40,10 +40,17 @@ const mandatorySMIFields = 10
 //
 // This is the cgo-free path (plan §3.2): the NVML bindings need cgo, which
 // would force the whole binary — server included — to be built with a C
-// toolchain. nvidia-smi gives everything except NVLink topology, which the
-// NVML build adds.
+// toolchain. The query gives everything except topology, which Discover
+// reads from `nvidia-smi topo -m` afterwards.
 func DiscoverGPUsSMI(ctx context.Context, binary string) ([]GPU, error) {
 	if binary == "" {
+		// Not configured and not installed is a node without an NVIDIA card,
+		// not a failed discovery. Reported as an error it also cut discovery
+		// short before InfiniBand was looked at, on every CPU node that had
+		// not been told by hand to skip GPUs.
+		if _, err := exec.LookPath("nvidia-smi"); err != nil {
+			return nil, nil
+		}
 		binary = "nvidia-smi"
 	}
 	out, err := exec.CommandContext(ctx, binary,

internal/agent/discovery/host.go

@@ -0,0 +1,174 @@
+package discovery
+
+import (
+	"bufio"
+	"os"
+	"runtime"
+	"strconv"
+	"strings"
+)
+
+// Host is what a node is, as far as installing software on it is concerned.
+type Host struct {
+	// OSID, OSVersion and OSCodename come from /etc/os-release: "ubuntu",
+	// "24.04", "noble".
+	OSID       string `json:"os_id,omitempty"`
+	OSVersion  string `json:"os_version,omitempty"`
+	OSCodename string `json:"os_codename,omitempty"`
+	// OSLike is ID_LIKE: the families this system says it follows, "debian"
+	// for Ubuntu, "rhel centos fedora" for Rocky. It is what decides the
+	// package manager, which the name alone does not for every derivative.
+	OSLike []string `json:"os_like,omitempty"`
+	// Arch is Debian's spelling, "amd64" or "arm64", because that is what
+	// package names and release archives are keyed by.
+	Arch string `json:"arch,omitempty"`
+	// CPUFlags are the vector extensions inference cares about, of those this
+	// processor has.
+	CPUFlags []string `json:"cpu_flags,omitempty"`
+	// NVIDIADriver is the loaded driver's version, empty when there is none.
+	NVIDIADriver string `json:"nvidia_driver,omitempty"`
+	// MinComputeCapability is the oldest GPU's, 0 when there are none or it
+	// is not known. Not read here but from the GPU inventory, by the agent.
+	MinComputeCapability float64 `json:"min_compute_capability,omitempty"`
+}
+
+// Family is the packaging family: "deb", "rpm", or "" when neither is known.
+func (h Host) Family() string {
+	for _, id := range append([]string{h.OSID}, h.OSLike...) {
+		switch id {
+		case "debian", "ubuntu":
+			return "deb"
+		case "rhel", "fedora", "centos", "rocky", "almalinux", "suse", "opensuse":
+			return "rpm"
+		}
+	}
+	return ""
+}
+
+// HasCPUFlag reports whether the processor has an extension.
+func (h Host) HasCPUFlag(f string) bool {
+	for _, have := range h.CPUFlags {
+		if have == f {
+			return true
+		}
+	}
+	return false
+}
+
+// interestingFlags are the extensions that decide which build of an engine
+// runs well. A VM on QEMU's default CPU model has none of them — and still
+// runs, five times slower, with nothing saying why.
+var interestingFlags = []string{"avx", "avx2", "fma", "f16c", "avx512f", "avx512_bf16", "amx_tile", "asimd", "sve"}
+
+// DiscoverHost reads what the machine is.
+func DiscoverHost() Host {
+	h := hostFromOSRelease("/etc/os-release")
+	h.Arch = debianArch(runtime.GOARCH)
+	h.CPUFlags = cpuFlagsFrom("/proc/cpuinfo")
+	h.NVIDIADriver = nvidiaDriverFrom("/proc/driver/nvidia/version")
+	return h
+}
+
+func hostFromOSRelease(path string) Host {
+	var h Host
+	f, err := os.Open(path) //nolint:gosec // a fixed system path
+	if err != nil {
+		return h
+	}
+	defer f.Close()
+	sc := bufio.NewScanner(f)
+	for sc.Scan() {
+		k, v, ok := strings.Cut(sc.Text(), "=")
+		if !ok {
+			continue
+		}
+		v = strings.Trim(v, `"'`)
+		switch k {
+		case "ID":
+			h.OSID = v
+		case "VERSION_ID":
+			h.OSVersion = v
+		case "VERSION_CODENAME":
+			h.OSCodename = v
+		case "ID_LIKE":
+			h.OSLike = strings.Fields(v)
+		}
+	}
+	return h
+}
+
+func debianArch(goarch string) string {
+	switch goarch {
+	case "amd64", "arm64":
+		return goarch
+	case "386":
+		return "i386"
+	case "arm":
+		return "armhf"
+	}
+	return goarch
+}
+
+func cpuFlagsFrom(path string) []string {
+	b, err := os.ReadFile(path) //nolint:gosec // a fixed system path
+	if err != nil {
+		return nil
+	}
+	have := map[string]bool{}
+	for _, line := range strings.Split(string(b), "\n") {
+		k, v, ok := strings.Cut(line, ":")
+		if !ok {
+			continue
+		}
+		// "flags" on x86, "Features" on ARM. The first processor is enough:
+		// a machine does not mix instruction sets across its cores.
+		if k = strings.TrimSpace(k); k != "flags" && k != "Features" {
+			continue
+		}
+		for _, f := range strings.Fields(v) {
+			have[f] = true
+		}
+		break
+	}
+	var out []string
+	for _, f := range interestingFlags {
+		if have[f] {
+			out = append(out, f)
+		}
+	}
+	return out
+}
+
+// nvidiaDriverFrom reads the loaded kernel module's version, which exists
+// exactly when a driver is loaded — unlike nvidia-smi, which can be installed
+// on a machine whose module failed to load.
+func nvidiaDriverFrom(path string) string {
+	b, err := os.ReadFile(path) //nolint:gosec // a fixed system path
+	if err != nil {
+		return ""
+	}
+	// "NVRM version: NVIDIA UNIX x86_64 Kernel Module  570.86.15  Thu ..."
+	fields := strings.Fields(string(b))
+	for i, f := range fields {
+		if f == "Module" && i+1 < len(fields) {
+			return fields[i+1]
+		}
+	}
+	return ""
+}
+
+// MinComputeCapability is the oldest GPU's compute capability, 0 when none
+// is known: an engine built for newer GPUs is refused by the oldest.
+func MinComputeCapability(gpus []GPU) float64 {
+	low := 0.0
+	for _, g := range gpus {
+		cc, err := strconv.ParseFloat(strings.TrimSpace(g.ComputeCapability), 64)
+		if err != nil || cc <= 0 {
+			continue
+		}
+		if low == 0 || cc < low {
+			low = cc
+		}
+	}
+	return low
+}

internal/agent/discovery/ib.go

@@ -5,6 +5,7 @@ package discovery
 import (
 	"errors"
 	"fmt"
+	"net"
 	"os"
 	"path/filepath"
 	"slices"
@@ -42,8 +43,66 @@ type Port struct {
 	GIDPrefix string `json:"gid_prefix"`
 	LID       int    `json:"lid"`
 	SMLID     int    `json:"sm_lid"`
+	// Counters are the port's error and traffic counters, by their sysfs
+	// names, as far as the driver has them. The data counters are converted
+	// to bytes; sysfs counts them in four-byte words.
+	Counters map[string]uint64 `json:"counters,omitempty"`
+	// RoCEv2GID is the GID table index NCCL should use on a RoCE port: the
+	// first RoCE v2 entry holding an IPv4 address, else the first RoCE v2
+	// entry. -1 on InfiniBand, and on a RoCE port with no v2 entry. NCCL's
+	// default of 0 is a RoCE v1 link-local entry on most setups, which does
+	// not route, and a job on it hangs.
+	RoCEv2GID int `json:"rocev2_gid"`
+	// PortGUID is the port's own identifier, the interface half of its GID.
+	PortGUID string `json:"port_guid,omitempty"`
+	// SM is the subnet manager this port sees as master, from sminfo. Nil on
+	// RoCE, which has none, and when infiniband-diags is not installed.
+	SM *SMInfo `json:"sm,omitempty"`
+	// Network is, on RoCE, the IP network of the RoCE v2 address: the only
+	// thing that says two RoCE ports are on the same fabric.
+	Network string `json:"network,omitempty"`
 }
 
+// SMInfo is a subnet manager as a port sees it.
+type SMInfo struct {
+	GUID     string `json:"guid"`
+	LID      int    `json:"lid"`
+	Priority int    `json:"priority"`
+	// State is SMINFO_MASTER, SMINFO_STANDBY, ...
+	State string `json:"state"`
+}
+
+// FabricKey is what identifies the physical fabric this port is on: the
+// master subnet manager on InfiniBand, since every port on one fabric sees
+// the same one and ports on separate fabrics cannot; the IP network on RoCE.
+// Empty when neither is known, and for a port that is not up.
+func (p Port) FabricKey() string {
+	if !p.Active() {
+		return ""
+	}
+	switch {
+	case p.LinkLayer == LinkEthernet && p.Network != "":
+		return "roce:" + p.Network
+	case p.SM != nil && p.SM.GUID != "":
+		return "sm:" + p.SM.GUID
+	}
+	return ""
+}
+
+// PortCounters are the counters worth watching, from each port's counters/
+// directory. The errors are the ones that say a cable, a transceiver or a
+// switch port is going bad before a job fails on it; the data counters say
+// whether the fabric is carrying anything at all.
+var PortCounters = []string{
+	"symbol_error", "link_error_recovery", "link_downed", "port_rcv_errors",
+	"port_rcv_remote_physical_errors", "port_xmit_discards",
+	"local_link_integrity_errors", "excessive_buffer_overrun_errors",
+	"port_xmit_data", "port_rcv_data",
+}
+
+// dataCounter reports whether a counter counts data in four-byte words.
+func dataCounter(name string) bool { return name == "port_xmit_data" || name == "port_rcv_data" }
+
 // Active reports whether the port can carry traffic.
 func (p Port) Active() bool { return strings.EqualFold(p.State, "ACTIVE") }
 
@@ -56,6 +115,9 @@ type HCA struct {
 	// FirmwareVersion is reported by the driver; worth capturing because
 	// NCCL behaviour differs between firmware revisions.
 	FirmwareVersion string `json:"firmware_version,omitempty"`
+	// NodeGUID identifies the adapter on its fabric: it is the name a
+	// switch's view of the fabric, and every other tool, knows it by.
+	NodeGUID string `json:"node_guid,omitempty"`
 }
 
 // ActivePorts returns the ports that can carry traffic.
@@ -102,6 +164,7 @@ func DiscoverIB(root string) ([]HCA, error) {
 func readHCA(dir, name string) (HCA, error) {
 	h := HCA{Name: name, NUMANode: -1}
 	h.FirmwareVersion = readTrimmed(filepath.Join(dir, "fw_ver"))
+	h.NodeGUID = normalizeGUID(readTrimmed(filepath.Join(dir, "node_guid")))
 
 	// device is a symlink into the PCI tree; its target's base name is the
 	// bus address, and numa_node lives beside it.
@@ -138,6 +201,30 @@ func readPort(dir string, num int) Port {
 		LID:       parseMaybeHex(readTrimmed(filepath.Join(dir, "lid"))),
 		SMLID:     parseMaybeHex(readTrimmed(filepath.Join(dir, "sm_lid"))),
 	}
+	p.RoCEv2GID = -1
+	p.PortGUID = portGUID(readTrimmed(filepath.Join(dir, "gids", "0")))
+	if p.LinkLayer == LinkEthernet {
+		p.RoCEv2GID = rocev2GID(dir)
+		if p.RoCEv2GID >= 0 {
+			idx := strconv.Itoa(p.RoCEv2GID)
+			p.Network = RoCENetwork(readTrimmed(filepath.Join(dir, "gid_attrs", "ndevs", idx)),
+				readTrimmed(filepath.Join(dir, "gids", idx)))
+		}
+	}
+	for _, name := range PortCounters {
+		raw := readTrimmed(filepath.Join(dir, "counters", name))
+		v, err := strconv.ParseUint(raw, 10, 64)
+		if err != nil {
+			continue // not every driver has every counter
+		}
+		if dataCounter(name) {
+			v *= 4
+		}
+		if p.Counters == nil {
+			p.Counters = map[string]uint64{}
+		}
+		p.Counters[name] = v
+	}
 	return p
 }
 
@@ -201,3 +288,91 @@ func readTrimmed(path string) string {
 	}
 	return strings.TrimSpace(string(b))
 }
+
+// rocev2GID finds the GID index a RoCE port's traffic should use.
+func rocev2GID(dir string) int {
+	first := -1
+	for i := 0; i < 64; i++ {
+		typ := readTrimmed(filepath.Join(dir, "gid_attrs", "types", strconv.Itoa(i)))
+		if !strings.EqualFold(typ, "RoCE v2") {
+			continue
+		}
+		gid := strings.ToLower(readTrimmed(filepath.Join(dir, "gids", strconv.Itoa(i))))
+		if strings.HasPrefix(gid, "0000:0000:0000:0000:0000:ffff:") {
+			return i
+		}
+		if first < 0 {
+			first = i
+		}
+	}
+	return first
+}
+
+// normalizeGUID writes a GUID one way: sysfs says "0c42:a103:0045:bd6c",
+// sminfo and ibnetdiscover say "0x0c42a1030045bd6c".
+func normalizeGUID(g string) string {
+	g = strings.ToLower(strings.TrimSpace(g))
+	g = strings.TrimPrefix(g, "0x")
+	g = strings.ReplaceAll(g, ":", "")
+	if g == "" || strings.Trim(g, "0") == "" {
+		return ""
+	}
+	for len(g) < 16 {
+		g = "0" + g
+	}
+	return "0x" + g
+}
+
+// NormalizeGUID is normalizeGUID for other packages.
+func NormalizeGUID(g string) string { return normalizeGUID(g) }
+
+// portGUID is the interface identifier of a GID: its last four groups.
+func portGUID(gid string) string {
+	parts := strings.Split(gid, ":")
+	if len(parts) != 8 {
+		return ""
+	}
+	return normalizeGUID(strings.Join(parts[4:], ""))
+}
+
+// RoCENetwork is the IP network an IPv4-mapped RoCE GID's address sits in,
+// read from the interface that carries it. Replaced in tests.
+var RoCENetwork = func(ndev, gid string) string {
+	ip := gidIPv4(gid)
+	if ip == nil || ndev == "" {
+		return ""
+	}
+	ifc, err := net.InterfaceByName(ndev)
+	if err != nil {
+		return ""
+	}
+	addrs, _ := ifc.Addrs()
+	for _, a := range addrs {
+		if n, ok := a.(*net.IPNet); ok && n.IP.Equal(ip) {
+			return (&net.IPNet{IP: ip.Mask(n.Mask), Mask: n.Mask}).String()
+		}
+	}
+	return ""
+}
+
+// gidIPv4 is the IPv4 address in an IPv4-mapped GID, nil otherwise.
+func gidIPv4(gid string) net.IP {
+	g := strings.ToLower(gid)
+	const mapped = "0000:0000:0000:0000:0000:ffff:"
+	if !strings.HasPrefix(g, mapped) {
+		return nil
+	}
+	hex := strings.ReplaceAll(strings.TrimPrefix(g, mapped), ":", "")
+	if len(hex) != 8 {
+		return nil
+	}
+	b := make(net.IP, 4)
+	for i := range 4 {
+		v, err := strconv.ParseUint(hex[i*2:i*2+2], 16, 8)
+		if err != nil {
+			return nil
+		}
+		b[i] = byte(v)
+	}
+	return b
+}

internal/agent/discovery/ib_test.go

@@ -31,6 +31,9 @@ func fakeSysfs(t *testing.T) string {
 	write("ib/mlx5_0/ports/1/gids/0", "FEC0:0000:0000:000A:0002:C903:00A0:1234")
 	write("ib/mlx5_0/ports/1/lid", "0x1a")
 	write("ib/mlx5_0/ports/1/sm_lid", "1")
+	write("ib/mlx5_0/ports/1/counters/symbol_error", "3")
+	write("ib/mlx5_0/ports/1/counters/link_downed", "1")
+	write("ib/mlx5_0/ports/1/counters/port_xmit_data", "1000")
 
 	// mlx5_1: two ports, the second one down.
 	write("ib/mlx5_1/ports/1/state", "4: ACTIVE")
@@ -46,7 +49,13 @@ func fakeSysfs(t *testing.T) string {
 	write("ib/mlx5_2/ports/1/state", "4: ACTIVE")
 	write("ib/mlx5_2/ports/1/rate", "100 Gb/sec (4X EDR)")
 	write("ib/mlx5_2/ports/1/link_layer", "Ethernet")
-	write("ib/mlx5_2/ports/1/gids/0", "0000:0000:0000:0000:0000:ffff:0a00:0001")
+	write("ib/mlx5_2/ports/1/gids/0", "fe80:0000:0000:0000:0000:0000:0000:0001")
+	write("ib/mlx5_2/ports/1/gid_attrs/types/0", "IB/RoCE v1")
+	write("ib/mlx5_2/ports/1/gids/1", "fe80:0000:0000:0000:0000:0000:0000:0001")
+	write("ib/mlx5_2/ports/1/gid_attrs/types/1", "RoCE v2")
+	write("ib/mlx5_2/ports/1/gids/3", "0000:0000:0000:0000:0000:ffff:0a00:0001")
+	write("ib/mlx5_2/ports/1/gid_attrs/types/3", "RoCE v2")
+	write("ib/mlx5_2/ports/1/gid_attrs/ndevs/3", "ens1f0")
 
 	// A PCI device with a NUMA node, linked the way the driver does it.
 	pci := filepath.Join(root, "pci", "0000:0b:00.0")
@@ -214,3 +223,131 @@ func TestParseHelpers(t *testing.T) {
 		}
 	})
 }
+
+func TestAPortsCountersAreRead(t *testing.T) {
+	hcas, err := DiscoverIB(fakeSysfs(t))
+	if err != nil {
+		t.Fatal(err)
+	}
+	c := hcas[0].Ports[0].Counters
+	// Data in bytes: sysfs counts four-byte words.
+	if c["symbol_error"] != 3 || c["link_downed"] != 1 || c["port_xmit_data"] != 4000 {
+		t.Errorf("counters = %v", c)
+	}
+	if _, ok := c["port_rcv_errors"]; ok {
+		t.Error("a counter the driver does not have should be absent, not zero")
+	}
+}
+
+func TestRoCEFindsItsRoutableGID(t *testing.T) {
+	// NCCL's default GID index is a v1 link-local entry on most hosts, which
+	// does not route: the job hangs. The v2 entry holding an IPv4 address is
+	// the one to use, over an earlier v2 link-local one.
+	hcas, err := DiscoverIB(fakeSysfs(t))
+	if err != nil {
+		t.Fatal(err)
+	}
+	for _, h := range hcas {
+		p := h.Ports[0]
+		switch h.Name {
+		case "mlx5_2":
+			if p.RoCEv2GID != 3 {
+				t.Errorf("RoCE GID index = %d, want 3", p.RoCEv2GID)
+			}
+		default:
+			if p.RoCEv2GID != -1 {
+				t.Errorf("%s is InfiniBand and has a GID index %d", h.Name, p.RoCEv2GID)
+			}
+		}
+	}
+}
+
+func TestTheSubnetManagerAPortSeesIsRead(t *testing.T) {
+	sm, err := ParseSMInfo("sminfo: sm lid 1 sm guid 0x98039B03009fcfd6, activity count 5452 priority 15 state 3 SMINFO_MASTER\n")
+	if err != nil {
+		t.Fatal(err)
+	}
+	if sm.GUID != "0x98039b03009fcfd6" || sm.LID != 1 || sm.Priority != 15 || sm.State != "SMINFO_MASTER" {
+		t.Errorf("sm = %+v", sm)
+	}
+	if _, err := ParseSMInfo("ibwarn: [1234] mad_rpc_open_port: can't open UMAD port"); err == nil {
+		t.Error("an error from sminfo should not parse")
+	}
+}
+
+func TestAPortsFabricIsItsSubnetManagerOrItsNetwork(t *testing.T) {
+	// Two ports on one InfiniBand fabric see one master SM, and on separate
+	// fabrics cannot; RoCE has no SM, and is its IP network.
+	hcas, err := DiscoverIB(fakeSysfs(t))
+	if err != nil {
+		t.Fatal(err)
+	}
+	if hcas[0].NodeGUID != "" {
+		t.Errorf("no node_guid file, yet %q", hcas[0].NodeGUID)
+	}
+	err = AttachSM(hcas, func(hca string, port int) (*SMInfo, error) {
+		return &SMInfo{GUID: "0x00000000000000aa", LID: 1, State: "SMINFO_MASTER"}, nil
+	})
+	if err != nil {
+		t.Fatal(err)
+	}
+	if got := hcas[0].Ports[0].FabricKey(); got != "sm:0x00000000000000aa" {
+		t.Errorf("IB key = %q", got)
+	}
+	if got := hcas[1].Ports[1].FabricKey(); got != "" {
+		t.Errorf("a port that is down is on no fabric, got %q", got)
+	}
+	if hcas[2].Ports[0].SM != nil {
+		t.Error("RoCE was asked for a subnet manager")
+	}
+	if got := hcas[0].Ports[0].PortGUID; got != "0x0002c90300a01234" {
+		t.Errorf("port GUID = %q", got)
+	}
+}
+
+func TestARoCEPortsNetworkIsItsFabric(t *testing.T) {
+	old := RoCENetwork
+	t.Cleanup(func() { RoCENetwork = old })
+	RoCENetwork = func(ndev, gid string) string {
+		if ip := gidIPv4(gid); ip != nil && ip.String() == "10.0.0.1" {
+			return "10.0.0.0/24"
+		}
+		return ""
+	}
+	hcas, err := DiscoverIB(fakeSysfs(t))
+	if err != nil {
+		t.Fatal(err)
+	}
+	roce := hcas[2].Ports[0]
+	if roce.Network != "10.0.0.0/24" || roce.FabricKey() != "roce:10.0.0.0/24" {
+		t.Errorf("RoCE network %q key %q", roce.Network, roce.FabricKey())
+	}
+}
+
+// smirDump is `saquery SMIR` on a fabric with a master and a standby, in the
+// layout infiniband-diags prints.
+const smirDump = "SMInfoRecord dump:\n\t\tRID\n\t\tLID...................1\n\t\tSMInfo dump:\n" +
+	"\t\tGUID..................0x248a07030030f0c0\n\t\tSM_Key................0x0000000000000000\n" +
+	"\t\tActCount..............386773\n\t\tPriority..............15\n\t\tSMState...............3\n" +
+	"SMInfoRecord dump:\n\t\tRID\n\t\tLID...................12\n\t\tSMInfo dump:\n" +
+	"\t\tGUID..................0x248a07030030f1d4\n\t\tSM_Key................0x0000000000000000\n" +
+	"\t\tActCount..............12\n\t\tPriority..............14\n\t\tSMState...............2\n"
+
+func TestEverySubnetManagerOnAFabricIsListed(t *testing.T) {
+	recs, err := ParseSMIR(smirDump)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if len(recs) != 2 {
+		t.Fatalf("records = %+v", recs)
+	}
+	if recs[0].GUID != "0x248a07030030f0c0" || recs[0].State != "master" || recs[0].Priority != 15 || recs[0].LID != 1 {
+		t.Errorf("master = %+v", recs[0])
+	}
+	if recs[1].State != "standby" || recs[1].LID != 12 {
+		t.Errorf("standby = %+v", recs[1])
+	}
+	if _, err := ParseSMIR("ibwarn: can't open UMAD port"); err == nil {
+		t.Error("an error should not parse")
+	}
+}

internal/agent/discovery/load.go

@@ -0,0 +1,260 @@
+package discovery
+
+import (
+	"bufio"
+	"bytes"
+	"os"
+	"path/filepath"
+	"strconv"
+	"strings"
+	"sync"
+	"time"
+)
+
+// Load is how busy a node is: the machine as a whole, and each engine's share.
+type Load struct {
+	CPUPercent float64
+	CPUCount   int
+	Load1      float64
+	Load5      float64
+	Load15     float64
+	MemUsed    int64
+	MemAvail   int64
+	Shards     []ShardLoad
+}
+
+// ShardLoad is one engine process group's use.
+type ShardLoad struct {
+	ID       string
+	CPUCores float64
+	RSSBytes int64
+}
+
+// userHZ is the unit /proc counts CPU time in. Fixed at 100 by the kernel's
+// ABI on every architecture a node runs, whatever the internal tick rate.
+const userHZ = 100
+
+// LoadSampler measures load as the difference between two readings of /proc.
+//
+// CPU time is a counter, so a percentage needs two samples and the time
+// between them. The sampler keeps the previous one; the first call has
+// nothing to compare with and reports no load at all, rather than a zero that
+// would read as an idle machine.
+//
+// Linux only, like every node. Elsewhere the files are missing and it reports
+// nothing, as an older agent does.
+type LoadSampler struct {
+	root string
+	now  func() time.Time
+
+	mu      sync.Mutex
+	prevAt  time.Time
+	prevCPU cpuTimes
+	prevPG  map[int]groupUse // process group -> what it has used
+}
+
+// groupUse is a process group's CPU time so far and its resident memory now.
+type groupUse struct {
+	jiffies  uint64
+	rssPages int64
+}
+
+type cpuTimes struct{ total, idle uint64 }
+
+// NewLoadSampler reads the real /proc.
+func NewLoadSampler() *LoadSampler { return newLoadSampler("/proc", time.Now) }
+
+func newLoadSampler(root string, now func() time.Time) *LoadSampler {
+	return &LoadSampler{root: root, now: now}
+}
+
+// Sample reads the machine and the process groups led by the given shards'
+// PIDs, keyed by shard ID. False when there is nothing to report yet.
+func (s *LoadSampler) Sample(shards map[string]int) (Load, bool) {
+	cpu, ok := s.cpu()
+	if !ok {
+		return Load{}, false
+	}
+	groups := s.groups()
+	at := s.now()
+
+	s.mu.Lock()
+	prevAt, prevCPU, prevPG := s.prevAt, s.prevCPU, s.prevPG
+	s.prevAt, s.prevCPU, s.prevPG = at, cpu, groups
+	s.mu.Unlock()
+
+	if prevAt.IsZero() || cpu.total <= prevCPU.total {
+		return Load{}, false
+	}
+	l := Load{CPUCount: s.cpuCount()}
+	dTotal := float64(cpu.total - prevCPU.total)
+	dIdle := float64(cpu.idle - prevCPU.idle)
+	l.CPUPercent = clamp((dTotal-dIdle)/dTotal*100, 0, 100)
+	l.Load1, l.Load5, l.Load15 = s.loadavg()
+	total, avail := s.meminfo()
+	if total > 0 {
+		l.MemUsed, l.MemAvail = total-avail, avail
+	}
+
+	secs := at.Sub(prevAt).Seconds()
+	for id, pid := range shards {
+		if pid <= 0 {
+			continue
+		}
+		now := groups[pid]
+		sl := ShardLoad{ID: id, RSSBytes: now.rssPages * int64(os.Getpagesize())}
+		if before, ok := prevPG[pid]; ok && secs > 0 && now.jiffies >= before.jiffies {
+			sl.CPUCores = float64(now.jiffies-before.jiffies) / userHZ / secs
+		}
+		l.Shards = append(l.Shards, sl)
+	}
+	return l, true
+}
+
+func clamp(v, lo, hi float64) float64 { return max(lo, min(v, hi)) }
+
+// cpu reads the aggregate line of /proc/stat. Idle includes iowait: a core
+// waiting on the disk is not doing work.
+func (s *LoadSampler) cpu() (cpuTimes, bool) {
+	b, err := os.ReadFile(filepath.Join(s.root, "stat"))
+	if err != nil {
+		return cpuTimes{}, false
+	}
+	line, _, _ := bytes.Cut(b, []byte("\n"))
+	f := strings.Fields(string(line))
+	if len(f) < 5 || f[0] != "cpu" {
+		return cpuTimes{}, false
+	}
+	var t cpuTimes
+	for i, v := range f[1:] {
+		// guest and guest_nice are already counted in user and nice.
+		if i >= 8 {
+			break
+		}
+		n, _ := strconv.ParseUint(v, 10, 64)
+		t.total += n
+		if i == 3 || i == 4 {
+			t.idle += n
+		}
+	}
+	return t, true
+}
+
+func (s *LoadSampler) cpuCount() int {
+	b, err := os.ReadFile(filepath.Join(s.root, "stat"))
+	if err != nil {
+		return 0
+	}
+	n := 0
+	for _, line := range strings.Split(string(b), "\n") {
+		if strings.HasPrefix(line, "cpu") && len(line) > 3 && line[3] >= '0' && line[3] <= '9' {
+			n++
+		}
+	}
+	return n
+}
+
+func (s *LoadSampler) loadavg() (l1, l5, l15 float64) {
+	b, err := os.ReadFile(filepath.Join(s.root, "loadavg"))
+	if err != nil {
+		return 0, 0, 0
+	}
+	f := strings.Fields(string(b))
+	if len(f) < 3 {
+		return 0, 0, 0
+	}
+	l1, _ = strconv.ParseFloat(f[0], 64)
+	l5, _ = strconv.ParseFloat(f[1], 64)
+	l15, _ = strconv.ParseFloat(f[2], 64)
+	return l1, l5, l15
+}
+
+// meminfo returns MemTotal and MemAvailable in bytes.
+func (s *LoadSampler) meminfo() (total, avail int64) {
+	f, err := os.Open(filepath.Join(s.root, "meminfo"))
+	if err != nil {
+		return 0, 0
+	}
+	defer f.Close()
+	sc := bufio.NewScanner(f)
+	for sc.Scan() {
+		k, v, ok := strings.Cut(sc.Text(), ":")
+		if !ok {
+			continue
+		}
+		fields := strings.Fields(v)
+		if len(fields) == 0 {
+			continue
+		}
+		kb, _ := strconv.ParseInt(fields[0], 10, 64)
+		switch k {
+		case "MemTotal":
+			total = kb * 1024
+		case "MemAvailable":
+			avail = kb * 1024
+		}
+	}
+	return total, avail
+}
+
+// procStat is what one /proc/<pid>/stat says that load needs.
+type procStat struct {
+	pgrp     int
+	jiffies  uint64
+	rssPages int64
+}
+
+// readProcStat parses /proc/<pid>/stat. The command name is in parentheses
+// and may itself hold spaces and parentheses, so fields are counted from the
+// last closing one.
+func readProcStat(path string) (procStat, bool) {
+	b, err := os.ReadFile(path)
+	if err != nil {
+		return procStat{}, false
+	}
+	i := bytes.LastIndexByte(b, ')')
+	if i < 0 {
+		return procStat{}, false
+	}
+	f := strings.Fields(string(b[i+1:]))
+	// After the name: state(0) ppid(1) pgrp(2) ... utime(11) stime(12) ... rss(21)
+	if len(f) < 22 {
+		return procStat{}, false
+	}
+	var st procStat
+	st.pgrp, _ = strconv.Atoi(f[2])
+	ut, _ := strconv.ParseUint(f[11], 10, 64)
+	stt, _ := strconv.ParseUint(f[12], 10, 64)
+	st.jiffies = ut + stt
+	st.rssPages, _ = strconv.ParseInt(f[21], 10, 64)
+	return st, true
+}
+
+// groups totals CPU time and memory per process group, in one pass over
+// /proc. An engine is started as the leader of its own group, so its workers
+// — vLLM starts several — are counted with it.
+func (s *LoadSampler) groups() map[int]groupUse {
+	out := map[int]groupUse{}
+	s.eachProc(func(st procStat) {
+		g := out[st.pgrp]
+		g.jiffies += st.jiffies
+		g.rssPages += st.rssPages
+		out[st.pgrp] = g
+	})
+	return out
+}
+
+func (s *LoadSampler) eachProc(fn func(procStat)) {
+	entries, err := os.ReadDir(s.root)
+	if err != nil {
+		return
+	}
+	for _, e := range entries {
+		if _, err := strconv.Atoi(e.Name()); err != nil {
+			continue
+		}
+		if st, ok := readProcStat(filepath.Join(s.root, e.Name(), "stat")); ok {
+			fn(st)
+		}
+	}
+}

internal/agent/discovery/load_test.go

@@ -0,0 +1,96 @@
+package discovery
+
+import (
+	"fmt"
+	"os"
+	"path/filepath"
+	"testing"
+	"time"
+)
+
+// fakeProc is a /proc with two CPUs, a memory picture and processes.
+type fakeProc struct {
+	t    *testing.T
+	root string
+}
+
+func newFakeProc(t *testing.T) *fakeProc {
+	f := &fakeProc{t: t, root: t.TempDir()}
+	f.write("loadavg", "1.50 0.75 0.25 2/300 4242\n")
+	f.write("meminfo", "MemTotal:       8000000 kB\nMemFree:        1000000 kB\nMemAvailable:   3000000 kB\n")
+	return f
+}
+
+func (f *fakeProc) write(name, body string) {
+	f.t.Helper()
+	p := filepath.Join(f.root, name)
+	if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
+		f.t.Fatal(err)
+	}
+	if err := os.WriteFile(p, []byte(body), 0o644); err != nil {
+		f.t.Fatal(err)
+	}
+}
+
+// cpu sets the aggregate counters: user and idle, the rest zero.
+func (f *fakeProc) cpu(user, idle uint64) {
+	f.write("stat", fmt.Sprintf("cpu  %d 0 0 %d 0 0 0 0 0 0\ncpu0 0 0 0 0 0 0 0 0 0 0\ncpu1 0 0 0 0 0 0 0 0 0 0\nintr 1\n", user, idle))
+}
+
+// proc sets one process: its group, CPU jiffies (as utime) and RSS in pages.
+func (f *fakeProc) proc(pid, pgrp int, jiffies uint64, rss int64) {
+	// The name holds a space and a parenthesis, as real ones can.
+	f.write(fmt.Sprintf("%d/stat", pid), fmt.Sprintf(
+		"%d (llama-server (x)) S 1 %d %d 0 -1 0 0 0 0 0 %d 0 0 0 20 0 1 0 100 1000 %d 0\n",
+		pid, pgrp, pgrp, jiffies, rss))
+}
+
+func TestLoadIsTheDifferenceBetweenTwoReadings(t *testing.T) {
+	p := newFakeProc(t)
+	at := time.Unix(1000, 0)
+	s := newLoadSampler(p.root, func() time.Time { return at })
+
+	p.cpu(100, 900)
+	p.proc(42, 42, 1000, 10)
+	p.proc(43, 42, 500, 5) // a worker in the engine's group
+	p.proc(7, 7, 99999, 1) // something else entirely
+	if _, ok := s.Sample(map[string]int{"shard-a": 42}); ok {
+		t.Fatal("the first reading has nothing to compare with and should report nothing")
+	}
+
+	// Two seconds later: 300 of the 400 jiffies that passed were busy, and
+	// the engine's group used 400 of them.
+	at = at.Add(2 * time.Second)
+	p.cpu(400, 1000)
+	p.proc(42, 42, 1300, 10)
+	p.proc(43, 42, 600, 6)
+	l, ok := s.Sample(map[string]int{"shard-a": 42})
+	if !ok {
+		t.Fatal("the second reading should report")
+	}
+	if l.CPUPercent != 75 || l.CPUCount != 2 {
+		t.Errorf("cpu %.1f%% of %d, want 75%% of 2", l.CPUPercent, l.CPUCount)
+	}
+	if l.Load1 != 1.5 || l.Load15 != 0.25 {
+		t.Errorf("load %v %v", l.Load1, l.Load15)
+	}
+	if l.MemUsed != 5000000*1024 || l.MemAvail != 3000000*1024 {
+		t.Errorf("memory used %d available %d", l.MemUsed, l.MemAvail)
+	}
+	if len(l.Shards) != 1 {
+		t.Fatalf("shards = %+v", l.Shards)
+	}
+	sh := l.Shards[0]
+	// 400 jiffies over 2 seconds at 100 a second is two cores.
+	if sh.ID != "shard-a" || sh.CPUCores != 2 || sh.RSSBytes != 16*int64(os.Getpagesize()) {
+		t.Errorf("shard = %+v", sh)
+	}
+}
+
+func TestNoProcMeansNoLoad(t *testing.T) {
+	s := newLoadSampler(filepath.Join(t.TempDir(), "missing"), time.Now)
+	s.Sample(nil)
+	if _, ok := s.Sample(nil); ok {
+		t.Error("a system without /proc should report nothing, not zeros")
+	}
+}

internal/agent/discovery/perftest.go

@@ -0,0 +1,133 @@
+package discovery
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"os/exec"
+	"regexp"
+	"strconv"
+	"strings"
+	"time"
+)
+
+// WriteBW is one side of an ib_write_bw run, from perftest: an RDMA write
+// bandwidth test between two adapters, which is what a collective spanning
+// nodes rides on.
+type WriteBW struct {
+	HCA  string
+	Port int
+	// GIDIndex is the RoCE v2 GID to use; negative on InfiniBand, where the
+	// port's LID is used.
+	GIDIndex int
+	// Peer is the server's address, for the client; empty makes the server.
+	Peer    string
+	TCPPort int
+	Seconds int
+}
+
+// Args are the command line: a duration rather than a count, so the test
+// takes as long on a slow link as a fast one, and results in Gb/s.
+func (w WriteBW) Args() []string {
+	args := []string{"-d", w.HCA, "-i", strconv.Itoa(max(w.Port, 1)),
+		"-p", strconv.Itoa(w.TCPPort), "-D", strconv.Itoa(max(w.Seconds, 1)),
+		"-F", "--report_gbits"}
+	if w.GIDIndex >= 0 {
+		args = append(args, "-x", strconv.Itoa(w.GIDIndex))
+	}
+	if w.Peer != "" {
+		args = append(args, w.Peer)
+	}
+	return args
+}
+
+// RunWriteBW runs one side and returns the average bandwidth the output
+// reports, in Gb/s, and the output. The client retries its connection for a
+// while: the server is started first, but a few seconds may pass before it
+// listens.
+func RunWriteBW(ctx context.Context, w WriteBW) (float64, string, error) {
+	bin, err := exec.LookPath("ib_write_bw")
+	if err != nil {
+		return 0, "", errors.New("ib_write_bw is not installed (the perftest package)")
+	}
+	limit := time.Duration(w.Seconds+60) * time.Second
+	var out []byte
+	for attempt := 0; ; attempt++ {
+		cctx, cancel := context.WithTimeout(ctx, limit)
+		out, err = exec.CommandContext(cctx, bin, w.Args()...).CombinedOutput() //nolint:gosec // fixed tool
+		cancel()
+		if err == nil || w.Peer == "" || !refused(string(out)) || attempt >= 10 {
+			break
+		}
+		select {
+		case <-ctx.Done():
+			return 0, string(out), ctx.Err()
+		case <-time.After(time.Second):
+		}
+	}
+	text := string(out)
+	if err != nil {
+		return 0, text, fmt.Errorf("ib_write_bw: %v: %s", err, lastNonEmpty(text))
+	}
+	gbps, perr := ParseWriteBW(text)
+	if perr != nil && w.Peer != "" {
+		// Only the client is sure to print the result; a server that does not
+		// is not a failure.
+		return 0, text, perr
+	}
+	return gbps, text, nil
+}
+
+func refused(out string) bool {
+	for _, s := range []string{"Couldn't connect", "Unable to init the socket connection", "Connection refused"} {
+		if strings.Contains(out, s) {
+			return true
+		}
+	}
+	return false
+}
+
+var columnSplit = regexp.MustCompile(`\s{2,}`)
+
+// ParseWriteBW finds the average bandwidth in perftest's result table:
+//
+//	#bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]
+//	65536      887036           0.00               96.93               0.184877
+//
+// by the heading's position rather than a fixed column, since versions add
+// columns and the peak reads 0.00 when the test runs for a duration.
+func ParseWriteBW(out string) (float64, error) {
+	lines := strings.Split(out, "\n")
+	for i, l := range lines {
+		if !strings.Contains(l, "BW average") {
+			continue
+		}
+		heads := columnSplit.Split(strings.TrimSpace(l), -1)
+		col := -1
+		for j, h := range heads {
+			if strings.HasPrefix(h, "BW average") {
+				col = j
+			}
+		}
+		for _, d := range lines[i+1:] {
+			f := strings.Fields(d)
+			if len(f) == 0 || strings.HasPrefix(f[0], "-") {
+				continue
+			}
+			if len(f) != len(heads) || col < 0 {
+				return 0, fmt.Errorf("perftest's result table has a shape this does not know: %q / %q", l, d)
+			}
+			v, err := strconv.ParseFloat(f[col], 64)
+			if err != nil {
+				return 0, fmt.Errorf("reading the average bandwidth %q: %w", f[col], err)
+			}
+			return v, nil
+		}
+	}
+	return 0, errors.New("perftest printed no result")
+}
+
+func lastNonEmpty(s string) string {
+	lines := strings.Split(strings.TrimSpace(s), "\n")
+	return strings.TrimSpace(lines[len(lines)-1])
+}

internal/agent/discovery/perftest_test.go

@@ -0,0 +1,49 @@
+package discovery
+
+import (
+	"slices"
+	"testing"
+)
+
+// The client's output of perftest 24.x, with a duration.
+const writeBWClient = `---------------------------------------------------------------------------------------
+                    RDMA_Write BW Test
+ Dual-port       : OFF		Device         : mlx5_0
+ Number of qps   : 1		Transport type : IB
+ Connection type : RC		Using SRQ      : OFF
+ TX depth        : 128
+ CQ Moderation   : 100
+ Mtu             : 4096[B]
+ Link type       : IB
+ Max inline data : 0[B]
+ rdma_cm QPs	 : OFF
+ Data ex. method : Ethernet
+---------------------------------------------------------------------------------------
+ local address: LID 0x05 QPN 0x0112 PSN 0x7b1f02 RKey 0x1fffbd VAddr 0x007f1d2a5f0000
+ remote address: LID 0x09 QPN 0x0113 PSN 0x2c1e44 RKey 0x1ffeaf VAddr 0x007fa4e1a60000
+---------------------------------------------------------------------------------------
+ #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]
+ 65536      1851214          0.00               388.21		   0.740422
+---------------------------------------------------------------------------------------
+`
+
+func TestTheAverageBandwidthIsReadFromPerftest(t *testing.T) {
+	got, err := ParseWriteBW(writeBWClient)
+	if err != nil || got != 388.21 {
+		t.Errorf("got %v, %v", got, err)
+	}
+	if _, err := ParseWriteBW(" Couldn't connect to 10.0.0.2:18515\n"); err == nil {
+		t.Error("output with no result table gave a number")
+	}
+}
+
+func TestWriteBWArguments(t *testing.T) {
+	ib := WriteBW{HCA: "mlx5_0", Port: 1, GIDIndex: -1, TCPPort: 18515, Seconds: 3}.Args()
+	if slices.Contains(ib, "-x") {
+		t.Errorf("InfiniBand was given a GID index: %v", ib)
+	}
+	roce := WriteBW{HCA: "mlx5_0", Port: 1, GIDIndex: 3, Peer: "10.0.0.2", TCPPort: 18515, Seconds: 3}.Args()
+	if !slices.Contains(roce, "-x") || roce[len(roce)-1] != "10.0.0.2" {
+		t.Errorf("RoCE client: %v", roce)
+	}
+}

internal/agent/discovery/sensors.go

@@ -0,0 +1,242 @@
+package discovery
+
+import (
+	"context"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"strconv"
+	"strings"
+	"sync"
+	"time"
+)
+
+// Sensors is what a node's own hardware says about heat and power: the CPUs,
+// the board where it has an inlet sensor, and each GPU. A zero is not
+// measured, never a reading of zero — a VM has none of these, and must not
+// look like a machine running at 0 °C on no power.
+type Sensors struct {
+	// CPUTempC is the hottest CPU sensor: the package, or the hottest core
+	// where no package reading exists.
+	CPUTempC float64
+	// CPUPowerW is the CPU packages' draw, from RAPL: the change in their
+	// energy counters over the time between two samples.
+	CPUPowerW float64
+	// InletTempC is the air coming in, where the board has a sensor it calls
+	// that. Most do not, and a CPU's temperature is not a substitute: it
+	// follows the load, not the room.
+	InletTempC float64
+	GPUs       []GPUSensor
+}
+
+// GPUSensor is one card's heat, power and whether it is slowing itself down.
+type GPUSensor struct {
+	Index       int
+	TempC       float64
+	MemoryTempC float64
+	PowerW      float64
+	PowerLimitW float64
+	// Throttle names what is holding the card's clocks down now:
+	// sw_power_cap, hw_slowdown, sw_thermal, hw_thermal, power_brake. Empty
+	// is running free. Idle and application clock settings are left out;
+	// they are not the card being held back.
+	Throttle []string
+}
+
+// sensorEvery bounds how often nvidia-smi is run. Heartbeats come more often
+// than temperatures change, and each run costs a few hundred milliseconds.
+const sensorEvery = 10 * time.Second
+
+// SensorSampler reads Sensors. The first sample has no CPU power, which needs
+// two readings of the energy counters.
+type SensorSampler struct {
+	sys string // /sys, or a test's copy of it
+	smi func(ctx context.Context, fields string) ([]byte, error)
+	now func() time.Time
+
+	mu         sync.Mutex
+	last       Sensors
+	lastAt     time.Time
+	prevEnergy map[string]uint64
+	prevAt     time.Time
+}
+
+// NewSensorSampler reads this machine.
+func NewSensorSampler() *SensorSampler {
+	return &SensorSampler{sys: "/sys", smi: runSMI, now: time.Now}
+}
+
+func runSMI(ctx context.Context, fields string) ([]byte, error) {
+	ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
+	defer cancel()
+	//nolint:gosec // fixed binary, fields from this file
+	return exec.CommandContext(ctx, "nvidia-smi", "--query-gpu="+fields,
+		"--format=csv,noheader,nounits").Output()
+}
+
+// Sample reads the sensors, or returns the last reading when that is recent.
+func (s *SensorSampler) Sample(ctx context.Context) Sensors {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	now := s.now()
+	if !s.lastAt.IsZero() && now.Sub(s.lastAt) < sensorEvery {
+		return s.last
+	}
+	out := Sensors{}
+	out.CPUTempC, out.InletTempC = s.hwmon()
+	out.CPUPowerW = s.rapl(now)
+	out.GPUs = s.gpus(ctx)
+	s.last, s.lastAt = out, now
+	return out
+}
+
+// hwmon reads the hottest CPU sensor and an inlet sensor, from every hwmon
+// device the kernel exposes — what lm_sensors reads, without it installed.
+func (s *SensorSampler) hwmon() (cpu, inlet float64) {
+	devs, _ := filepath.Glob(filepath.Join(s.sys, "class/hwmon/hwmon*"))
+	var pkg, core float64
+	for _, d := range devs {
+		name := readTrim(filepath.Join(d, "name"))
+		inputs, _ := filepath.Glob(filepath.Join(d, "temp*_input"))
+		for _, in := range inputs {
+			milli, err := strconv.ParseFloat(readTrim(in), 64)
+			if err != nil || milli <= 0 {
+				continue
+			}
+			c := milli / 1000
+			label := strings.ToLower(readTrim(strings.TrimSuffix(in, "_input") + "_label"))
+			switch {
+			case isCPUSensor(name) && (strings.HasPrefix(label, "package") || label == "tctl" || label == "tdie"):
+				pkg = max(pkg, c)
+			case isCPUSensor(name):
+				core = max(core, c)
+			case label == "cpu":
+				// A board's own reading of the CPU, as Dell's SMM gives.
+				core = max(core, c)
+			case strings.Contains(label, "inlet") || strings.Contains(label, "ambient") ||
+				strings.Contains(label, "intake") || label == "systin":
+				inlet = max(inlet, c)
+			}
+		}
+	}
+	if pkg > 0 {
+		return pkg, inlet
+	}
+	return core, inlet
+}
+
+func isCPUSensor(name string) bool {
+	switch name {
+	case "coretemp", "k10temp", "zenpower", "cpu_thermal":
+		return true
+	}
+	return false
+}
+
+// rapl is the CPU packages' power from their energy counters. Package
+// domains only: the dram and core subdomains are parts of a package, and
+// adding them would count the same energy twice.
+func (s *SensorSampler) rapl(now time.Time) float64 {
+	doms, _ := filepath.Glob(filepath.Join(s.sys, "class/powercap/intel-rapl:*"))
+	energy := map[string]uint64{}
+	wrap := map[string]uint64{}
+	for _, d := range doms {
+		if strings.Count(filepath.Base(d), ":") != 1 {
+			continue // intel-rapl:0:0 is a subdomain of intel-rapl:0
+		}
+		e, err := strconv.ParseUint(readTrim(filepath.Join(d, "energy_uj")), 10, 64)
+		if err != nil {
+			continue // unreadable without root on newer kernels
+		}
+		energy[d] = e
+		wrap[d], _ = strconv.ParseUint(readTrim(filepath.Join(d, "max_energy_range_uj")), 10, 64)
+	}
+	prev, prevAt := s.prevEnergy, s.prevAt
+	s.prevEnergy, s.prevAt = energy, now
+	dt := now.Sub(prevAt).Seconds()
+	if prev == nil || dt <= 0 || len(energy) == 0 {
+		return 0
+	}
+	var joules float64
+	for d, e := range energy {
+		p, ok := prev[d]
+		if !ok {
+			return 0 // a domain appeared: no fair difference this time
+		}
+		if e < p { // the counter wrapped
+			e += wrap[d]
+		}
+		joules += float64(e-p) / 1e6
+	}
+	return joules / dt
+}
+
+// The throttle reasons nvidia-smi reports, as bits. Only those that mean the
+// card is being held back are named.
+var throttleBits = []struct {
+	bit  uint64
+	name string
+}{
+	{0x4, "sw_power_cap"},
+	{0x8, "hw_slowdown"},
+	{0x20, "sw_thermal"},
+	{0x40, "hw_thermal"},
+	{0x80, "power_brake"},
+}
+
+// gpus asks nvidia-smi for each card's heat and power. The memory temperature
+// is not known to every driver, and a field nvidia-smi does not know fails the
+// whole query, so it is asked for first and dropped on failure.
+func (s *SensorSampler) gpus(ctx context.Context) []GPUSensor {
+	withMem := "index,temperature.gpu,power.draw,enforced.power.limit,clocks_throttle_reasons.active,temperature.memory"
+	out, err := s.smi(ctx, withMem)
+	if err != nil {
+		if out, err = s.smi(ctx, strings.TrimSuffix(withMem, ",temperature.memory")); err != nil {
+			return nil
+		}
+	}
+	var gs []GPUSensor
+	for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
+		f := strings.Split(line, ",")
+		if len(f) < 5 {
+			continue
+		}
+		for i := range f {
+			f[i] = strings.TrimSpace(f[i])
+		}
+		idx, err := strconv.Atoi(f[0])
+		if err != nil {
+			continue
+		}
+		g := GPUSensor{Index: idx, TempC: num(f[1]), PowerW: num(f[2]), PowerLimitW: num(f[3])}
+		if bits, err := strconv.ParseUint(strings.TrimPrefix(f[4], "0x"), 16, 64); err == nil {
+			for _, b := range throttleBits {
+				if bits&b.bit != 0 {
+					g.Throttle = append(g.Throttle, b.name)
+				}
+			}
+		}
+		if len(f) > 5 {
+			g.MemoryTempC = num(f[5])
+		}
+		gs = append(gs, g)
+	}
+	return gs
+}
+
+// num reads a figure nvidia-smi may give as "[N/A]" or "N/A".
+func num(s string) float64 {
+	v, err := strconv.ParseFloat(s, 64)
+	if err != nil || v < 0 {
+		return 0
+	}
+	return v
+}
+
+func readTrim(path string) string {
+	b, err := os.ReadFile(path) //nolint:gosec // sysfs
+	if err != nil {
+		return ""
+	}
+	return strings.TrimSpace(string(b))
+}

internal/agent/discovery/sensors_test.go

@@ -0,0 +1,169 @@
+package discovery
+
+import (
+	"context"
+	"errors"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+	"time"
+)
+
+func put(t *testing.T, root, rel, body string) {
+	t.Helper()
+	p := filepath.Join(root, rel)
+	if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
+		t.Fatal(err)
+	}
+	if err := os.WriteFile(p, []byte(body+"\n"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+}
+
+// dellHost is the layout of the Proxmox host these tests were written
+// against: Dell's SMM with a CPU and SODIMM sensors, coretemp with a package
+// and cores, and RAPL with a package and its dram subdomain.
+func dellHost(t *testing.T) string {
+	root := t.TempDir()
+	put(t, root, "class/hwmon/hwmon0/name", "dell_smm")
+	put(t, root, "class/hwmon/hwmon0/temp1_input", "55000")
+	put(t, root, "class/hwmon/hwmon0/temp1_label", "CPU")
+	put(t, root, "class/hwmon/hwmon0/temp2_input", "47000")
+	put(t, root, "class/hwmon/hwmon0/temp2_label", "SODIMM")
+	put(t, root, "class/hwmon/hwmon1/name", "coretemp")
+	put(t, root, "class/hwmon/hwmon1/temp1_input", "58000")
+	put(t, root, "class/hwmon/hwmon1/temp1_label", "Package id 0")
+	put(t, root, "class/hwmon/hwmon1/temp10_input", "61000")
+	put(t, root, "class/hwmon/hwmon1/temp10_label", "Core 8")
+	put(t, root, "class/powercap/intel-rapl:0/energy_uj", "1000000000")
+	put(t, root, "class/powercap/intel-rapl:0/max_energy_range_uj", "262143328850")
+	put(t, root, "class/powercap/intel-rapl:0:0/energy_uj", "5000000")
+	return root
+}
+
+func sampler(sys string, smi func(context.Context, string) ([]byte, error), clock *time.Time) *SensorSampler {
+	return &SensorSampler{sys: sys, smi: smi, now: func() time.Time { return *clock }}
+}
+
+func noSMI(context.Context, string) ([]byte, error) { return nil, errors.New("not found") }
+
+func TestAHostsCPUHeatAndPowerAreReadAsLmSensorsWould(t *testing.T) {
+	root := dellHost(t)
+	now := time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC)
+	s := sampler(root, noSMI, &now)
+
+	first := s.Sample(t.Context())
+	// The package reading, not the hotter core: the package is what the
+	// chip itself reports as its temperature.
+	if first.CPUTempC != 58 {
+		t.Errorf("CPU %.1f °C, expected the package's 58", first.CPUTempC)
+	}
+	if first.CPUPowerW != 0 {
+		t.Error("power from one reading of a counter is invented")
+	}
+	if first.InletTempC != 0 || first.GPUs != nil {
+		t.Errorf("sensors this host does not have were reported: %+v", first)
+	}
+
+	// 10 s later the package has used 650 J, and its dram subdomain some of
+	// that — counted once.
+	put(t, root, "class/powercap/intel-rapl:0/energy_uj", "1650000000")
+	put(t, root, "class/powercap/intel-rapl:0:0/energy_uj", "99000000")
+	now = now.Add(10 * time.Second)
+	if got := s.Sample(t.Context()).CPUPowerW; got < 64.9 || got > 65.1 {
+		t.Errorf("CPU power %.2f W, expected 65", got)
+	}
+}
+
+func TestACounterThatWrapsIsNotANegativeDraw(t *testing.T) {
+	root := dellHost(t)
+	now := time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC)
+	put(t, root, "class/powercap/intel-rapl:0/energy_uj", "262143000000")
+	s := sampler(root, noSMI, &now)
+	s.Sample(t.Context())
+	// 328,850 µJ to the wrap, then 499,671,150 past it: 500 J in 10 s.
+	put(t, root, "class/powercap/intel-rapl:0/energy_uj", "499671150")
+	now = now.Add(10 * time.Second)
+	if got := s.Sample(t.Context()).CPUPowerW; got < 49.9 || got > 50.1 {
+		t.Errorf("across a wrap: %.2f W, expected 50", got)
+	}
+}
+
+func TestAVMHasNoSensorsAndSaysNothing(t *testing.T) {
+	root := t.TempDir()
+	put(t, root, "class/thermal/cooling_device0/type", "Processor")
+	now := time.Now()
+	got := sampler(root, noSMI, &now).Sample(t.Context())
+	if got.CPUTempC != 0 || got.CPUPowerW != 0 || got.InletTempC != 0 || got.GPUs != nil {
+		t.Errorf("a machine with no sensors reported %+v", got)
+	}
+}
+
+func TestABoardsInletSensorIsTheRoomNotTheCPU(t *testing.T) {
+	root := t.TempDir()
+	put(t, root, "class/hwmon/hwmon2/name", "nct6798")
+	put(t, root, "class/hwmon/hwmon2/temp1_input", "24500")
+	put(t, root, "class/hwmon/hwmon2/temp1_label", "SYSTIN")
+	put(t, root, "class/hwmon/hwmon3/name", "k10temp")
+	put(t, root, "class/hwmon/hwmon3/temp1_input", "71250")
+	put(t, root, "class/hwmon/hwmon3/temp1_label", "Tctl")
+	now := time.Now()
+	got := sampler(root, noSMI, &now).Sample(t.Context())
+	if got.InletTempC != 24.5 || got.CPUTempC != 71.25 {
+		t.Errorf("inlet %.2f, CPU %.2f", got.InletTempC, got.CPUTempC)
+	}
+}
+
+func TestEachGPUSaysHowHotHowHungryAndWhatHoldsItBack(t *testing.T) {
+	var asked []string
+	smi := func(_ context.Context, fields string) ([]byte, error) {
+		asked = append(asked, fields)
+		return []byte(
+			"0, 71, 612.45, 700.00, 0x0000000000000000, 84\n" +
+				"1, 88, 698.10, 700.00, 0x0000000000000024, 95\n" + // sw_power_cap + sw_thermal
+				"2, 45, [N/A], 700.00, 0x0000000000000001, [N/A]\n"), nil // idle only
+	}
+	now := time.Now()
+	ss := sampler(t.TempDir(), smi, &now)
+	got := ss.Sample(t.Context())
+	if len(got.GPUs) != 3 {
+		t.Fatalf("%+v", got.GPUs)
+	}
+	g0, g1, g2 := got.GPUs[0], got.GPUs[1], got.GPUs[2]
+	if g0.TempC != 71 || g0.PowerW != 612.45 || g0.PowerLimitW != 700 || g0.MemoryTempC != 84 || g0.Throttle != nil {
+		t.Errorf("card 0 %+v", g0)
+	}
+	if strings.Join(g1.Throttle, ",") != "sw_power_cap,sw_thermal" {
+		t.Errorf("card 1 throttled by %v", g1.Throttle)
+	}
+	if g2.PowerW != 0 || g2.MemoryTempC != 0 || g2.Throttle != nil {
+		t.Errorf("card 2: N/A is not a reading, and idle is not being held back: %+v", g2)
+	}
+	// Within ten seconds the last reading is reused rather than running
+	// nvidia-smi on every heartbeat; after, it is asked again.
+	now = now.Add(5 * time.Second)
+	ss.Sample(t.Context())
+	if len(asked) != 1 {
+		t.Errorf("nvidia-smi was asked %d times within ten seconds", len(asked))
+	}
+	now = now.Add(6 * time.Second)
+	ss.Sample(t.Context())
+	if len(asked) != 2 {
+		t.Errorf("nvidia-smi was asked %d times after eleven seconds, expected 2", len(asked))
+	}
+}
+
+func TestADriverThatDoesNotKnowMemoryTemperatureStillReportsTheRest(t *testing.T) {
+	smi := func(_ context.Context, fields string) ([]byte, error) {
+		if strings.Contains(fields, "temperature.memory") {
+			return nil, errors.New(`Field "temperature.memory" is not a valid field to query.`)
+		}
+		return []byte("0, 66, 250.0, 300.0, 0x0000000000000040\n"), nil
+	}
+	now := time.Now()
+	got := sampler(t.TempDir(), smi, &now).Sample(t.Context())
+	if len(got.GPUs) != 1 || got.GPUs[0].TempC != 66 || strings.Join(got.GPUs[0].Throttle, ",") != "hw_thermal" {
+		t.Errorf("%+v", got.GPUs)
+	}
+}

internal/agent/discovery/sminfo.go

@@ -0,0 +1,172 @@
+package discovery
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"os/exec"
+	"regexp"
+	"strconv"
+	"strings"
+	"time"
+)
+
+// ErrNoDiags means infiniband-diags is not installed, so the subnet manager
+// cannot be asked who it is.
+var ErrNoDiags = errors.New("sminfo not found: install infiniband-diags")
+
+// smLine is sminfo's one line of answer:
+//
+//	sminfo: sm lid 1 sm guid 0x98039b03009fcfd6, activity count 5452 priority 15 state 3 SMINFO_MASTER
+var smLine = regexp.MustCompile(`sm lid (\d+) sm guid (0x[0-9a-fA-F]+), activity count \d+ priority (\d+) state \d+ (\S+)`)
+
+// ParseSMInfo reads sminfo's output.
+func ParseSMInfo(out string) (*SMInfo, error) {
+	m := smLine.FindStringSubmatch(out)
+	if m == nil {
+		return nil, fmt.Errorf("unrecognised sminfo output: %q", strings.TrimSpace(out))
+	}
+	lid, _ := strconv.Atoi(m[1])
+	prio, _ := strconv.Atoi(m[3])
+	return &SMInfo{GUID: normalizeGUID(m[2]), LID: lid, Priority: prio, State: m[4]}, nil
+}
+
+// QuerySM asks, through one port, which subnet manager is master. It is a
+// management datagram to the SM, not a scan: cheap, but not free, which is
+// why the agent asks once a minute rather than every heartbeat.
+func QuerySM(ctx context.Context, hca string, port int) (*SMInfo, error) {
+	bin, err := exec.LookPath("sminfo")
+	if err != nil {
+		return nil, ErrNoDiags
+	}
+	ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
+	defer cancel()
+	out, err := exec.CommandContext(ctx, bin, "-C", hca, "-P", strconv.Itoa(port)).CombinedOutput()
+	if err != nil {
+		return nil, fmt.Errorf("sminfo on %s/%d: %w: %s", hca, port, err, strings.TrimSpace(string(out)))
+	}
+	return ParseSMInfo(string(out))
+}
+
+// AttachSM fills in the subnet manager each active InfiniBand port sees.
+// RoCE has none. The first error is returned for reporting; the ports it
+// could answer for keep their answer.
+func AttachSM(hcas []HCA, query func(hca string, port int) (*SMInfo, error)) error {
+	var first error
+	for i := range hcas {
+		for j := range hcas[i].Ports {
+			p := &hcas[i].Ports[j]
+			if !p.Active() || p.LinkLayer == LinkEthernet {
+				continue
+			}
+			sm, err := query(hcas[i].Name, p.Num)
+			if err != nil {
+				if first == nil {
+					first = err
+				}
+				continue
+			}
+			p.SM = sm
+		}
+	}
+	return first
+}
+
+// SMRecord is one subnet manager on a fabric, as the subnet administrator
+// lists them: master and standbys alike.
+type SMRecord struct {
+	GUID     string `json:"guid"`
+	LID      int    `json:"lid"`
+	Priority int    `json:"priority"`
+	// State is master, standby, discovering or not active.
+	State string `json:"state"`
+}
+
+// smStates are SMInfo's state numbers.
+var smStates = map[string]string{"0": "not active", "1": "discovering", "2": "standby", "3": "master"}
+
+// smirField is one line of saquery's record dump: "LID.........1".
+var smirField = regexp.MustCompile(`^\s*([A-Za-z_]+)\.{2,}\s*(\S+)`)
+
+// ParseSMIR reads `saquery SMIR`: every subnet manager the fabric's subnet
+// administrator knows of, one "SMInfoRecord dump:" block each.
+func ParseSMIR(out string) ([]SMRecord, error) {
+	var recs []SMRecord
+	var cur *SMRecord
+	for _, line := range strings.Split(out, "\n") {
+		if strings.Contains(line, "SMInfoRecord dump") {
+			recs = append(recs, SMRecord{})
+			cur = &recs[len(recs)-1]
+			continue
+		}
+		m := smirField.FindStringSubmatch(line)
+		if m == nil || cur == nil {
+			continue
+		}
+		switch strings.ToLower(m[1]) {
+		case "lid":
+			cur.LID, _ = strconv.Atoi(m[2])
+		case "guid":
+			cur.GUID = normalizeGUID(m[2])
+		case "priority":
+			cur.Priority, _ = strconv.Atoi(m[2])
+		case "smstate":
+			cur.State = smStates[m[2]]
+			if cur.State == "" {
+				cur.State = m[2]
+			}
+		}
+	}
+	out2 := recs[:0]
+	for _, r := range recs {
+		if r.GUID != "" {
+			out2 = append(out2, r)
+		}
+	}
+	if len(out2) == 0 {
+		return nil, fmt.Errorf("no subnet manager records in saquery output")
+	}
+	return out2, nil
+}
+
+// QuerySMs asks the subnet administrator, through one port, for every subnet
+// manager on the fabric.
+func QuerySMs(ctx context.Context, hca string, port int) ([]SMRecord, error) {
+	bin, err := exec.LookPath("saquery")
+	if err != nil {
+		return nil, ErrNoDiags
+	}
+	ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
+	defer cancel()
+	out, err := exec.CommandContext(ctx, bin, "-C", hca, "-P", strconv.Itoa(port), "SMIR").CombinedOutput()
+	if err != nil {
+		return nil, fmt.Errorf("saquery on %s/%d: %w: %s", hca, port, err, strings.TrimSpace(string(out)))
+	}
+	return ParseSMIR(string(out))
+}
+
+// ScanFabric maps the fabric a port is on with ibnetdiscover: every switch,
+// adapter and link. It can take a while on a large fabric, so it is bounded,
+// and its output is capped: a fabric of thousands of ports is a few
+// megabytes of text.
+func ScanFabric(ctx context.Context, hca string, port int) (string, error) {
+	bin, err := exec.LookPath("ibnetdiscover")
+	if err != nil {
+		return "", ErrNoDiags
+	}
+	ctx, cancel := context.WithTimeout(ctx, 2*time.Minute)
+	defer cancel()
+	out, err := exec.CommandContext(ctx, bin, "-C", hca, "-P", strconv.Itoa(port)).Output()
+	if err != nil {
+		var ee *exec.ExitError
+		if errors.As(err, &ee) {
+			return "", fmt.Errorf("ibnetdiscover on %s/%d: %w: %s", hca, port, err, strings.TrimSpace(string(ee.Stderr)))
+		}
+		return "", fmt.Errorf("ibnetdiscover on %s/%d: %w", hca, port, err)
+	}
+	const limit = 16 << 20
+	if len(out) > limit {
+		return "", fmt.Errorf("ibnetdiscover printed %d MB, more than the %d MB a scan may send", len(out)>>20, limit>>20)
+	}
+	return string(out), nil
+}

internal/agent/discovery/topo.go

@@ -0,0 +1,185 @@
+package discovery
+
+import (
+	"context"
+	"fmt"
+	"os/exec"
+	"regexp"
+	"sort"
+	"strconv"
+	"strings"
+)
+
+// Topology is what `nvidia-smi topo -m` says about how a node's GPUs and
+// network adapters are wired together.
+//
+// Read from nvidia-smi rather than NVML so the portable, cgo-free binary knows
+// it too. Without it that binary saw no NVLink at all: every GPU reported no
+// peers, and a tensor-parallel group was chosen by index on machines where
+// NVLink decides whether it runs at full speed.
+type Topology struct {
+	// NVLink is, per GPU, the GPUs it reaches over NVLink and how many links
+	// are bonded to each.
+	NVLink map[int]map[int]int
+	// NICPath is, per GPU, how far each network adapter is, by its HCA name:
+	// PIX, PXB, PHB, NODE or SYS, nearest first. A GPU should talk to the
+	// fabric through the adapter nearest it; one across the socket costs a
+	// trip over the processor interconnect on every transfer.
+	NICPath map[int]map[string]string
+	// NUMA is the NUMA node each GPU is attached to, where reported.
+	NUMA map[int]int
+}
+
+// PathRank orders the relations nvidia-smi reports, nearest first. Lower is
+// closer; an unknown relation ranks last.
+func PathRank(rel string) int {
+	switch strings.ToUpper(rel) {
+	case "PIX":
+		return 0
+	case "PXB":
+		return 1
+	case "PHB":
+		return 2
+	case "NODE":
+		return 3
+	case "SYS":
+		return 4
+	}
+	return 5
+}
+
+// DiscoverTopology runs `nvidia-smi topo -m`. A node without nvidia-smi has
+// no topology to report, which is not an error.
+func DiscoverTopology(ctx context.Context, binary string) (Topology, error) {
+	if binary == "" {
+		if _, err := exec.LookPath("nvidia-smi"); err != nil {
+			return Topology{}, nil
+		}
+		binary = "nvidia-smi"
+	}
+	out, err := exec.CommandContext(ctx, binary, "topo", "-m").Output()
+	if err != nil {
+		return Topology{}, fmt.Errorf("running %s topo -m: %w", binary, err)
+	}
+	return parseTopo(string(out))
+}
+
+// ParseTopology reads `nvidia-smi topo -m` output.
+func ParseTopology(out string) (Topology, error) { return parseTopo(out) }
+
+var (
+	ansi       = regexp.MustCompile(`\x1b\[[0-9;]*m`)
+	nvLinks    = regexp.MustCompile(`^NV(\d+)$`)
+	nicLegend  = regexp.MustCompile(`^\s*(NIC\d+)\s*:\s*(\S+)\s*$`)
+	gpuColumn  = regexp.MustCompile(`^GPU(\d+)$`)
+	numaColumn = "NUMA Affinity"
+)
+
+// parseTopo reads the matrix. Columns are tab-separated; the header names
+// each one — GPUn, then the adapters, as NICn with a legend below on newer
+// drivers or by HCA name on older ones — then CPU and NUMA affinity.
+func parseTopo(out string) (Topology, error) {
+	out = ansi.ReplaceAllString(out, "")
+	lines := strings.Split(out, "\n")
+
+	legend := map[string]string{}
+	for _, l := range lines {
+		if m := nicLegend.FindStringSubmatch(l); m != nil {
+			legend[m[1]] = m[2]
+		}
+	}
+
+	var header []string
+	start := -1
+	for i, l := range lines {
+		cells := splitTabs(l)
+		if len(cells) > 1 && cells[0] == "" && gpuColumn.MatchString(cells[1]) {
+			header, start = cells, i
+			break
+		}
+	}
+	if start < 0 {
+		return Topology{}, fmt.Errorf("no GPU matrix in nvidia-smi topo output")
+	}
+
+	t := Topology{NVLink: map[int]map[int]int{}, NICPath: map[int]map[string]string{}, NUMA: map[int]int{}}
+	for _, l := range lines[start+1:] {
+		cells := splitTabs(l)
+		if len(cells) < 2 {
+			// The matrix ends at the first blank line, before the legend.
+			if strings.TrimSpace(l) == "" {
+				break
+			}
+			continue
+		}
+		m := gpuColumn.FindStringSubmatch(cells[0])
+		if m == nil {
+			continue // an adapter's row: its view of the GPUs is the same matrix
+		}
+		gpu, _ := strconv.Atoi(m[1])
+		for c := 1; c < len(cells) && c < len(header); c++ {
+			col, v := header[c], strings.TrimSpace(cells[c])
+			switch {
+			case gpuColumn.MatchString(col):
+				peer, _ := strconv.Atoi(gpuColumn.FindStringSubmatch(col)[1])
+				if n := nvLinks.FindStringSubmatch(v); n != nil && peer != gpu {
+					links, _ := strconv.Atoi(n[1])
+					if t.NVLink[gpu] == nil {
+						t.NVLink[gpu] = map[int]int{}
+					}
+					t.NVLink[gpu][peer] = links
+				}
+			case col == numaColumn:
+				if n, err := strconv.Atoi(v); err == nil {
+					t.NUMA[gpu] = n
+				}
+			case col == "CPU Affinity" || col == "GPU NUMA ID" || col == "":
+			default:
+				// An adapter: NICn named by the legend, or an HCA by name.
+				hca := col
+				if name, ok := legend[col]; ok {
+					hca = name
+				}
+				if PathRank(v) < 5 {
+					if t.NICPath[gpu] == nil {
+						t.NICPath[gpu] = map[string]string{}
+					}
+					t.NICPath[gpu][hca] = strings.ToUpper(v)
+				}
+			}
+		}
+	}
+	if len(t.NVLink) == 0 && len(t.NICPath) == 0 && len(t.NUMA) == 0 {
+		return t, fmt.Errorf("the nvidia-smi topo matrix had no rows")
+	}
+	return t, nil
+}
+
+func splitTabs(l string) []string {
+	l = strings.TrimRight(l, "\r")
+	cells := strings.Split(l, "\t")
+	for i := range cells {
+		cells[i] = strings.TrimSpace(cells[i])
+	}
+	return cells
+}
+
+// Apply puts the topology on the GPUs it describes. What a GPU already knows
+// — NVLink peers from NVML, which is the more exact source — is kept.
+func (t Topology) Apply(gpus []GPU) {
+	for i := range gpus {
+		g := &gpus[i]
+		if len(g.NVLinkPeers) == 0 {
+			for peer := range t.NVLink[g.Index] {
+				g.NVLinkPeers = append(g.NVLinkPeers, peer)
+			}
+			sort.Ints(g.NVLinkPeers)
+		}
+		if n, ok := t.NUMA[g.Index]; ok {
+			g.NUMANode = n
+		}
+		if paths := t.NICPath[g.Index]; len(paths) > 0 {
+			g.NICPaths = paths
+		}
+	}
+}

internal/agent/discovery/topo_test.go

@@ -0,0 +1,113 @@
+package discovery
+
+import (
+	"reflect"
+	"strings"
+	"testing"
+)
+
+// row joins cells the way nvidia-smi prints them: tab-separated, each padded.
+func row(cells ...string) string { return strings.Join(cells, "\t") }
+
+// dgxH100 is `nvidia-smi topo -m` on an eight-GPU NVSwitch machine with two
+// adapters, as newer drivers print it: NICn columns and a legend naming them.
+var dgxH100 = strings.Join([]string{
+	row("\x1b[4m", "GPU0", "GPU1", "GPU2", "GPU3", "NIC0", "NIC1", "CPU Affinity", "NUMA Affinity", "GPU NUMA ID\x1b[0m"),
+	row("GPU0", " X ", "NV18", "NV18", "NV18", "PXB", "SYS", "0-55,112-167", "0", "N/A"),
+	row("GPU1", "NV18", " X ", "NV18", "NV18", "PXB", "SYS", "0-55,112-167", "0", "N/A"),
+	row("GPU2", "NV18", "NV18", " X ", "NV18", "SYS", "PXB", "56-111,168-223", "1", "N/A"),
+	row("GPU3", "NV18", "NV18", "NV18", " X ", "SYS", "PIX", "56-111,168-223", "1", "N/A"),
+	row("NIC0", "PXB", "PXB", "SYS", "SYS", " X ", "SYS"),
+	row("NIC1", "SYS", "SYS", "PXB", "PIX", "SYS", " X "),
+	"",
+	"Legend:",
+	"",
+	"  X    = Self",
+	"  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)",
+	"  NV#  = Connection traversing a bonded set of # NVLinks",
+	"",
+	"NIC Legend:",
+	"",
+	"  NIC0: mlx5_0",
+	"  NIC1: mlx5_1",
+	"",
+}, "\n")
+
+func TestNVLinkAndAdapterDistanceAreReadFromTopo(t *testing.T) {
+	topo, err := parseTopo(dgxH100)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if got := topo.NVLink[0]; !reflect.DeepEqual(got, map[int]int{1: 18, 2: 18, 3: 18}) {
+		t.Errorf("GPU0's NVLink peers = %v", got)
+	}
+	// The legend turns NIC1 into the adapter's own name.
+	if got := topo.NICPath[3]; !reflect.DeepEqual(got, map[string]string{"mlx5_0": "SYS", "mlx5_1": "PIX"}) {
+		t.Errorf("GPU3's adapters = %v", got)
+	}
+	if topo.NUMA[2] != 1 || topo.NUMA[0] != 0 {
+		t.Errorf("NUMA = %v", topo.NUMA)
+	}
+
+	gpus := []GPU{{Index: 0}, {Index: 3}}
+	topo.Apply(gpus)
+	if !reflect.DeepEqual(gpus[0].NVLinkPeers, []int{1, 2, 3}) || gpus[1].NUMANode != 1 ||
+		gpus[1].NICPaths["mlx5_1"] != "PIX" {
+		t.Errorf("applied = %+v", gpus)
+	}
+}
+
+func TestOlderDriversNameAdaptersInTheHeader(t *testing.T) {
+	// No legend: the columns are the adapters' own names, and NVLink between
+	// pairs only, as on an A100 PCIe box with bridges.
+	out := strings.Join([]string{
+		row("", "GPU0", "GPU1", "GPU2", "GPU3", "mlx5_0", "CPU Affinity", "NUMA Affinity"),
+		row("GPU0", " X ", "NV12", "SYS", "SYS", "NODE", "0-31", "0"),
+		row("GPU1", "NV12", " X ", "SYS", "SYS", "NODE", "0-31", "0"),
+		row("GPU2", "SYS", "SYS", " X ", "NV12", "SYS", "32-63", "1"),
+		row("GPU3", "SYS", "SYS", "NV12", " X ", "SYS", "32-63", "1"),
+		row("mlx5_0", "NODE", "NODE", "SYS", "SYS", " X "),
+		"",
+	}, "\n")
+	topo, err := parseTopo(out)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if !reflect.DeepEqual(topo.NVLink[2], map[int]int{3: 12}) || len(topo.NVLink[0]) != 1 {
+		t.Errorf("NVLink = %v", topo.NVLink)
+	}
+	if topo.NICPath[0]["mlx5_0"] != "NODE" {
+		t.Errorf("adapters = %v", topo.NICPath)
+	}
+}
+
+func TestAPCIeMachineHasNoNVLink(t *testing.T) {
+	out := strings.Join([]string{
+		row("", "GPU0", "GPU1", "CPU Affinity", "NUMA Affinity"),
+		row("GPU0", " X ", "PHB", "0-15", "0"),
+		row("GPU1", "PHB", " X ", "0-15", "0"),
+		"",
+	}, "\n")
+	topo, err := parseTopo(out)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if len(topo.NVLink) != 0 {
+		t.Errorf("NVLink on a PCIe machine: %v", topo.NVLink)
+	}
+}
+
+func TestNVMLPeersAreKeptOverTopo(t *testing.T) {
+	topo, _ := parseTopo(dgxH100)
+	gpus := []GPU{{Index: 0, NVLinkPeers: []int{1}}}
+	topo.Apply(gpus)
+	if !reflect.DeepEqual(gpus[0].NVLinkPeers, []int{1}) {
+		t.Errorf("NVML's answer was replaced: %v", gpus[0].NVLinkPeers)
+	}
+}
+
+func TestSomethingThatIsNotAMatrixIsAnError(t *testing.T) {
+	if _, err := parseTopo("NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver."); err == nil {
+		t.Error("expected an error")
+	}
+}

internal/agent/engines/engine.go

@@ -40,6 +40,9 @@ type Profile struct {
 	ModelPath string
 	// ServedModelName is the name clients use, which need not match the path.
 	ServedModelName string
+	// Adapters are LoRA adapters to load beside the model, in order: an
+	// adapter's index is the id requests apply it by.
+	Adapters []Adapter
 }
 
 // ShardSpec is one shard of one replica on this node.
@@ -63,15 +66,35 @@ type ShardSpec struct {
 	HCAs []string
 	// MgmtInterface is the ethernet interface used for bootstrap traffic.
 	MgmtInterface string
+	// GIDIndex is the RoCE v2 GID index for the HCAs, set by the agent when
+	// they are RoCE; nil on InfiniBand, where NCCL needs none.
+	GIDIndex *int
 
 	HTTPPort int
 	Profile  Profile
 
+	// APIKey is what the engine demands of every request. Engines listen on
+	// all of a node's addresses; without a key anybody who can reach the port
+	// talks to the model directly, past the gateway's authentication, quotas
+	// and accounting. Given to the engine in its environment rather than on
+	// its command line, which every user on the node can read.
+	APIKey string `json:"api_key,omitempty"`
+	// Threads is the CPU cores placement reserved for a shard with no GPU,
+	// and so what its engine computes on. Zero leaves it to the engine's own
+	// configuration.
+	Threads int `json:"threads,omitempty"`
+
 	// Pool and Fabric are control plane concepts the agent cannot derive. They
-	// are carried so the agent can report them back after a restart, and are
-	// not used when building a command.
+	// are carried so the agent can report them back after a restart; Fabric
+	// also says, with an "nvl:" prefix, that the replica is inside a
+	// multi-node NVLink domain, which changes NCCL's environment.
 	Pool   string
 	Fabric string
+	// ProfileFP identifies the version of the profile this shard was started
+	// from. Carried for the same reason as Pool and Fabric, and never read: it
+	// lets the control plane tell, after its own restart, whether a shard that
+	// is still running matches the profile the catalog holds now.
+	ProfileFP string
 }
 
 // Leader reports whether this shard serves HTTP for the replica.
@@ -146,6 +169,16 @@ func NewRegistry(cfg Config) *Registry {
 	return r
 }
 
+// UseInstalled points an engine at a build the agent installed, and reports
+// whether the engine takes one.
+func (r *Registry) UseInstalled(name, path string) bool {
+	u, ok := r.engines[name].(interface{ UseInstalled(string) })
+	if ok {
+		u.UseInstalled(path)
+	}
+	return ok
+}
+
 // Add registers an engine, replacing any engine of the same name.
 func (r *Registry) Add(e Engine) { r.engines[e.Name()] = e }
 
@@ -178,6 +211,10 @@ type Config struct {
 
 // --- shared helpers ---
 
+// NCCLEnv is the environment a shard with this spec is given, for a tool that
+// shows an operator what an engine would see.
+func NCCLEnv(s ShardSpec) map[string]string { return baseEnv(s) }
+
 // baseEnv builds the environment every engine shares.
 //
 // This is the part of §5.3 the plan marks critical. The rules encoded here:
@@ -200,11 +237,24 @@ func baseEnv(s ShardSpec) map[string]string {
 	if len(s.HCAs) > 0 {
 		env["NCCL_IB_HCA"] = strings.Join(s.HCAs, ",")
 		env["NCCL_IB_DISABLE"] = "0"
+		// RoCE: the index of the routable RoCE v2 entry. NCCL's default is a
+		// v1 link-local one on most hosts, and a job on it hangs.
+		if s.GIDIndex != nil {
+			env["NCCL_IB_GID_INDEX"] = strconv.Itoa(*s.GIDIndex)
+		}
 	} else {
 		// Single node, or a node with no fabric. Either way IB is not the path.
 		env["NCCL_IB_DISABLE"] = "1"
 	}
 
+	// Placed inside a multi-node NVLink domain: NCCL is to carry the
+	// replica's collectives over NVLink between its nodes, which it only
+	// does when told to, and needs IMEX for.
+	if s.NNodes > 1 && strings.HasPrefix(s.Fabric, "nvl:") {
+		env["NCCL_MNNVL_ENABLE"] = "1"
+		env["NCCL_CUMEM_ENABLE"] = "1"
+	}
+
 	if s.MgmtInterface != "" {
 		env["NCCL_SOCKET_IFNAME"] = s.MgmtInterface
 		env["GLOO_SOCKET_IFNAME"] = s.MgmtInterface
@@ -213,6 +263,11 @@ func baseEnv(s ShardSpec) map[string]string {
 	// Profile overrides win: they are the escape hatch for the per-cluster
 	// tuning no adapter can guess.
 	for k, v := range s.Profile.Env {
+		// Checked again here, not only where the model was defined: this is
+		// the process about to run as root, whatever sent the profile.
+		if bad, _ := UnsafeEnv(k); bad {
+			continue
+		}
 		env[k] = v
 	}
 	return env
@@ -321,3 +376,10 @@ func quantizationOf(args []string) string {
 	}
 	return ""
 }
+
+// Adapter is a LoRA adapter an engine loads beside its model. Path is a
+// source until the agent has fetched it, a local file after.
+type Adapter struct {
+	Name string `json:"name"`
+	Path string `json:"path"`
+}

internal/agent/engines/engines_test.go

@@ -3,6 +3,7 @@ package engines
 import (
 	"fmt"
 	"slices"
+	"strconv"
 	"strings"
 	"testing"
 )
@@ -243,14 +244,41 @@ func TestVLLMRayWorkerRunsOnlyRay(t *testing.T) {
 }
 
 func TestVLLMMPMode(t *testing.T) {
+	// vLLM's multi-node launch: every node the same serve with its rank and
+	// the leader's address, workers headless. Workers used to be given the
+	// leader's address as their own VLLM_HOST_IP, and no rank at all.
 	v := &VLLM{cfg: VLLMConfig{LaunchMode: "mp"}}
-	cmd := buildOne(t, v, multiNodeSpec(0))
-	if argValue(cmd.Args, "--distributed-executor-backend") != "mp" {
-		t.Errorf("backend not set: %v", cmd.Args)
+	for _, rank := range []int{0, 1} {
+		cmd := buildOne(t, v, multiNodeSpec(rank))
+		for flag, want := range map[string]string{
+			"--distributed-executor-backend": "mp", "--nnodes": "2",
+			"--node-rank": strconv.Itoa(rank), "--master-addr": "10.0.0.10", "--master-port": "20001",
+		} {
+			if got := argValue(cmd.Args, flag); got != want {
+				t.Errorf("rank %d: %s = %q, want %q (%v)", rank, flag, got, want, cmd.Args)
+			}
+		}
+		if headless := slices.Contains(cmd.Args, "--headless"); headless != (rank > 0) {
+			t.Errorf("rank %d: headless = %v", rank, headless)
+		}
+		if ip, ok := envMap(cmd.Env)["VLLM_HOST_IP"]; ok && ip == "10.0.0.10" && rank > 0 {
+			t.Errorf("a worker was told the leader's address is its own")
+		}
 	}
-	env := envMap(cmd.Env)
-	if env["VLLM_HOST_IP"] != "10.0.0.10" || env["VLLM_PORT"] != "20001" {
-		t.Errorf("dist init not configured: %q %q", env["VLLM_HOST_IP"], env["VLLM_PORT"])
+}
+
+func TestRoCEGetsItsGIDIndex(t *testing.T) {
+	s := multiNodeSpec(0)
+	idx := 3
+	s.GIDIndex = &idx
+	env := envMap(buildOne(t, &VLLM{cfg: VLLMConfig{LaunchMode: "mp"}}, s).Env)
+	if env["NCCL_IB_GID_INDEX"] != "3" {
+		t.Errorf("NCCL_IB_GID_INDEX = %q", env["NCCL_IB_GID_INDEX"])
+	}
+	// InfiniBand needs none, and must not be given one.
+	env = envMap(buildOne(t, &VLLM{cfg: VLLMConfig{LaunchMode: "mp"}}, multiNodeSpec(0)).Env)
+	if _, ok := env["NCCL_IB_GID_INDEX"]; ok {
+		t.Error("an InfiniBand shard was given a GID index")
 	}
 }
 
@@ -423,3 +451,28 @@ func TestQuantizationOf(t *testing.T) {
 		t.Errorf("a dangling flag should yield nothing, got %q", got)
 	}
 }
+
+func TestAReplicaInAnNVLinkDomainIsToldToUseIt(t *testing.T) {
+	s := multiNodeSpec(0)
+	s.Fabric = "nvl:76b1ad76-7c24-4b6b-9f6c-2a1e0d8c11aa/7"
+	env := envMap(buildOne(t, &VLLM{cfg: VLLMConfig{LaunchMode: "mp"}}, s).Env)
+	if env["NCCL_MNNVL_ENABLE"] != "1" {
+		t.Errorf("NCCL_MNNVL_ENABLE = %q", env["NCCL_MNNVL_ENABLE"])
+	}
+	if env := envMap(buildOne(t, &VLLM{cfg: VLLMConfig{LaunchMode: "mp"}}, multiNodeSpec(0)).Env); env["NCCL_MNNVL_ENABLE"] != "" {
+		t.Error("a replica on InfiniBand was told to use multi-node NVLink")
+	}
+}
+
+func TestMultiNodeVLLMDefaultsToItsOwnLaunch(t *testing.T) {
+	// mp is what vLLM documents and needs nothing but vLLM; Ray was the
+	// default and was never checked against a real cluster.
+	cmds, err := (&VLLM{}).Commands(multiNodeSpec(1))
+	if err != nil {
+		t.Fatal(err)
+	}
+	if len(cmds) != 1 || cmds[0].Name != "vllm" || argValue(cmds[0].Args, "--distributed-executor-backend") != "mp" ||
+		!slices.Contains(cmds[0].Args, "--headless") {
+		t.Errorf("default worker = %+v", cmds)
+	}
+}

internal/agent/engines/llamacpp.go

@@ -2,28 +2,51 @@ package engines
 
 import (
 	"fmt"
+	"os"
+	"path/filepath"
+	"runtime"
+	"sort"
 	"strconv"
 	"strings"
+	"sync/atomic"
 )
 
 // LlamaCppConfig locates the llama.cpp installation.
 type LlamaCppConfig struct {
 	// Bin is the llama-server binary. It is the server, not the CLI: the
 	// gateway talks OpenAI over HTTP and needs the one that serves it.
+	// Empty means the agent installs llama.cpp itself, at the version the
+	// control plane asks for. Setting it opts out of that.
 	Bin string `koanf:"bin"`
 	// Parallel is how many requests a server handles at once, when a profile
 	// does not say. llama.cpp defaults to one, which would make every second
 	// request wait behind the first no matter what the router decided.
 	Parallel int `koanf:"parallel"`
+	// Threads is how many CPU threads a shard with no GPU computes on. Zero
+	// takes half this machine's CPUs.
+	//
+	// Not all of them, which is llama.cpp's own default: two busy servers
+	// that each spin on every CPU fight for them, and on an 8-vCPU node
+	// that made 7B generation fall from 10 tokens a second to under 1 each.
+	// At half, one server alone is about a third slower and two together
+	// keep 5.5 each. A node that only ever runs one CPU replica does better
+	// with all of them; say so here.
+	Threads int `koanf:"threads"`
 }
 
+// numCPU is the CPUs this process may run on; a variable for tests.
+var numCPU = runtime.NumCPU
+
 // LlamaCpp builds llama.cpp launch commands.
 //
 // It is the engine that makes a cluster without GPUs useful: llama.cpp runs a
 // small model on CPU at a speed that is fine for testing the whole path, and it
 // serves the same OpenAI API as the others, so nothing downstream can tell the
 // difference.
-type LlamaCpp struct{ cfg LlamaCppConfig }
+type LlamaCpp struct {
+	cfg       LlamaCppConfig
+	installed atomic.Pointer[string]
+}
 
 // Name identifies the engine in the catalog.
 func (*LlamaCpp) Name() string { return "llamacpp" }
@@ -65,14 +88,15 @@ func (*LlamaCpp) Validate(p Profile, gpus []GPUInfo) error {
 
 // Commands returns the processes for one shard, which is always one process.
 func (l *LlamaCpp) Commands(s ShardSpec) ([]Command, error) {
-	bin := l.cfg.Bin
-	if bin == "" {
-		bin = "llama-server"
-	}
+	bin := l.bin()
 	p := s.Profile
 
+	model, err := ggufIn(p.ModelPath)
+	if err != nil {
+		return nil, err
+	}
 	args := []string{
-		"--model", p.ModelPath,
+		"--model", model,
 		"--host", "0.0.0.0",
 		"--port", strconv.Itoa(s.HTTPPort),
 		// Without this there is no /metrics, and an endpoint the router cannot
@@ -108,10 +132,35 @@ func (l *LlamaCpp) Commands(s ShardSpec) ([]Command, error) {
 	if len(s.GPUIndices) > 0 && !hasFlag(p.Args, "--n-gpu-layers", "-ngl") {
 		args = append(args, "--n-gpu-layers", "999")
 	}
+	if len(s.GPUIndices) == 0 && !hasFlag(p.Args, "--threads", "-t") {
+		// What placement reserved first: it is what the node's other
+		// replicas were placed around.
+		n := s.Threads
+		if n <= 0 {
+			n = l.cfg.Threads
+		}
+		if n <= 0 {
+			n = max(numCPU()/2, 1)
+		}
+		args = append(args, "--threads", strconv.Itoa(n))
+	}
+	// Loaded at start, applied per request: a request that names an adapter
+	// carries "lora": [{"id": i, "scale": s}], and one that names none is the
+	// base model, since every adapter starts at scale zero.
+	if len(p.Adapters) > 0 {
+		args = append(args, "--lora-init-without-apply")
+		for _, a := range p.Adapters {
+			args = append(args, "--lora", a.Path)
+		}
+	}
 	args = append(args, p.Args...)
 
+	env := baseEnv(s)
+	if s.APIKey != "" {
+		env["LLAMA_API_KEY"] = s.APIKey
+	}
 	return []Command{{
-		Name: "llama-server", Path: bin, Args: args, Env: materializeEnv(baseEnv(s)),
+		Name: "llama-server", Path: bin, Args: args, Env: materializeEnv(env),
 	}}, nil
 }
 
@@ -127,3 +176,61 @@ func hasFlag(args []string, names ...string) bool {
 	}
 	return false
 }
+
+// ggufIn resolves a model path that may be a directory.
+//
+// A multi-file version arrives as the directory it was downloaded into, because
+// that is the only shape a set can take. llama.cpp wants a file: given the
+// first part of a split GGUF it finds the rest beside it, so the first part is
+// what it gets.
+//
+// A path that is already a file is returned untouched, which is every
+// single-file model and every hand-staged one.
+func ggufIn(path string) (string, error) {
+	st, err := os.Stat(path)
+	if err != nil || !st.IsDir() {
+		// Not a directory, or not there at all — the second is the engine's to
+		// report, with the message it would have given anyway.
+		return path, nil //nolint:nilerr // a missing path is the engine's error to give
+	}
+	entries, err := os.ReadDir(path)
+	if err != nil {
+		return "", err
+	}
+	var names []string
+	for _, e := range entries {
+		if !e.IsDir() && strings.HasSuffix(strings.ToLower(e.Name()), ".gguf") {
+			names = append(names, e.Name())
+		}
+	}
+	if len(names) == 0 {
+		return "", fmt.Errorf("%s holds no .gguf file", path)
+	}
+	// Sorted, so a split model's first part comes first: the parts are named
+	// "-00001-of-00004" and lexical order is shard order.
+	sort.Strings(names)
+	return filepath.Join(path, names[0]), nil
+}
+
+// UseInstalled points the engine at a build the agent installed and checked.
+// It takes precedence over everything else, because it is the only one that
+// was seen to run.
+func (l *LlamaCpp) UseInstalled(path string) { l.installed.Store(&path) }
+
+// Configured reports whether the agent's configuration names a binary. A
+// named binary is the operator's deliberate choice; one merely found on PATH
+// is not, and is not reported as installed.
+func (l *LlamaCpp) Configured() bool { return l.cfg.Bin != "" }
+
+// bin is the llama-server to run: the installed build, else the configured
+// one, else whatever PATH has — which is only ever reached by a node not
+// provisioned by the agent and not told where its engine is.
+func (l *LlamaCpp) bin() string {
+	if p := l.installed.Load(); p != nil {
+		return *p
+	}
+	if l.cfg.Bin != "" {
+		return l.cfg.Bin
+	}
+	return "llama-server"
+}

internal/agent/engines/llamacpp_test.go

@@ -1,6 +1,8 @@
 package engines
 
 import (
+	"os"
+	"path/filepath"
 	"slices"
 	"strings"
 	"testing"
@@ -187,3 +189,137 @@ func TestLlamaCppLetsAProfileTurnTheTemplateOff(t *testing.T) {
 		t.Errorf("the profile said --no-jinja and got both: %v", cmds[0].Args)
 	}
 }
+
+func TestASetDirectoryResolvesToItsFirstPart(t *testing.T) {
+	dir := t.TempDir()
+	// Deliberately out of order on disk, because readdir order is not shard
+	// order and the first part is the one llama.cpp must be given.
+	for _, n := range []string{
+		"qwen-q4_k_m-00003-of-00003.gguf",
+		"qwen-q4_k_m-00001-of-00003.gguf",
+		"qwen-q4_k_m-00002-of-00003.gguf",
+	} {
+		if err := os.WriteFile(filepath.Join(dir, n), []byte("x"), 0o644); err != nil {
+			t.Fatal(err)
+		}
+	}
+
+	got, err := ggufIn(dir)
+	if err != nil {
+		t.Fatal(err)
+	}
+	// Given the first part llama.cpp finds the rest beside it; given the third
+	// it loads a third of a model.
+	if filepath.Base(got) != "qwen-q4_k_m-00001-of-00003.gguf" {
+		t.Errorf("expected the first shard, got %s", filepath.Base(got))
+	}
+}
+
+func TestAPlainFilePathIsUntouched(t *testing.T) {
+	// Every single-file model and every hand-staged one takes this path.
+	p := filepath.Join(t.TempDir(), "model.gguf")
+	if err := os.WriteFile(p, []byte("x"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	if got, _ := ggufIn(p); got != p {
+		t.Errorf("expected the path unchanged, got %s", got)
+	}
+}
+
+func TestADirectoryWithNoWeightsSaysSo(t *testing.T) {
+	dir := t.TempDir()
+	if err := os.WriteFile(filepath.Join(dir, "config.json"), []byte("{}"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	if _, err := ggufIn(dir); err == nil {
+		t.Error("a directory with no .gguf in it is not a model llama.cpp can load")
+	}
+}
+
+func TestAMissingPathIsLeftToTheEngine(t *testing.T) {
+	// The engine's own "no such file" names the path and the flag it came
+	// from, which is more use than anything this could say.
+	p := "/nonexistent/model.gguf"
+	got, err := ggufIn(p)
+	if err != nil || got != p {
+		t.Errorf("expected it passed through, got %q %v", got, err)
+	}
+}
+
+func TestLlamaCppIsGivenItsKeyOutOfSightOfTheProcessList(t *testing.T) {
+	// The command line is readable by every user on the node; the
+	// environment of a root process is not.
+	cmds, err := (&LlamaCpp{}).Commands(ShardSpec{HTTPPort: 8080, APIKey: "k1",
+		Profile: Profile{ModelPath: "/m.gguf"}})
+	if err != nil {
+		t.Fatal(err)
+	}
+	if slices.Contains(cmds[0].Args, "k1") || slices.Contains(cmds[0].Args, "--api-key") {
+		t.Errorf("the key is on the command line: %v", cmds[0].Args)
+	}
+	if !slices.Contains(cmds[0].Env, "LLAMA_API_KEY=k1") {
+		t.Error("llama-server was not given its key")
+	}
+}
+
+func TestACPUShardLeavesHalfTheMachineForAnother(t *testing.T) {
+	prev := numCPU
+	numCPU = func() int { return 8 }
+	t.Cleanup(func() { numCPU = prev })
+
+	threads := func(l *LlamaCpp, s ShardSpec) (string, bool) {
+		cmds, err := l.Commands(s)
+		if err != nil {
+			t.Fatal(err)
+		}
+		return flagValue(cmds[0].Args, "--threads")
+	}
+	if v, _ := threads(&LlamaCpp{}, llamaSpec(nil)); v != "4" {
+		t.Errorf("--threads %q on 8 CPUs, expected 4", v)
+	}
+	if v, _ := threads(&LlamaCpp{cfg: LlamaCppConfig{Threads: 8}}, llamaSpec(nil)); v != "8" {
+		t.Errorf("--threads %q with 8 configured", v)
+	}
+	if _, ok := threads(&LlamaCpp{}, llamaSpec([]int{0})); ok {
+		t.Error("a GPU shard was given a thread count")
+	}
+	s := llamaSpec(nil)
+	s.Profile.Args = []string{"-t", "2"}
+	cmds, _ := (&LlamaCpp{}).Commands(s)
+	if slices.Contains(cmds[0].Args, "--threads") {
+		t.Error("the profile's own -t was overridden")
+	}
+	numCPU = func() int { return 1 }
+	if v, _ := threads(&LlamaCpp{}, llamaSpec(nil)); v != "1" {
+		t.Errorf("--threads %q on one CPU", v)
+	}
+}
+
+func TestTheCoresPlacementReservedAreTheThreads(t *testing.T) {
+	s := llamaSpec(nil)
+	s.Threads = 3
+	cmds, err := (&LlamaCpp{cfg: LlamaCppConfig{Threads: 8}}).Commands(s)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if v, _ := flagValue(cmds[0].Args, "--threads"); v != "3" {
+		t.Errorf("--threads %q; what placement reserved should win over the node's setting", v)
+	}
+}
+
+func TestAdaptersAreLoadedAtStartAndAppliedToNoneByDefault(t *testing.T) {
+	s := llamaSpec(nil)
+	s.Profile.Adapters = []Adapter{{Name: "sql", Path: "/w/sql.gguf"}, {Name: "docs", Path: "/w/docs.gguf"}}
+	cmds, err := (&LlamaCpp{}).Commands(s)
+	if err != nil {
+		t.Fatal(err)
+	}
+	args := strings.Join(cmds[0].Args, " ")
+	if !strings.Contains(args, "--lora-init-without-apply --lora /w/sql.gguf --lora /w/docs.gguf") {
+		t.Errorf("args %s: both adapters, in order, loaded unapplied", args)
+	}
+	plain, _ := (&LlamaCpp{}).Commands(llamaSpec(nil))
+	if strings.Contains(strings.Join(plain[0].Args, " "), "--lora") {
+		t.Error("a model with no adapters was given lora flags")
+	}
+}

internal/agent/engines/probe.go

@@ -0,0 +1,216 @@
+package engines
+
+import (
+	"context"
+	"errors"
+	"os/exec"
+	"regexp"
+	"sort"
+	"strings"
+	"time"
+)
+
+// Unknown is the version reported for an engine that is installed and would
+// not say which version it is.
+//
+// A value, not an absent key, because the two mean different things and
+// placement reads the keys: absent is "this node cannot run it", Unknown is
+// "it is here and the probe could not get a number out of it". Collapsing them
+// would make a slow engine look like a missing one and take a node out of
+// scheduling for being slow to answer --version.
+const Unknown = "unknown"
+
+// Prober is an engine that can say whether it is installed on this machine.
+//
+// Optional, and asserted for rather than part of Engine: an adapter that has no
+// way to check should report nothing rather than be forced to invent an answer.
+type Prober interface {
+	// Probe returns the engine's version, or an error when it is not installed.
+	Probe(ctx context.Context) (string, error)
+}
+
+// Probe asks every registered engine whether it is here.
+//
+// The result is what the node reports at registration, and its key set is what
+// placement filters on. Until this existed nobody knew: the registry registers
+// every adapter whether or not its binary exists, so a node with no vLLM
+// accepted a vLLM shard, registered healthy, and failed the shard minutes later
+// with "executable file not found" — in the engine's own words, on a machine
+// nobody was watching.
+//
+// Bounded, because it runs on the path to registering and an agent that cannot
+// reach the control plane is worse than one that reports an incomplete list.
+func (r *Registry) Probe(ctx context.Context) map[string]string {
+	out := map[string]string{}
+	for name, e := range r.engines {
+		p, ok := e.(Prober)
+		if !ok {
+			continue
+		}
+		v, err := p.Probe(ctx)
+		if err != nil {
+			continue
+		}
+		if v == "" {
+			v = Unknown
+		}
+		out[name] = v
+	}
+	return out
+}
+
+// Names of the engines in a probe result, sorted.
+func ProbedNames(m map[string]string) []string {
+	out := make([]string, 0, len(m))
+	for n := range m {
+		out = append(out, n)
+	}
+	sort.Strings(out)
+	return out
+}
+
+// probeTimeout bounds one engine's version check.
+//
+// Generous, because vLLM's --version imports torch and that is seconds on a
+// cold page cache, and mean, because this is startup: an engine that has not
+// answered in this long is reported as present-with-unknown-version rather
+// than waited for.
+const probeTimeout = 8 * time.Second
+
+// run executes a version command, saying whether the program was even there.
+//
+// The two failures are different answers and the caller has to tell them
+// apart. A binary that is absent is an engine this node cannot run. A binary
+// that is present and exits non-zero is an engine it probably can — unless the
+// command was a test of whether a module imports, where a non-zero exit is the
+// whole answer.
+func run(ctx context.Context, bin string, args ...string) (out string, found bool, ok bool) {
+	path, err := exec.LookPath(bin)
+	if err != nil {
+		return "", false, false
+	}
+	ctx, cancel := context.WithTimeout(ctx, probeTimeout)
+	defer cancel()
+
+	// Combined, not just stdout: engines print their version to either, and one
+	// that chose stderr would otherwise read as having no version at all.
+	b, err := exec.CommandContext(ctx, path, args...).CombinedOutput()
+	return string(b), true, err == nil
+}
+
+// installedBinary reports an engine that is its own executable.
+//
+// It resolves the path and runs nothing. The first version of this ran the
+// engine with --version, which is fine for an engine and not fine for the
+// wrapper scripts people put in front of one: a script that shifts past flags
+// it does not recognise and then execs the server will start an inference
+// server, at agent startup, on every node. The agent's own test suite installs
+// exactly such a script, and caught it.
+//
+// So the version is not read here. Presence is what placement needs, and what
+// the version is for — telling whether an engine still accepts the flags llmgw
+// builds — is `llmgw ctl engine-check`, which is run deliberately by somebody
+// who meant to run the engine.
+func installedBinary(bin string) (string, error) {
+	if _, err := exec.LookPath(bin); err != nil {
+		return "", errNotInstalled
+	}
+	return Unknown, nil
+}
+
+// versionOfModule probes an engine that is a module inside an interpreter.
+//
+// This one does execute, and safely: the command is llmgw's own one-line
+// import, not the engine's launcher, so there is no wrapper to shift past
+// flags and start a server. There is also no alternative — a module has no
+// path to look up.
+//
+// A non-zero exit is the answer: python3 exists on almost every machine, so
+// presence of the interpreter says nothing. Treating a failed import as
+// "installed" reported SGLang on every node in a fleet that had none, which
+// would take nodes into scheduling for an engine they cannot run — the exact
+// failure this whole change exists to stop.
+func versionOfModule(ctx context.Context, python string, args ...string) (string, error) {
+	out, found, ok := run(ctx, python, args...)
+	if !found || !ok {
+		return "", errNotInstalled
+	}
+	return firstVersion(out), nil
+}
+
+var errNotInstalled = errors.New("not installed on this node")
+
+var versionPattern = regexp.MustCompile(`\b\d+\.\d+(\.\d+)?\S*`)
+
+// firstVersion is the first thing in the output that looks like a version.
+//
+// Pattern rather than parsing each engine's format: they print "vllm 0.6.3",
+// "version: b4123 (…)", and a paragraph with the number in the middle, and all
+// of those change between releases. What this is for is telling one build from
+// another, which the first number does.
+func firstVersion(s string) string {
+	for _, line := range strings.Split(s, "\n") {
+		if m := versionPattern.FindString(line); m != "" {
+			return strings.TrimRight(m, ",;)")
+		}
+	}
+	return Unknown
+}
+
+// Configured is an engine whose location the agent's configuration names.
+type Configured interface {
+	Configured() bool
+}
+
+// ProbeConfigured probes only the engines the configuration names.
+//
+// Probe looks everywhere — PATH, the default interpreter — and reports what
+// it finds, which is exactly the trust in whatever happens to be on a machine
+// that installing engines replaces. A node the agent provisions reports what
+// it installed, plus what the operator named on purpose; nothing else.
+func (r *Registry) ProbeConfigured(ctx context.Context) map[string]string {
+	out := map[string]string{}
+	for name, e := range r.engines {
+		c, ok := e.(Configured)
+		if !ok || !c.Configured() {
+			continue
+		}
+		p, ok := e.(Prober)
+		if !ok {
+			continue
+		}
+		if v, err := p.Probe(ctx); err == nil {
+			if v == "" {
+				v = Unknown
+			}
+			out[name] = v
+		}
+	}
+	return out
+}
+
+// Probe reports whether the llama.cpp server is installed.
+func (l *LlamaCpp) Probe(context.Context) (string, error) {
+	return installedBinary(l.bin())
+}
+
+// Configured reports whether the configuration names vLLM.
+func (v *VLLM) Configured() bool { return v.cfg.Bin != "" }
+
+// Configured reports whether the configuration names SGLang's interpreter.
+func (g *SGLang) Configured() bool { return g.cfg.Python != "" }
+
+// Probe reports whether vLLM is installed.
+func (v *VLLM) Probe(context.Context) (string, error) {
+	return installedBinary(v.bin())
+}
+
+// Probe reports the SGLang version.
+//
+// Through the interpreter, because SGLang is a module rather than a binary:
+// asking the shell for "sglang" finds nothing on a machine that can run it
+// perfectly well.
+func (g *SGLang) Probe(ctx context.Context) (string, error) {
+	return versionOfModule(ctx, g.python(), "-c",
+		"import sglang, sys; sys.stdout.write(getattr(sglang, '__version__', ''))")
+}

internal/agent/engines/probe_test.go

@@ -0,0 +1,107 @@
+package engines
+
+import (
+	"context"
+	"os"
+	"path/filepath"
+	"testing"
+)
+
+// script writes an executable that stands in for an engine.
+func script(t *testing.T, name, body string) string {
+	t.Helper()
+	dir := t.TempDir()
+	p := filepath.Join(dir, name)
+	if err := os.WriteFile(p, []byte("#!/bin/sh\n"+body+"\n"), 0o755); err != nil {
+		t.Fatal(err)
+	}
+	return p
+}
+
+func TestAnEngineThatIsNotInstalledIsAbsentFromTheReport(t *testing.T) {
+	r := NewRegistry(Config{})
+	r.engines = map[string]Engine{
+		"vllm": &VLLM{cfg: VLLMConfig{Bin: "/nonexistent/vllm"}},
+	}
+	// Absent, not present-with-unknown-version. Placement reads the keys, and
+	// a node listed for an engine it cannot run is the failure being fixed.
+	if got := r.Probe(t.Context()); len(got) != 0 {
+		t.Errorf("expected nothing, got %v", got)
+	}
+}
+
+func TestProbingNeverRunsTheEngine(t *testing.T) {
+	// A wrapper that shifts past flags it does not recognise and then starts
+	// the server. People put these in front of engines, and the agent's own
+	// tests install one. Probing with --version would launch an inference
+	// server at agent startup, on every node.
+	marker := filepath.Join(t.TempDir(), "ran")
+	bin := script(t, "fake-vllm", "touch "+marker+"; sleep 60")
+
+	r := NewRegistry(Config{})
+	r.engines = map[string]Engine{"vllm": &VLLM{cfg: VLLMConfig{Bin: bin}}}
+
+	got := r.Probe(t.Context())
+	if _, ok := got["vllm"]; !ok {
+		t.Error("a binary that is there should be reported present")
+	}
+	if _, err := os.Stat(marker); err == nil {
+		t.Fatal("the engine was executed by a probe")
+	}
+}
+
+func TestAModuleThatWillNotImportIsNotInstalled(t *testing.T) {
+	// python3 is on almost every machine, so the interpreter being there says
+	// nothing about SGLang. Reporting it present put every node in a fleet
+	// forward for an engine none of them had.
+	bin := script(t, "python3", "exit 1")
+	r := NewRegistry(Config{})
+	r.engines = map[string]Engine{"sglang": &SGLang{cfg: SGLangConfig{Python: bin}}}
+
+	if got := r.Probe(t.Context()); len(got) != 0 {
+		t.Errorf("a failed import means not installed, got %v", got)
+	}
+}
+
+func TestAModuleThatImportsIsReportedWithItsVersion(t *testing.T) {
+	bin := script(t, "python3", `echo "0.4.1"`)
+	r := NewRegistry(Config{})
+	r.engines = map[string]Engine{"sglang": &SGLang{cfg: SGLangConfig{Python: bin}}}
+
+	got := r.Probe(t.Context())
+	if got["sglang"] != "0.4.1" {
+		t.Errorf("expected the version, got %v", got)
+	}
+}
+
+func TestAVersionIsPickedOutOfWhateverIsPrinted(t *testing.T) {
+	// Only the module probe reaches this now, and a Python package version is
+	// dotted. A bare build number is deliberately not matched: it would turn a
+	// stray number in a line of prose into a version.
+	for in, want := range map[string]string{
+		"0.4.1":                 "0.4.1",
+		"sglang 0.4.post1":      "0.4.post1",
+		"warning: noise\n0.4.1": "0.4.1",
+		"":                      Unknown,
+		"no numbers here":       Unknown,
+		"just 4123":             Unknown,
+	} {
+		if got := firstVersion(in); got != want {
+			t.Errorf("firstVersion(%q) = %q, want %q", in, got, want)
+		}
+	}
+}
+
+func TestProbingIsBoundedByTheContext(t *testing.T) {
+	ctx, cancel := context.WithCancel(context.Background())
+	cancel()
+	bin := script(t, "python3", "sleep 60")
+	r := NewRegistry(Config{})
+	r.engines = map[string]Engine{"sglang": &SGLang{cfg: SGLangConfig{Python: bin}}}
+
+	// Registration waits on this. An agent that cannot reach the control plane
+	// is worse than one reporting an incomplete list.
+	if got := r.Probe(ctx); len(got) != 0 {
+		t.Errorf("a cancelled probe should report nothing, got %v", got)
+	}
+}

internal/agent/engines/sglang.go

@@ -1,6 +1,9 @@
 package engines
 
-import "strconv"
+import (
+	"strconv"
+	"sync/atomic"
+)
 
 // SGLangConfig locates the SGLang installation.
 type SGLangConfig struct {
@@ -11,7 +14,25 @@ type SGLangConfig struct {
 }
 
 // SGLang builds SGLang launch commands.
-type SGLang struct{ cfg SGLangConfig }
+type SGLang struct {
+	cfg       SGLangConfig
+	installed atomic.Pointer[string]
+}
+
+// UseInstalled points the engine at the interpreter of the environment the
+// agent installed and checked, ahead of the configuration.
+func (g *SGLang) UseInstalled(path string) { g.installed.Store(&path) }
+
+// python is the interpreter to launch SGLang with.
+func (g *SGLang) python() string {
+	if p := g.installed.Load(); p != nil {
+		return *p
+	}
+	if g.cfg.Python != "" {
+		return g.cfg.Python
+	}
+	return "python3"
+}
 
 // Name identifies the engine in the catalog.
 func (*SGLang) Name() string { return "sglang" }
@@ -55,10 +76,7 @@ func (g *SGLang) Commands(s ShardSpec) ([]Command, error) {
 }
 
 func (g *SGLang) serverCommand(s ShardSpec) (Command, error) {
-	python := g.cfg.Python
-	if python == "" {
-		python = "python3"
-	}
+	python := g.python()
 	p := s.Profile
 
 	args := []string{"-m", "sglang.launch_server",
@@ -78,6 +96,11 @@ func (g *SGLang) serverCommand(s ShardSpec) (Command, error) {
 		)
 	}
 	args = append(args, p.Args...)
+	// SGLang reads its key only from the command line, so on a node with
+	// other users it is visible to them in the process list.
+	if s.APIKey != "" && s.Leader() {
+		args = append(args, "--api-key", s.APIKey)
+	}
 
 	return Command{Name: "sglang", Path: python, Args: args, Env: materializeEnv(baseEnv(s))}, nil
 }

internal/agent/engines/unsafeenv.go

@@ -0,0 +1,30 @@
+package engines
+
+import "strings"
+
+// UnsafeEnv reports whether a profile may not set an environment variable,
+// and why.
+//
+// A profile's env is the escape hatch for tuning no adapter can guess —
+// CUDA_*, NCCL_*, VLLM_*, HF_* — and it reaches an engine the agent starts as
+// root. Some variables are not tuning but a way to run other code in that
+// process: the dynamic loader's LD_PRELOAD, Python's PYTHONSTARTUP, a shell's
+// BASH_ENV. Set from a model definition, which an admin API or a config file
+// supplies, they would turn "add a model" into "run this as root on every
+// node". They are refused where a model is defined and again where an engine
+// starts.
+func UnsafeEnv(key string) (bool, string) {
+	k := strings.ToUpper(key)
+	switch {
+	case strings.HasPrefix(k, "LD_"), strings.HasPrefix(k, "DYLD_"):
+		return true, "it changes what the dynamic loader loads"
+	case k == "PATH":
+		return true, "it changes which binaries run"
+	case strings.HasPrefix(k, "PYTHON") && k != "PYTHONUNBUFFERED" && k != "PYTHONHASHSEED":
+		return true, "it changes what Python imports or runs at start"
+	case k == "BASH_ENV" || k == "ENV" || k == "PERL5OPT" || k == "PERL5LIB" ||
+		k == "NODE_OPTIONS" || k == "RUBYOPT" || k == "GCONV_PATH" || k == "OPENSSL_CONF":
+		return true, "it makes an interpreter or library load code"
+	}
+	return false, ""
+}

internal/agent/engines/vllm.go

@@ -3,24 +3,48 @@ package engines
 import (
 	"fmt"
 	"strconv"
+	"sync/atomic"
 )
 
 // VLLMConfig locates the vLLM installation and picks how it is launched.
 type VLLMConfig struct {
 	// Bin is the vllm executable.
 	Bin string `koanf:"bin"`
-	// LaunchMode selects the multi-node mechanism: "ray" or "mp".
+	// LaunchMode selects the multi-node mechanism: "mp", the default, or
+	// "ray".
 	//
-	// Which one works depends on the vLLM version, and the flags have changed
-	// more than once (plan §14). Both paths exist so a cluster can pick the one
-	// its version supports instead of waiting for a code change.
+	// mp is what vLLM documents today — every node runs serve with its rank,
+	// the workers headless — and needs nothing but vLLM. ray is the older way,
+	// kept for a vLLM too old for mp's multi-node flags; it needs Ray on every
+	// node and has not been checked against a real cluster.
 	LaunchMode string `koanf:"launch_mode"`
 	// RayBin is used when LaunchMode is "ray".
 	RayBin string `koanf:"ray_bin"`
 }
 
 // VLLM builds vLLM launch commands.
-type VLLM struct{ cfg VLLMConfig }
+type VLLM struct {
+	cfg       VLLMConfig
+	installed atomic.Pointer[string]
+}
+
+// UseInstalled points the engine at the environment the agent installed and
+// checked. It takes precedence over the configuration, as llama.cpp's does:
+// it is the one seen to run. An operator who wants their own sets the
+// control plane's engines.vllm to "".
+func (v *VLLM) UseInstalled(path string) { v.installed.Store(&path) }
+
+// bin is the vllm to run: the installed one, else the configured one, else
+// whatever PATH has.
+func (v *VLLM) bin() string {
+	if p := v.installed.Load(); p != nil {
+		return *p
+	}
+	if v.cfg.Bin != "" {
+		return v.cfg.Bin
+	}
+	return "vllm"
+}
 
 // Name identifies the engine in the catalog.
 func (*VLLM) Name() string { return "vllm" }
@@ -78,10 +102,7 @@ func (v *VLLM) Commands(s ShardSpec) ([]Command, error) {
 
 // serverCommand produces the vLLM server process.
 func (v *VLLM) serverCommand(s ShardSpec) (Command, error) {
-	bin := v.cfg.Bin
-	if bin == "" {
-		bin = "vllm"
-	}
+	bin := v.bin()
 	p := s.Profile
 
 	args := []string{"serve", p.ModelPath,
@@ -100,16 +121,27 @@ func (v *VLLM) serverCommand(s ShardSpec) (Command, error) {
 
 	if s.NNodes > 1 {
 		switch v.cfg.LaunchMode {
-		case "", "ray":
+		case "ray":
 			// Ray is started separately by the supervisor; vLLM then finds the
 			// cluster through the address in the environment.
 			env["VLLM_HOST_IP"] = s.LeaderAddr
 			env["RAY_ADDRESS"] = fmt.Sprintf("%s:%d", s.LeaderAddr, s.LeaderPort)
 			args = append(args, "--distributed-executor-backend", "ray")
-		case "mp":
-			env["VLLM_HOST_IP"] = s.LeaderAddr
-			env["VLLM_PORT"] = strconv.Itoa(s.LeaderPort)
-			args = append(args, "--distributed-executor-backend", "mp")
+		case "", "mp":
+			// vLLM's own multi-node launch: every node runs the same serve
+			// with its rank and the leader's address, and the workers run
+			// headless — no API server, only their GPUs. VLLM_HOST_IP is
+			// this node's address for vLLM to find itself, so it is left
+			// to vLLM; the leader's, which workers used to be given, made
+			// every worker announce itself as the leader.
+			args = append(args, "--distributed-executor-backend", "mp",
+				"--nnodes", strconv.Itoa(s.NNodes),
+				"--node-rank", strconv.Itoa(s.Rank),
+				"--master-addr", s.LeaderAddr,
+				"--master-port", strconv.Itoa(s.LeaderPort))
+			if !s.Leader() {
+				args = append(args, "--headless")
+			}
 		default:
 			return Command{}, fmt.Errorf("vllm: unknown launch_mode %q (want \"ray\" or \"mp\")", v.cfg.LaunchMode)
 		}
@@ -118,13 +150,16 @@ func (v *VLLM) serverCommand(s ShardSpec) (Command, error) {
 	// Profile arguments last, so an operator can override anything above.
 	args = append(args, p.Args...)
 
+	if s.APIKey != "" {
+		env["VLLM_API_KEY"] = s.APIKey
+	}
 	return Command{Name: "vllm", Path: bin, Args: args, Env: materializeEnv(env)}, nil
 }
 
 // rayCommand returns the command that starts or joins the Ray cluster, or
 // ok=false when the launch mode does not use Ray.
 func (v *VLLM) rayCommand(s ShardSpec) (Command, bool) {
-	if s.NNodes <= 1 || (v.cfg.LaunchMode != "" && v.cfg.LaunchMode != "ray") {
+	if s.NNodes <= 1 || v.cfg.LaunchMode != "ray" {
 		return Command{}, false
 	}
 	bin := v.cfg.RayBin
@@ -134,7 +169,10 @@ func (v *VLLM) rayCommand(s ShardSpec) (Command, bool) {
 
 	args := []string{"start", "--block"}
 	if s.Leader() {
-		args = append(args, "--head", "--port", strconv.Itoa(s.LeaderPort))
+		// Ray's head has no authentication: whoever reaches its port can run
+		// code on the node. Its dashboard, and the job API behind it, is the
+		// easiest way in, and nothing here uses it.
+		args = append(args, "--head", "--port", strconv.Itoa(s.LeaderPort), "--include-dashboard=false")
 	} else {
 		args = append(args, "--address", fmt.Sprintf("%s:%d", s.LeaderAddr, s.LeaderPort))
 	}

internal/agent/fabrics.go

@@ -0,0 +1,345 @@
+package agent
+
+import (
+	"context"
+	"fmt"
+	"os/exec"
+	"slices"
+	"strconv"
+	"strings"
+	"time"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
+)
+
+// usableFabrics reports the fabrics this node can actually carry traffic on.
+//
+// Loud about what it drops. A fabric silently missing from a registration looks
+// identical to one never configured, and the operator who typed the adapter
+// name is the only person who can fix it — so the reason is logged here, on the
+// machine where both halves are visible, rather than inferred later from a
+// replica that will not converge.
+func (a *Agent) usableFabrics(inv discovery.Inventory) []inventory.FabricAttachment {
+	out := a.checkFabrics(inv.HCAs, true)
+	a.mu.Lock()
+	a.fabricsSent = fabricsFingerprint(out)
+	a.mu.Unlock()
+	return out
+}
+
+// checkFabrics is the declared fabrics this node can carry traffic on, from
+// the adapters as they are now. Loud logs every problem; the heartbeat checks
+// quietly and speaks only when the answer changes.
+func (a *Agent) checkFabrics(hcas []discovery.HCA, loud bool) []inventory.FabricAttachment {
+	declared := map[string][]string{}
+	for _, f := range a.cfg.Fabrics {
+		declared[f.Fabric] = f.HCAs
+	}
+
+	var out []inventory.FabricAttachment
+	for _, c := range discovery.CheckFabrics(declared, hcas) {
+		if loud {
+			for _, p := range c.Problems {
+				a.log.Error("a declared fabric adapter cannot carry traffic",
+					"fabric", c.Fabric, "problem", p)
+			}
+		}
+		if !c.OK() {
+			if loud {
+				a.log.Error("fabric not advertised; no declared adapter on it can carry traffic",
+					"fabric", c.Fabric,
+					"consequence", "no multi-node replica will be placed here")
+			}
+			continue
+		}
+		out = append(out, inventory.FabricAttachment{
+			Fabric: c.Fabric, HCAs: c.Usable, RateGbps: c.RateGbps,
+		})
+	}
+	return out
+}
+
+// fabricsFingerprint is enough to notice a fabric, an adapter or a rate
+// changing.
+func fabricsFingerprint(fs []inventory.FabricAttachment) string {
+	var b strings.Builder
+	for _, f := range fs {
+		fmt.Fprintf(&b, "%s:%s:%d;", f.Fabric, strings.Join(f.HCAs, ","), f.RateGbps)
+	}
+	return b.String()
+}
+
+// sampleIB re-reads the adapters for the heartbeat: every port's state and
+// counters, and the fabrics again when they no longer match what the control
+// plane was told.
+//
+// Every heartbeat, because it is a few small files per port, and because a
+// port that went down used to be invisible until the agent restarted — the
+// node went on being offered multi-node work that would hang on its first
+// collective.
+func (a *Agent) sampleIB(hb *llmgwv1.Heartbeat) {
+	a.mu.Lock()
+	had := len(a.inv.HCAs) > 0
+	a.mu.Unlock()
+	if !had && len(a.cfg.Fabrics) == 0 {
+		return
+	}
+	hcas, err := discovery.DiscoverIB(a.cfg.Discovery.IBRoot)
+	if err != nil {
+		return
+	}
+	a.attachSM(hcas)
+	for _, h := range hcas {
+		for _, p := range h.Ports {
+			ip := &llmgwv1.IBPort{
+				Hca: h.Name, Port: int32(p.Num), State: p.State, PhysState: p.PhysState,
+				RateGbps: int32(p.RateGbps), LinkLayer: string(p.LinkLayer), Counters: p.Counters,
+				NodeGuid: h.NodeGUID, PortGuid: p.PortGUID, Network: p.Network, GidPrefix: p.GIDPrefix,
+			}
+			a.mu.Lock()
+			if p.SM != nil {
+				ip.SmGuid, ip.SmLid, ip.SmPriority, ip.SmState = p.SM.GUID, int32(p.SM.LID), int32(p.SM.Priority), p.SM.State
+			} else {
+				ip.SmError = a.smErr[smKey(h.Name, p.Num)]
+			}
+			for _, r := range a.sms[smKey(h.Name, p.Num)] {
+				ip.SubnetManagers = append(ip.SubnetManagers, &llmgwv1.SubnetManager{
+					Guid: r.GUID, Lid: int32(r.LID), Priority: int32(r.Priority), State: r.State})
+			}
+			ip.SubnetManagersError = a.smsErr[smKey(h.Name, p.Num)]
+			a.mu.Unlock()
+			hb.IbPorts = append(hb.IbPorts, ip)
+		}
+	}
+	a.mu.Lock()
+	a.inv.HCAs = hcas // a reconnect registers what is true now
+	a.mu.Unlock()
+	if len(a.cfg.Fabrics) == 0 {
+		return
+	}
+	now := a.checkFabrics(hcas, false)
+	fp := fabricsFingerprint(now)
+	a.mu.Lock()
+	changed := fp != a.fabricsSent
+	a.fabricsSent = fp
+	a.mu.Unlock()
+	if !changed {
+		return
+	}
+	// Said once, with the reasons, the moment it happens.
+	a.checkFabrics(hcas, true)
+	a.log.Warn("the fabrics this node can carry traffic on changed", "now", fp)
+	hb.FabricsChanged = true
+	for _, f := range now {
+		hb.Fabrics = append(hb.Fabrics, &llmgwv1.FabricAttachment{
+			Fabric: f.Fabric, Hcas: f.HCAs, RateGbps: int32(f.RateGbps),
+		})
+	}
+}
+
+func smKey(hca string, port int) string { return fmt.Sprintf("%s/%d", hca, port) }
+
+// scanFabric is discovery.ScanFabric, replaced in tests.
+var scanFabric = discovery.ScanFabric
+
+// querySMs is discovery.QuerySMs, replaced in tests.
+var querySMs = func(hca string, port int) ([]discovery.SMRecord, error) {
+	return discovery.QuerySMs(context.Background(), hca, port)
+}
+
+func splitSMKey(k string) (string, int) {
+	i := strings.LastIndexByte(k, '/')
+	port, _ := strconv.Atoi(k[i+1:])
+	return k[:i], port
+}
+
+// querySM is discovery.QuerySM, replaced in tests.
+var querySM = func(hca string, port int) (*discovery.SMInfo, error) {
+	return discovery.QuerySM(context.Background(), hca, port)
+}
+
+// attachSM puts on each active InfiniBand port the subnet manager it sees,
+// asking again once a minute and otherwise reusing the last answer.
+func (a *Agent) attachSM(hcas []discovery.HCA) {
+	a.mu.Lock()
+	stale := time.Since(a.smAsked) > time.Minute || a.sm == nil
+	a.mu.Unlock()
+	if stale {
+		sm, errs := map[string]*discovery.SMInfo{}, map[string]string{}
+		_ = discovery.AttachSM(hcas, func(hca string, port int) (*discovery.SMInfo, error) {
+			got, err := querySM(hca, port)
+			if err != nil {
+				errs[smKey(hca, port)] = err.Error()
+				return nil, err
+			}
+			sm[smKey(hca, port)] = got
+			return got, nil
+		})
+		// Every subnet manager on each fabric, through one port per master:
+		// the same list through every port of one fabric is one query too
+		// many for each.
+		sms, smsErr := map[string][]discovery.SMRecord{}, map[string]string{}
+		asked := map[string]bool{}
+		for k, v := range sm {
+			if asked[v.GUID] {
+				continue
+			}
+			asked[v.GUID] = true
+			hca, port := splitSMKey(k)
+			if recs, err := querySMs(hca, port); err != nil {
+				smsErr[k] = err.Error()
+			} else {
+				sms[k] = recs
+			}
+		}
+		a.mu.Lock()
+		for k, v := range sm {
+			if old := a.sm[k]; old != nil && old.GUID != v.GUID {
+				a.log.Warn("the master subnet manager changed", "port", k, "from", old.GUID, "to", v.GUID)
+			}
+		}
+		a.sm, a.smErr, a.smAsked = sm, errs, time.Now()
+		a.sms, a.smsErr = sms, smsErr
+		a.mu.Unlock()
+		return
+	}
+	a.mu.Lock()
+	defer a.mu.Unlock()
+	for i := range hcas {
+		for j := range hcas[i].Ports {
+			if sm := a.sm[smKey(hcas[i].Name, hcas[i].Ports[j].Num)]; sm != nil && hcas[i].Ports[j].Active() {
+				hcas[i].Ports[j].SM = sm
+			}
+		}
+	}
+}
+
+// ensureDiags installs infiniband-diags on a node with adapters and without
+// it. sminfo is what tells which physical fabric a port is on; without it,
+// two separate fabrics given one name cannot be caught. Installed the way
+// engine dependencies are, from the machine's own package sources.
+func (a *Agent) ensureDiags(ctx context.Context) {
+	a.mu.Lock()
+	hasIB := len(a.inv.HCAs) > 0
+	a.mu.Unlock()
+	if !hasIB || a.prov == nil {
+		return
+	}
+	if _, err := exec.LookPath("sminfo"); err == nil {
+		return
+	}
+	a.log.Info("installing infiniband-diags, to tell which fabric each port is on")
+	pkgs := map[string][]string{"deb": {"infiniband-diags"}, "rpm": {"infiniband-diags"}}
+	if err := a.prov.InstallPackages(ctx, pkgs); err != nil {
+		a.log.Warn("could not install infiniband-diags; fabrics cannot be told apart by their subnet manager",
+			"err", err)
+	}
+}
+
+// FeatureFabricTest is the agent answering FabricTestRequest.
+const FeatureFabricTest = "fabric-test"
+
+// runWriteBW is discovery.RunWriteBW, and haveWriteBW whether it is
+// installed; both replaced in tests.
+var (
+	runWriteBW  = discovery.RunWriteBW
+	haveWriteBW = func() bool { _, err := exec.LookPath("ib_write_bw"); return err == nil }
+)
+
+// fabricTest runs this node's side of a bandwidth test with another.
+func (a *Agent) fabricTest(ctx context.Context, req *llmgwv1.FabricTestRequest) *llmgwv1.FabricTestReply {
+	start := time.Now()
+	reply := &llmgwv1.FabricTestReply{RequestId: req.GetRequestId()}
+	defer func() { reply.TookMs = time.Since(start).Milliseconds() }()
+
+	w := discovery.WriteBW{HCA: req.GetHca(), Port: int(req.GetPort()), GIDIndex: -1,
+		TCPPort: int(req.GetTcpPort()), Seconds: int(req.GetSeconds())}
+	if req.GetRole() == "client" {
+		w.Peer = req.GetPeer()
+	}
+	a.mu.Lock()
+	found := false
+	for _, h := range a.inv.HCAs {
+		for _, p := range h.Ports {
+			if h.Name == w.HCA && p.Num == w.Port {
+				found = true
+				// RoCE needs its v2 GID named, as NCCL does; InfiniBand uses the LID.
+				if p.LinkLayer == discovery.LinkEthernet && p.RoCEv2GID >= 0 {
+					w.GIDIndex = p.RoCEv2GID
+				}
+			}
+		}
+	}
+	a.mu.Unlock()
+	if !found {
+		reply.Error = fmt.Sprintf("this node has no port %s/%d", w.HCA, w.Port)
+		return reply
+	}
+	if !haveWriteBW() && a.prov != nil {
+		a.log.Info("installing perftest, for a bandwidth test with another node")
+		pkgs := map[string][]string{"deb": {"perftest"}, "rpm": {"perftest"}}
+		if err := a.prov.InstallPackages(ctx, pkgs); err != nil {
+			reply.Error = "installing perftest: " + err.Error()
+			return reply
+		}
+	}
+	gbps, out, err := runWriteBW(ctx, w)
+	reply.Gbps = gbps
+	reply.Output = tailLines(out, 12)
+	if err != nil {
+		reply.Error = err.Error()
+	}
+	return reply
+}
+
+func tailLines(s string, n int) string {
+	lines := strings.Split(strings.TrimRight(s, "\n"), "\n")
+	if len(lines) > n {
+		lines = lines[len(lines)-n:]
+	}
+	return strings.Join(lines, "\n")
+}
+
+// discoverGPUFabric and discoverIMEX are replaced in tests.
+var (
+	discoverGPUFabric = discovery.DiscoverGPUFabric
+	discoverIMEX      = discovery.DiscoverIMEX
+)
+
+// sampleNVLink reads, once a minute, which NVLink domain each GPU is joined
+// to and whether IMEX is up. nvidia-smi -q takes a second or two, too long
+// for every heartbeat, and a fabric that was joined an hour ago rarely
+// changes in five seconds — but when a fabric manager restarts it does, for
+// every card at once.
+func (a *Agent) sampleNVLink(hb *llmgwv1.Heartbeat) {
+	a.mu.Lock()
+	gpus := slices.Clone(a.inv.GPUs)
+	due := time.Since(a.nvlAsked) > time.Minute
+	if due {
+		a.nvlAsked = time.Now()
+	}
+	a.mu.Unlock()
+	if len(gpus) == 0 || !due {
+		return
+	}
+	ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
+	defer cancel()
+	fab, err := discoverGPUFabric(ctx, a.cfg.Discovery.NvidiaSMI)
+	if err != nil || len(fab) == 0 {
+		return
+	}
+	discovery.ApplyFabric(gpus, fab)
+	for _, g := range gpus {
+		if g.Fabric == nil {
+			continue
+		}
+		hb.GpuFabric = append(hb.GpuFabric, &llmgwv1.GPUFabricState{
+			Index: int32(g.Index), State: g.Fabric.State, Status: g.Fabric.Status,
+			ClusterUuid: g.Fabric.ClusterUUID, CliqueId: g.Fabric.CliqueID, Bandwidth: g.Fabric.Bandwidth,
+		})
+	}
+	m := discoverIMEX(ctx)
+	hb.Imex = &llmgwv1.IMEX{State: m.State, Channels: int32(m.Channels)}
+}

internal/agent/logring.go

@@ -0,0 +1,84 @@
+package agent
+
+import (
+	"context"
+	"log/slog"
+	"strings"
+	"sync"
+)
+
+// agentLogLines is how much of its own log an agent keeps for the admin page.
+// Enough to cover an engine install or a failed start; the journal keeps the
+// rest for anyone who can reach it.
+const agentLogLines = 2000
+
+// logRing keeps the most recent lines written to it.
+type logRing struct {
+	mu    sync.Mutex
+	lines []string
+	next  int
+	full  bool
+}
+
+func newLogRing(n int) *logRing { return &logRing{lines: make([]string, n)} }
+
+// Write takes whole lines, as a text handler writes them.
+func (r *logRing) Write(p []byte) (int, error) {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	for _, line := range strings.Split(strings.TrimRight(string(p), "\n"), "\n") {
+		r.lines[r.next] = line
+		r.next = (r.next + 1) % len(r.lines)
+		if r.next == 0 {
+			r.full = true
+		}
+	}
+	return len(p), nil
+}
+
+// tail returns up to n of the most recent lines, oldest first.
+func (r *logRing) tail(n int) []string {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	size := r.next
+	if r.full {
+		size = len(r.lines)
+	}
+	n = min(n, size)
+	out := make([]string, 0, n)
+	for i := size - n; i < size; i++ {
+		idx := i
+		if r.full {
+			idx = (r.next + i) % len(r.lines)
+		}
+		out = append(out, r.lines[idx])
+	}
+	return out
+}
+
+// teeHandler sends every record to two handlers: wherever the agent already
+// logs, and the ring the admin page reads.
+type teeHandler struct{ a, b slog.Handler }
+
+func (t teeHandler) Enabled(ctx context.Context, l slog.Level) bool {
+	return t.a.Enabled(ctx, l) || t.b.Enabled(ctx, l)
+}
+
+func (t teeHandler) Handle(ctx context.Context, r slog.Record) error {
+	var err error
+	if t.a.Enabled(ctx, r.Level) {
+		err = t.a.Handle(ctx, r.Clone())
+	}
+	if t.b.Enabled(ctx, r.Level) {
+		_ = t.b.Handle(ctx, r.Clone())
+	}
+	return err
+}
+
+func (t teeHandler) WithAttrs(as []slog.Attr) slog.Handler {
+	return teeHandler{t.a.WithAttrs(as), t.b.WithAttrs(as)}
+}
+
+func (t teeHandler) WithGroup(name string) slog.Handler {
+	return teeHandler{t.a.WithGroup(name), t.b.WithGroup(name)}
+}

internal/agent/logring_test.go

@@ -0,0 +1,49 @@
+package agent
+
+import (
+	"fmt"
+	"slices"
+	"strings"
+	"testing"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+)
+
+func TestTheLogRingKeepsTheNewestLinesInOrder(t *testing.T) {
+	r := newLogRing(3)
+	if got := r.tail(10); len(got) != 0 {
+		t.Fatalf("empty ring gave %v", got)
+	}
+	for i := 1; i <= 5; i++ {
+		fmt.Fprintf(r, "line %d\n", i)
+	}
+	if got := r.tail(10); !slices.Equal(got, []string{"line 3", "line 4", "line 5"}) {
+		t.Errorf("after wrapping: %v", got)
+	}
+	if got := r.tail(2); !slices.Equal(got, []string{"line 4", "line 5"}) {
+		t.Errorf("tail(2): %v", got)
+	}
+}
+
+func TestANodeAnswersForItsOwnLog(t *testing.T) {
+	cfg := Config{Server: "127.0.0.1:7443", NodeID: "n1", WeightsCache: t.TempDir(), EnginesDir: t.TempDir()}
+	cfg.Superv.WorkDir = t.TempDir()
+	a, err := New(cfg, nil, nil)
+	if err != nil {
+		t.Fatal(err)
+	}
+	a.log.Info("engine installation", "state", "installing")
+
+	r := a.logsReply(&llmgwv1.LogsRequest{RequestId: "x", Source: "agent", Lines: 50})
+	if r.GetRequestId() != "x" || !strings.Contains(strings.Join(r.GetLines(), "\n"), "engine installation") {
+		t.Errorf("the agent's own line is not in its log: %v", r.GetLines())
+	}
+	if !slices.Contains(r.GetSources(), "agent") {
+		t.Errorf("sources %v", r.GetSources())
+	}
+
+	r = a.logsReply(&llmgwv1.LogsRequest{RequestId: "y", Source: "no-such-shard"})
+	if r.GetError() == "" {
+		t.Error("an unknown engine was answered as if it existed")
+	}
+}

internal/agent/logs.go

@@ -0,0 +1,34 @@
+package agent
+
+import (
+	"slices"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+)
+
+// logsReply answers a request for one of this node's logs.
+func (a *Agent) logsReply(req *llmgwv1.LogsRequest) *llmgwv1.LogsReply {
+	n := int(req.GetLines())
+	if n <= 0 {
+		n = 200
+	}
+	n = min(n, agentLogLines)
+	shards := a.sup.IDs()
+	slices.Sort(shards)
+	out := &llmgwv1.LogsReply{
+		RequestId: req.GetRequestId(),
+		Sources:   append([]string{"agent"}, shards...),
+	}
+	switch src := req.GetSource(); src {
+	case "", "agent":
+		out.Lines = a.logRing.tail(n)
+	default:
+		sh, err := a.sup.Get(src)
+		if err != nil {
+			out.Error = "no engine " + src + " runs on this node"
+			return out
+		}
+		out.Lines = sh.Logs(n)
+	}
+	return out
+}

internal/agent/provision.go

@@ -0,0 +1,197 @@
+package agent
+
+import (
+	"context"
+	"fmt"
+	"log/slog"
+	"maps"
+	"slices"
+	"strings"
+	"time"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
+)
+
+// engines is what this node can run: what the agent installed and saw run,
+// and what the configuration names and answers. Nothing merely found on the
+// machine — a llama-server on PATH is somebody's, not necessarily one that
+// works, and placing on it is how a replica fails on a node nobody watches.
+func (a *Agent) engines(ctx context.Context) map[string]string {
+	a.engineOnce.Do(func() {
+		vers := a.reg.ProbeConfigured(ctx)
+		a.mu.Lock()
+		a.configuredVers, a.lastEngineProbe = vers, time.Now()
+		a.mu.Unlock()
+		if len(vers) > 0 {
+			a.log.Info("engines named in the configuration",
+				"engines", engines.ProbedNames(vers))
+		}
+	})
+	a.mu.Lock()
+	defer a.mu.Unlock()
+	vers := a.engineReportLocked()
+	// Whatever registration sends is what the control plane now knows, so a
+	// heartbeat repeats it only once it differs.
+	a.enginesSent = enginesFingerprint(vers)
+	return vers
+}
+
+// engineReportLocked merges the two sources. Installed wins: it is the one
+// that was seen to run. Called with mu held.
+func (a *Agent) engineReportLocked() map[string]string {
+	out := maps.Clone(a.configuredVers)
+	if out == nil {
+		out = map[string]string{}
+	}
+	maps.Copy(out, a.installedVers)
+	return out
+}
+
+// provisionEvery is how often installed engines are checked again. A library
+// removed by an unrelated upgrade is found here rather than by the next
+// replica to start.
+const provisionEvery = 10 * time.Minute
+
+// setProvision takes the control plane's list of wanted engines.
+func (a *Agent) setProvision(p *llmgwv1.Provision) {
+	wants := []provision.Want{}
+	for _, e := range p.GetEngines() {
+		wants = append(wants, provision.Want{Engine: e.GetEngine(), Version: e.GetVersion()})
+	}
+	a.mu.Lock()
+	a.provWants = wants
+	a.mu.Unlock()
+	select {
+	case a.provKick <- struct{}{}:
+	default:
+	}
+}
+
+// provisionLoop installs and re-checks the wanted engines, one pass at a time.
+func (a *Agent) provisionLoop(ctx context.Context) {
+	// After a failure the next pass comes sooner, and backs off: a lock held
+	// by a new machine's own updates clears in a minute or two, while ten
+	// minutes' wait left a node idle for no reason.
+	next := provisionEvery
+	retry := time.Minute
+	for {
+		select {
+		case <-ctx.Done():
+			return
+		case <-a.provKick:
+		case <-time.After(next):
+		}
+		a.mu.Lock()
+		wants := slices.Clone(a.provWants)
+		managed := a.provWants != nil
+		a.mu.Unlock()
+		if !managed {
+			continue
+		}
+
+		statuses := a.prov.Converge(ctx, wants)
+		next = provisionEvery
+		for _, st := range statuses {
+			if st.State == provision.Failed {
+				next, retry = retry, min(retry*2, provisionEvery)
+				break
+			}
+		}
+		if next == provisionEvery {
+			retry = time.Minute
+		}
+		installed := map[string]string{}
+		keep := map[string]bool{}
+		for _, st := range statuses {
+			keep[st.Engine] = true
+			if st.State == provision.Ready {
+				a.reg.UseInstalled(st.Engine, st.Binary)
+				installed[st.Engine] = st.Version
+			}
+		}
+		a.mu.Lock()
+		if enginesFingerprint(installed) != enginesFingerprint(a.installedVers) {
+			a.log.Info("engines installed and running", "engines", engines.ProbedNames(installed))
+		}
+		a.installedVers = installed
+		// An engine no longer asked for is no longer reported on.
+		for e := range a.provStatus {
+			if !keep[e] {
+				delete(a.provStatus, e)
+			}
+		}
+		a.mu.Unlock()
+	}
+}
+
+// provisionStatus records a change in one engine's installation.
+func (a *Agent) provisionStatus(st provision.Status) {
+	a.mu.Lock()
+	prev, had := a.provStatus[st.Engine]
+	a.provStatus[st.Engine] = st
+	a.mu.Unlock()
+	if !had || prev.State != st.State || prev.Detail != st.Detail {
+		lvl := slog.LevelInfo
+		if st.State == provision.Failed || st.State == provision.Unsupported {
+			lvl = slog.LevelWarn
+		}
+		a.log.Log(context.Background(), lvl, "engine installation",
+			"engine", st.Engine, "version", st.Version, "variant", st.Variant,
+			"state", st.State, "detail", st.Detail)
+	}
+}
+
+func provisionFingerprint(m map[string]provision.Status) string {
+	var b strings.Builder
+	for _, k := range slices.Sorted(maps.Keys(m)) {
+		st := m[k]
+		fmt.Fprintf(&b, "%s|%s|%s|%s|%s\n", st.Engine, st.Version, st.Variant, st.State, st.Detail)
+	}
+	return b.String()
+}
+
+// engineReprobe is how often the installed engines are looked for again.
+//
+// Minutes, not seconds: an engine is installed by somebody at a keyboard, so a
+// minute's delay in noticing is invisible, while SGLang's probe starts a
+// Python interpreter that imports it.
+const engineReprobe = time.Minute
+
+// reprobeEngines looks for installed engines again when it is time, off the
+// heartbeat goroutine so a slow probe never delays a heartbeat.
+func (a *Agent) reprobeEngines() {
+	a.mu.Lock()
+	due := !a.probingEngines && !a.lastEngineProbe.IsZero() &&
+		time.Since(a.lastEngineProbe) >= engineReprobe
+	if due {
+		a.probingEngines = true
+	}
+	a.mu.Unlock()
+	if !due {
+		return
+	}
+	go func() {
+		ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
+		defer cancel()
+		vers := a.reg.ProbeConfigured(ctx)
+
+		a.mu.Lock()
+		changed := enginesFingerprint(vers) != enginesFingerprint(a.configuredVers)
+		a.configuredVers, a.lastEngineProbe, a.probingEngines = vers, time.Now(), false
+		a.mu.Unlock()
+		if changed {
+			a.log.Info("engines named in the configuration changed",
+				"engines", engines.ProbedNames(vers))
+		}
+	}()
+}
+
+func enginesFingerprint(vers map[string]string) string {
+	var b strings.Builder
+	for _, k := range slices.Sorted(maps.Keys(vers)) {
+		fmt.Fprintf(&b, "%s=%s\n", k, vers[k])
+	}
+	return b.String()
+}

internal/agent/provision/install.go

@@ -0,0 +1,738 @@
+package provision
+
+import (
+	"archive/tar"
+	"compress/gzip"
+	"context"
+	"crypto/sha256"
+	"encoding/hex"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"io/fs"
+	"log/slog"
+	"net/http"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"regexp"
+	"strings"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
+)
+
+// Want is one engine at one version.
+type Want struct {
+	Engine  string
+	Version string
+}
+
+// The states an engine's installation can be in.
+const (
+	Installing  = "installing"
+	Ready       = "ready"
+	Failed      = "failed"
+	Unsupported = "unsupported"
+	// Skipped is an engine this node is not meant to run, such as vLLM on a
+	// machine with no GPU. Nothing is wrong, and nothing is shown as wrong.
+	Skipped = "skipped"
+)
+
+// Status is where one engine's installation stands.
+type Status struct {
+	Engine  string
+	Version string
+	Variant string
+	State   string
+	// Detail is for a person: what is happening, or what went wrong.
+	Detail string
+	// Binary is the executable, once ready.
+	Binary string
+}
+
+// Installer installs engines under one directory.
+type Installer struct {
+	// Root is where engines are installed, one directory per build:
+	// <root>/<engine>/<version>-<variant>.
+	Root string
+	Host discovery.Host
+	Log  *slog.Logger
+	HTTP *http.Client
+	// OnStatus hears every change, so a download that takes minutes is seen
+	// to be under way rather than silent.
+	OnStatus func(Status)
+
+	// Run executes a command; a seam for tests. Nil runs it for real.
+	Run func(ctx context.Context, env []string, name string, args ...string) ([]byte, error)
+	// FreeSpace reports the gigabytes free under a directory; a seam for
+	// tests. Nil asks the filesystem.
+	FreeSpace func(dir string) (float64, error)
+	// RunStdin is Run with input; a seam for tests. Nil runs it for real.
+	RunStdin func(ctx context.Context, env []string, stdin, name string, args ...string) ([]byte, error)
+	// APIBase and DownloadBase default to GitHub, PyPIBase to PyPI; tests
+	// point them at a fake.
+	APIBase      string
+	DownloadBase string
+	PyPIBase     string
+}
+
+// validVersion is what a version may look like: "b10991", "0.29.0",
+// "0.11.0+cu128". No separators, and not starting with a dot.
+var validVersion = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._+-]{0,63}$`)
+
+// DefaultRoot is where engines are installed when nothing says otherwise.
+const DefaultRoot = "/opt/llmgw/engines"
+
+// marker is written into a build's directory once it has been installed and
+// seen to run. Its absence means the directory is not to be trusted.
+const marker = ".llmgw-installed"
+
+type markerFile struct {
+	Engine  string            `json:"engine"`
+	Version string            `json:"version"`
+	Variant string            `json:"variant"`
+	Binary  string            `json:"binary"`
+	SHA256  map[string]string `json:"sha256,omitempty"`
+	// For a Python engine: what was asked for, PyPI as of when, by which uv.
+	Requirements []string  `json:"requirements,omitempty"`
+	ExcludeNewer string    `json:"exclude_newer,omitempty"`
+	UV           string    `json:"uv,omitempty"`
+	Installed    time.Time `json:"installed"`
+}
+
+// Converge brings every wanted engine to ready, or says why it cannot.
+//
+// An engine already installed is checked again rather than believed: its
+// directory can be damaged, or a library it needs removed by somebody's
+// upgrade, and that is found here rather than by the next replica to fail.
+func (in *Installer) Converge(ctx context.Context, wants []Want) []Status {
+	out := make([]Status, 0, len(wants))
+	for _, w := range wants {
+		st := in.one(ctx, w)
+		in.report(st)
+		out = append(out, st)
+	}
+	return out
+}
+
+func (in *Installer) one(ctx context.Context, w Want) Status {
+	st := Status{Engine: w.Engine, Version: w.Version}
+	// Checked here, on the node, whatever the control plane checked: the
+	// version names a directory under the install root and is passed to a
+	// package manager, and "../x" would put an install — and the RemoveAll
+	// that replaces one — outside it, as root.
+	if !validVersion.MatchString(w.Version) {
+		st.State = Unsupported
+		st.Detail = fmt.Sprintf("%q is not a version this agent will install", w.Version)
+		return st
+	}
+	recipe, ok := Recipes[w.Engine]
+	if !ok {
+		st.State = Unsupported
+		st.Detail = "this agent does not know how to install " + w.Engine +
+			"; install it by hand and name it in the agent's configuration"
+		return st
+	}
+	build, err := recipe.Choose(w.Version, in.Host)
+	if err != nil {
+		st.State, st.Detail = Unsupported, err.Error()
+		var nh NotHere
+		if errors.As(err, &nh) {
+			st.State = Skipped
+		}
+		return st
+	}
+	st.Variant = build.Variant
+	dir := filepath.Join(in.root(), w.Engine, w.Version+"-"+build.Variant)
+
+	// Installed before: check it still runs, and trust nothing else about it.
+	if _, err := os.Stat(filepath.Join(dir, marker)); err == nil {
+		// The packages are checked too. Removing one is exactly the kind of
+		// change an unrelated upgrade makes, and ldd below would catch it only
+		// as a missing library with no hint of which package held it.
+		if err := in.packages(ctx, build.Packages, &st); err != nil {
+			st.State, st.Detail = Failed, err.Error()
+			return st
+		}
+		bin, version, err := in.verify(ctx, dir, build)
+		if err == nil {
+			return in.ready(st, build, bin, version)
+		}
+		in.log().Warn("an installed engine no longer runs; installing it again",
+			"engine", w.Engine, "version", w.Version, "err", err)
+	}
+
+	if err := in.install(ctx, recipe, w, build, dir, &st); err != nil {
+		st.State, st.Detail, st.Binary = Failed, err.Error(), ""
+		return st
+	}
+	bin, version, err := in.verify(ctx, dir, build)
+	if err != nil {
+		st.State, st.Detail = Failed, err.Error()
+		return st
+	}
+	return in.ready(st, build, bin, version)
+}
+
+func (in *Installer) ready(st Status, b Build, bin, version string) Status {
+	st.State, st.Binary = Ready, bin
+	st.Detail = strings.TrimSpace(filepath.Base(bin) + " " + version)
+	var missing []string
+	for _, f := range b.CPUFlags {
+		if !in.Host.HasCPUFlag(f) {
+			missing = append(missing, f)
+		}
+	}
+	if len(missing) > 0 {
+		// Said on every report, not once in a log: it is the difference between
+		// five tokens a second and forty, and a VM given QEMU's default CPU
+		// model looks entirely healthy otherwise.
+		st.Detail += "; this CPU lacks " + strings.Join(missing, ", ") +
+			", so it will run several times slower (a VM needs CPU type \"host\")"
+	}
+	return st
+}
+
+// install fetches, checks and unpacks a build into dir.
+//
+// Into a scratch directory renamed into place at the end, so a failure halfway
+// leaves nothing that looks installed.
+func (in *Installer) install(ctx context.Context, r Recipe, w Want, b Build, dir string, st *Status) error {
+	if b.Python != "" {
+		return in.installPython(ctx, r, w, b, dir, st)
+	}
+	if err := in.packages(ctx, b.Packages, st); err != nil {
+		return err
+	}
+
+	in.progress(st, "asking GitHub for the checksums of %s %s", w.Engine, w.Version)
+	sums, err := in.checksums(ctx, r.Repo, w.Version)
+	if err != nil {
+		return err
+	}
+
+	tmp := dir + ".partial"
+	_ = os.RemoveAll(tmp)
+	if err := os.MkdirAll(tmp, 0o755); err != nil {
+		return err
+	}
+	defer os.RemoveAll(tmp) //nolint:errcheck // gone either way once renamed
+
+	got := map[string]string{}
+	for _, asset := range b.Assets {
+		want, ok := sums[asset]
+		if !ok {
+			// Refused rather than installed unchecked: this is a binary that
+			// will run as root on every node.
+			return fmt.Errorf("%s %s publishes no checksum for %s, so it cannot be verified",
+				w.Engine, w.Version, asset)
+		}
+		archive := filepath.Join(tmp, asset+".download")
+		if err := in.download(ctx, r.Repo, w.Version, asset, archive, want, st); err != nil {
+			return err
+		}
+		in.progress(st, "unpacking %s", asset)
+		if err := untarGz(archive, tmp); err != nil {
+			return fmt.Errorf("unpacking %s: %w", asset, err)
+		}
+		_ = os.Remove(archive)
+		got[asset] = want
+	}
+
+	in.progress(st, "checking that %s runs", b.Binary)
+	bin, _, err := in.verify(ctx, tmp, b)
+	if err != nil {
+		return err
+	}
+	rel, _ := filepath.Rel(tmp, bin)
+	m, _ := json.MarshalIndent(markerFile{
+		Engine: w.Engine, Version: w.Version, Variant: b.Variant,
+		Binary: rel, SHA256: got, Installed: time.Now().UTC(),
+	}, "", "  ")
+	if err := os.WriteFile(filepath.Join(tmp, marker), m, 0o644); err != nil { //nolint:gosec // not secret
+		return err
+	}
+	_ = os.RemoveAll(dir)
+	if err := os.Rename(tmp, dir); err != nil {
+		return err
+	}
+	in.log().Info("installed an engine", "engine", w.Engine, "version", w.Version,
+		"variant", b.Variant, "dir", dir)
+	return nil
+}
+
+// verify finds the binary and runs it.
+//
+// Two checks, because they fail differently. ldd names a missing library
+// outright, where running the binary would only say it could not start. And
+// running it is the only proof that the rest — the right instruction set, the
+// right driver — is in place.
+func (in *Installer) verify(ctx context.Context, dir string, b Build) (bin, version string, err error) {
+	timeout := 60 * time.Second
+	if b.Python != "" {
+		// A script in the environment, not a binary to ldd; and importing
+		// torch and the engine behind it takes a while on a cold disk.
+		bin, timeout = filepath.Join(dir, "bin", b.Binary), 3*time.Minute
+		if _, err := os.Stat(bin); err != nil {
+			return "", "", fmt.Errorf("%s is not in the environment: %w", b.Binary, err)
+		}
+	} else {
+		bin, err = findFile(dir, b.Binary)
+		if err != nil {
+			return "", "", err
+		}
+		if out, err := in.run(ctx, nil, "ldd", bin); err == nil {
+			if missing := lddMissing(string(out)); len(missing) > 0 {
+				return "", "", fmt.Errorf("%s cannot run on this system: %s", b.Binary, strings.Join(missing, "; "))
+			}
+		}
+	}
+	args := b.VersionArgs
+	if len(args) == 0 {
+		args = []string{"--version"}
+	}
+	tctx, cancel := context.WithTimeout(ctx, timeout)
+	defer cancel()
+	out, err := in.run(tctx, nil, bin, args...)
+	if err != nil {
+		return "", "", fmt.Errorf("%s --version failed: %v: %s", b.Binary, err, lastLine(out))
+	}
+	return bin, firstVersionLine(out), nil
+}
+
+// packages installs the operating system packages a build needs.
+//
+// Only those missing, and from the node's own package sources: a package the
+// system already has is not touched, so an engine never upgrades something
+// the rest of the machine depends on.
+func (in *Installer) packages(ctx context.Context, byFamily map[string][]string, st *Status) error {
+	family := in.Host.Family()
+	pkgs := byFamily[family]
+	if len(pkgs) == 0 {
+		// A system of no family this knows is not refused: its packages may
+		// well be there, and if they are not, ldd names the missing library.
+		return nil
+	}
+	pm := packageManagers[family]
+	var missing []string
+	for _, p := range pkgs {
+		if _, err := in.run(ctx, nil, pm.query[0], append(pm.query[1:], p)...); err != nil {
+			missing = append(missing, p)
+		}
+	}
+	if len(missing) == 0 {
+		return nil
+	}
+	tool, cmds := pm.install(in, missing)
+	if tool == "" {
+		return fmt.Errorf("%s is missing and no package manager was found to install it",
+			strings.Join(missing, ", "))
+	}
+	in.progress(st, "installing %s with %s", strings.Join(missing, ", "), tool)
+	for _, c := range cmds {
+		if err := in.runWaitingForLock(ctx, pm.env, c, st); err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+// InstallPackages installs whichever of these packages the machine lacks,
+// from its own package sources, waiting out another package manager's lock.
+// byFamily is keyed by discovery.Host.Family: "deb" or "rpm".
+func (in *Installer) InstallPackages(ctx context.Context, byFamily map[string][]string) error {
+	var st Status
+	return in.packages(ctx, byFamily, &st)
+}
+
+// lockWait bounds how long a package command waits for another to finish.
+const lockWait = 10 * time.Minute
+
+// lockRetry is how often a locked package manager is tried again.
+var lockRetry = 10 * time.Second
+
+// runWaitingForLock runs a package command, waiting while another holds the
+// package manager's lock.
+//
+// A machine a minute old is running cloud-init's and unattended-upgrades' own
+// apt. apt's DPkg::Lock::Timeout covers dpkg's lock and not the one `apt-get
+// update` takes on its lists, so the first install on a new node failed with
+// "Unable to lock directory /var/lib/apt/lists/" — found by building one.
+func (in *Installer) runWaitingForLock(ctx context.Context, env, c []string, st *Status) error {
+	deadline := time.Now().Add(lockWait)
+	for {
+		out, err := in.run(ctx, env, c[0], c[1:]...)
+		if err == nil {
+			return nil
+		}
+		if !lockHeld(string(out)) || time.Now().After(deadline) {
+			return fmt.Errorf("%s: %v: %s", strings.Join(c, " "), err, lastLine(out))
+		}
+		in.progress(st, "waiting for another %s to finish (a new machine runs its own updates first)", c[0])
+		select {
+		case <-ctx.Done():
+			return ctx.Err()
+		case <-time.After(lockRetry):
+		}
+	}
+}
+
+// lockHeld reports whether a package command failed only because another
+// held the lock.
+func lockHeld(out string) bool {
+	for _, s := range []string{"Unable to lock", "Could not get lock", "is locked by another process",
+		"waiting for process with pid", "System management is locked"} {
+		if strings.Contains(out, s) {
+			return true
+		}
+	}
+	return false
+}
+
+// packageManager is how one packaging family asks and installs.
+type packageManager struct {
+	// query exits zero when the package named after it is installed.
+	query []string
+	env   []string
+	// install picks the tool present on this machine and the commands to run.
+	install func(in *Installer, pkgs []string) (tool string, cmds [][]string)
+}
+
+var packageManagers = map[string]packageManager{
+	"deb": {
+		// dpkg -s exits zero for a package merely known about, so the status
+		// is asked for and checked by the command itself: ${db:Status-Abbrev}
+		// is "ii " only when installed.
+		query: []string{"sh", "-c", `test "$(dpkg-query -W -f='${db:Status-Abbrev}' "$0" 2>/dev/null)" = "ii "`},
+		env:   []string{"DEBIAN_FRONTEND=noninteractive"},
+		install: func(_ *Installer, pkgs []string) (string, [][]string) {
+			// A lock timeout rather than failing on the lock: a machine a
+			// minute old is running cloud-init's and unattended-upgrades' own
+			// apt, and the first attempt on every new node would fail.
+			lock := "-o=DPkg::Lock::Timeout=600"
+			return "apt-get", [][]string{
+				{"apt-get", lock, "update"},
+				append([]string{"apt-get", lock, "install", "-y", "--no-install-recommends"}, pkgs...),
+			}
+		},
+	},
+	"rpm": {
+		query: []string{"rpm", "-q"},
+		install: func(in *Installer, pkgs []string) (string, [][]string) {
+			for _, tool := range []string{"dnf", "yum"} {
+				if in.has(tool) {
+					return tool, [][]string{append([]string{tool, "install", "-y"}, pkgs...)}
+				}
+			}
+			if in.has("zypper") {
+				return "zypper", [][]string{append([]string{"zypper", "--non-interactive", "install"}, pkgs...)}
+			}
+			return "", nil
+		},
+	},
+}
+
+// has reports whether a command is on PATH. Asked through the same seam as
+// every other command, so a test can say which tools a machine has.
+func (in *Installer) has(tool string) bool {
+	_, err := in.run(context.Background(), nil, "sh", "-c", `command -v "$0"`, tool)
+	return err == nil
+}
+
+// checksums reads the sha256 GitHub publishes for each asset of a release.
+func (in *Installer) checksums(ctx context.Context, repo, tag string) (map[string]string, error) {
+	url := strings.TrimSuffix(in.apiBase(), "/") + "/repos/" + repo + "/releases/tags/" + tag
+	req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
+	if err != nil {
+		return nil, err
+	}
+	req.Header.Set("Accept", "application/vnd.github+json")
+	resp, err := in.client().Do(req)
+	if err != nil {
+		return nil, fmt.Errorf("reaching GitHub for %s %s: %w", repo, tag, err)
+	}
+	defer resp.Body.Close()
+	switch {
+	case resp.StatusCode == http.StatusNotFound:
+		return nil, fmt.Errorf("%s has no release %s", repo, tag)
+	case resp.StatusCode == http.StatusForbidden || resp.StatusCode == http.StatusTooManyRequests:
+		// Unauthenticated callers get sixty requests an hour per address, and
+		// a whole fleet behind one NAT shares them.
+		return nil, fmt.Errorf("GitHub is rate-limiting this address (%s); it retries later", resp.Status)
+	case resp.StatusCode != http.StatusOK:
+		return nil, fmt.Errorf("GitHub answered %s for %s %s", resp.Status, repo, tag)
+	}
+	var rel struct {
+		Assets []struct {
+			Name   string `json:"name"`
+			Digest string `json:"digest"`
+		} `json:"assets"`
+	}
+	if err := json.NewDecoder(resp.Body).Decode(&rel); err != nil {
+		return nil, fmt.Errorf("reading %s %s from GitHub: %w", repo, tag, err)
+	}
+	out := map[string]string{}
+	for _, a := range rel.Assets {
+		if sum, ok := strings.CutPrefix(a.Digest, "sha256:"); ok && len(sum) == 64 {
+			out[a.Name] = sum
+		}
+	}
+	return out, nil
+}
+
+func (in *Installer) download(ctx context.Context, repo, tag, asset, dst, want string, st *Status) error {
+	url := strings.TrimSuffix(in.downloadBase(), "/") + "/" + repo + "/releases/download/" + tag + "/" + asset
+	req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
+	if err != nil {
+		return err
+	}
+	resp, err := in.client().Do(req)
+	if err != nil {
+		return fmt.Errorf("downloading %s: %w", asset, err)
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode != http.StatusOK {
+		return fmt.Errorf("downloading %s: %s", asset, resp.Status)
+	}
+	if resp.ContentLength > 0 {
+		in.progress(st, "downloading %s (%d MB)", asset, resp.ContentLength>>20)
+	} else {
+		in.progress(st, "downloading %s", asset)
+	}
+	f, err := os.Create(dst) //nolint:gosec // inside the install root
+	if err != nil {
+		return err
+	}
+	h := sha256.New()
+	_, err = io.Copy(io.MultiWriter(f, h), resp.Body)
+	if cerr := f.Close(); err == nil {
+		err = cerr
+	}
+	if err != nil {
+		return fmt.Errorf("downloading %s: %w", asset, err)
+	}
+	if got := hex.EncodeToString(h.Sum(nil)); got != want {
+		return fmt.Errorf("%s does not match the checksum its publisher gave (got %s, expected %s)",
+			asset, got[:12], want[:12])
+	}
+	return nil
+}
+
+// untarGz unpacks an archive under dir, refusing anything that would land
+// outside it. A release archive is somebody else's file, and a path of
+// "../../etc/cron.d/x" in it would otherwise be written as root.
+//
+// Every write goes through an os.Root, which refuses to leave dir however the
+// path gets there. Checking names alone was not enough: "a -> ." followed by
+// "a/a/b -> ../.." reads as inside and resolves two levels above it, and a
+// file written through that lands wherever the links point.
+func untarGz(archive, dir string) error {
+	f, err := os.Open(archive) //nolint:gosec // inside the install root
+	if err != nil {
+		return err
+	}
+	defer f.Close()
+	gz, err := gzip.NewReader(f)
+	if err != nil {
+		return err
+	}
+	root, err := os.OpenRoot(dir)
+	if err != nil {
+		return err
+	}
+	defer root.Close()
+	tr := tar.NewReader(gz)
+	for {
+		h, err := tr.Next()
+		if errors.Is(err, io.EOF) {
+			return nil
+		}
+		if err != nil {
+			return err
+		}
+		name := filepath.Clean(h.Name)
+		if _, err := within(dir, name); err != nil {
+			return err
+		}
+		switch h.Typeflag {
+		case tar.TypeDir:
+			if err := root.MkdirAll(name, 0o755); err != nil {
+				return err
+			}
+		case tar.TypeReg:
+			if err := root.MkdirAll(filepath.Dir(name), 0o755); err != nil {
+				return err
+			}
+			out, err := root.OpenFile(name, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, fs.FileMode(h.Mode)&0o755)
+			if err != nil {
+				return fmt.Errorf("%s: %w", h.Name, err)
+			}
+			if _, err := io.Copy(out, tr); err != nil { //nolint:gosec // a release archive, size bounded by its checksum
+				out.Close()
+				return err
+			}
+			if err := out.Close(); err != nil {
+				return err
+			}
+		case tar.TypeSymlink:
+			// Libraries are versioned through symlinks to a file beside them —
+			// libllama.so -> libllama.so.0 — and that is all a release needs.
+			// A target with a separator in it could point anywhere once other
+			// links are followed, and a binary found through one would be run
+			// as root, so it is refused rather than reasoned about.
+			if h.Linkname == "" || strings.ContainsAny(h.Linkname, `/\`) || h.Linkname == ".." || h.Linkname == "." {
+				return fmt.Errorf("%s links to %q; only a file beside it is allowed", h.Name, h.Linkname)
+			}
+			if err := root.MkdirAll(filepath.Dir(name), 0o755); err != nil {
+				return err
+			}
+			_ = root.Remove(name)
+			if err := root.Symlink(h.Linkname, name); err != nil {
+				return err
+			}
+		default:
+			// Devices, FIFOs and hard links have no business in a release.
+		}
+	}
+}
+
+func within(dir, name string) (string, error) {
+	target := filepath.Join(dir, name) //nolint:gosec // checked on the next line
+	rel, err := filepath.Rel(dir, target)
+	if err != nil || rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) {
+		return "", fmt.Errorf("%s would unpack outside the install directory", name)
+	}
+	return target, nil
+}
+
+// lddMissing reads what ldd could not satisfy, in two shapes:
+//
+//	libgomp.so.1 => not found
+//	/lib64/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by ...)
+//
+// The second is a system too old for the build — RHEL 9 against a build that
+// wants GCC 12's libstdc++ — and is said as such, rather than as a library
+// that is plainly there being "not found".
+func lddMissing(out string) []string {
+	seen := map[string]bool{}
+	var missing []string
+	for _, line := range strings.Split(out, "\n") {
+		line = strings.TrimSpace(line)
+		if !strings.Contains(line, "not found") {
+			continue
+		}
+		var what string
+		if lib, rest, ok := strings.Cut(line, ": version `"); ok {
+			sym, _, _ := strings.Cut(rest, "'")
+			what = filepath.Base(lib) + " is too old (needs " + sym + ")"
+		} else {
+			what = strings.TrimSpace(strings.SplitN(line, "=>", 2)[0]) + " is not installed"
+		}
+		if !seen[what] {
+			seen[what] = true
+			missing = append(missing, what)
+		}
+	}
+	return missing
+}
+
+func findFile(dir, name string) (string, error) {
+	var found string
+	_ = filepath.WalkDir(dir, func(p string, d fs.DirEntry, err error) error {
+		if err != nil || found != "" {
+			return nil //nolint:nilerr // an unreadable subtree is not the binary
+		}
+		if !d.IsDir() && d.Name() == name {
+			found = p
+		}
+		return nil
+	})
+	if found == "" {
+		return "", fmt.Errorf("the build has no %s in it", name)
+	}
+	return found, nil
+}
+
+func (in *Installer) progress(st *Status, format string, args ...any) {
+	st.State, st.Detail = Installing, fmt.Sprintf(format, args...)
+	in.report(*st)
+}
+
+func (in *Installer) report(st Status) {
+	if in.OnStatus != nil {
+		in.OnStatus(st)
+	}
+}
+
+func runWithStdin(ctx context.Context, env []string, stdin, name string, args ...string) ([]byte, error) {
+	cmd := exec.CommandContext(ctx, name, args...) //nolint:gosec // fixed commands and paths we installed
+	cmd.Env = append(os.Environ(), env...)
+	cmd.Stdin = strings.NewReader(stdin)
+	return cmd.CombinedOutput()
+}
+
+func (in *Installer) run(ctx context.Context, env []string, name string, args ...string) ([]byte, error) {
+	if in.Run != nil {
+		return in.Run(ctx, env, name, args...)
+	}
+	cmd := exec.CommandContext(ctx, name, args...) //nolint:gosec // fixed commands and paths we installed
+	cmd.Env = append(os.Environ(), env...)
+	return cmd.CombinedOutput()
+}
+
+func (in *Installer) root() string {
+	if in.Root != "" {
+		return in.Root
+	}
+	return DefaultRoot
+}
+
+func (in *Installer) log() *slog.Logger {
+	if in.Log != nil {
+		return in.Log
+	}
+	return slog.Default()
+}
+
+func (in *Installer) client() *http.Client {
+	if in.HTTP != nil {
+		return in.HTTP
+	}
+	return &http.Client{Timeout: 30 * time.Minute}
+}
+
+func (in *Installer) apiBase() string {
+	if in.APIBase != "" {
+		return in.APIBase
+	}
+	return "https://api.github.com"
+}
+
+func (in *Installer) downloadBase() string {
+	if in.DownloadBase != "" {
+		return in.DownloadBase
+	}
+	return "https://github.com"
+}
+
+func lastLine(b []byte) string {
+	lines := strings.Split(strings.TrimSpace(string(b)), "\n")
+	return strings.TrimSpace(lines[len(lines)-1])
+}
+
+func firstVersionLine(b []byte) string {
+	for _, l := range strings.Split(string(b), "\n") {
+		if strings.Contains(strings.ToLower(l), "version") {
+			return strings.TrimSpace(l)
+		}
+	}
+	// vllm --version says the number and nothing else.
+	for _, l := range strings.Split(string(b), "\n") {
+		if l = strings.TrimSpace(l); l != "" && l[0] >= '0' && l[0] <= '9' {
+			return l
+		}
+	}
+	return ""
+}

internal/agent/provision/install_test.go

@@ -0,0 +1,390 @@
+package provision
+
+import (
+	"archive/tar"
+	"bytes"
+	"compress/gzip"
+	"context"
+	"crypto/sha256"
+	"encoding/hex"
+	"encoding/json"
+	"errors"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"strings"
+	"sync"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
+)
+
+var ubuntu = discovery.Host{OSID: "ubuntu", OSVersion: "24.04", Arch: "amd64", CPUFlags: []string{"avx2"}}
+
+// release is a fake GitHub: one release, its assets and their digests.
+type release struct {
+	assets  map[string][]byte
+	digests map[string]string
+	gets    int
+	mu      sync.Mutex
+}
+
+func tarGz(t *testing.T, files map[string]string, links map[string]string) []byte {
+	t.Helper()
+	var buf bytes.Buffer
+	gz := gzip.NewWriter(&buf)
+	tw := tar.NewWriter(gz)
+	for name, body := range files {
+		_ = tw.WriteHeader(&tar.Header{Name: name, Mode: 0o755, Size: int64(len(body)), Typeflag: tar.TypeReg})
+		_, _ = tw.Write([]byte(body))
+	}
+	for name, target := range links {
+		_ = tw.WriteHeader(&tar.Header{Name: name, Linkname: target, Typeflag: tar.TypeSymlink})
+	}
+	_ = tw.Close()
+	_ = gz.Close()
+	return buf.Bytes()
+}
+
+func (r *release) serve(t *testing.T) *httptest.Server {
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
+		r.mu.Lock()
+		r.gets++
+		r.mu.Unlock()
+		if strings.HasPrefix(req.URL.Path, "/repos/") {
+			var out struct {
+				Assets []map[string]string `json:"assets"`
+			}
+			for name := range r.assets {
+				out.Assets = append(out.Assets, map[string]string{"name": name, "digest": r.digests[name]})
+			}
+			_ = json.NewEncoder(w).Encode(out)
+			return
+		}
+		body, ok := r.assets[filepath.Base(req.URL.Path)]
+		if !ok {
+			http.NotFound(w, req)
+			return
+		}
+		_, _ = w.Write(body)
+	}))
+	t.Cleanup(srv.Close)
+	return srv
+}
+
+func newRelease(assets map[string][]byte) *release {
+	r := &release{assets: assets, digests: map[string]string{}}
+	for name, b := range assets {
+		sum := sha256.Sum256(b)
+		r.digests[name] = "sha256:" + hex.EncodeToString(sum[:])
+	}
+	return r
+}
+
+// fakeSystem answers the package tools, ldd and the engine itself.
+type fakeSystem struct {
+	mu        sync.Mutex
+	installed map[string]bool
+	tools     map[string]bool
+	lddOut    string
+	calls     []string
+	// locked is how many more package commands find the lock held.
+	locked int
+}
+
+var errExit = errors.New("exit 1")
+
+func (f *fakeSystem) run(_ context.Context, _ []string, name string, args ...string) ([]byte, error) {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	f.calls = append(f.calls, name+" "+strings.Join(args, " "))
+	switch {
+	case name == "sh" && strings.Contains(args[1], "dpkg-query"),
+		name == "rpm" && args[0] == "-q":
+		if f.installed[args[len(args)-1]] {
+			return nil, nil
+		}
+		return nil, errExit
+	case name == "sh" && strings.Contains(args[1], "command -v"):
+		if f.tools[args[2]] {
+			return nil, nil
+		}
+		return nil, errExit
+	case name == "apt-get" || name == "dnf":
+		if f.locked > 0 {
+			f.locked--
+			return []byte("E: Unable to lock directory /var/lib/apt/lists/\n"), errExit
+		}
+		for i, a := range args {
+			if a == "install" {
+				for _, p := range args[i+1:] {
+					if !strings.HasPrefix(p, "-") {
+						f.installed[p] = true
+					}
+				}
+			}
+		}
+		return nil, nil
+	case name == "ldd":
+		return []byte(f.lddOut), nil
+	case strings.HasSuffix(name, "llama-server"):
+		return []byte("version: 10991 (abc)\n"), nil
+	}
+	return nil, errors.New("unexpected command " + name)
+}
+
+func installer(t *testing.T, srv *httptest.Server, sys *fakeSystem, host discovery.Host) (*Installer, *[]Status) {
+	var seen []Status
+	var mu sync.Mutex
+	return &Installer{
+		Root: t.TempDir(), Host: host, Run: sys.run,
+		APIBase: srv.URL, DownloadBase: srv.URL, HTTP: srv.Client(),
+		OnStatus: func(s Status) { mu.Lock(); seen = append(seen, s); mu.Unlock() },
+	}, &seen
+}
+
+func llamaRelease(t *testing.T) *release {
+	return newRelease(map[string][]byte{
+		"llama-b1-bin-ubuntu-x64.tar.gz": tarGz(t,
+			map[string]string{"llama-b1/llama-server": "#!/bin/sh\n", "llama-b1/libllama.so.0": "x"},
+			map[string]string{"llama-b1/libllama.so": "libllama.so.0"}),
+	})
+}
+
+func TestAnEngineIsInstalledFromNothing(t *testing.T) {
+	// A node rebuilt from a bare image: no engine, no OpenMP runtime. What it
+	// ends with has to have come from the publisher, checked, and been run.
+	rel := llamaRelease(t)
+	sys := &fakeSystem{installed: map[string]bool{}}
+	in, seen := installer(t, rel.serve(t), sys, ubuntu)
+
+	got := in.Converge(t.Context(), []Want{{Engine: "llamacpp", Version: "b1"}})
+	if len(got) != 1 || got[0].State != Ready {
+		t.Fatalf("not ready: %+v", got)
+	}
+	if !sys.installed["libgomp1"] {
+		t.Error("the OpenMP runtime the build links against was not installed")
+	}
+	if !strings.HasPrefix(got[0].Binary, in.Root) {
+		t.Errorf("binary %s is not the one installed under %s", got[0].Binary, in.Root)
+	}
+	if _, err := os.Lstat(filepath.Join(filepath.Dir(got[0].Binary), "libllama.so")); err != nil {
+		t.Errorf("the library symlink was not unpacked: %v", err)
+	}
+	var states []string
+	for _, s := range *seen {
+		states = append(states, s.State+": "+s.Detail)
+	}
+	if !strings.Contains(strings.Join(states, "\n"), "downloading") {
+		t.Errorf("a download was not reported while it ran:\n%s", strings.Join(states, "\n"))
+	}
+}
+
+func TestAnInstalledEngineIsCheckedNotFetchedAgain(t *testing.T) {
+	rel := llamaRelease(t)
+	sys := &fakeSystem{installed: map[string]bool{}}
+	in, _ := installer(t, rel.serve(t), sys, ubuntu)
+	w := []Want{{Engine: "llamacpp", Version: "b1"}}
+	in.Converge(t.Context(), w)
+	before := rel.gets
+
+	got := in.Converge(t.Context(), w)
+	if got[0].State != Ready {
+		t.Fatalf("second pass: %+v", got[0])
+	}
+	if rel.gets != before {
+		t.Errorf("a ready engine was downloaded again (%d requests)", rel.gets-before)
+	}
+}
+
+func TestAnEngineThatCannotLoadALibraryIsNotReady(t *testing.T) {
+	// Found on a node before: llama-server present, executable, and unable to
+	// start for want of a library — while the node reported ready.
+	rel := llamaRelease(t)
+	sys := &fakeSystem{installed: map[string]bool{"libgomp1": true},
+		lddOut: "\tlibcuda.so.1 => not found\n"}
+	in, _ := installer(t, rel.serve(t), sys, ubuntu)
+
+	got := in.Converge(t.Context(), []Want{{Engine: "llamacpp", Version: "b1"}})
+	if got[0].State != Failed || !strings.Contains(got[0].Detail, "libcuda.so.1") {
+		t.Fatalf("expected a failure naming the library: %+v", got[0])
+	}
+	if _, err := os.Stat(filepath.Join(in.Root, "llamacpp", "b1-cpu", marker)); err == nil {
+		t.Error("a build that does not run was marked installed")
+	}
+}
+
+func TestADownloadThatDoesNotMatchItsChecksumIsRefused(t *testing.T) {
+	rel := llamaRelease(t)
+	for name := range rel.digests {
+		rel.digests[name] = "sha256:" + strings.Repeat("0", 64)
+	}
+	sys := &fakeSystem{installed: map[string]bool{"libgomp1": true}}
+	in, _ := installer(t, rel.serve(t), sys, ubuntu)
+
+	got := in.Converge(t.Context(), []Want{{Engine: "llamacpp", Version: "b1"}})
+	if got[0].State != Failed || !strings.Contains(got[0].Detail, "checksum") {
+		t.Fatalf("a tampered download was not refused: %+v", got[0])
+	}
+}
+
+func TestAnArchiveCannotWriteOutsideItsDirectory(t *testing.T) {
+	rel := newRelease(map[string][]byte{
+		"llama-b1-bin-ubuntu-x64.tar.gz": tarGz(t,
+			map[string]string{"../../escaped": "x", "llama-server": ""}, nil),
+	})
+	sys := &fakeSystem{installed: map[string]bool{"libgomp1": true}}
+	in, _ := installer(t, rel.serve(t), sys, ubuntu)
+
+	got := in.Converge(t.Context(), []Want{{Engine: "llamacpp", Version: "b1"}})
+	if got[0].State != Failed {
+		t.Fatalf("an archive escaping its directory was installed: %+v", got[0])
+	}
+}
+
+func TestTheBuildFollowsTheHardware(t *testing.T) {
+	gpu := ubuntu
+	gpu.NVIDIADriver = "570.86.15"
+	b, err := chooseLlamaCpp("b1", gpu)
+	if err != nil || !strings.HasPrefix(b.Variant, "cuda") || len(b.Assets) != 2 {
+		t.Errorf("an NVIDIA node got %+v, %v; expected the CUDA build and its runtime", b, err)
+	}
+	if _, err := chooseLlamaCpp("b1", discovery.Host{OSID: "ubuntu", Arch: "riscv64"}); err == nil {
+		t.Error("a build was chosen for an architecture llama.cpp does not publish for")
+	}
+
+	slow := ubuntu
+	slow.CPUFlags = nil
+	st := (&Installer{Host: slow}).ready(Status{}, Build{CPUFlags: []string{"avx2"}}, "/x/llama-server", "")
+	if !strings.Contains(st.Detail, "avx2") {
+		t.Errorf("a CPU without AVX2 was not called out: %q", st.Detail)
+	}
+}
+
+func TestAnEngineTheAgentCannotInstallSaysSo(t *testing.T) {
+	in := &Installer{Root: t.TempDir(), Host: ubuntu}
+	got := in.Converge(t.Context(), []Want{{Engine: "tgi", Version: "3.0.0"}})
+	if got[0].State != Unsupported || !strings.Contains(got[0].Detail, "install it by hand") {
+		t.Errorf("got %+v", got[0])
+	}
+}
+
+func TestARedHatFamilyNodeInstallsWithItsOwnPackageManager(t *testing.T) {
+	// The build is not Ubuntu's in any way that matters; the package that
+	// carries OpenMP's runtime is named differently, and installed by dnf.
+	rocky := discovery.Host{OSID: "rocky", OSVersion: "10.0", OSLike: []string{"rhel", "centos", "fedora"},
+		Arch: "amd64", CPUFlags: []string{"avx2"}}
+	rel := llamaRelease(t)
+	sys := &fakeSystem{installed: map[string]bool{}, tools: map[string]bool{"dnf": true}}
+	in, _ := installer(t, rel.serve(t), sys, rocky)
+
+	got := in.Converge(t.Context(), []Want{{Engine: "llamacpp", Version: "b1"}})
+	if got[0].State != Ready {
+		t.Fatalf("not ready: %+v", got[0])
+	}
+	if !sys.installed["libgomp"] || sys.installed["libgomp1"] {
+		t.Errorf("installed %v; expected the rpm name, libgomp", sys.installed)
+	}
+}
+
+func TestASystemTooOldForTheBuildSaysWhatItLacks(t *testing.T) {
+	// RHEL 9: glibc is new enough, libstdc++ is one GCC release short. ldd
+	// finds the library and reports the symbol, and "libstdc++ not found"
+	// would send somebody looking for a package that is installed.
+	rel := llamaRelease(t)
+	sys := &fakeSystem{installed: map[string]bool{"libgomp1": true},
+		lddOut: "\t/lib64/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /x/libllama.so)\n" +
+			"\t/lib64/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /x/llama-server)\n"}
+	in, _ := installer(t, rel.serve(t), sys, ubuntu)
+
+	got := in.Converge(t.Context(), []Want{{Engine: "llamacpp", Version: "b1"}})
+	if got[0].State != Failed || !strings.Contains(got[0].Detail, "libstdc++.so.6 is too old (needs GLIBCXX_3.4.30)") {
+		t.Fatalf("detail %q", got[0].Detail)
+	}
+	if strings.Count(got[0].Detail, "GLIBCXX") != 1 {
+		t.Errorf("the same shortfall was listed more than once: %q", got[0].Detail)
+	}
+}
+
+func TestAPackageManagerBusyWithItsOwnUpdatesIsWaitedFor(t *testing.T) {
+	// A new node's first minutes: cloud-init and unattended-upgrades hold
+	// apt's lists lock, which DPkg::Lock::Timeout does not cover.
+	lockRetry = time.Millisecond
+	t.Cleanup(func() { lockRetry = 10 * time.Second })
+
+	rel := llamaRelease(t)
+	sys := &fakeSystem{installed: map[string]bool{}, locked: 3}
+	in, seen := installer(t, rel.serve(t), sys, ubuntu)
+
+	got := in.Converge(t.Context(), []Want{{Engine: "llamacpp", Version: "b1"}})
+	if got[0].State != Ready {
+		t.Fatalf("gave up on a lock another apt held: %+v", got[0])
+	}
+	waited := false
+	for _, s := range *seen {
+		waited = waited || strings.Contains(s.Detail, "waiting for another apt-get")
+	}
+	if !waited {
+		t.Error("the wait was not reported")
+	}
+}
+
+func TestAVersionThatIsAPathIsRefused(t *testing.T) {
+	// The version names a directory under the install root; "../" would put
+	// an install, and the RemoveAll that replaces one, outside it, as root.
+	in := &Installer{Root: t.TempDir(), Host: ubuntu}
+	for _, v := range []string{"../../etc", "..", ".hidden", "b1/../../x", "", "b1 && rm -rf /"} {
+		got := in.Converge(t.Context(), []Want{{Engine: "llamacpp", Version: v}})
+		if got[0].State != Unsupported {
+			t.Errorf("version %q: %+v", v, got[0])
+		}
+	}
+}
+
+func TestChainedLinksCannotCarryAFileOutOfTheInstall(t *testing.T) {
+	// Each link reads as inside the directory; together they point above it.
+	var buf bytes.Buffer
+	gz := gzip.NewWriter(&buf)
+	tw := tar.NewWriter(gz)
+	_ = tw.WriteHeader(&tar.Header{Name: "a", Linkname: ".", Typeflag: tar.TypeSymlink})
+	_ = tw.WriteHeader(&tar.Header{Name: "a/a/b", Linkname: "../..", Typeflag: tar.TypeSymlink})
+	_ = tw.WriteHeader(&tar.Header{Name: "a/a/b/escaped", Mode: 0o644, Size: 1, Typeflag: tar.TypeReg})
+	_, _ = tw.Write([]byte("x"))
+	_ = tw.Close()
+	_ = gz.Close()
+
+	parent := t.TempDir()
+	dir := filepath.Join(parent, "install")
+	_ = os.MkdirAll(dir, 0o755)
+	archive := filepath.Join(parent, "evil.tar.gz")
+	_ = os.WriteFile(archive, buf.Bytes(), 0o644)
+
+	if err := untarGz(archive, dir); err == nil {
+		t.Error("an archive of chained links was unpacked")
+	}
+	if _, err := os.Stat(filepath.Join(parent, "escaped")); err == nil {
+		t.Fatal("a file was written outside the install directory")
+	}
+}
+
+func TestALibrarysVersionLinkIsStillUnpacked(t *testing.T) {
+	var buf bytes.Buffer
+	gz := gzip.NewWriter(&buf)
+	tw := tar.NewWriter(gz)
+	_ = tw.WriteHeader(&tar.Header{Name: "lib/libllama.so.0", Mode: 0o755, Size: 1, Typeflag: tar.TypeReg})
+	_, _ = tw.Write([]byte("x"))
+	_ = tw.WriteHeader(&tar.Header{Name: "lib/libllama.so", Linkname: "libllama.so.0", Typeflag: tar.TypeSymlink})
+	_ = tw.Close()
+	_ = gz.Close()
+	dir := t.TempDir()
+	archive := filepath.Join(t.TempDir(), "ok.tar.gz")
+	_ = os.WriteFile(archive, buf.Bytes(), 0o644)
+	if err := untarGz(archive, dir); err != nil {
+		t.Fatal(err)
+	}
+	if target, err := os.Readlink(filepath.Join(dir, "lib", "libllama.so")); err != nil || target != "libllama.so.0" {
+		t.Errorf("link %q %v", target, err)
+	}
+}

internal/agent/provision/python.go

@@ -0,0 +1,205 @@
+package provision
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"net/http"
+	"os"
+	"path/filepath"
+	"strings"
+	"syscall"
+	"time"
+)
+
+// A Python engine is a virtual environment built by uv: an interpreter uv
+// fetches, and the engine with every dependency from PyPI.
+//
+// Two things make it as trustworthy as an archive checked against its
+// publisher's checksum. The dependencies are resolved as PyPI stood a day
+// after the engine was published (--exclude-newer), so a node built next
+// month gets what one built today got, not whatever a dependency released
+// since — the usual way a working vLLM stops working. And the resolution is
+// written out with every file's sha256 from PyPI and installed with
+// --require-hashes, so a file that does not match what PyPI published is
+// refused. The resolved list is kept in the environment, as the record of
+// exactly what is installed.
+
+// pythonInstallTimeout bounds one installation: several gigabytes, on a
+// network that may be slow, but not forever.
+const pythonInstallTimeout = time.Hour
+
+// lockName is the resolved requirements, kept in the environment.
+const lockName = "llmgw-requirements.txt"
+
+func (in *Installer) installPython(ctx context.Context, r Recipe, w Want, b Build, dir string, st *Status) error {
+	ctx, cancel := context.WithTimeout(ctx, pythonInstallTimeout)
+	defer cancel()
+
+	if err := in.packages(ctx, b.Packages, st); err != nil {
+		return err
+	}
+	if err := os.MkdirAll(in.root(), 0o755); err != nil {
+		return err
+	}
+	if b.FreeGB > 0 {
+		if free, err := in.free(in.root()); err == nil && free < float64(b.FreeGB) {
+			return fmt.Errorf("%s needs about %d GB free under %s and there are %.1f GB",
+				w.Engine, b.FreeGB, in.root(), free)
+		}
+	}
+
+	in.progress(st, "installing uv %s", UVVersion)
+	uv := in.one(ctx, Want{Engine: "uv", Version: UVVersion})
+	if uv.State != Ready {
+		return fmt.Errorf("installing uv, which installs %s: %s", w.Engine, uv.Detail)
+	}
+
+	in.progress(st, "asking PyPI when %s %s was published", r.PyPI, w.Version)
+	published, err := in.published(ctx, r.PyPI, w.Version)
+	if err != nil {
+		return err
+	}
+	asOf := published.Add(24 * time.Hour).UTC().Format(time.RFC3339)
+
+	cache := filepath.Join(in.root(), ".uv-cache")
+	env := []string{
+		"UV_CACHE_DIR=" + cache,
+		"UV_PYTHON_INSTALL_DIR=" + filepath.Join(in.root(), "python"),
+		// Only the interpreter uv fetched: the system's may be the wrong
+		// version, or change under the environment with the next upgrade.
+		"UV_PYTHON_PREFERENCE=only-managed",
+		// Nobody's uv.toml or pip.conf decides what is installed here.
+		"UV_NO_CONFIG=1",
+	}
+	tmp := dir + ".partial"
+	_ = os.RemoveAll(tmp)
+	defer os.RemoveAll(tmp) //nolint:errcheck // gone either way once renamed
+
+	in.progress(st, "creating a Python %s environment for %s", b.Python, w.Engine)
+	// Relocatable, because it is built under one name and renamed into place
+	// once it works: its scripts find their interpreter relative to
+	// themselves rather than by the path they were built at.
+	if out, err := in.run(ctx, env, uv.Binary, "venv", "--relocatable", "--python", b.Python, tmp); err != nil {
+		return fmt.Errorf("uv venv: %v: %s", err, tail(out))
+	}
+	python := filepath.Join(tmp, "bin", "python")
+
+	in.progress(st, "resolving %s against PyPI as of %s", strings.Join(b.Requirements, " "), asOf[:10])
+	lock := filepath.Join(tmp, lockName)
+	args := []string{"pip", "compile", "--python", python, "--exclude-newer", asOf,
+		"--generate-hashes", "--quiet", "--output-file", lock, "-"}
+	if out, err := in.runStdin(ctx, env, strings.Join(b.Requirements, "\n")+"\n", uv.Binary, args...); err != nil {
+		return fmt.Errorf("resolving %s: %v: %s", strings.Join(b.Requirements, " "), err, tail(out))
+	}
+
+	in.progress(st, "downloading and installing %s and its dependencies: several gigabytes, which takes minutes",
+		strings.Join(b.Requirements, " "))
+	if out, err := in.run(ctx, env, uv.Binary, "pip", "install", "--python", python,
+		"--require-hashes", "--no-deps", "-r", lock); err != nil {
+		return fmt.Errorf("installing %s: %v: %s", strings.Join(b.Requirements, " "), err, tail(out))
+	}
+
+	in.progress(st, "checking that %s runs", w.Engine)
+	if _, _, err := in.verify(ctx, tmp, b); err != nil {
+		return err
+	}
+	m, _ := json.MarshalIndent(markerFile{
+		Engine: w.Engine, Version: w.Version, Variant: b.Variant,
+		Binary: filepath.Join("bin", b.Binary), Requirements: b.Requirements,
+		ExcludeNewer: asOf, UV: UVVersion, Installed: time.Now().UTC(),
+	}, "", "  ")
+	if err := os.WriteFile(filepath.Join(tmp, marker), m, 0o644); err != nil { //nolint:gosec // not secret
+		return err
+	}
+	_ = os.RemoveAll(dir)
+	if err := os.Rename(tmp, dir); err != nil {
+		return err
+	}
+	// The cache is as large as the environment. Kept after a failure, so the
+	// next attempt does not download it all again; dropped after success.
+	_ = os.RemoveAll(cache)
+	in.log().Info("installed an engine", "engine", w.Engine, "version", w.Version,
+		"variant", b.Variant, "dir", dir, "as_of", asOf)
+	return nil
+}
+
+// published is when PyPI received a release's last file.
+func (in *Installer) published(ctx context.Context, project, version string) (time.Time, error) {
+	url := strings.TrimSuffix(in.pypiBase(), "/") + "/pypi/" + project + "/" + version + "/json"
+	req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
+	if err != nil {
+		return time.Time{}, err
+	}
+	resp, err := in.client().Do(req)
+	if err != nil {
+		return time.Time{}, fmt.Errorf("asking PyPI for %s %s: %w", project, version, err)
+	}
+	defer resp.Body.Close() //nolint:errcheck // read-only
+	if resp.StatusCode == http.StatusNotFound {
+		return time.Time{}, fmt.Errorf("PyPI has no %s %s", project, version)
+	}
+	if resp.StatusCode != http.StatusOK {
+		return time.Time{}, fmt.Errorf("asking PyPI for %s %s: %s", project, version, resp.Status)
+	}
+	var rel struct {
+		URLs []struct {
+			Uploaded time.Time `json:"upload_time_iso_8601"`
+		} `json:"urls"`
+	}
+	if err := json.NewDecoder(resp.Body).Decode(&rel); err != nil {
+		return time.Time{}, fmt.Errorf("reading PyPI's answer for %s %s: %w", project, version, err)
+	}
+	var last time.Time
+	for _, u := range rel.URLs {
+		if u.Uploaded.After(last) {
+			last = u.Uploaded
+		}
+	}
+	if last.IsZero() {
+		return time.Time{}, errors.New("PyPI lists no files for " + project + " " + version)
+	}
+	return last, nil
+}
+
+func (in *Installer) free(dir string) (float64, error) {
+	if in.FreeSpace != nil {
+		return in.FreeSpace(dir)
+	}
+	return FreeGB(dir)
+}
+
+func (in *Installer) pypiBase() string {
+	if in.PyPIBase != "" {
+		return in.PyPIBase
+	}
+	return "https://pypi.org"
+}
+
+// runStdin runs a command with input on its standard input.
+func (in *Installer) runStdin(ctx context.Context, env []string, stdin, name string, args ...string) ([]byte, error) {
+	if in.RunStdin != nil {
+		return in.RunStdin(ctx, env, stdin, name, args...)
+	}
+	return runWithStdin(ctx, env, stdin, name, args...)
+}
+
+// FreeGB is the space available to an unprivileged writer under dir.
+func FreeGB(dir string) (float64, error) {
+	var s syscall.Statfs_t
+	if err := syscall.Statfs(dir, &s); err != nil {
+		return 0, err
+	}
+	return float64(s.Bavail) * float64(s.Bsize) / (1 << 30), nil
+}
+
+// tail is the last few lines of a command's output: uv explains a failed
+// resolution over several, and the last alone is rarely the reason.
+func tail(b []byte) string {
+	lines := strings.Split(strings.TrimSpace(string(b)), "\n")
+	if len(lines) > 4 {
+		lines = lines[len(lines)-4:]
+	}
+	return strings.Join(lines, " / ")
+}

internal/agent/provision/python_test.go

@@ -0,0 +1,184 @@
+package provision
+
+import (
+	"context"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"slices"
+	"strings"
+	"sync"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
+)
+
+var gpuNode = discovery.Host{OSID: "ubuntu", OSVersion: "24.04", Arch: "amd64",
+	NVIDIADriver: "580.65.06", MinComputeCapability: 8.0}
+
+// fakePython is uv and the environments it builds.
+type fakePython struct {
+	fakeSystem
+	pmu      sync.Mutex
+	commands []string
+	failPip  bool
+}
+
+func (f *fakePython) run(ctx context.Context, env []string, name string, args ...string) ([]byte, error) {
+	line := name + " " + strings.Join(args, " ")
+	f.pmu.Lock()
+	f.commands = append(f.commands, line)
+	f.pmu.Unlock()
+	switch {
+	case strings.HasSuffix(name, "/uv") && len(args) == 1 && args[0] == "--version":
+		return []byte("uv 0.12.20\n"), nil
+	case strings.HasSuffix(name, "/uv") && args[0] == "venv":
+		return nil, os.MkdirAll(filepath.Join(args[len(args)-1], "bin"), 0o755)
+	case strings.HasSuffix(name, "/uv") && args[0] == "pip" && args[1] == "install":
+		if f.failPip {
+			return []byte("error: Hash mismatch for torch\n"), errExit
+		}
+		venv := filepath.Dir(filepath.Dir(args[slices.Index(args, "--python")+1]))
+		return nil, os.WriteFile(filepath.Join(venv, "bin", "vllm"), []byte("#!/bin/sh\n"), 0o755)
+	case strings.HasSuffix(name, "/bin/vllm") && args[0] == "--version":
+		return []byte("0.30.0\n"), nil
+	}
+	return f.fakeSystem.run(ctx, env, name, args...)
+}
+
+func (f *fakePython) runStdin(_ context.Context, _ []string, stdin, name string, args ...string) ([]byte, error) {
+	f.pmu.Lock()
+	f.commands = append(f.commands, name+" "+strings.Join(args, " ")+" <<< "+strings.TrimSpace(stdin))
+	f.pmu.Unlock()
+	out := args[slices.Index(args, "--output-file")+1]
+	return nil, os.WriteFile(out, []byte("vllm==0.30.0 --hash=sha256:ab\n"), 0o644)
+}
+
+func pythonInstaller(t *testing.T, host discovery.Host, sys *fakePython) (*Installer, *release) {
+	uv := newRelease(map[string][]byte{
+		"uv-x86_64-unknown-linux-gnu.tar.gz": tarGz(t,
+			map[string]string{"uv-x86_64-unknown-linux-gnu/uv": "#!/bin/sh\n"}, nil),
+	})
+	gh := uv.serve(t)
+	mux := http.NewServeMux()
+	mux.HandleFunc("/pypi/vllm/0.30.0/json", func(w http.ResponseWriter, _ *http.Request) {
+		_, _ = w.Write([]byte(`{"urls":[{"upload_time_iso_8601":"2026-09-22T05:22:21.647189Z"},
+			{"upload_time_iso_8601":"2026-09-22T05:23:10.039243Z"}]}`))
+	})
+	pypi := httptest.NewServer(mux)
+	t.Cleanup(pypi.Close)
+	return &Installer{
+		Root: t.TempDir(), Host: host, Run: sys.run, RunStdin: sys.runStdin,
+		APIBase: gh.URL, DownloadBase: gh.URL, PyPIBase: pypi.URL, HTTP: gh.Client(),
+		FreeSpace: func(string) (float64, error) { return 100, nil },
+	}, uv
+}
+
+func TestVLLMIsInstalledOnAGPUNode(t *testing.T) {
+	sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}}
+	in, _ := pythonInstaller(t, gpuNode, sys)
+
+	got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0"}})
+	if got[0].State != Ready {
+		t.Fatalf("not ready: %+v", got[0])
+	}
+	dir := filepath.Join(in.Root, "vllm", "0.30.0-cu13")
+	if got[0].Binary != filepath.Join(dir, "bin", "vllm") {
+		t.Errorf("binary %s, want the environment's vllm", got[0].Binary)
+	}
+	if !sys.installed["gcc"] {
+		t.Error("gcc, which Triton compiles its launcher with, was not installed")
+	}
+	all := strings.Join(sys.commands, "\n")
+	for _, want := range []string{
+		// PyPI as it stood a day after vLLM's last file, so the dependencies
+		// are the ones it was released against.
+		"--exclude-newer 2026-09-23T05:23:10Z",
+		"--generate-hashes",
+		"--require-hashes",
+		"venv --relocatable --python 3.12",
+	} {
+		if !strings.Contains(all, want) {
+			t.Errorf("no %q in:\n%s", want, all)
+		}
+	}
+	if _, err := os.Stat(filepath.Join(dir, lockName)); err != nil {
+		t.Errorf("the resolved requirements were not kept: %v", err)
+	}
+	if _, err := os.Stat(filepath.Join(in.Root, ".uv-cache")); err == nil {
+		t.Error("the download cache was left behind after a successful install")
+	}
+
+	// Checked again on the next pass, not installed again.
+	before := len(sys.commands)
+	if again := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0"}}); again[0].State != Ready {
+		t.Fatalf("second pass: %+v", again[0])
+	}
+	for _, c := range sys.commands[before:] {
+		if strings.Contains(c, "pip install") {
+			t.Errorf("a ready environment was installed again: %s", c)
+		}
+	}
+}
+
+func TestVLLMIsNotForEveryNode(t *testing.T) {
+	cases := []struct {
+		name  string
+		host  discovery.Host
+		state string
+		says  string
+	}{
+		{"no GPU", ubuntu, Skipped, "has none"},
+		{"old driver", discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15"}, Unsupported, "driver 580"},
+		{"V100", discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 7.0}, Unsupported, "7.5"},
+	}
+	for _, c := range cases {
+		t.Run(c.name, func(t *testing.T) {
+			sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}}
+			in, uv := pythonInstaller(t, c.host, sys)
+			got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0"}})
+			if got[0].State != c.state || !strings.Contains(got[0].Detail, c.says) {
+				t.Errorf("got %s %q, want %s saying %q", got[0].State, got[0].Detail, c.state, c.says)
+			}
+			if uv.gets != 0 || len(sys.commands) != 0 {
+				t.Errorf("a node that cannot run it still started installing: %v", sys.commands)
+			}
+		})
+	}
+}
+
+func TestAPythonEngineIsRefusedWithoutTheDiskForIt(t *testing.T) {
+	sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{"gcc": true}}}
+	in, uv := pythonInstaller(t, gpuNode, sys)
+	in.FreeSpace = func(string) (float64, error) { return 12.5, nil }
+	got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0"}})
+	if got[0].State != Failed || !strings.Contains(got[0].Detail, "12.5 GB") {
+		t.Errorf("got %s %q", got[0].State, got[0].Detail)
+	}
+	if uv.gets != 0 {
+		t.Error("it began downloading anyway")
+	}
+}
+
+func TestAFailedPythonInstallLeavesNothingThatLooksInstalled(t *testing.T) {
+	sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{"gcc": true}}, failPip: true}
+	in, _ := pythonInstaller(t, gpuNode, sys)
+	got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0"}})
+	if got[0].State != Failed || !strings.Contains(got[0].Detail, "Hash mismatch") {
+		t.Fatalf("got %s %q", got[0].State, got[0].Detail)
+	}
+	entries, _ := os.ReadDir(filepath.Join(in.Root, "vllm"))
+	for _, e := range entries {
+		t.Errorf("left behind: %s", e.Name())
+	}
+}
+
+func TestAVersionPyPIDoesNotHaveIsSaidSo(t *testing.T) {
+	sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{"gcc": true}}}
+	in, _ := pythonInstaller(t, gpuNode, sys)
+	got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "9.9.9"}})
+	if got[0].State != Failed || !strings.Contains(got[0].Detail, "PyPI has no vllm 9.9.9") {
+		t.Errorf("got %s %q", got[0].State, got[0].Detail)
+	}
+}

internal/agent/provision/recipe.go

@@ -0,0 +1,250 @@
+// Package provision installs the engines a node is asked to have.
+//
+// The control plane says which engines at which versions; everything else is
+// worked out here, on the node, because only the node knows it: which build
+// fits its processor and driver, what its operating system is missing, and
+// whether the result actually runs. Nothing already on the machine is taken
+// on trust. An engine counts as installed when this package installed it,
+// checked every file against the checksum its publisher gave, and watched it
+// start.
+package provision
+
+import (
+	"fmt"
+	"slices"
+	"strings"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
+)
+
+// Build is one published build of an engine, chosen for one machine.
+type Build struct {
+	// Variant names the build: "cpu", "cuda-12.8".
+	Variant string
+	// Assets are the release files to fetch, unpacked in order into one
+	// directory.
+	Assets []string
+	// Packages are operating system packages the build links against and a
+	// minimal install may not carry, by packaging family ("deb", "rpm"): the
+	// same library is libgomp1 on one and libgomp on the other.
+	Packages map[string][]string
+	// Binary is the executable's file name, found anywhere in the unpacked
+	// tree.
+	Binary string
+	// CPUFlags are extensions the build is slow without. Missing ones are
+	// reported, not refused: slow is still running.
+	CPUFlags []string
+
+	// Python, when set, makes the build a Python environment rather than an
+	// archive: that interpreter version, with Requirements installed from
+	// PyPI by uv. Binary is then looked for in the environment's bin.
+	Python       string
+	Requirements []string
+	// VersionArgs is what Binary is run with to prove the build works, when
+	// --version is not it.
+	VersionArgs []string
+	// FreeGB is the disk space the installation needs before it starts: a
+	// Python engine is several gigabytes, and one that fills the disk halfway
+	// leaves the node worse off than one that never began.
+	FreeGB int
+}
+
+// Recipe knows how to install one engine.
+type Recipe struct {
+	// Repo is the GitHub repository whose releases carry the builds; empty
+	// for an engine installed from PyPI.
+	Repo string
+	// PyPI is the project an engine installed from PyPI is published as.
+	PyPI string
+	// Choose picks the build for a machine, or says why none fits.
+	Choose func(version string, h discovery.Host) (Build, error)
+}
+
+// Recipes are the engines this agent can install.
+//
+// vLLM and SGLang are Python environments of several gigabytes, installed
+// from PyPI by uv, which is itself installed like llama.cpp: a GitHub release
+// checked against its published checksum.
+var Recipes = map[string]Recipe{
+	"llamacpp": {Repo: "ggml-org/llama.cpp", Choose: chooseLlamaCpp},
+	"vllm":     {PyPI: "vllm", Choose: chooseVLLM},
+	"sglang":   {PyPI: "sglang", Choose: chooseSGLang},
+	"uv":       {Repo: "astral-sh/uv", Choose: chooseUV},
+}
+
+// Installable are the engines this agent can install, as it tells the
+// control plane: every recipe but uv, which is a tool for the others.
+func Installable() []string {
+	var out []string
+	for name := range Recipes {
+		if name != "uv" {
+			out = append(out, name)
+		}
+	}
+	slices.Sort(out)
+	return out
+}
+
+// UVVersion is the uv that installs the Python engines.
+const UVVersion = "0.12.20"
+
+// pythonForEngines is the interpreter the Python engines run on. uv fetches
+// it, so the node's own Python — or its absence — does not matter.
+const pythonForEngines = "3.12"
+
+// NotHere is the error a recipe gives for a node the engine is not meant
+// for, as opposed to one it should run on and cannot: vLLM on a machine
+// with no GPU is not a fault, and is not reported as one.
+type NotHere struct{ Reason string }
+
+func (e NotHere) Error() string { return e.Reason }
+
+func chooseUV(version string, h discovery.Host) (Build, error) {
+	var arch string
+	switch h.Arch {
+	case "amd64":
+		arch = "x86_64"
+	case "arm64":
+		arch = "aarch64"
+	default:
+		return Build{}, fmt.Errorf("uv publishes no Linux build for %s", orUnknown(h.Arch))
+	}
+	return Build{
+		Variant: arch,
+		Assets:  []string{"uv-" + arch + "-unknown-linux-gnu.tar.gz"},
+		Binary:  "uv",
+	}, nil
+}
+
+// minDriverForCUDA13 is the oldest NVIDIA driver CUDA 13 runs on. Every vLLM
+// since 0.26 and SGLang 0.5 is built on torch for CUDA 13.
+const minDriverForCUDA13 = 580
+
+// minComputeForCUDA13 is the oldest GPU CUDA 13 supports: Turing. It dropped
+// Volta, so a V100 cannot run these builds at all.
+const minComputeForCUDA13 = 7.5
+
+// cudaGate says whether a node can run an engine built on CUDA 13.
+func cudaGate(engine string, h discovery.Host) error {
+	switch h.Arch {
+	case "amd64", "arm64":
+	default:
+		return fmt.Errorf("%s publishes no Linux build for %s", engine, orUnknown(h.Arch))
+	}
+	if h.NVIDIADriver == "" {
+		return NotHere{engine + " runs on NVIDIA GPUs and this node has none; it serves llama.cpp models"}
+	}
+	if major := leadingInt(h.NVIDIADriver); major > 0 && major < minDriverForCUDA13 {
+		return fmt.Errorf("%s is built for CUDA 13, which needs NVIDIA driver %d or newer; this node has %s",
+			engine, minDriverForCUDA13, h.NVIDIADriver)
+	}
+	if cc := h.MinComputeCapability; cc > 0 && cc < minComputeForCUDA13 {
+		return fmt.Errorf("%s is built for CUDA 13, which supports GPUs of compute capability %.1f and newer; this node has %.1f",
+			engine, minComputeForCUDA13, cc)
+	}
+	return nil
+}
+
+// Python engines need a C compiler at run time: Triton builds a small
+// launcher for its kernels the first time a model loads, and without gcc
+// that load fails long after the installation looked finished.
+var compilerPackages = map[string][]string{"deb": {"gcc"}, "rpm": {"gcc"}}
+
+func chooseVLLM(version string, h discovery.Host) (Build, error) {
+	if err := cudaGate("vLLM", h); err != nil {
+		return Build{}, err
+	}
+	return Build{
+		Variant:      "cu13",
+		Python:       pythonForEngines,
+		Requirements: []string{"vllm==" + version},
+		Binary:       "vllm",
+		Packages:     compilerPackages,
+		FreeGB:       25,
+	}, nil
+}
+
+func chooseSGLang(version string, h discovery.Host) (Build, error) {
+	if err := cudaGate("SGLang", h); err != nil {
+		return Build{}, err
+	}
+	// SGLang is launched as a module, so what the agent is given is the
+	// environment's interpreter, and the proof is importing it.
+	return Build{
+		Variant:      "cu13",
+		Python:       pythonForEngines,
+		Requirements: []string{"sglang==" + version},
+		Binary:       "python",
+		VersionArgs:  []string{"-c", "import sglang; print('sglang', sglang.__version__)"},
+		Packages:     compilerPackages,
+		FreeGB:       25,
+	}, nil
+}
+
+// leadingInt is the number a version string starts with: 580 of "580.65.06".
+func leadingInt(s string) int {
+	n := 0
+	for _, c := range strings.TrimSpace(s) {
+		if c < '0' || c > '9' {
+			break
+		}
+		n = n*10 + int(c-'0')
+	}
+	return n
+}
+
+// cudaForLlamaCpp is the CUDA build taken on an NVIDIA node. The release also
+// carries a matching cudart archive, so the node needs a driver and nothing
+// else — no toolkit.
+const cudaForLlamaCpp = "12.8"
+
+func chooseLlamaCpp(version string, h discovery.Host) (Build, error) {
+	// The published Linux builds are built on Ubuntu, but nothing about them
+	// is Ubuntu's: they need glibc 2.34 and the libstdc++ of GCC 12, which
+	// Debian 12 and RHEL 10 have and RHEL 9 does not. So the distribution's
+	// name decides nothing here. The build is tried everywhere, and ldd says
+	// whether this system can load it — in the words of the missing symbol.
+	var arch string
+	switch h.Arch {
+	case "amd64":
+		arch = "x64"
+	case "arm64":
+		arch = "arm64"
+	default:
+		return Build{}, fmt.Errorf("llama.cpp publishes no Linux build for %s", orUnknown(h.Arch))
+	}
+
+	b := Build{
+		Variant: "cpu",
+		Assets:  []string{fmt.Sprintf("llama-%s-bin-ubuntu-%s.tar.gz", version, arch)},
+		// Everything else the build links is in the archive or in any current
+		// glibc system; OpenMP's runtime is the one a cloud image lacks, and
+		// the server does not start without it.
+		Packages: map[string][]string{"deb": {"libgomp1"}, "rpm": {"libgomp"}},
+		Binary:   "llama-server",
+	}
+	if arch == "x64" {
+		// The archive carries a backend per CPU generation and loads the best
+		// one, down to SSE4.2. Below AVX2 it runs, several times slower.
+		b.CPUFlags = []string{"avx2"}
+	}
+	// The CUDA build is published for x64 at this version; on arm64 it is a
+	// newer CUDA that the driver may not support, so an arm64 GPU node runs
+	// the CPU build until someone decides otherwise.
+	if h.NVIDIADriver != "" && arch == "x64" {
+		name := fmt.Sprintf("bin-ubuntu-cuda-%s-x64.tar.gz", cudaForLlamaCpp)
+		b.Variant = "cuda-" + cudaForLlamaCpp
+		b.Assets = []string{
+			fmt.Sprintf("llama-%s-%s", version, name),
+			fmt.Sprintf("cudart-llama-%s-%s", version, name),
+		}
+	}
+	return b, nil
+}
+
+func orUnknown(s string) string {
+	if strings.TrimSpace(s) == "" {
+		return "an unrecognised system"
+	}
+	return s
+}

internal/agent/selfdescribe_test.go

@@ -0,0 +1,119 @@
+package agent
+
+import (
+	"context"
+	"os"
+	"slices"
+	"testing"
+	"time"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+)
+
+func TestAnAgentConfigNeedsOnlyWhereTheControlPlaneIs(t *testing.T) {
+	// Every node carried its own name, its own address and the same two
+	// directories written out by hand, so no two agent files were alike and a
+	// copy taken from one node registered the next as the first.
+	cfg := Config{Server: "127.0.0.1:7443"}
+	if err := cfg.normalize(); err != nil {
+		t.Fatal(err)
+	}
+	host, _ := os.Hostname()
+	if cfg.NodeID != host || cfg.Hostname != host {
+		t.Errorf("node_id %q, hostname %q; expected the machine's name %q",
+			cfg.NodeID, cfg.Hostname, host)
+	}
+	if cfg.MgmtAddr != "127.0.0.1" {
+		t.Errorf("mgmt_addr %q, expected the address on the route to the control plane",
+			cfg.MgmtAddr)
+	}
+	if cfg.WeightsCache != DefaultWeightsCache {
+		t.Errorf("weights land in %q", cfg.WeightsCache)
+	}
+	if !slices.Contains(cfg.ModelDirs, cfg.WeightsCache) {
+		t.Errorf("the download directory %q is not looked in: %v",
+			cfg.WeightsCache, cfg.ModelDirs)
+	}
+}
+
+func TestWhatIsWrittenStillWins(t *testing.T) {
+	cfg := Config{
+		Server: "127.0.0.1:7443", NodeID: "gpu-a-01", MgmtAddr: "10.9.9.9",
+		ModelDirs: []string{"/lustre/models"}, WeightsCache: "/scratch/w",
+	}
+	if err := cfg.normalize(); err != nil {
+		t.Fatal(err)
+	}
+	if cfg.NodeID != "gpu-a-01" || cfg.MgmtAddr != "10.9.9.9" {
+		t.Errorf("configured identity overridden: %q %q", cfg.NodeID, cfg.MgmtAddr)
+	}
+	if !slices.Equal(cfg.ModelDirs, []string{"/lustre/models", "/scratch/w"}) {
+		t.Errorf("model dirs %v", cfg.ModelDirs)
+	}
+}
+
+func TestAHeartbeatSaysWhenTheEnginesChanged(t *testing.T) {
+	cfg := Config{Server: "127.0.0.1:7443", NodeID: "n1", WeightsCache: t.TempDir()}
+	cfg.Superv.WorkDir = t.TempDir()
+	a, err := New(cfg, nil, nil)
+	if err != nil {
+		t.Fatal(err)
+	}
+	engines := func(m *llmgwv1.AgentMessage) *llmgwv1.Heartbeat { return m.GetHeartbeat() }
+
+	a.mu.Lock()
+	a.configuredVers = map[string]string{"llamacpp": "unknown"}
+	a.enginesSent = enginesFingerprint(a.configuredVers)
+	a.mu.Unlock()
+	if hb := engines(a.heartbeat()); hb.GetEnginesChanged() {
+		t.Errorf("unchanged engines were resent")
+	}
+
+	a.mu.Lock()
+	a.configuredVers = map[string]string{"llamacpp": "unknown", "vllm": "unknown"}
+	a.mu.Unlock()
+	hb := engines(a.heartbeat())
+	if !hb.GetEnginesChanged() || len(hb.GetEngineVersions()) != 2 {
+		t.Errorf("an installed engine was not reported: %v", hb.GetEngineVersions())
+	}
+	if engines(a.heartbeat()).GetEnginesChanged() {
+		t.Errorf("reported again on the next heartbeat")
+	}
+}
+
+func TestWhatTheControlPlaneAsksToInstallIsReportedBack(t *testing.T) {
+	cfg := Config{Server: "127.0.0.1:7443", NodeID: "n1", WeightsCache: t.TempDir(),
+		EnginesDir: t.TempDir()}
+	cfg.Superv.WorkDir = t.TempDir()
+	a, err := New(cfg, nil, nil)
+	if err != nil {
+		t.Fatal(err)
+	}
+	ctx, cancel := context.WithCancel(t.Context())
+	defer cancel()
+	go a.provisionLoop(ctx)
+
+	// An engine this agent cannot install: no network is needed to learn so,
+	// and the answer has to reach the control plane rather than a log.
+	a.setProvision(&llmgwv1.Provision{Engines: []*llmgwv1.EngineWant{{Engine: "vllm", Version: "0.29.0"}}})
+
+	deadline := time.Now().Add(3 * time.Second)
+	for time.Now().Before(deadline) {
+		hb := a.heartbeat().GetHeartbeat()
+		if hb.GetProvisionChanged() {
+			p := hb.GetProvision()
+			if len(p) != 1 || p[0].GetState() != "unsupported" {
+				t.Fatalf("reported %v", p)
+			}
+			a.mu.Lock()
+			_, reported := a.engineReportLocked()["vllm"]
+			a.mu.Unlock()
+			if reported {
+				t.Error("an engine that was not installed is reported as present")
+			}
+			return
+		}
+		time.Sleep(20 * time.Millisecond)
+	}
+	t.Fatal("the installation's outcome never reached a heartbeat")
+}

internal/agent/supervisor/health.go

@@ -53,8 +53,8 @@ func NewHTTPHealth(timeout time.Duration, opts ...HTTPHealthOption) *HTTPHealth
 }
 
 // Ready reports whether the engine at baseURL is serving model.
-func (h *HTTPHealth) Ready(ctx context.Context, baseURL, model string) error {
-	if err := h.get(ctx, baseURL+h.healthPath, nil); err != nil {
+func (h *HTTPHealth) Ready(ctx context.Context, baseURL, model, apiKey string) error {
+	if err := h.get(ctx, baseURL+h.healthPath, apiKey, nil); err != nil {
 		return fmt.Errorf("health: %w", err)
 	}
 	if !h.requireName || model == "" {
@@ -66,7 +66,7 @@ func (h *HTTPHealth) Ready(ctx context.Context, baseURL, model string) error {
 			ID string `json:"id"`
 		} `json:"data"`
 	}
-	if err := h.get(ctx, baseURL+h.modelsPath, &list); err != nil {
+	if err := h.get(ctx, baseURL+h.modelsPath, apiKey, &list); err != nil {
 		return fmt.Errorf("models: %w", err)
 	}
 	for _, d := range list.Data {
@@ -77,11 +77,14 @@ func (h *HTTPHealth) Ready(ctx context.Context, baseURL, model string) error {
 	return fmt.Errorf("engine is up but does not serve %q yet", model)
 }
 
-func (h *HTTPHealth) get(ctx context.Context, url string, out any) error {
+func (h *HTTPHealth) get(ctx context.Context, url, apiKey string, out any) error {
 	req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
 	if err != nil {
 		return err
 	}
+	if apiKey != "" {
+		req.Header.Set("Authorization", "Bearer "+apiKey)
+	}
 	resp, err := h.client.Do(req)
 	if err != nil {
 		return err

internal/agent/supervisor/health_test.go

@@ -39,7 +39,7 @@ func engineStub(t *testing.T, healthy bool, models ...string) *httptest.Server {
 
 func TestReadyWhenServingModel(t *testing.T) {
 	srv := engineStub(t, true, "llama-3.3-70b")
-	if err := NewHTTPHealth(time.Second).Ready(t.Context(), srv.URL, "llama-3.3-70b"); err != nil {
+	if err := NewHTTPHealth(time.Second).Ready(t.Context(), srv.URL, "llama-3.3-70b", ""); err != nil {
 		t.Errorf("a serving engine was reported unready: %v", err)
 	}
 }
@@ -48,7 +48,7 @@ func TestNotReadyWhileLoading(t *testing.T) {
 	// The case that matters: the process is up but the weights are not loaded.
 	// Routing on /health alone would send traffic into a loading model.
 	srv := engineStub(t, false, "llama-3.3-70b")
-	err := NewHTTPHealth(time.Second).Ready(t.Context(), srv.URL, "llama-3.3-70b")
+	err := NewHTTPHealth(time.Second).Ready(t.Context(), srv.URL, "llama-3.3-70b", "")
 	if err == nil {
 		t.Fatal("an unhealthy engine was reported ready")
 	}
@@ -61,7 +61,7 @@ func TestNotReadyWhenModelMissing(t *testing.T) {
 	// /health passes but the model is not listed: the engine is up and serving
 	// something else, or still registering.
 	srv := engineStub(t, true, "some-other-model")
-	err := NewHTTPHealth(time.Second).Ready(t.Context(), srv.URL, "llama-3.3-70b")
+	err := NewHTTPHealth(time.Second).Ready(t.Context(), srv.URL, "llama-3.3-70b", "")
 	if err == nil {
 		t.Fatal("a model that is not served was reported ready")
 	}
@@ -73,14 +73,33 @@ func TestNotReadyWhenModelMissing(t *testing.T) {
 func TestModelCheckCanBeSkipped(t *testing.T) {
 	srv := engineStub(t, true)
 	h := NewHTTPHealth(time.Second, WithoutModelCheck())
-	if err := h.Ready(t.Context(), srv.URL, "anything"); err != nil {
+	if err := h.Ready(t.Context(), srv.URL, "anything", ""); err != nil {
 		t.Errorf("the model check was not skipped: %v", err)
 	}
 }
 
 func TestUnreachableEngine(t *testing.T) {
 	h := NewHTTPHealth(200 * time.Millisecond)
-	if err := h.Ready(t.Context(), "http://127.0.0.1:1", "m"); err == nil {
+	if err := h.Ready(t.Context(), "http://127.0.0.1:1", "m", ""); err == nil {
 		t.Error("an unreachable engine was reported ready")
 	}
 }
+
+func TestTheHealthCheckSendsTheEnginesKey(t *testing.T) {
+	// An engine started with a key refuses /v1/models without it, and a
+	// shard that is serving would read as never ready.
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		if r.Header.Get("Authorization") != "Bearer k1" {
+			w.WriteHeader(http.StatusUnauthorized)
+			return
+		}
+		_, _ = w.Write([]byte(`{"data":[{"id":"m"}]}`))
+	}))
+	defer srv.Close()
+	if err := NewHTTPHealth(time.Second).Ready(t.Context(), srv.URL, "m", "k1"); err != nil {
+		t.Errorf("with the key: %v", err)
+	}
+	if err := NewHTTPHealth(time.Second).Ready(t.Context(), srv.URL, "m", ""); err == nil {
+		t.Error("without the key the engine refused, and that read as ready")
+	}
+}

internal/agent/supervisor/supervisor.go

@@ -79,7 +79,7 @@ func (c *Config) normalize() {
 // supervisor can be tested without an HTTP engine.
 type HealthChecker interface {
 	// Ready reports whether the shard at baseURL serves model.
-	Ready(ctx context.Context, baseURL, model string) error
+	Ready(ctx context.Context, baseURL, model, apiKey string) error
 }
 
 // Shard is one supervised engine process.
@@ -243,6 +243,16 @@ func (s *Supervisor) launch(ctx context.Context, sh *Shard, cmds []engines.Comma
 	}
 
 	cmd := cmds[len(cmds)-1]
+	// The main process can need the rendezvous point too: a Ray worker's
+	// join is its only process, and started before the head listens it
+	// exits, failing the shard and with it the replica.
+	if cmd.WaitForLeaderPort && !sh.Spec.Leader() {
+		addr := fmt.Sprintf("%s:%d", sh.Spec.LeaderAddr, sh.Spec.LeaderPort)
+		if err := waitForPort(runCtx, addr, s.cfg.RendezvousTimeout); err != nil {
+			cancel()
+			return fmt.Errorf("waiting for %s (%s): %w", addr, cmd.Name, err)
+		}
+	}
 
 	c := exec.CommandContext(runCtx, cmd.Path, cmd.Args...)
 	c.Env = cmd.Env
@@ -401,7 +411,7 @@ func (s *Supervisor) WaitReady(ctx context.Context, sh *Shard, baseURL, model st
 			last := strings.Join(sh.Logs(10), "\n")
 			return fmt.Errorf("shard %s not ready within %v; last output:\n%s", sh.ID, timeout, last)
 		case <-t.C:
-			if err := s.health.Ready(ctx, baseURL, model); err != nil {
+			if err := s.health.Ready(ctx, baseURL, model, sh.Spec.APIKey); err != nil {
 				continue
 			}
 			sh.setState(Ready, nil)

internal/agent/supervisor/supervisor_test.go

@@ -8,6 +8,7 @@ import (
 	"net"
 	"os"
 	"path/filepath"
+	"slices"
 	"strings"
 	"sync/atomic"
 	"syscall"
@@ -20,7 +21,7 @@ import (
 // scriptedHealth reports ready once the flag is set.
 type scriptedHealth struct{ ready atomic.Bool }
 
-func (h *scriptedHealth) Ready(context.Context, string, string) error {
+func (h *scriptedHealth) Ready(context.Context, string, string, string) error {
 	if h.ready.Load() {
 		return nil
 	}
@@ -242,7 +243,9 @@ func TestWaitReadyTimeoutIncludesLogs(t *testing.T) {
 	if err != nil {
 		t.Fatal(err)
 	}
-	time.Sleep(100 * time.Millisecond) // let the line be captured
+	eventually(t, 3*time.Second, "the line was never captured", func() bool {
+		return strings.Contains(strings.Join(sh.Logs(10), "\n"), "CUDA out of memory")
+	})
 
 	err = s.WaitReady(t.Context(), sh, "http://x", "m", 200*time.Millisecond)
 	if err == nil {
@@ -294,7 +297,10 @@ func TestLogsAreCaptured(t *testing.T) {
 	if err != nil {
 		t.Fatal(err)
 	}
-	time.Sleep(400 * time.Millisecond)
+	eventually(t, 3*time.Second, "not every line arrived", func() bool {
+		l := strings.Join(sh.Logs(10), "\n")
+		return strings.Contains(l, "line1") && strings.Contains(l, "line2") && strings.Contains(l, "line3")
+	})
 
 	logs := strings.Join(sh.Logs(10), "\n")
 	for _, want := range []string{"line1", "line2", "line3"} {
@@ -371,11 +377,11 @@ func TestStateFileRemovedOnExit(t *testing.T) {
 
 	sh, _ := s.Start(t.Context(), "a", engines.ShardSpec{}, shellCommand("exit 0"))
 	waitState(t, sh, Failed, 3*time.Second)
-	time.Sleep(50 * time.Millisecond)
-
-	if _, err := os.Stat(s.statePath("a")); !os.IsNotExist(err) {
-		t.Error("the state file outlived the process; a later scan would find a dead orphan")
-	}
+	eventually(t, 3*time.Second,
+		"the state file outlived the process; a later scan would find a dead orphan", func() bool {
+			_, err := os.Stat(s.statePath("a"))
+			return os.IsNotExist(err)
+		})
 }
 
 // --- state machine and ring buffer ---
@@ -536,9 +542,9 @@ func TestCorruptStateFileIsCleanedUp(t *testing.T) {
 
 // --- multi-process shards (plan §11, phase 4) ---
 
-func TestSidecarRunsBeforeEngine(t *testing.T) {
-	// The Ray shape: a supporting process starts first and keeps running, then
-	// the engine starts against it.
+func TestASidecarAndAnEngineBothRunUnderOneShard(t *testing.T) {
+	// The Ray shape: a supporting process runs alongside the engine for the
+	// shard's lifetime, but the engine is the one whose exit ends the shard.
 	s, _ := newSupervisor(t)
 	dir := t.TempDir()
 	order := filepath.Join(dir, "order")
@@ -571,13 +577,20 @@ func TestSidecarRunsBeforeEngine(t *testing.T) {
 		}
 	}
 	// Asserted rather than broken out of: the original fell through on timeout,
-	// so a slow machine turned the ordering check into a no-op and the failure
-	// surfaced later as something else.
+	// so a slow machine turned this into a no-op and the failure surfaced later
+	// as something else.
 	if len(started) != 2 {
 		t.Fatalf("only %d of 2 processes started: %v", len(started), started)
 	}
-	if started[0] != "sidecar" || started[1] != "engine" {
-		t.Fatalf("start order was %v", started)
+	// Which of the two appended first is deliberately not asserted. The
+	// supervisor spawns the sidecar first, but spawning is not sequencing:
+	// nothing here makes the engine wait, so two shells racing to append give
+	// either order and a test that demanded one would fail for the scheduler's
+	// reasons rather than the supervisor's. The wait that is a real guarantee
+	// is WaitForLeaderPort, and TestWaitForLeaderPortBlocksUntilListening is
+	// where it is checked.
+	if !slices.Contains(started, "sidecar") || !slices.Contains(started, "engine") {
+		t.Fatalf("expected both processes to have run, got %v", started)
 	}
 
 	// The shard's pid is the engine's, not the sidecar's: the engine's exit is
@@ -735,7 +748,9 @@ func TestWaitAliveFailsOnEarlyExit(t *testing.T) {
 	if err := s.WaitAlive(t.Context(), sh); err == nil {
 		t.Fatal("a worker that died immediately was reported ready")
 	}
-	if d := time.Since(start); d > 1500*time.Millisecond {
+	// Below AliveGrace (5s), which is what waiting it out would take; the
+	// headroom is for a loaded runner, where 1.5s was sometimes not enough.
+	if d := time.Since(start); d > 4*time.Second {
 		t.Errorf("waited %v for a process that had already exited", d)
 	}
 }
@@ -786,3 +801,50 @@ func TestAnAdoptedShardCanBecomeReady(t *testing.T) {
 	}
 	_ = s2.Stop(t.Context(), "a")
 }
+
+func TestAWorkersOnlyProcessWaitsForTheLeader(t *testing.T) {
+	// A Ray worker's join is its only process. The wait was honoured only for
+	// sidecars, so the join started at once, found no head, exited, and took
+	// the replica down with it.
+	s, _ := newSupervisor(t)
+	s.cfg.RendezvousTimeout = 5 * time.Second
+
+	ln, err := net.Listen("tcp", "127.0.0.1:0")
+	if err != nil {
+		t.Fatal(err)
+	}
+	port := ln.Addr().(*net.TCPAddr).Port
+	_ = ln.Close()
+	go func() {
+		time.Sleep(400 * time.Millisecond)
+		l, err := net.Listen("tcp", fmt.Sprintf("127.0.0.1:%d", port))
+		if err == nil {
+			t.Cleanup(func() { _ = l.Close() })
+		}
+	}()
+
+	spec := engines.ShardSpec{Rank: 1, LeaderAddr: "127.0.0.1", LeaderPort: port}
+	cmds := []engines.Command{{Name: "join", Path: "/bin/sh", Args: []string{"-c", "sleep 30"},
+		Env: os.Environ(), WaitForLeaderPort: true}}
+	start := time.Now()
+	if _, err := s.Start(t.Context(), "w", spec, cmds); err != nil {
+		t.Fatal(err)
+	}
+	if d := time.Since(start); d < 300*time.Millisecond {
+		t.Errorf("the worker started after %v, before the leader was listening", d)
+	}
+}
+
+// eventually waits for cond, failing with msg if it does not hold within
+// limit. Sleeping a fixed time and then checking measured the machine the
+// test ran on, and a loaded CI runner under the race detector lost that race.
+func eventually(t *testing.T, limit time.Duration, msg string, cond func() bool) {
+	t.Helper()
+	deadline := time.Now().Add(limit)
+	for !cond() {
+		if time.Now().After(deadline) {
+			t.Fatal(msg)
+		}
+		time.Sleep(10 * time.Millisecond)
+	}
+}

internal/agent/upgrade.go

@@ -0,0 +1,58 @@
+package agent
+
+import (
+	"context"
+	"os"
+	"runtime"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/release"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+)
+
+// upgradeExit is the exit status after an upgrade: non-zero, so systemd's
+// Restart=on-failure starts the new binary, and distinct, so the journal says
+// why the agent went.
+const upgradeExit = 75
+
+// exitForUpgrade ends the process; a variable so a test can see it called
+// rather than have its own process end.
+var exitForUpgrade = func() { os.Exit(upgradeExit) }
+
+// upgradeTarget is the binary an upgrade replaces; a variable so a test
+// replaces a file of its own rather than the test binary running it.
+var upgradeTarget = os.Executable
+
+// upgrade replaces this agent's binary with a release and restarts into it.
+//
+// Engines are not touched: an agent restart leaves them running and adopts
+// them again, which is what makes upgrading a node something that can be done
+// while it serves.
+func (a *Agent) upgrade(ctx context.Context, m *llmgwv1.UpgradeAgent, out chan<- *llmgwv1.AgentMessage) {
+	reply := &llmgwv1.UpgradeReply{RequestId: m.GetRequestId(), Version: m.GetVersion()}
+	send := func() {
+		select {
+		case out <- &llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_Upgrade{Upgrade: reply}}:
+		case <-ctx.Done():
+		}
+	}
+	a.log.Info("upgrading", "to", m.GetVersion(), "from", a.cfg.Version)
+	self, err := upgradeTarget()
+	if err == nil {
+		dctx, cancel := context.WithTimeout(ctx, 10*time.Minute)
+		_, err = release.Install(dctx, nil, m.GetReleases(), m.GetVersion(), runtime.GOOS, runtime.GOARCH, self)
+		cancel()
+	}
+	if err != nil {
+		reply.Error = err.Error()
+		a.log.Error("upgrade failed; still running the old version", "to", m.GetVersion(), "err", err)
+		send()
+		return
+	}
+	send()
+	a.log.Info("upgraded; restarting into the new version", "to", m.GetVersion())
+	// A moment for the reply to leave before the stream goes.
+	time.Sleep(500 * time.Millisecond)
+	exitForUpgrade()
+}

internal/agent/upgrade_test.go

@@ -0,0 +1,78 @@
+package agent
+
+import (
+	"crypto/sha256"
+	"encoding/hex"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"runtime"
+	"testing"
+	"time"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/release"
+)
+
+func TestAnAgentUpgradesItselfAndSaysSoBeforeRestarting(t *testing.T) {
+	body := []byte("the new agent")
+	sum := sha256.Sum256(body)
+	asset := release.AssetName("v9.9.9", runtime.GOOS, runtime.GOARCH)
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		switch r.URL.Path {
+		case "/v9.9.9/checksums.txt":
+			_, _ = w.Write([]byte(hex.EncodeToString(sum[:]) + "  " + asset + "\n"))
+		case "/v9.9.9/" + asset:
+			_, _ = w.Write(body)
+		default:
+			http.NotFound(w, r)
+		}
+	}))
+	defer srv.Close()
+
+	bin := filepath.Join(t.TempDir(), "llmgw")
+	_ = os.WriteFile(bin, []byte("the old agent"), 0o755)
+	exited := make(chan struct{}, 1)
+	oldTarget, oldExit := upgradeTarget, exitForUpgrade
+	upgradeTarget = func() (string, error) { return bin, nil }
+	exitForUpgrade = func() { exited <- struct{}{} }
+	t.Cleanup(func() { upgradeTarget, exitForUpgrade = oldTarget, oldExit })
+
+	cfg := Config{Server: "127.0.0.1:7443", NodeID: "n1", WeightsCache: t.TempDir(), EnginesDir: t.TempDir()}
+	cfg.Superv.WorkDir = t.TempDir()
+	a, err := New(cfg, nil, nil)
+	if err != nil {
+		t.Fatal(err)
+	}
+	out := make(chan *llmgwv1.AgentMessage, 1)
+	go a.upgrade(t.Context(), &llmgwv1.UpgradeAgent{RequestId: "u1", Version: "v9.9.9", Releases: srv.URL}, out)
+
+	select {
+	case m := <-out:
+		if r := m.GetUpgrade(); r.GetRequestId() != "u1" || r.GetError() != "" {
+			t.Fatalf("reply %+v", r)
+		}
+	case <-time.After(10 * time.Second):
+		t.Fatal("no reply")
+	}
+	select {
+	case <-exited:
+	case <-time.After(5 * time.Second):
+		t.Fatal("the agent did not restart into the new binary")
+	}
+	if b, _ := os.ReadFile(bin); string(b) != "the new agent" {
+		t.Errorf("binary holds %q", b)
+	}
+
+	// A version with no release answers with why, and does not restart.
+	go a.upgrade(t.Context(), &llmgwv1.UpgradeAgent{RequestId: "u2", Version: "v9.9.8", Releases: srv.URL}, out)
+	if r := (<-out).GetUpgrade(); r.GetError() == "" {
+		t.Error("an upgrade to a release that does not exist reported success")
+	}
+	select {
+	case <-exited:
+		t.Error("a failed upgrade restarted the agent")
+	case <-time.After(time.Second):
+	}
+}

internal/agent/weights/fetch.go

@@ -84,6 +84,11 @@ func (f *Fetcher) Ensure(ctx context.Context, source string) (string, error) {
 	if source == "" {
 		return "", fmt.Errorf("no source given")
 	}
+	// A version rather than a file. Everything below assumes one URL is one
+	// file, and a set has to be handled before any of it.
+	if IsSet(source) {
+		return f.ensureSet(ctx, source)
+	}
 	if !isRemote(source) {
 		if !isPath(source) {
 			// A bare model id — "Qwen/Qwen2.5-7B-Instruct" — is not a file and
@@ -134,6 +139,15 @@ func (f *Fetcher) Ensure(ctx context.Context, source string) (string, error) {
 
 // fetch downloads a source to a target, resuming and verifying.
 func (f *Fetcher) fetch(ctx context.Context, source, target string) (string, error) {
+	return f.fetchAs(ctx, source, source, target)
+}
+
+// fetchAs downloads one URL but reports progress under another name.
+//
+// The two differ for a set: the shards waiting on it asked for the version, not
+// for the fourth file in it, and progress reported under the file's own URL
+// reaches nobody. What they see is one number climbing across the whole set.
+func (f *Fetcher) fetchAs(ctx context.Context, source, reportAs, target string) (string, error) {
 	url, err := resolveURL(source)
 	if err != nil {
 		return "", err
@@ -192,8 +206,8 @@ func (f *Fetcher) fetch(ctx context.Context, source, target string) (string, err
 	}
 
 	total := existing + resp.ContentLength
-	f.report(Progress{Source: source, BytesDone: existing, BytesTotal: total})
-	written, err := f.copyWithProgress(ctx, out, resp.Body, source, existing, total)
+	f.report(Progress{Source: reportAs, BytesDone: existing, BytesTotal: total})
+	written, err := f.copyWithProgress(ctx, out, resp.Body, reportAs, existing, total)
 	closeErr := out.Close()
 	if err != nil {
 		// The part file is kept on purpose, so the next attempt resumes.

internal/agent/weights/set.go

@@ -0,0 +1,129 @@
+package weights
+
+import (
+	"context"
+	"encoding/json"
+	"fmt"
+	"io"
+	"net/http"
+	"os"
+	"path/filepath"
+	"strings"
+)
+
+// IsSet reports whether a source names a set of files rather than one.
+//
+// By the trailing slash, which is the whole signal: "…/v1/model.gguf" is a file
+// and "…/v1/" is the version it belongs to. A flag beside the source would be a
+// second place for the two to disagree, and the URL already distinguishes a
+// directory from a file everywhere else on the web.
+func IsSet(source string) bool {
+	return isRemote(source) && strings.HasSuffix(source, "/")
+}
+
+// manifest is what the mirror answers for a version.
+type manifest struct {
+	Model   string `json:"model"`
+	Version string `json:"version"`
+	Files   []struct {
+		Name  string `json:"name"`
+		Bytes int64  `json:"bytes"`
+	} `json:"files"`
+	Bytes int64 `json:"bytes"`
+}
+
+// ensureSet downloads every file of a version into one directory.
+//
+// A directory, because that is what a multi-file model is: vLLM is handed the
+// folder and reads the shards and the config out of it, and llama.cpp is handed
+// the first part and finds the rest beside it. Downloading them into one place
+// is what makes either possible, and a catalog entry naming one file could
+// express neither.
+//
+// The whole set or nothing. A directory with three of four shards in it loads
+// as a broken model rather than failing to load, so a partial set is left
+// unfinished under its temporary name and the next attempt starts again.
+func (f *Fetcher) ensureSet(ctx context.Context, source string) (string, error) {
+	dir, err := f.cachePath(strings.TrimSuffix(source, "/"))
+	if err != nil {
+		return "", err
+	}
+	// A complete set is marked by a file written last. Checking for the
+	// directory alone would treat an interrupted download as finished, which is
+	// exactly the failure the marker exists to prevent.
+	done := filepath.Join(dir, ".complete")
+	if _, err := os.Stat(done); err == nil {
+		return dir, nil
+	}
+
+	m, err := f.manifestFor(ctx, source)
+	if err != nil {
+		return "", err
+	}
+	if len(m.Files) == 0 {
+		return "", fmt.Errorf("%s holds no files", source)
+	}
+
+	if err := os.MkdirAll(dir, 0o755); err != nil {
+		return "", err
+	}
+	for _, file := range m.Files {
+		// Rejected rather than joined: a name with a separator in it would
+		// write outside the directory the mirror said it was in, and the
+		// mirror is a place an operator copies files into by hand.
+		if strings.ContainsAny(file.Name, `/\`) || file.Name == ".." {
+			return "", fmt.Errorf("%s: the mirror offered a file named %q", source, file.Name)
+		}
+		target := filepath.Join(dir, file.Name)
+		if st, err := os.Stat(target); err == nil && st.Size() == file.Bytes {
+			continue
+		}
+		// Reported under the set's own address, because that is what the
+		// shards are waiting on. Reusing fetch rather than writing a second
+		// downloader keeps resume, the .part rename and the progress callback
+		// in one place.
+		if _, err := f.fetchAs(ctx, source+file.Name, source, target); err != nil {
+			return "", err
+		}
+	}
+
+	if err := os.WriteFile(done, []byte(source+"\n"), 0o644); err != nil {
+		return "", err
+	}
+	return dir, nil
+}
+
+// manifestFor asks the mirror what a version consists of.
+func (f *Fetcher) manifestFor(ctx context.Context, source string) (manifest, error) {
+	req, err := http.NewRequestWithContext(ctx, http.MethodGet, source, nil)
+	if err != nil {
+		return manifest{}, err
+	}
+	client := f.Client
+	if client == nil {
+		client = &http.Client{}
+	}
+	resp, err := client.Do(req)
+	if err != nil {
+		return manifest{}, fmt.Errorf("asking %s what it holds: %w", source, err)
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode != http.StatusOK {
+		return manifest{}, fmt.Errorf("%s answered %d", source, resp.StatusCode)
+	}
+
+	var m manifest
+	body, err := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
+	if err != nil {
+		return manifest{}, err
+	}
+	if err := json.Unmarshal(body, &m); err != nil {
+		// The likely cause is a plain file server with a directory index, which
+		// answers HTML. Said plainly, because "invalid character '<'" sends
+		// somebody looking at the wrong thing.
+		return manifest{}, fmt.Errorf(
+			"%s did not answer with a manifest; a source ending in / has to be a "+
+				"mirror version, not a directory listing", source)
+	}
+	return m, nil
+}

internal/agent/weights/set_test.go

@@ -0,0 +1,167 @@
+package weights
+
+import (
+	"fmt"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+)
+
+// mirrorOf serves a manifest at "/m/v/" and the files beneath it.
+func mirrorOf(t *testing.T, files map[string]string) (*Fetcher, *httptest.Server, *int) {
+	t.Helper()
+	hits := 0
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		if strings.HasSuffix(r.URL.Path, "/") {
+			var b strings.Builder
+			b.WriteString(`{"model":"m","version":"v","files":[`)
+			first := true
+			total := 0
+			for name, body := range files {
+				if !first {
+					b.WriteString(",")
+				}
+				first = false
+				fmt.Fprintf(&b, `{"name":%q,"bytes":%d}`, name, len(body))
+				total += len(body)
+			}
+			fmt.Fprintf(&b, `],"bytes":%d}`, total)
+			_, _ = w.Write([]byte(b.String()))
+			return
+		}
+		body, ok := files[strings.TrimPrefix(r.URL.Path, "/m/v/")]
+		if !ok {
+			http.NotFound(w, r)
+			return
+		}
+		hits++
+		_, _ = w.Write([]byte(body))
+	}))
+	t.Cleanup(srv.Close)
+	return &Fetcher{CacheDir: t.TempDir()}, srv, &hits
+}
+
+func TestASetIsFetchedIntoOneDirectory(t *testing.T) {
+	f, srv, _ := mirrorOf(t, map[string]string{
+		"model-00001-of-00002.safetensors": "aaaa",
+		"model-00002-of-00002.safetensors": "bb",
+		"config.json":                      "{}",
+	})
+
+	got, err := f.Ensure(t.Context(), srv.URL+"/m/v/")
+	if err != nil {
+		t.Fatal(err)
+	}
+	st, err := os.Stat(got)
+	if err != nil || !st.IsDir() {
+		t.Fatalf("expected a directory, got %q: %v", got, err)
+	}
+	// A directory, because that is what a multi-file model is: vLLM reads the
+	// shards and the config out of the folder, and a catalog entry naming one
+	// file could express neither.
+	for _, name := range []string{"model-00001-of-00002.safetensors", "config.json"} {
+		if _, err := os.Stat(filepath.Join(got, name)); err != nil {
+			t.Errorf("%s did not arrive: %v", name, err)
+		}
+	}
+}
+
+func TestAFinishedSetIsNotFetchedAgain(t *testing.T) {
+	f, srv, hits := mirrorOf(t, map[string]string{"a.gguf": "aaaa", "b.gguf": "bb"})
+
+	if _, err := f.Ensure(t.Context(), srv.URL+"/m/v/"); err != nil {
+		t.Fatal(err)
+	}
+	after := *hits
+	if _, err := f.Ensure(t.Context(), srv.URL+"/m/v/"); err != nil {
+		t.Fatal(err)
+	}
+	if *hits != after {
+		t.Errorf("the second call downloaded again: %d then %d", after, *hits)
+	}
+}
+
+func TestAHalfFetchedSetIsNotTreatedAsFinished(t *testing.T) {
+	f, srv, _ := mirrorOf(t, map[string]string{"a.gguf": "aaaa", "b.gguf": "bb"})
+	dir, err := f.cachePath(srv.URL + "/m/v")
+	if err != nil {
+		t.Fatal(err)
+	}
+	// One file present, no completion marker — an interrupted download.
+	if err := os.MkdirAll(dir, 0o755); err != nil {
+		t.Fatal(err)
+	}
+	if err := os.WriteFile(filepath.Join(dir, "a.gguf"), []byte("aaaa"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+
+	got, err := f.Ensure(t.Context(), srv.URL+"/m/v/")
+	if err != nil {
+		t.Fatal(err)
+	}
+	// A directory with three of four shards loads as a broken model rather
+	// than failing to load, so the marker is what says the set is whole.
+	if _, err := os.Stat(filepath.Join(got, "b.gguf")); err != nil {
+		t.Errorf("the missing file was not fetched: %v", err)
+	}
+}
+
+func TestAFileAlreadyThereAtTheRightSizeIsNotRefetched(t *testing.T) {
+	f, srv, hits := mirrorOf(t, map[string]string{"a.gguf": "aaaa", "b.gguf": "bb"})
+	dir, _ := f.cachePath(srv.URL + "/m/v")
+	_ = os.MkdirAll(dir, 0o755)
+	_ = os.WriteFile(filepath.Join(dir, "a.gguf"), []byte("aaaa"), 0o644)
+
+	if _, err := f.Ensure(t.Context(), srv.URL+"/m/v/"); err != nil {
+		t.Fatal(err)
+	}
+	// Resuming a set means not paying for the shards that already landed.
+	if *hits != 1 {
+		t.Errorf("expected only the missing file to be fetched, got %d downloads", *hits)
+	}
+}
+
+func TestAFilenameThatEscapesTheDirectoryIsRefused(t *testing.T) {
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+		_, _ = w.Write([]byte(`{"files":[{"name":"../../etc/passwd","bytes":1}]}`))
+	}))
+	defer srv.Close()
+	f := &Fetcher{CacheDir: t.TempDir()}
+
+	_, err := f.Ensure(t.Context(), srv.URL+"/m/v/")
+	// The mirror is a place an operator copies files into by hand, so what it
+	// reports is not automatically safe to join onto a path.
+	if err == nil || !strings.Contains(err.Error(), "offered a file named") {
+		t.Errorf("expected the name to be refused, got %v", err)
+	}
+}
+
+func TestADirectoryListingIsNotAManifest(t *testing.T) {
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+		_, _ = w.Write([]byte("<pre><a href=\"a.gguf\">a.gguf</a></pre>"))
+	}))
+	defer srv.Close()
+	f := &Fetcher{CacheDir: t.TempDir()}
+
+	_, err := f.Ensure(t.Context(), srv.URL+"/m/v/")
+	// "invalid character '<'" sends somebody looking at the wrong thing.
+	if err == nil || !strings.Contains(err.Error(), "has to be a mirror version") {
+		t.Errorf("the reason should name the cause, got %v", err)
+	}
+}
+
+func TestOnlyATrailingSlashMeansASet(t *testing.T) {
+	for src, want := range map[string]bool{
+		"http://mirror/m/v/":           true,
+		"http://mirror/m/v/model.gguf": false,
+		"/var/lib/models/":             false, // a local directory is not fetched
+		"Qwen/Qwen2.5-7B-Instruct":     false, // a repo id, resolved by the engine
+	} {
+		if got := IsSet(src); got != want {
+			t.Errorf("IsSet(%q) = %v, want %v", src, got, want)
+		}
+	}
+}

internal/auth/auth.go

@@ -21,6 +21,9 @@ import (
 type Principal struct {
 	KeyID     string
 	KeyPrefix string
+	// UserID is the person the key belongs to, empty for a key an operator
+	// issued in a project. A personal key has a person and no project.
+	UserID    string
 	ProjectID string
 	TeamID    string
 	OrgID     string
@@ -186,7 +189,7 @@ func (a *Authenticator) resolve(ctx context.Context, hash []byte) (Principal, er
 		return Principal{}, err
 	}
 	return Principal{
-		KeyID: k.ID, KeyPrefix: k.Prefix, ProjectID: eff.ProjectID,
+		KeyID: k.ID, KeyPrefix: k.Prefix, UserID: eff.UserID, ProjectID: eff.ProjectID,
 		TeamID: eff.TeamID, OrgID: eff.OrgID,
 		Band: eff.Band, RoutingProfile: eff.RoutingProfile,
 		Limits: eff,

internal/auth/auth_test.go

@@ -16,7 +16,7 @@ import (
 )
 
 // fixture builds a store with one org/team/project and one key.
-func fixture(t *testing.T) (*store.SQLite, string, string) {
+func fixture(t *testing.T) (*store.DB, string, string) {
 	t.Helper()
 	ctx := context.Background()
 	s, err := store.OpenSQLite(ctx, filepath.Join(t.TempDir(), "t.db"))

internal/bench/bench.go

@@ -225,6 +225,7 @@ func runStep(ctx context.Context, client *http.Client, cfg Config, concurrency i
 	)
 
 	sampler := newSampler(cfg.MetricsURL, cfg.Engine)
+	sampler.apiKey = cfg.APIKey
 	sampler.start(ctx)
 
 	start := time.Now()

internal/bench/capacity.go

@@ -0,0 +1,77 @@
+package bench
+
+import "time"
+
+// Capacity is what one replica was measured to serve, kept so the measurement
+// outlives the terminal it was printed in.
+//
+// The conclusion of a run, not the run. The ramp itself — every concurrency
+// level, its latencies, its errors — is worth reading once, while deciding
+// whether the run was sound; it is not worth carrying around afterwards, and
+// storing it would grow without bound as a model is re-benchmarked. What a
+// later reader needs is the number and enough provenance to judge whether it is
+// still true.
+type Capacity struct {
+	Model   string `json:"model"`
+	Profile string `json:"profile"`
+
+	// MaxConcurrencyAtSLO is the capacity: the highest concurrency that still
+	// met the target. Zero means even one request did not, which is a real
+	// result and not a missing one — it says this hardware cannot serve this
+	// model at this SLO at all.
+	MaxConcurrencyAtSLO int `json:"max_concurrency_at_slo"`
+	// ThroughputTokPerSec and the latencies are taken at that level, not at the
+	// peak. Peak throughput arrives long after latency has become unusable, so
+	// reporting it would overstate what the hardware can serve.
+	ThroughputTokPerSec float64       `json:"throughput_tok_per_sec"`
+	TTFTp99             time.Duration `json:"ttft_p99"`
+	TPOTp99             time.Duration `json:"tpot_p99"`
+
+	// SuggestedTargetWaiting and SuggestedKVCeiling are the queue depth and
+	// cache pressure seen while the SLO was still being met, which is what
+	// admission should treat as healthy for this model. Admission currently
+	// applies one global default to every model; these are the per-model
+	// numbers it should be using instead.
+	SuggestedTargetWaiting int     `json:"suggested_target_waiting"`
+	SuggestedKVCeiling     float64 `json:"suggested_kv_ceiling"`
+
+	// SLO is recorded because capacity is meaningless without it. The same
+	// replica has a different capacity against a 200ms first-token target than
+	// against a 2s one, and a number stored without the target it was measured
+	// against cannot be compared with anything.
+	SLO SLO `json:"slo"`
+
+	// Target is the endpoint the load actually went to, kept so a profile
+	// measured against the gateway is never mistaken for one measured against
+	// a replica. Only the second says anything about the hardware.
+	Target     string    `json:"target"`
+	MeasuredAt time.Time `json:"measured_at"`
+	// Stale is not a field. Whether a measurement still holds depends on
+	// whether the hardware and the engine build have changed since, which this
+	// package cannot know; MeasuredAt is given so the reader can decide.
+}
+
+// Capacity summarizes a completed run for storage.
+func (r Result) Capacity(profile string) Capacity {
+	return Capacity{
+		Model:                  r.Model,
+		Profile:                profile,
+		MaxConcurrencyAtSLO:    r.MaxConcurrencyAtSLO,
+		ThroughputTokPerSec:    r.ThroughputTokPerSec,
+		TTFTp99:                r.TTFTp99,
+		TPOTp99:                r.TPOTp99,
+		SuggestedTargetWaiting: r.SuggestedTargetWaiting,
+		SuggestedKVCeiling:     r.SuggestedKVCeiling,
+		SLO:                    r.SLO,
+		Target:                 r.Target,
+		MeasuredAt:             r.RanAt,
+	}
+}
+
+// Usable reports whether this measurement says anything.
+//
+// A run that never met the SLO, or one stopped before it finished a level,
+// concludes at zero. That is worth recording — it is the answer to "can this
+// run here at all" — but a caller sizing a model against it would compute a
+// replica count of infinity, so it has to be asked about rather than used.
+func (c Capacity) Usable() bool { return c.MaxConcurrencyAtSLO > 0 }