Keep each held row on one line

The replica count, its nodes and the mode selector sit side by side instead
of stacking in a fixed-width cell, and node names drop the prefix every node
shares, with the full name on hover.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

CHANGELOG.md

@@ -20,6 +20,240 @@ is only released once its **exit criterion** is met, not when its code lands.
 Until v1.0.0 the minor version is the breaking one: configuration formats and Go
 APIs change between minors without a deprecation period.
 
+## Unreleased
+
+Changes since v0.10.4, published as release candidates (v0.10.5-rc.N) until
+there are enough for v0.10.5.
+
+### The mirror page in tabs
+
+/admin/mirror was one long page: what the mirror holds, the Hugging Face
+explorer and the fetch form, one under another, with every visit waiting on
+Hugging Face for a list it might not show. It is now three tabs — Held (with
+how many are held, downloading and failed), Explore Hugging Face, and Fetch by
+URL — each with its own address. Only the tab shown does its work, and only
+Held reloads while something downloads. A download started from the explorer
+links to Held, where its progress is; a fetch the mirror refused comes back on
+the form it was sent from.
+
+### A held model opens to where it runs, and how many
+
+Clicking a model under Held opens a card beneath it: each replica with its
+node, state, age and the CPU and memory its engine uses, and how many more
+replicas the cluster has room for now, on which nodes, and why not more. The
+replicas it always runs and the most it grows to under load are changed there.
+Room is found by trying the scheduler itself on a copy of the inventory — the
+same decisions the next passes would make, not an estimate beside them — and a
+number of always-running replicas it cannot place is refused with the reason,
+rather than accepted and left short. A ceiling above the room is taken, and
+said. The same answer is at `GET /api/v1/models/{name}/headroom` on the admin
+API.
+
+Each row also says how its model runs without being opened: replicas ready of
+those it should have, how many are still starting, and the nodes they are on —
+in amber when short, in red when parked or when it does not fit. It is one line
+per model beside its mode, and nodes drop the part of the name every node
+shares (node05 for llmgw-node05, the full name on hover).
+
+## v0.10.4 — 2026-09-30
+
+Published first as release candidates v0.10.4-rc.1 to rc.9.
+
+After upgrading, run `llmgw ctl restart --all` once: CPU replicas take the new
+thread count below only when they start.
+
+### CPU replicas no longer fight over the CPUs
+
+llama.cpp computes on every CPU of the machine unless told otherwise, and the
+agent did not tell it. Two busy CPU replicas on one node then spin against each
+other: on an 8-vCPU node, 7B generation fell from 10 tokens a second alone to
+0.8 each with both busy, in two rounds of the same test. The agent now starts a
+CPU replica on half the machine's CPUs, where one alone makes 6.5 and two busy
+make 5.5 each. `llamacpp.threads` in the agent's configuration sets it for a
+node — all of them suits a node that only ever runs one CPU replica — and a
+profile's own `--threads` still wins. Measured on VMs whose host is
+oversubscribed; the proportions, not the numbers, are the point.
+
+### Tracing
+
+With `tracing.endpoint` set (or `OTEL_EXPORTER_OTLP_ENDPOINT`), each inference
+request is traced over OTLP/HTTP: a span for the request, carrying its
+`X-Request-Id`, with admission, scheduling and each attempt at an engine under
+it. A caller's own `traceparent` is continued, and the engine is sent one for
+its attempt. `tracing.sample_ratio` traces a share of the requests that arrive
+without a decision. Off, it costs nothing.
+
+### Restarting replicas
+
+`llmgw ctl restart REPLICA... | --model M | --all` replaces replicas with new
+ones. Each replacement is started beside the replica it replaces, which keeps
+serving until the new one is ready; replicas of one model go one at a time.
+Where no second copy fits anywhere, the old replica is stopped after the drain
+grace period and the new one starts in its place. This is how running engines
+pick up the keys introduced in v0.10.3, an edited profile, or a new engine
+version.
+
+### A model that keeps failing is parked, and everybody is told
+
+A model whose replicas failed at every start was retried forever: kev-0.5b,
+whose architecture the nodes' llama.cpp did not know, was started 155 times in
+eight hours. After five failures in a row a model is now parked — not placed
+again until its definition changes or `llmgw ctl retry MODEL` is run — and
+`ctl status`, `ctl models` and the admin API say so with the last error.
+
+A request for a model that will not come up — parked, or fitting on no node —
+is answered at once with the reason, instead of "the model is loading; retry
+shortly", which a client would retry forever. Each request still makes the
+control plane try placing it afresh, so a model that fits once room is made is
+not refused on an old answer. "no placement for model" is logged when its
+reason changes or every ten minutes, not on every pass: it was 250 identical
+lines in two days for one model.
+
+### Metrics and alerts for what goes wrong
+
+New series: `llmgw_model_replicas_desired`, `llmgw_model_start_failures`,
+`llmgw_model_parked`, `llmgw_model_unplaceable`, `llmgw_node_engine_install`,
+`llmgw_control_plane_leader` and `llmgw_certificate_expiry_timestamp_seconds`.
+`llmgw_model_replicas_ready` is now published at 0 for every model that should
+be running; it was published only while a replica was ready, so
+ModelHasNothingServing could never fire. New alerts: ModelParked,
+ModelFailingToStart, ModelCannotBePlaced, EngineInstallFailed,
+ControlPlaneLeadership, CertificateExpiresSoon and CertificateExpiresVerySoon.
+
+`ctl status` counts a model against the replicas it should have now rather
+than its minimum, so an on-demand model that was asked for and fitted nowhere
+no longer reads as satisfied.
+
+### Backups
+
+`llmgw server backup` writes the store and `/etc/llmgw` — configuration,
+secrets, and the certificate authority's key — to one archive readable by root
+only, keeping the newest 14. `deploy/systemd/llmgw-backup.timer` runs it
+nightly. The control plane publishes the newest archive's time as
+`llmgw_backup_last_success_timestamp_seconds`, and BackupTooOld fires when it is
+more than 26 hours old or there has never been one. docs/install.md now says how
+to restore, including why running engines survive it: the engine-key secret
+comes back with the store.
+
+### Replicas follow load between min and max
+
+A model's `max` was a ceiling nothing reached: every model ran exactly its
+minimum however long its queue. Now, when requests are waiting at the engines —
+every slot of every replica busy — at one or more per replica for a minute, one
+replica is added, up to `max`; ten minutes with nothing waiting takes one of
+those away, down to `min`. One step at a time, and not while the previous one
+is still starting, so a spike that is over before a replica loads does not
+leave several behind. Each decision is logged with the load that made it.
+`llmgw ctl scale MODEL --min 1 --max 3` turns it on for a model.
+
+### A model from the registry in one command
+
+`llmgw ctl model serve OWNER/REPO` takes a model from the registry to a replica
+that answers. It picks the GGUF file (`--quant`, Q4_K_M by default), has the
+mirror fetch it and checks what arrived against the registry's published
+sha256, checks that the nodes' llama.cpp can load it, sizes its memory from its
+header, adds it to the catalog, waits for a replica — stopping early with the
+reason if it fits nowhere or keeps failing — and asks it to say hello. Each step
+is shown as it happens. It runs in the control plane as a job, so a download of
+hours does not depend on a terminal staying open. `--on-demand` adds it without
+starting a replica.
+
+### Upgrading from one command
+
+`sudo llmgw ctl upgrade VERSION`, run on the control plane, upgrades that
+machine and then every node in turn. Each agent is told the version over its
+existing stream, fetches the release for its own platform, checks it against
+the release's checksums, keeps the old binary beside the new and restarts,
+while its engines keep serving. A node that does not come back on the new
+version, ready, with its replicas, stops the upgrade before the next one.
+Running it again resumes. Agents from before this have to be moved by hand
+once.
+
+### Following a request
+
+Every response carries `X-Request-Id` — the client's own when it sends a plain
+one, otherwise one the gateway makes — and the engine receives it too. One
+access log line per request records it with the model asked for and served,
+the replica, status, time queued, time to the first token, total time,
+tokens, and the key, project and user. Usage records keep the id, replica,
+queue time and time to first token, and `llmgw ctl request ID` reads one back:
+the answer to "why was this slow?".
+
+### Rate-limit headers
+
+Responses carry OpenAI's `x-ratelimit-limit-*`, `x-ratelimit-remaining-*` and
+`x-ratelimit-reset-*` for requests and tokens, from whichever level of the
+tenancy tree will refuse first, so SDKs and editors can slow down before they
+are refused.
+
+### Memory sized from the model
+
+The mirror's serve form sizes a llama.cpp model's memory from its GGUF header:
+weights, plus the KV cache for the context pool from its layers, KV heads and
+head width, plus the engine's own. At a 16k pool that cache ranges from 0.44 GB
+to 5.25 GB across the models on the live mirror, which the old "weights plus
+half again plus two" could not see.
+
+### Security
+
+- **Alertmanager answered anyone on the network**, including requests to
+  silence alerts. The monitoring README now binds it to loopback, where
+  Prometheus beside it is its only client.
+- **Login and signup could be made to use gigabytes.** Each attempt hashes with
+  64 MiB of argon2, with no bound on how many ran at once, and the per-address
+  throttle grew by one entry per address tried. Hashes now run a few at a time
+  and the rest wait; the throttle expires old entries, and when it is still
+  full it refuses new addresses rather than dropping live ones — which would
+  have let a flood of made-up addresses reset the throttle on a real one.
+- **Any key holder could grow Prometheus without bound** by asking for made-up
+  model names, each of which became a metric label. Requests are counted under
+  the catalog's name for their model, or "unknown".
+- **A SQLite backup left a full copy of the store readable** by local users
+  while it ran. The snapshot is now made in a private directory, and
+  llmgw-backup.service sets `UMask=0077`.
+- **An engine archive could write outside its directory through chained
+  links.** Archives are unpacked through `os.Root`, and a link may only point
+  at a file beside it. Every llama.cpp b10991 archive, CPU and CUDA, already
+  follows that rule.
+- **Onboarding made every node trust the cluster's root for everything.** It
+  was added to the system store, so a certificate the control plane signed
+  would pass for any site with every TLS client on the node. The agent trusts
+  it through its own configuration only; onboarding no longer adds it, and
+  takes it out of a node that an older version onboarded.
+- **Any node could manage a mirror running mutual TLS**, since one root signs
+  every node's certificate. `llmgw mirror serve --api-client NAME` answers only
+  the certificate with that common name, and the mirror warns when mutual TLS
+  runs without one.
+- **The administrator's password stayed in `init.log`.** It is shown once, on
+  the screen, and the log keeps a placeholder.
+- **Auth off on a public address was silent.** The server now warns at start
+  when the API listens beyond loopback without keys.
+- **The actor recorded with a change was whatever the caller sent**, line
+  breaks included. It is cut to a short printable name, and `sudo llmgw ctl`
+  records the person who ran sudo rather than root.
+- **vLLM's Ray head ran its dashboard**, whose job API runs code for anyone who
+  reaches it. It is off. The head itself still has no authentication, and
+  SGLang still takes its key as a flag; docs/install.md says what to do about
+  both. The monitoring README says how to keep Prometheus's queries to the
+  control plane.
+
+### Fixes
+
+- Tests that slept a fixed time and then asserted now wait for their
+  condition. They are the likeliest cause of CI's intermittent failures on a
+  loaded runner.
+- `ctl` reported a change the API had queued (202 Accepted) as an error, as
+  `ctl restart` did on its first real use.
+- The status each failure is answered with — 499, 502, 503 with and without
+  Retry-After, 504, 429 — is now tested case by case.
+
+### Releases
+
+Small fixes are published as release candidates of the next version —
+v0.10.4-rc.1, rc.2 — which sort after v0.10.3 and before v0.10.4. Their release
+notes come from this section, and they are marked as pre-releases. Onboarding
+treats a release candidate as a release and downloads it.
+
 ## v0.10.3 — 2026-09-30
 
 Security fixes from a review of v0.10.2. Three of them were confirmed on the

api/gen/llmgw/v1/agent.pb.go

@@ -104,6 +104,7 @@ type AgentMessage struct {
 	//	*AgentMessage_Logs
 	//	*AgentMessage_FabricScan
 	//	*AgentMessage_FabricTest
+	//	*AgentMessage_Upgrade
 	Msg           isAgentMessage_Msg `protobuf_oneof:"msg"`
 	unknownFields protoimpl.UnknownFields
 	sizeCache     protoimpl.SizeCache
@@ -200,6 +201,15 @@ func (x *AgentMessage) GetFabricTest() *FabricTestReply {
 	return nil
 }
 
+func (x *AgentMessage) GetUpgrade() *UpgradeReply {
+	if x != nil {
+		if x, ok := x.Msg.(*AgentMessage_Upgrade); ok {
+			return x.Upgrade
+		}
+	}
+	return nil
+}
+
 type isAgentMessage_Msg interface {
 	isAgentMessage_Msg()
 }
@@ -228,6 +238,10 @@ type AgentMessage_FabricTest struct {
 	FabricTest *FabricTestReply `protobuf:"bytes,6,opt,name=fabric_test,json=fabricTest,proto3,oneof"`
 }
 
+type AgentMessage_Upgrade struct {
+	Upgrade *UpgradeReply `protobuf:"bytes,7,opt,name=upgrade,proto3,oneof"`
+}
+
 func (*AgentMessage_Register) isAgentMessage_Msg() {}
 
 func (*AgentMessage_Heartbeat) isAgentMessage_Msg() {}
@@ -240,6 +254,8 @@ func (*AgentMessage_FabricScan) isAgentMessage_Msg() {}
 
 func (*AgentMessage_FabricTest) isAgentMessage_Msg() {}
 
+func (*AgentMessage_Upgrade) isAgentMessage_Msg() {}
+
 // ControlMessage is anything the control plane tells a node.
 type ControlMessage struct {
 	state protoimpl.MessageState `protogen:"open.v1"`
@@ -254,6 +270,7 @@ type ControlMessage struct {
 	//	*ControlMessage_Logs
 	//	*ControlMessage_FabricScan
 	//	*ControlMessage_FabricTest
+	//	*ControlMessage_Upgrade
 	Msg           isControlMessage_Msg `protobuf_oneof:"msg"`
 	unknownFields protoimpl.UnknownFields
 	sizeCache     protoimpl.SizeCache
@@ -377,6 +394,15 @@ func (x *ControlMessage) GetFabricTest() *FabricTestRequest {
 	return nil
 }
 
+func (x *ControlMessage) GetUpgrade() *UpgradeAgent {
+	if x != nil {
+		if x, ok := x.Msg.(*ControlMessage_Upgrade); ok {
+			return x.Upgrade
+		}
+	}
+	return nil
+}
+
 type isControlMessage_Msg interface {
 	isControlMessage_Msg()
 }
@@ -417,6 +443,10 @@ type ControlMessage_FabricTest struct {
 	FabricTest *FabricTestRequest `protobuf:"bytes,9,opt,name=fabric_test,json=fabricTest,proto3,oneof"`
 }
 
+type ControlMessage_Upgrade struct {
+	Upgrade *UpgradeAgent `protobuf:"bytes,10,opt,name=upgrade,proto3,oneof"`
+}
+
 func (*ControlMessage_Start) isControlMessage_Msg() {}
 
 func (*ControlMessage_Stop) isControlMessage_Msg() {}
@@ -435,6 +465,8 @@ func (*ControlMessage_FabricScan) isControlMessage_Msg() {}
 
 func (*ControlMessage_FabricTest) isControlMessage_Msg() {}
 
+func (*ControlMessage_Upgrade) isControlMessage_Msg() {}
+
 // Register is the first message on every stream. It carries the node's full
 // hardware inventory, so the control plane needs no separate discovery call and
 // a reconnecting agent automatically refreshes what the scheduler knows.
@@ -3460,11 +3492,144 @@ func (x *FabricTestReply) GetTookMs() int64 {
 	return 0
 }
 
+// UpgradeAgent tells an agent to replace its own binary with a release and
+// restart.
+//
+// The agent fetches the release for its own platform and checks it against
+// the release's checksums before replacing anything, keeps the old binary
+// beside the new, then exits so systemd starts the new one. Engines keep
+// running throughout, as for any agent restart, and are adopted on the way
+// back. Over the stream the agent already holds, so upgrading a fleet needs
+// no ssh into it.
+type UpgradeAgent struct {
+	state     protoimpl.MessageState `protogen:"open.v1"`
+	RequestId string                 `protobuf:"bytes,1,opt,name=request_id,json=requestId,proto3" json:"request_id,omitempty"`
+	// version must be a release: v0.10.4 or v0.10.4-rc.2.
+	Version string `protobuf:"bytes,2,opt,name=version,proto3" json:"version,omitempty"`
+	// releases is the base URL release files are downloaded from.
+	Releases      string `protobuf:"bytes,3,opt,name=releases,proto3" json:"releases,omitempty"`
+	unknownFields protoimpl.UnknownFields
+	sizeCache     protoimpl.SizeCache
+}
+
+func (x *UpgradeAgent) Reset() {
+	*x = UpgradeAgent{}
+	mi := &file_llmgw_v1_agent_proto_msgTypes[33]
+	ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+	ms.StoreMessageInfo(mi)
+}
+
+func (x *UpgradeAgent) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*UpgradeAgent) ProtoMessage() {}
+
+func (x *UpgradeAgent) ProtoReflect() protoreflect.Message {
+	mi := &file_llmgw_v1_agent_proto_msgTypes[33]
+	if x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use UpgradeAgent.ProtoReflect.Descriptor instead.
+func (*UpgradeAgent) Descriptor() ([]byte, []int) {
+	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{33}
+}
+
+func (x *UpgradeAgent) GetRequestId() string {
+	if x != nil {
+		return x.RequestId
+	}
+	return ""
+}
+
+func (x *UpgradeAgent) GetVersion() string {
+	if x != nil {
+		return x.Version
+	}
+	return ""
+}
+
+func (x *UpgradeAgent) GetReleases() string {
+	if x != nil {
+		return x.Releases
+	}
+	return ""
+}
+
+// UpgradeReply says whether the new binary is in place. Sent before the
+// agent exits; an empty error means it will be back on the new version.
+type UpgradeReply struct {
+	state         protoimpl.MessageState `protogen:"open.v1"`
+	RequestId     string                 `protobuf:"bytes,1,opt,name=request_id,json=requestId,proto3" json:"request_id,omitempty"`
+	Version       string                 `protobuf:"bytes,2,opt,name=version,proto3" json:"version,omitempty"`
+	Error         string                 `protobuf:"bytes,3,opt,name=error,proto3" json:"error,omitempty"`
+	unknownFields protoimpl.UnknownFields
+	sizeCache     protoimpl.SizeCache
+}
+
+func (x *UpgradeReply) Reset() {
+	*x = UpgradeReply{}
+	mi := &file_llmgw_v1_agent_proto_msgTypes[34]
+	ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+	ms.StoreMessageInfo(mi)
+}
+
+func (x *UpgradeReply) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*UpgradeReply) ProtoMessage() {}
+
+func (x *UpgradeReply) ProtoReflect() protoreflect.Message {
+	mi := &file_llmgw_v1_agent_proto_msgTypes[34]
+	if x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use UpgradeReply.ProtoReflect.Descriptor instead.
+func (*UpgradeReply) Descriptor() ([]byte, []int) {
+	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{34}
+}
+
+func (x *UpgradeReply) GetRequestId() string {
+	if x != nil {
+		return x.RequestId
+	}
+	return ""
+}
+
+func (x *UpgradeReply) GetVersion() string {
+	if x != nil {
+		return x.Version
+	}
+	return ""
+}
+
+func (x *UpgradeReply) GetError() string {
+	if x != nil {
+		return x.Error
+	}
+	return ""
+}
+
 var File_llmgw_v1_agent_proto protoreflect.FileDescriptor
 
 const file_llmgw_v1_agent_proto_rawDesc = "" +
 	"\n" +
-	"\x14llmgw/v1/agent.proto\x12\bllmgw.v1\"\xd2\x02\n" +
+	"\x14llmgw/v1/agent.proto\x12\bllmgw.v1\"\x86\x03\n" +
 	"\fAgentMessage\x120\n" +
 	"\bregister\x18\x01 \x01(\v2\x12.llmgw.v1.RegisterH\x00R\bregister\x123\n" +
 	"\theartbeat\x18\x02 \x01(\v2\x13.llmgw.v1.HeartbeatH\x00R\theartbeat\x12-\n" +
@@ -3473,8 +3638,9 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
 	"\vfabric_scan\x18\x05 \x01(\v2\x19.llmgw.v1.FabricScanReplyH\x00R\n" +
 	"fabricScan\x12<\n" +
 	"\vfabric_test\x18\x06 \x01(\v2\x19.llmgw.v1.FabricTestReplyH\x00R\n" +
-	"fabricTestB\x05\n" +
-	"\x03msg\"\xe5\x03\n" +
+	"fabricTest\x122\n" +
+	"\aupgrade\x18\a \x01(\v2\x16.llmgw.v1.UpgradeReplyH\x00R\aupgradeB\x05\n" +
+	"\x03msg\"\x99\x04\n" +
 	"\x0eControlMessage\x12,\n" +
 	"\x05start\x18\x01 \x01(\v2\x14.llmgw.v1.StartShardH\x00R\x05start\x12)\n" +
 	"\x04stop\x18\x02 \x01(\v2\x13.llmgw.v1.StopShardH\x00R\x04stop\x12,\n" +
@@ -3486,7 +3652,9 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
 	"\vfabric_scan\x18\b \x01(\v2\x1b.llmgw.v1.FabricScanRequestH\x00R\n" +
 	"fabricScan\x12>\n" +
 	"\vfabric_test\x18\t \x01(\v2\x1b.llmgw.v1.FabricTestRequestH\x00R\n" +
-	"fabricTestB\x05\n" +
+	"fabricTest\x122\n" +
+	"\aupgrade\x18\n" +
+	" \x01(\v2\x16.llmgw.v1.UpgradeAgentH\x00R\aupgradeB\x05\n" +
 	"\x03msg\"\xe3\x06\n" +
 	"\bRegister\x12\x17\n" +
 	"\anode_id\x18\x01 \x01(\tR\x06nodeId\x12\x1a\n" +
@@ -3795,7 +3963,17 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
 	"\x04gbps\x18\x02 \x01(\x01R\x04gbps\x12\x16\n" +
 	"\x06output\x18\x03 \x01(\tR\x06output\x12\x14\n" +
 	"\x05error\x18\x04 \x01(\tR\x05error\x12\x17\n" +
-	"\atook_ms\x18\x05 \x01(\x03R\x06tookMs2O\n" +
+	"\atook_ms\x18\x05 \x01(\x03R\x06tookMs\"c\n" +
+	"\fUpgradeAgent\x12\x1d\n" +
+	"\n" +
+	"request_id\x18\x01 \x01(\tR\trequestId\x12\x18\n" +
+	"\aversion\x18\x02 \x01(\tR\aversion\x12\x1a\n" +
+	"\breleases\x18\x03 \x01(\tR\breleases\"]\n" +
+	"\fUpgradeReply\x12\x1d\n" +
+	"\n" +
+	"request_id\x18\x01 \x01(\tR\trequestId\x12\x18\n" +
+	"\aversion\x18\x02 \x01(\tR\aversion\x12\x14\n" +
+	"\x05error\x18\x03 \x01(\tR\x05error2O\n" +
 	"\fAgentService\x12?\n" +
 	"\aConnect\x12\x16.llmgw.v1.AgentMessage\x1a\x18.llmgw.v1.ControlMessage(\x010\x01B\x93\x01\n" +
 	"\fcom.llmgw.v1B\n" +
@@ -3814,7 +3992,7 @@ func file_llmgw_v1_agent_proto_rawDescGZIP() []byte {
 }
 
 var file_llmgw_v1_agent_proto_enumTypes = make([]protoimpl.EnumInfo, 1)
-var file_llmgw_v1_agent_proto_msgTypes = make([]protoimpl.MessageInfo, 39)
+var file_llmgw_v1_agent_proto_msgTypes = make([]protoimpl.MessageInfo, 41)
 var file_llmgw_v1_agent_proto_goTypes = []any{
 	(ShardStatus_State)(0),    // 0: llmgw.v1.ShardStatus.State
 	(*AgentMessage)(nil),      // 1: llmgw.v1.AgentMessage
@@ -3850,12 +4028,14 @@ var file_llmgw_v1_agent_proto_goTypes = []any{
 	(*FabricScanReply)(nil),   // 31: llmgw.v1.FabricScanReply
 	(*FabricTestRequest)(nil), // 32: llmgw.v1.FabricTestRequest
 	(*FabricTestReply)(nil),   // 33: llmgw.v1.FabricTestReply
-	nil,                       // 34: llmgw.v1.Register.LabelsEntry
-	nil,                       // 35: llmgw.v1.Register.EngineVersionsEntry
-	nil,                       // 36: llmgw.v1.GPU.NicPathsEntry
-	nil,                       // 37: llmgw.v1.IBPort.CountersEntry
-	nil,                       // 38: llmgw.v1.Heartbeat.EngineVersionsEntry
-	nil,                       // 39: llmgw.v1.EngineProfile.EnvEntry
+	(*UpgradeAgent)(nil),      // 34: llmgw.v1.UpgradeAgent
+	(*UpgradeReply)(nil),      // 35: llmgw.v1.UpgradeReply
+	nil,                       // 36: llmgw.v1.Register.LabelsEntry
+	nil,                       // 37: llmgw.v1.Register.EngineVersionsEntry
+	nil,                       // 38: llmgw.v1.GPU.NicPathsEntry
+	nil,                       // 39: llmgw.v1.IBPort.CountersEntry
+	nil,                       // 40: llmgw.v1.Heartbeat.EngineVersionsEntry
+	nil,                       // 41: llmgw.v1.EngineProfile.EnvEntry
 }
 var file_llmgw_v1_agent_proto_depIdxs = []int32{
 	3,  // 0: llmgw.v1.AgentMessage.register:type_name -> llmgw.v1.Register
@@ -3864,50 +4044,52 @@ var file_llmgw_v1_agent_proto_depIdxs = []int32{
 	29, // 3: llmgw.v1.AgentMessage.logs:type_name -> llmgw.v1.LogsReply
 	31, // 4: llmgw.v1.AgentMessage.fabric_scan:type_name -> llmgw.v1.FabricScanReply
 	33, // 5: llmgw.v1.AgentMessage.fabric_test:type_name -> llmgw.v1.FabricTestReply
-	22, // 6: llmgw.v1.ControlMessage.start:type_name -> llmgw.v1.StartShard
-	25, // 7: llmgw.v1.ControlMessage.stop:type_name -> llmgw.v1.StopShard
-	26, // 8: llmgw.v1.ControlMessage.drain:type_name -> llmgw.v1.DrainShard
-	27, // 9: llmgw.v1.ControlMessage.ack:type_name -> llmgw.v1.Ack
-	24, // 10: llmgw.v1.ControlMessage.remove_weights:type_name -> llmgw.v1.RemoveWeights
-	17, // 11: llmgw.v1.ControlMessage.provision:type_name -> llmgw.v1.Provision
-	28, // 12: llmgw.v1.ControlMessage.logs:type_name -> llmgw.v1.LogsRequest
-	30, // 13: llmgw.v1.ControlMessage.fabric_scan:type_name -> llmgw.v1.FabricScanRequest
-	32, // 14: llmgw.v1.ControlMessage.fabric_test:type_name -> llmgw.v1.FabricTestRequest
-	34, // 15: llmgw.v1.Register.labels:type_name -> llmgw.v1.Register.LabelsEntry
-	7,  // 16: llmgw.v1.Register.gpus:type_name -> llmgw.v1.GPU
-	8,  // 17: llmgw.v1.Register.fabrics:type_name -> llmgw.v1.FabricAttachment
-	35, // 18: llmgw.v1.Register.engine_versions:type_name -> llmgw.v1.Register.EngineVersionsEntry
-	6,  // 19: llmgw.v1.Register.running_shards:type_name -> llmgw.v1.RunningShard
-	9,  // 20: llmgw.v1.Register.weights:type_name -> llmgw.v1.LocalWeights
-	5,  // 21: llmgw.v1.Register.storage:type_name -> llmgw.v1.Storage
-	4,  // 22: llmgw.v1.Register.host:type_name -> llmgw.v1.Host
-	14, // 23: llmgw.v1.Register.imex:type_name -> llmgw.v1.IMEX
-	0,  // 24: llmgw.v1.RunningShard.state:type_name -> llmgw.v1.ShardStatus.State
-	36, // 25: llmgw.v1.GPU.nic_paths:type_name -> llmgw.v1.GPU.NicPathsEntry
-	11, // 26: llmgw.v1.NodeLoad.shards:type_name -> llmgw.v1.ShardLoad
-	37, // 27: llmgw.v1.IBPort.counters:type_name -> llmgw.v1.IBPort.CountersEntry
-	13, // 28: llmgw.v1.IBPort.subnet_managers:type_name -> llmgw.v1.SubnetManager
-	5,  // 29: llmgw.v1.Heartbeat.storage:type_name -> llmgw.v1.Storage
-	9,  // 30: llmgw.v1.Heartbeat.weights:type_name -> llmgw.v1.LocalWeights
-	38, // 31: llmgw.v1.Heartbeat.engine_versions:type_name -> llmgw.v1.Heartbeat.EngineVersionsEntry
-	19, // 32: llmgw.v1.Heartbeat.provision:type_name -> llmgw.v1.ProvisionStatus
-	10, // 33: llmgw.v1.Heartbeat.load:type_name -> llmgw.v1.NodeLoad
-	12, // 34: llmgw.v1.Heartbeat.ib_ports:type_name -> llmgw.v1.IBPort
-	8,  // 35: llmgw.v1.Heartbeat.fabrics:type_name -> llmgw.v1.FabricAttachment
-	15, // 36: llmgw.v1.Heartbeat.gpu_fabric:type_name -> llmgw.v1.GPUFabricState
-	14, // 37: llmgw.v1.Heartbeat.imex:type_name -> llmgw.v1.IMEX
-	18, // 38: llmgw.v1.Provision.engines:type_name -> llmgw.v1.EngineWant
-	0,  // 39: llmgw.v1.ShardStatus.state:type_name -> llmgw.v1.ShardStatus.State
-	21, // 40: llmgw.v1.ShardStatus.fetch:type_name -> llmgw.v1.FetchProgress
-	23, // 41: llmgw.v1.StartShard.profile:type_name -> llmgw.v1.EngineProfile
-	39, // 42: llmgw.v1.EngineProfile.env:type_name -> llmgw.v1.EngineProfile.EnvEntry
-	1,  // 43: llmgw.v1.AgentService.Connect:input_type -> llmgw.v1.AgentMessage
-	2,  // 44: llmgw.v1.AgentService.Connect:output_type -> llmgw.v1.ControlMessage
-	44, // [44:45] is the sub-list for method output_type
-	43, // [43:44] is the sub-list for method input_type
-	43, // [43:43] is the sub-list for extension type_name
-	43, // [43:43] is the sub-list for extension extendee
-	0,  // [0:43] is the sub-list for field type_name
+	35, // 6: llmgw.v1.AgentMessage.upgrade:type_name -> llmgw.v1.UpgradeReply
+	22, // 7: llmgw.v1.ControlMessage.start:type_name -> llmgw.v1.StartShard
+	25, // 8: llmgw.v1.ControlMessage.stop:type_name -> llmgw.v1.StopShard
+	26, // 9: llmgw.v1.ControlMessage.drain:type_name -> llmgw.v1.DrainShard
+	27, // 10: llmgw.v1.ControlMessage.ack:type_name -> llmgw.v1.Ack
+	24, // 11: llmgw.v1.ControlMessage.remove_weights:type_name -> llmgw.v1.RemoveWeights
+	17, // 12: llmgw.v1.ControlMessage.provision:type_name -> llmgw.v1.Provision
+	28, // 13: llmgw.v1.ControlMessage.logs:type_name -> llmgw.v1.LogsRequest
+	30, // 14: llmgw.v1.ControlMessage.fabric_scan:type_name -> llmgw.v1.FabricScanRequest
+	32, // 15: llmgw.v1.ControlMessage.fabric_test:type_name -> llmgw.v1.FabricTestRequest
+	34, // 16: llmgw.v1.ControlMessage.upgrade:type_name -> llmgw.v1.UpgradeAgent
+	36, // 17: llmgw.v1.Register.labels:type_name -> llmgw.v1.Register.LabelsEntry
+	7,  // 18: llmgw.v1.Register.gpus:type_name -> llmgw.v1.GPU
+	8,  // 19: llmgw.v1.Register.fabrics:type_name -> llmgw.v1.FabricAttachment
+	37, // 20: llmgw.v1.Register.engine_versions:type_name -> llmgw.v1.Register.EngineVersionsEntry
+	6,  // 21: llmgw.v1.Register.running_shards:type_name -> llmgw.v1.RunningShard
+	9,  // 22: llmgw.v1.Register.weights:type_name -> llmgw.v1.LocalWeights
+	5,  // 23: llmgw.v1.Register.storage:type_name -> llmgw.v1.Storage
+	4,  // 24: llmgw.v1.Register.host:type_name -> llmgw.v1.Host
+	14, // 25: llmgw.v1.Register.imex:type_name -> llmgw.v1.IMEX
+	0,  // 26: llmgw.v1.RunningShard.state:type_name -> llmgw.v1.ShardStatus.State
+	38, // 27: llmgw.v1.GPU.nic_paths:type_name -> llmgw.v1.GPU.NicPathsEntry
+	11, // 28: llmgw.v1.NodeLoad.shards:type_name -> llmgw.v1.ShardLoad
+	39, // 29: llmgw.v1.IBPort.counters:type_name -> llmgw.v1.IBPort.CountersEntry
+	13, // 30: llmgw.v1.IBPort.subnet_managers:type_name -> llmgw.v1.SubnetManager
+	5,  // 31: llmgw.v1.Heartbeat.storage:type_name -> llmgw.v1.Storage
+	9,  // 32: llmgw.v1.Heartbeat.weights:type_name -> llmgw.v1.LocalWeights
+	40, // 33: llmgw.v1.Heartbeat.engine_versions:type_name -> llmgw.v1.Heartbeat.EngineVersionsEntry
+	19, // 34: llmgw.v1.Heartbeat.provision:type_name -> llmgw.v1.ProvisionStatus
+	10, // 35: llmgw.v1.Heartbeat.load:type_name -> llmgw.v1.NodeLoad
+	12, // 36: llmgw.v1.Heartbeat.ib_ports:type_name -> llmgw.v1.IBPort
+	8,  // 37: llmgw.v1.Heartbeat.fabrics:type_name -> llmgw.v1.FabricAttachment
+	15, // 38: llmgw.v1.Heartbeat.gpu_fabric:type_name -> llmgw.v1.GPUFabricState
+	14, // 39: llmgw.v1.Heartbeat.imex:type_name -> llmgw.v1.IMEX
+	18, // 40: llmgw.v1.Provision.engines:type_name -> llmgw.v1.EngineWant
+	0,  // 41: llmgw.v1.ShardStatus.state:type_name -> llmgw.v1.ShardStatus.State
+	21, // 42: llmgw.v1.ShardStatus.fetch:type_name -> llmgw.v1.FetchProgress
+	23, // 43: llmgw.v1.StartShard.profile:type_name -> llmgw.v1.EngineProfile
+	41, // 44: llmgw.v1.EngineProfile.env:type_name -> llmgw.v1.EngineProfile.EnvEntry
+	1,  // 45: llmgw.v1.AgentService.Connect:input_type -> llmgw.v1.AgentMessage
+	2,  // 46: llmgw.v1.AgentService.Connect:output_type -> llmgw.v1.ControlMessage
+	46, // [46:47] is the sub-list for method output_type
+	45, // [45:46] is the sub-list for method input_type
+	45, // [45:45] is the sub-list for extension type_name
+	45, // [45:45] is the sub-list for extension extendee
+	0,  // [0:45] is the sub-list for field type_name
 }
 
 func init() { file_llmgw_v1_agent_proto_init() }
@@ -3922,6 +4104,7 @@ func file_llmgw_v1_agent_proto_init() {
 		(*AgentMessage_Logs)(nil),
 		(*AgentMessage_FabricScan)(nil),
 		(*AgentMessage_FabricTest)(nil),
+		(*AgentMessage_Upgrade)(nil),
 	}
 	file_llmgw_v1_agent_proto_msgTypes[1].OneofWrappers = []any{
 		(*ControlMessage_Start)(nil),
@@ -3933,6 +4116,7 @@ func file_llmgw_v1_agent_proto_init() {
 		(*ControlMessage_Logs)(nil),
 		(*ControlMessage_FabricScan)(nil),
 		(*ControlMessage_FabricTest)(nil),
+		(*ControlMessage_Upgrade)(nil),
 	}
 	type x struct{}
 	out := protoimpl.TypeBuilder{
@@ -3940,7 +4124,7 @@ func file_llmgw_v1_agent_proto_init() {
 			GoPackagePath: reflect.TypeOf(x{}).PkgPath(),
 			RawDescriptor: unsafe.Slice(unsafe.StringData(file_llmgw_v1_agent_proto_rawDesc), len(file_llmgw_v1_agent_proto_rawDesc)),
 			NumEnums:      1,
-			NumMessages:   39,
+			NumMessages:   41,
 			NumExtensions: 0,
 			NumServices:   1,
 		},

api/proto/llmgw/v1/agent.proto

@@ -26,6 +26,7 @@ message AgentMessage {
     LogsReply logs = 4;
     FabricScanReply fabric_scan = 5;
     FabricTestReply fabric_test = 6;
+    UpgradeReply upgrade = 7;
   }
 }
 
@@ -41,6 +42,7 @@ message ControlMessage {
     LogsRequest logs = 7;
     FabricScanRequest fabric_scan = 8;
     FabricTestRequest fabric_test = 9;
+    UpgradeAgent upgrade = 10;
   }
 }
 
@@ -649,3 +651,28 @@ message FabricTestReply {
   string error = 4;
   int64 took_ms = 5;
 }
+
+// UpgradeAgent tells an agent to replace its own binary with a release and
+// restart.
+//
+// The agent fetches the release for its own platform and checks it against
+// the release's checksums before replacing anything, keeps the old binary
+// beside the new, then exits so systemd starts the new one. Engines keep
+// running throughout, as for any agent restart, and are adopted on the way
+// back. Over the stream the agent already holds, so upgrading a fleet needs
+// no ssh into it.
+message UpgradeAgent {
+  string request_id = 1;
+  // version must be a release: v0.10.4 or v0.10.4-rc.2.
+  string version = 2;
+  // releases is the base URL release files are downloaded from.
+  string releases = 3;
+}
+
+// UpgradeReply says whether the new binary is in place. Sent before the
+// agent exits; an empty error means it will be back on the new version.
+message UpgradeReply {
+  string request_id = 1;
+  string version = 2;
+  string error = 3;
+}

deploy/bootstrap/onboard-node.sh

@@ -122,9 +122,10 @@ TMP="$(mktemp -d)"
 trap 'rm -rf "$TMP"' EXIT
 "$BINARY" ctl cert issue-client "$NODE" --dir "$PKI" --out "$TMP" >/dev/null
 
-# 2 and 3. Identity, and trusting the root so future tooling on that machine
-#    does not treat our certificates as strangers.
-say "installing the certificate and trusting the root"
+# 2 and 3. Identity. The root is trusted by the agent's configuration only,
+#    never system-wide: it signs whatever the control plane asks, and every
+#    TLS client on the machine would take it for any site.
+say "installing the certificate"
 scp -q $SSHOPTS "$PKI/ca.pem" "$TMP/$NODE.pem" "$TMP/$NODE-key.pem" "$SSH_USER@$ADDR:/tmp/"
 $SSH "set -e
   sudo install -d -m 0755 /etc/llmgw
@@ -132,8 +133,10 @@ $SSH "set -e
   sudo install -m 0644 /tmp/$NODE.pem /etc/llmgw/agent.pem
   sudo install -m 0600 /tmp/$NODE-key.pem /etc/llmgw/agent-key.pem
   sudo chown root:root /etc/llmgw/*
-  sudo install -m 0644 /tmp/ca.pem /usr/local/share/ca-certificates/llmgw-ca.crt
-  sudo update-ca-certificates >/dev/null 2>&1 || true
+  if [ -f /usr/local/share/ca-certificates/llmgw-ca.crt ]; then
+    sudo rm -f /usr/local/share/ca-certificates/llmgw-ca.crt
+    sudo update-ca-certificates --fresh >/dev/null 2>&1 || true
+  fi
   rm -f /tmp/ca.pem /tmp/$NODE.pem /tmp/$NODE-key.pem"
 
 # 4. The binary, the configuration and the unit.
@@ -143,7 +146,7 @@ $SSH "set -e
 # that is not a release (a developer's, with commits past the tag) has no such
 # file, and is copied as before.
 TAG="$("$BINARY" --version 2>/dev/null | awk '{print $NF}')"
-if printf '%s' "$TAG" | grep -Eq '^v[0-9]+\.[0-9]+\.[0-9]+$'; then
+if printf '%s' "$TAG" | grep -Eq '^v[0-9]+\.[0-9]+\.[0-9]+(-rc\.[0-9]+)?$'; then
   say "installing llmgw $TAG from the release"
   $SSH "set -e
     arch=\$(dpkg --print-architecture)

deploy/monitoring/README.md

@@ -46,6 +46,19 @@ apt-get install prometheus prometheus-alertmanager
 install -m 0644 prometheus.yml            /etc/prometheus/prometheus.yml
 install -m 0644 llmgw-rules.yml           /etc/prometheus/rules/llmgw.yml
 install -m 0644 alertmanager.yml          /etc/prometheus/alertmanager.yml
+# Alertmanager answers on loopback only. Its API takes silences from anyone who
+# can reach it, with no login, and silencing BackupTooOld or ModelParked is how
+# something that should be noticed is not. Prometheus beside it is its only
+# client; one Alertmanager needs no cluster port.
+echo 'ARGS="--web.listen-address=127.0.0.1:9093 --cluster.listen-address="' \
+    > /etc/default/prometheus-alertmanager
+# Prometheus itself is read-only as installed: its admin and lifecycle APIs
+# stay off (no --web.enable-admin-api, no --web.enable-lifecycle). It still
+# answers any query from anyone who reaches :9090, which says which models and
+# nodes there are and how busy each is. Its readers are the control plane
+# (monitoring.prometheus) and Grafana beside it; to keep it to those, allow
+# the control plane and drop the rest, e.g. with nftables:
+#   nft add rule inet filter input tcp dport 9090 ip saddr != { 127.0.0.1, CONTROL_PLANE_IP } drop
 promtool check config /etc/prometheus/prometheus.yml
 systemctl restart prometheus prometheus-alertmanager
 
@@ -75,7 +88,8 @@ Alertmanager groups and holds alerts and has no receiver.
 Where an alert should go — an address, a Slack workspace, a webhook — is a
 decision with credentials attached, and a monitoring system that quietly starts
 mailing people is worse than one that is visibly not wired up. Alerts are
-visible at `:9093` and in Prometheus; add a receiver to `alertmanager.yml` to
+visible in Prometheus, and at `:9093` on the monitoring host itself (`ssh -L
+9093:127.0.0.1:9093 <host>` to open its page); add a receiver to `alertmanager.yml` to
 make them travel. `critical` already routes separately so it can be pointed
 somewhere else without touching the rest.
 
@@ -86,7 +100,7 @@ An alerting system nobody has seen fire is an assumption.
 ```
 llmgw ctl cordon <node>     # work already there keeps running
 # ~2 minutes later
-curl -s http://<host>:9093/api/v2/alerts | jq '.[].labels'
+ssh <host> curl -s http://127.0.0.1:9093/api/v2/alerts | jq '.[].labels'
 llmgw ctl uncordon <node>   # the alert resolves on its own
 ```
 

deploy/monitoring/llmgw-rules.yml

@@ -42,10 +42,97 @@ groups:
         annotations:
           summary: "{{ $labels.model }} has no replica ready"
           description: >-
-            The model is placed and nothing is serving it. Ten minutes, because
-            a large model on CPU legitimately takes minutes to load. A model
-            that is deliberately stopped publishes no series at all, so this
-            cannot fire for one nobody asked for.
+            The model should be running and nothing is serving it. Ten minutes,
+            because a large model on CPU legitimately takes minutes to load. The
+            series is published at 0 for every model that should be running —
+            before v0.10.4 it was published only while a replica was ready, so
+            this could not fire at all. A model deliberately stopped, or an
+            on-demand one nobody is asking for, publishes nothing.
+
+      - alert: ModelParked
+        expr: llmgw_model_parked == 1
+        labels: { severity: critical }
+        annotations:
+          summary: "{{ $labels.model }} failed every time it started and is no longer retried"
+          description: >-
+            Five replicas in a row failed at start. Before this existed a model
+            whose weights its engine could not load was started 155 times over
+            eight hours. `llmgw ctl status` has the last error; once it is dealt
+            with, `llmgw ctl retry <model>`. Changing the model's definition
+            also retries it.
+
+      - alert: ModelFailingToStart
+        expr: llmgw_model_start_failures >= 2
+        for: 5m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.model }} has failed to start {{ $value }} times in a row"
+          description: >-
+            On its way to being parked. The engine's own words are in
+            `llmgw ctl logs <node> --source <shard>` for the next attempt, or in
+            the control plane's log for the last one.
+
+      - alert: ModelCannotBePlaced
+        expr: llmgw_model_unplaceable == 1
+        for: 10m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.model }} is wanted and fits on no node"
+          description: >-
+            Short of replicas, and the last attempt found nowhere with the
+            memory, cards or engine it needs. `llmgw ctl explain-placement
+            <model>` says what each node lacks. For an on-demand model, requests
+            for it are told this rather than asked to retry.
+
+      - alert: EngineInstallFailed
+        expr: llmgw_node_engine_install{state="failed"} == 1
+        for: 15m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.engine }} {{ $labels.version }} will not install on {{ $labels.node }}"
+          description: >-
+            The node takes no model needing it until it does. The agent retries
+            on its own; `llmgw ctl nodes` and the node's logs on the cluster
+            page say why it failed.
+
+      - alert: ControlPlaneLeadership
+        expr: sum(llmgw_control_plane_leader) != 1
+        for: 2m
+        labels: { severity: critical }
+        annotations:
+          summary: "{{ $value }} control planes believe they are placing work"
+          description: >-
+            None means nothing is placed or replaced; two means both are, each
+            into GPUs the other believes free. The lease is in the store: check
+            it is reachable from every control plane.
+
+      - alert: CertificateExpiresSoon
+        expr: (llmgw_certificate_expiry_timestamp_seconds - time()) < 30 * 86400
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.cert }} ({{ $labels.subject }}) expires in {{ $value | humanizeDuration }}"
+          description: >-
+            Certificates last a year and nothing renews them. When the agent
+            server's runs out every agent refuses to connect at once. Reissue
+            with `llmgw ctl cert` and restart the control plane.
+
+      - alert: CertificateExpiresVerySoon
+        expr: (llmgw_certificate_expiry_timestamp_seconds - time()) < 7 * 86400
+        labels: { severity: critical }
+        annotations:
+          summary: "{{ $labels.cert }} ({{ $labels.subject }}) expires in {{ $value | humanizeDuration }}"
+          description: See CertificateExpiresSoon; a week is not long.
+
+      - alert: BackupTooOld
+        expr: time() - llmgw_backup_last_success_timestamp_seconds > 26 * 3600
+        labels: { severity: warning }
+        annotations:
+          summary: "The newest llmgw backup is {{ $value | humanizeDuration }} old"
+          description: >-
+            llmgw-backup.timer runs nightly. Also fires when there has never
+            been a backup (the series is 0 then). `journalctl -u
+            llmgw-backup.service` on the control plane says why the last one
+            failed; docs/install.md, "Backing up", says how to set it up.
 
       - alert: PoolSaturated
         expr: llmgw_pool_saturation > 1

deploy/systemd/llmgw-backup.service

@@ -0,0 +1,14 @@
+[Unit]
+Description=llmgw backup: the store and /etc/llmgw, to /var/backups/llmgw-server
+After=llmgw-server.service postgresql.service
+
+[Service]
+Type=oneshot
+# Nothing it writes is for anyone else: the archive holds the CA's key and a
+# copy of the store.
+UMask=0077
+# Root, because the archive includes /etc/llmgw/secrets and the certificate
+# authority's key, which the llmgw user deliberately cannot read all of.
+ExecStart=/usr/local/bin/llmgw server backup --config /etc/llmgw/server.yaml --keep 14
+Nice=10
+IOSchedulingClass=idle

deploy/systemd/llmgw-backup.timer

@@ -0,0 +1,12 @@
+[Unit]
+Description=Back up llmgw every night
+
+[Timer]
+OnCalendar=*-*-* 03:15:00
+# A machine that was off at 03:15 backs up when it next starts, rather than
+# waiting another day with a gap nobody notices until the alert.
+Persistent=true
+RandomizedDelaySec=10m
+
+[Install]
+WantedBy=timers.target

docs/install.md

@@ -909,88 +909,123 @@ refuses every request.
 
 ## Upgrading
 
-One binary on every machine, so an upgrade is a copy and a restart. The part
-worth care is the copy, not the order.
+On the control plane:
+
+```
+sudo llmgw ctl upgrade v0.10.4
+```
+
+It upgrades this machine first — fetching the release for its platform,
+checking it against the release's checksums, keeping the old binary as
+`/usr/local/bin/llmgw.bak-pre-<version>`, restarting `llmgw-server` and waiting
+for every node to reconnect — and then each node in turn. A node's agent is told
+the version over the stream it already holds, fetches and checks the release
+itself, swaps it in and restarts; no ssh into the nodes is needed. Release
+candidates (`v0.10.4-rc.2`) work the same way.
+
+A node counts as done when it is back on the new version, ready, with as many
+ready replicas as before. One that is not within `--timeout` (three minutes)
+stops the upgrade there, so a bad release reaches one node rather than all of
+them; `llmgw ctl logs <node>` says why, and the node keeps its previous binary
+beside the new one. Nodes already on the version are skipped, so running the
+command again resumes where it stopped. `--nodes a,b` limits it to some nodes,
+and `--skip-control-plane` leaves this machine alone.
 
 **Nothing stops serving.** Engines are started with `KillMode=process`, so they
 keep running when the agent restarts, and the agent reports them back on its
 next connection. The control plane holds no replica state across its own
-restart either — it rebuilds the fleet from what the agents report. A rolling
-upgrade of the whole cluster interrupts no request that is not in flight at the
-moment of the restart.
+restart either — it rebuilds the fleet from what the agents report.
 
 **A mixed fleet is fine while you roll.** The agent protocol only ever gains
 fields, and both sides treat an absent one as unknown rather than as a default.
-Order does not matter; finish the roll the same day anyway, because
-`ctl nodes` reports each agent's version and a fleet that disagrees with itself
-is one more thing to hold in your head.
-
-```
-VERSION=v0.9.0
-# copy to every machine first, install nowhere yet
-for h in 10.0.0.140 10.0.0.141 10.0.0.142 10.0.0.146; do
-  ssh $h 'cat > /tmp/llmgw-new; stat -c %s /tmp/llmgw-new' < llmgw-$VERSION-linux-amd64
-done
-```
-
-**Check every size before installing anything.** This is not defensive writing:
-on a host whose login shell prints anything, `scp` fails outright, and the
-obvious workaround — `ssh -n host 'cat > file'` — points stdin at `/dev/null`
-and writes a *zero-byte file* while reporting success. Installing that on a
-fleet takes the fleet down. Compare the number on each machine with the local
-one, then:
-
-```
-sudo install -m0755 /tmp/llmgw-new /usr/local/bin/llmgw
-sudo systemctl restart llmgw-agent     # or llmgw-server, or llmgw-mirror
-llmgw --version
-```
+Agents older than v0.10.4 cannot upgrade themselves: move those by hand once,
+installing the release binary and restarting `llmgw-agent`, and every upgrade
+after that is the command above. The mirror, which is not an agent, is
+upgraded the same way by hand.
 
 Then confirm what you have, rather than assuming:
 
 ```
-llmgw ctl nodes       # every agent back, LAST SEEN in seconds
+llmgw ctl nodes       # every agent on the new version, LAST SEEN in seconds
 llmgw ctl replicas    # same replicas, adopted rather than replaced
 ```
 
 A replica may come back marked **`ready (stale profile)`**. That means it is
 running a profile that has since been edited — the reconciler leaves a serving
 replica alone on purpose, because churning one costs more than the difference.
-Restart it when you want the new profile to take effect; the flag is there so
-the choice is yours rather than silent.
+`llmgw ctl restart <replica>` replaces it with one on the new profile, starting
+the new one before stopping the old.
 
 ## Backing up
 
-Three things, in descending order of how badly you want them.
+`llmgw server backup` writes one archive holding everything a control plane
+cannot be rebuilt without:
 
-**The certificate authority.** `ca-key.pem`, `/etc/llmgw/pki` unless you kept it elsewhere (step 2).
-Losing it does not stop the cluster, and it means every node identity has to be
-reissued the next time you touch one. Nothing in llmgw can reconstruct it.
+- **the store** — users, API keys, the tenancy tree, limits and budgets, usage
+  history, models added at runtime, measured capacity, and the secret every
+  engine key is derived from (`pg_dump` in its custom format for PostgreSQL, a
+  consistent `VACUUM INTO` snapshot for SQLite; both are safe against a running
+  gateway);
+- **`/etc/llmgw`** — the configuration, the store's address, and the
+  certificate authority's key (`pki/`). Nothing in llmgw can reconstruct that
+  key; losing it means reissuing every node's identity.
 
-**The store.** Users, API keys, organisations, teams, projects, every limit and
-budget, all usage history, models added at runtime, and measured capacity.
+Archives go to `/var/backups/llmgw-server`, readable by root only because of the CA
+key, and the newest 14 are kept. Run it nightly with the timer:
 
 ```
-# SQLite
-sqlite3 /var/lib/llmgw/llmgw.db ".backup '/backup/llmgw-$(date +%F).db'"
-
-# PostgreSQL
-pg_dump --format=custom llmgw > /backup/llmgw-$(date +%F).dump
+sudo install -m 0644 deploy/systemd/llmgw-backup.service deploy/systemd/llmgw-backup.timer /etc/systemd/system/
+sudo systemctl daemon-reload
+sudo systemctl enable --now llmgw-backup.timer
+sudo systemctl start llmgw-backup.service     # one now, to see it work
+ls -l /var/backups/llmgw-server
 ```
 
-`.backup` and `pg_dump` are both safe against a running gateway. Copying the
-SQLite file with `cp` while it is being written is not.
+The control plane publishes when the newest archive was written as
+`llmgw_backup_last_success_timestamp_seconds`, and **BackupTooOld** fires when
+it is more than 26 hours old — including when there has never been one.
+
+**Copy the archives off the machine.** They sit on the disk they back up and do
+not survive losing it. Anything that copies files will do — rsync from the
+mirror, a NAS mount, an offsite bucket — as long as it keeps the archives
+readable by root only.
 
 **Keys are stored hashed and cannot be recovered from a backup or from
 anywhere else.** Restoring an old store restores which keys exist and who owns
 them, not their secrets. A key created after the backup is gone and has to be
-reissued — which is why the store is worth backing up on a schedule rather than
-before upgrades.
+reissued.
+
+### Restoring
+
+On a machine with llmgw installed (the same version or newer) and, for
+PostgreSQL, an empty database and role as `server init --postgres` would create
+them:
+
+```
+sudo systemctl stop llmgw-server
+mkdir /tmp/restore && sudo tar -xzf /var/backups/llmgw-server/llmgw-backup-<host>-<time>.tar.gz -C /tmp/restore
+cat /tmp/restore/MANIFEST                     # version and time it was taken
+
+# /etc/llmgw: configuration, secrets, certificates and the CA
+sudo cp -a /tmp/restore/etc/llmgw/. /etc/llmgw/
+
+# the store — PostgreSQL
+sudo -u postgres pg_restore --clean --if-exists --no-owner --role=llmgw -d llmgw /tmp/restore/store.pgdump
+# the store — SQLite
+sudo install -o llmgw -g llmgw -m 0600 /tmp/restore/store.sqlite /var/lib/llmgw/llmgw.db
+
+sudo systemctl start llmgw-server
+sudo rm -rf /tmp/restore
+```
+
+Running engines keep serving throughout: agents reconnect to the restored
+control plane and their replicas are adopted, and because the engine-key secret
+came back with the store, the router can still reach them.
 
 **The mirror's directory** is the one thing you can rebuild, by downloading it
-all again. Worth remembering that a pinned version may no longer be upstream to
-download: that is exactly what pinning it protected you from, and a backup is
-what makes the protection outlive the disk.
+all again. A pinned version may no longer be upstream to download: that is
+exactly what pinning it protected you from, and a copy of the directory is what
+makes the protection outlive the disk.
 
 ## Reclaiming disk
 
@@ -1089,3 +1124,11 @@ requests. On a slow model, set `timeouts.generation` higher.
   has them. Single-node placement on real GPUs is the well-trodden path; if you
   are the first to run a TP group across real cards, run `llmgw doctor --json`
   on each node first and keep the reports.
+- **Two engine setups are weaker than the rest.** An engine's key is passed
+  in its environment, where only root and its own user can read it, except
+  SGLang's, which takes it only as a flag: other users on the node can see it
+  in the process list. Mount `/proc` with `hidepid=2` on nodes where that
+  matters. vLLM's `launch_mode: ray` starts a Ray head without
+  authentication, so anyone who can reach its port can run code on the
+  node. The dashboard is off, but firewall the port to the other nodes in the
+  replica, or use the default `mp` launch mode, which needs no Ray at all.

docs/reference.md

@@ -81,6 +81,7 @@ rather than repeated.
 | `notice` | `string` | [Landing](#landing) |
 | `bin` | `string` | [LlamaCppConfig](#llamacppconfig) |
 | `parallel` | `int` | [LlamaCppConfig](#llamacppconfig) |
+| `threads` | `int` | [LlamaCppConfig](#llamacppconfig) |
 | `api` | `string` | [Mirror](#mirror) |
 | `tls` | `tlsconf.Config` | [Mirror](#mirror) |
 | `registry` | `string` | [Mirror](#mirror) |
@@ -167,8 +168,11 @@ rather than repeated.
 | `landing` | `Landing` | [Server](#server) |
 | `agent_tls` | `tlsconf.Config` | [Server](#server) |
 | `monitoring` | `Monitoring` | [Server](#server) |
+| `tracing` | `Tracing` | [Server](#server) |
 | `mirror` | `Mirror` | [Server](#server) |
 | `engines` | `map[string]string` | [Server](#server) |
+| `backup_dir` | `string` | [Server](#server) |
+| `releases` | `string` | [Server](#server) |
 | `router` | `Router` | [Server](#server) |
 | `control` | `Control` | [Server](#server) |
 | `models` | `[]catalog.Model` | [Server](#server) |
@@ -182,6 +186,8 @@ rather than repeated.
 | `first_token` | `time.Duration` | [Timeouts](#timeouts) |
 | `idle` | `time.Duration` | [Timeouts](#timeouts) |
 | `total` | `time.Duration` | [Timeouts](#timeouts) |
+| `endpoint` | `string` | [Tracing](#tracing) |
+| `sample_ratio` | `float64` | [Tracing](#tracing) |
 | `bin` | `string` | [VLLMConfig](#vllmconfig) |
 | `launch_mode` | `string` | [VLLMConfig](#vllmconfig) |
 | `ray_bin` | `string` | [VLLMConfig](#vllmconfig) |
@@ -632,6 +638,20 @@ Parallel is how many requests a server handles at once, when a profile
 does not say. llama.cpp defaults to one, which would make every second
 request wait behind the first no matter what the router decided.
 
+### `threads`
+
+`int`
+
+Threads is how many CPU threads a shard with no GPU computes on. Zero
+takes half this machine's CPUs.
+
+Not all of them, which is llama.cpp's own default: two busy servers
+that each spin on every CPU fight for them, and on an 8-vCPU node
+that made 7B generation fall from 10 tokens a second to under 1 each.
+At half, one server alone is about a third slower and two together
+keep 5.5 each. A node that only ever runs one CPU replica does better
+with all of them; say so here.
+
 ## Mirror
 
 ### `api`
@@ -1244,6 +1264,16 @@ Monitoring is where the dashboard reads its history from.
 
 Takes a [Monitoring](#monitoring) block.
 
+### `tracing`
+
+`Tracing`
+
+Tracing sends a trace of each inference request to an OpenTelemetry
+collector. Off unless an endpoint is set here or in
+OTEL_EXPORTER_OTLP_ENDPOINT.
+
+Takes a [Tracing](#tracing) block.
+
 ### `mirror`
 
 `Mirror`
@@ -1265,6 +1295,22 @@ never trusted to already have what somebody once put there.
 Unset keeps DefaultEngines. A version replaces the default; an empty
 one leaves that engine to be installed by hand.
 
+### `backup_dir`
+
+`string`
+
+BackupDir is where `llmgw server backup` writes, and where the newest
+archive's age is read for llmgw_backup_last_success_timestamp_seconds.
+Empty means /var/backups/llmgw-server.
+
+### `releases`
+
+`string`
+
+Releases is where `llmgw ctl upgrade` fetches release binaries from,
+for this machine and — through their agents — every node. Empty means
+the project's own releases.
+
 ### `router`
 
 `Router`
@@ -1348,6 +1394,22 @@ about a hundred tokens on a slow model.
 
 `time.Duration`
 
+## Tracing
+
+### `endpoint`
+
+`string`
+
+Endpoint is the collector, for example http://otel.internal:4318.
+Empty falls back to OTEL_EXPORTER_OTLP_ENDPOINT, and to no tracing.
+
+### `sample_ratio`
+
+`float64`
+
+SampleRatio is the share of requests traced, 0 to 1, when the caller
+did not decide it with a sampled traceparent of its own. Zero takes 1.
+
 ## VLLMConfig
 
 ### `bin`

go.mod

@@ -19,6 +19,10 @@ require (
 	github.com/prometheus/common v0.71.0
 	github.com/spf13/cobra v1.10.2
 	github.com/yuin/goldmark v1.8.6
+	go.opentelemetry.io/otel v1.46.0
+	go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.46.0
+	go.opentelemetry.io/otel/sdk v1.46.0
+	go.opentelemetry.io/otel/trace v1.46.0
 	golang.org/x/crypto v0.57.0
 	golang.org/x/net v0.59.0
 	golang.org/x/sync v0.23.0
@@ -29,6 +33,7 @@ require (
 require (
 	github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
 	github.com/beorn7/perks v1.0.1 // indirect
+	github.com/cenkalti/backoff/v5 v5.0.3 // indirect
 	github.com/charmbracelet/colorprofile v0.4.1 // indirect
 	github.com/charmbracelet/x/ansi v0.11.6 // indirect
 	github.com/charmbracelet/x/cellbuf v0.0.15 // indirect
@@ -39,8 +44,11 @@ require (
 	github.com/dustin/go-humanize v1.0.1 // indirect
 	github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f // indirect
 	github.com/fsnotify/fsnotify v1.9.0 // indirect
-	github.com/go-viper/mapstructure/v2 v2.4.0 // indirect
+	github.com/go-logr/logr v1.4.4 // indirect
+	github.com/go-logr/stdr v1.2.2 // indirect
+	github.com/go-viper/mapstructure/v2 v2.5.0 // indirect
 	github.com/google/uuid v1.6.0 // indirect
+	github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0 // indirect
 	github.com/inconshreveable/mousetrap v1.1.0 // indirect
 	github.com/jackc/pgpassfile v1.0.0 // indirect
 	github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
@@ -61,9 +69,16 @@ require (
 	github.com/rivo/uniseg v0.4.7 // indirect
 	github.com/spf13/pflag v1.0.9 // indirect
 	github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e // indirect
+	go.opentelemetry.io/auto/sdk v1.2.1 // indirect
+	go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.46.0 // indirect
+	go.opentelemetry.io/otel/metric v1.46.0 // indirect
+	go.opentelemetry.io/proto/otlp v1.11.0 // indirect
 	go.yaml.in/yaml/v3 v3.0.5 // indirect
 	golang.org/x/sys v0.48.0 // indirect
 	golang.org/x/text v0.42.0 // indirect
+	google.golang.org/genproto/googleapis/api v0.0.0-20260819154853-08b0e4226688 // indirect
+	google.golang.org/genproto/googleapis/rpc v0.0.0-20260819154853-08b0e4226688 // indirect
+	google.golang.org/grpc v1.83.1 // indirect
 	modernc.org/libc v1.75.7 // indirect
 	modernc.org/mathutil v1.7.1 // indirect
 	modernc.org/memory v1.12.1 // indirect

go.sum

@@ -6,6 +6,8 @@ github.com/aymanbagabas/go-osc52/v2 v2.0.1 h1:HwpRHbFMcZLEVr42D4p7XBqjyuxQH5SMiE
 github.com/aymanbagabas/go-osc52/v2 v2.0.1/go.mod h1:uYgXzlJ7ZpABp8OJ+exZzJJhRNQ2ASbcXHWsFqH8hp8=
 github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
 github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
+github.com/cenkalti/backoff/v5 v5.0.3 h1:ZN+IMa753KfX5hd8vVaMixjnqRZ3y8CuJKRKj1xcsSM=
+github.com/cenkalti/backoff/v5 v5.0.3/go.mod h1:rkhZdG3JZukswDf7f0cwqPNk4K0sa+F97BxZthm/crw=
 github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
 github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
 github.com/charmbracelet/bubbletea v1.3.10 h1:otUDHWMMzQSB0Pkc87rm691KZ3SWa4KUlvF9nRvCICw=
@@ -34,14 +36,23 @@ github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f h1:Y/CXytFA4m6
 github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f/go.mod h1:vw97MGsxSvLiUE2X8qFplwetxpGLQrlU1Q9AUEIzCaM=
 github.com/fsnotify/fsnotify v1.9.0 h1:2Ml+OJNzbYCTzsxtv8vKSFD9PbJjmhYF14k/jKC7S9k=
 github.com/fsnotify/fsnotify v1.9.0/go.mod h1:8jBTzvmWwFyi3Pb8djgCCO5IBqzKJ/Jwo8TRcHyHii0=
-github.com/go-viper/mapstructure/v2 v2.4.0 h1:EBsztssimR/CONLSZZ04E8qAkxNYq4Qp9LvH92wZUgs=
-github.com/go-viper/mapstructure/v2 v2.4.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM=
+github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
+github.com/go-logr/logr v1.4.4 h1:tG4xh9yMsRCAiodLVTxyrkzSZ9+o0L1Kg/+cPVcbP/8=
+github.com/go-logr/logr v1.4.4/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
+github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
+github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
+github.com/go-viper/mapstructure/v2 v2.5.0 h1:vM5IJoUAy3d7zRSVtIwQgBj7BiWtMPfmPEgAXnvj1Ro=
+github.com/go-viper/mapstructure/v2 v2.5.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM=
+github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek=
+github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps=
 github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
 github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
 github.com/google/pprof v0.0.0-20260802141513-ef3492d7dac3 h1:LMLX+LgTNWpfvCBdFebv6EsYotImrt/Ppc5cXIriCSo=
 github.com/google/pprof v0.0.0-20260802141513-ef3492d7dac3/go.mod h1:jl5iWTm0/hd5PjEYEOuwAJ57L/CibdZfrqZ5XA5GrCk=
 github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
 github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
+github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0 h1:/Tnpcb2E0Pz/tN9s3bfEY2Q8ePCEX9iuS+cneUwncnw=
+github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0/go.mod h1:zOBXOsUaBSjKgmH4OGzV1esUpR3oUSCPYVd2cUBjKYY=
 github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
 github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
 github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
@@ -117,6 +128,24 @@ github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e h1:JVG44RsyaB9T2KIHavM
 github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e/go.mod h1:RbqR21r5mrJuqunuUZ/Dhy/avygyECGrLceyNeo4LiM=
 github.com/yuin/goldmark v1.8.6 h1:d0VcaP1sx9GkFVkoW+KtggpGi2KZ965i14b0+bDQST4=
 github.com/yuin/goldmark v1.8.6/go.mod h1:ip/1k0VRfGynBgxOz0yCqHrbZXhcjxyuS66Brc7iBKg=
+go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64=
+go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y=
+go.opentelemetry.io/otel v1.46.0 h1:FHt5/CDyVxi/8IM1CH7VE/rRgq3kLHa2mSTVMO8AWyc=
+go.opentelemetry.io/otel v1.46.0/go.mod h1:Gj3SEScelsNC45tp4nSxRYlS+f5iez7W8XPMCt905kE=
+go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.46.0 h1:OFnwLJr+pF3iHrlGSzbxyuo6/6HyBlnlN1CWEJmBVcw=
+go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.46.0/go.mod h1:716wFneO0ov19A2beH5hjfh9AK5z/VWNAtDijp1Y0/g=
+go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.46.0 h1:KrC1YrQeSt46ITMWAbgQx1M1eV1/1TKzttrBzymPmss=
+go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.46.0/go.mod h1:zDSEzoEqsOrgBeGvH66KRgxh90VonFyJqBHA0Pk3+rM=
+go.opentelemetry.io/otel/metric v1.46.0 h1:yBnkXvgV7AXFILZc5K6IZe/CBFF3OS7BJ8ov6/lj0K8=
+go.opentelemetry.io/otel/metric v1.46.0/go.mod h1:iPmdWqifKUdzziPkvvzIJXITl56fQx2mGM/DHLB3/2o=
+go.opentelemetry.io/otel/sdk v1.46.0 h1:h5CNQQjEbuQXY/JfZtgt3i7HVFV3aHPO2OAwO2eTYPI=
+go.opentelemetry.io/otel/sdk v1.46.0/go.mod h1:GAERFXFt5SYCEB+YiKUbMBeza6UaDH7GmGOZEfh2gSM=
+go.opentelemetry.io/otel/sdk/metric v1.46.0 h1:0piZ26EG4RBfebb2jhDH6ERCYHoVWduc3kLgPCwSnSE=
+go.opentelemetry.io/otel/sdk/metric v1.46.0/go.mod h1:I1PbKrdVc8Qu8HYVDNtqVIwLwjNrhsV/uFuxfwg8mO4=
+go.opentelemetry.io/otel/trace v1.46.0 h1:OULy7ccdJnZtJ0UDYFOIGaCmiWzJ8Vi2G/Rsu60qs1c=
+go.opentelemetry.io/otel/trace v1.46.0/go.mod h1:J7GAXweO77XSFkB/rmAqk9D6ihszhFjLU+d9WuUxDLI=
+go.opentelemetry.io/proto/otlp v1.11.0 h1:5rrYs0Ykyj50sdU/JU0x8etU+LubXWb+gED6TbEdMIk=
+go.opentelemetry.io/proto/otlp v1.11.0/go.mod h1:SmVizdCOAm3XBtG1g1NnOdhW6jtddT72hLMhv8VwA8E=
 go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
 go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
 go.yaml.in/yaml/v2 v2.4.4 h1:tuyd0P+2Ont/d6e2rl3be67goVK4R6deVxCUX5vyPaQ=
@@ -141,6 +170,14 @@ golang.org/x/text v0.42.0 h1:JbOZXgfeCPU9gacVtYliJqOhD+zhrEqK4LfdpmlUZqI=
 golang.org/x/text v0.42.0/go.mod h1:ojzP1Z+2QtioaF8DTtO8K5q7JWVVYwZKenzujK0Zd0E=
 golang.org/x/tools v0.49.0 h1:3NI7VXzL9+1WZD52Dx2ttoPwD5DWrFGpl9mFZDlmisI=
 golang.org/x/tools v0.49.0/go.mod h1:SJNXV9DBKT0UbdttsQjbfJlAE/q+y36++zo3uL3N0Oo=
+gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4=
+gonum.org/v1/gonum v0.17.0/go.mod h1:El3tOrEuMpv2UdMrbNlKEh9vd86bmQ6vqIcDwxEOc1E=
+google.golang.org/genproto/googleapis/api v0.0.0-20260819154853-08b0e4226688 h1:ax2KzoSRIZU/M0cIxri3pKxy99vniH1PVxWC6si/eZI=
+google.golang.org/genproto/googleapis/api v0.0.0-20260819154853-08b0e4226688/go.mod h1:1RJ9BQGyNdZwkGc1eTqkErfRZ6RJyYPHZo73BZ1vQqI=
+google.golang.org/genproto/googleapis/rpc v0.0.0-20260819154853-08b0e4226688 h1:cYNAzI2sUwhmCcoj9TxvihSrqsxt6uIkj3rDRhSDmW4=
+google.golang.org/genproto/googleapis/rpc v0.0.0-20260819154853-08b0e4226688/go.mod h1:DjtHYE8FKJLivXcBEjGwndXfIC23G0VpXiXKqG179uA=
+google.golang.org/grpc v1.83.1 h1:HIO0+BEtBP6soyqvqC8sNUjZ7bTs+0hFQuFF+RAy++Y=
+google.golang.org/grpc v1.83.1/go.mod h1:kDyl6SKsiHKt0uylY5gtn5cEjkrIOhQOGDgIc4JGwzQ=
 google.golang.org/protobuf v1.36.12 h1:pJOKDDOyeXErUroCihFAd5LQuwXBSpVnKGrj5o/fwxc=
 google.golang.org/protobuf v1.36.12/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=

internal/account/password.go

@@ -15,6 +15,7 @@ import (
 	"encoding/base64"
 	"errors"
 	"fmt"
+	"runtime"
 	"strings"
 
 	"golang.org/x/crypto/argon2"
@@ -56,7 +57,7 @@ func HashPassword(password string) (string, error) {
 	if _, err := rand.Read(salt); err != nil {
 		return "", err
 	}
-	sum := argon2.IDKey([]byte(password), salt, argonTime, argonMemory, argonThreads, argonKeyLen)
+	sum := idKey([]byte(password), salt, argonTime, argonMemory, argonThreads, argonKeyLen)
 
 	b64 := base64.RawStdEncoding.EncodeToString
 	return fmt.Sprintf("$argon2id$v=%d$m=%d,t=%d,p=%d$%s$%s",
@@ -93,7 +94,7 @@ func VerifyPassword(password, encoded string) (bool, error) {
 		return false, ErrBadHash
 	}
 
-	got := argon2.IDKey([]byte(password), salt, time, memory, threads, uint32(len(want)))
+	got := idKey([]byte(password), salt, time, memory, threads, uint32(len(want)))
 	return subtle.ConstantTimeCompare(got, want) == 1, nil
 }
 
@@ -110,3 +111,16 @@ func init() {
 	}
 	dummyHash = h
 }
+
+// argonSlots bounds how many hashes run at once. Each takes 64 MiB, and the
+// login and signup forms are on the client-facing port: without a bound, a
+// few hundred parallel attempts at random addresses asked for gigabytes at
+// once, and the per-address throttle does not see them — each address is new.
+// Beyond the bound, attempts wait their turn instead.
+var argonSlots = make(chan struct{}, max(2, runtime.NumCPU()/2))
+
+func idKey(password, salt []byte, time, memory uint32, threads uint8, keyLen uint32) []byte {
+	argonSlots <- struct{}{}
+	defer func() { <-argonSlots }()
+	return argon2.IDKey(password, salt, time, memory, threads, keyLen)
+}

internal/account/service.go

@@ -230,6 +230,22 @@ func (s *Service) allowAttempt(email string) bool {
 	defer s.mu.Unlock()
 
 	w, ok := s.attempts[email]
+	if !ok && len(s.attempts) >= maxTrackedAddresses {
+		// Full of addresses nobody has signed in as for a while: forget the
+		// ones whose window has passed. Only expiring, never evicting early —
+		// dropping a live window is how an attacker spraying random addresses
+		// would buy unlimited guesses at the one they care about.
+		for k, v := range s.attempts {
+			if now.Sub(v.since) > s.cfg.AttemptWindow {
+				delete(s.attempts, k)
+			}
+		}
+		if len(s.attempts) >= maxTrackedAddresses {
+			// Still full, so somebody is spraying addresses faster than they
+			// expire. New ones wait; those already tracked are unaffected.
+			return false
+		}
+	}
 	if !ok || now.Sub(w.since) > s.cfg.AttemptWindow {
 		s.attempts[email] = &attemptWindow{count: 1, since: now}
 		return true
@@ -238,6 +254,10 @@ func (s *Service) allowAttempt(email string) bool {
 	return w.count <= s.cfg.MaxAttempts
 }
 
+// maxTrackedAddresses bounds the throttle's memory. Unbounded, it grew by one
+// entry per address tried and shrank only on a successful sign-in.
+const maxTrackedAddresses = 50000
+
 func (s *Service) clearAttempts(email string) {
 	s.mu.Lock()
 	delete(s.attempts, email)

internal/account/throttle_bound_test.go

@@ -0,0 +1,33 @@
+package account
+
+import (
+	"fmt"
+	"testing"
+	"time"
+)
+
+func TestSprayingAddressesCannotGrowTheThrottleOrEscapeIt(t *testing.T) {
+	now := time.Unix(0, 0)
+	s := &Service{cfg: DefaultConfig(), attempts: map[string]*attemptWindow{}, now: func() time.Time { return now }}
+
+	// The target is being guessed at.
+	for i := 0; i < s.cfg.MaxAttempts; i++ {
+		s.allowAttempt("victim@example.com")
+	}
+	// Then the table is flooded with made-up addresses.
+	for i := 0; i < maxTrackedAddresses+100; i++ {
+		s.allowAttempt(fmt.Sprintf("x%d@example.com", i))
+	}
+	if n := len(s.attempts); n > maxTrackedAddresses {
+		t.Errorf("the table grew to %d", n)
+	}
+	if s.allowAttempt("victim@example.com") {
+		t.Error("flooding the table reset the target's throttle")
+	}
+
+	// Once the window passes, everything expires and sign-ins work again.
+	now = now.Add(s.cfg.AttemptWindow + time.Second)
+	if !s.allowAttempt("new@example.com") {
+		t.Error("a new address was refused after the flood expired")
+	}
+}

internal/accounting/accounting.go

@@ -383,3 +383,39 @@ func keyCounter(keyID string) string {
 // they are asking is whether the budgets they are about to display mean
 // anything, and the accountant is what would enforce them.
 func (a *Accountant) AnythingPriced() bool { return a.pricing.AnythingPriced() }
+
+// Headroom is the tightest of a principal's request and token limits: how
+// much is left of each, and when the minute it is counted over rolls on.
+type Headroom struct {
+	RequestLimit, RequestsLeft int
+	TokenLimit, TokensLeft     int
+	Reset                      time.Duration
+}
+
+// Headroom reports what a principal has left, after whatever Check just
+// charged. Of several levels the one with least left is the one that will
+// refuse first, so that is the one reported. A limit of 0 means none is set.
+func (a *Accountant) Headroom(p auth.Principal) Headroom {
+	var h Headroom
+	firstReq, firstTok := true, true
+	for _, l := range levels(p.Limits, p.KeyID) {
+		if l.RPM != nil && *l.RPM > 0 {
+			left := max(0, *l.RPM-int(a.requests.Rate(l.Counter)))
+			if firstReq || left < h.RequestsLeft {
+				h.RequestLimit, h.RequestsLeft, firstReq = *l.RPM, left, false
+			}
+		}
+		if l.TPM != nil && *l.TPM > 0 {
+			left := max(0, *l.TPM-int(a.tokens.Rate(l.Counter)))
+			if firstTok || left < h.TokensLeft {
+				h.TokenLimit, h.TokensLeft, firstTok = *l.TPM, left, false
+			}
+		}
+	}
+	// The windows slide, so room comes back continuously; the next minute
+	// boundary is when a whole window's worth has moved on, and is the figure
+	// a client pacing itself can act on.
+	now := time.Now()
+	h.Reset = now.Truncate(windowLen).Add(windowLen).Sub(now)
+	return h
+}

internal/accounting/accounting_test.go

@@ -630,3 +630,31 @@ func TestAnEmptyPriceTableMeansNoBudgetCanEverBind(t *testing.T) {
 		t.Error("no price table at all is not priced")
 	}
 }
+
+func TestHeadroomIsTheTightestLimitAfterThisRequest(t *testing.T) {
+	// A project allowing 100 a minute under a team allowing 3: the team is
+	// what will refuse, so the team is what a client is told about.
+	a := New(pricing(t), nil, slog.New(slog.DiscardHandler))
+	p := principal("k1", func(e *store.Effective) {
+		e.Chain = []store.Limits{
+			{Scope: store.ScopeProject, ScopeID: "proj", RPM: intp(100), TPM: intp(50000)},
+			{Scope: store.ScopeTeam, ScopeID: "team", RPM: intp(3)},
+		}
+	})
+	if _, err := a.Check(p, "m", 10); err != nil {
+		t.Fatal(err)
+	}
+	h := a.Headroom(p)
+	if h.RequestLimit != 3 || h.RequestsLeft != 2 {
+		t.Errorf("requests %d left of %d; expected 2 of the team's 3", h.RequestsLeft, h.RequestLimit)
+	}
+	if h.TokenLimit != 50000 || h.TokensLeft != 50000-10 {
+		t.Errorf("tokens %d left of %d", h.TokensLeft, h.TokenLimit)
+	}
+	if h.Reset <= 0 || h.Reset > time.Minute {
+		t.Errorf("reset %v", h.Reset)
+	}
+	if none := a.Headroom(principal("k2", nil)); none.RequestLimit != 0 || none.TokenLimit != 0 {
+		t.Errorf("a key with no limits advertised some: %+v", none)
+	}
+}

internal/agent/agent.go

@@ -14,13 +14,9 @@ import (
 	"math/rand/v2"
 	"net"
 	"net/http"
-	"net/url"
 	"os"
-	"os/exec"
 	"path/filepath"
 	"slices"
-	"strconv"
-	"strings"
 	"sync"
 	"time"
 
@@ -34,278 +30,9 @@ import (
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/supervisor"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/weights"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/agentsvc"
-	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
 )
 
-// Config configures the agent.
-type Config struct {
-	// NodeID defaults to the hostname. It must equal the common name of the
-	// agent's certificate, so set it only where the two differ.
-	NodeID string `koanf:"node_id"`
-	// Server is the control plane, and Servers is the same thing when there is
-	// more than one. Both are accepted: every configuration written before
-	// there could be a second control plane names one address, and none of
-	// them should have to change.
-	//
-	// The order is tried in order, and a refusal moves to the next. Only the
-	// leader accepts an agent, so a standby is skipped in the time it takes to
-	// be told so — which is what makes failover a reconnect rather than an
-	// edit to every node's configuration.
-	Server  string   `koanf:"server"`
-	Servers []string `koanf:"servers"`
-	// Hostname defaults to the machine's.
-	Hostname string `koanf:"hostname"`
-	// MgmtAddr is the address engines are reached on. It defaults to this
-	// machine's address on the route to the control plane, which is the one
-	// the control plane and the router can reach.
-	MgmtAddr string `koanf:"mgmt_addr"`
-	// Labels are free-form facts about a node for a profile's node_selector.
-	// Nothing needs them: memory, GPUs, engines and disk are measured, and a
-	// label restating one of those goes stale the day the machine changes.
-	Labels map[string]string `koanf:"labels"`
-
-	// MgmtInterface is the ethernet interface NCCL bootstraps over. It
-	// defaults to the interface holding MgmtAddr, then eth0.
-	MgmtInterface string `koanf:"mgmt_interface"`
-	// Fabrics labels this node's HCAs. It is explicit on purpose: OpenSM's
-	// default subnet prefix is the same on every fabric, so nothing can be
-	// inferred from the wire (plan §5.2).
-	Fabrics []inventory.FabricAttachment `koanf:"fabrics"`
-
-	HeartbeatInterval time.Duration `koanf:"heartbeat_interval"`
-	// ReconnectBase is the first reconnect delay; it backs off from there.
-	ReconnectBase time.Duration `koanf:"reconnect_base"`
-	ReconnectMax  time.Duration `koanf:"reconnect_max"`
-
-	// AdoptReadyTimeout bounds how long an adopted engine is given to answer a
-	// health check before the agent stops waiting on it.
-	//
-	// Shorter than a startup timeout on purpose: this engine is already
-	// running, so it is not loading a model, it is either serving or it is not.
-	AdoptReadyTimeout time.Duration `koanf:"adopt_ready_timeout"`
-
-	// TLS authenticates both ends of the control-plane stream. Leaving it unset
-	// keeps the plaintext behaviour, which only belongs on a trusted network.
-	TLS tlsconf.Config `koanf:"tls"`
-
-	// ModelDirs are the directories this node keeps model weights in, beyond
-	// WeightsCache, which is always looked in. Storage differs at every installation,
-	// so llmgw observes it rather than owning it: a Lustre mount, an image
-	// exported over the fabric, local NVMe and a copy made by hand all look
-	// identical from here, and placement only needs to know whether the file
-	// is present.
-	ModelDirs []string `koanf:"model_dirs"`
-
-	// EnginesDir is where the agent installs engines, one directory per
-	// build. Defaults to /opt/llmgw/engines.
-	EnginesDir string `koanf:"engines_dir"`
-
-	// WeightsCache is where a remote source is downloaded to. Defaults to the
-	// first entry of ModelDirs, because a node that says where its models live
-	// has already answered this question, and otherwise to
-	// /var/lib/llmgw-weights.
-	//
-	// Fetching is the simple arrangement: every node keeps its own copy. The
-	// exotic ones need no code — weights on a shared filer or an image exported
-	// over the fabric are simply present, and present means nothing is fetched.
-	WeightsCache string `koanf:"weights_cache"`
-
-	Discovery discovery.Options `koanf:"discovery"`
-	Engines   engines.Config    `koanf:"engines"`
-	Superv    supervisor.Config `koanf:"supervisor"`
-
-	Version string
-}
-
-// DefaultConfig returns sensible defaults.
-func DefaultConfig() Config {
-	// No MgmtInterface: normalize finds the one holding the management
-	// address, and falls back to eth0 only when it cannot. Defaulted here, it
-	// was never found — every node said eth0 unless told otherwise, and a node
-	// on ens3 or bond0 had NCCL bootstrap over an interface it does not have.
-	return Config{
-		HeartbeatInterval: 5 * time.Second,
-		ReconnectBase:     time.Second,
-		ReconnectMax:      30 * time.Second,
-		AdoptReadyTimeout: 60 * time.Second,
-		Superv:            supervisor.DefaultConfig(),
-		// NVML gives NVLink topology, so prefer it when this build has it.
-		Discovery: discovery.Options{PreferNVML: true},
-	}
-}
-
-func (c *Config) normalize() error {
-	d := DefaultConfig()
-	if err := c.TLS.Validate(); err != nil {
-		return err
-	}
-	if err := c.normalizeServerURL(); err != nil {
-		return err
-	}
-	// Everything below is found rather than written. A file that has to say
-	// which machine it is on, and at what address, is a file that differs on
-	// every node — and a copy taken from one node to the next registers the
-	// second as the first.
-	host, _ := os.Hostname()
-	if c.NodeID == "" {
-		c.NodeID = host
-	}
-	if c.NodeID == "" {
-		return errors.New("node_id is not set and the hostname could not be read")
-	}
-	if c.Hostname == "" {
-		c.Hostname = host
-	}
-	if c.MgmtAddr == "" {
-		c.MgmtAddr = addrToward(c.Server)
-	}
-	if c.MgmtInterface == "" {
-		c.MgmtInterface = interfaceOf(c.MgmtAddr)
-	}
-	if c.MgmtInterface == "" {
-		c.MgmtInterface = "eth0"
-	}
-	// The directory downloads land in is always one that is looked in. When
-	// they were two settings, a node could fill its disk with fetched weights
-	// that placement never saw — which is what every node here was doing.
-	if c.WeightsCache == "" && len(c.ModelDirs) > 0 {
-		c.WeightsCache = c.ModelDirs[0]
-	}
-	if c.WeightsCache == "" {
-		c.WeightsCache = DefaultWeightsCache
-	}
-	if !slices.Contains(c.ModelDirs, c.WeightsCache) {
-		c.ModelDirs = append(c.ModelDirs, c.WeightsCache)
-	}
-	if c.HeartbeatInterval <= 0 {
-		c.HeartbeatInterval = d.HeartbeatInterval
-	}
-	if c.ReconnectBase <= 0 {
-		c.ReconnectBase = d.ReconnectBase
-	}
-	if c.AdoptReadyTimeout <= 0 {
-		c.AdoptReadyTimeout = d.AdoptReadyTimeout
-	}
-	if c.ReconnectMax < c.ReconnectBase {
-		c.ReconnectMax = max(c.ReconnectBase, d.ReconnectMax)
-	}
-	return nil
-}
-
-// normalizeServerURL makes the configured address a URL connect can use, and
-// refuses the combination that would quietly undo the certificates: a scheme
-// that says plaintext while TLS is configured. Silently connecting anyway would
-// mean a cluster believing it has mutual authentication when it has none.
-func (c *Config) normalizeServerURL() error {
-	// One list, however it was written. Server first when both are given,
-	// because an operator adding a standby to an existing file expects the
-	// one already there to stay preferred.
-	all := make([]string, 0, 1+len(c.Servers))
-	if c.Server != "" {
-		all = append(all, c.Server)
-	}
-	all = append(all, c.Servers...)
-
-	seen := map[string]struct{}{}
-	out := make([]string, 0, len(all))
-	for _, s := range all {
-		fixed, err := normalizeOne(s, c.TLS.Enabled())
-		if err != nil {
-			return err
-		}
-		// A duplicate would be tried twice in a row on failover, which reads
-		// as the rotation not working.
-		if _, dup := seen[fixed]; dup {
-			continue
-		}
-		seen[fixed] = struct{}{}
-		out = append(out, fixed)
-	}
-	if len(out) == 0 {
-		return errors.New("server is required")
-	}
-	c.Servers = out
-	c.Server = out[0]
-	return nil
-}
-
-// DefaultWeightsCache is where fetched weights go when nothing says otherwise.
-const DefaultWeightsCache = "/var/lib/llmgw-weights"
-
-// addrToward is this machine's address on the route to the control plane.
-//
-// The address engines are reached on is whichever one the control plane can
-// reach, and the kernel already knows which that is: a UDP "connection" picks
-// the source address for the route without sending a packet. Asking the
-// interfaces instead would have to guess among docker bridges, VPNs and
-// link-local addresses.
-func addrToward(server string) string {
-	u, err := url.Parse(server)
-	if err != nil || u.Host == "" {
-		return ""
-	}
-	host := u.Hostname()
-	port := u.Port()
-	if port == "" {
-		port = "443"
-	}
-	conn, err := net.Dial("udp", net.JoinHostPort(host, port))
-	if err != nil {
-		return ""
-	}
-	defer conn.Close()
-	if a, ok := conn.LocalAddr().(*net.UDPAddr); ok {
-		return a.IP.String()
-	}
-	return ""
-}
-
-// interfaceOf names the interface that holds an address, or "" when none does.
-func interfaceOf(addr string) string {
-	ip := net.ParseIP(addr)
-	if ip == nil {
-		return ""
-	}
-	ifaces, err := net.Interfaces()
-	if err != nil {
-		return ""
-	}
-	for _, ifc := range ifaces {
-		addrs, err := ifc.Addrs()
-		if err != nil {
-			continue
-		}
-		for _, a := range addrs {
-			if n, ok := a.(*net.IPNet); ok && n.IP.Equal(ip) {
-				return ifc.Name
-			}
-		}
-	}
-	return ""
-}
-
-// normalizeOne applies the scheme rules to a single address.
-func normalizeOne(addr string, secure bool) (string, error) {
-	switch {
-	case strings.HasPrefix(addr, "https://"):
-		if !secure {
-			return "", fmt.Errorf("%s is an https:// address but no tls certificates are configured", addr)
-		}
-	case strings.HasPrefix(addr, "http://"):
-		if secure {
-			return "", fmt.Errorf("tls is configured but %s is an http:// address; use https://", addr)
-		}
-	case strings.Contains(addr, "://"):
-		return "", fmt.Errorf("server %q must be an http:// or https:// address", addr)
-	case secure:
-		return "https://" + addr, nil
-	default:
-		return "http://" + addr, nil
-	}
-	return addr, nil
-}
-
 // Agent runs on a GPU node.
 type Agent struct {
 	cfg  Config
@@ -586,131 +313,6 @@ func (a *Agent) settleAdopted(sh *supervisor.Shard, o supervisor.Orphan) {
 		"shard", o.ShardID, "replica", o.ReplicaID)
 }
 
-// usableFabrics reports the fabrics this node can actually carry traffic on.
-//
-// Loud about what it drops. A fabric silently missing from a registration looks
-// identical to one never configured, and the operator who typed the adapter
-// name is the only person who can fix it — so the reason is logged here, on the
-// machine where both halves are visible, rather than inferred later from a
-// replica that will not converge.
-func (a *Agent) usableFabrics(inv discovery.Inventory) []inventory.FabricAttachment {
-	out := a.checkFabrics(inv.HCAs, true)
-	a.mu.Lock()
-	a.fabricsSent = fabricsFingerprint(out)
-	a.mu.Unlock()
-	return out
-}
-
-// checkFabrics is the declared fabrics this node can carry traffic on, from
-// the adapters as they are now. Loud logs every problem; the heartbeat checks
-// quietly and speaks only when the answer changes.
-func (a *Agent) checkFabrics(hcas []discovery.HCA, loud bool) []inventory.FabricAttachment {
-	declared := map[string][]string{}
-	for _, f := range a.cfg.Fabrics {
-		declared[f.Fabric] = f.HCAs
-	}
-
-	var out []inventory.FabricAttachment
-	for _, c := range discovery.CheckFabrics(declared, hcas) {
-		if loud {
-			for _, p := range c.Problems {
-				a.log.Error("a declared fabric adapter cannot carry traffic",
-					"fabric", c.Fabric, "problem", p)
-			}
-		}
-		if !c.OK() {
-			if loud {
-				a.log.Error("fabric not advertised; no declared adapter on it can carry traffic",
-					"fabric", c.Fabric,
-					"consequence", "no multi-node replica will be placed here")
-			}
-			continue
-		}
-		out = append(out, inventory.FabricAttachment{
-			Fabric: c.Fabric, HCAs: c.Usable, RateGbps: c.RateGbps,
-		})
-	}
-	return out
-}
-
-// fabricsFingerprint is enough to notice a fabric, an adapter or a rate
-// changing.
-func fabricsFingerprint(fs []inventory.FabricAttachment) string {
-	var b strings.Builder
-	for _, f := range fs {
-		fmt.Fprintf(&b, "%s:%s:%d;", f.Fabric, strings.Join(f.HCAs, ","), f.RateGbps)
-	}
-	return b.String()
-}
-
-// sampleIB re-reads the adapters for the heartbeat: every port's state and
-// counters, and the fabrics again when they no longer match what the control
-// plane was told.
-//
-// Every heartbeat, because it is a few small files per port, and because a
-// port that went down used to be invisible until the agent restarted — the
-// node went on being offered multi-node work that would hang on its first
-// collective.
-func (a *Agent) sampleIB(hb *llmgwv1.Heartbeat) {
-	a.mu.Lock()
-	had := len(a.inv.HCAs) > 0
-	a.mu.Unlock()
-	if !had && len(a.cfg.Fabrics) == 0 {
-		return
-	}
-	hcas, err := discovery.DiscoverIB(a.cfg.Discovery.IBRoot)
-	if err != nil {
-		return
-	}
-	a.attachSM(hcas)
-	for _, h := range hcas {
-		for _, p := range h.Ports {
-			ip := &llmgwv1.IBPort{
-				Hca: h.Name, Port: int32(p.Num), State: p.State, PhysState: p.PhysState,
-				RateGbps: int32(p.RateGbps), LinkLayer: string(p.LinkLayer), Counters: p.Counters,
-				NodeGuid: h.NodeGUID, PortGuid: p.PortGUID, Network: p.Network, GidPrefix: p.GIDPrefix,
-			}
-			a.mu.Lock()
-			if p.SM != nil {
-				ip.SmGuid, ip.SmLid, ip.SmPriority, ip.SmState = p.SM.GUID, int32(p.SM.LID), int32(p.SM.Priority), p.SM.State
-			} else {
-				ip.SmError = a.smErr[smKey(h.Name, p.Num)]
-			}
-			for _, r := range a.sms[smKey(h.Name, p.Num)] {
-				ip.SubnetManagers = append(ip.SubnetManagers, &llmgwv1.SubnetManager{
-					Guid: r.GUID, Lid: int32(r.LID), Priority: int32(r.Priority), State: r.State})
-			}
-			ip.SubnetManagersError = a.smsErr[smKey(h.Name, p.Num)]
-			a.mu.Unlock()
-			hb.IbPorts = append(hb.IbPorts, ip)
-		}
-	}
-	a.mu.Lock()
-	a.inv.HCAs = hcas // a reconnect registers what is true now
-	a.mu.Unlock()
-	if len(a.cfg.Fabrics) == 0 {
-		return
-	}
-	now := a.checkFabrics(hcas, false)
-	fp := fabricsFingerprint(now)
-	a.mu.Lock()
-	changed := fp != a.fabricsSent
-	a.fabricsSent = fp
-	a.mu.Unlock()
-	if !changed {
-		return
-	}
-	// Said once, with the reasons, the moment it happens.
-	a.checkFabrics(hcas, true)
-	a.log.Warn("the fabrics this node can carry traffic on changed", "now", fp)
-	hb.FabricsChanged = true
-	for _, f := range now {
-		hb.Fabrics = append(hb.Fabrics, &llmgwv1.FabricAttachment{
-			Fabric: f.Fabric, Hcas: f.HCAs, RateGbps: int32(f.RateGbps),
-		})
-	}
-}
-
 // weights builds the fetcher lazily, so an agent that never starts a shard
 // never needs a cache directory.
 func (a *Agent) weights() *weights.Fetcher {
@@ -1119,6 +721,8 @@ func (a *Agent) handle(ctx context.Context, msg *llmgwv1.ControlMessage, out cha
 		a.removeWeights(m.RemoveWeights.GetPaths())
 	case *llmgwv1.ControlMessage_Provision:
 		a.setProvision(m.Provision)
+	case *llmgwv1.ControlMessage_Upgrade:
+		go a.upgrade(ctx, m.Upgrade, out)
 	case *llmgwv1.ControlMessage_Logs:
 		reply := a.logsReply(m.Logs)
 		// Off the receive loop: a full send queue must not stop commands being
@@ -1410,215 +1014,6 @@ func Client(cfg tlsconf.Config, timeout time.Duration) (*http.Client, error) {
 	return &http.Client{Transport: &http2.Transport{TLSClientConfig: tc}}, nil
 }
 
-// engines is what this node can run: what the agent installed and saw run,
-// and what the configuration names and answers. Nothing merely found on the
-// machine — a llama-server on PATH is somebody's, not necessarily one that
-// works, and placing on it is how a replica fails on a node nobody watches.
-func (a *Agent) engines(ctx context.Context) map[string]string {
-	a.engineOnce.Do(func() {
-		vers := a.reg.ProbeConfigured(ctx)
-		a.mu.Lock()
-		a.configuredVers, a.lastEngineProbe = vers, time.Now()
-		a.mu.Unlock()
-		if len(vers) > 0 {
-			a.log.Info("engines named in the configuration",
-				"engines", engines.ProbedNames(vers))
-		}
-	})
-	a.mu.Lock()
-	defer a.mu.Unlock()
-	vers := a.engineReportLocked()
-	// Whatever registration sends is what the control plane now knows, so a
-	// heartbeat repeats it only once it differs.
-	a.enginesSent = enginesFingerprint(vers)
-	return vers
-}
-
-// engineReportLocked merges the two sources. Installed wins: it is the one
-// that was seen to run. Called with mu held.
-func (a *Agent) engineReportLocked() map[string]string {
-	out := maps.Clone(a.configuredVers)
-	if out == nil {
-		out = map[string]string{}
-	}
-	maps.Copy(out, a.installedVers)
-	return out
-}
-
-// provisionEvery is how often installed engines are checked again. A library
-// removed by an unrelated upgrade is found here rather than by the next
-// replica to start.
-const provisionEvery = 10 * time.Minute
-
-// setProvision takes the control plane's list of wanted engines.
-func (a *Agent) setProvision(p *llmgwv1.Provision) {
-	wants := []provision.Want{}
-	for _, e := range p.GetEngines() {
-		wants = append(wants, provision.Want{Engine: e.GetEngine(), Version: e.GetVersion()})
-	}
-	a.mu.Lock()
-	a.provWants = wants
-	a.mu.Unlock()
-	select {
-	case a.provKick <- struct{}{}:
-	default:
-	}
-}
-
-// provisionLoop installs and re-checks the wanted engines, one pass at a time.
-func (a *Agent) provisionLoop(ctx context.Context) {
-	// After a failure the next pass comes sooner, and backs off: a lock held
-	// by a new machine's own updates clears in a minute or two, while ten
-	// minutes' wait left a node idle for no reason.
-	next := provisionEvery
-	retry := time.Minute
-	for {
-		select {
-		case <-ctx.Done():
-			return
-		case <-a.provKick:
-		case <-time.After(next):
-		}
-		a.mu.Lock()
-		wants := slices.Clone(a.provWants)
-		managed := a.provWants != nil
-		a.mu.Unlock()
-		if !managed {
-			continue
-		}
-
-		statuses := a.prov.Converge(ctx, wants)
-		next = provisionEvery
-		for _, st := range statuses {
-			if st.State == provision.Failed {
-				next, retry = retry, min(retry*2, provisionEvery)
-				break
-			}
-		}
-		if next == provisionEvery {
-			retry = time.Minute
-		}
-		installed := map[string]string{}
-		keep := map[string]bool{}
-		for _, st := range statuses {
-			keep[st.Engine] = true
-			if st.State == provision.Ready {
-				a.reg.UseInstalled(st.Engine, st.Binary)
-				installed[st.Engine] = st.Version
-			}
-		}
-		a.mu.Lock()
-		if enginesFingerprint(installed) != enginesFingerprint(a.installedVers) {
-			a.log.Info("engines installed and running", "engines", engines.ProbedNames(installed))
-		}
-		a.installedVers = installed
-		// An engine no longer asked for is no longer reported on.
-		for e := range a.provStatus {
-			if !keep[e] {
-				delete(a.provStatus, e)
-			}
-		}
-		a.mu.Unlock()
-	}
-}
-
-// provisionStatus records a change in one engine's installation.
-func (a *Agent) provisionStatus(st provision.Status) {
-	a.mu.Lock()
-	prev, had := a.provStatus[st.Engine]
-	a.provStatus[st.Engine] = st
-	a.mu.Unlock()
-	if !had || prev.State != st.State || prev.Detail != st.Detail {
-		lvl := slog.LevelInfo
-		if st.State == provision.Failed || st.State == provision.Unsupported {
-			lvl = slog.LevelWarn
-		}
-		a.log.Log(context.Background(), lvl, "engine installation",
-			"engine", st.Engine, "version", st.Version, "variant", st.Variant,
-			"state", st.State, "detail", st.Detail)
-	}
-}
-
-func provisionFingerprint(m map[string]provision.Status) string {
-	var b strings.Builder
-	for _, k := range slices.Sorted(maps.Keys(m)) {
-		st := m[k]
-		fmt.Fprintf(&b, "%s|%s|%s|%s|%s\n", st.Engine, st.Version, st.Variant, st.State, st.Detail)
-	}
-	return b.String()
-}
-
-// engineReprobe is how often the installed engines are looked for again.
-//
-// Minutes, not seconds: an engine is installed by somebody at a keyboard, so a
-// minute's delay in noticing is invisible, while SGLang's probe starts a
-// Python interpreter that imports it.
-const engineReprobe = time.Minute
-
-// reprobeEngines looks for installed engines again when it is time, off the
-// heartbeat goroutine so a slow probe never delays a heartbeat.
-func (a *Agent) reprobeEngines() {
-	a.mu.Lock()
-	due := !a.probingEngines && !a.lastEngineProbe.IsZero() &&
-		time.Since(a.lastEngineProbe) >= engineReprobe
-	if due {
-		a.probingEngines = true
-	}
-	a.mu.Unlock()
-	if !due {
-		return
-	}
-	go func() {
-		ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
-		defer cancel()
-		vers := a.reg.ProbeConfigured(ctx)
-
-		a.mu.Lock()
-		changed := enginesFingerprint(vers) != enginesFingerprint(a.configuredVers)
-		a.configuredVers, a.lastEngineProbe, a.probingEngines = vers, time.Now(), false
-		a.mu.Unlock()
-		if changed {
-			a.log.Info("engines named in the configuration changed",
-				"engines", engines.ProbedNames(vers))
-		}
-	}()
-}
-
-func enginesFingerprint(vers map[string]string) string {
-	var b strings.Builder
-	for _, k := range slices.Sorted(maps.Keys(vers)) {
-		fmt.Fprintf(&b, "%s=%s\n", k, vers[k])
-	}
-	return b.String()
-}
-
-// logsReply answers a request for one of this node's logs.
-func (a *Agent) logsReply(req *llmgwv1.LogsRequest) *llmgwv1.LogsReply {
-	n := int(req.GetLines())
-	if n <= 0 {
-		n = 200
-	}
-	n = min(n, agentLogLines)
-	shards := a.sup.IDs()
-	slices.Sort(shards)
-	out := &llmgwv1.LogsReply{
-		RequestId: req.GetRequestId(),
-		Sources:   append([]string{"agent"}, shards...),
-	}
-	switch src := req.GetSource(); src {
-	case "", "agent":
-		out.Lines = a.logRing.tail(n)
-	default:
-		sh, err := a.sup.Get(src)
-		if err != nil {
-			out.Error = "no engine " + src + " runs on this node"
-			return out
-		}
-		out.Lines = sh.Logs(n)
-	}
-	return out
-}
-
 // localizeSpec fills in what only this node knows.
 //
 // The interface NCCL bootstraps over is this node's, not the cluster's: the
@@ -1669,208 +1064,3 @@ func ResolveManagement(c Config) (addr, iface string) {
 	}
 	return addr, iface
 }
-
-func smKey(hca string, port int) string { return fmt.Sprintf("%s/%d", hca, port) }
-
-// scanFabric is discovery.ScanFabric, replaced in tests.
-var scanFabric = discovery.ScanFabric
-
-// querySMs is discovery.QuerySMs, replaced in tests.
-var querySMs = func(hca string, port int) ([]discovery.SMRecord, error) {
-	return discovery.QuerySMs(context.Background(), hca, port)
-}
-
-func splitSMKey(k string) (string, int) {
-	i := strings.LastIndexByte(k, '/')
-	port, _ := strconv.Atoi(k[i+1:])
-	return k[:i], port
-}
-
-// querySM is discovery.QuerySM, replaced in tests.
-var querySM = func(hca string, port int) (*discovery.SMInfo, error) {
-	return discovery.QuerySM(context.Background(), hca, port)
-}
-
-// attachSM puts on each active InfiniBand port the subnet manager it sees,
-// asking again once a minute and otherwise reusing the last answer.
-func (a *Agent) attachSM(hcas []discovery.HCA) {
-	a.mu.Lock()
-	stale := time.Since(a.smAsked) > time.Minute || a.sm == nil
-	a.mu.Unlock()
-	if stale {
-		sm, errs := map[string]*discovery.SMInfo{}, map[string]string{}
-		_ = discovery.AttachSM(hcas, func(hca string, port int) (*discovery.SMInfo, error) {
-			got, err := querySM(hca, port)
-			if err != nil {
-				errs[smKey(hca, port)] = err.Error()
-				return nil, err
-			}
-			sm[smKey(hca, port)] = got
-			return got, nil
-		})
-		// Every subnet manager on each fabric, through one port per master:
-		// the same list through every port of one fabric is one query too
-		// many for each.
-		sms, smsErr := map[string][]discovery.SMRecord{}, map[string]string{}
-		asked := map[string]bool{}
-		for k, v := range sm {
-			if asked[v.GUID] {
-				continue
-			}
-			asked[v.GUID] = true
-			hca, port := splitSMKey(k)
-			if recs, err := querySMs(hca, port); err != nil {
-				smsErr[k] = err.Error()
-			} else {
-				sms[k] = recs
-			}
-		}
-		a.mu.Lock()
-		for k, v := range sm {
-			if old := a.sm[k]; old != nil && old.GUID != v.GUID {
-				a.log.Warn("the master subnet manager changed", "port", k, "from", old.GUID, "to", v.GUID)
-			}
-		}
-		a.sm, a.smErr, a.smAsked = sm, errs, time.Now()
-		a.sms, a.smsErr = sms, smsErr
-		a.mu.Unlock()
-		return
-	}
-	a.mu.Lock()
-	defer a.mu.Unlock()
-	for i := range hcas {
-		for j := range hcas[i].Ports {
-			if sm := a.sm[smKey(hcas[i].Name, hcas[i].Ports[j].Num)]; sm != nil && hcas[i].Ports[j].Active() {
-				hcas[i].Ports[j].SM = sm
-			}
-		}
-	}
-}
-
-// ensureDiags installs infiniband-diags on a node with adapters and without
-// it. sminfo is what tells which physical fabric a port is on; without it,
-// two separate fabrics given one name cannot be caught. Installed the way
-// engine dependencies are, from the machine's own package sources.
-func (a *Agent) ensureDiags(ctx context.Context) {
-	a.mu.Lock()
-	hasIB := len(a.inv.HCAs) > 0
-	a.mu.Unlock()
-	if !hasIB || a.prov == nil {
-		return
-	}
-	if _, err := exec.LookPath("sminfo"); err == nil {
-		return
-	}
-	a.log.Info("installing infiniband-diags, to tell which fabric each port is on")
-	pkgs := map[string][]string{"deb": {"infiniband-diags"}, "rpm": {"infiniband-diags"}}
-	if err := a.prov.InstallPackages(ctx, pkgs); err != nil {
-		a.log.Warn("could not install infiniband-diags; fabrics cannot be told apart by their subnet manager",
-			"err", err)
-	}
-}
-
-// FeatureFabricTest is the agent answering FabricTestRequest.
-const FeatureFabricTest = "fabric-test"
-
-// runWriteBW is discovery.RunWriteBW, and haveWriteBW whether it is
-// installed; both replaced in tests.
-var (
-	runWriteBW  = discovery.RunWriteBW
-	haveWriteBW = func() bool { _, err := exec.LookPath("ib_write_bw"); return err == nil }
-)
-
-// fabricTest runs this node's side of a bandwidth test with another.
-func (a *Agent) fabricTest(ctx context.Context, req *llmgwv1.FabricTestRequest) *llmgwv1.FabricTestReply {
-	start := time.Now()
-	reply := &llmgwv1.FabricTestReply{RequestId: req.GetRequestId()}
-	defer func() { reply.TookMs = time.Since(start).Milliseconds() }()
-
-	w := discovery.WriteBW{HCA: req.GetHca(), Port: int(req.GetPort()), GIDIndex: -1,
-		TCPPort: int(req.GetTcpPort()), Seconds: int(req.GetSeconds())}
-	if req.GetRole() == "client" {
-		w.Peer = req.GetPeer()
-	}
-	a.mu.Lock()
-	found := false
-	for _, h := range a.inv.HCAs {
-		for _, p := range h.Ports {
-			if h.Name == w.HCA && p.Num == w.Port {
-				found = true
-				// RoCE needs its v2 GID named, as NCCL does; InfiniBand uses the LID.
-				if p.LinkLayer == discovery.LinkEthernet && p.RoCEv2GID >= 0 {
-					w.GIDIndex = p.RoCEv2GID
-				}
-			}
-		}
-	}
-	a.mu.Unlock()
-	if !found {
-		reply.Error = fmt.Sprintf("this node has no port %s/%d", w.HCA, w.Port)
-		return reply
-	}
-	if !haveWriteBW() && a.prov != nil {
-		a.log.Info("installing perftest, for a bandwidth test with another node")
-		pkgs := map[string][]string{"deb": {"perftest"}, "rpm": {"perftest"}}
-		if err := a.prov.InstallPackages(ctx, pkgs); err != nil {
-			reply.Error = "installing perftest: " + err.Error()
-			return reply
-		}
-	}
-	gbps, out, err := runWriteBW(ctx, w)
-	reply.Gbps = gbps
-	reply.Output = tailLines(out, 12)
-	if err != nil {
-		reply.Error = err.Error()
-	}
-	return reply
-}
-
-func tailLines(s string, n int) string {
-	lines := strings.Split(strings.TrimRight(s, "\n"), "\n")
-	if len(lines) > n {
-		lines = lines[len(lines)-n:]
-	}
-	return strings.Join(lines, "\n")
-}
-
-// discoverGPUFabric and discoverIMEX are replaced in tests.
-var (
-	discoverGPUFabric = discovery.DiscoverGPUFabric
-	discoverIMEX      = discovery.DiscoverIMEX
-)
-
-// sampleNVLink reads, once a minute, which NVLink domain each GPU is joined
-// to and whether IMEX is up. nvidia-smi -q takes a second or two, too long
-// for every heartbeat, and a fabric that was joined an hour ago rarely
-// changes in five seconds — but when a fabric manager restarts it does, for
-// every card at once.
-func (a *Agent) sampleNVLink(hb *llmgwv1.Heartbeat) {
-	a.mu.Lock()
-	gpus := slices.Clone(a.inv.GPUs)
-	due := time.Since(a.nvlAsked) > time.Minute
-	if due {
-		a.nvlAsked = time.Now()
-	}
-	a.mu.Unlock()
-	if len(gpus) == 0 || !due {
-		return
-	}
-	ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
-	defer cancel()
-	fab, err := discoverGPUFabric(ctx, a.cfg.Discovery.NvidiaSMI)
-	if err != nil || len(fab) == 0 {
-		return
-	}
-	discovery.ApplyFabric(gpus, fab)
-	for _, g := range gpus {
-		if g.Fabric == nil {
-			continue
-		}
-		hb.GpuFabric = append(hb.GpuFabric, &llmgwv1.GPUFabricState{
-			Index: int32(g.Index), State: g.Fabric.State, Status: g.Fabric.Status,
-			ClusterUuid: g.Fabric.ClusterUUID, CliqueId: g.Fabric.CliqueID, Bandwidth: g.Fabric.Bandwidth,
-		})
-	}
-	m := discoverIMEX(ctx)
-	hb.Imex = &llmgwv1.IMEX{State: m.State, Channels: int32(m.Channels)}
-}

internal/agent/agent_test.go

@@ -400,11 +400,16 @@ func TestHeartbeatKeepsNodeAlive(t *testing.T) {
 	n, _ := h.inv.Get("n1")
 	first := n.LastSeen
 
-	time.Sleep(200 * time.Millisecond)
-	n, _ = h.inv.Get("n1")
-	if !n.LastSeen.After(first) {
-		t.Error("heartbeats are not refreshing LastSeen; the node would be declared lost")
+	// Waited for rather than slept through: four heartbeat intervals was not
+	// always enough on a loaded runner for one to have landed.
+	deadline := time.Now().Add(3 * time.Second)
+	for time.Now().Before(deadline) {
+		if n, _ = h.inv.Get("n1"); n.LastSeen.After(first) {
+			return
+		}
+		time.Sleep(20 * time.Millisecond)
 	}
+	t.Error("heartbeats are not refreshing LastSeen; the node would be declared lost")
 }
 
 func TestStartAndStopShardEndToEnd(t *testing.T) {

internal/agent/config.go

@@ -0,0 +1,286 @@
+package agent
+
+import (
+	"errors"
+	"fmt"
+	"net"
+	"net/url"
+	"os"
+	"slices"
+	"strings"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/supervisor"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
+)
+
+// Config configures the agent.
+type Config struct {
+	// NodeID defaults to the hostname. It must equal the common name of the
+	// agent's certificate, so set it only where the two differ.
+	NodeID string `koanf:"node_id"`
+	// Server is the control plane, and Servers is the same thing when there is
+	// more than one. Both are accepted: every configuration written before
+	// there could be a second control plane names one address, and none of
+	// them should have to change.
+	//
+	// The order is tried in order, and a refusal moves to the next. Only the
+	// leader accepts an agent, so a standby is skipped in the time it takes to
+	// be told so — which is what makes failover a reconnect rather than an
+	// edit to every node's configuration.
+	Server  string   `koanf:"server"`
+	Servers []string `koanf:"servers"`
+	// Hostname defaults to the machine's.
+	Hostname string `koanf:"hostname"`
+	// MgmtAddr is the address engines are reached on. It defaults to this
+	// machine's address on the route to the control plane, which is the one
+	// the control plane and the router can reach.
+	MgmtAddr string `koanf:"mgmt_addr"`
+	// Labels are free-form facts about a node for a profile's node_selector.
+	// Nothing needs them: memory, GPUs, engines and disk are measured, and a
+	// label restating one of those goes stale the day the machine changes.
+	Labels map[string]string `koanf:"labels"`
+
+	// MgmtInterface is the ethernet interface NCCL bootstraps over. It
+	// defaults to the interface holding MgmtAddr, then eth0.
+	MgmtInterface string `koanf:"mgmt_interface"`
+	// Fabrics labels this node's HCAs. It is explicit on purpose: OpenSM's
+	// default subnet prefix is the same on every fabric, so nothing can be
+	// inferred from the wire (plan §5.2).
+	Fabrics []inventory.FabricAttachment `koanf:"fabrics"`
+
+	HeartbeatInterval time.Duration `koanf:"heartbeat_interval"`
+	// ReconnectBase is the first reconnect delay; it backs off from there.
+	ReconnectBase time.Duration `koanf:"reconnect_base"`
+	ReconnectMax  time.Duration `koanf:"reconnect_max"`
+
+	// AdoptReadyTimeout bounds how long an adopted engine is given to answer a
+	// health check before the agent stops waiting on it.
+	//
+	// Shorter than a startup timeout on purpose: this engine is already
+	// running, so it is not loading a model, it is either serving or it is not.
+	AdoptReadyTimeout time.Duration `koanf:"adopt_ready_timeout"`
+
+	// TLS authenticates both ends of the control-plane stream. Leaving it unset
+	// keeps the plaintext behaviour, which only belongs on a trusted network.
+	TLS tlsconf.Config `koanf:"tls"`
+
+	// ModelDirs are the directories this node keeps model weights in, beyond
+	// WeightsCache, which is always looked in. Storage differs at every installation,
+	// so llmgw observes it rather than owning it: a Lustre mount, an image
+	// exported over the fabric, local NVMe and a copy made by hand all look
+	// identical from here, and placement only needs to know whether the file
+	// is present.
+	ModelDirs []string `koanf:"model_dirs"`
+
+	// EnginesDir is where the agent installs engines, one directory per
+	// build. Defaults to /opt/llmgw/engines.
+	EnginesDir string `koanf:"engines_dir"`
+
+	// WeightsCache is where a remote source is downloaded to. Defaults to the
+	// first entry of ModelDirs, because a node that says where its models live
+	// has already answered this question, and otherwise to
+	// /var/lib/llmgw-weights.
+	//
+	// Fetching is the simple arrangement: every node keeps its own copy. The
+	// exotic ones need no code — weights on a shared filer or an image exported
+	// over the fabric are simply present, and present means nothing is fetched.
+	WeightsCache string `koanf:"weights_cache"`
+
+	Discovery discovery.Options `koanf:"discovery"`
+	Engines   engines.Config    `koanf:"engines"`
+	Superv    supervisor.Config `koanf:"supervisor"`
+
+	Version string
+}
+
+// DefaultConfig returns sensible defaults.
+func DefaultConfig() Config {
+	// No MgmtInterface: normalize finds the one holding the management
+	// address, and falls back to eth0 only when it cannot. Defaulted here, it
+	// was never found — every node said eth0 unless told otherwise, and a node
+	// on ens3 or bond0 had NCCL bootstrap over an interface it does not have.
+	return Config{
+		HeartbeatInterval: 5 * time.Second,
+		ReconnectBase:     time.Second,
+		ReconnectMax:      30 * time.Second,
+		AdoptReadyTimeout: 60 * time.Second,
+		Superv:            supervisor.DefaultConfig(),
+		// NVML gives NVLink topology, so prefer it when this build has it.
+		Discovery: discovery.Options{PreferNVML: true},
+	}
+}
+
+func (c *Config) normalize() error {
+	d := DefaultConfig()
+	if err := c.TLS.Validate(); err != nil {
+		return err
+	}
+	if err := c.normalizeServerURL(); err != nil {
+		return err
+	}
+	// Everything below is found rather than written. A file that has to say
+	// which machine it is on, and at what address, is a file that differs on
+	// every node — and a copy taken from one node to the next registers the
+	// second as the first.
+	host, _ := os.Hostname()
+	if c.NodeID == "" {
+		c.NodeID = host
+	}
+	if c.NodeID == "" {
+		return errors.New("node_id is not set and the hostname could not be read")
+	}
+	if c.Hostname == "" {
+		c.Hostname = host
+	}
+	if c.MgmtAddr == "" {
+		c.MgmtAddr = addrToward(c.Server)
+	}
+	if c.MgmtInterface == "" {
+		c.MgmtInterface = interfaceOf(c.MgmtAddr)
+	}
+	if c.MgmtInterface == "" {
+		c.MgmtInterface = "eth0"
+	}
+	// The directory downloads land in is always one that is looked in. When
+	// they were two settings, a node could fill its disk with fetched weights
+	// that placement never saw — which is what every node here was doing.
+	if c.WeightsCache == "" && len(c.ModelDirs) > 0 {
+		c.WeightsCache = c.ModelDirs[0]
+	}
+	if c.WeightsCache == "" {
+		c.WeightsCache = DefaultWeightsCache
+	}
+	if !slices.Contains(c.ModelDirs, c.WeightsCache) {
+		c.ModelDirs = append(c.ModelDirs, c.WeightsCache)
+	}
+	if c.HeartbeatInterval <= 0 {
+		c.HeartbeatInterval = d.HeartbeatInterval
+	}
+	if c.ReconnectBase <= 0 {
+		c.ReconnectBase = d.ReconnectBase
+	}
+	if c.AdoptReadyTimeout <= 0 {
+		c.AdoptReadyTimeout = d.AdoptReadyTimeout
+	}
+	if c.ReconnectMax < c.ReconnectBase {
+		c.ReconnectMax = max(c.ReconnectBase, d.ReconnectMax)
+	}
+	return nil
+}
+
+// normalizeServerURL makes the configured address a URL connect can use, and
+// refuses the combination that would quietly undo the certificates: a scheme
+// that says plaintext while TLS is configured. Silently connecting anyway would
+// mean a cluster believing it has mutual authentication when it has none.
+func (c *Config) normalizeServerURL() error {
+	// One list, however it was written. Server first when both are given,
+	// because an operator adding a standby to an existing file expects the
+	// one already there to stay preferred.
+	all := make([]string, 0, 1+len(c.Servers))
+	if c.Server != "" {
+		all = append(all, c.Server)
+	}
+	all = append(all, c.Servers...)
+
+	seen := map[string]struct{}{}
+	out := make([]string, 0, len(all))
+	for _, s := range all {
+		fixed, err := normalizeOne(s, c.TLS.Enabled())
+		if err != nil {
+			return err
+		}
+		// A duplicate would be tried twice in a row on failover, which reads
+		// as the rotation not working.
+		if _, dup := seen[fixed]; dup {
+			continue
+		}
+		seen[fixed] = struct{}{}
+		out = append(out, fixed)
+	}
+	if len(out) == 0 {
+		return errors.New("server is required")
+	}
+	c.Servers = out
+	c.Server = out[0]
+	return nil
+}
+
+// DefaultWeightsCache is where fetched weights go when nothing says otherwise.
+const DefaultWeightsCache = "/var/lib/llmgw-weights"
+
+// addrToward is this machine's address on the route to the control plane.
+//
+// The address engines are reached on is whichever one the control plane can
+// reach, and the kernel already knows which that is: a UDP "connection" picks
+// the source address for the route without sending a packet. Asking the
+// interfaces instead would have to guess among docker bridges, VPNs and
+// link-local addresses.
+func addrToward(server string) string {
+	u, err := url.Parse(server)
+	if err != nil || u.Host == "" {
+		return ""
+	}
+	host := u.Hostname()
+	port := u.Port()
+	if port == "" {
+		port = "443"
+	}
+	conn, err := net.Dial("udp", net.JoinHostPort(host, port))
+	if err != nil {
+		return ""
+	}
+	defer conn.Close()
+	if a, ok := conn.LocalAddr().(*net.UDPAddr); ok {
+		return a.IP.String()
+	}
+	return ""
+}
+
+// interfaceOf names the interface that holds an address, or "" when none does.
+func interfaceOf(addr string) string {
+	ip := net.ParseIP(addr)
+	if ip == nil {
+		return ""
+	}
+	ifaces, err := net.Interfaces()
+	if err != nil {
+		return ""
+	}
+	for _, ifc := range ifaces {
+		addrs, err := ifc.Addrs()
+		if err != nil {
+			continue
+		}
+		for _, a := range addrs {
+			if n, ok := a.(*net.IPNet); ok && n.IP.Equal(ip) {
+				return ifc.Name
+			}
+		}
+	}
+	return ""
+}
+
+// normalizeOne applies the scheme rules to a single address.
+func normalizeOne(addr string, secure bool) (string, error) {
+	switch {
+	case strings.HasPrefix(addr, "https://"):
+		if !secure {
+			return "", fmt.Errorf("%s is an https:// address but no tls certificates are configured", addr)
+		}
+	case strings.HasPrefix(addr, "http://"):
+		if secure {
+			return "", fmt.Errorf("tls is configured but %s is an http:// address; use https://", addr)
+		}
+	case strings.Contains(addr, "://"):
+		return "", fmt.Errorf("server %q must be an http:// or https:// address", addr)
+	case secure:
+		return "https://" + addr, nil
+	default:
+		return "http://" + addr, nil
+	}
+	return addr, nil
+}

internal/agent/engines/llamacpp.go

@@ -4,6 +4,7 @@ import (
 	"fmt"
 	"os"
 	"path/filepath"
+	"runtime"
 	"sort"
 	"strconv"
 	"strings"
@@ -21,8 +22,21 @@ type LlamaCppConfig struct {
 	// does not say. llama.cpp defaults to one, which would make every second
 	// request wait behind the first no matter what the router decided.
 	Parallel int `koanf:"parallel"`
+	// Threads is how many CPU threads a shard with no GPU computes on. Zero
+	// takes half this machine's CPUs.
+	//
+	// Not all of them, which is llama.cpp's own default: two busy servers
+	// that each spin on every CPU fight for them, and on an 8-vCPU node
+	// that made 7B generation fall from 10 tokens a second to under 1 each.
+	// At half, one server alone is about a third slower and two together
+	// keep 5.5 each. A node that only ever runs one CPU replica does better
+	// with all of them; say so here.
+	Threads int `koanf:"threads"`
 }
 
+// numCPU is the CPUs this process may run on; a variable for tests.
+var numCPU = runtime.NumCPU
+
 // LlamaCpp builds llama.cpp launch commands.
 //
 // It is the engine that makes a cluster without GPUs useful: llama.cpp runs a
@@ -118,6 +132,13 @@ func (l *LlamaCpp) Commands(s ShardSpec) ([]Command, error) {
 	if len(s.GPUIndices) > 0 && !hasFlag(p.Args, "--n-gpu-layers", "-ngl") {
 		args = append(args, "--n-gpu-layers", "999")
 	}
+	if len(s.GPUIndices) == 0 && !hasFlag(p.Args, "--threads", "-t") {
+		n := l.cfg.Threads
+		if n <= 0 {
+			n = max(numCPU()/2, 1)
+		}
+		args = append(args, "--threads", strconv.Itoa(n))
+	}
 	args = append(args, p.Args...)
 
 	env := baseEnv(s)

internal/agent/engines/llamacpp_test.go

@@ -261,3 +261,36 @@ func TestLlamaCppIsGivenItsKeyOutOfSightOfTheProcessList(t *testing.T) {
 		t.Error("llama-server was not given its key")
 	}
 }
+
+func TestACPUShardLeavesHalfTheMachineForAnother(t *testing.T) {
+	prev := numCPU
+	numCPU = func() int { return 8 }
+	t.Cleanup(func() { numCPU = prev })
+
+	threads := func(l *LlamaCpp, s ShardSpec) (string, bool) {
+		cmds, err := l.Commands(s)
+		if err != nil {
+			t.Fatal(err)
+		}
+		return flagValue(cmds[0].Args, "--threads")
+	}
+	if v, _ := threads(&LlamaCpp{}, llamaSpec(nil)); v != "4" {
+		t.Errorf("--threads %q on 8 CPUs, expected 4", v)
+	}
+	if v, _ := threads(&LlamaCpp{cfg: LlamaCppConfig{Threads: 8}}, llamaSpec(nil)); v != "8" {
+		t.Errorf("--threads %q with 8 configured", v)
+	}
+	if _, ok := threads(&LlamaCpp{}, llamaSpec([]int{0})); ok {
+		t.Error("a GPU shard was given a thread count")
+	}
+	s := llamaSpec(nil)
+	s.Profile.Args = []string{"-t", "2"}
+	cmds, _ := (&LlamaCpp{}).Commands(s)
+	if slices.Contains(cmds[0].Args, "--threads") {
+		t.Error("the profile's own -t was overridden")
+	}
+	numCPU = func() int { return 1 }
+	if v, _ := threads(&LlamaCpp{}, llamaSpec(nil)); v != "1" {
+		t.Errorf("--threads %q on one CPU", v)
+	}
+}

internal/agent/engines/vllm.go

@@ -169,7 +169,10 @@ func (v *VLLM) rayCommand(s ShardSpec) (Command, bool) {
 
 	args := []string{"start", "--block"}
 	if s.Leader() {
-		args = append(args, "--head", "--port", strconv.Itoa(s.LeaderPort))
+		// Ray's head has no authentication: whoever reaches its port can run
+		// code on the node. Its dashboard, and the job API behind it, is the
+		// easiest way in, and nothing here uses it.
+		args = append(args, "--head", "--port", strconv.Itoa(s.LeaderPort), "--include-dashboard=false")
 	} else {
 		args = append(args, "--address", fmt.Sprintf("%s:%d", s.LeaderAddr, s.LeaderPort))
 	}

internal/agent/fabrics.go

@@ -0,0 +1,345 @@
+package agent
+
+import (
+	"context"
+	"fmt"
+	"os/exec"
+	"slices"
+	"strconv"
+	"strings"
+	"time"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
+)
+
+// usableFabrics reports the fabrics this node can actually carry traffic on.
+//
+// Loud about what it drops. A fabric silently missing from a registration looks
+// identical to one never configured, and the operator who typed the adapter
+// name is the only person who can fix it — so the reason is logged here, on the
+// machine where both halves are visible, rather than inferred later from a
+// replica that will not converge.
+func (a *Agent) usableFabrics(inv discovery.Inventory) []inventory.FabricAttachment {
+	out := a.checkFabrics(inv.HCAs, true)
+	a.mu.Lock()
+	a.fabricsSent = fabricsFingerprint(out)
+	a.mu.Unlock()
+	return out
+}
+
+// checkFabrics is the declared fabrics this node can carry traffic on, from
+// the adapters as they are now. Loud logs every problem; the heartbeat checks
+// quietly and speaks only when the answer changes.
+func (a *Agent) checkFabrics(hcas []discovery.HCA, loud bool) []inventory.FabricAttachment {
+	declared := map[string][]string{}
+	for _, f := range a.cfg.Fabrics {
+		declared[f.Fabric] = f.HCAs
+	}
+
+	var out []inventory.FabricAttachment
+	for _, c := range discovery.CheckFabrics(declared, hcas) {
+		if loud {
+			for _, p := range c.Problems {
+				a.log.Error("a declared fabric adapter cannot carry traffic",
+					"fabric", c.Fabric, "problem", p)
+			}
+		}
+		if !c.OK() {
+			if loud {
+				a.log.Error("fabric not advertised; no declared adapter on it can carry traffic",
+					"fabric", c.Fabric,
+					"consequence", "no multi-node replica will be placed here")
+			}
+			continue
+		}
+		out = append(out, inventory.FabricAttachment{
+			Fabric: c.Fabric, HCAs: c.Usable, RateGbps: c.RateGbps,
+		})
+	}
+	return out
+}
+
+// fabricsFingerprint is enough to notice a fabric, an adapter or a rate
+// changing.
+func fabricsFingerprint(fs []inventory.FabricAttachment) string {
+	var b strings.Builder
+	for _, f := range fs {
+		fmt.Fprintf(&b, "%s:%s:%d;", f.Fabric, strings.Join(f.HCAs, ","), f.RateGbps)
+	}
+	return b.String()
+}
+
+// sampleIB re-reads the adapters for the heartbeat: every port's state and
+// counters, and the fabrics again when they no longer match what the control
+// plane was told.
+//
+// Every heartbeat, because it is a few small files per port, and because a
+// port that went down used to be invisible until the agent restarted — the
+// node went on being offered multi-node work that would hang on its first
+// collective.
+func (a *Agent) sampleIB(hb *llmgwv1.Heartbeat) {
+	a.mu.Lock()
+	had := len(a.inv.HCAs) > 0
+	a.mu.Unlock()
+	if !had && len(a.cfg.Fabrics) == 0 {
+		return
+	}
+	hcas, err := discovery.DiscoverIB(a.cfg.Discovery.IBRoot)
+	if err != nil {
+		return
+	}
+	a.attachSM(hcas)
+	for _, h := range hcas {
+		for _, p := range h.Ports {
+			ip := &llmgwv1.IBPort{
+				Hca: h.Name, Port: int32(p.Num), State: p.State, PhysState: p.PhysState,
+				RateGbps: int32(p.RateGbps), LinkLayer: string(p.LinkLayer), Counters: p.Counters,
+				NodeGuid: h.NodeGUID, PortGuid: p.PortGUID, Network: p.Network, GidPrefix: p.GIDPrefix,
+			}
+			a.mu.Lock()
+			if p.SM != nil {
+				ip.SmGuid, ip.SmLid, ip.SmPriority, ip.SmState = p.SM.GUID, int32(p.SM.LID), int32(p.SM.Priority), p.SM.State
+			} else {
+				ip.SmError = a.smErr[smKey(h.Name, p.Num)]
+			}
+			for _, r := range a.sms[smKey(h.Name, p.Num)] {
+				ip.SubnetManagers = append(ip.SubnetManagers, &llmgwv1.SubnetManager{
+					Guid: r.GUID, Lid: int32(r.LID), Priority: int32(r.Priority), State: r.State})
+			}
+			ip.SubnetManagersError = a.smsErr[smKey(h.Name, p.Num)]
+			a.mu.Unlock()
+			hb.IbPorts = append(hb.IbPorts, ip)
+		}
+	}
+	a.mu.Lock()
+	a.inv.HCAs = hcas // a reconnect registers what is true now
+	a.mu.Unlock()
+	if len(a.cfg.Fabrics) == 0 {
+		return
+	}
+	now := a.checkFabrics(hcas, false)
+	fp := fabricsFingerprint(now)
+	a.mu.Lock()
+	changed := fp != a.fabricsSent
+	a.fabricsSent = fp
+	a.mu.Unlock()
+	if !changed {
+		return
+	}
+	// Said once, with the reasons, the moment it happens.
+	a.checkFabrics(hcas, true)
+	a.log.Warn("the fabrics this node can carry traffic on changed", "now", fp)
+	hb.FabricsChanged = true
+	for _, f := range now {
+		hb.Fabrics = append(hb.Fabrics, &llmgwv1.FabricAttachment{
+			Fabric: f.Fabric, Hcas: f.HCAs, RateGbps: int32(f.RateGbps),
+		})
+	}
+}
+
+func smKey(hca string, port int) string { return fmt.Sprintf("%s/%d", hca, port) }
+
+// scanFabric is discovery.ScanFabric, replaced in tests.
+var scanFabric = discovery.ScanFabric
+
+// querySMs is discovery.QuerySMs, replaced in tests.
+var querySMs = func(hca string, port int) ([]discovery.SMRecord, error) {
+	return discovery.QuerySMs(context.Background(), hca, port)
+}
+
+func splitSMKey(k string) (string, int) {
+	i := strings.LastIndexByte(k, '/')
+	port, _ := strconv.Atoi(k[i+1:])
+	return k[:i], port
+}
+
+// querySM is discovery.QuerySM, replaced in tests.
+var querySM = func(hca string, port int) (*discovery.SMInfo, error) {
+	return discovery.QuerySM(context.Background(), hca, port)
+}
+
+// attachSM puts on each active InfiniBand port the subnet manager it sees,
+// asking again once a minute and otherwise reusing the last answer.
+func (a *Agent) attachSM(hcas []discovery.HCA) {
+	a.mu.Lock()
+	stale := time.Since(a.smAsked) > time.Minute || a.sm == nil
+	a.mu.Unlock()
+	if stale {
+		sm, errs := map[string]*discovery.SMInfo{}, map[string]string{}
+		_ = discovery.AttachSM(hcas, func(hca string, port int) (*discovery.SMInfo, error) {
+			got, err := querySM(hca, port)
+			if err != nil {
+				errs[smKey(hca, port)] = err.Error()
+				return nil, err
+			}
+			sm[smKey(hca, port)] = got
+			return got, nil
+		})
+		// Every subnet manager on each fabric, through one port per master:
+		// the same list through every port of one fabric is one query too
+		// many for each.
+		sms, smsErr := map[string][]discovery.SMRecord{}, map[string]string{}
+		asked := map[string]bool{}
+		for k, v := range sm {
+			if asked[v.GUID] {
+				continue
+			}
+			asked[v.GUID] = true
+			hca, port := splitSMKey(k)
+			if recs, err := querySMs(hca, port); err != nil {
+				smsErr[k] = err.Error()
+			} else {
+				sms[k] = recs
+			}
+		}
+		a.mu.Lock()
+		for k, v := range sm {
+			if old := a.sm[k]; old != nil && old.GUID != v.GUID {
+				a.log.Warn("the master subnet manager changed", "port", k, "from", old.GUID, "to", v.GUID)
+			}
+		}
+		a.sm, a.smErr, a.smAsked = sm, errs, time.Now()
+		a.sms, a.smsErr = sms, smsErr
+		a.mu.Unlock()
+		return
+	}
+	a.mu.Lock()
+	defer a.mu.Unlock()
+	for i := range hcas {
+		for j := range hcas[i].Ports {
+			if sm := a.sm[smKey(hcas[i].Name, hcas[i].Ports[j].Num)]; sm != nil && hcas[i].Ports[j].Active() {
+				hcas[i].Ports[j].SM = sm
+			}
+		}
+	}
+}
+
+// ensureDiags installs infiniband-diags on a node with adapters and without
+// it. sminfo is what tells which physical fabric a port is on; without it,
+// two separate fabrics given one name cannot be caught. Installed the way
+// engine dependencies are, from the machine's own package sources.
+func (a *Agent) ensureDiags(ctx context.Context) {
+	a.mu.Lock()
+	hasIB := len(a.inv.HCAs) > 0
+	a.mu.Unlock()
+	if !hasIB || a.prov == nil {
+		return
+	}
+	if _, err := exec.LookPath("sminfo"); err == nil {
+		return
+	}
+	a.log.Info("installing infiniband-diags, to tell which fabric each port is on")
+	pkgs := map[string][]string{"deb": {"infiniband-diags"}, "rpm": {"infiniband-diags"}}
+	if err := a.prov.InstallPackages(ctx, pkgs); err != nil {
+		a.log.Warn("could not install infiniband-diags; fabrics cannot be told apart by their subnet manager",
+			"err", err)
+	}
+}
+
+// FeatureFabricTest is the agent answering FabricTestRequest.
+const FeatureFabricTest = "fabric-test"
+
+// runWriteBW is discovery.RunWriteBW, and haveWriteBW whether it is
+// installed; both replaced in tests.
+var (
+	runWriteBW  = discovery.RunWriteBW
+	haveWriteBW = func() bool { _, err := exec.LookPath("ib_write_bw"); return err == nil }
+)
+
+// fabricTest runs this node's side of a bandwidth test with another.
+func (a *Agent) fabricTest(ctx context.Context, req *llmgwv1.FabricTestRequest) *llmgwv1.FabricTestReply {
+	start := time.Now()
+	reply := &llmgwv1.FabricTestReply{RequestId: req.GetRequestId()}
+	defer func() { reply.TookMs = time.Since(start).Milliseconds() }()
+
+	w := discovery.WriteBW{HCA: req.GetHca(), Port: int(req.GetPort()), GIDIndex: -1,
+		TCPPort: int(req.GetTcpPort()), Seconds: int(req.GetSeconds())}
+	if req.GetRole() == "client" {
+		w.Peer = req.GetPeer()
+	}
+	a.mu.Lock()
+	found := false
+	for _, h := range a.inv.HCAs {
+		for _, p := range h.Ports {
+			if h.Name == w.HCA && p.Num == w.Port {
+				found = true
+				// RoCE needs its v2 GID named, as NCCL does; InfiniBand uses the LID.
+				if p.LinkLayer == discovery.LinkEthernet && p.RoCEv2GID >= 0 {
+					w.GIDIndex = p.RoCEv2GID
+				}
+			}
+		}
+	}
+	a.mu.Unlock()
+	if !found {
+		reply.Error = fmt.Sprintf("this node has no port %s/%d", w.HCA, w.Port)
+		return reply
+	}
+	if !haveWriteBW() && a.prov != nil {
+		a.log.Info("installing perftest, for a bandwidth test with another node")
+		pkgs := map[string][]string{"deb": {"perftest"}, "rpm": {"perftest"}}
+		if err := a.prov.InstallPackages(ctx, pkgs); err != nil {
+			reply.Error = "installing perftest: " + err.Error()
+			return reply
+		}
+	}
+	gbps, out, err := runWriteBW(ctx, w)
+	reply.Gbps = gbps
+	reply.Output = tailLines(out, 12)
+	if err != nil {
+		reply.Error = err.Error()
+	}
+	return reply
+}
+
+func tailLines(s string, n int) string {
+	lines := strings.Split(strings.TrimRight(s, "\n"), "\n")
+	if len(lines) > n {
+		lines = lines[len(lines)-n:]
+	}
+	return strings.Join(lines, "\n")
+}
+
+// discoverGPUFabric and discoverIMEX are replaced in tests.
+var (
+	discoverGPUFabric = discovery.DiscoverGPUFabric
+	discoverIMEX      = discovery.DiscoverIMEX
+)
+
+// sampleNVLink reads, once a minute, which NVLink domain each GPU is joined
+// to and whether IMEX is up. nvidia-smi -q takes a second or two, too long
+// for every heartbeat, and a fabric that was joined an hour ago rarely
+// changes in five seconds — but when a fabric manager restarts it does, for
+// every card at once.
+func (a *Agent) sampleNVLink(hb *llmgwv1.Heartbeat) {
+	a.mu.Lock()
+	gpus := slices.Clone(a.inv.GPUs)
+	due := time.Since(a.nvlAsked) > time.Minute
+	if due {
+		a.nvlAsked = time.Now()
+	}
+	a.mu.Unlock()
+	if len(gpus) == 0 || !due {
+		return
+	}
+	ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
+	defer cancel()
+	fab, err := discoverGPUFabric(ctx, a.cfg.Discovery.NvidiaSMI)
+	if err != nil || len(fab) == 0 {
+		return
+	}
+	discovery.ApplyFabric(gpus, fab)
+	for _, g := range gpus {
+		if g.Fabric == nil {
+			continue
+		}
+		hb.GpuFabric = append(hb.GpuFabric, &llmgwv1.GPUFabricState{
+			Index: int32(g.Index), State: g.Fabric.State, Status: g.Fabric.Status,
+			ClusterUuid: g.Fabric.ClusterUUID, CliqueId: g.Fabric.CliqueID, Bandwidth: g.Fabric.Bandwidth,
+		})
+	}
+	m := discoverIMEX(ctx)
+	hb.Imex = &llmgwv1.IMEX{State: m.State, Channels: int32(m.Channels)}
+}

internal/agent/logs.go

@@ -0,0 +1,34 @@
+package agent
+
+import (
+	"slices"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+)
+
+// logsReply answers a request for one of this node's logs.
+func (a *Agent) logsReply(req *llmgwv1.LogsRequest) *llmgwv1.LogsReply {
+	n := int(req.GetLines())
+	if n <= 0 {
+		n = 200
+	}
+	n = min(n, agentLogLines)
+	shards := a.sup.IDs()
+	slices.Sort(shards)
+	out := &llmgwv1.LogsReply{
+		RequestId: req.GetRequestId(),
+		Sources:   append([]string{"agent"}, shards...),
+	}
+	switch src := req.GetSource(); src {
+	case "", "agent":
+		out.Lines = a.logRing.tail(n)
+	default:
+		sh, err := a.sup.Get(src)
+		if err != nil {
+			out.Error = "no engine " + src + " runs on this node"
+			return out
+		}
+		out.Lines = sh.Logs(n)
+	}
+	return out
+}

internal/agent/provision.go

@@ -0,0 +1,197 @@
+package agent
+
+import (
+	"context"
+	"fmt"
+	"log/slog"
+	"maps"
+	"slices"
+	"strings"
+	"time"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
+)
+
+// engines is what this node can run: what the agent installed and saw run,
+// and what the configuration names and answers. Nothing merely found on the
+// machine — a llama-server on PATH is somebody's, not necessarily one that
+// works, and placing on it is how a replica fails on a node nobody watches.
+func (a *Agent) engines(ctx context.Context) map[string]string {
+	a.engineOnce.Do(func() {
+		vers := a.reg.ProbeConfigured(ctx)
+		a.mu.Lock()
+		a.configuredVers, a.lastEngineProbe = vers, time.Now()
+		a.mu.Unlock()
+		if len(vers) > 0 {
+			a.log.Info("engines named in the configuration",
+				"engines", engines.ProbedNames(vers))
+		}
+	})
+	a.mu.Lock()
+	defer a.mu.Unlock()
+	vers := a.engineReportLocked()
+	// Whatever registration sends is what the control plane now knows, so a
+	// heartbeat repeats it only once it differs.
+	a.enginesSent = enginesFingerprint(vers)
+	return vers
+}
+
+// engineReportLocked merges the two sources. Installed wins: it is the one
+// that was seen to run. Called with mu held.
+func (a *Agent) engineReportLocked() map[string]string {
+	out := maps.Clone(a.configuredVers)
+	if out == nil {
+		out = map[string]string{}
+	}
+	maps.Copy(out, a.installedVers)
+	return out
+}
+
+// provisionEvery is how often installed engines are checked again. A library
+// removed by an unrelated upgrade is found here rather than by the next
+// replica to start.
+const provisionEvery = 10 * time.Minute
+
+// setProvision takes the control plane's list of wanted engines.
+func (a *Agent) setProvision(p *llmgwv1.Provision) {
+	wants := []provision.Want{}
+	for _, e := range p.GetEngines() {
+		wants = append(wants, provision.Want{Engine: e.GetEngine(), Version: e.GetVersion()})
+	}
+	a.mu.Lock()
+	a.provWants = wants
+	a.mu.Unlock()
+	select {
+	case a.provKick <- struct{}{}:
+	default:
+	}
+}
+
+// provisionLoop installs and re-checks the wanted engines, one pass at a time.
+func (a *Agent) provisionLoop(ctx context.Context) {
+	// After a failure the next pass comes sooner, and backs off: a lock held
+	// by a new machine's own updates clears in a minute or two, while ten
+	// minutes' wait left a node idle for no reason.
+	next := provisionEvery
+	retry := time.Minute
+	for {
+		select {
+		case <-ctx.Done():
+			return
+		case <-a.provKick:
+		case <-time.After(next):
+		}
+		a.mu.Lock()
+		wants := slices.Clone(a.provWants)
+		managed := a.provWants != nil
+		a.mu.Unlock()
+		if !managed {
+			continue
+		}
+
+		statuses := a.prov.Converge(ctx, wants)
+		next = provisionEvery
+		for _, st := range statuses {
+			if st.State == provision.Failed {
+				next, retry = retry, min(retry*2, provisionEvery)
+				break
+			}
+		}
+		if next == provisionEvery {
+			retry = time.Minute
+		}
+		installed := map[string]string{}
+		keep := map[string]bool{}
+		for _, st := range statuses {
+			keep[st.Engine] = true
+			if st.State == provision.Ready {
+				a.reg.UseInstalled(st.Engine, st.Binary)
+				installed[st.Engine] = st.Version
+			}
+		}
+		a.mu.Lock()
+		if enginesFingerprint(installed) != enginesFingerprint(a.installedVers) {
+			a.log.Info("engines installed and running", "engines", engines.ProbedNames(installed))
+		}
+		a.installedVers = installed
+		// An engine no longer asked for is no longer reported on.
+		for e := range a.provStatus {
+			if !keep[e] {
+				delete(a.provStatus, e)
+			}
+		}
+		a.mu.Unlock()
+	}
+}
+
+// provisionStatus records a change in one engine's installation.
+func (a *Agent) provisionStatus(st provision.Status) {
+	a.mu.Lock()
+	prev, had := a.provStatus[st.Engine]
+	a.provStatus[st.Engine] = st
+	a.mu.Unlock()
+	if !had || prev.State != st.State || prev.Detail != st.Detail {
+		lvl := slog.LevelInfo
+		if st.State == provision.Failed || st.State == provision.Unsupported {
+			lvl = slog.LevelWarn
+		}
+		a.log.Log(context.Background(), lvl, "engine installation",
+			"engine", st.Engine, "version", st.Version, "variant", st.Variant,
+			"state", st.State, "detail", st.Detail)
+	}
+}
+
+func provisionFingerprint(m map[string]provision.Status) string {
+	var b strings.Builder
+	for _, k := range slices.Sorted(maps.Keys(m)) {
+		st := m[k]
+		fmt.Fprintf(&b, "%s|%s|%s|%s|%s\n", st.Engine, st.Version, st.Variant, st.State, st.Detail)
+	}
+	return b.String()
+}
+
+// engineReprobe is how often the installed engines are looked for again.
+//
+// Minutes, not seconds: an engine is installed by somebody at a keyboard, so a
+// minute's delay in noticing is invisible, while SGLang's probe starts a
+// Python interpreter that imports it.
+const engineReprobe = time.Minute
+
+// reprobeEngines looks for installed engines again when it is time, off the
+// heartbeat goroutine so a slow probe never delays a heartbeat.
+func (a *Agent) reprobeEngines() {
+	a.mu.Lock()
+	due := !a.probingEngines && !a.lastEngineProbe.IsZero() &&
+		time.Since(a.lastEngineProbe) >= engineReprobe
+	if due {
+		a.probingEngines = true
+	}
+	a.mu.Unlock()
+	if !due {
+		return
+	}
+	go func() {
+		ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
+		defer cancel()
+		vers := a.reg.ProbeConfigured(ctx)
+
+		a.mu.Lock()
+		changed := enginesFingerprint(vers) != enginesFingerprint(a.configuredVers)
+		a.configuredVers, a.lastEngineProbe, a.probingEngines = vers, time.Now(), false
+		a.mu.Unlock()
+		if changed {
+			a.log.Info("engines named in the configuration changed",
+				"engines", engines.ProbedNames(vers))
+		}
+	}()
+}
+
+func enginesFingerprint(vers map[string]string) string {
+	var b strings.Builder
+	for _, k := range slices.Sorted(maps.Keys(vers)) {
+		fmt.Fprintf(&b, "%s=%s\n", k, vers[k])
+	}
+	return b.String()
+}

internal/agent/provision/install.go

@@ -523,6 +523,11 @@ func (in *Installer) download(ctx context.Context, repo, tag, asset, dst, want s
 // untarGz unpacks an archive under dir, refusing anything that would land
 // outside it. A release archive is somebody else's file, and a path of
 // "../../etc/cron.d/x" in it would otherwise be written as root.
+//
+// Every write goes through an os.Root, which refuses to leave dir however the
+// path gets there. Checking names alone was not enough: "a -> ." followed by
+// "a/a/b -> ../.." reads as inside and resolves two levels above it, and a
+// file written through that lands wherever the links point.
 func untarGz(archive, dir string) error {
 	f, err := os.Open(archive) //nolint:gosec // inside the install root
 	if err != nil {
@@ -533,6 +538,11 @@ func untarGz(archive, dir string) error {
 	if err != nil {
 		return err
 	}
+	root, err := os.OpenRoot(dir)
+	if err != nil {
+		return err
+	}
+	defer root.Close()
 	tr := tar.NewReader(gz)
 	for {
 		h, err := tr.Next()
@@ -542,22 +552,22 @@ func untarGz(archive, dir string) error {
 		if err != nil {
 			return err
 		}
-		target, err := within(dir, h.Name)
-		if err != nil {
+		name := filepath.Clean(h.Name)
+		if _, err := within(dir, name); err != nil {
 			return err
 		}
 		switch h.Typeflag {
 		case tar.TypeDir:
-			if err := os.MkdirAll(target, 0o755); err != nil {
+			if err := root.MkdirAll(name, 0o755); err != nil {
 				return err
 			}
 		case tar.TypeReg:
-			if err := os.MkdirAll(filepath.Dir(target), 0o755); err != nil {
+			if err := root.MkdirAll(filepath.Dir(name), 0o755); err != nil {
 				return err
 			}
-			out, err := os.OpenFile(target, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, fs.FileMode(h.Mode)&0o755) //nolint:gosec // checked above
+			out, err := root.OpenFile(name, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, fs.FileMode(h.Mode)&0o755)
 			if err != nil {
-				return err
+				return fmt.Errorf("%s: %w", h.Name, err)
 			}
 			if _, err := io.Copy(out, tr); err != nil { //nolint:gosec // a release archive, size bounded by its checksum
 				out.Close()
@@ -567,19 +577,19 @@ func untarGz(archive, dir string) error {
 				return err
 			}
 		case tar.TypeSymlink:
-			// Libraries are versioned through symlinks, and those are
-			// relative. One pointing anywhere else is refused.
-			if filepath.IsAbs(h.Linkname) {
-				return fmt.Errorf("%s links outside the archive", h.Name)
+			// Libraries are versioned through symlinks to a file beside them —
+			// libllama.so -> libllama.so.0 — and that is all a release needs.
+			// A target with a separator in it could point anywhere once other
+			// links are followed, and a binary found through one would be run
+			// as root, so it is refused rather than reasoned about.
+			if h.Linkname == "" || strings.ContainsAny(h.Linkname, `/\`) || h.Linkname == ".." || h.Linkname == "." {
+				return fmt.Errorf("%s links to %q; only a file beside it is allowed", h.Name, h.Linkname)
 			}
-			if _, err := within(dir, filepath.Join(filepath.Dir(h.Name), h.Linkname)); err != nil {
-				return fmt.Errorf("%s links outside the archive", h.Name)
-			}
-			if err := os.MkdirAll(filepath.Dir(target), 0o755); err != nil {
+			if err := root.MkdirAll(filepath.Dir(name), 0o755); err != nil {
 				return err
 			}
-			_ = os.Remove(target)
-			if err := os.Symlink(h.Linkname, target); err != nil {
+			_ = root.Remove(name)
+			if err := root.Symlink(h.Linkname, name); err != nil {
 				return err
 			}
 		default:

internal/agent/provision/install_test.go

@@ -342,3 +342,49 @@ func TestAVersionThatIsAPathIsRefused(t *testing.T) {
 		}
 	}
 }
+
+func TestChainedLinksCannotCarryAFileOutOfTheInstall(t *testing.T) {
+	// Each link reads as inside the directory; together they point above it.
+	var buf bytes.Buffer
+	gz := gzip.NewWriter(&buf)
+	tw := tar.NewWriter(gz)
+	_ = tw.WriteHeader(&tar.Header{Name: "a", Linkname: ".", Typeflag: tar.TypeSymlink})
+	_ = tw.WriteHeader(&tar.Header{Name: "a/a/b", Linkname: "../..", Typeflag: tar.TypeSymlink})
+	_ = tw.WriteHeader(&tar.Header{Name: "a/a/b/escaped", Mode: 0o644, Size: 1, Typeflag: tar.TypeReg})
+	_, _ = tw.Write([]byte("x"))
+	_ = tw.Close()
+	_ = gz.Close()
+
+	parent := t.TempDir()
+	dir := filepath.Join(parent, "install")
+	_ = os.MkdirAll(dir, 0o755)
+	archive := filepath.Join(parent, "evil.tar.gz")
+	_ = os.WriteFile(archive, buf.Bytes(), 0o644)
+
+	if err := untarGz(archive, dir); err == nil {
+		t.Error("an archive of chained links was unpacked")
+	}
+	if _, err := os.Stat(filepath.Join(parent, "escaped")); err == nil {
+		t.Fatal("a file was written outside the install directory")
+	}
+}
+
+func TestALibrarysVersionLinkIsStillUnpacked(t *testing.T) {
+	var buf bytes.Buffer
+	gz := gzip.NewWriter(&buf)
+	tw := tar.NewWriter(gz)
+	_ = tw.WriteHeader(&tar.Header{Name: "lib/libllama.so.0", Mode: 0o755, Size: 1, Typeflag: tar.TypeReg})
+	_, _ = tw.Write([]byte("x"))
+	_ = tw.WriteHeader(&tar.Header{Name: "lib/libllama.so", Linkname: "libllama.so.0", Typeflag: tar.TypeSymlink})
+	_ = tw.Close()
+	_ = gz.Close()
+	dir := t.TempDir()
+	archive := filepath.Join(t.TempDir(), "ok.tar.gz")
+	_ = os.WriteFile(archive, buf.Bytes(), 0o644)
+	if err := untarGz(archive, dir); err != nil {
+		t.Fatal(err)
+	}
+	if target, err := os.Readlink(filepath.Join(dir, "lib", "libllama.so")); err != nil || target != "libllama.so.0" {
+		t.Errorf("link %q %v", target, err)
+	}
+}

internal/agent/supervisor/supervisor_test.go

@@ -243,7 +243,9 @@ func TestWaitReadyTimeoutIncludesLogs(t *testing.T) {
 	if err != nil {
 		t.Fatal(err)
 	}
-	time.Sleep(100 * time.Millisecond) // let the line be captured
+	eventually(t, 3*time.Second, "the line was never captured", func() bool {
+		return strings.Contains(strings.Join(sh.Logs(10), "\n"), "CUDA out of memory")
+	})
 
 	err = s.WaitReady(t.Context(), sh, "http://x", "m", 200*time.Millisecond)
 	if err == nil {
@@ -295,7 +297,10 @@ func TestLogsAreCaptured(t *testing.T) {
 	if err != nil {
 		t.Fatal(err)
 	}
-	time.Sleep(400 * time.Millisecond)
+	eventually(t, 3*time.Second, "not every line arrived", func() bool {
+		l := strings.Join(sh.Logs(10), "\n")
+		return strings.Contains(l, "line1") && strings.Contains(l, "line2") && strings.Contains(l, "line3")
+	})
 
 	logs := strings.Join(sh.Logs(10), "\n")
 	for _, want := range []string{"line1", "line2", "line3"} {
@@ -372,11 +377,11 @@ func TestStateFileRemovedOnExit(t *testing.T) {
 
 	sh, _ := s.Start(t.Context(), "a", engines.ShardSpec{}, shellCommand("exit 0"))
 	waitState(t, sh, Failed, 3*time.Second)
-	time.Sleep(50 * time.Millisecond)
-
-	if _, err := os.Stat(s.statePath("a")); !os.IsNotExist(err) {
-		t.Error("the state file outlived the process; a later scan would find a dead orphan")
-	}
+	eventually(t, 3*time.Second,
+		"the state file outlived the process; a later scan would find a dead orphan", func() bool {
+			_, err := os.Stat(s.statePath("a"))
+			return os.IsNotExist(err)
+		})
 }
 
 // --- state machine and ring buffer ---
@@ -743,7 +748,9 @@ func TestWaitAliveFailsOnEarlyExit(t *testing.T) {
 	if err := s.WaitAlive(t.Context(), sh); err == nil {
 		t.Fatal("a worker that died immediately was reported ready")
 	}
-	if d := time.Since(start); d > 1500*time.Millisecond {
+	// Below AliveGrace (5s), which is what waiting it out would take; the
+	// headroom is for a loaded runner, where 1.5s was sometimes not enough.
+	if d := time.Since(start); d > 4*time.Second {
 		t.Errorf("waited %v for a process that had already exited", d)
 	}
 }
@@ -827,3 +834,17 @@ func TestAWorkersOnlyProcessWaitsForTheLeader(t *testing.T) {
 		t.Errorf("the worker started after %v, before the leader was listening", d)
 	}
 }
+
+// eventually waits for cond, failing with msg if it does not hold within
+// limit. Sleeping a fixed time and then checking measured the machine the
+// test ran on, and a loaded CI runner under the race detector lost that race.
+func eventually(t *testing.T, limit time.Duration, msg string, cond func() bool) {
+	t.Helper()
+	deadline := time.Now().Add(limit)
+	for !cond() {
+		if time.Now().After(deadline) {
+			t.Fatal(msg)
+		}
+		time.Sleep(10 * time.Millisecond)
+	}
+}

internal/agent/upgrade.go

@@ -0,0 +1,58 @@
+package agent
+
+import (
+	"context"
+	"os"
+	"runtime"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/release"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+)
+
+// upgradeExit is the exit status after an upgrade: non-zero, so systemd's
+// Restart=on-failure starts the new binary, and distinct, so the journal says
+// why the agent went.
+const upgradeExit = 75
+
+// exitForUpgrade ends the process; a variable so a test can see it called
+// rather than have its own process end.
+var exitForUpgrade = func() { os.Exit(upgradeExit) }
+
+// upgradeTarget is the binary an upgrade replaces; a variable so a test
+// replaces a file of its own rather than the test binary running it.
+var upgradeTarget = os.Executable
+
+// upgrade replaces this agent's binary with a release and restarts into it.
+//
+// Engines are not touched: an agent restart leaves them running and adopts
+// them again, which is what makes upgrading a node something that can be done
+// while it serves.
+func (a *Agent) upgrade(ctx context.Context, m *llmgwv1.UpgradeAgent, out chan<- *llmgwv1.AgentMessage) {
+	reply := &llmgwv1.UpgradeReply{RequestId: m.GetRequestId(), Version: m.GetVersion()}
+	send := func() {
+		select {
+		case out <- &llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_Upgrade{Upgrade: reply}}:
+		case <-ctx.Done():
+		}
+	}
+	a.log.Info("upgrading", "to", m.GetVersion(), "from", a.cfg.Version)
+	self, err := upgradeTarget()
+	if err == nil {
+		dctx, cancel := context.WithTimeout(ctx, 10*time.Minute)
+		_, err = release.Install(dctx, nil, m.GetReleases(), m.GetVersion(), runtime.GOOS, runtime.GOARCH, self)
+		cancel()
+	}
+	if err != nil {
+		reply.Error = err.Error()
+		a.log.Error("upgrade failed; still running the old version", "to", m.GetVersion(), "err", err)
+		send()
+		return
+	}
+	send()
+	a.log.Info("upgraded; restarting into the new version", "to", m.GetVersion())
+	// A moment for the reply to leave before the stream goes.
+	time.Sleep(500 * time.Millisecond)
+	exitForUpgrade()
+}

internal/agent/upgrade_test.go

@@ -0,0 +1,78 @@
+package agent
+
+import (
+	"crypto/sha256"
+	"encoding/hex"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"runtime"
+	"testing"
+	"time"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/release"
+)
+
+func TestAnAgentUpgradesItselfAndSaysSoBeforeRestarting(t *testing.T) {
+	body := []byte("the new agent")
+	sum := sha256.Sum256(body)
+	asset := release.AssetName("v9.9.9", runtime.GOOS, runtime.GOARCH)
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		switch r.URL.Path {
+		case "/v9.9.9/checksums.txt":
+			_, _ = w.Write([]byte(hex.EncodeToString(sum[:]) + "  " + asset + "\n"))
+		case "/v9.9.9/" + asset:
+			_, _ = w.Write(body)
+		default:
+			http.NotFound(w, r)
+		}
+	}))
+	defer srv.Close()
+
+	bin := filepath.Join(t.TempDir(), "llmgw")
+	_ = os.WriteFile(bin, []byte("the old agent"), 0o755)
+	exited := make(chan struct{}, 1)
+	oldTarget, oldExit := upgradeTarget, exitForUpgrade
+	upgradeTarget = func() (string, error) { return bin, nil }
+	exitForUpgrade = func() { exited <- struct{}{} }
+	t.Cleanup(func() { upgradeTarget, exitForUpgrade = oldTarget, oldExit })
+
+	cfg := Config{Server: "127.0.0.1:7443", NodeID: "n1", WeightsCache: t.TempDir(), EnginesDir: t.TempDir()}
+	cfg.Superv.WorkDir = t.TempDir()
+	a, err := New(cfg, nil, nil)
+	if err != nil {
+		t.Fatal(err)
+	}
+	out := make(chan *llmgwv1.AgentMessage, 1)
+	go a.upgrade(t.Context(), &llmgwv1.UpgradeAgent{RequestId: "u1", Version: "v9.9.9", Releases: srv.URL}, out)
+
+	select {
+	case m := <-out:
+		if r := m.GetUpgrade(); r.GetRequestId() != "u1" || r.GetError() != "" {
+			t.Fatalf("reply %+v", r)
+		}
+	case <-time.After(10 * time.Second):
+		t.Fatal("no reply")
+	}
+	select {
+	case <-exited:
+	case <-time.After(5 * time.Second):
+		t.Fatal("the agent did not restart into the new binary")
+	}
+	if b, _ := os.ReadFile(bin); string(b) != "the new agent" {
+		t.Errorf("binary holds %q", b)
+	}
+
+	// A version with no release answers with why, and does not restart.
+	go a.upgrade(t.Context(), &llmgwv1.UpgradeAgent{RequestId: "u2", Version: "v9.9.8", Releases: srv.URL}, out)
+	if r := (<-out).GetUpgrade(); r.GetError() == "" {
+		t.Error("an upgrade to a release that does not exist reported success")
+	}
+	select {
+	case <-exited:
+		t.Error("a failed upgrade restarted the agent")
+	case <-time.After(time.Second):
+	}
+}

internal/cli/backup.go

@@ -0,0 +1,312 @@
+package cli
+
+import (
+	"archive/tar"
+	"compress/gzip"
+	"context"
+	"errors"
+	"fmt"
+	"io"
+	"io/fs"
+	"net/url"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"slices"
+	"strings"
+	"time"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/version"
+)
+
+// DefaultBackupDir is where backups are written and where the control plane
+// looks for the newest one.
+//
+// Its own directory rather than /var/backups/llmgw, which a PostgreSQL move
+// had already made for its own dumps, owned by postgres and closed to
+// everyone else — so the control plane could not see a backup landing in it.
+const DefaultBackupDir = "/var/backups/llmgw-server"
+
+func newBackupCmd() *cobra.Command {
+	var (
+		configPath string
+		etcDir     string
+		dir        string
+		keep       int
+	)
+	c := &cobra.Command{
+		Use:   "backup",
+		Short: "Write the store and /etc/llmgw to one archive, and keep the last few",
+		Long: "Everything a control plane cannot be rebuilt without: the store — keys,\n" +
+			"people, quotas, usage, the models added at runtime, the secret engine\n" +
+			"keys derive from — and /etc/llmgw, which holds the configuration, the\n" +
+			"store's address and the certificate authority's key.\n\n" +
+			"The archive holds that key, so it is written readable by root only. It\n" +
+			"is on the same disk as what it backs up: copy it elsewhere, or it does\n" +
+			"not survive the loss of this machine.\n\n" +
+			"Run daily by llmgw-backup.timer (`llmgw server backup`). The control plane publishes the time of\n" +
+			"the newest archive as llmgw_backup_last_success_timestamp_seconds, and\n" +
+			"BackupTooOld fires when it is more than a day old.",
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			cfg, err := config.Load(configPath)
+			if err != nil {
+				return err
+			}
+			path, err := writeBackup(cmd.Context(), cfg.Auth.Store, etcDir, dir)
+			if err != nil {
+				return err
+			}
+			if why := unlistable(dir); why != "" {
+				fmt.Fprintf(cmd.ErrOrStderr(), "warning: %s; the control plane will report no backup "+
+					"and BackupTooOld will fire\n", why)
+			}
+			removed, err := pruneBackups(dir, keep)
+			if err != nil {
+				return err
+			}
+			fmt.Fprintf(cmd.OutOrStdout(), "wrote %s", path)
+			if removed > 0 {
+				fmt.Fprintf(cmd.OutOrStdout(), "; removed %d older", removed)
+			}
+			fmt.Fprintln(cmd.OutOrStdout())
+			return nil
+		},
+	}
+	c.Flags().StringVar(&configPath, "config", "/etc/llmgw/server.yaml", "the control plane's configuration")
+	c.Flags().StringVar(&etcDir, "etc", "/etc/llmgw", "configuration directory to include")
+	c.Flags().StringVar(&dir, "dir", DefaultBackupDir, "where archives are written")
+	c.Flags().IntVar(&keep, "keep", 14, "how many archives to keep")
+	return c
+}
+
+const backupPrefix = "llmgw-backup-"
+
+// writeBackup writes one archive and returns its path. Written beside its
+// final name and renamed, so a backup interrupted halfway is never the newest
+// one the control plane reports.
+func writeBackup(ctx context.Context, storeAddr, etcDir, dir string) (string, error) {
+	// Listable by anyone, so the control plane — not root — can see when the
+	// newest archive was written. The archives themselves are root's alone.
+	if err := os.MkdirAll(dir, 0o755); err != nil {
+		return "", err
+	}
+	host, _ := os.Hostname()
+	name := backupPrefix + host + "-" + time.Now().UTC().Format("20060102T150405Z") + ".tar.gz"
+	final := filepath.Join(dir, name)
+	part := final + ".part"
+
+	f, err := os.OpenFile(part, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0o600)
+	if err != nil {
+		return "", err
+	}
+	ok := false
+	defer func() {
+		if !ok {
+			_ = os.Remove(part)
+		}
+	}()
+	gz := gzip.NewWriter(f)
+	tw := tar.NewWriter(gz)
+
+	manifest := fmt.Sprintf("llmgw %s\nhost %s\ntime %s\n", version.Version, host,
+		time.Now().UTC().Format(time.RFC3339))
+	if err := addBytes(tw, "MANIFEST", []byte(manifest)); err != nil {
+		return "", err
+	}
+	if storeAddr != "" {
+		if err := addStore(ctx, tw, storeAddr, dir); err != nil {
+			return "", fmt.Errorf("the store: %w", err)
+		}
+	}
+	if err := addTree(tw, etcDir, "etc/llmgw"); err != nil {
+		return "", fmt.Errorf("%s: %w", etcDir, err)
+	}
+	if err := errors.Join(tw.Close(), gz.Close(), f.Sync(), f.Close()); err != nil {
+		return "", err
+	}
+	if err := os.Rename(part, final); err != nil {
+		return "", err
+	}
+	ok = true
+	return final, nil
+}
+
+// addStore dumps the store into the archive.
+func addStore(ctx context.Context, tw *tar.Writer, addr, scratch string) error {
+	addr, err := store.ResolveAddress(addr)
+	if err != nil {
+		return err
+	}
+	// Inside a directory only its owner can enter. The archive directory is
+	// listable on purpose, and VACUUM INTO creates its file with the default
+	// mode: made there directly, a full copy of the store — password hashes,
+	// the engine-key secret — sat readable by any local user for as long as
+	// the backup took.
+	tmpDir, err := os.MkdirTemp(scratch, ".store-")
+	if err != nil {
+		return err
+	}
+	defer os.RemoveAll(tmpDir)
+	tmpPath := filepath.Join(tmpDir, "store")
+
+	name := "store.sqlite"
+	if store.IsPostgres(addr) {
+		name = "store.pgdump"
+		if err := pgDump(ctx, addr, tmpPath); err != nil {
+			return err
+		}
+	} else {
+		db, err := store.Open(ctx, addr)
+		if err != nil {
+			return err
+		}
+		err = db.SnapshotTo(ctx, tmpPath)
+		_ = db.Close()
+		if err != nil {
+			return err
+		}
+	}
+	return addFile(tw, tmpPath, name)
+}
+
+// pgDump runs pg_dump in its custom format, which pg_restore can restore
+// selectively. The connection is given through the environment, not the
+// command line: a DSN carries the password, and every user on the machine
+// can read another process's arguments.
+func pgDump(ctx context.Context, dsn, out string) error {
+	u, err := url.Parse(dsn)
+	if err != nil {
+		return fmt.Errorf("the store address is not a URL")
+	}
+	env := os.Environ()
+	set := func(k, v string) {
+		if v != "" {
+			env = append(env, k+"="+v)
+		}
+	}
+	set("PGHOST", u.Hostname())
+	set("PGPORT", u.Port())
+	set("PGUSER", u.User.Username())
+	if pw, ok := u.User.Password(); ok {
+		set("PGPASSWORD", pw)
+	}
+	set("PGDATABASE", strings.TrimPrefix(u.Path, "/"))
+	set("PGSSLMODE", u.Query().Get("sslmode"))
+	cmd := exec.CommandContext(ctx, "pg_dump", "--format=custom", "--no-owner", "--file="+out) //nolint:gosec // fixed binary, our own file
+	cmd.Env = env
+	if b, err := cmd.CombinedOutput(); err != nil {
+		return fmt.Errorf("pg_dump: %v: %s", err, strings.TrimSpace(string(b)))
+	}
+	return nil
+}
+
+func addTree(tw *tar.Writer, root, as string) error {
+	return filepath.WalkDir(root, func(p string, d fs.DirEntry, err error) error {
+		if err != nil {
+			return err
+		}
+		rel, _ := filepath.Rel(root, p)
+		name := filepath.ToSlash(filepath.Join(as, rel))
+		info, err := d.Info()
+		if err != nil {
+			return err
+		}
+		switch {
+		case d.IsDir():
+			return tw.WriteHeader(&tar.Header{Name: name + "/", Mode: int64(info.Mode().Perm()),
+				Typeflag: tar.TypeDir, ModTime: info.ModTime()})
+		case info.Mode().IsRegular():
+			return addFile(tw, p, name)
+		}
+		return nil // sockets, links: not configuration
+	})
+}
+
+func addFile(tw *tar.Writer, path, name string) error {
+	f, err := os.Open(path) //nolint:gosec // files we chose to back up
+	if err != nil {
+		return err
+	}
+	defer f.Close()
+	info, err := f.Stat()
+	if err != nil {
+		return err
+	}
+	if err := tw.WriteHeader(&tar.Header{Name: name, Mode: int64(info.Mode().Perm()),
+		Size: info.Size(), ModTime: info.ModTime(), Typeflag: tar.TypeReg}); err != nil {
+		return err
+	}
+	_, err = io.Copy(tw, f)
+	return err
+}
+
+func addBytes(tw *tar.Writer, name string, b []byte) error {
+	if err := tw.WriteHeader(&tar.Header{Name: name, Mode: 0o600, Size: int64(len(b)),
+		ModTime: time.Now(), Typeflag: tar.TypeReg}); err != nil {
+		return err
+	}
+	_, err := tw.Write(b)
+	return err
+}
+
+// pruneBackups keeps the newest keep archives. Only files this command
+// wrote are considered, so nothing else in the directory is ever removed.
+func pruneBackups(dir string, keep int) (int, error) {
+	entries, err := os.ReadDir(dir)
+	if err != nil {
+		return 0, err
+	}
+	var names []string
+	for _, e := range entries {
+		if n := e.Name(); strings.HasPrefix(n, backupPrefix) && strings.HasSuffix(n, ".tar.gz") {
+			names = append(names, n)
+		}
+	}
+	// The name carries a UTC timestamp, so name order is age order.
+	slices.Sort(names)
+	removed := 0
+	for len(names) > max(keep, 1) {
+		if err := os.Remove(filepath.Join(dir, names[0])); err != nil {
+			return removed, err
+		}
+		names = names[1:]
+		removed++
+	}
+	return removed, nil
+}
+
+// NewestBackup is the modification time of the newest complete archive in
+// dir, or zero when there is none.
+func NewestBackup(dir string) time.Time {
+	entries, err := os.ReadDir(dir)
+	if err != nil {
+		return time.Time{}
+	}
+	var newest time.Time
+	for _, e := range entries {
+		if n := e.Name(); !strings.HasPrefix(n, backupPrefix) || !strings.HasSuffix(n, ".tar.gz") {
+			continue
+		}
+		if info, err := e.Info(); err == nil && info.ModTime().After(newest) {
+			newest = info.ModTime()
+		}
+	}
+	return newest
+}
+
+// unlistable says why the control plane, which does not run as root, could
+// not see the archives in dir, or "" when it can.
+func unlistable(dir string) string {
+	info, err := os.Stat(dir)
+	if err != nil {
+		return err.Error()
+	}
+	if info.Mode().Perm()&0o005 != 0o005 {
+		return fmt.Sprintf("%s is %v, so only its owner can list it", dir, info.Mode().Perm())
+	}
+	return ""
+}

internal/cli/backup_test.go

@@ -0,0 +1,141 @@
+package cli
+
+import (
+	"archive/tar"
+	"compress/gzip"
+	"io"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+func TestABackupHoldsTheStoreAndTheConfiguration(t *testing.T) {
+	tmp := t.TempDir()
+	dbPath := filepath.Join(tmp, "store.db")
+	db, err := store.Open(t.Context(), dbPath)
+	if err != nil {
+		t.Fatal(err)
+	}
+	secret, err := db.ClusterSecret(t.Context(), "engine-key")
+	if err != nil {
+		t.Fatal(err)
+	}
+	_ = db.Close()
+
+	etc := filepath.Join(tmp, "etc")
+	_ = os.MkdirAll(filepath.Join(etc, "pki"), 0o700)
+	_ = os.WriteFile(filepath.Join(etc, "server.yaml"), []byte("listen: :8080\n"), 0o644)
+	_ = os.WriteFile(filepath.Join(etc, "pki", "ca-key.pem"), []byte("KEY"), 0o600)
+
+	out := filepath.Join(tmp, "backups")
+	path, err := writeBackup(t.Context(), dbPath, etc, out)
+	if err != nil {
+		t.Fatal(err)
+	}
+	info, _ := os.Stat(path)
+	if info.Mode().Perm() != 0o600 {
+		t.Errorf("the archive holds the CA's key and is %v", info.Mode().Perm())
+	}
+
+	files := map[string][]byte{}
+	f, _ := os.Open(path)
+	defer f.Close()
+	gz, err := gzip.NewReader(f)
+	if err != nil {
+		t.Fatal(err)
+	}
+	tr := tar.NewReader(gz)
+	for {
+		h, err := tr.Next()
+		if err == io.EOF {
+			break
+		}
+		if err != nil {
+			t.Fatal(err)
+		}
+		b, _ := io.ReadAll(tr)
+		files[h.Name] = b
+	}
+	for _, want := range []string{"MANIFEST", "store.sqlite", "etc/llmgw/server.yaml", "etc/llmgw/pki/ca-key.pem"} {
+		if _, ok := files[want]; !ok {
+			t.Errorf("%s is not in the archive", want)
+		}
+	}
+
+	// And the store in it is a working store, with what was in it.
+	restored := filepath.Join(tmp, "restored.db")
+	_ = os.WriteFile(restored, files["store.sqlite"], 0o600)
+	rdb, err := store.Open(t.Context(), restored)
+	if err != nil {
+		t.Fatalf("the backed-up store does not open: %v", err)
+	}
+	defer rdb.Close()
+	if got, _ := rdb.ClusterSecret(t.Context(), "engine-key"); got != secret {
+		t.Error("the restored store lost the engine key secret; every engine would be locked out")
+	}
+}
+
+func TestOldBackupsArePrunedAndTheNewestIsFound(t *testing.T) {
+	dir := t.TempDir()
+	for i, stamp := range []string{"20260101T000000Z", "20260102T000000Z", "20260103T000000Z"} {
+		p := filepath.Join(dir, backupPrefix+"h-"+stamp+".tar.gz")
+		_ = os.WriteFile(p, []byte("x"), 0o600)
+		mt := time.Date(2026, 1, 1+i, 0, 0, 0, 0, time.UTC)
+		_ = os.Chtimes(p, mt, mt)
+	}
+	_ = os.WriteFile(filepath.Join(dir, "someone-elses-file"), []byte("x"), 0o600)
+	_ = os.WriteFile(filepath.Join(dir, backupPrefix+"h-20260104T000000Z.tar.gz.part"), []byte("x"), 0o600)
+
+	removed, err := pruneBackups(dir, 2)
+	if err != nil || removed != 1 {
+		t.Fatalf("removed %d, %v", removed, err)
+	}
+	if _, err := os.Stat(filepath.Join(dir, "someone-elses-file")); err != nil {
+		t.Error("a file the backup did not write was removed")
+	}
+	newest := NewestBackup(dir)
+	if !newest.Equal(time.Date(2026, 1, 3, 0, 0, 0, 0, time.UTC)) {
+		t.Errorf("newest %v; an interrupted .part must not count", newest)
+	}
+	if !NewestBackup(filepath.Join(dir, "none")).IsZero() {
+		t.Error("a missing directory reported a backup")
+	}
+}
+
+func TestABackupDirectoryTheControlPlaneCannotListIsSaid(t *testing.T) {
+	dir := t.TempDir()
+	_ = os.Chmod(dir, 0o700)
+	if unlistable(dir) == "" {
+		t.Error("a directory only its owner can list was taken as visible")
+	}
+	_ = os.Chmod(dir, 0o755)
+	if why := unlistable(dir); why != "" {
+		t.Errorf("a listable directory: %s", why)
+	}
+}
+
+func TestNoScratchCopyOfTheStoreIsLeftBehind(t *testing.T) {
+	// The snapshot is made before it is archived. Made straight into the
+	// listable archive directory, it was readable by anyone for that time.
+	tmp := t.TempDir()
+	dbPath := filepath.Join(tmp, "store.db")
+	db, err := store.Open(t.Context(), dbPath)
+	if err != nil {
+		t.Fatal(err)
+	}
+	_ = db.Close()
+	out := filepath.Join(tmp, "backups")
+	if _, err := writeBackup(t.Context(), dbPath, t.TempDir(), out); err != nil {
+		t.Fatal(err)
+	}
+	entries, _ := os.ReadDir(out)
+	for _, e := range entries {
+		if strings.HasPrefix(e.Name(), ".store") {
+			t.Errorf("a scratch copy of the store was left behind: %s", e.Name())
+		}
+	}
+}

internal/cli/ctl.go

@@ -83,7 +83,10 @@ func (c *ctlClient) do(ctx context.Context, method, path string, body, out any)
 	}
 	defer resp.Body.Close()
 
-	if resp.StatusCode != http.StatusOK {
+	// Any 2xx. The API answers a change it has queued rather than made — a
+	// restart — with 202, and treating that as a failure printed
+	// "Error: 202 Accepted" over a request that had worked.
+	if resp.StatusCode < 200 || resp.StatusCode > 299 {
 		var e struct {
 			Error string `json:"error"`
 		}
@@ -119,6 +122,7 @@ func newCtlCmd() *cobra.Command {
 	cmd.AddCommand(
 		newNodesCmd(client, &asJSON),
 		newLogsCmd(client, &asJSON),
+		newRequestCmd(client, &asJSON),
 		newModelsCmd(client, &asJSON),
 		newReplicasCmd(client, &asJSON),
 		newExplainCmd(client, &asJSON),
@@ -135,6 +139,9 @@ func newCtlCmd() *cobra.Command {
 		newStatusCmd(client, &asJSON),
 		newModelCmd(client),
 		newScaleCmd(client),
+		newRestartCmd(client),
+		newRetryCmd(client),
+		newUpgradeCmd(client),
 		newEngineCheckCmd(),
 		newToolCheckCmd(),
 		newWeightsCmd(client, &asJSON),
@@ -244,6 +251,10 @@ func newNodesCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 // actually served, never what is merely catalogued.
 func modelStatus(m adminapi.ModelView) string {
 	switch {
+	case m.Parked:
+		return "parked"
+	case m.Unplaceable:
+		return "cannot place"
 	case m.Min == 0 && m.Total == 0:
 		return "stopped"
 	case m.Ready >= m.Min && m.Min > 0:
@@ -370,6 +381,9 @@ func newReplicasCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 				if r.Stale {
 					state += " (stale profile)"
 				}
+				if r.Restart != "" {
+					state += " (restart " + r.Restart + ")"
+				}
 				fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n",
 					r.ID, r.Model, state, r.Pool, join(nodes),
 					(time.Duration(r.AgeSec) * time.Second).String(),

internal/cli/ctl_request.go

@@ -0,0 +1,45 @@
+package cli
+
+import (
+	"encoding/json"
+	"fmt"
+	"net/url"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+)
+
+func newRequestCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
+	return &cobra.Command{
+		Use:   "request ID",
+		Short: "What one request went through: replica, queue, first token, tokens",
+		Long: "Every response carries its id in X-Request-Id, and a client may send its\n" +
+			"own. Given one, this says which replica answered, how long admission held\n" +
+			"it, how long the first token took and what it cost — the answer to \"why\n" +
+			"was this slow\". The access log has the same fields under \"request\".",
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			var v adminapi.RequestView
+			if err := client().get(cmd.Context(), "/api/v1/requests/"+url.PathEscape(args[0]), &v); err != nil {
+				return err
+			}
+			if *asJSON {
+				return json.NewEncoder(cmd.OutOrStdout()).Encode(v)
+			}
+			model := v.Model
+			if v.ServedModel != "" && v.ServedModel != v.Model {
+				model += " (served by " + v.ServedModel + ")"
+			}
+			out := cmd.OutOrStdout()
+			fmt.Fprintf(out, "request   %s at %s\n", v.RequestID, v.At.UTC().Format("2006-01-02 15:04:05Z"))
+			fmt.Fprintf(out, "model     %s\n", model)
+			fmt.Fprintf(out, "replica   %s\n", or(v.Replica, "- (none answered)"))
+			fmt.Fprintf(out, "status    %d\n", v.Status)
+			fmt.Fprintf(out, "time      %d ms total: %d queued, %d to the first token\n", v.LatencyMs, v.QueueMs, v.TTFTMs)
+			fmt.Fprintf(out, "tokens    %d in, %d out\n", v.PromptTokens, v.CompletionTokens)
+			fmt.Fprintf(out, "who       key %s, project %s%s\n", v.KeyID, v.ProjectID, prefixed(", user ", v.UserID))
+			return nil
+		},
+	}
+}

internal/cli/ctl_restart.go

@@ -0,0 +1,75 @@
+package cli
+
+import (
+	"encoding/json"
+	"errors"
+	"fmt"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+)
+
+func newRestartCmd(client func() *ctlClient) *cobra.Command {
+	var (
+		models []string
+		all    bool
+	)
+	c := &cobra.Command{
+		Use:   "restart [REPLICA...]",
+		Short: "Replace replicas with new ones, starting each replacement before stopping the old",
+		Long: "How a running engine picks up what only applies at start: an engine key,\n" +
+			"an edited profile, a new engine version.\n\n" +
+			"Each replacement is placed beside the replica it replaces, and the old one\n" +
+			"keeps serving until the new one is ready. Replicas of one model are\n" +
+			"replaced one at a time. Where nothing else has room for a second copy —\n" +
+			"the usual case on a small CPU fleet — the old one is stopped after the\n" +
+			"drain grace period (5 minutes by default) and the new one starts in its\n" +
+			"place; a model with only that replica is unserved while it loads.\n\n" +
+			"  llmgw ctl restart qwen2.5-7b-3f2a91c0\n" +
+			"  llmgw ctl restart --model qwen2.5-7b\n" +
+			"  llmgw ctl restart --all",
+		RunE: func(cmd *cobra.Command, args []string) error {
+			if len(args) == 0 && len(models) == 0 && !all {
+				return errors.New("name replicas, --model, or --all")
+			}
+			body, _ := json.Marshal(adminapi.RestartRequest{Replicas: args, Models: models, All: all})
+			var res adminapi.RestartResult
+			if err := client().post(cmd.Context(), "/api/v1/restart", body, &res); err != nil {
+				return err
+			}
+			fmt.Printf("%d replica(s) queued for restart; watch with: llmgw ctl replicas\n", len(res.Queued))
+			return nil
+		},
+	}
+	c.Flags().StringArrayVar(&models, "model", nil, "every replica of this model; repeatable")
+	c.Flags().BoolVar(&all, "all", false, "every ready replica in the cluster")
+	return c
+}
+
+func newRetryCmd(client func() *ctlClient) *cobra.Command {
+	return &cobra.Command{
+		Use:   "retry MODEL",
+		Short: "Try a parked model again, and clear its backoff",
+		Long: "A model whose replicas fail every time they start is parked after five\n" +
+			"failures in a row: it is not placed again until its definition changes\n" +
+			"or this is run. Run it once the cause — the weights, the engine, the\n" +
+			"profile — has been dealt with; `llmgw ctl status` lists parked models with\n" +
+			"their last error.",
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			var res struct {
+				WasParked bool `json:"was_parked"`
+			}
+			if err := client().post(cmd.Context(), "/api/v1/models/"+args[0]+"/retry", []byte(`{}`), &res); err != nil {
+				return err
+			}
+			if res.WasParked {
+				fmt.Printf("%s unparked; it is placed on the next pass\n", args[0])
+			} else {
+				fmt.Printf("%s was not parked; its backoff is cleared\n", args[0])
+			}
+			return nil
+		},
+	}
+}

internal/cli/ctl_status_test.go

@@ -0,0 +1,24 @@
+package cli
+
+import (
+	"net/http"
+	"net/http/httptest"
+	"testing"
+)
+
+func TestAQueuedChangeIsNotReportedAsAnError(t *testing.T) {
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+		w.WriteHeader(http.StatusAccepted)
+		_, _ = w.Write([]byte(`{"queued":["r1"]}`))
+	}))
+	defer srv.Close()
+	var out struct {
+		Queued []string `json:"queued"`
+	}
+	if err := newCtlClient(srv.URL).post(t.Context(), "/api/v1/restart", []byte(`{}`), &out); err != nil {
+		t.Fatalf("202 read as a failure: %v", err)
+	}
+	if len(out.Queued) != 1 {
+		t.Errorf("the body of a 202 was not read: %+v", out)
+	}
+}

internal/cli/ctl_upgrade.go

@@ -0,0 +1,197 @@
+package cli
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"os"
+	"os/exec"
+	"runtime"
+	"slices"
+	"strings"
+	"time"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/release"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/version"
+)
+
+func newUpgradeCmd(client func() *ctlClient) *cobra.Command {
+	var (
+		nodes     []string
+		skipCP    bool
+		releases  string
+		perNode   time.Duration
+		serverSvc string
+	)
+	c := &cobra.Command{
+		Use:   "upgrade VERSION",
+		Short: "Upgrade this control plane, then every node, one at a time",
+		Long: "Moves the cluster to a release: v0.10.4, or a candidate like v0.10.4-rc.2.\n\n" +
+			"This machine first — run it with sudo on the control plane, since it\n" +
+			"replaces /usr/local/bin/llmgw and restarts llmgw-server. Then each node, in\n" +
+			"turn: its agent fetches the release for its own platform, checks it\n" +
+			"against the release's checksums, swaps it in and restarts. Engines keep\n" +
+			"serving through all of it and are adopted again; no ssh into the nodes is\n" +
+			"needed.\n\n" +
+			"A node counts as done when it is back on the new version, ready, with as\n" +
+			"many ready replicas as before. One that is not within --timeout stops the\n" +
+			"upgrade there, so a bad release reaches one node rather than all of them.\n" +
+			"Nodes already on the version are skipped, so running it again resumes.",
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			v := args[0]
+			if !release.Tag.MatchString(v) {
+				return fmt.Errorf("%q is not a release version (v1.2.3 or v1.2.3-rc.4)", v)
+			}
+			out := cmd.OutOrStdout()
+			ctx := cmd.Context()
+			if !skipCP {
+				if err := upgradeControlPlane(ctx, out, client, v, releases, serverSvc); err != nil {
+					return err
+				}
+			}
+			return upgradeNodes(ctx, out, client, v, nodes, perNode)
+		},
+	}
+	c.Flags().StringSliceVar(&nodes, "nodes", nil, "only these nodes (default: all)")
+	c.Flags().BoolVar(&skipCP, "skip-control-plane", false, "leave this machine as it is and upgrade only nodes")
+	c.Flags().StringVar(&releases, "releases", release.DefaultBase, "where release binaries are fetched from, for this machine")
+	c.Flags().DurationVar(&perNode, "timeout", 3*time.Minute, "how long a node may take to come back")
+	c.Flags().StringVar(&serverSvc, "service", "llmgw-server", "this machine's systemd unit")
+	return c
+}
+
+func upgradeControlPlane(ctx context.Context, out io.Writer, client func() *ctlClient, v, releases, svc string) error {
+	if version.Version == v {
+		fmt.Fprintf(out, "this machine already runs %s\n", v)
+		return nil
+	}
+	if os.Geteuid() != 0 {
+		return errors.New("replacing this machine's binary needs root: run it with sudo, " +
+			"or pass --skip-control-plane to upgrade only the nodes")
+	}
+	self, err := os.Executable()
+	if err != nil {
+		return err
+	}
+	fmt.Fprintf(out, "== this machine: %s -> %s\n", version.Version, v)
+	backup, err := release.Install(ctx, nil, releases, v, runtime.GOOS, runtime.GOARCH, self)
+	if err != nil {
+		return err
+	}
+	fmt.Fprintf(out, "   installed, checksum verified; the old binary is %s\n", backup)
+	var before []adminapi.NodeView
+	_ = client().get(ctx, "/api/v1/nodes", &before)
+	if b, err := exec.CommandContext(ctx, "systemctl", "restart", svc).CombinedOutput(); err != nil { //nolint:gosec // fixed command
+		return fmt.Errorf("restarting %s: %v: %s", svc, err, strings.TrimSpace(string(b)))
+	}
+	fmt.Fprintf(out, "   %s restarted; waiting for its nodes to reconnect\n", svc)
+	// Back when the admin API answers and every node that was ready is ready
+	// again: agents reconnect within seconds of the control plane returning.
+	want := 0
+	for _, n := range before {
+		if n.State == "ready" {
+			want++
+		}
+	}
+	deadline := time.Now().Add(2 * time.Minute)
+	for {
+		var now []adminapi.NodeView
+		if err := client().get(ctx, "/api/v1/nodes", &now); err == nil {
+			ready := 0
+			for _, n := range now {
+				if n.State == "ready" {
+					ready++
+				}
+			}
+			if ready >= want {
+				fmt.Fprintf(out, "   back: %d of %d nodes ready\n", ready, len(now))
+				return nil
+			}
+		}
+		if time.Now().After(deadline) {
+			return fmt.Errorf("%s did not come back with its %d nodes within two minutes; "+
+				"see journalctl -u %s, and %s holds the previous binary", svc, want, svc, backup)
+		}
+		time.Sleep(2 * time.Second)
+	}
+}
+
+func upgradeNodes(ctx context.Context, out io.Writer, client func() *ctlClient, v string, only []string, perNode time.Duration) error {
+	var nodes []adminapi.NodeView
+	if err := client().get(ctx, "/api/v1/nodes", &nodes); err != nil {
+		return err
+	}
+	slices.SortFunc(nodes, func(a, b adminapi.NodeView) int { return strings.Compare(a.ID, b.ID) })
+	done, skipped := 0, 0
+	for _, n := range nodes {
+		if len(only) > 0 && !slices.Contains(only, n.ID) {
+			continue
+		}
+		if n.AgentVersion == v {
+			skipped++
+			continue
+		}
+		start := time.Now()
+		before := readyReplicasOn(ctx, client, n.ID)
+		fmt.Fprintf(out, "== %s: %s -> %s\n", n.ID, n.AgentVersion, v)
+		body, _ := json.Marshal(adminapi.UpgradeRequest{Version: v})
+		if err := client().post(ctx, "/api/v1/nodes/"+n.ID+"/upgrade", body, nil); err != nil {
+			return fmt.Errorf("%s: %w\nstopped here; `llmgw ctl logs %s` says more", n.ID, err, n.ID)
+		}
+		fmt.Fprintf(out, "   new binary in place; waiting for it to come back with its %d replica(s)\n", before)
+		deadline := time.Now().Add(perNode)
+		for {
+			var now []adminapi.NodeView
+			_ = client().get(ctx, "/api/v1/nodes", &now)
+			var cur adminapi.NodeView
+			for _, x := range now {
+				if x.ID == n.ID {
+					cur = x
+				}
+			}
+			after := readyReplicasOn(ctx, client, n.ID)
+			if cur.AgentVersion == v && cur.State == "ready" && after >= before {
+				fmt.Fprintf(out, "   done in %s: ready, replicas %d/%d\n",
+					time.Since(start).Round(time.Second), after, before)
+				done++
+				break
+			}
+			if time.Now().After(deadline) {
+				return fmt.Errorf("%s did not come back within %s (version %q, state %q, replicas %d/%d)\n"+
+					"stopped here, before the next node; `llmgw ctl logs %s` says why, and the node "+
+					"keeps its previous binary beside the new one", n.ID, perNode, cur.AgentVersion,
+					cur.State, after, before, n.ID)
+			}
+			time.Sleep(2 * time.Second)
+		}
+	}
+	fmt.Fprintf(out, "%d node(s) upgraded to %s, %d already on it\n", done, v, skipped)
+	return nil
+}
+
+// readyReplicasOn counts the ready replicas with a shard on a node.
+func readyReplicasOn(ctx context.Context, client func() *ctlClient, node string) int {
+	var reps []adminapi.ReplicaView
+	if err := client().get(ctx, "/api/v1/replicas", &reps); err != nil {
+		return 0
+	}
+	n := 0
+	for _, r := range reps {
+		if r.State != "ready" {
+			continue
+		}
+		for _, s := range r.Shards {
+			if s.Node == node {
+				n++
+				break
+			}
+		}
+	}
+	return n
+}

internal/cli/ctl_upgrade_test.go

@@ -0,0 +1,88 @@
+package cli
+
+import (
+	"bytes"
+	"encoding/json"
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"sync"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+)
+
+// fakeFleet is an admin API whose nodes come back on the version they are told
+// to upgrade to — except those listed in stuck.
+type fakeFleet struct {
+	mu       sync.Mutex
+	versions map[string]string
+	stuck    map[string]bool
+	told     []string
+}
+
+func (f *fakeFleet) server(t *testing.T) *httptest.Server {
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		f.mu.Lock()
+		defer f.mu.Unlock()
+		switch {
+		case r.URL.Path == "/api/v1/nodes":
+			var out []adminapi.NodeView
+			for id, v := range f.versions {
+				out = append(out, adminapi.NodeView{ID: id, State: "ready", AgentVersion: v})
+			}
+			_ = json.NewEncoder(w).Encode(out)
+		case r.URL.Path == "/api/v1/replicas":
+			var out []adminapi.ReplicaView
+			for id := range f.versions {
+				out = append(out, adminapi.ReplicaView{ID: "r-" + id, State: "ready",
+					Shards: []adminapi.ShardView{{Node: id}}})
+			}
+			_ = json.NewEncoder(w).Encode(out)
+		case strings.HasSuffix(r.URL.Path, "/upgrade"):
+			id := strings.TrimSuffix(strings.TrimPrefix(r.URL.Path, "/api/v1/nodes/"), "/upgrade")
+			var req adminapi.UpgradeRequest
+			_ = json.NewDecoder(r.Body).Decode(&req)
+			f.told = append(f.told, id)
+			if !f.stuck[id] {
+				f.versions[id] = req.Version
+			}
+			_, _ = w.Write([]byte(`{}`))
+		default:
+			http.NotFound(w, r)
+		}
+	}))
+	t.Cleanup(srv.Close)
+	return srv
+}
+
+func TestAnUpgradeGoesNodeByNodeAndSkipsTheDone(t *testing.T) {
+	f := &fakeFleet{versions: map[string]string{"a": "v1.0.0", "b": "v1.0.1", "c": "v1.0.0"}}
+	srv := f.server(t)
+	var out bytes.Buffer
+	err := upgradeNodes(t.Context(), &out, func() *ctlClient { return newCtlClient(srv.URL) }, "v1.0.1", nil, 5*time.Second)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if strings.Join(f.told, ",") != "a,c" {
+		t.Errorf("told %v; b was already on the version", f.told)
+	}
+	if !strings.Contains(out.String(), "2 node(s) upgraded to v1.0.1, 1 already on it") {
+		t.Errorf("output:\n%s", out.String())
+	}
+}
+
+func TestAnUpgradeStopsAtTheFirstNodeThatDoesNotComeBack(t *testing.T) {
+	// A bad release reaches one node, not all of them.
+	f := &fakeFleet{versions: map[string]string{"a": "v1.0.0", "b": "v1.0.0"}, stuck: map[string]bool{"a": true}}
+	srv := f.server(t)
+	var out bytes.Buffer
+	err := upgradeNodes(t.Context(), &out, func() *ctlClient { return newCtlClient(srv.URL) }, "v1.0.1", nil, 3*time.Second)
+	if err == nil || !strings.Contains(err.Error(), "a did not come back") {
+		t.Fatalf("err %v", err)
+	}
+	if len(f.told) != 1 {
+		t.Errorf("went on to %v after a node failed to come back", f.told)
+	}
+}

internal/cli/mirror.go

@@ -8,6 +8,7 @@ import (
 	"net/http"
 	"net/netip"
 	"os"
+	"slices"
 	"strings"
 	"time"
 
@@ -232,7 +233,7 @@ func newMirrorRemoveCmd(dir *string) *cobra.Command {
 func newMirrorServeCmd(dir *string) *cobra.Command {
 	var listen, apiListen, base string
 	var tlsc tlsconf.Config
-	var allow []string
+	var allow, clients []string
 	var allowPrivate bool
 
 	c := &cobra.Command{
@@ -259,6 +260,9 @@ func newMirrorServeCmd(dir *string) *cobra.Command {
 			if err := tlsc.Validate(); err != nil {
 				return err
 			}
+			if len(clients) > 0 && !tlsc.Enabled() {
+				return fmt.Errorf("--api-client names a certificate, and needs --tls-ca, --tls-cert and --tls-key")
+			}
 
 			out := cmd.OutOrStdout()
 			log := slog.Default()
@@ -282,7 +286,7 @@ func newMirrorServeCmd(dir *string) *cobra.Command {
 			}
 			admin := &http.Server{
 				Addr:              apiListen,
-				Handler:           apiGate(tlsc.Enabled(), allowed, api.Handler()),
+				Handler:           onlyClients(clients, apiGate(tlsc.Enabled(), allowed, api.Handler())),
 				ReadHeaderTimeout: 10 * time.Second,
 			}
 
@@ -294,6 +298,12 @@ func newMirrorServeCmd(dir *string) *cobra.Command {
 			switch {
 			case tlsc.Enabled():
 				fmt.Fprintf(out, "management API on %s, mutual TLS against %s\n", apiListen, tlsc.CA)
+				if len(clients) == 0 {
+					// The same root signs every node's certificate, so without a
+					// name any node could delete models here.
+					log.Warn("the management API takes any certificate from " + tlsc.CA +
+						", every node's included; name the control plane's with --api-client")
+				}
 			case len(allowed) > 0:
 				fmt.Fprintf(out, "management API on %s, for %s and this machine only\n", apiListen, strings.Join(allow, ", "))
 			default:
@@ -317,6 +327,8 @@ func newMirrorServeCmd(dir *string) *cobra.Command {
 		"the address nodes use to reach this mirror, as it should appear in catalog sources")
 	c.Flags().StringArrayVar(&allow, "api-allow", nil,
 		"an address or network the management API answers, besides this machine; repeatable (default: this machine only)")
+	c.Flags().StringArrayVar(&clients, "api-client", nil,
+		"with mutual TLS, the common name of a certificate the management API answers; repeatable (default: any from --tls-ca)")
 	c.Flags().BoolVar(&allowPrivate, "allow-private-sources", false,
 		"let downloads reach loopback, private and link-local addresses")
 	c.Flags().StringVar(&tlsc.CA, "tls-ca", "", "CA bundle that management clients must present a certificate from")
@@ -452,6 +464,23 @@ func apiGate(mtls bool, allowed []netip.Prefix, next http.Handler) http.Handler
 	return onlyFrom(allowed, next)
 }
 
+// onlyClients answers only certificates with one of these common names, when
+// any are given. The handshake has already checked the certificate against
+// the root; this narrows which of the root's certificates may manage.
+func onlyClients(names []string, next http.Handler) http.Handler {
+	if len(names) == 0 {
+		return next
+	}
+	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		if r.TLS == nil || len(r.TLS.VerifiedChains) == 0 ||
+			!slices.Contains(names, r.TLS.VerifiedChains[0][0].Subject.CommonName) {
+			http.Error(w, "this mirror is managed by its control plane only", http.StatusForbidden)
+			return
+		}
+		next.ServeHTTP(w, r)
+	})
+}
+
 // onlyFrom answers only this machine and the addresses allowed: the management
 // API can delete models and start downloads, and a mirror on its own machine
 // is reached over the network by one control plane and no one else.

internal/cli/mirror_gate_test.go

@@ -1,6 +1,9 @@
 package cli
 
 import (
+	"crypto/tls"
+	"crypto/x509"
+	"crypto/x509/pkix"
 	"net/http"
 	"net/http/httptest"
 	"net/netip"
@@ -37,3 +40,27 @@ func TestTheMirrorAPIAnswersOnlyThisMachineUnlessTold(t *testing.T) {
 		t.Errorf("mTLS without an allowlist refused a client: %d", got)
 	}
 }
+
+func TestOnlyANamedCertificateManagesTheMirror(t *testing.T) {
+	ok := http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {})
+	with := func(cn string) *http.Request {
+		r := httptest.NewRequest("GET", "/", nil)
+		if cn != "" {
+			r.TLS = &tls.ConnectionState{VerifiedChains: [][]*x509.Certificate{{{Subject: pkix.Name{CommonName: cn}}}}}
+		}
+		return r
+	}
+	h := onlyClients([]string{"control-plane"}, ok)
+	for cn, want := range map[string]int{"control-plane": 200, "node01": 403, "": 403} {
+		w := httptest.NewRecorder()
+		h.ServeHTTP(w, with(cn))
+		if w.Code != want {
+			t.Errorf("%q: %d, expected %d", cn, w.Code, want)
+		}
+	}
+	w := httptest.NewRecorder()
+	onlyClients(nil, ok).ServeHTTP(w, with("node01"))
+	if w.Code != 200 {
+		t.Error("no names given narrowed it anyway")
+	}
+}

internal/cli/model.go

@@ -3,6 +3,7 @@ package cli
 import (
 	"encoding/json"
 	"fmt"
+	"os"
 	"os/user"
 	"time"
 
@@ -28,7 +29,8 @@ func newModelCmd(client func() *ctlClient) *cobra.Command {
 		Use:   "model",
 		Short: "Add or remove models while the cluster is running",
 	}
-	c.AddCommand(newModelAddCmd(client), newModelRemoveCmd(client), newModelPlanCmd(client))
+	c.AddCommand(newModelAddCmd(client), newModelRemoveCmd(client), newModelPlanCmd(client),
+		newModelServeCmd(client))
 	return c
 }
 
@@ -126,6 +128,10 @@ func newModelRemoveCmd(client func() *ctlClient) *cobra.Command {
 
 // actor names who made a change, for the audit trail on a stored model.
 func actor() string {
+	// Under sudo, the person who ran it rather than root.
+	if u := os.Getenv("SUDO_USER"); u != "" && os.Geteuid() == 0 {
+		return u
+	}
 	if u, err := user.Current(); err == nil && u.Username != "" {
 		return u.Username
 	}

internal/cli/model_serve.go

@@ -0,0 +1,80 @@
+package cli
+
+import (
+	"encoding/json"
+	"errors"
+	"fmt"
+	"time"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
+)
+
+func newModelServeCmd(client func() *ctlClient) *cobra.Command {
+	var (
+		req      modelserve.Request
+		onDemand bool
+	)
+	c := &cobra.Command{
+		Use:   "serve OWNER/REPO",
+		Short: "Take a model from the registry to a replica that answers, in one command",
+		Long: "Picks the GGUF file (--quant, Q4_K_M by default, or --file), has the mirror\n" +
+			"fetch it and checks what arrived against the registry's checksum, checks\n" +
+			"that the nodes' llama.cpp can load it and sizes its memory from its own\n" +
+			"header, adds it to the catalog, waits for a replica, and asks it to say\n" +
+			"hello. Each step is shown as it happens; the first that fails stops it.\n\n" +
+			"  llmgw ctl model serve bartowski/Qwen2.5-3B-Instruct-GGUF\n" +
+			"  llmgw ctl model serve Qwen/Qwen2.5-7B-Instruct-GGUF --quant Q5_K_M --name qwen-7b\n" +
+			"  llmgw ctl model serve owner/big-GGUF --on-demand",
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			req.Repo = args[0]
+			if onDemand {
+				req.Replicas = 0
+			}
+			body, _ := json.Marshal(req)
+			var job modelserve.Job
+			if err := client().post(cmd.Context(), "/api/v1/models/serve", body, &job); err != nil {
+				return err
+			}
+			out := cmd.OutOrStdout()
+			fmt.Fprintf(out, "job %s\n", job.ID)
+			shown := map[string]string{}
+			for {
+				for _, s := range job.Steps {
+					line := s.State + " " + s.Detail
+					if shown[s.Name] == line {
+						continue
+					}
+					shown[s.Name] = line
+					mark := map[string]string{modelserve.Running: "..", modelserve.Done: "ok",
+						modelserve.Failed: "!!"}[s.State]
+					fmt.Fprintf(out, "%-3s %-11s %s\n", mark, s.Name, s.Detail)
+				}
+				switch job.State {
+				case modelserve.Done:
+					fmt.Fprintf(out, "\n%s is served.\n", job.Model)
+					return nil
+				case modelserve.Failed:
+					return errors.New(job.Error)
+				}
+				time.Sleep(2 * time.Second)
+				if err := client().get(cmd.Context(), "/api/v1/jobs/"+job.ID, &job); err != nil {
+					return err
+				}
+			}
+		},
+	}
+	f := c.Flags()
+	f.StringVar(&req.Quant, "quant", "", "the quantisation to serve (default Q4_K_M)")
+	f.StringVar(&req.File, "file", "", "the file to serve, instead of choosing by --quant")
+	f.StringVar(&req.Name, "name", "", "the catalog name (default: from the repository)")
+	f.StringSliceVar(&req.Aliases, "alias", nil, "other names clients may use; repeatable")
+	f.StringVar(&req.Description, "description", "", "what it is for, shown to clients in /v1/models")
+	f.IntVar(&req.Replicas, "replicas", 1, "how many replicas to run")
+	f.BoolVar(&onDemand, "on-demand", false, "run none until first asked for")
+	f.IntVar(&req.CtxPool, "ctx-pool", 16384, "llama.cpp's --ctx-size, shared by its parallel slots")
+	f.IntVar(&req.ContextLength, "context-length", 4096, "what one request may use")
+	return c
+}

internal/cli/server.go

@@ -5,8 +5,10 @@ import (
 	"crypto/tls"
 	"errors"
 	"fmt"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/sourcecheck"
+	"github.com/prometheus/client_golang/prometheus"
 	"log/slog"
 	"net"
 	"net/http"
@@ -33,9 +35,11 @@ import (
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/obs"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/promq"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/router"
 	storepkg "gitea.mynodes.xyz/ilker/llmgw/internal/store"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/version"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/web"
 )
@@ -87,6 +91,21 @@ func newServerCmd() *cobra.Command {
 					"or give --endpoint for a router-only run")
 			}
 
+			// Before the router, so its tracer is the configured one.
+			shutdownTracing, err := obs.SetupTracing(cmd.Context(), obs.TracingOptions{
+				Endpoint: cfg.Tracing.Endpoint, SampleRatio: cfg.Tracing.SampleRatio,
+				Service: "llmgw-server", Version: version.Version,
+			})
+			if err != nil {
+				return err
+			}
+			defer func() {
+				// Bounded: a collector that is gone must not hold up a shutdown.
+				sctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+				defer cancel()
+				_ = shutdownTracing(sctx)
+			}()
+
 			rt, err := router.New(cfg, log)
 			if err != nil {
 				return err
@@ -131,6 +150,28 @@ func newServerCmd() *cobra.Command {
 				// On the same /metrics as the traffic figures: correlating
 				// "the cluster was full" with "requests were queueing" should
 				// not need two systems.
+				// When the newest backup was written, so its absence alerts
+				// rather than being found on the day it is needed.
+				backupDir := cfg.BackupDir
+				if backupDir == "" {
+					backupDir = DefaultBackupDir
+				}
+				if err := rt.RegisterMetrics(prometheus.NewGaugeFunc(prometheus.GaugeOpts{
+					Name: "llmgw_backup_last_success_timestamp_seconds",
+					Help: "When the newest backup archive was written, as a Unix time; 0 when there is none.",
+				}, func() float64 {
+					if t := NewestBackup(backupDir); !t.IsZero() {
+						return float64(t.Unix())
+					}
+					return 0
+				})); err != nil {
+					return err
+				}
+				if err := rt.RegisterMetrics(tlsconf.ExpiryCollector(map[string]string{
+					"agent-server": cfg.AgentTLS.Cert, "agent-ca": cfg.AgentTLS.CA,
+				})); err != nil {
+					return err
+				}
 				if err := rt.RegisterMetrics(fleetmetrics.New(ctl)); err != nil {
 					return err
 				}
@@ -147,8 +188,17 @@ func newServerCmd() *cobra.Command {
 			if err != nil {
 				return err
 			}
+			// Kept past this block: serving a model from the registry needs
+			// both the admin API and the mirror, and the mirror is set up
+			// below.
+			var (
+				adminAPI   *adminapi.API
+				catalogSvc *modelsvc.Service
+			)
 			if admin != nil {
 				api := adminapi.New(admin)
+				adminAPI = api
+				api.SetReleases(cfg.Releases)
 				// Catalog changes need somewhere to persist, so this is wired
 				// only when there is a store. Without one, the endpoints say so
 				// rather than accepting a model that vanishes on restart.
@@ -165,6 +215,7 @@ func newServerCmd() *cobra.Command {
 						admin.ReplaceCatalog(ctx, built)
 					}
 					api.SetModels(svc)
+					catalogSvc = svc
 					// And the mirror page, so a model that has just been
 					// fetched can be served without writing YAML by hand.
 					rt.SetCatalogAdmin(svc)
@@ -234,7 +285,16 @@ func newServerCmd() *cobra.Command {
 				// Browsing the registry is what turns "paste a resolve URL"
 				// into "find the model". It needs a way out to the internet
 				// from this machine, which the mirror's own fetching does not.
-				rt.SetRegistry(hf.New(cfg.Mirror.Registry, cfg.Mirror.Timeout))
+				registry := hf.New(cfg.Mirror.Registry, cfg.Mirror.Timeout)
+				rt.SetRegistry(registry)
+				// One command from a registry repository to a replica that
+				// answers: `llmgw ctl model serve`.
+				if adminAPI != nil && catalogSvc != nil {
+					adminAPI.SetServe(modelserve.New(modelserve.Deps{
+						Registry: registry, Mirror: mc, Catalog: catalogSvc, Fleet: admin,
+						LlamaVersion: cfg.WantedEngines()["llamacpp"], Log: log,
+					}))
+				}
 				log.Info("model mirror configured",
 					"api", cfg.Mirror.API, "mtls", cfg.Mirror.TLS.Enabled())
 			}
@@ -361,6 +421,7 @@ func newServerCmd() *cobra.Command {
 	f.StringArrayVar(&endpoints, "endpoint", nil, "static endpoint: id=URL=model[,model] (repeatable)")
 
 	cmd.AddCommand(newServerInitCmd())
+	cmd.AddCommand(newBackupCmd())
 	return cmd
 }
 
@@ -521,6 +582,19 @@ func (r *dispatcherRef) FabricTest(ctx context.Context, nodeID string, req *llmg
 	return ft.FabricTest(ctx, nodeID, req)
 }
 
+// Upgrade forwards a node upgrade, as Logs does.
+func (r *dispatcherRef) Upgrade(ctx context.Context, nodeID, version, releases string) error {
+	d, err := r.get()
+	if err != nil {
+		return err
+	}
+	u, ok := d.(control.Upgrader)
+	if !ok {
+		return errors.New("this dispatcher cannot upgrade nodes")
+	}
+	return u.Upgrade(ctx, nodeID, version, releases)
+}
+
 func (r *dispatcherRef) StartShard(ctx context.Context, cmd reconciler.ShardCommand) error {
 	d, err := r.get()
 	if err != nil {

internal/cli/status.go

@@ -94,6 +94,14 @@ func printSummary(cmd *cobra.Command, s adminapi.Summary) {
 	fmt.Fprintf(out, "\nMODELS    %d of %d have the replicas they asked for\n",
 		s.Models.Satisfied, s.Models.Total)
 
+	if len(s.Parked) > 0 {
+		fmt.Fprintln(out, "\nPARKED — failed every time they started; not retried until fixed")
+		for _, p := range s.Parked {
+			fmt.Fprintf(out, "  %s (%d failures): %s\n", p.Model, p.Failures, p.Reason)
+		}
+		fmt.Fprintln(out, "\n  Once fixed: llmgw ctl retry <model>")
+	}
+
 	if len(s.Unplaceable) > 0 {
 		fmt.Fprintln(out, "\nNOT PLACED")
 		for _, m := range s.Unplaceable {

internal/cli/tui_init.go

@@ -389,17 +389,22 @@ func (in *serverInit) runInitTUI(ctx context.Context, out io.Writer) error {
 		return fmt.Errorf("stopped; run init again to finish what was started")
 	}
 
-	// The whole run is kept, root's only, since it holds the password. On a
-	// failure it is printed too, for the lines the screen had no room for;
-	// otherwise the screen's last frame says it, and only what must outlive
-	// the screen is printed again: the password, and what to do next.
+	// The whole run is kept, root's only, with the password taken out: it is
+	// shown once, on the screen, and a file on disk would outlive that. On a
+	// failure the run is printed too, for the lines the screen had no room
+	// for; otherwise the screen's last frame says it, and only what must
+	// outlive the screen is printed again: the password, and what to do next.
 	logMu.Lock()
 	defer logMu.Unlock()
 	whole := strings.Join(log, "\n") + "\n"
+	kept := whole
+	if fm.password != "" {
+		kept = strings.ReplaceAll(kept, fm.password, "[password shown once, not kept]")
+	}
 	saved := ""
 	if in.stateDir != "" {
 		p := filepath.Join(in.stateDir, "init.log")
-		if err := os.WriteFile(p, []byte(whole), 0o600); err == nil {
+		if err := os.WriteFile(p, []byte(kept), 0o600); err == nil {
 			saved = p
 		}
 	}

internal/config/config.go

@@ -67,6 +67,10 @@ type Server struct {
 
 	// Monitoring is where the dashboard reads its history from.
 	Monitoring Monitoring `koanf:"monitoring"`
+	// Tracing sends a trace of each inference request to an OpenTelemetry
+	// collector. Off unless an endpoint is set here or in
+	// OTEL_EXPORTER_OTLP_ENDPOINT.
+	Tracing Tracing `koanf:"tracing"`
 
 	// Mirror points the control plane at a model mirror it can administer.
 	// Empty means there is none, and /admin/mirror stays unmounted.
@@ -81,6 +85,16 @@ type Server struct {
 	// one leaves that engine to be installed by hand.
 	Engines map[string]string `koanf:"engines"`
 
+	// BackupDir is where `llmgw server backup` writes, and where the newest
+	// archive's age is read for llmgw_backup_last_success_timestamp_seconds.
+	// Empty means /var/backups/llmgw-server.
+	BackupDir string `koanf:"backup_dir"`
+
+	// Releases is where `llmgw ctl upgrade` fetches release binaries from,
+	// for this machine and — through their agents — every node. Empty means
+	// the project's own releases.
+	Releases string `koanf:"releases"`
+
 	Router  Router          `koanf:"router"`
 	Control Control         `koanf:"control"`
 	Models  []catalog.Model `koanf:"models"`
@@ -110,6 +124,21 @@ type Monitoring struct {
 	Timeout time.Duration `koanf:"timeout"`
 }
 
+// Tracing is where request traces go, over OTLP/HTTP.
+//
+// One span per inference request, with admission, scheduling and each attempt
+// at an engine beneath it, and the trace carried on to the engine in a
+// traceparent header. The request's X-Request-Id is on the span, so a trace
+// and `llmgw ctl request` name the same request.
+type Tracing struct {
+	// Endpoint is the collector, for example http://otel.internal:4318.
+	// Empty falls back to OTEL_EXPORTER_OTLP_ENDPOINT, and to no tracing.
+	Endpoint string `koanf:"endpoint"`
+	// SampleRatio is the share of requests traced, 0 to 1, when the caller
+	// did not decide it with a sampled traceparent of its own. Zero takes 1.
+	SampleRatio float64 `koanf:"sample_ratio"`
+}
+
 // Mirror is a model mirror this control plane administers over its API.
 //
 // The mirror is a separate machine with a large disk, not part of the gateway,

internal/control/adminapi/actor_test.go

@@ -0,0 +1,24 @@
+package adminapi
+
+import (
+	"net/http/httptest"
+	"strings"
+	"testing"
+)
+
+func TestAClaimedActorIsOneShortName(t *testing.T) {
+	for claimed, want := range map[string]string{
+		"ilker":                    "ilker",
+		"":                         "unknown",
+		"ilker\nlevel=ERROR msg=x": "ilkerlevelERRORmsgx",
+		strings.Repeat("a", 500):   strings.Repeat("a", 64),
+		"ops@example.org":          "ops@example.org",
+		"\x1b[31m":                 "31m",
+	} {
+		r := httptest.NewRequest("POST", "/", nil)
+		r.Header.Set("X-Llmgw-Actor", claimed)
+		if got := actorOf(r); got != want {
+			t.Errorf("%q: %q, expected %q", claimed, got, want)
+		}
+	}
+}

internal/control/adminapi/api.go

@@ -10,11 +10,15 @@ import (
 	"context"
 	"encoding/json"
 	"fmt"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/release"
 	"io"
 	"net/http"
 	"slices"
+	"strconv"
 	"strings"
 	"time"
+	"unicode"
 
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/bench"
@@ -30,6 +34,11 @@ import (
 
 // API serves the management endpoints.
 type API struct {
+	// releases is where release binaries are fetched from, for upgrades.
+	releases string
+	// serve runs registry-to-replica jobs; nil without a mirror and a store.
+	serve *modelserve.Server
+
 	models   Models
 	capacity Capacity
 	usage    Usage
@@ -74,9 +83,24 @@ func (a *API) Register(mux *http.ServeMux) {
 	mux.HandleFunc("POST /api/v1/weights/collect", a.collectWeights)
 	mux.HandleFunc("GET /api/v1/explain-placement", a.explainPlacement)
 	mux.HandleFunc("POST /api/v1/nodes/{id}/state", a.setNodeState)
+	mux.HandleFunc("POST /api/v1/nodes/{id}/upgrade", a.upgradeNode)
 	mux.HandleFunc("DELETE /api/v1/nodes/{id}", a.removeNode)
 	mux.HandleFunc("POST /api/v1/fabric-test", a.fabricTest)
 	mux.HandleFunc("GET /api/v1/nodes/{id}/logs", a.nodeLogs)
+	mux.HandleFunc("POST /api/v1/restart", a.restart)
+	mux.HandleFunc("POST /api/v1/models/serve", a.serveModel)
+	mux.HandleFunc("GET /api/v1/jobs/{id}", a.job)
+	mux.HandleFunc("GET /api/v1/requests/{id}", a.request)
+	mux.HandleFunc("POST /api/v1/models/{name}/retry", a.retryModel)
+	mux.HandleFunc("GET /api/v1/models/{name}/headroom", func(w http.ResponseWriter, r *http.Request) {
+		limit, _ := strconv.Atoi(r.URL.Query().Get("limit"))
+		v, ok := a.Headroom(r.PathValue("name"), limit)
+		if !ok {
+			http.Error(w, "no such model", http.StatusNotFound)
+			return
+		}
+		writeJSON(w, http.StatusOK, v)
+	})
 }
 
 // --- responses ---
@@ -232,6 +256,18 @@ type ModelView struct {
 	Max         int      `json:"max_replicas"`
 	Ready       int      `json:"ready_replicas"`
 	Total       int      `json:"total_replicas"`
+	// Desired is how many it should have now: Min, or for an on-demand
+	// model one while it is being asked for and none otherwise.
+	Desired int `json:"desired_replicas"`
+	// Failures counts replicas that failed to start in a row.
+	Failures int `json:"failures,omitempty"`
+	// Parked is set once Failures reached the limit; the model is not retried
+	// until its definition changes or `ctl retry` is run. Reason is the last
+	// error.
+	Parked bool   `json:"parked,omitempty"`
+	Reason string `json:"reason,omitempty"`
+	// Unplaceable is set when it is short and the last attempt found no node.
+	Unplaceable bool     `json:"unplaceable,omitempty"`
 	Fallbacks   []string `json:"fallbacks,omitempty"`
 	Profiles    []string `json:"profiles"`
 }
@@ -243,8 +279,11 @@ type ReplicaView struct {
 	// and anyone else holding the admin listener who needs to reach an engine
 	// directly. The client-facing pages never render it.
 	EngineKey string `json:"engine_key,omitempty"`
-	Model     string `json:"model"`
-	Profile   string `json:"profile"`
+	// Restart is "queued" or "replacing" while an operator's restart of this
+	// replica is under way.
+	Restart string `json:"restart,omitempty"`
+	Model   string `json:"model"`
+	Profile string `json:"profile"`
 	// Stale is true when the profile has been edited since this replica was
 	// placed, so it is running a definition the catalog no longer holds.
 	//
@@ -307,6 +346,22 @@ type ExplainView struct {
 	Rejections []RejectionView `json:"rejections,omitempty"`
 }
 
+// HeadroomView is how many more replicas of a model fit now, and where.
+type HeadroomView struct {
+	Model string `json:"model"`
+	// Current is the replicas holding capacity: starting or ready.
+	Current int `json:"current"`
+	// More is how many more placement would find room for now.
+	More int `json:"more"`
+	// Nodes is where each of those would go.
+	Nodes [][]string `json:"nodes,omitempty"`
+	// Limited is set when More stopped at the limit asked rather than at the
+	// cluster's edge: there may be room for more.
+	Limited bool `json:"limited,omitempty"`
+	// Blocked says why one more does not fit, a line per node considered.
+	Blocked []string `json:"blocked,omitempty"`
+}
+
 // RejectionView is one rejected candidate.
 type RejectionView struct {
 	Profile string `json:"profile"`
@@ -438,13 +493,20 @@ func (a *API) Models() []ModelView {
 		}
 	}
 
+	health := map[string]reconciler.ModelHealth{}
+	for _, h := range a.ctl.Reconciler().ModelHealth() {
+		health[h.Model] = h
+	}
 	out := make([]ModelView, 0)
 	for _, m := range a.ctl.Catalog().Models() {
+		h := health[m.Name]
 		v := ModelView{
 			Name: m.Name, Description: m.Description, Aliases: m.Aliases, Source: m.Source,
 			Min: m.Scaling.Min, Max: m.Scaling.Max,
 			Ready: ready[m.Name], Total: total[m.Name],
 			Fallbacks: m.Fallbacks,
+			Desired:   h.Desired, Failures: h.Failures,
+			Parked: h.Parked, Reason: h.Reason, Unplaceable: h.Unplaceable,
 		}
 		for _, p := range m.Profiles {
 			v.Profiles = append(v.Profiles, p.Name)
@@ -476,6 +538,7 @@ func (a *API) Replicas(model string) []ReplicaView {
 			State: rep.State.String(), Pool: rep.Pool, Fabric: rep.Fabric,
 			Endpoint: rep.Endpoint, Err: rep.Err,
 			EngineKey: a.ctl.EngineKey(rep.ID),
+			Restart:   a.ctl.Reconciler().RestartState(rep.ID),
 			Stale:     a.profileChanged(rep.Model, rep.Profile, rep.ProfileFP),
 			AgeSec:    int64(time.Since(rep.CreatedAt).Seconds()),
 			IdleSec:   -1,
@@ -649,7 +712,7 @@ func (a *API) putCapacity(w http.ResponseWriter, r *http.Request) {
 	}
 	if err := a.capacity.PutCapacity(r.Context(), store.StoredCapacity{
 		Model: c.Model, Profile: c.Profile, Measurement: body,
-		MeasuredAt: c.MeasuredAt, MeasuredBy: r.Header.Get("X-Llmgw-Actor"),
+		MeasuredAt: c.MeasuredAt, MeasuredBy: actorOf(r),
 	}); err != nil {
 		writeError(w, http.StatusInternalServerError, err.Error())
 		return
@@ -700,7 +763,7 @@ func (a *API) addModel(w http.ResponseWriter, r *http.Request) {
 		writeError(w, http.StatusBadRequest, "the model definition is not valid JSON: "+err.Error())
 		return
 	}
-	if err := a.models.Add(r.Context(), m, r.Header.Get("X-Llmgw-Actor")); err != nil {
+	if err := a.models.Add(r.Context(), m, actorOf(r)); err != nil {
 		// Every failure here is the definition being wrong, which is the
 		// caller's to fix — a collision, a missing fallback, a profile that
 		// cannot be placed. None of them are server faults.
@@ -749,7 +812,7 @@ func (a *API) scaleModel(w http.ResponseWriter, r *http.Request) {
 	name := r.PathValue("name")
 	m, err := a.models.Scale(r.Context(), name,
 		catalog.Scaling{Min: req.Min, Max: req.Max, IdleTimeout: req.IdleTimeout},
-		r.Header.Get("X-Llmgw-Actor"))
+		actorOf(r))
 	if err != nil {
 		// Like addModel: every refusal here is the request being wrong — a
 		// model declared in configuration, one that was never added, a max
@@ -799,6 +862,20 @@ func (a *API) explainPlacement(w http.ResponseWriter, r *http.Request) {
 // It does not check that the model exists; the handler above does that so it
 // can answer 404, while the dashboard only ever asks about models it just read
 // out of the catalog.
+// Headroom tries placing up to limit more replicas of a model, on a copy of
+// the inventory, and says how many fit. A limit of zero or less takes 16.
+func (a *API) Headroom(model string, limit int) (HeadroomView, bool) {
+	if limit <= 0 || limit > 64 {
+		limit = 16
+	}
+	h, ok := a.ctl.Reconciler().Headroom(model, limit)
+	if !ok {
+		return HeadroomView{}, false
+	}
+	return HeadroomView{Model: model, Current: h.Current, More: h.More, Nodes: h.Nodes,
+		Limited: h.Limited, Blocked: h.Blocked}, true
+}
+
 func (a *API) Explain(model string) ExplainView {
 	v := ExplainView{Model: model}
 	for _, rep := range a.ctl.Reconciler().Replicas() {
@@ -1044,3 +1121,189 @@ func (a *API) Topology(id string) (TopologyView, bool) {
 func (a *API) NVLink() nvlink.Report {
 	return nvlink.Analyze(a.ctl.Inventory().Nodes())
 }
+
+// RestartRequest names what to restart: replicas by id, every replica of some
+// models, or — All — every ready replica in the cluster.
+type RestartRequest struct {
+	Replicas []string `json:"replicas,omitempty"`
+	Models   []string `json:"models,omitempty"`
+	All      bool     `json:"all,omitempty"`
+}
+
+// RestartResult lists the replicas queued.
+type RestartResult struct {
+	Queued []string `json:"queued"`
+}
+
+func (a *API) restart(w http.ResponseWriter, r *http.Request) {
+	var req RestartRequest
+	if err := json.NewDecoder(io.LimitReader(r.Body, 1<<16)).Decode(&req); err != nil {
+		writeError(w, http.StatusBadRequest, "the restart request is not valid JSON: "+err.Error())
+		return
+	}
+	ids := slices.Clone(req.Replicas)
+	for _, rep := range a.ctl.Reconciler().ReadyReplicas() {
+		if req.All || slices.Contains(req.Models, rep.Model) {
+			ids = append(ids, rep.ID)
+		}
+	}
+	if len(ids) == 0 {
+		writeError(w, http.StatusBadRequest, "nothing ready matches; name replicas, models, or all")
+		return
+	}
+	if err := a.ctl.Reconciler().Restart(ids...); err != nil {
+		writeError(w, http.StatusNotFound, err.Error())
+		return
+	}
+	writeJSON(w, http.StatusAccepted, RestartResult{Queued: ids})
+}
+
+func (a *API) retryModel(w http.ResponseWriter, r *http.Request) {
+	name := r.PathValue("name")
+	if _, ok := a.ctl.Catalog().Resolve(name); !ok {
+		writeError(w, http.StatusNotFound, "no model called "+name)
+		return
+	}
+	was := a.ctl.Reconciler().Retry(name)
+	writeJSON(w, http.StatusOK, map[string]bool{"was_parked": was})
+}
+
+// requestFinder looks a request up by the id it was given. Asked of the usage
+// source by type assertion, so one without it answers 501 rather than every
+// source having to grow the method.
+type requestFinder interface {
+	UsageByRequest(ctx context.Context, requestID string) (store.Usage, bool, error)
+}
+
+// RequestView is what one request went through.
+type RequestView struct {
+	RequestID        string    `json:"request_id"`
+	At               time.Time `json:"at"`
+	KeyID            string    `json:"key_id"`
+	UserID           string    `json:"user_id,omitempty"`
+	ProjectID        string    `json:"project_id"`
+	Model            string    `json:"model"`
+	ServedModel      string    `json:"served_model"`
+	Replica          string    `json:"replica,omitempty"`
+	Status           int       `json:"status"`
+	LatencyMs        int64     `json:"latency_ms"`
+	QueueMs          int64     `json:"queue_ms"`
+	TTFTMs           int64     `json:"ttft_ms"`
+	PromptTokens     int64     `json:"prompt_tokens"`
+	CompletionTokens int64     `json:"completion_tokens"`
+	CostMicros       int64     `json:"cost_micros"`
+}
+
+func (a *API) request(w http.ResponseWriter, r *http.Request) {
+	f, ok := a.usage.(requestFinder)
+	if !ok {
+		writeError(w, http.StatusNotImplemented, "this control plane keeps no usage records")
+		return
+	}
+	u, found, err := f.UsageByRequest(r.Context(), r.PathValue("id"))
+	switch {
+	case err != nil:
+		writeError(w, http.StatusInternalServerError, err.Error())
+		return
+	case !found:
+		writeError(w, http.StatusNotFound, "no request with that id; records are written a few seconds after a request ends")
+		return
+	}
+	writeJSON(w, http.StatusOK, RequestView{
+		RequestID: u.RequestID, At: u.At, KeyID: u.KeyID, UserID: u.UserID, ProjectID: u.ProjectID,
+		Model: u.Model, ServedModel: u.ServedModel, Replica: u.Replica, Status: u.Status,
+		LatencyMs: u.LatencyMs, QueueMs: u.QueueMs, TTFTMs: u.TTFTMs,
+		PromptTokens: u.PromptTokens, CompletionTokens: u.CompletionTokens, CostMicros: u.CostMicros,
+	})
+}
+
+// SetReleases sets where upgrades fetch release binaries from.
+func (a *API) SetReleases(base string) { a.releases = base }
+
+// UpgradeRequest names the release a node should run.
+type UpgradeRequest struct {
+	Version string `json:"version"`
+}
+
+func (a *API) upgradeNode(w http.ResponseWriter, r *http.Request) {
+	var req UpgradeRequest
+	if err := json.NewDecoder(io.LimitReader(r.Body, 1<<12)).Decode(&req); err != nil {
+		writeError(w, http.StatusBadRequest, "the upgrade request is not valid JSON: "+err.Error())
+		return
+	}
+	if !release.Tag.MatchString(req.Version) {
+		writeError(w, http.StatusBadRequest, req.Version+" is not a release version")
+		return
+	}
+	u := a.ctl.Upgrader()
+	if u == nil {
+		writeError(w, http.StatusNotImplemented, "this control plane cannot upgrade nodes")
+		return
+	}
+	base := a.releases
+	if base == "" {
+		base = release.DefaultBase
+	}
+	if err := u.Upgrade(r.Context(), r.PathValue("id"), req.Version, base); err != nil {
+		writeError(w, http.StatusBadGateway, err.Error())
+		return
+	}
+	writeJSON(w, http.StatusOK, map[string]string{"node": r.PathValue("id"), "version": req.Version})
+}
+
+// SetServe supplies what serves a model from the registry.
+func (a *API) SetServe(s *modelserve.Server) { a.serve = s }
+
+func (a *API) serveModel(w http.ResponseWriter, r *http.Request) {
+	if a.serve == nil {
+		writeError(w, http.StatusNotImplemented,
+			"serving from the registry needs a mirror and a store, and this control plane lacks one")
+		return
+	}
+	var req modelserve.Request
+	if err := json.NewDecoder(io.LimitReader(r.Body, 1<<16)).Decode(&req); err != nil {
+		writeError(w, http.StatusBadRequest, "the request is not valid JSON: "+err.Error())
+		return
+	}
+	req.By = actorOf(r)
+	job, err := a.serve.Start(req)
+	if err != nil {
+		writeError(w, http.StatusBadRequest, err.Error())
+		return
+	}
+	writeJSON(w, http.StatusAccepted, job)
+}
+
+func (a *API) job(w http.ResponseWriter, r *http.Request) {
+	if a.serve == nil {
+		writeError(w, http.StatusNotFound, "no such job")
+		return
+	}
+	job, ok := a.serve.Get(r.PathValue("id"))
+	if !ok {
+		writeError(w, http.StatusNotFound, "no such job; jobs are kept only until the control plane restarts")
+		return
+	}
+	writeJSON(w, http.StatusOK, job)
+}
+
+// actorOf is who a change is recorded as made by: what the caller says, since
+// the admin API answers only this machine and trusts whoever reaches it. Kept
+// to a short printable name, so a claim cannot pass for more than one — no
+// line breaks to forge a second log line, no length to bury the rest.
+func actorOf(r *http.Request) string {
+	claimed := r.Header.Get("X-Llmgw-Actor")
+	var b strings.Builder
+	for _, c := range claimed {
+		if b.Len() >= 64 {
+			break
+		}
+		if unicode.IsLetter(c) || unicode.IsDigit(c) || strings.ContainsRune("._-@", c) {
+			b.WriteRune(c)
+		}
+	}
+	if b.Len() == 0 {
+		return "unknown"
+	}
+	return b.String()
+}

internal/control/adminapi/summary.go

@@ -25,6 +25,9 @@ type Summary struct {
 	// Unplaceable names models the scheduler currently cannot satisfy. This is
 	// the line that turns "something is wrong" into a place to look.
 	Unplaceable []string `json:"unplaceable,omitempty"`
+	// Parked are models that failed every time they started and are no
+	// longer retried, with the last error.
+	Parked []ParkedModel `json:"parked,omitempty"`
 	// IdleReplicas are serving replicas nothing has asked for in a while, which
 	// on a fleet with more models than GPUs is the first place to find room.
 	IdleReplicas []IdleReplica `json:"idle_replicas,omitempty"`
@@ -201,17 +204,30 @@ func (a *API) Summary() Summary {
 
 	for _, m := range a.Models() {
 		s.Models.Total++
-		if m.Ready >= m.Min {
+		if m.Parked {
+			s.Parked = append(s.Parked, ParkedModel{Model: m.Name, Failures: m.Failures, Reason: m.Reason})
+		}
+		// Against what it should have now, not its minimum: an on-demand
+		// model's minimum is 0, so one that was asked for and fitted nowhere
+		// used to count as satisfied.
+		if m.Ready >= m.Desired && !m.Parked {
 			s.Models.Satisfied++
 			continue
 		}
 		s.Models.Short++
 		// Short of replicas and unable to place more is the actionable case;
 		// short and still starting is just a cluster doing its job.
-		if m.Total <= m.Ready {
+		if m.Unplaceable || (m.Total <= m.Ready && !m.Parked) {
 			s.Unplaceable = append(s.Unplaceable, m.Name)
 		}
 	}
 	sort.Strings(s.Unplaceable)
 	return s
 }
+
+// ParkedModel is a model no longer retried, and why.
+type ParkedModel struct {
+	Model    string `json:"model"`
+	Failures int    `json:"failures"`
+	Reason   string `json:"reason"`
+}

internal/control/agentsvc/heartbeat_test.go

@@ -264,3 +264,22 @@ func TestAGPUFallingOutOfItsDomainTakesTheNodeOut(t *testing.T) {
 		t.Errorf("imex = %+v", n.IMEX)
 	}
 }
+
+func TestAnUpgradeWaitsForTheAgentsAnswer(t *testing.T) {
+	s := newTestService()
+	c := &conn{nodeID: "n1", send: make(chan *llmgwv1.ControlMessage, 4), closed: make(chan struct{})}
+	s.conns["n1"] = c
+	done := make(chan error, 1)
+	go func() { done <- s.Upgrade(t.Context(), "n1", "v1.2.3", "https://r") }()
+
+	req := (<-c.send).GetUpgrade()
+	if req.GetVersion() != "v1.2.3" || req.GetReleases() != "https://r" {
+		t.Fatalf("sent %v", req)
+	}
+	s.handle("n1", &llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_Upgrade{Upgrade: &llmgwv1.UpgradeReply{
+		RequestId: req.GetRequestId(), Version: "v1.2.3", Error: "checksum mismatch",
+	}}})
+	if err := <-done; err == nil || err.Error() != "checksum mismatch" {
+		t.Errorf("the agent's refusal was not passed on: %v", err)
+	}
+}

internal/control/agentsvc/service.go

@@ -60,6 +60,8 @@ type Service struct {
 	logMu    sync.Mutex
 	logWaits map[string]chan *llmgwv1.LogsReply
 	logSeq   uint64
+	// upgradeWaits are the upgrade requests awaiting a reply; guarded by logMu.
+	upgradeWaits map[string]chan *llmgwv1.UpgradeReply
 	// scanWaits are the fabric scans awaiting a reply, by request id.
 	scanWaits map[string]chan *llmgwv1.FabricScanReply
 	testWaits map[string]chan *llmgwv1.FabricTestReply
@@ -309,6 +311,15 @@ func (s *Service) handle(nodeID string, msg *llmgwv1.AgentMessage) {
 			ch <- m.FabricScan
 		}
 
+	case *llmgwv1.AgentMessage_Upgrade:
+		s.logMu.Lock()
+		ch, ok := s.upgradeWaits[m.Upgrade.GetRequestId()]
+		delete(s.upgradeWaits, m.Upgrade.GetRequestId())
+		s.logMu.Unlock()
+		if ok {
+			ch <- m.Upgrade
+		}
+
 	case *llmgwv1.AgentMessage_Logs:
 		s.logMu.Lock()
 		ch, ok := s.logWaits[m.Logs.GetRequestId()]
@@ -842,3 +853,44 @@ func installable(reg *llmgwv1.Register) []string {
 // The admin API finds the bandwidth test by asserting this; a signature that
 // drifted would make it quietly nil.
 var _ fabrictest.Asker = (*Service)(nil)
+
+// Upgrade tells a node's agent to replace its binary with a release and
+// restart, and returns once the agent says the new binary is in place — or
+// why it is not. The agent then exits and comes back on the new version; the
+// caller watches for that in the inventory.
+func (s *Service) Upgrade(ctx context.Context, nodeID, version, releases string) error {
+	s.logMu.Lock()
+	if s.upgradeWaits == nil {
+		s.upgradeWaits = map[string]chan *llmgwv1.UpgradeReply{}
+	}
+	s.logSeq++
+	id := fmt.Sprintf("upgrade-%d", s.logSeq)
+	ch := make(chan *llmgwv1.UpgradeReply, 1)
+	s.upgradeWaits[id] = ch
+	s.logMu.Unlock()
+	defer func() {
+		s.logMu.Lock()
+		delete(s.upgradeWaits, id)
+		s.logMu.Unlock()
+	}()
+
+	if err := s.send(nodeID, &llmgwv1.ControlMessage{Msg: &llmgwv1.ControlMessage_Upgrade{
+		Upgrade: &llmgwv1.UpgradeAgent{RequestId: id, Version: version, Releases: releases},
+	}}); err != nil {
+		return err
+	}
+	s.log.Info("told a node to upgrade", "node", nodeID, "version", version)
+	// A download of some tens of megabytes on a slow link, and no longer: an
+	// agent too old to know this message never answers.
+	ctx, cancel := context.WithTimeout(ctx, 5*time.Minute)
+	defer cancel()
+	select {
+	case r := <-ch:
+		if r.GetError() != "" {
+			return errors.New(r.GetError())
+		}
+		return nil
+	case <-ctx.Done():
+		return fmt.Errorf("node %s did not answer the upgrade; an agent older than v0.10.4 cannot upgrade itself", nodeID)
+	}
+}

internal/control/control.go

@@ -134,6 +134,16 @@ func (c *Controller) collectUsage() {
 		return
 	}
 	c.rec.RecordUsage(c.usage.LastUsed())
+	// And what the replicas are doing, when the source can say: what adds
+	// and removes replicas between a model's min and max.
+	if l, ok := c.usage.(LoadSource); ok {
+		c.rec.RecordLoad(l.ModelLoad())
+	}
+}
+
+// LoadSource reports each model's replicas' load, keyed by catalog name.
+type LoadSource interface {
+	ModelLoad() map[string]reconciler.Load
 }
 
 // ReplaceCatalog puts a new catalog in force, without a restart.
@@ -171,6 +181,9 @@ func (c *Controller) Nodes() []inventory.Node        { return c.inv.Nodes() }
 func (c *Controller) FreeGPUs(nodeID string) []int   { return c.inv.FreeGPUs(nodeID) }
 func (c *Controller) Replicas() []reconciler.Replica { return c.rec.Replicas() }
 
+// ModelHealth is how each model in the catalog stands.
+func (c *Controller) ModelHealth() []reconciler.ModelHealth { return c.rec.ModelHealth() }
+
 // Reconciler exposes the reconciliation loop, for status reports and ctl.
 func (c *Controller) Reconciler() *reconciler.Reconciler { return c.rec }
 
@@ -208,6 +221,17 @@ func (c *Controller) WeightRemover() WeightRemover {
 	return r
 }
 
+// Upgrader can tell a node's agent to upgrade itself.
+type Upgrader interface {
+	Upgrade(ctx context.Context, nodeID, version, releases string) error
+}
+
+// Upgrader returns the dispatcher's upgrade capability, or nil.
+func (c *Controller) Upgrader() Upgrader {
+	u, _ := c.disp.(Upgrader)
+	return u
+}
+
 // LogReader can ask a node for the recent lines of one of its logs.
 type LogReader interface {
 	Logs(ctx context.Context, nodeID, source string, lines int) (out, sources []string, err error)

internal/control/fleetmetrics/fleetmetrics.go

@@ -39,9 +39,25 @@ type Source interface {
 // derived from state that already exists, so computing them when asked is both
 // simpler and fresher than maintaining a parallel copy that can drift. Nothing
 // runs between scrapes.
+// HealthSource adds what the reconciler knows about each model and where
+// engine installs stand, and whether this control plane is the one placing
+// work. Asked for by type assertion, so a source without it — a test's —
+// publishes what it did before.
+type HealthSource interface {
+	ModelHealth() []reconciler.ModelHealth
+	IsLeader() bool
+}
+
 type Collector struct {
 	src Source
 
+	modelDesired     *prometheus.Desc
+	modelFailures    *prometheus.Desc
+	modelParked      *prometheus.Desc
+	modelUnplaceable *prometheus.Desc
+	engineInstall    *prometheus.Desc
+	leader           *prometheus.Desc
+
 	nodeUp       *prometheus.Desc
 	nodeGPUs     *prometheus.Desc
 	nodeGPUsFree *prometheus.Desc
@@ -137,7 +153,26 @@ func New(src Source) *Collector {
 		replicaState: prometheus.NewDesc("llmgw_replicas",
 			"Replicas by model and state.", []string{"model", "state"}, nil),
 		replicaReady: prometheus.NewDesc("llmgw_model_replicas_ready",
-			"Replicas of a model that are serving.", []string{"model"}, nil),
+			"Replicas of a model that are serving. Published, as 0 when none are, for every "+
+				"model that should be running; a stopped model, or an on-demand one nobody is "+
+				"asking for, publishes nothing.", []string{"model"}, nil),
+		modelDesired: prometheus.NewDesc("llmgw_model_replicas_desired",
+			"Replicas a model should have now: its minimum, or for an on-demand model 1 while "+
+				"it is being asked for.", []string{"model"}, nil),
+		modelFailures: prometheus.NewDesc("llmgw_model_start_failures",
+			"Replicas of a model that failed to start in a row. Resets when one comes up.",
+			[]string{"model"}, nil),
+		modelParked: prometheus.NewDesc("llmgw_model_parked",
+			"1 when a model failed so often in a row that it is no longer retried.",
+			[]string{"model"}, nil),
+		modelUnplaceable: prometheus.NewDesc("llmgw_model_unplaceable",
+			"1 when a model is short of replicas and the last attempt found no node to place one on.",
+			[]string{"model"}, nil),
+		engineInstall: prometheus.NewDesc("llmgw_node_engine_install",
+			"Where installing an engine stands on a node: 1 for the current state.",
+			[]string{"node", "engine", "version", "state"}, nil),
+		leader: prometheus.NewDesc("llmgw_control_plane_leader",
+			"1 when this control plane is the one placing work, 0 when it is a standby.", nil, nil),
 		replicaIdle: prometheus.NewDesc("llmgw_replica_idle_seconds",
 			"Seconds since a replica was last sent a request. Absent when nothing has been reported, "+
 				"which is not the same as idle.", []string{"model", "replica"}, nil),
@@ -201,6 +236,8 @@ func (c *Collector) Describe(ch chan<- *prometheus.Desc) {
 		c.gpuMemoryTotal, c.gpuMemoryUsed, c.gpuUtil, c.gpuTemp, c.gpuHealthy,
 		c.gpuNVLink, c.gpuECC,
 		c.replicaState, c.replicaReady, c.replicaIdle,
+		c.modelDesired, c.modelFailures, c.modelParked, c.modelUnplaceable,
+		c.engineInstall, c.leader,
 	} {
 		ch <- d
 	}
@@ -387,8 +424,43 @@ func (c *Collector) Collect(ch chan<- prometheus.Metric) {
 		ch <- prometheus.MustNewConstMetric(c.replicaState, prometheus.GaugeValue,
 			float64(v), k[0], k[1])
 	}
-	for model, v := range ready {
-		ch <- prometheus.MustNewConstMetric(c.replicaReady, prometheus.GaugeValue,
-			float64(v), model)
+	hs, ok := c.src.(HealthSource)
+	if !ok {
+		for model, v := range ready {
+			ch <- prometheus.MustNewConstMetric(c.replicaReady, prometheus.GaugeValue,
+				float64(v), model)
+		}
+		return
+	}
+	for _, n := range c.src.Nodes() {
+		for _, p := range n.Provision {
+			ch <- prometheus.MustNewConstMetric(c.engineInstall, prometheus.GaugeValue, 1,
+				n.ID, p.Engine, p.Version, p.State)
+		}
+	}
+	lead := 0.0
+	if hs.IsLeader() {
+		lead = 1
+	}
+	ch <- prometheus.MustNewConstMetric(c.leader, prometheus.GaugeValue, lead)
+	for _, h := range hs.ModelHealth() {
+		// Ready is published at zero for what should be running — the one
+		// value ModelHasNothingServing exists to catch, and which, emitted
+		// only for replicas that were Ready, it never saw.
+		if h.Desired > 0 || ready[h.Model] > 0 {
+			ch <- prometheus.MustNewConstMetric(c.replicaReady, prometheus.GaugeValue,
+				float64(ready[h.Model]), h.Model)
+		}
+		ch <- prometheus.MustNewConstMetric(c.modelDesired, prometheus.GaugeValue, float64(h.Desired), h.Model)
+		ch <- prometheus.MustNewConstMetric(c.modelFailures, prometheus.GaugeValue, float64(h.Failures), h.Model)
+		ch <- prometheus.MustNewConstMetric(c.modelParked, prometheus.GaugeValue, b2f(h.Parked), h.Model)
+		ch <- prometheus.MustNewConstMetric(c.modelUnplaceable, prometheus.GaugeValue, b2f(h.Unplaceable), h.Model)
 	}
 }
+
+func b2f(b bool) float64 {
+	if b {
+		return 1
+	}
+	return 0
+}

internal/control/fleetmetrics/fleetmetrics_test.go

@@ -196,3 +196,35 @@ func TestEveryIBPortIsReportedWithItsCounters(t *testing.T) {
 		}
 	}
 }
+
+type healthSource struct {
+	fakeSource
+	health []reconciler.ModelHealth
+}
+
+func (h healthSource) ModelHealth() []reconciler.ModelHealth { return h.health }
+func (h healthSource) IsLeader() bool                        { return true }
+
+func TestAModelThatShouldRunPublishesZeroReplicasReady(t *testing.T) {
+	// ModelHasNothingServing alerts on llmgw_model_replicas_ready == 0, and
+	// the series was only ever published while a replica was ready — so the
+	// alert could not fire for the case it was written for.
+	src := healthSource{health: []reconciler.ModelHealth{
+		{Model: "wanted", Desired: 1, Failures: 5, Parked: true, Reason: "boom"},
+		{Model: "stopped", Desired: 0},
+	}}
+	out := gather(t, src)
+	for _, want := range []string{
+		"llmgw_model_replicas_ready model=wanted -> 0",
+		"llmgw_model_parked model=wanted -> 1",
+		"llmgw_model_start_failures model=wanted -> 5",
+		"llmgw_control_plane_leader -> 1",
+	} {
+		if !strings.Contains(out, want) {
+			t.Errorf("missing %s", want)
+		}
+	}
+	if strings.Contains(out, "llmgw_model_replicas_ready model=stopped") {
+		t.Error("a stopped model publishes a ready count, so the alert would fire for it")
+	}
+}

internal/control/inventory/inventory.go

@@ -392,6 +392,23 @@ func NewStore() *Store {
 	return &Store{nodes: map[string]*Node{}, allocs: map[string][]Allocation{}}
 }
 
+// Clone is a copy to try decisions on without making them: allocations made
+// on it are its own. Nodes are copied by value; the slices inside them are
+// shared, and a clone must not write through them.
+func (s *Store) Clone() *Store {
+	s.mu.RLock()
+	defer s.mu.RUnlock()
+	c := NewStore()
+	for id, n := range s.nodes {
+		cp := *n
+		c.nodes[id] = &cp
+	}
+	for id, list := range s.allocs {
+		c.allocs[id] = slices.Clone(list)
+	}
+	return c
+}
+
 // Upsert records a node's registration, preserving operator-set state such as
 // a cordon: an agent reconnecting must not silently undo an operator's decision.
 // RecordSMChange keeps a port's change of master subnet manager, the last

internal/control/modelserve/modelserve.go

@@ -0,0 +1,552 @@
+// Package modelserve takes a model from a registry repository to a replica
+// answering requests, as one job.
+//
+// Adding a model was four commands and a hand-written YAML file: plan it,
+// fetch it into the mirror, write its catalog entry with a memory figure
+// somebody guessed, scale it — and then find out whether it loads. This runs
+// those steps in the control plane, where the mirror, the registry, the
+// catalog and the fleet already are, and reports each one as it goes, so
+// whatever asks — ctl, a page — only has to watch.
+package modelserve
+
+import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"log/slog"
+	"net/http"
+	"regexp"
+	"slices"
+	"strconv"
+	"strings"
+	"sync"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
+)
+
+// Registry reads a repository.
+type Registry interface {
+	Repo(ctx context.Context, id string) (hf.Repo, error)
+}
+
+// Mirror fetches files and reports on them.
+type Mirror interface {
+	Fetch(ctx context.Context, req mirror.FetchRequest) (mirror.FetchResponse, error)
+	List(ctx context.Context) (mirror.ModelsResponse, error)
+}
+
+// Catalog adds a model.
+type Catalog interface {
+	Add(ctx context.Context, m catalog.Model, by string) error
+}
+
+// Fleet is what the job watches once the model is in the catalog.
+type Fleet interface {
+	Replicas() []reconciler.Replica
+	ModelHealth() []reconciler.ModelHealth
+	EngineKey(replicaID string) string
+}
+
+// Deps are what a job needs.
+type Deps struct {
+	Registry Registry
+	Mirror   Mirror
+	Catalog  Catalog
+	Fleet    Fleet
+	// LlamaVersion is the llama.cpp build the nodes install, which decides
+	// which architectures load.
+	LlamaVersion string
+	HTTP         *http.Client
+	Log          *slog.Logger
+	// Poll is how often the job looks at the mirror and the fleet.
+	Poll time.Duration
+	// ReadyTimeout bounds the wait for a replica: the node fetching the
+	// weights from the mirror and the engine loading them.
+	ReadyTimeout time.Duration
+}
+
+// Request is what to serve.
+type Request struct {
+	// Repo is the registry repository, owner/name.
+	Repo string `json:"repo"`
+	// Quant picks the file by its quantisation, Q4_K_M unless given.
+	Quant string `json:"quant,omitempty"`
+	// File picks it by name instead.
+	File string `json:"file,omitempty"`
+	// Name is the catalog name; the repository suggests one otherwise.
+	Name        string   `json:"name,omitempty"`
+	Aliases     []string `json:"aliases,omitempty"`
+	Description string   `json:"description,omitempty"`
+	// Replicas to run; 0 makes it on demand, loaded when first asked for.
+	Replicas int `json:"replicas"`
+	// CtxPool is llama.cpp's --ctx-size; ContextLength what one request may
+	// use. They default to 16384 and 4096, what four parallel slots divide.
+	CtxPool       int `json:"ctx_pool,omitempty"`
+	ContextLength int `json:"context_length,omitempty"`
+	// By is who asked, for the catalog's record.
+	By string `json:"-"`
+}
+
+// The states a job and its steps can be in.
+const (
+	Running = "running"
+	Done    = "done"
+	Failed  = "failed"
+	Skipped = "skipped"
+)
+
+// Step is one stage of a job.
+type Step struct {
+	Name   string    `json:"name"`
+	State  string    `json:"state"`
+	Detail string    `json:"detail,omitempty"`
+	At     time.Time `json:"at"`
+}
+
+// Job is one model being served, and where it stands.
+type Job struct {
+	ID      string    `json:"id"`
+	Request Request   `json:"request"`
+	State   string    `json:"state"`
+	Error   string    `json:"error,omitempty"`
+	Model   string    `json:"model,omitempty"`
+	Steps   []Step    `json:"steps"`
+	Started time.Time `json:"started"`
+	// Reply is what the model said to the smoke test, and TokensPerSecond
+	// how fast it said it.
+	Reply           string  `json:"reply,omitempty"`
+	TokensPerSecond float64 `json:"tokens_per_second,omitempty"`
+}
+
+// Server runs jobs and remembers them.
+type Server struct {
+	deps Deps
+	mu   sync.Mutex
+	jobs map[string]*Job
+	seq  int
+}
+
+// New builds a server.
+func New(d Deps) *Server {
+	if d.Poll <= 0 {
+		d.Poll = 2 * time.Second
+	}
+	if d.ReadyTimeout <= 0 {
+		d.ReadyTimeout = 30 * time.Minute
+	}
+	if d.HTTP == nil {
+		d.HTTP = &http.Client{Timeout: 5 * time.Minute}
+	}
+	if d.Log == nil {
+		d.Log = slog.Default()
+	}
+	return &Server{deps: d, jobs: map[string]*Job{}}
+}
+
+var repoPattern = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._-]*/[A-Za-z0-9][A-Za-z0-9._-]*$`)
+
+// Start begins a job and returns at once; the job runs on its own.
+func (s *Server) Start(req Request) (Job, error) {
+	req.Repo = strings.Trim(strings.TrimSpace(req.Repo), "/")
+	if !repoPattern.MatchString(req.Repo) {
+		return Job{}, fmt.Errorf("%q is not a repository; it looks like owner/name", req.Repo)
+	}
+	if req.Replicas < 0 {
+		return Job{}, errors.New("replicas cannot be negative")
+	}
+	if req.Quant == "" && req.File == "" {
+		req.Quant = "Q4_K_M"
+	}
+	if req.CtxPool <= 0 {
+		req.CtxPool = 16384
+	}
+	if req.ContextLength <= 0 {
+		req.ContextLength = 4096
+	}
+	s.mu.Lock()
+	s.seq++
+	j := &Job{ID: fmt.Sprintf("serve-%d-%d", time.Now().Unix(), s.seq), Request: req,
+		State: Running, Started: time.Now()}
+	s.jobs[j.ID] = j
+	s.mu.Unlock()
+	go s.run(j)
+	return s.snapshot(j), nil
+}
+
+// Get returns a job as it stands.
+func (s *Server) Get(id string) (Job, bool) {
+	s.mu.Lock()
+	j, ok := s.jobs[id]
+	s.mu.Unlock()
+	if !ok {
+		return Job{}, false
+	}
+	return s.snapshot(j), true
+}
+
+func (s *Server) snapshot(j *Job) Job {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	out := *j
+	out.Steps = slices.Clone(j.Steps)
+	return out
+}
+
+// step starts a new step, or updates the detail of the current one.
+func (s *Server) step(j *Job, name, state, detail string) {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	if n := len(j.Steps); n > 0 && j.Steps[n-1].Name == name {
+		j.Steps[n-1].State, j.Steps[n-1].Detail, j.Steps[n-1].At = state, detail, time.Now()
+		return
+	}
+	j.Steps = append(j.Steps, Step{Name: name, State: state, Detail: detail, At: time.Now()})
+}
+
+func (s *Server) fail(j *Job, name string, err error) {
+	s.step(j, name, Failed, err.Error())
+	s.mu.Lock()
+	j.State, j.Error = Failed, err.Error()
+	s.mu.Unlock()
+	s.deps.Log.Warn("serving a model failed", "job", j.ID, "repo", j.Request.Repo, "step", name, "err", err)
+}
+
+func (s *Server) run(j *Job) {
+	ctx := context.Background()
+	req := j.Request
+
+	// 1. The registry: which files.
+	s.step(j, "registry", Running, "reading "+req.Repo)
+	repo, err := s.deps.Registry.Repo(ctx, req.Repo)
+	if err != nil {
+		s.fail(j, "registry", err)
+		return
+	}
+	if repo.Gated.Is() {
+		s.fail(j, "registry", fmt.Errorf("%s is gated: it needs an account's approval the mirror does not have", req.Repo))
+		return
+	}
+	set, err := pick(repo, req)
+	if err != nil {
+		s.fail(j, "registry", err)
+		return
+	}
+	name := req.Name
+	if name == "" {
+		name = hf.ModelName(repo.ID)
+	}
+	s.mu.Lock()
+	j.Model = name
+	s.mu.Unlock()
+	s.step(j, "registry", Done, fmt.Sprintf("%s, %.1f GB in %d file(s), revision %s",
+		set.Name, gb(set.Bytes), len(set.Files), repo.ShortRevision()))
+
+	// 2. The mirror: fetch, and check what arrived against what was published.
+	source, err := s.mirror(ctx, j, repo, set, name)
+	if err != nil {
+		s.fail(j, "mirror", err)
+		return
+	}
+
+	// 3. Whether the nodes' llama.cpp loads it, and how much memory it needs.
+	s.step(j, "check", Running, "reading its header from the mirror")
+	var header gguf.Header
+	if !strings.HasSuffix(source, "/") {
+		v := gguf.Check(ctx, source, s.deps.LlamaVersion)
+		if !v.Servable {
+			s.fail(j, "check", fmt.Errorf("llama.cpp %s on the nodes cannot run this: %s", s.deps.LlamaVersion, v.Reason))
+			return
+		}
+		header = v.Header
+	}
+	memGB := int((header.MemoryBytes(set.Bytes, req.CtxPool) + 1<<30 - 1) >> 30)
+	basis := fmt.Sprintf("weights %.1f GB + KV cache %.1f GB for %d tokens + engine",
+		gb(set.Bytes), gb(int64(req.CtxPool)*header.KVBytesPerToken()), req.CtxPool)
+	if memGB == 0 {
+		// A set of shards, or a header that did not say: weights plus half
+		// again plus two, the rough figure from before shapes were read.
+		memGB = int(gb(set.Bytes)*1.5) + 2
+		basis = "weights plus half again plus two; the header gave no shape"
+	}
+	s.step(j, "check", Done, fmt.Sprintf("%s, needs about %d GB (%s)", orUnknown(header.Architecture), memGB, basis))
+
+	// 4. The catalog.
+	m := catalog.Model{
+		Name: name, Aliases: req.Aliases, Description: req.Description, Source: source,
+		RoutingProfile: "chat", ContextLength: req.ContextLength, MaxOutput: 1024,
+		Profiles: []catalog.Profile{{
+			Name: "cpu", Engine: "llamacpp", Weight: 1, Priority: 100,
+			Args: []string{"--ctx-size", strconv.Itoa(req.CtxPool)}, MinMemoryGB: memGB,
+		}},
+		Scaling: catalog.Scaling{Min: req.Replicas, Max: max(req.Replicas, 1)},
+	}
+	s.step(j, "catalog", Running, "adding "+name)
+	if err := s.deps.Catalog.Add(ctx, m, req.By); err != nil {
+		s.fail(j, "catalog", err)
+		return
+	}
+	if req.Replicas == 0 {
+		s.step(j, "catalog", Done, name+" added on demand: it loads when first asked for")
+		s.finish(j)
+		return
+	}
+	s.step(j, "catalog", Done, fmt.Sprintf("%s added, %d replica(s)", name, req.Replicas))
+
+	// 5. A replica, ready.
+	rep, err := s.waitReady(ctx, j, name)
+	if err != nil {
+		s.fail(j, "placement", err)
+		return
+	}
+
+	// 6. Does it answer.
+	s.step(j, "smoke test", Running, "asking "+rep.ID+" to say hello")
+	reply, tps, err := s.smoke(ctx, rep, name)
+	if err != nil {
+		s.fail(j, "smoke test", err)
+		return
+	}
+	s.mu.Lock()
+	j.Reply, j.TokensPerSecond = reply, tps
+	s.mu.Unlock()
+	s.step(j, "smoke test", Done, fmt.Sprintf("answered at %.1f tokens/s: %q", tps, reply))
+	s.finish(j)
+}
+
+func (s *Server) finish(j *Job) {
+	s.mu.Lock()
+	j.State = Done
+	s.mu.Unlock()
+	s.deps.Log.Info("model served", "job", j.ID, "repo", j.Request.Repo, "model", j.Model)
+}
+
+// pick chooses the file set to serve.
+func pick(repo hf.Repo, req Request) (hf.FileSet, error) {
+	var weights []hf.File
+	for _, f := range repo.Files {
+		lower := strings.ToLower(f.Path)
+		if !strings.HasSuffix(lower, ".gguf") || strings.Contains(lower, "mmproj") ||
+			strings.Contains(lower, "imatrix") {
+			continue
+		}
+		weights = append(weights, f)
+	}
+	sets := hf.Group(weights)
+	if len(sets) == 0 {
+		return hf.FileSet{}, fmt.Errorf("%s has no GGUF files; the nodes run llama.cpp, which needs one", repo.ID)
+	}
+	var names []string
+	for _, st := range sets {
+		names = append(names, st.Name)
+		if req.File != "" && (st.Name == req.File || slices.ContainsFunc(st.Files, func(f hf.File) bool { return f.Path == req.File })) {
+			return st, nil
+		}
+	}
+	if req.File == "" {
+		want := strings.ToLower(req.Quant)
+		var match []hf.FileSet
+		for _, st := range sets {
+			if strings.Contains(strings.ToLower(st.Name), want) {
+				match = append(match, st)
+			}
+		}
+		if len(match) == 1 {
+			return match[0], nil
+		}
+		if len(match) > 1 {
+			// Both a split and a joined copy of one quantisation: the joined
+			// one, which is one download to verify rather than several.
+			slices.SortFunc(match, func(a, b hf.FileSet) int { return a.Shards - b.Shards })
+			return match[0], nil
+		}
+	}
+	return hf.FileSet{}, fmt.Errorf("%s has no file matching %q; it has: %s",
+		repo.ID, or(req.File, req.Quant), strings.Join(names, ", "))
+}
+
+// mirror fetches the set into the mirror and returns the source a catalog
+// entry names: the file, or the directory for a set of shards.
+func (s *Server) mirror(ctx context.Context, j *Job, repo hf.Repo, set hf.FileSet, name string) (string, error) {
+	version := repo.ShortRevision()
+	s.step(j, "mirror", Running, "asking the mirror for "+set.Name)
+	var base string
+	for _, f := range set.Files {
+		file := mirror.LastPathElement(f.Path)
+		resp, err := s.deps.Mirror.Fetch(ctx, mirror.FetchRequest{
+			Source: repo.DownloadURL("", f.Path), Model: name, Version: version, File: file,
+		})
+		// Already there is not a failure: the same commit fetched before.
+		if err != nil && !strings.Contains(err.Error(), "already") {
+			return "", err
+		}
+		base = resp.URL
+	}
+	deadline := time.Now().Add(6 * time.Hour)
+	for {
+		got, err := s.deps.Mirror.List(ctx)
+		if err != nil {
+			return "", err
+		}
+		var done, total int64
+		present, failed := 0, ""
+		sums := map[string]string{}
+		for _, st := range got.Models {
+			if st.Model != name || st.Version != version {
+				continue
+			}
+			done += st.BytesDone
+			total += max(st.BytesTotal, st.BytesDone)
+			switch st.State {
+			case mirror.Present:
+				present++
+				sums[st.File] = st.SHA256
+			case mirror.Failed:
+				failed = st.Err
+			}
+		}
+		if failed != "" {
+			return "", fmt.Errorf("the mirror could not fetch it: %s", failed)
+		}
+		if present >= len(set.Files) {
+			// What arrived against what the registry published: a mirror
+			// copies one file to every node, and is the one place a corrupted
+			// download is caught once instead of loaded everywhere.
+			for _, f := range set.Files {
+				file := mirror.LastPathElement(f.Path)
+				if f.SHA256 != "" && sums[file] != "" && !strings.EqualFold(f.SHA256, sums[file]) {
+					return "", fmt.Errorf("%s arrived with sha256 %s, and the registry published %s",
+						file, sums[file][:12], f.SHA256[:12])
+				}
+			}
+			source := base
+			if len(set.Files) > 1 {
+				source = strings.TrimSuffix(source, mirror.LastPathElement(source))
+			}
+			s.step(j, "mirror", Done, fmt.Sprintf("%.1f GB on the mirror, checksums match the registry", gb(total)))
+			return source, nil
+		}
+		if total > 0 {
+			s.step(j, "mirror", Running, fmt.Sprintf("fetching: %.1f of %.1f GB (%d%%)",
+				gb(done), gb(total), done*100/total))
+		}
+		if time.Now().After(deadline) {
+			return "", errors.New("the mirror had not finished after six hours")
+		}
+		time.Sleep(s.deps.Poll)
+	}
+}
+
+// waitReady waits for a replica of the model to serve, reporting what it is
+// doing and giving up early when it cannot be placed or keeps failing.
+func (s *Server) waitReady(ctx context.Context, j *Job, name string) (reconciler.Replica, error) {
+	s.step(j, "placement", Running, "waiting for the control plane to place it")
+	started := time.Now()
+	deadline := started.Add(s.deps.ReadyTimeout)
+	for {
+		for _, h := range s.deps.Fleet.ModelHealth() {
+			if h.Model != name {
+				continue
+			}
+			if h.Parked {
+				return reconciler.Replica{}, fmt.Errorf("it failed to start %d times and is parked: %s", h.Failures, h.Reason)
+			}
+			// A minute's grace from when this step began, not from the job:
+			// the download before it can take hours, and the first pass may
+			// not have run yet.
+			if h.Unplaceable && time.Since(started) > time.Minute {
+				return reconciler.Replica{}, fmt.Errorf("no node can host it:\n%s", h.Placement)
+			}
+		}
+		for _, r := range s.deps.Fleet.Replicas() {
+			if r.Model != name {
+				continue
+			}
+			if r.State == reconciler.Ready {
+				s.step(j, "placement", Done, fmt.Sprintf("%s ready on %s at %s", r.ID,
+					strings.Join(r.NodeIDs(), ","), r.Endpoint))
+				return r, nil
+			}
+			s.step(j, "placement", Running, fmt.Sprintf("%s on %s is %s", r.ID,
+				strings.Join(r.NodeIDs(), ","), r.State))
+		}
+		if time.Now().After(deadline) {
+			return reconciler.Replica{}, fmt.Errorf("no replica was ready after %s", s.deps.ReadyTimeout)
+		}
+		select {
+		case <-ctx.Done():
+			return reconciler.Replica{}, ctx.Err()
+		case <-time.After(s.deps.Poll):
+		}
+	}
+}
+
+// smoke sends one short request straight to the replica, with its engine key.
+func (s *Server) smoke(ctx context.Context, r reconciler.Replica, name string) (string, float64, error) {
+	body, _ := json.Marshal(map[string]any{
+		"model":      name,
+		"max_tokens": 32,
+		"messages":   []map[string]string{{"role": "user", "content": "Say hello in one short sentence."}},
+	})
+	req, err := http.NewRequestWithContext(ctx, http.MethodPost,
+		strings.TrimRight(r.Endpoint, "/")+"/v1/chat/completions", bytes.NewReader(body))
+	if err != nil {
+		return "", 0, err
+	}
+	req.Header.Set("Content-Type", "application/json")
+	if k := s.deps.Fleet.EngineKey(r.ID); k != "" {
+		req.Header.Set("Authorization", "Bearer "+k)
+	}
+	start := time.Now()
+	resp, err := s.deps.HTTP.Do(req)
+	if err != nil {
+		return "", 0, err
+	}
+	defer resp.Body.Close()
+	b, _ := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
+	if resp.StatusCode != http.StatusOK {
+		return "", 0, fmt.Errorf("the replica answered %s: %s", resp.Status, strings.TrimSpace(string(b)))
+	}
+	var out struct {
+		Choices []struct {
+			Message struct {
+				Content string `json:"content"`
+			} `json:"message"`
+		} `json:"choices"`
+		Usage struct {
+			CompletionTokens int `json:"completion_tokens"`
+		} `json:"usage"`
+	}
+	if err := json.Unmarshal(b, &out); err != nil || len(out.Choices) == 0 {
+		return "", 0, fmt.Errorf("the replica's answer was not a completion: %s", strings.TrimSpace(string(b)))
+	}
+	secs := time.Since(start).Seconds()
+	tps := 0.0
+	if secs > 0 {
+		tps = float64(out.Usage.CompletionTokens) / secs
+	}
+	return strings.TrimSpace(out.Choices[0].Message.Content), tps, nil
+}
+
+func gb(b int64) float64 { return float64(b) / (1 << 30) }
+
+func or(a, b string) string {
+	if a != "" {
+		return a
+	}
+	return b
+}
+
+func orUnknown(arch string) string {
+	if arch == "" {
+		return "architecture not read"
+	}
+	return arch
+}

internal/control/modelserve/modelserve_test.go

@@ -0,0 +1,225 @@
+package modelserve
+
+import (
+	"bytes"
+	"context"
+	"encoding/binary"
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"sync"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
+)
+
+// ggufHeader is enough of a GGUF file for its architecture and shape.
+func ggufHeader() []byte {
+	var b bytes.Buffer
+	w := func(v any) { _ = binary.Write(&b, binary.LittleEndian, v) }
+	str := func(s string) { w(uint64(len(s))); b.WriteString(s) }
+	u32 := func(k string, v uint32) { str(k); w(uint32(4)); w(v) }
+	b.WriteString("GGUF")
+	w(uint32(3))
+	w(uint64(0))
+	w(uint64(5))
+	str("general.architecture")
+	w(uint32(8))
+	str("qwen2")
+	u32("qwen2.block_count", 28)
+	u32("qwen2.attention.head_count", 12)
+	u32("qwen2.attention.head_count_kv", 2)
+	u32("qwen2.embedding_length", 1536)
+	return b.Bytes()
+}
+
+type fakeRegistry struct{ repo hf.Repo }
+
+func (f fakeRegistry) Repo(context.Context, string) (hf.Repo, error) { return f.repo, nil }
+
+// fakeMirror serves the header at every file URL and reports each fetched
+// file present, with the checksum it was told to.
+type fakeMirror struct {
+	mu   sync.Mutex
+	srv  *httptest.Server
+	got  []mirror.FetchRequest
+	sums map[string]string
+}
+
+func (m *fakeMirror) Fetch(_ context.Context, r mirror.FetchRequest) (mirror.FetchResponse, error) {
+	m.mu.Lock()
+	defer m.mu.Unlock()
+	m.got = append(m.got, r)
+	return mirror.FetchResponse{URL: m.srv.URL + "/" + r.Model + "/" + r.Version + "/" + r.File}, nil
+}
+
+func (m *fakeMirror) List(context.Context) (mirror.ModelsResponse, error) {
+	m.mu.Lock()
+	defer m.mu.Unlock()
+	var out mirror.ModelsResponse
+	for _, r := range m.got {
+		out.Models = append(out.Models, mirror.Status{Model: r.Model, Version: r.Version, File: r.File,
+			State: mirror.Present, BytesDone: 1 << 30, BytesTotal: 1 << 30, SHA256: m.sums[r.File]})
+	}
+	return out, nil
+}
+
+type fakeCatalog struct {
+	mu    sync.Mutex
+	added []catalog.Model
+}
+
+func (c *fakeCatalog) Add(_ context.Context, m catalog.Model, _ string) error {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+	c.added = append(c.added, m)
+	return nil
+}
+
+// fakeFleet has a replica ready at engine once the catalog holds the model.
+type fakeFleet struct {
+	cat    *fakeCatalog
+	engine string
+	health []reconciler.ModelHealth
+}
+
+func (f *fakeFleet) Replicas() []reconciler.Replica {
+	f.cat.mu.Lock()
+	defer f.cat.mu.Unlock()
+	var out []reconciler.Replica
+	for _, m := range f.cat.added {
+		out = append(out, reconciler.Replica{ID: m.Name + "-r1", Model: m.Name,
+			State: reconciler.Ready, Endpoint: f.engine,
+			Shards: []reconciler.Shard{{NodeID: "node01"}}})
+	}
+	return out
+}
+func (f *fakeFleet) ModelHealth() []reconciler.ModelHealth { return f.health }
+func (f *fakeFleet) EngineKey(string) string               { return "k" }
+
+func setup(t *testing.T, files []hf.File, sums map[string]string) (*Server, *fakeMirror, *fakeCatalog, *fakeFleet) {
+	t.Helper()
+	header := ggufHeader()
+	files0 := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+		_, _ = w.Write(header)
+	}))
+	t.Cleanup(files0.Close)
+	engine := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		if r.Header.Get("Authorization") != "Bearer k" {
+			w.WriteHeader(http.StatusUnauthorized)
+			return
+		}
+		_, _ = w.Write([]byte(`{"choices":[{"message":{"content":"Hello!"}}],"usage":{"completion_tokens":3}}`))
+	}))
+	t.Cleanup(engine.Close)
+
+	m := &fakeMirror{srv: files0, sums: sums}
+	c := &fakeCatalog{}
+	f := &fakeFleet{cat: c, engine: engine.URL}
+	s := New(Deps{
+		Registry: fakeRegistry{repo: hf.Repo{ID: "bartowski/Qwen2.5-1.5B-Instruct-GGUF", Revision: "abcdef123456", Files: files}},
+		Mirror:   m, Catalog: c, Fleet: f, Poll: 10 * time.Millisecond, ReadyTimeout: 5 * time.Second,
+	})
+	return s, m, c, f
+}
+
+func wait(t *testing.T, s *Server, id string) Job {
+	t.Helper()
+	deadline := time.Now().Add(10 * time.Second)
+	for time.Now().Before(deadline) {
+		if j, _ := s.Get(id); j.State != Running {
+			return j
+		}
+		time.Sleep(10 * time.Millisecond)
+	}
+	t.Fatal("the job never finished")
+	return Job{}
+}
+
+func TestAModelGoesFromTheRegistryToAnAnsweringReplica(t *testing.T) {
+	files := []hf.File{
+		{Path: "Qwen2.5-1.5B-Instruct-Q8_0.gguf", Bytes: 2 << 30, SHA256: "aa"},
+		{Path: "Qwen2.5-1.5B-Instruct-Q4_K_M.gguf", Bytes: 1 << 30, SHA256: "bb"},
+		{Path: "mmproj-Q4_K_M.gguf", Bytes: 1 << 20},
+	}
+	s, m, c, _ := setup(t, files, map[string]string{"Qwen2.5-1.5B-Instruct-Q4_K_M.gguf": "BB"})
+	job, err := s.Start(Request{Repo: "bartowski/Qwen2.5-1.5B-Instruct-GGUF", Replicas: 1})
+	if err != nil {
+		t.Fatal(err)
+	}
+	j := wait(t, s, job.ID)
+	if j.State != Done {
+		t.Fatalf("job %s: %s\n%+v", j.State, j.Error, j.Steps)
+	}
+	if len(m.got) != 1 || m.got[0].File != "Qwen2.5-1.5B-Instruct-Q4_K_M.gguf" || m.got[0].Version != "abcdef12" {
+		t.Errorf("fetched %+v; expected the Q4_K_M file at its revision", m.got)
+	}
+	mod := c.added[0]
+	if mod.Name != "qwen2.5-1.5b-instruct" || mod.Profiles[0].MinMemoryGB == 0 || mod.Scaling.Min != 1 {
+		t.Errorf("catalog entry %+v", mod)
+	}
+	if j.Reply != "Hello!" {
+		t.Errorf("the smoke test's answer was %q", j.Reply)
+	}
+	var names []string
+	for _, st := range j.Steps {
+		names = append(names, st.Name)
+	}
+	if strings.Join(names, ",") != "registry,mirror,check,catalog,placement,smoke test" {
+		t.Errorf("steps %v", names)
+	}
+}
+
+func TestAQuantTheRepositoryDoesNotHaveIsNamedWithWhatItHas(t *testing.T) {
+	s, _, c, _ := setup(t, []hf.File{{Path: "m-Q8_0.gguf", Bytes: 1}}, nil)
+	job, _ := s.Start(Request{Repo: "a/b", Replicas: 1})
+	j := wait(t, s, job.ID)
+	if j.State != Failed || !strings.Contains(j.Error, "m-Q8_0.gguf") {
+		t.Errorf("%s: %s", j.State, j.Error)
+	}
+	if len(c.added) != 0 {
+		t.Error("something was added to the catalog")
+	}
+}
+
+func TestADownloadThatDoesNotMatchTheRegistryIsNotServed(t *testing.T) {
+	files := []hf.File{{Path: "m-Q4_K_M.gguf", Bytes: 1 << 30, SHA256: "aaaaaaaaaaaaaaaa"}}
+	s, _, c, _ := setup(t, files, map[string]string{"m-Q4_K_M.gguf": "bbbbbbbbbbbbbbbb"})
+	job, _ := s.Start(Request{Repo: "a/b", Replicas: 1})
+	j := wait(t, s, job.ID)
+	if j.State != Failed || !strings.Contains(j.Error, "registry published") {
+		t.Errorf("%s: %s", j.State, j.Error)
+	}
+	if len(c.added) != 0 {
+		t.Error("a corrupted download reached the catalog")
+	}
+}
+
+func TestAParkedModelStopsTheWait(t *testing.T) {
+	s, _, _, f := setup(t, []hf.File{{Path: "m-Q4_K_M.gguf", Bytes: 1 << 30}}, nil)
+	f.health = []reconciler.ModelHealth{{Model: "qwen2.5-1.5b-instruct", Parked: true, Failures: 5, Reason: "unknown model architecture"}}
+	// A fleet where it never becomes ready, only parked.
+	s.deps.Fleet = parkedFleet{f}
+	job, _ := s.Start(Request{Repo: "a/b", Replicas: 1})
+	j := wait(t, s, job.ID)
+	if j.State != Failed || !strings.Contains(j.Error, "parked") {
+		t.Errorf("%s: %s", j.State, j.Error)
+	}
+}
+
+type parkedFleet struct{ *fakeFleet }
+
+func (parkedFleet) Replicas() []reconciler.Replica { return nil }
+
+func TestARepositoryThatIsNotOneIsRefused(t *testing.T) {
+	s := New(Deps{})
+	for _, r := range []string{"", "noslash", "../etc/passwd", "a/b/c"} {
+		if _, err := s.Start(Request{Repo: r}); err == nil {
+			t.Errorf("%q accepted", r)
+		}
+	}
+}

internal/control/reconciler/adopt.go

@@ -0,0 +1,255 @@
+package reconciler
+
+import (
+	"slices"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/supervisor"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
+)
+
+// Adopt rebuilds replica state from shards an agent reports it is already
+// running (plan §6.5).
+//
+// This is what makes a control plane restart survivable. Without it the
+// reconciler sees an empty cluster, places duplicates, and the engines already
+// running keep holding GPUs the inventory believes are free — so the duplicates
+// fail to allocate, with an error naming the new process rather than the old.
+//
+// Shards whose model or profile is no longer in the catalog are returned for
+// the caller to stop: nothing will ever claim them.
+func (r *Reconciler) Adopt(nodeID string, shards []AdoptedShard) (unclaimed []AdoptedShard) {
+	defer r.forgetShardsTheAgentDoesNotHave(nodeID, shards)
+
+	byReplica := map[string][]AdoptedShard{}
+	for _, s := range shards {
+		byReplica[s.ReplicaID] = append(byReplica[s.ReplicaID], s)
+	}
+
+	r.mu.Lock()
+	defer r.mu.Unlock()
+
+	for replicaID, group := range byReplica {
+		if existing, known := r.replicas[replicaID]; known {
+			// The other half of a replica we are still assembling. Merge rather
+			// than choose: these shards belong to the same replica as the ones
+			// already adopted, they are simply on a different node whose agent
+			// registered a moment later. Treating them as a competing claim is
+			// what tore multi-node replicas apart on every control plane
+			// restart — one half adopted and failed for being short, the other
+			// stopped as unclaimed when it finally arrived.
+			if !existing.AdoptedAt.IsZero() && len(existing.Shards) < existing.WantShards {
+				r.mergeAdopted(existing, group)
+				continue
+			}
+			// Usually this is an agent reconnecting rather than a control plane
+			// restart, and its status reports keep the state current.
+			//
+			// Not always. Replica ids are derived from the model name and a
+			// counter, so a control plane that restarts and places before the
+			// agents reconnect produces a *different* replica under the same
+			// id, on a freshly allocated port. Skipping the adoption then
+			// leaves the cluster pointing at a port nothing listens on, while
+			// the engine that is actually serving is treated as a stray.
+			//
+			// The fleet is authoritative about what is running (plan §6.5), so
+			// what we merely placed loses to what an agent reports — unless our
+			// copy has a shard that has actually come up, in which case two
+			// things really are running and the one we did not ask for is the
+			// stray.
+			if existing.hasLiveShard() {
+				// Only what we do not recognise. A reconnecting agent reports
+				// the shards we already know, and disowning those would tell it
+				// to stop the very replica that is serving.
+				known := map[string]struct{}{}
+				for _, sh := range existing.Shards {
+					known[sh.ID] = struct{}{}
+				}
+				var strays []AdoptedShard
+				for _, s := range group {
+					if _, ok := known[s.ShardID]; !ok {
+						strays = append(strays, s)
+					}
+				}
+				unclaimed = append(unclaimed, strays...)
+				r.reservePorts(strays)
+				continue
+			}
+			r.log.Warn("a running replica reported by the fleet replaces one we placed",
+				"replica", replicaID, "placed_endpoint", existing.Endpoint)
+			for _, s := range existing.Shards {
+				r.inv.Release(s.ID)
+			}
+			delete(r.replicas, replicaID)
+		}
+
+		first := group[0]
+		m, ok := r.cat.Load().Resolve(first.Model)
+		if !ok {
+			unclaimed = append(unclaimed, group...)
+			r.reservePorts(group)
+			continue
+		}
+		prof, ok := m.ProfileByName(first.Profile)
+		if !ok {
+			unclaimed = append(unclaimed, group...)
+			r.reservePorts(group)
+			continue
+		}
+
+		rep := &Replica{
+			ID: replicaID, Model: m.Name, Profile: prof.Name,
+			Fabric: first.Fabric, Pool: first.Pool,
+			// From the agent, which was told it when the shard started. Empty
+			// from an agent too old to report it, which reads as unknown.
+			ProfileFP: first.ProfileFP,
+			Weight:    prof.Weight, CreatedAt: r.now(),
+			Endpoint: first.Endpoint,
+			// From the profile, not from what was reported: the point is to
+			// notice when what was reported is short.
+			WantShards: max(prof.Nodes, 1),
+			AdoptedAt:  r.now(),
+		}
+		r.mergeAdopted(rep, group)
+		r.replicas[replicaID] = rep
+
+	}
+	return unclaimed
+}
+
+// mergeAdopted folds shards an agent reports into a replica being reassembled.
+//
+// Called both for the first group, which creates the replica, and for every
+// later group belonging to it, so a replica spanning N nodes is built up over N
+// registrations. Shard ids already present are skipped: an agent that
+// reconnects repeats what it is running, and counting those again would carry a
+// replica past its shard count without a second node ever appearing.
+func (r *Reconciler) mergeAdopted(rep *Replica, group []AdoptedShard) {
+	have := make(map[string]struct{}, len(rep.Shards))
+	for _, sh := range rep.Shards {
+		have[sh.ID] = struct{}{}
+	}
+	for _, s := range group {
+		if _, dup := have[s.ShardID]; dup {
+			continue
+		}
+		have[s.ShardID] = struct{}{}
+		rep.Shards = append(rep.Shards, Shard{
+			ID: s.ShardID, NodeID: s.NodeID, Rank: s.Rank,
+			GPUIndices: s.GPUIndices, HCAs: s.HCAs,
+			HTTPPort: s.HTTPPort, State: s.State,
+		})
+		// Reclaim the GPUs and the memory, or the next placement hands them
+		// out twice. The memory comes from the catalog rather than the agent:
+		// what a profile asked for is not something a running process reports.
+		r.inv.Allocate(s.NodeID, inventory.Allocation{
+			ShardID: s.ShardID, ReplicaID: rep.ID, GPUIndices: s.GPUIndices,
+			MemoryGB: r.profileMemory(rep.Model, rep.Profile),
+		})
+		// The endpoint comes from the leader, which may be in any of the
+		// groups — the rank 0 node is not necessarily the first to register.
+		if s.Rank == 0 && s.Endpoint != "" {
+			rep.Endpoint = s.Endpoint
+		}
+	}
+	slices.SortFunc(rep.Shards, func(a, b Shard) int { return a.Rank - b.Rank })
+	rep.deriveState(r.now())
+
+	r.log.Info("adopted replica from the fleet",
+		"replica", rep.ID, "model", rep.Model,
+		"shards", len(rep.Shards), "want", rep.WantShards, "state", rep.State)
+}
+
+// forgetShardsTheAgentDoesNotHave fails shards we believe are on a node that the
+// agent, having just told us everything it is running, did not mention.
+//
+// The mirror image of adoption, and the half that was missing. Adoption handles
+// "the agent has something we do not know about"; this handles "we think the
+// agent has something it does not". Both follow from the same rule: the fleet is
+// authoritative about what is running (plan §6.5).
+//
+// Without it, an engine that dies while its agent survives leaves the control
+// plane believing a replica is Ready forever. Observed on a real cluster: the
+// engine was killed, the agent restarted and registered with nothing running,
+// and the replica stayed Ready pointing at a dead port for as long as anyone
+// watched. Traffic survived — the breaker took the endpoint out — but the
+// cluster ran a third short with no sign of it.
+//
+// Safe against the obvious race. A registration follows a fresh connection, so
+// anything we dispatched over a previous one is not running by definition;
+// there is no window where a shard is legitimately started but unreported.
+func (r *Reconciler) forgetShardsTheAgentDoesNotHave(nodeID string, running []AdoptedShard) {
+	if nodeID == "" {
+		return
+	}
+	reported := make(map[string]struct{}, len(running))
+	for _, s := range running {
+		reported[s.ShardID] = struct{}{}
+	}
+
+	var lost []string
+	r.mu.Lock()
+	for _, rep := range r.replicas {
+		if rep.State == Stopped {
+			continue
+		}
+		changed := false
+		for i := range rep.Shards {
+			sh := &rep.Shards[i]
+			if sh.NodeID != nodeID || sh.State.Terminal() {
+				continue
+			}
+			if _, ok := reported[sh.ID]; ok {
+				continue
+			}
+			sh.State = supervisor.Failed
+			sh.Err = "the agent reports it is not running"
+			r.inv.Release(sh.ID)
+			changed = true
+			lost = append(lost, sh.ID)
+		}
+		if changed {
+			rep.deriveState(r.now())
+		}
+	}
+	r.mu.Unlock()
+
+	for _, id := range lost {
+		r.log.Warn("a shard we believed was running is not; the agent does not have it",
+			"node", nodeID, "shard", id)
+	}
+}
+
+// reservePorts records ports held by shards nothing owns.
+//
+// They are being stopped, but stopping is not instant and the engine keeps its
+// socket until it exits. Handing the port to a new shard in the meantime makes
+// that shard fail to bind, which fails its replica, which places another one on
+// the same port — a loop that does not end on its own.
+//
+// Called with the lock held.
+func (r *Reconciler) reservePorts(shards []AdoptedShard) {
+	for _, s := range shards {
+		if s.HTTPPort > 0 {
+			r.reservedPorts[s.HTTPPort] = struct{}{}
+		}
+	}
+}
+
+// AdoptedShard is one running shard an agent reported at registration.
+type AdoptedShard struct {
+	NodeID     string
+	ReplicaID  string
+	ShardID    string
+	Rank       int
+	GPUIndices []int
+	HCAs       []string
+	HTTPPort   int
+	State      supervisor.State
+	Model      string
+	Profile    string
+	Pool       string
+	Fabric     string
+	ProfileFP  string
+	// Endpoint is the leader's base URL, set on the rank 0 shard.
+	Endpoint string
+}

internal/control/reconciler/demand.go

@@ -0,0 +1,214 @@
+package reconciler
+
+import (
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+)
+
+// RecordUsage stamps replicas with when each was last sent a request.
+//
+// Replicas absent from the map are left alone rather than zeroed: a router that
+// has just restarted knows nothing about what happened before it, and treating
+// that as "never used" would make every replica look abandoned at once —
+// exactly when a fleet is least able to afford acting on that.
+func (r *Reconciler) RecordUsage(lastUsed map[string]time.Time) {
+	if len(lastUsed) == 0 {
+		return
+	}
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	for id, at := range lastUsed {
+		rep, ok := r.replicas[id]
+		if !ok {
+			continue
+		}
+		// Only forwards. Two routers reporting, or a stale snapshot arriving
+		// late, must not move a replica's last use backwards.
+		if at.After(rep.LastUsedAt) {
+			rep.LastUsedAt = at
+		}
+	}
+	r.noteUsage(lastUsed)
+}
+
+// IdleFor reports how long a replica has gone without a request, and whether
+// that is known at all.
+//
+// The second return is not a detail. "Idle for an hour" and "nobody has told me
+// yet" are different states, and anything that acts on idleness — eviction,
+// most obviously — has to refuse to act on the second.
+func (r *Reconciler) IdleFor(id string) (time.Duration, bool) {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	rep, ok := r.replicas[id]
+	if !ok || rep.LastUsedAt.IsZero() {
+		return 0, false
+	}
+	return r.now().Sub(rep.LastUsedAt), true
+}
+
+// Want marks an on-demand model as asked for, so the next pass places it.
+//
+// Called by whatever fields the request — the router, in practice. It returns
+// whether the model is one this applies to, so a caller can tell "being loaded"
+// from "there is genuinely nothing serving this", which are different answers
+// to give a client.
+func (r *Reconciler) Want(model string) bool {
+	m, ok := r.cat.Load().Resolve(model)
+	if !ok || !m.Scaling.OnDemand() {
+		return false
+	}
+	r.mu.Lock()
+	r.wanted[m.Name] = r.now()
+	// A placement that failed before this request says nothing about now:
+	// forgotten, so the pass this triggers decides afresh and Unservable
+	// does not answer from a stale result.
+	if res, ok := r.lastExplain[m.Name]; ok && res.Placement == nil {
+		delete(r.lastExplain, m.Name)
+	}
+	r.mu.Unlock()
+	// Immediately, not on the next tick. Someone is waiting on this request,
+	// and five seconds of ticker is five seconds they spend staring at nothing.
+	r.Notify()
+	return true
+}
+
+// desiredFor is how many replicas a model should have right now.
+//
+// For an ordinary model that is its configured minimum. For an on-demand one it
+// is one while somebody is asking and none once they stop, which is the whole
+// feature: a cluster with more models than GPUs can hold every model in its
+// catalog and only the wanted ones in its cards.
+func (r *Reconciler) desiredFor(m catalog.Model) int {
+	base := r.baseFor(m)
+	if base == 0 {
+		return 0
+	}
+	r.mu.Lock()
+	extra := r.extra[m.Name]
+	r.mu.Unlock()
+	return min(base+extra, max(m.Scaling.Max, base))
+}
+
+// baseFor is what a model runs without load: its minimum, or for an on-demand
+// model one while somebody is asking and none once they stop.
+func (r *Reconciler) baseFor(m catalog.Model) int {
+	if !m.Scaling.OnDemand() {
+		return m.Scaling.Min
+	}
+	r.mu.Lock()
+	last, asked := r.wanted[m.Name]
+	r.mu.Unlock()
+	if !asked {
+		return 0
+	}
+	if r.now().Sub(last) > r.idleTimeout(m) {
+		return 0
+	}
+	return 1
+}
+
+// idleTimeout is how long an on-demand model stays up unasked-for.
+func (r *Reconciler) idleTimeout(m catalog.Model) time.Duration {
+	if m.Scaling.IdleTimeout > 0 {
+		return m.Scaling.IdleTimeout
+	}
+	return r.cfg.IdleTimeout
+}
+
+// noteUsage keeps an on-demand model alive while it is being used.
+//
+// Without this a model would be evicted exactly IdleTimeout after the request
+// that loaded it, however busy it had been since — the timer would measure
+// time since the first request rather than the last.
+func (r *Reconciler) noteUsage(lastUsed map[string]time.Time) {
+	byModel := map[string]time.Time{}
+	for id, at := range lastUsed {
+		rep, ok := r.replicas[id]
+		if !ok {
+			continue
+		}
+		if at.After(byModel[rep.Model]) {
+			byModel[rep.Model] = at
+		}
+	}
+	for model, at := range byModel {
+		if cur, ok := r.wanted[model]; ok && at.After(cur) {
+			r.wanted[model] = at
+		}
+	}
+}
+
+// Load is what a model's replicas are doing, as the router sees it.
+type Load struct {
+	Replicas int
+	Running  int
+	Waiting  int
+}
+
+// RecordLoad adds or takes away replicas between a model's min and max as its
+// load says.
+//
+// Max was a ceiling nothing reached: every model ran exactly its minimum
+// however long its queue. Now requests waiting at the engines — every slot of
+// every replica busy — held for ScaleUpAfter add one replica, and a quiet spell
+// of ScaleDownAfter takes one of those back. One step at a time, and not while
+// the last one is still starting: on a small CPU fleet each replica is minutes
+// of weights and gigabytes of memory, and a spike that is over by the time it
+// loads should not leave three behind.
+func (r *Reconciler) RecordLoad(loads map[string]Load) {
+	now := r.now()
+	for _, m := range r.cat.Load().Models() {
+		base := r.baseFor(m)
+		r.mu.Lock()
+		if base == 0 || m.Scaling.Max <= base {
+			delete(r.extra, m.Name)
+			delete(r.busySince, m.Name)
+			delete(r.quietSince, m.Name)
+			r.mu.Unlock()
+			continue
+		}
+		starting := false
+		for _, rep := range r.replicas {
+			if rep.Model == m.Name && (rep.State == Placing || rep.State == Starting) {
+				starting = true
+			}
+		}
+		l := loads[m.Name]
+		perReplica := float64(l.Waiting) / float64(max(l.Replicas, 1))
+		extra := r.extra[m.Name]
+		switch {
+		case perReplica >= r.cfg.ScaleUpWaiting:
+			delete(r.quietSince, m.Name)
+			since, ok := r.busySince[m.Name]
+			if !ok {
+				r.busySince[m.Name] = now
+				break
+			}
+			if now.Sub(since) >= r.cfg.ScaleUpAfter && base+extra < m.Scaling.Max && !starting {
+				r.extra[m.Name] = extra + 1
+				r.busySince[m.Name] = now
+				r.log.Info("scaling up for load", "model", m.Name, "replicas", base+extra+1,
+					"waiting_per_replica", perReplica, "for", now.Sub(since).Round(time.Second))
+			}
+		case l.Waiting == 0 && extra > 0:
+			delete(r.busySince, m.Name)
+			since, ok := r.quietSince[m.Name]
+			if !ok {
+				r.quietSince[m.Name] = now
+				break
+			}
+			if now.Sub(since) >= r.cfg.ScaleDownAfter {
+				r.extra[m.Name] = extra - 1
+				r.quietSince[m.Name] = now
+				r.log.Info("scaling down after a quiet spell", "model", m.Name,
+					"replicas", base+extra-1, "quiet_for", now.Sub(since).Round(time.Second))
+			}
+		default:
+			delete(r.busySince, m.Name)
+			delete(r.quietSince, m.Name)
+		}
+		r.mu.Unlock()
+	}
+}

internal/control/reconciler/headroom.go

@@ -0,0 +1,86 @@
+package reconciler
+
+import (
+	"slices"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/placement"
+)
+
+// Headroom is how many more replicas of a model the cluster has room for now.
+type Headroom struct {
+	// Current is the replicas the model has that hold capacity: starting or
+	// ready, not those on their way out.
+	Current int
+	// More is how many more could be placed now, at most the limit asked.
+	More int
+	// Nodes is where each of those would go, in order, a replica's nodes
+	// joined when it spans several.
+	Nodes [][]string
+	// Limited is set when More reached the limit asked, so there may be room
+	// for more than that. A CPU profile that declares no memory is always
+	// limited: nothing it asks for runs out.
+	Limited bool
+	// Blocked is why one more than More does not fit, a line per node that
+	// was considered. Empty when Limited.
+	Blocked []string
+}
+
+// Headroom tries placing up to limit more replicas of a model on a copy of
+// the inventory, each one's allocation made on the copy before the next is
+// tried, exactly as the reconciler would place them one pass at a time.
+//
+// The scheduler's own decisions, not an estimate beside them: what the page
+// offers is then what the next passes will do, and a count that the
+// arithmetic of a second implementation allowed but placement refuses would
+// leave a model short with nobody told why.
+func (r *Reconciler) Headroom(model string, limit int) (Headroom, bool) {
+	m, ok := r.cat.Load().Resolve(model)
+	if !ok {
+		return Headroom{}, false
+	}
+	var h Headroom
+	var avoid []string
+	r.mu.Lock()
+	for _, rep := range r.replicas {
+		if rep.Model == m.Name && rep.State.Counts() && rep.RetiringAt.IsZero() {
+			h.Current++
+			avoid = append(avoid, rep.NodeIDs()...)
+		}
+	}
+	r.mu.Unlock()
+
+	inv := r.inv.Clone()
+	sched := placement.New(inv)
+	for h.More < limit {
+		res := sched.Place(m.PlacementProfiles(), avoid)
+		if res.Placement == nil && len(avoid) > 0 {
+			// As scaleUp does: spreading is a preference, not a rule.
+			res = sched.Place(m.PlacementProfiles(), nil)
+		}
+		if res.Placement == nil {
+			for _, rej := range res.Rejections {
+				h.Blocked = append(h.Blocked, rej.String())
+			}
+			slices.Sort(h.Blocked)
+			h.Blocked = slices.Compact(h.Blocked)
+			return h, true
+		}
+		mem := 0
+		if prof, ok := m.ProfileByName(res.Placement.Profile.Name); ok {
+			mem = prof.MinMemoryGB
+		}
+		var nodes []string
+		for _, a := range res.Placement.Nodes {
+			inv.Allocate(a.NodeID, inventory.Allocation{
+				ShardID: "headroom", ReplicaID: "headroom", GPUIndices: a.GPUIndices, MemoryGB: mem,
+			})
+			nodes = append(nodes, a.NodeID)
+		}
+		avoid = append(avoid, nodes...)
+		h.Nodes = append(h.Nodes, nodes)
+		h.More++
+	}
+	h.Limited = true
+	return h, true
+}

internal/control/reconciler/headroom_test.go

@@ -0,0 +1,59 @@
+package reconciler
+
+import (
+	"strings"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
+)
+
+func TestHeadroomIsWhatPlacementWouldDoNext(t *testing.T) {
+	// Two 8-GPU nodes and a 4-GPU profile: four replicas in all, one placed.
+	h := newHarness(t, testModel(1), gpuNode("1", 8), gpuNode("2", 8))
+	h.r.Reconcile(t.Context())
+	h.markReady()
+
+	got, ok := h.r.Headroom("llama", 10)
+	if !ok {
+		t.Fatal("the model was not found")
+	}
+	if got.Current != 1 || got.More != 3 || got.Limited {
+		t.Fatalf("headroom %+v, expected 1 placed and room for 3 more", got)
+	}
+	// Spread first: the next one goes to the node without one.
+	if first := got.Nodes[0][0]; first == h.r.Replicas()[0].NodeIDs()[0] {
+		t.Errorf("the next replica would stack on %s beside the first", first)
+	}
+	if len(got.Blocked) == 0 || !strings.Contains(strings.Join(got.Blocked, "\n"), "GPU") {
+		t.Errorf("no reason given for the fifth: %v", got.Blocked)
+	}
+	// Tried on a copy: nothing was taken from the real inventory.
+	if free := len(h.inv.FreeGPUs("1")) + len(h.inv.FreeGPUs("2")); free != 12 {
+		t.Errorf("%d GPUs free after asking, expected the 12 the one replica leaves", free)
+	}
+
+	capped, _ := h.r.Headroom("llama", 2)
+	if capped.More != 2 || !capped.Limited || capped.Blocked != nil {
+		t.Errorf("asked for at most 2: %+v", capped)
+	}
+	if _, ok := h.r.Headroom("nope", 5); ok {
+		t.Error("an unknown model has headroom")
+	}
+}
+
+func TestHeadroomCountsMemoryAlreadyClaimed(t *testing.T) {
+	cpu := func(id string) inventory.Node {
+		return inventory.Node{ID: id, MgmtAddr: "10.0.0." + id, State: inventory.Ready, MemoryTotalMB: 16 * 1024}
+	}
+	m := catalog.Model{Name: "small", Source: "/m.gguf", Scaling: catalog.Scaling{Min: 1, Max: 1},
+		Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 6}}}
+	h := newHarness(t, m, cpu("a"), cpu("b"))
+	h.r.Reconcile(t.Context())
+
+	// 16 GB each at 6 GB a replica: two per node, one of them taken.
+	got, _ := h.r.Headroom("small", 10)
+	if got.Current != 1 || got.More != 3 {
+		t.Errorf("headroom %+v, expected room for 3 more", got)
+	}
+}

internal/control/reconciler/health.go

@@ -0,0 +1,151 @@
+package reconciler
+
+import (
+	"crypto/sha256"
+	"encoding/hex"
+	"encoding/json"
+	"fmt"
+	"strings"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+)
+
+// parkRecord is why a model stopped being retried.
+type parkRecord struct {
+	Since    time.Time
+	Reason   string
+	Failures int
+	// fp is the model's definition when it was parked; a different one
+	// unparks it.
+	fp string
+}
+
+type placementNote struct {
+	text string
+	at   time.Time
+}
+
+// modelFingerprint identifies a model's definition, so a change to it can be
+// told from a retry of the same thing.
+func modelFingerprint(m catalog.Model) string {
+	b, _ := json.Marshal(m)
+	sum := sha256.Sum256(b)
+	return hex.EncodeToString(sum[:8])
+}
+
+// placementLogEvery is how often an unchanged placement failure is repeated.
+const placementLogEvery = 10 * time.Minute
+
+// notePlacementFailure logs why a model could not be placed, when the reason
+// is new or has not been said for a while. Said on every pass it was 250
+// identical lines in two days for one model, burying everything else.
+func (r *Reconciler) notePlacementFailure(model, why string) {
+	r.mu.Lock()
+	last := r.placementLogged[model]
+	say := last.text != why || r.now().Sub(last.at) >= placementLogEvery
+	if say {
+		r.placementLogged[model] = placementNote{text: why, at: r.now()}
+	}
+	r.mu.Unlock()
+	if say {
+		r.log.Warn("no placement for model", "model", model, "reasons", why)
+	}
+}
+
+// Retry clears a parked model and its backoff, so it is placed on the next
+// pass.
+func (r *Reconciler) Retry(model string) bool {
+	r.mu.Lock()
+	_, was := r.parked[model]
+	delete(r.parked, model)
+	r.attempts[model] = 0
+	delete(r.nextAttempt, model)
+	r.mu.Unlock()
+	r.Notify()
+	return was
+}
+
+// ModelHealth is how one model stands, for status and metrics.
+type ModelHealth struct {
+	Model string
+	// Desired is how many replicas it should have now; 0 for a stopped model
+	// or an on-demand one nobody is asking for.
+	Desired int
+	Ready   int
+	// Failures counts replicas that failed in a row.
+	Failures int
+	// Parked is set when Failures reached ParkAfter; Reason is the last error.
+	Parked      bool
+	ParkedSince time.Time
+	Reason      string
+	// Unplaceable is set when it is short of replicas and the last attempt to
+	// place one found nowhere; Placement says why.
+	Unplaceable bool
+	Placement   string
+}
+
+// ModelHealth reports every model in the catalog.
+func (r *Reconciler) ModelHealth() []ModelHealth {
+	models := r.cat.Load().Models()
+	desired := map[string]int{}
+	for _, m := range models {
+		desired[m.Name] = r.desiredFor(m)
+	}
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	ready, live := map[string]int{}, map[string]int{}
+	for _, rep := range r.replicas {
+		if rep.State == Ready {
+			ready[rep.Model]++
+		}
+		if rep.State.Counts() {
+			live[rep.Model]++
+		}
+	}
+	out := make([]ModelHealth, 0, len(models))
+	for _, m := range models {
+		h := ModelHealth{Model: m.Name, Desired: desired[m.Name], Ready: ready[m.Name],
+			Failures: r.attempts[m.Name]}
+		if p, ok := r.parked[m.Name]; ok {
+			h.Parked, h.ParkedSince, h.Reason = true, p.Since, p.Reason
+		}
+		if res, ok := r.lastExplain[m.Name]; ok && res.Placement == nil && live[m.Name] < h.Desired {
+			h.Unplaceable, h.Placement = true, res.Explain()
+		}
+		out = append(out, h)
+	}
+	return out
+}
+
+// Unservable says why a model cannot be served at all, so a request for it is
+// told now rather than asked to retry something that will never come up.
+func (r *Reconciler) Unservable(model string) (string, bool) {
+	m, ok := r.cat.Load().Resolve(model)
+	if !ok {
+		return "", false
+	}
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	if p, ok := r.parked[m.Name]; ok {
+		return fmt.Sprintf("%s failed to start %d times in a row and is not being retried: %s",
+			m.Name, p.Failures, p.Reason), true
+	}
+	for _, rep := range r.replicas {
+		if rep.Model == m.Name && rep.State.Counts() {
+			return "", false // something is on its way
+		}
+	}
+	if res, ok := r.lastExplain[m.Name]; ok && res.Placement == nil {
+		return m.Name + " cannot be placed on any node: " + firstLine(res.Explain()), true
+	}
+	return "", false
+}
+
+func firstLine(s string) string {
+	s = strings.TrimSpace(s)
+	if i := strings.IndexByte(s, '\n'); i >= 0 {
+		return strings.TrimSpace(s[:i]) + " (and others; see llmgw ctl explain-placement)"
+	}
+	return s
+}

internal/control/reconciler/reconciler.go

@@ -65,6 +65,22 @@ type Config struct {
 	// Without a bound, draining a node whose model cannot be placed anywhere
 	// else would never finish.
 	DrainGrace time.Duration
+	// ScaleUpWaiting is the average number of requests waiting per ready
+	// replica, held for ScaleUpAfter, that adds a replica — up to the model's
+	// max. ScaleDownAfter is how long with nothing waiting before one of the
+	// replicas added that way is taken away again, down to its min.
+	ScaleUpWaiting float64
+	ScaleUpAfter   time.Duration
+	ScaleDownAfter time.Duration
+	// ParkAfter is how many failures in a row stop a model being retried.
+	//
+	// Backoff alone never gives up: it grows to MaxBackoff and keeps trying,
+	// and a model whose weights its engine cannot load — an architecture the
+	// installed llama.cpp does not know — was started 155 times over eight
+	// hours on the live cluster, each time failing the same way, with nothing
+	// to show for it but the log. A parked model is not placed again until its
+	// definition changes or an operator runs `llmgw ctl retry`.
+	ParkAfter int
 	// PortRange is the per-node port range for engine HTTP and dist-init.
 	PortRange [2]int
 	// MgmtInterface is the ethernet interface used for NCCL bootstrap.
@@ -125,6 +141,10 @@ func DefaultConfig() Config {
 		EvictionGrace:    2 * time.Minute,
 		MinLifetime:      5 * time.Minute,
 		DrainGrace:       5 * time.Minute,
+		ParkAfter:        5,
+		ScaleUpWaiting:   1,
+		ScaleUpAfter:     time.Minute,
+		ScaleDownAfter:   10 * time.Minute,
 		PortRange:        [2]int{20000, 20999},
 		MgmtInterface:    "eth0",
 	}
@@ -164,6 +184,18 @@ func (c *Config) normalize() {
 	if c.DrainGrace <= 0 {
 		c.DrainGrace = d.DrainGrace
 	}
+	if c.ParkAfter <= 0 {
+		c.ParkAfter = d.ParkAfter
+	}
+	if c.ScaleUpWaiting <= 0 {
+		c.ScaleUpWaiting = d.ScaleUpWaiting
+	}
+	if c.ScaleUpAfter <= 0 {
+		c.ScaleUpAfter = d.ScaleUpAfter
+	}
+	if c.ScaleDownAfter <= 0 {
+		c.ScaleDownAfter = d.ScaleDownAfter
+	}
 	if c.PortRange[1] <= c.PortRange[0] {
 		c.PortRange = d.PortRange
 	}
@@ -204,6 +236,20 @@ type Reconciler struct {
 	// because it is state about this moment and not about what the model is.
 	// A model nobody has wanted has no entry and is simply not running.
 	wanted map[string]time.Time
+	// restartQueue holds, per model, the replicas an operator asked to be
+	// restarted and not yet begun. One is replaced at a time: replacing them
+	// all at once would need the memory of every replica twice over.
+	restartQueue map[string][]string
+	// parked holds models that failed ParkAfter times in a row.
+	parked map[string]parkRecord
+	// extra is how many replicas load has added above a model's base, and
+	// busySince/quietSince when its load last crossed the thresholds.
+	extra      map[string]int
+	busySince  map[string]time.Time
+	quietSince map[string]time.Time
+	// placementLogged remembers the last "no placement" said per model, so it
+	// is said when it changes rather than on every five-second pass.
+	placementLogged map[string]placementNote
 
 	wake chan struct{}
 	stop chan struct{}
@@ -225,18 +271,24 @@ func New(cfg Config, cat *catalog.Live, inv *inventory.Store, disp Dispatcher, l
 	}
 	return &Reconciler{
 		cfg: cfg, cat: cat, inv: inv, disp: disp, log: log,
-		sched:         placement.New(inv),
-		now:           time.Now,
-		replicas:      map[string]*Replica{},
-		nextAttempt:   map[string]time.Time{},
-		attempts:      map[string]int{},
-		reservedPorts: map[int]struct{}{},
-		wanted:        map[string]time.Time{},
-		lastExplain:   map[string]placement.Result{},
-		wake:          make(chan struct{}, 1),
-		stop:          make(chan struct{}),
-		done:          make(chan struct{}),
-		startedAt:     time.Now(),
+		sched:           placement.New(inv),
+		now:             time.Now,
+		replicas:        map[string]*Replica{},
+		nextAttempt:     map[string]time.Time{},
+		attempts:        map[string]int{},
+		reservedPorts:   map[int]struct{}{},
+		wanted:          map[string]time.Time{},
+		restartQueue:    map[string][]string{},
+		parked:          map[string]parkRecord{},
+		extra:           map[string]int{},
+		busySince:       map[string]time.Time{},
+		quietSince:      map[string]time.Time{},
+		placementLogged: map[string]placementNote{},
+		lastExplain:     map[string]placement.Result{},
+		wake:            make(chan struct{}, 1),
+		stop:            make(chan struct{}),
+		done:            make(chan struct{}),
+		startedAt:       time.Now(),
 	}
 }
 
@@ -403,121 +455,6 @@ func (r *Reconciler) fleetHasSettled() bool {
 	return settled
 }
 
-// RecordUsage stamps replicas with when each was last sent a request.
-//
-// Replicas absent from the map are left alone rather than zeroed: a router that
-// has just restarted knows nothing about what happened before it, and treating
-// that as "never used" would make every replica look abandoned at once —
-// exactly when a fleet is least able to afford acting on that.
-func (r *Reconciler) RecordUsage(lastUsed map[string]time.Time) {
-	if len(lastUsed) == 0 {
-		return
-	}
-	r.mu.Lock()
-	defer r.mu.Unlock()
-	for id, at := range lastUsed {
-		rep, ok := r.replicas[id]
-		if !ok {
-			continue
-		}
-		// Only forwards. Two routers reporting, or a stale snapshot arriving
-		// late, must not move a replica's last use backwards.
-		if at.After(rep.LastUsedAt) {
-			rep.LastUsedAt = at
-		}
-	}
-	r.noteUsage(lastUsed)
-}
-
-// IdleFor reports how long a replica has gone without a request, and whether
-// that is known at all.
-//
-// The second return is not a detail. "Idle for an hour" and "nobody has told me
-// yet" are different states, and anything that acts on idleness — eviction,
-// most obviously — has to refuse to act on the second.
-func (r *Reconciler) IdleFor(id string) (time.Duration, bool) {
-	r.mu.Lock()
-	defer r.mu.Unlock()
-	rep, ok := r.replicas[id]
-	if !ok || rep.LastUsedAt.IsZero() {
-		return 0, false
-	}
-	return r.now().Sub(rep.LastUsedAt), true
-}
-
-// Want marks an on-demand model as asked for, so the next pass places it.
-//
-// Called by whatever fields the request — the router, in practice. It returns
-// whether the model is one this applies to, so a caller can tell "being loaded"
-// from "there is genuinely nothing serving this", which are different answers
-// to give a client.
-func (r *Reconciler) Want(model string) bool {
-	m, ok := r.cat.Load().Resolve(model)
-	if !ok || !m.Scaling.OnDemand() {
-		return false
-	}
-	r.mu.Lock()
-	r.wanted[m.Name] = r.now()
-	r.mu.Unlock()
-	// Immediately, not on the next tick. Someone is waiting on this request,
-	// and five seconds of ticker is five seconds they spend staring at nothing.
-	r.Notify()
-	return true
-}
-
-// desiredFor is how many replicas a model should have right now.
-//
-// For an ordinary model that is its configured minimum. For an on-demand one it
-// is one while somebody is asking and none once they stop, which is the whole
-// feature: a cluster with more models than GPUs can hold every model in its
-// catalog and only the wanted ones in its cards.
-func (r *Reconciler) desiredFor(m catalog.Model) int {
-	if !m.Scaling.OnDemand() {
-		return m.Scaling.Min
-	}
-	r.mu.Lock()
-	last, asked := r.wanted[m.Name]
-	r.mu.Unlock()
-	if !asked {
-		return 0
-	}
-	if r.now().Sub(last) > r.idleTimeout(m) {
-		return 0
-	}
-	return 1
-}
-
-// idleTimeout is how long an on-demand model stays up unasked-for.
-func (r *Reconciler) idleTimeout(m catalog.Model) time.Duration {
-	if m.Scaling.IdleTimeout > 0 {
-		return m.Scaling.IdleTimeout
-	}
-	return r.cfg.IdleTimeout
-}
-
-// noteUsage keeps an on-demand model alive while it is being used.
-//
-// Without this a model would be evicted exactly IdleTimeout after the request
-// that loaded it, however busy it had been since — the timer would measure
-// time since the first request rather than the last.
-func (r *Reconciler) noteUsage(lastUsed map[string]time.Time) {
-	byModel := map[string]time.Time{}
-	for id, at := range lastUsed {
-		rep, ok := r.replicas[id]
-		if !ok {
-			continue
-		}
-		if at.After(byModel[rep.Model]) {
-			byModel[rep.Model] = at
-		}
-	}
-	for model, at := range byModel {
-		if cur, ok := r.wanted[model]; ok && at.After(cur) {
-			r.wanted[model] = at
-		}
-	}
-}
-
 // expireStuckReplicas fails replicas that never finished coming up.
 //
 // Every other path out of Starting depends on the agent reporting something. A
@@ -663,6 +600,17 @@ func (r *Reconciler) reapFailed(ctx context.Context) {
 			// replica is about to disappear, and the backoff has to survive it.
 			r.attempts[rep.Model]++
 			r.nextAttempt[rep.Model] = r.now().Add(r.backoff(r.attempts[rep.Model]))
+			if n := r.attempts[rep.Model]; n >= r.cfg.ParkAfter {
+				if _, already := r.parked[rep.Model]; !already {
+					m, _ := r.cat.Load().Resolve(rep.Model)
+					r.parked[rep.Model] = parkRecord{
+						Since: r.now(), Reason: rep.Err, Failures: n, fp: modelFingerprint(m),
+					}
+					r.log.Error("model parked: it failed every time it started, and is not retried "+
+						"until its definition changes or `llmgw ctl retry` is run",
+						"model", rep.Model, "failures", n, "last_error", rep.Err)
+				}
+			}
 		}
 	}
 	r.mu.Unlock()
@@ -708,24 +656,54 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
 	// pointer stops being safe the moment it is in the map: an agent's status
 	// report can mutate it from another goroutine at any time.
 	r.mu.Lock()
-	var liveIDs, drainingIDs, avoid []string
+	var liveIDs, drainingIDs, retiringIDs, avoid []string
+	readyLive := 0
 	for _, rep := range r.replicas {
 		if rep.Model != m.Name {
 			continue
 		}
 		switch {
+		case rep.State.Counts() && !rep.RetiringAt.IsZero():
+			retiringIDs = append(retiringIDs, rep.ID)
+			avoid = append(avoid, rep.NodeIDs()...)
 		case rep.State.Counts():
 			liveIDs = append(liveIDs, rep.ID)
 			avoid = append(avoid, rep.NodeIDs()...)
+			if rep.State == Ready {
+				readyLive++
+			}
 		case rep.State == Draining:
 			drainingIDs = append(drainingIDs, rep.ID)
 		}
 	}
+	// The next restart an operator queued, once the last one is done.
+	if len(retiringIDs) == 0 {
+		if id, ok := r.nextRestartLocked(m.Name); ok {
+			r.replicas[id].RetiringAt = r.now()
+			retiringIDs = append(retiringIDs, id)
+			liveIDs = slices.DeleteFunc(liveIDs, func(x string) bool { return x == id })
+			readyLive--
+			r.log.Info("restarting a replica: placing its replacement first",
+				"replica", id, "model", m.Name)
+		}
+	}
 	blockedUntil := r.nextAttempt[m.Name]
+	park, isParked := r.parked[m.Name]
+	if isParked && park.fp != modelFingerprint(m) {
+		// Somebody changed the model — a new source, an edited profile — so
+		// the reason it kept failing may be gone. Worth one more streak.
+		delete(r.parked, m.Name)
+		r.attempts[m.Name] = 0
+		delete(r.nextAttempt, m.Name)
+		isParked, blockedUntil = false, time.Time{}
+		r.log.Info("model unparked: its definition changed", "model", m.Name)
+	}
 	r.mu.Unlock()
 
 	desired := r.desiredFor(m)
 	switch {
+	case len(liveIDs) < desired && isParked:
+		// Not placed. Its failure is on every status the operator reads.
 	case len(liveIDs) < desired:
 		if r.now().Before(blockedUntil) {
 			// Still in backoff. Rebuilding immediately would hot-loop on a
@@ -763,6 +741,8 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
 			r.stopReplica(ctx, id)
 		}
 	}
+
+	r.retire(ctx, retiringIDs, readyLive, len(liveIDs), desired)
 }
 
 // scaleUp places and starts one replica per pass.
@@ -795,7 +775,7 @@ func (r *Reconciler) scaleUp(ctx context.Context, m catalog.Model, avoid []strin
 		if m.Scaling.OnDemand() && r.makeRoomFor(ctx, m) {
 			return
 		}
-		r.log.Warn("no placement for model", "model", m.Name, "reasons", res.Explain())
+		r.notePlacementFailure(m.Name, res.Explain())
 		return
 	}
 	if err := r.startReplica(ctx, m, res.Placement); err != nil {
@@ -1006,253 +986,6 @@ func (r *Reconciler) stopReplica(ctx context.Context, id string) {
 	r.mu.Unlock()
 }
 
-// Adopt rebuilds replica state from shards an agent reports it is already
-// running (plan §6.5).
-//
-// This is what makes a control plane restart survivable. Without it the
-// reconciler sees an empty cluster, places duplicates, and the engines already
-// running keep holding GPUs the inventory believes are free — so the duplicates
-// fail to allocate, with an error naming the new process rather than the old.
-//
-// Shards whose model or profile is no longer in the catalog are returned for
-// the caller to stop: nothing will ever claim them.
-func (r *Reconciler) Adopt(nodeID string, shards []AdoptedShard) (unclaimed []AdoptedShard) {
-	defer r.forgetShardsTheAgentDoesNotHave(nodeID, shards)
-
-	byReplica := map[string][]AdoptedShard{}
-	for _, s := range shards {
-		byReplica[s.ReplicaID] = append(byReplica[s.ReplicaID], s)
-	}
-
-	r.mu.Lock()
-	defer r.mu.Unlock()
-
-	for replicaID, group := range byReplica {
-		if existing, known := r.replicas[replicaID]; known {
-			// The other half of a replica we are still assembling. Merge rather
-			// than choose: these shards belong to the same replica as the ones
-			// already adopted, they are simply on a different node whose agent
-			// registered a moment later. Treating them as a competing claim is
-			// what tore multi-node replicas apart on every control plane
-			// restart — one half adopted and failed for being short, the other
-			// stopped as unclaimed when it finally arrived.
-			if !existing.AdoptedAt.IsZero() && len(existing.Shards) < existing.WantShards {
-				r.mergeAdopted(existing, group)
-				continue
-			}
-			// Usually this is an agent reconnecting rather than a control plane
-			// restart, and its status reports keep the state current.
-			//
-			// Not always. Replica ids are derived from the model name and a
-			// counter, so a control plane that restarts and places before the
-			// agents reconnect produces a *different* replica under the same
-			// id, on a freshly allocated port. Skipping the adoption then
-			// leaves the cluster pointing at a port nothing listens on, while
-			// the engine that is actually serving is treated as a stray.
-			//
-			// The fleet is authoritative about what is running (plan §6.5), so
-			// what we merely placed loses to what an agent reports — unless our
-			// copy has a shard that has actually come up, in which case two
-			// things really are running and the one we did not ask for is the
-			// stray.
-			if existing.hasLiveShard() {
-				// Only what we do not recognise. A reconnecting agent reports
-				// the shards we already know, and disowning those would tell it
-				// to stop the very replica that is serving.
-				known := map[string]struct{}{}
-				for _, sh := range existing.Shards {
-					known[sh.ID] = struct{}{}
-				}
-				var strays []AdoptedShard
-				for _, s := range group {
-					if _, ok := known[s.ShardID]; !ok {
-						strays = append(strays, s)
-					}
-				}
-				unclaimed = append(unclaimed, strays...)
-				r.reservePorts(strays)
-				continue
-			}
-			r.log.Warn("a running replica reported by the fleet replaces one we placed",
-				"replica", replicaID, "placed_endpoint", existing.Endpoint)
-			for _, s := range existing.Shards {
-				r.inv.Release(s.ID)
-			}
-			delete(r.replicas, replicaID)
-		}
-
-		first := group[0]
-		m, ok := r.cat.Load().Resolve(first.Model)
-		if !ok {
-			unclaimed = append(unclaimed, group...)
-			r.reservePorts(group)
-			continue
-		}
-		prof, ok := m.ProfileByName(first.Profile)
-		if !ok {
-			unclaimed = append(unclaimed, group...)
-			r.reservePorts(group)
-			continue
-		}
-
-		rep := &Replica{
-			ID: replicaID, Model: m.Name, Profile: prof.Name,
-			Fabric: first.Fabric, Pool: first.Pool,
-			// From the agent, which was told it when the shard started. Empty
-			// from an agent too old to report it, which reads as unknown.
-			ProfileFP: first.ProfileFP,
-			Weight:    prof.Weight, CreatedAt: r.now(),
-			Endpoint: first.Endpoint,
-			// From the profile, not from what was reported: the point is to
-			// notice when what was reported is short.
-			WantShards: max(prof.Nodes, 1),
-			AdoptedAt:  r.now(),
-		}
-		r.mergeAdopted(rep, group)
-		r.replicas[replicaID] = rep
-
-	}
-	return unclaimed
-}
-
-// mergeAdopted folds shards an agent reports into a replica being reassembled.
-//
-// Called both for the first group, which creates the replica, and for every
-// later group belonging to it, so a replica spanning N nodes is built up over N
-// registrations. Shard ids already present are skipped: an agent that
-// reconnects repeats what it is running, and counting those again would carry a
-// replica past its shard count without a second node ever appearing.
-func (r *Reconciler) mergeAdopted(rep *Replica, group []AdoptedShard) {
-	have := make(map[string]struct{}, len(rep.Shards))
-	for _, sh := range rep.Shards {
-		have[sh.ID] = struct{}{}
-	}
-	for _, s := range group {
-		if _, dup := have[s.ShardID]; dup {
-			continue
-		}
-		have[s.ShardID] = struct{}{}
-		rep.Shards = append(rep.Shards, Shard{
-			ID: s.ShardID, NodeID: s.NodeID, Rank: s.Rank,
-			GPUIndices: s.GPUIndices, HCAs: s.HCAs,
-			HTTPPort: s.HTTPPort, State: s.State,
-		})
-		// Reclaim the GPUs and the memory, or the next placement hands them
-		// out twice. The memory comes from the catalog rather than the agent:
-		// what a profile asked for is not something a running process reports.
-		r.inv.Allocate(s.NodeID, inventory.Allocation{
-			ShardID: s.ShardID, ReplicaID: rep.ID, GPUIndices: s.GPUIndices,
-			MemoryGB: r.profileMemory(rep.Model, rep.Profile),
-		})
-		// The endpoint comes from the leader, which may be in any of the
-		// groups — the rank 0 node is not necessarily the first to register.
-		if s.Rank == 0 && s.Endpoint != "" {
-			rep.Endpoint = s.Endpoint
-		}
-	}
-	slices.SortFunc(rep.Shards, func(a, b Shard) int { return a.Rank - b.Rank })
-	rep.deriveState(r.now())
-
-	r.log.Info("adopted replica from the fleet",
-		"replica", rep.ID, "model", rep.Model,
-		"shards", len(rep.Shards), "want", rep.WantShards, "state", rep.State)
-}
-
-// forgetShardsTheAgentDoesNotHave fails shards we believe are on a node that the
-// agent, having just told us everything it is running, did not mention.
-//
-// The mirror image of adoption, and the half that was missing. Adoption handles
-// "the agent has something we do not know about"; this handles "we think the
-// agent has something it does not". Both follow from the same rule: the fleet is
-// authoritative about what is running (plan §6.5).
-//
-// Without it, an engine that dies while its agent survives leaves the control
-// plane believing a replica is Ready forever. Observed on a real cluster: the
-// engine was killed, the agent restarted and registered with nothing running,
-// and the replica stayed Ready pointing at a dead port for as long as anyone
-// watched. Traffic survived — the breaker took the endpoint out — but the
-// cluster ran a third short with no sign of it.
-//
-// Safe against the obvious race. A registration follows a fresh connection, so
-// anything we dispatched over a previous one is not running by definition;
-// there is no window where a shard is legitimately started but unreported.
-func (r *Reconciler) forgetShardsTheAgentDoesNotHave(nodeID string, running []AdoptedShard) {
-	if nodeID == "" {
-		return
-	}
-	reported := make(map[string]struct{}, len(running))
-	for _, s := range running {
-		reported[s.ShardID] = struct{}{}
-	}
-
-	var lost []string
-	r.mu.Lock()
-	for _, rep := range r.replicas {
-		if rep.State == Stopped {
-			continue
-		}
-		changed := false
-		for i := range rep.Shards {
-			sh := &rep.Shards[i]
-			if sh.NodeID != nodeID || sh.State.Terminal() {
-				continue
-			}
-			if _, ok := reported[sh.ID]; ok {
-				continue
-			}
-			sh.State = supervisor.Failed
-			sh.Err = "the agent reports it is not running"
-			r.inv.Release(sh.ID)
-			changed = true
-			lost = append(lost, sh.ID)
-		}
-		if changed {
-			rep.deriveState(r.now())
-		}
-	}
-	r.mu.Unlock()
-
-	for _, id := range lost {
-		r.log.Warn("a shard we believed was running is not; the agent does not have it",
-			"node", nodeID, "shard", id)
-	}
-}
-
-// reservePorts records ports held by shards nothing owns.
-//
-// They are being stopped, but stopping is not instant and the engine keeps its
-// socket until it exits. Handing the port to a new shard in the meantime makes
-// that shard fail to bind, which fails its replica, which places another one on
-// the same port — a loop that does not end on its own.
-//
-// Called with the lock held.
-func (r *Reconciler) reservePorts(shards []AdoptedShard) {
-	for _, s := range shards {
-		if s.HTTPPort > 0 {
-			r.reservedPorts[s.HTTPPort] = struct{}{}
-		}
-	}
-}
-
-// AdoptedShard is one running shard an agent reported at registration.
-type AdoptedShard struct {
-	NodeID     string
-	ReplicaID  string
-	ShardID    string
-	Rank       int
-	GPUIndices []int
-	HCAs       []string
-	HTTPPort   int
-	State      supervisor.State
-	Model      string
-	Profile    string
-	Pool       string
-	Fabric     string
-	ProfileFP  string
-	// Endpoint is the leader's base URL, set on the rank 0 shard.
-	Endpoint string
-}
-
 // OnFetchProgress records how far a shard's download has got.
 //
 // Separate from OnShardStatus because it is not a transition: the shard is

internal/control/reconciler/reconciler_test.go

@@ -1,6 +1,7 @@
 package reconciler
 
 import (
+	"bytes"
 	"context"
 	"fmt"
 	"log/slog"
@@ -1741,3 +1742,264 @@ func TestAnEngineIsStartedWithItsReplicasKey(t *testing.T) {
 		}
 	}
 }
+
+func replicaByID(h *harness, id string) (Replica, bool) {
+	for _, rep := range h.r.Replicas() {
+		if rep.ID == id {
+			return rep, true
+		}
+	}
+	return Replica{}, false
+}
+
+func TestARestartedReplicaServesUntilItsReplacementIsReady(t *testing.T) {
+	// Restarting is how a running engine picks up a key, an edited profile or
+	// a new engine version. Stopping first would leave a one-replica model
+	// unserved for as long as its replacement takes to load.
+	h := newHarness(t, testModel(1), gpuNode("1", 4), gpuNode("2", 4))
+	h.r.Reconcile(t.Context())
+	h.markReady()
+	old := h.r.ReadyReplicas()[0].ID
+
+	if err := h.r.Restart(old); err != nil {
+		t.Fatal(err)
+	}
+	h.r.Reconcile(t.Context())
+	if rep, _ := replicaByID(h, old); rep.State != Ready || h.r.RestartState(old) != "replacing" {
+		t.Fatalf("the old replica stopped serving before its replacement was up: %+v", rep)
+	}
+	if n := len(h.r.Replicas()); n != 2 {
+		t.Fatalf("no replacement was placed beside it: %d replicas", n)
+	}
+
+	h.markReady()
+	h.r.Reconcile(t.Context())
+	h.r.Reconcile(t.Context())
+	if _, still := replicaByID(h, old); still {
+		if rep, _ := replicaByID(h, old); rep.State != Draining && rep.State != Stopped {
+			t.Errorf("the old replica is %s after its replacement became ready", rep.State)
+		}
+	}
+	if got := len(h.r.ReadyReplicas()); got != 1 {
+		t.Errorf("%d ready replicas at the end; the model asks for 1", got)
+	}
+}
+
+func TestARestartWithNoRoomBesideItHappensInPlace(t *testing.T) {
+	// One node, room for one replica: the replacement can only start once the
+	// old one has gone. Waiting forever would mean never restarting at all.
+	h := newHarness(t, testModel(1), gpuNode("1", 4))
+	h.r.Reconcile(t.Context())
+	h.markReady()
+	old := h.r.ReadyReplicas()[0].ID
+	if err := h.r.Restart(old); err != nil {
+		t.Fatal(err)
+	}
+	h.r.Reconcile(t.Context())
+	if rep, _ := replicaByID(h, old); rep.State != Ready {
+		t.Fatalf("stopped before the grace period: %s", rep.State)
+	}
+
+	h.advance(h.r.cfg.DrainGrace + time.Second)
+	h.r.Reconcile(t.Context())
+	if rep, ok := replicaByID(h, old); ok && rep.State == Ready {
+		t.Fatal("still serving after the grace period with no replacement possible")
+	}
+	for i := 0; i < 3; i++ {
+		h.r.Reconcile(t.Context())
+		h.markReady()
+	}
+	ready := h.r.ReadyReplicas()
+	if len(ready) != 1 || ready[0].ID == old {
+		t.Errorf("expected one new replica in its place, got %+v", ready)
+	}
+}
+
+func TestReplicasOfAModelAreRestartedOneAtATime(t *testing.T) {
+	h := newHarness(t, testModel(2), gpuNode("1", 8), gpuNode("2", 8), gpuNode("3", 8))
+	for i := 0; i < 3; i++ {
+		h.r.Reconcile(t.Context())
+		h.markReady()
+	}
+	var ids []string
+	for _, rep := range h.r.ReadyReplicas() {
+		ids = append(ids, rep.ID)
+	}
+	if err := h.r.Restart(ids...); err != nil {
+		t.Fatal(err)
+	}
+	h.r.Reconcile(t.Context())
+	states := []string{h.r.RestartState(ids[0]), h.r.RestartState(ids[1])}
+	if !slices.Contains(states, "replacing") || !slices.Contains(states, "queued") {
+		t.Errorf("expected one replacing and one queued, got %v", states)
+	}
+	if err := h.r.Restart("no-such"); err == nil {
+		t.Error("an unknown replica was accepted")
+	}
+}
+
+// failRepeatedly starts and fails the model's replica n times over.
+func failRepeatedly(t *testing.T, h *harness, n int, why string) {
+	t.Helper()
+	for i := 0; i < n; i++ {
+		h.disp.reset()
+		h.r.Reconcile(t.Context())
+		started := h.disp.startedShards()
+		if len(started) == 0 {
+			return // parked, or still in backoff
+		}
+		s := started[0]
+		h.r.OnShardStatus(s.ReplicaID, s.ShardID, supervisor.Failed, why)
+		h.r.Reconcile(t.Context())
+		h.advance(DefaultConfig().MaxBackoff + time.Second)
+	}
+}
+
+func TestAModelThatAlwaysFailsIsParkedNotRetriedForever(t *testing.T) {
+	// kev-0.5b: an architecture the node's llama.cpp did not know, started
+	// 155 times in eight hours, the same error every time.
+	h := newHarness(t, testModel(1), gpuNode("1", 8))
+	failRepeatedly(t, h, DefaultConfig().ParkAfter, "unknown model architecture: 'ggmlc'")
+
+	h.disp.reset()
+	h.advance(DefaultConfig().MaxBackoff + time.Second)
+	h.r.Reconcile(t.Context())
+	if n := len(h.disp.startedShards()); n != 0 {
+		t.Fatalf("a parked model was started again (%d shards)", n)
+	}
+	var mh ModelHealth
+	for _, x := range h.r.ModelHealth() {
+		if x.Model == "llama" {
+			mh = x
+		}
+	}
+	if !mh.Parked || !strings.Contains(mh.Reason, "ggmlc") || mh.Failures < DefaultConfig().ParkAfter {
+		t.Errorf("health %+v", mh)
+	}
+	if why, no := h.r.Unservable("llama"); !no || !strings.Contains(why, "ggmlc") {
+		t.Errorf("a request would not be told why: %q %v", why, no)
+	}
+
+	// Retried by the operator once fixed.
+	if !h.r.Retry("llama") {
+		t.Error("Retry did not report the model as parked")
+	}
+	h.r.Reconcile(t.Context())
+	if len(h.disp.startedShards()) == 0 {
+		t.Error("a retried model was not placed")
+	}
+}
+
+func TestChangingAParkedModelsDefinitionTriesItAgain(t *testing.T) {
+	h := newHarness(t, testModel(1), gpuNode("1", 8))
+	failRepeatedly(t, h, DefaultConfig().ParkAfter, "boom")
+
+	fixed := testModel(1)
+	fixed.Source = "/models/llama-fixed"
+	cat, err := catalog.New([]catalog.Model{fixed})
+	if err != nil {
+		t.Fatal(err)
+	}
+	h.r.cat.Store(cat)
+	h.disp.reset()
+	h.r.Reconcile(t.Context())
+	if len(h.disp.startedShards()) == 0 {
+		t.Error("a model whose definition changed stayed parked")
+	}
+}
+
+func TestAPlacementFailureIsLoggedWhenItChangesNotEveryPass(t *testing.T) {
+	var buf bytes.Buffer
+	h := newHarness(t, testModel(1), gpuNode("1", 1)) // tp4 needs 4 cards
+	h.r.log = slog.New(slog.NewTextHandler(&buf, nil))
+	for i := 0; i < 20; i++ {
+		h.r.Reconcile(t.Context())
+		h.advance(5 * time.Second)
+	}
+	if n := strings.Count(buf.String(), "no placement for model"); n != 1 {
+		t.Errorf("logged %d times over 20 passes", n)
+	}
+	h.advance(placementLogEvery)
+	h.r.Reconcile(t.Context())
+	if n := strings.Count(buf.String(), "no placement for model"); n != 2 {
+		t.Errorf("not repeated after %s: %d", placementLogEvery, n)
+	}
+	var mh ModelHealth
+	for _, x := range h.r.ModelHealth() {
+		mh = x
+	}
+	if !mh.Unplaceable || mh.Desired != 1 {
+		t.Errorf("health %+v", mh)
+	}
+}
+
+func liveCount(h *harness) int {
+	n := 0
+	for _, r := range h.r.Replicas() {
+		if r.State.Counts() {
+			n++
+		}
+	}
+	return n
+}
+
+func TestLoadAddsReplicasUpToMaxAndQuietTakesThemBack(t *testing.T) {
+	m := testModel(1)
+	m.Scaling.Max = 3
+	h := newHarness(t, m, gpuNode("1", 8), gpuNode("2", 8), gpuNode("3", 8))
+	h.r.Reconcile(t.Context())
+	h.markReady()
+	busy := map[string]Load{"llama": {Replicas: 1, Running: 4, Waiting: 3}}
+
+	// A spike shorter than ScaleUpAfter adds nothing.
+	h.r.RecordLoad(busy)
+	h.advance(30 * time.Second)
+	h.r.RecordLoad(busy)
+	h.r.Reconcile(t.Context())
+	if n := liveCount(h); n != 1 {
+		t.Fatalf("%d replicas after 30s of load", n)
+	}
+
+	// Held past it, one more.
+	h.advance(31 * time.Second)
+	h.r.RecordLoad(busy)
+	h.r.Reconcile(t.Context())
+	if n := liveCount(h); n != 2 {
+		t.Fatalf("%d replicas after a minute of load; expected 2", n)
+	}
+
+	// Not another while that one is still starting.
+	h.advance(2 * time.Minute)
+	h.r.RecordLoad(busy)
+	h.r.Reconcile(t.Context())
+	if n := liveCount(h); n != 2 {
+		t.Fatalf("stacked a replica on one still starting: %d", n)
+	}
+
+	// Once it is up and load holds, the third — and never a fourth.
+	h.markReady()
+	for i := 0; i < 5; i++ {
+		h.r.RecordLoad(busy)
+		h.advance(61 * time.Second)
+		h.r.RecordLoad(busy)
+		h.r.Reconcile(t.Context())
+		h.markReady()
+	}
+	if n := liveCount(h); n != 3 {
+		t.Fatalf("%d replicas under sustained load; max is 3", n)
+	}
+
+	// Quiet: back down one step per ScaleDownAfter, to min and no further.
+	quiet := map[string]Load{"llama": {Replicas: 3}}
+	for i := 0; i < 6; i++ {
+		h.r.RecordLoad(quiet)
+		h.advance(DefaultConfig().ScaleDownAfter + time.Second)
+		h.r.RecordLoad(quiet)
+		for j := 0; j < 3; j++ {
+			h.r.Reconcile(t.Context())
+		}
+	}
+	if n := liveCount(h); n != 1 {
+		t.Errorf("%d replicas after a long quiet spell; min is 1", n)
+	}
+}

internal/control/reconciler/replica.go

@@ -156,6 +156,10 @@ type Replica struct {
 	// DrainingAt is when the replica stopped taking new work, so the wait for a
 	// replacement can be bounded.
 	DrainingAt time.Time
+	// RetiringAt is set while the replica is being restarted: it keeps serving
+	// but no longer counts toward the model's replicas, so a replacement is
+	// placed beside it, and it is drained only once that replacement is ready.
+	RetiringAt time.Time
 	// Attempt counts how many times this model has failed to come up in a row,
 	// carried across rebuilds so the backoff does not reset on every attempt.
 	Attempt int

internal/control/reconciler/restart.go

@@ -0,0 +1,113 @@
+package reconciler
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"slices"
+	"time"
+)
+
+// retire finishes restarting replicas.
+//
+// A retiring replica keeps serving until its replacement is ready, and is then
+// drained like any other. When no replacement could be placed at all —
+// nowhere else has the memory, which on a small CPU fleet is the usual case —
+// it is stopped after DrainGrace so its replacement can take its place: a
+// restart in place, with the model unserved while it loads if it had no other
+// replica. Said in the log, since that is the one case with an outage.
+func (r *Reconciler) retire(ctx context.Context, ids []string, readyLive, live, desired int) {
+	for _, id := range ids {
+		r.mu.Lock()
+		rep, ok := r.replicas[id]
+		var since time.Time
+		if ok {
+			since = rep.RetiringAt
+		}
+		r.mu.Unlock()
+		if !ok {
+			continue
+		}
+		switch {
+		case readyLive >= desired:
+			r.drainForRestart(ctx, id)
+		case live < desired && r.now().Sub(since) > r.cfg.DrainGrace:
+			r.log.Warn("restarting a replica in place: no room for its replacement beside it",
+				"replica", id, "after", r.cfg.DrainGrace)
+			r.drainForRestart(ctx, id)
+			r.stopReplica(ctx, id)
+		}
+	}
+}
+
+func (r *Reconciler) drainForRestart(ctx context.Context, id string) {
+	r.mu.Lock()
+	rep, ok := r.replicas[id]
+	var shards []Shard
+	if ok {
+		rep.State, rep.DrainingAt, rep.RetiringAt = Draining, r.now(), time.Time{}
+		shards = slices.Clone(rep.Shards)
+	}
+	r.mu.Unlock()
+	for _, s := range shards {
+		if err := r.disp.DrainShard(ctx, s.NodeID, s.ID); err != nil {
+			r.log.Warn("draining a shard failed", "shard", s.ID, "err", err)
+		}
+	}
+}
+
+// nextRestartLocked pops the next queued replica of a model that is still
+// ready to be restarted. Called with mu held.
+func (r *Reconciler) nextRestartLocked(model string) (string, bool) {
+	q := r.restartQueue[model]
+	for len(q) > 0 {
+		id := q[0]
+		q = q[1:]
+		if rep, ok := r.replicas[id]; ok && rep.State == Ready && rep.RetiringAt.IsZero() {
+			r.restartQueue[model] = q
+			return id, true
+		}
+	}
+	delete(r.restartQueue, model)
+	return "", false
+}
+
+// ErrNoSuchReplica means no ready replica has that id.
+var ErrNoSuchReplica = errors.New("no ready replica with that id")
+
+// Restart queues replicas to be replaced, each by a new one started beside it
+// before it is stopped — to pick up an edited profile, an engine key, a new
+// engine version. Replicas of one model are replaced one at a time.
+func (r *Reconciler) Restart(ids ...string) error {
+	r.mu.Lock()
+	for _, id := range ids {
+		rep, ok := r.replicas[id]
+		if !ok || rep.State != Ready {
+			r.mu.Unlock()
+			return fmt.Errorf("%w: %s", ErrNoSuchReplica, id)
+		}
+		if !slices.Contains(r.restartQueue[rep.Model], id) && rep.RetiringAt.IsZero() {
+			r.restartQueue[rep.Model] = append(r.restartQueue[rep.Model], id)
+		}
+	}
+	r.mu.Unlock()
+	r.Notify()
+	return nil
+}
+
+// RestartState says where a replica's restart stands: "queued", "replacing",
+// or "" when none was asked for.
+func (r *Reconciler) RestartState(id string) string {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	rep, ok := r.replicas[id]
+	switch {
+	case !ok:
+		return ""
+	case !rep.RetiringAt.IsZero():
+		return "replacing"
+	case slices.Contains(r.restartQueue[rep.Model], id):
+		return "queued"
+	}
+	return ""
+}

internal/docgen/render.go

@@ -14,7 +14,7 @@ import (
 // a new configuration file is a failure here rather than a silent omission.
 var Sources = []string{
 	"internal/config/config.go",
-	"internal/agent/agent.go",
+	"internal/agent/config.go",
 	"internal/agent/discovery/discover.go",
 	"internal/agent/engines/engine.go",
 	"internal/agent/engines/llamacpp.go",

internal/gguf/gguf.go

@@ -25,6 +25,59 @@ type Header struct {
 	Version      uint32
 	Architecture string
 	Name         string
+
+	// The shape the KV cache is sized from, from the architecture's own keys
+	// (llama.block_count, qwen2.attention.head_count_kv, ...). Zero when the
+	// file does not say, which a caller has to treat as unknown.
+	BlockCount      uint64
+	HeadCount       uint64
+	HeadCountKV     uint64
+	EmbeddingLength uint64
+	// KeyLength and ValueLength are the per-head sizes when the file gives
+	// them; otherwise a head is EmbeddingLength / HeadCount.
+	KeyLength   uint64
+	ValueLength uint64
+}
+
+// KVBytesPerToken is what one token of context costs in llama.cpp's default
+// f16 cache, or 0 when the header did not give the shape.
+//
+// Keys and values, per layer, per KV head — the KV heads rather than the
+// attention heads, because with grouped-query attention a model with forty
+// heads may keep only eight, and the difference is a factor of five.
+func (h Header) KVBytesPerToken() int64 {
+	kvHeads := h.HeadCountKV
+	if kvHeads == 0 {
+		kvHeads = h.HeadCount
+	}
+	if h.BlockCount == 0 || kvHeads == 0 {
+		return 0
+	}
+	k, v := h.KeyLength, h.ValueLength
+	if k == 0 && h.HeadCount > 0 {
+		k = h.EmbeddingLength / h.HeadCount
+	}
+	if v == 0 {
+		v = k
+	}
+	if k == 0 {
+		return 0
+	}
+	const f16 = 2
+	return int64(h.BlockCount * kvHeads * (k + v) * f16) //nolint:gosec // header-sized numbers
+}
+
+// MemoryBytes is what one llama.cpp server needs for this model: the weights,
+// a KV cache for a context pool of ctxPool tokens (llama.cpp's --ctx-size,
+// which --parallel divides among its slots rather than multiplies), and room
+// for compute buffers and the server itself. 0 when the shape is unknown.
+func (h Header) MemoryBytes(weights int64, ctxPool int) int64 {
+	perToken := h.KVBytesPerToken()
+	if perToken == 0 || weights <= 0 {
+		return 0
+	}
+	overhead := int64(1<<30) + weights/20
+	return weights + int64(ctxPool)*perToken + overhead
 }
 
 // ErrNotGGUF means the bytes are not a GGUF file.
@@ -87,16 +140,42 @@ func Read(r io.Reader) (Header, error) {
 		_, err := io.CopyN(io.Discard, br, n)
 		return err
 	}
+	// num holds the last integer value read, for the shape keys.
+	var num uint64
+	readUint := func(size int) error {
+		b := make([]byte, size)
+		if _, err := io.ReadFull(br, b); err != nil {
+			return err
+		}
+		switch size {
+		case 1:
+			num = uint64(b[0])
+		case 2:
+			num = uint64(binary.LittleEndian.Uint16(b))
+		case 4:
+			num = uint64(binary.LittleEndian.Uint32(b))
+		default:
+			num = binary.LittleEndian.Uint64(b)
+		}
+		return nil
+	}
 	var value func(t uint32) (string, error)
 	value = func(t uint32) (string, error) {
+		num = 0
 		switch t {
-		case tUint8, tInt8, tBool:
+		case tUint8, tInt8:
+			return "", readUint(1)
+		case tBool:
 			return "", skip(1)
 		case tUint16, tInt16:
-			return "", skip(2)
-		case tUint32, tInt32, tFloat32:
+			return "", readUint(2)
+		case tUint32, tInt32:
+			return "", readUint(4)
+		case tFloat32:
 			return "", skip(4)
-		case tUint64, tInt64, tFloat64:
+		case tUint64, tInt64:
+			return "", readUint(8)
+		case tFloat64:
 			return "", skip(8)
 		case tString:
 			return str()
@@ -129,18 +208,27 @@ func Read(r io.Reader) (Header, error) {
 	if err != nil {
 		return h, err
 	}
+	// Once the architecture is known, running out of the fetched bytes is
+	// not a failure: what the caller came for is in hand, and the shape keys
+	// are a bonus that a file with its tokenizer first may not reach.
+	partial := func(err error) (Header, error) {
+		if h.Architecture != "" {
+			return h, nil
+		}
+		return h, err
+	}
 	for i := uint64(0); i < kvs; i++ {
 		key, err := str()
 		if err != nil {
-			return h, err
+			return partial(err)
 		}
 		var t uint32
 		if err := binary.Read(br, binary.LittleEndian, &t); err != nil {
-			return h, err
+			return partial(err)
 		}
 		v, err := value(t)
 		if err != nil {
-			return h, err
+			return partial(err)
 		}
 		switch key {
 		case "general.architecture":
@@ -148,7 +236,25 @@ func Read(r io.Reader) (Header, error) {
 		case "general.name":
 			h.Name = v
 		}
-		if h.Architecture != "" && h.Name != "" {
+		if h.Architecture != "" {
+			switch strings.TrimPrefix(key, h.Architecture+".") {
+			case "block_count":
+				h.BlockCount = num
+			case "attention.head_count":
+				h.HeadCount = num
+			case "attention.head_count_kv":
+				h.HeadCountKV = num
+			case "embedding_length":
+				h.EmbeddingLength = num
+			case "attention.key_length":
+				h.KeyLength = num
+			case "attention.value_length":
+				h.ValueLength = num
+			}
+		}
+		// The tokenizer's arrays — megabytes of vocabulary — come after the
+		// architecture's keys, and nothing past them is wanted.
+		if strings.HasPrefix(key, "tokenizer.") && h.BlockCount > 0 {
 			break
 		}
 	}
@@ -209,5 +315,7 @@ func Check(ctx context.Context, url, llamaVersion string) Verdict {
 	if err != nil {
 		return Verdict{Servable: true, Reason: "its header could not be read: " + err.Error()}
 	}
-	return LlamaCpp(llamaVersion, h.Architecture)
+	v := LlamaCpp(llamaVersion, h.Architecture)
+	v.Header = h
+	return v
 }

internal/gguf/gguf_test.go

@@ -92,3 +92,54 @@ func TestLlamaCppKnowsWhatItCanLoad(t *testing.T) {
 }
 
 var timeZero = time.Time{}
+
+// shapedHeader is a header with the keys a KV cache is sized from, written
+// with the integer types llama.cpp uses for them.
+func shapedHeader(arch string, layers, heads, kvHeads, embd uint32, truncateTokenizer bool) []byte {
+	var b bytes.Buffer
+	w := func(v any) { _ = binary.Write(&b, binary.LittleEndian, v) }
+	str := func(s string) { w(uint64(len(s))); b.WriteString(s) }
+	u32 := func(k string, v uint32) { str(k); w(uint32(tUint32)); w(v) }
+	b.WriteString("GGUF")
+	w(uint32(3))
+	w(uint64(0))
+	w(uint64(7))
+	str("general.architecture")
+	w(uint32(tString))
+	str(arch)
+	u32(arch+".block_count", layers)
+	u32(arch+".attention.head_count", heads)
+	u32(arch+".attention.head_count_kv", kvHeads)
+	u32(arch+".embedding_length", embd)
+	u32(arch+".context_length", 32768)
+	str("tokenizer.ggml.tokens")
+	w(uint32(tArray))
+	w(uint32(tString))
+	w(uint64(1 << 20))
+	if !truncateTokenizer {
+		for i := 0; i < 1<<20; i++ {
+			str("t")
+		}
+	}
+	return b.Bytes()
+}
+
+func TestTheKVCacheIsSizedFromTheModelsShape(t *testing.T) {
+	// Qwen2.5-14B: 48 layers, 40 heads of which 8 are kept for K and V,
+	// 5120 wide — 196608 bytes a token in f16, as read from the live mirror.
+	h, err := Read(bytes.NewReader(shapedHeader("qwen2", 48, 40, 8, 5120, true)))
+	if err != nil {
+		t.Fatalf("a header cut off in its tokenizer failed to read: %v", err)
+	}
+	if got := h.KVBytesPerToken(); got != 196608 {
+		t.Errorf("%d bytes a token, expected 196608", got)
+	}
+	weights := int64(9 << 30)
+	want := weights + 16384*196608 + (1<<30 + weights/20)
+	if got := h.MemoryBytes(weights, 16384); got != want {
+		t.Errorf("memory %d, expected %d", got, want)
+	}
+	if (Header{}).MemoryBytes(weights, 16384) != 0 {
+		t.Error("a header with no shape produced a figure")
+	}
+}

internal/gguf/llamacpp.go

@@ -180,6 +180,9 @@ type Verdict struct {
 	// checked.
 	Checked bool
 	Reason  string
+	// Header is what was read, zero when nothing was; its shape sizes the
+	// model's memory.
+	Header Header
 }
 
 // LlamaCpp judges an architecture against a llama.cpp build.

internal/gguf/remote_test.go

@@ -0,0 +1,30 @@
+package gguf
+
+import (
+	"os"
+	"strings"
+	"testing"
+)
+
+// TestRealHeaders reads real files when LLMGW_GGUF_URLS names them (space
+// separated), and prints the shape and memory estimate for a 16384-token pool.
+// Skipped otherwise: CI has no mirror to read.
+func TestRealHeaders(t *testing.T) {
+	urls := strings.Fields(os.Getenv("LLMGW_GGUF_URLS"))
+	if len(urls) == 0 {
+		t.Skip("LLMGW_GGUF_URLS not set")
+	}
+	for _, u := range urls {
+		h, err := FromURL(t.Context(), nil, u)
+		if err != nil {
+			t.Errorf("%s: %v", u, err)
+			continue
+		}
+		t.Logf("%s: arch=%s layers=%d heads=%d kv_heads=%d embd=%d key_len=%d kv/token=%d B, kv@16k=%.2f GiB",
+			u[strings.LastIndex(u, "/")+1:], h.Architecture, h.BlockCount, h.HeadCount, h.HeadCountKV,
+			h.EmbeddingLength, h.KeyLength, h.KVBytesPerToken(), float64(16384*h.KVBytesPerToken())/(1<<30))
+		if h.KVBytesPerToken() == 0 {
+			t.Errorf("%s: no KV shape read", u)
+		}
+	}
+}

internal/hf/hf.go

@@ -558,3 +558,16 @@ func Group(files []File) []FileSet {
 	}
 	return out
 }
+
+// ModelName is the catalog name a repository suggests: its last element,
+// lowercased, with the -GGUF marker dropped. The owner is not part of what
+// anybody calls the model, and "-GGUF" describes the file format.
+func ModelName(repoID string) string {
+	name := repoID
+	if i := strings.LastIndex(name, "/"); i >= 0 {
+		name = name[i+1:]
+	}
+	name = strings.TrimSuffix(name, "-GGUF")
+	name = strings.TrimSuffix(name, "-gguf")
+	return strings.ToLower(name)
+}

internal/obs/trace.go

@@ -0,0 +1,72 @@
+package obs
+
+import (
+	"context"
+	"fmt"
+	"net/url"
+	"os"
+	"time"
+
+	"go.opentelemetry.io/otel"
+	"go.opentelemetry.io/otel/attribute"
+	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
+	"go.opentelemetry.io/otel/propagation"
+	"go.opentelemetry.io/otel/sdk/resource"
+	sdktrace "go.opentelemetry.io/otel/sdk/trace"
+)
+
+// TracingOptions says where traces go.
+type TracingOptions struct {
+	Endpoint    string  // http(s)://host:4318; empty reads OTEL_EXPORTER_OTLP_ENDPOINT
+	SampleRatio float64 // of traces the caller did not already decide; zero takes 1
+	Service     string
+	Version     string
+}
+
+// SetupTracing installs the global tracer provider and the W3C propagator,
+// and returns what flushes it on the way out. With no endpoint it installs
+// nothing, and every span is the no-op one: tracing that is off costs a
+// function call per span.
+func SetupTracing(ctx context.Context, o TracingOptions) (func(context.Context) error, error) {
+	noop := func(context.Context) error { return nil }
+	endpoint := o.Endpoint
+	if endpoint == "" {
+		endpoint = os.Getenv("OTEL_EXPORTER_OTLP_ENDPOINT")
+	}
+	if endpoint == "" {
+		return noop, nil
+	}
+	u, err := url.Parse(endpoint)
+	if err != nil || (u.Scheme != "http" && u.Scheme != "https") || u.Host == "" {
+		return noop, fmt.Errorf("tracing endpoint %q is not an http(s) address", endpoint)
+	}
+	opts := []otlptracehttp.Option{otlptracehttp.WithEndpoint(u.Host)}
+	if u.Scheme == "http" {
+		opts = append(opts, otlptracehttp.WithInsecure())
+	}
+	if u.Path != "" && u.Path != "/" {
+		opts = append(opts, otlptracehttp.WithURLPath(u.Path))
+	}
+	exp, err := otlptracehttp.New(ctx, opts...)
+	if err != nil {
+		return noop, err
+	}
+	ratio := o.SampleRatio
+	if ratio <= 0 || ratio > 1 {
+		ratio = 1
+	}
+	res := resource.NewSchemaless(
+		attribute.String("service.name", o.Service),
+		attribute.String("service.version", o.Version),
+	)
+	tp := sdktrace.NewTracerProvider(
+		sdktrace.WithBatcher(exp, sdktrace.WithBatchTimeout(5*time.Second)),
+		sdktrace.WithResource(res),
+		// A caller that sent a sampled traceparent has already decided; the
+		// ratio is for requests that arrive without one.
+		sdktrace.WithSampler(sdktrace.ParentBased(sdktrace.TraceIDRatioBased(ratio))),
+	)
+	otel.SetTracerProvider(tp)
+	otel.SetTextMapPropagator(propagation.TraceContext{})
+	return tp.Shutdown, nil
+}

internal/obs/trace_test.go

@@ -0,0 +1,19 @@
+package obs
+
+import (
+	"context"
+	"testing"
+)
+
+func TestTracingIsOffUntilGivenSomewhereToSend(t *testing.T) {
+	t.Setenv("OTEL_EXPORTER_OTLP_ENDPOINT", "")
+	stop, err := SetupTracing(context.Background(), TracingOptions{Service: "t"})
+	if err != nil || stop(context.Background()) != nil {
+		t.Fatal(err)
+	}
+	for _, bad := range []string{"otel:4318", "ftp://otel:4318", "http://"} {
+		if _, err := SetupTracing(context.Background(), TracingOptions{Endpoint: bad}); err == nil {
+			t.Errorf("%q accepted", bad)
+		}
+	}
+}

internal/onboard/onboard.go

@@ -18,6 +18,7 @@ import (
 	"encoding/pem"
 	"errors"
 	"fmt"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/release"
 	"net"
 	"os"
 	"path/filepath"
@@ -332,7 +333,10 @@ func certNames(path string) ([]string, error) {
 
 // --- binary ---
 
-var releaseTag = regexp.MustCompile(`^v\d+\.\d+\.\d+$`)
+// releaseTag is a version CI published: v0.10.4, or a release candidate of
+// it, v0.10.4-rc.2. Anything else — a build with commits past its tag — has no
+// release to download.
+var releaseTag = release.Tag
 
 func (r *Runner) binary(ctx context.Context, res *Result) error {
 	v := r.Opts.Version
@@ -494,14 +498,17 @@ func (r *Runner) certificate(ctx context.Context, res *Result) error {
 			return err
 		}
 	}
-	// Trusting the root too, so other tools on the machine do not treat the
-	// cluster's certificates as strangers. Best effort: either family's way.
-	_, _ = r.Remote.Run(ctx, `if [ -d /usr/local/share/ca-certificates ]; then
-  sudo install -m 0644 /etc/llmgw/ca.pem /usr/local/share/ca-certificates/llmgw-ca.crt && sudo update-ca-certificates >/dev/null 2>&1
-elif [ -d /etc/pki/ca-trust/source/anchors ]; then
-  sudo install -m 0644 /etc/llmgw/ca.pem /etc/pki/ca-trust/source/anchors/llmgw-ca.pem && sudo update-ca-trust >/dev/null 2>&1
+	// The agent trusts the cluster's root through its own configuration only.
+	// Not system-wide: that root signs whatever the control plane asks it to,
+	// and every TLS client on the machine would then take it for any site.
+	// A node onboarded by an older version had it added; take it back out.
+	_, _ = r.Remote.Run(ctx, `if [ -f /usr/local/share/ca-certificates/llmgw-ca.crt ]; then
+  sudo rm -f /usr/local/share/ca-certificates/llmgw-ca.crt && sudo update-ca-certificates --fresh >/dev/null 2>&1
+fi
+if [ -f /etc/pki/ca-trust/source/anchors/llmgw-ca.pem ]; then
+  sudo rm -f /etc/pki/ca-trust/source/anchors/llmgw-ca.pem && sudo update-ca-trust >/dev/null 2>&1
 fi; true`, nil)
-	r.emit(StepCert, Done, "certificate for "+res.Node+" installed, the cluster's root trusted", "")
+	r.emit(StepCert, Done, "certificate for "+res.Node+" installed", "")
 	return nil
 }
 

internal/onboard/releasetag_test.go

@@ -0,0 +1,14 @@
+package onboard
+
+import "testing"
+
+func TestAReleaseCandidateIsARelease(t *testing.T) {
+	for v, want := range map[string]bool{
+		"v0.10.4": true, "v0.10.4-rc.1": true, "v0.10.4-rc.12": true,
+		"v0.10.3-5-gad6d2d3": false, "v0.10.4-rc": false, "v0.10.3-a": false, "dev": false,
+	} {
+		if got := releaseTag.MatchString(v); got != want {
+			t.Errorf("%s: %v", v, got)
+		}
+	}
+}

internal/release/release.go

@@ -0,0 +1,148 @@
+// Package release fetches and installs llmgw's own release binaries.
+//
+// One implementation for everywhere a binary is replaced — the control plane
+// upgrading itself, and an agent told to upgrade by it — so the checks are
+// the same on both: the release's own checksum is verified before anything is
+// installed, the old binary is kept beside the new one, and the swap is a
+// rename, so a machine is never left without a binary that runs.
+package release
+
+import (
+	"bufio"
+	"context"
+	"crypto/sha256"
+	"encoding/hex"
+	"fmt"
+	"io"
+	"net/http"
+	"os"
+	"path/filepath"
+	"regexp"
+	"strings"
+	"time"
+)
+
+// DefaultBase is where CI publishes releases.
+const DefaultBase = "https://gitea.mynodes.xyz/ilker/llmgw/releases/download"
+
+// Tag is a version CI published: v0.10.4, or a release candidate v0.10.4-rc.2.
+var Tag = regexp.MustCompile(`^v\d+\.\d+\.\d+(-rc\.\d+)?$`)
+
+// AssetName is the release file for a platform.
+func AssetName(version, goos, goarch string) string {
+	return "llmgw-" + version + "-" + goos + "-" + goarch
+}
+
+// Install fetches version for this platform from base, checks it against the
+// release's checksums, and puts it at target, keeping the old one as
+// target.bak-pre-<version>. It returns the backup's path.
+func Install(ctx context.Context, client *http.Client, base, version, goos, goarch, target string) (string, error) {
+	if !Tag.MatchString(version) {
+		return "", fmt.Errorf("%q is not a release version", version)
+	}
+	if client == nil {
+		client = &http.Client{Timeout: 10 * time.Minute}
+	}
+	target, err := filepath.EvalSymlinks(target)
+	if err != nil {
+		return "", err
+	}
+	dir := filepath.Dir(target)
+	asset := AssetName(version, goos, goarch)
+	root := strings.TrimSuffix(base, "/") + "/" + version + "/"
+
+	want, err := checksum(ctx, client, root+"checksums.txt", asset)
+	if err != nil {
+		return "", err
+	}
+	// Beside the target, so the final rename is on one filesystem and atomic.
+	tmp, err := os.CreateTemp(dir, ".llmgw-"+version+"-*")
+	if err != nil {
+		return "", err
+	}
+	defer os.Remove(tmp.Name())
+	got, err := download(ctx, client, root+asset, tmp)
+	if cerr := tmp.Close(); err == nil {
+		err = cerr
+	}
+	if err != nil {
+		return "", err
+	}
+	if got != want {
+		return "", fmt.Errorf("%s does not match the release's checksum (got %s, expected %s)",
+			asset, got[:12], want[:12])
+	}
+	if err := os.Chmod(tmp.Name(), 0o755); err != nil {
+		return "", err
+	}
+	backup := target + ".bak-pre-" + version
+	if err := copyFile(target, backup); err != nil {
+		return "", fmt.Errorf("keeping the old binary: %w", err)
+	}
+	if err := os.Rename(tmp.Name(), target); err != nil {
+		return "", err
+	}
+	return backup, nil
+}
+
+func checksum(ctx context.Context, client *http.Client, url, asset string) (string, error) {
+	body, err := get(ctx, client, url)
+	if err != nil {
+		return "", err
+	}
+	defer body.Close()
+	sc := bufio.NewScanner(io.LimitReader(body, 1<<20))
+	for sc.Scan() {
+		f := strings.Fields(sc.Text())
+		if len(f) == 2 && f[1] == asset && len(f[0]) == 64 {
+			return f[0], nil
+		}
+	}
+	return "", fmt.Errorf("the release's checksums list no %s", asset)
+}
+
+func download(ctx context.Context, client *http.Client, url string, dst io.Writer) (string, error) {
+	body, err := get(ctx, client, url)
+	if err != nil {
+		return "", err
+	}
+	defer body.Close()
+	h := sha256.New()
+	if _, err := io.Copy(io.MultiWriter(dst, h), body); err != nil {
+		return "", fmt.Errorf("downloading %s: %w", url, err)
+	}
+	return hex.EncodeToString(h.Sum(nil)), nil
+}
+
+func get(ctx context.Context, client *http.Client, url string) (io.ReadCloser, error) {
+	req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
+	if err != nil {
+		return nil, err
+	}
+	resp, err := client.Do(req)
+	if err != nil {
+		return nil, fmt.Errorf("fetching %s: %w", url, err)
+	}
+	if resp.StatusCode != http.StatusOK {
+		resp.Body.Close()
+		return nil, fmt.Errorf("fetching %s: %s", url, resp.Status)
+	}
+	return resp.Body, nil
+}
+
+func copyFile(src, dst string) error {
+	in, err := os.Open(src) //nolint:gosec // our own binary
+	if err != nil {
+		return err
+	}
+	defer in.Close()
+	out, err := os.OpenFile(dst, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0o755) //nolint:gosec // beside it
+	if err != nil {
+		return err
+	}
+	if _, err := io.Copy(out, in); err != nil {
+		out.Close()
+		return err
+	}
+	return out.Close()
+}

internal/release/release_test.go

@@ -0,0 +1,61 @@
+package release
+
+import (
+	"crypto/sha256"
+	"encoding/hex"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"testing"
+)
+
+func releaseServer(t *testing.T, version, asset string, body []byte, sum string) *httptest.Server {
+	t.Helper()
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		switch r.URL.Path {
+		case "/" + version + "/checksums.txt":
+			_, _ = w.Write([]byte(sum + "  " + asset + "\n"))
+		case "/" + version + "/" + asset:
+			_, _ = w.Write(body)
+		default:
+			http.NotFound(w, r)
+		}
+	}))
+	t.Cleanup(srv.Close)
+	return srv
+}
+
+func TestABinaryIsReplacedOnlyWithWhatTheReleasePublished(t *testing.T) {
+	dir := t.TempDir()
+	target := filepath.Join(dir, "llmgw")
+	_ = os.WriteFile(target, []byte("old"), 0o755)
+	body := []byte("new binary")
+	sum := sha256.Sum256(body)
+	asset := AssetName("v1.2.3", "linux", "amd64")
+
+	srv := releaseServer(t, "v1.2.3", asset, body, hex.EncodeToString(sum[:]))
+	backup, err := Install(t.Context(), srv.Client(), srv.URL, "v1.2.3", "linux", "amd64", target)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if b, _ := os.ReadFile(target); string(b) != "new binary" {
+		t.Errorf("target holds %q", b)
+	}
+	if b, _ := os.ReadFile(backup); string(b) != "old" {
+		t.Errorf("the old binary was not kept: %q", b)
+	}
+
+	// A download that does not match its checksum changes nothing.
+	bad := releaseServer(t, "v1.2.4", AssetName("v1.2.4", "linux", "amd64"), []byte("tampered"),
+		hex.EncodeToString(sum[:]))
+	if _, err := Install(t.Context(), bad.Client(), bad.URL, "v1.2.4", "linux", "amd64", target); err == nil {
+		t.Fatal("a binary not matching its checksum was installed")
+	}
+	if b, _ := os.ReadFile(target); string(b) != "new binary" {
+		t.Errorf("a failed install touched the binary: %q", b)
+	}
+	if _, err := Install(t.Context(), nil, srv.URL, "../../x", "linux", "amd64", target); err == nil {
+		t.Error("a version that is not one was fetched")
+	}
+}

internal/router/activate.go

@@ -17,6 +17,22 @@ type Activator interface {
 	Want(model string) bool
 }
 
+// unservable is an activator that can also say a model will not come up at
+// all — it failed every time it started, or fits on no node — so a request is
+// told so at once rather than asked to retry something that never arrives.
+type unservable interface {
+	Unservable(model string) (string, bool)
+}
+
+// whyUnservable is the activator's reason a model cannot be served, if any.
+func (r *Router) whyUnservable(model string) (string, bool) {
+	u, ok := r.activator.(unservable)
+	if !ok {
+		return "", false
+	}
+	return u.Unservable(model)
+}
+
 // SetActivator supplies the control plane that can load models on demand.
 func (r *Router) SetActivator(a Activator) { r.activator = a }
 
@@ -32,6 +48,8 @@ func (r *Router) SetActivator(a Activator) { r.activator = a }
 // a slow request. A fifteen-minute wait is a job pretending to be one, and the
 // client should be told to come back rather than left holding a socket.
 func (r *Router) activate(ctx context.Context, model string) bool {
+	// Asked first, always: asking is what makes the control plane try again,
+	// and a model that fitted nowhere an hour ago may fit now.
 	if r.activator == nil || !r.activator.Want(model) {
 		return false
 	}
@@ -56,6 +74,11 @@ func (r *Router) activate(ctx context.Context, model string) bool {
 			if len(r.reg.Snapshots(model)) > 0 {
 				return true
 			}
+			// Asking set a placement going; if it found nowhere, there is
+			// nothing to wait for.
+			if _, no := r.whyUnservable(model); no {
+				return false
+			}
 		}
 	}
 }

internal/router/admission/admission_test.go

@@ -42,14 +42,20 @@ func newController(t *testing.T, cfg Config, sat SaturationFunc) *Controller {
 
 func TestAdmitsImmediatelyWhenIdle(t *testing.T) {
 	s := &fixedSat{}
-	c := newController(t, testConfig(), s.fn)
+	// A queue that is looked at every ten seconds, so a request that went
+	// through it rather than the fast path would take that long. A bound of
+	// 50ms against a 1ms tick measured the machine the test ran on, and a
+	// loaded CI runner under the race detector was slower than that.
+	cfg := testConfig()
+	cfg.Interval = 10 * time.Second
+	c := newController(t, cfg, s.fn)
 
 	start := time.Now()
 	lease, err := c.Acquire(t.Context(), Request{Model: "m", Band: "bronze"})
 	if err != nil {
 		t.Fatal(err)
 	}
-	if d := time.Since(start); d > 50*time.Millisecond {
+	if d := time.Since(start); d > 2*time.Second {
 		t.Errorf("idle admission took %v; the fast path is not being used", d)
 	}
 	lease.Release()
@@ -131,8 +137,13 @@ func TestQueuedRequestAdmittedWhenSaturationDrops(t *testing.T) {
 		done <- err
 	}()
 
-	// Give it time to actually queue, then open the gate.
-	time.Sleep(50 * time.Millisecond)
+	// Wait for it to actually queue, then open the gate. Waited for, not
+	// slept through: on a loaded runner 50ms was sometimes not long enough
+	// for the goroutine to have been scheduled at all.
+	deadline := time.Now().Add(2 * time.Second)
+	for c.Depths()[3].Depth != 1 && time.Now().Before(deadline) {
+		time.Sleep(5 * time.Millisecond)
+	}
 	if d := c.Depths(); d[3].Depth != 1 {
 		t.Fatalf("expected 1 queued bronze request, got %+v", d)
 	}

internal/router/finish_test.go

@@ -0,0 +1,119 @@
+package router
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"net/http"
+	"net/http/httptest"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/accounting"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/router/admission"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/router/proxy"
+)
+
+// What a client is told for each way a dispatch can end. Each status is
+// something a client acts on: 503 with Retry-After is "try again", 503 alone
+// is "not here", 502 is "the engine failed", and a stream already under way
+// cannot be given a status at all.
+func TestEachWayADispatchEndsIsToldToTheClientAsItShouldBe(t *testing.T) {
+	rt := newTestRouter(t, baseConfig())
+	for _, c := range []struct {
+		name       string
+		res        dispatchResult
+		status     int
+		body       bool // an error body was written
+		retryAfter bool
+	}{
+		{"served", dispatchResult{outcome: proxy.Outcome{Status: 200, Wrote: true}}, 200, false, false},
+		{"engine refused, relayed", dispatchResult{outcome: proxy.Outcome{Status: 400, Wrote: true}}, 400, false, false},
+		{"client went away", dispatchResult{err: proxy.ErrClientGone}, 499, false, false},
+		{"stream cut short", dispatchResult{outcome: proxy.Outcome{Status: 200, Wrote: true}, err: errors.New("eof")}, 200, false, false},
+		{"stream cut before a status", dispatchResult{outcome: proxy.Outcome{Wrote: true}, err: errors.New("eof")}, 200, false, false},
+		{"still loading", dispatchResult{err: fmt.Errorf("%w: retry shortly", errLoading)}, 503, true, true},
+		{"nothing serves it", dispatchResult{err: fmt.Errorf("%w: no replica", errNoCapacity)}, 503, true, false},
+		{"engine failed", dispatchResult{err: errors.New("connection reset")}, 502, true, false},
+	} {
+		t.Run(c.name, func(t *testing.T) {
+			w := httptest.NewRecorder()
+			got := rt.finish(w, &parsedRequest{Model: "m"}, c.res, time.Now())
+			if got != c.status {
+				t.Errorf("status %d, expected %d", got, c.status)
+			}
+			if c.body {
+				if w.Code != c.status {
+					t.Errorf("written %d, expected %d", w.Code, c.status)
+				}
+				var e struct {
+					Error struct {
+						Message string `json:"message"`
+					} `json:"error"`
+				}
+				if err := json.Unmarshal(w.Body.Bytes(), &e); err != nil || e.Error.Message == "" {
+					t.Errorf("error body %q", w.Body.String())
+				}
+			} else if w.Body.Len() != 0 {
+				t.Errorf("wrote %q over a response that was not ours to write", w.Body.String())
+			}
+			if (w.Header().Get("Retry-After") != "") != c.retryAfter {
+				t.Errorf("Retry-After %q", w.Header().Get("Retry-After"))
+			}
+		})
+	}
+}
+
+func TestAnUpstreamStatusIsKeptWhenTheStreamBreaks(t *testing.T) {
+	rt := newTestRouter(t, baseConfig())
+	w := httptest.NewRecorder()
+	res := dispatchResult{outcome: proxy.Outcome{Status: http.StatusTooManyRequests, Wrote: true}, err: errors.New("eof")}
+	if got := rt.finish(w, &parsedRequest{Model: "m"}, res, time.Now()); got != 429 {
+		t.Errorf("status %d", got)
+	}
+}
+
+func TestARefusalBeforeDispatchSaysWhetherWaitingHelps(t *testing.T) {
+	rt := newTestRouter(t, baseConfig())
+	for _, c := range []struct {
+		name       string
+		refuse     func(http.ResponseWriter)
+		status     int
+		retryAfter string
+	}{
+		{"queue full", func(w http.ResponseWriter) {
+			rt.rejectAdmission(w, &parsedRequest{Model: "m"}, admission.ErrQueueFull, time.Now())
+		}, 503, ""},
+		{"queued too long", func(w http.ResponseWriter) {
+			rt.rejectAdmission(w, &parsedRequest{Model: "m"}, admission.ErrQueueTimeout, time.Now())
+		}, 504, ""},
+		{"shutting down", func(w http.ResponseWriter) {
+			rt.rejectAdmission(w, &parsedRequest{Model: "m"}, admission.ErrClosed, time.Now())
+		}, 503, ""},
+		{"client left the queue", func(w http.ResponseWriter) {
+			rt.rejectAdmission(w, &parsedRequest{Model: "m"}, context.Canceled, time.Now())
+		}, 200, ""}, // nothing written: nobody is listening
+		{"rate limited", func(w http.ResponseWriter) {
+			rt.rejectLimit(w, &parsedRequest{Model: "m"},
+				&accounting.LimitError{Err: accounting.ErrRateLimited, RetryAfter: 3 * time.Second}, time.Now())
+		}, 429, "3"},
+		{"budget spent", func(w http.ResponseWriter) {
+			rt.rejectLimit(w, &parsedRequest{Model: "m"}, &accounting.LimitError{Err: accounting.ErrBudgetSpent}, time.Now())
+		}, 429, ""},
+		{"not a limit at all", func(w http.ResponseWriter) {
+			rt.rejectLimit(w, &parsedRequest{Model: "m"}, errors.New("store down"), time.Now())
+		}, 500, ""},
+	} {
+		t.Run(c.name, func(t *testing.T) {
+			w := httptest.NewRecorder()
+			c.refuse(w)
+			if w.Code != c.status {
+				t.Errorf("status %d, expected %d", w.Code, c.status)
+			}
+			if got := w.Header().Get("Retry-After"); got != c.retryAfter {
+				t.Errorf("Retry-After %q, expected %q", got, c.retryAfter)
+			}
+		})
+	}
+}

internal/router/handlers.go

@@ -3,6 +3,8 @@ package router
 import (
 	"bytes"
 	"context"
+	"crypto/rand"
+	"encoding/hex"
 	"encoding/json"
 	"errors"
 	"fmt"
@@ -13,6 +15,9 @@ import (
 	"strings"
 	"time"
 
+	"go.opentelemetry.io/otel/attribute"
+	"go.opentelemetry.io/otel/codes"
+
 	"gitea.mynodes.xyz/ilker/llmgw/internal/accounting"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/auth"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/router/admission"
@@ -51,7 +56,7 @@ func (r *Router) rejectAdmission(w http.ResponseWriter, req *parsedRequest, err
 	case errors.Is(err, context.Canceled), errors.Is(err, context.DeadlineExceeded):
 		// The client gave up while queued; nobody is listening.
 		r.m.observeAdmissionRejected(req.Band, "client_gone")
-		r.m.observeRequest(req.Model, "", 499, time.Since(start), nil, 0)
+		r.m.observeRequest(r.metricModel(req.Model), "", 499, time.Since(start), nil, 0)
 		return
 	case errors.Is(err, admission.ErrQueueTimeout):
 		status = http.StatusGatewayTimeout
@@ -67,7 +72,8 @@ func (r *Router) rejectAdmission(w http.ResponseWriter, req *parsedRequest, err
 	}
 	r.m.observeAdmissionRejected(req.Band, reason)
 	writeError(w, status, "server_error", err.Error())
-	r.m.observeRequest(req.Model, "", status, time.Since(start), nil, 0)
+	r.m.observeRequest(r.metricModel(req.Model), "", status, time.Since(start), nil, 0)
+	r.accessLog(req, dispatchResult{err: err}, status, time.Since(start))
 }
 
 // prefixBytes is how much of the prompt is kept for prefix-aware routing.
@@ -224,6 +230,12 @@ type parsedRequest struct {
 	UpstreamBody       []byte
 	Header             http.Header
 
+	// RequestID follows the request through the access log, the usage record
+	// and the engine, and goes back to the client in X-Request-Id.
+	RequestID string
+	// QueueMs is how long admission held it.
+	QueueMs int64
+
 	SessionID string
 	Band      string
 	// KeyID, ProjectID and Tenant are empty when authentication is off.
@@ -275,8 +287,23 @@ type inferenceBody struct {
 func (r *Router) handleInference(path string) http.HandlerFunc {
 	return func(w http.ResponseWriter, req *http.Request) {
 		start := time.Now()
+		// First, so even a request refused as malformed can be named when
+		// somebody asks what happened to it.
+		rid := requestID(req.Header.Get("X-Request-Id"))
+		w.Header().Set("X-Request-Id", rid)
+		ctx, span := startInference(req, path, rid)
+		defer span.End()
+		req = req.WithContext(ctx)
 
 		parsed, err := r.parse(req, path)
+		if err == nil {
+			span.SetAttributes(attribute.String("llmgw.model", parsed.Model))
+			parsed.RequestID = rid
+			if parsed.Header == nil {
+				parsed.Header = http.Header{}
+			}
+			parsed.Header.Set("X-Request-Id", rid)
+		}
 		if err != nil {
 			writeError(w, http.StatusBadRequest, "invalid_request_error", err.Error())
 			r.m.observeRequest("", "", http.StatusBadRequest, time.Since(start), nil, 0)
@@ -289,7 +316,7 @@ func (r *Router) handleInference(path string) http.HandlerFunc {
 		if _, entry := r.resolve(parsed.Model); entry.ContextLength > 0 {
 			if why := checkContextFits(entry, parsed.PromptTokens, parsed.MaxTokens); why != "" {
 				writeError(w, http.StatusBadRequest, "invalid_request_error", why)
-				r.m.observeRequest(parsed.Model, "", http.StatusBadRequest, time.Since(start), nil, 0)
+				r.m.observeRequest(r.metricModel(parsed.Model), "", http.StatusBadRequest, time.Since(start), nil, 0)
 				return
 			}
 		}
@@ -297,7 +324,7 @@ func (r *Router) handleInference(path string) http.HandlerFunc {
 		if p, ok := auth.FromContext(req.Context()); ok {
 			if !r.permits(p, parsed.Model) {
 				auth.WriteForbiddenModel(w, parsed.Model)
-				r.m.observeRequest(parsed.Model, "", http.StatusForbidden, time.Since(start), nil, 0)
+				r.m.observeRequest(r.metricModel(parsed.Model), "", http.StatusForbidden, time.Since(start), nil, 0)
 				return
 			}
 			// Limits are checked before admission, not after: a request that is
@@ -323,6 +350,9 @@ func (r *Router) handleInference(path string) http.HandlerFunc {
 						w.Header().Set("x-llmgw-served-model", parsed.Model)
 					}
 				}
+				// On every answer, refused or not, so a client can pace itself
+				// before it is refused rather than learn the limit from a 429.
+				r.rateLimitHeaders(w, p)
 				if err != nil {
 					r.rejectLimit(w, parsed, err, start)
 					return
@@ -332,14 +362,17 @@ func (r *Router) handleInference(path string) http.HandlerFunc {
 		}
 
 		// Step 5 of the lifecycle: "now?" before "where?" (plan §7.1).
+		_, aspan := tracer().Start(req.Context(), "admission")
 		lease, err := r.admit(req.Context(), parsed)
+		endSpan(aspan, err)
 		if err != nil {
 			r.rejectAdmission(w, parsed, err, start)
 			return
 		}
 		defer lease.Release()
 		queued := time.Since(start)
-		r.m.observeAdmission(lease.Band(), parsed.Model, queued)
+		parsed.QueueMs = queued.Milliseconds()
+		r.m.observeAdmission(lease.Band(), r.metricModel(parsed.Model), queued)
 		if queued > time.Millisecond {
 			w.Header().Set("x-llmgw-queue-ms", strconv.FormatInt(queued.Milliseconds(), 10))
 		}
@@ -349,7 +382,7 @@ func (r *Router) handleInference(path string) http.HandlerFunc {
 		// model that will actually serve, not the one that was asked for.
 		if dropped, err := r.loadThread(req.Context(), parsed); err != nil {
 			writeError(w, http.StatusBadRequest, "invalid_request_error", err.Error())
-			r.m.observeRequest(parsed.Model, "", http.StatusBadRequest, time.Since(start), nil, 0)
+			r.m.observeRequest(r.metricModel(parsed.Model), "", http.StatusBadRequest, time.Since(start), nil, 0)
 			return
 		} else if dropped > 0 {
 			// Dropping turns is the right answer and must not be a silent one.
@@ -357,7 +390,13 @@ func (r *Router) handleInference(path string) http.HandlerFunc {
 		}
 
 		res := r.dispatch(req.Context(), w, parsed)
-		r.finish(w, parsed, res, start)
+		status := r.finish(w, parsed, res, start)
+		span.SetAttributes(attribute.Int("http.response.status_code", status),
+			attribute.String("llmgw.served_model", res.model),
+			attribute.String("llmgw.endpoint", res.endpointID))
+		if status >= 500 {
+			span.SetStatus(codes.Error, http.StatusText(status))
+		}
 		// After the client has been served. A conversation that failed to
 		// record is a worse outcome than a slow reply, but only just, and the
 		// caller has their answer either way.
@@ -389,7 +428,8 @@ func (r *Router) permits(p auth.Principal, requested string) bool {
 }
 
 // finish records metrics and, when nothing was written, produces the error body.
-func (r *Router) finish(w http.ResponseWriter, req *parsedRequest, res dispatchResult, start time.Time) {
+// It returns the status the request ended with.
+func (r *Router) finish(w http.ResponseWriter, req *parsedRequest, res dispatchResult, start time.Time) int {
 	status := res.outcome.Status
 	switch {
 	case res.err == nil:
@@ -420,8 +460,10 @@ func (r *Router) finish(w http.ResponseWriter, req *parsedRequest, res dispatchR
 		writeError(w, status, "server_error", res.err.Error())
 	}
 
-	r.m.observeRequest(req.Model, res.pool, status, time.Since(start), res.outcome.Usage, res.outcome.TTFT)
+	r.m.observeRequest(r.metricModel(req.Model), res.pool, status, time.Since(start), res.outcome.Usage, res.outcome.TTFT)
 	r.account(req, res, status, time.Since(start))
+	r.accessLog(req, res, status, time.Since(start))
+	return status
 }
 
 // account settles a finished request against its key.
@@ -437,6 +479,8 @@ func (r *Router) account(req *parsedRequest, res dispatchResult, status int, too
 		At: time.Now(), KeyID: req.KeyID, UserID: req.UserID, ProjectID: req.ProjectID,
 		Model: req.Model, ServedModel: res.model, Status: status,
 		LatencyMs: took.Milliseconds(),
+		RequestID: req.RequestID, Replica: res.endpointID,
+		QueueMs: req.QueueMs, TTFTMs: res.outcome.TTFT.Milliseconds(),
 	}
 	if u.ServedModel == "" {
 		u.ServedModel = req.Model
@@ -467,7 +511,8 @@ func (r *Router) rejectLimit(w http.ResponseWriter, req *parsedRequest, err erro
 	// already know how to surface that code. What differs is Retry-After,
 	// which is absent when waiting does not help.
 	writeError(w, http.StatusTooManyRequests, "rate_limit_exceeded", le.Error())
-	r.m.observeRequest(req.Model, "", http.StatusTooManyRequests, time.Since(start), nil, 0)
+	r.m.observeRequest(r.metricModel(req.Model), "", http.StatusTooManyRequests, time.Since(start), nil, 0)
+	r.accessLog(req, dispatchResult{err: err}, http.StatusTooManyRequests, time.Since(start))
 }
 
 // parse reads the body and extracts the routing-relevant fields.
@@ -854,3 +899,77 @@ func (r *Router) handleMe(w http.ResponseWriter, req *http.Request) {
 	}
 	writeJSON(w, http.StatusOK, out)
 }
+
+// requestID is the client's X-Request-Id when it is one worth keeping, or a
+// new one. Kept only when short and plain: it is written into logs and a
+// header, and a client-chosen string with newlines or megabytes in it is not
+// an id.
+func requestID(sent string) string {
+	if n := len(sent); n > 0 && n <= 128 && strings.IndexFunc(sent, func(c rune) bool {
+		return !(c >= 'a' && c <= 'z' || c >= 'A' && c <= 'Z' || c >= '0' && c <= '9' ||
+			c == '-' || c == '_' || c == '.' || c == ':')
+	}) < 0 {
+		return sent
+	}
+	var b [12]byte
+	_, _ = rand.Read(b[:])
+	return "req_" + hex.EncodeToString(b[:])
+}
+
+// accessLog writes one line per finished request: enough to follow a slow or
+// failed request from the id a client reports to the replica that served it,
+// and where its time went. Before this the only per-request lines were for
+// disconnects, early stream ends and upstream retries.
+func (r *Router) accessLog(req *parsedRequest, res dispatchResult, status int, took time.Duration) {
+	attrs := []any{
+		"request_id", req.RequestID, "status", status, "model", req.Model,
+		"duration_ms", took.Milliseconds(), "queue_ms", req.QueueMs, "stream", req.Stream,
+	}
+	if req.RequestedModel != "" && req.RequestedModel != req.Model {
+		attrs = append(attrs, "requested_model", req.RequestedModel)
+	}
+	if res.model != "" && res.model != req.Model {
+		attrs = append(attrs, "served_model", res.model)
+	}
+	if res.endpointID != "" {
+		attrs = append(attrs, "replica", res.endpointID)
+	}
+	if res.outcome.TTFT > 0 {
+		attrs = append(attrs, "ttft_ms", res.outcome.TTFT.Milliseconds())
+	}
+	if u := res.outcome.Usage; u != nil {
+		attrs = append(attrs, "prompt_tokens", u.PromptTokens, "completion_tokens", u.CompletionTokens)
+	}
+	if req.KeyID != "" {
+		attrs = append(attrs, "key", req.KeyID, "project", req.ProjectID)
+	}
+	if req.UserID != "" {
+		attrs = append(attrs, "user", req.UserID)
+	}
+	if res.err != nil {
+		attrs = append(attrs, "err", res.err.Error())
+	}
+	r.log.Info("request", attrs...)
+}
+
+// rateLimitHeaders sets OpenAI's x-ratelimit-* headers from the tightest
+// limit that applies. SDKs and editors read these to slow down before they
+// are refused; only Retry-After, and only on a refusal, was sent before. A
+// limit that is not set is not advertised.
+func (r *Router) rateLimitHeaders(w http.ResponseWriter, p auth.Principal) {
+	if r.acct == nil {
+		return
+	}
+	h := r.acct.Headroom(p)
+	reset := strconv.FormatFloat(h.Reset.Seconds(), 'f', 0, 64) + "s"
+	if h.RequestLimit > 0 {
+		w.Header().Set("x-ratelimit-limit-requests", strconv.Itoa(h.RequestLimit))
+		w.Header().Set("x-ratelimit-remaining-requests", strconv.Itoa(h.RequestsLeft))
+		w.Header().Set("x-ratelimit-reset-requests", reset)
+	}
+	if h.TokenLimit > 0 {
+		w.Header().Set("x-ratelimit-limit-tokens", strconv.Itoa(h.TokenLimit))
+		w.Header().Set("x-ratelimit-remaining-tokens", strconv.Itoa(h.TokensLeft))
+		w.Header().Set("x-ratelimit-reset-tokens", reset)
+	}
+}

internal/router/metrics.go

@@ -151,3 +151,17 @@ func (m *metrics) observeScores(d scheduler.Decision) {
 	}
 	m.considered.Observe(float64(d.Considered))
 }
+
+// metricModel is the model label a request is counted under: the catalog's
+// name for it, or "unknown". Taken as sent, any key holder could mint a new
+// series per request — a refused request counted its made-up model name —
+// and grow Prometheus without bound.
+func (r *Router) metricModel(name string) string {
+	if name == "" {
+		return ""
+	}
+	if m, ok := r.modelMap()[name]; ok {
+		return m.Name
+	}
+	return "unknown"
+}

internal/router/pricing_test.go

@@ -140,3 +140,51 @@ func waitForUsage(t *testing.T, rt *Router) store.KeyUsage {
 	t.Fatal("no usage was recorded")
 	return store.KeyUsage{}
 }
+
+func TestARequestCanBeFollowedByItsID(t *testing.T) {
+	// "Why was my request slow?" had no answer: the queue time went out in a
+	// header and nowhere else, and nothing tied a request to its replica.
+	e := newFakeEngine(t, "qwen2.5-14b-instruct")
+	rt, secret := pricedRouter(t, e.srv.URL)
+
+	rec := post(t, rt.Handler(),
+		map[string]any{"model": "q14", "messages": []any{
+			map[string]any{"role": "user", "content": "hello"}}},
+		map[string]string{"Authorization": "Bearer " + secret, "X-Request-Id": "client-abc.1"})
+	if rec.Code != 200 {
+		t.Fatalf("status %d: %s", rec.Code, rec.Body.String())
+	}
+	if got := rec.Header().Get("X-Request-Id"); got != "client-abc.1" {
+		t.Errorf("the client's id came back as %q", got)
+	}
+
+	deadline := time.Now().Add(5 * time.Second)
+	for {
+		u, ok, err := rt.store.UsageByRequest(t.Context(), "client-abc.1")
+		if err != nil {
+			t.Fatal(err)
+		}
+		if ok {
+			if u.Replica == "" || u.Status != 200 {
+				t.Errorf("recorded %+v; expected the replica that answered", u)
+			}
+			break
+		}
+		if time.Now().After(deadline) {
+			t.Fatal("the request's usage cannot be found by its id")
+		}
+		time.Sleep(20 * time.Millisecond)
+	}
+}
+
+func TestAnIDThatIsNotOneIsReplaced(t *testing.T) {
+	for _, bad := range []string{"", "has space", "new\nline", strings.Repeat("x", 200)} {
+		got := requestID(bad)
+		if got == bad || !strings.HasPrefix(got, "req_") {
+			t.Errorf("%q became %q", bad, got)
+		}
+	}
+	if requestID("req_1") != "req_1" {
+		t.Error("a plain id was not kept")
+	}
+}

internal/router/proxy/proxy.go

@@ -526,7 +526,7 @@ func copyHeaders(dst, src http.Header) {
 // llmgw, not llmgw to the engine. llmgw's own key for the engine is set by Do.
 func cloneForUpstream(h http.Header) http.Header {
 	out := make(http.Header, 4)
-	for _, k := range []string{"Accept", "User-Agent", "Traceparent", "Tracestate"} {
+	for _, k := range []string{"Accept", "User-Agent", "Traceparent", "Tracestate", "X-Request-Id"} {
 		if v := h.Get(k); v != "" {
 			out.Set(k, v)
 		}

internal/router/router.go

@@ -11,19 +11,24 @@ import (
 	"fmt"
 	"log/slog"
 	"math"
+	"net"
 	"net/http"
+	"net/netip"
 	"slices"
 	"strings"
 	"sync/atomic"
 	"time"
 
 	"github.com/prometheus/client_golang/prometheus"
+	"go.opentelemetry.io/otel/attribute"
+	"go.opentelemetry.io/otel/trace"
 
 	"gitea.mynodes.xyz/ilker/llmgw/internal/account"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/accounting"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/auth"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/router/admission"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/router/breaker"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint"
@@ -230,6 +235,11 @@ func New(cfg config.Server, log *slog.Logger) (*Router, error) {
 		Timeout:  cfg.Router.ScrapeTimeout,
 	}, log)
 
+	if !cfg.Auth.Enabled && !loopbackOnly(cfg.Listen) {
+		log.Warn("auth is off and the API listens beyond this machine: anyone who can reach "+
+			cfg.Listen+" can use every model; set auth.enabled, or listen on 127.0.0.1",
+			"listen", cfg.Listen)
+	}
 	if cfg.Auth.Enabled {
 		st, err := storepkg.Open(context.Background(), cfg.Auth.Store)
 		if err != nil {
@@ -510,6 +520,24 @@ func (r *Router) SetCatalog(c *catalog.Catalog) {
 // holding its cards out of habit.
 func (r *Router) LastUsed() map[string]time.Time { return r.reg.LastUsed() }
 
+// ModelLoad sums what each model's replicas report: how many there are, and
+// requests running and waiting at their engines. Keyed by the catalog name,
+// which the control plane publishes first among each endpoint's names.
+func (r *Router) ModelLoad() map[string]reconciler.Load {
+	out := map[string]reconciler.Load{}
+	for _, s := range r.reg.Snapshots("") {
+		if len(s.Models) == 0 {
+			continue
+		}
+		l := out[s.Models[0]]
+		l.Replicas++
+		l.Running += s.Stats.Running
+		l.Waiting += s.Stats.Waiting
+		out[s.Models[0]] = l
+	}
+	return out
+}
+
 func (r *Router) SetEndpoints(ctx context.Context, infos []endpoint.Info) {
 	if r.prefix != nil {
 		// A replica that has gone takes its cache with it. Leaving its entries
@@ -629,6 +657,11 @@ func (r *Router) dispatchOne(ctx context.Context, w http.ResponseWriter, req *pa
 		}
 	}
 	if len(candidates) == 0 {
+		if why, no := r.whyUnservable(model); no && r.activator != nil {
+			// Not "loading; retry": it will not load, and a client told to
+			// retry keeps retrying something that never arrives.
+			return dispatchResult{err: fmt.Errorf("%w: %s", errNoCapacity, why)}
+		}
 		if r.activator != nil && r.activator.Want(model) {
 			// Asked for, still loading. Saying "no endpoint serves this" would
 			// be a lie, and a client told to retry will succeed where one told
@@ -658,7 +691,13 @@ func (r *Router) dispatchOne(ctx context.Context, w http.ResponseWriter, req *pa
 		if len(candidates) == 0 {
 			break
 		}
+		_, sspan := tracer().Start(ctx, "schedule", trace.WithAttributes(
+			attribute.String("llmgw.model", model), attribute.Int("llmgw.candidates", len(candidates))))
 		dec, err := profile.Schedule(sctx, candidates)
+		if err == nil {
+			sspan.SetAttributes(attribute.String("llmgw.endpoint", dec.Endpoint.ID))
+		}
+		endSpan(sspan, err)
 		if err != nil {
 			last = dispatchResult{err: fmt.Errorf("%w: %v", errNoCapacity, err)}
 			break
@@ -672,15 +711,20 @@ func (r *Router) dispatchOne(ctx context.Context, w http.ResponseWriter, req *pa
 			continue
 		}
 
-		out, perr := r.px.Do(ctx, w, proxy.Request{
+		uctx, uspan := tracer().Start(ctx, "upstream", trace.WithSpanKind(trace.SpanKindClient),
+			trace.WithAttributes(attribute.String("llmgw.endpoint", dec.Endpoint.ID),
+				attribute.String("url.full", dec.Endpoint.URL+req.Path)))
+		out, perr := r.px.Do(uctx, w, proxy.Request{
 			URL:                dec.Endpoint.URL + req.Path,
 			APIKey:             dec.Endpoint.APIKey,
 			Body:               req.UpstreamBody,
-			Header:             req.Header,
+			Header:             withTraceparent(uctx, req.Header),
 			Stream:             req.Stream,
 			SuppressUsageChunk: req.SuppressUsageChunk,
 			CaptureReply:       req.captureBytes(),
 		})
+		uspan.SetAttributes(attribute.Int("http.response.status_code", out.Status))
+		endSpan(uspan, perr)
 		// A retryable failure is the endpoint's fault; a client disconnect is not.
 		release(perr == nil || errors.Is(perr, proxy.ErrClientGone))
 
@@ -740,3 +784,17 @@ func personalLimits(p config.Personal) *storepkg.Limits {
 	}
 	return &l
 }
+
+// loopbackOnly reports whether a listen address is reachable from this
+// machine alone. An empty host (":8080") is every interface.
+func loopbackOnly(listen string) bool {
+	host, _, err := net.SplitHostPort(listen)
+	if err != nil {
+		return false
+	}
+	if host == "localhost" {
+		return true
+	}
+	a, err := netip.ParseAddr(host)
+	return err == nil && a.IsLoopback()
+}

internal/router/router_listen_test.go

@@ -0,0 +1,14 @@
+package router
+
+import "testing"
+
+func TestOnlyALoopbackListenCountsAsPrivate(t *testing.T) {
+	for addr, want := range map[string]bool{
+		"127.0.0.1:8080": true, "[::1]:8080": true, "localhost:8080": true,
+		":8080": false, "0.0.0.0:8080": false, "10.0.0.140:8080": false, "junk": false,
+	} {
+		if got := loopbackOnly(addr); got != want {
+			t.Errorf("%s: %v", addr, got)
+		}
+	}
+}

internal/router/router_test.go

@@ -4,6 +4,7 @@ import (
 	"bytes"
 	"encoding/json"
 	"fmt"
+	"github.com/prometheus/client_golang/prometheus"
 	"io"
 	"log/slog"
 	"net/http"
@@ -35,6 +36,8 @@ type fakeEngine struct {
 	status atomic.Int64
 	// last is the body of the last completion asked for.
 	last atomic.Value
+	// traceparent is the one the last completion came with.
+	traceparent atomic.Value
 }
 
 func newFakeEngine(t *testing.T, model string) *fakeEngine {
@@ -48,6 +51,7 @@ func newFakeEngine(t *testing.T, model string) *fakeEngine {
 	})
 	mux.HandleFunc("POST /v1/chat/completions", func(w http.ResponseWriter, r *http.Request) {
 		f.calls.Add(1)
+		f.traceparent.Store(r.Header.Get("traceparent"))
 		if b, err := io.ReadAll(r.Body); err == nil {
 			f.last.Store(b)
 		}
@@ -619,3 +623,68 @@ func TestTheServedModelHeaderArrivesBeforeTheBody(t *testing.T) {
 		t.Errorf("served-model header = %q, want backup", got)
 	}
 }
+
+type unservableActivator struct {
+	fakeActivator
+	why map[string]string
+}
+
+func (u *unservableActivator) Unservable(model string) (string, bool) {
+	w, ok := u.why[model]
+	return w, ok
+}
+
+func TestAModelThatCannotComeUpIsSaidSoAtOnce(t *testing.T) {
+	// A model that fits on no node, or failed every time it started, was
+	// answered "loading; retry shortly" — and a client that retried did so
+	// forever.
+	e := newFakeEngine(t, "m")
+	cfg := baseConfig(config.Endpoint{ID: "a", URL: e.srv.URL, Models: []string{"m"}})
+	cfg.Router.ActivationWait = 10 * time.Second
+	rt := newTestRouter(t, cfg)
+	rt.SetActivator(&unservableActivator{
+		fakeActivator: fakeActivator{onDemand: map[string]bool{"phi-4": true}},
+		why:           map[string]string{"phi-4": "phi-4 cannot be placed on any node: needs 15 GB"},
+	})
+
+	start := time.Now()
+	rec := post(t, rt.Handler(), map[string]any{"model": "phi-4", "messages": []any{}}, nil)
+	if took := time.Since(start); took > 2*time.Second {
+		t.Errorf("the request waited %s for a model that will not load", took)
+	}
+	if rec.Code != http.StatusServiceUnavailable || rec.Header().Get("Retry-After") != "" {
+		t.Errorf("status %d, Retry-After %q", rec.Code, rec.Header().Get("Retry-After"))
+	}
+	if !strings.Contains(rec.Body.String(), "needs 15 GB") {
+		t.Errorf("the reason was not given: %s", rec.Body.String())
+	}
+}
+
+func TestAMadeUpModelNameDoesNotBecomeAMetricLabel(t *testing.T) {
+	// Counted as sent, every request for a nonsense model minted a new series.
+	e := newFakeEngine(t, "m")
+	cfg := baseConfig(config.Endpoint{ID: "a", URL: e.srv.URL, Models: []string{"m"}})
+	cfg.Models = []catalog.Model{{Name: "m", Aliases: []string{"mm"}}}
+	rt := newTestRouter(t, cfg)
+
+	if got := rt.metricModel("mm"); got != "m" {
+		t.Errorf("an alias is counted as %q, not the model it names", got)
+	}
+	for i := 0; i < 20; i++ {
+		post(t, rt.Handler(), map[string]any{"model": fmt.Sprintf("junk-%d", i), "messages": []any{}}, nil)
+	}
+	// Twenty names, one series: all of them "unknown".
+	reg := prometheus.NewRegistry()
+	reg.MustRegister(rt.m.requests)
+	mfs, err := reg.Gather()
+	if err != nil {
+		t.Fatal(err)
+	}
+	series := 0
+	for _, mf := range mfs {
+		series += len(mf.GetMetric())
+	}
+	if series != 1 {
+		t.Errorf("%d series after twenty made-up names", series)
+	}
+}

internal/router/scrape/llamacpp_test.go

@@ -175,7 +175,10 @@ func TestABrokenContractIsReportedOnceAndLoudly(t *testing.T) {
 	in := endpoint.Info{ID: "e1", URL: srv.URL, Engine: "vllm", Models: []string{"m"}}
 	reg.Replace([]endpoint.Info{in})
 
-	p := NewPoller(reg, Config{Interval: 20 * time.Millisecond, Timeout: 10 * time.Millisecond},
+	// A generous timeout: a scrape that timed out would be reported as
+	// unreachable, and the violation reported again once it answered — two
+	// warnings where the test counts one, on a slow runner only.
+	p := NewPoller(reg, Config{Interval: 20 * time.Millisecond, Timeout: 2 * time.Second},
 		slog.New(cap))
 	ctx, cancel := context.WithCancel(context.Background())
 	defer cancel()

internal/router/trace.go

@@ -0,0 +1,51 @@
+package router
+
+import (
+	"context"
+	"net/http"
+
+	"go.opentelemetry.io/otel"
+	"go.opentelemetry.io/otel/attribute"
+	"go.opentelemetry.io/otel/codes"
+	"go.opentelemetry.io/otel/propagation"
+	"go.opentelemetry.io/otel/trace"
+)
+
+// tracer is looked up per use rather than kept, so a provider installed after
+// the router was built — as tests do — is the one spans go to.
+func tracer() trace.Tracer { return otel.Tracer("gitea.mynodes.xyz/ilker/llmgw/internal/router") }
+
+// startInference opens the span for one inference request, continuing the
+// caller's trace when it sent a traceparent.
+func startInference(req *http.Request, path, requestID string) (context.Context, trace.Span) {
+	ctx := otel.GetTextMapPropagator().Extract(req.Context(), propagation.HeaderCarrier(req.Header))
+	return tracer().Start(ctx, "inference "+path,
+		trace.WithSpanKind(trace.SpanKindServer),
+		trace.WithAttributes(
+			attribute.String("llmgw.request_id", requestID),
+			attribute.String("http.route", path),
+		))
+}
+
+// endSpan closes a span, marking it failed when err is set.
+func endSpan(span trace.Span, err error) {
+	if err != nil {
+		span.RecordError(err)
+		span.SetStatus(codes.Error, err.Error())
+	}
+	span.End()
+}
+
+// withTraceparent is h with the current span's context in it, for the engine.
+// A copy: the request's header is sent on every attempt, each its own span.
+func withTraceparent(ctx context.Context, h http.Header) http.Header {
+	if !trace.SpanContextFromContext(ctx).IsValid() {
+		return h
+	}
+	out := h.Clone()
+	if out == nil {
+		out = http.Header{}
+	}
+	otel.GetTextMapPropagator().Inject(ctx, propagation.HeaderCarrier(out))
+	return out
+}

internal/router/trace_test.go

@@ -0,0 +1,83 @@
+package router
+
+import (
+	"testing"
+
+	"go.opentelemetry.io/otel"
+	"go.opentelemetry.io/otel/propagation"
+	sdktrace "go.opentelemetry.io/otel/sdk/trace"
+	"go.opentelemetry.io/otel/sdk/trace/tracetest"
+	"go.opentelemetry.io/otel/trace/noop"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+)
+
+func TestARequestIsTracedIntoTheEngine(t *testing.T) {
+	rec := tracetest.NewSpanRecorder()
+	prevTP, prevProp := otel.GetTracerProvider(), otel.GetTextMapPropagator()
+	otel.SetTracerProvider(sdktrace.NewTracerProvider(sdktrace.WithSpanProcessor(rec)))
+	otel.SetTextMapPropagator(propagation.TraceContext{})
+	t.Cleanup(func() { otel.SetTracerProvider(prevTP); otel.SetTextMapPropagator(prevProp) })
+
+	e := newFakeEngine(t, "m")
+	rt := newTestRouter(t, baseConfig(config.Endpoint{ID: "a", URL: e.srv.URL, Models: []string{"m"}}))
+	// The caller's own trace, continued rather than replaced.
+	const caller = "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01"
+	res := post(t, rt.Handler(), map[string]any{"model": "m", "messages": []any{}},
+		map[string]string{"traceparent": caller, "X-Request-Id": "req-1"})
+	if res.Code != 200 {
+		t.Fatalf("status %d: %s", res.Code, res.Body.String())
+	}
+
+	spans := map[string]sdktrace.ReadOnlySpan{}
+	for _, s := range rec.Ended() {
+		spans[s.Name()] = s
+	}
+	root, ok := spans["inference /v1/chat/completions"]
+	if !ok {
+		t.Fatalf("no inference span among %v", spans)
+	}
+	if got := root.SpanContext().TraceID().String(); got != "4bf92f3577b34da6a3ce929d0e0e4736" {
+		t.Errorf("the caller's trace was not continued: %s", got)
+	}
+	attrs := map[string]string{}
+	for _, a := range root.Attributes() {
+		attrs[string(a.Key)] = a.Value.Emit()
+	}
+	if attrs["llmgw.request_id"] != "req-1" || attrs["llmgw.model"] != "m" ||
+		attrs["http.response.status_code"] != "200" || attrs["llmgw.endpoint"] != "a" {
+		t.Errorf("inference span attributes %v", attrs)
+	}
+	for _, name := range []string{"admission", "schedule", "upstream"} {
+		s, ok := spans[name]
+		if !ok {
+			t.Errorf("no %s span", name)
+			continue
+		}
+		if s.Parent().SpanID() != root.SpanContext().SpanID() {
+			t.Errorf("%s is not under the request's span", name)
+		}
+	}
+	up := spans["upstream"]
+	tp, _ := e.traceparent.Load().(string)
+	if want := "00-" + up.SpanContext().TraceID().String() + "-" + up.SpanContext().SpanID().String() + "-01"; tp != want {
+		t.Errorf("the engine got traceparent %q, expected %q", tp, want)
+	}
+}
+
+func TestWithoutTracingTheEngineGetsNoTraceparent(t *testing.T) {
+	// What an unconfigured process has. Set explicitly, since the default
+	// global delegates for good to the first real provider another test set.
+	prevTP, prevProp := otel.GetTracerProvider(), otel.GetTextMapPropagator()
+	otel.SetTracerProvider(noop.NewTracerProvider())
+	otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator())
+	t.Cleanup(func() { otel.SetTracerProvider(prevTP); otel.SetTextMapPropagator(prevProp) })
+	e := newFakeEngine(t, "m")
+	rt := newTestRouter(t, baseConfig(config.Endpoint{ID: "a", URL: e.srv.URL, Models: []string{"m"}}))
+	if res := post(t, rt.Handler(), map[string]any{"model": "m", "messages": []any{}}, nil); res.Code != 200 {
+		t.Fatalf("status %d", res.Code)
+	}
+	if tp, _ := e.traceparent.Load().(string); tp != "" {
+		t.Errorf("traceparent %q sent with tracing off", tp)
+	}
+}

internal/store/migrations/postgres/0012_usage_diagnostics.sql

@@ -0,0 +1,14 @@
+-- Ported from the SQLite migration of the same number.
+-- What a request went through, beside what it cost: which replica answered,
+-- how long it queued in admission, and how long the first token took. Without
+-- them "why was my request slow" had no answer — the queue time went out in a
+-- response header and nowhere else.
+--
+-- request_id is the id the client sent or the gateway made, echoed in the
+-- X-Request-Id header and the access log, so one request can be followed from
+-- the client's error to the replica that served it.
+ALTER TABLE usage ADD COLUMN request_id TEXT NOT NULL DEFAULT '';
+ALTER TABLE usage ADD COLUMN replica TEXT NOT NULL DEFAULT '';
+ALTER TABLE usage ADD COLUMN queue_ms BIGINT NOT NULL DEFAULT 0;
+ALTER TABLE usage ADD COLUMN ttft_ms BIGINT NOT NULL DEFAULT 0;
+CREATE INDEX usage_by_request ON usage(request_id) WHERE request_id <> '';

internal/store/migrations/sqlite/0012_usage_diagnostics.sql

@@ -0,0 +1,13 @@
+-- What a request went through, beside what it cost: which replica answered,
+-- how long it queued in admission, and how long the first token took. Without
+-- them "why was my request slow" had no answer — the queue time went out in a
+-- response header and nowhere else.
+--
+-- request_id is the id the client sent or the gateway made, echoed in the
+-- X-Request-Id header and the access log, so one request can be followed from
+-- the client's error to the replica that served it.
+ALTER TABLE usage ADD COLUMN request_id TEXT NOT NULL DEFAULT '';
+ALTER TABLE usage ADD COLUMN replica TEXT NOT NULL DEFAULT '';
+ALTER TABLE usage ADD COLUMN queue_ms INTEGER NOT NULL DEFAULT 0;
+ALTER TABLE usage ADD COLUMN ttft_ms INTEGER NOT NULL DEFAULT 0;
+CREATE INDEX usage_by_request ON usage(request_id) WHERE request_id <> '';

internal/store/open.go

@@ -59,3 +59,9 @@ func resolveAddress(address string) (string, error) {
 	}
 	return out, nil
 }
+
+// ResolveAddress expands an @file store address, as Open does.
+func ResolveAddress(address string) (string, error) { return resolveAddress(address) }
+
+// IsPostgres reports whether a resolved address names a PostgreSQL server.
+func IsPostgres(address string) bool { return isPostgres(address) }

internal/store/sqlite.go

@@ -424,8 +424,9 @@ func (s *DB) WriteUsage(ctx context.Context, records []Usage) error {
 	return s.inTx(ctx, func(tx conn) error {
 		stmt, err := tx.PrepareContext(ctx,
 			`INSERT INTO usage (ts, key_id, project_id, model, served_model,
-			   prompt_tokens, completion_tokens, cost_micros, status, latency_ms, user_id)
-			 VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`)
+			   prompt_tokens, completion_tokens, cost_micros, status, latency_ms, user_id,
+			   request_id, replica, queue_ms, ttft_ms)
+			 VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`)
 		if err != nil {
 			return err
 		}
@@ -433,7 +434,8 @@ func (s *DB) WriteUsage(ctx context.Context, records []Usage) error {
 		for _, r := range records {
 			if _, err := stmt.ExecContext(ctx, millis(r.At), r.KeyID, r.ProjectID,
 				r.Model, r.ServedModel, r.PromptTokens, r.CompletionTokens,
-				r.CostMicros, r.Status, r.LatencyMs, r.UserID); err != nil {
+				r.CostMicros, r.Status, r.LatencyMs, r.UserID,
+				r.RequestID, r.Replica, r.QueueMs, r.TTFTMs); err != nil {
 				return err
 			}
 		}
@@ -817,3 +819,36 @@ func (s *DB) UsageByModel(ctx context.Context, since time.Time) ([]ModelUsage, e
 	}
 	return out, rows.Err()
 }
+
+// SnapshotTo writes a consistent copy of a SQLite store to path, while it is
+// in use. VACUUM INTO takes a read transaction, so the copy is of one moment
+// even with writes going on, which copying the file is not.
+func (s *DB) SnapshotTo(ctx context.Context, path string) error {
+	if s.dialect != SQLiteDialect {
+		return fmt.Errorf("a snapshot is for SQLite; back up PostgreSQL with pg_dump")
+	}
+	_, err := s.raw.ExecContext(ctx, `VACUUM INTO ?`, path)
+	return err
+}
+
+// UsageByRequest finds the record of one request by the id it was given, or
+// ok=false when none has that id.
+func (s *DB) UsageByRequest(ctx context.Context, requestID string) (Usage, bool, error) {
+	var u Usage
+	var ts int64
+	err := s.db.QueryRowContext(ctx,
+		`SELECT ts, key_id, project_id, COALESCE(user_id, ''), model, served_model, prompt_tokens,
+		        completion_tokens, cost_micros, status, latency_ms, request_id, replica, queue_ms, ttft_ms
+		 FROM usage WHERE request_id = ? ORDER BY ts DESC LIMIT 1`, requestID).
+		Scan(&ts, &u.KeyID, &u.ProjectID, &u.UserID, &u.Model, &u.ServedModel, &u.PromptTokens,
+			&u.CompletionTokens, &u.CostMicros, &u.Status, &u.LatencyMs, &u.RequestID, &u.Replica,
+			&u.QueueMs, &u.TTFTMs)
+	if errors.Is(err, sql.ErrNoRows) {
+		return Usage{}, false, nil
+	}
+	if err != nil {
+		return Usage{}, false, err
+	}
+	u.At = time.UnixMilli(ts)
+	return u, true, nil
+}

internal/store/store.go

@@ -244,6 +244,15 @@ type Usage struct {
 	CostMicros int64
 	Status     int
 	LatencyMs  int64
+
+	// RequestID is the id the client sent in X-Request-Id or the gateway made.
+	RequestID string
+	// Replica answered the request; empty when none did.
+	Replica string
+	// QueueMs is time spent waiting in admission; TTFTMs is time to the first
+	// token, 0 when none came.
+	QueueMs int64
+	TTFTMs  int64
 }
 
 // Store is the persistent half of tenancy and accounting.

internal/tlsconf/expiry.go

@@ -0,0 +1,58 @@
+package tlsconf
+
+import (
+	"crypto/x509"
+	"encoding/pem"
+	"os"
+
+	"github.com/prometheus/client_golang/prometheus"
+)
+
+// ExpiryCollector publishes when each certificate in a set of files expires.
+//
+// Certificates here last a year and nothing renews them; the first sign of one
+// running out was going to be every agent refusing to connect on the same day.
+// Read at every scrape rather than once, so a certificate replaced on disk is
+// seen without a restart. A file that cannot be read publishes nothing, which
+// is its own thing to notice.
+func ExpiryCollector(files map[string]string) prometheus.Collector {
+	return &expiry{files: files, desc: prometheus.NewDesc("llmgw_certificate_expiry_timestamp_seconds",
+		"When a certificate this control plane uses expires, as a Unix time.",
+		[]string{"cert", "subject"}, nil)}
+}
+
+type expiry struct {
+	files map[string]string
+	desc  *prometheus.Desc
+}
+
+func (e *expiry) Describe(ch chan<- *prometheus.Desc) { ch <- e.desc }
+
+func (e *expiry) Collect(ch chan<- prometheus.Metric) {
+	for name, path := range e.files {
+		if path == "" {
+			continue
+		}
+		b, err := os.ReadFile(path) //nolint:gosec // configured certificate paths
+		if err != nil {
+			continue
+		}
+		// Every certificate in the file: a CA bundle may hold several.
+		for {
+			var blk *pem.Block
+			blk, b = pem.Decode(b)
+			if blk == nil {
+				break
+			}
+			if blk.Type != "CERTIFICATE" {
+				continue
+			}
+			c, err := x509.ParseCertificate(blk.Bytes)
+			if err != nil {
+				continue
+			}
+			ch <- prometheus.MustNewConstMetric(e.desc, prometheus.GaugeValue,
+				float64(c.NotAfter.Unix()), name, c.Subject.CommonName)
+		}
+	}
+}

internal/web/admin_mirror.go

@@ -7,11 +7,13 @@ import (
 	"fmt"
 	"html/template"
 	"net/http"
+	"slices"
 	"sort"
 	"strconv"
 	"strings"
 	"time"
 
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
@@ -67,6 +69,18 @@ type modelUse struct {
 	Name     string
 	Mode     string // always, ondemand or stopped
 	Replicas int
+	// Min and Max are its scaling as the catalog holds it.
+	Min, Max int
+	// Ready, Wanted and Nodes are how it runs now, for the row itself:
+	// replicas ready of those it should have, and the nodes they are on.
+	// Fleet is false where nothing could be asked, so the row says nothing
+	// rather than "0 of 0".
+	Fleet         bool
+	Ready, Wanted int
+	Starting      int
+	Nodes         []nodeName
+	Parked        bool
+	Unplaceable   bool
 	// Declared models are read from server.yaml on every start, so a mode
 	// changed here would revert; the page says where to change it instead.
 	Declared bool
@@ -83,6 +97,11 @@ type mirrorRow struct {
 	// operator clicked. Nil on a row that is still arriving: a catalog entry
 	// pointing at a partial file is the thing versioned paths exist to prevent.
 	Serve *serveForm
+	// Detail is the card opened beneath the row: what runs it, on which
+	// nodes, and how many replicas it runs.
+	Detail *heldDetail
+	// Open is the row's ?open= value.
+	Open string
 
 	// HasPercent is false when the upstream never sent a length. The bar is
 	// then not drawn at all: a progress bar that has to invent its own total is
@@ -100,7 +119,11 @@ type mirrorRow struct {
 }
 
 type adminMirrorData struct {
-	Nav  Nav
+	Nav Nav
+	// Tab is the part of the page shown: held, explore or fetch. Tabs are the
+	// links to each.
+	Tab  string
+	Tabs []mirrorTab
 	Addr string
 	Base string
 	Rows []mirrorRow
@@ -270,6 +293,8 @@ func (p *AccountPages) RegisterAdminMirror(mux *http.ServeMux, d MirrorDeps) {
 				return name + " now runs all the time", nil
 			}))
 
+		mux.HandleFunc("POST /admin/mirror/replicas", p.mirrorAction(m, p.setReplicas))
+
 		mux.HandleFunc("POST /admin/mirror/serve", p.mirrorAction(m,
 			func(ctx context.Context, admin store.User, r *http.Request) (string, error) {
 				model, err := buildModel(r.FormValue)
@@ -361,7 +386,14 @@ func (p *AccountPages) renderMirror(w http.ResponseWriter, r *http.Request,
 	}
 
 	d.HasCatalog = p.catalog != nil
-	p.explore(r, &d)
+	d.HasRegistry = p.registry != nil
+	d.Tab = mirrorTabFor(r, errMsg)
+	// Only the tab shown does its work: the registry is a server across the
+	// internet, and asking it for sixty repositories to show somebody who came
+	// to look at a download's progress made every visit wait on it.
+	if d.Tab == tabExplore {
+		p.explore(r, &d)
+	}
 
 	got, err := m.List(r.Context())
 	if err != nil {
@@ -370,6 +402,7 @@ func (p *AccountPages) renderMirror(w http.ResponseWriter, r *http.Request,
 		if d.Error == "" {
 			d.Error = err.Error()
 		}
+		d.Tabs = mirrorTabs(d)
 		render(w, "admin_mirror", d)
 		return
 	}
@@ -412,12 +445,46 @@ func (p *AccountPages) renderMirror(w http.ResponseWriter, r *http.Request,
 		}
 		d.Rows = append(d.Rows, row)
 	}
+	var views map[string]adminapi.ModelView
+	prefix := ""
+	if p.fleet != nil && len(sources) > 0 {
+		views = map[string]adminapi.ModelView{}
+		for _, v := range p.fleet.Models() {
+			views[v.Name] = v
+		}
+		var ids []string
+		for _, n := range p.fleet.Nodes() {
+			ids = append(ids, n.ID)
+		}
+		prefix = sharedPrefix(ids)
+	}
 	for i := range d.Rows {
 		for _, m := range sources[d.Rows[i].URL] {
-			d.Rows[i].UsedBy = append(d.Rows[i].UsedBy, modelUse{
+			u := modelUse{
 				Name: m.Name, Mode: modeOf(m.Scaling), Replicas: max(m.Scaling.Max, 1),
+				Min: m.Scaling.Min, Max: m.Scaling.Max,
 				Declared: p.catalog.Declared(m.Name),
-			})
+			}
+			if v, ok := views[m.Name]; ok {
+				u.Fleet, u.Ready, u.Wanted = true, v.Ready, v.Desired
+				u.Parked, u.Unplaceable = v.Parked, v.Unplaceable
+				var nodes []string
+				nodes, u.Starting = runningOn(p.fleet.Replicas(m.Name))
+				for _, n := range nodes {
+					u.Nodes = append(u.Nodes, nodeName{Full: n, Short: strings.TrimPrefix(n, prefix)})
+				}
+			}
+			d.Rows[i].UsedBy = append(d.Rows[i].UsedBy, u)
+		}
+	}
+
+	// The one row whose card is open. From the form too, so a change made in
+	// the card comes back with the card still open to show it.
+	open := r.FormValue("open")
+	for i := range d.Rows {
+		d.Rows[i].Open = openKey(d.Rows[i].Model, d.Rows[i].Version)
+		if open != "" && d.Rows[i].Open == open && d.Rows[i].State == mirror.Present {
+			d.Rows[i].Detail = p.heldDetailFor(d.Rows[i])
 		}
 	}
 
@@ -462,12 +529,78 @@ func (p *AccountPages) renderMirror(w http.ResponseWriter, r *http.Request,
 
 	d.Groups = groupByModel(d.Rows)
 
-	if d.Fetching > 0 {
+	// Only on the list: a reload anywhere else takes with it a search, an
+	// open repository or a URL half typed.
+	if d.Fetching > 0 && d.Tab == tabHeld && open == "" {
 		d.RefreshSec = 5
 	}
+	d.Tabs = mirrorTabs(d)
 	render(w, "admin_mirror", d)
 }
 
+const (
+	tabHeld    = "held"
+	tabExplore = "explore"
+	tabFetch   = "fetch"
+)
+
+// mirrorTab is one of the page's tabs.
+type mirrorTab struct {
+	Label  string
+	Count  string // beside the label, empty for none
+	Href   string
+	Active bool
+}
+
+// mirrorTabFor is the tab a request is for.
+//
+// Named in the address when it is chosen, and otherwise read from what the
+// address carries, so links made before the page had tabs — a search, an
+// open repository — still land where they meant. After a change the list is
+// shown, where its effect is, unless it was refused: then the form it came
+// from, to try again.
+func mirrorTabFor(r *http.Request, errMsg string) string {
+	valid := func(t string) bool { return t == tabHeld || t == tabExplore || t == tabFetch }
+	if r.Method == http.MethodPost {
+		if t := r.PostFormValue("tab"); errMsg != "" && valid(t) {
+			return t
+		}
+		return tabHeld
+	}
+	q := r.URL.Query()
+	if t := q.Get("tab"); valid(t) {
+		return t
+	}
+	for _, k := range []string{"q", "repo", "view", "task", "size", "x"} {
+		if q.Has(k) {
+			return tabExplore
+		}
+	}
+	return tabHeld
+}
+
+func mirrorTabs(d adminMirrorData) []mirrorTab {
+	held := mirrorTab{Label: "Held", Href: "/admin/mirror?tab=held", Active: d.Tab == tabHeld}
+	switch {
+	case d.Unreachable:
+		held.Count = "unreachable"
+	case d.Fetching > 0:
+		held.Count = fmt.Sprintf("%d · %d downloading", d.Present, d.Fetching)
+	default:
+		held.Count = strconv.Itoa(d.Present)
+	}
+	if d.Failed > 0 {
+		held.Count += fmt.Sprintf(" · %d failed", d.Failed)
+	}
+	tabs := []mirrorTab{held}
+	if d.HasRegistry {
+		tabs = append(tabs, mirrorTab{Label: "Explore Hugging Face",
+			Href: "/admin/mirror?tab=explore", Active: d.Tab == tabExplore})
+	}
+	return append(tabs, mirrorTab{Label: "Fetch by URL",
+		Href: "/admin/mirror?tab=fetch", Active: d.Tab == tabFetch})
+}
+
 // mirrorGroup is a family of models the mirror holds — every qwen, every
 // llama — and their versions.
 type mirrorGroup struct {
@@ -621,12 +754,7 @@ func (p *AccountPages) openRepo(r *http.Request, d *adminMirrorData, repo string
 // is not part of what anybody calls the model, and "-GGUF" describes the file
 // format rather than the model, which is already clear from the file being
 // fetched.
-func defaultModelName(repoID string) string {
-	name := mirror.LastPathElement(repoID)
-	name = strings.TrimSuffix(name, "-GGUF")
-	name = strings.TrimSuffix(name, "-gguf")
-	return strings.ToLower(name)
-}
+func defaultModelName(repoID string) string { return hf.ModelName(repoID) }
 
 // verdictFor judges a registry file set by the repository's architecture. A
 // vision projector is its own architecture whatever the repository says.
@@ -668,3 +796,43 @@ func writeJSONStatus(w http.ResponseWriter, code int, v any) {
 	w.WriteHeader(code)
 	_ = json.NewEncoder(w).Encode(v)
 }
+
+// runningOn is the nodes a model's replicas are on, each once and in order,
+// and how many of its replicas are not ready yet.
+func runningOn(reps []adminapi.ReplicaView) (nodes []string, starting int) {
+	for _, r := range reps {
+		if r.State != "ready" {
+			starting++
+		}
+		for _, sh := range r.Shards {
+			if !slices.Contains(nodes, sh.Node) {
+				nodes = append(nodes, sh.Node)
+			}
+		}
+	}
+	slices.Sort(nodes)
+	return nodes, starting
+}
+
+// nodeName is a node as a row names it: short, with the full name beside.
+type nodeName struct{ Short, Full string }
+
+// sharedPrefix is what every node's name starts with, up to and including a
+// separator — "llmgw-" of llmgw-node01 and llmgw-node02 — so a row can say
+// node01 and fit on a line. Nothing when the names share no such part, or
+// when there is one node and dropping it would leave only a number.
+func sharedPrefix(ids []string) string {
+	if len(ids) < 2 {
+		return ""
+	}
+	p := ids[0]
+	for _, id := range ids[1:] {
+		for !strings.HasPrefix(id, p) {
+			p = p[:len(p)-1]
+		}
+	}
+	if i := strings.LastIndexAny(p, "-._"); i >= 0 {
+		return p[:i+1]
+	}
+	return ""
+}

internal/web/admin_mirror_test.go

@@ -341,16 +341,18 @@ func TestAFailedRowOffersRetryRatherThanRemove(t *testing.T) {
 }
 
 type fakeRegistry struct {
-	results []hf.Model
-	repo    hf.Repo
-	err     error
-	asked   string
-	query   hf.Query
+	results  []hf.Model
+	repo     hf.Repo
+	err      error
+	asked    string
+	query    hf.Query
+	explored int
 }
 
 func (f *fakeRegistry) Explore(_ context.Context, q hf.Query) ([]hf.Model, error) {
 	f.asked = q.Search
 	f.query = q
+	f.explored++
 	return f.results, f.err
 }
 
@@ -441,7 +443,11 @@ func TestWithNoRegistryThereIsNoSearchBox(t *testing.T) {
 	if strings.Contains(body, "Search Hugging Face") {
 		t.Error("the explorer should be absent without a registry")
 	}
-	if !strings.Contains(body, "Fetch a model") {
+	if strings.Contains(body, "Explore Hugging Face") {
+		t.Error("a tab for an explorer that is not there")
+	}
+	fetch := mirrorGet(t, mirrorPage(t, &fakeMirror{}, true), "/admin/mirror?tab=fetch").Body.String()
+	if !strings.Contains(fetch, `name="source" required`) {
 		t.Error("fetching by hand must still be there")
 	}
 }
@@ -464,8 +470,7 @@ func TestAShardedModelIsOneRowAndOneButton(t *testing.T) {
 		"/admin/mirror?repo=Qwen/Qwen2.5-7B-Instruct").Body.String()
 
 	// Fifteen shards is one decision, not fifteen. Four here, and one form.
-	if n := strings.Count(body, `action="/admin/mirror/fetch"`); n != 2 {
-		// One for the set, one for the manual form at the bottom.
+	if n := strings.Count(body, `action="/admin/mirror/fetch"`); n != 1 {
 		t.Errorf("expected the four shards as a single form, found %d forms", n)
 	}
 	if !strings.Contains(body, "4 shards, fetched together") {
@@ -534,8 +539,8 @@ func TestAlternativeQuantisationsStaySeparateRows(t *testing.T) {
 
 	// Two GGUFs are alternatives, not halves. Offering "fetch all" here would
 	// spend gigabytes on a copy nobody asked for.
-	if n := strings.Count(body, `action="/admin/mirror/fetch"`); n != 3 {
-		t.Errorf("expected a form each plus the manual one, found %d", n)
+	if n := strings.Count(body, `action="/admin/mirror/fetch"`); n != 2 {
+		t.Errorf("expected a form each, found %d", n)
 	}
 }
 
@@ -587,7 +592,7 @@ func TestHowAModelRunsIsChangedWhereItsFileIsShown(t *testing.T) {
 		t.Error("the running model's mode is not shown as chosen")
 	}
 	// Declared in server.yaml: shown, not offered, since it would revert.
-	if !strings.Contains(body, "on demand<br>set in server.yaml") {
+	if !strings.Contains(body, `title="set in server.yaml, which is read again on every start">on demand · yaml</span>`) {
 		t.Error("a declared model should say where its mode is set")
 	}
 	if strings.Count(body, `name="mode" class="mode"`) != 1 {
@@ -655,3 +660,53 @@ func privateOK(m *mirror.Manager) *mirror.Manager {
 	m.AllowPrivateSources()
 	return m
 }
+
+func TestEachTabDoesOnlyItsOwnWork(t *testing.T) {
+	reg := &fakeRegistry{}
+	mux := mirrorPageWith(t, &fakeMirror{models: []mirror.Status{
+		{Model: "m", Version: "v", File: "m.gguf", State: mirror.Present, BytesDone: 1},
+		{Model: "n", Version: "v", File: "n.gguf", State: mirror.Fetching, BytesDone: 1},
+	}}, reg, true)
+
+	held := mirrorGet(t, mux, "/admin/mirror").Body.String()
+	if reg.explored != 0 {
+		t.Error("the list asked the registry for repositories it does not show")
+	}
+	if !strings.Contains(held, `class="on" aria-current="page">Held <span class="count">1 · 1 downloading</span>`) {
+		t.Errorf("the Held tab should be chosen and count what it holds: %s", held)
+	}
+	if strings.Contains(held, `name="source" required`) || strings.Contains(held, "Search Hugging Face") {
+		t.Error("other tabs' forms are on the list")
+	}
+	if !strings.Contains(held, `http-equiv="refresh"`) {
+		t.Error("the list stopped following a download")
+	}
+
+	explore := mirrorGet(t, mux, "/admin/mirror?tab=explore").Body.String()
+	if reg.explored != 1 || !strings.Contains(explore, "Search Hugging Face") {
+		t.Error("the explorer tab did not explore")
+	}
+	if strings.Contains(explore, `id="held"`) || strings.Contains(explore, `http-equiv="refresh"`) {
+		t.Error("the explorer shows the list, or reloads away a search")
+	}
+	// A link from before there were tabs still lands on the explorer.
+	if old := mirrorGet(t, mux, "/admin/mirror?q=qwen").Body.String(); !strings.Contains(old, "Search Hugging Face") {
+		t.Error("a search address no longer opens the explorer")
+	}
+}
+
+func TestARefusedFetchReturnsToItsFormAndAStartedOneToTheList(t *testing.T) {
+	m := &fakeMirror{fetchErr: errors.New("upstream said 404")}
+	mux := mirrorPage(t, m, true)
+	body := mirrorPost(t, mux, "/admin/mirror/fetch", url.Values{"csrf": {"tok"}, "source": {"https://h.example/m.gguf"}, "model": {"m"},
+		"version": {"v"}, "force": {"1"}, "tab": {"fetch"}}).Body.String()
+	if !strings.Contains(body, "upstream said 404") || !strings.Contains(body, `name="source" required`) {
+		t.Errorf("a refusal should come back on the form it came from: %s", body)
+	}
+	m.fetchErr = nil
+	body = mirrorPost(t, mux, "/admin/mirror/fetch", url.Values{"csrf": {"tok"}, "source": {"https://h.example/m.gguf"}, "model": {"m"},
+		"version": {"v"}, "force": {"1"}, "tab": {"fetch"}}).Body.String()
+	if !strings.Contains(body, `id="held"`) {
+		t.Errorf("a started download should be shown where it is followed: %s", body)
+	}
+}