Release v0.10.4: CPU threads, tracing, and the rest of the review

CPU replicas start on half the machine's CPUs, not all of them: two busy
llama.cpp servers each spinning on all 8 vCPUs fell to 0.8 tok/s apiece,
where at 4 threads they keep 5.5. llamacpp.threads sets it per node.

Inference requests can be traced over OTLP/HTTP (tracing.endpoint), with
admission, schedule and upstream spans and traceparent sent to the engine.

Security: onboarding no longer puts the cluster root in the system trust
store and removes it where it was; the mirror's mutual-TLS API can be
limited to named certificates (--api-client); init.log no longer keeps the
administrator's password; a warning when auth is off on a public listen;
the recorded actor is sanitised and sudo-aware; Ray's dashboard is off, and
the remaining engine gaps are documented.

reconciler.go and agent.go are split by topic, the status mapping of each
failure is tested, and the plan's checklist is brought up to date.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

CHANGELOG.md

@@ -20,10 +20,33 @@ is only released once its **exit criterion** is met, not when its code lands.
 Until v1.0.0 the minor version is the breaking one: configuration formats and Go
 APIs change between minors without a deprecation period.
 
-## Unreleased
+## v0.10.4 — 2026-09-30
 
-Changes since v0.10.3, published as release candidates (v0.10.4-rc.N) until
-there are enough for v0.10.4.
+Published first as release candidates v0.10.4-rc.1 to rc.9.
+
+After upgrading, run `llmgw ctl restart --all` once: CPU replicas take the new
+thread count below only when they start.
+
+### CPU replicas no longer fight over the CPUs
+
+llama.cpp computes on every CPU of the machine unless told otherwise, and the
+agent did not tell it. Two busy CPU replicas on one node then spin against each
+other: on an 8-vCPU node, 7B generation fell from 10 tokens a second alone to
+0.8 each with both busy, in two rounds of the same test. The agent now starts a
+CPU replica on half the machine's CPUs, where one alone makes 6.5 and two busy
+make 5.5 each. `llamacpp.threads` in the agent's configuration sets it for a
+node — all of them suits a node that only ever runs one CPU replica — and a
+profile's own `--threads` still wins. Measured on VMs whose host is
+oversubscribed; the proportions, not the numbers, are the point.
+
+### Tracing
+
+With `tracing.endpoint` set (or `OTEL_EXPORTER_OTLP_ENDPOINT`), each inference
+request is traced over OTLP/HTTP: a span for the request, carrying its
+`X-Request-Id`, with admission, scheduling and each attempt at an engine under
+it. A caller's own `traceparent` is continued, and the engine is sent one for
+its attempt. `tracing.sample_ratio` traces a share of the requests that arrive
+without a decision. Off, it costs nothing.
 
 ### Restarting replicas
 
@@ -77,6 +100,108 @@ more than 26 hours old or there has never been one. docs/install.md now says how
 to restore, including why running engines survive it: the engine-key secret
 comes back with the store.
 
+### Replicas follow load between min and max
+
+A model's `max` was a ceiling nothing reached: every model ran exactly its
+minimum however long its queue. Now, when requests are waiting at the engines —
+every slot of every replica busy — at one or more per replica for a minute, one
+replica is added, up to `max`; ten minutes with nothing waiting takes one of
+those away, down to `min`. One step at a time, and not while the previous one
+is still starting, so a spike that is over before a replica loads does not
+leave several behind. Each decision is logged with the load that made it.
+`llmgw ctl scale MODEL --min 1 --max 3` turns it on for a model.
+
+### A model from the registry in one command
+
+`llmgw ctl model serve OWNER/REPO` takes a model from the registry to a replica
+that answers. It picks the GGUF file (`--quant`, Q4_K_M by default), has the
+mirror fetch it and checks what arrived against the registry's published
+sha256, checks that the nodes' llama.cpp can load it, sizes its memory from its
+header, adds it to the catalog, waits for a replica — stopping early with the
+reason if it fits nowhere or keeps failing — and asks it to say hello. Each step
+is shown as it happens. It runs in the control plane as a job, so a download of
+hours does not depend on a terminal staying open. `--on-demand` adds it without
+starting a replica.
+
+### Upgrading from one command
+
+`sudo llmgw ctl upgrade VERSION`, run on the control plane, upgrades that
+machine and then every node in turn. Each agent is told the version over its
+existing stream, fetches the release for its own platform, checks it against
+the release's checksums, keeps the old binary beside the new and restarts,
+while its engines keep serving. A node that does not come back on the new
+version, ready, with its replicas, stops the upgrade before the next one.
+Running it again resumes. Agents from before this have to be moved by hand
+once.
+
+### Following a request
+
+Every response carries `X-Request-Id` — the client's own when it sends a plain
+one, otherwise one the gateway makes — and the engine receives it too. One
+access log line per request records it with the model asked for and served,
+the replica, status, time queued, time to the first token, total time,
+tokens, and the key, project and user. Usage records keep the id, replica,
+queue time and time to first token, and `llmgw ctl request ID` reads one back:
+the answer to "why was this slow?".
+
+### Rate-limit headers
+
+Responses carry OpenAI's `x-ratelimit-limit-*`, `x-ratelimit-remaining-*` and
+`x-ratelimit-reset-*` for requests and tokens, from whichever level of the
+tenancy tree will refuse first, so SDKs and editors can slow down before they
+are refused.
+
+### Memory sized from the model
+
+The mirror's serve form sizes a llama.cpp model's memory from its GGUF header:
+weights, plus the KV cache for the context pool from its layers, KV heads and
+head width, plus the engine's own. At a 16k pool that cache ranges from 0.44 GB
+to 5.25 GB across the models on the live mirror, which the old "weights plus
+half again plus two" could not see.
+
+### Security
+
+- **Alertmanager answered anyone on the network**, including requests to
+  silence alerts. The monitoring README now binds it to loopback, where
+  Prometheus beside it is its only client.
+- **Login and signup could be made to use gigabytes.** Each attempt hashes with
+  64 MiB of argon2, with no bound on how many ran at once, and the per-address
+  throttle grew by one entry per address tried. Hashes now run a few at a time
+  and the rest wait; the throttle expires old entries, and when it is still
+  full it refuses new addresses rather than dropping live ones — which would
+  have let a flood of made-up addresses reset the throttle on a real one.
+- **Any key holder could grow Prometheus without bound** by asking for made-up
+  model names, each of which became a metric label. Requests are counted under
+  the catalog's name for their model, or "unknown".
+- **A SQLite backup left a full copy of the store readable** by local users
+  while it ran. The snapshot is now made in a private directory, and
+  llmgw-backup.service sets `UMask=0077`.
+- **An engine archive could write outside its directory through chained
+  links.** Archives are unpacked through `os.Root`, and a link may only point
+  at a file beside it. Every llama.cpp b10991 archive, CPU and CUDA, already
+  follows that rule.
+- **Onboarding made every node trust the cluster's root for everything.** It
+  was added to the system store, so a certificate the control plane signed
+  would pass for any site with every TLS client on the node. The agent trusts
+  it through its own configuration only; onboarding no longer adds it, and
+  takes it out of a node that an older version onboarded.
+- **Any node could manage a mirror running mutual TLS**, since one root signs
+  every node's certificate. `llmgw mirror serve --api-client NAME` answers only
+  the certificate with that common name, and the mirror warns when mutual TLS
+  runs without one.
+- **The administrator's password stayed in `init.log`.** It is shown once, on
+  the screen, and the log keeps a placeholder.
+- **Auth off on a public address was silent.** The server now warns at start
+  when the API listens beyond loopback without keys.
+- **The actor recorded with a change was whatever the caller sent**, line
+  breaks included. It is cut to a short printable name, and `sudo llmgw ctl`
+  records the person who ran sudo rather than root.
+- **vLLM's Ray head ran its dashboard**, whose job API runs code for anyone who
+  reaches it. It is off. The head itself still has no authentication, and
+  SGLang still takes its key as a flag; docs/install.md says what to do about
+  both. The monitoring README says how to keep Prometheus's queries to the
+  control plane.
+
 ### Fixes
 
 - Tests that slept a fixed time and then asserted now wait for their
@@ -84,6 +209,8 @@ comes back with the store.
   loaded runner.
 - `ctl` reported a change the API had queued (202 Accepted) as an error, as
   `ctl restart` did on its first real use.
+- The status each failure is answered with — 499, 502, 503 with and without
+  Retry-After, 504, 429 — is now tested case by case.
 
 ### Releases
 

api/gen/llmgw/v1/agent.pb.go

@@ -104,6 +104,7 @@ type AgentMessage struct {
 	//	*AgentMessage_Logs
 	//	*AgentMessage_FabricScan
 	//	*AgentMessage_FabricTest
+	//	*AgentMessage_Upgrade
 	Msg           isAgentMessage_Msg `protobuf_oneof:"msg"`
 	unknownFields protoimpl.UnknownFields
 	sizeCache     protoimpl.SizeCache
@@ -200,6 +201,15 @@ func (x *AgentMessage) GetFabricTest() *FabricTestReply {
 	return nil
 }
 
+func (x *AgentMessage) GetUpgrade() *UpgradeReply {
+	if x != nil {
+		if x, ok := x.Msg.(*AgentMessage_Upgrade); ok {
+			return x.Upgrade
+		}
+	}
+	return nil
+}
+
 type isAgentMessage_Msg interface {
 	isAgentMessage_Msg()
 }
@@ -228,6 +238,10 @@ type AgentMessage_FabricTest struct {
 	FabricTest *FabricTestReply `protobuf:"bytes,6,opt,name=fabric_test,json=fabricTest,proto3,oneof"`
 }
 
+type AgentMessage_Upgrade struct {
+	Upgrade *UpgradeReply `protobuf:"bytes,7,opt,name=upgrade,proto3,oneof"`
+}
+
 func (*AgentMessage_Register) isAgentMessage_Msg() {}
 
 func (*AgentMessage_Heartbeat) isAgentMessage_Msg() {}
@@ -240,6 +254,8 @@ func (*AgentMessage_FabricScan) isAgentMessage_Msg() {}
 
 func (*AgentMessage_FabricTest) isAgentMessage_Msg() {}
 
+func (*AgentMessage_Upgrade) isAgentMessage_Msg() {}
+
 // ControlMessage is anything the control plane tells a node.
 type ControlMessage struct {
 	state protoimpl.MessageState `protogen:"open.v1"`
@@ -254,6 +270,7 @@ type ControlMessage struct {
 	//	*ControlMessage_Logs
 	//	*ControlMessage_FabricScan
 	//	*ControlMessage_FabricTest
+	//	*ControlMessage_Upgrade
 	Msg           isControlMessage_Msg `protobuf_oneof:"msg"`
 	unknownFields protoimpl.UnknownFields
 	sizeCache     protoimpl.SizeCache
@@ -377,6 +394,15 @@ func (x *ControlMessage) GetFabricTest() *FabricTestRequest {
 	return nil
 }
 
+func (x *ControlMessage) GetUpgrade() *UpgradeAgent {
+	if x != nil {
+		if x, ok := x.Msg.(*ControlMessage_Upgrade); ok {
+			return x.Upgrade
+		}
+	}
+	return nil
+}
+
 type isControlMessage_Msg interface {
 	isControlMessage_Msg()
 }
@@ -417,6 +443,10 @@ type ControlMessage_FabricTest struct {
 	FabricTest *FabricTestRequest `protobuf:"bytes,9,opt,name=fabric_test,json=fabricTest,proto3,oneof"`
 }
 
+type ControlMessage_Upgrade struct {
+	Upgrade *UpgradeAgent `protobuf:"bytes,10,opt,name=upgrade,proto3,oneof"`
+}
+
 func (*ControlMessage_Start) isControlMessage_Msg() {}
 
 func (*ControlMessage_Stop) isControlMessage_Msg() {}
@@ -435,6 +465,8 @@ func (*ControlMessage_FabricScan) isControlMessage_Msg() {}
 
 func (*ControlMessage_FabricTest) isControlMessage_Msg() {}
 
+func (*ControlMessage_Upgrade) isControlMessage_Msg() {}
+
 // Register is the first message on every stream. It carries the node's full
 // hardware inventory, so the control plane needs no separate discovery call and
 // a reconnecting agent automatically refreshes what the scheduler knows.
@@ -3460,11 +3492,144 @@ func (x *FabricTestReply) GetTookMs() int64 {
 	return 0
 }
 
+// UpgradeAgent tells an agent to replace its own binary with a release and
+// restart.
+//
+// The agent fetches the release for its own platform and checks it against
+// the release's checksums before replacing anything, keeps the old binary
+// beside the new, then exits so systemd starts the new one. Engines keep
+// running throughout, as for any agent restart, and are adopted on the way
+// back. Over the stream the agent already holds, so upgrading a fleet needs
+// no ssh into it.
+type UpgradeAgent struct {
+	state     protoimpl.MessageState `protogen:"open.v1"`
+	RequestId string                 `protobuf:"bytes,1,opt,name=request_id,json=requestId,proto3" json:"request_id,omitempty"`
+	// version must be a release: v0.10.4 or v0.10.4-rc.2.
+	Version string `protobuf:"bytes,2,opt,name=version,proto3" json:"version,omitempty"`
+	// releases is the base URL release files are downloaded from.
+	Releases      string `protobuf:"bytes,3,opt,name=releases,proto3" json:"releases,omitempty"`
+	unknownFields protoimpl.UnknownFields
+	sizeCache     protoimpl.SizeCache
+}
+
+func (x *UpgradeAgent) Reset() {
+	*x = UpgradeAgent{}
+	mi := &file_llmgw_v1_agent_proto_msgTypes[33]
+	ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+	ms.StoreMessageInfo(mi)
+}
+
+func (x *UpgradeAgent) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*UpgradeAgent) ProtoMessage() {}
+
+func (x *UpgradeAgent) ProtoReflect() protoreflect.Message {
+	mi := &file_llmgw_v1_agent_proto_msgTypes[33]
+	if x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use UpgradeAgent.ProtoReflect.Descriptor instead.
+func (*UpgradeAgent) Descriptor() ([]byte, []int) {
+	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{33}
+}
+
+func (x *UpgradeAgent) GetRequestId() string {
+	if x != nil {
+		return x.RequestId
+	}
+	return ""
+}
+
+func (x *UpgradeAgent) GetVersion() string {
+	if x != nil {
+		return x.Version
+	}
+	return ""
+}
+
+func (x *UpgradeAgent) GetReleases() string {
+	if x != nil {
+		return x.Releases
+	}
+	return ""
+}
+
+// UpgradeReply says whether the new binary is in place. Sent before the
+// agent exits; an empty error means it will be back on the new version.
+type UpgradeReply struct {
+	state         protoimpl.MessageState `protogen:"open.v1"`
+	RequestId     string                 `protobuf:"bytes,1,opt,name=request_id,json=requestId,proto3" json:"request_id,omitempty"`
+	Version       string                 `protobuf:"bytes,2,opt,name=version,proto3" json:"version,omitempty"`
+	Error         string                 `protobuf:"bytes,3,opt,name=error,proto3" json:"error,omitempty"`
+	unknownFields protoimpl.UnknownFields
+	sizeCache     protoimpl.SizeCache
+}
+
+func (x *UpgradeReply) Reset() {
+	*x = UpgradeReply{}
+	mi := &file_llmgw_v1_agent_proto_msgTypes[34]
+	ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+	ms.StoreMessageInfo(mi)
+}
+
+func (x *UpgradeReply) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*UpgradeReply) ProtoMessage() {}
+
+func (x *UpgradeReply) ProtoReflect() protoreflect.Message {
+	mi := &file_llmgw_v1_agent_proto_msgTypes[34]
+	if x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use UpgradeReply.ProtoReflect.Descriptor instead.
+func (*UpgradeReply) Descriptor() ([]byte, []int) {
+	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{34}
+}
+
+func (x *UpgradeReply) GetRequestId() string {
+	if x != nil {
+		return x.RequestId
+	}
+	return ""
+}
+
+func (x *UpgradeReply) GetVersion() string {
+	if x != nil {
+		return x.Version
+	}
+	return ""
+}
+
+func (x *UpgradeReply) GetError() string {
+	if x != nil {
+		return x.Error
+	}
+	return ""
+}
+
 var File_llmgw_v1_agent_proto protoreflect.FileDescriptor
 
 const file_llmgw_v1_agent_proto_rawDesc = "" +
 	"\n" +
-	"\x14llmgw/v1/agent.proto\x12\bllmgw.v1\"\xd2\x02\n" +
+	"\x14llmgw/v1/agent.proto\x12\bllmgw.v1\"\x86\x03\n" +
 	"\fAgentMessage\x120\n" +
 	"\bregister\x18\x01 \x01(\v2\x12.llmgw.v1.RegisterH\x00R\bregister\x123\n" +
 	"\theartbeat\x18\x02 \x01(\v2\x13.llmgw.v1.HeartbeatH\x00R\theartbeat\x12-\n" +
@@ -3473,8 +3638,9 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
 	"\vfabric_scan\x18\x05 \x01(\v2\x19.llmgw.v1.FabricScanReplyH\x00R\n" +
 	"fabricScan\x12<\n" +
 	"\vfabric_test\x18\x06 \x01(\v2\x19.llmgw.v1.FabricTestReplyH\x00R\n" +
-	"fabricTestB\x05\n" +
-	"\x03msg\"\xe5\x03\n" +
+	"fabricTest\x122\n" +
+	"\aupgrade\x18\a \x01(\v2\x16.llmgw.v1.UpgradeReplyH\x00R\aupgradeB\x05\n" +
+	"\x03msg\"\x99\x04\n" +
 	"\x0eControlMessage\x12,\n" +
 	"\x05start\x18\x01 \x01(\v2\x14.llmgw.v1.StartShardH\x00R\x05start\x12)\n" +
 	"\x04stop\x18\x02 \x01(\v2\x13.llmgw.v1.StopShardH\x00R\x04stop\x12,\n" +
@@ -3486,7 +3652,9 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
 	"\vfabric_scan\x18\b \x01(\v2\x1b.llmgw.v1.FabricScanRequestH\x00R\n" +
 	"fabricScan\x12>\n" +
 	"\vfabric_test\x18\t \x01(\v2\x1b.llmgw.v1.FabricTestRequestH\x00R\n" +
-	"fabricTestB\x05\n" +
+	"fabricTest\x122\n" +
+	"\aupgrade\x18\n" +
+	" \x01(\v2\x16.llmgw.v1.UpgradeAgentH\x00R\aupgradeB\x05\n" +
 	"\x03msg\"\xe3\x06\n" +
 	"\bRegister\x12\x17\n" +
 	"\anode_id\x18\x01 \x01(\tR\x06nodeId\x12\x1a\n" +
@@ -3795,7 +3963,17 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
 	"\x04gbps\x18\x02 \x01(\x01R\x04gbps\x12\x16\n" +
 	"\x06output\x18\x03 \x01(\tR\x06output\x12\x14\n" +
 	"\x05error\x18\x04 \x01(\tR\x05error\x12\x17\n" +
-	"\atook_ms\x18\x05 \x01(\x03R\x06tookMs2O\n" +
+	"\atook_ms\x18\x05 \x01(\x03R\x06tookMs\"c\n" +
+	"\fUpgradeAgent\x12\x1d\n" +
+	"\n" +
+	"request_id\x18\x01 \x01(\tR\trequestId\x12\x18\n" +
+	"\aversion\x18\x02 \x01(\tR\aversion\x12\x1a\n" +
+	"\breleases\x18\x03 \x01(\tR\breleases\"]\n" +
+	"\fUpgradeReply\x12\x1d\n" +
+	"\n" +
+	"request_id\x18\x01 \x01(\tR\trequestId\x12\x18\n" +
+	"\aversion\x18\x02 \x01(\tR\aversion\x12\x14\n" +
+	"\x05error\x18\x03 \x01(\tR\x05error2O\n" +
 	"\fAgentService\x12?\n" +
 	"\aConnect\x12\x16.llmgw.v1.AgentMessage\x1a\x18.llmgw.v1.ControlMessage(\x010\x01B\x93\x01\n" +
 	"\fcom.llmgw.v1B\n" +
@@ -3814,7 +3992,7 @@ func file_llmgw_v1_agent_proto_rawDescGZIP() []byte {
 }
 
 var file_llmgw_v1_agent_proto_enumTypes = make([]protoimpl.EnumInfo, 1)
-var file_llmgw_v1_agent_proto_msgTypes = make([]protoimpl.MessageInfo, 39)
+var file_llmgw_v1_agent_proto_msgTypes = make([]protoimpl.MessageInfo, 41)
 var file_llmgw_v1_agent_proto_goTypes = []any{
 	(ShardStatus_State)(0),    // 0: llmgw.v1.ShardStatus.State
 	(*AgentMessage)(nil),      // 1: llmgw.v1.AgentMessage
@@ -3850,12 +4028,14 @@ var file_llmgw_v1_agent_proto_goTypes = []any{
 	(*FabricScanReply)(nil),   // 31: llmgw.v1.FabricScanReply
 	(*FabricTestRequest)(nil), // 32: llmgw.v1.FabricTestRequest
 	(*FabricTestReply)(nil),   // 33: llmgw.v1.FabricTestReply
-	nil,                       // 34: llmgw.v1.Register.LabelsEntry
-	nil,                       // 35: llmgw.v1.Register.EngineVersionsEntry
-	nil,                       // 36: llmgw.v1.GPU.NicPathsEntry
-	nil,                       // 37: llmgw.v1.IBPort.CountersEntry
-	nil,                       // 38: llmgw.v1.Heartbeat.EngineVersionsEntry
-	nil,                       // 39: llmgw.v1.EngineProfile.EnvEntry
+	(*UpgradeAgent)(nil),      // 34: llmgw.v1.UpgradeAgent
+	(*UpgradeReply)(nil),      // 35: llmgw.v1.UpgradeReply
+	nil,                       // 36: llmgw.v1.Register.LabelsEntry
+	nil,                       // 37: llmgw.v1.Register.EngineVersionsEntry
+	nil,                       // 38: llmgw.v1.GPU.NicPathsEntry
+	nil,                       // 39: llmgw.v1.IBPort.CountersEntry
+	nil,                       // 40: llmgw.v1.Heartbeat.EngineVersionsEntry
+	nil,                       // 41: llmgw.v1.EngineProfile.EnvEntry
 }
 var file_llmgw_v1_agent_proto_depIdxs = []int32{
 	3,  // 0: llmgw.v1.AgentMessage.register:type_name -> llmgw.v1.Register
@@ -3864,50 +4044,52 @@ var file_llmgw_v1_agent_proto_depIdxs = []int32{
 	29, // 3: llmgw.v1.AgentMessage.logs:type_name -> llmgw.v1.LogsReply
 	31, // 4: llmgw.v1.AgentMessage.fabric_scan:type_name -> llmgw.v1.FabricScanReply
 	33, // 5: llmgw.v1.AgentMessage.fabric_test:type_name -> llmgw.v1.FabricTestReply
-	22, // 6: llmgw.v1.ControlMessage.start:type_name -> llmgw.v1.StartShard
-	25, // 7: llmgw.v1.ControlMessage.stop:type_name -> llmgw.v1.StopShard
-	26, // 8: llmgw.v1.ControlMessage.drain:type_name -> llmgw.v1.DrainShard
-	27, // 9: llmgw.v1.ControlMessage.ack:type_name -> llmgw.v1.Ack
-	24, // 10: llmgw.v1.ControlMessage.remove_weights:type_name -> llmgw.v1.RemoveWeights
-	17, // 11: llmgw.v1.ControlMessage.provision:type_name -> llmgw.v1.Provision
-	28, // 12: llmgw.v1.ControlMessage.logs:type_name -> llmgw.v1.LogsRequest
-	30, // 13: llmgw.v1.ControlMessage.fabric_scan:type_name -> llmgw.v1.FabricScanRequest
-	32, // 14: llmgw.v1.ControlMessage.fabric_test:type_name -> llmgw.v1.FabricTestRequest
-	34, // 15: llmgw.v1.Register.labels:type_name -> llmgw.v1.Register.LabelsEntry
-	7,  // 16: llmgw.v1.Register.gpus:type_name -> llmgw.v1.GPU
-	8,  // 17: llmgw.v1.Register.fabrics:type_name -> llmgw.v1.FabricAttachment
-	35, // 18: llmgw.v1.Register.engine_versions:type_name -> llmgw.v1.Register.EngineVersionsEntry
-	6,  // 19: llmgw.v1.Register.running_shards:type_name -> llmgw.v1.RunningShard
-	9,  // 20: llmgw.v1.Register.weights:type_name -> llmgw.v1.LocalWeights
-	5,  // 21: llmgw.v1.Register.storage:type_name -> llmgw.v1.Storage
-	4,  // 22: llmgw.v1.Register.host:type_name -> llmgw.v1.Host
-	14, // 23: llmgw.v1.Register.imex:type_name -> llmgw.v1.IMEX
-	0,  // 24: llmgw.v1.RunningShard.state:type_name -> llmgw.v1.ShardStatus.State
-	36, // 25: llmgw.v1.GPU.nic_paths:type_name -> llmgw.v1.GPU.NicPathsEntry
-	11, // 26: llmgw.v1.NodeLoad.shards:type_name -> llmgw.v1.ShardLoad
-	37, // 27: llmgw.v1.IBPort.counters:type_name -> llmgw.v1.IBPort.CountersEntry
-	13, // 28: llmgw.v1.IBPort.subnet_managers:type_name -> llmgw.v1.SubnetManager
-	5,  // 29: llmgw.v1.Heartbeat.storage:type_name -> llmgw.v1.Storage
-	9,  // 30: llmgw.v1.Heartbeat.weights:type_name -> llmgw.v1.LocalWeights
-	38, // 31: llmgw.v1.Heartbeat.engine_versions:type_name -> llmgw.v1.Heartbeat.EngineVersionsEntry
-	19, // 32: llmgw.v1.Heartbeat.provision:type_name -> llmgw.v1.ProvisionStatus
-	10, // 33: llmgw.v1.Heartbeat.load:type_name -> llmgw.v1.NodeLoad
-	12, // 34: llmgw.v1.Heartbeat.ib_ports:type_name -> llmgw.v1.IBPort
-	8,  // 35: llmgw.v1.Heartbeat.fabrics:type_name -> llmgw.v1.FabricAttachment
-	15, // 36: llmgw.v1.Heartbeat.gpu_fabric:type_name -> llmgw.v1.GPUFabricState
-	14, // 37: llmgw.v1.Heartbeat.imex:type_name -> llmgw.v1.IMEX
-	18, // 38: llmgw.v1.Provision.engines:type_name -> llmgw.v1.EngineWant
-	0,  // 39: llmgw.v1.ShardStatus.state:type_name -> llmgw.v1.ShardStatus.State
-	21, // 40: llmgw.v1.ShardStatus.fetch:type_name -> llmgw.v1.FetchProgress
-	23, // 41: llmgw.v1.StartShard.profile:type_name -> llmgw.v1.EngineProfile
-	39, // 42: llmgw.v1.EngineProfile.env:type_name -> llmgw.v1.EngineProfile.EnvEntry
-	1,  // 43: llmgw.v1.AgentService.Connect:input_type -> llmgw.v1.AgentMessage
-	2,  // 44: llmgw.v1.AgentService.Connect:output_type -> llmgw.v1.ControlMessage
-	44, // [44:45] is the sub-list for method output_type
-	43, // [43:44] is the sub-list for method input_type
-	43, // [43:43] is the sub-list for extension type_name
-	43, // [43:43] is the sub-list for extension extendee
-	0,  // [0:43] is the sub-list for field type_name
+	35, // 6: llmgw.v1.AgentMessage.upgrade:type_name -> llmgw.v1.UpgradeReply
+	22, // 7: llmgw.v1.ControlMessage.start:type_name -> llmgw.v1.StartShard
+	25, // 8: llmgw.v1.ControlMessage.stop:type_name -> llmgw.v1.StopShard
+	26, // 9: llmgw.v1.ControlMessage.drain:type_name -> llmgw.v1.DrainShard
+	27, // 10: llmgw.v1.ControlMessage.ack:type_name -> llmgw.v1.Ack
+	24, // 11: llmgw.v1.ControlMessage.remove_weights:type_name -> llmgw.v1.RemoveWeights
+	17, // 12: llmgw.v1.ControlMessage.provision:type_name -> llmgw.v1.Provision
+	28, // 13: llmgw.v1.ControlMessage.logs:type_name -> llmgw.v1.LogsRequest
+	30, // 14: llmgw.v1.ControlMessage.fabric_scan:type_name -> llmgw.v1.FabricScanRequest
+	32, // 15: llmgw.v1.ControlMessage.fabric_test:type_name -> llmgw.v1.FabricTestRequest
+	34, // 16: llmgw.v1.ControlMessage.upgrade:type_name -> llmgw.v1.UpgradeAgent
+	36, // 17: llmgw.v1.Register.labels:type_name -> llmgw.v1.Register.LabelsEntry
+	7,  // 18: llmgw.v1.Register.gpus:type_name -> llmgw.v1.GPU
+	8,  // 19: llmgw.v1.Register.fabrics:type_name -> llmgw.v1.FabricAttachment
+	37, // 20: llmgw.v1.Register.engine_versions:type_name -> llmgw.v1.Register.EngineVersionsEntry
+	6,  // 21: llmgw.v1.Register.running_shards:type_name -> llmgw.v1.RunningShard
+	9,  // 22: llmgw.v1.Register.weights:type_name -> llmgw.v1.LocalWeights
+	5,  // 23: llmgw.v1.Register.storage:type_name -> llmgw.v1.Storage
+	4,  // 24: llmgw.v1.Register.host:type_name -> llmgw.v1.Host
+	14, // 25: llmgw.v1.Register.imex:type_name -> llmgw.v1.IMEX
+	0,  // 26: llmgw.v1.RunningShard.state:type_name -> llmgw.v1.ShardStatus.State
+	38, // 27: llmgw.v1.GPU.nic_paths:type_name -> llmgw.v1.GPU.NicPathsEntry
+	11, // 28: llmgw.v1.NodeLoad.shards:type_name -> llmgw.v1.ShardLoad
+	39, // 29: llmgw.v1.IBPort.counters:type_name -> llmgw.v1.IBPort.CountersEntry
+	13, // 30: llmgw.v1.IBPort.subnet_managers:type_name -> llmgw.v1.SubnetManager
+	5,  // 31: llmgw.v1.Heartbeat.storage:type_name -> llmgw.v1.Storage
+	9,  // 32: llmgw.v1.Heartbeat.weights:type_name -> llmgw.v1.LocalWeights
+	40, // 33: llmgw.v1.Heartbeat.engine_versions:type_name -> llmgw.v1.Heartbeat.EngineVersionsEntry
+	19, // 34: llmgw.v1.Heartbeat.provision:type_name -> llmgw.v1.ProvisionStatus
+	10, // 35: llmgw.v1.Heartbeat.load:type_name -> llmgw.v1.NodeLoad
+	12, // 36: llmgw.v1.Heartbeat.ib_ports:type_name -> llmgw.v1.IBPort
+	8,  // 37: llmgw.v1.Heartbeat.fabrics:type_name -> llmgw.v1.FabricAttachment
+	15, // 38: llmgw.v1.Heartbeat.gpu_fabric:type_name -> llmgw.v1.GPUFabricState
+	14, // 39: llmgw.v1.Heartbeat.imex:type_name -> llmgw.v1.IMEX
+	18, // 40: llmgw.v1.Provision.engines:type_name -> llmgw.v1.EngineWant
+	0,  // 41: llmgw.v1.ShardStatus.state:type_name -> llmgw.v1.ShardStatus.State
+	21, // 42: llmgw.v1.ShardStatus.fetch:type_name -> llmgw.v1.FetchProgress
+	23, // 43: llmgw.v1.StartShard.profile:type_name -> llmgw.v1.EngineProfile
+	41, // 44: llmgw.v1.EngineProfile.env:type_name -> llmgw.v1.EngineProfile.EnvEntry
+	1,  // 45: llmgw.v1.AgentService.Connect:input_type -> llmgw.v1.AgentMessage
+	2,  // 46: llmgw.v1.AgentService.Connect:output_type -> llmgw.v1.ControlMessage
+	46, // [46:47] is the sub-list for method output_type
+	45, // [45:46] is the sub-list for method input_type
+	45, // [45:45] is the sub-list for extension type_name
+	45, // [45:45] is the sub-list for extension extendee
+	0,  // [0:45] is the sub-list for field type_name
 }
 
 func init() { file_llmgw_v1_agent_proto_init() }
@@ -3922,6 +4104,7 @@ func file_llmgw_v1_agent_proto_init() {
 		(*AgentMessage_Logs)(nil),
 		(*AgentMessage_FabricScan)(nil),
 		(*AgentMessage_FabricTest)(nil),
+		(*AgentMessage_Upgrade)(nil),
 	}
 	file_llmgw_v1_agent_proto_msgTypes[1].OneofWrappers = []any{
 		(*ControlMessage_Start)(nil),
@@ -3933,6 +4116,7 @@ func file_llmgw_v1_agent_proto_init() {
 		(*ControlMessage_Logs)(nil),
 		(*ControlMessage_FabricScan)(nil),
 		(*ControlMessage_FabricTest)(nil),
+		(*ControlMessage_Upgrade)(nil),
 	}
 	type x struct{}
 	out := protoimpl.TypeBuilder{
@@ -3940,7 +4124,7 @@ func file_llmgw_v1_agent_proto_init() {
 			GoPackagePath: reflect.TypeOf(x{}).PkgPath(),
 			RawDescriptor: unsafe.Slice(unsafe.StringData(file_llmgw_v1_agent_proto_rawDesc), len(file_llmgw_v1_agent_proto_rawDesc)),
 			NumEnums:      1,
-			NumMessages:   39,
+			NumMessages:   41,
 			NumExtensions: 0,
 			NumServices:   1,
 		},

api/proto/llmgw/v1/agent.proto

@@ -26,6 +26,7 @@ message AgentMessage {
     LogsReply logs = 4;
     FabricScanReply fabric_scan = 5;
     FabricTestReply fabric_test = 6;
+    UpgradeReply upgrade = 7;
   }
 }
 
@@ -41,6 +42,7 @@ message ControlMessage {
     LogsRequest logs = 7;
     FabricScanRequest fabric_scan = 8;
     FabricTestRequest fabric_test = 9;
+    UpgradeAgent upgrade = 10;
   }
 }
 
@@ -649,3 +651,28 @@ message FabricTestReply {
   string error = 4;
   int64 took_ms = 5;
 }
+
+// UpgradeAgent tells an agent to replace its own binary with a release and
+// restart.
+//
+// The agent fetches the release for its own platform and checks it against
+// the release's checksums before replacing anything, keeps the old binary
+// beside the new, then exits so systemd starts the new one. Engines keep
+// running throughout, as for any agent restart, and are adopted on the way
+// back. Over the stream the agent already holds, so upgrading a fleet needs
+// no ssh into it.
+message UpgradeAgent {
+  string request_id = 1;
+  // version must be a release: v0.10.4 or v0.10.4-rc.2.
+  string version = 2;
+  // releases is the base URL release files are downloaded from.
+  string releases = 3;
+}
+
+// UpgradeReply says whether the new binary is in place. Sent before the
+// agent exits; an empty error means it will be back on the new version.
+message UpgradeReply {
+  string request_id = 1;
+  string version = 2;
+  string error = 3;
+}

deploy/bootstrap/onboard-node.sh

@@ -122,9 +122,10 @@ TMP="$(mktemp -d)"
 trap 'rm -rf "$TMP"' EXIT
 "$BINARY" ctl cert issue-client "$NODE" --dir "$PKI" --out "$TMP" >/dev/null
 
-# 2 and 3. Identity, and trusting the root so future tooling on that machine
-#    does not treat our certificates as strangers.
-say "installing the certificate and trusting the root"
+# 2 and 3. Identity. The root is trusted by the agent's configuration only,
+#    never system-wide: it signs whatever the control plane asks, and every
+#    TLS client on the machine would take it for any site.
+say "installing the certificate"
 scp -q $SSHOPTS "$PKI/ca.pem" "$TMP/$NODE.pem" "$TMP/$NODE-key.pem" "$SSH_USER@$ADDR:/tmp/"
 $SSH "set -e
   sudo install -d -m 0755 /etc/llmgw
@@ -132,8 +133,10 @@ $SSH "set -e
   sudo install -m 0644 /tmp/$NODE.pem /etc/llmgw/agent.pem
   sudo install -m 0600 /tmp/$NODE-key.pem /etc/llmgw/agent-key.pem
   sudo chown root:root /etc/llmgw/*
-  sudo install -m 0644 /tmp/ca.pem /usr/local/share/ca-certificates/llmgw-ca.crt
-  sudo update-ca-certificates >/dev/null 2>&1 || true
+  if [ -f /usr/local/share/ca-certificates/llmgw-ca.crt ]; then
+    sudo rm -f /usr/local/share/ca-certificates/llmgw-ca.crt
+    sudo update-ca-certificates --fresh >/dev/null 2>&1 || true
+  fi
   rm -f /tmp/ca.pem /tmp/$NODE.pem /tmp/$NODE-key.pem"
 
 # 4. The binary, the configuration and the unit.

deploy/monitoring/README.md

@@ -46,6 +46,19 @@ apt-get install prometheus prometheus-alertmanager
 install -m 0644 prometheus.yml            /etc/prometheus/prometheus.yml
 install -m 0644 llmgw-rules.yml           /etc/prometheus/rules/llmgw.yml
 install -m 0644 alertmanager.yml          /etc/prometheus/alertmanager.yml
+# Alertmanager answers on loopback only. Its API takes silences from anyone who
+# can reach it, with no login, and silencing BackupTooOld or ModelParked is how
+# something that should be noticed is not. Prometheus beside it is its only
+# client; one Alertmanager needs no cluster port.
+echo 'ARGS="--web.listen-address=127.0.0.1:9093 --cluster.listen-address="' \
+    > /etc/default/prometheus-alertmanager
+# Prometheus itself is read-only as installed: its admin and lifecycle APIs
+# stay off (no --web.enable-admin-api, no --web.enable-lifecycle). It still
+# answers any query from anyone who reaches :9090, which says which models and
+# nodes there are and how busy each is. Its readers are the control plane
+# (monitoring.prometheus) and Grafana beside it; to keep it to those, allow
+# the control plane and drop the rest, e.g. with nftables:
+#   nft add rule inet filter input tcp dport 9090 ip saddr != { 127.0.0.1, CONTROL_PLANE_IP } drop
 promtool check config /etc/prometheus/prometheus.yml
 systemctl restart prometheus prometheus-alertmanager
 
@@ -75,7 +88,8 @@ Alertmanager groups and holds alerts and has no receiver.
 Where an alert should go — an address, a Slack workspace, a webhook — is a
 decision with credentials attached, and a monitoring system that quietly starts
 mailing people is worse than one that is visibly not wired up. Alerts are
-visible at `:9093` and in Prometheus; add a receiver to `alertmanager.yml` to
+visible in Prometheus, and at `:9093` on the monitoring host itself (`ssh -L
+9093:127.0.0.1:9093 <host>` to open its page); add a receiver to `alertmanager.yml` to
 make them travel. `critical` already routes separately so it can be pointed
 somewhere else without touching the rest.
 
@@ -86,7 +100,7 @@ An alerting system nobody has seen fire is an assumption.
 ```
 llmgw ctl cordon <node>     # work already there keeps running
 # ~2 minutes later
-curl -s http://<host>:9093/api/v2/alerts | jq '.[].labels'
+ssh <host> curl -s http://127.0.0.1:9093/api/v2/alerts | jq '.[].labels'
 llmgw ctl uncordon <node>   # the alert resolves on its own
 ```
 

deploy/systemd/llmgw-backup.service

@@ -4,6 +4,9 @@ After=llmgw-server.service postgresql.service
 
 [Service]
 Type=oneshot
+# Nothing it writes is for anyone else: the archive holds the CA's key and a
+# copy of the store.
+UMask=0077
 # Root, because the archive includes /etc/llmgw/secrets and the certificate
 # authority's key, which the llmgw user deliberately cannot read all of.
 ExecStart=/usr/local/bin/llmgw server backup --config /etc/llmgw/server.yaml --keep 14

docs/install.md

@@ -909,55 +909,52 @@ refuses every request.
 
 ## Upgrading
 
-One binary on every machine, so an upgrade is a copy and a restart. The part
-worth care is the copy, not the order.
+On the control plane:
+
+```
+sudo llmgw ctl upgrade v0.10.4
+```
+
+It upgrades this machine first — fetching the release for its platform,
+checking it against the release's checksums, keeping the old binary as
+`/usr/local/bin/llmgw.bak-pre-<version>`, restarting `llmgw-server` and waiting
+for every node to reconnect — and then each node in turn. A node's agent is told
+the version over the stream it already holds, fetches and checks the release
+itself, swaps it in and restarts; no ssh into the nodes is needed. Release
+candidates (`v0.10.4-rc.2`) work the same way.
+
+A node counts as done when it is back on the new version, ready, with as many
+ready replicas as before. One that is not within `--timeout` (three minutes)
+stops the upgrade there, so a bad release reaches one node rather than all of
+them; `llmgw ctl logs <node>` says why, and the node keeps its previous binary
+beside the new one. Nodes already on the version are skipped, so running the
+command again resumes where it stopped. `--nodes a,b` limits it to some nodes,
+and `--skip-control-plane` leaves this machine alone.
 
 **Nothing stops serving.** Engines are started with `KillMode=process`, so they
 keep running when the agent restarts, and the agent reports them back on its
 next connection. The control plane holds no replica state across its own
-restart either — it rebuilds the fleet from what the agents report. A rolling
-upgrade of the whole cluster interrupts no request that is not in flight at the
-moment of the restart.
+restart either — it rebuilds the fleet from what the agents report.
 
 **A mixed fleet is fine while you roll.** The agent protocol only ever gains
 fields, and both sides treat an absent one as unknown rather than as a default.
-Order does not matter; finish the roll the same day anyway, because
-`ctl nodes` reports each agent's version and a fleet that disagrees with itself
-is one more thing to hold in your head.
-
-```
-VERSION=v0.9.0
-# copy to every machine first, install nowhere yet
-for h in 10.0.0.140 10.0.0.141 10.0.0.142 10.0.0.146; do
-  ssh $h 'cat > /tmp/llmgw-new; stat -c %s /tmp/llmgw-new' < llmgw-$VERSION-linux-amd64
-done
-```
-
-**Check every size before installing anything.** This is not defensive writing:
-on a host whose login shell prints anything, `scp` fails outright, and the
-obvious workaround — `ssh -n host 'cat > file'` — points stdin at `/dev/null`
-and writes a *zero-byte file* while reporting success. Installing that on a
-fleet takes the fleet down. Compare the number on each machine with the local
-one, then:
-
-```
-sudo install -m0755 /tmp/llmgw-new /usr/local/bin/llmgw
-sudo systemctl restart llmgw-agent     # or llmgw-server, or llmgw-mirror
-llmgw --version
-```
+Agents older than v0.10.4 cannot upgrade themselves: move those by hand once,
+installing the release binary and restarting `llmgw-agent`, and every upgrade
+after that is the command above. The mirror, which is not an agent, is
+upgraded the same way by hand.
 
 Then confirm what you have, rather than assuming:
 
 ```
-llmgw ctl nodes       # every agent back, LAST SEEN in seconds
+llmgw ctl nodes       # every agent on the new version, LAST SEEN in seconds
 llmgw ctl replicas    # same replicas, adopted rather than replaced
 ```
 
 A replica may come back marked **`ready (stale profile)`**. That means it is
 running a profile that has since been edited — the reconciler leaves a serving
 replica alone on purpose, because churning one costs more than the difference.
-Restart it when you want the new profile to take effect; the flag is there so
-the choice is yours rather than silent.
+`llmgw ctl restart <replica>` replaces it with one on the new profile, starting
+the new one before stopping the old.
 
 ## Backing up
 
@@ -1127,3 +1124,11 @@ requests. On a slow model, set `timeouts.generation` higher.
   has them. Single-node placement on real GPUs is the well-trodden path; if you
   are the first to run a TP group across real cards, run `llmgw doctor --json`
   on each node first and keep the reports.
+- **Two engine setups are weaker than the rest.** An engine's key is passed
+  in its environment, where only root and its own user can read it, except
+  SGLang's, which takes it only as a flag: other users on the node can see it
+  in the process list. Mount `/proc` with `hidepid=2` on nodes where that
+  matters. vLLM's `launch_mode: ray` starts a Ray head without
+  authentication, so anyone who can reach its port can run code on the
+  node. The dashboard is off, but firewall the port to the other nodes in the
+  replica, or use the default `mp` launch mode, which needs no Ray at all.

docs/reference.md

@@ -81,6 +81,7 @@ rather than repeated.
 | `notice` | `string` | [Landing](#landing) |
 | `bin` | `string` | [LlamaCppConfig](#llamacppconfig) |
 | `parallel` | `int` | [LlamaCppConfig](#llamacppconfig) |
+| `threads` | `int` | [LlamaCppConfig](#llamacppconfig) |
 | `api` | `string` | [Mirror](#mirror) |
 | `tls` | `tlsconf.Config` | [Mirror](#mirror) |
 | `registry` | `string` | [Mirror](#mirror) |
@@ -167,9 +168,11 @@ rather than repeated.
 | `landing` | `Landing` | [Server](#server) |
 | `agent_tls` | `tlsconf.Config` | [Server](#server) |
 | `monitoring` | `Monitoring` | [Server](#server) |
+| `tracing` | `Tracing` | [Server](#server) |
 | `mirror` | `Mirror` | [Server](#server) |
 | `engines` | `map[string]string` | [Server](#server) |
 | `backup_dir` | `string` | [Server](#server) |
+| `releases` | `string` | [Server](#server) |
 | `router` | `Router` | [Server](#server) |
 | `control` | `Control` | [Server](#server) |
 | `models` | `[]catalog.Model` | [Server](#server) |
@@ -183,6 +186,8 @@ rather than repeated.
 | `first_token` | `time.Duration` | [Timeouts](#timeouts) |
 | `idle` | `time.Duration` | [Timeouts](#timeouts) |
 | `total` | `time.Duration` | [Timeouts](#timeouts) |
+| `endpoint` | `string` | [Tracing](#tracing) |
+| `sample_ratio` | `float64` | [Tracing](#tracing) |
 | `bin` | `string` | [VLLMConfig](#vllmconfig) |
 | `launch_mode` | `string` | [VLLMConfig](#vllmconfig) |
 | `ray_bin` | `string` | [VLLMConfig](#vllmconfig) |
@@ -633,6 +638,20 @@ Parallel is how many requests a server handles at once, when a profile
 does not say. llama.cpp defaults to one, which would make every second
 request wait behind the first no matter what the router decided.
 
+### `threads`
+
+`int`
+
+Threads is how many CPU threads a shard with no GPU computes on. Zero
+takes half this machine's CPUs.
+
+Not all of them, which is llama.cpp's own default: two busy servers
+that each spin on every CPU fight for them, and on an 8-vCPU node
+that made 7B generation fall from 10 tokens a second to under 1 each.
+At half, one server alone is about a third slower and two together
+keep 5.5 each. A node that only ever runs one CPU replica does better
+with all of them; say so here.
+
 ## Mirror
 
 ### `api`
@@ -1245,6 +1264,16 @@ Monitoring is where the dashboard reads its history from.
 
 Takes a [Monitoring](#monitoring) block.
 
+### `tracing`
+
+`Tracing`
+
+Tracing sends a trace of each inference request to an OpenTelemetry
+collector. Off unless an endpoint is set here or in
+OTEL_EXPORTER_OTLP_ENDPOINT.
+
+Takes a [Tracing](#tracing) block.
+
 ### `mirror`
 
 `Mirror`
@@ -1274,6 +1303,14 @@ BackupDir is where `llmgw server backup` writes, and where the newest
 archive's age is read for llmgw_backup_last_success_timestamp_seconds.
 Empty means /var/backups/llmgw-server.
 
+### `releases`
+
+`string`
+
+Releases is where `llmgw ctl upgrade` fetches release binaries from,
+for this machine and — through their agents — every node. Empty means
+the project's own releases.
+
 ### `router`
 
 `Router`
@@ -1357,6 +1394,22 @@ about a hundred tokens on a slow model.
 
 `time.Duration`
 
+## Tracing
+
+### `endpoint`
+
+`string`
+
+Endpoint is the collector, for example http://otel.internal:4318.
+Empty falls back to OTEL_EXPORTER_OTLP_ENDPOINT, and to no tracing.
+
+### `sample_ratio`
+
+`float64`
+
+SampleRatio is the share of requests traced, 0 to 1, when the caller
+did not decide it with a sampled traceparent of its own. Zero takes 1.
+
 ## VLLMConfig
 
 ### `bin`

go.mod

@@ -19,6 +19,10 @@ require (
 	github.com/prometheus/common v0.71.0
 	github.com/spf13/cobra v1.10.2
 	github.com/yuin/goldmark v1.8.6
+	go.opentelemetry.io/otel v1.46.0
+	go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.46.0
+	go.opentelemetry.io/otel/sdk v1.46.0
+	go.opentelemetry.io/otel/trace v1.46.0
 	golang.org/x/crypto v0.57.0
 	golang.org/x/net v0.59.0
 	golang.org/x/sync v0.23.0
@@ -29,6 +33,7 @@ require (
 require (
 	github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
 	github.com/beorn7/perks v1.0.1 // indirect
+	github.com/cenkalti/backoff/v5 v5.0.3 // indirect
 	github.com/charmbracelet/colorprofile v0.4.1 // indirect
 	github.com/charmbracelet/x/ansi v0.11.6 // indirect
 	github.com/charmbracelet/x/cellbuf v0.0.15 // indirect
@@ -39,8 +44,11 @@ require (
 	github.com/dustin/go-humanize v1.0.1 // indirect
 	github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f // indirect
 	github.com/fsnotify/fsnotify v1.9.0 // indirect
-	github.com/go-viper/mapstructure/v2 v2.4.0 // indirect
+	github.com/go-logr/logr v1.4.4 // indirect
+	github.com/go-logr/stdr v1.2.2 // indirect
+	github.com/go-viper/mapstructure/v2 v2.5.0 // indirect
 	github.com/google/uuid v1.6.0 // indirect
+	github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0 // indirect
 	github.com/inconshreveable/mousetrap v1.1.0 // indirect
 	github.com/jackc/pgpassfile v1.0.0 // indirect
 	github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
@@ -61,9 +69,16 @@ require (
 	github.com/rivo/uniseg v0.4.7 // indirect
 	github.com/spf13/pflag v1.0.9 // indirect
 	github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e // indirect
+	go.opentelemetry.io/auto/sdk v1.2.1 // indirect
+	go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.46.0 // indirect
+	go.opentelemetry.io/otel/metric v1.46.0 // indirect
+	go.opentelemetry.io/proto/otlp v1.11.0 // indirect
 	go.yaml.in/yaml/v3 v3.0.5 // indirect
 	golang.org/x/sys v0.48.0 // indirect
 	golang.org/x/text v0.42.0 // indirect
+	google.golang.org/genproto/googleapis/api v0.0.0-20260819154853-08b0e4226688 // indirect
+	google.golang.org/genproto/googleapis/rpc v0.0.0-20260819154853-08b0e4226688 // indirect
+	google.golang.org/grpc v1.83.1 // indirect
 	modernc.org/libc v1.75.7 // indirect
 	modernc.org/mathutil v1.7.1 // indirect
 	modernc.org/memory v1.12.1 // indirect

go.sum

@@ -6,6 +6,8 @@ github.com/aymanbagabas/go-osc52/v2 v2.0.1 h1:HwpRHbFMcZLEVr42D4p7XBqjyuxQH5SMiE
 github.com/aymanbagabas/go-osc52/v2 v2.0.1/go.mod h1:uYgXzlJ7ZpABp8OJ+exZzJJhRNQ2ASbcXHWsFqH8hp8=
 github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
 github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
+github.com/cenkalti/backoff/v5 v5.0.3 h1:ZN+IMa753KfX5hd8vVaMixjnqRZ3y8CuJKRKj1xcsSM=
+github.com/cenkalti/backoff/v5 v5.0.3/go.mod h1:rkhZdG3JZukswDf7f0cwqPNk4K0sa+F97BxZthm/crw=
 github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
 github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
 github.com/charmbracelet/bubbletea v1.3.10 h1:otUDHWMMzQSB0Pkc87rm691KZ3SWa4KUlvF9nRvCICw=
@@ -34,14 +36,23 @@ github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f h1:Y/CXytFA4m6
 github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f/go.mod h1:vw97MGsxSvLiUE2X8qFplwetxpGLQrlU1Q9AUEIzCaM=
 github.com/fsnotify/fsnotify v1.9.0 h1:2Ml+OJNzbYCTzsxtv8vKSFD9PbJjmhYF14k/jKC7S9k=
 github.com/fsnotify/fsnotify v1.9.0/go.mod h1:8jBTzvmWwFyi3Pb8djgCCO5IBqzKJ/Jwo8TRcHyHii0=
-github.com/go-viper/mapstructure/v2 v2.4.0 h1:EBsztssimR/CONLSZZ04E8qAkxNYq4Qp9LvH92wZUgs=
-github.com/go-viper/mapstructure/v2 v2.4.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM=
+github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
+github.com/go-logr/logr v1.4.4 h1:tG4xh9yMsRCAiodLVTxyrkzSZ9+o0L1Kg/+cPVcbP/8=
+github.com/go-logr/logr v1.4.4/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
+github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
+github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
+github.com/go-viper/mapstructure/v2 v2.5.0 h1:vM5IJoUAy3d7zRSVtIwQgBj7BiWtMPfmPEgAXnvj1Ro=
+github.com/go-viper/mapstructure/v2 v2.5.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM=
+github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek=
+github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps=
 github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
 github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
 github.com/google/pprof v0.0.0-20260802141513-ef3492d7dac3 h1:LMLX+LgTNWpfvCBdFebv6EsYotImrt/Ppc5cXIriCSo=
 github.com/google/pprof v0.0.0-20260802141513-ef3492d7dac3/go.mod h1:jl5iWTm0/hd5PjEYEOuwAJ57L/CibdZfrqZ5XA5GrCk=
 github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
 github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
+github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0 h1:/Tnpcb2E0Pz/tN9s3bfEY2Q8ePCEX9iuS+cneUwncnw=
+github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0/go.mod h1:zOBXOsUaBSjKgmH4OGzV1esUpR3oUSCPYVd2cUBjKYY=
 github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
 github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
 github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
@@ -117,6 +128,24 @@ github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e h1:JVG44RsyaB9T2KIHavM
 github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e/go.mod h1:RbqR21r5mrJuqunuUZ/Dhy/avygyECGrLceyNeo4LiM=
 github.com/yuin/goldmark v1.8.6 h1:d0VcaP1sx9GkFVkoW+KtggpGi2KZ965i14b0+bDQST4=
 github.com/yuin/goldmark v1.8.6/go.mod h1:ip/1k0VRfGynBgxOz0yCqHrbZXhcjxyuS66Brc7iBKg=
+go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64=
+go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y=
+go.opentelemetry.io/otel v1.46.0 h1:FHt5/CDyVxi/8IM1CH7VE/rRgq3kLHa2mSTVMO8AWyc=
+go.opentelemetry.io/otel v1.46.0/go.mod h1:Gj3SEScelsNC45tp4nSxRYlS+f5iez7W8XPMCt905kE=
+go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.46.0 h1:OFnwLJr+pF3iHrlGSzbxyuo6/6HyBlnlN1CWEJmBVcw=
+go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.46.0/go.mod h1:716wFneO0ov19A2beH5hjfh9AK5z/VWNAtDijp1Y0/g=
+go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.46.0 h1:KrC1YrQeSt46ITMWAbgQx1M1eV1/1TKzttrBzymPmss=
+go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.46.0/go.mod h1:zDSEzoEqsOrgBeGvH66KRgxh90VonFyJqBHA0Pk3+rM=
+go.opentelemetry.io/otel/metric v1.46.0 h1:yBnkXvgV7AXFILZc5K6IZe/CBFF3OS7BJ8ov6/lj0K8=
+go.opentelemetry.io/otel/metric v1.46.0/go.mod h1:iPmdWqifKUdzziPkvvzIJXITl56fQx2mGM/DHLB3/2o=
+go.opentelemetry.io/otel/sdk v1.46.0 h1:h5CNQQjEbuQXY/JfZtgt3i7HVFV3aHPO2OAwO2eTYPI=
+go.opentelemetry.io/otel/sdk v1.46.0/go.mod h1:GAERFXFt5SYCEB+YiKUbMBeza6UaDH7GmGOZEfh2gSM=
+go.opentelemetry.io/otel/sdk/metric v1.46.0 h1:0piZ26EG4RBfebb2jhDH6ERCYHoVWduc3kLgPCwSnSE=
+go.opentelemetry.io/otel/sdk/metric v1.46.0/go.mod h1:I1PbKrdVc8Qu8HYVDNtqVIwLwjNrhsV/uFuxfwg8mO4=
+go.opentelemetry.io/otel/trace v1.46.0 h1:OULy7ccdJnZtJ0UDYFOIGaCmiWzJ8Vi2G/Rsu60qs1c=
+go.opentelemetry.io/otel/trace v1.46.0/go.mod h1:J7GAXweO77XSFkB/rmAqk9D6ihszhFjLU+d9WuUxDLI=
+go.opentelemetry.io/proto/otlp v1.11.0 h1:5rrYs0Ykyj50sdU/JU0x8etU+LubXWb+gED6TbEdMIk=
+go.opentelemetry.io/proto/otlp v1.11.0/go.mod h1:SmVizdCOAm3XBtG1g1NnOdhW6jtddT72hLMhv8VwA8E=
 go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
 go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
 go.yaml.in/yaml/v2 v2.4.4 h1:tuyd0P+2Ont/d6e2rl3be67goVK4R6deVxCUX5vyPaQ=
@@ -141,6 +170,14 @@ golang.org/x/text v0.42.0 h1:JbOZXgfeCPU9gacVtYliJqOhD+zhrEqK4LfdpmlUZqI=
 golang.org/x/text v0.42.0/go.mod h1:ojzP1Z+2QtioaF8DTtO8K5q7JWVVYwZKenzujK0Zd0E=
 golang.org/x/tools v0.49.0 h1:3NI7VXzL9+1WZD52Dx2ttoPwD5DWrFGpl9mFZDlmisI=
 golang.org/x/tools v0.49.0/go.mod h1:SJNXV9DBKT0UbdttsQjbfJlAE/q+y36++zo3uL3N0Oo=
+gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4=
+gonum.org/v1/gonum v0.17.0/go.mod h1:El3tOrEuMpv2UdMrbNlKEh9vd86bmQ6vqIcDwxEOc1E=
+google.golang.org/genproto/googleapis/api v0.0.0-20260819154853-08b0e4226688 h1:ax2KzoSRIZU/M0cIxri3pKxy99vniH1PVxWC6si/eZI=
+google.golang.org/genproto/googleapis/api v0.0.0-20260819154853-08b0e4226688/go.mod h1:1RJ9BQGyNdZwkGc1eTqkErfRZ6RJyYPHZo73BZ1vQqI=
+google.golang.org/genproto/googleapis/rpc v0.0.0-20260819154853-08b0e4226688 h1:cYNAzI2sUwhmCcoj9TxvihSrqsxt6uIkj3rDRhSDmW4=
+google.golang.org/genproto/googleapis/rpc v0.0.0-20260819154853-08b0e4226688/go.mod h1:DjtHYE8FKJLivXcBEjGwndXfIC23G0VpXiXKqG179uA=
+google.golang.org/grpc v1.83.1 h1:HIO0+BEtBP6soyqvqC8sNUjZ7bTs+0hFQuFF+RAy++Y=
+google.golang.org/grpc v1.83.1/go.mod h1:kDyl6SKsiHKt0uylY5gtn5cEjkrIOhQOGDgIc4JGwzQ=
 google.golang.org/protobuf v1.36.12 h1:pJOKDDOyeXErUroCihFAd5LQuwXBSpVnKGrj5o/fwxc=
 google.golang.org/protobuf v1.36.12/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=

internal/account/password.go

@@ -15,6 +15,7 @@ import (
 	"encoding/base64"
 	"errors"
 	"fmt"
+	"runtime"
 	"strings"
 
 	"golang.org/x/crypto/argon2"
@@ -56,7 +57,7 @@ func HashPassword(password string) (string, error) {
 	if _, err := rand.Read(salt); err != nil {
 		return "", err
 	}
-	sum := argon2.IDKey([]byte(password), salt, argonTime, argonMemory, argonThreads, argonKeyLen)
+	sum := idKey([]byte(password), salt, argonTime, argonMemory, argonThreads, argonKeyLen)
 
 	b64 := base64.RawStdEncoding.EncodeToString
 	return fmt.Sprintf("$argon2id$v=%d$m=%d,t=%d,p=%d$%s$%s",
@@ -93,7 +94,7 @@ func VerifyPassword(password, encoded string) (bool, error) {
 		return false, ErrBadHash
 	}
 
-	got := argon2.IDKey([]byte(password), salt, time, memory, threads, uint32(len(want)))
+	got := idKey([]byte(password), salt, time, memory, threads, uint32(len(want)))
 	return subtle.ConstantTimeCompare(got, want) == 1, nil
 }
 
@@ -110,3 +111,16 @@ func init() {
 	}
 	dummyHash = h
 }
+
+// argonSlots bounds how many hashes run at once. Each takes 64 MiB, and the
+// login and signup forms are on the client-facing port: without a bound, a
+// few hundred parallel attempts at random addresses asked for gigabytes at
+// once, and the per-address throttle does not see them — each address is new.
+// Beyond the bound, attempts wait their turn instead.
+var argonSlots = make(chan struct{}, max(2, runtime.NumCPU()/2))
+
+func idKey(password, salt []byte, time, memory uint32, threads uint8, keyLen uint32) []byte {
+	argonSlots <- struct{}{}
+	defer func() { <-argonSlots }()
+	return argon2.IDKey(password, salt, time, memory, threads, keyLen)
+}

internal/account/service.go

@@ -230,6 +230,22 @@ func (s *Service) allowAttempt(email string) bool {
 	defer s.mu.Unlock()
 
 	w, ok := s.attempts[email]
+	if !ok && len(s.attempts) >= maxTrackedAddresses {
+		// Full of addresses nobody has signed in as for a while: forget the
+		// ones whose window has passed. Only expiring, never evicting early —
+		// dropping a live window is how an attacker spraying random addresses
+		// would buy unlimited guesses at the one they care about.
+		for k, v := range s.attempts {
+			if now.Sub(v.since) > s.cfg.AttemptWindow {
+				delete(s.attempts, k)
+			}
+		}
+		if len(s.attempts) >= maxTrackedAddresses {
+			// Still full, so somebody is spraying addresses faster than they
+			// expire. New ones wait; those already tracked are unaffected.
+			return false
+		}
+	}
 	if !ok || now.Sub(w.since) > s.cfg.AttemptWindow {
 		s.attempts[email] = &attemptWindow{count: 1, since: now}
 		return true
@@ -238,6 +254,10 @@ func (s *Service) allowAttempt(email string) bool {
 	return w.count <= s.cfg.MaxAttempts
 }
 
+// maxTrackedAddresses bounds the throttle's memory. Unbounded, it grew by one
+// entry per address tried and shrank only on a successful sign-in.
+const maxTrackedAddresses = 50000
+
 func (s *Service) clearAttempts(email string) {
 	s.mu.Lock()
 	delete(s.attempts, email)

internal/account/throttle_bound_test.go

@@ -0,0 +1,33 @@
+package account
+
+import (
+	"fmt"
+	"testing"
+	"time"
+)
+
+func TestSprayingAddressesCannotGrowTheThrottleOrEscapeIt(t *testing.T) {
+	now := time.Unix(0, 0)
+	s := &Service{cfg: DefaultConfig(), attempts: map[string]*attemptWindow{}, now: func() time.Time { return now }}
+
+	// The target is being guessed at.
+	for i := 0; i < s.cfg.MaxAttempts; i++ {
+		s.allowAttempt("victim@example.com")
+	}
+	// Then the table is flooded with made-up addresses.
+	for i := 0; i < maxTrackedAddresses+100; i++ {
+		s.allowAttempt(fmt.Sprintf("x%d@example.com", i))
+	}
+	if n := len(s.attempts); n > maxTrackedAddresses {
+		t.Errorf("the table grew to %d", n)
+	}
+	if s.allowAttempt("victim@example.com") {
+		t.Error("flooding the table reset the target's throttle")
+	}
+
+	// Once the window passes, everything expires and sign-ins work again.
+	now = now.Add(s.cfg.AttemptWindow + time.Second)
+	if !s.allowAttempt("new@example.com") {
+		t.Error("a new address was refused after the flood expired")
+	}
+}

internal/accounting/accounting.go

@@ -383,3 +383,39 @@ func keyCounter(keyID string) string {
 // they are asking is whether the budgets they are about to display mean
 // anything, and the accountant is what would enforce them.
 func (a *Accountant) AnythingPriced() bool { return a.pricing.AnythingPriced() }
+
+// Headroom is the tightest of a principal's request and token limits: how
+// much is left of each, and when the minute it is counted over rolls on.
+type Headroom struct {
+	RequestLimit, RequestsLeft int
+	TokenLimit, TokensLeft     int
+	Reset                      time.Duration
+}
+
+// Headroom reports what a principal has left, after whatever Check just
+// charged. Of several levels the one with least left is the one that will
+// refuse first, so that is the one reported. A limit of 0 means none is set.
+func (a *Accountant) Headroom(p auth.Principal) Headroom {
+	var h Headroom
+	firstReq, firstTok := true, true
+	for _, l := range levels(p.Limits, p.KeyID) {
+		if l.RPM != nil && *l.RPM > 0 {
+			left := max(0, *l.RPM-int(a.requests.Rate(l.Counter)))
+			if firstReq || left < h.RequestsLeft {
+				h.RequestLimit, h.RequestsLeft, firstReq = *l.RPM, left, false
+			}
+		}
+		if l.TPM != nil && *l.TPM > 0 {
+			left := max(0, *l.TPM-int(a.tokens.Rate(l.Counter)))
+			if firstTok || left < h.TokensLeft {
+				h.TokenLimit, h.TokensLeft, firstTok = *l.TPM, left, false
+			}
+		}
+	}
+	// The windows slide, so room comes back continuously; the next minute
+	// boundary is when a whole window's worth has moved on, and is the figure
+	// a client pacing itself can act on.
+	now := time.Now()
+	h.Reset = now.Truncate(windowLen).Add(windowLen).Sub(now)
+	return h
+}

internal/accounting/accounting_test.go

@@ -630,3 +630,31 @@ func TestAnEmptyPriceTableMeansNoBudgetCanEverBind(t *testing.T) {
 		t.Error("no price table at all is not priced")
 	}
 }
+
+func TestHeadroomIsTheTightestLimitAfterThisRequest(t *testing.T) {
+	// A project allowing 100 a minute under a team allowing 3: the team is
+	// what will refuse, so the team is what a client is told about.
+	a := New(pricing(t), nil, slog.New(slog.DiscardHandler))
+	p := principal("k1", func(e *store.Effective) {
+		e.Chain = []store.Limits{
+			{Scope: store.ScopeProject, ScopeID: "proj", RPM: intp(100), TPM: intp(50000)},
+			{Scope: store.ScopeTeam, ScopeID: "team", RPM: intp(3)},
+		}
+	})
+	if _, err := a.Check(p, "m", 10); err != nil {
+		t.Fatal(err)
+	}
+	h := a.Headroom(p)
+	if h.RequestLimit != 3 || h.RequestsLeft != 2 {
+		t.Errorf("requests %d left of %d; expected 2 of the team's 3", h.RequestsLeft, h.RequestLimit)
+	}
+	if h.TokenLimit != 50000 || h.TokensLeft != 50000-10 {
+		t.Errorf("tokens %d left of %d", h.TokensLeft, h.TokenLimit)
+	}
+	if h.Reset <= 0 || h.Reset > time.Minute {
+		t.Errorf("reset %v", h.Reset)
+	}
+	if none := a.Headroom(principal("k2", nil)); none.RequestLimit != 0 || none.TokenLimit != 0 {
+		t.Errorf("a key with no limits advertised some: %+v", none)
+	}
+}

internal/agent/agent.go

@@ -14,13 +14,9 @@ import (
 	"math/rand/v2"
 	"net"
 	"net/http"
-	"net/url"
 	"os"
-	"os/exec"
 	"path/filepath"
 	"slices"
-	"strconv"
-	"strings"
 	"sync"
 	"time"
 
@@ -34,278 +30,9 @@ import (
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/supervisor"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/weights"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/agentsvc"
-	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
 )
 
-// Config configures the agent.
-type Config struct {
-	// NodeID defaults to the hostname. It must equal the common name of the
-	// agent's certificate, so set it only where the two differ.
-	NodeID string `koanf:"node_id"`
-	// Server is the control plane, and Servers is the same thing when there is
-	// more than one. Both are accepted: every configuration written before
-	// there could be a second control plane names one address, and none of
-	// them should have to change.
-	//
-	// The order is tried in order, and a refusal moves to the next. Only the
-	// leader accepts an agent, so a standby is skipped in the time it takes to
-	// be told so — which is what makes failover a reconnect rather than an
-	// edit to every node's configuration.
-	Server  string   `koanf:"server"`
-	Servers []string `koanf:"servers"`
-	// Hostname defaults to the machine's.
-	Hostname string `koanf:"hostname"`
-	// MgmtAddr is the address engines are reached on. It defaults to this
-	// machine's address on the route to the control plane, which is the one
-	// the control plane and the router can reach.
-	MgmtAddr string `koanf:"mgmt_addr"`
-	// Labels are free-form facts about a node for a profile's node_selector.
-	// Nothing needs them: memory, GPUs, engines and disk are measured, and a
-	// label restating one of those goes stale the day the machine changes.
-	Labels map[string]string `koanf:"labels"`
-
-	// MgmtInterface is the ethernet interface NCCL bootstraps over. It
-	// defaults to the interface holding MgmtAddr, then eth0.
-	MgmtInterface string `koanf:"mgmt_interface"`
-	// Fabrics labels this node's HCAs. It is explicit on purpose: OpenSM's
-	// default subnet prefix is the same on every fabric, so nothing can be
-	// inferred from the wire (plan §5.2).
-	Fabrics []inventory.FabricAttachment `koanf:"fabrics"`
-
-	HeartbeatInterval time.Duration `koanf:"heartbeat_interval"`
-	// ReconnectBase is the first reconnect delay; it backs off from there.
-	ReconnectBase time.Duration `koanf:"reconnect_base"`
-	ReconnectMax  time.Duration `koanf:"reconnect_max"`
-
-	// AdoptReadyTimeout bounds how long an adopted engine is given to answer a
-	// health check before the agent stops waiting on it.
-	//
-	// Shorter than a startup timeout on purpose: this engine is already
-	// running, so it is not loading a model, it is either serving or it is not.
-	AdoptReadyTimeout time.Duration `koanf:"adopt_ready_timeout"`
-
-	// TLS authenticates both ends of the control-plane stream. Leaving it unset
-	// keeps the plaintext behaviour, which only belongs on a trusted network.
-	TLS tlsconf.Config `koanf:"tls"`
-
-	// ModelDirs are the directories this node keeps model weights in, beyond
-	// WeightsCache, which is always looked in. Storage differs at every installation,
-	// so llmgw observes it rather than owning it: a Lustre mount, an image
-	// exported over the fabric, local NVMe and a copy made by hand all look
-	// identical from here, and placement only needs to know whether the file
-	// is present.
-	ModelDirs []string `koanf:"model_dirs"`
-
-	// EnginesDir is where the agent installs engines, one directory per
-	// build. Defaults to /opt/llmgw/engines.
-	EnginesDir string `koanf:"engines_dir"`
-
-	// WeightsCache is where a remote source is downloaded to. Defaults to the
-	// first entry of ModelDirs, because a node that says where its models live
-	// has already answered this question, and otherwise to
-	// /var/lib/llmgw-weights.
-	//
-	// Fetching is the simple arrangement: every node keeps its own copy. The
-	// exotic ones need no code — weights on a shared filer or an image exported
-	// over the fabric are simply present, and present means nothing is fetched.
-	WeightsCache string `koanf:"weights_cache"`
-
-	Discovery discovery.Options `koanf:"discovery"`
-	Engines   engines.Config    `koanf:"engines"`
-	Superv    supervisor.Config `koanf:"supervisor"`
-
-	Version string
-}
-
-// DefaultConfig returns sensible defaults.
-func DefaultConfig() Config {
-	// No MgmtInterface: normalize finds the one holding the management
-	// address, and falls back to eth0 only when it cannot. Defaulted here, it
-	// was never found — every node said eth0 unless told otherwise, and a node
-	// on ens3 or bond0 had NCCL bootstrap over an interface it does not have.
-	return Config{
-		HeartbeatInterval: 5 * time.Second,
-		ReconnectBase:     time.Second,
-		ReconnectMax:      30 * time.Second,
-		AdoptReadyTimeout: 60 * time.Second,
-		Superv:            supervisor.DefaultConfig(),
-		// NVML gives NVLink topology, so prefer it when this build has it.
-		Discovery: discovery.Options{PreferNVML: true},
-	}
-}
-
-func (c *Config) normalize() error {
-	d := DefaultConfig()
-	if err := c.TLS.Validate(); err != nil {
-		return err
-	}
-	if err := c.normalizeServerURL(); err != nil {
-		return err
-	}
-	// Everything below is found rather than written. A file that has to say
-	// which machine it is on, and at what address, is a file that differs on
-	// every node — and a copy taken from one node to the next registers the
-	// second as the first.
-	host, _ := os.Hostname()
-	if c.NodeID == "" {
-		c.NodeID = host
-	}
-	if c.NodeID == "" {
-		return errors.New("node_id is not set and the hostname could not be read")
-	}
-	if c.Hostname == "" {
-		c.Hostname = host
-	}
-	if c.MgmtAddr == "" {
-		c.MgmtAddr = addrToward(c.Server)
-	}
-	if c.MgmtInterface == "" {
-		c.MgmtInterface = interfaceOf(c.MgmtAddr)
-	}
-	if c.MgmtInterface == "" {
-		c.MgmtInterface = "eth0"
-	}
-	// The directory downloads land in is always one that is looked in. When
-	// they were two settings, a node could fill its disk with fetched weights
-	// that placement never saw — which is what every node here was doing.
-	if c.WeightsCache == "" && len(c.ModelDirs) > 0 {
-		c.WeightsCache = c.ModelDirs[0]
-	}
-	if c.WeightsCache == "" {
-		c.WeightsCache = DefaultWeightsCache
-	}
-	if !slices.Contains(c.ModelDirs, c.WeightsCache) {
-		c.ModelDirs = append(c.ModelDirs, c.WeightsCache)
-	}
-	if c.HeartbeatInterval <= 0 {
-		c.HeartbeatInterval = d.HeartbeatInterval
-	}
-	if c.ReconnectBase <= 0 {
-		c.ReconnectBase = d.ReconnectBase
-	}
-	if c.AdoptReadyTimeout <= 0 {
-		c.AdoptReadyTimeout = d.AdoptReadyTimeout
-	}
-	if c.ReconnectMax < c.ReconnectBase {
-		c.ReconnectMax = max(c.ReconnectBase, d.ReconnectMax)
-	}
-	return nil
-}
-
-// normalizeServerURL makes the configured address a URL connect can use, and
-// refuses the combination that would quietly undo the certificates: a scheme
-// that says plaintext while TLS is configured. Silently connecting anyway would
-// mean a cluster believing it has mutual authentication when it has none.
-func (c *Config) normalizeServerURL() error {
-	// One list, however it was written. Server first when both are given,
-	// because an operator adding a standby to an existing file expects the
-	// one already there to stay preferred.
-	all := make([]string, 0, 1+len(c.Servers))
-	if c.Server != "" {
-		all = append(all, c.Server)
-	}
-	all = append(all, c.Servers...)
-
-	seen := map[string]struct{}{}
-	out := make([]string, 0, len(all))
-	for _, s := range all {
-		fixed, err := normalizeOne(s, c.TLS.Enabled())
-		if err != nil {
-			return err
-		}
-		// A duplicate would be tried twice in a row on failover, which reads
-		// as the rotation not working.
-		if _, dup := seen[fixed]; dup {
-			continue
-		}
-		seen[fixed] = struct{}{}
-		out = append(out, fixed)
-	}
-	if len(out) == 0 {
-		return errors.New("server is required")
-	}
-	c.Servers = out
-	c.Server = out[0]
-	return nil
-}
-
-// DefaultWeightsCache is where fetched weights go when nothing says otherwise.
-const DefaultWeightsCache = "/var/lib/llmgw-weights"
-
-// addrToward is this machine's address on the route to the control plane.
-//
-// The address engines are reached on is whichever one the control plane can
-// reach, and the kernel already knows which that is: a UDP "connection" picks
-// the source address for the route without sending a packet. Asking the
-// interfaces instead would have to guess among docker bridges, VPNs and
-// link-local addresses.
-func addrToward(server string) string {
-	u, err := url.Parse(server)
-	if err != nil || u.Host == "" {
-		return ""
-	}
-	host := u.Hostname()
-	port := u.Port()
-	if port == "" {
-		port = "443"
-	}
-	conn, err := net.Dial("udp", net.JoinHostPort(host, port))
-	if err != nil {
-		return ""
-	}
-	defer conn.Close()
-	if a, ok := conn.LocalAddr().(*net.UDPAddr); ok {
-		return a.IP.String()
-	}
-	return ""
-}
-
-// interfaceOf names the interface that holds an address, or "" when none does.
-func interfaceOf(addr string) string {
-	ip := net.ParseIP(addr)
-	if ip == nil {
-		return ""
-	}
-	ifaces, err := net.Interfaces()
-	if err != nil {
-		return ""
-	}
-	for _, ifc := range ifaces {
-		addrs, err := ifc.Addrs()
-		if err != nil {
-			continue
-		}
-		for _, a := range addrs {
-			if n, ok := a.(*net.IPNet); ok && n.IP.Equal(ip) {
-				return ifc.Name
-			}
-		}
-	}
-	return ""
-}
-
-// normalizeOne applies the scheme rules to a single address.
-func normalizeOne(addr string, secure bool) (string, error) {
-	switch {
-	case strings.HasPrefix(addr, "https://"):
-		if !secure {
-			return "", fmt.Errorf("%s is an https:// address but no tls certificates are configured", addr)
-		}
-	case strings.HasPrefix(addr, "http://"):
-		if secure {
-			return "", fmt.Errorf("tls is configured but %s is an http:// address; use https://", addr)
-		}
-	case strings.Contains(addr, "://"):
-		return "", fmt.Errorf("server %q must be an http:// or https:// address", addr)
-	case secure:
-		return "https://" + addr, nil
-	default:
-		return "http://" + addr, nil
-	}
-	return addr, nil
-}
-
 // Agent runs on a GPU node.
 type Agent struct {
 	cfg  Config
@@ -586,131 +313,6 @@ func (a *Agent) settleAdopted(sh *supervisor.Shard, o supervisor.Orphan) {
 		"shard", o.ShardID, "replica", o.ReplicaID)
 }
 
-// usableFabrics reports the fabrics this node can actually carry traffic on.
-//
-// Loud about what it drops. A fabric silently missing from a registration looks
-// identical to one never configured, and the operator who typed the adapter
-// name is the only person who can fix it — so the reason is logged here, on the
-// machine where both halves are visible, rather than inferred later from a
-// replica that will not converge.
-func (a *Agent) usableFabrics(inv discovery.Inventory) []inventory.FabricAttachment {
-	out := a.checkFabrics(inv.HCAs, true)
-	a.mu.Lock()
-	a.fabricsSent = fabricsFingerprint(out)
-	a.mu.Unlock()
-	return out
-}
-
-// checkFabrics is the declared fabrics this node can carry traffic on, from
-// the adapters as they are now. Loud logs every problem; the heartbeat checks
-// quietly and speaks only when the answer changes.
-func (a *Agent) checkFabrics(hcas []discovery.HCA, loud bool) []inventory.FabricAttachment {
-	declared := map[string][]string{}
-	for _, f := range a.cfg.Fabrics {
-		declared[f.Fabric] = f.HCAs
-	}
-
-	var out []inventory.FabricAttachment
-	for _, c := range discovery.CheckFabrics(declared, hcas) {
-		if loud {
-			for _, p := range c.Problems {
-				a.log.Error("a declared fabric adapter cannot carry traffic",
-					"fabric", c.Fabric, "problem", p)
-			}
-		}
-		if !c.OK() {
-			if loud {
-				a.log.Error("fabric not advertised; no declared adapter on it can carry traffic",
-					"fabric", c.Fabric,
-					"consequence", "no multi-node replica will be placed here")
-			}
-			continue
-		}
-		out = append(out, inventory.FabricAttachment{
-			Fabric: c.Fabric, HCAs: c.Usable, RateGbps: c.RateGbps,
-		})
-	}
-	return out
-}
-
-// fabricsFingerprint is enough to notice a fabric, an adapter or a rate
-// changing.
-func fabricsFingerprint(fs []inventory.FabricAttachment) string {
-	var b strings.Builder
-	for _, f := range fs {
-		fmt.Fprintf(&b, "%s:%s:%d;", f.Fabric, strings.Join(f.HCAs, ","), f.RateGbps)
-	}
-	return b.String()
-}
-
-// sampleIB re-reads the adapters for the heartbeat: every port's state and
-// counters, and the fabrics again when they no longer match what the control
-// plane was told.
-//
-// Every heartbeat, because it is a few small files per port, and because a
-// port that went down used to be invisible until the agent restarted — the
-// node went on being offered multi-node work that would hang on its first
-// collective.
-func (a *Agent) sampleIB(hb *llmgwv1.Heartbeat) {
-	a.mu.Lock()
-	had := len(a.inv.HCAs) > 0
-	a.mu.Unlock()
-	if !had && len(a.cfg.Fabrics) == 0 {
-		return
-	}
-	hcas, err := discovery.DiscoverIB(a.cfg.Discovery.IBRoot)
-	if err != nil {
-		return
-	}
-	a.attachSM(hcas)
-	for _, h := range hcas {
-		for _, p := range h.Ports {
-			ip := &llmgwv1.IBPort{
-				Hca: h.Name, Port: int32(p.Num), State: p.State, PhysState: p.PhysState,
-				RateGbps: int32(p.RateGbps), LinkLayer: string(p.LinkLayer), Counters: p.Counters,
-				NodeGuid: h.NodeGUID, PortGuid: p.PortGUID, Network: p.Network, GidPrefix: p.GIDPrefix,
-			}
-			a.mu.Lock()
-			if p.SM != nil {
-				ip.SmGuid, ip.SmLid, ip.SmPriority, ip.SmState = p.SM.GUID, int32(p.SM.LID), int32(p.SM.Priority), p.SM.State
-			} else {
-				ip.SmError = a.smErr[smKey(h.Name, p.Num)]
-			}
-			for _, r := range a.sms[smKey(h.Name, p.Num)] {
-				ip.SubnetManagers = append(ip.SubnetManagers, &llmgwv1.SubnetManager{
-					Guid: r.GUID, Lid: int32(r.LID), Priority: int32(r.Priority), State: r.State})
-			}
-			ip.SubnetManagersError = a.smsErr[smKey(h.Name, p.Num)]
-			a.mu.Unlock()
-			hb.IbPorts = append(hb.IbPorts, ip)
-		}
-	}
-	a.mu.Lock()
-	a.inv.HCAs = hcas // a reconnect registers what is true now
-	a.mu.Unlock()
-	if len(a.cfg.Fabrics) == 0 {
-		return
-	}
-	now := a.checkFabrics(hcas, false)
-	fp := fabricsFingerprint(now)
-	a.mu.Lock()
-	changed := fp != a.fabricsSent
-	a.fabricsSent = fp
-	a.mu.Unlock()
-	if !changed {
-		return
-	}
-	// Said once, with the reasons, the moment it happens.
-	a.checkFabrics(hcas, true)
-	a.log.Warn("the fabrics this node can carry traffic on changed", "now", fp)
-	hb.FabricsChanged = true
-	for _, f := range now {
-		hb.Fabrics = append(hb.Fabrics, &llmgwv1.FabricAttachment{
-			Fabric: f.Fabric, Hcas: f.HCAs, RateGbps: int32(f.RateGbps),
-		})
-	}
-}
-
 // weights builds the fetcher lazily, so an agent that never starts a shard
 // never needs a cache directory.
 func (a *Agent) weights() *weights.Fetcher {
@@ -1119,6 +721,8 @@ func (a *Agent) handle(ctx context.Context, msg *llmgwv1.ControlMessage, out cha
 		a.removeWeights(m.RemoveWeights.GetPaths())
 	case *llmgwv1.ControlMessage_Provision:
 		a.setProvision(m.Provision)
+	case *llmgwv1.ControlMessage_Upgrade:
+		go a.upgrade(ctx, m.Upgrade, out)
 	case *llmgwv1.ControlMessage_Logs:
 		reply := a.logsReply(m.Logs)
 		// Off the receive loop: a full send queue must not stop commands being
@@ -1410,215 +1014,6 @@ func Client(cfg tlsconf.Config, timeout time.Duration) (*http.Client, error) {
 	return &http.Client{Transport: &http2.Transport{TLSClientConfig: tc}}, nil
 }
 
-// engines is what this node can run: what the agent installed and saw run,
-// and what the configuration names and answers. Nothing merely found on the
-// machine — a llama-server on PATH is somebody's, not necessarily one that
-// works, and placing on it is how a replica fails on a node nobody watches.
-func (a *Agent) engines(ctx context.Context) map[string]string {
-	a.engineOnce.Do(func() {
-		vers := a.reg.ProbeConfigured(ctx)
-		a.mu.Lock()
-		a.configuredVers, a.lastEngineProbe = vers, time.Now()
-		a.mu.Unlock()
-		if len(vers) > 0 {
-			a.log.Info("engines named in the configuration",
-				"engines", engines.ProbedNames(vers))
-		}
-	})
-	a.mu.Lock()
-	defer a.mu.Unlock()
-	vers := a.engineReportLocked()
-	// Whatever registration sends is what the control plane now knows, so a
-	// heartbeat repeats it only once it differs.
-	a.enginesSent = enginesFingerprint(vers)
-	return vers
-}
-
-// engineReportLocked merges the two sources. Installed wins: it is the one
-// that was seen to run. Called with mu held.
-func (a *Agent) engineReportLocked() map[string]string {
-	out := maps.Clone(a.configuredVers)
-	if out == nil {
-		out = map[string]string{}
-	}
-	maps.Copy(out, a.installedVers)
-	return out
-}
-
-// provisionEvery is how often installed engines are checked again. A library
-// removed by an unrelated upgrade is found here rather than by the next
-// replica to start.
-const provisionEvery = 10 * time.Minute
-
-// setProvision takes the control plane's list of wanted engines.
-func (a *Agent) setProvision(p *llmgwv1.Provision) {
-	wants := []provision.Want{}
-	for _, e := range p.GetEngines() {
-		wants = append(wants, provision.Want{Engine: e.GetEngine(), Version: e.GetVersion()})
-	}
-	a.mu.Lock()
-	a.provWants = wants
-	a.mu.Unlock()
-	select {
-	case a.provKick <- struct{}{}:
-	default:
-	}
-}
-
-// provisionLoop installs and re-checks the wanted engines, one pass at a time.
-func (a *Agent) provisionLoop(ctx context.Context) {
-	// After a failure the next pass comes sooner, and backs off: a lock held
-	// by a new machine's own updates clears in a minute or two, while ten
-	// minutes' wait left a node idle for no reason.
-	next := provisionEvery
-	retry := time.Minute
-	for {
-		select {
-		case <-ctx.Done():
-			return
-		case <-a.provKick:
-		case <-time.After(next):
-		}
-		a.mu.Lock()
-		wants := slices.Clone(a.provWants)
-		managed := a.provWants != nil
-		a.mu.Unlock()
-		if !managed {
-			continue
-		}
-
-		statuses := a.prov.Converge(ctx, wants)
-		next = provisionEvery
-		for _, st := range statuses {
-			if st.State == provision.Failed {
-				next, retry = retry, min(retry*2, provisionEvery)
-				break
-			}
-		}
-		if next == provisionEvery {
-			retry = time.Minute
-		}
-		installed := map[string]string{}
-		keep := map[string]bool{}
-		for _, st := range statuses {
-			keep[st.Engine] = true
-			if st.State == provision.Ready {
-				a.reg.UseInstalled(st.Engine, st.Binary)
-				installed[st.Engine] = st.Version
-			}
-		}
-		a.mu.Lock()
-		if enginesFingerprint(installed) != enginesFingerprint(a.installedVers) {
-			a.log.Info("engines installed and running", "engines", engines.ProbedNames(installed))
-		}
-		a.installedVers = installed
-		// An engine no longer asked for is no longer reported on.
-		for e := range a.provStatus {
-			if !keep[e] {
-				delete(a.provStatus, e)
-			}
-		}
-		a.mu.Unlock()
-	}
-}
-
-// provisionStatus records a change in one engine's installation.
-func (a *Agent) provisionStatus(st provision.Status) {
-	a.mu.Lock()
-	prev, had := a.provStatus[st.Engine]
-	a.provStatus[st.Engine] = st
-	a.mu.Unlock()
-	if !had || prev.State != st.State || prev.Detail != st.Detail {
-		lvl := slog.LevelInfo
-		if st.State == provision.Failed || st.State == provision.Unsupported {
-			lvl = slog.LevelWarn
-		}
-		a.log.Log(context.Background(), lvl, "engine installation",
-			"engine", st.Engine, "version", st.Version, "variant", st.Variant,
-			"state", st.State, "detail", st.Detail)
-	}
-}
-
-func provisionFingerprint(m map[string]provision.Status) string {
-	var b strings.Builder
-	for _, k := range slices.Sorted(maps.Keys(m)) {
-		st := m[k]
-		fmt.Fprintf(&b, "%s|%s|%s|%s|%s\n", st.Engine, st.Version, st.Variant, st.State, st.Detail)
-	}
-	return b.String()
-}
-
-// engineReprobe is how often the installed engines are looked for again.
-//
-// Minutes, not seconds: an engine is installed by somebody at a keyboard, so a
-// minute's delay in noticing is invisible, while SGLang's probe starts a
-// Python interpreter that imports it.
-const engineReprobe = time.Minute
-
-// reprobeEngines looks for installed engines again when it is time, off the
-// heartbeat goroutine so a slow probe never delays a heartbeat.
-func (a *Agent) reprobeEngines() {
-	a.mu.Lock()
-	due := !a.probingEngines && !a.lastEngineProbe.IsZero() &&
-		time.Since(a.lastEngineProbe) >= engineReprobe
-	if due {
-		a.probingEngines = true
-	}
-	a.mu.Unlock()
-	if !due {
-		return
-	}
-	go func() {
-		ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
-		defer cancel()
-		vers := a.reg.ProbeConfigured(ctx)
-
-		a.mu.Lock()
-		changed := enginesFingerprint(vers) != enginesFingerprint(a.configuredVers)
-		a.configuredVers, a.lastEngineProbe, a.probingEngines = vers, time.Now(), false
-		a.mu.Unlock()
-		if changed {
-			a.log.Info("engines named in the configuration changed",
-				"engines", engines.ProbedNames(vers))
-		}
-	}()
-}
-
-func enginesFingerprint(vers map[string]string) string {
-	var b strings.Builder
-	for _, k := range slices.Sorted(maps.Keys(vers)) {
-		fmt.Fprintf(&b, "%s=%s\n", k, vers[k])
-	}
-	return b.String()
-}
-
-// logsReply answers a request for one of this node's logs.
-func (a *Agent) logsReply(req *llmgwv1.LogsRequest) *llmgwv1.LogsReply {
-	n := int(req.GetLines())
-	if n <= 0 {
-		n = 200
-	}
-	n = min(n, agentLogLines)
-	shards := a.sup.IDs()
-	slices.Sort(shards)
-	out := &llmgwv1.LogsReply{
-		RequestId: req.GetRequestId(),
-		Sources:   append([]string{"agent"}, shards...),
-	}
-	switch src := req.GetSource(); src {
-	case "", "agent":
-		out.Lines = a.logRing.tail(n)
-	default:
-		sh, err := a.sup.Get(src)
-		if err != nil {
-			out.Error = "no engine " + src + " runs on this node"
-			return out
-		}
-		out.Lines = sh.Logs(n)
-	}
-	return out
-}
-
 // localizeSpec fills in what only this node knows.
 //
 // The interface NCCL bootstraps over is this node's, not the cluster's: the
@@ -1669,208 +1064,3 @@ func ResolveManagement(c Config) (addr, iface string) {
 	}
 	return addr, iface
 }
-
-func smKey(hca string, port int) string { return fmt.Sprintf("%s/%d", hca, port) }
-
-// scanFabric is discovery.ScanFabric, replaced in tests.
-var scanFabric = discovery.ScanFabric
-
-// querySMs is discovery.QuerySMs, replaced in tests.
-var querySMs = func(hca string, port int) ([]discovery.SMRecord, error) {
-	return discovery.QuerySMs(context.Background(), hca, port)
-}
-
-func splitSMKey(k string) (string, int) {
-	i := strings.LastIndexByte(k, '/')
-	port, _ := strconv.Atoi(k[i+1:])
-	return k[:i], port
-}
-
-// querySM is discovery.QuerySM, replaced in tests.
-var querySM = func(hca string, port int) (*discovery.SMInfo, error) {
-	return discovery.QuerySM(context.Background(), hca, port)
-}
-
-// attachSM puts on each active InfiniBand port the subnet manager it sees,
-// asking again once a minute and otherwise reusing the last answer.
-func (a *Agent) attachSM(hcas []discovery.HCA) {
-	a.mu.Lock()
-	stale := time.Since(a.smAsked) > time.Minute || a.sm == nil
-	a.mu.Unlock()
-	if stale {
-		sm, errs := map[string]*discovery.SMInfo{}, map[string]string{}
-		_ = discovery.AttachSM(hcas, func(hca string, port int) (*discovery.SMInfo, error) {
-			got, err := querySM(hca, port)
-			if err != nil {
-				errs[smKey(hca, port)] = err.Error()
-				return nil, err
-			}
-			sm[smKey(hca, port)] = got
-			return got, nil
-		})
-		// Every subnet manager on each fabric, through one port per master:
-		// the same list through every port of one fabric is one query too
-		// many for each.
-		sms, smsErr := map[string][]discovery.SMRecord{}, map[string]string{}
-		asked := map[string]bool{}
-		for k, v := range sm {
-			if asked[v.GUID] {
-				continue
-			}
-			asked[v.GUID] = true
-			hca, port := splitSMKey(k)
-			if recs, err := querySMs(hca, port); err != nil {
-				smsErr[k] = err.Error()
-			} else {
-				sms[k] = recs
-			}
-		}
-		a.mu.Lock()
-		for k, v := range sm {
-			if old := a.sm[k]; old != nil && old.GUID != v.GUID {
-				a.log.Warn("the master subnet manager changed", "port", k, "from", old.GUID, "to", v.GUID)
-			}
-		}
-		a.sm, a.smErr, a.smAsked = sm, errs, time.Now()
-		a.sms, a.smsErr = sms, smsErr
-		a.mu.Unlock()
-		return
-	}
-	a.mu.Lock()
-	defer a.mu.Unlock()
-	for i := range hcas {
-		for j := range hcas[i].Ports {
-			if sm := a.sm[smKey(hcas[i].Name, hcas[i].Ports[j].Num)]; sm != nil && hcas[i].Ports[j].Active() {
-				hcas[i].Ports[j].SM = sm
-			}
-		}
-	}
-}
-
-// ensureDiags installs infiniband-diags on a node with adapters and without
-// it. sminfo is what tells which physical fabric a port is on; without it,
-// two separate fabrics given one name cannot be caught. Installed the way
-// engine dependencies are, from the machine's own package sources.
-func (a *Agent) ensureDiags(ctx context.Context) {
-	a.mu.Lock()
-	hasIB := len(a.inv.HCAs) > 0
-	a.mu.Unlock()
-	if !hasIB || a.prov == nil {
-		return
-	}
-	if _, err := exec.LookPath("sminfo"); err == nil {
-		return
-	}
-	a.log.Info("installing infiniband-diags, to tell which fabric each port is on")
-	pkgs := map[string][]string{"deb": {"infiniband-diags"}, "rpm": {"infiniband-diags"}}
-	if err := a.prov.InstallPackages(ctx, pkgs); err != nil {
-		a.log.Warn("could not install infiniband-diags; fabrics cannot be told apart by their subnet manager",
-			"err", err)
-	}
-}
-
-// FeatureFabricTest is the agent answering FabricTestRequest.
-const FeatureFabricTest = "fabric-test"
-
-// runWriteBW is discovery.RunWriteBW, and haveWriteBW whether it is
-// installed; both replaced in tests.
-var (
-	runWriteBW  = discovery.RunWriteBW
-	haveWriteBW = func() bool { _, err := exec.LookPath("ib_write_bw"); return err == nil }
-)
-
-// fabricTest runs this node's side of a bandwidth test with another.
-func (a *Agent) fabricTest(ctx context.Context, req *llmgwv1.FabricTestRequest) *llmgwv1.FabricTestReply {
-	start := time.Now()
-	reply := &llmgwv1.FabricTestReply{RequestId: req.GetRequestId()}
-	defer func() { reply.TookMs = time.Since(start).Milliseconds() }()
-
-	w := discovery.WriteBW{HCA: req.GetHca(), Port: int(req.GetPort()), GIDIndex: -1,
-		TCPPort: int(req.GetTcpPort()), Seconds: int(req.GetSeconds())}
-	if req.GetRole() == "client" {
-		w.Peer = req.GetPeer()
-	}
-	a.mu.Lock()
-	found := false
-	for _, h := range a.inv.HCAs {
-		for _, p := range h.Ports {
-			if h.Name == w.HCA && p.Num == w.Port {
-				found = true
-				// RoCE needs its v2 GID named, as NCCL does; InfiniBand uses the LID.
-				if p.LinkLayer == discovery.LinkEthernet && p.RoCEv2GID >= 0 {
-					w.GIDIndex = p.RoCEv2GID
-				}
-			}
-		}
-	}
-	a.mu.Unlock()
-	if !found {
-		reply.Error = fmt.Sprintf("this node has no port %s/%d", w.HCA, w.Port)
-		return reply
-	}
-	if !haveWriteBW() && a.prov != nil {
-		a.log.Info("installing perftest, for a bandwidth test with another node")
-		pkgs := map[string][]string{"deb": {"perftest"}, "rpm": {"perftest"}}
-		if err := a.prov.InstallPackages(ctx, pkgs); err != nil {
-			reply.Error = "installing perftest: " + err.Error()
-			return reply
-		}
-	}
-	gbps, out, err := runWriteBW(ctx, w)
-	reply.Gbps = gbps
-	reply.Output = tailLines(out, 12)
-	if err != nil {
-		reply.Error = err.Error()
-	}
-	return reply
-}
-
-func tailLines(s string, n int) string {
-	lines := strings.Split(strings.TrimRight(s, "\n"), "\n")
-	if len(lines) > n {
-		lines = lines[len(lines)-n:]
-	}
-	return strings.Join(lines, "\n")
-}
-
-// discoverGPUFabric and discoverIMEX are replaced in tests.
-var (
-	discoverGPUFabric = discovery.DiscoverGPUFabric
-	discoverIMEX      = discovery.DiscoverIMEX
-)
-
-// sampleNVLink reads, once a minute, which NVLink domain each GPU is joined
-// to and whether IMEX is up. nvidia-smi -q takes a second or two, too long
-// for every heartbeat, and a fabric that was joined an hour ago rarely
-// changes in five seconds — but when a fabric manager restarts it does, for
-// every card at once.
-func (a *Agent) sampleNVLink(hb *llmgwv1.Heartbeat) {
-	a.mu.Lock()
-	gpus := slices.Clone(a.inv.GPUs)
-	due := time.Since(a.nvlAsked) > time.Minute
-	if due {
-		a.nvlAsked = time.Now()
-	}
-	a.mu.Unlock()
-	if len(gpus) == 0 || !due {
-		return
-	}
-	ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
-	defer cancel()
-	fab, err := discoverGPUFabric(ctx, a.cfg.Discovery.NvidiaSMI)
-	if err != nil || len(fab) == 0 {
-		return
-	}
-	discovery.ApplyFabric(gpus, fab)
-	for _, g := range gpus {
-		if g.Fabric == nil {
-			continue
-		}
-		hb.GpuFabric = append(hb.GpuFabric, &llmgwv1.GPUFabricState{
-			Index: int32(g.Index), State: g.Fabric.State, Status: g.Fabric.Status,
-			ClusterUuid: g.Fabric.ClusterUUID, CliqueId: g.Fabric.CliqueID, Bandwidth: g.Fabric.Bandwidth,
-		})
-	}
-	m := discoverIMEX(ctx)
-	hb.Imex = &llmgwv1.IMEX{State: m.State, Channels: int32(m.Channels)}
-}

internal/agent/config.go

@@ -0,0 +1,286 @@
+package agent
+
+import (
+	"errors"
+	"fmt"
+	"net"
+	"net/url"
+	"os"
+	"slices"
+	"strings"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/supervisor"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
+)
+
+// Config configures the agent.
+type Config struct {
+	// NodeID defaults to the hostname. It must equal the common name of the
+	// agent's certificate, so set it only where the two differ.
+	NodeID string `koanf:"node_id"`
+	// Server is the control plane, and Servers is the same thing when there is
+	// more than one. Both are accepted: every configuration written before
+	// there could be a second control plane names one address, and none of
+	// them should have to change.
+	//
+	// The order is tried in order, and a refusal moves to the next. Only the
+	// leader accepts an agent, so a standby is skipped in the time it takes to
+	// be told so — which is what makes failover a reconnect rather than an
+	// edit to every node's configuration.
+	Server  string   `koanf:"server"`
+	Servers []string `koanf:"servers"`
+	// Hostname defaults to the machine's.
+	Hostname string `koanf:"hostname"`
+	// MgmtAddr is the address engines are reached on. It defaults to this
+	// machine's address on the route to the control plane, which is the one
+	// the control plane and the router can reach.
+	MgmtAddr string `koanf:"mgmt_addr"`
+	// Labels are free-form facts about a node for a profile's node_selector.
+	// Nothing needs them: memory, GPUs, engines and disk are measured, and a
+	// label restating one of those goes stale the day the machine changes.
+	Labels map[string]string `koanf:"labels"`
+
+	// MgmtInterface is the ethernet interface NCCL bootstraps over. It
+	// defaults to the interface holding MgmtAddr, then eth0.
+	MgmtInterface string `koanf:"mgmt_interface"`
+	// Fabrics labels this node's HCAs. It is explicit on purpose: OpenSM's
+	// default subnet prefix is the same on every fabric, so nothing can be
+	// inferred from the wire (plan §5.2).
+	Fabrics []inventory.FabricAttachment `koanf:"fabrics"`
+
+	HeartbeatInterval time.Duration `koanf:"heartbeat_interval"`
+	// ReconnectBase is the first reconnect delay; it backs off from there.
+	ReconnectBase time.Duration `koanf:"reconnect_base"`
+	ReconnectMax  time.Duration `koanf:"reconnect_max"`
+
+	// AdoptReadyTimeout bounds how long an adopted engine is given to answer a
+	// health check before the agent stops waiting on it.
+	//
+	// Shorter than a startup timeout on purpose: this engine is already
+	// running, so it is not loading a model, it is either serving or it is not.
+	AdoptReadyTimeout time.Duration `koanf:"adopt_ready_timeout"`
+
+	// TLS authenticates both ends of the control-plane stream. Leaving it unset
+	// keeps the plaintext behaviour, which only belongs on a trusted network.
+	TLS tlsconf.Config `koanf:"tls"`
+
+	// ModelDirs are the directories this node keeps model weights in, beyond
+	// WeightsCache, which is always looked in. Storage differs at every installation,
+	// so llmgw observes it rather than owning it: a Lustre mount, an image
+	// exported over the fabric, local NVMe and a copy made by hand all look
+	// identical from here, and placement only needs to know whether the file
+	// is present.
+	ModelDirs []string `koanf:"model_dirs"`
+
+	// EnginesDir is where the agent installs engines, one directory per
+	// build. Defaults to /opt/llmgw/engines.
+	EnginesDir string `koanf:"engines_dir"`
+
+	// WeightsCache is where a remote source is downloaded to. Defaults to the
+	// first entry of ModelDirs, because a node that says where its models live
+	// has already answered this question, and otherwise to
+	// /var/lib/llmgw-weights.
+	//
+	// Fetching is the simple arrangement: every node keeps its own copy. The
+	// exotic ones need no code — weights on a shared filer or an image exported
+	// over the fabric are simply present, and present means nothing is fetched.
+	WeightsCache string `koanf:"weights_cache"`
+
+	Discovery discovery.Options `koanf:"discovery"`
+	Engines   engines.Config    `koanf:"engines"`
+	Superv    supervisor.Config `koanf:"supervisor"`
+
+	Version string
+}
+
+// DefaultConfig returns sensible defaults.
+func DefaultConfig() Config {
+	// No MgmtInterface: normalize finds the one holding the management
+	// address, and falls back to eth0 only when it cannot. Defaulted here, it
+	// was never found — every node said eth0 unless told otherwise, and a node
+	// on ens3 or bond0 had NCCL bootstrap over an interface it does not have.
+	return Config{
+		HeartbeatInterval: 5 * time.Second,
+		ReconnectBase:     time.Second,
+		ReconnectMax:      30 * time.Second,
+		AdoptReadyTimeout: 60 * time.Second,
+		Superv:            supervisor.DefaultConfig(),
+		// NVML gives NVLink topology, so prefer it when this build has it.
+		Discovery: discovery.Options{PreferNVML: true},
+	}
+}
+
+func (c *Config) normalize() error {
+	d := DefaultConfig()
+	if err := c.TLS.Validate(); err != nil {
+		return err
+	}
+	if err := c.normalizeServerURL(); err != nil {
+		return err
+	}
+	// Everything below is found rather than written. A file that has to say
+	// which machine it is on, and at what address, is a file that differs on
+	// every node — and a copy taken from one node to the next registers the
+	// second as the first.
+	host, _ := os.Hostname()
+	if c.NodeID == "" {
+		c.NodeID = host
+	}
+	if c.NodeID == "" {
+		return errors.New("node_id is not set and the hostname could not be read")
+	}
+	if c.Hostname == "" {
+		c.Hostname = host
+	}
+	if c.MgmtAddr == "" {
+		c.MgmtAddr = addrToward(c.Server)
+	}
+	if c.MgmtInterface == "" {
+		c.MgmtInterface = interfaceOf(c.MgmtAddr)
+	}
+	if c.MgmtInterface == "" {
+		c.MgmtInterface = "eth0"
+	}
+	// The directory downloads land in is always one that is looked in. When
+	// they were two settings, a node could fill its disk with fetched weights
+	// that placement never saw — which is what every node here was doing.
+	if c.WeightsCache == "" && len(c.ModelDirs) > 0 {
+		c.WeightsCache = c.ModelDirs[0]
+	}
+	if c.WeightsCache == "" {
+		c.WeightsCache = DefaultWeightsCache
+	}
+	if !slices.Contains(c.ModelDirs, c.WeightsCache) {
+		c.ModelDirs = append(c.ModelDirs, c.WeightsCache)
+	}
+	if c.HeartbeatInterval <= 0 {
+		c.HeartbeatInterval = d.HeartbeatInterval
+	}
+	if c.ReconnectBase <= 0 {
+		c.ReconnectBase = d.ReconnectBase
+	}
+	if c.AdoptReadyTimeout <= 0 {
+		c.AdoptReadyTimeout = d.AdoptReadyTimeout
+	}
+	if c.ReconnectMax < c.ReconnectBase {
+		c.ReconnectMax = max(c.ReconnectBase, d.ReconnectMax)
+	}
+	return nil
+}
+
+// normalizeServerURL makes the configured address a URL connect can use, and
+// refuses the combination that would quietly undo the certificates: a scheme
+// that says plaintext while TLS is configured. Silently connecting anyway would
+// mean a cluster believing it has mutual authentication when it has none.
+func (c *Config) normalizeServerURL() error {
+	// One list, however it was written. Server first when both are given,
+	// because an operator adding a standby to an existing file expects the
+	// one already there to stay preferred.
+	all := make([]string, 0, 1+len(c.Servers))
+	if c.Server != "" {
+		all = append(all, c.Server)
+	}
+	all = append(all, c.Servers...)
+
+	seen := map[string]struct{}{}
+	out := make([]string, 0, len(all))
+	for _, s := range all {
+		fixed, err := normalizeOne(s, c.TLS.Enabled())
+		if err != nil {
+			return err
+		}
+		// A duplicate would be tried twice in a row on failover, which reads
+		// as the rotation not working.
+		if _, dup := seen[fixed]; dup {
+			continue
+		}
+		seen[fixed] = struct{}{}
+		out = append(out, fixed)
+	}
+	if len(out) == 0 {
+		return errors.New("server is required")
+	}
+	c.Servers = out
+	c.Server = out[0]
+	return nil
+}
+
+// DefaultWeightsCache is where fetched weights go when nothing says otherwise.
+const DefaultWeightsCache = "/var/lib/llmgw-weights"
+
+// addrToward is this machine's address on the route to the control plane.
+//
+// The address engines are reached on is whichever one the control plane can
+// reach, and the kernel already knows which that is: a UDP "connection" picks
+// the source address for the route without sending a packet. Asking the
+// interfaces instead would have to guess among docker bridges, VPNs and
+// link-local addresses.
+func addrToward(server string) string {
+	u, err := url.Parse(server)
+	if err != nil || u.Host == "" {
+		return ""
+	}
+	host := u.Hostname()
+	port := u.Port()
+	if port == "" {
+		port = "443"
+	}
+	conn, err := net.Dial("udp", net.JoinHostPort(host, port))
+	if err != nil {
+		return ""
+	}
+	defer conn.Close()
+	if a, ok := conn.LocalAddr().(*net.UDPAddr); ok {
+		return a.IP.String()
+	}
+	return ""
+}
+
+// interfaceOf names the interface that holds an address, or "" when none does.
+func interfaceOf(addr string) string {
+	ip := net.ParseIP(addr)
+	if ip == nil {
+		return ""
+	}
+	ifaces, err := net.Interfaces()
+	if err != nil {
+		return ""
+	}
+	for _, ifc := range ifaces {
+		addrs, err := ifc.Addrs()
+		if err != nil {
+			continue
+		}
+		for _, a := range addrs {
+			if n, ok := a.(*net.IPNet); ok && n.IP.Equal(ip) {
+				return ifc.Name
+			}
+		}
+	}
+	return ""
+}
+
+// normalizeOne applies the scheme rules to a single address.
+func normalizeOne(addr string, secure bool) (string, error) {
+	switch {
+	case strings.HasPrefix(addr, "https://"):
+		if !secure {
+			return "", fmt.Errorf("%s is an https:// address but no tls certificates are configured", addr)
+		}
+	case strings.HasPrefix(addr, "http://"):
+		if secure {
+			return "", fmt.Errorf("tls is configured but %s is an http:// address; use https://", addr)
+		}
+	case strings.Contains(addr, "://"):
+		return "", fmt.Errorf("server %q must be an http:// or https:// address", addr)
+	case secure:
+		return "https://" + addr, nil
+	default:
+		return "http://" + addr, nil
+	}
+	return addr, nil
+}

internal/agent/engines/llamacpp.go

@@ -4,6 +4,7 @@ import (
 	"fmt"
 	"os"
 	"path/filepath"
+	"runtime"
 	"sort"
 	"strconv"
 	"strings"
@@ -21,8 +22,21 @@ type LlamaCppConfig struct {
 	// does not say. llama.cpp defaults to one, which would make every second
 	// request wait behind the first no matter what the router decided.
 	Parallel int `koanf:"parallel"`
+	// Threads is how many CPU threads a shard with no GPU computes on. Zero
+	// takes half this machine's CPUs.
+	//
+	// Not all of them, which is llama.cpp's own default: two busy servers
+	// that each spin on every CPU fight for them, and on an 8-vCPU node
+	// that made 7B generation fall from 10 tokens a second to under 1 each.
+	// At half, one server alone is about a third slower and two together
+	// keep 5.5 each. A node that only ever runs one CPU replica does better
+	// with all of them; say so here.
+	Threads int `koanf:"threads"`
 }
 
+// numCPU is the CPUs this process may run on; a variable for tests.
+var numCPU = runtime.NumCPU
+
 // LlamaCpp builds llama.cpp launch commands.
 //
 // It is the engine that makes a cluster without GPUs useful: llama.cpp runs a
@@ -118,6 +132,13 @@ func (l *LlamaCpp) Commands(s ShardSpec) ([]Command, error) {
 	if len(s.GPUIndices) > 0 && !hasFlag(p.Args, "--n-gpu-layers", "-ngl") {
 		args = append(args, "--n-gpu-layers", "999")
 	}
+	if len(s.GPUIndices) == 0 && !hasFlag(p.Args, "--threads", "-t") {
+		n := l.cfg.Threads
+		if n <= 0 {
+			n = max(numCPU()/2, 1)
+		}
+		args = append(args, "--threads", strconv.Itoa(n))
+	}
 	args = append(args, p.Args...)
 
 	env := baseEnv(s)

internal/agent/engines/llamacpp_test.go

@@ -261,3 +261,36 @@ func TestLlamaCppIsGivenItsKeyOutOfSightOfTheProcessList(t *testing.T) {
 		t.Error("llama-server was not given its key")
 	}
 }
+
+func TestACPUShardLeavesHalfTheMachineForAnother(t *testing.T) {
+	prev := numCPU
+	numCPU = func() int { return 8 }
+	t.Cleanup(func() { numCPU = prev })
+
+	threads := func(l *LlamaCpp, s ShardSpec) (string, bool) {
+		cmds, err := l.Commands(s)
+		if err != nil {
+			t.Fatal(err)
+		}
+		return flagValue(cmds[0].Args, "--threads")
+	}
+	if v, _ := threads(&LlamaCpp{}, llamaSpec(nil)); v != "4" {
+		t.Errorf("--threads %q on 8 CPUs, expected 4", v)
+	}
+	if v, _ := threads(&LlamaCpp{cfg: LlamaCppConfig{Threads: 8}}, llamaSpec(nil)); v != "8" {
+		t.Errorf("--threads %q with 8 configured", v)
+	}
+	if _, ok := threads(&LlamaCpp{}, llamaSpec([]int{0})); ok {
+		t.Error("a GPU shard was given a thread count")
+	}
+	s := llamaSpec(nil)
+	s.Profile.Args = []string{"-t", "2"}
+	cmds, _ := (&LlamaCpp{}).Commands(s)
+	if slices.Contains(cmds[0].Args, "--threads") {
+		t.Error("the profile's own -t was overridden")
+	}
+	numCPU = func() int { return 1 }
+	if v, _ := threads(&LlamaCpp{}, llamaSpec(nil)); v != "1" {
+		t.Errorf("--threads %q on one CPU", v)
+	}
+}

internal/agent/engines/vllm.go

@@ -169,7 +169,10 @@ func (v *VLLM) rayCommand(s ShardSpec) (Command, bool) {
 
 	args := []string{"start", "--block"}
 	if s.Leader() {
-		args = append(args, "--head", "--port", strconv.Itoa(s.LeaderPort))
+		// Ray's head has no authentication: whoever reaches its port can run
+		// code on the node. Its dashboard, and the job API behind it, is the
+		// easiest way in, and nothing here uses it.
+		args = append(args, "--head", "--port", strconv.Itoa(s.LeaderPort), "--include-dashboard=false")
 	} else {
 		args = append(args, "--address", fmt.Sprintf("%s:%d", s.LeaderAddr, s.LeaderPort))
 	}

internal/agent/fabrics.go

@@ -0,0 +1,345 @@
+package agent
+
+import (
+	"context"
+	"fmt"
+	"os/exec"
+	"slices"
+	"strconv"
+	"strings"
+	"time"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
+)
+
+// usableFabrics reports the fabrics this node can actually carry traffic on.
+//
+// Loud about what it drops. A fabric silently missing from a registration looks
+// identical to one never configured, and the operator who typed the adapter
+// name is the only person who can fix it — so the reason is logged here, on the
+// machine where both halves are visible, rather than inferred later from a
+// replica that will not converge.
+func (a *Agent) usableFabrics(inv discovery.Inventory) []inventory.FabricAttachment {
+	out := a.checkFabrics(inv.HCAs, true)
+	a.mu.Lock()
+	a.fabricsSent = fabricsFingerprint(out)
+	a.mu.Unlock()
+	return out
+}
+
+// checkFabrics is the declared fabrics this node can carry traffic on, from
+// the adapters as they are now. Loud logs every problem; the heartbeat checks
+// quietly and speaks only when the answer changes.
+func (a *Agent) checkFabrics(hcas []discovery.HCA, loud bool) []inventory.FabricAttachment {
+	declared := map[string][]string{}
+	for _, f := range a.cfg.Fabrics {
+		declared[f.Fabric] = f.HCAs
+	}
+
+	var out []inventory.FabricAttachment
+	for _, c := range discovery.CheckFabrics(declared, hcas) {
+		if loud {
+			for _, p := range c.Problems {
+				a.log.Error("a declared fabric adapter cannot carry traffic",
+					"fabric", c.Fabric, "problem", p)
+			}
+		}
+		if !c.OK() {
+			if loud {
+				a.log.Error("fabric not advertised; no declared adapter on it can carry traffic",
+					"fabric", c.Fabric,
+					"consequence", "no multi-node replica will be placed here")
+			}
+			continue
+		}
+		out = append(out, inventory.FabricAttachment{
+			Fabric: c.Fabric, HCAs: c.Usable, RateGbps: c.RateGbps,
+		})
+	}
+	return out
+}
+
+// fabricsFingerprint is enough to notice a fabric, an adapter or a rate
+// changing.
+func fabricsFingerprint(fs []inventory.FabricAttachment) string {
+	var b strings.Builder
+	for _, f := range fs {
+		fmt.Fprintf(&b, "%s:%s:%d;", f.Fabric, strings.Join(f.HCAs, ","), f.RateGbps)
+	}
+	return b.String()
+}
+
+// sampleIB re-reads the adapters for the heartbeat: every port's state and
+// counters, and the fabrics again when they no longer match what the control
+// plane was told.
+//
+// Every heartbeat, because it is a few small files per port, and because a
+// port that went down used to be invisible until the agent restarted — the
+// node went on being offered multi-node work that would hang on its first
+// collective.
+func (a *Agent) sampleIB(hb *llmgwv1.Heartbeat) {
+	a.mu.Lock()
+	had := len(a.inv.HCAs) > 0
+	a.mu.Unlock()
+	if !had && len(a.cfg.Fabrics) == 0 {
+		return
+	}
+	hcas, err := discovery.DiscoverIB(a.cfg.Discovery.IBRoot)
+	if err != nil {
+		return
+	}
+	a.attachSM(hcas)
+	for _, h := range hcas {
+		for _, p := range h.Ports {
+			ip := &llmgwv1.IBPort{
+				Hca: h.Name, Port: int32(p.Num), State: p.State, PhysState: p.PhysState,
+				RateGbps: int32(p.RateGbps), LinkLayer: string(p.LinkLayer), Counters: p.Counters,
+				NodeGuid: h.NodeGUID, PortGuid: p.PortGUID, Network: p.Network, GidPrefix: p.GIDPrefix,
+			}
+			a.mu.Lock()
+			if p.SM != nil {
+				ip.SmGuid, ip.SmLid, ip.SmPriority, ip.SmState = p.SM.GUID, int32(p.SM.LID), int32(p.SM.Priority), p.SM.State
+			} else {
+				ip.SmError = a.smErr[smKey(h.Name, p.Num)]
+			}
+			for _, r := range a.sms[smKey(h.Name, p.Num)] {
+				ip.SubnetManagers = append(ip.SubnetManagers, &llmgwv1.SubnetManager{
+					Guid: r.GUID, Lid: int32(r.LID), Priority: int32(r.Priority), State: r.State})
+			}
+			ip.SubnetManagersError = a.smsErr[smKey(h.Name, p.Num)]
+			a.mu.Unlock()
+			hb.IbPorts = append(hb.IbPorts, ip)
+		}
+	}
+	a.mu.Lock()
+	a.inv.HCAs = hcas // a reconnect registers what is true now
+	a.mu.Unlock()
+	if len(a.cfg.Fabrics) == 0 {
+		return
+	}
+	now := a.checkFabrics(hcas, false)
+	fp := fabricsFingerprint(now)
+	a.mu.Lock()
+	changed := fp != a.fabricsSent
+	a.fabricsSent = fp
+	a.mu.Unlock()
+	if !changed {
+		return
+	}
+	// Said once, with the reasons, the moment it happens.
+	a.checkFabrics(hcas, true)
+	a.log.Warn("the fabrics this node can carry traffic on changed", "now", fp)
+	hb.FabricsChanged = true
+	for _, f := range now {
+		hb.Fabrics = append(hb.Fabrics, &llmgwv1.FabricAttachment{
+			Fabric: f.Fabric, Hcas: f.HCAs, RateGbps: int32(f.RateGbps),
+		})
+	}
+}
+
+func smKey(hca string, port int) string { return fmt.Sprintf("%s/%d", hca, port) }
+
+// scanFabric is discovery.ScanFabric, replaced in tests.
+var scanFabric = discovery.ScanFabric
+
+// querySMs is discovery.QuerySMs, replaced in tests.
+var querySMs = func(hca string, port int) ([]discovery.SMRecord, error) {
+	return discovery.QuerySMs(context.Background(), hca, port)
+}
+
+func splitSMKey(k string) (string, int) {
+	i := strings.LastIndexByte(k, '/')
+	port, _ := strconv.Atoi(k[i+1:])
+	return k[:i], port
+}
+
+// querySM is discovery.QuerySM, replaced in tests.
+var querySM = func(hca string, port int) (*discovery.SMInfo, error) {
+	return discovery.QuerySM(context.Background(), hca, port)
+}
+
+// attachSM puts on each active InfiniBand port the subnet manager it sees,
+// asking again once a minute and otherwise reusing the last answer.
+func (a *Agent) attachSM(hcas []discovery.HCA) {
+	a.mu.Lock()
+	stale := time.Since(a.smAsked) > time.Minute || a.sm == nil
+	a.mu.Unlock()
+	if stale {
+		sm, errs := map[string]*discovery.SMInfo{}, map[string]string{}
+		_ = discovery.AttachSM(hcas, func(hca string, port int) (*discovery.SMInfo, error) {
+			got, err := querySM(hca, port)
+			if err != nil {
+				errs[smKey(hca, port)] = err.Error()
+				return nil, err
+			}
+			sm[smKey(hca, port)] = got
+			return got, nil
+		})
+		// Every subnet manager on each fabric, through one port per master:
+		// the same list through every port of one fabric is one query too
+		// many for each.
+		sms, smsErr := map[string][]discovery.SMRecord{}, map[string]string{}
+		asked := map[string]bool{}
+		for k, v := range sm {
+			if asked[v.GUID] {
+				continue
+			}
+			asked[v.GUID] = true
+			hca, port := splitSMKey(k)
+			if recs, err := querySMs(hca, port); err != nil {
+				smsErr[k] = err.Error()
+			} else {
+				sms[k] = recs
+			}
+		}
+		a.mu.Lock()
+		for k, v := range sm {
+			if old := a.sm[k]; old != nil && old.GUID != v.GUID {
+				a.log.Warn("the master subnet manager changed", "port", k, "from", old.GUID, "to", v.GUID)
+			}
+		}
+		a.sm, a.smErr, a.smAsked = sm, errs, time.Now()
+		a.sms, a.smsErr = sms, smsErr
+		a.mu.Unlock()
+		return
+	}
+	a.mu.Lock()
+	defer a.mu.Unlock()
+	for i := range hcas {
+		for j := range hcas[i].Ports {
+			if sm := a.sm[smKey(hcas[i].Name, hcas[i].Ports[j].Num)]; sm != nil && hcas[i].Ports[j].Active() {
+				hcas[i].Ports[j].SM = sm
+			}
+		}
+	}
+}
+
+// ensureDiags installs infiniband-diags on a node with adapters and without
+// it. sminfo is what tells which physical fabric a port is on; without it,
+// two separate fabrics given one name cannot be caught. Installed the way
+// engine dependencies are, from the machine's own package sources.
+func (a *Agent) ensureDiags(ctx context.Context) {
+	a.mu.Lock()
+	hasIB := len(a.inv.HCAs) > 0
+	a.mu.Unlock()
+	if !hasIB || a.prov == nil {
+		return
+	}
+	if _, err := exec.LookPath("sminfo"); err == nil {
+		return
+	}
+	a.log.Info("installing infiniband-diags, to tell which fabric each port is on")
+	pkgs := map[string][]string{"deb": {"infiniband-diags"}, "rpm": {"infiniband-diags"}}
+	if err := a.prov.InstallPackages(ctx, pkgs); err != nil {
+		a.log.Warn("could not install infiniband-diags; fabrics cannot be told apart by their subnet manager",
+			"err", err)
+	}
+}
+
+// FeatureFabricTest is the agent answering FabricTestRequest.
+const FeatureFabricTest = "fabric-test"
+
+// runWriteBW is discovery.RunWriteBW, and haveWriteBW whether it is
+// installed; both replaced in tests.
+var (
+	runWriteBW  = discovery.RunWriteBW
+	haveWriteBW = func() bool { _, err := exec.LookPath("ib_write_bw"); return err == nil }
+)
+
+// fabricTest runs this node's side of a bandwidth test with another.
+func (a *Agent) fabricTest(ctx context.Context, req *llmgwv1.FabricTestRequest) *llmgwv1.FabricTestReply {
+	start := time.Now()
+	reply := &llmgwv1.FabricTestReply{RequestId: req.GetRequestId()}
+	defer func() { reply.TookMs = time.Since(start).Milliseconds() }()
+
+	w := discovery.WriteBW{HCA: req.GetHca(), Port: int(req.GetPort()), GIDIndex: -1,
+		TCPPort: int(req.GetTcpPort()), Seconds: int(req.GetSeconds())}
+	if req.GetRole() == "client" {
+		w.Peer = req.GetPeer()
+	}
+	a.mu.Lock()
+	found := false
+	for _, h := range a.inv.HCAs {
+		for _, p := range h.Ports {
+			if h.Name == w.HCA && p.Num == w.Port {
+				found = true
+				// RoCE needs its v2 GID named, as NCCL does; InfiniBand uses the LID.
+				if p.LinkLayer == discovery.LinkEthernet && p.RoCEv2GID >= 0 {
+					w.GIDIndex = p.RoCEv2GID
+				}
+			}
+		}
+	}
+	a.mu.Unlock()
+	if !found {
+		reply.Error = fmt.Sprintf("this node has no port %s/%d", w.HCA, w.Port)
+		return reply
+	}
+	if !haveWriteBW() && a.prov != nil {
+		a.log.Info("installing perftest, for a bandwidth test with another node")
+		pkgs := map[string][]string{"deb": {"perftest"}, "rpm": {"perftest"}}
+		if err := a.prov.InstallPackages(ctx, pkgs); err != nil {
+			reply.Error = "installing perftest: " + err.Error()
+			return reply
+		}
+	}
+	gbps, out, err := runWriteBW(ctx, w)
+	reply.Gbps = gbps
+	reply.Output = tailLines(out, 12)
+	if err != nil {
+		reply.Error = err.Error()
+	}
+	return reply
+}
+
+func tailLines(s string, n int) string {
+	lines := strings.Split(strings.TrimRight(s, "\n"), "\n")
+	if len(lines) > n {
+		lines = lines[len(lines)-n:]
+	}
+	return strings.Join(lines, "\n")
+}
+
+// discoverGPUFabric and discoverIMEX are replaced in tests.
+var (
+	discoverGPUFabric = discovery.DiscoverGPUFabric
+	discoverIMEX      = discovery.DiscoverIMEX
+)
+
+// sampleNVLink reads, once a minute, which NVLink domain each GPU is joined
+// to and whether IMEX is up. nvidia-smi -q takes a second or two, too long
+// for every heartbeat, and a fabric that was joined an hour ago rarely
+// changes in five seconds — but when a fabric manager restarts it does, for
+// every card at once.
+func (a *Agent) sampleNVLink(hb *llmgwv1.Heartbeat) {
+	a.mu.Lock()
+	gpus := slices.Clone(a.inv.GPUs)
+	due := time.Since(a.nvlAsked) > time.Minute
+	if due {
+		a.nvlAsked = time.Now()
+	}
+	a.mu.Unlock()
+	if len(gpus) == 0 || !due {
+		return
+	}
+	ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
+	defer cancel()
+	fab, err := discoverGPUFabric(ctx, a.cfg.Discovery.NvidiaSMI)
+	if err != nil || len(fab) == 0 {
+		return
+	}
+	discovery.ApplyFabric(gpus, fab)
+	for _, g := range gpus {
+		if g.Fabric == nil {
+			continue
+		}
+		hb.GpuFabric = append(hb.GpuFabric, &llmgwv1.GPUFabricState{
+			Index: int32(g.Index), State: g.Fabric.State, Status: g.Fabric.Status,
+			ClusterUuid: g.Fabric.ClusterUUID, CliqueId: g.Fabric.CliqueID, Bandwidth: g.Fabric.Bandwidth,
+		})
+	}
+	m := discoverIMEX(ctx)
+	hb.Imex = &llmgwv1.IMEX{State: m.State, Channels: int32(m.Channels)}
+}

internal/agent/logs.go

@@ -0,0 +1,34 @@
+package agent
+
+import (
+	"slices"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+)
+
+// logsReply answers a request for one of this node's logs.
+func (a *Agent) logsReply(req *llmgwv1.LogsRequest) *llmgwv1.LogsReply {
+	n := int(req.GetLines())
+	if n <= 0 {
+		n = 200
+	}
+	n = min(n, agentLogLines)
+	shards := a.sup.IDs()
+	slices.Sort(shards)
+	out := &llmgwv1.LogsReply{
+		RequestId: req.GetRequestId(),
+		Sources:   append([]string{"agent"}, shards...),
+	}
+	switch src := req.GetSource(); src {
+	case "", "agent":
+		out.Lines = a.logRing.tail(n)
+	default:
+		sh, err := a.sup.Get(src)
+		if err != nil {
+			out.Error = "no engine " + src + " runs on this node"
+			return out
+		}
+		out.Lines = sh.Logs(n)
+	}
+	return out
+}

internal/agent/provision.go

@@ -0,0 +1,197 @@
+package agent
+
+import (
+	"context"
+	"fmt"
+	"log/slog"
+	"maps"
+	"slices"
+	"strings"
+	"time"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
+)
+
+// engines is what this node can run: what the agent installed and saw run,
+// and what the configuration names and answers. Nothing merely found on the
+// machine — a llama-server on PATH is somebody's, not necessarily one that
+// works, and placing on it is how a replica fails on a node nobody watches.
+func (a *Agent) engines(ctx context.Context) map[string]string {
+	a.engineOnce.Do(func() {
+		vers := a.reg.ProbeConfigured(ctx)
+		a.mu.Lock()
+		a.configuredVers, a.lastEngineProbe = vers, time.Now()
+		a.mu.Unlock()
+		if len(vers) > 0 {
+			a.log.Info("engines named in the configuration",
+				"engines", engines.ProbedNames(vers))
+		}
+	})
+	a.mu.Lock()
+	defer a.mu.Unlock()
+	vers := a.engineReportLocked()
+	// Whatever registration sends is what the control plane now knows, so a
+	// heartbeat repeats it only once it differs.
+	a.enginesSent = enginesFingerprint(vers)
+	return vers
+}
+
+// engineReportLocked merges the two sources. Installed wins: it is the one
+// that was seen to run. Called with mu held.
+func (a *Agent) engineReportLocked() map[string]string {
+	out := maps.Clone(a.configuredVers)
+	if out == nil {
+		out = map[string]string{}
+	}
+	maps.Copy(out, a.installedVers)
+	return out
+}
+
+// provisionEvery is how often installed engines are checked again. A library
+// removed by an unrelated upgrade is found here rather than by the next
+// replica to start.
+const provisionEvery = 10 * time.Minute
+
+// setProvision takes the control plane's list of wanted engines.
+func (a *Agent) setProvision(p *llmgwv1.Provision) {
+	wants := []provision.Want{}
+	for _, e := range p.GetEngines() {
+		wants = append(wants, provision.Want{Engine: e.GetEngine(), Version: e.GetVersion()})
+	}
+	a.mu.Lock()
+	a.provWants = wants
+	a.mu.Unlock()
+	select {
+	case a.provKick <- struct{}{}:
+	default:
+	}
+}
+
+// provisionLoop installs and re-checks the wanted engines, one pass at a time.
+func (a *Agent) provisionLoop(ctx context.Context) {
+	// After a failure the next pass comes sooner, and backs off: a lock held
+	// by a new machine's own updates clears in a minute or two, while ten
+	// minutes' wait left a node idle for no reason.
+	next := provisionEvery
+	retry := time.Minute
+	for {
+		select {
+		case <-ctx.Done():
+			return
+		case <-a.provKick:
+		case <-time.After(next):
+		}
+		a.mu.Lock()
+		wants := slices.Clone(a.provWants)
+		managed := a.provWants != nil
+		a.mu.Unlock()
+		if !managed {
+			continue
+		}
+
+		statuses := a.prov.Converge(ctx, wants)
+		next = provisionEvery
+		for _, st := range statuses {
+			if st.State == provision.Failed {
+				next, retry = retry, min(retry*2, provisionEvery)
+				break
+			}
+		}
+		if next == provisionEvery {
+			retry = time.Minute
+		}
+		installed := map[string]string{}
+		keep := map[string]bool{}
+		for _, st := range statuses {
+			keep[st.Engine] = true
+			if st.State == provision.Ready {
+				a.reg.UseInstalled(st.Engine, st.Binary)
+				installed[st.Engine] = st.Version
+			}
+		}
+		a.mu.Lock()
+		if enginesFingerprint(installed) != enginesFingerprint(a.installedVers) {
+			a.log.Info("engines installed and running", "engines", engines.ProbedNames(installed))
+		}
+		a.installedVers = installed
+		// An engine no longer asked for is no longer reported on.
+		for e := range a.provStatus {
+			if !keep[e] {
+				delete(a.provStatus, e)
+			}
+		}
+		a.mu.Unlock()
+	}
+}
+
+// provisionStatus records a change in one engine's installation.
+func (a *Agent) provisionStatus(st provision.Status) {
+	a.mu.Lock()
+	prev, had := a.provStatus[st.Engine]
+	a.provStatus[st.Engine] = st
+	a.mu.Unlock()
+	if !had || prev.State != st.State || prev.Detail != st.Detail {
+		lvl := slog.LevelInfo
+		if st.State == provision.Failed || st.State == provision.Unsupported {
+			lvl = slog.LevelWarn
+		}
+		a.log.Log(context.Background(), lvl, "engine installation",
+			"engine", st.Engine, "version", st.Version, "variant", st.Variant,
+			"state", st.State, "detail", st.Detail)
+	}
+}
+
+func provisionFingerprint(m map[string]provision.Status) string {
+	var b strings.Builder
+	for _, k := range slices.Sorted(maps.Keys(m)) {
+		st := m[k]
+		fmt.Fprintf(&b, "%s|%s|%s|%s|%s\n", st.Engine, st.Version, st.Variant, st.State, st.Detail)
+	}
+	return b.String()
+}
+
+// engineReprobe is how often the installed engines are looked for again.
+//
+// Minutes, not seconds: an engine is installed by somebody at a keyboard, so a
+// minute's delay in noticing is invisible, while SGLang's probe starts a
+// Python interpreter that imports it.
+const engineReprobe = time.Minute
+
+// reprobeEngines looks for installed engines again when it is time, off the
+// heartbeat goroutine so a slow probe never delays a heartbeat.
+func (a *Agent) reprobeEngines() {
+	a.mu.Lock()
+	due := !a.probingEngines && !a.lastEngineProbe.IsZero() &&
+		time.Since(a.lastEngineProbe) >= engineReprobe
+	if due {
+		a.probingEngines = true
+	}
+	a.mu.Unlock()
+	if !due {
+		return
+	}
+	go func() {
+		ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
+		defer cancel()
+		vers := a.reg.ProbeConfigured(ctx)
+
+		a.mu.Lock()
+		changed := enginesFingerprint(vers) != enginesFingerprint(a.configuredVers)
+		a.configuredVers, a.lastEngineProbe, a.probingEngines = vers, time.Now(), false
+		a.mu.Unlock()
+		if changed {
+			a.log.Info("engines named in the configuration changed",
+				"engines", engines.ProbedNames(vers))
+		}
+	}()
+}
+
+func enginesFingerprint(vers map[string]string) string {
+	var b strings.Builder
+	for _, k := range slices.Sorted(maps.Keys(vers)) {
+		fmt.Fprintf(&b, "%s=%s\n", k, vers[k])
+	}
+	return b.String()
+}

internal/agent/provision/install.go

@@ -523,6 +523,11 @@ func (in *Installer) download(ctx context.Context, repo, tag, asset, dst, want s
 // untarGz unpacks an archive under dir, refusing anything that would land
 // outside it. A release archive is somebody else's file, and a path of
 // "../../etc/cron.d/x" in it would otherwise be written as root.
+//
+// Every write goes through an os.Root, which refuses to leave dir however the
+// path gets there. Checking names alone was not enough: "a -> ." followed by
+// "a/a/b -> ../.." reads as inside and resolves two levels above it, and a
+// file written through that lands wherever the links point.
 func untarGz(archive, dir string) error {
 	f, err := os.Open(archive) //nolint:gosec // inside the install root
 	if err != nil {
@@ -533,6 +538,11 @@ func untarGz(archive, dir string) error {
 	if err != nil {
 		return err
 	}
+	root, err := os.OpenRoot(dir)
+	if err != nil {
+		return err
+	}
+	defer root.Close()
 	tr := tar.NewReader(gz)
 	for {
 		h, err := tr.Next()
@@ -542,22 +552,22 @@ func untarGz(archive, dir string) error {
 		if err != nil {
 			return err
 		}
-		target, err := within(dir, h.Name)
-		if err != nil {
+		name := filepath.Clean(h.Name)
+		if _, err := within(dir, name); err != nil {
 			return err
 		}
 		switch h.Typeflag {
 		case tar.TypeDir:
-			if err := os.MkdirAll(target, 0o755); err != nil {
+			if err := root.MkdirAll(name, 0o755); err != nil {
 				return err
 			}
 		case tar.TypeReg:
-			if err := os.MkdirAll(filepath.Dir(target), 0o755); err != nil {
+			if err := root.MkdirAll(filepath.Dir(name), 0o755); err != nil {
 				return err
 			}
-			out, err := os.OpenFile(target, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, fs.FileMode(h.Mode)&0o755) //nolint:gosec // checked above
+			out, err := root.OpenFile(name, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, fs.FileMode(h.Mode)&0o755)
 			if err != nil {
-				return err
+				return fmt.Errorf("%s: %w", h.Name, err)
 			}
 			if _, err := io.Copy(out, tr); err != nil { //nolint:gosec // a release archive, size bounded by its checksum
 				out.Close()
@@ -567,19 +577,19 @@ func untarGz(archive, dir string) error {
 				return err
 			}
 		case tar.TypeSymlink:
-			// Libraries are versioned through symlinks, and those are
-			// relative. One pointing anywhere else is refused.
-			if filepath.IsAbs(h.Linkname) {
-				return fmt.Errorf("%s links outside the archive", h.Name)
+			// Libraries are versioned through symlinks to a file beside them —
+			// libllama.so -> libllama.so.0 — and that is all a release needs.
+			// A target with a separator in it could point anywhere once other
+			// links are followed, and a binary found through one would be run
+			// as root, so it is refused rather than reasoned about.
+			if h.Linkname == "" || strings.ContainsAny(h.Linkname, `/\`) || h.Linkname == ".." || h.Linkname == "." {
+				return fmt.Errorf("%s links to %q; only a file beside it is allowed", h.Name, h.Linkname)
 			}
-			if _, err := within(dir, filepath.Join(filepath.Dir(h.Name), h.Linkname)); err != nil {
-				return fmt.Errorf("%s links outside the archive", h.Name)
-			}
-			if err := os.MkdirAll(filepath.Dir(target), 0o755); err != nil {
+			if err := root.MkdirAll(filepath.Dir(name), 0o755); err != nil {
 				return err
 			}
-			_ = os.Remove(target)
-			if err := os.Symlink(h.Linkname, target); err != nil {
+			_ = root.Remove(name)
+			if err := root.Symlink(h.Linkname, name); err != nil {
 				return err
 			}
 		default:

internal/agent/provision/install_test.go

@@ -342,3 +342,49 @@ func TestAVersionThatIsAPathIsRefused(t *testing.T) {
 		}
 	}
 }
+
+func TestChainedLinksCannotCarryAFileOutOfTheInstall(t *testing.T) {
+	// Each link reads as inside the directory; together they point above it.
+	var buf bytes.Buffer
+	gz := gzip.NewWriter(&buf)
+	tw := tar.NewWriter(gz)
+	_ = tw.WriteHeader(&tar.Header{Name: "a", Linkname: ".", Typeflag: tar.TypeSymlink})
+	_ = tw.WriteHeader(&tar.Header{Name: "a/a/b", Linkname: "../..", Typeflag: tar.TypeSymlink})
+	_ = tw.WriteHeader(&tar.Header{Name: "a/a/b/escaped", Mode: 0o644, Size: 1, Typeflag: tar.TypeReg})
+	_, _ = tw.Write([]byte("x"))
+	_ = tw.Close()
+	_ = gz.Close()
+
+	parent := t.TempDir()
+	dir := filepath.Join(parent, "install")
+	_ = os.MkdirAll(dir, 0o755)
+	archive := filepath.Join(parent, "evil.tar.gz")
+	_ = os.WriteFile(archive, buf.Bytes(), 0o644)
+
+	if err := untarGz(archive, dir); err == nil {
+		t.Error("an archive of chained links was unpacked")
+	}
+	if _, err := os.Stat(filepath.Join(parent, "escaped")); err == nil {
+		t.Fatal("a file was written outside the install directory")
+	}
+}
+
+func TestALibrarysVersionLinkIsStillUnpacked(t *testing.T) {
+	var buf bytes.Buffer
+	gz := gzip.NewWriter(&buf)
+	tw := tar.NewWriter(gz)
+	_ = tw.WriteHeader(&tar.Header{Name: "lib/libllama.so.0", Mode: 0o755, Size: 1, Typeflag: tar.TypeReg})
+	_, _ = tw.Write([]byte("x"))
+	_ = tw.WriteHeader(&tar.Header{Name: "lib/libllama.so", Linkname: "libllama.so.0", Typeflag: tar.TypeSymlink})
+	_ = tw.Close()
+	_ = gz.Close()
+	dir := t.TempDir()
+	archive := filepath.Join(t.TempDir(), "ok.tar.gz")
+	_ = os.WriteFile(archive, buf.Bytes(), 0o644)
+	if err := untarGz(archive, dir); err != nil {
+		t.Fatal(err)
+	}
+	if target, err := os.Readlink(filepath.Join(dir, "lib", "libllama.so")); err != nil || target != "libllama.so.0" {
+		t.Errorf("link %q %v", target, err)
+	}
+}

internal/agent/upgrade.go

@@ -0,0 +1,58 @@
+package agent
+
+import (
+	"context"
+	"os"
+	"runtime"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/release"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+)
+
+// upgradeExit is the exit status after an upgrade: non-zero, so systemd's
+// Restart=on-failure starts the new binary, and distinct, so the journal says
+// why the agent went.
+const upgradeExit = 75
+
+// exitForUpgrade ends the process; a variable so a test can see it called
+// rather than have its own process end.
+var exitForUpgrade = func() { os.Exit(upgradeExit) }
+
+// upgradeTarget is the binary an upgrade replaces; a variable so a test
+// replaces a file of its own rather than the test binary running it.
+var upgradeTarget = os.Executable
+
+// upgrade replaces this agent's binary with a release and restarts into it.
+//
+// Engines are not touched: an agent restart leaves them running and adopts
+// them again, which is what makes upgrading a node something that can be done
+// while it serves.
+func (a *Agent) upgrade(ctx context.Context, m *llmgwv1.UpgradeAgent, out chan<- *llmgwv1.AgentMessage) {
+	reply := &llmgwv1.UpgradeReply{RequestId: m.GetRequestId(), Version: m.GetVersion()}
+	send := func() {
+		select {
+		case out <- &llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_Upgrade{Upgrade: reply}}:
+		case <-ctx.Done():
+		}
+	}
+	a.log.Info("upgrading", "to", m.GetVersion(), "from", a.cfg.Version)
+	self, err := upgradeTarget()
+	if err == nil {
+		dctx, cancel := context.WithTimeout(ctx, 10*time.Minute)
+		_, err = release.Install(dctx, nil, m.GetReleases(), m.GetVersion(), runtime.GOOS, runtime.GOARCH, self)
+		cancel()
+	}
+	if err != nil {
+		reply.Error = err.Error()
+		a.log.Error("upgrade failed; still running the old version", "to", m.GetVersion(), "err", err)
+		send()
+		return
+	}
+	send()
+	a.log.Info("upgraded; restarting into the new version", "to", m.GetVersion())
+	// A moment for the reply to leave before the stream goes.
+	time.Sleep(500 * time.Millisecond)
+	exitForUpgrade()
+}

internal/agent/upgrade_test.go

@@ -0,0 +1,78 @@
+package agent
+
+import (
+	"crypto/sha256"
+	"encoding/hex"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"runtime"
+	"testing"
+	"time"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/release"
+)
+
+func TestAnAgentUpgradesItselfAndSaysSoBeforeRestarting(t *testing.T) {
+	body := []byte("the new agent")
+	sum := sha256.Sum256(body)
+	asset := release.AssetName("v9.9.9", runtime.GOOS, runtime.GOARCH)
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		switch r.URL.Path {
+		case "/v9.9.9/checksums.txt":
+			_, _ = w.Write([]byte(hex.EncodeToString(sum[:]) + "  " + asset + "\n"))
+		case "/v9.9.9/" + asset:
+			_, _ = w.Write(body)
+		default:
+			http.NotFound(w, r)
+		}
+	}))
+	defer srv.Close()
+
+	bin := filepath.Join(t.TempDir(), "llmgw")
+	_ = os.WriteFile(bin, []byte("the old agent"), 0o755)
+	exited := make(chan struct{}, 1)
+	oldTarget, oldExit := upgradeTarget, exitForUpgrade
+	upgradeTarget = func() (string, error) { return bin, nil }
+	exitForUpgrade = func() { exited <- struct{}{} }
+	t.Cleanup(func() { upgradeTarget, exitForUpgrade = oldTarget, oldExit })
+
+	cfg := Config{Server: "127.0.0.1:7443", NodeID: "n1", WeightsCache: t.TempDir(), EnginesDir: t.TempDir()}
+	cfg.Superv.WorkDir = t.TempDir()
+	a, err := New(cfg, nil, nil)
+	if err != nil {
+		t.Fatal(err)
+	}
+	out := make(chan *llmgwv1.AgentMessage, 1)
+	go a.upgrade(t.Context(), &llmgwv1.UpgradeAgent{RequestId: "u1", Version: "v9.9.9", Releases: srv.URL}, out)
+
+	select {
+	case m := <-out:
+		if r := m.GetUpgrade(); r.GetRequestId() != "u1" || r.GetError() != "" {
+			t.Fatalf("reply %+v", r)
+		}
+	case <-time.After(10 * time.Second):
+		t.Fatal("no reply")
+	}
+	select {
+	case <-exited:
+	case <-time.After(5 * time.Second):
+		t.Fatal("the agent did not restart into the new binary")
+	}
+	if b, _ := os.ReadFile(bin); string(b) != "the new agent" {
+		t.Errorf("binary holds %q", b)
+	}
+
+	// A version with no release answers with why, and does not restart.
+	go a.upgrade(t.Context(), &llmgwv1.UpgradeAgent{RequestId: "u2", Version: "v9.9.8", Releases: srv.URL}, out)
+	if r := (<-out).GetUpgrade(); r.GetError() == "" {
+		t.Error("an upgrade to a release that does not exist reported success")
+	}
+	select {
+	case <-exited:
+		t.Error("a failed upgrade restarted the agent")
+	case <-time.After(time.Second):
+	}
+}

internal/cli/backup.go

@@ -141,13 +141,17 @@ func addStore(ctx context.Context, tw *tar.Writer, addr, scratch string) error {
 	if err != nil {
 		return err
 	}
-	tmp, err := os.CreateTemp(scratch, ".store-*")
+	// Inside a directory only its owner can enter. The archive directory is
+	// listable on purpose, and VACUUM INTO creates its file with the default
+	// mode: made there directly, a full copy of the store — password hashes,
+	// the engine-key secret — sat readable by any local user for as long as
+	// the backup took.
+	tmpDir, err := os.MkdirTemp(scratch, ".store-")
 	if err != nil {
 		return err
 	}
-	tmpPath := tmp.Name()
-	_ = tmp.Close()
-	defer os.Remove(tmpPath)
+	defer os.RemoveAll(tmpDir)
+	tmpPath := filepath.Join(tmpDir, "store")
 
 	name := "store.sqlite"
 	if store.IsPostgres(addr) {
@@ -156,8 +160,6 @@ func addStore(ctx context.Context, tw *tar.Writer, addr, scratch string) error {
 			return err
 		}
 	} else {
-		// VACUUM INTO refuses a file that exists.
-		_ = os.Remove(tmpPath)
 		db, err := store.Open(ctx, addr)
 		if err != nil {
 			return err

internal/cli/backup_test.go

@@ -6,6 +6,7 @@ import (
 	"io"
 	"os"
 	"path/filepath"
+	"strings"
 	"testing"
 	"time"
 
@@ -116,3 +117,25 @@ func TestABackupDirectoryTheControlPlaneCannotListIsSaid(t *testing.T) {
 		t.Errorf("a listable directory: %s", why)
 	}
 }
+
+func TestNoScratchCopyOfTheStoreIsLeftBehind(t *testing.T) {
+	// The snapshot is made before it is archived. Made straight into the
+	// listable archive directory, it was readable by anyone for that time.
+	tmp := t.TempDir()
+	dbPath := filepath.Join(tmp, "store.db")
+	db, err := store.Open(t.Context(), dbPath)
+	if err != nil {
+		t.Fatal(err)
+	}
+	_ = db.Close()
+	out := filepath.Join(tmp, "backups")
+	if _, err := writeBackup(t.Context(), dbPath, t.TempDir(), out); err != nil {
+		t.Fatal(err)
+	}
+	entries, _ := os.ReadDir(out)
+	for _, e := range entries {
+		if strings.HasPrefix(e.Name(), ".store") {
+			t.Errorf("a scratch copy of the store was left behind: %s", e.Name())
+		}
+	}
+}

internal/cli/ctl.go

@@ -122,6 +122,7 @@ func newCtlCmd() *cobra.Command {
 	cmd.AddCommand(
 		newNodesCmd(client, &asJSON),
 		newLogsCmd(client, &asJSON),
+		newRequestCmd(client, &asJSON),
 		newModelsCmd(client, &asJSON),
 		newReplicasCmd(client, &asJSON),
 		newExplainCmd(client, &asJSON),
@@ -140,6 +141,7 @@ func newCtlCmd() *cobra.Command {
 		newScaleCmd(client),
 		newRestartCmd(client),
 		newRetryCmd(client),
+		newUpgradeCmd(client),
 		newEngineCheckCmd(),
 		newToolCheckCmd(),
 		newWeightsCmd(client, &asJSON),

internal/cli/ctl_request.go

@@ -0,0 +1,45 @@
+package cli
+
+import (
+	"encoding/json"
+	"fmt"
+	"net/url"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+)
+
+func newRequestCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
+	return &cobra.Command{
+		Use:   "request ID",
+		Short: "What one request went through: replica, queue, first token, tokens",
+		Long: "Every response carries its id in X-Request-Id, and a client may send its\n" +
+			"own. Given one, this says which replica answered, how long admission held\n" +
+			"it, how long the first token took and what it cost — the answer to \"why\n" +
+			"was this slow\". The access log has the same fields under \"request\".",
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			var v adminapi.RequestView
+			if err := client().get(cmd.Context(), "/api/v1/requests/"+url.PathEscape(args[0]), &v); err != nil {
+				return err
+			}
+			if *asJSON {
+				return json.NewEncoder(cmd.OutOrStdout()).Encode(v)
+			}
+			model := v.Model
+			if v.ServedModel != "" && v.ServedModel != v.Model {
+				model += " (served by " + v.ServedModel + ")"
+			}
+			out := cmd.OutOrStdout()
+			fmt.Fprintf(out, "request   %s at %s\n", v.RequestID, v.At.UTC().Format("2006-01-02 15:04:05Z"))
+			fmt.Fprintf(out, "model     %s\n", model)
+			fmt.Fprintf(out, "replica   %s\n", or(v.Replica, "- (none answered)"))
+			fmt.Fprintf(out, "status    %d\n", v.Status)
+			fmt.Fprintf(out, "time      %d ms total: %d queued, %d to the first token\n", v.LatencyMs, v.QueueMs, v.TTFTMs)
+			fmt.Fprintf(out, "tokens    %d in, %d out\n", v.PromptTokens, v.CompletionTokens)
+			fmt.Fprintf(out, "who       key %s, project %s%s\n", v.KeyID, v.ProjectID, prefixed(", user ", v.UserID))
+			return nil
+		},
+	}
+}

internal/cli/ctl_upgrade.go

@@ -0,0 +1,197 @@
+package cli
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"os"
+	"os/exec"
+	"runtime"
+	"slices"
+	"strings"
+	"time"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/release"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/version"
+)
+
+func newUpgradeCmd(client func() *ctlClient) *cobra.Command {
+	var (
+		nodes     []string
+		skipCP    bool
+		releases  string
+		perNode   time.Duration
+		serverSvc string
+	)
+	c := &cobra.Command{
+		Use:   "upgrade VERSION",
+		Short: "Upgrade this control plane, then every node, one at a time",
+		Long: "Moves the cluster to a release: v0.10.4, or a candidate like v0.10.4-rc.2.\n\n" +
+			"This machine first — run it with sudo on the control plane, since it\n" +
+			"replaces /usr/local/bin/llmgw and restarts llmgw-server. Then each node, in\n" +
+			"turn: its agent fetches the release for its own platform, checks it\n" +
+			"against the release's checksums, swaps it in and restarts. Engines keep\n" +
+			"serving through all of it and are adopted again; no ssh into the nodes is\n" +
+			"needed.\n\n" +
+			"A node counts as done when it is back on the new version, ready, with as\n" +
+			"many ready replicas as before. One that is not within --timeout stops the\n" +
+			"upgrade there, so a bad release reaches one node rather than all of them.\n" +
+			"Nodes already on the version are skipped, so running it again resumes.",
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			v := args[0]
+			if !release.Tag.MatchString(v) {
+				return fmt.Errorf("%q is not a release version (v1.2.3 or v1.2.3-rc.4)", v)
+			}
+			out := cmd.OutOrStdout()
+			ctx := cmd.Context()
+			if !skipCP {
+				if err := upgradeControlPlane(ctx, out, client, v, releases, serverSvc); err != nil {
+					return err
+				}
+			}
+			return upgradeNodes(ctx, out, client, v, nodes, perNode)
+		},
+	}
+	c.Flags().StringSliceVar(&nodes, "nodes", nil, "only these nodes (default: all)")
+	c.Flags().BoolVar(&skipCP, "skip-control-plane", false, "leave this machine as it is and upgrade only nodes")
+	c.Flags().StringVar(&releases, "releases", release.DefaultBase, "where release binaries are fetched from, for this machine")
+	c.Flags().DurationVar(&perNode, "timeout", 3*time.Minute, "how long a node may take to come back")
+	c.Flags().StringVar(&serverSvc, "service", "llmgw-server", "this machine's systemd unit")
+	return c
+}
+
+func upgradeControlPlane(ctx context.Context, out io.Writer, client func() *ctlClient, v, releases, svc string) error {
+	if version.Version == v {
+		fmt.Fprintf(out, "this machine already runs %s\n", v)
+		return nil
+	}
+	if os.Geteuid() != 0 {
+		return errors.New("replacing this machine's binary needs root: run it with sudo, " +
+			"or pass --skip-control-plane to upgrade only the nodes")
+	}
+	self, err := os.Executable()
+	if err != nil {
+		return err
+	}
+	fmt.Fprintf(out, "== this machine: %s -> %s\n", version.Version, v)
+	backup, err := release.Install(ctx, nil, releases, v, runtime.GOOS, runtime.GOARCH, self)
+	if err != nil {
+		return err
+	}
+	fmt.Fprintf(out, "   installed, checksum verified; the old binary is %s\n", backup)
+	var before []adminapi.NodeView
+	_ = client().get(ctx, "/api/v1/nodes", &before)
+	if b, err := exec.CommandContext(ctx, "systemctl", "restart", svc).CombinedOutput(); err != nil { //nolint:gosec // fixed command
+		return fmt.Errorf("restarting %s: %v: %s", svc, err, strings.TrimSpace(string(b)))
+	}
+	fmt.Fprintf(out, "   %s restarted; waiting for its nodes to reconnect\n", svc)
+	// Back when the admin API answers and every node that was ready is ready
+	// again: agents reconnect within seconds of the control plane returning.
+	want := 0
+	for _, n := range before {
+		if n.State == "ready" {
+			want++
+		}
+	}
+	deadline := time.Now().Add(2 * time.Minute)
+	for {
+		var now []adminapi.NodeView
+		if err := client().get(ctx, "/api/v1/nodes", &now); err == nil {
+			ready := 0
+			for _, n := range now {
+				if n.State == "ready" {
+					ready++
+				}
+			}
+			if ready >= want {
+				fmt.Fprintf(out, "   back: %d of %d nodes ready\n", ready, len(now))
+				return nil
+			}
+		}
+		if time.Now().After(deadline) {
+			return fmt.Errorf("%s did not come back with its %d nodes within two minutes; "+
+				"see journalctl -u %s, and %s holds the previous binary", svc, want, svc, backup)
+		}
+		time.Sleep(2 * time.Second)
+	}
+}
+
+func upgradeNodes(ctx context.Context, out io.Writer, client func() *ctlClient, v string, only []string, perNode time.Duration) error {
+	var nodes []adminapi.NodeView
+	if err := client().get(ctx, "/api/v1/nodes", &nodes); err != nil {
+		return err
+	}
+	slices.SortFunc(nodes, func(a, b adminapi.NodeView) int { return strings.Compare(a.ID, b.ID) })
+	done, skipped := 0, 0
+	for _, n := range nodes {
+		if len(only) > 0 && !slices.Contains(only, n.ID) {
+			continue
+		}
+		if n.AgentVersion == v {
+			skipped++
+			continue
+		}
+		start := time.Now()
+		before := readyReplicasOn(ctx, client, n.ID)
+		fmt.Fprintf(out, "== %s: %s -> %s\n", n.ID, n.AgentVersion, v)
+		body, _ := json.Marshal(adminapi.UpgradeRequest{Version: v})
+		if err := client().post(ctx, "/api/v1/nodes/"+n.ID+"/upgrade", body, nil); err != nil {
+			return fmt.Errorf("%s: %w\nstopped here; `llmgw ctl logs %s` says more", n.ID, err, n.ID)
+		}
+		fmt.Fprintf(out, "   new binary in place; waiting for it to come back with its %d replica(s)\n", before)
+		deadline := time.Now().Add(perNode)
+		for {
+			var now []adminapi.NodeView
+			_ = client().get(ctx, "/api/v1/nodes", &now)
+			var cur adminapi.NodeView
+			for _, x := range now {
+				if x.ID == n.ID {
+					cur = x
+				}
+			}
+			after := readyReplicasOn(ctx, client, n.ID)
+			if cur.AgentVersion == v && cur.State == "ready" && after >= before {
+				fmt.Fprintf(out, "   done in %s: ready, replicas %d/%d\n",
+					time.Since(start).Round(time.Second), after, before)
+				done++
+				break
+			}
+			if time.Now().After(deadline) {
+				return fmt.Errorf("%s did not come back within %s (version %q, state %q, replicas %d/%d)\n"+
+					"stopped here, before the next node; `llmgw ctl logs %s` says why, and the node "+
+					"keeps its previous binary beside the new one", n.ID, perNode, cur.AgentVersion,
+					cur.State, after, before, n.ID)
+			}
+			time.Sleep(2 * time.Second)
+		}
+	}
+	fmt.Fprintf(out, "%d node(s) upgraded to %s, %d already on it\n", done, v, skipped)
+	return nil
+}
+
+// readyReplicasOn counts the ready replicas with a shard on a node.
+func readyReplicasOn(ctx context.Context, client func() *ctlClient, node string) int {
+	var reps []adminapi.ReplicaView
+	if err := client().get(ctx, "/api/v1/replicas", &reps); err != nil {
+		return 0
+	}
+	n := 0
+	for _, r := range reps {
+		if r.State != "ready" {
+			continue
+		}
+		for _, s := range r.Shards {
+			if s.Node == node {
+				n++
+				break
+			}
+		}
+	}
+	return n
+}

internal/cli/ctl_upgrade_test.go

@@ -0,0 +1,88 @@
+package cli
+
+import (
+	"bytes"
+	"encoding/json"
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"sync"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+)
+
+// fakeFleet is an admin API whose nodes come back on the version they are told
+// to upgrade to — except those listed in stuck.
+type fakeFleet struct {
+	mu       sync.Mutex
+	versions map[string]string
+	stuck    map[string]bool
+	told     []string
+}
+
+func (f *fakeFleet) server(t *testing.T) *httptest.Server {
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		f.mu.Lock()
+		defer f.mu.Unlock()
+		switch {
+		case r.URL.Path == "/api/v1/nodes":
+			var out []adminapi.NodeView
+			for id, v := range f.versions {
+				out = append(out, adminapi.NodeView{ID: id, State: "ready", AgentVersion: v})
+			}
+			_ = json.NewEncoder(w).Encode(out)
+		case r.URL.Path == "/api/v1/replicas":
+			var out []adminapi.ReplicaView
+			for id := range f.versions {
+				out = append(out, adminapi.ReplicaView{ID: "r-" + id, State: "ready",
+					Shards: []adminapi.ShardView{{Node: id}}})
+			}
+			_ = json.NewEncoder(w).Encode(out)
+		case strings.HasSuffix(r.URL.Path, "/upgrade"):
+			id := strings.TrimSuffix(strings.TrimPrefix(r.URL.Path, "/api/v1/nodes/"), "/upgrade")
+			var req adminapi.UpgradeRequest
+			_ = json.NewDecoder(r.Body).Decode(&req)
+			f.told = append(f.told, id)
+			if !f.stuck[id] {
+				f.versions[id] = req.Version
+			}
+			_, _ = w.Write([]byte(`{}`))
+		default:
+			http.NotFound(w, r)
+		}
+	}))
+	t.Cleanup(srv.Close)
+	return srv
+}
+
+func TestAnUpgradeGoesNodeByNodeAndSkipsTheDone(t *testing.T) {
+	f := &fakeFleet{versions: map[string]string{"a": "v1.0.0", "b": "v1.0.1", "c": "v1.0.0"}}
+	srv := f.server(t)
+	var out bytes.Buffer
+	err := upgradeNodes(t.Context(), &out, func() *ctlClient { return newCtlClient(srv.URL) }, "v1.0.1", nil, 5*time.Second)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if strings.Join(f.told, ",") != "a,c" {
+		t.Errorf("told %v; b was already on the version", f.told)
+	}
+	if !strings.Contains(out.String(), "2 node(s) upgraded to v1.0.1, 1 already on it") {
+		t.Errorf("output:\n%s", out.String())
+	}
+}
+
+func TestAnUpgradeStopsAtTheFirstNodeThatDoesNotComeBack(t *testing.T) {
+	// A bad release reaches one node, not all of them.
+	f := &fakeFleet{versions: map[string]string{"a": "v1.0.0", "b": "v1.0.0"}, stuck: map[string]bool{"a": true}}
+	srv := f.server(t)
+	var out bytes.Buffer
+	err := upgradeNodes(t.Context(), &out, func() *ctlClient { return newCtlClient(srv.URL) }, "v1.0.1", nil, 3*time.Second)
+	if err == nil || !strings.Contains(err.Error(), "a did not come back") {
+		t.Fatalf("err %v", err)
+	}
+	if len(f.told) != 1 {
+		t.Errorf("went on to %v after a node failed to come back", f.told)
+	}
+}

internal/cli/mirror.go

@@ -8,6 +8,7 @@ import (
 	"net/http"
 	"net/netip"
 	"os"
+	"slices"
 	"strings"
 	"time"
 
@@ -232,7 +233,7 @@ func newMirrorRemoveCmd(dir *string) *cobra.Command {
 func newMirrorServeCmd(dir *string) *cobra.Command {
 	var listen, apiListen, base string
 	var tlsc tlsconf.Config
-	var allow []string
+	var allow, clients []string
 	var allowPrivate bool
 
 	c := &cobra.Command{
@@ -259,6 +260,9 @@ func newMirrorServeCmd(dir *string) *cobra.Command {
 			if err := tlsc.Validate(); err != nil {
 				return err
 			}
+			if len(clients) > 0 && !tlsc.Enabled() {
+				return fmt.Errorf("--api-client names a certificate, and needs --tls-ca, --tls-cert and --tls-key")
+			}
 
 			out := cmd.OutOrStdout()
 			log := slog.Default()
@@ -282,7 +286,7 @@ func newMirrorServeCmd(dir *string) *cobra.Command {
 			}
 			admin := &http.Server{
 				Addr:              apiListen,
-				Handler:           apiGate(tlsc.Enabled(), allowed, api.Handler()),
+				Handler:           onlyClients(clients, apiGate(tlsc.Enabled(), allowed, api.Handler())),
 				ReadHeaderTimeout: 10 * time.Second,
 			}
 
@@ -294,6 +298,12 @@ func newMirrorServeCmd(dir *string) *cobra.Command {
 			switch {
 			case tlsc.Enabled():
 				fmt.Fprintf(out, "management API on %s, mutual TLS against %s\n", apiListen, tlsc.CA)
+				if len(clients) == 0 {
+					// The same root signs every node's certificate, so without a
+					// name any node could delete models here.
+					log.Warn("the management API takes any certificate from " + tlsc.CA +
+						", every node's included; name the control plane's with --api-client")
+				}
 			case len(allowed) > 0:
 				fmt.Fprintf(out, "management API on %s, for %s and this machine only\n", apiListen, strings.Join(allow, ", "))
 			default:
@@ -317,6 +327,8 @@ func newMirrorServeCmd(dir *string) *cobra.Command {
 		"the address nodes use to reach this mirror, as it should appear in catalog sources")
 	c.Flags().StringArrayVar(&allow, "api-allow", nil,
 		"an address or network the management API answers, besides this machine; repeatable (default: this machine only)")
+	c.Flags().StringArrayVar(&clients, "api-client", nil,
+		"with mutual TLS, the common name of a certificate the management API answers; repeatable (default: any from --tls-ca)")
 	c.Flags().BoolVar(&allowPrivate, "allow-private-sources", false,
 		"let downloads reach loopback, private and link-local addresses")
 	c.Flags().StringVar(&tlsc.CA, "tls-ca", "", "CA bundle that management clients must present a certificate from")
@@ -452,6 +464,23 @@ func apiGate(mtls bool, allowed []netip.Prefix, next http.Handler) http.Handler
 	return onlyFrom(allowed, next)
 }
 
+// onlyClients answers only certificates with one of these common names, when
+// any are given. The handshake has already checked the certificate against
+// the root; this narrows which of the root's certificates may manage.
+func onlyClients(names []string, next http.Handler) http.Handler {
+	if len(names) == 0 {
+		return next
+	}
+	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		if r.TLS == nil || len(r.TLS.VerifiedChains) == 0 ||
+			!slices.Contains(names, r.TLS.VerifiedChains[0][0].Subject.CommonName) {
+			http.Error(w, "this mirror is managed by its control plane only", http.StatusForbidden)
+			return
+		}
+		next.ServeHTTP(w, r)
+	})
+}
+
 // onlyFrom answers only this machine and the addresses allowed: the management
 // API can delete models and start downloads, and a mirror on its own machine
 // is reached over the network by one control plane and no one else.

internal/cli/mirror_gate_test.go

@@ -1,6 +1,9 @@
 package cli
 
 import (
+	"crypto/tls"
+	"crypto/x509"
+	"crypto/x509/pkix"
 	"net/http"
 	"net/http/httptest"
 	"net/netip"
@@ -37,3 +40,27 @@ func TestTheMirrorAPIAnswersOnlyThisMachineUnlessTold(t *testing.T) {
 		t.Errorf("mTLS without an allowlist refused a client: %d", got)
 	}
 }
+
+func TestOnlyANamedCertificateManagesTheMirror(t *testing.T) {
+	ok := http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {})
+	with := func(cn string) *http.Request {
+		r := httptest.NewRequest("GET", "/", nil)
+		if cn != "" {
+			r.TLS = &tls.ConnectionState{VerifiedChains: [][]*x509.Certificate{{{Subject: pkix.Name{CommonName: cn}}}}}
+		}
+		return r
+	}
+	h := onlyClients([]string{"control-plane"}, ok)
+	for cn, want := range map[string]int{"control-plane": 200, "node01": 403, "": 403} {
+		w := httptest.NewRecorder()
+		h.ServeHTTP(w, with(cn))
+		if w.Code != want {
+			t.Errorf("%q: %d, expected %d", cn, w.Code, want)
+		}
+	}
+	w := httptest.NewRecorder()
+	onlyClients(nil, ok).ServeHTTP(w, with("node01"))
+	if w.Code != 200 {
+		t.Error("no names given narrowed it anyway")
+	}
+}

internal/cli/model.go

@@ -3,6 +3,7 @@ package cli
 import (
 	"encoding/json"
 	"fmt"
+	"os"
 	"os/user"
 	"time"
 
@@ -28,7 +29,8 @@ func newModelCmd(client func() *ctlClient) *cobra.Command {
 		Use:   "model",
 		Short: "Add or remove models while the cluster is running",
 	}
-	c.AddCommand(newModelAddCmd(client), newModelRemoveCmd(client), newModelPlanCmd(client))
+	c.AddCommand(newModelAddCmd(client), newModelRemoveCmd(client), newModelPlanCmd(client),
+		newModelServeCmd(client))
 	return c
 }
 
@@ -126,6 +128,10 @@ func newModelRemoveCmd(client func() *ctlClient) *cobra.Command {
 
 // actor names who made a change, for the audit trail on a stored model.
 func actor() string {
+	// Under sudo, the person who ran it rather than root.
+	if u := os.Getenv("SUDO_USER"); u != "" && os.Geteuid() == 0 {
+		return u
+	}
 	if u, err := user.Current(); err == nil && u.Username != "" {
 		return u.Username
 	}

internal/cli/model_serve.go

@@ -0,0 +1,80 @@
+package cli
+
+import (
+	"encoding/json"
+	"errors"
+	"fmt"
+	"time"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
+)
+
+func newModelServeCmd(client func() *ctlClient) *cobra.Command {
+	var (
+		req      modelserve.Request
+		onDemand bool
+	)
+	c := &cobra.Command{
+		Use:   "serve OWNER/REPO",
+		Short: "Take a model from the registry to a replica that answers, in one command",
+		Long: "Picks the GGUF file (--quant, Q4_K_M by default, or --file), has the mirror\n" +
+			"fetch it and checks what arrived against the registry's checksum, checks\n" +
+			"that the nodes' llama.cpp can load it and sizes its memory from its own\n" +
+			"header, adds it to the catalog, waits for a replica, and asks it to say\n" +
+			"hello. Each step is shown as it happens; the first that fails stops it.\n\n" +
+			"  llmgw ctl model serve bartowski/Qwen2.5-3B-Instruct-GGUF\n" +
+			"  llmgw ctl model serve Qwen/Qwen2.5-7B-Instruct-GGUF --quant Q5_K_M --name qwen-7b\n" +
+			"  llmgw ctl model serve owner/big-GGUF --on-demand",
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			req.Repo = args[0]
+			if onDemand {
+				req.Replicas = 0
+			}
+			body, _ := json.Marshal(req)
+			var job modelserve.Job
+			if err := client().post(cmd.Context(), "/api/v1/models/serve", body, &job); err != nil {
+				return err
+			}
+			out := cmd.OutOrStdout()
+			fmt.Fprintf(out, "job %s\n", job.ID)
+			shown := map[string]string{}
+			for {
+				for _, s := range job.Steps {
+					line := s.State + " " + s.Detail
+					if shown[s.Name] == line {
+						continue
+					}
+					shown[s.Name] = line
+					mark := map[string]string{modelserve.Running: "..", modelserve.Done: "ok",
+						modelserve.Failed: "!!"}[s.State]
+					fmt.Fprintf(out, "%-3s %-11s %s\n", mark, s.Name, s.Detail)
+				}
+				switch job.State {
+				case modelserve.Done:
+					fmt.Fprintf(out, "\n%s is served.\n", job.Model)
+					return nil
+				case modelserve.Failed:
+					return errors.New(job.Error)
+				}
+				time.Sleep(2 * time.Second)
+				if err := client().get(cmd.Context(), "/api/v1/jobs/"+job.ID, &job); err != nil {
+					return err
+				}
+			}
+		},
+	}
+	f := c.Flags()
+	f.StringVar(&req.Quant, "quant", "", "the quantisation to serve (default Q4_K_M)")
+	f.StringVar(&req.File, "file", "", "the file to serve, instead of choosing by --quant")
+	f.StringVar(&req.Name, "name", "", "the catalog name (default: from the repository)")
+	f.StringSliceVar(&req.Aliases, "alias", nil, "other names clients may use; repeatable")
+	f.StringVar(&req.Description, "description", "", "what it is for, shown to clients in /v1/models")
+	f.IntVar(&req.Replicas, "replicas", 1, "how many replicas to run")
+	f.BoolVar(&onDemand, "on-demand", false, "run none until first asked for")
+	f.IntVar(&req.CtxPool, "ctx-pool", 16384, "llama.cpp's --ctx-size, shared by its parallel slots")
+	f.IntVar(&req.ContextLength, "context-length", 4096, "what one request may use")
+	return c
+}

internal/cli/server.go

@@ -5,6 +5,7 @@ import (
 	"crypto/tls"
 	"errors"
 	"fmt"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/sourcecheck"
 	"github.com/prometheus/client_golang/prometheus"
@@ -34,6 +35,7 @@ import (
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/obs"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/promq"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/router"
 	storepkg "gitea.mynodes.xyz/ilker/llmgw/internal/store"
@@ -89,6 +91,21 @@ func newServerCmd() *cobra.Command {
 					"or give --endpoint for a router-only run")
 			}
 
+			// Before the router, so its tracer is the configured one.
+			shutdownTracing, err := obs.SetupTracing(cmd.Context(), obs.TracingOptions{
+				Endpoint: cfg.Tracing.Endpoint, SampleRatio: cfg.Tracing.SampleRatio,
+				Service: "llmgw-server", Version: version.Version,
+			})
+			if err != nil {
+				return err
+			}
+			defer func() {
+				// Bounded: a collector that is gone must not hold up a shutdown.
+				sctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+				defer cancel()
+				_ = shutdownTracing(sctx)
+			}()
+
 			rt, err := router.New(cfg, log)
 			if err != nil {
 				return err
@@ -171,8 +188,17 @@ func newServerCmd() *cobra.Command {
 			if err != nil {
 				return err
 			}
+			// Kept past this block: serving a model from the registry needs
+			// both the admin API and the mirror, and the mirror is set up
+			// below.
+			var (
+				adminAPI   *adminapi.API
+				catalogSvc *modelsvc.Service
+			)
 			if admin != nil {
 				api := adminapi.New(admin)
+				adminAPI = api
+				api.SetReleases(cfg.Releases)
 				// Catalog changes need somewhere to persist, so this is wired
 				// only when there is a store. Without one, the endpoints say so
 				// rather than accepting a model that vanishes on restart.
@@ -189,6 +215,7 @@ func newServerCmd() *cobra.Command {
 						admin.ReplaceCatalog(ctx, built)
 					}
 					api.SetModels(svc)
+					catalogSvc = svc
 					// And the mirror page, so a model that has just been
 					// fetched can be served without writing YAML by hand.
 					rt.SetCatalogAdmin(svc)
@@ -258,7 +285,16 @@ func newServerCmd() *cobra.Command {
 				// Browsing the registry is what turns "paste a resolve URL"
 				// into "find the model". It needs a way out to the internet
 				// from this machine, which the mirror's own fetching does not.
-				rt.SetRegistry(hf.New(cfg.Mirror.Registry, cfg.Mirror.Timeout))
+				registry := hf.New(cfg.Mirror.Registry, cfg.Mirror.Timeout)
+				rt.SetRegistry(registry)
+				// One command from a registry repository to a replica that
+				// answers: `llmgw ctl model serve`.
+				if adminAPI != nil && catalogSvc != nil {
+					adminAPI.SetServe(modelserve.New(modelserve.Deps{
+						Registry: registry, Mirror: mc, Catalog: catalogSvc, Fleet: admin,
+						LlamaVersion: cfg.WantedEngines()["llamacpp"], Log: log,
+					}))
+				}
 				log.Info("model mirror configured",
 					"api", cfg.Mirror.API, "mtls", cfg.Mirror.TLS.Enabled())
 			}
@@ -546,6 +582,19 @@ func (r *dispatcherRef) FabricTest(ctx context.Context, nodeID string, req *llmg
 	return ft.FabricTest(ctx, nodeID, req)
 }
 
+// Upgrade forwards a node upgrade, as Logs does.
+func (r *dispatcherRef) Upgrade(ctx context.Context, nodeID, version, releases string) error {
+	d, err := r.get()
+	if err != nil {
+		return err
+	}
+	u, ok := d.(control.Upgrader)
+	if !ok {
+		return errors.New("this dispatcher cannot upgrade nodes")
+	}
+	return u.Upgrade(ctx, nodeID, version, releases)
+}
+
 func (r *dispatcherRef) StartShard(ctx context.Context, cmd reconciler.ShardCommand) error {
 	d, err := r.get()
 	if err != nil {

internal/cli/tui_init.go

@@ -389,17 +389,22 @@ func (in *serverInit) runInitTUI(ctx context.Context, out io.Writer) error {
 		return fmt.Errorf("stopped; run init again to finish what was started")
 	}
 
-	// The whole run is kept, root's only, since it holds the password. On a
-	// failure it is printed too, for the lines the screen had no room for;
-	// otherwise the screen's last frame says it, and only what must outlive
-	// the screen is printed again: the password, and what to do next.
+	// The whole run is kept, root's only, with the password taken out: it is
+	// shown once, on the screen, and a file on disk would outlive that. On a
+	// failure the run is printed too, for the lines the screen had no room
+	// for; otherwise the screen's last frame says it, and only what must
+	// outlive the screen is printed again: the password, and what to do next.
 	logMu.Lock()
 	defer logMu.Unlock()
 	whole := strings.Join(log, "\n") + "\n"
+	kept := whole
+	if fm.password != "" {
+		kept = strings.ReplaceAll(kept, fm.password, "[password shown once, not kept]")
+	}
 	saved := ""
 	if in.stateDir != "" {
 		p := filepath.Join(in.stateDir, "init.log")
-		if err := os.WriteFile(p, []byte(whole), 0o600); err == nil {
+		if err := os.WriteFile(p, []byte(kept), 0o600); err == nil {
 			saved = p
 		}
 	}

internal/config/config.go

@@ -67,6 +67,10 @@ type Server struct {
 
 	// Monitoring is where the dashboard reads its history from.
 	Monitoring Monitoring `koanf:"monitoring"`
+	// Tracing sends a trace of each inference request to an OpenTelemetry
+	// collector. Off unless an endpoint is set here or in
+	// OTEL_EXPORTER_OTLP_ENDPOINT.
+	Tracing Tracing `koanf:"tracing"`
 
 	// Mirror points the control plane at a model mirror it can administer.
 	// Empty means there is none, and /admin/mirror stays unmounted.
@@ -86,6 +90,11 @@ type Server struct {
 	// Empty means /var/backups/llmgw-server.
 	BackupDir string `koanf:"backup_dir"`
 
+	// Releases is where `llmgw ctl upgrade` fetches release binaries from,
+	// for this machine and — through their agents — every node. Empty means
+	// the project's own releases.
+	Releases string `koanf:"releases"`
+
 	Router  Router          `koanf:"router"`
 	Control Control         `koanf:"control"`
 	Models  []catalog.Model `koanf:"models"`
@@ -115,6 +124,21 @@ type Monitoring struct {
 	Timeout time.Duration `koanf:"timeout"`
 }
 
+// Tracing is where request traces go, over OTLP/HTTP.
+//
+// One span per inference request, with admission, scheduling and each attempt
+// at an engine beneath it, and the trace carried on to the engine in a
+// traceparent header. The request's X-Request-Id is on the span, so a trace
+// and `llmgw ctl request` name the same request.
+type Tracing struct {
+	// Endpoint is the collector, for example http://otel.internal:4318.
+	// Empty falls back to OTEL_EXPORTER_OTLP_ENDPOINT, and to no tracing.
+	Endpoint string `koanf:"endpoint"`
+	// SampleRatio is the share of requests traced, 0 to 1, when the caller
+	// did not decide it with a sampled traceparent of its own. Zero takes 1.
+	SampleRatio float64 `koanf:"sample_ratio"`
+}
+
 // Mirror is a model mirror this control plane administers over its API.
 //
 // The mirror is a separate machine with a large disk, not part of the gateway,

internal/control/adminapi/actor_test.go

@@ -0,0 +1,24 @@
+package adminapi
+
+import (
+	"net/http/httptest"
+	"strings"
+	"testing"
+)
+
+func TestAClaimedActorIsOneShortName(t *testing.T) {
+	for claimed, want := range map[string]string{
+		"ilker":                    "ilker",
+		"":                         "unknown",
+		"ilker\nlevel=ERROR msg=x": "ilkerlevelERRORmsgx",
+		strings.Repeat("a", 500):   strings.Repeat("a", 64),
+		"ops@example.org":          "ops@example.org",
+		"\x1b[31m":                 "31m",
+	} {
+		r := httptest.NewRequest("POST", "/", nil)
+		r.Header.Set("X-Llmgw-Actor", claimed)
+		if got := actorOf(r); got != want {
+			t.Errorf("%q: %q, expected %q", claimed, got, want)
+		}
+	}
+}

internal/control/adminapi/api.go

@@ -10,11 +10,14 @@ import (
 	"context"
 	"encoding/json"
 	"fmt"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/release"
 	"io"
 	"net/http"
 	"slices"
 	"strings"
 	"time"
+	"unicode"
 
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/bench"
@@ -30,6 +33,11 @@ import (
 
 // API serves the management endpoints.
 type API struct {
+	// releases is where release binaries are fetched from, for upgrades.
+	releases string
+	// serve runs registry-to-replica jobs; nil without a mirror and a store.
+	serve *modelserve.Server
+
 	models   Models
 	capacity Capacity
 	usage    Usage
@@ -74,10 +82,14 @@ func (a *API) Register(mux *http.ServeMux) {
 	mux.HandleFunc("POST /api/v1/weights/collect", a.collectWeights)
 	mux.HandleFunc("GET /api/v1/explain-placement", a.explainPlacement)
 	mux.HandleFunc("POST /api/v1/nodes/{id}/state", a.setNodeState)
+	mux.HandleFunc("POST /api/v1/nodes/{id}/upgrade", a.upgradeNode)
 	mux.HandleFunc("DELETE /api/v1/nodes/{id}", a.removeNode)
 	mux.HandleFunc("POST /api/v1/fabric-test", a.fabricTest)
 	mux.HandleFunc("GET /api/v1/nodes/{id}/logs", a.nodeLogs)
 	mux.HandleFunc("POST /api/v1/restart", a.restart)
+	mux.HandleFunc("POST /api/v1/models/serve", a.serveModel)
+	mux.HandleFunc("GET /api/v1/jobs/{id}", a.job)
+	mux.HandleFunc("GET /api/v1/requests/{id}", a.request)
 	mux.HandleFunc("POST /api/v1/models/{name}/retry", a.retryModel)
 }
 
@@ -674,7 +686,7 @@ func (a *API) putCapacity(w http.ResponseWriter, r *http.Request) {
 	}
 	if err := a.capacity.PutCapacity(r.Context(), store.StoredCapacity{
 		Model: c.Model, Profile: c.Profile, Measurement: body,
-		MeasuredAt: c.MeasuredAt, MeasuredBy: r.Header.Get("X-Llmgw-Actor"),
+		MeasuredAt: c.MeasuredAt, MeasuredBy: actorOf(r),
 	}); err != nil {
 		writeError(w, http.StatusInternalServerError, err.Error())
 		return
@@ -725,7 +737,7 @@ func (a *API) addModel(w http.ResponseWriter, r *http.Request) {
 		writeError(w, http.StatusBadRequest, "the model definition is not valid JSON: "+err.Error())
 		return
 	}
-	if err := a.models.Add(r.Context(), m, r.Header.Get("X-Llmgw-Actor")); err != nil {
+	if err := a.models.Add(r.Context(), m, actorOf(r)); err != nil {
 		// Every failure here is the definition being wrong, which is the
 		// caller's to fix — a collision, a missing fallback, a profile that
 		// cannot be placed. None of them are server faults.
@@ -774,7 +786,7 @@ func (a *API) scaleModel(w http.ResponseWriter, r *http.Request) {
 	name := r.PathValue("name")
 	m, err := a.models.Scale(r.Context(), name,
 		catalog.Scaling{Min: req.Min, Max: req.Max, IdleTimeout: req.IdleTimeout},
-		r.Header.Get("X-Llmgw-Actor"))
+		actorOf(r))
 	if err != nil {
 		// Like addModel: every refusal here is the request being wrong — a
 		// model declared in configuration, one that was never added, a max
@@ -1115,3 +1127,143 @@ func (a *API) retryModel(w http.ResponseWriter, r *http.Request) {
 	was := a.ctl.Reconciler().Retry(name)
 	writeJSON(w, http.StatusOK, map[string]bool{"was_parked": was})
 }
+
+// requestFinder looks a request up by the id it was given. Asked of the usage
+// source by type assertion, so one without it answers 501 rather than every
+// source having to grow the method.
+type requestFinder interface {
+	UsageByRequest(ctx context.Context, requestID string) (store.Usage, bool, error)
+}
+
+// RequestView is what one request went through.
+type RequestView struct {
+	RequestID        string    `json:"request_id"`
+	At               time.Time `json:"at"`
+	KeyID            string    `json:"key_id"`
+	UserID           string    `json:"user_id,omitempty"`
+	ProjectID        string    `json:"project_id"`
+	Model            string    `json:"model"`
+	ServedModel      string    `json:"served_model"`
+	Replica          string    `json:"replica,omitempty"`
+	Status           int       `json:"status"`
+	LatencyMs        int64     `json:"latency_ms"`
+	QueueMs          int64     `json:"queue_ms"`
+	TTFTMs           int64     `json:"ttft_ms"`
+	PromptTokens     int64     `json:"prompt_tokens"`
+	CompletionTokens int64     `json:"completion_tokens"`
+	CostMicros       int64     `json:"cost_micros"`
+}
+
+func (a *API) request(w http.ResponseWriter, r *http.Request) {
+	f, ok := a.usage.(requestFinder)
+	if !ok {
+		writeError(w, http.StatusNotImplemented, "this control plane keeps no usage records")
+		return
+	}
+	u, found, err := f.UsageByRequest(r.Context(), r.PathValue("id"))
+	switch {
+	case err != nil:
+		writeError(w, http.StatusInternalServerError, err.Error())
+		return
+	case !found:
+		writeError(w, http.StatusNotFound, "no request with that id; records are written a few seconds after a request ends")
+		return
+	}
+	writeJSON(w, http.StatusOK, RequestView{
+		RequestID: u.RequestID, At: u.At, KeyID: u.KeyID, UserID: u.UserID, ProjectID: u.ProjectID,
+		Model: u.Model, ServedModel: u.ServedModel, Replica: u.Replica, Status: u.Status,
+		LatencyMs: u.LatencyMs, QueueMs: u.QueueMs, TTFTMs: u.TTFTMs,
+		PromptTokens: u.PromptTokens, CompletionTokens: u.CompletionTokens, CostMicros: u.CostMicros,
+	})
+}
+
+// SetReleases sets where upgrades fetch release binaries from.
+func (a *API) SetReleases(base string) { a.releases = base }
+
+// UpgradeRequest names the release a node should run.
+type UpgradeRequest struct {
+	Version string `json:"version"`
+}
+
+func (a *API) upgradeNode(w http.ResponseWriter, r *http.Request) {
+	var req UpgradeRequest
+	if err := json.NewDecoder(io.LimitReader(r.Body, 1<<12)).Decode(&req); err != nil {
+		writeError(w, http.StatusBadRequest, "the upgrade request is not valid JSON: "+err.Error())
+		return
+	}
+	if !release.Tag.MatchString(req.Version) {
+		writeError(w, http.StatusBadRequest, req.Version+" is not a release version")
+		return
+	}
+	u := a.ctl.Upgrader()
+	if u == nil {
+		writeError(w, http.StatusNotImplemented, "this control plane cannot upgrade nodes")
+		return
+	}
+	base := a.releases
+	if base == "" {
+		base = release.DefaultBase
+	}
+	if err := u.Upgrade(r.Context(), r.PathValue("id"), req.Version, base); err != nil {
+		writeError(w, http.StatusBadGateway, err.Error())
+		return
+	}
+	writeJSON(w, http.StatusOK, map[string]string{"node": r.PathValue("id"), "version": req.Version})
+}
+
+// SetServe supplies what serves a model from the registry.
+func (a *API) SetServe(s *modelserve.Server) { a.serve = s }
+
+func (a *API) serveModel(w http.ResponseWriter, r *http.Request) {
+	if a.serve == nil {
+		writeError(w, http.StatusNotImplemented,
+			"serving from the registry needs a mirror and a store, and this control plane lacks one")
+		return
+	}
+	var req modelserve.Request
+	if err := json.NewDecoder(io.LimitReader(r.Body, 1<<16)).Decode(&req); err != nil {
+		writeError(w, http.StatusBadRequest, "the request is not valid JSON: "+err.Error())
+		return
+	}
+	req.By = actorOf(r)
+	job, err := a.serve.Start(req)
+	if err != nil {
+		writeError(w, http.StatusBadRequest, err.Error())
+		return
+	}
+	writeJSON(w, http.StatusAccepted, job)
+}
+
+func (a *API) job(w http.ResponseWriter, r *http.Request) {
+	if a.serve == nil {
+		writeError(w, http.StatusNotFound, "no such job")
+		return
+	}
+	job, ok := a.serve.Get(r.PathValue("id"))
+	if !ok {
+		writeError(w, http.StatusNotFound, "no such job; jobs are kept only until the control plane restarts")
+		return
+	}
+	writeJSON(w, http.StatusOK, job)
+}
+
+// actorOf is who a change is recorded as made by: what the caller says, since
+// the admin API answers only this machine and trusts whoever reaches it. Kept
+// to a short printable name, so a claim cannot pass for more than one — no
+// line breaks to forge a second log line, no length to bury the rest.
+func actorOf(r *http.Request) string {
+	claimed := r.Header.Get("X-Llmgw-Actor")
+	var b strings.Builder
+	for _, c := range claimed {
+		if b.Len() >= 64 {
+			break
+		}
+		if unicode.IsLetter(c) || unicode.IsDigit(c) || strings.ContainsRune("._-@", c) {
+			b.WriteRune(c)
+		}
+	}
+	if b.Len() == 0 {
+		return "unknown"
+	}
+	return b.String()
+}

internal/control/agentsvc/heartbeat_test.go

@@ -264,3 +264,22 @@ func TestAGPUFallingOutOfItsDomainTakesTheNodeOut(t *testing.T) {
 		t.Errorf("imex = %+v", n.IMEX)
 	}
 }
+
+func TestAnUpgradeWaitsForTheAgentsAnswer(t *testing.T) {
+	s := newTestService()
+	c := &conn{nodeID: "n1", send: make(chan *llmgwv1.ControlMessage, 4), closed: make(chan struct{})}
+	s.conns["n1"] = c
+	done := make(chan error, 1)
+	go func() { done <- s.Upgrade(t.Context(), "n1", "v1.2.3", "https://r") }()
+
+	req := (<-c.send).GetUpgrade()
+	if req.GetVersion() != "v1.2.3" || req.GetReleases() != "https://r" {
+		t.Fatalf("sent %v", req)
+	}
+	s.handle("n1", &llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_Upgrade{Upgrade: &llmgwv1.UpgradeReply{
+		RequestId: req.GetRequestId(), Version: "v1.2.3", Error: "checksum mismatch",
+	}}})
+	if err := <-done; err == nil || err.Error() != "checksum mismatch" {
+		t.Errorf("the agent's refusal was not passed on: %v", err)
+	}
+}

internal/control/agentsvc/service.go

@@ -60,6 +60,8 @@ type Service struct {
 	logMu    sync.Mutex
 	logWaits map[string]chan *llmgwv1.LogsReply
 	logSeq   uint64
+	// upgradeWaits are the upgrade requests awaiting a reply; guarded by logMu.
+	upgradeWaits map[string]chan *llmgwv1.UpgradeReply
 	// scanWaits are the fabric scans awaiting a reply, by request id.
 	scanWaits map[string]chan *llmgwv1.FabricScanReply
 	testWaits map[string]chan *llmgwv1.FabricTestReply
@@ -309,6 +311,15 @@ func (s *Service) handle(nodeID string, msg *llmgwv1.AgentMessage) {
 			ch <- m.FabricScan
 		}
 
+	case *llmgwv1.AgentMessage_Upgrade:
+		s.logMu.Lock()
+		ch, ok := s.upgradeWaits[m.Upgrade.GetRequestId()]
+		delete(s.upgradeWaits, m.Upgrade.GetRequestId())
+		s.logMu.Unlock()
+		if ok {
+			ch <- m.Upgrade
+		}
+
 	case *llmgwv1.AgentMessage_Logs:
 		s.logMu.Lock()
 		ch, ok := s.logWaits[m.Logs.GetRequestId()]
@@ -842,3 +853,44 @@ func installable(reg *llmgwv1.Register) []string {
 // The admin API finds the bandwidth test by asserting this; a signature that
 // drifted would make it quietly nil.
 var _ fabrictest.Asker = (*Service)(nil)
+
+// Upgrade tells a node's agent to replace its binary with a release and
+// restart, and returns once the agent says the new binary is in place — or
+// why it is not. The agent then exits and comes back on the new version; the
+// caller watches for that in the inventory.
+func (s *Service) Upgrade(ctx context.Context, nodeID, version, releases string) error {
+	s.logMu.Lock()
+	if s.upgradeWaits == nil {
+		s.upgradeWaits = map[string]chan *llmgwv1.UpgradeReply{}
+	}
+	s.logSeq++
+	id := fmt.Sprintf("upgrade-%d", s.logSeq)
+	ch := make(chan *llmgwv1.UpgradeReply, 1)
+	s.upgradeWaits[id] = ch
+	s.logMu.Unlock()
+	defer func() {
+		s.logMu.Lock()
+		delete(s.upgradeWaits, id)
+		s.logMu.Unlock()
+	}()
+
+	if err := s.send(nodeID, &llmgwv1.ControlMessage{Msg: &llmgwv1.ControlMessage_Upgrade{
+		Upgrade: &llmgwv1.UpgradeAgent{RequestId: id, Version: version, Releases: releases},
+	}}); err != nil {
+		return err
+	}
+	s.log.Info("told a node to upgrade", "node", nodeID, "version", version)
+	// A download of some tens of megabytes on a slow link, and no longer: an
+	// agent too old to know this message never answers.
+	ctx, cancel := context.WithTimeout(ctx, 5*time.Minute)
+	defer cancel()
+	select {
+	case r := <-ch:
+		if r.GetError() != "" {
+			return errors.New(r.GetError())
+		}
+		return nil
+	case <-ctx.Done():
+		return fmt.Errorf("node %s did not answer the upgrade; an agent older than v0.10.4 cannot upgrade itself", nodeID)
+	}
+}

internal/control/control.go

@@ -134,6 +134,16 @@ func (c *Controller) collectUsage() {
 		return
 	}
 	c.rec.RecordUsage(c.usage.LastUsed())
+	// And what the replicas are doing, when the source can say: what adds
+	// and removes replicas between a model's min and max.
+	if l, ok := c.usage.(LoadSource); ok {
+		c.rec.RecordLoad(l.ModelLoad())
+	}
+}
+
+// LoadSource reports each model's replicas' load, keyed by catalog name.
+type LoadSource interface {
+	ModelLoad() map[string]reconciler.Load
 }
 
 // ReplaceCatalog puts a new catalog in force, without a restart.
@@ -211,6 +221,17 @@ func (c *Controller) WeightRemover() WeightRemover {
 	return r
 }
 
+// Upgrader can tell a node's agent to upgrade itself.
+type Upgrader interface {
+	Upgrade(ctx context.Context, nodeID, version, releases string) error
+}
+
+// Upgrader returns the dispatcher's upgrade capability, or nil.
+func (c *Controller) Upgrader() Upgrader {
+	u, _ := c.disp.(Upgrader)
+	return u
+}
+
 // LogReader can ask a node for the recent lines of one of its logs.
 type LogReader interface {
 	Logs(ctx context.Context, nodeID, source string, lines int) (out, sources []string, err error)

internal/control/modelserve/modelserve.go

@@ -0,0 +1,552 @@
+// Package modelserve takes a model from a registry repository to a replica
+// answering requests, as one job.
+//
+// Adding a model was four commands and a hand-written YAML file: plan it,
+// fetch it into the mirror, write its catalog entry with a memory figure
+// somebody guessed, scale it — and then find out whether it loads. This runs
+// those steps in the control plane, where the mirror, the registry, the
+// catalog and the fleet already are, and reports each one as it goes, so
+// whatever asks — ctl, a page — only has to watch.
+package modelserve
+
+import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"log/slog"
+	"net/http"
+	"regexp"
+	"slices"
+	"strconv"
+	"strings"
+	"sync"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
+)
+
+// Registry reads a repository.
+type Registry interface {
+	Repo(ctx context.Context, id string) (hf.Repo, error)
+}
+
+// Mirror fetches files and reports on them.
+type Mirror interface {
+	Fetch(ctx context.Context, req mirror.FetchRequest) (mirror.FetchResponse, error)
+	List(ctx context.Context) (mirror.ModelsResponse, error)
+}
+
+// Catalog adds a model.
+type Catalog interface {
+	Add(ctx context.Context, m catalog.Model, by string) error
+}
+
+// Fleet is what the job watches once the model is in the catalog.
+type Fleet interface {
+	Replicas() []reconciler.Replica
+	ModelHealth() []reconciler.ModelHealth
+	EngineKey(replicaID string) string
+}
+
+// Deps are what a job needs.
+type Deps struct {
+	Registry Registry
+	Mirror   Mirror
+	Catalog  Catalog
+	Fleet    Fleet
+	// LlamaVersion is the llama.cpp build the nodes install, which decides
+	// which architectures load.
+	LlamaVersion string
+	HTTP         *http.Client
+	Log          *slog.Logger
+	// Poll is how often the job looks at the mirror and the fleet.
+	Poll time.Duration
+	// ReadyTimeout bounds the wait for a replica: the node fetching the
+	// weights from the mirror and the engine loading them.
+	ReadyTimeout time.Duration
+}
+
+// Request is what to serve.
+type Request struct {
+	// Repo is the registry repository, owner/name.
+	Repo string `json:"repo"`
+	// Quant picks the file by its quantisation, Q4_K_M unless given.
+	Quant string `json:"quant,omitempty"`
+	// File picks it by name instead.
+	File string `json:"file,omitempty"`
+	// Name is the catalog name; the repository suggests one otherwise.
+	Name        string   `json:"name,omitempty"`
+	Aliases     []string `json:"aliases,omitempty"`
+	Description string   `json:"description,omitempty"`
+	// Replicas to run; 0 makes it on demand, loaded when first asked for.
+	Replicas int `json:"replicas"`
+	// CtxPool is llama.cpp's --ctx-size; ContextLength what one request may
+	// use. They default to 16384 and 4096, what four parallel slots divide.
+	CtxPool       int `json:"ctx_pool,omitempty"`
+	ContextLength int `json:"context_length,omitempty"`
+	// By is who asked, for the catalog's record.
+	By string `json:"-"`
+}
+
+// The states a job and its steps can be in.
+const (
+	Running = "running"
+	Done    = "done"
+	Failed  = "failed"
+	Skipped = "skipped"
+)
+
+// Step is one stage of a job.
+type Step struct {
+	Name   string    `json:"name"`
+	State  string    `json:"state"`
+	Detail string    `json:"detail,omitempty"`
+	At     time.Time `json:"at"`
+}
+
+// Job is one model being served, and where it stands.
+type Job struct {
+	ID      string    `json:"id"`
+	Request Request   `json:"request"`
+	State   string    `json:"state"`
+	Error   string    `json:"error,omitempty"`
+	Model   string    `json:"model,omitempty"`
+	Steps   []Step    `json:"steps"`
+	Started time.Time `json:"started"`
+	// Reply is what the model said to the smoke test, and TokensPerSecond
+	// how fast it said it.
+	Reply           string  `json:"reply,omitempty"`
+	TokensPerSecond float64 `json:"tokens_per_second,omitempty"`
+}
+
+// Server runs jobs and remembers them.
+type Server struct {
+	deps Deps
+	mu   sync.Mutex
+	jobs map[string]*Job
+	seq  int
+}
+
+// New builds a server.
+func New(d Deps) *Server {
+	if d.Poll <= 0 {
+		d.Poll = 2 * time.Second
+	}
+	if d.ReadyTimeout <= 0 {
+		d.ReadyTimeout = 30 * time.Minute
+	}
+	if d.HTTP == nil {
+		d.HTTP = &http.Client{Timeout: 5 * time.Minute}
+	}
+	if d.Log == nil {
+		d.Log = slog.Default()
+	}
+	return &Server{deps: d, jobs: map[string]*Job{}}
+}
+
+var repoPattern = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._-]*/[A-Za-z0-9][A-Za-z0-9._-]*$`)
+
+// Start begins a job and returns at once; the job runs on its own.
+func (s *Server) Start(req Request) (Job, error) {
+	req.Repo = strings.Trim(strings.TrimSpace(req.Repo), "/")
+	if !repoPattern.MatchString(req.Repo) {
+		return Job{}, fmt.Errorf("%q is not a repository; it looks like owner/name", req.Repo)
+	}
+	if req.Replicas < 0 {
+		return Job{}, errors.New("replicas cannot be negative")
+	}
+	if req.Quant == "" && req.File == "" {
+		req.Quant = "Q4_K_M"
+	}
+	if req.CtxPool <= 0 {
+		req.CtxPool = 16384
+	}
+	if req.ContextLength <= 0 {
+		req.ContextLength = 4096
+	}
+	s.mu.Lock()
+	s.seq++
+	j := &Job{ID: fmt.Sprintf("serve-%d-%d", time.Now().Unix(), s.seq), Request: req,
+		State: Running, Started: time.Now()}
+	s.jobs[j.ID] = j
+	s.mu.Unlock()
+	go s.run(j)
+	return s.snapshot(j), nil
+}
+
+// Get returns a job as it stands.
+func (s *Server) Get(id string) (Job, bool) {
+	s.mu.Lock()
+	j, ok := s.jobs[id]
+	s.mu.Unlock()
+	if !ok {
+		return Job{}, false
+	}
+	return s.snapshot(j), true
+}
+
+func (s *Server) snapshot(j *Job) Job {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	out := *j
+	out.Steps = slices.Clone(j.Steps)
+	return out
+}
+
+// step starts a new step, or updates the detail of the current one.
+func (s *Server) step(j *Job, name, state, detail string) {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	if n := len(j.Steps); n > 0 && j.Steps[n-1].Name == name {
+		j.Steps[n-1].State, j.Steps[n-1].Detail, j.Steps[n-1].At = state, detail, time.Now()
+		return
+	}
+	j.Steps = append(j.Steps, Step{Name: name, State: state, Detail: detail, At: time.Now()})
+}
+
+func (s *Server) fail(j *Job, name string, err error) {
+	s.step(j, name, Failed, err.Error())
+	s.mu.Lock()
+	j.State, j.Error = Failed, err.Error()
+	s.mu.Unlock()
+	s.deps.Log.Warn("serving a model failed", "job", j.ID, "repo", j.Request.Repo, "step", name, "err", err)
+}
+
+func (s *Server) run(j *Job) {
+	ctx := context.Background()
+	req := j.Request
+
+	// 1. The registry: which files.
+	s.step(j, "registry", Running, "reading "+req.Repo)
+	repo, err := s.deps.Registry.Repo(ctx, req.Repo)
+	if err != nil {
+		s.fail(j, "registry", err)
+		return
+	}
+	if repo.Gated.Is() {
+		s.fail(j, "registry", fmt.Errorf("%s is gated: it needs an account's approval the mirror does not have", req.Repo))
+		return
+	}
+	set, err := pick(repo, req)
+	if err != nil {
+		s.fail(j, "registry", err)
+		return
+	}
+	name := req.Name
+	if name == "" {
+		name = hf.ModelName(repo.ID)
+	}
+	s.mu.Lock()
+	j.Model = name
+	s.mu.Unlock()
+	s.step(j, "registry", Done, fmt.Sprintf("%s, %.1f GB in %d file(s), revision %s",
+		set.Name, gb(set.Bytes), len(set.Files), repo.ShortRevision()))
+
+	// 2. The mirror: fetch, and check what arrived against what was published.
+	source, err := s.mirror(ctx, j, repo, set, name)
+	if err != nil {
+		s.fail(j, "mirror", err)
+		return
+	}
+
+	// 3. Whether the nodes' llama.cpp loads it, and how much memory it needs.
+	s.step(j, "check", Running, "reading its header from the mirror")
+	var header gguf.Header
+	if !strings.HasSuffix(source, "/") {
+		v := gguf.Check(ctx, source, s.deps.LlamaVersion)
+		if !v.Servable {
+			s.fail(j, "check", fmt.Errorf("llama.cpp %s on the nodes cannot run this: %s", s.deps.LlamaVersion, v.Reason))
+			return
+		}
+		header = v.Header
+	}
+	memGB := int((header.MemoryBytes(set.Bytes, req.CtxPool) + 1<<30 - 1) >> 30)
+	basis := fmt.Sprintf("weights %.1f GB + KV cache %.1f GB for %d tokens + engine",
+		gb(set.Bytes), gb(int64(req.CtxPool)*header.KVBytesPerToken()), req.CtxPool)
+	if memGB == 0 {
+		// A set of shards, or a header that did not say: weights plus half
+		// again plus two, the rough figure from before shapes were read.
+		memGB = int(gb(set.Bytes)*1.5) + 2
+		basis = "weights plus half again plus two; the header gave no shape"
+	}
+	s.step(j, "check", Done, fmt.Sprintf("%s, needs about %d GB (%s)", orUnknown(header.Architecture), memGB, basis))
+
+	// 4. The catalog.
+	m := catalog.Model{
+		Name: name, Aliases: req.Aliases, Description: req.Description, Source: source,
+		RoutingProfile: "chat", ContextLength: req.ContextLength, MaxOutput: 1024,
+		Profiles: []catalog.Profile{{
+			Name: "cpu", Engine: "llamacpp", Weight: 1, Priority: 100,
+			Args: []string{"--ctx-size", strconv.Itoa(req.CtxPool)}, MinMemoryGB: memGB,
+		}},
+		Scaling: catalog.Scaling{Min: req.Replicas, Max: max(req.Replicas, 1)},
+	}
+	s.step(j, "catalog", Running, "adding "+name)
+	if err := s.deps.Catalog.Add(ctx, m, req.By); err != nil {
+		s.fail(j, "catalog", err)
+		return
+	}
+	if req.Replicas == 0 {
+		s.step(j, "catalog", Done, name+" added on demand: it loads when first asked for")
+		s.finish(j)
+		return
+	}
+	s.step(j, "catalog", Done, fmt.Sprintf("%s added, %d replica(s)", name, req.Replicas))
+
+	// 5. A replica, ready.
+	rep, err := s.waitReady(ctx, j, name)
+	if err != nil {
+		s.fail(j, "placement", err)
+		return
+	}
+
+	// 6. Does it answer.
+	s.step(j, "smoke test", Running, "asking "+rep.ID+" to say hello")
+	reply, tps, err := s.smoke(ctx, rep, name)
+	if err != nil {
+		s.fail(j, "smoke test", err)
+		return
+	}
+	s.mu.Lock()
+	j.Reply, j.TokensPerSecond = reply, tps
+	s.mu.Unlock()
+	s.step(j, "smoke test", Done, fmt.Sprintf("answered at %.1f tokens/s: %q", tps, reply))
+	s.finish(j)
+}
+
+func (s *Server) finish(j *Job) {
+	s.mu.Lock()
+	j.State = Done
+	s.mu.Unlock()
+	s.deps.Log.Info("model served", "job", j.ID, "repo", j.Request.Repo, "model", j.Model)
+}
+
+// pick chooses the file set to serve.
+func pick(repo hf.Repo, req Request) (hf.FileSet, error) {
+	var weights []hf.File
+	for _, f := range repo.Files {
+		lower := strings.ToLower(f.Path)
+		if !strings.HasSuffix(lower, ".gguf") || strings.Contains(lower, "mmproj") ||
+			strings.Contains(lower, "imatrix") {
+			continue
+		}
+		weights = append(weights, f)
+	}
+	sets := hf.Group(weights)
+	if len(sets) == 0 {
+		return hf.FileSet{}, fmt.Errorf("%s has no GGUF files; the nodes run llama.cpp, which needs one", repo.ID)
+	}
+	var names []string
+	for _, st := range sets {
+		names = append(names, st.Name)
+		if req.File != "" && (st.Name == req.File || slices.ContainsFunc(st.Files, func(f hf.File) bool { return f.Path == req.File })) {
+			return st, nil
+		}
+	}
+	if req.File == "" {
+		want := strings.ToLower(req.Quant)
+		var match []hf.FileSet
+		for _, st := range sets {
+			if strings.Contains(strings.ToLower(st.Name), want) {
+				match = append(match, st)
+			}
+		}
+		if len(match) == 1 {
+			return match[0], nil
+		}
+		if len(match) > 1 {
+			// Both a split and a joined copy of one quantisation: the joined
+			// one, which is one download to verify rather than several.
+			slices.SortFunc(match, func(a, b hf.FileSet) int { return a.Shards - b.Shards })
+			return match[0], nil
+		}
+	}
+	return hf.FileSet{}, fmt.Errorf("%s has no file matching %q; it has: %s",
+		repo.ID, or(req.File, req.Quant), strings.Join(names, ", "))
+}
+
+// mirror fetches the set into the mirror and returns the source a catalog
+// entry names: the file, or the directory for a set of shards.
+func (s *Server) mirror(ctx context.Context, j *Job, repo hf.Repo, set hf.FileSet, name string) (string, error) {
+	version := repo.ShortRevision()
+	s.step(j, "mirror", Running, "asking the mirror for "+set.Name)
+	var base string
+	for _, f := range set.Files {
+		file := mirror.LastPathElement(f.Path)
+		resp, err := s.deps.Mirror.Fetch(ctx, mirror.FetchRequest{
+			Source: repo.DownloadURL("", f.Path), Model: name, Version: version, File: file,
+		})
+		// Already there is not a failure: the same commit fetched before.
+		if err != nil && !strings.Contains(err.Error(), "already") {
+			return "", err
+		}
+		base = resp.URL
+	}
+	deadline := time.Now().Add(6 * time.Hour)
+	for {
+		got, err := s.deps.Mirror.List(ctx)
+		if err != nil {
+			return "", err
+		}
+		var done, total int64
+		present, failed := 0, ""
+		sums := map[string]string{}
+		for _, st := range got.Models {
+			if st.Model != name || st.Version != version {
+				continue
+			}
+			done += st.BytesDone
+			total += max(st.BytesTotal, st.BytesDone)
+			switch st.State {
+			case mirror.Present:
+				present++
+				sums[st.File] = st.SHA256
+			case mirror.Failed:
+				failed = st.Err
+			}
+		}
+		if failed != "" {
+			return "", fmt.Errorf("the mirror could not fetch it: %s", failed)
+		}
+		if present >= len(set.Files) {
+			// What arrived against what the registry published: a mirror
+			// copies one file to every node, and is the one place a corrupted
+			// download is caught once instead of loaded everywhere.
+			for _, f := range set.Files {
+				file := mirror.LastPathElement(f.Path)
+				if f.SHA256 != "" && sums[file] != "" && !strings.EqualFold(f.SHA256, sums[file]) {
+					return "", fmt.Errorf("%s arrived with sha256 %s, and the registry published %s",
+						file, sums[file][:12], f.SHA256[:12])
+				}
+			}
+			source := base
+			if len(set.Files) > 1 {
+				source = strings.TrimSuffix(source, mirror.LastPathElement(source))
+			}
+			s.step(j, "mirror", Done, fmt.Sprintf("%.1f GB on the mirror, checksums match the registry", gb(total)))
+			return source, nil
+		}
+		if total > 0 {
+			s.step(j, "mirror", Running, fmt.Sprintf("fetching: %.1f of %.1f GB (%d%%)",
+				gb(done), gb(total), done*100/total))
+		}
+		if time.Now().After(deadline) {
+			return "", errors.New("the mirror had not finished after six hours")
+		}
+		time.Sleep(s.deps.Poll)
+	}
+}
+
+// waitReady waits for a replica of the model to serve, reporting what it is
+// doing and giving up early when it cannot be placed or keeps failing.
+func (s *Server) waitReady(ctx context.Context, j *Job, name string) (reconciler.Replica, error) {
+	s.step(j, "placement", Running, "waiting for the control plane to place it")
+	started := time.Now()
+	deadline := started.Add(s.deps.ReadyTimeout)
+	for {
+		for _, h := range s.deps.Fleet.ModelHealth() {
+			if h.Model != name {
+				continue
+			}
+			if h.Parked {
+				return reconciler.Replica{}, fmt.Errorf("it failed to start %d times and is parked: %s", h.Failures, h.Reason)
+			}
+			// A minute's grace from when this step began, not from the job:
+			// the download before it can take hours, and the first pass may
+			// not have run yet.
+			if h.Unplaceable && time.Since(started) > time.Minute {
+				return reconciler.Replica{}, fmt.Errorf("no node can host it:\n%s", h.Placement)
+			}
+		}
+		for _, r := range s.deps.Fleet.Replicas() {
+			if r.Model != name {
+				continue
+			}
+			if r.State == reconciler.Ready {
+				s.step(j, "placement", Done, fmt.Sprintf("%s ready on %s at %s", r.ID,
+					strings.Join(r.NodeIDs(), ","), r.Endpoint))
+				return r, nil
+			}
+			s.step(j, "placement", Running, fmt.Sprintf("%s on %s is %s", r.ID,
+				strings.Join(r.NodeIDs(), ","), r.State))
+		}
+		if time.Now().After(deadline) {
+			return reconciler.Replica{}, fmt.Errorf("no replica was ready after %s", s.deps.ReadyTimeout)
+		}
+		select {
+		case <-ctx.Done():
+			return reconciler.Replica{}, ctx.Err()
+		case <-time.After(s.deps.Poll):
+		}
+	}
+}
+
+// smoke sends one short request straight to the replica, with its engine key.
+func (s *Server) smoke(ctx context.Context, r reconciler.Replica, name string) (string, float64, error) {
+	body, _ := json.Marshal(map[string]any{
+		"model":      name,
+		"max_tokens": 32,
+		"messages":   []map[string]string{{"role": "user", "content": "Say hello in one short sentence."}},
+	})
+	req, err := http.NewRequestWithContext(ctx, http.MethodPost,
+		strings.TrimRight(r.Endpoint, "/")+"/v1/chat/completions", bytes.NewReader(body))
+	if err != nil {
+		return "", 0, err
+	}
+	req.Header.Set("Content-Type", "application/json")
+	if k := s.deps.Fleet.EngineKey(r.ID); k != "" {
+		req.Header.Set("Authorization", "Bearer "+k)
+	}
+	start := time.Now()
+	resp, err := s.deps.HTTP.Do(req)
+	if err != nil {
+		return "", 0, err
+	}
+	defer resp.Body.Close()
+	b, _ := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
+	if resp.StatusCode != http.StatusOK {
+		return "", 0, fmt.Errorf("the replica answered %s: %s", resp.Status, strings.TrimSpace(string(b)))
+	}
+	var out struct {
+		Choices []struct {
+			Message struct {
+				Content string `json:"content"`
+			} `json:"message"`
+		} `json:"choices"`
+		Usage struct {
+			CompletionTokens int `json:"completion_tokens"`
+		} `json:"usage"`
+	}
+	if err := json.Unmarshal(b, &out); err != nil || len(out.Choices) == 0 {
+		return "", 0, fmt.Errorf("the replica's answer was not a completion: %s", strings.TrimSpace(string(b)))
+	}
+	secs := time.Since(start).Seconds()
+	tps := 0.0
+	if secs > 0 {
+		tps = float64(out.Usage.CompletionTokens) / secs
+	}
+	return strings.TrimSpace(out.Choices[0].Message.Content), tps, nil
+}
+
+func gb(b int64) float64 { return float64(b) / (1 << 30) }
+
+func or(a, b string) string {
+	if a != "" {
+		return a
+	}
+	return b
+}
+
+func orUnknown(arch string) string {
+	if arch == "" {
+		return "architecture not read"
+	}
+	return arch
+}

internal/control/modelserve/modelserve_test.go

@@ -0,0 +1,225 @@
+package modelserve
+
+import (
+	"bytes"
+	"context"
+	"encoding/binary"
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"sync"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
+)
+
+// ggufHeader is enough of a GGUF file for its architecture and shape.
+func ggufHeader() []byte {
+	var b bytes.Buffer
+	w := func(v any) { _ = binary.Write(&b, binary.LittleEndian, v) }
+	str := func(s string) { w(uint64(len(s))); b.WriteString(s) }
+	u32 := func(k string, v uint32) { str(k); w(uint32(4)); w(v) }
+	b.WriteString("GGUF")
+	w(uint32(3))
+	w(uint64(0))
+	w(uint64(5))
+	str("general.architecture")
+	w(uint32(8))
+	str("qwen2")
+	u32("qwen2.block_count", 28)
+	u32("qwen2.attention.head_count", 12)
+	u32("qwen2.attention.head_count_kv", 2)
+	u32("qwen2.embedding_length", 1536)
+	return b.Bytes()
+}
+
+type fakeRegistry struct{ repo hf.Repo }
+
+func (f fakeRegistry) Repo(context.Context, string) (hf.Repo, error) { return f.repo, nil }
+
+// fakeMirror serves the header at every file URL and reports each fetched
+// file present, with the checksum it was told to.
+type fakeMirror struct {
+	mu   sync.Mutex
+	srv  *httptest.Server
+	got  []mirror.FetchRequest
+	sums map[string]string
+}
+
+func (m *fakeMirror) Fetch(_ context.Context, r mirror.FetchRequest) (mirror.FetchResponse, error) {
+	m.mu.Lock()
+	defer m.mu.Unlock()
+	m.got = append(m.got, r)
+	return mirror.FetchResponse{URL: m.srv.URL + "/" + r.Model + "/" + r.Version + "/" + r.File}, nil
+}
+
+func (m *fakeMirror) List(context.Context) (mirror.ModelsResponse, error) {
+	m.mu.Lock()
+	defer m.mu.Unlock()
+	var out mirror.ModelsResponse
+	for _, r := range m.got {
+		out.Models = append(out.Models, mirror.Status{Model: r.Model, Version: r.Version, File: r.File,
+			State: mirror.Present, BytesDone: 1 << 30, BytesTotal: 1 << 30, SHA256: m.sums[r.File]})
+	}
+	return out, nil
+}
+
+type fakeCatalog struct {
+	mu    sync.Mutex
+	added []catalog.Model
+}
+
+func (c *fakeCatalog) Add(_ context.Context, m catalog.Model, _ string) error {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+	c.added = append(c.added, m)
+	return nil
+}
+
+// fakeFleet has a replica ready at engine once the catalog holds the model.
+type fakeFleet struct {
+	cat    *fakeCatalog
+	engine string
+	health []reconciler.ModelHealth
+}
+
+func (f *fakeFleet) Replicas() []reconciler.Replica {
+	f.cat.mu.Lock()
+	defer f.cat.mu.Unlock()
+	var out []reconciler.Replica
+	for _, m := range f.cat.added {
+		out = append(out, reconciler.Replica{ID: m.Name + "-r1", Model: m.Name,
+			State: reconciler.Ready, Endpoint: f.engine,
+			Shards: []reconciler.Shard{{NodeID: "node01"}}})
+	}
+	return out
+}
+func (f *fakeFleet) ModelHealth() []reconciler.ModelHealth { return f.health }
+func (f *fakeFleet) EngineKey(string) string               { return "k" }
+
+func setup(t *testing.T, files []hf.File, sums map[string]string) (*Server, *fakeMirror, *fakeCatalog, *fakeFleet) {
+	t.Helper()
+	header := ggufHeader()
+	files0 := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+		_, _ = w.Write(header)
+	}))
+	t.Cleanup(files0.Close)
+	engine := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		if r.Header.Get("Authorization") != "Bearer k" {
+			w.WriteHeader(http.StatusUnauthorized)
+			return
+		}
+		_, _ = w.Write([]byte(`{"choices":[{"message":{"content":"Hello!"}}],"usage":{"completion_tokens":3}}`))
+	}))
+	t.Cleanup(engine.Close)
+
+	m := &fakeMirror{srv: files0, sums: sums}
+	c := &fakeCatalog{}
+	f := &fakeFleet{cat: c, engine: engine.URL}
+	s := New(Deps{
+		Registry: fakeRegistry{repo: hf.Repo{ID: "bartowski/Qwen2.5-1.5B-Instruct-GGUF", Revision: "abcdef123456", Files: files}},
+		Mirror:   m, Catalog: c, Fleet: f, Poll: 10 * time.Millisecond, ReadyTimeout: 5 * time.Second,
+	})
+	return s, m, c, f
+}
+
+func wait(t *testing.T, s *Server, id string) Job {
+	t.Helper()
+	deadline := time.Now().Add(10 * time.Second)
+	for time.Now().Before(deadline) {
+		if j, _ := s.Get(id); j.State != Running {
+			return j
+		}
+		time.Sleep(10 * time.Millisecond)
+	}
+	t.Fatal("the job never finished")
+	return Job{}
+}
+
+func TestAModelGoesFromTheRegistryToAnAnsweringReplica(t *testing.T) {
+	files := []hf.File{
+		{Path: "Qwen2.5-1.5B-Instruct-Q8_0.gguf", Bytes: 2 << 30, SHA256: "aa"},
+		{Path: "Qwen2.5-1.5B-Instruct-Q4_K_M.gguf", Bytes: 1 << 30, SHA256: "bb"},
+		{Path: "mmproj-Q4_K_M.gguf", Bytes: 1 << 20},
+	}
+	s, m, c, _ := setup(t, files, map[string]string{"Qwen2.5-1.5B-Instruct-Q4_K_M.gguf": "BB"})
+	job, err := s.Start(Request{Repo: "bartowski/Qwen2.5-1.5B-Instruct-GGUF", Replicas: 1})
+	if err != nil {
+		t.Fatal(err)
+	}
+	j := wait(t, s, job.ID)
+	if j.State != Done {
+		t.Fatalf("job %s: %s\n%+v", j.State, j.Error, j.Steps)
+	}
+	if len(m.got) != 1 || m.got[0].File != "Qwen2.5-1.5B-Instruct-Q4_K_M.gguf" || m.got[0].Version != "abcdef12" {
+		t.Errorf("fetched %+v; expected the Q4_K_M file at its revision", m.got)
+	}
+	mod := c.added[0]
+	if mod.Name != "qwen2.5-1.5b-instruct" || mod.Profiles[0].MinMemoryGB == 0 || mod.Scaling.Min != 1 {
+		t.Errorf("catalog entry %+v", mod)
+	}
+	if j.Reply != "Hello!" {
+		t.Errorf("the smoke test's answer was %q", j.Reply)
+	}
+	var names []string
+	for _, st := range j.Steps {
+		names = append(names, st.Name)
+	}
+	if strings.Join(names, ",") != "registry,mirror,check,catalog,placement,smoke test" {
+		t.Errorf("steps %v", names)
+	}
+}
+
+func TestAQuantTheRepositoryDoesNotHaveIsNamedWithWhatItHas(t *testing.T) {
+	s, _, c, _ := setup(t, []hf.File{{Path: "m-Q8_0.gguf", Bytes: 1}}, nil)
+	job, _ := s.Start(Request{Repo: "a/b", Replicas: 1})
+	j := wait(t, s, job.ID)
+	if j.State != Failed || !strings.Contains(j.Error, "m-Q8_0.gguf") {
+		t.Errorf("%s: %s", j.State, j.Error)
+	}
+	if len(c.added) != 0 {
+		t.Error("something was added to the catalog")
+	}
+}
+
+func TestADownloadThatDoesNotMatchTheRegistryIsNotServed(t *testing.T) {
+	files := []hf.File{{Path: "m-Q4_K_M.gguf", Bytes: 1 << 30, SHA256: "aaaaaaaaaaaaaaaa"}}
+	s, _, c, _ := setup(t, files, map[string]string{"m-Q4_K_M.gguf": "bbbbbbbbbbbbbbbb"})
+	job, _ := s.Start(Request{Repo: "a/b", Replicas: 1})
+	j := wait(t, s, job.ID)
+	if j.State != Failed || !strings.Contains(j.Error, "registry published") {
+		t.Errorf("%s: %s", j.State, j.Error)
+	}
+	if len(c.added) != 0 {
+		t.Error("a corrupted download reached the catalog")
+	}
+}
+
+func TestAParkedModelStopsTheWait(t *testing.T) {
+	s, _, _, f := setup(t, []hf.File{{Path: "m-Q4_K_M.gguf", Bytes: 1 << 30}}, nil)
+	f.health = []reconciler.ModelHealth{{Model: "qwen2.5-1.5b-instruct", Parked: true, Failures: 5, Reason: "unknown model architecture"}}
+	// A fleet where it never becomes ready, only parked.
+	s.deps.Fleet = parkedFleet{f}
+	job, _ := s.Start(Request{Repo: "a/b", Replicas: 1})
+	j := wait(t, s, job.ID)
+	if j.State != Failed || !strings.Contains(j.Error, "parked") {
+		t.Errorf("%s: %s", j.State, j.Error)
+	}
+}
+
+type parkedFleet struct{ *fakeFleet }
+
+func (parkedFleet) Replicas() []reconciler.Replica { return nil }
+
+func TestARepositoryThatIsNotOneIsRefused(t *testing.T) {
+	s := New(Deps{})
+	for _, r := range []string{"", "noslash", "../etc/passwd", "a/b/c"} {
+		if _, err := s.Start(Request{Repo: r}); err == nil {
+			t.Errorf("%q accepted", r)
+		}
+	}
+}

internal/control/reconciler/adopt.go

@@ -0,0 +1,255 @@
+package reconciler
+
+import (
+	"slices"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/supervisor"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
+)
+
+// Adopt rebuilds replica state from shards an agent reports it is already
+// running (plan §6.5).
+//
+// This is what makes a control plane restart survivable. Without it the
+// reconciler sees an empty cluster, places duplicates, and the engines already
+// running keep holding GPUs the inventory believes are free — so the duplicates
+// fail to allocate, with an error naming the new process rather than the old.
+//
+// Shards whose model or profile is no longer in the catalog are returned for
+// the caller to stop: nothing will ever claim them.
+func (r *Reconciler) Adopt(nodeID string, shards []AdoptedShard) (unclaimed []AdoptedShard) {
+	defer r.forgetShardsTheAgentDoesNotHave(nodeID, shards)
+
+	byReplica := map[string][]AdoptedShard{}
+	for _, s := range shards {
+		byReplica[s.ReplicaID] = append(byReplica[s.ReplicaID], s)
+	}
+
+	r.mu.Lock()
+	defer r.mu.Unlock()
+
+	for replicaID, group := range byReplica {
+		if existing, known := r.replicas[replicaID]; known {
+			// The other half of a replica we are still assembling. Merge rather
+			// than choose: these shards belong to the same replica as the ones
+			// already adopted, they are simply on a different node whose agent
+			// registered a moment later. Treating them as a competing claim is
+			// what tore multi-node replicas apart on every control plane
+			// restart — one half adopted and failed for being short, the other
+			// stopped as unclaimed when it finally arrived.
+			if !existing.AdoptedAt.IsZero() && len(existing.Shards) < existing.WantShards {
+				r.mergeAdopted(existing, group)
+				continue
+			}
+			// Usually this is an agent reconnecting rather than a control plane
+			// restart, and its status reports keep the state current.
+			//
+			// Not always. Replica ids are derived from the model name and a
+			// counter, so a control plane that restarts and places before the
+			// agents reconnect produces a *different* replica under the same
+			// id, on a freshly allocated port. Skipping the adoption then
+			// leaves the cluster pointing at a port nothing listens on, while
+			// the engine that is actually serving is treated as a stray.
+			//
+			// The fleet is authoritative about what is running (plan §6.5), so
+			// what we merely placed loses to what an agent reports — unless our
+			// copy has a shard that has actually come up, in which case two
+			// things really are running and the one we did not ask for is the
+			// stray.
+			if existing.hasLiveShard() {
+				// Only what we do not recognise. A reconnecting agent reports
+				// the shards we already know, and disowning those would tell it
+				// to stop the very replica that is serving.
+				known := map[string]struct{}{}
+				for _, sh := range existing.Shards {
+					known[sh.ID] = struct{}{}
+				}
+				var strays []AdoptedShard
+				for _, s := range group {
+					if _, ok := known[s.ShardID]; !ok {
+						strays = append(strays, s)
+					}
+				}
+				unclaimed = append(unclaimed, strays...)
+				r.reservePorts(strays)
+				continue
+			}
+			r.log.Warn("a running replica reported by the fleet replaces one we placed",
+				"replica", replicaID, "placed_endpoint", existing.Endpoint)
+			for _, s := range existing.Shards {
+				r.inv.Release(s.ID)
+			}
+			delete(r.replicas, replicaID)
+		}
+
+		first := group[0]
+		m, ok := r.cat.Load().Resolve(first.Model)
+		if !ok {
+			unclaimed = append(unclaimed, group...)
+			r.reservePorts(group)
+			continue
+		}
+		prof, ok := m.ProfileByName(first.Profile)
+		if !ok {
+			unclaimed = append(unclaimed, group...)
+			r.reservePorts(group)
+			continue
+		}
+
+		rep := &Replica{
+			ID: replicaID, Model: m.Name, Profile: prof.Name,
+			Fabric: first.Fabric, Pool: first.Pool,
+			// From the agent, which was told it when the shard started. Empty
+			// from an agent too old to report it, which reads as unknown.
+			ProfileFP: first.ProfileFP,
+			Weight:    prof.Weight, CreatedAt: r.now(),
+			Endpoint: first.Endpoint,
+			// From the profile, not from what was reported: the point is to
+			// notice when what was reported is short.
+			WantShards: max(prof.Nodes, 1),
+			AdoptedAt:  r.now(),
+		}
+		r.mergeAdopted(rep, group)
+		r.replicas[replicaID] = rep
+
+	}
+	return unclaimed
+}
+
+// mergeAdopted folds shards an agent reports into a replica being reassembled.
+//
+// Called both for the first group, which creates the replica, and for every
+// later group belonging to it, so a replica spanning N nodes is built up over N
+// registrations. Shard ids already present are skipped: an agent that
+// reconnects repeats what it is running, and counting those again would carry a
+// replica past its shard count without a second node ever appearing.
+func (r *Reconciler) mergeAdopted(rep *Replica, group []AdoptedShard) {
+	have := make(map[string]struct{}, len(rep.Shards))
+	for _, sh := range rep.Shards {
+		have[sh.ID] = struct{}{}
+	}
+	for _, s := range group {
+		if _, dup := have[s.ShardID]; dup {
+			continue
+		}
+		have[s.ShardID] = struct{}{}
+		rep.Shards = append(rep.Shards, Shard{
+			ID: s.ShardID, NodeID: s.NodeID, Rank: s.Rank,
+			GPUIndices: s.GPUIndices, HCAs: s.HCAs,
+			HTTPPort: s.HTTPPort, State: s.State,
+		})
+		// Reclaim the GPUs and the memory, or the next placement hands them
+		// out twice. The memory comes from the catalog rather than the agent:
+		// what a profile asked for is not something a running process reports.
+		r.inv.Allocate(s.NodeID, inventory.Allocation{
+			ShardID: s.ShardID, ReplicaID: rep.ID, GPUIndices: s.GPUIndices,
+			MemoryGB: r.profileMemory(rep.Model, rep.Profile),
+		})
+		// The endpoint comes from the leader, which may be in any of the
+		// groups — the rank 0 node is not necessarily the first to register.
+		if s.Rank == 0 && s.Endpoint != "" {
+			rep.Endpoint = s.Endpoint
+		}
+	}
+	slices.SortFunc(rep.Shards, func(a, b Shard) int { return a.Rank - b.Rank })
+	rep.deriveState(r.now())
+
+	r.log.Info("adopted replica from the fleet",
+		"replica", rep.ID, "model", rep.Model,
+		"shards", len(rep.Shards), "want", rep.WantShards, "state", rep.State)
+}
+
+// forgetShardsTheAgentDoesNotHave fails shards we believe are on a node that the
+// agent, having just told us everything it is running, did not mention.
+//
+// The mirror image of adoption, and the half that was missing. Adoption handles
+// "the agent has something we do not know about"; this handles "we think the
+// agent has something it does not". Both follow from the same rule: the fleet is
+// authoritative about what is running (plan §6.5).
+//
+// Without it, an engine that dies while its agent survives leaves the control
+// plane believing a replica is Ready forever. Observed on a real cluster: the
+// engine was killed, the agent restarted and registered with nothing running,
+// and the replica stayed Ready pointing at a dead port for as long as anyone
+// watched. Traffic survived — the breaker took the endpoint out — but the
+// cluster ran a third short with no sign of it.
+//
+// Safe against the obvious race. A registration follows a fresh connection, so
+// anything we dispatched over a previous one is not running by definition;
+// there is no window where a shard is legitimately started but unreported.
+func (r *Reconciler) forgetShardsTheAgentDoesNotHave(nodeID string, running []AdoptedShard) {
+	if nodeID == "" {
+		return
+	}
+	reported := make(map[string]struct{}, len(running))
+	for _, s := range running {
+		reported[s.ShardID] = struct{}{}
+	}
+
+	var lost []string
+	r.mu.Lock()
+	for _, rep := range r.replicas {
+		if rep.State == Stopped {
+			continue
+		}
+		changed := false
+		for i := range rep.Shards {
+			sh := &rep.Shards[i]
+			if sh.NodeID != nodeID || sh.State.Terminal() {
+				continue
+			}
+			if _, ok := reported[sh.ID]; ok {
+				continue
+			}
+			sh.State = supervisor.Failed
+			sh.Err = "the agent reports it is not running"
+			r.inv.Release(sh.ID)
+			changed = true
+			lost = append(lost, sh.ID)
+		}
+		if changed {
+			rep.deriveState(r.now())
+		}
+	}
+	r.mu.Unlock()
+
+	for _, id := range lost {
+		r.log.Warn("a shard we believed was running is not; the agent does not have it",
+			"node", nodeID, "shard", id)
+	}
+}
+
+// reservePorts records ports held by shards nothing owns.
+//
+// They are being stopped, but stopping is not instant and the engine keeps its
+// socket until it exits. Handing the port to a new shard in the meantime makes
+// that shard fail to bind, which fails its replica, which places another one on
+// the same port — a loop that does not end on its own.
+//
+// Called with the lock held.
+func (r *Reconciler) reservePorts(shards []AdoptedShard) {
+	for _, s := range shards {
+		if s.HTTPPort > 0 {
+			r.reservedPorts[s.HTTPPort] = struct{}{}
+		}
+	}
+}
+
+// AdoptedShard is one running shard an agent reported at registration.
+type AdoptedShard struct {
+	NodeID     string
+	ReplicaID  string
+	ShardID    string
+	Rank       int
+	GPUIndices []int
+	HCAs       []string
+	HTTPPort   int
+	State      supervisor.State
+	Model      string
+	Profile    string
+	Pool       string
+	Fabric     string
+	ProfileFP  string
+	// Endpoint is the leader's base URL, set on the rank 0 shard.
+	Endpoint string
+}

internal/control/reconciler/demand.go

@@ -0,0 +1,214 @@
+package reconciler
+
+import (
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+)
+
+// RecordUsage stamps replicas with when each was last sent a request.
+//
+// Replicas absent from the map are left alone rather than zeroed: a router that
+// has just restarted knows nothing about what happened before it, and treating
+// that as "never used" would make every replica look abandoned at once —
+// exactly when a fleet is least able to afford acting on that.
+func (r *Reconciler) RecordUsage(lastUsed map[string]time.Time) {
+	if len(lastUsed) == 0 {
+		return
+	}
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	for id, at := range lastUsed {
+		rep, ok := r.replicas[id]
+		if !ok {
+			continue
+		}
+		// Only forwards. Two routers reporting, or a stale snapshot arriving
+		// late, must not move a replica's last use backwards.
+		if at.After(rep.LastUsedAt) {
+			rep.LastUsedAt = at
+		}
+	}
+	r.noteUsage(lastUsed)
+}
+
+// IdleFor reports how long a replica has gone without a request, and whether
+// that is known at all.
+//
+// The second return is not a detail. "Idle for an hour" and "nobody has told me
+// yet" are different states, and anything that acts on idleness — eviction,
+// most obviously — has to refuse to act on the second.
+func (r *Reconciler) IdleFor(id string) (time.Duration, bool) {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	rep, ok := r.replicas[id]
+	if !ok || rep.LastUsedAt.IsZero() {
+		return 0, false
+	}
+	return r.now().Sub(rep.LastUsedAt), true
+}
+
+// Want marks an on-demand model as asked for, so the next pass places it.
+//
+// Called by whatever fields the request — the router, in practice. It returns
+// whether the model is one this applies to, so a caller can tell "being loaded"
+// from "there is genuinely nothing serving this", which are different answers
+// to give a client.
+func (r *Reconciler) Want(model string) bool {
+	m, ok := r.cat.Load().Resolve(model)
+	if !ok || !m.Scaling.OnDemand() {
+		return false
+	}
+	r.mu.Lock()
+	r.wanted[m.Name] = r.now()
+	// A placement that failed before this request says nothing about now:
+	// forgotten, so the pass this triggers decides afresh and Unservable
+	// does not answer from a stale result.
+	if res, ok := r.lastExplain[m.Name]; ok && res.Placement == nil {
+		delete(r.lastExplain, m.Name)
+	}
+	r.mu.Unlock()
+	// Immediately, not on the next tick. Someone is waiting on this request,
+	// and five seconds of ticker is five seconds they spend staring at nothing.
+	r.Notify()
+	return true
+}
+
+// desiredFor is how many replicas a model should have right now.
+//
+// For an ordinary model that is its configured minimum. For an on-demand one it
+// is one while somebody is asking and none once they stop, which is the whole
+// feature: a cluster with more models than GPUs can hold every model in its
+// catalog and only the wanted ones in its cards.
+func (r *Reconciler) desiredFor(m catalog.Model) int {
+	base := r.baseFor(m)
+	if base == 0 {
+		return 0
+	}
+	r.mu.Lock()
+	extra := r.extra[m.Name]
+	r.mu.Unlock()
+	return min(base+extra, max(m.Scaling.Max, base))
+}
+
+// baseFor is what a model runs without load: its minimum, or for an on-demand
+// model one while somebody is asking and none once they stop.
+func (r *Reconciler) baseFor(m catalog.Model) int {
+	if !m.Scaling.OnDemand() {
+		return m.Scaling.Min
+	}
+	r.mu.Lock()
+	last, asked := r.wanted[m.Name]
+	r.mu.Unlock()
+	if !asked {
+		return 0
+	}
+	if r.now().Sub(last) > r.idleTimeout(m) {
+		return 0
+	}
+	return 1
+}
+
+// idleTimeout is how long an on-demand model stays up unasked-for.
+func (r *Reconciler) idleTimeout(m catalog.Model) time.Duration {
+	if m.Scaling.IdleTimeout > 0 {
+		return m.Scaling.IdleTimeout
+	}
+	return r.cfg.IdleTimeout
+}
+
+// noteUsage keeps an on-demand model alive while it is being used.
+//
+// Without this a model would be evicted exactly IdleTimeout after the request
+// that loaded it, however busy it had been since — the timer would measure
+// time since the first request rather than the last.
+func (r *Reconciler) noteUsage(lastUsed map[string]time.Time) {
+	byModel := map[string]time.Time{}
+	for id, at := range lastUsed {
+		rep, ok := r.replicas[id]
+		if !ok {
+			continue
+		}
+		if at.After(byModel[rep.Model]) {
+			byModel[rep.Model] = at
+		}
+	}
+	for model, at := range byModel {
+		if cur, ok := r.wanted[model]; ok && at.After(cur) {
+			r.wanted[model] = at
+		}
+	}
+}
+
+// Load is what a model's replicas are doing, as the router sees it.
+type Load struct {
+	Replicas int
+	Running  int
+	Waiting  int
+}
+
+// RecordLoad adds or takes away replicas between a model's min and max as its
+// load says.
+//
+// Max was a ceiling nothing reached: every model ran exactly its minimum
+// however long its queue. Now requests waiting at the engines — every slot of
+// every replica busy — held for ScaleUpAfter add one replica, and a quiet spell
+// of ScaleDownAfter takes one of those back. One step at a time, and not while
+// the last one is still starting: on a small CPU fleet each replica is minutes
+// of weights and gigabytes of memory, and a spike that is over by the time it
+// loads should not leave three behind.
+func (r *Reconciler) RecordLoad(loads map[string]Load) {
+	now := r.now()
+	for _, m := range r.cat.Load().Models() {
+		base := r.baseFor(m)
+		r.mu.Lock()
+		if base == 0 || m.Scaling.Max <= base {
+			delete(r.extra, m.Name)
+			delete(r.busySince, m.Name)
+			delete(r.quietSince, m.Name)
+			r.mu.Unlock()
+			continue
+		}
+		starting := false
+		for _, rep := range r.replicas {
+			if rep.Model == m.Name && (rep.State == Placing || rep.State == Starting) {
+				starting = true
+			}
+		}
+		l := loads[m.Name]
+		perReplica := float64(l.Waiting) / float64(max(l.Replicas, 1))
+		extra := r.extra[m.Name]
+		switch {
+		case perReplica >= r.cfg.ScaleUpWaiting:
+			delete(r.quietSince, m.Name)
+			since, ok := r.busySince[m.Name]
+			if !ok {
+				r.busySince[m.Name] = now
+				break
+			}
+			if now.Sub(since) >= r.cfg.ScaleUpAfter && base+extra < m.Scaling.Max && !starting {
+				r.extra[m.Name] = extra + 1
+				r.busySince[m.Name] = now
+				r.log.Info("scaling up for load", "model", m.Name, "replicas", base+extra+1,
+					"waiting_per_replica", perReplica, "for", now.Sub(since).Round(time.Second))
+			}
+		case l.Waiting == 0 && extra > 0:
+			delete(r.busySince, m.Name)
+			since, ok := r.quietSince[m.Name]
+			if !ok {
+				r.quietSince[m.Name] = now
+				break
+			}
+			if now.Sub(since) >= r.cfg.ScaleDownAfter {
+				r.extra[m.Name] = extra - 1
+				r.quietSince[m.Name] = now
+				r.log.Info("scaling down after a quiet spell", "model", m.Name,
+					"replicas", base+extra-1, "quiet_for", now.Sub(since).Round(time.Second))
+			}
+		default:
+			delete(r.busySince, m.Name)
+			delete(r.quietSince, m.Name)
+		}
+		r.mu.Unlock()
+	}
+}

internal/control/reconciler/health.go

@@ -0,0 +1,151 @@
+package reconciler
+
+import (
+	"crypto/sha256"
+	"encoding/hex"
+	"encoding/json"
+	"fmt"
+	"strings"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+)
+
+// parkRecord is why a model stopped being retried.
+type parkRecord struct {
+	Since    time.Time
+	Reason   string
+	Failures int
+	// fp is the model's definition when it was parked; a different one
+	// unparks it.
+	fp string
+}
+
+type placementNote struct {
+	text string
+	at   time.Time
+}
+
+// modelFingerprint identifies a model's definition, so a change to it can be
+// told from a retry of the same thing.
+func modelFingerprint(m catalog.Model) string {
+	b, _ := json.Marshal(m)
+	sum := sha256.Sum256(b)
+	return hex.EncodeToString(sum[:8])
+}
+
+// placementLogEvery is how often an unchanged placement failure is repeated.
+const placementLogEvery = 10 * time.Minute
+
+// notePlacementFailure logs why a model could not be placed, when the reason
+// is new or has not been said for a while. Said on every pass it was 250
+// identical lines in two days for one model, burying everything else.
+func (r *Reconciler) notePlacementFailure(model, why string) {
+	r.mu.Lock()
+	last := r.placementLogged[model]
+	say := last.text != why || r.now().Sub(last.at) >= placementLogEvery
+	if say {
+		r.placementLogged[model] = placementNote{text: why, at: r.now()}
+	}
+	r.mu.Unlock()
+	if say {
+		r.log.Warn("no placement for model", "model", model, "reasons", why)
+	}
+}
+
+// Retry clears a parked model and its backoff, so it is placed on the next
+// pass.
+func (r *Reconciler) Retry(model string) bool {
+	r.mu.Lock()
+	_, was := r.parked[model]
+	delete(r.parked, model)
+	r.attempts[model] = 0
+	delete(r.nextAttempt, model)
+	r.mu.Unlock()
+	r.Notify()
+	return was
+}
+
+// ModelHealth is how one model stands, for status and metrics.
+type ModelHealth struct {
+	Model string
+	// Desired is how many replicas it should have now; 0 for a stopped model
+	// or an on-demand one nobody is asking for.
+	Desired int
+	Ready   int
+	// Failures counts replicas that failed in a row.
+	Failures int
+	// Parked is set when Failures reached ParkAfter; Reason is the last error.
+	Parked      bool
+	ParkedSince time.Time
+	Reason      string
+	// Unplaceable is set when it is short of replicas and the last attempt to
+	// place one found nowhere; Placement says why.
+	Unplaceable bool
+	Placement   string
+}
+
+// ModelHealth reports every model in the catalog.
+func (r *Reconciler) ModelHealth() []ModelHealth {
+	models := r.cat.Load().Models()
+	desired := map[string]int{}
+	for _, m := range models {
+		desired[m.Name] = r.desiredFor(m)
+	}
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	ready, live := map[string]int{}, map[string]int{}
+	for _, rep := range r.replicas {
+		if rep.State == Ready {
+			ready[rep.Model]++
+		}
+		if rep.State.Counts() {
+			live[rep.Model]++
+		}
+	}
+	out := make([]ModelHealth, 0, len(models))
+	for _, m := range models {
+		h := ModelHealth{Model: m.Name, Desired: desired[m.Name], Ready: ready[m.Name],
+			Failures: r.attempts[m.Name]}
+		if p, ok := r.parked[m.Name]; ok {
+			h.Parked, h.ParkedSince, h.Reason = true, p.Since, p.Reason
+		}
+		if res, ok := r.lastExplain[m.Name]; ok && res.Placement == nil && live[m.Name] < h.Desired {
+			h.Unplaceable, h.Placement = true, res.Explain()
+		}
+		out = append(out, h)
+	}
+	return out
+}
+
+// Unservable says why a model cannot be served at all, so a request for it is
+// told now rather than asked to retry something that will never come up.
+func (r *Reconciler) Unservable(model string) (string, bool) {
+	m, ok := r.cat.Load().Resolve(model)
+	if !ok {
+		return "", false
+	}
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	if p, ok := r.parked[m.Name]; ok {
+		return fmt.Sprintf("%s failed to start %d times in a row and is not being retried: %s",
+			m.Name, p.Failures, p.Reason), true
+	}
+	for _, rep := range r.replicas {
+		if rep.Model == m.Name && rep.State.Counts() {
+			return "", false // something is on its way
+		}
+	}
+	if res, ok := r.lastExplain[m.Name]; ok && res.Placement == nil {
+		return m.Name + " cannot be placed on any node: " + firstLine(res.Explain()), true
+	}
+	return "", false
+}
+
+func firstLine(s string) string {
+	s = strings.TrimSpace(s)
+	if i := strings.IndexByte(s, '\n'); i >= 0 {
+		return strings.TrimSpace(s[:i]) + " (and others; see llmgw ctl explain-placement)"
+	}
+	return s
+}

internal/control/reconciler/reconciler.go

@@ -3,15 +3,10 @@ package reconciler
 import (
 	"context"
 	crypterand "crypto/rand"
-	"crypto/sha256"
-	"encoding/hex"
-	"encoding/json"
-	"errors"
 	"fmt"
 	"log/slog"
 	"math/rand/v2"
 	"slices"
-	"strings"
 	"sync"
 	"time"
 
@@ -70,6 +65,13 @@ type Config struct {
 	// Without a bound, draining a node whose model cannot be placed anywhere
 	// else would never finish.
 	DrainGrace time.Duration
+	// ScaleUpWaiting is the average number of requests waiting per ready
+	// replica, held for ScaleUpAfter, that adds a replica — up to the model's
+	// max. ScaleDownAfter is how long with nothing waiting before one of the
+	// replicas added that way is taken away again, down to its min.
+	ScaleUpWaiting float64
+	ScaleUpAfter   time.Duration
+	ScaleDownAfter time.Duration
 	// ParkAfter is how many failures in a row stop a model being retried.
 	//
 	// Backoff alone never gives up: it grows to MaxBackoff and keeps trying,
@@ -140,6 +142,9 @@ func DefaultConfig() Config {
 		MinLifetime:      5 * time.Minute,
 		DrainGrace:       5 * time.Minute,
 		ParkAfter:        5,
+		ScaleUpWaiting:   1,
+		ScaleUpAfter:     time.Minute,
+		ScaleDownAfter:   10 * time.Minute,
 		PortRange:        [2]int{20000, 20999},
 		MgmtInterface:    "eth0",
 	}
@@ -182,6 +187,15 @@ func (c *Config) normalize() {
 	if c.ParkAfter <= 0 {
 		c.ParkAfter = d.ParkAfter
 	}
+	if c.ScaleUpWaiting <= 0 {
+		c.ScaleUpWaiting = d.ScaleUpWaiting
+	}
+	if c.ScaleUpAfter <= 0 {
+		c.ScaleUpAfter = d.ScaleUpAfter
+	}
+	if c.ScaleDownAfter <= 0 {
+		c.ScaleDownAfter = d.ScaleDownAfter
+	}
 	if c.PortRange[1] <= c.PortRange[0] {
 		c.PortRange = d.PortRange
 	}
@@ -228,6 +242,11 @@ type Reconciler struct {
 	restartQueue map[string][]string
 	// parked holds models that failed ParkAfter times in a row.
 	parked map[string]parkRecord
+	// extra is how many replicas load has added above a model's base, and
+	// busySince/quietSince when its load last crossed the thresholds.
+	extra      map[string]int
+	busySince  map[string]time.Time
+	quietSince map[string]time.Time
 	// placementLogged remembers the last "no placement" said per model, so it
 	// is said when it changes rather than on every five-second pass.
 	placementLogged map[string]placementNote
@@ -261,6 +280,9 @@ func New(cfg Config, cat *catalog.Live, inv *inventory.Store, disp Dispatcher, l
 		wanted:          map[string]time.Time{},
 		restartQueue:    map[string][]string{},
 		parked:          map[string]parkRecord{},
+		extra:           map[string]int{},
+		busySince:       map[string]time.Time{},
+		quietSince:      map[string]time.Time{},
 		placementLogged: map[string]placementNote{},
 		lastExplain:     map[string]placement.Result{},
 		wake:            make(chan struct{}, 1),
@@ -433,127 +455,6 @@ func (r *Reconciler) fleetHasSettled() bool {
 	return settled
 }
 
-// RecordUsage stamps replicas with when each was last sent a request.
-//
-// Replicas absent from the map are left alone rather than zeroed: a router that
-// has just restarted knows nothing about what happened before it, and treating
-// that as "never used" would make every replica look abandoned at once —
-// exactly when a fleet is least able to afford acting on that.
-func (r *Reconciler) RecordUsage(lastUsed map[string]time.Time) {
-	if len(lastUsed) == 0 {
-		return
-	}
-	r.mu.Lock()
-	defer r.mu.Unlock()
-	for id, at := range lastUsed {
-		rep, ok := r.replicas[id]
-		if !ok {
-			continue
-		}
-		// Only forwards. Two routers reporting, or a stale snapshot arriving
-		// late, must not move a replica's last use backwards.
-		if at.After(rep.LastUsedAt) {
-			rep.LastUsedAt = at
-		}
-	}
-	r.noteUsage(lastUsed)
-}
-
-// IdleFor reports how long a replica has gone without a request, and whether
-// that is known at all.
-//
-// The second return is not a detail. "Idle for an hour" and "nobody has told me
-// yet" are different states, and anything that acts on idleness — eviction,
-// most obviously — has to refuse to act on the second.
-func (r *Reconciler) IdleFor(id string) (time.Duration, bool) {
-	r.mu.Lock()
-	defer r.mu.Unlock()
-	rep, ok := r.replicas[id]
-	if !ok || rep.LastUsedAt.IsZero() {
-		return 0, false
-	}
-	return r.now().Sub(rep.LastUsedAt), true
-}
-
-// Want marks an on-demand model as asked for, so the next pass places it.
-//
-// Called by whatever fields the request — the router, in practice. It returns
-// whether the model is one this applies to, so a caller can tell "being loaded"
-// from "there is genuinely nothing serving this", which are different answers
-// to give a client.
-func (r *Reconciler) Want(model string) bool {
-	m, ok := r.cat.Load().Resolve(model)
-	if !ok || !m.Scaling.OnDemand() {
-		return false
-	}
-	r.mu.Lock()
-	r.wanted[m.Name] = r.now()
-	// A placement that failed before this request says nothing about now:
-	// forgotten, so the pass this triggers decides afresh and Unservable
-	// does not answer from a stale result.
-	if res, ok := r.lastExplain[m.Name]; ok && res.Placement == nil {
-		delete(r.lastExplain, m.Name)
-	}
-	r.mu.Unlock()
-	// Immediately, not on the next tick. Someone is waiting on this request,
-	// and five seconds of ticker is five seconds they spend staring at nothing.
-	r.Notify()
-	return true
-}
-
-// desiredFor is how many replicas a model should have right now.
-//
-// For an ordinary model that is its configured minimum. For an on-demand one it
-// is one while somebody is asking and none once they stop, which is the whole
-// feature: a cluster with more models than GPUs can hold every model in its
-// catalog and only the wanted ones in its cards.
-func (r *Reconciler) desiredFor(m catalog.Model) int {
-	if !m.Scaling.OnDemand() {
-		return m.Scaling.Min
-	}
-	r.mu.Lock()
-	last, asked := r.wanted[m.Name]
-	r.mu.Unlock()
-	if !asked {
-		return 0
-	}
-	if r.now().Sub(last) > r.idleTimeout(m) {
-		return 0
-	}
-	return 1
-}
-
-// idleTimeout is how long an on-demand model stays up unasked-for.
-func (r *Reconciler) idleTimeout(m catalog.Model) time.Duration {
-	if m.Scaling.IdleTimeout > 0 {
-		return m.Scaling.IdleTimeout
-	}
-	return r.cfg.IdleTimeout
-}
-
-// noteUsage keeps an on-demand model alive while it is being used.
-//
-// Without this a model would be evicted exactly IdleTimeout after the request
-// that loaded it, however busy it had been since — the timer would measure
-// time since the first request rather than the last.
-func (r *Reconciler) noteUsage(lastUsed map[string]time.Time) {
-	byModel := map[string]time.Time{}
-	for id, at := range lastUsed {
-		rep, ok := r.replicas[id]
-		if !ok {
-			continue
-		}
-		if at.After(byModel[rep.Model]) {
-			byModel[rep.Model] = at
-		}
-	}
-	for model, at := range byModel {
-		if cur, ok := r.wanted[model]; ok && at.After(cur) {
-			r.wanted[model] = at
-		}
-	}
-}
-
 // expireStuckReplicas fails replicas that never finished coming up.
 //
 // Every other path out of Starting depends on the agent reporting something. A
@@ -844,110 +745,6 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
 	r.retire(ctx, retiringIDs, readyLive, len(liveIDs), desired)
 }
 
-// retire finishes restarting replicas.
-//
-// A retiring replica keeps serving until its replacement is ready, and is then
-// drained like any other. When no replacement could be placed at all —
-// nowhere else has the memory, which on a small CPU fleet is the usual case —
-// it is stopped after DrainGrace so its replacement can take its place: a
-// restart in place, with the model unserved while it loads if it had no other
-// replica. Said in the log, since that is the one case with an outage.
-func (r *Reconciler) retire(ctx context.Context, ids []string, readyLive, live, desired int) {
-	for _, id := range ids {
-		r.mu.Lock()
-		rep, ok := r.replicas[id]
-		var since time.Time
-		if ok {
-			since = rep.RetiringAt
-		}
-		r.mu.Unlock()
-		if !ok {
-			continue
-		}
-		switch {
-		case readyLive >= desired:
-			r.drainForRestart(ctx, id)
-		case live < desired && r.now().Sub(since) > r.cfg.DrainGrace:
-			r.log.Warn("restarting a replica in place: no room for its replacement beside it",
-				"replica", id, "after", r.cfg.DrainGrace)
-			r.drainForRestart(ctx, id)
-			r.stopReplica(ctx, id)
-		}
-	}
-}
-
-func (r *Reconciler) drainForRestart(ctx context.Context, id string) {
-	r.mu.Lock()
-	rep, ok := r.replicas[id]
-	var shards []Shard
-	if ok {
-		rep.State, rep.DrainingAt, rep.RetiringAt = Draining, r.now(), time.Time{}
-		shards = slices.Clone(rep.Shards)
-	}
-	r.mu.Unlock()
-	for _, s := range shards {
-		if err := r.disp.DrainShard(ctx, s.NodeID, s.ID); err != nil {
-			r.log.Warn("draining a shard failed", "shard", s.ID, "err", err)
-		}
-	}
-}
-
-// nextRestartLocked pops the next queued replica of a model that is still
-// ready to be restarted. Called with mu held.
-func (r *Reconciler) nextRestartLocked(model string) (string, bool) {
-	q := r.restartQueue[model]
-	for len(q) > 0 {
-		id := q[0]
-		q = q[1:]
-		if rep, ok := r.replicas[id]; ok && rep.State == Ready && rep.RetiringAt.IsZero() {
-			r.restartQueue[model] = q
-			return id, true
-		}
-	}
-	delete(r.restartQueue, model)
-	return "", false
-}
-
-// ErrNoSuchReplica means no ready replica has that id.
-var ErrNoSuchReplica = errors.New("no ready replica with that id")
-
-// Restart queues replicas to be replaced, each by a new one started beside it
-// before it is stopped — to pick up an edited profile, an engine key, a new
-// engine version. Replicas of one model are replaced one at a time.
-func (r *Reconciler) Restart(ids ...string) error {
-	r.mu.Lock()
-	for _, id := range ids {
-		rep, ok := r.replicas[id]
-		if !ok || rep.State != Ready {
-			r.mu.Unlock()
-			return fmt.Errorf("%w: %s", ErrNoSuchReplica, id)
-		}
-		if !slices.Contains(r.restartQueue[rep.Model], id) && rep.RetiringAt.IsZero() {
-			r.restartQueue[rep.Model] = append(r.restartQueue[rep.Model], id)
-		}
-	}
-	r.mu.Unlock()
-	r.Notify()
-	return nil
-}
-
-// RestartState says where a replica's restart stands: "queued", "replacing",
-// or "" when none was asked for.
-func (r *Reconciler) RestartState(id string) string {
-	r.mu.Lock()
-	defer r.mu.Unlock()
-	rep, ok := r.replicas[id]
-	switch {
-	case !ok:
-		return ""
-	case !rep.RetiringAt.IsZero():
-		return "replacing"
-	case slices.Contains(r.restartQueue[rep.Model], id):
-		return "queued"
-	}
-	return ""
-}
-
 // scaleUp places and starts one replica per pass.
 //
 // One at a time on purpose: placement decisions depend on what is already
@@ -1189,253 +986,6 @@ func (r *Reconciler) stopReplica(ctx context.Context, id string) {
 	r.mu.Unlock()
 }
 
-// Adopt rebuilds replica state from shards an agent reports it is already
-// running (plan §6.5).
-//
-// This is what makes a control plane restart survivable. Without it the
-// reconciler sees an empty cluster, places duplicates, and the engines already
-// running keep holding GPUs the inventory believes are free — so the duplicates
-// fail to allocate, with an error naming the new process rather than the old.
-//
-// Shards whose model or profile is no longer in the catalog are returned for
-// the caller to stop: nothing will ever claim them.
-func (r *Reconciler) Adopt(nodeID string, shards []AdoptedShard) (unclaimed []AdoptedShard) {
-	defer r.forgetShardsTheAgentDoesNotHave(nodeID, shards)
-
-	byReplica := map[string][]AdoptedShard{}
-	for _, s := range shards {
-		byReplica[s.ReplicaID] = append(byReplica[s.ReplicaID], s)
-	}
-
-	r.mu.Lock()
-	defer r.mu.Unlock()
-
-	for replicaID, group := range byReplica {
-		if existing, known := r.replicas[replicaID]; known {
-			// The other half of a replica we are still assembling. Merge rather
-			// than choose: these shards belong to the same replica as the ones
-			// already adopted, they are simply on a different node whose agent
-			// registered a moment later. Treating them as a competing claim is
-			// what tore multi-node replicas apart on every control plane
-			// restart — one half adopted and failed for being short, the other
-			// stopped as unclaimed when it finally arrived.
-			if !existing.AdoptedAt.IsZero() && len(existing.Shards) < existing.WantShards {
-				r.mergeAdopted(existing, group)
-				continue
-			}
-			// Usually this is an agent reconnecting rather than a control plane
-			// restart, and its status reports keep the state current.
-			//
-			// Not always. Replica ids are derived from the model name and a
-			// counter, so a control plane that restarts and places before the
-			// agents reconnect produces a *different* replica under the same
-			// id, on a freshly allocated port. Skipping the adoption then
-			// leaves the cluster pointing at a port nothing listens on, while
-			// the engine that is actually serving is treated as a stray.
-			//
-			// The fleet is authoritative about what is running (plan §6.5), so
-			// what we merely placed loses to what an agent reports — unless our
-			// copy has a shard that has actually come up, in which case two
-			// things really are running and the one we did not ask for is the
-			// stray.
-			if existing.hasLiveShard() {
-				// Only what we do not recognise. A reconnecting agent reports
-				// the shards we already know, and disowning those would tell it
-				// to stop the very replica that is serving.
-				known := map[string]struct{}{}
-				for _, sh := range existing.Shards {
-					known[sh.ID] = struct{}{}
-				}
-				var strays []AdoptedShard
-				for _, s := range group {
-					if _, ok := known[s.ShardID]; !ok {
-						strays = append(strays, s)
-					}
-				}
-				unclaimed = append(unclaimed, strays...)
-				r.reservePorts(strays)
-				continue
-			}
-			r.log.Warn("a running replica reported by the fleet replaces one we placed",
-				"replica", replicaID, "placed_endpoint", existing.Endpoint)
-			for _, s := range existing.Shards {
-				r.inv.Release(s.ID)
-			}
-			delete(r.replicas, replicaID)
-		}
-
-		first := group[0]
-		m, ok := r.cat.Load().Resolve(first.Model)
-		if !ok {
-			unclaimed = append(unclaimed, group...)
-			r.reservePorts(group)
-			continue
-		}
-		prof, ok := m.ProfileByName(first.Profile)
-		if !ok {
-			unclaimed = append(unclaimed, group...)
-			r.reservePorts(group)
-			continue
-		}
-
-		rep := &Replica{
-			ID: replicaID, Model: m.Name, Profile: prof.Name,
-			Fabric: first.Fabric, Pool: first.Pool,
-			// From the agent, which was told it when the shard started. Empty
-			// from an agent too old to report it, which reads as unknown.
-			ProfileFP: first.ProfileFP,
-			Weight:    prof.Weight, CreatedAt: r.now(),
-			Endpoint: first.Endpoint,
-			// From the profile, not from what was reported: the point is to
-			// notice when what was reported is short.
-			WantShards: max(prof.Nodes, 1),
-			AdoptedAt:  r.now(),
-		}
-		r.mergeAdopted(rep, group)
-		r.replicas[replicaID] = rep
-
-	}
-	return unclaimed
-}
-
-// mergeAdopted folds shards an agent reports into a replica being reassembled.
-//
-// Called both for the first group, which creates the replica, and for every
-// later group belonging to it, so a replica spanning N nodes is built up over N
-// registrations. Shard ids already present are skipped: an agent that
-// reconnects repeats what it is running, and counting those again would carry a
-// replica past its shard count without a second node ever appearing.
-func (r *Reconciler) mergeAdopted(rep *Replica, group []AdoptedShard) {
-	have := make(map[string]struct{}, len(rep.Shards))
-	for _, sh := range rep.Shards {
-		have[sh.ID] = struct{}{}
-	}
-	for _, s := range group {
-		if _, dup := have[s.ShardID]; dup {
-			continue
-		}
-		have[s.ShardID] = struct{}{}
-		rep.Shards = append(rep.Shards, Shard{
-			ID: s.ShardID, NodeID: s.NodeID, Rank: s.Rank,
-			GPUIndices: s.GPUIndices, HCAs: s.HCAs,
-			HTTPPort: s.HTTPPort, State: s.State,
-		})
-		// Reclaim the GPUs and the memory, or the next placement hands them
-		// out twice. The memory comes from the catalog rather than the agent:
-		// what a profile asked for is not something a running process reports.
-		r.inv.Allocate(s.NodeID, inventory.Allocation{
-			ShardID: s.ShardID, ReplicaID: rep.ID, GPUIndices: s.GPUIndices,
-			MemoryGB: r.profileMemory(rep.Model, rep.Profile),
-		})
-		// The endpoint comes from the leader, which may be in any of the
-		// groups — the rank 0 node is not necessarily the first to register.
-		if s.Rank == 0 && s.Endpoint != "" {
-			rep.Endpoint = s.Endpoint
-		}
-	}
-	slices.SortFunc(rep.Shards, func(a, b Shard) int { return a.Rank - b.Rank })
-	rep.deriveState(r.now())
-
-	r.log.Info("adopted replica from the fleet",
-		"replica", rep.ID, "model", rep.Model,
-		"shards", len(rep.Shards), "want", rep.WantShards, "state", rep.State)
-}
-
-// forgetShardsTheAgentDoesNotHave fails shards we believe are on a node that the
-// agent, having just told us everything it is running, did not mention.
-//
-// The mirror image of adoption, and the half that was missing. Adoption handles
-// "the agent has something we do not know about"; this handles "we think the
-// agent has something it does not". Both follow from the same rule: the fleet is
-// authoritative about what is running (plan §6.5).
-//
-// Without it, an engine that dies while its agent survives leaves the control
-// plane believing a replica is Ready forever. Observed on a real cluster: the
-// engine was killed, the agent restarted and registered with nothing running,
-// and the replica stayed Ready pointing at a dead port for as long as anyone
-// watched. Traffic survived — the breaker took the endpoint out — but the
-// cluster ran a third short with no sign of it.
-//
-// Safe against the obvious race. A registration follows a fresh connection, so
-// anything we dispatched over a previous one is not running by definition;
-// there is no window where a shard is legitimately started but unreported.
-func (r *Reconciler) forgetShardsTheAgentDoesNotHave(nodeID string, running []AdoptedShard) {
-	if nodeID == "" {
-		return
-	}
-	reported := make(map[string]struct{}, len(running))
-	for _, s := range running {
-		reported[s.ShardID] = struct{}{}
-	}
-
-	var lost []string
-	r.mu.Lock()
-	for _, rep := range r.replicas {
-		if rep.State == Stopped {
-			continue
-		}
-		changed := false
-		for i := range rep.Shards {
-			sh := &rep.Shards[i]
-			if sh.NodeID != nodeID || sh.State.Terminal() {
-				continue
-			}
-			if _, ok := reported[sh.ID]; ok {
-				continue
-			}
-			sh.State = supervisor.Failed
-			sh.Err = "the agent reports it is not running"
-			r.inv.Release(sh.ID)
-			changed = true
-			lost = append(lost, sh.ID)
-		}
-		if changed {
-			rep.deriveState(r.now())
-		}
-	}
-	r.mu.Unlock()
-
-	for _, id := range lost {
-		r.log.Warn("a shard we believed was running is not; the agent does not have it",
-			"node", nodeID, "shard", id)
-	}
-}
-
-// reservePorts records ports held by shards nothing owns.
-//
-// They are being stopped, but stopping is not instant and the engine keeps its
-// socket until it exits. Handing the port to a new shard in the meantime makes
-// that shard fail to bind, which fails its replica, which places another one on
-// the same port — a loop that does not end on its own.
-//
-// Called with the lock held.
-func (r *Reconciler) reservePorts(shards []AdoptedShard) {
-	for _, s := range shards {
-		if s.HTTPPort > 0 {
-			r.reservedPorts[s.HTTPPort] = struct{}{}
-		}
-	}
-}
-
-// AdoptedShard is one running shard an agent reported at registration.
-type AdoptedShard struct {
-	NodeID     string
-	ReplicaID  string
-	ShardID    string
-	Rank       int
-	GPUIndices []int
-	HCAs       []string
-	HTTPPort   int
-	State      supervisor.State
-	Model      string
-	Profile    string
-	Pool       string
-	Fabric     string
-	ProfileFP  string
-	// Endpoint is the leader's base URL, set on the rank 0 shard.
-	Endpoint string
-}
-
 // OnFetchProgress records how far a shard's download has got.
 //
 // Separate from OnShardStatus because it is not a transition: the shard is
@@ -1576,142 +1126,3 @@ func (r *Reconciler) engineKey(replicaID string) string {
 	}
 	return r.cfg.EngineKey(replicaID)
 }
-
-// parkRecord is why a model stopped being retried.
-type parkRecord struct {
-	Since    time.Time
-	Reason   string
-	Failures int
-	// fp is the model's definition when it was parked; a different one
-	// unparks it.
-	fp string
-}
-
-type placementNote struct {
-	text string
-	at   time.Time
-}
-
-// modelFingerprint identifies a model's definition, so a change to it can be
-// told from a retry of the same thing.
-func modelFingerprint(m catalog.Model) string {
-	b, _ := json.Marshal(m)
-	sum := sha256.Sum256(b)
-	return hex.EncodeToString(sum[:8])
-}
-
-// placementLogEvery is how often an unchanged placement failure is repeated.
-const placementLogEvery = 10 * time.Minute
-
-// notePlacementFailure logs why a model could not be placed, when the reason
-// is new or has not been said for a while. Said on every pass it was 250
-// identical lines in two days for one model, burying everything else.
-func (r *Reconciler) notePlacementFailure(model, why string) {
-	r.mu.Lock()
-	last := r.placementLogged[model]
-	say := last.text != why || r.now().Sub(last.at) >= placementLogEvery
-	if say {
-		r.placementLogged[model] = placementNote{text: why, at: r.now()}
-	}
-	r.mu.Unlock()
-	if say {
-		r.log.Warn("no placement for model", "model", model, "reasons", why)
-	}
-}
-
-// Retry clears a parked model and its backoff, so it is placed on the next
-// pass.
-func (r *Reconciler) Retry(model string) bool {
-	r.mu.Lock()
-	_, was := r.parked[model]
-	delete(r.parked, model)
-	r.attempts[model] = 0
-	delete(r.nextAttempt, model)
-	r.mu.Unlock()
-	r.Notify()
-	return was
-}
-
-// ModelHealth is how one model stands, for status and metrics.
-type ModelHealth struct {
-	Model string
-	// Desired is how many replicas it should have now; 0 for a stopped model
-	// or an on-demand one nobody is asking for.
-	Desired int
-	Ready   int
-	// Failures counts replicas that failed in a row.
-	Failures int
-	// Parked is set when Failures reached ParkAfter; Reason is the last error.
-	Parked      bool
-	ParkedSince time.Time
-	Reason      string
-	// Unplaceable is set when it is short of replicas and the last attempt to
-	// place one found nowhere; Placement says why.
-	Unplaceable bool
-	Placement   string
-}
-
-// ModelHealth reports every model in the catalog.
-func (r *Reconciler) ModelHealth() []ModelHealth {
-	models := r.cat.Load().Models()
-	desired := map[string]int{}
-	for _, m := range models {
-		desired[m.Name] = r.desiredFor(m)
-	}
-	r.mu.Lock()
-	defer r.mu.Unlock()
-	ready, live := map[string]int{}, map[string]int{}
-	for _, rep := range r.replicas {
-		if rep.State == Ready {
-			ready[rep.Model]++
-		}
-		if rep.State.Counts() {
-			live[rep.Model]++
-		}
-	}
-	out := make([]ModelHealth, 0, len(models))
-	for _, m := range models {
-		h := ModelHealth{Model: m.Name, Desired: desired[m.Name], Ready: ready[m.Name],
-			Failures: r.attempts[m.Name]}
-		if p, ok := r.parked[m.Name]; ok {
-			h.Parked, h.ParkedSince, h.Reason = true, p.Since, p.Reason
-		}
-		if res, ok := r.lastExplain[m.Name]; ok && res.Placement == nil && live[m.Name] < h.Desired {
-			h.Unplaceable, h.Placement = true, res.Explain()
-		}
-		out = append(out, h)
-	}
-	return out
-}
-
-// Unservable says why a model cannot be served at all, so a request for it is
-// told now rather than asked to retry something that will never come up.
-func (r *Reconciler) Unservable(model string) (string, bool) {
-	m, ok := r.cat.Load().Resolve(model)
-	if !ok {
-		return "", false
-	}
-	r.mu.Lock()
-	defer r.mu.Unlock()
-	if p, ok := r.parked[m.Name]; ok {
-		return fmt.Sprintf("%s failed to start %d times in a row and is not being retried: %s",
-			m.Name, p.Failures, p.Reason), true
-	}
-	for _, rep := range r.replicas {
-		if rep.Model == m.Name && rep.State.Counts() {
-			return "", false // something is on its way
-		}
-	}
-	if res, ok := r.lastExplain[m.Name]; ok && res.Placement == nil {
-		return m.Name + " cannot be placed on any node: " + firstLine(res.Explain()), true
-	}
-	return "", false
-}
-
-func firstLine(s string) string {
-	s = strings.TrimSpace(s)
-	if i := strings.IndexByte(s, '\n'); i >= 0 {
-		return strings.TrimSpace(s[:i]) + " (and others; see llmgw ctl explain-placement)"
-	}
-	return s
-}

internal/control/reconciler/reconciler_test.go

@@ -1932,3 +1932,74 @@ func TestAPlacementFailureIsLoggedWhenItChangesNotEveryPass(t *testing.T) {
 		t.Errorf("health %+v", mh)
 	}
 }
+
+func liveCount(h *harness) int {
+	n := 0
+	for _, r := range h.r.Replicas() {
+		if r.State.Counts() {
+			n++
+		}
+	}
+	return n
+}
+
+func TestLoadAddsReplicasUpToMaxAndQuietTakesThemBack(t *testing.T) {
+	m := testModel(1)
+	m.Scaling.Max = 3
+	h := newHarness(t, m, gpuNode("1", 8), gpuNode("2", 8), gpuNode("3", 8))
+	h.r.Reconcile(t.Context())
+	h.markReady()
+	busy := map[string]Load{"llama": {Replicas: 1, Running: 4, Waiting: 3}}
+
+	// A spike shorter than ScaleUpAfter adds nothing.
+	h.r.RecordLoad(busy)
+	h.advance(30 * time.Second)
+	h.r.RecordLoad(busy)
+	h.r.Reconcile(t.Context())
+	if n := liveCount(h); n != 1 {
+		t.Fatalf("%d replicas after 30s of load", n)
+	}
+
+	// Held past it, one more.
+	h.advance(31 * time.Second)
+	h.r.RecordLoad(busy)
+	h.r.Reconcile(t.Context())
+	if n := liveCount(h); n != 2 {
+		t.Fatalf("%d replicas after a minute of load; expected 2", n)
+	}
+
+	// Not another while that one is still starting.
+	h.advance(2 * time.Minute)
+	h.r.RecordLoad(busy)
+	h.r.Reconcile(t.Context())
+	if n := liveCount(h); n != 2 {
+		t.Fatalf("stacked a replica on one still starting: %d", n)
+	}
+
+	// Once it is up and load holds, the third — and never a fourth.
+	h.markReady()
+	for i := 0; i < 5; i++ {
+		h.r.RecordLoad(busy)
+		h.advance(61 * time.Second)
+		h.r.RecordLoad(busy)
+		h.r.Reconcile(t.Context())
+		h.markReady()
+	}
+	if n := liveCount(h); n != 3 {
+		t.Fatalf("%d replicas under sustained load; max is 3", n)
+	}
+
+	// Quiet: back down one step per ScaleDownAfter, to min and no further.
+	quiet := map[string]Load{"llama": {Replicas: 3}}
+	for i := 0; i < 6; i++ {
+		h.r.RecordLoad(quiet)
+		h.advance(DefaultConfig().ScaleDownAfter + time.Second)
+		h.r.RecordLoad(quiet)
+		for j := 0; j < 3; j++ {
+			h.r.Reconcile(t.Context())
+		}
+	}
+	if n := liveCount(h); n != 1 {
+		t.Errorf("%d replicas after a long quiet spell; min is 1", n)
+	}
+}

internal/control/reconciler/restart.go

@@ -0,0 +1,113 @@
+package reconciler
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"slices"
+	"time"
+)
+
+// retire finishes restarting replicas.
+//
+// A retiring replica keeps serving until its replacement is ready, and is then
+// drained like any other. When no replacement could be placed at all —
+// nowhere else has the memory, which on a small CPU fleet is the usual case —
+// it is stopped after DrainGrace so its replacement can take its place: a
+// restart in place, with the model unserved while it loads if it had no other
+// replica. Said in the log, since that is the one case with an outage.
+func (r *Reconciler) retire(ctx context.Context, ids []string, readyLive, live, desired int) {
+	for _, id := range ids {
+		r.mu.Lock()
+		rep, ok := r.replicas[id]
+		var since time.Time
+		if ok {
+			since = rep.RetiringAt
+		}
+		r.mu.Unlock()
+		if !ok {
+			continue
+		}
+		switch {
+		case readyLive >= desired:
+			r.drainForRestart(ctx, id)
+		case live < desired && r.now().Sub(since) > r.cfg.DrainGrace:
+			r.log.Warn("restarting a replica in place: no room for its replacement beside it",
+				"replica", id, "after", r.cfg.DrainGrace)
+			r.drainForRestart(ctx, id)
+			r.stopReplica(ctx, id)
+		}
+	}
+}
+
+func (r *Reconciler) drainForRestart(ctx context.Context, id string) {
+	r.mu.Lock()
+	rep, ok := r.replicas[id]
+	var shards []Shard
+	if ok {
+		rep.State, rep.DrainingAt, rep.RetiringAt = Draining, r.now(), time.Time{}
+		shards = slices.Clone(rep.Shards)
+	}
+	r.mu.Unlock()
+	for _, s := range shards {
+		if err := r.disp.DrainShard(ctx, s.NodeID, s.ID); err != nil {
+			r.log.Warn("draining a shard failed", "shard", s.ID, "err", err)
+		}
+	}
+}
+
+// nextRestartLocked pops the next queued replica of a model that is still
+// ready to be restarted. Called with mu held.
+func (r *Reconciler) nextRestartLocked(model string) (string, bool) {
+	q := r.restartQueue[model]
+	for len(q) > 0 {
+		id := q[0]
+		q = q[1:]
+		if rep, ok := r.replicas[id]; ok && rep.State == Ready && rep.RetiringAt.IsZero() {
+			r.restartQueue[model] = q
+			return id, true
+		}
+	}
+	delete(r.restartQueue, model)
+	return "", false
+}
+
+// ErrNoSuchReplica means no ready replica has that id.
+var ErrNoSuchReplica = errors.New("no ready replica with that id")
+
+// Restart queues replicas to be replaced, each by a new one started beside it
+// before it is stopped — to pick up an edited profile, an engine key, a new
+// engine version. Replicas of one model are replaced one at a time.
+func (r *Reconciler) Restart(ids ...string) error {
+	r.mu.Lock()
+	for _, id := range ids {
+		rep, ok := r.replicas[id]
+		if !ok || rep.State != Ready {
+			r.mu.Unlock()
+			return fmt.Errorf("%w: %s", ErrNoSuchReplica, id)
+		}
+		if !slices.Contains(r.restartQueue[rep.Model], id) && rep.RetiringAt.IsZero() {
+			r.restartQueue[rep.Model] = append(r.restartQueue[rep.Model], id)
+		}
+	}
+	r.mu.Unlock()
+	r.Notify()
+	return nil
+}
+
+// RestartState says where a replica's restart stands: "queued", "replacing",
+// or "" when none was asked for.
+func (r *Reconciler) RestartState(id string) string {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	rep, ok := r.replicas[id]
+	switch {
+	case !ok:
+		return ""
+	case !rep.RetiringAt.IsZero():
+		return "replacing"
+	case slices.Contains(r.restartQueue[rep.Model], id):
+		return "queued"
+	}
+	return ""
+}

internal/docgen/render.go

@@ -14,7 +14,7 @@ import (
 // a new configuration file is a failure here rather than a silent omission.
 var Sources = []string{
 	"internal/config/config.go",
-	"internal/agent/agent.go",
+	"internal/agent/config.go",
 	"internal/agent/discovery/discover.go",
 	"internal/agent/engines/engine.go",
 	"internal/agent/engines/llamacpp.go",

internal/gguf/gguf.go

@@ -25,6 +25,59 @@ type Header struct {
 	Version      uint32
 	Architecture string
 	Name         string
+
+	// The shape the KV cache is sized from, from the architecture's own keys
+	// (llama.block_count, qwen2.attention.head_count_kv, ...). Zero when the
+	// file does not say, which a caller has to treat as unknown.
+	BlockCount      uint64
+	HeadCount       uint64
+	HeadCountKV     uint64
+	EmbeddingLength uint64
+	// KeyLength and ValueLength are the per-head sizes when the file gives
+	// them; otherwise a head is EmbeddingLength / HeadCount.
+	KeyLength   uint64
+	ValueLength uint64
+}
+
+// KVBytesPerToken is what one token of context costs in llama.cpp's default
+// f16 cache, or 0 when the header did not give the shape.
+//
+// Keys and values, per layer, per KV head — the KV heads rather than the
+// attention heads, because with grouped-query attention a model with forty
+// heads may keep only eight, and the difference is a factor of five.
+func (h Header) KVBytesPerToken() int64 {
+	kvHeads := h.HeadCountKV
+	if kvHeads == 0 {
+		kvHeads = h.HeadCount
+	}
+	if h.BlockCount == 0 || kvHeads == 0 {
+		return 0
+	}
+	k, v := h.KeyLength, h.ValueLength
+	if k == 0 && h.HeadCount > 0 {
+		k = h.EmbeddingLength / h.HeadCount
+	}
+	if v == 0 {
+		v = k
+	}
+	if k == 0 {
+		return 0
+	}
+	const f16 = 2
+	return int64(h.BlockCount * kvHeads * (k + v) * f16) //nolint:gosec // header-sized numbers
+}
+
+// MemoryBytes is what one llama.cpp server needs for this model: the weights,
+// a KV cache for a context pool of ctxPool tokens (llama.cpp's --ctx-size,
+// which --parallel divides among its slots rather than multiplies), and room
+// for compute buffers and the server itself. 0 when the shape is unknown.
+func (h Header) MemoryBytes(weights int64, ctxPool int) int64 {
+	perToken := h.KVBytesPerToken()
+	if perToken == 0 || weights <= 0 {
+		return 0
+	}
+	overhead := int64(1<<30) + weights/20
+	return weights + int64(ctxPool)*perToken + overhead
 }
 
 // ErrNotGGUF means the bytes are not a GGUF file.
@@ -87,16 +140,42 @@ func Read(r io.Reader) (Header, error) {
 		_, err := io.CopyN(io.Discard, br, n)
 		return err
 	}
+	// num holds the last integer value read, for the shape keys.
+	var num uint64
+	readUint := func(size int) error {
+		b := make([]byte, size)
+		if _, err := io.ReadFull(br, b); err != nil {
+			return err
+		}
+		switch size {
+		case 1:
+			num = uint64(b[0])
+		case 2:
+			num = uint64(binary.LittleEndian.Uint16(b))
+		case 4:
+			num = uint64(binary.LittleEndian.Uint32(b))
+		default:
+			num = binary.LittleEndian.Uint64(b)
+		}
+		return nil
+	}
 	var value func(t uint32) (string, error)
 	value = func(t uint32) (string, error) {
+		num = 0
 		switch t {
-		case tUint8, tInt8, tBool:
+		case tUint8, tInt8:
+			return "", readUint(1)
+		case tBool:
 			return "", skip(1)
 		case tUint16, tInt16:
-			return "", skip(2)
-		case tUint32, tInt32, tFloat32:
+			return "", readUint(2)
+		case tUint32, tInt32:
+			return "", readUint(4)
+		case tFloat32:
 			return "", skip(4)
-		case tUint64, tInt64, tFloat64:
+		case tUint64, tInt64:
+			return "", readUint(8)
+		case tFloat64:
 			return "", skip(8)
 		case tString:
 			return str()
@@ -129,18 +208,27 @@ func Read(r io.Reader) (Header, error) {
 	if err != nil {
 		return h, err
 	}
+	// Once the architecture is known, running out of the fetched bytes is
+	// not a failure: what the caller came for is in hand, and the shape keys
+	// are a bonus that a file with its tokenizer first may not reach.
+	partial := func(err error) (Header, error) {
+		if h.Architecture != "" {
+			return h, nil
+		}
+		return h, err
+	}
 	for i := uint64(0); i < kvs; i++ {
 		key, err := str()
 		if err != nil {
-			return h, err
+			return partial(err)
 		}
 		var t uint32
 		if err := binary.Read(br, binary.LittleEndian, &t); err != nil {
-			return h, err
+			return partial(err)
 		}
 		v, err := value(t)
 		if err != nil {
-			return h, err
+			return partial(err)
 		}
 		switch key {
 		case "general.architecture":
@@ -148,7 +236,25 @@ func Read(r io.Reader) (Header, error) {
 		case "general.name":
 			h.Name = v
 		}
-		if h.Architecture != "" && h.Name != "" {
+		if h.Architecture != "" {
+			switch strings.TrimPrefix(key, h.Architecture+".") {
+			case "block_count":
+				h.BlockCount = num
+			case "attention.head_count":
+				h.HeadCount = num
+			case "attention.head_count_kv":
+				h.HeadCountKV = num
+			case "embedding_length":
+				h.EmbeddingLength = num
+			case "attention.key_length":
+				h.KeyLength = num
+			case "attention.value_length":
+				h.ValueLength = num
+			}
+		}
+		// The tokenizer's arrays — megabytes of vocabulary — come after the
+		// architecture's keys, and nothing past them is wanted.
+		if strings.HasPrefix(key, "tokenizer.") && h.BlockCount > 0 {
 			break
 		}
 	}
@@ -209,5 +315,7 @@ func Check(ctx context.Context, url, llamaVersion string) Verdict {
 	if err != nil {
 		return Verdict{Servable: true, Reason: "its header could not be read: " + err.Error()}
 	}
-	return LlamaCpp(llamaVersion, h.Architecture)
+	v := LlamaCpp(llamaVersion, h.Architecture)
+	v.Header = h
+	return v
 }

internal/gguf/gguf_test.go

@@ -92,3 +92,54 @@ func TestLlamaCppKnowsWhatItCanLoad(t *testing.T) {
 }
 
 var timeZero = time.Time{}
+
+// shapedHeader is a header with the keys a KV cache is sized from, written
+// with the integer types llama.cpp uses for them.
+func shapedHeader(arch string, layers, heads, kvHeads, embd uint32, truncateTokenizer bool) []byte {
+	var b bytes.Buffer
+	w := func(v any) { _ = binary.Write(&b, binary.LittleEndian, v) }
+	str := func(s string) { w(uint64(len(s))); b.WriteString(s) }
+	u32 := func(k string, v uint32) { str(k); w(uint32(tUint32)); w(v) }
+	b.WriteString("GGUF")
+	w(uint32(3))
+	w(uint64(0))
+	w(uint64(7))
+	str("general.architecture")
+	w(uint32(tString))
+	str(arch)
+	u32(arch+".block_count", layers)
+	u32(arch+".attention.head_count", heads)
+	u32(arch+".attention.head_count_kv", kvHeads)
+	u32(arch+".embedding_length", embd)
+	u32(arch+".context_length", 32768)
+	str("tokenizer.ggml.tokens")
+	w(uint32(tArray))
+	w(uint32(tString))
+	w(uint64(1 << 20))
+	if !truncateTokenizer {
+		for i := 0; i < 1<<20; i++ {
+			str("t")
+		}
+	}
+	return b.Bytes()
+}
+
+func TestTheKVCacheIsSizedFromTheModelsShape(t *testing.T) {
+	// Qwen2.5-14B: 48 layers, 40 heads of which 8 are kept for K and V,
+	// 5120 wide — 196608 bytes a token in f16, as read from the live mirror.
+	h, err := Read(bytes.NewReader(shapedHeader("qwen2", 48, 40, 8, 5120, true)))
+	if err != nil {
+		t.Fatalf("a header cut off in its tokenizer failed to read: %v", err)
+	}
+	if got := h.KVBytesPerToken(); got != 196608 {
+		t.Errorf("%d bytes a token, expected 196608", got)
+	}
+	weights := int64(9 << 30)
+	want := weights + 16384*196608 + (1<<30 + weights/20)
+	if got := h.MemoryBytes(weights, 16384); got != want {
+		t.Errorf("memory %d, expected %d", got, want)
+	}
+	if (Header{}).MemoryBytes(weights, 16384) != 0 {
+		t.Error("a header with no shape produced a figure")
+	}
+}

internal/gguf/llamacpp.go

@@ -180,6 +180,9 @@ type Verdict struct {
 	// checked.
 	Checked bool
 	Reason  string
+	// Header is what was read, zero when nothing was; its shape sizes the
+	// model's memory.
+	Header Header
 }
 
 // LlamaCpp judges an architecture against a llama.cpp build.

internal/gguf/remote_test.go

@@ -0,0 +1,30 @@
+package gguf
+
+import (
+	"os"
+	"strings"
+	"testing"
+)
+
+// TestRealHeaders reads real files when LLMGW_GGUF_URLS names them (space
+// separated), and prints the shape and memory estimate for a 16384-token pool.
+// Skipped otherwise: CI has no mirror to read.
+func TestRealHeaders(t *testing.T) {
+	urls := strings.Fields(os.Getenv("LLMGW_GGUF_URLS"))
+	if len(urls) == 0 {
+		t.Skip("LLMGW_GGUF_URLS not set")
+	}
+	for _, u := range urls {
+		h, err := FromURL(t.Context(), nil, u)
+		if err != nil {
+			t.Errorf("%s: %v", u, err)
+			continue
+		}
+		t.Logf("%s: arch=%s layers=%d heads=%d kv_heads=%d embd=%d key_len=%d kv/token=%d B, kv@16k=%.2f GiB",
+			u[strings.LastIndex(u, "/")+1:], h.Architecture, h.BlockCount, h.HeadCount, h.HeadCountKV,
+			h.EmbeddingLength, h.KeyLength, h.KVBytesPerToken(), float64(16384*h.KVBytesPerToken())/(1<<30))
+		if h.KVBytesPerToken() == 0 {
+			t.Errorf("%s: no KV shape read", u)
+		}
+	}
+}

internal/hf/hf.go

@@ -558,3 +558,16 @@ func Group(files []File) []FileSet {
 	}
 	return out
 }
+
+// ModelName is the catalog name a repository suggests: its last element,
+// lowercased, with the -GGUF marker dropped. The owner is not part of what
+// anybody calls the model, and "-GGUF" describes the file format.
+func ModelName(repoID string) string {
+	name := repoID
+	if i := strings.LastIndex(name, "/"); i >= 0 {
+		name = name[i+1:]
+	}
+	name = strings.TrimSuffix(name, "-GGUF")
+	name = strings.TrimSuffix(name, "-gguf")
+	return strings.ToLower(name)
+}

internal/obs/trace.go

@@ -0,0 +1,72 @@
+package obs
+
+import (
+	"context"
+	"fmt"
+	"net/url"
+	"os"
+	"time"
+
+	"go.opentelemetry.io/otel"
+	"go.opentelemetry.io/otel/attribute"
+	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
+	"go.opentelemetry.io/otel/propagation"
+	"go.opentelemetry.io/otel/sdk/resource"
+	sdktrace "go.opentelemetry.io/otel/sdk/trace"
+)
+
+// TracingOptions says where traces go.
+type TracingOptions struct {
+	Endpoint    string  // http(s)://host:4318; empty reads OTEL_EXPORTER_OTLP_ENDPOINT
+	SampleRatio float64 // of traces the caller did not already decide; zero takes 1
+	Service     string
+	Version     string
+}
+
+// SetupTracing installs the global tracer provider and the W3C propagator,
+// and returns what flushes it on the way out. With no endpoint it installs
+// nothing, and every span is the no-op one: tracing that is off costs a
+// function call per span.
+func SetupTracing(ctx context.Context, o TracingOptions) (func(context.Context) error, error) {
+	noop := func(context.Context) error { return nil }
+	endpoint := o.Endpoint
+	if endpoint == "" {
+		endpoint = os.Getenv("OTEL_EXPORTER_OTLP_ENDPOINT")
+	}
+	if endpoint == "" {
+		return noop, nil
+	}
+	u, err := url.Parse(endpoint)
+	if err != nil || (u.Scheme != "http" && u.Scheme != "https") || u.Host == "" {
+		return noop, fmt.Errorf("tracing endpoint %q is not an http(s) address", endpoint)
+	}
+	opts := []otlptracehttp.Option{otlptracehttp.WithEndpoint(u.Host)}
+	if u.Scheme == "http" {
+		opts = append(opts, otlptracehttp.WithInsecure())
+	}
+	if u.Path != "" && u.Path != "/" {
+		opts = append(opts, otlptracehttp.WithURLPath(u.Path))
+	}
+	exp, err := otlptracehttp.New(ctx, opts...)
+	if err != nil {
+		return noop, err
+	}
+	ratio := o.SampleRatio
+	if ratio <= 0 || ratio > 1 {
+		ratio = 1
+	}
+	res := resource.NewSchemaless(
+		attribute.String("service.name", o.Service),
+		attribute.String("service.version", o.Version),
+	)
+	tp := sdktrace.NewTracerProvider(
+		sdktrace.WithBatcher(exp, sdktrace.WithBatchTimeout(5*time.Second)),
+		sdktrace.WithResource(res),
+		// A caller that sent a sampled traceparent has already decided; the
+		// ratio is for requests that arrive without one.
+		sdktrace.WithSampler(sdktrace.ParentBased(sdktrace.TraceIDRatioBased(ratio))),
+	)
+	otel.SetTracerProvider(tp)
+	otel.SetTextMapPropagator(propagation.TraceContext{})
+	return tp.Shutdown, nil
+}

internal/obs/trace_test.go

@@ -0,0 +1,19 @@
+package obs
+
+import (
+	"context"
+	"testing"
+)
+
+func TestTracingIsOffUntilGivenSomewhereToSend(t *testing.T) {
+	t.Setenv("OTEL_EXPORTER_OTLP_ENDPOINT", "")
+	stop, err := SetupTracing(context.Background(), TracingOptions{Service: "t"})
+	if err != nil || stop(context.Background()) != nil {
+		t.Fatal(err)
+	}
+	for _, bad := range []string{"otel:4318", "ftp://otel:4318", "http://"} {
+		if _, err := SetupTracing(context.Background(), TracingOptions{Endpoint: bad}); err == nil {
+			t.Errorf("%q accepted", bad)
+		}
+	}
+}

internal/onboard/onboard.go

@@ -18,6 +18,7 @@ import (
 	"encoding/pem"
 	"errors"
 	"fmt"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/release"
 	"net"
 	"os"
 	"path/filepath"
@@ -335,7 +336,7 @@ func certNames(path string) ([]string, error) {
 // releaseTag is a version CI published: v0.10.4, or a release candidate of
 // it, v0.10.4-rc.2. Anything else — a build with commits past its tag — has no
 // release to download.
-var releaseTag = regexp.MustCompile(`^v\d+\.\d+\.\d+(-rc\.\d+)?$`)
+var releaseTag = release.Tag
 
 func (r *Runner) binary(ctx context.Context, res *Result) error {
 	v := r.Opts.Version
@@ -497,14 +498,17 @@ func (r *Runner) certificate(ctx context.Context, res *Result) error {
 			return err
 		}
 	}
-	// Trusting the root too, so other tools on the machine do not treat the
-	// cluster's certificates as strangers. Best effort: either family's way.
-	_, _ = r.Remote.Run(ctx, `if [ -d /usr/local/share/ca-certificates ]; then
-  sudo install -m 0644 /etc/llmgw/ca.pem /usr/local/share/ca-certificates/llmgw-ca.crt && sudo update-ca-certificates >/dev/null 2>&1
-elif [ -d /etc/pki/ca-trust/source/anchors ]; then
-  sudo install -m 0644 /etc/llmgw/ca.pem /etc/pki/ca-trust/source/anchors/llmgw-ca.pem && sudo update-ca-trust >/dev/null 2>&1
+	// The agent trusts the cluster's root through its own configuration only.
+	// Not system-wide: that root signs whatever the control plane asks it to,
+	// and every TLS client on the machine would then take it for any site.
+	// A node onboarded by an older version had it added; take it back out.
+	_, _ = r.Remote.Run(ctx, `if [ -f /usr/local/share/ca-certificates/llmgw-ca.crt ]; then
+  sudo rm -f /usr/local/share/ca-certificates/llmgw-ca.crt && sudo update-ca-certificates --fresh >/dev/null 2>&1
+fi
+if [ -f /etc/pki/ca-trust/source/anchors/llmgw-ca.pem ]; then
+  sudo rm -f /etc/pki/ca-trust/source/anchors/llmgw-ca.pem && sudo update-ca-trust >/dev/null 2>&1
 fi; true`, nil)
-	r.emit(StepCert, Done, "certificate for "+res.Node+" installed, the cluster's root trusted", "")
+	r.emit(StepCert, Done, "certificate for "+res.Node+" installed", "")
 	return nil
 }
 

internal/release/release.go

@@ -0,0 +1,148 @@
+// Package release fetches and installs llmgw's own release binaries.
+//
+// One implementation for everywhere a binary is replaced — the control plane
+// upgrading itself, and an agent told to upgrade by it — so the checks are
+// the same on both: the release's own checksum is verified before anything is
+// installed, the old binary is kept beside the new one, and the swap is a
+// rename, so a machine is never left without a binary that runs.
+package release
+
+import (
+	"bufio"
+	"context"
+	"crypto/sha256"
+	"encoding/hex"
+	"fmt"
+	"io"
+	"net/http"
+	"os"
+	"path/filepath"
+	"regexp"
+	"strings"
+	"time"
+)
+
+// DefaultBase is where CI publishes releases.
+const DefaultBase = "https://gitea.mynodes.xyz/ilker/llmgw/releases/download"
+
+// Tag is a version CI published: v0.10.4, or a release candidate v0.10.4-rc.2.
+var Tag = regexp.MustCompile(`^v\d+\.\d+\.\d+(-rc\.\d+)?$`)
+
+// AssetName is the release file for a platform.
+func AssetName(version, goos, goarch string) string {
+	return "llmgw-" + version + "-" + goos + "-" + goarch
+}
+
+// Install fetches version for this platform from base, checks it against the
+// release's checksums, and puts it at target, keeping the old one as
+// target.bak-pre-<version>. It returns the backup's path.
+func Install(ctx context.Context, client *http.Client, base, version, goos, goarch, target string) (string, error) {
+	if !Tag.MatchString(version) {
+		return "", fmt.Errorf("%q is not a release version", version)
+	}
+	if client == nil {
+		client = &http.Client{Timeout: 10 * time.Minute}
+	}
+	target, err := filepath.EvalSymlinks(target)
+	if err != nil {
+		return "", err
+	}
+	dir := filepath.Dir(target)
+	asset := AssetName(version, goos, goarch)
+	root := strings.TrimSuffix(base, "/") + "/" + version + "/"
+
+	want, err := checksum(ctx, client, root+"checksums.txt", asset)
+	if err != nil {
+		return "", err
+	}
+	// Beside the target, so the final rename is on one filesystem and atomic.
+	tmp, err := os.CreateTemp(dir, ".llmgw-"+version+"-*")
+	if err != nil {
+		return "", err
+	}
+	defer os.Remove(tmp.Name())
+	got, err := download(ctx, client, root+asset, tmp)
+	if cerr := tmp.Close(); err == nil {
+		err = cerr
+	}
+	if err != nil {
+		return "", err
+	}
+	if got != want {
+		return "", fmt.Errorf("%s does not match the release's checksum (got %s, expected %s)",
+			asset, got[:12], want[:12])
+	}
+	if err := os.Chmod(tmp.Name(), 0o755); err != nil {
+		return "", err
+	}
+	backup := target + ".bak-pre-" + version
+	if err := copyFile(target, backup); err != nil {
+		return "", fmt.Errorf("keeping the old binary: %w", err)
+	}
+	if err := os.Rename(tmp.Name(), target); err != nil {
+		return "", err
+	}
+	return backup, nil
+}
+
+func checksum(ctx context.Context, client *http.Client, url, asset string) (string, error) {
+	body, err := get(ctx, client, url)
+	if err != nil {
+		return "", err
+	}
+	defer body.Close()
+	sc := bufio.NewScanner(io.LimitReader(body, 1<<20))
+	for sc.Scan() {
+		f := strings.Fields(sc.Text())
+		if len(f) == 2 && f[1] == asset && len(f[0]) == 64 {
+			return f[0], nil
+		}
+	}
+	return "", fmt.Errorf("the release's checksums list no %s", asset)
+}
+
+func download(ctx context.Context, client *http.Client, url string, dst io.Writer) (string, error) {
+	body, err := get(ctx, client, url)
+	if err != nil {
+		return "", err
+	}
+	defer body.Close()
+	h := sha256.New()
+	if _, err := io.Copy(io.MultiWriter(dst, h), body); err != nil {
+		return "", fmt.Errorf("downloading %s: %w", url, err)
+	}
+	return hex.EncodeToString(h.Sum(nil)), nil
+}
+
+func get(ctx context.Context, client *http.Client, url string) (io.ReadCloser, error) {
+	req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
+	if err != nil {
+		return nil, err
+	}
+	resp, err := client.Do(req)
+	if err != nil {
+		return nil, fmt.Errorf("fetching %s: %w", url, err)
+	}
+	if resp.StatusCode != http.StatusOK {
+		resp.Body.Close()
+		return nil, fmt.Errorf("fetching %s: %s", url, resp.Status)
+	}
+	return resp.Body, nil
+}
+
+func copyFile(src, dst string) error {
+	in, err := os.Open(src) //nolint:gosec // our own binary
+	if err != nil {
+		return err
+	}
+	defer in.Close()
+	out, err := os.OpenFile(dst, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0o755) //nolint:gosec // beside it
+	if err != nil {
+		return err
+	}
+	if _, err := io.Copy(out, in); err != nil {
+		out.Close()
+		return err
+	}
+	return out.Close()
+}

internal/release/release_test.go

@@ -0,0 +1,61 @@
+package release
+
+import (
+	"crypto/sha256"
+	"encoding/hex"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"testing"
+)
+
+func releaseServer(t *testing.T, version, asset string, body []byte, sum string) *httptest.Server {
+	t.Helper()
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		switch r.URL.Path {
+		case "/" + version + "/checksums.txt":
+			_, _ = w.Write([]byte(sum + "  " + asset + "\n"))
+		case "/" + version + "/" + asset:
+			_, _ = w.Write(body)
+		default:
+			http.NotFound(w, r)
+		}
+	}))
+	t.Cleanup(srv.Close)
+	return srv
+}
+
+func TestABinaryIsReplacedOnlyWithWhatTheReleasePublished(t *testing.T) {
+	dir := t.TempDir()
+	target := filepath.Join(dir, "llmgw")
+	_ = os.WriteFile(target, []byte("old"), 0o755)
+	body := []byte("new binary")
+	sum := sha256.Sum256(body)
+	asset := AssetName("v1.2.3", "linux", "amd64")
+
+	srv := releaseServer(t, "v1.2.3", asset, body, hex.EncodeToString(sum[:]))
+	backup, err := Install(t.Context(), srv.Client(), srv.URL, "v1.2.3", "linux", "amd64", target)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if b, _ := os.ReadFile(target); string(b) != "new binary" {
+		t.Errorf("target holds %q", b)
+	}
+	if b, _ := os.ReadFile(backup); string(b) != "old" {
+		t.Errorf("the old binary was not kept: %q", b)
+	}
+
+	// A download that does not match its checksum changes nothing.
+	bad := releaseServer(t, "v1.2.4", AssetName("v1.2.4", "linux", "amd64"), []byte("tampered"),
+		hex.EncodeToString(sum[:]))
+	if _, err := Install(t.Context(), bad.Client(), bad.URL, "v1.2.4", "linux", "amd64", target); err == nil {
+		t.Fatal("a binary not matching its checksum was installed")
+	}
+	if b, _ := os.ReadFile(target); string(b) != "new binary" {
+		t.Errorf("a failed install touched the binary: %q", b)
+	}
+	if _, err := Install(t.Context(), nil, srv.URL, "../../x", "linux", "amd64", target); err == nil {
+		t.Error("a version that is not one was fetched")
+	}
+}

internal/router/finish_test.go

@@ -0,0 +1,119 @@
+package router
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"net/http"
+	"net/http/httptest"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/accounting"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/router/admission"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/router/proxy"
+)
+
+// What a client is told for each way a dispatch can end. Each status is
+// something a client acts on: 503 with Retry-After is "try again", 503 alone
+// is "not here", 502 is "the engine failed", and a stream already under way
+// cannot be given a status at all.
+func TestEachWayADispatchEndsIsToldToTheClientAsItShouldBe(t *testing.T) {
+	rt := newTestRouter(t, baseConfig())
+	for _, c := range []struct {
+		name       string
+		res        dispatchResult
+		status     int
+		body       bool // an error body was written
+		retryAfter bool
+	}{
+		{"served", dispatchResult{outcome: proxy.Outcome{Status: 200, Wrote: true}}, 200, false, false},
+		{"engine refused, relayed", dispatchResult{outcome: proxy.Outcome{Status: 400, Wrote: true}}, 400, false, false},
+		{"client went away", dispatchResult{err: proxy.ErrClientGone}, 499, false, false},
+		{"stream cut short", dispatchResult{outcome: proxy.Outcome{Status: 200, Wrote: true}, err: errors.New("eof")}, 200, false, false},
+		{"stream cut before a status", dispatchResult{outcome: proxy.Outcome{Wrote: true}, err: errors.New("eof")}, 200, false, false},
+		{"still loading", dispatchResult{err: fmt.Errorf("%w: retry shortly", errLoading)}, 503, true, true},
+		{"nothing serves it", dispatchResult{err: fmt.Errorf("%w: no replica", errNoCapacity)}, 503, true, false},
+		{"engine failed", dispatchResult{err: errors.New("connection reset")}, 502, true, false},
+	} {
+		t.Run(c.name, func(t *testing.T) {
+			w := httptest.NewRecorder()
+			got := rt.finish(w, &parsedRequest{Model: "m"}, c.res, time.Now())
+			if got != c.status {
+				t.Errorf("status %d, expected %d", got, c.status)
+			}
+			if c.body {
+				if w.Code != c.status {
+					t.Errorf("written %d, expected %d", w.Code, c.status)
+				}
+				var e struct {
+					Error struct {
+						Message string `json:"message"`
+					} `json:"error"`
+				}
+				if err := json.Unmarshal(w.Body.Bytes(), &e); err != nil || e.Error.Message == "" {
+					t.Errorf("error body %q", w.Body.String())
+				}
+			} else if w.Body.Len() != 0 {
+				t.Errorf("wrote %q over a response that was not ours to write", w.Body.String())
+			}
+			if (w.Header().Get("Retry-After") != "") != c.retryAfter {
+				t.Errorf("Retry-After %q", w.Header().Get("Retry-After"))
+			}
+		})
+	}
+}
+
+func TestAnUpstreamStatusIsKeptWhenTheStreamBreaks(t *testing.T) {
+	rt := newTestRouter(t, baseConfig())
+	w := httptest.NewRecorder()
+	res := dispatchResult{outcome: proxy.Outcome{Status: http.StatusTooManyRequests, Wrote: true}, err: errors.New("eof")}
+	if got := rt.finish(w, &parsedRequest{Model: "m"}, res, time.Now()); got != 429 {
+		t.Errorf("status %d", got)
+	}
+}
+
+func TestARefusalBeforeDispatchSaysWhetherWaitingHelps(t *testing.T) {
+	rt := newTestRouter(t, baseConfig())
+	for _, c := range []struct {
+		name       string
+		refuse     func(http.ResponseWriter)
+		status     int
+		retryAfter string
+	}{
+		{"queue full", func(w http.ResponseWriter) {
+			rt.rejectAdmission(w, &parsedRequest{Model: "m"}, admission.ErrQueueFull, time.Now())
+		}, 503, ""},
+		{"queued too long", func(w http.ResponseWriter) {
+			rt.rejectAdmission(w, &parsedRequest{Model: "m"}, admission.ErrQueueTimeout, time.Now())
+		}, 504, ""},
+		{"shutting down", func(w http.ResponseWriter) {
+			rt.rejectAdmission(w, &parsedRequest{Model: "m"}, admission.ErrClosed, time.Now())
+		}, 503, ""},
+		{"client left the queue", func(w http.ResponseWriter) {
+			rt.rejectAdmission(w, &parsedRequest{Model: "m"}, context.Canceled, time.Now())
+		}, 200, ""}, // nothing written: nobody is listening
+		{"rate limited", func(w http.ResponseWriter) {
+			rt.rejectLimit(w, &parsedRequest{Model: "m"},
+				&accounting.LimitError{Err: accounting.ErrRateLimited, RetryAfter: 3 * time.Second}, time.Now())
+		}, 429, "3"},
+		{"budget spent", func(w http.ResponseWriter) {
+			rt.rejectLimit(w, &parsedRequest{Model: "m"}, &accounting.LimitError{Err: accounting.ErrBudgetSpent}, time.Now())
+		}, 429, ""},
+		{"not a limit at all", func(w http.ResponseWriter) {
+			rt.rejectLimit(w, &parsedRequest{Model: "m"}, errors.New("store down"), time.Now())
+		}, 500, ""},
+	} {
+		t.Run(c.name, func(t *testing.T) {
+			w := httptest.NewRecorder()
+			c.refuse(w)
+			if w.Code != c.status {
+				t.Errorf("status %d, expected %d", w.Code, c.status)
+			}
+			if got := w.Header().Get("Retry-After"); got != c.retryAfter {
+				t.Errorf("Retry-After %q, expected %q", got, c.retryAfter)
+			}
+		})
+	}
+}

internal/router/handlers.go

@@ -3,6 +3,8 @@ package router
 import (
 	"bytes"
 	"context"
+	"crypto/rand"
+	"encoding/hex"
 	"encoding/json"
 	"errors"
 	"fmt"
@@ -13,6 +15,9 @@ import (
 	"strings"
 	"time"
 
+	"go.opentelemetry.io/otel/attribute"
+	"go.opentelemetry.io/otel/codes"
+
 	"gitea.mynodes.xyz/ilker/llmgw/internal/accounting"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/auth"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/router/admission"
@@ -51,7 +56,7 @@ func (r *Router) rejectAdmission(w http.ResponseWriter, req *parsedRequest, err
 	case errors.Is(err, context.Canceled), errors.Is(err, context.DeadlineExceeded):
 		// The client gave up while queued; nobody is listening.
 		r.m.observeAdmissionRejected(req.Band, "client_gone")
-		r.m.observeRequest(req.Model, "", 499, time.Since(start), nil, 0)
+		r.m.observeRequest(r.metricModel(req.Model), "", 499, time.Since(start), nil, 0)
 		return
 	case errors.Is(err, admission.ErrQueueTimeout):
 		status = http.StatusGatewayTimeout
@@ -67,7 +72,8 @@ func (r *Router) rejectAdmission(w http.ResponseWriter, req *parsedRequest, err
 	}
 	r.m.observeAdmissionRejected(req.Band, reason)
 	writeError(w, status, "server_error", err.Error())
-	r.m.observeRequest(req.Model, "", status, time.Since(start), nil, 0)
+	r.m.observeRequest(r.metricModel(req.Model), "", status, time.Since(start), nil, 0)
+	r.accessLog(req, dispatchResult{err: err}, status, time.Since(start))
 }
 
 // prefixBytes is how much of the prompt is kept for prefix-aware routing.
@@ -224,6 +230,12 @@ type parsedRequest struct {
 	UpstreamBody       []byte
 	Header             http.Header
 
+	// RequestID follows the request through the access log, the usage record
+	// and the engine, and goes back to the client in X-Request-Id.
+	RequestID string
+	// QueueMs is how long admission held it.
+	QueueMs int64
+
 	SessionID string
 	Band      string
 	// KeyID, ProjectID and Tenant are empty when authentication is off.
@@ -275,8 +287,23 @@ type inferenceBody struct {
 func (r *Router) handleInference(path string) http.HandlerFunc {
 	return func(w http.ResponseWriter, req *http.Request) {
 		start := time.Now()
+		// First, so even a request refused as malformed can be named when
+		// somebody asks what happened to it.
+		rid := requestID(req.Header.Get("X-Request-Id"))
+		w.Header().Set("X-Request-Id", rid)
+		ctx, span := startInference(req, path, rid)
+		defer span.End()
+		req = req.WithContext(ctx)
 
 		parsed, err := r.parse(req, path)
+		if err == nil {
+			span.SetAttributes(attribute.String("llmgw.model", parsed.Model))
+			parsed.RequestID = rid
+			if parsed.Header == nil {
+				parsed.Header = http.Header{}
+			}
+			parsed.Header.Set("X-Request-Id", rid)
+		}
 		if err != nil {
 			writeError(w, http.StatusBadRequest, "invalid_request_error", err.Error())
 			r.m.observeRequest("", "", http.StatusBadRequest, time.Since(start), nil, 0)
@@ -289,7 +316,7 @@ func (r *Router) handleInference(path string) http.HandlerFunc {
 		if _, entry := r.resolve(parsed.Model); entry.ContextLength > 0 {
 			if why := checkContextFits(entry, parsed.PromptTokens, parsed.MaxTokens); why != "" {
 				writeError(w, http.StatusBadRequest, "invalid_request_error", why)
-				r.m.observeRequest(parsed.Model, "", http.StatusBadRequest, time.Since(start), nil, 0)
+				r.m.observeRequest(r.metricModel(parsed.Model), "", http.StatusBadRequest, time.Since(start), nil, 0)
 				return
 			}
 		}
@@ -297,7 +324,7 @@ func (r *Router) handleInference(path string) http.HandlerFunc {
 		if p, ok := auth.FromContext(req.Context()); ok {
 			if !r.permits(p, parsed.Model) {
 				auth.WriteForbiddenModel(w, parsed.Model)
-				r.m.observeRequest(parsed.Model, "", http.StatusForbidden, time.Since(start), nil, 0)
+				r.m.observeRequest(r.metricModel(parsed.Model), "", http.StatusForbidden, time.Since(start), nil, 0)
 				return
 			}
 			// Limits are checked before admission, not after: a request that is
@@ -323,6 +350,9 @@ func (r *Router) handleInference(path string) http.HandlerFunc {
 						w.Header().Set("x-llmgw-served-model", parsed.Model)
 					}
 				}
+				// On every answer, refused or not, so a client can pace itself
+				// before it is refused rather than learn the limit from a 429.
+				r.rateLimitHeaders(w, p)
 				if err != nil {
 					r.rejectLimit(w, parsed, err, start)
 					return
@@ -332,14 +362,17 @@ func (r *Router) handleInference(path string) http.HandlerFunc {
 		}
 
 		// Step 5 of the lifecycle: "now?" before "where?" (plan §7.1).
+		_, aspan := tracer().Start(req.Context(), "admission")
 		lease, err := r.admit(req.Context(), parsed)
+		endSpan(aspan, err)
 		if err != nil {
 			r.rejectAdmission(w, parsed, err, start)
 			return
 		}
 		defer lease.Release()
 		queued := time.Since(start)
-		r.m.observeAdmission(lease.Band(), parsed.Model, queued)
+		parsed.QueueMs = queued.Milliseconds()
+		r.m.observeAdmission(lease.Band(), r.metricModel(parsed.Model), queued)
 		if queued > time.Millisecond {
 			w.Header().Set("x-llmgw-queue-ms", strconv.FormatInt(queued.Milliseconds(), 10))
 		}
@@ -349,7 +382,7 @@ func (r *Router) handleInference(path string) http.HandlerFunc {
 		// model that will actually serve, not the one that was asked for.
 		if dropped, err := r.loadThread(req.Context(), parsed); err != nil {
 			writeError(w, http.StatusBadRequest, "invalid_request_error", err.Error())
-			r.m.observeRequest(parsed.Model, "", http.StatusBadRequest, time.Since(start), nil, 0)
+			r.m.observeRequest(r.metricModel(parsed.Model), "", http.StatusBadRequest, time.Since(start), nil, 0)
 			return
 		} else if dropped > 0 {
 			// Dropping turns is the right answer and must not be a silent one.
@@ -357,7 +390,13 @@ func (r *Router) handleInference(path string) http.HandlerFunc {
 		}
 
 		res := r.dispatch(req.Context(), w, parsed)
-		r.finish(w, parsed, res, start)
+		status := r.finish(w, parsed, res, start)
+		span.SetAttributes(attribute.Int("http.response.status_code", status),
+			attribute.String("llmgw.served_model", res.model),
+			attribute.String("llmgw.endpoint", res.endpointID))
+		if status >= 500 {
+			span.SetStatus(codes.Error, http.StatusText(status))
+		}
 		// After the client has been served. A conversation that failed to
 		// record is a worse outcome than a slow reply, but only just, and the
 		// caller has their answer either way.
@@ -389,7 +428,8 @@ func (r *Router) permits(p auth.Principal, requested string) bool {
 }
 
 // finish records metrics and, when nothing was written, produces the error body.
-func (r *Router) finish(w http.ResponseWriter, req *parsedRequest, res dispatchResult, start time.Time) {
+// It returns the status the request ended with.
+func (r *Router) finish(w http.ResponseWriter, req *parsedRequest, res dispatchResult, start time.Time) int {
 	status := res.outcome.Status
 	switch {
 	case res.err == nil:
@@ -420,8 +460,10 @@ func (r *Router) finish(w http.ResponseWriter, req *parsedRequest, res dispatchR
 		writeError(w, status, "server_error", res.err.Error())
 	}
 
-	r.m.observeRequest(req.Model, res.pool, status, time.Since(start), res.outcome.Usage, res.outcome.TTFT)
+	r.m.observeRequest(r.metricModel(req.Model), res.pool, status, time.Since(start), res.outcome.Usage, res.outcome.TTFT)
 	r.account(req, res, status, time.Since(start))
+	r.accessLog(req, res, status, time.Since(start))
+	return status
 }
 
 // account settles a finished request against its key.
@@ -437,6 +479,8 @@ func (r *Router) account(req *parsedRequest, res dispatchResult, status int, too
 		At: time.Now(), KeyID: req.KeyID, UserID: req.UserID, ProjectID: req.ProjectID,
 		Model: req.Model, ServedModel: res.model, Status: status,
 		LatencyMs: took.Milliseconds(),
+		RequestID: req.RequestID, Replica: res.endpointID,
+		QueueMs: req.QueueMs, TTFTMs: res.outcome.TTFT.Milliseconds(),
 	}
 	if u.ServedModel == "" {
 		u.ServedModel = req.Model
@@ -467,7 +511,8 @@ func (r *Router) rejectLimit(w http.ResponseWriter, req *parsedRequest, err erro
 	// already know how to surface that code. What differs is Retry-After,
 	// which is absent when waiting does not help.
 	writeError(w, http.StatusTooManyRequests, "rate_limit_exceeded", le.Error())
-	r.m.observeRequest(req.Model, "", http.StatusTooManyRequests, time.Since(start), nil, 0)
+	r.m.observeRequest(r.metricModel(req.Model), "", http.StatusTooManyRequests, time.Since(start), nil, 0)
+	r.accessLog(req, dispatchResult{err: err}, http.StatusTooManyRequests, time.Since(start))
 }
 
 // parse reads the body and extracts the routing-relevant fields.
@@ -854,3 +899,77 @@ func (r *Router) handleMe(w http.ResponseWriter, req *http.Request) {
 	}
 	writeJSON(w, http.StatusOK, out)
 }
+
+// requestID is the client's X-Request-Id when it is one worth keeping, or a
+// new one. Kept only when short and plain: it is written into logs and a
+// header, and a client-chosen string with newlines or megabytes in it is not
+// an id.
+func requestID(sent string) string {
+	if n := len(sent); n > 0 && n <= 128 && strings.IndexFunc(sent, func(c rune) bool {
+		return !(c >= 'a' && c <= 'z' || c >= 'A' && c <= 'Z' || c >= '0' && c <= '9' ||
+			c == '-' || c == '_' || c == '.' || c == ':')
+	}) < 0 {
+		return sent
+	}
+	var b [12]byte
+	_, _ = rand.Read(b[:])
+	return "req_" + hex.EncodeToString(b[:])
+}
+
+// accessLog writes one line per finished request: enough to follow a slow or
+// failed request from the id a client reports to the replica that served it,
+// and where its time went. Before this the only per-request lines were for
+// disconnects, early stream ends and upstream retries.
+func (r *Router) accessLog(req *parsedRequest, res dispatchResult, status int, took time.Duration) {
+	attrs := []any{
+		"request_id", req.RequestID, "status", status, "model", req.Model,
+		"duration_ms", took.Milliseconds(), "queue_ms", req.QueueMs, "stream", req.Stream,
+	}
+	if req.RequestedModel != "" && req.RequestedModel != req.Model {
+		attrs = append(attrs, "requested_model", req.RequestedModel)
+	}
+	if res.model != "" && res.model != req.Model {
+		attrs = append(attrs, "served_model", res.model)
+	}
+	if res.endpointID != "" {
+		attrs = append(attrs, "replica", res.endpointID)
+	}
+	if res.outcome.TTFT > 0 {
+		attrs = append(attrs, "ttft_ms", res.outcome.TTFT.Milliseconds())
+	}
+	if u := res.outcome.Usage; u != nil {
+		attrs = append(attrs, "prompt_tokens", u.PromptTokens, "completion_tokens", u.CompletionTokens)
+	}
+	if req.KeyID != "" {
+		attrs = append(attrs, "key", req.KeyID, "project", req.ProjectID)
+	}
+	if req.UserID != "" {
+		attrs = append(attrs, "user", req.UserID)
+	}
+	if res.err != nil {
+		attrs = append(attrs, "err", res.err.Error())
+	}
+	r.log.Info("request", attrs...)
+}
+
+// rateLimitHeaders sets OpenAI's x-ratelimit-* headers from the tightest
+// limit that applies. SDKs and editors read these to slow down before they
+// are refused; only Retry-After, and only on a refusal, was sent before. A
+// limit that is not set is not advertised.
+func (r *Router) rateLimitHeaders(w http.ResponseWriter, p auth.Principal) {
+	if r.acct == nil {
+		return
+	}
+	h := r.acct.Headroom(p)
+	reset := strconv.FormatFloat(h.Reset.Seconds(), 'f', 0, 64) + "s"
+	if h.RequestLimit > 0 {
+		w.Header().Set("x-ratelimit-limit-requests", strconv.Itoa(h.RequestLimit))
+		w.Header().Set("x-ratelimit-remaining-requests", strconv.Itoa(h.RequestsLeft))
+		w.Header().Set("x-ratelimit-reset-requests", reset)
+	}
+	if h.TokenLimit > 0 {
+		w.Header().Set("x-ratelimit-limit-tokens", strconv.Itoa(h.TokenLimit))
+		w.Header().Set("x-ratelimit-remaining-tokens", strconv.Itoa(h.TokensLeft))
+		w.Header().Set("x-ratelimit-reset-tokens", reset)
+	}
+}

internal/router/metrics.go

@@ -151,3 +151,17 @@ func (m *metrics) observeScores(d scheduler.Decision) {
 	}
 	m.considered.Observe(float64(d.Considered))
 }
+
+// metricModel is the model label a request is counted under: the catalog's
+// name for it, or "unknown". Taken as sent, any key holder could mint a new
+// series per request — a refused request counted its made-up model name —
+// and grow Prometheus without bound.
+func (r *Router) metricModel(name string) string {
+	if name == "" {
+		return ""
+	}
+	if m, ok := r.modelMap()[name]; ok {
+		return m.Name
+	}
+	return "unknown"
+}

internal/router/pricing_test.go

@@ -140,3 +140,51 @@ func waitForUsage(t *testing.T, rt *Router) store.KeyUsage {
 	t.Fatal("no usage was recorded")
 	return store.KeyUsage{}
 }
+
+func TestARequestCanBeFollowedByItsID(t *testing.T) {
+	// "Why was my request slow?" had no answer: the queue time went out in a
+	// header and nowhere else, and nothing tied a request to its replica.
+	e := newFakeEngine(t, "qwen2.5-14b-instruct")
+	rt, secret := pricedRouter(t, e.srv.URL)
+
+	rec := post(t, rt.Handler(),
+		map[string]any{"model": "q14", "messages": []any{
+			map[string]any{"role": "user", "content": "hello"}}},
+		map[string]string{"Authorization": "Bearer " + secret, "X-Request-Id": "client-abc.1"})
+	if rec.Code != 200 {
+		t.Fatalf("status %d: %s", rec.Code, rec.Body.String())
+	}
+	if got := rec.Header().Get("X-Request-Id"); got != "client-abc.1" {
+		t.Errorf("the client's id came back as %q", got)
+	}
+
+	deadline := time.Now().Add(5 * time.Second)
+	for {
+		u, ok, err := rt.store.UsageByRequest(t.Context(), "client-abc.1")
+		if err != nil {
+			t.Fatal(err)
+		}
+		if ok {
+			if u.Replica == "" || u.Status != 200 {
+				t.Errorf("recorded %+v; expected the replica that answered", u)
+			}
+			break
+		}
+		if time.Now().After(deadline) {
+			t.Fatal("the request's usage cannot be found by its id")
+		}
+		time.Sleep(20 * time.Millisecond)
+	}
+}
+
+func TestAnIDThatIsNotOneIsReplaced(t *testing.T) {
+	for _, bad := range []string{"", "has space", "new\nline", strings.Repeat("x", 200)} {
+		got := requestID(bad)
+		if got == bad || !strings.HasPrefix(got, "req_") {
+			t.Errorf("%q became %q", bad, got)
+		}
+	}
+	if requestID("req_1") != "req_1" {
+		t.Error("a plain id was not kept")
+	}
+}

internal/router/proxy/proxy.go

@@ -526,7 +526,7 @@ func copyHeaders(dst, src http.Header) {
 // llmgw, not llmgw to the engine. llmgw's own key for the engine is set by Do.
 func cloneForUpstream(h http.Header) http.Header {
 	out := make(http.Header, 4)
-	for _, k := range []string{"Accept", "User-Agent", "Traceparent", "Tracestate"} {
+	for _, k := range []string{"Accept", "User-Agent", "Traceparent", "Tracestate", "X-Request-Id"} {
 		if v := h.Get(k); v != "" {
 			out.Set(k, v)
 		}

internal/router/router.go

@@ -11,19 +11,24 @@ import (
 	"fmt"
 	"log/slog"
 	"math"
+	"net"
 	"net/http"
+	"net/netip"
 	"slices"
 	"strings"
 	"sync/atomic"
 	"time"
 
 	"github.com/prometheus/client_golang/prometheus"
+	"go.opentelemetry.io/otel/attribute"
+	"go.opentelemetry.io/otel/trace"
 
 	"gitea.mynodes.xyz/ilker/llmgw/internal/account"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/accounting"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/auth"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/router/admission"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/router/breaker"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint"
@@ -230,6 +235,11 @@ func New(cfg config.Server, log *slog.Logger) (*Router, error) {
 		Timeout:  cfg.Router.ScrapeTimeout,
 	}, log)
 
+	if !cfg.Auth.Enabled && !loopbackOnly(cfg.Listen) {
+		log.Warn("auth is off and the API listens beyond this machine: anyone who can reach "+
+			cfg.Listen+" can use every model; set auth.enabled, or listen on 127.0.0.1",
+			"listen", cfg.Listen)
+	}
 	if cfg.Auth.Enabled {
 		st, err := storepkg.Open(context.Background(), cfg.Auth.Store)
 		if err != nil {
@@ -510,6 +520,24 @@ func (r *Router) SetCatalog(c *catalog.Catalog) {
 // holding its cards out of habit.
 func (r *Router) LastUsed() map[string]time.Time { return r.reg.LastUsed() }
 
+// ModelLoad sums what each model's replicas report: how many there are, and
+// requests running and waiting at their engines. Keyed by the catalog name,
+// which the control plane publishes first among each endpoint's names.
+func (r *Router) ModelLoad() map[string]reconciler.Load {
+	out := map[string]reconciler.Load{}
+	for _, s := range r.reg.Snapshots("") {
+		if len(s.Models) == 0 {
+			continue
+		}
+		l := out[s.Models[0]]
+		l.Replicas++
+		l.Running += s.Stats.Running
+		l.Waiting += s.Stats.Waiting
+		out[s.Models[0]] = l
+	}
+	return out
+}
+
 func (r *Router) SetEndpoints(ctx context.Context, infos []endpoint.Info) {
 	if r.prefix != nil {
 		// A replica that has gone takes its cache with it. Leaving its entries
@@ -663,7 +691,13 @@ func (r *Router) dispatchOne(ctx context.Context, w http.ResponseWriter, req *pa
 		if len(candidates) == 0 {
 			break
 		}
+		_, sspan := tracer().Start(ctx, "schedule", trace.WithAttributes(
+			attribute.String("llmgw.model", model), attribute.Int("llmgw.candidates", len(candidates))))
 		dec, err := profile.Schedule(sctx, candidates)
+		if err == nil {
+			sspan.SetAttributes(attribute.String("llmgw.endpoint", dec.Endpoint.ID))
+		}
+		endSpan(sspan, err)
 		if err != nil {
 			last = dispatchResult{err: fmt.Errorf("%w: %v", errNoCapacity, err)}
 			break
@@ -677,15 +711,20 @@ func (r *Router) dispatchOne(ctx context.Context, w http.ResponseWriter, req *pa
 			continue
 		}
 
-		out, perr := r.px.Do(ctx, w, proxy.Request{
+		uctx, uspan := tracer().Start(ctx, "upstream", trace.WithSpanKind(trace.SpanKindClient),
+			trace.WithAttributes(attribute.String("llmgw.endpoint", dec.Endpoint.ID),
+				attribute.String("url.full", dec.Endpoint.URL+req.Path)))
+		out, perr := r.px.Do(uctx, w, proxy.Request{
 			URL:                dec.Endpoint.URL + req.Path,
 			APIKey:             dec.Endpoint.APIKey,
 			Body:               req.UpstreamBody,
-			Header:             req.Header,
+			Header:             withTraceparent(uctx, req.Header),
 			Stream:             req.Stream,
 			SuppressUsageChunk: req.SuppressUsageChunk,
 			CaptureReply:       req.captureBytes(),
 		})
+		uspan.SetAttributes(attribute.Int("http.response.status_code", out.Status))
+		endSpan(uspan, perr)
 		// A retryable failure is the endpoint's fault; a client disconnect is not.
 		release(perr == nil || errors.Is(perr, proxy.ErrClientGone))
 
@@ -745,3 +784,17 @@ func personalLimits(p config.Personal) *storepkg.Limits {
 	}
 	return &l
 }
+
+// loopbackOnly reports whether a listen address is reachable from this
+// machine alone. An empty host (":8080") is every interface.
+func loopbackOnly(listen string) bool {
+	host, _, err := net.SplitHostPort(listen)
+	if err != nil {
+		return false
+	}
+	if host == "localhost" {
+		return true
+	}
+	a, err := netip.ParseAddr(host)
+	return err == nil && a.IsLoopback()
+}

internal/router/router_listen_test.go

@@ -0,0 +1,14 @@
+package router
+
+import "testing"
+
+func TestOnlyALoopbackListenCountsAsPrivate(t *testing.T) {
+	for addr, want := range map[string]bool{
+		"127.0.0.1:8080": true, "[::1]:8080": true, "localhost:8080": true,
+		":8080": false, "0.0.0.0:8080": false, "10.0.0.140:8080": false, "junk": false,
+	} {
+		if got := loopbackOnly(addr); got != want {
+			t.Errorf("%s: %v", addr, got)
+		}
+	}
+}

internal/router/router_test.go

@@ -4,6 +4,7 @@ import (
 	"bytes"
 	"encoding/json"
 	"fmt"
+	"github.com/prometheus/client_golang/prometheus"
 	"io"
 	"log/slog"
 	"net/http"
@@ -35,6 +36,8 @@ type fakeEngine struct {
 	status atomic.Int64
 	// last is the body of the last completion asked for.
 	last atomic.Value
+	// traceparent is the one the last completion came with.
+	traceparent atomic.Value
 }
 
 func newFakeEngine(t *testing.T, model string) *fakeEngine {
@@ -48,6 +51,7 @@ func newFakeEngine(t *testing.T, model string) *fakeEngine {
 	})
 	mux.HandleFunc("POST /v1/chat/completions", func(w http.ResponseWriter, r *http.Request) {
 		f.calls.Add(1)
+		f.traceparent.Store(r.Header.Get("traceparent"))
 		if b, err := io.ReadAll(r.Body); err == nil {
 			f.last.Store(b)
 		}
@@ -655,3 +659,32 @@ func TestAModelThatCannotComeUpIsSaidSoAtOnce(t *testing.T) {
 		t.Errorf("the reason was not given: %s", rec.Body.String())
 	}
 }
+
+func TestAMadeUpModelNameDoesNotBecomeAMetricLabel(t *testing.T) {
+	// Counted as sent, every request for a nonsense model minted a new series.
+	e := newFakeEngine(t, "m")
+	cfg := baseConfig(config.Endpoint{ID: "a", URL: e.srv.URL, Models: []string{"m"}})
+	cfg.Models = []catalog.Model{{Name: "m", Aliases: []string{"mm"}}}
+	rt := newTestRouter(t, cfg)
+
+	if got := rt.metricModel("mm"); got != "m" {
+		t.Errorf("an alias is counted as %q, not the model it names", got)
+	}
+	for i := 0; i < 20; i++ {
+		post(t, rt.Handler(), map[string]any{"model": fmt.Sprintf("junk-%d", i), "messages": []any{}}, nil)
+	}
+	// Twenty names, one series: all of them "unknown".
+	reg := prometheus.NewRegistry()
+	reg.MustRegister(rt.m.requests)
+	mfs, err := reg.Gather()
+	if err != nil {
+		t.Fatal(err)
+	}
+	series := 0
+	for _, mf := range mfs {
+		series += len(mf.GetMetric())
+	}
+	if series != 1 {
+		t.Errorf("%d series after twenty made-up names", series)
+	}
+}

internal/router/trace.go

@@ -0,0 +1,51 @@
+package router
+
+import (
+	"context"
+	"net/http"
+
+	"go.opentelemetry.io/otel"
+	"go.opentelemetry.io/otel/attribute"
+	"go.opentelemetry.io/otel/codes"
+	"go.opentelemetry.io/otel/propagation"
+	"go.opentelemetry.io/otel/trace"
+)
+
+// tracer is looked up per use rather than kept, so a provider installed after
+// the router was built — as tests do — is the one spans go to.
+func tracer() trace.Tracer { return otel.Tracer("gitea.mynodes.xyz/ilker/llmgw/internal/router") }
+
+// startInference opens the span for one inference request, continuing the
+// caller's trace when it sent a traceparent.
+func startInference(req *http.Request, path, requestID string) (context.Context, trace.Span) {
+	ctx := otel.GetTextMapPropagator().Extract(req.Context(), propagation.HeaderCarrier(req.Header))
+	return tracer().Start(ctx, "inference "+path,
+		trace.WithSpanKind(trace.SpanKindServer),
+		trace.WithAttributes(
+			attribute.String("llmgw.request_id", requestID),
+			attribute.String("http.route", path),
+		))
+}
+
+// endSpan closes a span, marking it failed when err is set.
+func endSpan(span trace.Span, err error) {
+	if err != nil {
+		span.RecordError(err)
+		span.SetStatus(codes.Error, err.Error())
+	}
+	span.End()
+}
+
+// withTraceparent is h with the current span's context in it, for the engine.
+// A copy: the request's header is sent on every attempt, each its own span.
+func withTraceparent(ctx context.Context, h http.Header) http.Header {
+	if !trace.SpanContextFromContext(ctx).IsValid() {
+		return h
+	}
+	out := h.Clone()
+	if out == nil {
+		out = http.Header{}
+	}
+	otel.GetTextMapPropagator().Inject(ctx, propagation.HeaderCarrier(out))
+	return out
+}

internal/router/trace_test.go

@@ -0,0 +1,83 @@
+package router
+
+import (
+	"testing"
+
+	"go.opentelemetry.io/otel"
+	"go.opentelemetry.io/otel/propagation"
+	sdktrace "go.opentelemetry.io/otel/sdk/trace"
+	"go.opentelemetry.io/otel/sdk/trace/tracetest"
+	"go.opentelemetry.io/otel/trace/noop"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+)
+
+func TestARequestIsTracedIntoTheEngine(t *testing.T) {
+	rec := tracetest.NewSpanRecorder()
+	prevTP, prevProp := otel.GetTracerProvider(), otel.GetTextMapPropagator()
+	otel.SetTracerProvider(sdktrace.NewTracerProvider(sdktrace.WithSpanProcessor(rec)))
+	otel.SetTextMapPropagator(propagation.TraceContext{})
+	t.Cleanup(func() { otel.SetTracerProvider(prevTP); otel.SetTextMapPropagator(prevProp) })
+
+	e := newFakeEngine(t, "m")
+	rt := newTestRouter(t, baseConfig(config.Endpoint{ID: "a", URL: e.srv.URL, Models: []string{"m"}}))
+	// The caller's own trace, continued rather than replaced.
+	const caller = "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01"
+	res := post(t, rt.Handler(), map[string]any{"model": "m", "messages": []any{}},
+		map[string]string{"traceparent": caller, "X-Request-Id": "req-1"})
+	if res.Code != 200 {
+		t.Fatalf("status %d: %s", res.Code, res.Body.String())
+	}
+
+	spans := map[string]sdktrace.ReadOnlySpan{}
+	for _, s := range rec.Ended() {
+		spans[s.Name()] = s
+	}
+	root, ok := spans["inference /v1/chat/completions"]
+	if !ok {
+		t.Fatalf("no inference span among %v", spans)
+	}
+	if got := root.SpanContext().TraceID().String(); got != "4bf92f3577b34da6a3ce929d0e0e4736" {
+		t.Errorf("the caller's trace was not continued: %s", got)
+	}
+	attrs := map[string]string{}
+	for _, a := range root.Attributes() {
+		attrs[string(a.Key)] = a.Value.Emit()
+	}
+	if attrs["llmgw.request_id"] != "req-1" || attrs["llmgw.model"] != "m" ||
+		attrs["http.response.status_code"] != "200" || attrs["llmgw.endpoint"] != "a" {
+		t.Errorf("inference span attributes %v", attrs)
+	}
+	for _, name := range []string{"admission", "schedule", "upstream"} {
+		s, ok := spans[name]
+		if !ok {
+			t.Errorf("no %s span", name)
+			continue
+		}
+		if s.Parent().SpanID() != root.SpanContext().SpanID() {
+			t.Errorf("%s is not under the request's span", name)
+		}
+	}
+	up := spans["upstream"]
+	tp, _ := e.traceparent.Load().(string)
+	if want := "00-" + up.SpanContext().TraceID().String() + "-" + up.SpanContext().SpanID().String() + "-01"; tp != want {
+		t.Errorf("the engine got traceparent %q, expected %q", tp, want)
+	}
+}
+
+func TestWithoutTracingTheEngineGetsNoTraceparent(t *testing.T) {
+	// What an unconfigured process has. Set explicitly, since the default
+	// global delegates for good to the first real provider another test set.
+	prevTP, prevProp := otel.GetTracerProvider(), otel.GetTextMapPropagator()
+	otel.SetTracerProvider(noop.NewTracerProvider())
+	otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator())
+	t.Cleanup(func() { otel.SetTracerProvider(prevTP); otel.SetTextMapPropagator(prevProp) })
+	e := newFakeEngine(t, "m")
+	rt := newTestRouter(t, baseConfig(config.Endpoint{ID: "a", URL: e.srv.URL, Models: []string{"m"}}))
+	if res := post(t, rt.Handler(), map[string]any{"model": "m", "messages": []any{}}, nil); res.Code != 200 {
+		t.Fatalf("status %d", res.Code)
+	}
+	if tp, _ := e.traceparent.Load().(string); tp != "" {
+		t.Errorf("traceparent %q sent with tracing off", tp)
+	}
+}

internal/store/migrations/postgres/0012_usage_diagnostics.sql

@@ -0,0 +1,14 @@
+-- Ported from the SQLite migration of the same number.
+-- What a request went through, beside what it cost: which replica answered,
+-- how long it queued in admission, and how long the first token took. Without
+-- them "why was my request slow" had no answer — the queue time went out in a
+-- response header and nowhere else.
+--
+-- request_id is the id the client sent or the gateway made, echoed in the
+-- X-Request-Id header and the access log, so one request can be followed from
+-- the client's error to the replica that served it.
+ALTER TABLE usage ADD COLUMN request_id TEXT NOT NULL DEFAULT '';
+ALTER TABLE usage ADD COLUMN replica TEXT NOT NULL DEFAULT '';
+ALTER TABLE usage ADD COLUMN queue_ms BIGINT NOT NULL DEFAULT 0;
+ALTER TABLE usage ADD COLUMN ttft_ms BIGINT NOT NULL DEFAULT 0;
+CREATE INDEX usage_by_request ON usage(request_id) WHERE request_id <> '';

internal/store/migrations/sqlite/0012_usage_diagnostics.sql

@@ -0,0 +1,13 @@
+-- What a request went through, beside what it cost: which replica answered,
+-- how long it queued in admission, and how long the first token took. Without
+-- them "why was my request slow" had no answer — the queue time went out in a
+-- response header and nowhere else.
+--
+-- request_id is the id the client sent or the gateway made, echoed in the
+-- X-Request-Id header and the access log, so one request can be followed from
+-- the client's error to the replica that served it.
+ALTER TABLE usage ADD COLUMN request_id TEXT NOT NULL DEFAULT '';
+ALTER TABLE usage ADD COLUMN replica TEXT NOT NULL DEFAULT '';
+ALTER TABLE usage ADD COLUMN queue_ms INTEGER NOT NULL DEFAULT 0;
+ALTER TABLE usage ADD COLUMN ttft_ms INTEGER NOT NULL DEFAULT 0;
+CREATE INDEX usage_by_request ON usage(request_id) WHERE request_id <> '';

internal/store/sqlite.go

@@ -424,8 +424,9 @@ func (s *DB) WriteUsage(ctx context.Context, records []Usage) error {
 	return s.inTx(ctx, func(tx conn) error {
 		stmt, err := tx.PrepareContext(ctx,
 			`INSERT INTO usage (ts, key_id, project_id, model, served_model,
-			   prompt_tokens, completion_tokens, cost_micros, status, latency_ms, user_id)
-			 VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`)
+			   prompt_tokens, completion_tokens, cost_micros, status, latency_ms, user_id,
+			   request_id, replica, queue_ms, ttft_ms)
+			 VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`)
 		if err != nil {
 			return err
 		}
@@ -433,7 +434,8 @@ func (s *DB) WriteUsage(ctx context.Context, records []Usage) error {
 		for _, r := range records {
 			if _, err := stmt.ExecContext(ctx, millis(r.At), r.KeyID, r.ProjectID,
 				r.Model, r.ServedModel, r.PromptTokens, r.CompletionTokens,
-				r.CostMicros, r.Status, r.LatencyMs, r.UserID); err != nil {
+				r.CostMicros, r.Status, r.LatencyMs, r.UserID,
+				r.RequestID, r.Replica, r.QueueMs, r.TTFTMs); err != nil {
 				return err
 			}
 		}
@@ -828,3 +830,25 @@ func (s *DB) SnapshotTo(ctx context.Context, path string) error {
 	_, err := s.raw.ExecContext(ctx, `VACUUM INTO ?`, path)
 	return err
 }
+
+// UsageByRequest finds the record of one request by the id it was given, or
+// ok=false when none has that id.
+func (s *DB) UsageByRequest(ctx context.Context, requestID string) (Usage, bool, error) {
+	var u Usage
+	var ts int64
+	err := s.db.QueryRowContext(ctx,
+		`SELECT ts, key_id, project_id, COALESCE(user_id, ''), model, served_model, prompt_tokens,
+		        completion_tokens, cost_micros, status, latency_ms, request_id, replica, queue_ms, ttft_ms
+		 FROM usage WHERE request_id = ? ORDER BY ts DESC LIMIT 1`, requestID).
+		Scan(&ts, &u.KeyID, &u.ProjectID, &u.UserID, &u.Model, &u.ServedModel, &u.PromptTokens,
+			&u.CompletionTokens, &u.CostMicros, &u.Status, &u.LatencyMs, &u.RequestID, &u.Replica,
+			&u.QueueMs, &u.TTFTMs)
+	if errors.Is(err, sql.ErrNoRows) {
+		return Usage{}, false, nil
+	}
+	if err != nil {
+		return Usage{}, false, err
+	}
+	u.At = time.UnixMilli(ts)
+	return u, true, nil
+}

internal/store/store.go

@@ -244,6 +244,15 @@ type Usage struct {
 	CostMicros int64
 	Status     int
 	LatencyMs  int64
+
+	// RequestID is the id the client sent in X-Request-Id or the gateway made.
+	RequestID string
+	// Replica answered the request; empty when none did.
+	Replica string
+	// QueueMs is time spent waiting in admission; TTFTMs is time to the first
+	// token, 0 when none came.
+	QueueMs int64
+	TTFTMs  int64
 }
 
 // Store is the persistent half of tenancy and accounting.

internal/web/admin_mirror.go

@@ -621,12 +621,7 @@ func (p *AccountPages) openRepo(r *http.Request, d *adminMirrorData, repo string
 // is not part of what anybody calls the model, and "-GGUF" describes the file
 // format rather than the model, which is already clear from the file being
 // fetched.
-func defaultModelName(repoID string) string {
-	name := mirror.LastPathElement(repoID)
-	name = strings.TrimSuffix(name, "-GGUF")
-	name = strings.TrimSuffix(name, "-gguf")
-	return strings.ToLower(name)
-}
+func defaultModelName(repoID string) string { return hf.ModelName(repoID) }
 
 // verdictFor judges a registry file set by the repository's architecture. A
 // vision projector is its own architecture whatever the repository says.

internal/web/admin_serve.go

@@ -49,6 +49,11 @@ type serveForm struct {
 	// 3.5 GB for a KV pool. Nothing was wrong with placement — it was never
 	// told what the model needed.
 	MinMemoryGB int
+	// MemoryBasis says how MinMemoryGB was arrived at, for the form to show.
+	MemoryBasis string
+	// CtxPool is the engine context pool the form starts with, and the one
+	// the memory figure assumes.
+	CtxPool int
 
 	// Refusal is set when this version cannot be served as it stands, and says
 	// why. Offered and then failing at placement is worse than not offered.
@@ -63,6 +68,7 @@ type serveForm struct {
 func planServe(st mirror.Status, files []mirror.Status, base string,
 	fleet Source, cat CatalogAdmin, check func(source string) gguf.Verdict,
 ) serveForm {
+	var header gguf.Header
 	f := serveForm{
 		Model:   st.Model,
 		Version: st.Version,
@@ -101,10 +107,12 @@ func planServe(st mirror.Status, files []mirror.Status, base string,
 	case "llamacpp":
 		f.Profile = "cpu"
 		if check != nil {
-			if v := check(f.Source); !v.Servable {
+			v := check(f.Source)
+			if !v.Servable {
 				f.Refusal = "llama.cpp on the nodes cannot run this: " + v.Reason
 				return f
 			}
+			header = v.Header
 		}
 	default:
 		f.Profile = "gpu"
@@ -123,7 +131,27 @@ func planServe(st mirror.Status, files []mirror.Status, base string,
 				"same model runs on the CPU nodes"
 		}
 	}
-	f.MinMemoryGB = memoryFor(st.BytesTotal, st.BytesDone)
+	f.CtxPool = defaultCtxPool
+	f.MinMemoryGB, f.MemoryBasis = memoryFor(st.BytesTotal, st.BytesDone), "weights plus half again plus two"
+	weights := st.BytesTotal
+	if weights <= 0 {
+		weights = st.BytesDone
+	}
+	if b := header.MemoryBytes(weights, defaultCtxPool); b > 0 {
+		kvHeads := header.HeadCountKV
+		if kvHeads == 0 {
+			kvHeads = header.HeadCount
+		}
+		// From the model's own shape: its KV cache for the pool the form
+		// starts with, which ranges from under half a gigabyte to over five
+		// across the models on the live mirror at the same 16k — a spread the
+		// flat "half again" could not see.
+		f.MinMemoryGB = int((b + 1<<30 - 1) >> 30)
+		f.MemoryBasis = fmt.Sprintf("weights %.1f GB + KV cache %.1f GB for %d tokens (%d layers × %d KV heads) + %.1f GB for the engine",
+			float64(weights)/(1<<30), float64(int64(defaultCtxPool)*header.KVBytesPerToken())/(1<<30),
+			defaultCtxPool, header.BlockCount, kvHeads,
+			float64(int64(1<<30)+weights/20)/(1<<30))
+	}
 	if biggest := biggestNode(fleet); biggest > 0 && f.MinMemoryGB > biggest {
 		f.Warning = strings.TrimSuffix(f.Warning, ".") +
 			fmt.Sprintf("%sthe largest node has %d GB and this needs about %d",
@@ -353,3 +381,7 @@ func modeOf(sc catalog.Scaling) string {
 		return modeStopped
 	}
 }
+
+// defaultCtxPool is the llama.cpp context pool the serve form starts with.
+// --parallel divides it among slots; it does not multiply it.
+const defaultCtxPool = 16384

internal/web/templates/admin_mirror.html

@@ -147,11 +147,11 @@ reachable from this machine.</p>
     <label>Output limit <span class="opt">optional</span>
       <input type="text" name="max_output" value="1024"></label>
     <label>Engine context pool <span class="opt">divided across slots</span>
-      <input type="text" name="ctx_size" value="16384"></label>
+      <input type="text" name="ctx_size" value="{{.CtxPool}}"></label>
   </div>
   <div class="field-row">
     <label>Profile name <input type="text" name="profile" value="{{.Profile}}"></label>
-    <label>Needs RAM (GB) <span class="opt">estimated from the weights</span>
+    <label>Needs RAM (GB) <span class="opt">for the context pool beside it</span>
       <input type="text" name="min_memory_gb" value="{{.MinMemoryGB}}"></label>
     <label>Only nodes labelled <span class="opt">key / value, optional</span>
       <input type="text" name="selector_key" placeholder="role" size="8">
@@ -162,8 +162,8 @@ reachable from this machine.</p>
 <p class="note">The context pool is what the engine is given; llama.cpp divides
 it across its parallel slots, so the number one request may use is the smaller
 one above.</p>
-<p class="note">The memory figure is the weights plus half again for the KV pool
-plus two for the engine — an estimate, and editable. It is filled in rather than
+<p class="note">Memory: {{.MemoryBasis}}. Change the context pool and this changes
+with it, by the KV cache's share — an estimate, and editable. It is filled in rather than
 left blank because a CPU profile has no card to be short of: with no figure,
 every ready node is a candidate and the first one picked is whichever the
 scheduler reached, not whichever can hold the model. A node label does the same

llmgw-plan.md

@@ -1349,10 +1349,9 @@ work_dir: /var/lib/llmgw-agent
 - [x] Reconciler: desired vs actual, replica lifecycle, node expiry, backoff
 - [x] `EndpointSource` wired from the reconciler to the router
 - [x] `ctl`: nodes, models, replicas, cordon, uncordon, drain, explain-placement
-- [ ] `ctl logs`: needs a request/response pair on the agent stream, which the
-      protocol does not have. The failure path already carries an engine log
-      tail with every status report, so the common case is covered; what is
-      missing is reading a healthy shard's output on demand.
+- [x] `ctl logs`: a request/response pair on the agent stream (LogsRequest,
+      LogsReply) reads a node's agent log and any shard's engine output on
+      demand, from the CLI and from the node's page under /admin.
 - **Exit criterion (met against a simulated agent fleet):** shutting down a node
   moves its replicas elsewhere while the router keeps serving. Measured in
   `test/e2e`: a node hosting a live replica is killed mid-traffic; the replica is
@@ -1452,32 +1451,39 @@ The launch path is now built too:
 - [x] Model allowlists and key→band mapping. With auth on, the band comes from
       the key and the header is ignored — it was trusted only while there was
       no identity to consult, which let any client promote itself.
-- [ ] Budget fallback
-- [ ] Routing profiles from configuration plus the `x-llmgw-profile` header
+- [x] Budget fallback
+- [x] Routing profiles from configuration plus the `x-llmgw-profile` header
 - [x] Approximate radix tree (§7.5 tier 1), with its thresholds and the
       `llmgw_prefix_decisions_total` breakdown. Measured against sim-engines
       whose caches were too small for the working set: 22.5% → 81.7% cache
       hits. It does nothing when the working set fits in one replica's cache,
       which is worth knowing rather than assuming.
-- [ ] Fallback models and pool spillover
+- [x] Fallback models
+- [ ] Pool spillover
 - [x] `ctl bench`: a load ramp that reports capacity as the last concurrency
       level meeting the SLO, plus the queue depth and KV usage observed there,
       so `weight`, `target_waiting` and `kv_ceiling` stop being guesses
 - [ ] `ctl plan <model> --target-rps N`, which needs capacity profiles stored
       rather than printed
-- [ ] OTel tracing
+- [x] OTel tracing: an inference span with admission, schedule and each
+      upstream attempt under it, over OTLP/HTTP when `tracing.endpoint` is
+      set, and the traceparent carried on to the engine
 - [x] systemd units and `docs/onboarding.md`
 - [x] Release binaries: static linux/amd64, linux/arm64 and darwin builds
       attached to each tag, so installing on a node does not need a Go toolchain
-- [ ] Reference documentation beyond the plan itself
+- [x] Reference documentation beyond the plan itself: `docs/`, with
+      `docs/reference.md` generated from the configuration structs
 
 ### Phase 6 — v2 ideas
 - [ ] Precise KV index (§7.5 tier 2) — vLLM KV events plus tokenizer integration
 - [ ] Dynamo-style block cost function, tuning the `softmax` picker
 - [ ] Admission displacement (dropping requests from lower bands)
-- [ ] Autoscaling and priority-based preemption
+- [x] Autoscaling on load: a replica added while requests wait, up to a
+      model's max, and taken back after ten quiet minutes
+- [ ] Priority-based preemption
 - [ ] Control plane HA (Raft)
-- [ ] Weight downloading and cache management
+- [x] Weight downloading and cache management: the mirror, agent fetches
+      checked against their digest, and reclaiming what nothing uses
 - [ ] Prefill/decode disaggregation (§12)
 - [ ] LoRA adapter routing (affinity to a replica that already has the adapter)
 - [ ] External provider support (as a fallback)