Tell clients how much of their rate limit is left

SDKs and editors pace themselves from OpenAI's x-ratelimit-* headers, and
the gateway sent none: only Retry-After, and only on the refusal they would
have avoided. A limit can be set at several levels of the tenancy tree, each
counted in its own pool, so the headers report the one with least left —
the one that will refuse first — after this request's own charge, on
refusals and successes alike. The windows slide, so the reset figure is the
next minute boundary, when a whole window has moved on. A limit that is not
set is not advertised.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

CHANGELOG.md

@@ -20,6 +20,231 @@ is only released once its **exit criterion** is met, not when its code lands.
 Until v1.0.0 the minor version is the breaking one: configuration formats and Go
 APIs change between minors without a deprecation period.
 
+## Unreleased
+
+Changes since v0.10.3, published as release candidates (v0.10.4-rc.N) until
+there are enough for v0.10.4.
+
+### Restarting replicas
+
+`llmgw ctl restart REPLICA... | --model M | --all` replaces replicas with new
+ones. Each replacement is started beside the replica it replaces, which keeps
+serving until the new one is ready; replicas of one model go one at a time.
+Where no second copy fits anywhere, the old replica is stopped after the drain
+grace period and the new one starts in its place. This is how running engines
+pick up the keys introduced in v0.10.3, an edited profile, or a new engine
+version.
+
+### A model that keeps failing is parked, and everybody is told
+
+A model whose replicas failed at every start was retried forever: kev-0.5b,
+whose architecture the nodes' llama.cpp did not know, was started 155 times in
+eight hours. After five failures in a row a model is now parked — not placed
+again until its definition changes or `llmgw ctl retry MODEL` is run — and
+`ctl status`, `ctl models` and the admin API say so with the last error.
+
+A request for a model that will not come up — parked, or fitting on no node —
+is answered at once with the reason, instead of "the model is loading; retry
+shortly", which a client would retry forever. Each request still makes the
+control plane try placing it afresh, so a model that fits once room is made is
+not refused on an old answer. "no placement for model" is logged when its
+reason changes or every ten minutes, not on every pass: it was 250 identical
+lines in two days for one model.
+
+### Metrics and alerts for what goes wrong
+
+New series: `llmgw_model_replicas_desired`, `llmgw_model_start_failures`,
+`llmgw_model_parked`, `llmgw_model_unplaceable`, `llmgw_node_engine_install`,
+`llmgw_control_plane_leader` and `llmgw_certificate_expiry_timestamp_seconds`.
+`llmgw_model_replicas_ready` is now published at 0 for every model that should
+be running; it was published only while a replica was ready, so
+ModelHasNothingServing could never fire. New alerts: ModelParked,
+ModelFailingToStart, ModelCannotBePlaced, EngineInstallFailed,
+ControlPlaneLeadership, CertificateExpiresSoon and CertificateExpiresVerySoon.
+
+`ctl status` counts a model against the replicas it should have now rather
+than its minimum, so an on-demand model that was asked for and fitted nowhere
+no longer reads as satisfied.
+
+### Backups
+
+`llmgw server backup` writes the store and `/etc/llmgw` — configuration,
+secrets, and the certificate authority's key — to one archive readable by root
+only, keeping the newest 14. `deploy/systemd/llmgw-backup.timer` runs it
+nightly. The control plane publishes the newest archive's time as
+`llmgw_backup_last_success_timestamp_seconds`, and BackupTooOld fires when it is
+more than 26 hours old or there has never been one. docs/install.md now says how
+to restore, including why running engines survive it: the engine-key secret
+comes back with the store.
+
+### Following a request
+
+Every response carries `X-Request-Id` — the client's own when it sends a plain
+one, otherwise one the gateway makes — and the engine receives it too. One
+access log line per request records it with the model asked for and served,
+the replica, status, time queued, time to the first token, total time,
+tokens, and the key, project and user. Usage records keep the id, replica,
+queue time and time to first token, and `llmgw ctl request ID` reads one back:
+the answer to "why was this slow?".
+
+### Rate-limit headers
+
+Responses carry OpenAI's `x-ratelimit-limit-*`, `x-ratelimit-remaining-*` and
+`x-ratelimit-reset-*` for requests and tokens, from whichever level of the
+tenancy tree will refuse first, so SDKs and editors can slow down before they
+are refused.
+
+### Memory sized from the model
+
+The mirror's serve form sizes a llama.cpp model's memory from its GGUF header:
+weights, plus the KV cache for the context pool from its layers, KV heads and
+head width, plus the engine's own. At a 16k pool that cache ranges from 0.44 GB
+to 5.25 GB across the models on the live mirror, which the old "weights plus
+half again plus two" could not see.
+
+### Security
+
+- **Alertmanager answered anyone on the network**, including requests to
+  silence alerts. The monitoring README now binds it to loopback, where
+  Prometheus beside it is its only client.
+- **Login and signup could be made to use gigabytes.** Each attempt hashes with
+  64 MiB of argon2, with no bound on how many ran at once, and the per-address
+  throttle grew by one entry per address tried. Hashes now run a few at a time
+  and the rest wait; the throttle expires old entries, and when it is still
+  full it refuses new addresses rather than dropping live ones — which would
+  have let a flood of made-up addresses reset the throttle on a real one.
+- **Any key holder could grow Prometheus without bound** by asking for made-up
+  model names, each of which became a metric label. Requests are counted under
+  the catalog's name for their model, or "unknown".
+- **A SQLite backup left a full copy of the store readable** by local users
+  while it ran. The snapshot is now made in a private directory, and
+  llmgw-backup.service sets `UMask=0077`.
+- **An engine archive could write outside its directory through chained
+  links.** Archives are unpacked through `os.Root`, and a link may only point
+  at a file beside it. Every llama.cpp b10991 archive, CPU and CUDA, already
+  follows that rule.
+
+### Fixes
+
+- Tests that slept a fixed time and then asserted now wait for their
+  condition. They are the likeliest cause of CI's intermittent failures on a
+  loaded runner.
+- `ctl` reported a change the API had queued (202 Accepted) as an error, as
+  `ctl restart` did on its first real use.
+
+### Releases
+
+Small fixes are published as release candidates of the next version —
+v0.10.4-rc.1, rc.2 — which sort after v0.10.3 and before v0.10.4. Their release
+notes come from this section, and they are marked as pre-releases. Onboarding
+treats a release candidate as a release and downloads it.
+
+## v0.10.3 — 2026-09-30
+
+Security fixes from a review of v0.10.2. Three of them were confirmed on the
+live cluster before being fixed.
+
+### Engines demand a key only the router holds
+
+Engines listened on every address of their node with no key. A request from a
+neighbouring node reached the model directly, past the gateway's
+authentication, quotas and accounting. Each replica's engine is now started
+with a key derived from the replica id and a secret in the store. The router,
+its metrics scrape, the agent's readiness check and `ctl bench` send it.
+llama.cpp and vLLM receive it through their environment, not their command
+line. Engines started before this keep running without a key until their
+replica is next started.
+
+### The mirror's management API answers only who it was told to
+
+Without `--api-allow`, the management API answered anyone who could reach it,
+including every node. It now answers this machine only; the control plane is
+let in with `--api-allow` or mutual TLS. **Upgrading a mirror set up by hand:
+add `--api-allow <control plane address>` to its unit.** Downloads started
+through the API go to http and https only, and never to loopback, private or
+link-local addresses (`--allow-private-sources` to permit that). Model,
+version and file names can no longer be `..` or start with a dot.
+
+### The admin listener refuses what a browser page could send
+
+The admin API has no login and trusts whoever reaches it. A page open in an
+operator's browser, reaching it over an ssh tunnel, could post a model
+definition. The listener now requires Host to be an address or localhost,
+refuses cross-site requests, and takes changes only as `application/json`.
+A profile's `env` may no longer set variables that make an engine run other
+code (`LD_*`, `PATH`, `PYTHON*` start-up variables, `BASH_ENV` and similar).
+They are refused where a model is defined and dropped again before an engine
+starts.
+
+### Fixes
+
+- The agent installs only engine versions that look like one, whatever the
+  control plane sent; a version of `../x` could have installed outside
+  `/opt/llmgw/engines`.
+
+## v0.10.2 — 2026-09-29
+
+- After `server init`'s screen closes, only the administrator's password and
+  the next steps are printed, not the whole run again under the screen that
+  just showed it. The whole run is kept in `/var/lib/llmgw/init.log`, root's
+  only, and printed in full when init fails.
+- The screens' spinner and text field are llmgw's own rather than from
+  bubbles, one dependency fewer. Every llmgw command still asks the terminal
+  for its colours as it starts, which bubbletea v1 does on purpose; a terminal
+  that never answers waits for its timeout.
+
+## v0.10.1 — 2026-09-29
+
+### `llmgw ctl fabric-test`: two nodes, rail by rail
+
+`llmgw doctor` checks a node on its own, and every check can pass while two
+nodes still cannot reach each other: a cable into the wrong switch, a
+partition key, a RoCE network with no route. `llmgw ctl fabric-test A B` runs
+`ib_write_bw` from each of A's adapters on a fabric to the same adapter on B,
+one rail at a time, through the agents — no SSH between nodes, and perftest is
+installed the first time — and grades each against its link's rate: ok at 80%,
+slow at 50%, fail below or when nothing got through.
+
+Agents now say which requests they answer beyond the usual (`features`), so
+one too old for the test is refused at once instead of waited for.
+
+Written against perftest's real output and tested with it, but not yet run
+over real InfiniBand or RoCE: this cluster has neither.
+
+### The mirror on its own machine, in two commands
+
+`llmgw mirror init --control-plane <address>` sets a machine up as the model
+mirror: the user, `/srv/models`, and a service serving files to every node on
+:8081 and the management API on :8082. `server init --mirror
+http://<mirror>:8082` uses it instead of running one beside the control plane.
+It took a unit written by hand and an edit to `server.yaml` before.
+
+The management API can fetch and delete models; `mirror serve --api-allow`
+makes it answer only the addresses given and the machine itself, and `mirror
+init` gives it the control plane's. With `--no-node`, the starter model is
+fetched and catalogued anyway, unpinned, and starts on the first node added.
+
+Tried on three empty VMs: mirror, control plane and a node, and a request
+answered by the starter model on the node.
+
+### `server init` asks, and `node add` takes several nodes
+
+- **`sudo llmgw server init`** in a terminal, without `--admin`, asks for the
+  administrator's email, the other names nodes will reach the machine by
+  (showing the addresses it already found), SQLite or PostgreSQL, and whether
+  to run the mirror, a node and the starter model here. Then its steps fill
+  in as a list, downloads updating in place, and the administrator's password
+  stays in a box until the end — and is printed again after, with the whole
+  run, so it is in the terminal's history. The last screen says how to add a
+  node and to back up the authority's key. `--plain`, or no terminal, prints
+  lines as before.
+- **`llmgw node add a b c`** adds each in turn and ends with which were added.
+  A fabric named for one node's adapters is recognised on the next by what the
+  adapters report, and not asked again.
+
+Both tried on empty VMs: init from the form to a starter model answering, and
+two nodes added to it in one command in 40 seconds.
+
 ## v0.10.0 — 2026-09-29
 
 Independent InfiniBand fabrics and NVLink domains, told apart, drawn and

api/gen/llmgw/v1/agent.pb.go

@@ -103,6 +103,7 @@ type AgentMessage struct {
 	//	*AgentMessage_Shard
 	//	*AgentMessage_Logs
 	//	*AgentMessage_FabricScan
+	//	*AgentMessage_FabricTest
 	Msg           isAgentMessage_Msg `protobuf_oneof:"msg"`
 	unknownFields protoimpl.UnknownFields
 	sizeCache     protoimpl.SizeCache
@@ -190,6 +191,15 @@ func (x *AgentMessage) GetFabricScan() *FabricScanReply {
 	return nil
 }
 
+func (x *AgentMessage) GetFabricTest() *FabricTestReply {
+	if x != nil {
+		if x, ok := x.Msg.(*AgentMessage_FabricTest); ok {
+			return x.FabricTest
+		}
+	}
+	return nil
+}
+
 type isAgentMessage_Msg interface {
 	isAgentMessage_Msg()
 }
@@ -214,6 +224,10 @@ type AgentMessage_FabricScan struct {
 	FabricScan *FabricScanReply `protobuf:"bytes,5,opt,name=fabric_scan,json=fabricScan,proto3,oneof"`
 }
 
+type AgentMessage_FabricTest struct {
+	FabricTest *FabricTestReply `protobuf:"bytes,6,opt,name=fabric_test,json=fabricTest,proto3,oneof"`
+}
+
 func (*AgentMessage_Register) isAgentMessage_Msg() {}
 
 func (*AgentMessage_Heartbeat) isAgentMessage_Msg() {}
@@ -224,6 +238,8 @@ func (*AgentMessage_Logs) isAgentMessage_Msg() {}
 
 func (*AgentMessage_FabricScan) isAgentMessage_Msg() {}
 
+func (*AgentMessage_FabricTest) isAgentMessage_Msg() {}
+
 // ControlMessage is anything the control plane tells a node.
 type ControlMessage struct {
 	state protoimpl.MessageState `protogen:"open.v1"`
@@ -237,6 +253,7 @@ type ControlMessage struct {
 	//	*ControlMessage_Provision
 	//	*ControlMessage_Logs
 	//	*ControlMessage_FabricScan
+	//	*ControlMessage_FabricTest
 	Msg           isControlMessage_Msg `protobuf_oneof:"msg"`
 	unknownFields protoimpl.UnknownFields
 	sizeCache     protoimpl.SizeCache
@@ -351,6 +368,15 @@ func (x *ControlMessage) GetFabricScan() *FabricScanRequest {
 	return nil
 }
 
+func (x *ControlMessage) GetFabricTest() *FabricTestRequest {
+	if x != nil {
+		if x, ok := x.Msg.(*ControlMessage_FabricTest); ok {
+			return x.FabricTest
+		}
+	}
+	return nil
+}
+
 type isControlMessage_Msg interface {
 	isControlMessage_Msg()
 }
@@ -387,6 +413,10 @@ type ControlMessage_FabricScan struct {
 	FabricScan *FabricScanRequest `protobuf:"bytes,8,opt,name=fabric_scan,json=fabricScan,proto3,oneof"`
 }
 
+type ControlMessage_FabricTest struct {
+	FabricTest *FabricTestRequest `protobuf:"bytes,9,opt,name=fabric_test,json=fabricTest,proto3,oneof"`
+}
+
 func (*ControlMessage_Start) isControlMessage_Msg() {}
 
 func (*ControlMessage_Stop) isControlMessage_Msg() {}
@@ -403,6 +433,8 @@ func (*ControlMessage_Logs) isControlMessage_Msg() {}
 
 func (*ControlMessage_FabricScan) isControlMessage_Msg() {}
 
+func (*ControlMessage_FabricTest) isControlMessage_Msg() {}
+
 // Register is the first message on every stream. It carries the node's full
 // hardware inventory, so the control plane needs no separate discovery call and
 // a reconnecting agent automatically refreshes what the scheduler knows.
@@ -478,8 +510,13 @@ type Register struct {
 	// vLLM, it answered "unsupported", and every node of a cluster half way
 	// through an upgrade showed as failing.
 	InstallableEngines []string `protobuf:"bytes,16,rep,name=installable_engines,json=installableEngines,proto3" json:"installable_engines,omitempty"`
-	unknownFields      protoimpl.UnknownFields
-	sizeCache          protoimpl.SizeCache
+	// features are the requests this agent answers beyond the ones every agent
+	// does: "fabric-test". An agent that does not know a request ignores it, so
+	// without this the control plane would wait minutes for an answer that is
+	// never coming.
+	Features      []string `protobuf:"bytes,17,rep,name=features,proto3" json:"features,omitempty"`
+	unknownFields protoimpl.UnknownFields
+	sizeCache     protoimpl.SizeCache
 }
 
 func (x *Register) Reset() {
@@ -624,6 +661,13 @@ func (x *Register) GetInstallableEngines() []string {
 	return nil
 }
 
+func (x *Register) GetFeatures() []string {
+	if x != nil {
+		return x.Features
+	}
+	return nil
+}
+
 // Host describes the operating system and processor of a node.
 type Host struct {
 	state protoimpl.MessageState `protogen:"open.v1"`
@@ -2481,7 +2525,11 @@ type StartShard struct {
 	Fabric string `protobuf:"bytes,14,opt,name=fabric,proto3" json:"fabric,omitempty"`
 	// profile_fp identifies the version of the profile, so the agent can say
 	// which one a running shard was started from. The agent never interprets it.
-	ProfileFp     string `protobuf:"bytes,15,opt,name=profile_fp,json=profileFp,proto3" json:"profile_fp,omitempty"`
+	ProfileFp string `protobuf:"bytes,15,opt,name=profile_fp,json=profileFp,proto3" json:"profile_fp,omitempty"`
+	// api_key is what the engine demands of every request, derived by the
+	// control plane from the replica id. The router sends it; nothing else
+	// that can reach the node's ports has it.
+	ApiKey        string `protobuf:"bytes,16,opt,name=api_key,json=apiKey,proto3" json:"api_key,omitempty"`
 	unknownFields protoimpl.UnknownFields
 	sizeCache     protoimpl.SizeCache
 }
@@ -2621,6 +2669,13 @@ func (x *StartShard) GetProfileFp() string {
 	return ""
 }
 
+func (x *StartShard) GetApiKey() string {
+	if x != nil {
+		return x.ApiKey
+	}
+	return ""
+}
+
 type EngineProfile struct {
 	state           protoimpl.MessageState `protogen:"open.v1"`
 	Name            string                 `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"`
@@ -3223,19 +3278,203 @@ func (x *FabricScanReply) GetTookMs() int64 {
 	return 0
 }
 
+// FabricTestRequest asks a node for one side of an RDMA bandwidth test
+// between two nodes: ib_write_bw, from perftest, over one adapter. The
+// server side is asked first and answers when the test is over; the client
+// dials it. What it proves is what a replica spanning nodes needs and nothing
+// on one node can show: that these two adapters reach each other through the
+// fabric, and how fast.
+type FabricTestRequest struct {
+	state     protoimpl.MessageState `protogen:"open.v1"`
+	RequestId string                 `protobuf:"bytes,1,opt,name=request_id,json=requestId,proto3" json:"request_id,omitempty"`
+	// role is "server" or "client".
+	Role string `protobuf:"bytes,2,opt,name=role,proto3" json:"role,omitempty"`
+	Hca  string `protobuf:"bytes,3,opt,name=hca,proto3" json:"hca,omitempty"`
+	Port int32  `protobuf:"varint,4,opt,name=port,proto3" json:"port,omitempty"`
+	// peer is the server's management address, for the client. perftest
+	// exchanges its connection details over TCP before any RDMA.
+	Peer string `protobuf:"bytes,5,opt,name=peer,proto3" json:"peer,omitempty"`
+	// tcp_port is that exchange's port, the same on both sides.
+	TcpPort       int32 `protobuf:"varint,6,opt,name=tcp_port,json=tcpPort,proto3" json:"tcp_port,omitempty"`
+	Seconds       int32 `protobuf:"varint,7,opt,name=seconds,proto3" json:"seconds,omitempty"`
+	unknownFields protoimpl.UnknownFields
+	sizeCache     protoimpl.SizeCache
+}
+
+func (x *FabricTestRequest) Reset() {
+	*x = FabricTestRequest{}
+	mi := &file_llmgw_v1_agent_proto_msgTypes[31]
+	ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+	ms.StoreMessageInfo(mi)
+}
+
+func (x *FabricTestRequest) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*FabricTestRequest) ProtoMessage() {}
+
+func (x *FabricTestRequest) ProtoReflect() protoreflect.Message {
+	mi := &file_llmgw_v1_agent_proto_msgTypes[31]
+	if x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use FabricTestRequest.ProtoReflect.Descriptor instead.
+func (*FabricTestRequest) Descriptor() ([]byte, []int) {
+	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{31}
+}
+
+func (x *FabricTestRequest) GetRequestId() string {
+	if x != nil {
+		return x.RequestId
+	}
+	return ""
+}
+
+func (x *FabricTestRequest) GetRole() string {
+	if x != nil {
+		return x.Role
+	}
+	return ""
+}
+
+func (x *FabricTestRequest) GetHca() string {
+	if x != nil {
+		return x.Hca
+	}
+	return ""
+}
+
+func (x *FabricTestRequest) GetPort() int32 {
+	if x != nil {
+		return x.Port
+	}
+	return 0
+}
+
+func (x *FabricTestRequest) GetPeer() string {
+	if x != nil {
+		return x.Peer
+	}
+	return ""
+}
+
+func (x *FabricTestRequest) GetTcpPort() int32 {
+	if x != nil {
+		return x.TcpPort
+	}
+	return 0
+}
+
+func (x *FabricTestRequest) GetSeconds() int32 {
+	if x != nil {
+		return x.Seconds
+	}
+	return 0
+}
+
+// FabricTestReply is what one side measured.
+type FabricTestReply struct {
+	state     protoimpl.MessageState `protogen:"open.v1"`
+	RequestId string                 `protobuf:"bytes,1,opt,name=request_id,json=requestId,proto3" json:"request_id,omitempty"`
+	// gbps is the average bandwidth, in gigabits a second.
+	Gbps float64 `protobuf:"fixed64,2,opt,name=gbps,proto3" json:"gbps,omitempty"`
+	// output is the end of the tool's output, for when the number is not
+	// enough.
+	Output        string `protobuf:"bytes,3,opt,name=output,proto3" json:"output,omitempty"`
+	Error         string `protobuf:"bytes,4,opt,name=error,proto3" json:"error,omitempty"`
+	TookMs        int64  `protobuf:"varint,5,opt,name=took_ms,json=tookMs,proto3" json:"took_ms,omitempty"`
+	unknownFields protoimpl.UnknownFields
+	sizeCache     protoimpl.SizeCache
+}
+
+func (x *FabricTestReply) Reset() {
+	*x = FabricTestReply{}
+	mi := &file_llmgw_v1_agent_proto_msgTypes[32]
+	ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+	ms.StoreMessageInfo(mi)
+}
+
+func (x *FabricTestReply) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*FabricTestReply) ProtoMessage() {}
+
+func (x *FabricTestReply) ProtoReflect() protoreflect.Message {
+	mi := &file_llmgw_v1_agent_proto_msgTypes[32]
+	if x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use FabricTestReply.ProtoReflect.Descriptor instead.
+func (*FabricTestReply) Descriptor() ([]byte, []int) {
+	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{32}
+}
+
+func (x *FabricTestReply) GetRequestId() string {
+	if x != nil {
+		return x.RequestId
+	}
+	return ""
+}
+
+func (x *FabricTestReply) GetGbps() float64 {
+	if x != nil {
+		return x.Gbps
+	}
+	return 0
+}
+
+func (x *FabricTestReply) GetOutput() string {
+	if x != nil {
+		return x.Output
+	}
+	return ""
+}
+
+func (x *FabricTestReply) GetError() string {
+	if x != nil {
+		return x.Error
+	}
+	return ""
+}
+
+func (x *FabricTestReply) GetTookMs() int64 {
+	if x != nil {
+		return x.TookMs
+	}
+	return 0
+}
+
 var File_llmgw_v1_agent_proto protoreflect.FileDescriptor
 
 const file_llmgw_v1_agent_proto_rawDesc = "" +
 	"\n" +
-	"\x14llmgw/v1/agent.proto\x12\bllmgw.v1\"\x94\x02\n" +
+	"\x14llmgw/v1/agent.proto\x12\bllmgw.v1\"\xd2\x02\n" +
 	"\fAgentMessage\x120\n" +
 	"\bregister\x18\x01 \x01(\v2\x12.llmgw.v1.RegisterH\x00R\bregister\x123\n" +
 	"\theartbeat\x18\x02 \x01(\v2\x13.llmgw.v1.HeartbeatH\x00R\theartbeat\x12-\n" +
 	"\x05shard\x18\x03 \x01(\v2\x15.llmgw.v1.ShardStatusH\x00R\x05shard\x12)\n" +
 	"\x04logs\x18\x04 \x01(\v2\x13.llmgw.v1.LogsReplyH\x00R\x04logs\x12<\n" +
 	"\vfabric_scan\x18\x05 \x01(\v2\x19.llmgw.v1.FabricScanReplyH\x00R\n" +
-	"fabricScanB\x05\n" +
-	"\x03msg\"\xa5\x03\n" +
+	"fabricScan\x12<\n" +
+	"\vfabric_test\x18\x06 \x01(\v2\x19.llmgw.v1.FabricTestReplyH\x00R\n" +
+	"fabricTestB\x05\n" +
+	"\x03msg\"\xe5\x03\n" +
 	"\x0eControlMessage\x12,\n" +
 	"\x05start\x18\x01 \x01(\v2\x14.llmgw.v1.StartShardH\x00R\x05start\x12)\n" +
 	"\x04stop\x18\x02 \x01(\v2\x13.llmgw.v1.StopShardH\x00R\x04stop\x12,\n" +
@@ -3245,8 +3484,10 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
 	"\tprovision\x18\x06 \x01(\v2\x13.llmgw.v1.ProvisionH\x00R\tprovision\x12+\n" +
 	"\x04logs\x18\a \x01(\v2\x15.llmgw.v1.LogsRequestH\x00R\x04logs\x12>\n" +
 	"\vfabric_scan\x18\b \x01(\v2\x1b.llmgw.v1.FabricScanRequestH\x00R\n" +
-	"fabricScanB\x05\n" +
-	"\x03msg\"\xc7\x06\n" +
+	"fabricScan\x12>\n" +
+	"\vfabric_test\x18\t \x01(\v2\x1b.llmgw.v1.FabricTestRequestH\x00R\n" +
+	"fabricTestB\x05\n" +
+	"\x03msg\"\xe3\x06\n" +
 	"\bRegister\x12\x17\n" +
 	"\anode_id\x18\x01 \x01(\tR\x06nodeId\x12\x1a\n" +
 	"\bhostname\x18\x02 \x01(\tR\bhostname\x12\x1b\n" +
@@ -3264,7 +3505,8 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
 	"\x04host\x18\r \x01(\v2\x0e.llmgw.v1.HostR\x04host\x12%\n" +
 	"\x0eengines_probed\x18\x0e \x01(\bR\renginesProbed\x12\"\n" +
 	"\x04imex\x18\x0f \x01(\v2\x0e.llmgw.v1.IMEXR\x04imex\x12/\n" +
-	"\x13installable_engines\x18\x10 \x03(\tR\x12installableEngines\x1a9\n" +
+	"\x13installable_engines\x18\x10 \x03(\tR\x12installableEngines\x12\x1a\n" +
+	"\bfeatures\x18\x11 \x03(\tR\bfeatures\x1a9\n" +
 	"\vLabelsEntry\x12\x10\n" +
 	"\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +
 	"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\x1aA\n" +
@@ -3462,7 +3704,7 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
 	"\n" +
 	"bytes_done\x18\x02 \x01(\x03R\tbytesDone\x12\x1f\n" +
 	"\vbytes_total\x18\x03 \x01(\x03R\n" +
-	"bytesTotal\"\xd9\x03\n" +
+	"bytesTotal\"\xf2\x03\n" +
 	"\n" +
 	"StartShard\x12\x1d\n" +
 	"\n" +
@@ -3485,7 +3727,8 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
 	"\x04pool\x18\r \x01(\tR\x04pool\x12\x16\n" +
 	"\x06fabric\x18\x0e \x01(\tR\x06fabric\x12\x1d\n" +
 	"\n" +
-	"profile_fp\x18\x0f \x01(\tR\tprofileFp\"\xfd\x02\n" +
+	"profile_fp\x18\x0f \x01(\tR\tprofileFp\x12\x17\n" +
+	"\aapi_key\x18\x10 \x01(\tR\x06apiKey\"\xfd\x02\n" +
 	"\rEngineProfile\x12\x12\n" +
 	"\x04name\x18\x01 \x01(\tR\x04name\x12\x16\n" +
 	"\x06engine\x18\x02 \x01(\tR\x06engine\x12\x1b\n" +
@@ -3536,7 +3779,23 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
 	"request_id\x18\x01 \x01(\tR\trequestId\x12\x16\n" +
 	"\x06output\x18\x02 \x01(\tR\x06output\x12\x14\n" +
 	"\x05error\x18\x03 \x01(\tR\x05error\x12\x17\n" +
-	"\atook_ms\x18\x04 \x01(\x03R\x06tookMs2O\n" +
+	"\atook_ms\x18\x04 \x01(\x03R\x06tookMs\"\xb5\x01\n" +
+	"\x11FabricTestRequest\x12\x1d\n" +
+	"\n" +
+	"request_id\x18\x01 \x01(\tR\trequestId\x12\x12\n" +
+	"\x04role\x18\x02 \x01(\tR\x04role\x12\x10\n" +
+	"\x03hca\x18\x03 \x01(\tR\x03hca\x12\x12\n" +
+	"\x04port\x18\x04 \x01(\x05R\x04port\x12\x12\n" +
+	"\x04peer\x18\x05 \x01(\tR\x04peer\x12\x19\n" +
+	"\btcp_port\x18\x06 \x01(\x05R\atcpPort\x12\x18\n" +
+	"\aseconds\x18\a \x01(\x05R\aseconds\"\x8b\x01\n" +
+	"\x0fFabricTestReply\x12\x1d\n" +
+	"\n" +
+	"request_id\x18\x01 \x01(\tR\trequestId\x12\x12\n" +
+	"\x04gbps\x18\x02 \x01(\x01R\x04gbps\x12\x16\n" +
+	"\x06output\x18\x03 \x01(\tR\x06output\x12\x14\n" +
+	"\x05error\x18\x04 \x01(\tR\x05error\x12\x17\n" +
+	"\atook_ms\x18\x05 \x01(\x03R\x06tookMs2O\n" +
 	"\fAgentService\x12?\n" +
 	"\aConnect\x12\x16.llmgw.v1.AgentMessage\x1a\x18.llmgw.v1.ControlMessage(\x010\x01B\x93\x01\n" +
 	"\fcom.llmgw.v1B\n" +
@@ -3555,7 +3814,7 @@ func file_llmgw_v1_agent_proto_rawDescGZIP() []byte {
 }
 
 var file_llmgw_v1_agent_proto_enumTypes = make([]protoimpl.EnumInfo, 1)
-var file_llmgw_v1_agent_proto_msgTypes = make([]protoimpl.MessageInfo, 37)
+var file_llmgw_v1_agent_proto_msgTypes = make([]protoimpl.MessageInfo, 39)
 var file_llmgw_v1_agent_proto_goTypes = []any{
 	(ShardStatus_State)(0),    // 0: llmgw.v1.ShardStatus.State
 	(*AgentMessage)(nil),      // 1: llmgw.v1.AgentMessage
@@ -3589,12 +3848,14 @@ var file_llmgw_v1_agent_proto_goTypes = []any{
 	(*LogsReply)(nil),         // 29: llmgw.v1.LogsReply
 	(*FabricScanRequest)(nil), // 30: llmgw.v1.FabricScanRequest
 	(*FabricScanReply)(nil),   // 31: llmgw.v1.FabricScanReply
-	nil,                       // 32: llmgw.v1.Register.LabelsEntry
-	nil,                       // 33: llmgw.v1.Register.EngineVersionsEntry
-	nil,                       // 34: llmgw.v1.GPU.NicPathsEntry
-	nil,                       // 35: llmgw.v1.IBPort.CountersEntry
-	nil,                       // 36: llmgw.v1.Heartbeat.EngineVersionsEntry
-	nil,                       // 37: llmgw.v1.EngineProfile.EnvEntry
+	(*FabricTestRequest)(nil), // 32: llmgw.v1.FabricTestRequest
+	(*FabricTestReply)(nil),   // 33: llmgw.v1.FabricTestReply
+	nil,                       // 34: llmgw.v1.Register.LabelsEntry
+	nil,                       // 35: llmgw.v1.Register.EngineVersionsEntry
+	nil,                       // 36: llmgw.v1.GPU.NicPathsEntry
+	nil,                       // 37: llmgw.v1.IBPort.CountersEntry
+	nil,                       // 38: llmgw.v1.Heartbeat.EngineVersionsEntry
+	nil,                       // 39: llmgw.v1.EngineProfile.EnvEntry
 }
 var file_llmgw_v1_agent_proto_depIdxs = []int32{
 	3,  // 0: llmgw.v1.AgentMessage.register:type_name -> llmgw.v1.Register
@@ -3602,49 +3863,51 @@ var file_llmgw_v1_agent_proto_depIdxs = []int32{
 	20, // 2: llmgw.v1.AgentMessage.shard:type_name -> llmgw.v1.ShardStatus
 	29, // 3: llmgw.v1.AgentMessage.logs:type_name -> llmgw.v1.LogsReply
 	31, // 4: llmgw.v1.AgentMessage.fabric_scan:type_name -> llmgw.v1.FabricScanReply
-	22, // 5: llmgw.v1.ControlMessage.start:type_name -> llmgw.v1.StartShard
-	25, // 6: llmgw.v1.ControlMessage.stop:type_name -> llmgw.v1.StopShard
-	26, // 7: llmgw.v1.ControlMessage.drain:type_name -> llmgw.v1.DrainShard
-	27, // 8: llmgw.v1.ControlMessage.ack:type_name -> llmgw.v1.Ack
-	24, // 9: llmgw.v1.ControlMessage.remove_weights:type_name -> llmgw.v1.RemoveWeights
-	17, // 10: llmgw.v1.ControlMessage.provision:type_name -> llmgw.v1.Provision
-	28, // 11: llmgw.v1.ControlMessage.logs:type_name -> llmgw.v1.LogsRequest
-	30, // 12: llmgw.v1.ControlMessage.fabric_scan:type_name -> llmgw.v1.FabricScanRequest
-	32, // 13: llmgw.v1.Register.labels:type_name -> llmgw.v1.Register.LabelsEntry
-	7,  // 14: llmgw.v1.Register.gpus:type_name -> llmgw.v1.GPU
-	8,  // 15: llmgw.v1.Register.fabrics:type_name -> llmgw.v1.FabricAttachment
-	33, // 16: llmgw.v1.Register.engine_versions:type_name -> llmgw.v1.Register.EngineVersionsEntry
-	6,  // 17: llmgw.v1.Register.running_shards:type_name -> llmgw.v1.RunningShard
-	9,  // 18: llmgw.v1.Register.weights:type_name -> llmgw.v1.LocalWeights
-	5,  // 19: llmgw.v1.Register.storage:type_name -> llmgw.v1.Storage
-	4,  // 20: llmgw.v1.Register.host:type_name -> llmgw.v1.Host
-	14, // 21: llmgw.v1.Register.imex:type_name -> llmgw.v1.IMEX
-	0,  // 22: llmgw.v1.RunningShard.state:type_name -> llmgw.v1.ShardStatus.State
-	34, // 23: llmgw.v1.GPU.nic_paths:type_name -> llmgw.v1.GPU.NicPathsEntry
-	11, // 24: llmgw.v1.NodeLoad.shards:type_name -> llmgw.v1.ShardLoad
-	35, // 25: llmgw.v1.IBPort.counters:type_name -> llmgw.v1.IBPort.CountersEntry
-	13, // 26: llmgw.v1.IBPort.subnet_managers:type_name -> llmgw.v1.SubnetManager
-	5,  // 27: llmgw.v1.Heartbeat.storage:type_name -> llmgw.v1.Storage
-	9,  // 28: llmgw.v1.Heartbeat.weights:type_name -> llmgw.v1.LocalWeights
-	36, // 29: llmgw.v1.Heartbeat.engine_versions:type_name -> llmgw.v1.Heartbeat.EngineVersionsEntry
-	19, // 30: llmgw.v1.Heartbeat.provision:type_name -> llmgw.v1.ProvisionStatus
-	10, // 31: llmgw.v1.Heartbeat.load:type_name -> llmgw.v1.NodeLoad
-	12, // 32: llmgw.v1.Heartbeat.ib_ports:type_name -> llmgw.v1.IBPort
-	8,  // 33: llmgw.v1.Heartbeat.fabrics:type_name -> llmgw.v1.FabricAttachment
-	15, // 34: llmgw.v1.Heartbeat.gpu_fabric:type_name -> llmgw.v1.GPUFabricState
-	14, // 35: llmgw.v1.Heartbeat.imex:type_name -> llmgw.v1.IMEX
-	18, // 36: llmgw.v1.Provision.engines:type_name -> llmgw.v1.EngineWant
-	0,  // 37: llmgw.v1.ShardStatus.state:type_name -> llmgw.v1.ShardStatus.State
-	21, // 38: llmgw.v1.ShardStatus.fetch:type_name -> llmgw.v1.FetchProgress
-	23, // 39: llmgw.v1.StartShard.profile:type_name -> llmgw.v1.EngineProfile
-	37, // 40: llmgw.v1.EngineProfile.env:type_name -> llmgw.v1.EngineProfile.EnvEntry
-	1,  // 41: llmgw.v1.AgentService.Connect:input_type -> llmgw.v1.AgentMessage
-	2,  // 42: llmgw.v1.AgentService.Connect:output_type -> llmgw.v1.ControlMessage
-	42, // [42:43] is the sub-list for method output_type
-	41, // [41:42] is the sub-list for method input_type
-	41, // [41:41] is the sub-list for extension type_name
-	41, // [41:41] is the sub-list for extension extendee
-	0,  // [0:41] is the sub-list for field type_name
+	33, // 5: llmgw.v1.AgentMessage.fabric_test:type_name -> llmgw.v1.FabricTestReply
+	22, // 6: llmgw.v1.ControlMessage.start:type_name -> llmgw.v1.StartShard
+	25, // 7: llmgw.v1.ControlMessage.stop:type_name -> llmgw.v1.StopShard
+	26, // 8: llmgw.v1.ControlMessage.drain:type_name -> llmgw.v1.DrainShard
+	27, // 9: llmgw.v1.ControlMessage.ack:type_name -> llmgw.v1.Ack
+	24, // 10: llmgw.v1.ControlMessage.remove_weights:type_name -> llmgw.v1.RemoveWeights
+	17, // 11: llmgw.v1.ControlMessage.provision:type_name -> llmgw.v1.Provision
+	28, // 12: llmgw.v1.ControlMessage.logs:type_name -> llmgw.v1.LogsRequest
+	30, // 13: llmgw.v1.ControlMessage.fabric_scan:type_name -> llmgw.v1.FabricScanRequest
+	32, // 14: llmgw.v1.ControlMessage.fabric_test:type_name -> llmgw.v1.FabricTestRequest
+	34, // 15: llmgw.v1.Register.labels:type_name -> llmgw.v1.Register.LabelsEntry
+	7,  // 16: llmgw.v1.Register.gpus:type_name -> llmgw.v1.GPU
+	8,  // 17: llmgw.v1.Register.fabrics:type_name -> llmgw.v1.FabricAttachment
+	35, // 18: llmgw.v1.Register.engine_versions:type_name -> llmgw.v1.Register.EngineVersionsEntry
+	6,  // 19: llmgw.v1.Register.running_shards:type_name -> llmgw.v1.RunningShard
+	9,  // 20: llmgw.v1.Register.weights:type_name -> llmgw.v1.LocalWeights
+	5,  // 21: llmgw.v1.Register.storage:type_name -> llmgw.v1.Storage
+	4,  // 22: llmgw.v1.Register.host:type_name -> llmgw.v1.Host
+	14, // 23: llmgw.v1.Register.imex:type_name -> llmgw.v1.IMEX
+	0,  // 24: llmgw.v1.RunningShard.state:type_name -> llmgw.v1.ShardStatus.State
+	36, // 25: llmgw.v1.GPU.nic_paths:type_name -> llmgw.v1.GPU.NicPathsEntry
+	11, // 26: llmgw.v1.NodeLoad.shards:type_name -> llmgw.v1.ShardLoad
+	37, // 27: llmgw.v1.IBPort.counters:type_name -> llmgw.v1.IBPort.CountersEntry
+	13, // 28: llmgw.v1.IBPort.subnet_managers:type_name -> llmgw.v1.SubnetManager
+	5,  // 29: llmgw.v1.Heartbeat.storage:type_name -> llmgw.v1.Storage
+	9,  // 30: llmgw.v1.Heartbeat.weights:type_name -> llmgw.v1.LocalWeights
+	38, // 31: llmgw.v1.Heartbeat.engine_versions:type_name -> llmgw.v1.Heartbeat.EngineVersionsEntry
+	19, // 32: llmgw.v1.Heartbeat.provision:type_name -> llmgw.v1.ProvisionStatus
+	10, // 33: llmgw.v1.Heartbeat.load:type_name -> llmgw.v1.NodeLoad
+	12, // 34: llmgw.v1.Heartbeat.ib_ports:type_name -> llmgw.v1.IBPort
+	8,  // 35: llmgw.v1.Heartbeat.fabrics:type_name -> llmgw.v1.FabricAttachment
+	15, // 36: llmgw.v1.Heartbeat.gpu_fabric:type_name -> llmgw.v1.GPUFabricState
+	14, // 37: llmgw.v1.Heartbeat.imex:type_name -> llmgw.v1.IMEX
+	18, // 38: llmgw.v1.Provision.engines:type_name -> llmgw.v1.EngineWant
+	0,  // 39: llmgw.v1.ShardStatus.state:type_name -> llmgw.v1.ShardStatus.State
+	21, // 40: llmgw.v1.ShardStatus.fetch:type_name -> llmgw.v1.FetchProgress
+	23, // 41: llmgw.v1.StartShard.profile:type_name -> llmgw.v1.EngineProfile
+	39, // 42: llmgw.v1.EngineProfile.env:type_name -> llmgw.v1.EngineProfile.EnvEntry
+	1,  // 43: llmgw.v1.AgentService.Connect:input_type -> llmgw.v1.AgentMessage
+	2,  // 44: llmgw.v1.AgentService.Connect:output_type -> llmgw.v1.ControlMessage
+	44, // [44:45] is the sub-list for method output_type
+	43, // [43:44] is the sub-list for method input_type
+	43, // [43:43] is the sub-list for extension type_name
+	43, // [43:43] is the sub-list for extension extendee
+	0,  // [0:43] is the sub-list for field type_name
 }
 
 func init() { file_llmgw_v1_agent_proto_init() }
@@ -3658,6 +3921,7 @@ func file_llmgw_v1_agent_proto_init() {
 		(*AgentMessage_Shard)(nil),
 		(*AgentMessage_Logs)(nil),
 		(*AgentMessage_FabricScan)(nil),
+		(*AgentMessage_FabricTest)(nil),
 	}
 	file_llmgw_v1_agent_proto_msgTypes[1].OneofWrappers = []any{
 		(*ControlMessage_Start)(nil),
@@ -3668,6 +3932,7 @@ func file_llmgw_v1_agent_proto_init() {
 		(*ControlMessage_Provision)(nil),
 		(*ControlMessage_Logs)(nil),
 		(*ControlMessage_FabricScan)(nil),
+		(*ControlMessage_FabricTest)(nil),
 	}
 	type x struct{}
 	out := protoimpl.TypeBuilder{
@@ -3675,7 +3940,7 @@ func file_llmgw_v1_agent_proto_init() {
 			GoPackagePath: reflect.TypeOf(x{}).PkgPath(),
 			RawDescriptor: unsafe.Slice(unsafe.StringData(file_llmgw_v1_agent_proto_rawDesc), len(file_llmgw_v1_agent_proto_rawDesc)),
 			NumEnums:      1,
-			NumMessages:   37,
+			NumMessages:   39,
 			NumExtensions: 0,
 			NumServices:   1,
 		},

api/proto/llmgw/v1/agent.proto

@@ -25,6 +25,7 @@ message AgentMessage {
     ShardStatus shard = 3;
     LogsReply logs = 4;
     FabricScanReply fabric_scan = 5;
+    FabricTestReply fabric_test = 6;
   }
 }
 
@@ -39,6 +40,7 @@ message ControlMessage {
     Provision provision = 6;
     LogsRequest logs = 7;
     FabricScanRequest fabric_scan = 8;
+    FabricTestRequest fabric_test = 9;
   }
 }
 
@@ -125,6 +127,12 @@ message Register {
   // vLLM, it answered "unsupported", and every node of a cluster half way
   // through an upgrade showed as failing.
   repeated string installable_engines = 16;
+
+  // features are the requests this agent answers beyond the ones every agent
+  // does: "fabric-test". An agent that does not know a request ignores it, so
+  // without this the control plane would wait minutes for an answer that is
+  // never coming.
+  repeated string features = 17;
 }
 
 // Host describes the operating system and processor of a node.
@@ -516,6 +524,10 @@ message StartShard {
   // profile_fp identifies the version of the profile, so the agent can say
   // which one a running shard was started from. The agent never interprets it.
   string profile_fp = 15;
+  // api_key is what the engine demands of every request, derived by the
+  // control plane from the replica id. The router sends it; nothing else
+  // that can reach the node's ports has it.
+  string api_key = 16;
 }
 
 message EngineProfile {
@@ -605,3 +617,35 @@ message FabricScanReply {
   string error = 3;
   int64 took_ms = 4;
 }
+
+// FabricTestRequest asks a node for one side of an RDMA bandwidth test
+// between two nodes: ib_write_bw, from perftest, over one adapter. The
+// server side is asked first and answers when the test is over; the client
+// dials it. What it proves is what a replica spanning nodes needs and nothing
+// on one node can show: that these two adapters reach each other through the
+// fabric, and how fast.
+message FabricTestRequest {
+  string request_id = 1;
+  // role is "server" or "client".
+  string role = 2;
+  string hca = 3;
+  int32 port = 4;
+  // peer is the server's management address, for the client. perftest
+  // exchanges its connection details over TCP before any RDMA.
+  string peer = 5;
+  // tcp_port is that exchange's port, the same on both sides.
+  int32 tcp_port = 6;
+  int32 seconds = 7;
+}
+
+// FabricTestReply is what one side measured.
+message FabricTestReply {
+  string request_id = 1;
+  // gbps is the average bandwidth, in gigabits a second.
+  double gbps = 2;
+  // output is the end of the tool's output, for when the number is not
+  // enough.
+  string output = 3;
+  string error = 4;
+  int64 took_ms = 5;
+}

deploy/bootstrap/onboard-node.sh

@@ -143,7 +143,7 @@ $SSH "set -e
 # that is not a release (a developer's, with commits past the tag) has no such
 # file, and is copied as before.
 TAG="$("$BINARY" --version 2>/dev/null | awk '{print $NF}')"
-if printf '%s' "$TAG" | grep -Eq '^v[0-9]+\.[0-9]+\.[0-9]+$'; then
+if printf '%s' "$TAG" | grep -Eq '^v[0-9]+\.[0-9]+\.[0-9]+(-rc\.[0-9]+)?$'; then
   say "installing llmgw $TAG from the release"
   $SSH "set -e
     arch=\$(dpkg --print-architecture)

deploy/monitoring/README.md

@@ -46,6 +46,12 @@ apt-get install prometheus prometheus-alertmanager
 install -m 0644 prometheus.yml            /etc/prometheus/prometheus.yml
 install -m 0644 llmgw-rules.yml           /etc/prometheus/rules/llmgw.yml
 install -m 0644 alertmanager.yml          /etc/prometheus/alertmanager.yml
+# Alertmanager answers on loopback only. Its API takes silences from anyone who
+# can reach it, with no login, and silencing BackupTooOld or ModelParked is how
+# something that should be noticed is not. Prometheus beside it is its only
+# client; one Alertmanager needs no cluster port.
+echo 'ARGS="--web.listen-address=127.0.0.1:9093 --cluster.listen-address="' \
+    > /etc/default/prometheus-alertmanager
 promtool check config /etc/prometheus/prometheus.yml
 systemctl restart prometheus prometheus-alertmanager
 
@@ -75,7 +81,8 @@ Alertmanager groups and holds alerts and has no receiver.
 Where an alert should go — an address, a Slack workspace, a webhook — is a
 decision with credentials attached, and a monitoring system that quietly starts
 mailing people is worse than one that is visibly not wired up. Alerts are
-visible at `:9093` and in Prometheus; add a receiver to `alertmanager.yml` to
+visible in Prometheus, and at `:9093` on the monitoring host itself (`ssh -L
+9093:127.0.0.1:9093 <host>` to open its page); add a receiver to `alertmanager.yml` to
 make them travel. `critical` already routes separately so it can be pointed
 somewhere else without touching the rest.
 
@@ -86,7 +93,7 @@ An alerting system nobody has seen fire is an assumption.
 ```
 llmgw ctl cordon <node>     # work already there keeps running
 # ~2 minutes later
-curl -s http://<host>:9093/api/v2/alerts | jq '.[].labels'
+ssh <host> curl -s http://127.0.0.1:9093/api/v2/alerts | jq '.[].labels'
 llmgw ctl uncordon <node>   # the alert resolves on its own
 ```
 

deploy/monitoring/llmgw-rules.yml

@@ -42,10 +42,97 @@ groups:
         annotations:
           summary: "{{ $labels.model }} has no replica ready"
           description: >-
-            The model is placed and nothing is serving it. Ten minutes, because
-            a large model on CPU legitimately takes minutes to load. A model
-            that is deliberately stopped publishes no series at all, so this
-            cannot fire for one nobody asked for.
+            The model should be running and nothing is serving it. Ten minutes,
+            because a large model on CPU legitimately takes minutes to load. The
+            series is published at 0 for every model that should be running —
+            before v0.10.4 it was published only while a replica was ready, so
+            this could not fire at all. A model deliberately stopped, or an
+            on-demand one nobody is asking for, publishes nothing.
+
+      - alert: ModelParked
+        expr: llmgw_model_parked == 1
+        labels: { severity: critical }
+        annotations:
+          summary: "{{ $labels.model }} failed every time it started and is no longer retried"
+          description: >-
+            Five replicas in a row failed at start. Before this existed a model
+            whose weights its engine could not load was started 155 times over
+            eight hours. `llmgw ctl status` has the last error; once it is dealt
+            with, `llmgw ctl retry <model>`. Changing the model's definition
+            also retries it.
+
+      - alert: ModelFailingToStart
+        expr: llmgw_model_start_failures >= 2
+        for: 5m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.model }} has failed to start {{ $value }} times in a row"
+          description: >-
+            On its way to being parked. The engine's own words are in
+            `llmgw ctl logs <node> --source <shard>` for the next attempt, or in
+            the control plane's log for the last one.
+
+      - alert: ModelCannotBePlaced
+        expr: llmgw_model_unplaceable == 1
+        for: 10m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.model }} is wanted and fits on no node"
+          description: >-
+            Short of replicas, and the last attempt found nowhere with the
+            memory, cards or engine it needs. `llmgw ctl explain-placement
+            <model>` says what each node lacks. For an on-demand model, requests
+            for it are told this rather than asked to retry.
+
+      - alert: EngineInstallFailed
+        expr: llmgw_node_engine_install{state="failed"} == 1
+        for: 15m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.engine }} {{ $labels.version }} will not install on {{ $labels.node }}"
+          description: >-
+            The node takes no model needing it until it does. The agent retries
+            on its own; `llmgw ctl nodes` and the node's logs on the cluster
+            page say why it failed.
+
+      - alert: ControlPlaneLeadership
+        expr: sum(llmgw_control_plane_leader) != 1
+        for: 2m
+        labels: { severity: critical }
+        annotations:
+          summary: "{{ $value }} control planes believe they are placing work"
+          description: >-
+            None means nothing is placed or replaced; two means both are, each
+            into GPUs the other believes free. The lease is in the store: check
+            it is reachable from every control plane.
+
+      - alert: CertificateExpiresSoon
+        expr: (llmgw_certificate_expiry_timestamp_seconds - time()) < 30 * 86400
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.cert }} ({{ $labels.subject }}) expires in {{ $value | humanizeDuration }}"
+          description: >-
+            Certificates last a year and nothing renews them. When the agent
+            server's runs out every agent refuses to connect at once. Reissue
+            with `llmgw ctl cert` and restart the control plane.
+
+      - alert: CertificateExpiresVerySoon
+        expr: (llmgw_certificate_expiry_timestamp_seconds - time()) < 7 * 86400
+        labels: { severity: critical }
+        annotations:
+          summary: "{{ $labels.cert }} ({{ $labels.subject }}) expires in {{ $value | humanizeDuration }}"
+          description: See CertificateExpiresSoon; a week is not long.
+
+      - alert: BackupTooOld
+        expr: time() - llmgw_backup_last_success_timestamp_seconds > 26 * 3600
+        labels: { severity: warning }
+        annotations:
+          summary: "The newest llmgw backup is {{ $value | humanizeDuration }} old"
+          description: >-
+            llmgw-backup.timer runs nightly. Also fires when there has never
+            been a backup (the series is 0 then). `journalctl -u
+            llmgw-backup.service` on the control plane says why the last one
+            failed; docs/install.md, "Backing up", says how to set it up.
 
       - alert: PoolSaturated
         expr: llmgw_pool_saturation > 1

deploy/systemd/llmgw-backup.service

@@ -0,0 +1,14 @@
+[Unit]
+Description=llmgw backup: the store and /etc/llmgw, to /var/backups/llmgw-server
+After=llmgw-server.service postgresql.service
+
+[Service]
+Type=oneshot
+# Nothing it writes is for anyone else: the archive holds the CA's key and a
+# copy of the store.
+UMask=0077
+# Root, because the archive includes /etc/llmgw/secrets and the certificate
+# authority's key, which the llmgw user deliberately cannot read all of.
+ExecStart=/usr/local/bin/llmgw server backup --config /etc/llmgw/server.yaml --keep 14
+Nice=10
+IOSchedulingClass=idle

deploy/systemd/llmgw-backup.timer

@@ -0,0 +1,12 @@
+[Unit]
+Description=Back up llmgw every night
+
+[Timer]
+OnCalendar=*-*-* 03:15:00
+# A machine that was off at 03:15 backs up when it next starts, rather than
+# waiting another day with a gap nobody notices until the alert.
+Persistent=true
+RandomizedDelaySec=10m
+
+[Install]
+WantedBy=timers.target

docs/fabrics.md

@@ -121,6 +121,39 @@ rate and errors, and the NCCL environment a shard on each fabric would be given.
 `--json` writes every fact it read as well as what it concluded — send that
 file when the report is not enough.
 
+## Checking two nodes: `llmgw ctl fabric-test`
+
+Every check on one node can pass while two nodes still cannot reach each other
+— a cable into the wrong switch, a partition key, a RoCE network with no route
+— and a replica spanning them then hangs on its first all-reduce. Before
+placing one, on the control plane:
+
+```
+llmgw ctl fabric-test gpu01 gpu02
+```
+
+```
+gpu01 → gpu02 on ib-compute, 41s
+
+RAIL              LINK      MEASURED    VERDICT  DETAIL
+mlx5_0 → mlx5_0   400 Gb/s  388.2 Gb/s  ok       97% of 400 Gb/s
+mlx5_1 → mlx5_1   400 Gb/s  241.0 Gb/s  slow     60% of 400 Gb/s: check the cable, and the card's PCIe width and NUMA node
+mlx5_2 → mlx5_2   400 Gb/s  0.0 Gb/s    fail     gpu01: ib_write_bw: Couldn't connect to 10.1.0.22:18517
+```
+
+It runs `ib_write_bw` (perftest, installed on a node the first time) from each
+of the first node's adapters on the fabric to the same adapter on the second —
+mlx5_0 to mlx5_0, the way NCCL pairs them — one rail at a time, through the
+agents, so no SSH between nodes is needed. Each rail is graded against its
+link's rate: **ok** at 80% or more, **slow** at 50%, **fail** below that or when
+nothing got through. `--fabric` picks one when the nodes share several,
+`--seconds` the length of each rail's run. The traffic is real: run it when the
+fabric is quiet.
+
+It measures RDMA between adapters, not GPUDirect or NCCL itself: a pass says
+the fabric carries traffic between these two nodes at its rate, which is what
+a first all-reduce needs and what nothing on one node can show.
+
 ## Watching them
 
 - **Cluster page, Fabrics.** Each declared fabric and what it physically is;
@@ -172,6 +205,6 @@ workers headless. `launch_mode: ray` is there for a vLLM too old for that.
 | It | Look at |
 |---|---|
 | is not placed | `llmgw ctl explain-placement <model>`: a fabric split by identity, a node whose IMEX is down, a node refused because its adapters are cabled to two fabrics |
-| starts and hangs on its first collective | the cluster page's fabric problems; `doctor` on each node — a port down, the wrong interface, RoCE without a v2 GID |
+| starts and hangs on its first collective | `llmgw ctl fabric-test` between two of its nodes; the cluster page's fabric problems; `doctor` on each node — a port down, the wrong interface, RoCE without a v2 GID |
 | runs, slowly | `nvidia_peermem`; the map for a degraded cable; the node page's NVLink matrix for a group that crossed PCIe |
 | worked yesterday | the subnet manager table for a failover; `link_downed` on the node's ports |

docs/install.md

@@ -961,36 +961,74 @@ the choice is yours rather than silent.
 
 ## Backing up
 
-Three things, in descending order of how badly you want them.
+`llmgw server backup` writes one archive holding everything a control plane
+cannot be rebuilt without:
 
-**The certificate authority.** `ca-key.pem`, `/etc/llmgw/pki` unless you kept it elsewhere (step 2).
-Losing it does not stop the cluster, and it means every node identity has to be
-reissued the next time you touch one. Nothing in llmgw can reconstruct it.
+- **the store** — users, API keys, the tenancy tree, limits and budgets, usage
+  history, models added at runtime, measured capacity, and the secret every
+  engine key is derived from (`pg_dump` in its custom format for PostgreSQL, a
+  consistent `VACUUM INTO` snapshot for SQLite; both are safe against a running
+  gateway);
+- **`/etc/llmgw`** — the configuration, the store's address, and the
+  certificate authority's key (`pki/`). Nothing in llmgw can reconstruct that
+  key; losing it means reissuing every node's identity.
 
-**The store.** Users, API keys, organisations, teams, projects, every limit and
-budget, all usage history, models added at runtime, and measured capacity.
+Archives go to `/var/backups/llmgw-server`, readable by root only because of the CA
+key, and the newest 14 are kept. Run it nightly with the timer:
 
 ```
-# SQLite
-sqlite3 /var/lib/llmgw/llmgw.db ".backup '/backup/llmgw-$(date +%F).db'"
-
-# PostgreSQL
-pg_dump --format=custom llmgw > /backup/llmgw-$(date +%F).dump
+sudo install -m 0644 deploy/systemd/llmgw-backup.service deploy/systemd/llmgw-backup.timer /etc/systemd/system/
+sudo systemctl daemon-reload
+sudo systemctl enable --now llmgw-backup.timer
+sudo systemctl start llmgw-backup.service     # one now, to see it work
+ls -l /var/backups/llmgw-server
 ```
 
-`.backup` and `pg_dump` are both safe against a running gateway. Copying the
-SQLite file with `cp` while it is being written is not.
+The control plane publishes when the newest archive was written as
+`llmgw_backup_last_success_timestamp_seconds`, and **BackupTooOld** fires when
+it is more than 26 hours old — including when there has never been one.
+
+**Copy the archives off the machine.** They sit on the disk they back up and do
+not survive losing it. Anything that copies files will do — rsync from the
+mirror, a NAS mount, an offsite bucket — as long as it keeps the archives
+readable by root only.
 
 **Keys are stored hashed and cannot be recovered from a backup or from
 anywhere else.** Restoring an old store restores which keys exist and who owns
 them, not their secrets. A key created after the backup is gone and has to be
-reissued — which is why the store is worth backing up on a schedule rather than
-before upgrades.
+reissued.
+
+### Restoring
+
+On a machine with llmgw installed (the same version or newer) and, for
+PostgreSQL, an empty database and role as `server init --postgres` would create
+them:
+
+```
+sudo systemctl stop llmgw-server
+mkdir /tmp/restore && sudo tar -xzf /var/backups/llmgw-server/llmgw-backup-<host>-<time>.tar.gz -C /tmp/restore
+cat /tmp/restore/MANIFEST                     # version and time it was taken
+
+# /etc/llmgw: configuration, secrets, certificates and the CA
+sudo cp -a /tmp/restore/etc/llmgw/. /etc/llmgw/
+
+# the store — PostgreSQL
+sudo -u postgres pg_restore --clean --if-exists --no-owner --role=llmgw -d llmgw /tmp/restore/store.pgdump
+# the store — SQLite
+sudo install -o llmgw -g llmgw -m 0600 /tmp/restore/store.sqlite /var/lib/llmgw/llmgw.db
+
+sudo systemctl start llmgw-server
+sudo rm -rf /tmp/restore
+```
+
+Running engines keep serving throughout: agents reconnect to the restored
+control plane and their replicas are adopted, and because the engine-key secret
+came back with the store, the router can still reach them.
 
 **The mirror's directory** is the one thing you can rebuild, by downloading it
-all again. Worth remembering that a pinned version may no longer be upstream to
-download: that is exactly what pinning it protected you from, and a backup is
-what makes the protection outlive the disk.
+all again. A pinned version may no longer be upstream to download: that is
+exactly what pinning it protected you from, and a copy of the directory is what
+makes the protection outlive the disk.
 
 ## Reclaiming disk
 

docs/quick-install.md

@@ -35,10 +35,12 @@ install it.
 ## 2. Set up the control plane
 
 ```
-sudo llmgw server init --admin you@example.com
+sudo llmgw server init
 ```
 
-This one command sets up everything on this machine, and takes a few minutes:
+It asks for your email and a few choices, then sets up everything on this
+machine, in a few minutes. In a script, give them as flags instead:
+`sudo llmgw server init --admin you@example.com`.
 
 - the service, its certificates and its configuration;
 - your administrator account. **The password is printed once. Copy it now.**
@@ -54,18 +56,33 @@ sudo llmgw server init --admin you@example.com --host llm.example.com
 
 It is safe to run again. It keeps what exists and finishes what is missing.
 
+**The mirror on its own machine**, when it wants a larger disk than the
+control plane has: first on that machine, with llmgw installed as in step 1,
+
+```
+sudo llmgw mirror init --control-plane <control plane address>
+```
+
+then on the control plane `sudo llmgw server init --mirror http://<mirror
+address>:8082`. The mirror serves model files to every node, and takes
+orders only from the control plane. Untick "This machine as a node" in the
+form (or add `--no-node`) to keep models off the control plane; the starter
+model then starts on the first node you add.
+
 **Check:** open `http://<control-plane>:8080/chat`, sign in with the printed
 password, and ask something. The small model answers. Change the password on
 the account page.
 
 ## 3. Add the GPU nodes
 
-On the control plane, for each node:
+On the control plane:
 
 ```
 sudo llmgw node add ubuntu@10.0.0.21
 ```
 
+Several at once: `sudo llmgw node add ubuntu@10.0.0.21 ubuntu@10.0.0.22 ubuntu@10.0.0.23`.
+
 Replace `ubuntu` with the user you log in to the node as, and `10.0.0.21` with
 its address. That user needs `sudo` without a password on the node.
 
@@ -89,6 +106,13 @@ the same fabric the same name. In a script, give it instead:
 sudo llmgw node add ubuntu@10.0.0.21 --fabric ib-a:mlx5_0,mlx5_1,mlx5_2,mlx5_3
 ```
 
+Once two nodes on a fabric are in, check they reach each other at the link's
+rate before placing anything across them:
+
+```
+llmgw ctl fabric-test gpu01 gpu02
+```
+
 **If it cannot log in.** `sudo` runs SSH as root, with root's keys. To use your
 own keys instead, log in to the control plane with `ssh -A` and keep your
 agent through sudo:

docs/reference.md

@@ -71,6 +71,7 @@ rather than repeated.
 | `lease_ttl` | `time.Duration` | [Control](#control) |
 | `id` | `string` | [Endpoint](#endpoint) |
 | `url` | `string` | [Endpoint](#endpoint) |
+| `api_key` | `string` | [Endpoint](#endpoint) |
 | `models` | `[]string` | [Endpoint](#endpoint) |
 | `engine` | `string` | [Endpoint](#endpoint) |
 | `pool` | `string` | [Endpoint](#endpoint) |
@@ -168,6 +169,7 @@ rather than repeated.
 | `monitoring` | `Monitoring` | [Server](#server) |
 | `mirror` | `Mirror` | [Server](#server) |
 | `engines` | `map[string]string` | [Server](#server) |
+| `backup_dir` | `string` | [Server](#server) |
 | `router` | `Router` | [Server](#server) |
 | `control` | `Control` | [Server](#server) |
 | `models` | `[]catalog.Model` | [Server](#server) |
@@ -566,6 +568,13 @@ reading the alert. Zero takes the default.
 
 `string`
 
+### `api_key`
+
+`string`
+
+APIKey is sent to the engine as a bearer token, for an engine started
+with one.
+
 ### `models`
 
 `[]string`
@@ -1257,6 +1266,14 @@ never trusted to already have what somebody once put there.
 Unset keeps DefaultEngines. A version replaces the default; an empty
 one leaves that engine to be installed by hand.
 
+### `backup_dir`
+
+`string`
+
+BackupDir is where `llmgw server backup` writes, and where the newest
+archive's age is read for llmgw_backup_last_success_timestamp_seconds.
+Empty means /var/backups/llmgw-server.
+
 ### `router`
 
 `Router`

go.mod

@@ -6,7 +6,6 @@ require (
 	connectrpc.com/connect v1.21.0
 	github.com/NVIDIA/go-nvml v0.13.4-0
 	github.com/cespare/xxhash/v2 v2.3.0
-	github.com/charmbracelet/bubbles v1.0.0
 	github.com/charmbracelet/bubbletea v1.3.10
 	github.com/charmbracelet/lipgloss v1.1.0
 	github.com/jackc/pgx/v5 v5.11.0
@@ -28,7 +27,6 @@ require (
 )
 
 require (
-	github.com/atotto/clipboard v0.1.4 // indirect
 	github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
 	github.com/beorn7/perks v1.0.1 // indirect
 	github.com/charmbracelet/colorprofile v0.4.1 // indirect

go.sum

@@ -2,16 +2,12 @@ connectrpc.com/connect v1.21.0 h1:LhqSJt7jHf5NJBo9Jq/t/9FjcYAideif0mg+qe2jCUs=
 connectrpc.com/connect v1.21.0/go.mod h1:A2ygJrukXwWy32vkCAAHNVguZrqZ+jeZ9rGRnGR4dN4=
 github.com/NVIDIA/go-nvml v0.13.4-0 h1:o3jp9u2x1R9ShFE3v+Aesp55XOSIQFMJz/VGNUcJaNE=
 github.com/NVIDIA/go-nvml v0.13.4-0/go.mod h1:id63qwpoDWpFXwnwM6psDCSqW4BmNu6mWpr4YeQtPGo=
-github.com/atotto/clipboard v0.1.4 h1:EH0zSVneZPSuFR11BlR9YppQTVDbh5+16AmcJi4g1z4=
-github.com/atotto/clipboard v0.1.4/go.mod h1:ZY9tmq7sm5xIbd9bOK4onWV4S6X0u6GY7Vn0Yu86PYI=
 github.com/aymanbagabas/go-osc52/v2 v2.0.1 h1:HwpRHbFMcZLEVr42D4p7XBqjyuxQH5SMiErDT4WkJ2k=
 github.com/aymanbagabas/go-osc52/v2 v2.0.1/go.mod h1:uYgXzlJ7ZpABp8OJ+exZzJJhRNQ2ASbcXHWsFqH8hp8=
 github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
 github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
 github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
 github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
-github.com/charmbracelet/bubbles v1.0.0 h1:12J8/ak/uCZEMQ6KU7pcfwceyjLlWsDLAxB5fXonfvc=
-github.com/charmbracelet/bubbles v1.0.0/go.mod h1:9d/Zd5GdnauMI5ivUIVisuEm3ave1XwXtD1ckyV6r3E=
 github.com/charmbracelet/bubbletea v1.3.10 h1:otUDHWMMzQSB0Pkc87rm691KZ3SWa4KUlvF9nRvCICw=
 github.com/charmbracelet/bubbletea v1.3.10/go.mod h1:ORQfo0fk8U+po9VaNvnV95UPWA1BitP1E0N6xJPlHr4=
 github.com/charmbracelet/colorprofile v0.4.1 h1:a1lO03qTrSIRaK8c3JRxJDZOvhvIeSco3ej+ngLk1kk=

internal/account/password.go

@@ -15,6 +15,7 @@ import (
 	"encoding/base64"
 	"errors"
 	"fmt"
+	"runtime"
 	"strings"
 
 	"golang.org/x/crypto/argon2"
@@ -56,7 +57,7 @@ func HashPassword(password string) (string, error) {
 	if _, err := rand.Read(salt); err != nil {
 		return "", err
 	}
-	sum := argon2.IDKey([]byte(password), salt, argonTime, argonMemory, argonThreads, argonKeyLen)
+	sum := idKey([]byte(password), salt, argonTime, argonMemory, argonThreads, argonKeyLen)
 
 	b64 := base64.RawStdEncoding.EncodeToString
 	return fmt.Sprintf("$argon2id$v=%d$m=%d,t=%d,p=%d$%s$%s",
@@ -93,7 +94,7 @@ func VerifyPassword(password, encoded string) (bool, error) {
 		return false, ErrBadHash
 	}
 
-	got := argon2.IDKey([]byte(password), salt, time, memory, threads, uint32(len(want)))
+	got := idKey([]byte(password), salt, time, memory, threads, uint32(len(want)))
 	return subtle.ConstantTimeCompare(got, want) == 1, nil
 }
 
@@ -110,3 +111,16 @@ func init() {
 	}
 	dummyHash = h
 }
+
+// argonSlots bounds how many hashes run at once. Each takes 64 MiB, and the
+// login and signup forms are on the client-facing port: without a bound, a
+// few hundred parallel attempts at random addresses asked for gigabytes at
+// once, and the per-address throttle does not see them — each address is new.
+// Beyond the bound, attempts wait their turn instead.
+var argonSlots = make(chan struct{}, max(2, runtime.NumCPU()/2))
+
+func idKey(password, salt []byte, time, memory uint32, threads uint8, keyLen uint32) []byte {
+	argonSlots <- struct{}{}
+	defer func() { <-argonSlots }()
+	return argon2.IDKey(password, salt, time, memory, threads, keyLen)
+}

internal/account/service.go

@@ -230,6 +230,22 @@ func (s *Service) allowAttempt(email string) bool {
 	defer s.mu.Unlock()
 
 	w, ok := s.attempts[email]
+	if !ok && len(s.attempts) >= maxTrackedAddresses {
+		// Full of addresses nobody has signed in as for a while: forget the
+		// ones whose window has passed. Only expiring, never evicting early —
+		// dropping a live window is how an attacker spraying random addresses
+		// would buy unlimited guesses at the one they care about.
+		for k, v := range s.attempts {
+			if now.Sub(v.since) > s.cfg.AttemptWindow {
+				delete(s.attempts, k)
+			}
+		}
+		if len(s.attempts) >= maxTrackedAddresses {
+			// Still full, so somebody is spraying addresses faster than they
+			// expire. New ones wait; those already tracked are unaffected.
+			return false
+		}
+	}
 	if !ok || now.Sub(w.since) > s.cfg.AttemptWindow {
 		s.attempts[email] = &attemptWindow{count: 1, since: now}
 		return true
@@ -238,6 +254,10 @@ func (s *Service) allowAttempt(email string) bool {
 	return w.count <= s.cfg.MaxAttempts
 }
 
+// maxTrackedAddresses bounds the throttle's memory. Unbounded, it grew by one
+// entry per address tried and shrank only on a successful sign-in.
+const maxTrackedAddresses = 50000
+
 func (s *Service) clearAttempts(email string) {
 	s.mu.Lock()
 	delete(s.attempts, email)

internal/account/throttle_bound_test.go

@@ -0,0 +1,33 @@
+package account
+
+import (
+	"fmt"
+	"testing"
+	"time"
+)
+
+func TestSprayingAddressesCannotGrowTheThrottleOrEscapeIt(t *testing.T) {
+	now := time.Unix(0, 0)
+	s := &Service{cfg: DefaultConfig(), attempts: map[string]*attemptWindow{}, now: func() time.Time { return now }}
+
+	// The target is being guessed at.
+	for i := 0; i < s.cfg.MaxAttempts; i++ {
+		s.allowAttempt("victim@example.com")
+	}
+	// Then the table is flooded with made-up addresses.
+	for i := 0; i < maxTrackedAddresses+100; i++ {
+		s.allowAttempt(fmt.Sprintf("x%d@example.com", i))
+	}
+	if n := len(s.attempts); n > maxTrackedAddresses {
+		t.Errorf("the table grew to %d", n)
+	}
+	if s.allowAttempt("victim@example.com") {
+		t.Error("flooding the table reset the target's throttle")
+	}
+
+	// Once the window passes, everything expires and sign-ins work again.
+	now = now.Add(s.cfg.AttemptWindow + time.Second)
+	if !s.allowAttempt("new@example.com") {
+		t.Error("a new address was refused after the flood expired")
+	}
+}

internal/accounting/accounting.go

@@ -383,3 +383,39 @@ func keyCounter(keyID string) string {
 // they are asking is whether the budgets they are about to display mean
 // anything, and the accountant is what would enforce them.
 func (a *Accountant) AnythingPriced() bool { return a.pricing.AnythingPriced() }
+
+// Headroom is the tightest of a principal's request and token limits: how
+// much is left of each, and when the minute it is counted over rolls on.
+type Headroom struct {
+	RequestLimit, RequestsLeft int
+	TokenLimit, TokensLeft     int
+	Reset                      time.Duration
+}
+
+// Headroom reports what a principal has left, after whatever Check just
+// charged. Of several levels the one with least left is the one that will
+// refuse first, so that is the one reported. A limit of 0 means none is set.
+func (a *Accountant) Headroom(p auth.Principal) Headroom {
+	var h Headroom
+	firstReq, firstTok := true, true
+	for _, l := range levels(p.Limits, p.KeyID) {
+		if l.RPM != nil && *l.RPM > 0 {
+			left := max(0, *l.RPM-int(a.requests.Rate(l.Counter)))
+			if firstReq || left < h.RequestsLeft {
+				h.RequestLimit, h.RequestsLeft, firstReq = *l.RPM, left, false
+			}
+		}
+		if l.TPM != nil && *l.TPM > 0 {
+			left := max(0, *l.TPM-int(a.tokens.Rate(l.Counter)))
+			if firstTok || left < h.TokensLeft {
+				h.TokenLimit, h.TokensLeft, firstTok = *l.TPM, left, false
+			}
+		}
+	}
+	// The windows slide, so room comes back continuously; the next minute
+	// boundary is when a whole window's worth has moved on, and is the figure
+	// a client pacing itself can act on.
+	now := time.Now()
+	h.Reset = now.Truncate(windowLen).Add(windowLen).Sub(now)
+	return h
+}

internal/accounting/accounting_test.go

@@ -630,3 +630,31 @@ func TestAnEmptyPriceTableMeansNoBudgetCanEverBind(t *testing.T) {
 		t.Error("no price table at all is not priced")
 	}
 }
+
+func TestHeadroomIsTheTightestLimitAfterThisRequest(t *testing.T) {
+	// A project allowing 100 a minute under a team allowing 3: the team is
+	// what will refuse, so the team is what a client is told about.
+	a := New(pricing(t), nil, slog.New(slog.DiscardHandler))
+	p := principal("k1", func(e *store.Effective) {
+		e.Chain = []store.Limits{
+			{Scope: store.ScopeProject, ScopeID: "proj", RPM: intp(100), TPM: intp(50000)},
+			{Scope: store.ScopeTeam, ScopeID: "team", RPM: intp(3)},
+		}
+	})
+	if _, err := a.Check(p, "m", 10); err != nil {
+		t.Fatal(err)
+	}
+	h := a.Headroom(p)
+	if h.RequestLimit != 3 || h.RequestsLeft != 2 {
+		t.Errorf("requests %d left of %d; expected 2 of the team's 3", h.RequestsLeft, h.RequestLimit)
+	}
+	if h.TokenLimit != 50000 || h.TokensLeft != 50000-10 {
+		t.Errorf("tokens %d left of %d", h.TokensLeft, h.TokenLimit)
+	}
+	if h.Reset <= 0 || h.Reset > time.Minute {
+		t.Errorf("reset %v", h.Reset)
+	}
+	if none := a.Headroom(principal("k2", nil)); none.RequestLimit != 0 || none.TokenLimit != 0 {
+		t.Errorf("a key with no limits advertised some: %+v", none)
+	}
+}

internal/agent/agent.go

@@ -844,6 +844,7 @@ func (a *Agent) session(ctx context.Context, server string) error {
 	reg.EngineVersions = a.engines(parent)
 	reg.EnginesProbed = true
 	reg.InstallableEngines = provision.Installable()
+	reg.Features = []string{FeatureFabricTest}
 	reg.RunningShards = a.runningShards()
 	a.mu.Lock()
 	h := a.prov.Host
@@ -1145,6 +1146,16 @@ func (a *Agent) handle(ctx context.Context, msg *llmgwv1.ControlMessage, out cha
 			case <-ctx.Done():
 			}
 		}(m.FabricScan)
+	case *llmgwv1.ControlMessage_FabricTest:
+		// Seconds of traffic, and a package install the first time: off the
+		// receive loop too.
+		go func(req *llmgwv1.FabricTestRequest) {
+			reply := a.fabricTest(ctx, req)
+			select {
+			case out <- &llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_FabricTest{FabricTest: reply}}:
+			case <-ctx.Done():
+			}
+		}(m.FabricTest)
 	case *llmgwv1.ControlMessage_Ack:
 		if !m.Ack.GetOk() {
 			a.log.Warn("control plane rejected a message",
@@ -1758,6 +1769,70 @@ func (a *Agent) ensureDiags(ctx context.Context) {
 	}
 }
 
+// FeatureFabricTest is the agent answering FabricTestRequest.
+const FeatureFabricTest = "fabric-test"
+
+// runWriteBW is discovery.RunWriteBW, and haveWriteBW whether it is
+// installed; both replaced in tests.
+var (
+	runWriteBW  = discovery.RunWriteBW
+	haveWriteBW = func() bool { _, err := exec.LookPath("ib_write_bw"); return err == nil }
+)
+
+// fabricTest runs this node's side of a bandwidth test with another.
+func (a *Agent) fabricTest(ctx context.Context, req *llmgwv1.FabricTestRequest) *llmgwv1.FabricTestReply {
+	start := time.Now()
+	reply := &llmgwv1.FabricTestReply{RequestId: req.GetRequestId()}
+	defer func() { reply.TookMs = time.Since(start).Milliseconds() }()
+
+	w := discovery.WriteBW{HCA: req.GetHca(), Port: int(req.GetPort()), GIDIndex: -1,
+		TCPPort: int(req.GetTcpPort()), Seconds: int(req.GetSeconds())}
+	if req.GetRole() == "client" {
+		w.Peer = req.GetPeer()
+	}
+	a.mu.Lock()
+	found := false
+	for _, h := range a.inv.HCAs {
+		for _, p := range h.Ports {
+			if h.Name == w.HCA && p.Num == w.Port {
+				found = true
+				// RoCE needs its v2 GID named, as NCCL does; InfiniBand uses the LID.
+				if p.LinkLayer == discovery.LinkEthernet && p.RoCEv2GID >= 0 {
+					w.GIDIndex = p.RoCEv2GID
+				}
+			}
+		}
+	}
+	a.mu.Unlock()
+	if !found {
+		reply.Error = fmt.Sprintf("this node has no port %s/%d", w.HCA, w.Port)
+		return reply
+	}
+	if !haveWriteBW() && a.prov != nil {
+		a.log.Info("installing perftest, for a bandwidth test with another node")
+		pkgs := map[string][]string{"deb": {"perftest"}, "rpm": {"perftest"}}
+		if err := a.prov.InstallPackages(ctx, pkgs); err != nil {
+			reply.Error = "installing perftest: " + err.Error()
+			return reply
+		}
+	}
+	gbps, out, err := runWriteBW(ctx, w)
+	reply.Gbps = gbps
+	reply.Output = tailLines(out, 12)
+	if err != nil {
+		reply.Error = err.Error()
+	}
+	return reply
+}
+
+func tailLines(s string, n int) string {
+	lines := strings.Split(strings.TrimRight(s, "\n"), "\n")
+	if len(lines) > n {
+		lines = lines[len(lines)-n:]
+	}
+	return strings.Join(lines, "\n")
+}
+
 // discoverGPUFabric and discoverIMEX are replaced in tests.
 var (
 	discoverGPUFabric = discovery.DiscoverGPUFabric

internal/agent/agent_test.go

@@ -23,6 +23,7 @@ import (
 	"golang.org/x/net/http2"
 	"golang.org/x/net/http2/h2c"
 
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/supervisor"
@@ -399,11 +400,16 @@ func TestHeartbeatKeepsNodeAlive(t *testing.T) {
 	n, _ := h.inv.Get("n1")
 	first := n.LastSeen
 
-	time.Sleep(200 * time.Millisecond)
-	n, _ = h.inv.Get("n1")
-	if !n.LastSeen.After(first) {
-		t.Error("heartbeats are not refreshing LastSeen; the node would be declared lost")
+	// Waited for rather than slept through: four heartbeat intervals was not
+	// always enough on a loaded runner for one to have landed.
+	deadline := time.Now().Add(3 * time.Second)
+	for time.Now().Before(deadline) {
+		if n, _ = h.inv.Get("n1"); n.LastSeen.After(first) {
+			return
+		}
+		time.Sleep(20 * time.Millisecond)
 	}
+	t.Error("heartbeats are not refreshing LastSeen; the node would be declared lost")
 }
 
 func TestStartAndStopShardEndToEnd(t *testing.T) {
@@ -881,3 +887,34 @@ func TestAFabricScanIsAskedForAndAnswered(t *testing.T) {
 		t.Errorf("asked %q, got %+v", asked, r)
 	}
 }
+
+func TestABandwidthTestIsRunOnTheAdapterAskedFor(t *testing.T) {
+	oldRun, oldHave := runWriteBW, haveWriteBW
+	t.Cleanup(func() { runWriteBW, haveWriteBW = oldRun, oldHave })
+	haveWriteBW = func() bool { return true }
+	var got discovery.WriteBW
+	runWriteBW = func(_ context.Context, w discovery.WriteBW) (float64, string, error) {
+		got = w
+		return 388.2, " #bytes  #iterations  BW peak[Gb/sec]  BW average[Gb/sec]\n", nil
+	}
+	h := start(t, func(c *Config) { c.Discovery.IBRoot = ibTreeWithActivePorts(t, "mlx5_0") })
+
+	r, err := h.svc.FabricTest(t.Context(), "n1", &llmgwv1.FabricTestRequest{
+		Role: "client", Hca: "mlx5_0", Port: 1, Peer: "10.0.0.2", TcpPort: 18515, Seconds: 3})
+	if err != nil {
+		t.Fatal(err)
+	}
+	if r.GetGbps() != 388.2 || r.GetError() != "" {
+		t.Errorf("reply %+v", r)
+	}
+	if got.HCA != "mlx5_0" || got.Peer != "10.0.0.2" || got.TCPPort != 18515 || got.GIDIndex != -1 {
+		t.Errorf("ran %+v", got)
+	}
+
+	// An adapter the node does not have is said so, without running anything.
+	got = discovery.WriteBW{}
+	r, _ = h.svc.FabricTest(t.Context(), "n1", &llmgwv1.FabricTestRequest{Role: "server", Hca: "mlx5_9", Port: 1, Seconds: 1})
+	if !strings.Contains(r.GetError(), "no port mlx5_9/1") || got.HCA != "" {
+		t.Errorf("reply %+v, ran %+v", r, got)
+	}
+}

internal/agent/discovery/perftest.go

@@ -0,0 +1,133 @@
+package discovery
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"os/exec"
+	"regexp"
+	"strconv"
+	"strings"
+	"time"
+)
+
+// WriteBW is one side of an ib_write_bw run, from perftest: an RDMA write
+// bandwidth test between two adapters, which is what a collective spanning
+// nodes rides on.
+type WriteBW struct {
+	HCA  string
+	Port int
+	// GIDIndex is the RoCE v2 GID to use; negative on InfiniBand, where the
+	// port's LID is used.
+	GIDIndex int
+	// Peer is the server's address, for the client; empty makes the server.
+	Peer    string
+	TCPPort int
+	Seconds int
+}
+
+// Args are the command line: a duration rather than a count, so the test
+// takes as long on a slow link as a fast one, and results in Gb/s.
+func (w WriteBW) Args() []string {
+	args := []string{"-d", w.HCA, "-i", strconv.Itoa(max(w.Port, 1)),
+		"-p", strconv.Itoa(w.TCPPort), "-D", strconv.Itoa(max(w.Seconds, 1)),
+		"-F", "--report_gbits"}
+	if w.GIDIndex >= 0 {
+		args = append(args, "-x", strconv.Itoa(w.GIDIndex))
+	}
+	if w.Peer != "" {
+		args = append(args, w.Peer)
+	}
+	return args
+}
+
+// RunWriteBW runs one side and returns the average bandwidth the output
+// reports, in Gb/s, and the output. The client retries its connection for a
+// while: the server is started first, but a few seconds may pass before it
+// listens.
+func RunWriteBW(ctx context.Context, w WriteBW) (float64, string, error) {
+	bin, err := exec.LookPath("ib_write_bw")
+	if err != nil {
+		return 0, "", errors.New("ib_write_bw is not installed (the perftest package)")
+	}
+	limit := time.Duration(w.Seconds+60) * time.Second
+	var out []byte
+	for attempt := 0; ; attempt++ {
+		cctx, cancel := context.WithTimeout(ctx, limit)
+		out, err = exec.CommandContext(cctx, bin, w.Args()...).CombinedOutput() //nolint:gosec // fixed tool
+		cancel()
+		if err == nil || w.Peer == "" || !refused(string(out)) || attempt >= 10 {
+			break
+		}
+		select {
+		case <-ctx.Done():
+			return 0, string(out), ctx.Err()
+		case <-time.After(time.Second):
+		}
+	}
+	text := string(out)
+	if err != nil {
+		return 0, text, fmt.Errorf("ib_write_bw: %v: %s", err, lastNonEmpty(text))
+	}
+	gbps, perr := ParseWriteBW(text)
+	if perr != nil && w.Peer != "" {
+		// Only the client is sure to print the result; a server that does not
+		// is not a failure.
+		return 0, text, perr
+	}
+	return gbps, text, nil
+}
+
+func refused(out string) bool {
+	for _, s := range []string{"Couldn't connect", "Unable to init the socket connection", "Connection refused"} {
+		if strings.Contains(out, s) {
+			return true
+		}
+	}
+	return false
+}
+
+var columnSplit = regexp.MustCompile(`\s{2,}`)
+
+// ParseWriteBW finds the average bandwidth in perftest's result table:
+//
+//	#bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]
+//	65536      887036           0.00               96.93               0.184877
+//
+// by the heading's position rather than a fixed column, since versions add
+// columns and the peak reads 0.00 when the test runs for a duration.
+func ParseWriteBW(out string) (float64, error) {
+	lines := strings.Split(out, "\n")
+	for i, l := range lines {
+		if !strings.Contains(l, "BW average") {
+			continue
+		}
+		heads := columnSplit.Split(strings.TrimSpace(l), -1)
+		col := -1
+		for j, h := range heads {
+			if strings.HasPrefix(h, "BW average") {
+				col = j
+			}
+		}
+		for _, d := range lines[i+1:] {
+			f := strings.Fields(d)
+			if len(f) == 0 || strings.HasPrefix(f[0], "-") {
+				continue
+			}
+			if len(f) != len(heads) || col < 0 {
+				return 0, fmt.Errorf("perftest's result table has a shape this does not know: %q / %q", l, d)
+			}
+			v, err := strconv.ParseFloat(f[col], 64)
+			if err != nil {
+				return 0, fmt.Errorf("reading the average bandwidth %q: %w", f[col], err)
+			}
+			return v, nil
+		}
+	}
+	return 0, errors.New("perftest printed no result")
+}
+
+func lastNonEmpty(s string) string {
+	lines := strings.Split(strings.TrimSpace(s), "\n")
+	return strings.TrimSpace(lines[len(lines)-1])
+}

internal/agent/discovery/perftest_test.go

@@ -0,0 +1,49 @@
+package discovery
+
+import (
+	"slices"
+	"testing"
+)
+
+// The client's output of perftest 24.x, with a duration.
+const writeBWClient = `---------------------------------------------------------------------------------------
+                    RDMA_Write BW Test
+ Dual-port       : OFF		Device         : mlx5_0
+ Number of qps   : 1		Transport type : IB
+ Connection type : RC		Using SRQ      : OFF
+ TX depth        : 128
+ CQ Moderation   : 100
+ Mtu             : 4096[B]
+ Link type       : IB
+ Max inline data : 0[B]
+ rdma_cm QPs	 : OFF
+ Data ex. method : Ethernet
+---------------------------------------------------------------------------------------
+ local address: LID 0x05 QPN 0x0112 PSN 0x7b1f02 RKey 0x1fffbd VAddr 0x007f1d2a5f0000
+ remote address: LID 0x09 QPN 0x0113 PSN 0x2c1e44 RKey 0x1ffeaf VAddr 0x007fa4e1a60000
+---------------------------------------------------------------------------------------
+ #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]
+ 65536      1851214          0.00               388.21		   0.740422
+---------------------------------------------------------------------------------------
+`
+
+func TestTheAverageBandwidthIsReadFromPerftest(t *testing.T) {
+	got, err := ParseWriteBW(writeBWClient)
+	if err != nil || got != 388.21 {
+		t.Errorf("got %v, %v", got, err)
+	}
+	if _, err := ParseWriteBW(" Couldn't connect to 10.0.0.2:18515\n"); err == nil {
+		t.Error("output with no result table gave a number")
+	}
+}
+
+func TestWriteBWArguments(t *testing.T) {
+	ib := WriteBW{HCA: "mlx5_0", Port: 1, GIDIndex: -1, TCPPort: 18515, Seconds: 3}.Args()
+	if slices.Contains(ib, "-x") {
+		t.Errorf("InfiniBand was given a GID index: %v", ib)
+	}
+	roce := WriteBW{HCA: "mlx5_0", Port: 1, GIDIndex: 3, Peer: "10.0.0.2", TCPPort: 18515, Seconds: 3}.Args()
+	if !slices.Contains(roce, "-x") || roce[len(roce)-1] != "10.0.0.2" {
+		t.Errorf("RoCE client: %v", roce)
+	}
+}

internal/agent/engines/engine.go

@@ -70,6 +70,13 @@ type ShardSpec struct {
 	HTTPPort int
 	Profile  Profile
 
+	// APIKey is what the engine demands of every request. Engines listen on
+	// all of a node's addresses; without a key anybody who can reach the port
+	// talks to the model directly, past the gateway's authentication, quotas
+	// and accounting. Given to the engine in its environment rather than on
+	// its command line, which every user on the node can read.
+	APIKey string `json:"api_key,omitempty"`
+
 	// Pool and Fabric are control plane concepts the agent cannot derive. They
 	// are carried so the agent can report them back after a restart; Fabric
 	// also says, with an "nvl:" prefix, that the replica is inside a
@@ -249,6 +256,11 @@ func baseEnv(s ShardSpec) map[string]string {
 	// Profile overrides win: they are the escape hatch for the per-cluster
 	// tuning no adapter can guess.
 	for k, v := range s.Profile.Env {
+		// Checked again here, not only where the model was defined: this is
+		// the process about to run as root, whatever sent the profile.
+		if bad, _ := UnsafeEnv(k); bad {
+			continue
+		}
 		env[k] = v
 	}
 	return env

internal/agent/engines/llamacpp.go

@@ -120,8 +120,12 @@ func (l *LlamaCpp) Commands(s ShardSpec) ([]Command, error) {
 	}
 	args = append(args, p.Args...)
 
+	env := baseEnv(s)
+	if s.APIKey != "" {
+		env["LLAMA_API_KEY"] = s.APIKey
+	}
 	return []Command{{
-		Name: "llama-server", Path: bin, Args: args, Env: materializeEnv(baseEnv(s)),
+		Name: "llama-server", Path: bin, Args: args, Env: materializeEnv(env),
 	}}, nil
 }
 

internal/agent/engines/llamacpp_test.go

@@ -245,3 +245,19 @@ func TestAMissingPathIsLeftToTheEngine(t *testing.T) {
 		t.Errorf("expected it passed through, got %q %v", got, err)
 	}
 }
+
+func TestLlamaCppIsGivenItsKeyOutOfSightOfTheProcessList(t *testing.T) {
+	// The command line is readable by every user on the node; the
+	// environment of a root process is not.
+	cmds, err := (&LlamaCpp{}).Commands(ShardSpec{HTTPPort: 8080, APIKey: "k1",
+		Profile: Profile{ModelPath: "/m.gguf"}})
+	if err != nil {
+		t.Fatal(err)
+	}
+	if slices.Contains(cmds[0].Args, "k1") || slices.Contains(cmds[0].Args, "--api-key") {
+		t.Errorf("the key is on the command line: %v", cmds[0].Args)
+	}
+	if !slices.Contains(cmds[0].Env, "LLAMA_API_KEY=k1") {
+		t.Error("llama-server was not given its key")
+	}
+}

internal/agent/engines/sglang.go

@@ -96,6 +96,11 @@ func (g *SGLang) serverCommand(s ShardSpec) (Command, error) {
 		)
 	}
 	args = append(args, p.Args...)
+	// SGLang reads its key only from the command line, so on a node with
+	// other users it is visible to them in the process list.
+	if s.APIKey != "" && s.Leader() {
+		args = append(args, "--api-key", s.APIKey)
+	}
 
 	return Command{Name: "sglang", Path: python, Args: args, Env: materializeEnv(baseEnv(s))}, nil
 }

internal/agent/engines/unsafeenv.go

@@ -0,0 +1,30 @@
+package engines
+
+import "strings"
+
+// UnsafeEnv reports whether a profile may not set an environment variable,
+// and why.
+//
+// A profile's env is the escape hatch for tuning no adapter can guess —
+// CUDA_*, NCCL_*, VLLM_*, HF_* — and it reaches an engine the agent starts as
+// root. Some variables are not tuning but a way to run other code in that
+// process: the dynamic loader's LD_PRELOAD, Python's PYTHONSTARTUP, a shell's
+// BASH_ENV. Set from a model definition, which an admin API or a config file
+// supplies, they would turn "add a model" into "run this as root on every
+// node". They are refused where a model is defined and again where an engine
+// starts.
+func UnsafeEnv(key string) (bool, string) {
+	k := strings.ToUpper(key)
+	switch {
+	case strings.HasPrefix(k, "LD_"), strings.HasPrefix(k, "DYLD_"):
+		return true, "it changes what the dynamic loader loads"
+	case k == "PATH":
+		return true, "it changes which binaries run"
+	case strings.HasPrefix(k, "PYTHON") && k != "PYTHONUNBUFFERED" && k != "PYTHONHASHSEED":
+		return true, "it changes what Python imports or runs at start"
+	case k == "BASH_ENV" || k == "ENV" || k == "PERL5OPT" || k == "PERL5LIB" ||
+		k == "NODE_OPTIONS" || k == "RUBYOPT" || k == "GCONV_PATH" || k == "OPENSSL_CONF":
+		return true, "it makes an interpreter or library load code"
+	}
+	return false, ""
+}

internal/agent/engines/vllm.go

@@ -150,6 +150,9 @@ func (v *VLLM) serverCommand(s ShardSpec) (Command, error) {
 	// Profile arguments last, so an operator can override anything above.
 	args = append(args, p.Args...)
 
+	if s.APIKey != "" {
+		env["VLLM_API_KEY"] = s.APIKey
+	}
 	return Command{Name: "vllm", Path: bin, Args: args, Env: materializeEnv(env)}, nil
 }
 

internal/agent/provision/install.go

@@ -16,6 +16,7 @@ import (
 	"os"
 	"os/exec"
 	"path/filepath"
+	"regexp"
 	"strings"
 	"time"
 
@@ -77,6 +78,10 @@ type Installer struct {
 	PyPIBase     string
 }
 
+// validVersion is what a version may look like: "b10991", "0.29.0",
+// "0.11.0+cu128". No separators, and not starting with a dot.
+var validVersion = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._+-]{0,63}$`)
+
 // DefaultRoot is where engines are installed when nothing says otherwise.
 const DefaultRoot = "/opt/llmgw/engines"
 
@@ -114,6 +119,15 @@ func (in *Installer) Converge(ctx context.Context, wants []Want) []Status {
 
 func (in *Installer) one(ctx context.Context, w Want) Status {
 	st := Status{Engine: w.Engine, Version: w.Version}
+	// Checked here, on the node, whatever the control plane checked: the
+	// version names a directory under the install root and is passed to a
+	// package manager, and "../x" would put an install — and the RemoveAll
+	// that replaces one — outside it, as root.
+	if !validVersion.MatchString(w.Version) {
+		st.State = Unsupported
+		st.Detail = fmt.Sprintf("%q is not a version this agent will install", w.Version)
+		return st
+	}
 	recipe, ok := Recipes[w.Engine]
 	if !ok {
 		st.State = Unsupported
@@ -509,6 +523,11 @@ func (in *Installer) download(ctx context.Context, repo, tag, asset, dst, want s
 // untarGz unpacks an archive under dir, refusing anything that would land
 // outside it. A release archive is somebody else's file, and a path of
 // "../../etc/cron.d/x" in it would otherwise be written as root.
+//
+// Every write goes through an os.Root, which refuses to leave dir however the
+// path gets there. Checking names alone was not enough: "a -> ." followed by
+// "a/a/b -> ../.." reads as inside and resolves two levels above it, and a
+// file written through that lands wherever the links point.
 func untarGz(archive, dir string) error {
 	f, err := os.Open(archive) //nolint:gosec // inside the install root
 	if err != nil {
@@ -519,6 +538,11 @@ func untarGz(archive, dir string) error {
 	if err != nil {
 		return err
 	}
+	root, err := os.OpenRoot(dir)
+	if err != nil {
+		return err
+	}
+	defer root.Close()
 	tr := tar.NewReader(gz)
 	for {
 		h, err := tr.Next()
@@ -528,22 +552,22 @@ func untarGz(archive, dir string) error {
 		if err != nil {
 			return err
 		}
-		target, err := within(dir, h.Name)
-		if err != nil {
+		name := filepath.Clean(h.Name)
+		if _, err := within(dir, name); err != nil {
 			return err
 		}
 		switch h.Typeflag {
 		case tar.TypeDir:
-			if err := os.MkdirAll(target, 0o755); err != nil {
+			if err := root.MkdirAll(name, 0o755); err != nil {
 				return err
 			}
 		case tar.TypeReg:
-			if err := os.MkdirAll(filepath.Dir(target), 0o755); err != nil {
+			if err := root.MkdirAll(filepath.Dir(name), 0o755); err != nil {
 				return err
 			}
-			out, err := os.OpenFile(target, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, fs.FileMode(h.Mode)&0o755) //nolint:gosec // checked above
+			out, err := root.OpenFile(name, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, fs.FileMode(h.Mode)&0o755)
 			if err != nil {
-				return err
+				return fmt.Errorf("%s: %w", h.Name, err)
 			}
 			if _, err := io.Copy(out, tr); err != nil { //nolint:gosec // a release archive, size bounded by its checksum
 				out.Close()
@@ -553,19 +577,19 @@ func untarGz(archive, dir string) error {
 				return err
 			}
 		case tar.TypeSymlink:
-			// Libraries are versioned through symlinks, and those are
-			// relative. One pointing anywhere else is refused.
-			if filepath.IsAbs(h.Linkname) {
-				return fmt.Errorf("%s links outside the archive", h.Name)
+			// Libraries are versioned through symlinks to a file beside them —
+			// libllama.so -> libllama.so.0 — and that is all a release needs.
+			// A target with a separator in it could point anywhere once other
+			// links are followed, and a binary found through one would be run
+			// as root, so it is refused rather than reasoned about.
+			if h.Linkname == "" || strings.ContainsAny(h.Linkname, `/\`) || h.Linkname == ".." || h.Linkname == "." {
+				return fmt.Errorf("%s links to %q; only a file beside it is allowed", h.Name, h.Linkname)
 			}
-			if _, err := within(dir, filepath.Join(filepath.Dir(h.Name), h.Linkname)); err != nil {
-				return fmt.Errorf("%s links outside the archive", h.Name)
-			}
-			if err := os.MkdirAll(filepath.Dir(target), 0o755); err != nil {
+			if err := root.MkdirAll(filepath.Dir(name), 0o755); err != nil {
 				return err
 			}
-			_ = os.Remove(target)
-			if err := os.Symlink(h.Linkname, target); err != nil {
+			_ = root.Remove(name)
+			if err := root.Symlink(h.Linkname, name); err != nil {
 				return err
 			}
 		default:

internal/agent/provision/install_test.go

@@ -330,3 +330,61 @@ func TestAPackageManagerBusyWithItsOwnUpdatesIsWaitedFor(t *testing.T) {
 		t.Error("the wait was not reported")
 	}
 }
+
+func TestAVersionThatIsAPathIsRefused(t *testing.T) {
+	// The version names a directory under the install root; "../" would put
+	// an install, and the RemoveAll that replaces one, outside it, as root.
+	in := &Installer{Root: t.TempDir(), Host: ubuntu}
+	for _, v := range []string{"../../etc", "..", ".hidden", "b1/../../x", "", "b1 && rm -rf /"} {
+		got := in.Converge(t.Context(), []Want{{Engine: "llamacpp", Version: v}})
+		if got[0].State != Unsupported {
+			t.Errorf("version %q: %+v", v, got[0])
+		}
+	}
+}
+
+func TestChainedLinksCannotCarryAFileOutOfTheInstall(t *testing.T) {
+	// Each link reads as inside the directory; together they point above it.
+	var buf bytes.Buffer
+	gz := gzip.NewWriter(&buf)
+	tw := tar.NewWriter(gz)
+	_ = tw.WriteHeader(&tar.Header{Name: "a", Linkname: ".", Typeflag: tar.TypeSymlink})
+	_ = tw.WriteHeader(&tar.Header{Name: "a/a/b", Linkname: "../..", Typeflag: tar.TypeSymlink})
+	_ = tw.WriteHeader(&tar.Header{Name: "a/a/b/escaped", Mode: 0o644, Size: 1, Typeflag: tar.TypeReg})
+	_, _ = tw.Write([]byte("x"))
+	_ = tw.Close()
+	_ = gz.Close()
+
+	parent := t.TempDir()
+	dir := filepath.Join(parent, "install")
+	_ = os.MkdirAll(dir, 0o755)
+	archive := filepath.Join(parent, "evil.tar.gz")
+	_ = os.WriteFile(archive, buf.Bytes(), 0o644)
+
+	if err := untarGz(archive, dir); err == nil {
+		t.Error("an archive of chained links was unpacked")
+	}
+	if _, err := os.Stat(filepath.Join(parent, "escaped")); err == nil {
+		t.Fatal("a file was written outside the install directory")
+	}
+}
+
+func TestALibrarysVersionLinkIsStillUnpacked(t *testing.T) {
+	var buf bytes.Buffer
+	gz := gzip.NewWriter(&buf)
+	tw := tar.NewWriter(gz)
+	_ = tw.WriteHeader(&tar.Header{Name: "lib/libllama.so.0", Mode: 0o755, Size: 1, Typeflag: tar.TypeReg})
+	_, _ = tw.Write([]byte("x"))
+	_ = tw.WriteHeader(&tar.Header{Name: "lib/libllama.so", Linkname: "libllama.so.0", Typeflag: tar.TypeSymlink})
+	_ = tw.Close()
+	_ = gz.Close()
+	dir := t.TempDir()
+	archive := filepath.Join(t.TempDir(), "ok.tar.gz")
+	_ = os.WriteFile(archive, buf.Bytes(), 0o644)
+	if err := untarGz(archive, dir); err != nil {
+		t.Fatal(err)
+	}
+	if target, err := os.Readlink(filepath.Join(dir, "lib", "libllama.so")); err != nil || target != "libllama.so.0" {
+		t.Errorf("link %q %v", target, err)
+	}
+}

internal/agent/supervisor/health.go

@@ -53,8 +53,8 @@ func NewHTTPHealth(timeout time.Duration, opts ...HTTPHealthOption) *HTTPHealth
 }
 
 // Ready reports whether the engine at baseURL is serving model.
-func (h *HTTPHealth) Ready(ctx context.Context, baseURL, model string) error {
-	if err := h.get(ctx, baseURL+h.healthPath, nil); err != nil {
+func (h *HTTPHealth) Ready(ctx context.Context, baseURL, model, apiKey string) error {
+	if err := h.get(ctx, baseURL+h.healthPath, apiKey, nil); err != nil {
 		return fmt.Errorf("health: %w", err)
 	}
 	if !h.requireName || model == "" {
@@ -66,7 +66,7 @@ func (h *HTTPHealth) Ready(ctx context.Context, baseURL, model string) error {
 			ID string `json:"id"`
 		} `json:"data"`
 	}
-	if err := h.get(ctx, baseURL+h.modelsPath, &list); err != nil {
+	if err := h.get(ctx, baseURL+h.modelsPath, apiKey, &list); err != nil {
 		return fmt.Errorf("models: %w", err)
 	}
 	for _, d := range list.Data {
@@ -77,11 +77,14 @@ func (h *HTTPHealth) Ready(ctx context.Context, baseURL, model string) error {
 	return fmt.Errorf("engine is up but does not serve %q yet", model)
 }
 
-func (h *HTTPHealth) get(ctx context.Context, url string, out any) error {
+func (h *HTTPHealth) get(ctx context.Context, url, apiKey string, out any) error {
 	req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
 	if err != nil {
 		return err
 	}
+	if apiKey != "" {
+		req.Header.Set("Authorization", "Bearer "+apiKey)
+	}
 	resp, err := h.client.Do(req)
 	if err != nil {
 		return err

internal/agent/supervisor/health_test.go

@@ -39,7 +39,7 @@ func engineStub(t *testing.T, healthy bool, models ...string) *httptest.Server {
 
 func TestReadyWhenServingModel(t *testing.T) {
 	srv := engineStub(t, true, "llama-3.3-70b")
-	if err := NewHTTPHealth(time.Second).Ready(t.Context(), srv.URL, "llama-3.3-70b"); err != nil {
+	if err := NewHTTPHealth(time.Second).Ready(t.Context(), srv.URL, "llama-3.3-70b", ""); err != nil {
 		t.Errorf("a serving engine was reported unready: %v", err)
 	}
 }
@@ -48,7 +48,7 @@ func TestNotReadyWhileLoading(t *testing.T) {
 	// The case that matters: the process is up but the weights are not loaded.
 	// Routing on /health alone would send traffic into a loading model.
 	srv := engineStub(t, false, "llama-3.3-70b")
-	err := NewHTTPHealth(time.Second).Ready(t.Context(), srv.URL, "llama-3.3-70b")
+	err := NewHTTPHealth(time.Second).Ready(t.Context(), srv.URL, "llama-3.3-70b", "")
 	if err == nil {
 		t.Fatal("an unhealthy engine was reported ready")
 	}
@@ -61,7 +61,7 @@ func TestNotReadyWhenModelMissing(t *testing.T) {
 	// /health passes but the model is not listed: the engine is up and serving
 	// something else, or still registering.
 	srv := engineStub(t, true, "some-other-model")
-	err := NewHTTPHealth(time.Second).Ready(t.Context(), srv.URL, "llama-3.3-70b")
+	err := NewHTTPHealth(time.Second).Ready(t.Context(), srv.URL, "llama-3.3-70b", "")
 	if err == nil {
 		t.Fatal("a model that is not served was reported ready")
 	}
@@ -73,14 +73,33 @@ func TestNotReadyWhenModelMissing(t *testing.T) {
 func TestModelCheckCanBeSkipped(t *testing.T) {
 	srv := engineStub(t, true)
 	h := NewHTTPHealth(time.Second, WithoutModelCheck())
-	if err := h.Ready(t.Context(), srv.URL, "anything"); err != nil {
+	if err := h.Ready(t.Context(), srv.URL, "anything", ""); err != nil {
 		t.Errorf("the model check was not skipped: %v", err)
 	}
 }
 
 func TestUnreachableEngine(t *testing.T) {
 	h := NewHTTPHealth(200 * time.Millisecond)
-	if err := h.Ready(t.Context(), "http://127.0.0.1:1", "m"); err == nil {
+	if err := h.Ready(t.Context(), "http://127.0.0.1:1", "m", ""); err == nil {
 		t.Error("an unreachable engine was reported ready")
 	}
 }
+
+func TestTheHealthCheckSendsTheEnginesKey(t *testing.T) {
+	// An engine started with a key refuses /v1/models without it, and a
+	// shard that is serving would read as never ready.
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		if r.Header.Get("Authorization") != "Bearer k1" {
+			w.WriteHeader(http.StatusUnauthorized)
+			return
+		}
+		_, _ = w.Write([]byte(`{"data":[{"id":"m"}]}`))
+	}))
+	defer srv.Close()
+	if err := NewHTTPHealth(time.Second).Ready(t.Context(), srv.URL, "m", "k1"); err != nil {
+		t.Errorf("with the key: %v", err)
+	}
+	if err := NewHTTPHealth(time.Second).Ready(t.Context(), srv.URL, "m", ""); err == nil {
+		t.Error("without the key the engine refused, and that read as ready")
+	}
+}

internal/agent/supervisor/supervisor.go

@@ -79,7 +79,7 @@ func (c *Config) normalize() {
 // supervisor can be tested without an HTTP engine.
 type HealthChecker interface {
 	// Ready reports whether the shard at baseURL serves model.
-	Ready(ctx context.Context, baseURL, model string) error
+	Ready(ctx context.Context, baseURL, model, apiKey string) error
 }
 
 // Shard is one supervised engine process.
@@ -411,7 +411,7 @@ func (s *Supervisor) WaitReady(ctx context.Context, sh *Shard, baseURL, model st
 			last := strings.Join(sh.Logs(10), "\n")
 			return fmt.Errorf("shard %s not ready within %v; last output:\n%s", sh.ID, timeout, last)
 		case <-t.C:
-			if err := s.health.Ready(ctx, baseURL, model); err != nil {
+			if err := s.health.Ready(ctx, baseURL, model, sh.Spec.APIKey); err != nil {
 				continue
 			}
 			sh.setState(Ready, nil)

internal/agent/supervisor/supervisor_test.go

@@ -21,7 +21,7 @@ import (
 // scriptedHealth reports ready once the flag is set.
 type scriptedHealth struct{ ready atomic.Bool }
 
-func (h *scriptedHealth) Ready(context.Context, string, string) error {
+func (h *scriptedHealth) Ready(context.Context, string, string, string) error {
 	if h.ready.Load() {
 		return nil
 	}
@@ -243,7 +243,9 @@ func TestWaitReadyTimeoutIncludesLogs(t *testing.T) {
 	if err != nil {
 		t.Fatal(err)
 	}
-	time.Sleep(100 * time.Millisecond) // let the line be captured
+	eventually(t, 3*time.Second, "the line was never captured", func() bool {
+		return strings.Contains(strings.Join(sh.Logs(10), "\n"), "CUDA out of memory")
+	})
 
 	err = s.WaitReady(t.Context(), sh, "http://x", "m", 200*time.Millisecond)
 	if err == nil {
@@ -295,7 +297,10 @@ func TestLogsAreCaptured(t *testing.T) {
 	if err != nil {
 		t.Fatal(err)
 	}
-	time.Sleep(400 * time.Millisecond)
+	eventually(t, 3*time.Second, "not every line arrived", func() bool {
+		l := strings.Join(sh.Logs(10), "\n")
+		return strings.Contains(l, "line1") && strings.Contains(l, "line2") && strings.Contains(l, "line3")
+	})
 
 	logs := strings.Join(sh.Logs(10), "\n")
 	for _, want := range []string{"line1", "line2", "line3"} {
@@ -372,11 +377,11 @@ func TestStateFileRemovedOnExit(t *testing.T) {
 
 	sh, _ := s.Start(t.Context(), "a", engines.ShardSpec{}, shellCommand("exit 0"))
 	waitState(t, sh, Failed, 3*time.Second)
-	time.Sleep(50 * time.Millisecond)
-
-	if _, err := os.Stat(s.statePath("a")); !os.IsNotExist(err) {
-		t.Error("the state file outlived the process; a later scan would find a dead orphan")
-	}
+	eventually(t, 3*time.Second,
+		"the state file outlived the process; a later scan would find a dead orphan", func() bool {
+			_, err := os.Stat(s.statePath("a"))
+			return os.IsNotExist(err)
+		})
 }
 
 // --- state machine and ring buffer ---
@@ -743,7 +748,9 @@ func TestWaitAliveFailsOnEarlyExit(t *testing.T) {
 	if err := s.WaitAlive(t.Context(), sh); err == nil {
 		t.Fatal("a worker that died immediately was reported ready")
 	}
-	if d := time.Since(start); d > 1500*time.Millisecond {
+	// Below AliveGrace (5s), which is what waiting it out would take; the
+	// headroom is for a loaded runner, where 1.5s was sometimes not enough.
+	if d := time.Since(start); d > 4*time.Second {
 		t.Errorf("waited %v for a process that had already exited", d)
 	}
 }
@@ -827,3 +834,17 @@ func TestAWorkersOnlyProcessWaitsForTheLeader(t *testing.T) {
 		t.Errorf("the worker started after %v, before the leader was listening", d)
 	}
 }
+
+// eventually waits for cond, failing with msg if it does not hold within
+// limit. Sleeping a fixed time and then checking measured the machine the
+// test ran on, and a loaded CI runner under the race detector lost that race.
+func eventually(t *testing.T, limit time.Duration, msg string, cond func() bool) {
+	t.Helper()
+	deadline := time.Now().Add(limit)
+	for !cond() {
+		if time.Now().After(deadline) {
+			t.Fatal(msg)
+		}
+		time.Sleep(10 * time.Millisecond)
+	}
+}

internal/bench/bench.go

@@ -225,6 +225,7 @@ func runStep(ctx context.Context, client *http.Client, cfg Config, concurrency i
 	)
 
 	sampler := newSampler(cfg.MetricsURL, cfg.Engine)
+	sampler.apiKey = cfg.APIKey
 	sampler.start(ctx)
 
 	start := time.Now()

internal/bench/sampler.go

@@ -19,6 +19,7 @@ import (
 // Optional, because a run pointed at the gateway has no single engine to read.
 type sampler struct {
 	url    string
+	apiKey string
 	engine string
 	client *http.Client
 	cancel context.CancelFunc
@@ -65,6 +66,9 @@ func (s *sampler) sample(ctx context.Context) {
 	if err != nil {
 		return
 	}
+	if s.apiKey != "" {
+		req.Header.Set("Authorization", "Bearer "+s.apiKey)
+	}
 	resp, err := s.client.Do(req)
 	if err != nil {
 		return

internal/cli/backup.go

@@ -0,0 +1,312 @@
+package cli
+
+import (
+	"archive/tar"
+	"compress/gzip"
+	"context"
+	"errors"
+	"fmt"
+	"io"
+	"io/fs"
+	"net/url"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"slices"
+	"strings"
+	"time"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/version"
+)
+
+// DefaultBackupDir is where backups are written and where the control plane
+// looks for the newest one.
+//
+// Its own directory rather than /var/backups/llmgw, which a PostgreSQL move
+// had already made for its own dumps, owned by postgres and closed to
+// everyone else — so the control plane could not see a backup landing in it.
+const DefaultBackupDir = "/var/backups/llmgw-server"
+
+func newBackupCmd() *cobra.Command {
+	var (
+		configPath string
+		etcDir     string
+		dir        string
+		keep       int
+	)
+	c := &cobra.Command{
+		Use:   "backup",
+		Short: "Write the store and /etc/llmgw to one archive, and keep the last few",
+		Long: "Everything a control plane cannot be rebuilt without: the store — keys,\n" +
+			"people, quotas, usage, the models added at runtime, the secret engine\n" +
+			"keys derive from — and /etc/llmgw, which holds the configuration, the\n" +
+			"store's address and the certificate authority's key.\n\n" +
+			"The archive holds that key, so it is written readable by root only. It\n" +
+			"is on the same disk as what it backs up: copy it elsewhere, or it does\n" +
+			"not survive the loss of this machine.\n\n" +
+			"Run daily by llmgw-backup.timer (`llmgw server backup`). The control plane publishes the time of\n" +
+			"the newest archive as llmgw_backup_last_success_timestamp_seconds, and\n" +
+			"BackupTooOld fires when it is more than a day old.",
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			cfg, err := config.Load(configPath)
+			if err != nil {
+				return err
+			}
+			path, err := writeBackup(cmd.Context(), cfg.Auth.Store, etcDir, dir)
+			if err != nil {
+				return err
+			}
+			if why := unlistable(dir); why != "" {
+				fmt.Fprintf(cmd.ErrOrStderr(), "warning: %s; the control plane will report no backup "+
+					"and BackupTooOld will fire\n", why)
+			}
+			removed, err := pruneBackups(dir, keep)
+			if err != nil {
+				return err
+			}
+			fmt.Fprintf(cmd.OutOrStdout(), "wrote %s", path)
+			if removed > 0 {
+				fmt.Fprintf(cmd.OutOrStdout(), "; removed %d older", removed)
+			}
+			fmt.Fprintln(cmd.OutOrStdout())
+			return nil
+		},
+	}
+	c.Flags().StringVar(&configPath, "config", "/etc/llmgw/server.yaml", "the control plane's configuration")
+	c.Flags().StringVar(&etcDir, "etc", "/etc/llmgw", "configuration directory to include")
+	c.Flags().StringVar(&dir, "dir", DefaultBackupDir, "where archives are written")
+	c.Flags().IntVar(&keep, "keep", 14, "how many archives to keep")
+	return c
+}
+
+const backupPrefix = "llmgw-backup-"
+
+// writeBackup writes one archive and returns its path. Written beside its
+// final name and renamed, so a backup interrupted halfway is never the newest
+// one the control plane reports.
+func writeBackup(ctx context.Context, storeAddr, etcDir, dir string) (string, error) {
+	// Listable by anyone, so the control plane — not root — can see when the
+	// newest archive was written. The archives themselves are root's alone.
+	if err := os.MkdirAll(dir, 0o755); err != nil {
+		return "", err
+	}
+	host, _ := os.Hostname()
+	name := backupPrefix + host + "-" + time.Now().UTC().Format("20060102T150405Z") + ".tar.gz"
+	final := filepath.Join(dir, name)
+	part := final + ".part"
+
+	f, err := os.OpenFile(part, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0o600)
+	if err != nil {
+		return "", err
+	}
+	ok := false
+	defer func() {
+		if !ok {
+			_ = os.Remove(part)
+		}
+	}()
+	gz := gzip.NewWriter(f)
+	tw := tar.NewWriter(gz)
+
+	manifest := fmt.Sprintf("llmgw %s\nhost %s\ntime %s\n", version.Version, host,
+		time.Now().UTC().Format(time.RFC3339))
+	if err := addBytes(tw, "MANIFEST", []byte(manifest)); err != nil {
+		return "", err
+	}
+	if storeAddr != "" {
+		if err := addStore(ctx, tw, storeAddr, dir); err != nil {
+			return "", fmt.Errorf("the store: %w", err)
+		}
+	}
+	if err := addTree(tw, etcDir, "etc/llmgw"); err != nil {
+		return "", fmt.Errorf("%s: %w", etcDir, err)
+	}
+	if err := errors.Join(tw.Close(), gz.Close(), f.Sync(), f.Close()); err != nil {
+		return "", err
+	}
+	if err := os.Rename(part, final); err != nil {
+		return "", err
+	}
+	ok = true
+	return final, nil
+}
+
+// addStore dumps the store into the archive.
+func addStore(ctx context.Context, tw *tar.Writer, addr, scratch string) error {
+	addr, err := store.ResolveAddress(addr)
+	if err != nil {
+		return err
+	}
+	// Inside a directory only its owner can enter. The archive directory is
+	// listable on purpose, and VACUUM INTO creates its file with the default
+	// mode: made there directly, a full copy of the store — password hashes,
+	// the engine-key secret — sat readable by any local user for as long as
+	// the backup took.
+	tmpDir, err := os.MkdirTemp(scratch, ".store-")
+	if err != nil {
+		return err
+	}
+	defer os.RemoveAll(tmpDir)
+	tmpPath := filepath.Join(tmpDir, "store")
+
+	name := "store.sqlite"
+	if store.IsPostgres(addr) {
+		name = "store.pgdump"
+		if err := pgDump(ctx, addr, tmpPath); err != nil {
+			return err
+		}
+	} else {
+		db, err := store.Open(ctx, addr)
+		if err != nil {
+			return err
+		}
+		err = db.SnapshotTo(ctx, tmpPath)
+		_ = db.Close()
+		if err != nil {
+			return err
+		}
+	}
+	return addFile(tw, tmpPath, name)
+}
+
+// pgDump runs pg_dump in its custom format, which pg_restore can restore
+// selectively. The connection is given through the environment, not the
+// command line: a DSN carries the password, and every user on the machine
+// can read another process's arguments.
+func pgDump(ctx context.Context, dsn, out string) error {
+	u, err := url.Parse(dsn)
+	if err != nil {
+		return fmt.Errorf("the store address is not a URL")
+	}
+	env := os.Environ()
+	set := func(k, v string) {
+		if v != "" {
+			env = append(env, k+"="+v)
+		}
+	}
+	set("PGHOST", u.Hostname())
+	set("PGPORT", u.Port())
+	set("PGUSER", u.User.Username())
+	if pw, ok := u.User.Password(); ok {
+		set("PGPASSWORD", pw)
+	}
+	set("PGDATABASE", strings.TrimPrefix(u.Path, "/"))
+	set("PGSSLMODE", u.Query().Get("sslmode"))
+	cmd := exec.CommandContext(ctx, "pg_dump", "--format=custom", "--no-owner", "--file="+out) //nolint:gosec // fixed binary, our own file
+	cmd.Env = env
+	if b, err := cmd.CombinedOutput(); err != nil {
+		return fmt.Errorf("pg_dump: %v: %s", err, strings.TrimSpace(string(b)))
+	}
+	return nil
+}
+
+func addTree(tw *tar.Writer, root, as string) error {
+	return filepath.WalkDir(root, func(p string, d fs.DirEntry, err error) error {
+		if err != nil {
+			return err
+		}
+		rel, _ := filepath.Rel(root, p)
+		name := filepath.ToSlash(filepath.Join(as, rel))
+		info, err := d.Info()
+		if err != nil {
+			return err
+		}
+		switch {
+		case d.IsDir():
+			return tw.WriteHeader(&tar.Header{Name: name + "/", Mode: int64(info.Mode().Perm()),
+				Typeflag: tar.TypeDir, ModTime: info.ModTime()})
+		case info.Mode().IsRegular():
+			return addFile(tw, p, name)
+		}
+		return nil // sockets, links: not configuration
+	})
+}
+
+func addFile(tw *tar.Writer, path, name string) error {
+	f, err := os.Open(path) //nolint:gosec // files we chose to back up
+	if err != nil {
+		return err
+	}
+	defer f.Close()
+	info, err := f.Stat()
+	if err != nil {
+		return err
+	}
+	if err := tw.WriteHeader(&tar.Header{Name: name, Mode: int64(info.Mode().Perm()),
+		Size: info.Size(), ModTime: info.ModTime(), Typeflag: tar.TypeReg}); err != nil {
+		return err
+	}
+	_, err = io.Copy(tw, f)
+	return err
+}
+
+func addBytes(tw *tar.Writer, name string, b []byte) error {
+	if err := tw.WriteHeader(&tar.Header{Name: name, Mode: 0o600, Size: int64(len(b)),
+		ModTime: time.Now(), Typeflag: tar.TypeReg}); err != nil {
+		return err
+	}
+	_, err := tw.Write(b)
+	return err
+}
+
+// pruneBackups keeps the newest keep archives. Only files this command
+// wrote are considered, so nothing else in the directory is ever removed.
+func pruneBackups(dir string, keep int) (int, error) {
+	entries, err := os.ReadDir(dir)
+	if err != nil {
+		return 0, err
+	}
+	var names []string
+	for _, e := range entries {
+		if n := e.Name(); strings.HasPrefix(n, backupPrefix) && strings.HasSuffix(n, ".tar.gz") {
+			names = append(names, n)
+		}
+	}
+	// The name carries a UTC timestamp, so name order is age order.
+	slices.Sort(names)
+	removed := 0
+	for len(names) > max(keep, 1) {
+		if err := os.Remove(filepath.Join(dir, names[0])); err != nil {
+			return removed, err
+		}
+		names = names[1:]
+		removed++
+	}
+	return removed, nil
+}
+
+// NewestBackup is the modification time of the newest complete archive in
+// dir, or zero when there is none.
+func NewestBackup(dir string) time.Time {
+	entries, err := os.ReadDir(dir)
+	if err != nil {
+		return time.Time{}
+	}
+	var newest time.Time
+	for _, e := range entries {
+		if n := e.Name(); !strings.HasPrefix(n, backupPrefix) || !strings.HasSuffix(n, ".tar.gz") {
+			continue
+		}
+		if info, err := e.Info(); err == nil && info.ModTime().After(newest) {
+			newest = info.ModTime()
+		}
+	}
+	return newest
+}
+
+// unlistable says why the control plane, which does not run as root, could
+// not see the archives in dir, or "" when it can.
+func unlistable(dir string) string {
+	info, err := os.Stat(dir)
+	if err != nil {
+		return err.Error()
+	}
+	if info.Mode().Perm()&0o005 != 0o005 {
+		return fmt.Sprintf("%s is %v, so only its owner can list it", dir, info.Mode().Perm())
+	}
+	return ""
+}

internal/cli/backup_test.go

@@ -0,0 +1,141 @@
+package cli
+
+import (
+	"archive/tar"
+	"compress/gzip"
+	"io"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+func TestABackupHoldsTheStoreAndTheConfiguration(t *testing.T) {
+	tmp := t.TempDir()
+	dbPath := filepath.Join(tmp, "store.db")
+	db, err := store.Open(t.Context(), dbPath)
+	if err != nil {
+		t.Fatal(err)
+	}
+	secret, err := db.ClusterSecret(t.Context(), "engine-key")
+	if err != nil {
+		t.Fatal(err)
+	}
+	_ = db.Close()
+
+	etc := filepath.Join(tmp, "etc")
+	_ = os.MkdirAll(filepath.Join(etc, "pki"), 0o700)
+	_ = os.WriteFile(filepath.Join(etc, "server.yaml"), []byte("listen: :8080\n"), 0o644)
+	_ = os.WriteFile(filepath.Join(etc, "pki", "ca-key.pem"), []byte("KEY"), 0o600)
+
+	out := filepath.Join(tmp, "backups")
+	path, err := writeBackup(t.Context(), dbPath, etc, out)
+	if err != nil {
+		t.Fatal(err)
+	}
+	info, _ := os.Stat(path)
+	if info.Mode().Perm() != 0o600 {
+		t.Errorf("the archive holds the CA's key and is %v", info.Mode().Perm())
+	}
+
+	files := map[string][]byte{}
+	f, _ := os.Open(path)
+	defer f.Close()
+	gz, err := gzip.NewReader(f)
+	if err != nil {
+		t.Fatal(err)
+	}
+	tr := tar.NewReader(gz)
+	for {
+		h, err := tr.Next()
+		if err == io.EOF {
+			break
+		}
+		if err != nil {
+			t.Fatal(err)
+		}
+		b, _ := io.ReadAll(tr)
+		files[h.Name] = b
+	}
+	for _, want := range []string{"MANIFEST", "store.sqlite", "etc/llmgw/server.yaml", "etc/llmgw/pki/ca-key.pem"} {
+		if _, ok := files[want]; !ok {
+			t.Errorf("%s is not in the archive", want)
+		}
+	}
+
+	// And the store in it is a working store, with what was in it.
+	restored := filepath.Join(tmp, "restored.db")
+	_ = os.WriteFile(restored, files["store.sqlite"], 0o600)
+	rdb, err := store.Open(t.Context(), restored)
+	if err != nil {
+		t.Fatalf("the backed-up store does not open: %v", err)
+	}
+	defer rdb.Close()
+	if got, _ := rdb.ClusterSecret(t.Context(), "engine-key"); got != secret {
+		t.Error("the restored store lost the engine key secret; every engine would be locked out")
+	}
+}
+
+func TestOldBackupsArePrunedAndTheNewestIsFound(t *testing.T) {
+	dir := t.TempDir()
+	for i, stamp := range []string{"20260101T000000Z", "20260102T000000Z", "20260103T000000Z"} {
+		p := filepath.Join(dir, backupPrefix+"h-"+stamp+".tar.gz")
+		_ = os.WriteFile(p, []byte("x"), 0o600)
+		mt := time.Date(2026, 1, 1+i, 0, 0, 0, 0, time.UTC)
+		_ = os.Chtimes(p, mt, mt)
+	}
+	_ = os.WriteFile(filepath.Join(dir, "someone-elses-file"), []byte("x"), 0o600)
+	_ = os.WriteFile(filepath.Join(dir, backupPrefix+"h-20260104T000000Z.tar.gz.part"), []byte("x"), 0o600)
+
+	removed, err := pruneBackups(dir, 2)
+	if err != nil || removed != 1 {
+		t.Fatalf("removed %d, %v", removed, err)
+	}
+	if _, err := os.Stat(filepath.Join(dir, "someone-elses-file")); err != nil {
+		t.Error("a file the backup did not write was removed")
+	}
+	newest := NewestBackup(dir)
+	if !newest.Equal(time.Date(2026, 1, 3, 0, 0, 0, 0, time.UTC)) {
+		t.Errorf("newest %v; an interrupted .part must not count", newest)
+	}
+	if !NewestBackup(filepath.Join(dir, "none")).IsZero() {
+		t.Error("a missing directory reported a backup")
+	}
+}
+
+func TestABackupDirectoryTheControlPlaneCannotListIsSaid(t *testing.T) {
+	dir := t.TempDir()
+	_ = os.Chmod(dir, 0o700)
+	if unlistable(dir) == "" {
+		t.Error("a directory only its owner can list was taken as visible")
+	}
+	_ = os.Chmod(dir, 0o755)
+	if why := unlistable(dir); why != "" {
+		t.Errorf("a listable directory: %s", why)
+	}
+}
+
+func TestNoScratchCopyOfTheStoreIsLeftBehind(t *testing.T) {
+	// The snapshot is made before it is archived. Made straight into the
+	// listable archive directory, it was readable by anyone for that time.
+	tmp := t.TempDir()
+	dbPath := filepath.Join(tmp, "store.db")
+	db, err := store.Open(t.Context(), dbPath)
+	if err != nil {
+		t.Fatal(err)
+	}
+	_ = db.Close()
+	out := filepath.Join(tmp, "backups")
+	if _, err := writeBackup(t.Context(), dbPath, t.TempDir(), out); err != nil {
+		t.Fatal(err)
+	}
+	entries, _ := os.ReadDir(out)
+	for _, e := range entries {
+		if strings.HasPrefix(e.Name(), ".store") {
+			t.Errorf("a scratch copy of the store was left behind: %s", e.Name())
+		}
+	}
+}

internal/cli/bench.go

@@ -41,6 +41,11 @@ func newBenchCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 			cfg.MetricsURL = metrics
 			cfg.Engine = engine
 			cfg.APIKey = key
+			// A replica's engine demands its own key. Asked for rather than
+			// typed: the admin API knows which replica answers at the target.
+			if cfg.APIKey == "" {
+				cfg.APIKey = replicaKey(cmd, client, target)
+			}
 			if len(steps) > 0 {
 				cfg.Concurrency = steps
 			}
@@ -227,3 +232,18 @@ func ms(d time.Duration) string {
 		return d.Round(10 * time.Millisecond).String()
 	}
 }
+
+// replicaKey is the engine key of the replica at target, or "" when target is
+// not a replica (the gateway, say) or the admin API cannot be asked.
+func replicaKey(cmd *cobra.Command, client func() *ctlClient, target string) string {
+	var reps []adminapi.ReplicaView
+	if err := client().get(cmd.Context(), "/api/v1/replicas", &reps); err != nil {
+		return ""
+	}
+	for _, rep := range reps {
+		if sameEndpoint(rep.Endpoint, target) {
+			return rep.EngineKey
+		}
+	}
+	return ""
+}

internal/cli/ctl.go

@@ -67,7 +67,9 @@ func (c *ctlClient) do(ctx context.Context, method, path string, body, out any)
 	if err != nil {
 		return err
 	}
-	if body != nil {
+	// On every change, body or not: the admin listener refuses anything else,
+	// because a browser can send a bodiless POST across origins unasked.
+	if method != http.MethodGet {
 		req.Header.Set("Content-Type", "application/json")
 	}
 	// Who made the change, kept with a stored model so a catalog that grew
@@ -81,7 +83,10 @@ func (c *ctlClient) do(ctx context.Context, method, path string, body, out any)
 	}
 	defer resp.Body.Close()
 
-	if resp.StatusCode != http.StatusOK {
+	// Any 2xx. The API answers a change it has queued rather than made — a
+	// restart — with 202, and treating that as a failure printed
+	// "Error: 202 Accepted" over a request that had worked.
+	if resp.StatusCode < 200 || resp.StatusCode > 299 {
 		var e struct {
 			Error string `json:"error"`
 		}
@@ -117,9 +122,11 @@ func newCtlCmd() *cobra.Command {
 	cmd.AddCommand(
 		newNodesCmd(client, &asJSON),
 		newLogsCmd(client, &asJSON),
+		newRequestCmd(client, &asJSON),
 		newModelsCmd(client, &asJSON),
 		newReplicasCmd(client, &asJSON),
 		newExplainCmd(client, &asJSON),
+		newFabricTestCmd(client, &asJSON),
 		newNodeStateCmd(client, "cordon", "cordoned",
 			"Stop placing new work on a node; what runs there keeps running"),
 		newNodeStateCmd(client, "uncordon", "ready",
@@ -132,6 +139,8 @@ func newCtlCmd() *cobra.Command {
 		newStatusCmd(client, &asJSON),
 		newModelCmd(client),
 		newScaleCmd(client),
+		newRestartCmd(client),
+		newRetryCmd(client),
 		newEngineCheckCmd(),
 		newToolCheckCmd(),
 		newWeightsCmd(client, &asJSON),
@@ -241,6 +250,10 @@ func newNodesCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 // actually served, never what is merely catalogued.
 func modelStatus(m adminapi.ModelView) string {
 	switch {
+	case m.Parked:
+		return "parked"
+	case m.Unplaceable:
+		return "cannot place"
 	case m.Min == 0 && m.Total == 0:
 		return "stopped"
 	case m.Ready >= m.Min && m.Min > 0:
@@ -367,6 +380,9 @@ func newReplicasCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 				if r.Stale {
 					state += " (stale profile)"
 				}
+				if r.Restart != "" {
+					state += " (restart " + r.Restart + ")"
+				}
 				fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n",
 					r.ID, r.Model, state, r.Pool, join(nodes),
 					(time.Duration(r.AgeSec) * time.Second).String(),

internal/cli/ctl_fabrictest.go

@@ -0,0 +1,75 @@
+package cli
+
+import (
+	"encoding/json"
+	"fmt"
+	"text/tabwriter"
+	"time"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+)
+
+func newFabricTestCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
+	var (
+		fabric  string
+		seconds int
+	)
+	c := &cobra.Command{
+		Use:   "fabric-test NODE-A NODE-B",
+		Short: "Measure RDMA bandwidth between two nodes, rail by rail",
+		Long: `Runs ib_write_bw from each of NODE-A's adapters on a fabric to the same
+adapter on NODE-B — mlx5_0 to mlx5_0, and so on, the way NCCL pairs them —
+one rail at a time, and grades each against its link's rate: ok at 80% or
+more, slow at 50%, fail below or when nothing got through.
+
+Every check on one node can pass while two nodes still cannot reach each
+other: a cable into the wrong switch, a partition key, a RoCE network with no
+route. A replica spanning them then hangs on its first all-reduce. Run this
+before placing one.
+
+It goes through the agents, so it needs no ssh between nodes. perftest is
+installed on a node the first time it is asked. The traffic is real and
+takes a few seconds a rail; run it when the fabric is quiet.`,
+		Example: `  llmgw ctl fabric-test gpu01 gpu02
+  llmgw ctl fabric-test gpu01 gpu02 --fabric ib-compute --seconds 10`,
+		Args: cobra.ExactArgs(2),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			body, _ := json.Marshal(map[string]any{"a": args[0], "b": args[1], "fabric": fabric, "seconds": seconds})
+			cl := client()
+			// Rails one after another, each for its seconds and a settle; the
+			// client's usual timeout is for questions, not measurements.
+			cl.http.Timeout = 30 * time.Minute
+			var v adminapi.FabricTestView
+			if err := cl.post(cmd.Context(), "/api/v1/fabric-test", body, &v); err != nil {
+				return err
+			}
+			if *asJSON {
+				return json.NewEncoder(cmd.OutOrStdout()).Encode(v)
+			}
+			out := cmd.OutOrStdout()
+			fmt.Fprintf(out, "%s → %s on %s, %s\n\n", v.A, v.B, v.Fabric, v.Took)
+			w := tabwriter.NewWriter(out, 0, 0, 2, ' ', 0)
+			fmt.Fprintln(w, "RAIL\tLINK\tMEASURED\tVERDICT\tDETAIL")
+			for _, r := range v.Rails {
+				link := "-"
+				if r.RateGbps > 0 {
+					link = fmt.Sprintf("%d Gb/s", r.RateGbps)
+				}
+				fmt.Fprintf(w, "%s → %s\t%s\t%.1f Gb/s\t%s\t%s\n", r.A, r.B, link, r.Gbps, r.Verdict, r.Detail)
+			}
+			_ = w.Flush()
+			for _, n := range v.Notes {
+				fmt.Fprintf(out, "\nnote: %s\n", n)
+			}
+			if !v.OK {
+				return fmt.Errorf("not every rail passed")
+			}
+			return nil
+		},
+	}
+	c.Flags().StringVar(&fabric, "fabric", "", "the fabric to test (default: the one the two nodes share)")
+	c.Flags().IntVar(&seconds, "seconds", 3, "how long each rail runs, up to 30")
+	return c
+}

internal/cli/ctl_request.go

@@ -0,0 +1,45 @@
+package cli
+
+import (
+	"encoding/json"
+	"fmt"
+	"net/url"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+)
+
+func newRequestCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
+	return &cobra.Command{
+		Use:   "request ID",
+		Short: "What one request went through: replica, queue, first token, tokens",
+		Long: "Every response carries its id in X-Request-Id, and a client may send its\n" +
+			"own. Given one, this says which replica answered, how long admission held\n" +
+			"it, how long the first token took and what it cost — the answer to \"why\n" +
+			"was this slow\". The access log has the same fields under \"request\".",
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			var v adminapi.RequestView
+			if err := client().get(cmd.Context(), "/api/v1/requests/"+url.PathEscape(args[0]), &v); err != nil {
+				return err
+			}
+			if *asJSON {
+				return json.NewEncoder(cmd.OutOrStdout()).Encode(v)
+			}
+			model := v.Model
+			if v.ServedModel != "" && v.ServedModel != v.Model {
+				model += " (served by " + v.ServedModel + ")"
+			}
+			out := cmd.OutOrStdout()
+			fmt.Fprintf(out, "request   %s at %s\n", v.RequestID, v.At.UTC().Format("2006-01-02 15:04:05Z"))
+			fmt.Fprintf(out, "model     %s\n", model)
+			fmt.Fprintf(out, "replica   %s\n", or(v.Replica, "- (none answered)"))
+			fmt.Fprintf(out, "status    %d\n", v.Status)
+			fmt.Fprintf(out, "time      %d ms total: %d queued, %d to the first token\n", v.LatencyMs, v.QueueMs, v.TTFTMs)
+			fmt.Fprintf(out, "tokens    %d in, %d out\n", v.PromptTokens, v.CompletionTokens)
+			fmt.Fprintf(out, "who       key %s, project %s%s\n", v.KeyID, v.ProjectID, prefixed(", user ", v.UserID))
+			return nil
+		},
+	}
+}

internal/cli/ctl_restart.go

@@ -0,0 +1,75 @@
+package cli
+
+import (
+	"encoding/json"
+	"errors"
+	"fmt"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+)
+
+func newRestartCmd(client func() *ctlClient) *cobra.Command {
+	var (
+		models []string
+		all    bool
+	)
+	c := &cobra.Command{
+		Use:   "restart [REPLICA...]",
+		Short: "Replace replicas with new ones, starting each replacement before stopping the old",
+		Long: "How a running engine picks up what only applies at start: an engine key,\n" +
+			"an edited profile, a new engine version.\n\n" +
+			"Each replacement is placed beside the replica it replaces, and the old one\n" +
+			"keeps serving until the new one is ready. Replicas of one model are\n" +
+			"replaced one at a time. Where nothing else has room for a second copy —\n" +
+			"the usual case on a small CPU fleet — the old one is stopped after the\n" +
+			"drain grace period (5 minutes by default) and the new one starts in its\n" +
+			"place; a model with only that replica is unserved while it loads.\n\n" +
+			"  llmgw ctl restart qwen2.5-7b-3f2a91c0\n" +
+			"  llmgw ctl restart --model qwen2.5-7b\n" +
+			"  llmgw ctl restart --all",
+		RunE: func(cmd *cobra.Command, args []string) error {
+			if len(args) == 0 && len(models) == 0 && !all {
+				return errors.New("name replicas, --model, or --all")
+			}
+			body, _ := json.Marshal(adminapi.RestartRequest{Replicas: args, Models: models, All: all})
+			var res adminapi.RestartResult
+			if err := client().post(cmd.Context(), "/api/v1/restart", body, &res); err != nil {
+				return err
+			}
+			fmt.Printf("%d replica(s) queued for restart; watch with: llmgw ctl replicas\n", len(res.Queued))
+			return nil
+		},
+	}
+	c.Flags().StringArrayVar(&models, "model", nil, "every replica of this model; repeatable")
+	c.Flags().BoolVar(&all, "all", false, "every ready replica in the cluster")
+	return c
+}
+
+func newRetryCmd(client func() *ctlClient) *cobra.Command {
+	return &cobra.Command{
+		Use:   "retry MODEL",
+		Short: "Try a parked model again, and clear its backoff",
+		Long: "A model whose replicas fail every time they start is parked after five\n" +
+			"failures in a row: it is not placed again until its definition changes\n" +
+			"or this is run. Run it once the cause — the weights, the engine, the\n" +
+			"profile — has been dealt with; `llmgw ctl status` lists parked models with\n" +
+			"their last error.",
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			var res struct {
+				WasParked bool `json:"was_parked"`
+			}
+			if err := client().post(cmd.Context(), "/api/v1/models/"+args[0]+"/retry", []byte(`{}`), &res); err != nil {
+				return err
+			}
+			if res.WasParked {
+				fmt.Printf("%s unparked; it is placed on the next pass\n", args[0])
+			} else {
+				fmt.Printf("%s was not parked; its backoff is cleared\n", args[0])
+			}
+			return nil
+		},
+	}
+}

internal/cli/ctl_status_test.go

@@ -0,0 +1,24 @@
+package cli
+
+import (
+	"net/http"
+	"net/http/httptest"
+	"testing"
+)
+
+func TestAQueuedChangeIsNotReportedAsAnError(t *testing.T) {
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+		w.WriteHeader(http.StatusAccepted)
+		_, _ = w.Write([]byte(`{"queued":["r1"]}`))
+	}))
+	defer srv.Close()
+	var out struct {
+		Queued []string `json:"queued"`
+	}
+	if err := newCtlClient(srv.URL).post(t.Context(), "/api/v1/restart", []byte(`{}`), &out); err != nil {
+		t.Fatalf("202 read as a failure: %v", err)
+	}
+	if len(out.Queued) != 1 {
+		t.Errorf("the body of a 202 was not read: %+v", out)
+	}
+}

internal/cli/dispatcherref_test.go

@@ -5,6 +5,7 @@ import (
 
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/agentsvc"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/fabrictest"
 )
 
 func TestTheForwardingDispatcherCarriesEveryCapability(t *testing.T) {
@@ -23,10 +24,17 @@ func TestTheForwardingDispatcherCarriesEveryCapability(t *testing.T) {
 		t.Error("dispatcherRef drops WeightRemover, so the control plane cannot reclaim disk")
 	}
 
+	if _, ok := ref.(fabrictest.Asker); !ok {
+		t.Error("dispatcherRef drops the bandwidth test, so ctl fabric-test cannot reach the agents")
+	}
+
 	// And the thing it forwards to must have it, or the forwarding has nothing
 	// to reach.
 	var svc any = &agentsvc.Service{}
 	if _, ok := svc.(control.WeightRemover); !ok {
 		t.Error("the agent service does not implement WeightRemover")
 	}
+	if _, ok := svc.(fabrictest.Asker); !ok {
+		t.Error("the agent service does not implement the bandwidth test")
+	}
 }

internal/cli/mirror.go

@@ -4,7 +4,9 @@ import (
 	"encoding/json"
 	"fmt"
 	"log/slog"
+	"net"
 	"net/http"
+	"net/netip"
 	"os"
 	"strings"
 	"time"
@@ -44,6 +46,7 @@ func newMirrorCmd() *cobra.Command {
 		newMirrorUnusedCmd(&dir),
 		newMirrorRemoveCmd(&dir),
 		newMirrorServeCmd(&dir),
+		newMirrorInitCmd(&dir),
 	)
 	return c
 }
@@ -229,6 +232,8 @@ func newMirrorRemoveCmd(dir *string) *cobra.Command {
 func newMirrorServeCmd(dir *string) *cobra.Command {
 	var listen, apiListen, base string
 	var tlsc tlsconf.Config
+	var allow []string
+	var allowPrivate bool
 
 	c := &cobra.Command{
 		Use:   "serve",
@@ -258,6 +263,9 @@ func newMirrorServeCmd(dir *string) *cobra.Command {
 			out := cmd.OutOrStdout()
 			log := slog.Default()
 			mgr := mirror.NewManager(*dir, log)
+			if allowPrivate {
+				mgr.AllowPrivateSources()
+			}
 			api := mirror.NewAPI(mgr, strings.TrimSuffix(base, "/"))
 
 			// http.Dir refuses paths that escape the root, which is the whole
@@ -268,9 +276,13 @@ func newMirrorServeCmd(dir *string) *cobra.Command {
 				Handler:           readOnly(withManifest(*dir, http.FileServer(http.Dir(*dir)))),
 				ReadHeaderTimeout: 10 * time.Second,
 			}
+			allowed, err := parseAllow(allow)
+			if err != nil {
+				return err
+			}
 			admin := &http.Server{
 				Addr:              apiListen,
-				Handler:           api.Handler(),
+				Handler:           apiGate(tlsc.Enabled(), allowed, api.Handler()),
 				ReadHeaderTimeout: 10 * time.Second,
 			}
 
@@ -279,14 +291,17 @@ func newMirrorServeCmd(dir *string) *cobra.Command {
 			fmt.Fprintf(out, "serving %s on %s\n", *dir, listen)
 
 			go func() { errs <- fmt.Errorf("serving the API: %w", serveAPI(admin, tlsc)) }()
-			if tlsc.Enabled() {
+			switch {
+			case tlsc.Enabled():
 				fmt.Fprintf(out, "management API on %s, mutual TLS against %s\n", apiListen, tlsc.CA)
-			} else {
-				// Said plainly rather than left to be discovered. An API that
-				// can delete models and start downloads, open to anyone who can
-				// reach the port, is a choice the operator should have made
-				// rather than inherited from a default.
-				fmt.Fprintf(out, "management API on %s, UNAUTHENTICATED — keep it on a trusted network\n", apiListen)
+			case len(allowed) > 0:
+				fmt.Fprintf(out, "management API on %s, for %s and this machine only\n", apiListen, strings.Join(allow, ", "))
+			default:
+				// This machine only, which was once "anyone": an API that can
+				// delete models and start downloads, open to every node that
+				// could reach the port, is a choice the operator should make
+				// with --api-allow rather than inherit.
+				fmt.Fprintf(out, "management API on %s, for this machine only (--api-allow to add the control plane)\n", apiListen)
 			}
 			fmt.Fprintf(out, "catalog sources should point at %s\n", base)
 
@@ -300,6 +315,10 @@ func newMirrorServeCmd(dir *string) *cobra.Command {
 	c.Flags().StringVar(&apiListen, "api-listen", ":8082", "address to serve the management API on")
 	c.Flags().StringVar(&base, "base", "http://localhost:8081",
 		"the address nodes use to reach this mirror, as it should appear in catalog sources")
+	c.Flags().StringArrayVar(&allow, "api-allow", nil,
+		"an address or network the management API answers, besides this machine; repeatable (default: this machine only)")
+	c.Flags().BoolVar(&allowPrivate, "allow-private-sources", false,
+		"let downloads reach loopback, private and link-local addresses")
 	c.Flags().StringVar(&tlsc.CA, "tls-ca", "", "CA bundle that management clients must present a certificate from")
 	c.Flags().StringVar(&tlsc.Cert, "tls-cert", "", "this mirror's certificate")
 	c.Flags().StringVar(&tlsc.Key, "tls-key", "", "this mirror's private key")
@@ -398,3 +417,61 @@ func versionPath(p string) (model, version string, ok bool) {
 	}
 	return parts[0], parts[1], true
 }
+
+// parseAllow reads addresses and networks: 10.0.0.140, 10.0.0.0/24.
+func parseAllow(list []string) ([]netip.Prefix, error) {
+	var out []netip.Prefix
+	for _, s := range list {
+		for _, one := range strings.Split(s, ",") {
+			one = strings.TrimSpace(one)
+			if one == "" {
+				continue
+			}
+			if p, err := netip.ParsePrefix(one); err == nil {
+				out = append(out, p.Masked())
+				continue
+			}
+			a, err := netip.ParseAddr(one)
+			if err != nil {
+				return nil, fmt.Errorf("--api-allow %q is not an address or a network", one)
+			}
+			out = append(out, netip.PrefixFrom(a.Unmap(), a.Unmap().BitLen()))
+		}
+	}
+	return out, nil
+}
+
+// apiGate decides who reaches the management API. With mutual TLS the
+// certificate decides, narrowed by --api-allow when given. Without it, only
+// this machine and the addresses allowed: nothing allowed used to mean anyone,
+// and every node that could reach the port could delete models.
+func apiGate(mtls bool, allowed []netip.Prefix, next http.Handler) http.Handler {
+	if mtls && len(allowed) == 0 {
+		return next
+	}
+	return onlyFrom(allowed, next)
+}
+
+// onlyFrom answers only this machine and the addresses allowed: the management
+// API can delete models and start downloads, and a mirror on its own machine
+// is reached over the network by one control plane and no one else.
+func onlyFrom(allowed []netip.Prefix, next http.Handler) http.Handler {
+	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		host, _, err := net.SplitHostPort(r.RemoteAddr)
+		a, perr := netip.ParseAddr(host)
+		if err != nil || perr != nil {
+			http.Error(w, "forbidden", http.StatusForbidden)
+			return
+		}
+		a = a.Unmap()
+		ok := a.IsLoopback()
+		for _, p := range allowed {
+			ok = ok || p.Contains(a)
+		}
+		if !ok {
+			http.Error(w, "this mirror is managed by its control plane only", http.StatusForbidden)
+			return
+		}
+		next.ServeHTTP(w, r)
+	})
+}

internal/cli/mirror_gate_test.go

@@ -0,0 +1,39 @@
+package cli
+
+import (
+	"net/http"
+	"net/http/httptest"
+	"net/netip"
+	"testing"
+)
+
+func TestTheMirrorAPIAnswersOnlyThisMachineUnlessTold(t *testing.T) {
+	ok := http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {})
+	call := func(h http.Handler, from string) int {
+		r := httptest.NewRequest("DELETE", "/v1/models/m/v", nil)
+		r.RemoteAddr = from + ":5000"
+		w := httptest.NewRecorder()
+		h.ServeHTTP(w, r)
+		return w.Code
+	}
+	cp := []netip.Prefix{netip.MustParsePrefix("10.0.0.140/32")}
+
+	// Nothing allowed and no TLS used to mean anyone: every node could delete
+	// models.
+	if got := call(apiGate(false, nil, ok), "10.0.0.141"); got != http.StatusForbidden {
+		t.Errorf("a node reached an unconfigured mirror API: %d", got)
+	}
+	if got := call(apiGate(false, nil, ok), "127.0.0.1"); got != http.StatusOK {
+		t.Errorf("the mirror's own machine was refused: %d", got)
+	}
+	if got := call(apiGate(false, cp, ok), "10.0.0.140"); got != http.StatusOK {
+		t.Errorf("the allowed control plane was refused: %d", got)
+	}
+	if got := call(apiGate(false, cp, ok), "10.0.0.141"); got != http.StatusForbidden {
+		t.Errorf("a node outside the allowlist got through: %d", got)
+	}
+	// With mutual TLS the certificate decides.
+	if got := call(apiGate(true, nil, ok), "10.0.0.141"); got != http.StatusOK {
+		t.Errorf("mTLS without an allowlist refused a client: %d", got)
+	}
+}

internal/cli/mirrorinit.go

@@ -0,0 +1,125 @@
+package cli
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"io"
+	"net"
+	"os"
+	"runtime"
+	"strings"
+	"time"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
+)
+
+// mirror init sets a machine up as the cluster's model mirror, on its own:
+// the bytes for every node on :8081, the management API on :8082 for the
+// control plane alone. server init does the same beside the control plane;
+// this is for when the mirror wants its own disk.
+
+func newMirrorInitCmd(dir *string) *cobra.Command {
+	var (
+		controlPlanes []string
+		base          string
+	)
+	cmd := &cobra.Command{
+		Use:   "init",
+		Short: "Set this machine up as the cluster's model mirror and start it",
+		Long: `Creates the llmgw user and the mirror's directory, and a systemd unit that
+serves the model files to every node on :8081 and the management API on
+:8082, and starts it.
+
+The management API can fetch and delete models, so it answers this machine
+and the control planes given with --control-plane, and nobody else. Then on
+the control plane:
+
+  sudo llmgw server init --mirror http://<this machine>:8082
+
+Safe to run again: it changes only what differs.`,
+		Example: `  sudo llmgw mirror init --control-plane 10.0.0.10`,
+		Args:    cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			if runtime.GOOS != "linux" {
+				return fmt.Errorf("mirror init sets up a Linux machine; this is %s", runtime.GOOS)
+			}
+			if os.Geteuid() != 0 {
+				return errors.New("mirror init creates a user, writes /etc and starts a service: run it as root")
+			}
+			if len(controlPlanes) == 0 {
+				return errors.New("--control-plane: which control plane manages this mirror (its address)")
+			}
+			if _, err := parseAllow(controlPlanes); err != nil {
+				return err
+			}
+			return mirrorInit(cmd.Context(), cmd.OutOrStdout(), *dir, base, controlPlanes)
+		},
+	}
+	cmd.Flags().StringArrayVar(&controlPlanes, "control-plane", nil,
+		"address of the control plane that manages this mirror; repeatable")
+	cmd.Flags().StringVar(&base, "base", "",
+		"the address nodes fetch from, http://host:8081 (default: this machine's first address)")
+	return cmd
+}
+
+func mirrorInit(ctx context.Context, out io.Writer, dir, base string, controlPlanes []string) error {
+	in := &serverInit{user: "llmgw", stateDir: "/var/lib/llmgw", out: out}
+	if err := in.ensureUser(ctx); err != nil {
+		return err
+	}
+	if err := os.MkdirAll(dir, 0o755); err != nil {
+		return err
+	}
+	if err := in.chown(dir, true); err != nil {
+		return err
+	}
+	addr := ""
+	if ips := localAddrs(); len(ips) > 0 {
+		addr = ips[0].String()
+	}
+	if base == "" {
+		if addr == "" {
+			return errors.New("this machine has no address to give nodes; give one with --base")
+		}
+		base = "http://" + net.JoinHostPort(addr, "8081")
+	}
+	unit := mirrorUnitFor(dir, ":8082", base, controlPlanes,
+		"'llmgw mirror init'. Bytes on :8081 for every node; the\n# management API on :8082, for the control plane only.")
+	changed, err := in.writeUnit(mirrorUnitPath, unit)
+	if err != nil {
+		return err
+	}
+	for _, args := range [][]string{{"daemon-reload"}, {"enable", "--now", "llmgw-mirror"}} {
+		if err := in.systemctl(ctx, args...); err != nil {
+			return err
+		}
+	}
+	if changed {
+		if err := in.systemctl(ctx, "restart", "llmgw-mirror"); err != nil {
+			return err
+		}
+		in.done("model mirror in %s: files on :8081 at %s, management on :8082 for %s", dir, base, strings.Join(controlPlanes, ", "))
+	} else {
+		in.kept("model mirror in %s", dir)
+	}
+	m, err := mirror.NewClient("http://127.0.0.1:8082", tlsconf.Config{}, 5*time.Second)
+	if err != nil {
+		return err
+	}
+	if err := waitFor(ctx, 30*time.Second, "the mirror to answer", func() (bool, error) {
+		return m.Health(ctx) == nil, nil
+	}); err != nil {
+		return err
+	}
+	in.kept("answering")
+	api := "http://<this machine>:8082"
+	if addr != "" {
+		api = "http://" + net.JoinHostPort(addr, "8082")
+	}
+	fmt.Fprintf(out, "\nThe mirror is up. On the control plane:\n\n  sudo llmgw server init --mirror %s\n\n", api)
+	return nil
+}

internal/cli/node.go

@@ -6,6 +6,7 @@ import (
 	"fmt"
 	"io"
 	"os"
+	"slices"
 	"strings"
 	"time"
 
@@ -64,12 +65,15 @@ type nodeAdd struct {
 	noWait    bool
 	plain     bool
 	serverCfg string
+	// named are the fabric names given in this run, by what the adapters
+	// report the fabric to be.
+	named map[string]string
 }
 
 func newNodeAddCmd() *cobra.Command {
 	var na nodeAdd
 	cmd := &cobra.Command{
-		Use:   "add [user@]address",
+		Use:   "add [user@]address...",
 		Short: "Make a machine a node: check it, give it an identity, start its agent",
 		Long: `Run on the machine holding the certificate authority — the control plane,
 after 'llmgw server init' — as root. It logs in to the machine with ssh,
@@ -87,14 +91,18 @@ Nothing else needs giving, except --fabric for InfiniBand or RoCE: which
 fabric an adapter is on has a name only you can choose, and it must be the
 same on every node attached to it. Doctor suggests one when it sees adapters.
 
+Several addresses add several nodes, one after another; a fabric named for
+one is recognised on the next and not asked again.
+
 The user needs sudo on the machine without a password. Run it again at any
 time: the node gets a fresh certificate and configuration, and its running
 engines are kept.`,
 		Example: `  sudo llmgw node add 10.0.0.21
+  sudo llmgw node add ubuntu@10.0.0.21 ubuntu@10.0.0.22 ubuntu@10.0.0.23
   sudo llmgw node add ubuntu@gpu01 --fabric ib-a:mlx5_0,mlx5_1,mlx5_2,mlx5_3`,
-		Args: cobra.ExactArgs(1),
+		Args: cobra.MinimumNArgs(1),
 		RunE: func(cmd *cobra.Command, args []string) error {
-			return na.run(cmd.Context(), args[0], cmd.OutOrStdout())
+			return na.run(cmd.Context(), args, cmd.OutOrStdout())
 		},
 	}
 	f := cmd.Flags()
@@ -120,18 +128,9 @@ engines are kept.`,
 	return cmd
 }
 
-func (na *nodeAdd) prepare(addr string) error {
+// prepare works out what every node added in this run shares.
+func (na *nodeAdd) prepare() error {
 	o := &na.opts
-	o.Addr = addr
-	if user, host, ok := strings.Cut(addr, "@"); ok {
-		o.User, o.Addr = user, host
-	}
-	if o.User == "" {
-		// Under sudo, the person who ran it — their account is usually the
-		// one that exists on the nodes too. root's own keys still do the
-		// logging in, since that is who is running ssh.
-		o.User = os.Getenv("SUDO_USER")
-	}
 	for _, s := range na.fabrics {
 		fab, err := onboard.ParseFabric(s)
 		if err != nil {
@@ -158,32 +157,124 @@ func (na *nodeAdd) prepare(addr string) error {
 		o.Binary = exe
 	}
 	onboard.AgentUnit = agentUnit
+	na.named = map[string]string{}
 	return nil
 }
 
-func (na *nodeAdd) run(ctx context.Context, addr string, out io.Writer) error {
-	if err := na.prepare(addr); err != nil {
+// target is the options for one address, [user@]host.
+func (na *nodeAdd) target(addr string) onboard.Options {
+	o := na.opts
+	o.Fabrics = slices.Clone(na.opts.Fabrics)
+	o.Addr = addr
+	if user, host, ok := strings.Cut(addr, "@"); ok {
+		o.User, o.Addr = user, host
+	}
+	if o.User == "" {
+		// Under sudo, the person who ran it — their account is usually the
+		// one that exists on the nodes too. root's own keys still do the
+		// logging in, since that is who is running ssh.
+		o.User = os.Getenv("SUDO_USER")
+	}
+	return o
+}
+
+// rememberFabrics wraps the question of which fabric a node's adapters are
+// on, so a name given for one node is used for the next whose adapters
+// report the same fabric, without asking again. ask nil is a script, which
+// is asked nothing and gets only what an earlier answer said.
+func (na *nodeAdd) rememberFabrics(ask func([]onboard.Fabric) []onboard.Fabric) func([]onboard.Fabric) []onboard.Fabric {
+	return func(s []onboard.Fabric) []onboard.Fabric {
+		var known []onboard.Fabric
+		all := true
+		for i, f := range s {
+			if name, ok := na.named[f.Key]; ok && f.Key != "" && f.Key != "unknown" {
+				f.Name = name
+				s[i].Name = name
+				known = append(known, f)
+			} else {
+				all = false
+			}
+		}
+		if all || ask == nil {
+			return known
+		}
+		got := ask(s)
+		for _, f := range got {
+			if f.Key != "" && f.Key != "unknown" {
+				na.named[f.Key] = f.Name
+			}
+		}
+		return got
+	}
+}
+
+func (na *nodeAdd) run(ctx context.Context, addrs []string, out io.Writer) error {
+	if len(addrs) > 1 && na.opts.Node != "" {
+		return errors.New("--node names one node; give one address with it")
+	}
+	if err := na.prepare(); err != nil {
 		return err
 	}
-	if !na.plain && isTerminal(out) {
-		return na.runTUI(ctx)
+	tui := !na.plain && isTerminal(out)
+	type outcome struct {
+		addr, node string
+		err        error
 	}
+	var results []outcome
+	for i, addr := range addrs {
+		o := na.target(addr)
+		title := "llmgw · adding " + o.Addr
+		if len(addrs) > 1 {
+			title += fmt.Sprintf(" (%d of %d)", i+1, len(addrs))
+		}
+		var node string
+		var err error
+		if tui {
+			node, err = na.runTUI(ctx, o, title)
+		} else {
+			node, err = na.runPlain(ctx, o, title, out)
+		}
+		results = append(results, outcome{addr, node, err})
+		if ctx.Err() != nil {
+			break
+		}
+	}
+	if len(addrs) == 1 {
+		return results[0].err
+	}
+	fmt.Fprintln(out)
+	failed := 0
+	for _, r := range results {
+		if r.err != nil {
+			failed++
+			fmt.Fprintf(out, "  FAIL  %-20s %s\n", r.addr, r.err)
+		} else {
+			fmt.Fprintf(out, "  ok    %-20s %s\n", r.addr, r.node)
+		}
+	}
+	if failed > 0 {
+		return fmt.Errorf("%d of %d not added; running the same command again finishes them and leaves the rest as they are", failed, len(addrs))
+	}
+	return nil
+}
+
+func (na *nodeAdd) runPlain(ctx context.Context, o onboard.Options, title string, out io.Writer) (string, error) {
 	p := &plainProgress{out: out}
-	fmt.Fprintf(out, "adding %s\n", na.opts.Addr)
-	r := onboard.New(na.opts, p.event)
-	res, err := r.Run(ctx)
+	fmt.Fprintf(out, "%s\n", title)
+	o.AskFabrics = na.rememberFabrics(nil)
+	res, err := onboard.New(o, p.event).Run(ctx)
 	if err != nil {
-		return err
+		return "", err
 	}
 	if na.noWait {
-		fmt.Fprintf(out, "\n%s is added. 'llmgw ctl nodes' shows it once it joins.\n", res.Node)
-		return nil
+		fmt.Fprintf(out, "\n%s is added. 'llmgw ctl nodes' shows it once it joins.\n\n", res.Node)
+		return res.Node, nil
 	}
 	if err := na.watch(ctx, res.Node, p.event); err != nil {
-		return err
+		return res.Node, err
 	}
-	fmt.Fprintf(out, "\n%s is in the cluster.\n", res.Node)
-	return nil
+	fmt.Fprintf(out, "\n%s is in the cluster.\n\n", res.Node)
+	return res.Node, nil
 }
 
 // adminAPI is the control plane's admin API, from its configuration when

internal/cli/server.go

@@ -7,6 +7,7 @@ import (
 	"fmt"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/sourcecheck"
+	"github.com/prometheus/client_golang/prometheus"
 	"log/slog"
 	"net"
 	"net/http"
@@ -21,11 +22,13 @@ import (
 	"golang.org/x/net/http2/h2c"
 	"golang.org/x/sync/errgroup"
 
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/agentsvc"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/fabrictest"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/fleetmetrics"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelsvc"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
@@ -34,6 +37,7 @@ import (
 	"gitea.mynodes.xyz/ilker/llmgw/internal/promq"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/router"
 	storepkg "gitea.mynodes.xyz/ilker/llmgw/internal/store"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/version"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/web"
 )
@@ -129,6 +133,28 @@ func newServerCmd() *cobra.Command {
 				// On the same /metrics as the traffic figures: correlating
 				// "the cluster was full" with "requests were queueing" should
 				// not need two systems.
+				// When the newest backup was written, so its absence alerts
+				// rather than being found on the day it is needed.
+				backupDir := cfg.BackupDir
+				if backupDir == "" {
+					backupDir = DefaultBackupDir
+				}
+				if err := rt.RegisterMetrics(prometheus.NewGaugeFunc(prometheus.GaugeOpts{
+					Name: "llmgw_backup_last_success_timestamp_seconds",
+					Help: "When the newest backup archive was written, as a Unix time; 0 when there is none.",
+				}, func() float64 {
+					if t := NewestBackup(backupDir); !t.IsZero() {
+						return float64(t.Unix())
+					}
+					return 0
+				})); err != nil {
+					return err
+				}
+				if err := rt.RegisterMetrics(tlsconf.ExpiryCollector(map[string]string{
+					"agent-server": cfg.AgentTLS.Cert, "agent-ca": cfg.AgentTLS.CA,
+				})); err != nil {
+					return err
+				}
 				if err := rt.RegisterMetrics(fleetmetrics.New(ctl)); err != nil {
 					return err
 				}
@@ -244,7 +270,7 @@ func newServerCmd() *cobra.Command {
 				// enforces its own per-phase deadlines.
 			}
 			adminSrv := &http.Server{
-				Handler:           adminMux,
+				Handler:           router.AdminGuard(adminMux),
 				ReadHeaderTimeout: 10 * time.Second,
 			}
 
@@ -359,6 +385,7 @@ func newServerCmd() *cobra.Command {
 	f.StringArrayVar(&endpoints, "endpoint", nil, "static endpoint: id=URL=model[,model] (repeatable)")
 
 	cmd.AddCommand(newServerInitCmd())
+	cmd.AddCommand(newBackupCmd())
 	return cmd
 }
 
@@ -383,6 +410,20 @@ func startControlPlane(ctx context.Context, cfg config.Server, cat *catalog.Live
 		ccfg.Reconciler.PortRange = [2]int{cfg.Control.PortRange[0], cfg.Control.PortRange[1]}
 	}
 
+	// Each replica's engine demands a key only the router has. The secret it
+	// is derived from lives in the store, so every control plane against it —
+	// and this one after a restart — computes the same key for a replica.
+	if h, ok := sink.(interface{ Store() *storepkg.DB }); ok && h.Store() != nil {
+		secret, err := h.Store().ClusterSecret(ctx, "engine-key")
+		if err != nil {
+			return nil, nil, fmt.Errorf("reading the engine key secret: %w", err)
+		}
+		ccfg.Reconciler.EngineKey = control.EngineKeys([]byte(secret))
+	} else {
+		log.Warn("engines start without a key: there is no store to keep the secret " +
+			"they are derived from, so anyone who can reach a node's ports reaches its models")
+	}
+
 	// The service needs the inventory and the status sink, both of which the
 	// controller owns; the controller needs the service as its dispatcher. The
 	// cycle is broken by giving the controller a dispatcher that forwards to
@@ -493,6 +534,18 @@ func (r *dispatcherRef) Logs(ctx context.Context, nodeID, source string, lines i
 	return lr.Logs(ctx, nodeID, source, lines)
 }
 
+func (r *dispatcherRef) FabricTest(ctx context.Context, nodeID string, req *llmgwv1.FabricTestRequest) (*llmgwv1.FabricTestReply, error) {
+	d, err := r.get()
+	if err != nil {
+		return nil, err
+	}
+	ft, ok := d.(fabrictest.Asker)
+	if !ok {
+		return nil, errors.New("this dispatcher cannot run a bandwidth test")
+	}
+	return ft.FabricTest(ctx, nodeID, req)
+}
+
 func (r *dispatcherRef) StartShard(ctx context.Context, cmd reconciler.ShardCommand) error {
 	d, err := r.get()
 	if err != nil {

internal/cli/serverinit.go

@@ -73,6 +73,8 @@ type serverInit struct {
 	// make it serve something: a mirror, this machine as a node, and a small
 	// model on it.
 	noMirror, noNode, noModel bool
+	// mirrorURL is a mirror on another machine to use instead of running one.
+	mirrorURL string
 	// pgSetup prepares PostgreSQL and returns the store address; a seam for
 	// tests. Nil does it for real.
 	pgSetup func(context.Context) (string, error)
@@ -92,6 +94,7 @@ func newServerInitCmd() *cobra.Command {
 		unitPath: "/etc/systemd/system/llmgw-server.service",
 		user:     "llmgw",
 	}
+	var plain bool
 	cmd := &cobra.Command{
 		Use:   "init",
 		Short: "Set this machine up as a control plane and start it",
@@ -122,15 +125,21 @@ func newServerInitCmd() *cobra.Command {
 				return errors.New("init creates a user, writes /etc and starts a service: run it as root")
 			}
 			in.out = cmd.OutOrStdout()
+			if !plain && isTerminal(in.out) {
+				return in.runInitTUI(cmd.Context(), in.out)
+			}
 			return in.run(cmd.Context())
 		},
 	}
 	f := cmd.Flags()
+	f.BoolVar(&plain, "plain", false, "plain output, even in a terminal")
 	f.StringArrayVar(&in.hosts, "host", nil,
 		"an address agents dial besides the ones found on this machine; repeatable")
 	f.StringVar(&in.admin, "admin", "", "email of the first administrator; a password is generated and printed once")
 	f.StringVar(&in.pkiDir, "pki", in.pkiDir, "directory holding the certificate authority")
 	f.BoolVar(&in.noMirror, "no-mirror", false, "do not run a model mirror on this machine")
+	f.StringVar(&in.mirrorURL, "mirror", "",
+		"use the mirror on another machine, set up there by 'llmgw mirror init': its management API, http://host:8082")
 	f.BoolVar(&in.noNode, "no-node", false, "do not make this machine a node (and serve no model on it)")
 	f.BoolVar(&in.noModel, "no-model", false, "do not fetch and serve the starter model ("+starterName+")")
 	f.BoolVar(&in.postgres, "postgres", false,
@@ -163,7 +172,12 @@ func (in *serverInit) run(ctx context.Context) error {
 		hasSystemd = false
 	}
 	// Before the service starts, because the mirror goes into server.yaml.
-	if hasSystemd && !in.noMirror {
+	switch {
+	case in.mirrorURL != "":
+		if err := in.useMirror(ctx); err != nil {
+			return fmt.Errorf("mirror: %w", err)
+		}
+	case hasSystemd && !in.noMirror:
 		if err := in.ensureMirror(ctx); err != nil {
 			return fmt.Errorf("mirror: %w", err)
 		}
@@ -175,14 +189,14 @@ func (in *serverInit) run(ctx context.Context) error {
 		if _, err := in.ensureNode(ctx); err != nil {
 			return fmt.Errorf("node: %w", err)
 		}
-		switch {
-		case in.noModel:
-		case in.noMirror:
-			fmt.Fprintln(in.out, "  skip  no starter model without the mirror")
-		default:
-			if err := in.ensureStarter(ctx); err != nil {
-				return fmt.Errorf("starter model: %w", err)
-			}
+	}
+	switch {
+	case !hasSystemd || in.noModel:
+	case in.noMirror && in.mirrorURL == "":
+		fmt.Fprintln(in.out, "  skip  no starter model without the mirror")
+	default:
+		if err := in.ensureStarter(ctx); err != nil {
+			return fmt.Errorf("starter model: %w", err)
 		}
 	}
 	in.nextSteps()

internal/cli/serverinit_cluster.go

@@ -111,15 +111,25 @@ func (in *serverInit) writeUnit(path, body string) (bool, error) {
 
 func (in *serverInit) mirrorUnit() string {
 	base := "http://" + net.JoinHostPort(in.reachableAddr(), "8081")
+	return mirrorUnitFor(mirrorDir, mirrorAPI, base, nil,
+		"'llmgw server init'. Bytes on :8081 for every node; the\n# management API on loopback only, for the control plane beside it.")
+}
+
+// mirrorUnitFor is a mirror's systemd unit. allow is who, besides the
+// machine itself, may use the management API.
+func mirrorUnitFor(dir, apiListen, base string, allow []string, about string) string {
+	exec := "/usr/local/bin/llmgw mirror serve --dir " + dir + " --listen :8081 --api-listen " + apiListen + " --base " + base
+	for _, a := range allow {
+		exec += " --api-allow " + a
+	}
 	return `[Unit]
 Description=llmgw model mirror
 After=network-online.target
 Wants=network-online.target
 
 [Service]
-# Written by 'llmgw server init'. Bytes on :8081 for every node; the
-# management API on loopback only, for the control plane beside it.
-ExecStart=/usr/local/bin/llmgw mirror serve --dir ` + mirrorDir + ` --listen :8081 --api-listen ` + mirrorAPI + ` --base ` + base + `
+# Written by ` + about + `
+ExecStart=` + exec + `
 User=llmgw
 Group=llmgw
 Restart=on-failure
@@ -128,13 +138,53 @@ NoNewPrivileges=true
 ProtectSystem=strict
 ProtectHome=true
 PrivateTmp=true
-ReadWritePaths=` + mirrorDir + `
+ReadWritePaths=` + dir + `
 
 [Install]
 WantedBy=multi-user.target
 `
 }
 
+// mirrorAPIURL is the mirror's management API: one on its own machine, or
+// the one beside this control plane.
+func (in *serverInit) mirrorAPIURL() string {
+	if in.mirrorURL != "" {
+		return in.mirrorURL
+	}
+	return "http://" + mirrorAPI
+}
+
+// useMirror points the control plane at a mirror on another machine, set up
+// there by 'llmgw mirror init'.
+func (in *serverInit) useMirror(ctx context.Context) error {
+	m, err := mirror.NewClient(in.mirrorURL, tlsconf.Config{}, 5*time.Second)
+	if err != nil {
+		return err
+	}
+	if err := m.Health(ctx); err != nil {
+		return fmt.Errorf("the mirror at %s does not answer: %v. On that machine: sudo llmgw mirror init --control-plane <this machine's address>", in.mirrorURL, err)
+	}
+	cfg, err := config.Load(in.configPath())
+	if err != nil {
+		return err
+	}
+	switch cfg.Mirror.API {
+	case in.mirrorURL:
+		in.kept("mirror at %s", in.mirrorURL)
+		return nil
+	case "":
+	default:
+		in.kept("mirror at %s, as server.yaml says; change it there to use %s", cfg.Mirror.API, in.mirrorURL)
+		return nil
+	}
+	if err := appendMirrorBlockFor(in.configPath(), in.mirrorURL,
+		"# The model mirror, on its own machine, added by 'llmgw server init'."); err != nil {
+		return err
+	}
+	in.done("server.yaml points at the mirror at %s", in.mirrorURL)
+	return nil
+}
+
 // ensureMirror runs a mirror on this machine and points the control plane at
 // it. Its management API is on loopback, which is why it needs no certificate:
 // nothing but this machine can reach it.
@@ -174,7 +224,9 @@ func (in *serverInit) ensureMirror(ctx context.Context) error {
 	}
 
 	if cfg.Mirror.API == "" {
-		if err := appendMirrorBlock(in.configPath()); err != nil {
+		if err := appendMirrorBlockFor(in.configPath(), "http://"+mirrorAPI,
+			"# The model mirror on this machine, added by 'llmgw server init'. Its\n"+
+				"# management API is on loopback, so it needs no certificate."); err != nil {
 			return err
 		}
 		in.done("server.yaml points at the mirror")
@@ -189,9 +241,9 @@ func (in *serverInit) ensureMirror(ctx context.Context) error {
 	})
 }
 
-// appendMirrorBlock adds the mirror to server.yaml without touching the rest,
-// keeping the file as it was beside it.
-func appendMirrorBlock(path string) error {
+// appendMirrorBlockFor adds the mirror to server.yaml without touching the
+// rest, keeping the file as it was beside it.
+func appendMirrorBlockFor(path, api, comment string) error {
 	b, err := os.ReadFile(path)
 	if err != nil {
 		return err
@@ -200,9 +252,7 @@ func appendMirrorBlock(path string) error {
 	if err := os.WriteFile(backup, b, 0o640); err != nil {
 		return err
 	}
-	block := "\n# The model mirror on this machine, added by 'llmgw server init'. Its\n" +
-		"# management API is on loopback, so it needs no certificate.\n" +
-		"mirror:\n  api: http://" + mirrorAPI + "\n"
+	block := "\n" + comment + "\nmirror:\n  api: " + api + "\n"
 	info, err := os.Stat(path)
 	if err != nil {
 		return err
@@ -356,8 +406,11 @@ func hasNVIDIADevice() bool {
 
 // starterModel is the catalog entry: a GPU profile first when there is a card,
 // and a CPU profile always, both pinned to this machine.
-func starterModel(source string, gpuClasses []string) catalog.Model {
-	pin := map[string]string{localLabelKey: localLabelValue}
+func starterModel(source string, gpuClasses []string, pinned bool) catalog.Model {
+	var pin map[string]string
+	if pinned {
+		pin = map[string]string{localLabelKey: localLabelValue}
+	}
 	var profiles []catalog.Profile
 	for _, class := range gpuClasses {
 		profiles = append(profiles, catalog.Profile{
@@ -401,7 +454,7 @@ func (in *serverInit) ensureStarter(ctx context.Context) error {
 	if !ok {
 		return fmt.Errorf("cannot turn %s into a download", starterSource)
 	}
-	m, err := mirror.NewClient("http://"+mirrorAPI, tlsconf.Config{}, 15*time.Second)
+	m, err := mirror.NewClient(in.mirrorAPIURL(), tlsconf.Config{}, 15*time.Second)
 	if err != nil {
 		return err
 	}
@@ -444,14 +497,24 @@ func (in *serverInit) ensureStarter(ctx context.Context) error {
 	}
 	in.done("%s in the mirror", starterName)
 
-	classes, _ := gpuClasses(ctx)
-	body, err := json.Marshal(starterModel(st.URL, classes))
+	// Served here when this machine is a node, pinned so it does not wander
+	// to the first GPU node that joins; otherwise on whichever node joins.
+	pinned := !in.noNode
+	var classes []string
+	if pinned {
+		classes, _ = gpuClasses(ctx)
+	}
+	body, err := json.Marshal(starterModel(st.URL, classes, pinned))
 	if err != nil {
 		return err
 	}
 	if err := client.post(ctx, "/api/v1/models", body, &map[string]string{}); err != nil {
 		return fmt.Errorf("adding %s to the catalog: %w", starterName, err)
 	}
+	if !pinned {
+		in.done("%s in the catalog; it starts on the first node that joins", starterName)
+		return nil
+	}
 	where := "on CPU"
 	if len(classes) > 0 {
 		where = "on " + strings.Join(classes, ", ") + ", or CPU if the card is busy"

internal/cli/serverinit_cluster_test.go

@@ -1,6 +1,8 @@
 package cli
 
 import (
+	"net/http"
+	"net/http/httptest"
 	"os"
 	"path/filepath"
 	"strings"
@@ -21,7 +23,7 @@ func TestTheInitAgentUnitIsTheShippedUnit(t *testing.T) {
 }
 
 func TestTheStarterUsesTheGPUAndFallsBackToCPUOnThisMachine(t *testing.T) {
-	m := starterModel("http://10.0.0.5:8081/qwen2.5-0.5b-instruct/starter/q.gguf", []string{"4090-24g"})
+	m := starterModel("http://10.0.0.5:8081/qwen2.5-0.5b-instruct/starter/q.gguf", []string{"4090-24g"}, true)
 	c, err := catalog.New([]catalog.Model{m})
 	if err != nil {
 		t.Fatalf("the starter is not a valid catalog entry: %v", err)
@@ -46,7 +48,7 @@ func TestTheStarterUsesTheGPUAndFallsBackToCPUOnThisMachine(t *testing.T) {
 		t.Error("the starter should be running, not waiting to be asked for")
 	}
 
-	if m := starterModel("http://x/q.gguf", nil); len(m.Profiles) != 1 || m.Profiles[0].Name != "cpu" {
+	if m := starterModel("http://x/q.gguf", nil, true); len(m.Profiles) != 1 || m.Profiles[0].Name != "cpu" {
 		t.Errorf("without a GPU only the CPU profile: %+v", m.Profiles)
 	}
 }
@@ -56,7 +58,7 @@ func TestTheMirrorBlockIsAddedAndReads(t *testing.T) {
 	if err := in.run(t.Context()); err != nil {
 		t.Fatalf("%v\n%s", err, out)
 	}
-	if err := appendMirrorBlock(in.configPath()); err != nil {
+	if err := appendMirrorBlockFor(in.configPath(), "http://"+mirrorAPI, "# the mirror"); err != nil {
 		t.Fatal(err)
 	}
 	cfg, err := config.Load(in.configPath())
@@ -87,3 +89,35 @@ func TestTheMirrorUnitAdvertisesAnAddressNodesCanReach(t *testing.T) {
 		t.Errorf("the management API is not on loopback: %s", u)
 	}
 }
+
+func TestWithNoNodeHereTheStarterGoesToTheFirstNodeThatJoins(t *testing.T) {
+	m := starterModel("http://x/q.gguf", nil, false)
+	for _, p := range m.Profiles {
+		if len(p.NodeSelector) > 0 {
+			t.Errorf("profile %s is pinned to a machine that is not a node: %v", p.Name, p.NodeSelector)
+		}
+	}
+}
+
+func TestTheMirrorAPIOnlyAnswersItsControlPlane(t *testing.T) {
+	allowed, err := parseAllow([]string{"10.0.0.140", "10.1.0.0/24"})
+	if err != nil {
+		t.Fatal(err)
+	}
+	h := onlyFrom(allowed, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {}))
+	for addr, want := range map[string]int{
+		"10.0.0.140:5000": 200, "10.1.0.7:5000": 200, "127.0.0.1:5000": 200, "[::1]:5000": 200,
+		"10.0.0.141:5000": 403, "192.168.1.9:5000": 403,
+	} {
+		r := httptest.NewRequest("GET", "/api/v1/health", nil)
+		r.RemoteAddr = addr
+		w := httptest.NewRecorder()
+		h.ServeHTTP(w, r)
+		if w.Code != want {
+			t.Errorf("%s: %d, want %d", addr, w.Code, want)
+		}
+	}
+	if _, err := parseAllow([]string{"not-an-address"}); err == nil {
+		t.Error("nonsense was accepted")
+	}
+}

internal/cli/status.go

@@ -94,6 +94,14 @@ func printSummary(cmd *cobra.Command, s adminapi.Summary) {
 	fmt.Fprintf(out, "\nMODELS    %d of %d have the replicas they asked for\n",
 		s.Models.Satisfied, s.Models.Total)
 
+	if len(s.Parked) > 0 {
+		fmt.Fprintln(out, "\nPARKED — failed every time they started; not retried until fixed")
+		for _, p := range s.Parked {
+			fmt.Fprintf(out, "  %s (%d failures): %s\n", p.Model, p.Failures, p.Reason)
+		}
+		fmt.Fprintln(out, "\n  Once fixed: llmgw ctl retry <model>")
+	}
+
 	if len(s.Unplaceable) > 0 {
 		fmt.Fprintln(out, "\nNOT PLACED")
 		for _, m := range s.Unplaceable {

internal/cli/tui_init.go

@@ -0,0 +1,421 @@
+package cli
+
+import (
+	"context"
+	"fmt"
+	"io"
+	"net/mail"
+	"os"
+	"path/filepath"
+	"strings"
+	"sync"
+
+	tea "github.com/charmbracelet/bubbletea"
+	"github.com/charmbracelet/lipgloss"
+)
+
+// The full-screen way through 'llmgw server init': a short form for what
+// init would otherwise take as flags, then init's own lines as a list that
+// fills in, with the administrator's password kept in view — and printed
+// again once the screen closes, so it is in the terminal's history.
+
+// initLineMsg is one line init wrote.
+type initLineMsg string
+
+// initDoneMsg is init finishing.
+type initDoneMsg struct{ err error }
+
+// lineWriter turns init's output into one message per line.
+type lineWriter struct {
+	mu   sync.Mutex
+	buf  []byte
+	send func(string)
+}
+
+func (w *lineWriter) Write(p []byte) (int, error) {
+	w.mu.Lock()
+	defer w.mu.Unlock()
+	w.buf = append(w.buf, p...)
+	for {
+		i := strings.IndexByte(string(w.buf), '\n')
+		if i < 0 {
+			return len(p), nil
+		}
+		w.send(string(w.buf[:i]))
+		w.buf = w.buf[i+1:]
+	}
+}
+
+// Form fields, in order.
+const (
+	fAdmin = iota
+	fHosts
+	fPostgres
+	fMirror
+	fNode
+	fModel
+	fStart
+	fCount
+)
+
+type initModel struct {
+	in    *serverInit
+	found []string // this machine's addresses
+
+	// The form, until it is submitted.
+	form      bool
+	focus     int
+	admin     textField
+	hosts     textField
+	toggles   [fCount]bool
+	formError string
+
+	// Then the run.
+	running  bool
+	lines    []string // init's lines, as it wrote them
+	password string
+	next     []string // the closing "Next:" text
+	spin     spinnerModel
+	width    int
+	done     bool
+	err      error
+	quit     bool
+	start    func()
+}
+
+func newInitModel(in *serverInit, found []string, askForm bool) initModel {
+	admin := newTextField()
+	admin.Placeholder = "you@example.com"
+	admin.SetValue(in.admin)
+	admin.CharLimit = 200
+	admin.Width = 40
+	hosts := newTextField()
+	hosts.Placeholder = "llm.example.com, 203.0.113.10"
+	hosts.SetValue(strings.Join(in.hosts, ", "))
+	hosts.CharLimit = 400
+	hosts.Width = 40
+	sp := newSpinner()
+	m := initModel{in: in, found: found, form: askForm, admin: admin, hosts: hosts, spin: sp, width: 100}
+	m.toggles[fPostgres] = in.postgres
+	m.toggles[fMirror] = !in.noMirror
+	m.toggles[fNode] = !in.noNode
+	m.toggles[fModel] = !in.noModel
+	if askForm {
+		m.admin.Focus()
+	}
+	return m
+}
+
+func (m initModel) Init() tea.Cmd {
+	if m.form {
+		return nil
+	}
+	return tea.Batch(m.spin.Tick, func() tea.Msg { return startInitMsg{} })
+}
+
+type startInitMsg struct{}
+
+func (m initModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
+	switch msg := msg.(type) {
+	case tea.WindowSizeMsg:
+		m.width = msg.Width
+	case spinTickMsg:
+		var cmd tea.Cmd
+		m.spin, cmd = m.spin.Update(msg)
+		return m, cmd
+	case startInitMsg:
+		m.running = true
+		if m.start != nil {
+			m.start()
+		}
+	case initLineMsg:
+		m.addLine(string(msg))
+	case initDoneMsg:
+		m.done, m.err = true, msg.err
+		return m, tea.Quit
+	case tea.KeyMsg:
+		if msg.Type == tea.KeyCtrlC {
+			m.quit = true
+			return m, tea.Quit
+		}
+		if m.form {
+			return m.formKey(msg)
+		}
+	}
+	return m, nil
+}
+
+// addLine files one of init's lines: the password aside, the closing text
+// aside, a progress line replacing the one before it.
+func (m *initModel) addLine(l string) {
+	t := strings.TrimSpace(l)
+	switch {
+	case t == "":
+		return
+	case strings.HasPrefix(t, "password: "):
+		m.password = strings.TrimPrefix(t, "password: ")
+		return
+	case strings.HasPrefix(t, "shown once"):
+		return
+	case !strings.HasPrefix(l, "  ") || len(m.next) > 0:
+		// Unindented lines are the closing text, and everything after its
+		// first line belongs to it.
+		m.next = append(m.next, l)
+		return
+	}
+	if strings.HasPrefix(t, "..") && len(m.lines) > 0 && strings.HasPrefix(strings.TrimSpace(m.lines[len(m.lines)-1]), "..") {
+		m.lines[len(m.lines)-1] = l
+		return
+	}
+	m.lines = append(m.lines, l)
+}
+
+func (m initModel) formKey(k tea.KeyMsg) (tea.Model, tea.Cmd) {
+	switch k.Type {
+	case tea.KeyTab, tea.KeyDown:
+		m.moveFocus(1)
+		return m, nil
+	case tea.KeyShiftTab, tea.KeyUp:
+		m.moveFocus(-1)
+		return m, nil
+	case tea.KeyEnter:
+		if m.focus != fStart {
+			m.moveFocus(1)
+			return m, nil
+		}
+		admin := strings.TrimSpace(m.admin.Value())
+		if admin != "" {
+			if _, err := mail.ParseAddress(admin); err != nil {
+				m.formError = "the administrator's email is not an address"
+				m.focus = fAdmin
+				m.admin.Focus()
+				return m, nil
+			}
+		}
+		m.in.admin = admin
+		m.in.hosts = nil
+		for _, h := range strings.Split(m.hosts.Value(), ",") {
+			if h = strings.TrimSpace(h); h != "" {
+				m.in.hosts = append(m.in.hosts, h)
+			}
+		}
+		m.in.postgres = m.toggles[fPostgres]
+		m.in.noMirror = !m.toggles[fMirror]
+		m.in.noNode = !m.toggles[fNode]
+		m.in.noModel = !m.toggles[fModel]
+		m.form = false
+		return m, tea.Batch(m.spin.Tick, func() tea.Msg { return startInitMsg{} })
+	}
+	switch m.focus {
+	case fAdmin:
+		var cmd tea.Cmd
+		m.admin, cmd = m.admin.Update(k)
+		return m, cmd
+	case fHosts:
+		var cmd tea.Cmd
+		m.hosts, cmd = m.hosts.Update(k)
+		return m, cmd
+	case fPostgres, fMirror, fNode, fModel:
+		if k.String() == " " || k.String() == "x" {
+			m.toggles[m.focus] = !m.toggles[m.focus]
+		}
+	}
+	return m, nil
+}
+
+func (m *initModel) moveFocus(d int) {
+	m.focus = (m.focus + d + fCount) % fCount
+	// With --mirror there is no mirror to run here, so nothing to tick.
+	if m.focus == fMirror && m.in.mirrorURL != "" {
+		m.focus = (m.focus + d + fCount) % fCount
+	}
+	m.admin.Blur()
+	m.hosts.Blur()
+	switch m.focus {
+	case fAdmin:
+		m.admin.Focus()
+	case fHosts:
+		m.hosts.Focus()
+	}
+}
+
+func (m initModel) View() string {
+	var b strings.Builder
+	b.WriteString(sTitle.Render("llmgw · setting up this machine as a control plane") + "\n\n")
+	if m.form {
+		return b.String() + m.formView()
+	}
+	for _, l := range m.visibleLines() {
+		b.WriteString(m.lineView(l) + "\n")
+	}
+	if m.password != "" {
+		box := sBox.BorderForeground(tuiWarn).Render(
+			fmt.Sprintf("Administrator %s\npassword  %s\n%s", m.in.admin,
+				lipgloss.NewStyle().Bold(true).Render(m.password),
+				sDim.Render("Shown once. Copy it now; change it on the account page after signing in.")))
+		b.WriteString("\n" + box + "\n")
+	}
+	switch {
+	case m.done && m.err != nil:
+		b.WriteString("\n" + sBad.Render("✗ ") + wrap(m.err.Error(), m.width-4, "  ") + "\n" +
+			sDim.Render("  It is safe to run init again once that is fixed: it finishes what is missing.") + "\n")
+	case m.done:
+		b.WriteString("\n" + sOK.Render("✓ ") + "The control plane is up. Add a node: " +
+			lipgloss.NewStyle().Bold(true).Render("sudo llmgw node add <user>@<address>") + "\n")
+		b.WriteString("  " + sDim.Render("Back up "+m.in.pkiDir+": its ca-key.pem signs every node's identity, and nothing recovers it.") + "\n")
+	case m.quit:
+		b.WriteString("\n" + sDim.Render("stopped; run init again to finish") + "\n")
+	case m.running:
+		b.WriteString("\n" + sDim.Render("ctrl+c to stop") + "\n")
+	}
+	return b.String()
+}
+
+// visibleLines are the last lines, enough to follow without the screen
+// outgrowing the terminal: the whole log is printed when it closes.
+func (m initModel) visibleLines() []string {
+	const keep = 16
+	if len(m.lines) > keep {
+		return m.lines[len(m.lines)-keep:]
+	}
+	return m.lines
+}
+
+func (m initModel) lineView(l string) string {
+	t := strings.TrimSpace(l)
+	word, rest, _ := strings.Cut(t, " ")
+	rest = truncate(strings.TrimSpace(rest), max(20, m.width-6))
+	switch word {
+	case "ok":
+		return "  " + sOK.Render("✓") + " " + rest
+	case "kept":
+		return "  " + sDim.Render("· "+rest)
+	case "skip":
+		return "  " + sDim.Render("- "+rest)
+	case "!!":
+		return "  " + sWarn.Render("! ") + rest
+	case "..":
+		if m.done {
+			return "  " + sDim.Render("· "+rest)
+		}
+		return "  " + m.spin.View() + " " + rest
+	}
+	return "    " + t
+}
+
+func (m initModel) formView() string {
+	var b strings.Builder
+	label := func(i int, s string) string {
+		if m.focus == i {
+			return sTitle.Render("› " + s)
+		}
+		return "  " + s
+	}
+	check := func(i int, s, hint string) string {
+		box := "[ ]"
+		if m.toggles[i] {
+			box = "[" + sOK.Render("x") + "]"
+		}
+		return fmt.Sprintf("%s %s %s\n", label(i, box), s, sDim.Render(hint))
+	}
+	fmt.Fprintf(&b, "%s\n    %s\n", label(fAdmin, "Administrator's email"), m.admin.View())
+	b.WriteString("    " + sDim.Render("A password is generated and shown once. Leave empty to create accounts later.") + "\n\n")
+	fmt.Fprintf(&b, "%s\n    %s\n", label(fHosts, "Other names nodes will reach this machine by"), m.hosts.View())
+	found := "none found"
+	if len(m.found) > 0 {
+		found = strings.Join(m.found, ", ")
+	}
+	b.WriteString("    " + sDim.Render("Already in the certificate: "+found+". Add a DNS name or NAT address here.") + "\n\n")
+	b.WriteString(check(fPostgres, "PostgreSQL", "rather than SQLite; installed here, for several control planes or a larger store"))
+	if m.in.mirrorURL != "" {
+		b.WriteString("  Model mirror " + sDim.Render("at "+m.in.mirrorURL+", from --mirror") + "\n")
+	} else {
+		b.WriteString(check(fMirror, "Model mirror", "downloads model files once for every node"))
+	}
+	b.WriteString(check(fNode, "This machine as a node", "serves models here too, on its GPUs or CPU"))
+	b.WriteString(check(fModel, "Starter model", "a small Qwen, so there is something to ask at once"))
+	b.WriteString("\n" + label(fStart, lipgloss.NewStyle().Bold(true).Render("Set up")) + "\n")
+	if m.formError != "" {
+		b.WriteString("\n" + sBad.Render(m.formError) + "\n")
+	}
+	b.WriteString("\n" + sDim.Render("tab or ↑↓ to move · space to tick · enter on Set up to start · ctrl+c to leave") + "\n")
+	return b.String()
+}
+
+// runInitTUI runs init with the screen above, asking the form first when
+// the flags left the administrator unset.
+func (in *serverInit) runInitTUI(ctx context.Context, out io.Writer) error {
+	var found []string
+	for _, ip := range localAddrs() {
+		found = append(found, ip.String())
+	}
+	m := newInitModel(in, found, in.admin == "")
+	var p *tea.Program
+
+	var (
+		logMu sync.Mutex
+		log   []string
+	)
+	w := &lineWriter{send: func(l string) {
+		logMu.Lock()
+		log = append(log, l)
+		logMu.Unlock()
+		p.Send(initLineMsg(l))
+	}}
+	ctx, cancel := context.WithCancel(ctx)
+	defer cancel()
+	started := make(chan struct{})
+	m.start = func() {
+		close(started)
+		go func() {
+			in.out = w
+			err := in.run(ctx)
+			p.Send(initDoneMsg{err})
+		}()
+	}
+	p = tea.NewProgram(m, tea.WithOutput(out))
+	final, err := p.Run()
+	if err != nil {
+		return err
+	}
+	fm := final.(initModel)
+	select {
+	case <-started:
+	default:
+		return fmt.Errorf("left before anything was done")
+	}
+	if fm.quit && !fm.done {
+		cancel()
+		return fmt.Errorf("stopped; run init again to finish what was started")
+	}
+
+	// The whole run is kept, root's only, since it holds the password. On a
+	// failure it is printed too, for the lines the screen had no room for;
+	// otherwise the screen's last frame says it, and only what must outlive
+	// the screen is printed again: the password, and what to do next.
+	logMu.Lock()
+	defer logMu.Unlock()
+	whole := strings.Join(log, "\n") + "\n"
+	saved := ""
+	if in.stateDir != "" {
+		p := filepath.Join(in.stateDir, "init.log")
+		if err := os.WriteFile(p, []byte(whole), 0o600); err == nil {
+			saved = p
+		}
+	}
+	fmt.Fprintln(out)
+	if fm.err != nil {
+		fmt.Fprint(out, whole)
+		return fm.err
+	}
+	if fm.password != "" {
+		fmt.Fprintf(out, "Administrator %s, password %s (shown once)\n\n", in.admin, fm.password)
+	}
+	for _, l := range fm.next {
+		fmt.Fprintln(out, l)
+	}
+	if saved != "" {
+		fmt.Fprintf(out, "\nThe whole run is in %s.\n", saved)
+	}
+	return fm.err
+}

internal/cli/tui_init_test.go

@@ -0,0 +1,103 @@
+package cli
+
+import (
+	"strings"
+	"testing"
+
+	tea "github.com/charmbracelet/bubbletea"
+)
+
+func typeIn(m tea.Model, s string) tea.Model {
+	return send(m, tea.KeyMsg{Type: tea.KeyRunes, Runes: []rune(s)})
+}
+
+func TestTheInitFormBecomesInitsSettings(t *testing.T) {
+	in := &serverInit{pkiDir: "/etc/llmgw/pki"}
+	var m tea.Model = newInitModel(in, []string{"10.0.0.140"}, true)
+	if v := m.View(); !strings.Contains(v, "Already in the certificate: 10.0.0.140") {
+		t.Fatalf("the addresses found are not shown:\n%s", v)
+	}
+	m = typeIn(m, "ops@example.com")
+	m = send(m, tea.KeyMsg{Type: tea.KeyTab})
+	m = typeIn(m, "llm.example.com, 203.0.113.10")
+	m = send(m, tea.KeyMsg{Type: tea.KeyTab}, tea.KeyMsg{Type: tea.KeyRunes, Runes: []rune(" ")}) // PostgreSQL on
+	m = send(m, tea.KeyMsg{Type: tea.KeyTab}, tea.KeyMsg{Type: tea.KeyTab}, tea.KeyMsg{Type: tea.KeyTab},
+		tea.KeyMsg{Type: tea.KeyRunes, Runes: []rune(" ")}) // starter model off
+	m = send(m, tea.KeyMsg{Type: tea.KeyTab}, tea.KeyMsg{Type: tea.KeyEnter})
+
+	if in.admin != "ops@example.com" || strings.Join(in.hosts, "|") != "llm.example.com|203.0.113.10" {
+		t.Errorf("admin %q hosts %q", in.admin, in.hosts)
+	}
+	if !in.postgres || in.noMirror || in.noNode || !in.noModel {
+		t.Errorf("toggles: postgres %v mirror %v node %v model %v", in.postgres, !in.noMirror, !in.noNode, !in.noModel)
+	}
+	if m.(initModel).form {
+		t.Error("the form stayed up after Set up")
+	}
+}
+
+func TestAnEmailThatIsNotOneIsSaidSo(t *testing.T) {
+	in := &serverInit{}
+	var m tea.Model = newInitModel(in, nil, true)
+	m = typeIn(m, "not an address")
+	for range fStart {
+		m = send(m, tea.KeyMsg{Type: tea.KeyTab})
+	}
+	m = send(m, tea.KeyMsg{Type: tea.KeyEnter})
+	if !m.(initModel).form || !strings.Contains(m.View(), "not an address") {
+		t.Errorf("a bad email was accepted:\n%s", m.View())
+	}
+}
+
+func TestInitsLinesAreFiledAndThePasswordKeptInView(t *testing.T) {
+	in := &serverInit{admin: "ops@example.com", pkiDir: "/etc/llmgw/pki"}
+	var m tea.Model = newInitModel(in, nil, false)
+	for _, l := range []string{
+		"  ok    system user llmgw",
+		"  kept  certificate authority in /etc/llmgw/pki",
+		"  ok    administrator ops@example.com (u1)",
+		"",
+		"        password: Tr0ub4dor-3",
+		"        shown once — sign in and change it from /profile",
+		"",
+		"  ..    downloading qwen2.5-0.5b-instruct into the mirror",
+		"  ..    40%",
+		"  ..    80%",
+		"  !!    an NVIDIA card is on the PCI bus but nvidia-smi does not answer",
+		"The control plane is up. Next:",
+		"  llmgw ctl status",
+	} {
+		m = send(m, initLineMsg(l))
+	}
+	m = send(m, initDoneMsg{})
+	im := m.(initModel)
+	if im.password != "Tr0ub4dor-3" {
+		t.Errorf("password %q", im.password)
+	}
+	v := m.View()
+	for _, want := range []string{"✓ system user llmgw", "Tr0ub4dor-3", "Shown once", "80%", "! an NVIDIA card", "Back up /etc/llmgw/pki", "node add"} {
+		if !strings.Contains(v, want) {
+			t.Errorf("screen lacks %q:\n%s", want, v)
+		}
+	}
+	if strings.Contains(v, "40%") {
+		t.Error("progress lines piled up instead of replacing each other")
+	}
+	if strings.Contains(v, "llmgw ctl status") {
+		t.Error("the closing text was listed as a step")
+	}
+}
+
+func TestWithAMirrorElsewhereTheFormDoesNotOfferOne(t *testing.T) {
+	in := &serverInit{mirrorURL: "http://10.0.0.150:8082"}
+	var m tea.Model = newInitModel(in, nil, true)
+	if v := m.View(); !strings.Contains(v, "at http://10.0.0.150:8082, from --mirror") {
+		t.Errorf("the mirror in use is not shown:\n%s", v)
+	}
+	// Tab from the addresses goes past the mirror line to PostgreSQL, then
+	// the node.
+	m = send(m, tea.KeyMsg{Type: tea.KeyTab}, tea.KeyMsg{Type: tea.KeyTab}, tea.KeyMsg{Type: tea.KeyTab})
+	if f := m.(initModel).focus; f != fNode {
+		t.Errorf("focus %d, want the node toggle %d", f, fNode)
+	}
+}

internal/cli/tui_node.go

@@ -7,8 +7,6 @@ import (
 	"os"
 	"strings"
 
-	"github.com/charmbracelet/bubbles/spinner"
-	"github.com/charmbracelet/bubbles/textinput"
 	tea "github.com/charmbracelet/bubbletea"
 	"github.com/charmbracelet/lipgloss"
 	"github.com/mattn/go-isatty"
@@ -67,7 +65,7 @@ type nodeModel struct {
 	title   string
 	steps   map[string]stepState
 	notes   []onboard.Event // warnings and failures, with their fixes
-	spin    spinner.Model
+	spin    spinnerModel
 	cancel  context.CancelFunc
 	width   int
 	done    bool
@@ -77,16 +75,14 @@ type nodeModel struct {
 
 	// The question being asked, if any.
 	fabrics   *askFabricsMsg
-	inputs    []textinput.Model
+	inputs    []textField
 	focus     int
 	confirm   *askContinueMsg
 	quitAsked bool
 }
 
 func newNodeModel(title string, cancel context.CancelFunc) nodeModel {
-	sp := spinner.New()
-	sp.Spinner = spinner.MiniDot
-	sp.Style = lipgloss.NewStyle().Foreground(tuiAccent)
+	sp := newSpinner()
 	return nodeModel{title: title, steps: map[string]stepState{}, spin: sp, cancel: cancel, width: 100}
 }
 
@@ -96,7 +92,7 @@ func (m nodeModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
 	switch msg := msg.(type) {
 	case tea.WindowSizeMsg:
 		m.width = msg.Width
-	case spinner.TickMsg:
+	case spinTickMsg:
 		var cmd tea.Cmd
 		m.spin, cmd = m.spin.Update(msg)
 		return m, cmd
@@ -121,7 +117,7 @@ func (m nodeModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
 		m.fabrics = &msg
 		m.inputs = nil
 		for i, f := range msg.suggested {
-			in := textinput.New()
+			in := newTextField()
 			in.Prompt = ""
 			in.SetValue(f.Name)
 			in.CharLimit = 40
@@ -176,7 +172,7 @@ func (m nodeModel) key(k tea.KeyMsg) (tea.Model, tea.Cmd) {
 			var out []onboard.Fabric
 			for i, in := range m.inputs {
 				if name := strings.TrimSpace(in.Value()); name != "" {
-					out = append(out, onboard.Fabric{Name: name, HCAs: m.fabrics.suggested[i].HCAs})
+					out = append(out, onboard.Fabric{Name: name, HCAs: m.fabrics.suggested[i].HCAs, Key: m.fabrics.suggested[i].Key})
 				}
 			}
 			m.fabrics.reply <- out
@@ -297,15 +293,15 @@ func wrap(s string, width int, indent string) string {
 	return strings.Join(lines, "\n"+indent)
 }
 
-// runTUI adds the node with the screen above.
-func (na *nodeAdd) runTUI(ctx context.Context) error {
+// runTUI adds one node with the screen above.
+func (na *nodeAdd) runTUI(ctx context.Context, o onboard.Options, title string) (string, error) {
 	ctx, cancel := context.WithCancel(ctx)
 	defer cancel()
-	m := newNodeModel("llmgw · adding "+na.opts.Addr, cancel)
+	m := newNodeModel(title, cancel)
 	p := tea.NewProgram(m)
 
 	emit := func(e onboard.Event) { p.Send(evMsg(e)) }
-	na.opts.AskFabrics = func(s []onboard.Fabric) []onboard.Fabric {
+	o.AskFabrics = na.rememberFabrics(func(s []onboard.Fabric) []onboard.Fabric {
 		reply := make(chan []onboard.Fabric)
 		p.Send(askFabricsMsg{s, reply})
 		select {
@@ -314,8 +310,8 @@ func (na *nodeAdd) runTUI(ctx context.Context) error {
 		case <-ctx.Done():
 			return nil
 		}
-	}
-	na.opts.AskContinue = func(fails int) bool {
+	})
+	o.AskContinue = func(fails int) bool {
 		reply := make(chan bool)
 		p.Send(askContinueMsg{fails, reply})
 		select {
@@ -327,7 +323,7 @@ func (na *nodeAdd) runTUI(ctx context.Context) error {
 	}
 
 	go func() {
-		res, err := onboard.New(na.opts, emit).Run(ctx)
+		res, err := onboard.New(o, emit).Run(ctx)
 		node := ""
 		if res != nil {
 			node = res.Node
@@ -344,11 +340,12 @@ func (na *nodeAdd) runTUI(ctx context.Context) error {
 
 	final, err := p.Run()
 	if err != nil {
-		return err
+		return "", err
 	}
 	fm := final.(nodeModel)
 	if fm.quitAsked && !fm.done {
-		return fmt.Errorf("stopped; the node may be half set up, and running this again finishes it")
+		cancel()
+		return fm.node, fmt.Errorf("stopped; the node may be half set up, and running this again finishes it")
 	}
-	return fm.err
+	return fm.node, fm.err
 }

internal/cli/tui_node_test.go

@@ -94,3 +94,24 @@ func TestANodeCountsAsJoinedOnlyWhenHeardSinceItsAgentStarted(t *testing.T) {
 		}
 	}
 }
+
+func TestAFabricNamedForOneNodeIsNotAskedAgainForTheNext(t *testing.T) {
+	na := &nodeAdd{named: map[string]string{}}
+	asked := 0
+	ask := func(s []onboard.Fabric) []onboard.Fabric {
+		asked++
+		return []onboard.Fabric{{Name: "ib-compute", HCAs: s[0].HCAs, Key: s[0].Key}}
+	}
+	first := na.rememberFabrics(ask)([]onboard.Fabric{{Name: "fabric", HCAs: []string{"mlx5_0"}, Key: "sm:0xa"}})
+	second := na.rememberFabrics(ask)([]onboard.Fabric{{Name: "fabric", HCAs: []string{"mlx5_0", "mlx5_1"}, Key: "sm:0xa"}})
+	if asked != 1 {
+		t.Errorf("asked %d times", asked)
+	}
+	if len(second) != 1 || second[0].Name != "ib-compute" || len(second[0].HCAs) != 2 || first[0].Name != "ib-compute" {
+		t.Errorf("first %+v second %+v", first, second)
+	}
+	// A fabric not seen before is asked about; in a script nothing is asked.
+	if got := na.rememberFabrics(nil)([]onboard.Fabric{{Name: "fabric", Key: "sm:0xb"}}); len(got) != 0 {
+		t.Errorf("a script was given a fabric nobody named: %+v", got)
+	}
+}

internal/cli/tui_widgets.go

@@ -0,0 +1,98 @@
+package cli
+
+import (
+	"strings"
+	"time"
+
+	tea "github.com/charmbracelet/bubbletea"
+	"github.com/charmbracelet/lipgloss"
+)
+
+// Two small widgets, written here rather than taken from bubbles: its
+// packages ask the terminal for its colours as they load, so every llmgw
+// command — llmgw --version included — sent a query and, in a terminal that
+// never answers one, waited five seconds for it. Only the screens need to
+// know the colours, and lipgloss asks when they first draw.
+
+// spinTickMsg advances a spinner.
+type spinTickMsg struct{}
+
+var spinFrames = []string{"⠋", "⠙", "⠹", "⠸", "⠼", "⠴", "⠦", "⠧", "⠇", "⠏"}
+
+type spinnerModel struct {
+	frame int
+	Style lipgloss.Style
+}
+
+func newSpinner() spinnerModel { return spinnerModel{Style: lipgloss.NewStyle().Foreground(tuiAccent)} }
+
+// Tick starts the spinner turning; it is a tea.Cmd.
+func (s spinnerModel) Tick() tea.Msg { return spinTickMsg{} }
+
+func (s spinnerModel) Update(msg tea.Msg) (spinnerModel, tea.Cmd) {
+	if _, ok := msg.(spinTickMsg); !ok {
+		return s, nil
+	}
+	s.frame = (s.frame + 1) % len(spinFrames)
+	return s, tea.Tick(100*time.Millisecond, func(time.Time) tea.Msg { return spinTickMsg{} })
+}
+
+func (s spinnerModel) View() string { return s.Style.Render(spinFrames[s.frame]) }
+
+// textField is one line of text typed at its end: all a form here needs.
+type textField struct {
+	value       []rune
+	focused     bool
+	Prompt      string
+	Placeholder string
+	CharLimit   int
+	Width       int
+}
+
+func newTextField() textField { return textField{Prompt: "> "} }
+
+func (f *textField) SetValue(s string) { f.value = []rune(s) }
+func (f textField) Value() string      { return string(f.value) }
+func (f *textField) Focus()            { f.focused = true }
+func (f *textField) Blur()             { f.focused = false }
+
+func (f textField) Update(msg tea.Msg) (textField, tea.Cmd) {
+	k, ok := msg.(tea.KeyMsg)
+	if !ok || !f.focused {
+		return f, nil
+	}
+	switch k.Type {
+	case tea.KeyRunes, tea.KeySpace:
+		r := k.Runes
+		if k.Type == tea.KeySpace {
+			r = []rune{' '}
+		}
+		for _, c := range r {
+			if f.CharLimit == 0 || len(f.value) < f.CharLimit {
+				f.value = append(f.value, c)
+			}
+		}
+	case tea.KeyBackspace:
+		if len(f.value) > 0 {
+			f.value = f.value[:len(f.value)-1]
+		}
+	case tea.KeyCtrlU:
+		f.value = nil
+	}
+	return f, nil
+}
+
+func (f textField) View() string {
+	text := string(f.value)
+	shown := text
+	if text == "" && f.Placeholder != "" {
+		shown = sDim.Render(f.Placeholder)
+	}
+	if f.focused {
+		shown += "█"
+	}
+	if pad := f.Width - len([]rune(text)); pad > 0 && text != "" {
+		shown += strings.Repeat(" ", pad)
+	}
+	return f.Prompt + shown
+}

internal/config/config.go

@@ -81,6 +81,11 @@ type Server struct {
 	// one leaves that engine to be installed by hand.
 	Engines map[string]string `koanf:"engines"`
 
+	// BackupDir is where `llmgw server backup` writes, and where the newest
+	// archive's age is read for llmgw_backup_last_success_timestamp_seconds.
+	// Empty means /var/backups/llmgw-server.
+	BackupDir string `koanf:"backup_dir"`
+
 	Router  Router          `koanf:"router"`
 	Control Control         `koanf:"control"`
 	Models  []catalog.Model `koanf:"models"`
@@ -404,8 +409,11 @@ type Scoring struct {
 
 // Endpoint is one engine replica. Static in phase 1.
 type Endpoint struct {
-	ID     string   `koanf:"id"`
-	URL    string   `koanf:"url"`
+	ID  string `koanf:"id"`
+	URL string `koanf:"url"`
+	// APIKey is sent to the engine as a bearer token, for an engine started
+	// with one.
+	APIKey string   `koanf:"api_key"`
 	Models []string `koanf:"models"`
 	// Engine names the inference engine, which decides what its metrics are
 	// called. Empty means vLLM's names.

internal/control/adminapi/api.go

@@ -75,7 +75,11 @@ func (a *API) Register(mux *http.ServeMux) {
 	mux.HandleFunc("GET /api/v1/explain-placement", a.explainPlacement)
 	mux.HandleFunc("POST /api/v1/nodes/{id}/state", a.setNodeState)
 	mux.HandleFunc("DELETE /api/v1/nodes/{id}", a.removeNode)
+	mux.HandleFunc("POST /api/v1/fabric-test", a.fabricTest)
 	mux.HandleFunc("GET /api/v1/nodes/{id}/logs", a.nodeLogs)
+	mux.HandleFunc("POST /api/v1/restart", a.restart)
+	mux.HandleFunc("GET /api/v1/requests/{id}", a.request)
+	mux.HandleFunc("POST /api/v1/models/{name}/retry", a.retryModel)
 }
 
 // --- responses ---
@@ -133,7 +137,10 @@ type NodeView struct {
 
 	LastSeenAgo  string `json:"last_seen_ago"`
 	AgentVersion string `json:"agent_version,omitempty"`
-	Replicas     int    `json:"replicas"`
+	// Features are the requests the agent answers beyond the usual:
+	// "fabric-test". Absent is an agent too old to say.
+	Features []string `json:"features,omitempty"`
+	Replicas int      `json:"replicas"`
 
 	// Load is how busy the machine is, absent when its agent has not
 	// measured it lately — which is unknown, not idle.
@@ -228,13 +235,32 @@ type ModelView struct {
 	Max         int      `json:"max_replicas"`
 	Ready       int      `json:"ready_replicas"`
 	Total       int      `json:"total_replicas"`
+	// Desired is how many it should have now: Min, or for an on-demand
+	// model one while it is being asked for and none otherwise.
+	Desired int `json:"desired_replicas"`
+	// Failures counts replicas that failed to start in a row.
+	Failures int `json:"failures,omitempty"`
+	// Parked is set once Failures reached the limit; the model is not retried
+	// until its definition changes or `ctl retry` is run. Reason is the last
+	// error.
+	Parked bool   `json:"parked,omitempty"`
+	Reason string `json:"reason,omitempty"`
+	// Unplaceable is set when it is short and the last attempt found no node.
+	Unplaceable bool     `json:"unplaceable,omitempty"`
 	Fallbacks   []string `json:"fallbacks,omitempty"`
 	Profiles    []string `json:"profiles"`
 }
 
 // ReplicaView is a replica as `ctl replicas` shows it.
 type ReplicaView struct {
-	ID      string `json:"id"`
+	ID string `json:"id"`
+	// EngineKey is the key this replica's engine demands, for `ctl bench`
+	// and anyone else holding the admin listener who needs to reach an engine
+	// directly. The client-facing pages never render it.
+	EngineKey string `json:"engine_key,omitempty"`
+	// Restart is "queued" or "replacing" while an operator's restart of this
+	// replica is under way.
+	Restart string `json:"restart,omitempty"`
 	Model   string `json:"model"`
 	Profile string `json:"profile"`
 	// Stale is true when the profile has been edited since this replica was
@@ -334,7 +360,7 @@ func (a *API) Nodes() []NodeView {
 			State: n.State.String(), Labels: n.Labels,
 			GPUTotal: len(n.GPUs), GPUFree: len(inv.FreeGPUs(n.ID)),
 			MemoryGB: n.MemoryTotalMB / 1024,
-			Fabrics:  n.FabricNames(), AgentVersion: n.AgentVersion,
+			Fabrics:  n.FabricNames(), AgentVersion: n.AgentVersion, Features: n.Features,
 			Replicas: replicasOn[n.ID],
 		}
 		if free, total := n.LocalDisk(); total > 0 {
@@ -430,13 +456,20 @@ func (a *API) Models() []ModelView {
 		}
 	}
 
+	health := map[string]reconciler.ModelHealth{}
+	for _, h := range a.ctl.Reconciler().ModelHealth() {
+		health[h.Model] = h
+	}
 	out := make([]ModelView, 0)
 	for _, m := range a.ctl.Catalog().Models() {
+		h := health[m.Name]
 		v := ModelView{
 			Name: m.Name, Description: m.Description, Aliases: m.Aliases, Source: m.Source,
 			Min: m.Scaling.Min, Max: m.Scaling.Max,
 			Ready: ready[m.Name], Total: total[m.Name],
 			Fallbacks: m.Fallbacks,
+			Desired:   h.Desired, Failures: h.Failures,
+			Parked: h.Parked, Reason: h.Reason, Unplaceable: h.Unplaceable,
 		}
 		for _, p := range m.Profiles {
 			v.Profiles = append(v.Profiles, p.Name)
@@ -467,9 +500,11 @@ func (a *API) Replicas(model string) []ReplicaView {
 			ID: rep.ID, Model: rep.Model, Profile: rep.Profile,
 			State: rep.State.String(), Pool: rep.Pool, Fabric: rep.Fabric,
 			Endpoint: rep.Endpoint, Err: rep.Err,
-			Stale:   a.profileChanged(rep.Model, rep.Profile, rep.ProfileFP),
-			AgeSec:  int64(time.Since(rep.CreatedAt).Seconds()),
-			IdleSec: -1,
+			EngineKey: a.ctl.EngineKey(rep.ID),
+			Restart:   a.ctl.Reconciler().RestartState(rep.ID),
+			Stale:     a.profileChanged(rep.Model, rep.Profile, rep.ProfileFP),
+			AgeSec:    int64(time.Since(rep.CreatedAt).Seconds()),
+			IdleSec:   -1,
 		}
 		if rep.LastUsedAt.IsZero() {
 			v.IdleSec = -1
@@ -1035,3 +1070,98 @@ func (a *API) Topology(id string) (TopologyView, bool) {
 func (a *API) NVLink() nvlink.Report {
 	return nvlink.Analyze(a.ctl.Inventory().Nodes())
 }
+
+// RestartRequest names what to restart: replicas by id, every replica of some
+// models, or — All — every ready replica in the cluster.
+type RestartRequest struct {
+	Replicas []string `json:"replicas,omitempty"`
+	Models   []string `json:"models,omitempty"`
+	All      bool     `json:"all,omitempty"`
+}
+
+// RestartResult lists the replicas queued.
+type RestartResult struct {
+	Queued []string `json:"queued"`
+}
+
+func (a *API) restart(w http.ResponseWriter, r *http.Request) {
+	var req RestartRequest
+	if err := json.NewDecoder(io.LimitReader(r.Body, 1<<16)).Decode(&req); err != nil {
+		writeError(w, http.StatusBadRequest, "the restart request is not valid JSON: "+err.Error())
+		return
+	}
+	ids := slices.Clone(req.Replicas)
+	for _, rep := range a.ctl.Reconciler().ReadyReplicas() {
+		if req.All || slices.Contains(req.Models, rep.Model) {
+			ids = append(ids, rep.ID)
+		}
+	}
+	if len(ids) == 0 {
+		writeError(w, http.StatusBadRequest, "nothing ready matches; name replicas, models, or all")
+		return
+	}
+	if err := a.ctl.Reconciler().Restart(ids...); err != nil {
+		writeError(w, http.StatusNotFound, err.Error())
+		return
+	}
+	writeJSON(w, http.StatusAccepted, RestartResult{Queued: ids})
+}
+
+func (a *API) retryModel(w http.ResponseWriter, r *http.Request) {
+	name := r.PathValue("name")
+	if _, ok := a.ctl.Catalog().Resolve(name); !ok {
+		writeError(w, http.StatusNotFound, "no model called "+name)
+		return
+	}
+	was := a.ctl.Reconciler().Retry(name)
+	writeJSON(w, http.StatusOK, map[string]bool{"was_parked": was})
+}
+
+// requestFinder looks a request up by the id it was given. Asked of the usage
+// source by type assertion, so one without it answers 501 rather than every
+// source having to grow the method.
+type requestFinder interface {
+	UsageByRequest(ctx context.Context, requestID string) (store.Usage, bool, error)
+}
+
+// RequestView is what one request went through.
+type RequestView struct {
+	RequestID        string    `json:"request_id"`
+	At               time.Time `json:"at"`
+	KeyID            string    `json:"key_id"`
+	UserID           string    `json:"user_id,omitempty"`
+	ProjectID        string    `json:"project_id"`
+	Model            string    `json:"model"`
+	ServedModel      string    `json:"served_model"`
+	Replica          string    `json:"replica,omitempty"`
+	Status           int       `json:"status"`
+	LatencyMs        int64     `json:"latency_ms"`
+	QueueMs          int64     `json:"queue_ms"`
+	TTFTMs           int64     `json:"ttft_ms"`
+	PromptTokens     int64     `json:"prompt_tokens"`
+	CompletionTokens int64     `json:"completion_tokens"`
+	CostMicros       int64     `json:"cost_micros"`
+}
+
+func (a *API) request(w http.ResponseWriter, r *http.Request) {
+	f, ok := a.usage.(requestFinder)
+	if !ok {
+		writeError(w, http.StatusNotImplemented, "this control plane keeps no usage records")
+		return
+	}
+	u, found, err := f.UsageByRequest(r.Context(), r.PathValue("id"))
+	switch {
+	case err != nil:
+		writeError(w, http.StatusInternalServerError, err.Error())
+		return
+	case !found:
+		writeError(w, http.StatusNotFound, "no request with that id; records are written a few seconds after a request ends")
+		return
+	}
+	writeJSON(w, http.StatusOK, RequestView{
+		RequestID: u.RequestID, At: u.At, KeyID: u.KeyID, UserID: u.UserID, ProjectID: u.ProjectID,
+		Model: u.Model, ServedModel: u.ServedModel, Replica: u.Replica, Status: u.Status,
+		LatencyMs: u.LatencyMs, QueueMs: u.QueueMs, TTFTMs: u.TTFTMs,
+		PromptTokens: u.PromptTokens, CompletionTokens: u.CompletionTokens, CostMicros: u.CostMicros,
+	})
+}

internal/control/adminapi/fabrictest.go

@@ -0,0 +1,71 @@
+package adminapi
+
+import (
+	"encoding/json"
+	"fmt"
+	"net/http"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/fabrictest"
+)
+
+// FabricTestView is a bandwidth test between two nodes.
+type FabricTestView struct {
+	A      string     `json:"a"`
+	B      string     `json:"b"`
+	Fabric string     `json:"fabric"`
+	OK     bool       `json:"ok"`
+	Rails  []RailView `json:"rails"`
+	Notes  []string   `json:"notes,omitempty"`
+	Took   string     `json:"took"`
+}
+
+// RailView is one rail's result.
+type RailView struct {
+	A        string  `json:"a"`
+	B        string  `json:"b"`
+	RateGbps int     `json:"rate_gbps"`
+	Gbps     float64 `json:"gbps"`
+	Verdict  string  `json:"verdict"`
+	Detail   string  `json:"detail,omitempty"`
+}
+
+// fabricTest measures RDMA bandwidth between two nodes, rail by rail. It
+// sends traffic over the fabric for seconds a rail, so it is a POST and only
+// ever run when asked.
+func (a *API) fabricTest(w http.ResponseWriter, r *http.Request) {
+	var body struct {
+		A, B, Fabric string
+		Seconds      int
+	}
+	if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
+		writeError(w, http.StatusBadRequest, err.Error())
+		return
+	}
+	ask := a.ctl.FabricTester()
+	if ask == nil {
+		writeError(w, http.StatusNotImplemented, "this control plane cannot reach its nodes' agents")
+		return
+	}
+	inv := a.ctl.Inventory()
+	na, okA := inv.Get(body.A)
+	nb, okB := inv.Get(body.B)
+	if !okA || !okB {
+		writeError(w, http.StatusNotFound, fmt.Sprintf("no node %q", map[bool]string{true: body.B, false: body.A}[okA]))
+		return
+	}
+	fabric, rails, notes, err := fabrictest.Plan(na, nb, body.Fabric)
+	if err != nil {
+		writeError(w, http.StatusBadRequest, err.Error())
+		return
+	}
+	start := time.Now()
+	rep := fabrictest.Run(r.Context(), ask, na, nb, fabric, rails,
+		fabrictest.Options{Seconds: min(max(body.Seconds, 1), 30), Settle: 2 * time.Second})
+	v := FabricTestView{A: na.ID, B: nb.ID, Fabric: fabric, OK: rep.OK(), Notes: append(notes, rep.Notes...),
+		Took: time.Since(start).Round(time.Second).String()}
+	for _, x := range rep.Results {
+		v.Rails = append(v.Rails, RailView{x.A, x.B, x.RateGbps, x.Gbps, x.Verdict, x.Detail})
+	}
+	writeJSON(w, http.StatusOK, v)
+}

internal/control/adminapi/summary.go

@@ -25,6 +25,9 @@ type Summary struct {
 	// Unplaceable names models the scheduler currently cannot satisfy. This is
 	// the line that turns "something is wrong" into a place to look.
 	Unplaceable []string `json:"unplaceable,omitempty"`
+	// Parked are models that failed every time they started and are no
+	// longer retried, with the last error.
+	Parked []ParkedModel `json:"parked,omitempty"`
 	// IdleReplicas are serving replicas nothing has asked for in a while, which
 	// on a fleet with more models than GPUs is the first place to find room.
 	IdleReplicas []IdleReplica `json:"idle_replicas,omitempty"`
@@ -201,17 +204,30 @@ func (a *API) Summary() Summary {
 
 	for _, m := range a.Models() {
 		s.Models.Total++
-		if m.Ready >= m.Min {
+		if m.Parked {
+			s.Parked = append(s.Parked, ParkedModel{Model: m.Name, Failures: m.Failures, Reason: m.Reason})
+		}
+		// Against what it should have now, not its minimum: an on-demand
+		// model's minimum is 0, so one that was asked for and fitted nowhere
+		// used to count as satisfied.
+		if m.Ready >= m.Desired && !m.Parked {
 			s.Models.Satisfied++
 			continue
 		}
 		s.Models.Short++
 		// Short of replicas and unable to place more is the actionable case;
 		// short and still starting is just a cluster doing its job.
-		if m.Total <= m.Ready {
+		if m.Unplaceable || (m.Total <= m.Ready && !m.Parked) {
 			s.Unplaceable = append(s.Unplaceable, m.Name)
 		}
 	}
 	sort.Strings(s.Unplaceable)
 	return s
 }
+
+// ParkedModel is a model no longer retried, and why.
+type ParkedModel struct {
+	Model    string `json:"model"`
+	Failures int    `json:"failures"`
+	Reason   string `json:"reason"`
+}

internal/control/agentsvc/convert.go

@@ -23,6 +23,7 @@ func NodeFromRegister(r *llmgwv1.Register) inventory.Node {
 		Labels:         r.GetLabels(),
 		State:          inventory.Ready,
 		AgentVersion:   r.GetAgentVersion(),
+		Features:       r.GetFeatures(),
 		EngineVersions: r.GetEngineVersions(),
 		EnginesProbed:  r.GetEnginesProbed(),
 		MemoryTotalMB:  int(r.GetMemoryTotalMb()),
@@ -190,6 +191,7 @@ func StartShardFromCommand(cmd reconciler.ShardCommand) *llmgwv1.StartShard {
 		Pool:             cmd.Pool,
 		Fabric:           cmd.Fabric,
 		ProfileFp:        cmd.ProfileFP,
+		ApiKey:           cmd.Spec.APIKey,
 	}
 }
 
@@ -210,6 +212,7 @@ func ShardSpecFromStart(m *llmgwv1.StartShard) (engines.ShardSpec, time.Duration
 		Pool:          m.GetPool(),
 		Fabric:        m.GetFabric(),
 		ProfileFP:     m.GetProfileFp(),
+		APIKey:        m.GetApiKey(),
 		Profile: engines.Profile{
 			Name: p.GetName(), Engine: p.GetEngine(), GPUClass: p.GetGpuClass(),
 			TP: int(p.GetTp()), PP: int(p.GetPp()),

internal/control/agentsvc/service.go

@@ -19,6 +19,7 @@ import (
 	"gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1/llmgwv1connect"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/supervisor"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/fabrictest"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
 )
@@ -61,6 +62,7 @@ type Service struct {
 	logSeq   uint64
 	// scanWaits are the fabric scans awaiting a reply, by request id.
 	scanWaits map[string]chan *llmgwv1.FabricScanReply
+	testWaits map[string]chan *llmgwv1.FabricTestReply
 }
 
 // SetEngines sets the engines every node installs, by name and version.
@@ -289,6 +291,15 @@ func (s *Service) handle(nodeID string, msg *llmgwv1.AgentMessage) {
 				"last_output", m.Shard.GetLastOutput())
 		}
 
+	case *llmgwv1.AgentMessage_FabricTest:
+		s.logMu.Lock()
+		ch, ok := s.testWaits[m.FabricTest.GetRequestId()]
+		delete(s.testWaits, m.FabricTest.GetRequestId())
+		s.logMu.Unlock()
+		if ok {
+			ch <- m.FabricTest
+		}
+
 	case *llmgwv1.AgentMessage_FabricScan:
 		s.logMu.Lock()
 		ch, ok := s.scanWaits[m.FabricScan.GetRequestId()]
@@ -717,6 +728,38 @@ func (s *Service) ScanFabric(ctx context.Context, nodeID, hca string, port int)
 	}
 }
 
+// FabricTest asks a node for one side of a bandwidth test, and waits for
+// what it measured. The request's id is filled in here.
+func (s *Service) FabricTest(ctx context.Context, nodeID string, req *llmgwv1.FabricTestRequest) (*llmgwv1.FabricTestReply, error) {
+	s.logMu.Lock()
+	if s.testWaits == nil {
+		s.testWaits = map[string]chan *llmgwv1.FabricTestReply{}
+	}
+	s.logSeq++
+	id := fmt.Sprintf("test-%d", s.logSeq)
+	ch := make(chan *llmgwv1.FabricTestReply, 1)
+	s.testWaits[id] = ch
+	s.logMu.Unlock()
+	defer func() {
+		s.logMu.Lock()
+		delete(s.testWaits, id)
+		s.logMu.Unlock()
+	}()
+	req.RequestId = id
+	if err := s.send(nodeID, &llmgwv1.ControlMessage{Msg: &llmgwv1.ControlMessage_FabricTest{FabricTest: req}}); err != nil {
+		return nil, err
+	}
+	// The test's own length, and a package install the first time.
+	ctx, cancel := context.WithTimeout(ctx, time.Duration(req.GetSeconds())*time.Second+5*time.Minute)
+	defer cancel()
+	select {
+	case r := <-ch:
+		return r, nil
+	case <-ctx.Done():
+		return nil, fmt.Errorf("node %s did not answer the bandwidth test", nodeID)
+	}
+}
+
 // ScanFabrics maps every physical InfiniBand fabric every so often, asking
 // one node on each: the first, by name, whose port on it is up. A fabric
 // looks the same from all its ports, so one is enough, and the rest of its
@@ -795,3 +838,7 @@ func installable(reg *llmgwv1.Register) []string {
 	}
 	return []string{"llamacpp"}
 }
+
+// The admin API finds the bandwidth test by asserting this; a signature that
+// drifted would make it quietly nil.
+var _ fabrictest.Asker = (*Service)(nil)

internal/control/catalog/catalog.go

@@ -302,6 +302,13 @@ func (m Model) validate() error {
 		}
 		seen[p.Name] = struct{}{}
 
+		for k := range p.Env {
+			if bad, why := engines.UnsafeEnv(k); bad {
+				return fmt.Errorf("model %q profile %q: env %s is not allowed: %s",
+					m.Name, p.Name, k, why)
+			}
+		}
+
 		if p.GPUClass == "" && !cpuCapable(p.Engine) {
 			return fmt.Errorf("model %q profile %q: gpu_class is required for engine %q",
 				m.Name, p.Name, p.Engine)

internal/control/catalog/catalog_test.go

@@ -302,3 +302,22 @@ func TestSimulatedNeedsEveryProfileToBeSimulated(t *testing.T) {
 		})
 	}
 }
+
+func TestAProfileCannotSetWhatRunsCodeInTheEngine(t *testing.T) {
+	// A profile's env reaches an engine started as root. LD_PRELOAD from a
+	// model definition would make "add a model" into "run this everywhere".
+	for _, k := range []string{"LD_PRELOAD", "ld_library_path", "PYTHONSTARTUP", "PATH", "BASH_ENV"} {
+		m := Model{Name: "m", Source: "http://x/m.gguf", Profiles: []Profile{{
+			Name: "cpu", Engine: "llamacpp", Env: map[string]string{k: "/tmp/x"},
+		}}}
+		if _, err := New([]Model{m}); err == nil || !strings.Contains(err.Error(), "not allowed") {
+			t.Errorf("%s: %v", k, err)
+		}
+	}
+	ok := Model{Name: "m", Source: "http://x/m.gguf", Profiles: []Profile{{
+		Name: "cpu", Engine: "llamacpp", Env: map[string]string{"OMP_NUM_THREADS": "4", "PYTHONUNBUFFERED": "1"},
+	}}}
+	if _, err := New([]Model{ok}); err != nil {
+		t.Errorf("ordinary tuning refused: %v", err)
+	}
+}

internal/control/control.go

@@ -4,6 +4,9 @@ package control
 
 import (
 	"context"
+	"crypto/hmac"
+	"crypto/sha256"
+	"encoding/hex"
 	"fmt"
 	"log/slog"
 	"strings"
@@ -11,6 +14,7 @@ import (
 	"time"
 
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/fabrictest"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint"
@@ -167,6 +171,9 @@ func (c *Controller) Nodes() []inventory.Node        { return c.inv.Nodes() }
 func (c *Controller) FreeGPUs(nodeID string) []int   { return c.inv.FreeGPUs(nodeID) }
 func (c *Controller) Replicas() []reconciler.Replica { return c.rec.Replicas() }
 
+// ModelHealth is how each model in the catalog stands.
+func (c *Controller) ModelHealth() []reconciler.ModelHealth { return c.rec.ModelHealth() }
+
 // Reconciler exposes the reconciliation loop, for status reports and ctl.
 func (c *Controller) Reconciler() *reconciler.Reconciler { return c.rec }
 
@@ -215,6 +222,12 @@ func (c *Controller) LogReader() LogReader {
 	return r
 }
 
+// FabricTester returns the dispatcher's bandwidth test capability, or nil.
+func (c *Controller) FabricTester() fabrictest.Asker {
+	r, _ := c.disp.(fabrictest.Asker)
+	return r
+}
+
 // Start runs the reconciler and begins publishing endpoints.
 func (c *Controller) Start(ctx context.Context) {
 	if c.lead != nil {
@@ -297,6 +310,7 @@ func (c *Controller) endpoints() []endpoint.Info {
 			ID:       rep.ID,
 			URL:      strings.TrimRight(rep.Endpoint, "/"),
 			Models:   names,
+			APIKey:   c.engineKey(rep.ID),
 			Engine:   prof.Engine,
 			Pool:     rep.Pool,
 			Weight:   rep.Weight,
@@ -314,3 +328,29 @@ func fingerprint(infos []endpoint.Info) string {
 	}
 	return b.String()
 }
+
+// EngineKey is the key a replica's engine demands, empty when engines run
+// without one.
+func (c *Controller) EngineKey(replicaID string) string { return c.engineKey(replicaID) }
+
+func (c *Controller) engineKey(replicaID string) string {
+	if c.cfg.Reconciler.EngineKey == nil {
+		return ""
+	}
+	return c.cfg.Reconciler.EngineKey(replicaID)
+}
+
+// EngineKeys derives each replica's engine key from one secret.
+//
+// HMAC rather than a stored random key per replica: the router needs the key
+// for every replica it sends to, including ones adopted after a restart that
+// this process never started, and a second control plane has to agree. A
+// function of a shared secret and the replica id gives all three without a
+// table to keep in step. A key taken from one node opens that replica only.
+func EngineKeys(secret []byte) func(replicaID string) string {
+	return func(replicaID string) string {
+		m := hmac.New(sha256.New, secret)
+		m.Write([]byte("llmgw engine key\x00" + replicaID))
+		return "llmgw-" + hex.EncodeToString(m.Sum(nil))[:40]
+	}
+}

internal/control/control_test.go

@@ -0,0 +1,19 @@
+package control
+
+import "testing"
+
+func TestEngineKeysAreTheSameForTheSameReplicaAndNoOther(t *testing.T) {
+	// A control plane that restarts, or a second one, must compute the key a
+	// running engine was started with; a key taken from one node must open
+	// that replica only.
+	a, b := EngineKeys([]byte("s")), EngineKeys([]byte("s"))
+	if a("r1") != b("r1") {
+		t.Error("the same secret gave two keys for one replica")
+	}
+	if a("r1") == a("r2") {
+		t.Error("two replicas share a key")
+	}
+	if a("r1") == EngineKeys([]byte("other"))("r1") {
+		t.Error("the key does not depend on the secret")
+	}
+}

internal/control/fabrictest/fabrictest.go

@@ -0,0 +1,235 @@
+// Package fabrictest measures the fabric between two nodes, rail by rail.
+//
+// Every check on one node — ports up, a subnet manager, the right modules —
+// can pass while the two nodes still cannot reach each other: a cable into the
+// wrong switch, a partition key, a RoCE network with no route. A replica
+// spanning them then starts, and hangs on its first all-reduce with nothing to
+// say why. This runs what that all-reduce would ride on — an RDMA write from
+// each adapter to its counterpart — and says how it went against the link's
+// rate, before a replica does.
+package fabrictest
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"slices"
+	"strings"
+	"time"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
+)
+
+// Asker sends one side of a test to a node and waits for its answer.
+type Asker interface {
+	FabricTest(ctx context.Context, node string, req *llmgwv1.FabricTestRequest) (*llmgwv1.FabricTestReply, error)
+}
+
+// Rail is one adapter on each node, paired by their position in the fabric's
+// declaration: mlx5_0 with mlx5_0, the way NCCL pairs them.
+type Rail struct {
+	A, B         string // adapters
+	PortA, PortB int
+	// RateGbps is the slower of the two links.
+	RateGbps int
+}
+
+// Result is one rail's test.
+type Result struct {
+	Rail
+	Gbps float64 `json:"gbps"`
+	// Verdict is "ok" at 80% of the link's rate or more, "slow" at 50%,
+	// "fail" below that or when it did not run.
+	Verdict string `json:"verdict"`
+	Detail  string `json:"detail,omitempty"`
+}
+
+// Report is a whole test.
+type Report struct {
+	Fabric  string   `json:"fabric"`
+	A, B    string   `json:"-"`
+	Results []Result `json:"results"`
+	// Notes say what the test could not cover.
+	Notes []string `json:"notes,omitempty"`
+}
+
+// OK reports whether every rail passed.
+func (r Report) OK() bool {
+	for _, x := range r.Results {
+		if x.Verdict != "ok" {
+			return false
+		}
+	}
+	return len(r.Results) > 0
+}
+
+// Plan pairs the adapters two nodes have on a fabric. With no fabric named,
+// the one fabric they share is used.
+func Plan(a, b inventory.Node, fabric string) (string, []Rail, []string, error) {
+	if a.ID == b.ID {
+		return "", nil, nil, errors.New("give two different nodes")
+	}
+	if fabric == "" {
+		var common []string
+		for _, f := range a.FabricNames() {
+			if b.HasFabric(f) {
+				common = append(common, f)
+			}
+		}
+		switch len(common) {
+		case 0:
+			return "", nil, nil, fmt.Errorf("%s (%s) and %s (%s) declare no fabric in common",
+				a.ID, orNone(a.FabricNames()), b.ID, orNone(b.FabricNames()))
+		case 1:
+			fabric = common[0]
+		default:
+			return "", nil, nil, fmt.Errorf("%s and %s share %s: name one with --fabric", a.ID, b.ID, strings.Join(common, ", "))
+		}
+	}
+	for _, n := range []inventory.Node{a, b} {
+		if !slices.Contains(n.Features, "fabric-test") {
+			return "", nil, nil, fmt.Errorf("%s's agent (%s) is too old to run a bandwidth test: upgrade it", n.ID, orNone([]string{n.AgentVersion}))
+		}
+	}
+	ha, hb := a.HCAsFor(fabric), b.HCAsFor(fabric)
+	if len(ha) == 0 || len(hb) == 0 {
+		return "", nil, nil, fmt.Errorf("%s is not declared on both %s and %s", fabric, a.ID, b.ID)
+	}
+	var notes []string
+	if len(ha) != len(hb) {
+		notes = append(notes, fmt.Sprintf("%s has %d adapters on %s and %s has %d: only the first %d are paired",
+			a.ID, len(ha), fabric, b.ID, len(hb), min(len(ha), len(hb))))
+	}
+	var rails []Rail
+	for i := range min(len(ha), len(hb)) {
+		pa, ra := port(a, ha[i])
+		pb, rb := port(b, hb[i])
+		rails = append(rails, Rail{A: ha[i], B: hb[i], PortA: pa, PortB: pb, RateGbps: minPositive(ra, rb)})
+	}
+	return fabric, rails, notes, nil
+}
+
+// port is an adapter's first active port and its rate, or port 1 when the
+// node reported none.
+func port(n inventory.Node, hca string) (int, int) {
+	for _, p := range n.IBPorts {
+		if p.HCA == hca && strings.EqualFold(p.State, "ACTIVE") {
+			return p.Port, p.RateGbps
+		}
+	}
+	return 1, 0
+}
+
+// Options for a run.
+type Options struct {
+	Seconds int
+	// BasePort is perftest's TCP exchange port for the first rail; each rail
+	// takes the next, so a rail whose server is still closing never meets the
+	// next one's.
+	BasePort int
+	// Settle is how long the server is given to listen before the client
+	// dials; the client retries besides.
+	Settle time.Duration
+}
+
+// Run tests every rail, one after another: rails at once would share the
+// PCIe and memory bandwidth they are measuring.
+func Run(ctx context.Context, ask Asker, a, b inventory.Node, fabric string, rails []Rail, o Options) Report {
+	if o.Seconds <= 0 {
+		o.Seconds = 3
+	}
+	if o.BasePort == 0 {
+		o.BasePort = 18515
+	}
+	rep := Report{Fabric: fabric, A: a.ID, B: b.ID}
+	for i, r := range rails {
+		rep.Results = append(rep.Results, runRail(ctx, ask, a, b, r, o, o.BasePort+i))
+	}
+	return rep
+}
+
+func runRail(ctx context.Context, ask Asker, a, b inventory.Node, r Rail, o Options, tcp int) Result {
+	res := Result{Rail: r, Verdict: "fail"}
+	if b.MgmtAddr == "" {
+		res.Detail = b.ID + " reported no management address for the client to dial"
+		return res
+	}
+	type answer struct {
+		r   *llmgwv1.FabricTestReply
+		err error
+	}
+	server := make(chan answer, 1)
+	go func() {
+		rep, err := ask.FabricTest(ctx, b.ID, &llmgwv1.FabricTestRequest{Role: "server", Hca: r.B,
+			Port: int32(r.PortB), TcpPort: int32(tcp), Seconds: int32(o.Seconds)}) //nolint:gosec // small numbers
+		server <- answer{rep, err}
+	}()
+	select {
+	case <-ctx.Done():
+		res.Detail = ctx.Err().Error()
+		return res
+	case <-time.After(o.Settle):
+	}
+	client, err := ask.FabricTest(ctx, a.ID, &llmgwv1.FabricTestRequest{Role: "client", Hca: r.A,
+		Port: int32(r.PortA), Peer: b.MgmtAddr, TcpPort: int32(tcp), Seconds: int32(o.Seconds)}) //nolint:gosec // small numbers
+	var srv answer
+	select {
+	case srv = <-server:
+	case <-time.After(time.Duration(o.Seconds)*time.Second + time.Minute):
+		srv.err = errors.New("the server side did not finish")
+	}
+	switch {
+	case err != nil:
+		res.Detail = a.ID + ": " + err.Error()
+	case client.GetError() != "":
+		res.Detail = a.ID + ": " + client.GetError()
+		// The server's reason is often the better one: it saw no one arrive.
+		if srv.r != nil && srv.r.GetError() != "" {
+			res.Detail += "; " + b.ID + ": " + srv.r.GetError()
+		}
+	default:
+		res.Gbps = client.GetGbps()
+		res.Verdict, res.Detail = grade(res.Gbps, r.RateGbps)
+	}
+	return res
+}
+
+// grade compares a measurement with the link's rate. RDMA writes of 64 KiB
+// reach 90–97% of the signalling rate on a healthy link; well under that is
+// a degraded cable, a card in a narrow PCIe slot, or traffic crossing a
+// socket.
+func grade(gbps float64, rate int) (string, string) {
+	if gbps <= 0 {
+		return "fail", "no traffic got through"
+	}
+	if rate <= 0 {
+		return "ok", "the link's rate is not known to compare with"
+	}
+	pct := gbps / float64(rate) * 100
+	switch {
+	case pct >= 80:
+		return "ok", fmt.Sprintf("%.0f%% of %d Gb/s", pct, rate)
+	case pct >= 50:
+		return "slow", fmt.Sprintf("%.0f%% of %d Gb/s: check the cable, and the card's PCIe width and NUMA node", pct, rate)
+	default:
+		return "fail", fmt.Sprintf("%.0f%% of %d Gb/s", pct, rate)
+	}
+}
+
+func minPositive(a, b int) int {
+	switch {
+	case a <= 0:
+		return b
+	case b <= 0:
+		return a
+	}
+	return min(a, b)
+}
+
+func orNone(s []string) string {
+	if len(s) == 0 {
+		return "none"
+	}
+	return strings.Join(slices.Clone(s), ", ")
+}

internal/control/fabrictest/fabrictest_test.go

@@ -0,0 +1,100 @@
+package fabrictest
+
+import (
+	"context"
+	"strings"
+	"sync"
+	"testing"
+	"time"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
+)
+
+func node(id string, hcas ...string) inventory.Node {
+	n := inventory.Node{ID: id, MgmtAddr: "10.0.0." + id[len(id)-1:], Features: []string{"fabric-test"},
+		Fabrics: []inventory.FabricAttachment{{Fabric: "ib-a", HCAs: hcas}}}
+	for _, h := range hcas {
+		n.IBPorts = append(n.IBPorts, inventory.IBPort{HCA: h, Port: 1, State: "ACTIVE", RateGbps: 400})
+	}
+	return n
+}
+
+// fakeFabric answers as two agents would: the server waits for its client.
+type fakeFabric struct {
+	mu    sync.Mutex
+	gbps  map[string]float64 // by client adapter
+	fails map[string]string
+	asked []string
+}
+
+func (f *fakeFabric) FabricTest(_ context.Context, n string, req *llmgwv1.FabricTestRequest) (*llmgwv1.FabricTestReply, error) {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	f.asked = append(f.asked, n+" "+req.GetRole()+" "+req.GetHca())
+	if req.GetRole() == "server" {
+		return &llmgwv1.FabricTestReply{}, nil
+	}
+	if e := f.fails[req.GetHca()]; e != "" {
+		return &llmgwv1.FabricTestReply{Error: e}, nil
+	}
+	return &llmgwv1.FabricTestReply{Gbps: f.gbps[req.GetHca()]}, nil
+}
+
+func TestRailsArePairedByPositionOnTheSharedFabric(t *testing.T) {
+	a, b := node("n1", "mlx5_0", "mlx5_1"), node("n2", "mlx5_0", "mlx5_1", "mlx5_2")
+	fabric, rails, notes, err := Plan(a, b, "")
+	if err != nil {
+		t.Fatal(err)
+	}
+	if fabric != "ib-a" || len(rails) != 2 || rails[1].A != "mlx5_1" || rails[1].B != "mlx5_1" || rails[0].RateGbps != 400 {
+		t.Errorf("got %s %+v", fabric, rails)
+	}
+	if len(notes) != 1 || !strings.Contains(notes[0], "only the first 2") {
+		t.Errorf("an unequal count went unmentioned: %v", notes)
+	}
+}
+
+func TestNodesWithNoFabricInCommonAreSaidSo(t *testing.T) {
+	b := node("n2", "mlx5_0")
+	b.Fabrics[0].Fabric = "ib-b"
+	if _, _, _, err := Plan(node("n1", "mlx5_0"), b, ""); err == nil || !strings.Contains(err.Error(), "no fabric in common") {
+		t.Errorf("got %v", err)
+	}
+}
+
+func TestEachRailIsGradedAgainstItsLink(t *testing.T) {
+	a, b := node("n1", "mlx5_0", "mlx5_1", "mlx5_2"), node("n2", "mlx5_0", "mlx5_1", "mlx5_2")
+	_, rails, _, _ := Plan(a, b, "ib-a")
+	f := &fakeFabric{
+		gbps:  map[string]float64{"mlx5_0": 388, "mlx5_1": 240},
+		fails: map[string]string{"mlx5_2": "ib_write_bw: Couldn't connect to 10.0.0.2:18517"},
+	}
+	rep := Run(context.Background(), f, a, b, "ib-a", rails, Options{Seconds: 1, Settle: time.Millisecond})
+	want := []string{"ok", "slow", "fail"}
+	for i, r := range rep.Results {
+		if r.Verdict != want[i] {
+			t.Errorf("rail %d: %s (%s), want %s", i, r.Verdict, r.Detail, want[i])
+		}
+	}
+	if rep.OK() {
+		t.Error("a report with a failed rail is OK")
+	}
+	if !strings.Contains(rep.Results[2].Detail, "Couldn't connect") {
+		t.Errorf("the reason was lost: %q", rep.Results[2].Detail)
+	}
+	// Servers on the second node, clients on the first, one rail at a time.
+	if f.asked[0] != "n2 server mlx5_0" || f.asked[1] != "n1 client mlx5_0" {
+		t.Errorf("asked %v", f.asked)
+	}
+}
+
+func TestAnAgentTooOldToAnswerIsRefusedAtOnce(t *testing.T) {
+	// It would ignore the request, and the control plane would wait minutes.
+	old := node("n2", "mlx5_0")
+	old.Features, old.AgentVersion = nil, "v0.10.0"
+	_, _, _, err := Plan(node("n1", "mlx5_0"), old, "")
+	if err == nil || !strings.Contains(err.Error(), "n2's agent (v0.10.0) is too old") {
+		t.Errorf("got %v", err)
+	}
+}

internal/control/fleetmetrics/fleetmetrics.go

@@ -39,9 +39,25 @@ type Source interface {
 // derived from state that already exists, so computing them when asked is both
 // simpler and fresher than maintaining a parallel copy that can drift. Nothing
 // runs between scrapes.
+// HealthSource adds what the reconciler knows about each model and where
+// engine installs stand, and whether this control plane is the one placing
+// work. Asked for by type assertion, so a source without it — a test's —
+// publishes what it did before.
+type HealthSource interface {
+	ModelHealth() []reconciler.ModelHealth
+	IsLeader() bool
+}
+
 type Collector struct {
 	src Source
 
+	modelDesired     *prometheus.Desc
+	modelFailures    *prometheus.Desc
+	modelParked      *prometheus.Desc
+	modelUnplaceable *prometheus.Desc
+	engineInstall    *prometheus.Desc
+	leader           *prometheus.Desc
+
 	nodeUp       *prometheus.Desc
 	nodeGPUs     *prometheus.Desc
 	nodeGPUsFree *prometheus.Desc
@@ -137,7 +153,26 @@ func New(src Source) *Collector {
 		replicaState: prometheus.NewDesc("llmgw_replicas",
 			"Replicas by model and state.", []string{"model", "state"}, nil),
 		replicaReady: prometheus.NewDesc("llmgw_model_replicas_ready",
-			"Replicas of a model that are serving.", []string{"model"}, nil),
+			"Replicas of a model that are serving. Published, as 0 when none are, for every "+
+				"model that should be running; a stopped model, or an on-demand one nobody is "+
+				"asking for, publishes nothing.", []string{"model"}, nil),
+		modelDesired: prometheus.NewDesc("llmgw_model_replicas_desired",
+			"Replicas a model should have now: its minimum, or for an on-demand model 1 while "+
+				"it is being asked for.", []string{"model"}, nil),
+		modelFailures: prometheus.NewDesc("llmgw_model_start_failures",
+			"Replicas of a model that failed to start in a row. Resets when one comes up.",
+			[]string{"model"}, nil),
+		modelParked: prometheus.NewDesc("llmgw_model_parked",
+			"1 when a model failed so often in a row that it is no longer retried.",
+			[]string{"model"}, nil),
+		modelUnplaceable: prometheus.NewDesc("llmgw_model_unplaceable",
+			"1 when a model is short of replicas and the last attempt found no node to place one on.",
+			[]string{"model"}, nil),
+		engineInstall: prometheus.NewDesc("llmgw_node_engine_install",
+			"Where installing an engine stands on a node: 1 for the current state.",
+			[]string{"node", "engine", "version", "state"}, nil),
+		leader: prometheus.NewDesc("llmgw_control_plane_leader",
+			"1 when this control plane is the one placing work, 0 when it is a standby.", nil, nil),
 		replicaIdle: prometheus.NewDesc("llmgw_replica_idle_seconds",
 			"Seconds since a replica was last sent a request. Absent when nothing has been reported, "+
 				"which is not the same as idle.", []string{"model", "replica"}, nil),
@@ -201,6 +236,8 @@ func (c *Collector) Describe(ch chan<- *prometheus.Desc) {
 		c.gpuMemoryTotal, c.gpuMemoryUsed, c.gpuUtil, c.gpuTemp, c.gpuHealthy,
 		c.gpuNVLink, c.gpuECC,
 		c.replicaState, c.replicaReady, c.replicaIdle,
+		c.modelDesired, c.modelFailures, c.modelParked, c.modelUnplaceable,
+		c.engineInstall, c.leader,
 	} {
 		ch <- d
 	}
@@ -387,8 +424,43 @@ func (c *Collector) Collect(ch chan<- prometheus.Metric) {
 		ch <- prometheus.MustNewConstMetric(c.replicaState, prometheus.GaugeValue,
 			float64(v), k[0], k[1])
 	}
-	for model, v := range ready {
-		ch <- prometheus.MustNewConstMetric(c.replicaReady, prometheus.GaugeValue,
-			float64(v), model)
+	hs, ok := c.src.(HealthSource)
+	if !ok {
+		for model, v := range ready {
+			ch <- prometheus.MustNewConstMetric(c.replicaReady, prometheus.GaugeValue,
+				float64(v), model)
+		}
+		return
+	}
+	for _, n := range c.src.Nodes() {
+		for _, p := range n.Provision {
+			ch <- prometheus.MustNewConstMetric(c.engineInstall, prometheus.GaugeValue, 1,
+				n.ID, p.Engine, p.Version, p.State)
+		}
+	}
+	lead := 0.0
+	if hs.IsLeader() {
+		lead = 1
+	}
+	ch <- prometheus.MustNewConstMetric(c.leader, prometheus.GaugeValue, lead)
+	for _, h := range hs.ModelHealth() {
+		// Ready is published at zero for what should be running — the one
+		// value ModelHasNothingServing exists to catch, and which, emitted
+		// only for replicas that were Ready, it never saw.
+		if h.Desired > 0 || ready[h.Model] > 0 {
+			ch <- prometheus.MustNewConstMetric(c.replicaReady, prometheus.GaugeValue,
+				float64(ready[h.Model]), h.Model)
+		}
+		ch <- prometheus.MustNewConstMetric(c.modelDesired, prometheus.GaugeValue, float64(h.Desired), h.Model)
+		ch <- prometheus.MustNewConstMetric(c.modelFailures, prometheus.GaugeValue, float64(h.Failures), h.Model)
+		ch <- prometheus.MustNewConstMetric(c.modelParked, prometheus.GaugeValue, b2f(h.Parked), h.Model)
+		ch <- prometheus.MustNewConstMetric(c.modelUnplaceable, prometheus.GaugeValue, b2f(h.Unplaceable), h.Model)
 	}
 }
+
+func b2f(b bool) float64 {
+	if b {
+		return 1
+	}
+	return 0
+}

internal/control/fleetmetrics/fleetmetrics_test.go

@@ -196,3 +196,35 @@ func TestEveryIBPortIsReportedWithItsCounters(t *testing.T) {
 		}
 	}
 }
+
+type healthSource struct {
+	fakeSource
+	health []reconciler.ModelHealth
+}
+
+func (h healthSource) ModelHealth() []reconciler.ModelHealth { return h.health }
+func (h healthSource) IsLeader() bool                        { return true }
+
+func TestAModelThatShouldRunPublishesZeroReplicasReady(t *testing.T) {
+	// ModelHasNothingServing alerts on llmgw_model_replicas_ready == 0, and
+	// the series was only ever published while a replica was ready — so the
+	// alert could not fire for the case it was written for.
+	src := healthSource{health: []reconciler.ModelHealth{
+		{Model: "wanted", Desired: 1, Failures: 5, Parked: true, Reason: "boom"},
+		{Model: "stopped", Desired: 0},
+	}}
+	out := gather(t, src)
+	for _, want := range []string{
+		"llmgw_model_replicas_ready model=wanted -> 0",
+		"llmgw_model_parked model=wanted -> 1",
+		"llmgw_model_start_failures model=wanted -> 5",
+		"llmgw_control_plane_leader -> 1",
+	} {
+		if !strings.Contains(out, want) {
+			t.Errorf("missing %s", want)
+		}
+	}
+	if strings.Contains(out, "llmgw_model_replicas_ready model=stopped") {
+		t.Error("a stopped model publishes a ready count, so the alert would fire for it")
+	}
+}

internal/control/inventory/inventory.go

@@ -100,7 +100,10 @@ type Node struct {
 	LastSeen time.Time `json:"last_seen"`
 	// AgentVersion and EngineVersions are reported at registration. The engine
 	// version decides which metric names and CLI flags apply (plan §14).
-	AgentVersion   string            `json:"agent_version,omitempty"`
+	AgentVersion string `json:"agent_version,omitempty"`
+	// Features are the requests the agent answers beyond the ones every
+	// agent does.
+	Features       []string          `json:"features,omitempty"`
 	EngineVersions map[string]string `json:"engine_versions,omitempty"`
 	// EnginesProbed says EngineVersions is an answer, so an empty map means
 	// "has none" rather than "an agent too old to say".

internal/control/reconciler/reconciler.go

@@ -3,10 +3,15 @@ package reconciler
 import (
 	"context"
 	crypterand "crypto/rand"
+	"crypto/sha256"
+	"encoding/hex"
+	"encoding/json"
+	"errors"
 	"fmt"
 	"log/slog"
 	"math/rand/v2"
 	"slices"
+	"strings"
 	"sync"
 	"time"
 
@@ -65,10 +70,24 @@ type Config struct {
 	// Without a bound, draining a node whose model cannot be placed anywhere
 	// else would never finish.
 	DrainGrace time.Duration
+	// ParkAfter is how many failures in a row stop a model being retried.
+	//
+	// Backoff alone never gives up: it grows to MaxBackoff and keeps trying,
+	// and a model whose weights its engine cannot load — an architecture the
+	// installed llama.cpp does not know — was started 155 times over eight
+	// hours on the live cluster, each time failing the same way, with nothing
+	// to show for it but the log. A parked model is not placed again until its
+	// definition changes or an operator runs `llmgw ctl retry`.
+	ParkAfter int
 	// PortRange is the per-node port range for engine HTTP and dist-init.
 	PortRange [2]int
 	// MgmtInterface is the ethernet interface used for NCCL bootstrap.
 	MgmtInterface string
+	// EngineKey gives each replica the key its engine demands. Derived from
+	// the replica id rather than stored, so a control plane that restarts —
+	// or a second one — computes the same key for a replica it adopts. Nil
+	// starts engines without one.
+	EngineKey func(replicaID string) string
 	// EvictionGrace is how long a replica must have gone unasked-for before it
 	// may be stopped to make room for another model.
 	//
@@ -120,6 +139,7 @@ func DefaultConfig() Config {
 		EvictionGrace:    2 * time.Minute,
 		MinLifetime:      5 * time.Minute,
 		DrainGrace:       5 * time.Minute,
+		ParkAfter:        5,
 		PortRange:        [2]int{20000, 20999},
 		MgmtInterface:    "eth0",
 	}
@@ -159,6 +179,9 @@ func (c *Config) normalize() {
 	if c.DrainGrace <= 0 {
 		c.DrainGrace = d.DrainGrace
 	}
+	if c.ParkAfter <= 0 {
+		c.ParkAfter = d.ParkAfter
+	}
 	if c.PortRange[1] <= c.PortRange[0] {
 		c.PortRange = d.PortRange
 	}
@@ -199,6 +222,15 @@ type Reconciler struct {
 	// because it is state about this moment and not about what the model is.
 	// A model nobody has wanted has no entry and is simply not running.
 	wanted map[string]time.Time
+	// restartQueue holds, per model, the replicas an operator asked to be
+	// restarted and not yet begun. One is replaced at a time: replacing them
+	// all at once would need the memory of every replica twice over.
+	restartQueue map[string][]string
+	// parked holds models that failed ParkAfter times in a row.
+	parked map[string]parkRecord
+	// placementLogged remembers the last "no placement" said per model, so it
+	// is said when it changes rather than on every five-second pass.
+	placementLogged map[string]placementNote
 
 	wake chan struct{}
 	stop chan struct{}
@@ -220,18 +252,21 @@ func New(cfg Config, cat *catalog.Live, inv *inventory.Store, disp Dispatcher, l
 	}
 	return &Reconciler{
 		cfg: cfg, cat: cat, inv: inv, disp: disp, log: log,
-		sched:         placement.New(inv),
-		now:           time.Now,
-		replicas:      map[string]*Replica{},
-		nextAttempt:   map[string]time.Time{},
-		attempts:      map[string]int{},
-		reservedPorts: map[int]struct{}{},
-		wanted:        map[string]time.Time{},
-		lastExplain:   map[string]placement.Result{},
-		wake:          make(chan struct{}, 1),
-		stop:          make(chan struct{}),
-		done:          make(chan struct{}),
-		startedAt:     time.Now(),
+		sched:           placement.New(inv),
+		now:             time.Now,
+		replicas:        map[string]*Replica{},
+		nextAttempt:     map[string]time.Time{},
+		attempts:        map[string]int{},
+		reservedPorts:   map[int]struct{}{},
+		wanted:          map[string]time.Time{},
+		restartQueue:    map[string][]string{},
+		parked:          map[string]parkRecord{},
+		placementLogged: map[string]placementNote{},
+		lastExplain:     map[string]placement.Result{},
+		wake:            make(chan struct{}, 1),
+		stop:            make(chan struct{}),
+		done:            make(chan struct{}),
+		startedAt:       time.Now(),
 	}
 }
 
@@ -453,6 +488,12 @@ func (r *Reconciler) Want(model string) bool {
 	}
 	r.mu.Lock()
 	r.wanted[m.Name] = r.now()
+	// A placement that failed before this request says nothing about now:
+	// forgotten, so the pass this triggers decides afresh and Unservable
+	// does not answer from a stale result.
+	if res, ok := r.lastExplain[m.Name]; ok && res.Placement == nil {
+		delete(r.lastExplain, m.Name)
+	}
 	r.mu.Unlock()
 	// Immediately, not on the next tick. Someone is waiting on this request,
 	// and five seconds of ticker is five seconds they spend staring at nothing.
@@ -658,6 +699,17 @@ func (r *Reconciler) reapFailed(ctx context.Context) {
 			// replica is about to disappear, and the backoff has to survive it.
 			r.attempts[rep.Model]++
 			r.nextAttempt[rep.Model] = r.now().Add(r.backoff(r.attempts[rep.Model]))
+			if n := r.attempts[rep.Model]; n >= r.cfg.ParkAfter {
+				if _, already := r.parked[rep.Model]; !already {
+					m, _ := r.cat.Load().Resolve(rep.Model)
+					r.parked[rep.Model] = parkRecord{
+						Since: r.now(), Reason: rep.Err, Failures: n, fp: modelFingerprint(m),
+					}
+					r.log.Error("model parked: it failed every time it started, and is not retried "+
+						"until its definition changes or `llmgw ctl retry` is run",
+						"model", rep.Model, "failures", n, "last_error", rep.Err)
+				}
+			}
 		}
 	}
 	r.mu.Unlock()
@@ -703,24 +755,54 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
 	// pointer stops being safe the moment it is in the map: an agent's status
 	// report can mutate it from another goroutine at any time.
 	r.mu.Lock()
-	var liveIDs, drainingIDs, avoid []string
+	var liveIDs, drainingIDs, retiringIDs, avoid []string
+	readyLive := 0
 	for _, rep := range r.replicas {
 		if rep.Model != m.Name {
 			continue
 		}
 		switch {
+		case rep.State.Counts() && !rep.RetiringAt.IsZero():
+			retiringIDs = append(retiringIDs, rep.ID)
+			avoid = append(avoid, rep.NodeIDs()...)
 		case rep.State.Counts():
 			liveIDs = append(liveIDs, rep.ID)
 			avoid = append(avoid, rep.NodeIDs()...)
+			if rep.State == Ready {
+				readyLive++
+			}
 		case rep.State == Draining:
 			drainingIDs = append(drainingIDs, rep.ID)
 		}
 	}
+	// The next restart an operator queued, once the last one is done.
+	if len(retiringIDs) == 0 {
+		if id, ok := r.nextRestartLocked(m.Name); ok {
+			r.replicas[id].RetiringAt = r.now()
+			retiringIDs = append(retiringIDs, id)
+			liveIDs = slices.DeleteFunc(liveIDs, func(x string) bool { return x == id })
+			readyLive--
+			r.log.Info("restarting a replica: placing its replacement first",
+				"replica", id, "model", m.Name)
+		}
+	}
 	blockedUntil := r.nextAttempt[m.Name]
+	park, isParked := r.parked[m.Name]
+	if isParked && park.fp != modelFingerprint(m) {
+		// Somebody changed the model — a new source, an edited profile — so
+		// the reason it kept failing may be gone. Worth one more streak.
+		delete(r.parked, m.Name)
+		r.attempts[m.Name] = 0
+		delete(r.nextAttempt, m.Name)
+		isParked, blockedUntil = false, time.Time{}
+		r.log.Info("model unparked: its definition changed", "model", m.Name)
+	}
 	r.mu.Unlock()
 
 	desired := r.desiredFor(m)
 	switch {
+	case len(liveIDs) < desired && isParked:
+		// Not placed. Its failure is on every status the operator reads.
 	case len(liveIDs) < desired:
 		if r.now().Before(blockedUntil) {
 			// Still in backoff. Rebuilding immediately would hot-loop on a
@@ -758,6 +840,112 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
 			r.stopReplica(ctx, id)
 		}
 	}
+
+	r.retire(ctx, retiringIDs, readyLive, len(liveIDs), desired)
+}
+
+// retire finishes restarting replicas.
+//
+// A retiring replica keeps serving until its replacement is ready, and is then
+// drained like any other. When no replacement could be placed at all —
+// nowhere else has the memory, which on a small CPU fleet is the usual case —
+// it is stopped after DrainGrace so its replacement can take its place: a
+// restart in place, with the model unserved while it loads if it had no other
+// replica. Said in the log, since that is the one case with an outage.
+func (r *Reconciler) retire(ctx context.Context, ids []string, readyLive, live, desired int) {
+	for _, id := range ids {
+		r.mu.Lock()
+		rep, ok := r.replicas[id]
+		var since time.Time
+		if ok {
+			since = rep.RetiringAt
+		}
+		r.mu.Unlock()
+		if !ok {
+			continue
+		}
+		switch {
+		case readyLive >= desired:
+			r.drainForRestart(ctx, id)
+		case live < desired && r.now().Sub(since) > r.cfg.DrainGrace:
+			r.log.Warn("restarting a replica in place: no room for its replacement beside it",
+				"replica", id, "after", r.cfg.DrainGrace)
+			r.drainForRestart(ctx, id)
+			r.stopReplica(ctx, id)
+		}
+	}
+}
+
+func (r *Reconciler) drainForRestart(ctx context.Context, id string) {
+	r.mu.Lock()
+	rep, ok := r.replicas[id]
+	var shards []Shard
+	if ok {
+		rep.State, rep.DrainingAt, rep.RetiringAt = Draining, r.now(), time.Time{}
+		shards = slices.Clone(rep.Shards)
+	}
+	r.mu.Unlock()
+	for _, s := range shards {
+		if err := r.disp.DrainShard(ctx, s.NodeID, s.ID); err != nil {
+			r.log.Warn("draining a shard failed", "shard", s.ID, "err", err)
+		}
+	}
+}
+
+// nextRestartLocked pops the next queued replica of a model that is still
+// ready to be restarted. Called with mu held.
+func (r *Reconciler) nextRestartLocked(model string) (string, bool) {
+	q := r.restartQueue[model]
+	for len(q) > 0 {
+		id := q[0]
+		q = q[1:]
+		if rep, ok := r.replicas[id]; ok && rep.State == Ready && rep.RetiringAt.IsZero() {
+			r.restartQueue[model] = q
+			return id, true
+		}
+	}
+	delete(r.restartQueue, model)
+	return "", false
+}
+
+// ErrNoSuchReplica means no ready replica has that id.
+var ErrNoSuchReplica = errors.New("no ready replica with that id")
+
+// Restart queues replicas to be replaced, each by a new one started beside it
+// before it is stopped — to pick up an edited profile, an engine key, a new
+// engine version. Replicas of one model are replaced one at a time.
+func (r *Reconciler) Restart(ids ...string) error {
+	r.mu.Lock()
+	for _, id := range ids {
+		rep, ok := r.replicas[id]
+		if !ok || rep.State != Ready {
+			r.mu.Unlock()
+			return fmt.Errorf("%w: %s", ErrNoSuchReplica, id)
+		}
+		if !slices.Contains(r.restartQueue[rep.Model], id) && rep.RetiringAt.IsZero() {
+			r.restartQueue[rep.Model] = append(r.restartQueue[rep.Model], id)
+		}
+	}
+	r.mu.Unlock()
+	r.Notify()
+	return nil
+}
+
+// RestartState says where a replica's restart stands: "queued", "replacing",
+// or "" when none was asked for.
+func (r *Reconciler) RestartState(id string) string {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	rep, ok := r.replicas[id]
+	switch {
+	case !ok:
+		return ""
+	case !rep.RetiringAt.IsZero():
+		return "replacing"
+	case slices.Contains(r.restartQueue[rep.Model], id):
+		return "queued"
+	}
+	return ""
 }
 
 // scaleUp places and starts one replica per pass.
@@ -790,7 +978,7 @@ func (r *Reconciler) scaleUp(ctx context.Context, m catalog.Model, avoid []strin
 		if m.Scaling.OnDemand() && r.makeRoomFor(ctx, m) {
 			return
 		}
-		r.log.Warn("no placement for model", "model", m.Name, "reasons", res.Explain())
+		r.notePlacementFailure(m.Name, res.Explain())
 		return
 	}
 	if err := r.startReplica(ctx, m, res.Placement); err != nil {
@@ -883,6 +1071,7 @@ func (r *Reconciler) startReplica(ctx context.Context, m catalog.Model, pl *plac
 				MgmtInterface: r.cfg.MgmtInterface,
 				HTTPPort:      s.HTTPPort,
 				Profile:       espec,
+				APIKey:        r.engineKey(id),
 			},
 			StartupTimeout: prof.StartupTimeout,
 			Pool:           pl.Pool,
@@ -1380,3 +1569,149 @@ func (r *Reconciler) profileMemory(model, profile string) int {
 	}
 	return 0
 }
+
+func (r *Reconciler) engineKey(replicaID string) string {
+	if r.cfg.EngineKey == nil {
+		return ""
+	}
+	return r.cfg.EngineKey(replicaID)
+}
+
+// parkRecord is why a model stopped being retried.
+type parkRecord struct {
+	Since    time.Time
+	Reason   string
+	Failures int
+	// fp is the model's definition when it was parked; a different one
+	// unparks it.
+	fp string
+}
+
+type placementNote struct {
+	text string
+	at   time.Time
+}
+
+// modelFingerprint identifies a model's definition, so a change to it can be
+// told from a retry of the same thing.
+func modelFingerprint(m catalog.Model) string {
+	b, _ := json.Marshal(m)
+	sum := sha256.Sum256(b)
+	return hex.EncodeToString(sum[:8])
+}
+
+// placementLogEvery is how often an unchanged placement failure is repeated.
+const placementLogEvery = 10 * time.Minute
+
+// notePlacementFailure logs why a model could not be placed, when the reason
+// is new or has not been said for a while. Said on every pass it was 250
+// identical lines in two days for one model, burying everything else.
+func (r *Reconciler) notePlacementFailure(model, why string) {
+	r.mu.Lock()
+	last := r.placementLogged[model]
+	say := last.text != why || r.now().Sub(last.at) >= placementLogEvery
+	if say {
+		r.placementLogged[model] = placementNote{text: why, at: r.now()}
+	}
+	r.mu.Unlock()
+	if say {
+		r.log.Warn("no placement for model", "model", model, "reasons", why)
+	}
+}
+
+// Retry clears a parked model and its backoff, so it is placed on the next
+// pass.
+func (r *Reconciler) Retry(model string) bool {
+	r.mu.Lock()
+	_, was := r.parked[model]
+	delete(r.parked, model)
+	r.attempts[model] = 0
+	delete(r.nextAttempt, model)
+	r.mu.Unlock()
+	r.Notify()
+	return was
+}
+
+// ModelHealth is how one model stands, for status and metrics.
+type ModelHealth struct {
+	Model string
+	// Desired is how many replicas it should have now; 0 for a stopped model
+	// or an on-demand one nobody is asking for.
+	Desired int
+	Ready   int
+	// Failures counts replicas that failed in a row.
+	Failures int
+	// Parked is set when Failures reached ParkAfter; Reason is the last error.
+	Parked      bool
+	ParkedSince time.Time
+	Reason      string
+	// Unplaceable is set when it is short of replicas and the last attempt to
+	// place one found nowhere; Placement says why.
+	Unplaceable bool
+	Placement   string
+}
+
+// ModelHealth reports every model in the catalog.
+func (r *Reconciler) ModelHealth() []ModelHealth {
+	models := r.cat.Load().Models()
+	desired := map[string]int{}
+	for _, m := range models {
+		desired[m.Name] = r.desiredFor(m)
+	}
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	ready, live := map[string]int{}, map[string]int{}
+	for _, rep := range r.replicas {
+		if rep.State == Ready {
+			ready[rep.Model]++
+		}
+		if rep.State.Counts() {
+			live[rep.Model]++
+		}
+	}
+	out := make([]ModelHealth, 0, len(models))
+	for _, m := range models {
+		h := ModelHealth{Model: m.Name, Desired: desired[m.Name], Ready: ready[m.Name],
+			Failures: r.attempts[m.Name]}
+		if p, ok := r.parked[m.Name]; ok {
+			h.Parked, h.ParkedSince, h.Reason = true, p.Since, p.Reason
+		}
+		if res, ok := r.lastExplain[m.Name]; ok && res.Placement == nil && live[m.Name] < h.Desired {
+			h.Unplaceable, h.Placement = true, res.Explain()
+		}
+		out = append(out, h)
+	}
+	return out
+}
+
+// Unservable says why a model cannot be served at all, so a request for it is
+// told now rather than asked to retry something that will never come up.
+func (r *Reconciler) Unservable(model string) (string, bool) {
+	m, ok := r.cat.Load().Resolve(model)
+	if !ok {
+		return "", false
+	}
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	if p, ok := r.parked[m.Name]; ok {
+		return fmt.Sprintf("%s failed to start %d times in a row and is not being retried: %s",
+			m.Name, p.Failures, p.Reason), true
+	}
+	for _, rep := range r.replicas {
+		if rep.Model == m.Name && rep.State.Counts() {
+			return "", false // something is on its way
+		}
+	}
+	if res, ok := r.lastExplain[m.Name]; ok && res.Placement == nil {
+		return m.Name + " cannot be placed on any node: " + firstLine(res.Explain()), true
+	}
+	return "", false
+}
+
+func firstLine(s string) string {
+	s = strings.TrimSpace(s)
+	if i := strings.IndexByte(s, '\n'); i >= 0 {
+		return strings.TrimSpace(s[:i]) + " (and others; see llmgw ctl explain-placement)"
+	}
+	return s
+}

internal/control/reconciler/reconciler_test.go

@@ -1,6 +1,7 @@
 package reconciler
 
 import (
+	"bytes"
 	"context"
 	"fmt"
 	"log/slog"
@@ -1721,3 +1722,213 @@ func TestAnAgentTooOldToReportTheProfileVersionIsNotGuessedAt(t *testing.T) {
 		t.Errorf("an unreported fingerprint should stay empty, got %q", fp)
 	}
 }
+
+func TestAnEngineIsStartedWithItsReplicasKey(t *testing.T) {
+	// Engines listen on every address a node has. Started without a key, any
+	// machine that could reach the port reached the model past the gateway's
+	// authentication, quotas and accounting — checked from a neighbouring node
+	// on the live cluster.
+	h := newHarness(t, testModel(1), gpuNode("1", 8))
+	h.r.cfg.EngineKey = func(id string) string { return "key-for-" + id }
+
+	h.r.Reconcile(t.Context())
+	started := h.disp.startedShards()
+	if len(started) == 0 {
+		t.Fatal("nothing was started")
+	}
+	for _, c := range started {
+		if c.Spec.APIKey != "key-for-"+c.ReplicaID {
+			t.Errorf("shard %s started with key %q", c.ShardID, c.Spec.APIKey)
+		}
+	}
+}
+
+func replicaByID(h *harness, id string) (Replica, bool) {
+	for _, rep := range h.r.Replicas() {
+		if rep.ID == id {
+			return rep, true
+		}
+	}
+	return Replica{}, false
+}
+
+func TestARestartedReplicaServesUntilItsReplacementIsReady(t *testing.T) {
+	// Restarting is how a running engine picks up a key, an edited profile or
+	// a new engine version. Stopping first would leave a one-replica model
+	// unserved for as long as its replacement takes to load.
+	h := newHarness(t, testModel(1), gpuNode("1", 4), gpuNode("2", 4))
+	h.r.Reconcile(t.Context())
+	h.markReady()
+	old := h.r.ReadyReplicas()[0].ID
+
+	if err := h.r.Restart(old); err != nil {
+		t.Fatal(err)
+	}
+	h.r.Reconcile(t.Context())
+	if rep, _ := replicaByID(h, old); rep.State != Ready || h.r.RestartState(old) != "replacing" {
+		t.Fatalf("the old replica stopped serving before its replacement was up: %+v", rep)
+	}
+	if n := len(h.r.Replicas()); n != 2 {
+		t.Fatalf("no replacement was placed beside it: %d replicas", n)
+	}
+
+	h.markReady()
+	h.r.Reconcile(t.Context())
+	h.r.Reconcile(t.Context())
+	if _, still := replicaByID(h, old); still {
+		if rep, _ := replicaByID(h, old); rep.State != Draining && rep.State != Stopped {
+			t.Errorf("the old replica is %s after its replacement became ready", rep.State)
+		}
+	}
+	if got := len(h.r.ReadyReplicas()); got != 1 {
+		t.Errorf("%d ready replicas at the end; the model asks for 1", got)
+	}
+}
+
+func TestARestartWithNoRoomBesideItHappensInPlace(t *testing.T) {
+	// One node, room for one replica: the replacement can only start once the
+	// old one has gone. Waiting forever would mean never restarting at all.
+	h := newHarness(t, testModel(1), gpuNode("1", 4))
+	h.r.Reconcile(t.Context())
+	h.markReady()
+	old := h.r.ReadyReplicas()[0].ID
+	if err := h.r.Restart(old); err != nil {
+		t.Fatal(err)
+	}
+	h.r.Reconcile(t.Context())
+	if rep, _ := replicaByID(h, old); rep.State != Ready {
+		t.Fatalf("stopped before the grace period: %s", rep.State)
+	}
+
+	h.advance(h.r.cfg.DrainGrace + time.Second)
+	h.r.Reconcile(t.Context())
+	if rep, ok := replicaByID(h, old); ok && rep.State == Ready {
+		t.Fatal("still serving after the grace period with no replacement possible")
+	}
+	for i := 0; i < 3; i++ {
+		h.r.Reconcile(t.Context())
+		h.markReady()
+	}
+	ready := h.r.ReadyReplicas()
+	if len(ready) != 1 || ready[0].ID == old {
+		t.Errorf("expected one new replica in its place, got %+v", ready)
+	}
+}
+
+func TestReplicasOfAModelAreRestartedOneAtATime(t *testing.T) {
+	h := newHarness(t, testModel(2), gpuNode("1", 8), gpuNode("2", 8), gpuNode("3", 8))
+	for i := 0; i < 3; i++ {
+		h.r.Reconcile(t.Context())
+		h.markReady()
+	}
+	var ids []string
+	for _, rep := range h.r.ReadyReplicas() {
+		ids = append(ids, rep.ID)
+	}
+	if err := h.r.Restart(ids...); err != nil {
+		t.Fatal(err)
+	}
+	h.r.Reconcile(t.Context())
+	states := []string{h.r.RestartState(ids[0]), h.r.RestartState(ids[1])}
+	if !slices.Contains(states, "replacing") || !slices.Contains(states, "queued") {
+		t.Errorf("expected one replacing and one queued, got %v", states)
+	}
+	if err := h.r.Restart("no-such"); err == nil {
+		t.Error("an unknown replica was accepted")
+	}
+}
+
+// failRepeatedly starts and fails the model's replica n times over.
+func failRepeatedly(t *testing.T, h *harness, n int, why string) {
+	t.Helper()
+	for i := 0; i < n; i++ {
+		h.disp.reset()
+		h.r.Reconcile(t.Context())
+		started := h.disp.startedShards()
+		if len(started) == 0 {
+			return // parked, or still in backoff
+		}
+		s := started[0]
+		h.r.OnShardStatus(s.ReplicaID, s.ShardID, supervisor.Failed, why)
+		h.r.Reconcile(t.Context())
+		h.advance(DefaultConfig().MaxBackoff + time.Second)
+	}
+}
+
+func TestAModelThatAlwaysFailsIsParkedNotRetriedForever(t *testing.T) {
+	// kev-0.5b: an architecture the node's llama.cpp did not know, started
+	// 155 times in eight hours, the same error every time.
+	h := newHarness(t, testModel(1), gpuNode("1", 8))
+	failRepeatedly(t, h, DefaultConfig().ParkAfter, "unknown model architecture: 'ggmlc'")
+
+	h.disp.reset()
+	h.advance(DefaultConfig().MaxBackoff + time.Second)
+	h.r.Reconcile(t.Context())
+	if n := len(h.disp.startedShards()); n != 0 {
+		t.Fatalf("a parked model was started again (%d shards)", n)
+	}
+	var mh ModelHealth
+	for _, x := range h.r.ModelHealth() {
+		if x.Model == "llama" {
+			mh = x
+		}
+	}
+	if !mh.Parked || !strings.Contains(mh.Reason, "ggmlc") || mh.Failures < DefaultConfig().ParkAfter {
+		t.Errorf("health %+v", mh)
+	}
+	if why, no := h.r.Unservable("llama"); !no || !strings.Contains(why, "ggmlc") {
+		t.Errorf("a request would not be told why: %q %v", why, no)
+	}
+
+	// Retried by the operator once fixed.
+	if !h.r.Retry("llama") {
+		t.Error("Retry did not report the model as parked")
+	}
+	h.r.Reconcile(t.Context())
+	if len(h.disp.startedShards()) == 0 {
+		t.Error("a retried model was not placed")
+	}
+}
+
+func TestChangingAParkedModelsDefinitionTriesItAgain(t *testing.T) {
+	h := newHarness(t, testModel(1), gpuNode("1", 8))
+	failRepeatedly(t, h, DefaultConfig().ParkAfter, "boom")
+
+	fixed := testModel(1)
+	fixed.Source = "/models/llama-fixed"
+	cat, err := catalog.New([]catalog.Model{fixed})
+	if err != nil {
+		t.Fatal(err)
+	}
+	h.r.cat.Store(cat)
+	h.disp.reset()
+	h.r.Reconcile(t.Context())
+	if len(h.disp.startedShards()) == 0 {
+		t.Error("a model whose definition changed stayed parked")
+	}
+}
+
+func TestAPlacementFailureIsLoggedWhenItChangesNotEveryPass(t *testing.T) {
+	var buf bytes.Buffer
+	h := newHarness(t, testModel(1), gpuNode("1", 1)) // tp4 needs 4 cards
+	h.r.log = slog.New(slog.NewTextHandler(&buf, nil))
+	for i := 0; i < 20; i++ {
+		h.r.Reconcile(t.Context())
+		h.advance(5 * time.Second)
+	}
+	if n := strings.Count(buf.String(), "no placement for model"); n != 1 {
+		t.Errorf("logged %d times over 20 passes", n)
+	}
+	h.advance(placementLogEvery)
+	h.r.Reconcile(t.Context())
+	if n := strings.Count(buf.String(), "no placement for model"); n != 2 {
+		t.Errorf("not repeated after %s: %d", placementLogEvery, n)
+	}
+	var mh ModelHealth
+	for _, x := range h.r.ModelHealth() {
+		mh = x
+	}
+	if !mh.Unplaceable || mh.Desired != 1 {
+		t.Errorf("health %+v", mh)
+	}
+}

internal/control/reconciler/replica.go

@@ -156,6 +156,10 @@ type Replica struct {
 	// DrainingAt is when the replica stopped taking new work, so the wait for a
 	// replacement can be bounded.
 	DrainingAt time.Time
+	// RetiringAt is set while the replica is being restarted: it keeps serving
+	// but no longer counts toward the model's replicas, so a replacement is
+	// placed beside it, and it is drained only once that replacement is ready.
+	RetiringAt time.Time
 	// Attempt counts how many times this model has failed to come up in a row,
 	// carried across rebuilds so the backoff does not reset on every attempt.
 	Attempt int

internal/gguf/gguf.go

@@ -25,6 +25,59 @@ type Header struct {
 	Version      uint32
 	Architecture string
 	Name         string
+
+	// The shape the KV cache is sized from, from the architecture's own keys
+	// (llama.block_count, qwen2.attention.head_count_kv, ...). Zero when the
+	// file does not say, which a caller has to treat as unknown.
+	BlockCount      uint64
+	HeadCount       uint64
+	HeadCountKV     uint64
+	EmbeddingLength uint64
+	// KeyLength and ValueLength are the per-head sizes when the file gives
+	// them; otherwise a head is EmbeddingLength / HeadCount.
+	KeyLength   uint64
+	ValueLength uint64
+}
+
+// KVBytesPerToken is what one token of context costs in llama.cpp's default
+// f16 cache, or 0 when the header did not give the shape.
+//
+// Keys and values, per layer, per KV head — the KV heads rather than the
+// attention heads, because with grouped-query attention a model with forty
+// heads may keep only eight, and the difference is a factor of five.
+func (h Header) KVBytesPerToken() int64 {
+	kvHeads := h.HeadCountKV
+	if kvHeads == 0 {
+		kvHeads = h.HeadCount
+	}
+	if h.BlockCount == 0 || kvHeads == 0 {
+		return 0
+	}
+	k, v := h.KeyLength, h.ValueLength
+	if k == 0 && h.HeadCount > 0 {
+		k = h.EmbeddingLength / h.HeadCount
+	}
+	if v == 0 {
+		v = k
+	}
+	if k == 0 {
+		return 0
+	}
+	const f16 = 2
+	return int64(h.BlockCount * kvHeads * (k + v) * f16) //nolint:gosec // header-sized numbers
+}
+
+// MemoryBytes is what one llama.cpp server needs for this model: the weights,
+// a KV cache for a context pool of ctxPool tokens (llama.cpp's --ctx-size,
+// which --parallel divides among its slots rather than multiplies), and room
+// for compute buffers and the server itself. 0 when the shape is unknown.
+func (h Header) MemoryBytes(weights int64, ctxPool int) int64 {
+	perToken := h.KVBytesPerToken()
+	if perToken == 0 || weights <= 0 {
+		return 0
+	}
+	overhead := int64(1<<30) + weights/20
+	return weights + int64(ctxPool)*perToken + overhead
 }
 
 // ErrNotGGUF means the bytes are not a GGUF file.
@@ -87,16 +140,42 @@ func Read(r io.Reader) (Header, error) {
 		_, err := io.CopyN(io.Discard, br, n)
 		return err
 	}
+	// num holds the last integer value read, for the shape keys.
+	var num uint64
+	readUint := func(size int) error {
+		b := make([]byte, size)
+		if _, err := io.ReadFull(br, b); err != nil {
+			return err
+		}
+		switch size {
+		case 1:
+			num = uint64(b[0])
+		case 2:
+			num = uint64(binary.LittleEndian.Uint16(b))
+		case 4:
+			num = uint64(binary.LittleEndian.Uint32(b))
+		default:
+			num = binary.LittleEndian.Uint64(b)
+		}
+		return nil
+	}
 	var value func(t uint32) (string, error)
 	value = func(t uint32) (string, error) {
+		num = 0
 		switch t {
-		case tUint8, tInt8, tBool:
+		case tUint8, tInt8:
+			return "", readUint(1)
+		case tBool:
 			return "", skip(1)
 		case tUint16, tInt16:
-			return "", skip(2)
-		case tUint32, tInt32, tFloat32:
+			return "", readUint(2)
+		case tUint32, tInt32:
+			return "", readUint(4)
+		case tFloat32:
 			return "", skip(4)
-		case tUint64, tInt64, tFloat64:
+		case tUint64, tInt64:
+			return "", readUint(8)
+		case tFloat64:
 			return "", skip(8)
 		case tString:
 			return str()
@@ -129,18 +208,27 @@ func Read(r io.Reader) (Header, error) {
 	if err != nil {
 		return h, err
 	}
+	// Once the architecture is known, running out of the fetched bytes is
+	// not a failure: what the caller came for is in hand, and the shape keys
+	// are a bonus that a file with its tokenizer first may not reach.
+	partial := func(err error) (Header, error) {
+		if h.Architecture != "" {
+			return h, nil
+		}
+		return h, err
+	}
 	for i := uint64(0); i < kvs; i++ {
 		key, err := str()
 		if err != nil {
-			return h, err
+			return partial(err)
 		}
 		var t uint32
 		if err := binary.Read(br, binary.LittleEndian, &t); err != nil {
-			return h, err
+			return partial(err)
 		}
 		v, err := value(t)
 		if err != nil {
-			return h, err
+			return partial(err)
 		}
 		switch key {
 		case "general.architecture":
@@ -148,7 +236,25 @@ func Read(r io.Reader) (Header, error) {
 		case "general.name":
 			h.Name = v
 		}
-		if h.Architecture != "" && h.Name != "" {
+		if h.Architecture != "" {
+			switch strings.TrimPrefix(key, h.Architecture+".") {
+			case "block_count":
+				h.BlockCount = num
+			case "attention.head_count":
+				h.HeadCount = num
+			case "attention.head_count_kv":
+				h.HeadCountKV = num
+			case "embedding_length":
+				h.EmbeddingLength = num
+			case "attention.key_length":
+				h.KeyLength = num
+			case "attention.value_length":
+				h.ValueLength = num
+			}
+		}
+		// The tokenizer's arrays — megabytes of vocabulary — come after the
+		// architecture's keys, and nothing past them is wanted.
+		if strings.HasPrefix(key, "tokenizer.") && h.BlockCount > 0 {
 			break
 		}
 	}
@@ -209,5 +315,7 @@ func Check(ctx context.Context, url, llamaVersion string) Verdict {
 	if err != nil {
 		return Verdict{Servable: true, Reason: "its header could not be read: " + err.Error()}
 	}
-	return LlamaCpp(llamaVersion, h.Architecture)
+	v := LlamaCpp(llamaVersion, h.Architecture)
+	v.Header = h
+	return v
 }

internal/gguf/gguf_test.go

@@ -92,3 +92,54 @@ func TestLlamaCppKnowsWhatItCanLoad(t *testing.T) {
 }
 
 var timeZero = time.Time{}
+
+// shapedHeader is a header with the keys a KV cache is sized from, written
+// with the integer types llama.cpp uses for them.
+func shapedHeader(arch string, layers, heads, kvHeads, embd uint32, truncateTokenizer bool) []byte {
+	var b bytes.Buffer
+	w := func(v any) { _ = binary.Write(&b, binary.LittleEndian, v) }
+	str := func(s string) { w(uint64(len(s))); b.WriteString(s) }
+	u32 := func(k string, v uint32) { str(k); w(uint32(tUint32)); w(v) }
+	b.WriteString("GGUF")
+	w(uint32(3))
+	w(uint64(0))
+	w(uint64(7))
+	str("general.architecture")
+	w(uint32(tString))
+	str(arch)
+	u32(arch+".block_count", layers)
+	u32(arch+".attention.head_count", heads)
+	u32(arch+".attention.head_count_kv", kvHeads)
+	u32(arch+".embedding_length", embd)
+	u32(arch+".context_length", 32768)
+	str("tokenizer.ggml.tokens")
+	w(uint32(tArray))
+	w(uint32(tString))
+	w(uint64(1 << 20))
+	if !truncateTokenizer {
+		for i := 0; i < 1<<20; i++ {
+			str("t")
+		}
+	}
+	return b.Bytes()
+}
+
+func TestTheKVCacheIsSizedFromTheModelsShape(t *testing.T) {
+	// Qwen2.5-14B: 48 layers, 40 heads of which 8 are kept for K and V,
+	// 5120 wide — 196608 bytes a token in f16, as read from the live mirror.
+	h, err := Read(bytes.NewReader(shapedHeader("qwen2", 48, 40, 8, 5120, true)))
+	if err != nil {
+		t.Fatalf("a header cut off in its tokenizer failed to read: %v", err)
+	}
+	if got := h.KVBytesPerToken(); got != 196608 {
+		t.Errorf("%d bytes a token, expected 196608", got)
+	}
+	weights := int64(9 << 30)
+	want := weights + 16384*196608 + (1<<30 + weights/20)
+	if got := h.MemoryBytes(weights, 16384); got != want {
+		t.Errorf("memory %d, expected %d", got, want)
+	}
+	if (Header{}).MemoryBytes(weights, 16384) != 0 {
+		t.Error("a header with no shape produced a figure")
+	}
+}

internal/gguf/llamacpp.go

@@ -180,6 +180,9 @@ type Verdict struct {
 	// checked.
 	Checked bool
 	Reason  string
+	// Header is what was read, zero when nothing was; its shape sizes the
+	// model's memory.
+	Header Header
 }
 
 // LlamaCpp judges an architecture against a llama.cpp build.

internal/gguf/remote_test.go

@@ -0,0 +1,30 @@
+package gguf
+
+import (
+	"os"
+	"strings"
+	"testing"
+)
+
+// TestRealHeaders reads real files when LLMGW_GGUF_URLS names them (space
+// separated), and prints the shape and memory estimate for a 16384-token pool.
+// Skipped otherwise: CI has no mirror to read.
+func TestRealHeaders(t *testing.T) {
+	urls := strings.Fields(os.Getenv("LLMGW_GGUF_URLS"))
+	if len(urls) == 0 {
+		t.Skip("LLMGW_GGUF_URLS not set")
+	}
+	for _, u := range urls {
+		h, err := FromURL(t.Context(), nil, u)
+		if err != nil {
+			t.Errorf("%s: %v", u, err)
+			continue
+		}
+		t.Logf("%s: arch=%s layers=%d heads=%d kv_heads=%d embd=%d key_len=%d kv/token=%d B, kv@16k=%.2f GiB",
+			u[strings.LastIndex(u, "/")+1:], h.Architecture, h.BlockCount, h.HeadCount, h.HeadCountKV,
+			h.EmbeddingLength, h.KeyLength, h.KVBytesPerToken(), float64(16384*h.KVBytesPerToken())/(1<<30))
+		if h.KVBytesPerToken() == 0 {
+			t.Errorf("%s: no KV shape read", u)
+		}
+	}
+}

internal/mirror/add.go

@@ -7,7 +7,6 @@ import (
 	"net/http"
 	"os"
 	"path/filepath"
-	"strings"
 	"time"
 )
 
@@ -29,13 +28,22 @@ type Added struct {
 // the layout — a version that can be overwritten takes away the ability to
 // change the catalog back and still have the old weights.
 func Add(ctx context.Context, root, source string, e Entry, progress func(done, total int64)) (Added, error) {
+	return addWith(ctx, http.DefaultClient, root, source, e, progress)
+}
+
+// addWith is Add through a given client. The management API passes one that
+// will not reach private addresses; `llmgw mirror add`, run by somebody on
+// the mirror, passes the ordinary one.
+func addWith(ctx context.Context, client *http.Client, root, source string, e Entry,
+	progress func(done, total int64),
+) (Added, error) {
 	if e.Model == "" || e.Version == "" || e.File == "" {
 		return Added{}, fmt.Errorf("a model, a version and a filename are all required")
 	}
-	if strings.ContainsAny(e.Model+e.Version+e.File, `/\`) {
-		// Each is one path element. A slash would put the file somewhere the
-		// layout does not describe, and List would not find it again.
-		return Added{}, fmt.Errorf("model, version and filename are single names, not paths")
+	for _, n := range []string{e.Model, e.Version, e.File} {
+		if err := ValidName(n); err != nil {
+			return Added{}, err
+		}
 	}
 
 	final := e.Path(root)
@@ -50,7 +58,7 @@ func Add(ctx context.Context, root, source string, e Entry, progress func(done,
 	if err != nil {
 		return Added{}, err
 	}
-	resp, err := http.DefaultClient.Do(req)
+	resp, err := client.Do(req)
 	if err != nil {
 		return Added{}, fmt.Errorf("fetching %s: %w", source, err)
 	}

internal/mirror/api.go

@@ -130,8 +130,8 @@ func (a *API) status(w http.ResponseWriter, r *http.Request) {
 
 func (a *API) remove(w http.ResponseWriter, r *http.Request) {
 	model, version := r.PathValue("model"), r.PathValue("version")
-	if strings.ContainsAny(model+version, `/\`) {
-		fail(w, http.StatusBadRequest, errors.New("model and version are single names, not paths"))
+	if err := errors.Join(ValidName(model), ValidName(version)); err != nil {
+		fail(w, http.StatusBadRequest, err)
 		return
 	}
 

internal/mirror/api_test.go

@@ -17,7 +17,10 @@ import (
 func testAPI(t *testing.T) (*API, string) {
 	t.Helper()
 	root := t.TempDir()
-	a := NewAPI(NewManager(root, discardLog()), "http://mirror:8081")
+	m := NewManager(root, discardLog())
+	// The upstreams in these tests are httptest servers on loopback.
+	m.AllowPrivateSources()
+	a := NewAPI(m, "http://mirror:8081")
 
 	// A download outlives the request that started it — that is the point of
 	// the manager. So the temp directory must not be torn down while one is
@@ -434,3 +437,33 @@ func TestRemovingAVersionIsRefusedWhileAnyShardIsArriving(t *testing.T) {
 		t.Errorf("expected 409 while a shard is arriving, got %d", code)
 	}
 }
+
+func TestTheAPIWillNotFetchFromThePrivateNetwork(t *testing.T) {
+	// The file server hands whatever was fetched to anyone, so a fetch that
+	// could reach 169.254.169.254 or a database on the LAN was a way to read
+	// them through the mirror.
+	m := NewManager(t.TempDir(), discardLog())
+	for _, src := range []string{
+		"http://127.0.0.1:1/x.gguf", "http://169.254.169.254/latest/meta-data",
+		"http://10.0.0.140:9090/api/v1/nodes", "http://[::1]:1/x",
+	} {
+		st, err := m.Start(src, Entry{Model: "m", Version: "v", File: "x.gguf"})
+		if err != nil {
+			t.Fatalf("%s: %v", src, err)
+		}
+		deadline := time.Now().Add(5 * time.Second)
+		for st.State == Fetching && time.Now().Before(deadline) {
+			time.Sleep(10 * time.Millisecond)
+			st, _ = m.Status("m", "v")
+		}
+		if st.State != Failed || !strings.Contains(st.Err, "refusing to fetch") {
+			t.Errorf("%s: %+v", src, st)
+		}
+		m.Forget("m", "v")
+	}
+	for _, src := range []string{"file:///etc/passwd", "gopher://x/", "not a url"} {
+		if _, err := m.Start(src, Entry{Model: "m", Version: "v", File: "x"}); err == nil {
+			t.Errorf("%s was accepted as a source", src)
+		}
+	}
+}

internal/mirror/manager.go

@@ -5,6 +5,7 @@ import (
 	"errors"
 	"fmt"
 	"log/slog"
+	"net/http"
 	"os"
 	"sync"
 	"time"
@@ -72,8 +73,18 @@ type Manager struct {
 	// until the first had finished. A fifteen-shard model had to be pulled one
 	// shard at a time, each waiting on the last.
 	running map[string]*Status
+
+	// client fetches sources. It will not reach private addresses unless
+	// AllowPrivateSources was called.
+	client *http.Client
 }
 
+// AllowPrivateSources lets downloads reach loopback, private and link-local
+// addresses — a model server on the same network, say. Off by default: the
+// file server hands whatever was fetched to anyone, so a fetch that can reach
+// the private network is a way to read it.
+func (m *Manager) AllowPrivateSources() { m.client = fetchClient(true) }
+
 // runKey identifies one download.
 func runKey(e Entry) string { return e.Model + "/" + e.Version + "/" + e.File }
 
@@ -82,7 +93,7 @@ func NewManager(root string, log *slog.Logger) *Manager {
 	if log == nil {
 		log = slog.Default()
 	}
-	return &Manager{root: root, log: log, running: map[string]*Status{}}
+	return &Manager{root: root, log: log, running: map[string]*Status{}, client: fetchClient(false)}
 }
 
 // ErrAlreadyFetching means a download for that model and version is running.
@@ -97,6 +108,9 @@ func (m *Manager) Start(source string, e Entry) (Status, error) {
 	if source == "" {
 		return Status{}, fmt.Errorf("a source URL is required")
 	}
+	if err := validSource(source); err != nil {
+		return Status{}, err
+	}
 	key := runKey(e)
 
 	m.mu.Lock()
@@ -125,7 +139,7 @@ func (m *Manager) run(source string, e Entry, key string) {
 	// minutes to hours and the thing that asked for it is long gone.
 	ctx := context.Background()
 
-	added, err := Add(ctx, m.root, source, e, func(done, total int64) {
+	added, err := addWith(ctx, m.client, m.root, source, e, func(done, total int64) {
 		m.mu.Lock()
 		if st, ok := m.running[key]; ok {
 			st.BytesDone, st.BytesTotal = done, total

internal/mirror/mirror.go

@@ -37,6 +37,21 @@ type Entry struct {
 	Bytes   int64
 }
 
+// ValidName checks one element of an entry — a model, version or file name.
+//
+// Each is one path element, and a name that is not one writes somewhere the
+// layout does not describe. Rejecting slashes was not enough: ".." is a
+// single element too, and a model called ".." put its file beside the mirror
+// root rather than inside it. A name starting with a dot is refused as well,
+// because that is where the mirror keeps its own bookkeeping.
+func ValidName(n string) error {
+	if n == "" || n[0] == '.' || strings.ContainsAny(n, "/\\\x00") {
+		return fmt.Errorf("%q is not a usable name: it must be a single path element "+
+			"that does not start with a dot", n)
+	}
+	return nil
+}
+
 // Path is where this entry lives under a root.
 func (e Entry) Path(root string) string {
 	return filepath.Join(root, e.Model, e.Version, e.File)

internal/mirror/mirror_test.go

@@ -162,3 +162,18 @@ func TestProgressIsReportedForALongDownload(t *testing.T) {
 		t.Error("a download reported no progress at all")
 	}
 }
+
+func TestANameThatLeavesTheMirrorIsRefused(t *testing.T) {
+	root := t.TempDir()
+	for _, e := range []Entry{
+		{Model: "..", Version: "v", File: "f.gguf"},
+		{Model: "m", Version: "..", File: "f.gguf"},
+		{Model: "m", Version: "v", File: ".f.sha256"},
+		{Model: "a/b", Version: "v", File: "f.gguf"},
+	} {
+		if _, err := Add(t.Context(), root, "http://127.0.0.1:1/x", e, nil); err == nil ||
+			!strings.Contains(err.Error(), "not a usable name") {
+			t.Errorf("%+v: %v", e, err)
+		}
+	}
+}

internal/mirror/safefetch.go

@@ -0,0 +1,86 @@
+package mirror
+
+import (
+	"errors"
+	"fmt"
+	"net"
+	"net/http"
+	"net/netip"
+	"net/url"
+	"syscall"
+	"time"
+)
+
+// errPrivateSource is a fetch refused because it would reach this machine or
+// the network behind it rather than the internet.
+var errPrivateSource = errors.New("refusing to fetch from a loopback, private or link-local address")
+
+// fetchClient downloads sources for the management API.
+//
+// The API fetches whatever URL it is given and the file server then hands the
+// result to anyone. Without this, "fetch http://169.254.169.254/..." was a way
+// to read a cloud's instance metadata — or anything else on the network the
+// mirror sits in — through the mirror. The check is made on the address
+// actually dialled, after resolution and on every redirect, so a name that
+// resolves somewhere public and then somewhere private is caught too.
+func fetchClient(allowPrivate bool) *http.Client {
+	d := &net.Dialer{Timeout: 30 * time.Second, KeepAlive: 30 * time.Second}
+	if !allowPrivate {
+		d.Control = func(_, address string, _ syscall.RawConn) error {
+			host, _, err := net.SplitHostPort(address)
+			if err != nil {
+				return err
+			}
+			a, err := netip.ParseAddr(host)
+			if err != nil {
+				return err
+			}
+			if blockedAddr(a.Unmap()) {
+				return fmt.Errorf("%w (%s)", errPrivateSource, a)
+			}
+			return nil
+		}
+	}
+	return &http.Client{
+		Transport: &http.Transport{
+			Proxy:                 http.ProxyFromEnvironment,
+			DialContext:           d.DialContext,
+			TLSHandshakeTimeout:   30 * time.Second,
+			ResponseHeaderTimeout: 2 * time.Minute,
+			ForceAttemptHTTP2:     true,
+		},
+		// No overall timeout: a model is tens of gigabytes. The dial, the
+		// handshake and the first byte are bounded above; after that a stall
+		// is the context's to end.
+		CheckRedirect: func(req *http.Request, via []*http.Request) error {
+			if len(via) >= 10 {
+				return errors.New("too many redirects")
+			}
+			return checkScheme(req.URL)
+		},
+	}
+}
+
+func blockedAddr(a netip.Addr) bool {
+	cgnat := netip.MustParsePrefix("100.64.0.0/10")
+	return a.IsLoopback() || a.IsPrivate() || a.IsLinkLocalUnicast() ||
+		a.IsLinkLocalMulticast() || a.IsMulticast() || a.IsUnspecified() || cgnat.Contains(a)
+}
+
+// checkScheme allows the two schemes a model is published over. file:// and
+// the rest would read this machine's own disk.
+func checkScheme(u *url.URL) error {
+	if u.Scheme != "http" && u.Scheme != "https" {
+		return fmt.Errorf("%q is not an http or https source", u.Redacted())
+	}
+	return nil
+}
+
+// validSource checks a source before a download is started for it.
+func validSource(source string) error {
+	u, err := url.Parse(source)
+	if err != nil || u.Host == "" {
+		return fmt.Errorf("%q is not a URL", source)
+	}
+	return checkScheme(u)
+}

internal/onboard/onboard.go

@@ -35,6 +35,10 @@ import (
 type Fabric struct {
 	Name string
 	HCAs []string
+	// Key is what the adapters report the fabric to be — its subnet manager,
+	// or a RoCE network — on a suggestion, so a name given for one node can
+	// be given again to the next on the same fabric.
+	Key string
 }
 
 // ParseFabric reads "name:hca1,hca2", or "name" alone.
@@ -328,7 +332,10 @@ func certNames(path string) ([]string, error) {
 
 // --- binary ---
 
-var releaseTag = regexp.MustCompile(`^v\d+\.\d+\.\d+$`)
+// releaseTag is a version CI published: v0.10.4, or a release candidate of
+// it, v0.10.4-rc.2. Anything else — a build with commits past its tag — has no
+// release to download.
+var releaseTag = regexp.MustCompile(`^v\d+\.\d+\.\d+(-rc\.\d+)?$`)
 
 func (r *Runner) binary(ctx context.Context, res *Result) error {
 	v := r.Opts.Version
@@ -462,7 +469,7 @@ func suggestFabrics(hcas []discovery.HCA) []Fabric {
 		if len(keys) > 1 {
 			name = fmt.Sprintf("fabric%d", i+1)
 		}
-		out = append(out, Fabric{Name: name, HCAs: byKey[k]})
+		out = append(out, Fabric{Name: name, HCAs: byKey[k], Key: k})
 	}
 	return out
 }

internal/onboard/releasetag_test.go

@@ -0,0 +1,14 @@
+package onboard
+
+import "testing"
+
+func TestAReleaseCandidateIsARelease(t *testing.T) {
+	for v, want := range map[string]bool{
+		"v0.10.4": true, "v0.10.4-rc.1": true, "v0.10.4-rc.12": true,
+		"v0.10.3-5-gad6d2d3": false, "v0.10.4-rc": false, "v0.10.3-a": false, "dev": false,
+	} {
+		if got := releaseTag.MatchString(v); got != want {
+			t.Errorf("%s: %v", v, got)
+		}
+	}
+}

internal/router/activate.go

@@ -17,6 +17,22 @@ type Activator interface {
 	Want(model string) bool
 }
 
+// unservable is an activator that can also say a model will not come up at
+// all — it failed every time it started, or fits on no node — so a request is
+// told so at once rather than asked to retry something that never arrives.
+type unservable interface {
+	Unservable(model string) (string, bool)
+}
+
+// whyUnservable is the activator's reason a model cannot be served, if any.
+func (r *Router) whyUnservable(model string) (string, bool) {
+	u, ok := r.activator.(unservable)
+	if !ok {
+		return "", false
+	}
+	return u.Unservable(model)
+}
+
 // SetActivator supplies the control plane that can load models on demand.
 func (r *Router) SetActivator(a Activator) { r.activator = a }
 
@@ -32,6 +48,8 @@ func (r *Router) SetActivator(a Activator) { r.activator = a }
 // a slow request. A fifteen-minute wait is a job pretending to be one, and the
 // client should be told to come back rather than left holding a socket.
 func (r *Router) activate(ctx context.Context, model string) bool {
+	// Asked first, always: asking is what makes the control plane try again,
+	// and a model that fitted nowhere an hour ago may fit now.
 	if r.activator == nil || !r.activator.Want(model) {
 		return false
 	}
@@ -56,6 +74,11 @@ func (r *Router) activate(ctx context.Context, model string) bool {
 			if len(r.reg.Snapshots(model)) > 0 {
 				return true
 			}
+			// Asking set a placement going; if it found nowhere, there is
+			// nothing to wait for.
+			if _, no := r.whyUnservable(model); no {
+				return false
+			}
 		}
 	}
 }

internal/router/adminguard.go

@@ -0,0 +1,75 @@
+package router
+
+import (
+	"net"
+	"net/http"
+	"net/netip"
+	"net/url"
+	"strings"
+)
+
+// AdminGuard keeps a browser from being used against the admin listener.
+//
+// The listener has no login: reaching it is meant to mean holding the machine,
+// so it binds to loopback and an operator reaches it over an ssh tunnel. But a
+// browser on that operator's machine reaches it too, from any page it has
+// open. A page could POST a model whose profile runs as root on every node,
+// or — by pointing its own domain at 127.0.0.1 — read pprof's heap, which
+// holds API keys in the clear. Three checks close that without touching what
+// ctl and curl do:
+//
+//   - Host must be an address or localhost. A page that rebinds its own name
+//     to 127.0.0.1 still sends that name.
+//   - A request carrying Origin or Sec-Fetch-Site from another site is
+//     refused. Browsers send them; ctl and curl do not.
+//   - Anything but GET and HEAD must be application/json. A page can send a
+//     form or text/plain anywhere without asking; JSON across origins needs a
+//     preflight this listener never grants.
+func AdminGuard(next http.Handler) http.Handler {
+	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		if !addressHost(r.Host) {
+			http.Error(w, "the admin listener answers to an address or localhost, not a name", http.StatusMisdirectedRequest)
+			return
+		}
+		if crossSite(r) {
+			http.Error(w, "cross-site request refused", http.StatusForbidden)
+			return
+		}
+		if r.Method != http.MethodGet && r.Method != http.MethodHead {
+			ct := r.Header.Get("Content-Type")
+			if mt, _, _ := strings.Cut(ct, ";"); strings.TrimSpace(strings.ToLower(mt)) != "application/json" {
+				http.Error(w, "a change to the admin API must be sent as application/json", http.StatusUnsupportedMediaType)
+				return
+			}
+		}
+		next.ServeHTTP(w, r)
+	})
+}
+
+func addressHost(hostport string) bool {
+	host := hostport
+	if h, _, err := net.SplitHostPort(hostport); err == nil {
+		host = h
+	}
+	host = strings.Trim(host, "[]")
+	if strings.EqualFold(host, "localhost") {
+		return true
+	}
+	_, err := netip.ParseAddr(host)
+	return err == nil
+}
+
+func crossSite(r *http.Request) bool {
+	switch r.Header.Get("Sec-Fetch-Site") {
+	case "", "same-origin", "none":
+	default:
+		return true
+	}
+	if o := r.Header.Get("Origin"); o != "" {
+		u, err := url.Parse(o)
+		if err != nil || !strings.EqualFold(u.Host, r.Host) {
+			return true
+		}
+	}
+	return false
+}

internal/router/adminguard_test.go

@@ -0,0 +1,57 @@
+package router
+
+import (
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"testing"
+)
+
+func TestTheAdminListenerRefusesWhatABrowserPageCouldSend(t *testing.T) {
+	h := AdminGuard(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {}))
+	do := func(method, host, ct string, hdr map[string]string) int {
+		r := httptest.NewRequest(method, "/api/v1/models", strings.NewReader("{}"))
+		r.Host = host
+		if ct != "" {
+			r.Header.Set("Content-Type", ct)
+		}
+		for k, v := range hdr {
+			r.Header.Set(k, v)
+		}
+		w := httptest.NewRecorder()
+		h.ServeHTTP(w, r)
+		return w.Code
+	}
+
+	// What ctl and curl send.
+	if got := do("POST", "127.0.0.1:9090", "application/json", nil); got != 200 {
+		t.Errorf("ctl's request was refused: %d", got)
+	}
+	if got := do("GET", "localhost:9090", "", nil); got != 200 {
+		t.Errorf("a GET through an ssh tunnel was refused: %d", got)
+	}
+	if got := do("DELETE", "[::1]:9090", "application/json", nil); got != 200 {
+		t.Errorf("IPv6 loopback refused: %d", got)
+	}
+
+	// A page posting a form, or text/plain, needs no preflight.
+	if got := do("POST", "127.0.0.1:9090", "text/plain", nil); got != http.StatusUnsupportedMediaType {
+		t.Errorf("text/plain POST accepted: %d", got)
+	}
+	if got := do("POST", "127.0.0.1:9090", "", nil); got != http.StatusUnsupportedMediaType {
+		t.Errorf("a bodiless POST accepted: %d", got)
+	}
+	// DNS rebinding: the page's own name, now resolving to 127.0.0.1.
+	if got := do("GET", "evil.example:9090", "", nil); got != http.StatusMisdirectedRequest {
+		t.Errorf("a rebound name was answered: %d", got)
+	}
+	// And anything a browser marks as coming from elsewhere.
+	if got := do("POST", "127.0.0.1:9090", "application/json",
+		map[string]string{"Sec-Fetch-Site": "cross-site"}); got != http.StatusForbidden {
+		t.Errorf("cross-site fetch accepted: %d", got)
+	}
+	if got := do("POST", "127.0.0.1:9090", "application/json",
+		map[string]string{"Origin": "https://evil.example"}); got != http.StatusForbidden {
+		t.Errorf("foreign Origin accepted: %d", got)
+	}
+}

internal/router/admission/admission_test.go

@@ -42,14 +42,20 @@ func newController(t *testing.T, cfg Config, sat SaturationFunc) *Controller {
 
 func TestAdmitsImmediatelyWhenIdle(t *testing.T) {
 	s := &fixedSat{}
-	c := newController(t, testConfig(), s.fn)
+	// A queue that is looked at every ten seconds, so a request that went
+	// through it rather than the fast path would take that long. A bound of
+	// 50ms against a 1ms tick measured the machine the test ran on, and a
+	// loaded CI runner under the race detector was slower than that.
+	cfg := testConfig()
+	cfg.Interval = 10 * time.Second
+	c := newController(t, cfg, s.fn)
 
 	start := time.Now()
 	lease, err := c.Acquire(t.Context(), Request{Model: "m", Band: "bronze"})
 	if err != nil {
 		t.Fatal(err)
 	}
-	if d := time.Since(start); d > 50*time.Millisecond {
+	if d := time.Since(start); d > 2*time.Second {
 		t.Errorf("idle admission took %v; the fast path is not being used", d)
 	}
 	lease.Release()
@@ -131,8 +137,13 @@ func TestQueuedRequestAdmittedWhenSaturationDrops(t *testing.T) {
 		done <- err
 	}()
 
-	// Give it time to actually queue, then open the gate.
-	time.Sleep(50 * time.Millisecond)
+	// Wait for it to actually queue, then open the gate. Waited for, not
+	// slept through: on a loaded runner 50ms was sometimes not long enough
+	// for the goroutine to have been scheduled at all.
+	deadline := time.Now().Add(2 * time.Second)
+	for c.Depths()[3].Depth != 1 && time.Now().Before(deadline) {
+		time.Sleep(5 * time.Millisecond)
+	}
 	if d := c.Depths(); d[3].Depth != 1 {
 		t.Fatalf("expected 1 queued bronze request, got %+v", d)
 	}

internal/router/endpoint/endpoint.go

@@ -22,6 +22,9 @@ type Info struct {
 	ID string
 	// URL is the engine's root address (e.g. http://gpu-a-01:8000), no trailing slash.
 	URL string
+	// APIKey is what the engine demands of every request, sent as a bearer
+	// token. Empty for an engine that asks for none.
+	APIKey string
 	// Models lists the model names this replica serves.
 	Models []string
 	// Engine names the inference engine serving here, which decides what its

internal/router/handlers.go

@@ -3,6 +3,8 @@ package router
 import (
 	"bytes"
 	"context"
+	"crypto/rand"
+	"encoding/hex"
 	"encoding/json"
 	"errors"
 	"fmt"
@@ -51,7 +53,7 @@ func (r *Router) rejectAdmission(w http.ResponseWriter, req *parsedRequest, err
 	case errors.Is(err, context.Canceled), errors.Is(err, context.DeadlineExceeded):
 		// The client gave up while queued; nobody is listening.
 		r.m.observeAdmissionRejected(req.Band, "client_gone")
-		r.m.observeRequest(req.Model, "", 499, time.Since(start), nil, 0)
+		r.m.observeRequest(r.metricModel(req.Model), "", 499, time.Since(start), nil, 0)
 		return
 	case errors.Is(err, admission.ErrQueueTimeout):
 		status = http.StatusGatewayTimeout
@@ -67,7 +69,8 @@ func (r *Router) rejectAdmission(w http.ResponseWriter, req *parsedRequest, err
 	}
 	r.m.observeAdmissionRejected(req.Band, reason)
 	writeError(w, status, "server_error", err.Error())
-	r.m.observeRequest(req.Model, "", status, time.Since(start), nil, 0)
+	r.m.observeRequest(r.metricModel(req.Model), "", status, time.Since(start), nil, 0)
+	r.accessLog(req, dispatchResult{err: err}, status, time.Since(start))
 }
 
 // prefixBytes is how much of the prompt is kept for prefix-aware routing.
@@ -224,6 +227,12 @@ type parsedRequest struct {
 	UpstreamBody       []byte
 	Header             http.Header
 
+	// RequestID follows the request through the access log, the usage record
+	// and the engine, and goes back to the client in X-Request-Id.
+	RequestID string
+	// QueueMs is how long admission held it.
+	QueueMs int64
+
 	SessionID string
 	Band      string
 	// KeyID, ProjectID and Tenant are empty when authentication is off.
@@ -275,8 +284,19 @@ type inferenceBody struct {
 func (r *Router) handleInference(path string) http.HandlerFunc {
 	return func(w http.ResponseWriter, req *http.Request) {
 		start := time.Now()
+		// First, so even a request refused as malformed can be named when
+		// somebody asks what happened to it.
+		rid := requestID(req.Header.Get("X-Request-Id"))
+		w.Header().Set("X-Request-Id", rid)
 
 		parsed, err := r.parse(req, path)
+		if err == nil {
+			parsed.RequestID = rid
+			if parsed.Header == nil {
+				parsed.Header = http.Header{}
+			}
+			parsed.Header.Set("X-Request-Id", rid)
+		}
 		if err != nil {
 			writeError(w, http.StatusBadRequest, "invalid_request_error", err.Error())
 			r.m.observeRequest("", "", http.StatusBadRequest, time.Since(start), nil, 0)
@@ -289,7 +309,7 @@ func (r *Router) handleInference(path string) http.HandlerFunc {
 		if _, entry := r.resolve(parsed.Model); entry.ContextLength > 0 {
 			if why := checkContextFits(entry, parsed.PromptTokens, parsed.MaxTokens); why != "" {
 				writeError(w, http.StatusBadRequest, "invalid_request_error", why)
-				r.m.observeRequest(parsed.Model, "", http.StatusBadRequest, time.Since(start), nil, 0)
+				r.m.observeRequest(r.metricModel(parsed.Model), "", http.StatusBadRequest, time.Since(start), nil, 0)
 				return
 			}
 		}
@@ -297,7 +317,7 @@ func (r *Router) handleInference(path string) http.HandlerFunc {
 		if p, ok := auth.FromContext(req.Context()); ok {
 			if !r.permits(p, parsed.Model) {
 				auth.WriteForbiddenModel(w, parsed.Model)
-				r.m.observeRequest(parsed.Model, "", http.StatusForbidden, time.Since(start), nil, 0)
+				r.m.observeRequest(r.metricModel(parsed.Model), "", http.StatusForbidden, time.Since(start), nil, 0)
 				return
 			}
 			// Limits are checked before admission, not after: a request that is
@@ -323,6 +343,9 @@ func (r *Router) handleInference(path string) http.HandlerFunc {
 						w.Header().Set("x-llmgw-served-model", parsed.Model)
 					}
 				}
+				// On every answer, refused or not, so a client can pace itself
+				// before it is refused rather than learn the limit from a 429.
+				r.rateLimitHeaders(w, p)
 				if err != nil {
 					r.rejectLimit(w, parsed, err, start)
 					return
@@ -339,7 +362,8 @@ func (r *Router) handleInference(path string) http.HandlerFunc {
 		}
 		defer lease.Release()
 		queued := time.Since(start)
-		r.m.observeAdmission(lease.Band(), parsed.Model, queued)
+		parsed.QueueMs = queued.Milliseconds()
+		r.m.observeAdmission(lease.Band(), r.metricModel(parsed.Model), queued)
 		if queued > time.Millisecond {
 			w.Header().Set("x-llmgw-queue-ms", strconv.FormatInt(queued.Milliseconds(), 10))
 		}
@@ -349,7 +373,7 @@ func (r *Router) handleInference(path string) http.HandlerFunc {
 		// model that will actually serve, not the one that was asked for.
 		if dropped, err := r.loadThread(req.Context(), parsed); err != nil {
 			writeError(w, http.StatusBadRequest, "invalid_request_error", err.Error())
-			r.m.observeRequest(parsed.Model, "", http.StatusBadRequest, time.Since(start), nil, 0)
+			r.m.observeRequest(r.metricModel(parsed.Model), "", http.StatusBadRequest, time.Since(start), nil, 0)
 			return
 		} else if dropped > 0 {
 			// Dropping turns is the right answer and must not be a silent one.
@@ -420,8 +444,9 @@ func (r *Router) finish(w http.ResponseWriter, req *parsedRequest, res dispatchR
 		writeError(w, status, "server_error", res.err.Error())
 	}
 
-	r.m.observeRequest(req.Model, res.pool, status, time.Since(start), res.outcome.Usage, res.outcome.TTFT)
+	r.m.observeRequest(r.metricModel(req.Model), res.pool, status, time.Since(start), res.outcome.Usage, res.outcome.TTFT)
 	r.account(req, res, status, time.Since(start))
+	r.accessLog(req, res, status, time.Since(start))
 }
 
 // account settles a finished request against its key.
@@ -437,6 +462,8 @@ func (r *Router) account(req *parsedRequest, res dispatchResult, status int, too
 		At: time.Now(), KeyID: req.KeyID, UserID: req.UserID, ProjectID: req.ProjectID,
 		Model: req.Model, ServedModel: res.model, Status: status,
 		LatencyMs: took.Milliseconds(),
+		RequestID: req.RequestID, Replica: res.endpointID,
+		QueueMs: req.QueueMs, TTFTMs: res.outcome.TTFT.Milliseconds(),
 	}
 	if u.ServedModel == "" {
 		u.ServedModel = req.Model
@@ -467,7 +494,8 @@ func (r *Router) rejectLimit(w http.ResponseWriter, req *parsedRequest, err erro
 	// already know how to surface that code. What differs is Retry-After,
 	// which is absent when waiting does not help.
 	writeError(w, http.StatusTooManyRequests, "rate_limit_exceeded", le.Error())
-	r.m.observeRequest(req.Model, "", http.StatusTooManyRequests, time.Since(start), nil, 0)
+	r.m.observeRequest(r.metricModel(req.Model), "", http.StatusTooManyRequests, time.Since(start), nil, 0)
+	r.accessLog(req, dispatchResult{err: err}, http.StatusTooManyRequests, time.Since(start))
 }
 
 // parse reads the body and extracts the routing-relevant fields.
@@ -854,3 +882,77 @@ func (r *Router) handleMe(w http.ResponseWriter, req *http.Request) {
 	}
 	writeJSON(w, http.StatusOK, out)
 }
+
+// requestID is the client's X-Request-Id when it is one worth keeping, or a
+// new one. Kept only when short and plain: it is written into logs and a
+// header, and a client-chosen string with newlines or megabytes in it is not
+// an id.
+func requestID(sent string) string {
+	if n := len(sent); n > 0 && n <= 128 && strings.IndexFunc(sent, func(c rune) bool {
+		return !(c >= 'a' && c <= 'z' || c >= 'A' && c <= 'Z' || c >= '0' && c <= '9' ||
+			c == '-' || c == '_' || c == '.' || c == ':')
+	}) < 0 {
+		return sent
+	}
+	var b [12]byte
+	_, _ = rand.Read(b[:])
+	return "req_" + hex.EncodeToString(b[:])
+}
+
+// accessLog writes one line per finished request: enough to follow a slow or
+// failed request from the id a client reports to the replica that served it,
+// and where its time went. Before this the only per-request lines were for
+// disconnects, early stream ends and upstream retries.
+func (r *Router) accessLog(req *parsedRequest, res dispatchResult, status int, took time.Duration) {
+	attrs := []any{
+		"request_id", req.RequestID, "status", status, "model", req.Model,
+		"duration_ms", took.Milliseconds(), "queue_ms", req.QueueMs, "stream", req.Stream,
+	}
+	if req.RequestedModel != "" && req.RequestedModel != req.Model {
+		attrs = append(attrs, "requested_model", req.RequestedModel)
+	}
+	if res.model != "" && res.model != req.Model {
+		attrs = append(attrs, "served_model", res.model)
+	}
+	if res.endpointID != "" {
+		attrs = append(attrs, "replica", res.endpointID)
+	}
+	if res.outcome.TTFT > 0 {
+		attrs = append(attrs, "ttft_ms", res.outcome.TTFT.Milliseconds())
+	}
+	if u := res.outcome.Usage; u != nil {
+		attrs = append(attrs, "prompt_tokens", u.PromptTokens, "completion_tokens", u.CompletionTokens)
+	}
+	if req.KeyID != "" {
+		attrs = append(attrs, "key", req.KeyID, "project", req.ProjectID)
+	}
+	if req.UserID != "" {
+		attrs = append(attrs, "user", req.UserID)
+	}
+	if res.err != nil {
+		attrs = append(attrs, "err", res.err.Error())
+	}
+	r.log.Info("request", attrs...)
+}
+
+// rateLimitHeaders sets OpenAI's x-ratelimit-* headers from the tightest
+// limit that applies. SDKs and editors read these to slow down before they
+// are refused; only Retry-After, and only on a refusal, was sent before. A
+// limit that is not set is not advertised.
+func (r *Router) rateLimitHeaders(w http.ResponseWriter, p auth.Principal) {
+	if r.acct == nil {
+		return
+	}
+	h := r.acct.Headroom(p)
+	reset := strconv.FormatFloat(h.Reset.Seconds(), 'f', 0, 64) + "s"
+	if h.RequestLimit > 0 {
+		w.Header().Set("x-ratelimit-limit-requests", strconv.Itoa(h.RequestLimit))
+		w.Header().Set("x-ratelimit-remaining-requests", strconv.Itoa(h.RequestsLeft))
+		w.Header().Set("x-ratelimit-reset-requests", reset)
+	}
+	if h.TokenLimit > 0 {
+		w.Header().Set("x-ratelimit-limit-tokens", strconv.Itoa(h.TokenLimit))
+		w.Header().Set("x-ratelimit-remaining-tokens", strconv.Itoa(h.TokensLeft))
+		w.Header().Set("x-ratelimit-reset-tokens", reset)
+	}
+}

internal/router/metrics.go

@@ -151,3 +151,17 @@ func (m *metrics) observeScores(d scheduler.Decision) {
 	}
 	m.considered.Observe(float64(d.Considered))
 }
+
+// metricModel is the model label a request is counted under: the catalog's
+// name for it, or "unknown". Taken as sent, any key holder could mint a new
+// series per request — a refused request counted its made-up model name —
+// and grow Prometheus without bound.
+func (r *Router) metricModel(name string) string {
+	if name == "" {
+		return ""
+	}
+	if m, ok := r.modelMap()[name]; ok {
+		return m.Name
+	}
+	return "unknown"
+}

internal/router/pricing_test.go

@@ -140,3 +140,51 @@ func waitForUsage(t *testing.T, rt *Router) store.KeyUsage {
 	t.Fatal("no usage was recorded")
 	return store.KeyUsage{}
 }
+
+func TestARequestCanBeFollowedByItsID(t *testing.T) {
+	// "Why was my request slow?" had no answer: the queue time went out in a
+	// header and nowhere else, and nothing tied a request to its replica.
+	e := newFakeEngine(t, "qwen2.5-14b-instruct")
+	rt, secret := pricedRouter(t, e.srv.URL)
+
+	rec := post(t, rt.Handler(),
+		map[string]any{"model": "q14", "messages": []any{
+			map[string]any{"role": "user", "content": "hello"}}},
+		map[string]string{"Authorization": "Bearer " + secret, "X-Request-Id": "client-abc.1"})
+	if rec.Code != 200 {
+		t.Fatalf("status %d: %s", rec.Code, rec.Body.String())
+	}
+	if got := rec.Header().Get("X-Request-Id"); got != "client-abc.1" {
+		t.Errorf("the client's id came back as %q", got)
+	}
+
+	deadline := time.Now().Add(5 * time.Second)
+	for {
+		u, ok, err := rt.store.UsageByRequest(t.Context(), "client-abc.1")
+		if err != nil {
+			t.Fatal(err)
+		}
+		if ok {
+			if u.Replica == "" || u.Status != 200 {
+				t.Errorf("recorded %+v; expected the replica that answered", u)
+			}
+			break
+		}
+		if time.Now().After(deadline) {
+			t.Fatal("the request's usage cannot be found by its id")
+		}
+		time.Sleep(20 * time.Millisecond)
+	}
+}
+
+func TestAnIDThatIsNotOneIsReplaced(t *testing.T) {
+	for _, bad := range []string{"", "has space", "new\nline", strings.Repeat("x", 200)} {
+		got := requestID(bad)
+		if got == bad || !strings.HasPrefix(got, "req_") {
+			t.Errorf("%q became %q", bad, got)
+		}
+	}
+	if requestID("req_1") != "req_1" {
+		t.Error("a plain id was not kept")
+	}
+}

internal/router/proxy/proxy.go

@@ -148,6 +148,8 @@ type Request struct {
 	Body []byte
 	// Header carries headers to forward (authorization is deliberately not).
 	Header http.Header
+	// APIKey is the engine's own key, sent in place of the client's.
+	APIKey string
 	// Stream says whether an SSE response is expected.
 	Stream bool
 	// SuppressUsageChunk drops the final usage-only chunk from the client while
@@ -218,6 +220,9 @@ func (p *Proxy) Do(ctx context.Context, w http.ResponseWriter, r Request) (Outco
 	}
 	req.Header = cloneForUpstream(r.Header)
 	req.Header.Set("Content-Type", "application/json")
+	if r.APIKey != "" {
+		req.Header.Set("Authorization", "Bearer "+r.APIKey)
+	}
 	req.ContentLength = int64(len(r.Body))
 
 	// A non-streaming request waits for the whole completion before a single
@@ -518,11 +523,10 @@ func copyHeaders(dst, src http.Header) {
 
 // cloneForUpstream forwards only what the engine needs. The client's
 // Authorization header is deliberately dropped: it authenticates the caller to
-// llmgw, not llmgw to the engine, and engines are reached over a trusted
-// network (plan §14).
+// llmgw, not llmgw to the engine. llmgw's own key for the engine is set by Do.
 func cloneForUpstream(h http.Header) http.Header {
 	out := make(http.Header, 4)
-	for _, k := range []string{"Accept", "User-Agent", "Traceparent", "Tracestate"} {
+	for _, k := range []string{"Accept", "User-Agent", "Traceparent", "Tracestate", "X-Request-Id"} {
 		if v := h.Get(k); v != "" {
 			out.Set(k, v)
 		}