Park a model that fails every time it starts, and say so everywhere

On the live cluster kev-0.5b — an architecture the nodes' llama.cpp did not
know — was started 155 times in eight hours, failing the same way each time,
and phi-4, which fitted on no node, logged "no placement" 250 times in two
days. Neither showed up anywhere an operator looked: the backoff never gave
up, no metric said a model was failing or unplaceable, and the one alert for
a model with nothing serving could not fire, because its series was published
only while a replica was ready.

After five failures in a row a model is parked: not placed again until its
definition changes — a new source or profile — or `llmgw ctl retry` is run.
Status, the model list and the admin API show it with its last error. The
placement warning is logged when its reason changes or every ten minutes.

A request for a parked or unplaceable model is answered at once with why,
rather than "loading; retry shortly", which a client retries forever. Each
request still clears the previous placement result and triggers a fresh
attempt, so a model that fits once room is made is not refused on a stale
answer; the router waits for that attempt, not for its full activation
window.

Metrics now cover model health (desired, ready at 0 for what should run,
consecutive start failures, parked, unplaceable), engine installs per node,
leadership, and certificate expiry read from disk at each scrape, with alerts
for each. `ctl status` counts a model against what it should have now, so an
on-demand model that was asked for and fitted nowhere no longer reads as
satisfied.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

CHANGELOG.md

@@ -20,6 +20,118 @@ is only released once its **exit criterion** is met, not when its code lands.
 Until v1.0.0 the minor version is the breaking one: configuration formats and Go
 APIs change between minors without a deprecation period.
 
+## Unreleased
+
+Changes since v0.10.3, published as release candidates (v0.10.4-rc.N) until
+there are enough for v0.10.4.
+
+### Restarting replicas
+
+`llmgw ctl restart REPLICA... | --model M | --all` replaces replicas with new
+ones. Each replacement is started beside the replica it replaces, which keeps
+serving until the new one is ready; replicas of one model go one at a time.
+Where no second copy fits anywhere, the old replica is stopped after the drain
+grace period and the new one starts in its place. This is how running engines
+pick up the keys introduced in v0.10.3, an edited profile, or a new engine
+version.
+
+### A model that keeps failing is parked, and everybody is told
+
+A model whose replicas failed at every start was retried forever: kev-0.5b,
+whose architecture the nodes' llama.cpp did not know, was started 155 times in
+eight hours. After five failures in a row a model is now parked — not placed
+again until its definition changes or `llmgw ctl retry MODEL` is run — and
+`ctl status`, `ctl models` and the admin API say so with the last error.
+
+A request for a model that will not come up — parked, or fitting on no node —
+is answered at once with the reason, instead of "the model is loading; retry
+shortly", which a client would retry forever. Each request still makes the
+control plane try placing it afresh, so a model that fits once room is made is
+not refused on an old answer. "no placement for model" is logged when its
+reason changes or every ten minutes, not on every pass: it was 250 identical
+lines in two days for one model.
+
+### Metrics and alerts for what goes wrong
+
+New series: `llmgw_model_replicas_desired`, `llmgw_model_start_failures`,
+`llmgw_model_parked`, `llmgw_model_unplaceable`, `llmgw_node_engine_install`,
+`llmgw_control_plane_leader` and `llmgw_certificate_expiry_timestamp_seconds`.
+`llmgw_model_replicas_ready` is now published at 0 for every model that should
+be running; it was published only while a replica was ready, so
+ModelHasNothingServing could never fire. New alerts: ModelParked,
+ModelFailingToStart, ModelCannotBePlaced, EngineInstallFailed,
+ControlPlaneLeadership, CertificateExpiresSoon and CertificateExpiresVerySoon.
+
+`ctl status` counts a model against the replicas it should have now rather
+than its minimum, so an on-demand model that was asked for and fitted nowhere
+no longer reads as satisfied.
+
+### Fixes
+
+- `ctl` reported a change the API had queued (202 Accepted) as an error, as
+  `ctl restart` did on its first real use.
+
+### Releases
+
+Small fixes are published as release candidates of the next version —
+v0.10.4-rc.1, rc.2 — which sort after v0.10.3 and before v0.10.4. Their release
+notes come from this section, and they are marked as pre-releases. Onboarding
+treats a release candidate as a release and downloads it.
+
+## v0.10.3 — 2026-09-30
+
+Security fixes from a review of v0.10.2. Three of them were confirmed on the
+live cluster before being fixed.
+
+### Engines demand a key only the router holds
+
+Engines listened on every address of their node with no key. A request from a
+neighbouring node reached the model directly, past the gateway's
+authentication, quotas and accounting. Each replica's engine is now started
+with a key derived from the replica id and a secret in the store. The router,
+its metrics scrape, the agent's readiness check and `ctl bench` send it.
+llama.cpp and vLLM receive it through their environment, not their command
+line. Engines started before this keep running without a key until their
+replica is next started.
+
+### The mirror's management API answers only who it was told to
+
+Without `--api-allow`, the management API answered anyone who could reach it,
+including every node. It now answers this machine only; the control plane is
+let in with `--api-allow` or mutual TLS. **Upgrading a mirror set up by hand:
+add `--api-allow <control plane address>` to its unit.** Downloads started
+through the API go to http and https only, and never to loopback, private or
+link-local addresses (`--allow-private-sources` to permit that). Model,
+version and file names can no longer be `..` or start with a dot.
+
+### The admin listener refuses what a browser page could send
+
+The admin API has no login and trusts whoever reaches it. A page open in an
+operator's browser, reaching it over an ssh tunnel, could post a model
+definition. The listener now requires Host to be an address or localhost,
+refuses cross-site requests, and takes changes only as `application/json`.
+A profile's `env` may no longer set variables that make an engine run other
+code (`LD_*`, `PATH`, `PYTHON*` start-up variables, `BASH_ENV` and similar).
+They are refused where a model is defined and dropped again before an engine
+starts.
+
+### Fixes
+
+- The agent installs only engine versions that look like one, whatever the
+  control plane sent; a version of `../x` could have installed outside
+  `/opt/llmgw/engines`.
+
+## v0.10.2 — 2026-09-29
+
+- After `server init`'s screen closes, only the administrator's password and
+  the next steps are printed, not the whole run again under the screen that
+  just showed it. The whole run is kept in `/var/lib/llmgw/init.log`, root's
+  only, and printed in full when init fails.
+- The screens' spinner and text field are llmgw's own rather than from
+  bubbles, one dependency fewer. Every llmgw command still asks the terminal
+  for its colours as it starts, which bubbletea v1 does on purpose; a terminal
+  that never answers waits for its timeout.
+
 ## v0.10.1 — 2026-09-29
 
 ### `llmgw ctl fabric-test`: two nodes, rail by rail

api/gen/llmgw/v1/agent.pb.go

@@ -2525,7 +2525,11 @@ type StartShard struct {
 	Fabric string `protobuf:"bytes,14,opt,name=fabric,proto3" json:"fabric,omitempty"`
 	// profile_fp identifies the version of the profile, so the agent can say
 	// which one a running shard was started from. The agent never interprets it.
-	ProfileFp     string `protobuf:"bytes,15,opt,name=profile_fp,json=profileFp,proto3" json:"profile_fp,omitempty"`
+	ProfileFp string `protobuf:"bytes,15,opt,name=profile_fp,json=profileFp,proto3" json:"profile_fp,omitempty"`
+	// api_key is what the engine demands of every request, derived by the
+	// control plane from the replica id. The router sends it; nothing else
+	// that can reach the node's ports has it.
+	ApiKey        string `protobuf:"bytes,16,opt,name=api_key,json=apiKey,proto3" json:"api_key,omitempty"`
 	unknownFields protoimpl.UnknownFields
 	sizeCache     protoimpl.SizeCache
 }
@@ -2665,6 +2669,13 @@ func (x *StartShard) GetProfileFp() string {
 	return ""
 }
 
+func (x *StartShard) GetApiKey() string {
+	if x != nil {
+		return x.ApiKey
+	}
+	return ""
+}
+
 type EngineProfile struct {
 	state           protoimpl.MessageState `protogen:"open.v1"`
 	Name            string                 `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"`
@@ -3693,7 +3704,7 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
 	"\n" +
 	"bytes_done\x18\x02 \x01(\x03R\tbytesDone\x12\x1f\n" +
 	"\vbytes_total\x18\x03 \x01(\x03R\n" +
-	"bytesTotal\"\xd9\x03\n" +
+	"bytesTotal\"\xf2\x03\n" +
 	"\n" +
 	"StartShard\x12\x1d\n" +
 	"\n" +
@@ -3716,7 +3727,8 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
 	"\x04pool\x18\r \x01(\tR\x04pool\x12\x16\n" +
 	"\x06fabric\x18\x0e \x01(\tR\x06fabric\x12\x1d\n" +
 	"\n" +
-	"profile_fp\x18\x0f \x01(\tR\tprofileFp\"\xfd\x02\n" +
+	"profile_fp\x18\x0f \x01(\tR\tprofileFp\x12\x17\n" +
+	"\aapi_key\x18\x10 \x01(\tR\x06apiKey\"\xfd\x02\n" +
 	"\rEngineProfile\x12\x12\n" +
 	"\x04name\x18\x01 \x01(\tR\x04name\x12\x16\n" +
 	"\x06engine\x18\x02 \x01(\tR\x06engine\x12\x1b\n" +

api/proto/llmgw/v1/agent.proto

@@ -524,6 +524,10 @@ message StartShard {
   // profile_fp identifies the version of the profile, so the agent can say
   // which one a running shard was started from. The agent never interprets it.
   string profile_fp = 15;
+  // api_key is what the engine demands of every request, derived by the
+  // control plane from the replica id. The router sends it; nothing else
+  // that can reach the node's ports has it.
+  string api_key = 16;
 }
 
 message EngineProfile {

deploy/bootstrap/onboard-node.sh

@@ -143,7 +143,7 @@ $SSH "set -e
 # that is not a release (a developer's, with commits past the tag) has no such
 # file, and is copied as before.
 TAG="$("$BINARY" --version 2>/dev/null | awk '{print $NF}')"
-if printf '%s' "$TAG" | grep -Eq '^v[0-9]+\.[0-9]+\.[0-9]+$'; then
+if printf '%s' "$TAG" | grep -Eq '^v[0-9]+\.[0-9]+\.[0-9]+(-rc\.[0-9]+)?$'; then
   say "installing llmgw $TAG from the release"
   $SSH "set -e
     arch=\$(dpkg --print-architecture)

deploy/monitoring/llmgw-rules.yml

@@ -42,10 +42,86 @@ groups:
         annotations:
           summary: "{{ $labels.model }} has no replica ready"
           description: >-
-            The model is placed and nothing is serving it. Ten minutes, because
-            a large model on CPU legitimately takes minutes to load. A model
-            that is deliberately stopped publishes no series at all, so this
-            cannot fire for one nobody asked for.
+            The model should be running and nothing is serving it. Ten minutes,
+            because a large model on CPU legitimately takes minutes to load. The
+            series is published at 0 for every model that should be running —
+            before v0.10.4 it was published only while a replica was ready, so
+            this could not fire at all. A model deliberately stopped, or an
+            on-demand one nobody is asking for, publishes nothing.
+
+      - alert: ModelParked
+        expr: llmgw_model_parked == 1
+        labels: { severity: critical }
+        annotations:
+          summary: "{{ $labels.model }} failed every time it started and is no longer retried"
+          description: >-
+            Five replicas in a row failed at start. Before this existed a model
+            whose weights its engine could not load was started 155 times over
+            eight hours. `llmgw ctl status` has the last error; once it is dealt
+            with, `llmgw ctl retry <model>`. Changing the model's definition
+            also retries it.
+
+      - alert: ModelFailingToStart
+        expr: llmgw_model_start_failures >= 2
+        for: 5m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.model }} has failed to start {{ $value }} times in a row"
+          description: >-
+            On its way to being parked. The engine's own words are in
+            `llmgw ctl logs <node> --source <shard>` for the next attempt, or in
+            the control plane's log for the last one.
+
+      - alert: ModelCannotBePlaced
+        expr: llmgw_model_unplaceable == 1
+        for: 10m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.model }} is wanted and fits on no node"
+          description: >-
+            Short of replicas, and the last attempt found nowhere with the
+            memory, cards or engine it needs. `llmgw ctl explain-placement
+            <model>` says what each node lacks. For an on-demand model, requests
+            for it are told this rather than asked to retry.
+
+      - alert: EngineInstallFailed
+        expr: llmgw_node_engine_install{state="failed"} == 1
+        for: 15m
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.engine }} {{ $labels.version }} will not install on {{ $labels.node }}"
+          description: >-
+            The node takes no model needing it until it does. The agent retries
+            on its own; `llmgw ctl nodes` and the node's logs on the cluster
+            page say why it failed.
+
+      - alert: ControlPlaneLeadership
+        expr: sum(llmgw_control_plane_leader) != 1
+        for: 2m
+        labels: { severity: critical }
+        annotations:
+          summary: "{{ $value }} control planes believe they are placing work"
+          description: >-
+            None means nothing is placed or replaced; two means both are, each
+            into GPUs the other believes free. The lease is in the store: check
+            it is reachable from every control plane.
+
+      - alert: CertificateExpiresSoon
+        expr: (llmgw_certificate_expiry_timestamp_seconds - time()) < 30 * 86400
+        labels: { severity: warning }
+        annotations:
+          summary: "{{ $labels.cert }} ({{ $labels.subject }}) expires in {{ $value | humanizeDuration }}"
+          description: >-
+            Certificates last a year and nothing renews them. When the agent
+            server's runs out every agent refuses to connect at once. Reissue
+            with `llmgw ctl cert` and restart the control plane.
+
+      - alert: CertificateExpiresVerySoon
+        expr: (llmgw_certificate_expiry_timestamp_seconds - time()) < 7 * 86400
+        labels: { severity: critical }
+        annotations:
+          summary: "{{ $labels.cert }} ({{ $labels.subject }}) expires in {{ $value | humanizeDuration }}"
+          description: See CertificateExpiresSoon; a week is not long.
 
       - alert: PoolSaturated
         expr: llmgw_pool_saturation > 1

docs/reference.md

@@ -71,6 +71,7 @@ rather than repeated.
 | `lease_ttl` | `time.Duration` | [Control](#control) |
 | `id` | `string` | [Endpoint](#endpoint) |
 | `url` | `string` | [Endpoint](#endpoint) |
+| `api_key` | `string` | [Endpoint](#endpoint) |
 | `models` | `[]string` | [Endpoint](#endpoint) |
 | `engine` | `string` | [Endpoint](#endpoint) |
 | `pool` | `string` | [Endpoint](#endpoint) |
@@ -566,6 +567,13 @@ reading the alert. Zero takes the default.
 
 `string`
 
+### `api_key`
+
+`string`
+
+APIKey is sent to the engine as a bearer token, for an engine started
+with one.
+
 ### `models`
 
 `[]string`

go.mod

@@ -6,7 +6,6 @@ require (
 	connectrpc.com/connect v1.21.0
 	github.com/NVIDIA/go-nvml v0.13.4-0
 	github.com/cespare/xxhash/v2 v2.3.0
-	github.com/charmbracelet/bubbles v1.0.0
 	github.com/charmbracelet/bubbletea v1.3.10
 	github.com/charmbracelet/lipgloss v1.1.0
 	github.com/jackc/pgx/v5 v5.11.0
@@ -28,7 +27,6 @@ require (
 )
 
 require (
-	github.com/atotto/clipboard v0.1.4 // indirect
 	github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
 	github.com/beorn7/perks v1.0.1 // indirect
 	github.com/charmbracelet/colorprofile v0.4.1 // indirect

go.sum

@@ -2,16 +2,12 @@ connectrpc.com/connect v1.21.0 h1:LhqSJt7jHf5NJBo9Jq/t/9FjcYAideif0mg+qe2jCUs=
 connectrpc.com/connect v1.21.0/go.mod h1:A2ygJrukXwWy32vkCAAHNVguZrqZ+jeZ9rGRnGR4dN4=
 github.com/NVIDIA/go-nvml v0.13.4-0 h1:o3jp9u2x1R9ShFE3v+Aesp55XOSIQFMJz/VGNUcJaNE=
 github.com/NVIDIA/go-nvml v0.13.4-0/go.mod h1:id63qwpoDWpFXwnwM6psDCSqW4BmNu6mWpr4YeQtPGo=
-github.com/atotto/clipboard v0.1.4 h1:EH0zSVneZPSuFR11BlR9YppQTVDbh5+16AmcJi4g1z4=
-github.com/atotto/clipboard v0.1.4/go.mod h1:ZY9tmq7sm5xIbd9bOK4onWV4S6X0u6GY7Vn0Yu86PYI=
 github.com/aymanbagabas/go-osc52/v2 v2.0.1 h1:HwpRHbFMcZLEVr42D4p7XBqjyuxQH5SMiErDT4WkJ2k=
 github.com/aymanbagabas/go-osc52/v2 v2.0.1/go.mod h1:uYgXzlJ7ZpABp8OJ+exZzJJhRNQ2ASbcXHWsFqH8hp8=
 github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
 github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
 github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
 github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
-github.com/charmbracelet/bubbles v1.0.0 h1:12J8/ak/uCZEMQ6KU7pcfwceyjLlWsDLAxB5fXonfvc=
-github.com/charmbracelet/bubbles v1.0.0/go.mod h1:9d/Zd5GdnauMI5ivUIVisuEm3ave1XwXtD1ckyV6r3E=
 github.com/charmbracelet/bubbletea v1.3.10 h1:otUDHWMMzQSB0Pkc87rm691KZ3SWa4KUlvF9nRvCICw=
 github.com/charmbracelet/bubbletea v1.3.10/go.mod h1:ORQfo0fk8U+po9VaNvnV95UPWA1BitP1E0N6xJPlHr4=
 github.com/charmbracelet/colorprofile v0.4.1 h1:a1lO03qTrSIRaK8c3JRxJDZOvhvIeSco3ej+ngLk1kk=

internal/agent/engines/engine.go

@@ -70,6 +70,13 @@ type ShardSpec struct {
 	HTTPPort int
 	Profile  Profile
 
+	// APIKey is what the engine demands of every request. Engines listen on
+	// all of a node's addresses; without a key anybody who can reach the port
+	// talks to the model directly, past the gateway's authentication, quotas
+	// and accounting. Given to the engine in its environment rather than on
+	// its command line, which every user on the node can read.
+	APIKey string `json:"api_key,omitempty"`
+
 	// Pool and Fabric are control plane concepts the agent cannot derive. They
 	// are carried so the agent can report them back after a restart; Fabric
 	// also says, with an "nvl:" prefix, that the replica is inside a
@@ -249,6 +256,11 @@ func baseEnv(s ShardSpec) map[string]string {
 	// Profile overrides win: they are the escape hatch for the per-cluster
 	// tuning no adapter can guess.
 	for k, v := range s.Profile.Env {
+		// Checked again here, not only where the model was defined: this is
+		// the process about to run as root, whatever sent the profile.
+		if bad, _ := UnsafeEnv(k); bad {
+			continue
+		}
 		env[k] = v
 	}
 	return env

internal/agent/engines/llamacpp.go

@@ -120,8 +120,12 @@ func (l *LlamaCpp) Commands(s ShardSpec) ([]Command, error) {
 	}
 	args = append(args, p.Args...)
 
+	env := baseEnv(s)
+	if s.APIKey != "" {
+		env["LLAMA_API_KEY"] = s.APIKey
+	}
 	return []Command{{
-		Name: "llama-server", Path: bin, Args: args, Env: materializeEnv(baseEnv(s)),
+		Name: "llama-server", Path: bin, Args: args, Env: materializeEnv(env),
 	}}, nil
 }
 

internal/agent/engines/llamacpp_test.go

@@ -245,3 +245,19 @@ func TestAMissingPathIsLeftToTheEngine(t *testing.T) {
 		t.Errorf("expected it passed through, got %q %v", got, err)
 	}
 }
+
+func TestLlamaCppIsGivenItsKeyOutOfSightOfTheProcessList(t *testing.T) {
+	// The command line is readable by every user on the node; the
+	// environment of a root process is not.
+	cmds, err := (&LlamaCpp{}).Commands(ShardSpec{HTTPPort: 8080, APIKey: "k1",
+		Profile: Profile{ModelPath: "/m.gguf"}})
+	if err != nil {
+		t.Fatal(err)
+	}
+	if slices.Contains(cmds[0].Args, "k1") || slices.Contains(cmds[0].Args, "--api-key") {
+		t.Errorf("the key is on the command line: %v", cmds[0].Args)
+	}
+	if !slices.Contains(cmds[0].Env, "LLAMA_API_KEY=k1") {
+		t.Error("llama-server was not given its key")
+	}
+}

internal/agent/engines/sglang.go

@@ -96,6 +96,11 @@ func (g *SGLang) serverCommand(s ShardSpec) (Command, error) {
 		)
 	}
 	args = append(args, p.Args...)
+	// SGLang reads its key only from the command line, so on a node with
+	// other users it is visible to them in the process list.
+	if s.APIKey != "" && s.Leader() {
+		args = append(args, "--api-key", s.APIKey)
+	}
 
 	return Command{Name: "sglang", Path: python, Args: args, Env: materializeEnv(baseEnv(s))}, nil
 }

internal/agent/engines/unsafeenv.go

@@ -0,0 +1,30 @@
+package engines
+
+import "strings"
+
+// UnsafeEnv reports whether a profile may not set an environment variable,
+// and why.
+//
+// A profile's env is the escape hatch for tuning no adapter can guess —
+// CUDA_*, NCCL_*, VLLM_*, HF_* — and it reaches an engine the agent starts as
+// root. Some variables are not tuning but a way to run other code in that
+// process: the dynamic loader's LD_PRELOAD, Python's PYTHONSTARTUP, a shell's
+// BASH_ENV. Set from a model definition, which an admin API or a config file
+// supplies, they would turn "add a model" into "run this as root on every
+// node". They are refused where a model is defined and again where an engine
+// starts.
+func UnsafeEnv(key string) (bool, string) {
+	k := strings.ToUpper(key)
+	switch {
+	case strings.HasPrefix(k, "LD_"), strings.HasPrefix(k, "DYLD_"):
+		return true, "it changes what the dynamic loader loads"
+	case k == "PATH":
+		return true, "it changes which binaries run"
+	case strings.HasPrefix(k, "PYTHON") && k != "PYTHONUNBUFFERED" && k != "PYTHONHASHSEED":
+		return true, "it changes what Python imports or runs at start"
+	case k == "BASH_ENV" || k == "ENV" || k == "PERL5OPT" || k == "PERL5LIB" ||
+		k == "NODE_OPTIONS" || k == "RUBYOPT" || k == "GCONV_PATH" || k == "OPENSSL_CONF":
+		return true, "it makes an interpreter or library load code"
+	}
+	return false, ""
+}

internal/agent/engines/vllm.go

@@ -150,6 +150,9 @@ func (v *VLLM) serverCommand(s ShardSpec) (Command, error) {
 	// Profile arguments last, so an operator can override anything above.
 	args = append(args, p.Args...)
 
+	if s.APIKey != "" {
+		env["VLLM_API_KEY"] = s.APIKey
+	}
 	return Command{Name: "vllm", Path: bin, Args: args, Env: materializeEnv(env)}, nil
 }
 

internal/agent/provision/install.go

@@ -16,6 +16,7 @@ import (
 	"os"
 	"os/exec"
 	"path/filepath"
+	"regexp"
 	"strings"
 	"time"
 
@@ -77,6 +78,10 @@ type Installer struct {
 	PyPIBase     string
 }
 
+// validVersion is what a version may look like: "b10991", "0.29.0",
+// "0.11.0+cu128". No separators, and not starting with a dot.
+var validVersion = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._+-]{0,63}$`)
+
 // DefaultRoot is where engines are installed when nothing says otherwise.
 const DefaultRoot = "/opt/llmgw/engines"
 
@@ -114,6 +119,15 @@ func (in *Installer) Converge(ctx context.Context, wants []Want) []Status {
 
 func (in *Installer) one(ctx context.Context, w Want) Status {
 	st := Status{Engine: w.Engine, Version: w.Version}
+	// Checked here, on the node, whatever the control plane checked: the
+	// version names a directory under the install root and is passed to a
+	// package manager, and "../x" would put an install — and the RemoveAll
+	// that replaces one — outside it, as root.
+	if !validVersion.MatchString(w.Version) {
+		st.State = Unsupported
+		st.Detail = fmt.Sprintf("%q is not a version this agent will install", w.Version)
+		return st
+	}
 	recipe, ok := Recipes[w.Engine]
 	if !ok {
 		st.State = Unsupported

internal/agent/provision/install_test.go

@@ -330,3 +330,15 @@ func TestAPackageManagerBusyWithItsOwnUpdatesIsWaitedFor(t *testing.T) {
 		t.Error("the wait was not reported")
 	}
 }
+
+func TestAVersionThatIsAPathIsRefused(t *testing.T) {
+	// The version names a directory under the install root; "../" would put
+	// an install, and the RemoveAll that replaces one, outside it, as root.
+	in := &Installer{Root: t.TempDir(), Host: ubuntu}
+	for _, v := range []string{"../../etc", "..", ".hidden", "b1/../../x", "", "b1 && rm -rf /"} {
+		got := in.Converge(t.Context(), []Want{{Engine: "llamacpp", Version: v}})
+		if got[0].State != Unsupported {
+			t.Errorf("version %q: %+v", v, got[0])
+		}
+	}
+}

internal/agent/supervisor/health.go

@@ -53,8 +53,8 @@ func NewHTTPHealth(timeout time.Duration, opts ...HTTPHealthOption) *HTTPHealth
 }
 
 // Ready reports whether the engine at baseURL is serving model.
-func (h *HTTPHealth) Ready(ctx context.Context, baseURL, model string) error {
-	if err := h.get(ctx, baseURL+h.healthPath, nil); err != nil {
+func (h *HTTPHealth) Ready(ctx context.Context, baseURL, model, apiKey string) error {
+	if err := h.get(ctx, baseURL+h.healthPath, apiKey, nil); err != nil {
 		return fmt.Errorf("health: %w", err)
 	}
 	if !h.requireName || model == "" {
@@ -66,7 +66,7 @@ func (h *HTTPHealth) Ready(ctx context.Context, baseURL, model string) error {
 			ID string `json:"id"`
 		} `json:"data"`
 	}
-	if err := h.get(ctx, baseURL+h.modelsPath, &list); err != nil {
+	if err := h.get(ctx, baseURL+h.modelsPath, apiKey, &list); err != nil {
 		return fmt.Errorf("models: %w", err)
 	}
 	for _, d := range list.Data {
@@ -77,11 +77,14 @@ func (h *HTTPHealth) Ready(ctx context.Context, baseURL, model string) error {
 	return fmt.Errorf("engine is up but does not serve %q yet", model)
 }
 
-func (h *HTTPHealth) get(ctx context.Context, url string, out any) error {
+func (h *HTTPHealth) get(ctx context.Context, url, apiKey string, out any) error {
 	req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
 	if err != nil {
 		return err
 	}
+	if apiKey != "" {
+		req.Header.Set("Authorization", "Bearer "+apiKey)
+	}
 	resp, err := h.client.Do(req)
 	if err != nil {
 		return err

internal/agent/supervisor/health_test.go

@@ -39,7 +39,7 @@ func engineStub(t *testing.T, healthy bool, models ...string) *httptest.Server {
 
 func TestReadyWhenServingModel(t *testing.T) {
 	srv := engineStub(t, true, "llama-3.3-70b")
-	if err := NewHTTPHealth(time.Second).Ready(t.Context(), srv.URL, "llama-3.3-70b"); err != nil {
+	if err := NewHTTPHealth(time.Second).Ready(t.Context(), srv.URL, "llama-3.3-70b", ""); err != nil {
 		t.Errorf("a serving engine was reported unready: %v", err)
 	}
 }
@@ -48,7 +48,7 @@ func TestNotReadyWhileLoading(t *testing.T) {
 	// The case that matters: the process is up but the weights are not loaded.
 	// Routing on /health alone would send traffic into a loading model.
 	srv := engineStub(t, false, "llama-3.3-70b")
-	err := NewHTTPHealth(time.Second).Ready(t.Context(), srv.URL, "llama-3.3-70b")
+	err := NewHTTPHealth(time.Second).Ready(t.Context(), srv.URL, "llama-3.3-70b", "")
 	if err == nil {
 		t.Fatal("an unhealthy engine was reported ready")
 	}
@@ -61,7 +61,7 @@ func TestNotReadyWhenModelMissing(t *testing.T) {
 	// /health passes but the model is not listed: the engine is up and serving
 	// something else, or still registering.
 	srv := engineStub(t, true, "some-other-model")
-	err := NewHTTPHealth(time.Second).Ready(t.Context(), srv.URL, "llama-3.3-70b")
+	err := NewHTTPHealth(time.Second).Ready(t.Context(), srv.URL, "llama-3.3-70b", "")
 	if err == nil {
 		t.Fatal("a model that is not served was reported ready")
 	}
@@ -73,14 +73,33 @@ func TestNotReadyWhenModelMissing(t *testing.T) {
 func TestModelCheckCanBeSkipped(t *testing.T) {
 	srv := engineStub(t, true)
 	h := NewHTTPHealth(time.Second, WithoutModelCheck())
-	if err := h.Ready(t.Context(), srv.URL, "anything"); err != nil {
+	if err := h.Ready(t.Context(), srv.URL, "anything", ""); err != nil {
 		t.Errorf("the model check was not skipped: %v", err)
 	}
 }
 
 func TestUnreachableEngine(t *testing.T) {
 	h := NewHTTPHealth(200 * time.Millisecond)
-	if err := h.Ready(t.Context(), "http://127.0.0.1:1", "m"); err == nil {
+	if err := h.Ready(t.Context(), "http://127.0.0.1:1", "m", ""); err == nil {
 		t.Error("an unreachable engine was reported ready")
 	}
 }
+
+func TestTheHealthCheckSendsTheEnginesKey(t *testing.T) {
+	// An engine started with a key refuses /v1/models without it, and a
+	// shard that is serving would read as never ready.
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		if r.Header.Get("Authorization") != "Bearer k1" {
+			w.WriteHeader(http.StatusUnauthorized)
+			return
+		}
+		_, _ = w.Write([]byte(`{"data":[{"id":"m"}]}`))
+	}))
+	defer srv.Close()
+	if err := NewHTTPHealth(time.Second).Ready(t.Context(), srv.URL, "m", "k1"); err != nil {
+		t.Errorf("with the key: %v", err)
+	}
+	if err := NewHTTPHealth(time.Second).Ready(t.Context(), srv.URL, "m", ""); err == nil {
+		t.Error("without the key the engine refused, and that read as ready")
+	}
+}

internal/agent/supervisor/supervisor.go

@@ -79,7 +79,7 @@ func (c *Config) normalize() {
 // supervisor can be tested without an HTTP engine.
 type HealthChecker interface {
 	// Ready reports whether the shard at baseURL serves model.
-	Ready(ctx context.Context, baseURL, model string) error
+	Ready(ctx context.Context, baseURL, model, apiKey string) error
 }
 
 // Shard is one supervised engine process.
@@ -411,7 +411,7 @@ func (s *Supervisor) WaitReady(ctx context.Context, sh *Shard, baseURL, model st
 			last := strings.Join(sh.Logs(10), "\n")
 			return fmt.Errorf("shard %s not ready within %v; last output:\n%s", sh.ID, timeout, last)
 		case <-t.C:
-			if err := s.health.Ready(ctx, baseURL, model); err != nil {
+			if err := s.health.Ready(ctx, baseURL, model, sh.Spec.APIKey); err != nil {
 				continue
 			}
 			sh.setState(Ready, nil)

internal/agent/supervisor/supervisor_test.go

@@ -21,7 +21,7 @@ import (
 // scriptedHealth reports ready once the flag is set.
 type scriptedHealth struct{ ready atomic.Bool }
 
-func (h *scriptedHealth) Ready(context.Context, string, string) error {
+func (h *scriptedHealth) Ready(context.Context, string, string, string) error {
 	if h.ready.Load() {
 		return nil
 	}

internal/bench/bench.go

@@ -225,6 +225,7 @@ func runStep(ctx context.Context, client *http.Client, cfg Config, concurrency i
 	)
 
 	sampler := newSampler(cfg.MetricsURL, cfg.Engine)
+	sampler.apiKey = cfg.APIKey
 	sampler.start(ctx)
 
 	start := time.Now()

internal/bench/sampler.go

@@ -19,6 +19,7 @@ import (
 // Optional, because a run pointed at the gateway has no single engine to read.
 type sampler struct {
 	url    string
+	apiKey string
 	engine string
 	client *http.Client
 	cancel context.CancelFunc
@@ -65,6 +66,9 @@ func (s *sampler) sample(ctx context.Context) {
 	if err != nil {
 		return
 	}
+	if s.apiKey != "" {
+		req.Header.Set("Authorization", "Bearer "+s.apiKey)
+	}
 	resp, err := s.client.Do(req)
 	if err != nil {
 		return

internal/cli/bench.go

@@ -41,6 +41,11 @@ func newBenchCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 			cfg.MetricsURL = metrics
 			cfg.Engine = engine
 			cfg.APIKey = key
+			// A replica's engine demands its own key. Asked for rather than
+			// typed: the admin API knows which replica answers at the target.
+			if cfg.APIKey == "" {
+				cfg.APIKey = replicaKey(cmd, client, target)
+			}
 			if len(steps) > 0 {
 				cfg.Concurrency = steps
 			}
@@ -227,3 +232,18 @@ func ms(d time.Duration) string {
 		return d.Round(10 * time.Millisecond).String()
 	}
 }
+
+// replicaKey is the engine key of the replica at target, or "" when target is
+// not a replica (the gateway, say) or the admin API cannot be asked.
+func replicaKey(cmd *cobra.Command, client func() *ctlClient, target string) string {
+	var reps []adminapi.ReplicaView
+	if err := client().get(cmd.Context(), "/api/v1/replicas", &reps); err != nil {
+		return ""
+	}
+	for _, rep := range reps {
+		if sameEndpoint(rep.Endpoint, target) {
+			return rep.EngineKey
+		}
+	}
+	return ""
+}

internal/cli/ctl.go

@@ -67,7 +67,9 @@ func (c *ctlClient) do(ctx context.Context, method, path string, body, out any)
 	if err != nil {
 		return err
 	}
-	if body != nil {
+	// On every change, body or not: the admin listener refuses anything else,
+	// because a browser can send a bodiless POST across origins unasked.
+	if method != http.MethodGet {
 		req.Header.Set("Content-Type", "application/json")
 	}
 	// Who made the change, kept with a stored model so a catalog that grew
@@ -81,7 +83,10 @@ func (c *ctlClient) do(ctx context.Context, method, path string, body, out any)
 	}
 	defer resp.Body.Close()
 
-	if resp.StatusCode != http.StatusOK {
+	// Any 2xx. The API answers a change it has queued rather than made — a
+	// restart — with 202, and treating that as a failure printed
+	// "Error: 202 Accepted" over a request that had worked.
+	if resp.StatusCode < 200 || resp.StatusCode > 299 {
 		var e struct {
 			Error string `json:"error"`
 		}
@@ -133,6 +138,8 @@ func newCtlCmd() *cobra.Command {
 		newStatusCmd(client, &asJSON),
 		newModelCmd(client),
 		newScaleCmd(client),
+		newRestartCmd(client),
+		newRetryCmd(client),
 		newEngineCheckCmd(),
 		newToolCheckCmd(),
 		newWeightsCmd(client, &asJSON),
@@ -242,6 +249,10 @@ func newNodesCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 // actually served, never what is merely catalogued.
 func modelStatus(m adminapi.ModelView) string {
 	switch {
+	case m.Parked:
+		return "parked"
+	case m.Unplaceable:
+		return "cannot place"
 	case m.Min == 0 && m.Total == 0:
 		return "stopped"
 	case m.Ready >= m.Min && m.Min > 0:
@@ -368,6 +379,9 @@ func newReplicasCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 				if r.Stale {
 					state += " (stale profile)"
 				}
+				if r.Restart != "" {
+					state += " (restart " + r.Restart + ")"
+				}
 				fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n",
 					r.ID, r.Model, state, r.Pool, join(nodes),
 					(time.Duration(r.AgeSec) * time.Second).String(),

internal/cli/ctl_restart.go

@@ -0,0 +1,75 @@
+package cli
+
+import (
+	"encoding/json"
+	"errors"
+	"fmt"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+)
+
+func newRestartCmd(client func() *ctlClient) *cobra.Command {
+	var (
+		models []string
+		all    bool
+	)
+	c := &cobra.Command{
+		Use:   "restart [REPLICA...]",
+		Short: "Replace replicas with new ones, starting each replacement before stopping the old",
+		Long: "How a running engine picks up what only applies at start: an engine key,\n" +
+			"an edited profile, a new engine version.\n\n" +
+			"Each replacement is placed beside the replica it replaces, and the old one\n" +
+			"keeps serving until the new one is ready. Replicas of one model are\n" +
+			"replaced one at a time. Where nothing else has room for a second copy —\n" +
+			"the usual case on a small CPU fleet — the old one is stopped after the\n" +
+			"drain grace period (5 minutes by default) and the new one starts in its\n" +
+			"place; a model with only that replica is unserved while it loads.\n\n" +
+			"  llmgw ctl restart qwen2.5-7b-3f2a91c0\n" +
+			"  llmgw ctl restart --model qwen2.5-7b\n" +
+			"  llmgw ctl restart --all",
+		RunE: func(cmd *cobra.Command, args []string) error {
+			if len(args) == 0 && len(models) == 0 && !all {
+				return errors.New("name replicas, --model, or --all")
+			}
+			body, _ := json.Marshal(adminapi.RestartRequest{Replicas: args, Models: models, All: all})
+			var res adminapi.RestartResult
+			if err := client().post(cmd.Context(), "/api/v1/restart", body, &res); err != nil {
+				return err
+			}
+			fmt.Printf("%d replica(s) queued for restart; watch with: llmgw ctl replicas\n", len(res.Queued))
+			return nil
+		},
+	}
+	c.Flags().StringArrayVar(&models, "model", nil, "every replica of this model; repeatable")
+	c.Flags().BoolVar(&all, "all", false, "every ready replica in the cluster")
+	return c
+}
+
+func newRetryCmd(client func() *ctlClient) *cobra.Command {
+	return &cobra.Command{
+		Use:   "retry MODEL",
+		Short: "Try a parked model again, and clear its backoff",
+		Long: "A model whose replicas fail every time they start is parked after five\n" +
+			"failures in a row: it is not placed again until its definition changes\n" +
+			"or this is run. Run it once the cause — the weights, the engine, the\n" +
+			"profile — has been dealt with; `llmgw ctl status` lists parked models with\n" +
+			"their last error.",
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			var res struct {
+				WasParked bool `json:"was_parked"`
+			}
+			if err := client().post(cmd.Context(), "/api/v1/models/"+args[0]+"/retry", []byte(`{}`), &res); err != nil {
+				return err
+			}
+			if res.WasParked {
+				fmt.Printf("%s unparked; it is placed on the next pass\n", args[0])
+			} else {
+				fmt.Printf("%s was not parked; its backoff is cleared\n", args[0])
+			}
+			return nil
+		},
+	}
+}

internal/cli/ctl_status_test.go

@@ -0,0 +1,24 @@
+package cli
+
+import (
+	"net/http"
+	"net/http/httptest"
+	"testing"
+)
+
+func TestAQueuedChangeIsNotReportedAsAnError(t *testing.T) {
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+		w.WriteHeader(http.StatusAccepted)
+		_, _ = w.Write([]byte(`{"queued":["r1"]}`))
+	}))
+	defer srv.Close()
+	var out struct {
+		Queued []string `json:"queued"`
+	}
+	if err := newCtlClient(srv.URL).post(t.Context(), "/api/v1/restart", []byte(`{}`), &out); err != nil {
+		t.Fatalf("202 read as a failure: %v", err)
+	}
+	if len(out.Queued) != 1 {
+		t.Errorf("the body of a 202 was not read: %+v", out)
+	}
+}

internal/cli/mirror.go

@@ -233,6 +233,7 @@ func newMirrorServeCmd(dir *string) *cobra.Command {
 	var listen, apiListen, base string
 	var tlsc tlsconf.Config
 	var allow []string
+	var allowPrivate bool
 
 	c := &cobra.Command{
 		Use:   "serve",
@@ -262,6 +263,9 @@ func newMirrorServeCmd(dir *string) *cobra.Command {
 			out := cmd.OutOrStdout()
 			log := slog.Default()
 			mgr := mirror.NewManager(*dir, log)
+			if allowPrivate {
+				mgr.AllowPrivateSources()
+			}
 			api := mirror.NewAPI(mgr, strings.TrimSuffix(base, "/"))
 
 			// http.Dir refuses paths that escape the root, which is the whole
@@ -278,7 +282,7 @@ func newMirrorServeCmd(dir *string) *cobra.Command {
 			}
 			admin := &http.Server{
 				Addr:              apiListen,
-				Handler:           onlyFrom(allowed, api.Handler()),
+				Handler:           apiGate(tlsc.Enabled(), allowed, api.Handler()),
 				ReadHeaderTimeout: 10 * time.Second,
 			}
 
@@ -293,11 +297,11 @@ func newMirrorServeCmd(dir *string) *cobra.Command {
 			case len(allowed) > 0:
 				fmt.Fprintf(out, "management API on %s, for %s and this machine only\n", apiListen, strings.Join(allow, ", "))
 			default:
-				// Said plainly rather than left to be discovered. An API that
-				// can delete models and start downloads, open to anyone who can
-				// reach the port, is a choice the operator should have made
-				// rather than inherited from a default.
-				fmt.Fprintf(out, "management API on %s, UNAUTHENTICATED — keep it on a trusted network\n", apiListen)
+				// This machine only, which was once "anyone": an API that can
+				// delete models and start downloads, open to every node that
+				// could reach the port, is a choice the operator should make
+				// with --api-allow rather than inherit.
+				fmt.Fprintf(out, "management API on %s, for this machine only (--api-allow to add the control plane)\n", apiListen)
 			}
 			fmt.Fprintf(out, "catalog sources should point at %s\n", base)
 
@@ -312,7 +316,9 @@ func newMirrorServeCmd(dir *string) *cobra.Command {
 	c.Flags().StringVar(&base, "base", "http://localhost:8081",
 		"the address nodes use to reach this mirror, as it should appear in catalog sources")
 	c.Flags().StringArrayVar(&allow, "api-allow", nil,
-		"an address or network the management API answers, besides this machine; repeatable (default: anyone)")
+		"an address or network the management API answers, besides this machine; repeatable (default: this machine only)")
+	c.Flags().BoolVar(&allowPrivate, "allow-private-sources", false,
+		"let downloads reach loopback, private and link-local addresses")
 	c.Flags().StringVar(&tlsc.CA, "tls-ca", "", "CA bundle that management clients must present a certificate from")
 	c.Flags().StringVar(&tlsc.Cert, "tls-cert", "", "this mirror's certificate")
 	c.Flags().StringVar(&tlsc.Key, "tls-key", "", "this mirror's private key")
@@ -435,14 +441,21 @@ func parseAllow(list []string) ([]netip.Prefix, error) {
 	return out, nil
 }
 
-// onlyFrom answers only this machine and the addresses allowed, when any
-// are: the management API can delete models and start downloads, and a
-// mirror on its own machine is reached over the network by one control plane
-// and no one else. Nothing allowed is the old behaviour, anyone.
-func onlyFrom(allowed []netip.Prefix, next http.Handler) http.Handler {
-	if len(allowed) == 0 {
+// apiGate decides who reaches the management API. With mutual TLS the
+// certificate decides, narrowed by --api-allow when given. Without it, only
+// this machine and the addresses allowed: nothing allowed used to mean anyone,
+// and every node that could reach the port could delete models.
+func apiGate(mtls bool, allowed []netip.Prefix, next http.Handler) http.Handler {
+	if mtls && len(allowed) == 0 {
 		return next
 	}
+	return onlyFrom(allowed, next)
+}
+
+// onlyFrom answers only this machine and the addresses allowed: the management
+// API can delete models and start downloads, and a mirror on its own machine
+// is reached over the network by one control plane and no one else.
+func onlyFrom(allowed []netip.Prefix, next http.Handler) http.Handler {
 	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 		host, _, err := net.SplitHostPort(r.RemoteAddr)
 		a, perr := netip.ParseAddr(host)

internal/cli/mirror_gate_test.go

@@ -0,0 +1,39 @@
+package cli
+
+import (
+	"net/http"
+	"net/http/httptest"
+	"net/netip"
+	"testing"
+)
+
+func TestTheMirrorAPIAnswersOnlyThisMachineUnlessTold(t *testing.T) {
+	ok := http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {})
+	call := func(h http.Handler, from string) int {
+		r := httptest.NewRequest("DELETE", "/v1/models/m/v", nil)
+		r.RemoteAddr = from + ":5000"
+		w := httptest.NewRecorder()
+		h.ServeHTTP(w, r)
+		return w.Code
+	}
+	cp := []netip.Prefix{netip.MustParsePrefix("10.0.0.140/32")}
+
+	// Nothing allowed and no TLS used to mean anyone: every node could delete
+	// models.
+	if got := call(apiGate(false, nil, ok), "10.0.0.141"); got != http.StatusForbidden {
+		t.Errorf("a node reached an unconfigured mirror API: %d", got)
+	}
+	if got := call(apiGate(false, nil, ok), "127.0.0.1"); got != http.StatusOK {
+		t.Errorf("the mirror's own machine was refused: %d", got)
+	}
+	if got := call(apiGate(false, cp, ok), "10.0.0.140"); got != http.StatusOK {
+		t.Errorf("the allowed control plane was refused: %d", got)
+	}
+	if got := call(apiGate(false, cp, ok), "10.0.0.141"); got != http.StatusForbidden {
+		t.Errorf("a node outside the allowlist got through: %d", got)
+	}
+	// With mutual TLS the certificate decides.
+	if got := call(apiGate(true, nil, ok), "10.0.0.141"); got != http.StatusOK {
+		t.Errorf("mTLS without an allowlist refused a client: %d", got)
+	}
+}

internal/cli/server.go

@@ -36,6 +36,7 @@ import (
 	"gitea.mynodes.xyz/ilker/llmgw/internal/promq"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/router"
 	storepkg "gitea.mynodes.xyz/ilker/llmgw/internal/store"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/version"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/web"
 )
@@ -131,6 +132,11 @@ func newServerCmd() *cobra.Command {
 				// On the same /metrics as the traffic figures: correlating
 				// "the cluster was full" with "requests were queueing" should
 				// not need two systems.
+				if err := rt.RegisterMetrics(tlsconf.ExpiryCollector(map[string]string{
+					"agent-server": cfg.AgentTLS.Cert, "agent-ca": cfg.AgentTLS.CA,
+				})); err != nil {
+					return err
+				}
 				if err := rt.RegisterMetrics(fleetmetrics.New(ctl)); err != nil {
 					return err
 				}
@@ -246,7 +252,7 @@ func newServerCmd() *cobra.Command {
 				// enforces its own per-phase deadlines.
 			}
 			adminSrv := &http.Server{
-				Handler:           adminMux,
+				Handler:           router.AdminGuard(adminMux),
 				ReadHeaderTimeout: 10 * time.Second,
 			}
 
@@ -385,6 +391,20 @@ func startControlPlane(ctx context.Context, cfg config.Server, cat *catalog.Live
 		ccfg.Reconciler.PortRange = [2]int{cfg.Control.PortRange[0], cfg.Control.PortRange[1]}
 	}
 
+	// Each replica's engine demands a key only the router has. The secret it
+	// is derived from lives in the store, so every control plane against it —
+	// and this one after a restart — computes the same key for a replica.
+	if h, ok := sink.(interface{ Store() *storepkg.DB }); ok && h.Store() != nil {
+		secret, err := h.Store().ClusterSecret(ctx, "engine-key")
+		if err != nil {
+			return nil, nil, fmt.Errorf("reading the engine key secret: %w", err)
+		}
+		ccfg.Reconciler.EngineKey = control.EngineKeys([]byte(secret))
+	} else {
+		log.Warn("engines start without a key: there is no store to keep the secret " +
+			"they are derived from, so anyone who can reach a node's ports reaches its models")
+	}
+
 	// The service needs the inventory and the status sink, both of which the
 	// controller owns; the controller needs the service as its dispatcher. The
 	// cycle is broken by giving the controller a dispatcher that forwards to

internal/cli/status.go

@@ -94,6 +94,14 @@ func printSummary(cmd *cobra.Command, s adminapi.Summary) {
 	fmt.Fprintf(out, "\nMODELS    %d of %d have the replicas they asked for\n",
 		s.Models.Satisfied, s.Models.Total)
 
+	if len(s.Parked) > 0 {
+		fmt.Fprintln(out, "\nPARKED — failed every time they started; not retried until fixed")
+		for _, p := range s.Parked {
+			fmt.Fprintf(out, "  %s (%d failures): %s\n", p.Model, p.Failures, p.Reason)
+		}
+		fmt.Fprintln(out, "\n  Once fixed: llmgw ctl retry <model>")
+	}
+
 	if len(s.Unplaceable) > 0 {
 		fmt.Fprintln(out, "\nNOT PLACED")
 		for _, m := range s.Unplaceable {

internal/cli/tui_init.go

@@ -5,11 +5,11 @@ import (
 	"fmt"
 	"io"
 	"net/mail"
+	"os"
+	"path/filepath"
 	"strings"
 	"sync"
 
-	"github.com/charmbracelet/bubbles/spinner"
-	"github.com/charmbracelet/bubbles/textinput"
 	tea "github.com/charmbracelet/bubbletea"
 	"github.com/charmbracelet/lipgloss"
 )
@@ -65,8 +65,8 @@ type initModel struct {
 	// The form, until it is submitted.
 	form      bool
 	focus     int
-	admin     textinput.Model
-	hosts     textinput.Model
+	admin     textField
+	hosts     textField
 	toggles   [fCount]bool
 	formError string
 
@@ -75,7 +75,7 @@ type initModel struct {
 	lines    []string // init's lines, as it wrote them
 	password string
 	next     []string // the closing "Next:" text
-	spin     spinner.Model
+	spin     spinnerModel
 	width    int
 	done     bool
 	err      error
@@ -84,19 +84,17 @@ type initModel struct {
 }
 
 func newInitModel(in *serverInit, found []string, askForm bool) initModel {
-	admin := textinput.New()
+	admin := newTextField()
 	admin.Placeholder = "you@example.com"
 	admin.SetValue(in.admin)
 	admin.CharLimit = 200
 	admin.Width = 40
-	hosts := textinput.New()
+	hosts := newTextField()
 	hosts.Placeholder = "llm.example.com, 203.0.113.10"
 	hosts.SetValue(strings.Join(in.hosts, ", "))
 	hosts.CharLimit = 400
 	hosts.Width = 40
-	sp := spinner.New()
-	sp.Spinner = spinner.MiniDot
-	sp.Style = lipgloss.NewStyle().Foreground(tuiAccent)
+	sp := newSpinner()
 	m := initModel{in: in, found: found, form: askForm, admin: admin, hosts: hosts, spin: sp, width: 100}
 	m.toggles[fPostgres] = in.postgres
 	m.toggles[fMirror] = !in.noMirror
@@ -110,7 +108,7 @@ func newInitModel(in *serverInit, found []string, askForm bool) initModel {
 
 func (m initModel) Init() tea.Cmd {
 	if m.form {
-		return textinput.Blink
+		return nil
 	}
 	return tea.Batch(m.spin.Tick, func() tea.Msg { return startInitMsg{} })
 }
@@ -121,7 +119,7 @@ func (m initModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
 	switch msg := msg.(type) {
 	case tea.WindowSizeMsg:
 		m.width = msg.Width
-	case spinner.TickMsg:
+	case spinTickMsg:
 		var cmd tea.Cmd
 		m.spin, cmd = m.spin.Update(msg)
 		return m, cmd
@@ -391,13 +389,33 @@ func (in *serverInit) runInitTUI(ctx context.Context, out io.Writer) error {
 		return fmt.Errorf("stopped; run init again to finish what was started")
 	}
 
-	// The screen showed the last lines; the whole run goes to the terminal's
-	// history, the password with it.
+	// The whole run is kept, root's only, since it holds the password. On a
+	// failure it is printed too, for the lines the screen had no room for;
+	// otherwise the screen's last frame says it, and only what must outlive
+	// the screen is printed again: the password, and what to do next.
 	logMu.Lock()
 	defer logMu.Unlock()
+	whole := strings.Join(log, "\n") + "\n"
+	saved := ""
+	if in.stateDir != "" {
+		p := filepath.Join(in.stateDir, "init.log")
+		if err := os.WriteFile(p, []byte(whole), 0o600); err == nil {
+			saved = p
+		}
+	}
 	fmt.Fprintln(out)
-	for _, l := range log {
+	if fm.err != nil {
+		fmt.Fprint(out, whole)
+		return fm.err
+	}
+	if fm.password != "" {
+		fmt.Fprintf(out, "Administrator %s, password %s (shown once)\n\n", in.admin, fm.password)
+	}
+	for _, l := range fm.next {
 		fmt.Fprintln(out, l)
 	}
+	if saved != "" {
+		fmt.Fprintf(out, "\nThe whole run is in %s.\n", saved)
+	}
 	return fm.err
 }

internal/cli/tui_node.go

@@ -7,8 +7,6 @@ import (
 	"os"
 	"strings"
 
-	"github.com/charmbracelet/bubbles/spinner"
-	"github.com/charmbracelet/bubbles/textinput"
 	tea "github.com/charmbracelet/bubbletea"
 	"github.com/charmbracelet/lipgloss"
 	"github.com/mattn/go-isatty"
@@ -67,7 +65,7 @@ type nodeModel struct {
 	title   string
 	steps   map[string]stepState
 	notes   []onboard.Event // warnings and failures, with their fixes
-	spin    spinner.Model
+	spin    spinnerModel
 	cancel  context.CancelFunc
 	width   int
 	done    bool
@@ -77,16 +75,14 @@ type nodeModel struct {
 
 	// The question being asked, if any.
 	fabrics   *askFabricsMsg
-	inputs    []textinput.Model
+	inputs    []textField
 	focus     int
 	confirm   *askContinueMsg
 	quitAsked bool
 }
 
 func newNodeModel(title string, cancel context.CancelFunc) nodeModel {
-	sp := spinner.New()
-	sp.Spinner = spinner.MiniDot
-	sp.Style = lipgloss.NewStyle().Foreground(tuiAccent)
+	sp := newSpinner()
 	return nodeModel{title: title, steps: map[string]stepState{}, spin: sp, cancel: cancel, width: 100}
 }
 
@@ -96,7 +92,7 @@ func (m nodeModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
 	switch msg := msg.(type) {
 	case tea.WindowSizeMsg:
 		m.width = msg.Width
-	case spinner.TickMsg:
+	case spinTickMsg:
 		var cmd tea.Cmd
 		m.spin, cmd = m.spin.Update(msg)
 		return m, cmd
@@ -121,7 +117,7 @@ func (m nodeModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
 		m.fabrics = &msg
 		m.inputs = nil
 		for i, f := range msg.suggested {
-			in := textinput.New()
+			in := newTextField()
 			in.Prompt = ""
 			in.SetValue(f.Name)
 			in.CharLimit = 40

internal/cli/tui_widgets.go

@@ -0,0 +1,98 @@
+package cli
+
+import (
+	"strings"
+	"time"
+
+	tea "github.com/charmbracelet/bubbletea"
+	"github.com/charmbracelet/lipgloss"
+)
+
+// Two small widgets, written here rather than taken from bubbles: its
+// packages ask the terminal for its colours as they load, so every llmgw
+// command — llmgw --version included — sent a query and, in a terminal that
+// never answers one, waited five seconds for it. Only the screens need to
+// know the colours, and lipgloss asks when they first draw.
+
+// spinTickMsg advances a spinner.
+type spinTickMsg struct{}
+
+var spinFrames = []string{"⠋", "⠙", "⠹", "⠸", "⠼", "⠴", "⠦", "⠧", "⠇", "⠏"}
+
+type spinnerModel struct {
+	frame int
+	Style lipgloss.Style
+}
+
+func newSpinner() spinnerModel { return spinnerModel{Style: lipgloss.NewStyle().Foreground(tuiAccent)} }
+
+// Tick starts the spinner turning; it is a tea.Cmd.
+func (s spinnerModel) Tick() tea.Msg { return spinTickMsg{} }
+
+func (s spinnerModel) Update(msg tea.Msg) (spinnerModel, tea.Cmd) {
+	if _, ok := msg.(spinTickMsg); !ok {
+		return s, nil
+	}
+	s.frame = (s.frame + 1) % len(spinFrames)
+	return s, tea.Tick(100*time.Millisecond, func(time.Time) tea.Msg { return spinTickMsg{} })
+}
+
+func (s spinnerModel) View() string { return s.Style.Render(spinFrames[s.frame]) }
+
+// textField is one line of text typed at its end: all a form here needs.
+type textField struct {
+	value       []rune
+	focused     bool
+	Prompt      string
+	Placeholder string
+	CharLimit   int
+	Width       int
+}
+
+func newTextField() textField { return textField{Prompt: "> "} }
+
+func (f *textField) SetValue(s string) { f.value = []rune(s) }
+func (f textField) Value() string      { return string(f.value) }
+func (f *textField) Focus()            { f.focused = true }
+func (f *textField) Blur()             { f.focused = false }
+
+func (f textField) Update(msg tea.Msg) (textField, tea.Cmd) {
+	k, ok := msg.(tea.KeyMsg)
+	if !ok || !f.focused {
+		return f, nil
+	}
+	switch k.Type {
+	case tea.KeyRunes, tea.KeySpace:
+		r := k.Runes
+		if k.Type == tea.KeySpace {
+			r = []rune{' '}
+		}
+		for _, c := range r {
+			if f.CharLimit == 0 || len(f.value) < f.CharLimit {
+				f.value = append(f.value, c)
+			}
+		}
+	case tea.KeyBackspace:
+		if len(f.value) > 0 {
+			f.value = f.value[:len(f.value)-1]
+		}
+	case tea.KeyCtrlU:
+		f.value = nil
+	}
+	return f, nil
+}
+
+func (f textField) View() string {
+	text := string(f.value)
+	shown := text
+	if text == "" && f.Placeholder != "" {
+		shown = sDim.Render(f.Placeholder)
+	}
+	if f.focused {
+		shown += "█"
+	}
+	if pad := f.Width - len([]rune(text)); pad > 0 && text != "" {
+		shown += strings.Repeat(" ", pad)
+	}
+	return f.Prompt + shown
+}

internal/config/config.go

@@ -404,8 +404,11 @@ type Scoring struct {
 
 // Endpoint is one engine replica. Static in phase 1.
 type Endpoint struct {
-	ID     string   `koanf:"id"`
-	URL    string   `koanf:"url"`
+	ID  string `koanf:"id"`
+	URL string `koanf:"url"`
+	// APIKey is sent to the engine as a bearer token, for an engine started
+	// with one.
+	APIKey string   `koanf:"api_key"`
 	Models []string `koanf:"models"`
 	// Engine names the inference engine, which decides what its metrics are
 	// called. Empty means vLLM's names.

internal/control/adminapi/api.go

@@ -77,6 +77,8 @@ func (a *API) Register(mux *http.ServeMux) {
 	mux.HandleFunc("DELETE /api/v1/nodes/{id}", a.removeNode)
 	mux.HandleFunc("POST /api/v1/fabric-test", a.fabricTest)
 	mux.HandleFunc("GET /api/v1/nodes/{id}/logs", a.nodeLogs)
+	mux.HandleFunc("POST /api/v1/restart", a.restart)
+	mux.HandleFunc("POST /api/v1/models/{name}/retry", a.retryModel)
 }
 
 // --- responses ---
@@ -232,13 +234,32 @@ type ModelView struct {
 	Max         int      `json:"max_replicas"`
 	Ready       int      `json:"ready_replicas"`
 	Total       int      `json:"total_replicas"`
+	// Desired is how many it should have now: Min, or for an on-demand
+	// model one while it is being asked for and none otherwise.
+	Desired int `json:"desired_replicas"`
+	// Failures counts replicas that failed to start in a row.
+	Failures int `json:"failures,omitempty"`
+	// Parked is set once Failures reached the limit; the model is not retried
+	// until its definition changes or `ctl retry` is run. Reason is the last
+	// error.
+	Parked bool   `json:"parked,omitempty"`
+	Reason string `json:"reason,omitempty"`
+	// Unplaceable is set when it is short and the last attempt found no node.
+	Unplaceable bool     `json:"unplaceable,omitempty"`
 	Fallbacks   []string `json:"fallbacks,omitempty"`
 	Profiles    []string `json:"profiles"`
 }
 
 // ReplicaView is a replica as `ctl replicas` shows it.
 type ReplicaView struct {
-	ID      string `json:"id"`
+	ID string `json:"id"`
+	// EngineKey is the key this replica's engine demands, for `ctl bench`
+	// and anyone else holding the admin listener who needs to reach an engine
+	// directly. The client-facing pages never render it.
+	EngineKey string `json:"engine_key,omitempty"`
+	// Restart is "queued" or "replacing" while an operator's restart of this
+	// replica is under way.
+	Restart string `json:"restart,omitempty"`
 	Model   string `json:"model"`
 	Profile string `json:"profile"`
 	// Stale is true when the profile has been edited since this replica was
@@ -434,13 +455,20 @@ func (a *API) Models() []ModelView {
 		}
 	}
 
+	health := map[string]reconciler.ModelHealth{}
+	for _, h := range a.ctl.Reconciler().ModelHealth() {
+		health[h.Model] = h
+	}
 	out := make([]ModelView, 0)
 	for _, m := range a.ctl.Catalog().Models() {
+		h := health[m.Name]
 		v := ModelView{
 			Name: m.Name, Description: m.Description, Aliases: m.Aliases, Source: m.Source,
 			Min: m.Scaling.Min, Max: m.Scaling.Max,
 			Ready: ready[m.Name], Total: total[m.Name],
 			Fallbacks: m.Fallbacks,
+			Desired:   h.Desired, Failures: h.Failures,
+			Parked: h.Parked, Reason: h.Reason, Unplaceable: h.Unplaceable,
 		}
 		for _, p := range m.Profiles {
 			v.Profiles = append(v.Profiles, p.Name)
@@ -471,9 +499,11 @@ func (a *API) Replicas(model string) []ReplicaView {
 			ID: rep.ID, Model: rep.Model, Profile: rep.Profile,
 			State: rep.State.String(), Pool: rep.Pool, Fabric: rep.Fabric,
 			Endpoint: rep.Endpoint, Err: rep.Err,
-			Stale:   a.profileChanged(rep.Model, rep.Profile, rep.ProfileFP),
-			AgeSec:  int64(time.Since(rep.CreatedAt).Seconds()),
-			IdleSec: -1,
+			EngineKey: a.ctl.EngineKey(rep.ID),
+			Restart:   a.ctl.Reconciler().RestartState(rep.ID),
+			Stale:     a.profileChanged(rep.Model, rep.Profile, rep.ProfileFP),
+			AgeSec:    int64(time.Since(rep.CreatedAt).Seconds()),
+			IdleSec:   -1,
 		}
 		if rep.LastUsedAt.IsZero() {
 			v.IdleSec = -1
@@ -1039,3 +1069,49 @@ func (a *API) Topology(id string) (TopologyView, bool) {
 func (a *API) NVLink() nvlink.Report {
 	return nvlink.Analyze(a.ctl.Inventory().Nodes())
 }
+
+// RestartRequest names what to restart: replicas by id, every replica of some
+// models, or — All — every ready replica in the cluster.
+type RestartRequest struct {
+	Replicas []string `json:"replicas,omitempty"`
+	Models   []string `json:"models,omitempty"`
+	All      bool     `json:"all,omitempty"`
+}
+
+// RestartResult lists the replicas queued.
+type RestartResult struct {
+	Queued []string `json:"queued"`
+}
+
+func (a *API) restart(w http.ResponseWriter, r *http.Request) {
+	var req RestartRequest
+	if err := json.NewDecoder(io.LimitReader(r.Body, 1<<16)).Decode(&req); err != nil {
+		writeError(w, http.StatusBadRequest, "the restart request is not valid JSON: "+err.Error())
+		return
+	}
+	ids := slices.Clone(req.Replicas)
+	for _, rep := range a.ctl.Reconciler().ReadyReplicas() {
+		if req.All || slices.Contains(req.Models, rep.Model) {
+			ids = append(ids, rep.ID)
+		}
+	}
+	if len(ids) == 0 {
+		writeError(w, http.StatusBadRequest, "nothing ready matches; name replicas, models, or all")
+		return
+	}
+	if err := a.ctl.Reconciler().Restart(ids...); err != nil {
+		writeError(w, http.StatusNotFound, err.Error())
+		return
+	}
+	writeJSON(w, http.StatusAccepted, RestartResult{Queued: ids})
+}
+
+func (a *API) retryModel(w http.ResponseWriter, r *http.Request) {
+	name := r.PathValue("name")
+	if _, ok := a.ctl.Catalog().Resolve(name); !ok {
+		writeError(w, http.StatusNotFound, "no model called "+name)
+		return
+	}
+	was := a.ctl.Reconciler().Retry(name)
+	writeJSON(w, http.StatusOK, map[string]bool{"was_parked": was})
+}

internal/control/adminapi/summary.go

@@ -25,6 +25,9 @@ type Summary struct {
 	// Unplaceable names models the scheduler currently cannot satisfy. This is
 	// the line that turns "something is wrong" into a place to look.
 	Unplaceable []string `json:"unplaceable,omitempty"`
+	// Parked are models that failed every time they started and are no
+	// longer retried, with the last error.
+	Parked []ParkedModel `json:"parked,omitempty"`
 	// IdleReplicas are serving replicas nothing has asked for in a while, which
 	// on a fleet with more models than GPUs is the first place to find room.
 	IdleReplicas []IdleReplica `json:"idle_replicas,omitempty"`
@@ -201,17 +204,30 @@ func (a *API) Summary() Summary {
 
 	for _, m := range a.Models() {
 		s.Models.Total++
-		if m.Ready >= m.Min {
+		if m.Parked {
+			s.Parked = append(s.Parked, ParkedModel{Model: m.Name, Failures: m.Failures, Reason: m.Reason})
+		}
+		// Against what it should have now, not its minimum: an on-demand
+		// model's minimum is 0, so one that was asked for and fitted nowhere
+		// used to count as satisfied.
+		if m.Ready >= m.Desired && !m.Parked {
 			s.Models.Satisfied++
 			continue
 		}
 		s.Models.Short++
 		// Short of replicas and unable to place more is the actionable case;
 		// short and still starting is just a cluster doing its job.
-		if m.Total <= m.Ready {
+		if m.Unplaceable || (m.Total <= m.Ready && !m.Parked) {
 			s.Unplaceable = append(s.Unplaceable, m.Name)
 		}
 	}
 	sort.Strings(s.Unplaceable)
 	return s
 }
+
+// ParkedModel is a model no longer retried, and why.
+type ParkedModel struct {
+	Model    string `json:"model"`
+	Failures int    `json:"failures"`
+	Reason   string `json:"reason"`
+}

internal/control/agentsvc/convert.go

@@ -191,6 +191,7 @@ func StartShardFromCommand(cmd reconciler.ShardCommand) *llmgwv1.StartShard {
 		Pool:             cmd.Pool,
 		Fabric:           cmd.Fabric,
 		ProfileFp:        cmd.ProfileFP,
+		ApiKey:           cmd.Spec.APIKey,
 	}
 }
 
@@ -211,6 +212,7 @@ func ShardSpecFromStart(m *llmgwv1.StartShard) (engines.ShardSpec, time.Duration
 		Pool:          m.GetPool(),
 		Fabric:        m.GetFabric(),
 		ProfileFP:     m.GetProfileFp(),
+		APIKey:        m.GetApiKey(),
 		Profile: engines.Profile{
 			Name: p.GetName(), Engine: p.GetEngine(), GPUClass: p.GetGpuClass(),
 			TP: int(p.GetTp()), PP: int(p.GetPp()),

internal/control/catalog/catalog.go

@@ -302,6 +302,13 @@ func (m Model) validate() error {
 		}
 		seen[p.Name] = struct{}{}
 
+		for k := range p.Env {
+			if bad, why := engines.UnsafeEnv(k); bad {
+				return fmt.Errorf("model %q profile %q: env %s is not allowed: %s",
+					m.Name, p.Name, k, why)
+			}
+		}
+
 		if p.GPUClass == "" && !cpuCapable(p.Engine) {
 			return fmt.Errorf("model %q profile %q: gpu_class is required for engine %q",
 				m.Name, p.Name, p.Engine)

internal/control/catalog/catalog_test.go

@@ -302,3 +302,22 @@ func TestSimulatedNeedsEveryProfileToBeSimulated(t *testing.T) {
 		})
 	}
 }
+
+func TestAProfileCannotSetWhatRunsCodeInTheEngine(t *testing.T) {
+	// A profile's env reaches an engine started as root. LD_PRELOAD from a
+	// model definition would make "add a model" into "run this everywhere".
+	for _, k := range []string{"LD_PRELOAD", "ld_library_path", "PYTHONSTARTUP", "PATH", "BASH_ENV"} {
+		m := Model{Name: "m", Source: "http://x/m.gguf", Profiles: []Profile{{
+			Name: "cpu", Engine: "llamacpp", Env: map[string]string{k: "/tmp/x"},
+		}}}
+		if _, err := New([]Model{m}); err == nil || !strings.Contains(err.Error(), "not allowed") {
+			t.Errorf("%s: %v", k, err)
+		}
+	}
+	ok := Model{Name: "m", Source: "http://x/m.gguf", Profiles: []Profile{{
+		Name: "cpu", Engine: "llamacpp", Env: map[string]string{"OMP_NUM_THREADS": "4", "PYTHONUNBUFFERED": "1"},
+	}}}
+	if _, err := New([]Model{ok}); err != nil {
+		t.Errorf("ordinary tuning refused: %v", err)
+	}
+}

internal/control/control.go

@@ -4,6 +4,9 @@ package control
 
 import (
 	"context"
+	"crypto/hmac"
+	"crypto/sha256"
+	"encoding/hex"
 	"fmt"
 	"log/slog"
 	"strings"
@@ -168,6 +171,9 @@ func (c *Controller) Nodes() []inventory.Node        { return c.inv.Nodes() }
 func (c *Controller) FreeGPUs(nodeID string) []int   { return c.inv.FreeGPUs(nodeID) }
 func (c *Controller) Replicas() []reconciler.Replica { return c.rec.Replicas() }
 
+// ModelHealth is how each model in the catalog stands.
+func (c *Controller) ModelHealth() []reconciler.ModelHealth { return c.rec.ModelHealth() }
+
 // Reconciler exposes the reconciliation loop, for status reports and ctl.
 func (c *Controller) Reconciler() *reconciler.Reconciler { return c.rec }
 
@@ -304,6 +310,7 @@ func (c *Controller) endpoints() []endpoint.Info {
 			ID:       rep.ID,
 			URL:      strings.TrimRight(rep.Endpoint, "/"),
 			Models:   names,
+			APIKey:   c.engineKey(rep.ID),
 			Engine:   prof.Engine,
 			Pool:     rep.Pool,
 			Weight:   rep.Weight,
@@ -321,3 +328,29 @@ func fingerprint(infos []endpoint.Info) string {
 	}
 	return b.String()
 }
+
+// EngineKey is the key a replica's engine demands, empty when engines run
+// without one.
+func (c *Controller) EngineKey(replicaID string) string { return c.engineKey(replicaID) }
+
+func (c *Controller) engineKey(replicaID string) string {
+	if c.cfg.Reconciler.EngineKey == nil {
+		return ""
+	}
+	return c.cfg.Reconciler.EngineKey(replicaID)
+}
+
+// EngineKeys derives each replica's engine key from one secret.
+//
+// HMAC rather than a stored random key per replica: the router needs the key
+// for every replica it sends to, including ones adopted after a restart that
+// this process never started, and a second control plane has to agree. A
+// function of a shared secret and the replica id gives all three without a
+// table to keep in step. A key taken from one node opens that replica only.
+func EngineKeys(secret []byte) func(replicaID string) string {
+	return func(replicaID string) string {
+		m := hmac.New(sha256.New, secret)
+		m.Write([]byte("llmgw engine key\x00" + replicaID))
+		return "llmgw-" + hex.EncodeToString(m.Sum(nil))[:40]
+	}
+}

internal/control/control_test.go

@@ -0,0 +1,19 @@
+package control
+
+import "testing"
+
+func TestEngineKeysAreTheSameForTheSameReplicaAndNoOther(t *testing.T) {
+	// A control plane that restarts, or a second one, must compute the key a
+	// running engine was started with; a key taken from one node must open
+	// that replica only.
+	a, b := EngineKeys([]byte("s")), EngineKeys([]byte("s"))
+	if a("r1") != b("r1") {
+		t.Error("the same secret gave two keys for one replica")
+	}
+	if a("r1") == a("r2") {
+		t.Error("two replicas share a key")
+	}
+	if a("r1") == EngineKeys([]byte("other"))("r1") {
+		t.Error("the key does not depend on the secret")
+	}
+}

internal/control/fleetmetrics/fleetmetrics.go

@@ -39,9 +39,25 @@ type Source interface {
 // derived from state that already exists, so computing them when asked is both
 // simpler and fresher than maintaining a parallel copy that can drift. Nothing
 // runs between scrapes.
+// HealthSource adds what the reconciler knows about each model and where
+// engine installs stand, and whether this control plane is the one placing
+// work. Asked for by type assertion, so a source without it — a test's —
+// publishes what it did before.
+type HealthSource interface {
+	ModelHealth() []reconciler.ModelHealth
+	IsLeader() bool
+}
+
 type Collector struct {
 	src Source
 
+	modelDesired     *prometheus.Desc
+	modelFailures    *prometheus.Desc
+	modelParked      *prometheus.Desc
+	modelUnplaceable *prometheus.Desc
+	engineInstall    *prometheus.Desc
+	leader           *prometheus.Desc
+
 	nodeUp       *prometheus.Desc
 	nodeGPUs     *prometheus.Desc
 	nodeGPUsFree *prometheus.Desc
@@ -137,7 +153,26 @@ func New(src Source) *Collector {
 		replicaState: prometheus.NewDesc("llmgw_replicas",
 			"Replicas by model and state.", []string{"model", "state"}, nil),
 		replicaReady: prometheus.NewDesc("llmgw_model_replicas_ready",
-			"Replicas of a model that are serving.", []string{"model"}, nil),
+			"Replicas of a model that are serving. Published, as 0 when none are, for every "+
+				"model that should be running; a stopped model, or an on-demand one nobody is "+
+				"asking for, publishes nothing.", []string{"model"}, nil),
+		modelDesired: prometheus.NewDesc("llmgw_model_replicas_desired",
+			"Replicas a model should have now: its minimum, or for an on-demand model 1 while "+
+				"it is being asked for.", []string{"model"}, nil),
+		modelFailures: prometheus.NewDesc("llmgw_model_start_failures",
+			"Replicas of a model that failed to start in a row. Resets when one comes up.",
+			[]string{"model"}, nil),
+		modelParked: prometheus.NewDesc("llmgw_model_parked",
+			"1 when a model failed so often in a row that it is no longer retried.",
+			[]string{"model"}, nil),
+		modelUnplaceable: prometheus.NewDesc("llmgw_model_unplaceable",
+			"1 when a model is short of replicas and the last attempt found no node to place one on.",
+			[]string{"model"}, nil),
+		engineInstall: prometheus.NewDesc("llmgw_node_engine_install",
+			"Where installing an engine stands on a node: 1 for the current state.",
+			[]string{"node", "engine", "version", "state"}, nil),
+		leader: prometheus.NewDesc("llmgw_control_plane_leader",
+			"1 when this control plane is the one placing work, 0 when it is a standby.", nil, nil),
 		replicaIdle: prometheus.NewDesc("llmgw_replica_idle_seconds",
 			"Seconds since a replica was last sent a request. Absent when nothing has been reported, "+
 				"which is not the same as idle.", []string{"model", "replica"}, nil),
@@ -201,6 +236,8 @@ func (c *Collector) Describe(ch chan<- *prometheus.Desc) {
 		c.gpuMemoryTotal, c.gpuMemoryUsed, c.gpuUtil, c.gpuTemp, c.gpuHealthy,
 		c.gpuNVLink, c.gpuECC,
 		c.replicaState, c.replicaReady, c.replicaIdle,
+		c.modelDesired, c.modelFailures, c.modelParked, c.modelUnplaceable,
+		c.engineInstall, c.leader,
 	} {
 		ch <- d
 	}
@@ -387,8 +424,43 @@ func (c *Collector) Collect(ch chan<- prometheus.Metric) {
 		ch <- prometheus.MustNewConstMetric(c.replicaState, prometheus.GaugeValue,
 			float64(v), k[0], k[1])
 	}
-	for model, v := range ready {
-		ch <- prometheus.MustNewConstMetric(c.replicaReady, prometheus.GaugeValue,
-			float64(v), model)
+	hs, ok := c.src.(HealthSource)
+	if !ok {
+		for model, v := range ready {
+			ch <- prometheus.MustNewConstMetric(c.replicaReady, prometheus.GaugeValue,
+				float64(v), model)
+		}
+		return
+	}
+	for _, n := range c.src.Nodes() {
+		for _, p := range n.Provision {
+			ch <- prometheus.MustNewConstMetric(c.engineInstall, prometheus.GaugeValue, 1,
+				n.ID, p.Engine, p.Version, p.State)
+		}
+	}
+	lead := 0.0
+	if hs.IsLeader() {
+		lead = 1
+	}
+	ch <- prometheus.MustNewConstMetric(c.leader, prometheus.GaugeValue, lead)
+	for _, h := range hs.ModelHealth() {
+		// Ready is published at zero for what should be running — the one
+		// value ModelHasNothingServing exists to catch, and which, emitted
+		// only for replicas that were Ready, it never saw.
+		if h.Desired > 0 || ready[h.Model] > 0 {
+			ch <- prometheus.MustNewConstMetric(c.replicaReady, prometheus.GaugeValue,
+				float64(ready[h.Model]), h.Model)
+		}
+		ch <- prometheus.MustNewConstMetric(c.modelDesired, prometheus.GaugeValue, float64(h.Desired), h.Model)
+		ch <- prometheus.MustNewConstMetric(c.modelFailures, prometheus.GaugeValue, float64(h.Failures), h.Model)
+		ch <- prometheus.MustNewConstMetric(c.modelParked, prometheus.GaugeValue, b2f(h.Parked), h.Model)
+		ch <- prometheus.MustNewConstMetric(c.modelUnplaceable, prometheus.GaugeValue, b2f(h.Unplaceable), h.Model)
 	}
 }
+
+func b2f(b bool) float64 {
+	if b {
+		return 1
+	}
+	return 0
+}

internal/control/fleetmetrics/fleetmetrics_test.go

@@ -196,3 +196,35 @@ func TestEveryIBPortIsReportedWithItsCounters(t *testing.T) {
 		}
 	}
 }
+
+type healthSource struct {
+	fakeSource
+	health []reconciler.ModelHealth
+}
+
+func (h healthSource) ModelHealth() []reconciler.ModelHealth { return h.health }
+func (h healthSource) IsLeader() bool                        { return true }
+
+func TestAModelThatShouldRunPublishesZeroReplicasReady(t *testing.T) {
+	// ModelHasNothingServing alerts on llmgw_model_replicas_ready == 0, and
+	// the series was only ever published while a replica was ready — so the
+	// alert could not fire for the case it was written for.
+	src := healthSource{health: []reconciler.ModelHealth{
+		{Model: "wanted", Desired: 1, Failures: 5, Parked: true, Reason: "boom"},
+		{Model: "stopped", Desired: 0},
+	}}
+	out := gather(t, src)
+	for _, want := range []string{
+		"llmgw_model_replicas_ready model=wanted -> 0",
+		"llmgw_model_parked model=wanted -> 1",
+		"llmgw_model_start_failures model=wanted -> 5",
+		"llmgw_control_plane_leader -> 1",
+	} {
+		if !strings.Contains(out, want) {
+			t.Errorf("missing %s", want)
+		}
+	}
+	if strings.Contains(out, "llmgw_model_replicas_ready model=stopped") {
+		t.Error("a stopped model publishes a ready count, so the alert would fire for it")
+	}
+}

internal/control/reconciler/reconciler.go

@@ -3,10 +3,15 @@ package reconciler
 import (
 	"context"
 	crypterand "crypto/rand"
+	"crypto/sha256"
+	"encoding/hex"
+	"encoding/json"
+	"errors"
 	"fmt"
 	"log/slog"
 	"math/rand/v2"
 	"slices"
+	"strings"
 	"sync"
 	"time"
 
@@ -65,10 +70,24 @@ type Config struct {
 	// Without a bound, draining a node whose model cannot be placed anywhere
 	// else would never finish.
 	DrainGrace time.Duration
+	// ParkAfter is how many failures in a row stop a model being retried.
+	//
+	// Backoff alone never gives up: it grows to MaxBackoff and keeps trying,
+	// and a model whose weights its engine cannot load — an architecture the
+	// installed llama.cpp does not know — was started 155 times over eight
+	// hours on the live cluster, each time failing the same way, with nothing
+	// to show for it but the log. A parked model is not placed again until its
+	// definition changes or an operator runs `llmgw ctl retry`.
+	ParkAfter int
 	// PortRange is the per-node port range for engine HTTP and dist-init.
 	PortRange [2]int
 	// MgmtInterface is the ethernet interface used for NCCL bootstrap.
 	MgmtInterface string
+	// EngineKey gives each replica the key its engine demands. Derived from
+	// the replica id rather than stored, so a control plane that restarts —
+	// or a second one — computes the same key for a replica it adopts. Nil
+	// starts engines without one.
+	EngineKey func(replicaID string) string
 	// EvictionGrace is how long a replica must have gone unasked-for before it
 	// may be stopped to make room for another model.
 	//
@@ -120,6 +139,7 @@ func DefaultConfig() Config {
 		EvictionGrace:    2 * time.Minute,
 		MinLifetime:      5 * time.Minute,
 		DrainGrace:       5 * time.Minute,
+		ParkAfter:        5,
 		PortRange:        [2]int{20000, 20999},
 		MgmtInterface:    "eth0",
 	}
@@ -159,6 +179,9 @@ func (c *Config) normalize() {
 	if c.DrainGrace <= 0 {
 		c.DrainGrace = d.DrainGrace
 	}
+	if c.ParkAfter <= 0 {
+		c.ParkAfter = d.ParkAfter
+	}
 	if c.PortRange[1] <= c.PortRange[0] {
 		c.PortRange = d.PortRange
 	}
@@ -199,6 +222,15 @@ type Reconciler struct {
 	// because it is state about this moment and not about what the model is.
 	// A model nobody has wanted has no entry and is simply not running.
 	wanted map[string]time.Time
+	// restartQueue holds, per model, the replicas an operator asked to be
+	// restarted and not yet begun. One is replaced at a time: replacing them
+	// all at once would need the memory of every replica twice over.
+	restartQueue map[string][]string
+	// parked holds models that failed ParkAfter times in a row.
+	parked map[string]parkRecord
+	// placementLogged remembers the last "no placement" said per model, so it
+	// is said when it changes rather than on every five-second pass.
+	placementLogged map[string]placementNote
 
 	wake chan struct{}
 	stop chan struct{}
@@ -220,18 +252,21 @@ func New(cfg Config, cat *catalog.Live, inv *inventory.Store, disp Dispatcher, l
 	}
 	return &Reconciler{
 		cfg: cfg, cat: cat, inv: inv, disp: disp, log: log,
-		sched:         placement.New(inv),
-		now:           time.Now,
-		replicas:      map[string]*Replica{},
-		nextAttempt:   map[string]time.Time{},
-		attempts:      map[string]int{},
-		reservedPorts: map[int]struct{}{},
-		wanted:        map[string]time.Time{},
-		lastExplain:   map[string]placement.Result{},
-		wake:          make(chan struct{}, 1),
-		stop:          make(chan struct{}),
-		done:          make(chan struct{}),
-		startedAt:     time.Now(),
+		sched:           placement.New(inv),
+		now:             time.Now,
+		replicas:        map[string]*Replica{},
+		nextAttempt:     map[string]time.Time{},
+		attempts:        map[string]int{},
+		reservedPorts:   map[int]struct{}{},
+		wanted:          map[string]time.Time{},
+		restartQueue:    map[string][]string{},
+		parked:          map[string]parkRecord{},
+		placementLogged: map[string]placementNote{},
+		lastExplain:     map[string]placement.Result{},
+		wake:            make(chan struct{}, 1),
+		stop:            make(chan struct{}),
+		done:            make(chan struct{}),
+		startedAt:       time.Now(),
 	}
 }
 
@@ -453,6 +488,12 @@ func (r *Reconciler) Want(model string) bool {
 	}
 	r.mu.Lock()
 	r.wanted[m.Name] = r.now()
+	// A placement that failed before this request says nothing about now:
+	// forgotten, so the pass this triggers decides afresh and Unservable
+	// does not answer from a stale result.
+	if res, ok := r.lastExplain[m.Name]; ok && res.Placement == nil {
+		delete(r.lastExplain, m.Name)
+	}
 	r.mu.Unlock()
 	// Immediately, not on the next tick. Someone is waiting on this request,
 	// and five seconds of ticker is five seconds they spend staring at nothing.
@@ -658,6 +699,17 @@ func (r *Reconciler) reapFailed(ctx context.Context) {
 			// replica is about to disappear, and the backoff has to survive it.
 			r.attempts[rep.Model]++
 			r.nextAttempt[rep.Model] = r.now().Add(r.backoff(r.attempts[rep.Model]))
+			if n := r.attempts[rep.Model]; n >= r.cfg.ParkAfter {
+				if _, already := r.parked[rep.Model]; !already {
+					m, _ := r.cat.Load().Resolve(rep.Model)
+					r.parked[rep.Model] = parkRecord{
+						Since: r.now(), Reason: rep.Err, Failures: n, fp: modelFingerprint(m),
+					}
+					r.log.Error("model parked: it failed every time it started, and is not retried "+
+						"until its definition changes or `llmgw ctl retry` is run",
+						"model", rep.Model, "failures", n, "last_error", rep.Err)
+				}
+			}
 		}
 	}
 	r.mu.Unlock()
@@ -703,24 +755,54 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
 	// pointer stops being safe the moment it is in the map: an agent's status
 	// report can mutate it from another goroutine at any time.
 	r.mu.Lock()
-	var liveIDs, drainingIDs, avoid []string
+	var liveIDs, drainingIDs, retiringIDs, avoid []string
+	readyLive := 0
 	for _, rep := range r.replicas {
 		if rep.Model != m.Name {
 			continue
 		}
 		switch {
+		case rep.State.Counts() && !rep.RetiringAt.IsZero():
+			retiringIDs = append(retiringIDs, rep.ID)
+			avoid = append(avoid, rep.NodeIDs()...)
 		case rep.State.Counts():
 			liveIDs = append(liveIDs, rep.ID)
 			avoid = append(avoid, rep.NodeIDs()...)
+			if rep.State == Ready {
+				readyLive++
+			}
 		case rep.State == Draining:
 			drainingIDs = append(drainingIDs, rep.ID)
 		}
 	}
+	// The next restart an operator queued, once the last one is done.
+	if len(retiringIDs) == 0 {
+		if id, ok := r.nextRestartLocked(m.Name); ok {
+			r.replicas[id].RetiringAt = r.now()
+			retiringIDs = append(retiringIDs, id)
+			liveIDs = slices.DeleteFunc(liveIDs, func(x string) bool { return x == id })
+			readyLive--
+			r.log.Info("restarting a replica: placing its replacement first",
+				"replica", id, "model", m.Name)
+		}
+	}
 	blockedUntil := r.nextAttempt[m.Name]
+	park, isParked := r.parked[m.Name]
+	if isParked && park.fp != modelFingerprint(m) {
+		// Somebody changed the model — a new source, an edited profile — so
+		// the reason it kept failing may be gone. Worth one more streak.
+		delete(r.parked, m.Name)
+		r.attempts[m.Name] = 0
+		delete(r.nextAttempt, m.Name)
+		isParked, blockedUntil = false, time.Time{}
+		r.log.Info("model unparked: its definition changed", "model", m.Name)
+	}
 	r.mu.Unlock()
 
 	desired := r.desiredFor(m)
 	switch {
+	case len(liveIDs) < desired && isParked:
+		// Not placed. Its failure is on every status the operator reads.
 	case len(liveIDs) < desired:
 		if r.now().Before(blockedUntil) {
 			// Still in backoff. Rebuilding immediately would hot-loop on a
@@ -758,6 +840,112 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
 			r.stopReplica(ctx, id)
 		}
 	}
+
+	r.retire(ctx, retiringIDs, readyLive, len(liveIDs), desired)
+}
+
+// retire finishes restarting replicas.
+//
+// A retiring replica keeps serving until its replacement is ready, and is then
+// drained like any other. When no replacement could be placed at all —
+// nowhere else has the memory, which on a small CPU fleet is the usual case —
+// it is stopped after DrainGrace so its replacement can take its place: a
+// restart in place, with the model unserved while it loads if it had no other
+// replica. Said in the log, since that is the one case with an outage.
+func (r *Reconciler) retire(ctx context.Context, ids []string, readyLive, live, desired int) {
+	for _, id := range ids {
+		r.mu.Lock()
+		rep, ok := r.replicas[id]
+		var since time.Time
+		if ok {
+			since = rep.RetiringAt
+		}
+		r.mu.Unlock()
+		if !ok {
+			continue
+		}
+		switch {
+		case readyLive >= desired:
+			r.drainForRestart(ctx, id)
+		case live < desired && r.now().Sub(since) > r.cfg.DrainGrace:
+			r.log.Warn("restarting a replica in place: no room for its replacement beside it",
+				"replica", id, "after", r.cfg.DrainGrace)
+			r.drainForRestart(ctx, id)
+			r.stopReplica(ctx, id)
+		}
+	}
+}
+
+func (r *Reconciler) drainForRestart(ctx context.Context, id string) {
+	r.mu.Lock()
+	rep, ok := r.replicas[id]
+	var shards []Shard
+	if ok {
+		rep.State, rep.DrainingAt, rep.RetiringAt = Draining, r.now(), time.Time{}
+		shards = slices.Clone(rep.Shards)
+	}
+	r.mu.Unlock()
+	for _, s := range shards {
+		if err := r.disp.DrainShard(ctx, s.NodeID, s.ID); err != nil {
+			r.log.Warn("draining a shard failed", "shard", s.ID, "err", err)
+		}
+	}
+}
+
+// nextRestartLocked pops the next queued replica of a model that is still
+// ready to be restarted. Called with mu held.
+func (r *Reconciler) nextRestartLocked(model string) (string, bool) {
+	q := r.restartQueue[model]
+	for len(q) > 0 {
+		id := q[0]
+		q = q[1:]
+		if rep, ok := r.replicas[id]; ok && rep.State == Ready && rep.RetiringAt.IsZero() {
+			r.restartQueue[model] = q
+			return id, true
+		}
+	}
+	delete(r.restartQueue, model)
+	return "", false
+}
+
+// ErrNoSuchReplica means no ready replica has that id.
+var ErrNoSuchReplica = errors.New("no ready replica with that id")
+
+// Restart queues replicas to be replaced, each by a new one started beside it
+// before it is stopped — to pick up an edited profile, an engine key, a new
+// engine version. Replicas of one model are replaced one at a time.
+func (r *Reconciler) Restart(ids ...string) error {
+	r.mu.Lock()
+	for _, id := range ids {
+		rep, ok := r.replicas[id]
+		if !ok || rep.State != Ready {
+			r.mu.Unlock()
+			return fmt.Errorf("%w: %s", ErrNoSuchReplica, id)
+		}
+		if !slices.Contains(r.restartQueue[rep.Model], id) && rep.RetiringAt.IsZero() {
+			r.restartQueue[rep.Model] = append(r.restartQueue[rep.Model], id)
+		}
+	}
+	r.mu.Unlock()
+	r.Notify()
+	return nil
+}
+
+// RestartState says where a replica's restart stands: "queued", "replacing",
+// or "" when none was asked for.
+func (r *Reconciler) RestartState(id string) string {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	rep, ok := r.replicas[id]
+	switch {
+	case !ok:
+		return ""
+	case !rep.RetiringAt.IsZero():
+		return "replacing"
+	case slices.Contains(r.restartQueue[rep.Model], id):
+		return "queued"
+	}
+	return ""
 }
 
 // scaleUp places and starts one replica per pass.
@@ -790,7 +978,7 @@ func (r *Reconciler) scaleUp(ctx context.Context, m catalog.Model, avoid []strin
 		if m.Scaling.OnDemand() && r.makeRoomFor(ctx, m) {
 			return
 		}
-		r.log.Warn("no placement for model", "model", m.Name, "reasons", res.Explain())
+		r.notePlacementFailure(m.Name, res.Explain())
 		return
 	}
 	if err := r.startReplica(ctx, m, res.Placement); err != nil {
@@ -883,6 +1071,7 @@ func (r *Reconciler) startReplica(ctx context.Context, m catalog.Model, pl *plac
 				MgmtInterface: r.cfg.MgmtInterface,
 				HTTPPort:      s.HTTPPort,
 				Profile:       espec,
+				APIKey:        r.engineKey(id),
 			},
 			StartupTimeout: prof.StartupTimeout,
 			Pool:           pl.Pool,
@@ -1380,3 +1569,149 @@ func (r *Reconciler) profileMemory(model, profile string) int {
 	}
 	return 0
 }
+
+func (r *Reconciler) engineKey(replicaID string) string {
+	if r.cfg.EngineKey == nil {
+		return ""
+	}
+	return r.cfg.EngineKey(replicaID)
+}
+
+// parkRecord is why a model stopped being retried.
+type parkRecord struct {
+	Since    time.Time
+	Reason   string
+	Failures int
+	// fp is the model's definition when it was parked; a different one
+	// unparks it.
+	fp string
+}
+
+type placementNote struct {
+	text string
+	at   time.Time
+}
+
+// modelFingerprint identifies a model's definition, so a change to it can be
+// told from a retry of the same thing.
+func modelFingerprint(m catalog.Model) string {
+	b, _ := json.Marshal(m)
+	sum := sha256.Sum256(b)
+	return hex.EncodeToString(sum[:8])
+}
+
+// placementLogEvery is how often an unchanged placement failure is repeated.
+const placementLogEvery = 10 * time.Minute
+
+// notePlacementFailure logs why a model could not be placed, when the reason
+// is new or has not been said for a while. Said on every pass it was 250
+// identical lines in two days for one model, burying everything else.
+func (r *Reconciler) notePlacementFailure(model, why string) {
+	r.mu.Lock()
+	last := r.placementLogged[model]
+	say := last.text != why || r.now().Sub(last.at) >= placementLogEvery
+	if say {
+		r.placementLogged[model] = placementNote{text: why, at: r.now()}
+	}
+	r.mu.Unlock()
+	if say {
+		r.log.Warn("no placement for model", "model", model, "reasons", why)
+	}
+}
+
+// Retry clears a parked model and its backoff, so it is placed on the next
+// pass.
+func (r *Reconciler) Retry(model string) bool {
+	r.mu.Lock()
+	_, was := r.parked[model]
+	delete(r.parked, model)
+	r.attempts[model] = 0
+	delete(r.nextAttempt, model)
+	r.mu.Unlock()
+	r.Notify()
+	return was
+}
+
+// ModelHealth is how one model stands, for status and metrics.
+type ModelHealth struct {
+	Model string
+	// Desired is how many replicas it should have now; 0 for a stopped model
+	// or an on-demand one nobody is asking for.
+	Desired int
+	Ready   int
+	// Failures counts replicas that failed in a row.
+	Failures int
+	// Parked is set when Failures reached ParkAfter; Reason is the last error.
+	Parked      bool
+	ParkedSince time.Time
+	Reason      string
+	// Unplaceable is set when it is short of replicas and the last attempt to
+	// place one found nowhere; Placement says why.
+	Unplaceable bool
+	Placement   string
+}
+
+// ModelHealth reports every model in the catalog.
+func (r *Reconciler) ModelHealth() []ModelHealth {
+	models := r.cat.Load().Models()
+	desired := map[string]int{}
+	for _, m := range models {
+		desired[m.Name] = r.desiredFor(m)
+	}
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	ready, live := map[string]int{}, map[string]int{}
+	for _, rep := range r.replicas {
+		if rep.State == Ready {
+			ready[rep.Model]++
+		}
+		if rep.State.Counts() {
+			live[rep.Model]++
+		}
+	}
+	out := make([]ModelHealth, 0, len(models))
+	for _, m := range models {
+		h := ModelHealth{Model: m.Name, Desired: desired[m.Name], Ready: ready[m.Name],
+			Failures: r.attempts[m.Name]}
+		if p, ok := r.parked[m.Name]; ok {
+			h.Parked, h.ParkedSince, h.Reason = true, p.Since, p.Reason
+		}
+		if res, ok := r.lastExplain[m.Name]; ok && res.Placement == nil && live[m.Name] < h.Desired {
+			h.Unplaceable, h.Placement = true, res.Explain()
+		}
+		out = append(out, h)
+	}
+	return out
+}
+
+// Unservable says why a model cannot be served at all, so a request for it is
+// told now rather than asked to retry something that will never come up.
+func (r *Reconciler) Unservable(model string) (string, bool) {
+	m, ok := r.cat.Load().Resolve(model)
+	if !ok {
+		return "", false
+	}
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	if p, ok := r.parked[m.Name]; ok {
+		return fmt.Sprintf("%s failed to start %d times in a row and is not being retried: %s",
+			m.Name, p.Failures, p.Reason), true
+	}
+	for _, rep := range r.replicas {
+		if rep.Model == m.Name && rep.State.Counts() {
+			return "", false // something is on its way
+		}
+	}
+	if res, ok := r.lastExplain[m.Name]; ok && res.Placement == nil {
+		return m.Name + " cannot be placed on any node: " + firstLine(res.Explain()), true
+	}
+	return "", false
+}
+
+func firstLine(s string) string {
+	s = strings.TrimSpace(s)
+	if i := strings.IndexByte(s, '\n'); i >= 0 {
+		return strings.TrimSpace(s[:i]) + " (and others; see llmgw ctl explain-placement)"
+	}
+	return s
+}

internal/control/reconciler/reconciler_test.go

@@ -1,6 +1,7 @@
 package reconciler
 
 import (
+	"bytes"
 	"context"
 	"fmt"
 	"log/slog"
@@ -1721,3 +1722,213 @@ func TestAnAgentTooOldToReportTheProfileVersionIsNotGuessedAt(t *testing.T) {
 		t.Errorf("an unreported fingerprint should stay empty, got %q", fp)
 	}
 }
+
+func TestAnEngineIsStartedWithItsReplicasKey(t *testing.T) {
+	// Engines listen on every address a node has. Started without a key, any
+	// machine that could reach the port reached the model past the gateway's
+	// authentication, quotas and accounting — checked from a neighbouring node
+	// on the live cluster.
+	h := newHarness(t, testModel(1), gpuNode("1", 8))
+	h.r.cfg.EngineKey = func(id string) string { return "key-for-" + id }
+
+	h.r.Reconcile(t.Context())
+	started := h.disp.startedShards()
+	if len(started) == 0 {
+		t.Fatal("nothing was started")
+	}
+	for _, c := range started {
+		if c.Spec.APIKey != "key-for-"+c.ReplicaID {
+			t.Errorf("shard %s started with key %q", c.ShardID, c.Spec.APIKey)
+		}
+	}
+}
+
+func replicaByID(h *harness, id string) (Replica, bool) {
+	for _, rep := range h.r.Replicas() {
+		if rep.ID == id {
+			return rep, true
+		}
+	}
+	return Replica{}, false
+}
+
+func TestARestartedReplicaServesUntilItsReplacementIsReady(t *testing.T) {
+	// Restarting is how a running engine picks up a key, an edited profile or
+	// a new engine version. Stopping first would leave a one-replica model
+	// unserved for as long as its replacement takes to load.
+	h := newHarness(t, testModel(1), gpuNode("1", 4), gpuNode("2", 4))
+	h.r.Reconcile(t.Context())
+	h.markReady()
+	old := h.r.ReadyReplicas()[0].ID
+
+	if err := h.r.Restart(old); err != nil {
+		t.Fatal(err)
+	}
+	h.r.Reconcile(t.Context())
+	if rep, _ := replicaByID(h, old); rep.State != Ready || h.r.RestartState(old) != "replacing" {
+		t.Fatalf("the old replica stopped serving before its replacement was up: %+v", rep)
+	}
+	if n := len(h.r.Replicas()); n != 2 {
+		t.Fatalf("no replacement was placed beside it: %d replicas", n)
+	}
+
+	h.markReady()
+	h.r.Reconcile(t.Context())
+	h.r.Reconcile(t.Context())
+	if _, still := replicaByID(h, old); still {
+		if rep, _ := replicaByID(h, old); rep.State != Draining && rep.State != Stopped {
+			t.Errorf("the old replica is %s after its replacement became ready", rep.State)
+		}
+	}
+	if got := len(h.r.ReadyReplicas()); got != 1 {
+		t.Errorf("%d ready replicas at the end; the model asks for 1", got)
+	}
+}
+
+func TestARestartWithNoRoomBesideItHappensInPlace(t *testing.T) {
+	// One node, room for one replica: the replacement can only start once the
+	// old one has gone. Waiting forever would mean never restarting at all.
+	h := newHarness(t, testModel(1), gpuNode("1", 4))
+	h.r.Reconcile(t.Context())
+	h.markReady()
+	old := h.r.ReadyReplicas()[0].ID
+	if err := h.r.Restart(old); err != nil {
+		t.Fatal(err)
+	}
+	h.r.Reconcile(t.Context())
+	if rep, _ := replicaByID(h, old); rep.State != Ready {
+		t.Fatalf("stopped before the grace period: %s", rep.State)
+	}
+
+	h.advance(h.r.cfg.DrainGrace + time.Second)
+	h.r.Reconcile(t.Context())
+	if rep, ok := replicaByID(h, old); ok && rep.State == Ready {
+		t.Fatal("still serving after the grace period with no replacement possible")
+	}
+	for i := 0; i < 3; i++ {
+		h.r.Reconcile(t.Context())
+		h.markReady()
+	}
+	ready := h.r.ReadyReplicas()
+	if len(ready) != 1 || ready[0].ID == old {
+		t.Errorf("expected one new replica in its place, got %+v", ready)
+	}
+}
+
+func TestReplicasOfAModelAreRestartedOneAtATime(t *testing.T) {
+	h := newHarness(t, testModel(2), gpuNode("1", 8), gpuNode("2", 8), gpuNode("3", 8))
+	for i := 0; i < 3; i++ {
+		h.r.Reconcile(t.Context())
+		h.markReady()
+	}
+	var ids []string
+	for _, rep := range h.r.ReadyReplicas() {
+		ids = append(ids, rep.ID)
+	}
+	if err := h.r.Restart(ids...); err != nil {
+		t.Fatal(err)
+	}
+	h.r.Reconcile(t.Context())
+	states := []string{h.r.RestartState(ids[0]), h.r.RestartState(ids[1])}
+	if !slices.Contains(states, "replacing") || !slices.Contains(states, "queued") {
+		t.Errorf("expected one replacing and one queued, got %v", states)
+	}
+	if err := h.r.Restart("no-such"); err == nil {
+		t.Error("an unknown replica was accepted")
+	}
+}
+
+// failRepeatedly starts and fails the model's replica n times over.
+func failRepeatedly(t *testing.T, h *harness, n int, why string) {
+	t.Helper()
+	for i := 0; i < n; i++ {
+		h.disp.reset()
+		h.r.Reconcile(t.Context())
+		started := h.disp.startedShards()
+		if len(started) == 0 {
+			return // parked, or still in backoff
+		}
+		s := started[0]
+		h.r.OnShardStatus(s.ReplicaID, s.ShardID, supervisor.Failed, why)
+		h.r.Reconcile(t.Context())
+		h.advance(DefaultConfig().MaxBackoff + time.Second)
+	}
+}
+
+func TestAModelThatAlwaysFailsIsParkedNotRetriedForever(t *testing.T) {
+	// kev-0.5b: an architecture the node's llama.cpp did not know, started
+	// 155 times in eight hours, the same error every time.
+	h := newHarness(t, testModel(1), gpuNode("1", 8))
+	failRepeatedly(t, h, DefaultConfig().ParkAfter, "unknown model architecture: 'ggmlc'")
+
+	h.disp.reset()
+	h.advance(DefaultConfig().MaxBackoff + time.Second)
+	h.r.Reconcile(t.Context())
+	if n := len(h.disp.startedShards()); n != 0 {
+		t.Fatalf("a parked model was started again (%d shards)", n)
+	}
+	var mh ModelHealth
+	for _, x := range h.r.ModelHealth() {
+		if x.Model == "llama" {
+			mh = x
+		}
+	}
+	if !mh.Parked || !strings.Contains(mh.Reason, "ggmlc") || mh.Failures < DefaultConfig().ParkAfter {
+		t.Errorf("health %+v", mh)
+	}
+	if why, no := h.r.Unservable("llama"); !no || !strings.Contains(why, "ggmlc") {
+		t.Errorf("a request would not be told why: %q %v", why, no)
+	}
+
+	// Retried by the operator once fixed.
+	if !h.r.Retry("llama") {
+		t.Error("Retry did not report the model as parked")
+	}
+	h.r.Reconcile(t.Context())
+	if len(h.disp.startedShards()) == 0 {
+		t.Error("a retried model was not placed")
+	}
+}
+
+func TestChangingAParkedModelsDefinitionTriesItAgain(t *testing.T) {
+	h := newHarness(t, testModel(1), gpuNode("1", 8))
+	failRepeatedly(t, h, DefaultConfig().ParkAfter, "boom")
+
+	fixed := testModel(1)
+	fixed.Source = "/models/llama-fixed"
+	cat, err := catalog.New([]catalog.Model{fixed})
+	if err != nil {
+		t.Fatal(err)
+	}
+	h.r.cat.Store(cat)
+	h.disp.reset()
+	h.r.Reconcile(t.Context())
+	if len(h.disp.startedShards()) == 0 {
+		t.Error("a model whose definition changed stayed parked")
+	}
+}
+
+func TestAPlacementFailureIsLoggedWhenItChangesNotEveryPass(t *testing.T) {
+	var buf bytes.Buffer
+	h := newHarness(t, testModel(1), gpuNode("1", 1)) // tp4 needs 4 cards
+	h.r.log = slog.New(slog.NewTextHandler(&buf, nil))
+	for i := 0; i < 20; i++ {
+		h.r.Reconcile(t.Context())
+		h.advance(5 * time.Second)
+	}
+	if n := strings.Count(buf.String(), "no placement for model"); n != 1 {
+		t.Errorf("logged %d times over 20 passes", n)
+	}
+	h.advance(placementLogEvery)
+	h.r.Reconcile(t.Context())
+	if n := strings.Count(buf.String(), "no placement for model"); n != 2 {
+		t.Errorf("not repeated after %s: %d", placementLogEvery, n)
+	}
+	var mh ModelHealth
+	for _, x := range h.r.ModelHealth() {
+		mh = x
+	}
+	if !mh.Unplaceable || mh.Desired != 1 {
+		t.Errorf("health %+v", mh)
+	}
+}

internal/control/reconciler/replica.go

@@ -156,6 +156,10 @@ type Replica struct {
 	// DrainingAt is when the replica stopped taking new work, so the wait for a
 	// replacement can be bounded.
 	DrainingAt time.Time
+	// RetiringAt is set while the replica is being restarted: it keeps serving
+	// but no longer counts toward the model's replicas, so a replacement is
+	// placed beside it, and it is drained only once that replacement is ready.
+	RetiringAt time.Time
 	// Attempt counts how many times this model has failed to come up in a row,
 	// carried across rebuilds so the backoff does not reset on every attempt.
 	Attempt int

internal/mirror/add.go

@@ -7,7 +7,6 @@ import (
 	"net/http"
 	"os"
 	"path/filepath"
-	"strings"
 	"time"
 )
 
@@ -29,13 +28,22 @@ type Added struct {
 // the layout — a version that can be overwritten takes away the ability to
 // change the catalog back and still have the old weights.
 func Add(ctx context.Context, root, source string, e Entry, progress func(done, total int64)) (Added, error) {
+	return addWith(ctx, http.DefaultClient, root, source, e, progress)
+}
+
+// addWith is Add through a given client. The management API passes one that
+// will not reach private addresses; `llmgw mirror add`, run by somebody on
+// the mirror, passes the ordinary one.
+func addWith(ctx context.Context, client *http.Client, root, source string, e Entry,
+	progress func(done, total int64),
+) (Added, error) {
 	if e.Model == "" || e.Version == "" || e.File == "" {
 		return Added{}, fmt.Errorf("a model, a version and a filename are all required")
 	}
-	if strings.ContainsAny(e.Model+e.Version+e.File, `/\`) {
-		// Each is one path element. A slash would put the file somewhere the
-		// layout does not describe, and List would not find it again.
-		return Added{}, fmt.Errorf("model, version and filename are single names, not paths")
+	for _, n := range []string{e.Model, e.Version, e.File} {
+		if err := ValidName(n); err != nil {
+			return Added{}, err
+		}
 	}
 
 	final := e.Path(root)
@@ -50,7 +58,7 @@ func Add(ctx context.Context, root, source string, e Entry, progress func(done,
 	if err != nil {
 		return Added{}, err
 	}
-	resp, err := http.DefaultClient.Do(req)
+	resp, err := client.Do(req)
 	if err != nil {
 		return Added{}, fmt.Errorf("fetching %s: %w", source, err)
 	}

internal/mirror/api.go

@@ -130,8 +130,8 @@ func (a *API) status(w http.ResponseWriter, r *http.Request) {
 
 func (a *API) remove(w http.ResponseWriter, r *http.Request) {
 	model, version := r.PathValue("model"), r.PathValue("version")
-	if strings.ContainsAny(model+version, `/\`) {
-		fail(w, http.StatusBadRequest, errors.New("model and version are single names, not paths"))
+	if err := errors.Join(ValidName(model), ValidName(version)); err != nil {
+		fail(w, http.StatusBadRequest, err)
 		return
 	}
 

internal/mirror/api_test.go

@@ -17,7 +17,10 @@ import (
 func testAPI(t *testing.T) (*API, string) {
 	t.Helper()
 	root := t.TempDir()
-	a := NewAPI(NewManager(root, discardLog()), "http://mirror:8081")
+	m := NewManager(root, discardLog())
+	// The upstreams in these tests are httptest servers on loopback.
+	m.AllowPrivateSources()
+	a := NewAPI(m, "http://mirror:8081")
 
 	// A download outlives the request that started it — that is the point of
 	// the manager. So the temp directory must not be torn down while one is
@@ -434,3 +437,33 @@ func TestRemovingAVersionIsRefusedWhileAnyShardIsArriving(t *testing.T) {
 		t.Errorf("expected 409 while a shard is arriving, got %d", code)
 	}
 }
+
+func TestTheAPIWillNotFetchFromThePrivateNetwork(t *testing.T) {
+	// The file server hands whatever was fetched to anyone, so a fetch that
+	// could reach 169.254.169.254 or a database on the LAN was a way to read
+	// them through the mirror.
+	m := NewManager(t.TempDir(), discardLog())
+	for _, src := range []string{
+		"http://127.0.0.1:1/x.gguf", "http://169.254.169.254/latest/meta-data",
+		"http://10.0.0.140:9090/api/v1/nodes", "http://[::1]:1/x",
+	} {
+		st, err := m.Start(src, Entry{Model: "m", Version: "v", File: "x.gguf"})
+		if err != nil {
+			t.Fatalf("%s: %v", src, err)
+		}
+		deadline := time.Now().Add(5 * time.Second)
+		for st.State == Fetching && time.Now().Before(deadline) {
+			time.Sleep(10 * time.Millisecond)
+			st, _ = m.Status("m", "v")
+		}
+		if st.State != Failed || !strings.Contains(st.Err, "refusing to fetch") {
+			t.Errorf("%s: %+v", src, st)
+		}
+		m.Forget("m", "v")
+	}
+	for _, src := range []string{"file:///etc/passwd", "gopher://x/", "not a url"} {
+		if _, err := m.Start(src, Entry{Model: "m", Version: "v", File: "x"}); err == nil {
+			t.Errorf("%s was accepted as a source", src)
+		}
+	}
+}

internal/mirror/manager.go

@@ -5,6 +5,7 @@ import (
 	"errors"
 	"fmt"
 	"log/slog"
+	"net/http"
 	"os"
 	"sync"
 	"time"
@@ -72,8 +73,18 @@ type Manager struct {
 	// until the first had finished. A fifteen-shard model had to be pulled one
 	// shard at a time, each waiting on the last.
 	running map[string]*Status
+
+	// client fetches sources. It will not reach private addresses unless
+	// AllowPrivateSources was called.
+	client *http.Client
 }
 
+// AllowPrivateSources lets downloads reach loopback, private and link-local
+// addresses — a model server on the same network, say. Off by default: the
+// file server hands whatever was fetched to anyone, so a fetch that can reach
+// the private network is a way to read it.
+func (m *Manager) AllowPrivateSources() { m.client = fetchClient(true) }
+
 // runKey identifies one download.
 func runKey(e Entry) string { return e.Model + "/" + e.Version + "/" + e.File }
 
@@ -82,7 +93,7 @@ func NewManager(root string, log *slog.Logger) *Manager {
 	if log == nil {
 		log = slog.Default()
 	}
-	return &Manager{root: root, log: log, running: map[string]*Status{}}
+	return &Manager{root: root, log: log, running: map[string]*Status{}, client: fetchClient(false)}
 }
 
 // ErrAlreadyFetching means a download for that model and version is running.
@@ -97,6 +108,9 @@ func (m *Manager) Start(source string, e Entry) (Status, error) {
 	if source == "" {
 		return Status{}, fmt.Errorf("a source URL is required")
 	}
+	if err := validSource(source); err != nil {
+		return Status{}, err
+	}
 	key := runKey(e)
 
 	m.mu.Lock()
@@ -125,7 +139,7 @@ func (m *Manager) run(source string, e Entry, key string) {
 	// minutes to hours and the thing that asked for it is long gone.
 	ctx := context.Background()
 
-	added, err := Add(ctx, m.root, source, e, func(done, total int64) {
+	added, err := addWith(ctx, m.client, m.root, source, e, func(done, total int64) {
 		m.mu.Lock()
 		if st, ok := m.running[key]; ok {
 			st.BytesDone, st.BytesTotal = done, total

internal/mirror/mirror.go

@@ -37,6 +37,21 @@ type Entry struct {
 	Bytes   int64
 }
 
+// ValidName checks one element of an entry — a model, version or file name.
+//
+// Each is one path element, and a name that is not one writes somewhere the
+// layout does not describe. Rejecting slashes was not enough: ".." is a
+// single element too, and a model called ".." put its file beside the mirror
+// root rather than inside it. A name starting with a dot is refused as well,
+// because that is where the mirror keeps its own bookkeeping.
+func ValidName(n string) error {
+	if n == "" || n[0] == '.' || strings.ContainsAny(n, "/\\\x00") {
+		return fmt.Errorf("%q is not a usable name: it must be a single path element "+
+			"that does not start with a dot", n)
+	}
+	return nil
+}
+
 // Path is where this entry lives under a root.
 func (e Entry) Path(root string) string {
 	return filepath.Join(root, e.Model, e.Version, e.File)

internal/mirror/mirror_test.go

@@ -162,3 +162,18 @@ func TestProgressIsReportedForALongDownload(t *testing.T) {
 		t.Error("a download reported no progress at all")
 	}
 }
+
+func TestANameThatLeavesTheMirrorIsRefused(t *testing.T) {
+	root := t.TempDir()
+	for _, e := range []Entry{
+		{Model: "..", Version: "v", File: "f.gguf"},
+		{Model: "m", Version: "..", File: "f.gguf"},
+		{Model: "m", Version: "v", File: ".f.sha256"},
+		{Model: "a/b", Version: "v", File: "f.gguf"},
+	} {
+		if _, err := Add(t.Context(), root, "http://127.0.0.1:1/x", e, nil); err == nil ||
+			!strings.Contains(err.Error(), "not a usable name") {
+			t.Errorf("%+v: %v", e, err)
+		}
+	}
+}

internal/mirror/safefetch.go

@@ -0,0 +1,86 @@
+package mirror
+
+import (
+	"errors"
+	"fmt"
+	"net"
+	"net/http"
+	"net/netip"
+	"net/url"
+	"syscall"
+	"time"
+)
+
+// errPrivateSource is a fetch refused because it would reach this machine or
+// the network behind it rather than the internet.
+var errPrivateSource = errors.New("refusing to fetch from a loopback, private or link-local address")
+
+// fetchClient downloads sources for the management API.
+//
+// The API fetches whatever URL it is given and the file server then hands the
+// result to anyone. Without this, "fetch http://169.254.169.254/..." was a way
+// to read a cloud's instance metadata — or anything else on the network the
+// mirror sits in — through the mirror. The check is made on the address
+// actually dialled, after resolution and on every redirect, so a name that
+// resolves somewhere public and then somewhere private is caught too.
+func fetchClient(allowPrivate bool) *http.Client {
+	d := &net.Dialer{Timeout: 30 * time.Second, KeepAlive: 30 * time.Second}
+	if !allowPrivate {
+		d.Control = func(_, address string, _ syscall.RawConn) error {
+			host, _, err := net.SplitHostPort(address)
+			if err != nil {
+				return err
+			}
+			a, err := netip.ParseAddr(host)
+			if err != nil {
+				return err
+			}
+			if blockedAddr(a.Unmap()) {
+				return fmt.Errorf("%w (%s)", errPrivateSource, a)
+			}
+			return nil
+		}
+	}
+	return &http.Client{
+		Transport: &http.Transport{
+			Proxy:                 http.ProxyFromEnvironment,
+			DialContext:           d.DialContext,
+			TLSHandshakeTimeout:   30 * time.Second,
+			ResponseHeaderTimeout: 2 * time.Minute,
+			ForceAttemptHTTP2:     true,
+		},
+		// No overall timeout: a model is tens of gigabytes. The dial, the
+		// handshake and the first byte are bounded above; after that a stall
+		// is the context's to end.
+		CheckRedirect: func(req *http.Request, via []*http.Request) error {
+			if len(via) >= 10 {
+				return errors.New("too many redirects")
+			}
+			return checkScheme(req.URL)
+		},
+	}
+}
+
+func blockedAddr(a netip.Addr) bool {
+	cgnat := netip.MustParsePrefix("100.64.0.0/10")
+	return a.IsLoopback() || a.IsPrivate() || a.IsLinkLocalUnicast() ||
+		a.IsLinkLocalMulticast() || a.IsMulticast() || a.IsUnspecified() || cgnat.Contains(a)
+}
+
+// checkScheme allows the two schemes a model is published over. file:// and
+// the rest would read this machine's own disk.
+func checkScheme(u *url.URL) error {
+	if u.Scheme != "http" && u.Scheme != "https" {
+		return fmt.Errorf("%q is not an http or https source", u.Redacted())
+	}
+	return nil
+}
+
+// validSource checks a source before a download is started for it.
+func validSource(source string) error {
+	u, err := url.Parse(source)
+	if err != nil || u.Host == "" {
+		return fmt.Errorf("%q is not a URL", source)
+	}
+	return checkScheme(u)
+}

internal/onboard/onboard.go

@@ -332,7 +332,10 @@ func certNames(path string) ([]string, error) {
 
 // --- binary ---
 
-var releaseTag = regexp.MustCompile(`^v\d+\.\d+\.\d+$`)
+// releaseTag is a version CI published: v0.10.4, or a release candidate of
+// it, v0.10.4-rc.2. Anything else — a build with commits past its tag — has no
+// release to download.
+var releaseTag = regexp.MustCompile(`^v\d+\.\d+\.\d+(-rc\.\d+)?$`)
 
 func (r *Runner) binary(ctx context.Context, res *Result) error {
 	v := r.Opts.Version

internal/onboard/releasetag_test.go

@@ -0,0 +1,14 @@
+package onboard
+
+import "testing"
+
+func TestAReleaseCandidateIsARelease(t *testing.T) {
+	for v, want := range map[string]bool{
+		"v0.10.4": true, "v0.10.4-rc.1": true, "v0.10.4-rc.12": true,
+		"v0.10.3-5-gad6d2d3": false, "v0.10.4-rc": false, "v0.10.3-a": false, "dev": false,
+	} {
+		if got := releaseTag.MatchString(v); got != want {
+			t.Errorf("%s: %v", v, got)
+		}
+	}
+}

internal/router/activate.go

@@ -17,6 +17,22 @@ type Activator interface {
 	Want(model string) bool
 }
 
+// unservable is an activator that can also say a model will not come up at
+// all — it failed every time it started, or fits on no node — so a request is
+// told so at once rather than asked to retry something that never arrives.
+type unservable interface {
+	Unservable(model string) (string, bool)
+}
+
+// whyUnservable is the activator's reason a model cannot be served, if any.
+func (r *Router) whyUnservable(model string) (string, bool) {
+	u, ok := r.activator.(unservable)
+	if !ok {
+		return "", false
+	}
+	return u.Unservable(model)
+}
+
 // SetActivator supplies the control plane that can load models on demand.
 func (r *Router) SetActivator(a Activator) { r.activator = a }
 
@@ -32,6 +48,8 @@ func (r *Router) SetActivator(a Activator) { r.activator = a }
 // a slow request. A fifteen-minute wait is a job pretending to be one, and the
 // client should be told to come back rather than left holding a socket.
 func (r *Router) activate(ctx context.Context, model string) bool {
+	// Asked first, always: asking is what makes the control plane try again,
+	// and a model that fitted nowhere an hour ago may fit now.
 	if r.activator == nil || !r.activator.Want(model) {
 		return false
 	}
@@ -56,6 +74,11 @@ func (r *Router) activate(ctx context.Context, model string) bool {
 			if len(r.reg.Snapshots(model)) > 0 {
 				return true
 			}
+			// Asking set a placement going; if it found nowhere, there is
+			// nothing to wait for.
+			if _, no := r.whyUnservable(model); no {
+				return false
+			}
 		}
 	}
 }

internal/router/adminguard.go

@@ -0,0 +1,75 @@
+package router
+
+import (
+	"net"
+	"net/http"
+	"net/netip"
+	"net/url"
+	"strings"
+)
+
+// AdminGuard keeps a browser from being used against the admin listener.
+//
+// The listener has no login: reaching it is meant to mean holding the machine,
+// so it binds to loopback and an operator reaches it over an ssh tunnel. But a
+// browser on that operator's machine reaches it too, from any page it has
+// open. A page could POST a model whose profile runs as root on every node,
+// or — by pointing its own domain at 127.0.0.1 — read pprof's heap, which
+// holds API keys in the clear. Three checks close that without touching what
+// ctl and curl do:
+//
+//   - Host must be an address or localhost. A page that rebinds its own name
+//     to 127.0.0.1 still sends that name.
+//   - A request carrying Origin or Sec-Fetch-Site from another site is
+//     refused. Browsers send them; ctl and curl do not.
+//   - Anything but GET and HEAD must be application/json. A page can send a
+//     form or text/plain anywhere without asking; JSON across origins needs a
+//     preflight this listener never grants.
+func AdminGuard(next http.Handler) http.Handler {
+	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		if !addressHost(r.Host) {
+			http.Error(w, "the admin listener answers to an address or localhost, not a name", http.StatusMisdirectedRequest)
+			return
+		}
+		if crossSite(r) {
+			http.Error(w, "cross-site request refused", http.StatusForbidden)
+			return
+		}
+		if r.Method != http.MethodGet && r.Method != http.MethodHead {
+			ct := r.Header.Get("Content-Type")
+			if mt, _, _ := strings.Cut(ct, ";"); strings.TrimSpace(strings.ToLower(mt)) != "application/json" {
+				http.Error(w, "a change to the admin API must be sent as application/json", http.StatusUnsupportedMediaType)
+				return
+			}
+		}
+		next.ServeHTTP(w, r)
+	})
+}
+
+func addressHost(hostport string) bool {
+	host := hostport
+	if h, _, err := net.SplitHostPort(hostport); err == nil {
+		host = h
+	}
+	host = strings.Trim(host, "[]")
+	if strings.EqualFold(host, "localhost") {
+		return true
+	}
+	_, err := netip.ParseAddr(host)
+	return err == nil
+}
+
+func crossSite(r *http.Request) bool {
+	switch r.Header.Get("Sec-Fetch-Site") {
+	case "", "same-origin", "none":
+	default:
+		return true
+	}
+	if o := r.Header.Get("Origin"); o != "" {
+		u, err := url.Parse(o)
+		if err != nil || !strings.EqualFold(u.Host, r.Host) {
+			return true
+		}
+	}
+	return false
+}

internal/router/adminguard_test.go

@@ -0,0 +1,57 @@
+package router
+
+import (
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"testing"
+)
+
+func TestTheAdminListenerRefusesWhatABrowserPageCouldSend(t *testing.T) {
+	h := AdminGuard(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {}))
+	do := func(method, host, ct string, hdr map[string]string) int {
+		r := httptest.NewRequest(method, "/api/v1/models", strings.NewReader("{}"))
+		r.Host = host
+		if ct != "" {
+			r.Header.Set("Content-Type", ct)
+		}
+		for k, v := range hdr {
+			r.Header.Set(k, v)
+		}
+		w := httptest.NewRecorder()
+		h.ServeHTTP(w, r)
+		return w.Code
+	}
+
+	// What ctl and curl send.
+	if got := do("POST", "127.0.0.1:9090", "application/json", nil); got != 200 {
+		t.Errorf("ctl's request was refused: %d", got)
+	}
+	if got := do("GET", "localhost:9090", "", nil); got != 200 {
+		t.Errorf("a GET through an ssh tunnel was refused: %d", got)
+	}
+	if got := do("DELETE", "[::1]:9090", "application/json", nil); got != 200 {
+		t.Errorf("IPv6 loopback refused: %d", got)
+	}
+
+	// A page posting a form, or text/plain, needs no preflight.
+	if got := do("POST", "127.0.0.1:9090", "text/plain", nil); got != http.StatusUnsupportedMediaType {
+		t.Errorf("text/plain POST accepted: %d", got)
+	}
+	if got := do("POST", "127.0.0.1:9090", "", nil); got != http.StatusUnsupportedMediaType {
+		t.Errorf("a bodiless POST accepted: %d", got)
+	}
+	// DNS rebinding: the page's own name, now resolving to 127.0.0.1.
+	if got := do("GET", "evil.example:9090", "", nil); got != http.StatusMisdirectedRequest {
+		t.Errorf("a rebound name was answered: %d", got)
+	}
+	// And anything a browser marks as coming from elsewhere.
+	if got := do("POST", "127.0.0.1:9090", "application/json",
+		map[string]string{"Sec-Fetch-Site": "cross-site"}); got != http.StatusForbidden {
+		t.Errorf("cross-site fetch accepted: %d", got)
+	}
+	if got := do("POST", "127.0.0.1:9090", "application/json",
+		map[string]string{"Origin": "https://evil.example"}); got != http.StatusForbidden {
+		t.Errorf("foreign Origin accepted: %d", got)
+	}
+}

internal/router/endpoint/endpoint.go

@@ -22,6 +22,9 @@ type Info struct {
 	ID string
 	// URL is the engine's root address (e.g. http://gpu-a-01:8000), no trailing slash.
 	URL string
+	// APIKey is what the engine demands of every request, sent as a bearer
+	// token. Empty for an engine that asks for none.
+	APIKey string
 	// Models lists the model names this replica serves.
 	Models []string
 	// Engine names the inference engine serving here, which decides what its

internal/router/proxy/proxy.go

@@ -148,6 +148,8 @@ type Request struct {
 	Body []byte
 	// Header carries headers to forward (authorization is deliberately not).
 	Header http.Header
+	// APIKey is the engine's own key, sent in place of the client's.
+	APIKey string
 	// Stream says whether an SSE response is expected.
 	Stream bool
 	// SuppressUsageChunk drops the final usage-only chunk from the client while
@@ -218,6 +220,9 @@ func (p *Proxy) Do(ctx context.Context, w http.ResponseWriter, r Request) (Outco
 	}
 	req.Header = cloneForUpstream(r.Header)
 	req.Header.Set("Content-Type", "application/json")
+	if r.APIKey != "" {
+		req.Header.Set("Authorization", "Bearer "+r.APIKey)
+	}
 	req.ContentLength = int64(len(r.Body))
 
 	// A non-streaming request waits for the whole completion before a single
@@ -518,8 +523,7 @@ func copyHeaders(dst, src http.Header) {
 
 // cloneForUpstream forwards only what the engine needs. The client's
 // Authorization header is deliberately dropped: it authenticates the caller to
-// llmgw, not llmgw to the engine, and engines are reached over a trusted
-// network (plan §14).
+// llmgw, not llmgw to the engine. llmgw's own key for the engine is set by Do.
 func cloneForUpstream(h http.Header) http.Header {
 	out := make(http.Header, 4)
 	for _, k := range []string{"Accept", "User-Agent", "Traceparent", "Tracestate"} {

internal/router/proxy/proxy_test.go

@@ -355,3 +355,22 @@ func TestAStreamingRequestKeepsTheShortLivenessTimeout(t *testing.T) {
 		t.Errorf("took %s; the streaming liveness timeout is not being applied", elapsed)
 	}
 }
+
+func TestTheEngineGetsLlmgwsKeyNotTheClients(t *testing.T) {
+	var got string
+	up := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		got = r.Header.Get("Authorization")
+		io.WriteString(w, `{}`)
+	}))
+	defer up.Close()
+
+	client := http.Header{"Authorization": {"Bearer sk-the-callers-own"}}
+	_, err := testProxy().Do(t.Context(), httptest.NewRecorder(),
+		Request{URL: up.URL, Body: []byte(`{}`), Header: client, APIKey: "llmgw-engine"})
+	if err != nil {
+		t.Fatal(err)
+	}
+	if got != "Bearer llmgw-engine" {
+		t.Errorf("the engine was sent %q", got)
+	}
+}

internal/router/router.go

@@ -373,6 +373,7 @@ func toInfos(eps []config.Endpoint) []endpoint.Info {
 		out = append(out, endpoint.Info{
 			ID:       e.ID,
 			URL:      strings.TrimRight(e.URL, "/"),
+			APIKey:   e.APIKey,
 			Models:   e.Models,
 			Engine:   e.Engine,
 			Pool:     e.Pool,
@@ -628,6 +629,11 @@ func (r *Router) dispatchOne(ctx context.Context, w http.ResponseWriter, req *pa
 		}
 	}
 	if len(candidates) == 0 {
+		if why, no := r.whyUnservable(model); no && r.activator != nil {
+			// Not "loading; retry": it will not load, and a client told to
+			// retry keeps retrying something that never arrives.
+			return dispatchResult{err: fmt.Errorf("%w: %s", errNoCapacity, why)}
+		}
 		if r.activator != nil && r.activator.Want(model) {
 			// Asked for, still loading. Saying "no endpoint serves this" would
 			// be a lie, and a client told to retry will succeed where one told
@@ -673,6 +679,7 @@ func (r *Router) dispatchOne(ctx context.Context, w http.ResponseWriter, req *pa
 
 		out, perr := r.px.Do(ctx, w, proxy.Request{
 			URL:                dec.Endpoint.URL + req.Path,
+			APIKey:             dec.Endpoint.APIKey,
 			Body:               req.UpstreamBody,
 			Header:             req.Header,
 			Stream:             req.Stream,

internal/router/router_test.go

@@ -619,3 +619,39 @@ func TestTheServedModelHeaderArrivesBeforeTheBody(t *testing.T) {
 		t.Errorf("served-model header = %q, want backup", got)
 	}
 }
+
+type unservableActivator struct {
+	fakeActivator
+	why map[string]string
+}
+
+func (u *unservableActivator) Unservable(model string) (string, bool) {
+	w, ok := u.why[model]
+	return w, ok
+}
+
+func TestAModelThatCannotComeUpIsSaidSoAtOnce(t *testing.T) {
+	// A model that fits on no node, or failed every time it started, was
+	// answered "loading; retry shortly" — and a client that retried did so
+	// forever.
+	e := newFakeEngine(t, "m")
+	cfg := baseConfig(config.Endpoint{ID: "a", URL: e.srv.URL, Models: []string{"m"}})
+	cfg.Router.ActivationWait = 10 * time.Second
+	rt := newTestRouter(t, cfg)
+	rt.SetActivator(&unservableActivator{
+		fakeActivator: fakeActivator{onDemand: map[string]bool{"phi-4": true}},
+		why:           map[string]string{"phi-4": "phi-4 cannot be placed on any node: needs 15 GB"},
+	})
+
+	start := time.Now()
+	rec := post(t, rt.Handler(), map[string]any{"model": "phi-4", "messages": []any{}}, nil)
+	if took := time.Since(start); took > 2*time.Second {
+		t.Errorf("the request waited %s for a model that will not load", took)
+	}
+	if rec.Code != http.StatusServiceUnavailable || rec.Header().Get("Retry-After") != "" {
+		t.Errorf("status %d, Retry-After %q", rec.Code, rec.Header().Get("Retry-After"))
+	}
+	if !strings.Contains(rec.Body.String(), "needs 15 GB") {
+		t.Errorf("the reason was not given: %s", rec.Body.String())
+	}
+}

internal/router/scrape/poller.go

@@ -157,7 +157,7 @@ func (p *Poller) loop(ctx context.Context, in endpoint.Info) {
 }
 
 func (p *Poller) once(ctx context.Context, in endpoint.Info, reported *string) {
-	stats, err := p.fetch(ctx, in.URL, p.mappingFor(in))
+	stats, err := p.fetch(ctx, in.URL, in.APIKey, p.mappingFor(in))
 	if err != nil {
 		if ctx.Err() != nil {
 			return
@@ -201,7 +201,7 @@ func (p *Poller) report(in endpoint.Info, err error, reported *string) {
 	*reported = "unreachable"
 }
 
-func (p *Poller) fetch(ctx context.Context, base string, mapping Mapping) (endpoint.Stats, error) {
+func (p *Poller) fetch(ctx context.Context, base, apiKey string, mapping Mapping) (endpoint.Stats, error) {
 	ctx, cancel := context.WithTimeout(ctx, p.cfg.Timeout)
 	defer cancel()
 
@@ -209,6 +209,10 @@ func (p *Poller) fetch(ctx context.Context, base string, mapping Mapping) (endpo
 	if err != nil {
 		return endpoint.Stats{}, err
 	}
+	// llama.cpp guards /metrics with its key; vLLM does not, and ignores one.
+	if apiKey != "" {
+		req.Header.Set("Authorization", "Bearer "+apiKey)
+	}
 	resp, err := p.http.Do(req)
 	if err != nil {
 		return endpoint.Stats{}, err

internal/store/copy.go

@@ -38,6 +38,10 @@ var copyOrder = []string{
 	"threads",
 	"thread_messages",
 	"memory_notes",
+	// Copied, not regenerated: every running engine was started with a key
+	// derived from this, and a new secret would lock the router out of all
+	// of them at the moment of the move.
+	"cluster_secrets",
 }
 
 // notCopied are tables deliberately left behind.

internal/store/migrations/postgres/0011_cluster_secrets.sql

@@ -0,0 +1,12 @@
+-- Secrets the control planes share and nobody types: generated on first use,
+-- read by every control plane against this store afterwards.
+--
+-- The first is the one each replica's engine key is derived from. It has to be
+-- the same for every control plane — the router sends a replica the key it was
+-- started with, whichever control plane started it — and the same after a
+-- restart, when running replicas are adopted rather than restarted.
+CREATE TABLE cluster_secrets (
+  name       TEXT PRIMARY KEY,
+  value      TEXT NOT NULL,
+  created_at BIGINT NOT NULL
+);

internal/store/migrations/sqlite/0011_cluster_secrets.sql

@@ -0,0 +1,12 @@
+-- Secrets the control planes share and nobody types: generated on first use,
+-- read by every control plane against this store afterwards.
+--
+-- The first is the one each replica's engine key is derived from. It has to be
+-- the same for every control plane — the router sends a replica the key it was
+-- started with, whichever control plane started it — and the same after a
+-- restart, when running replicas are adopted rather than restarted.
+CREATE TABLE cluster_secrets (
+  name       TEXT PRIMARY KEY,
+  value      TEXT NOT NULL,
+  created_at BIGINT NOT NULL
+);

internal/store/secrets.go

@@ -0,0 +1,31 @@
+package store
+
+import (
+	"context"
+	"crypto/rand"
+	"encoding/hex"
+	"time"
+)
+
+// ClusterSecret returns the named secret, creating a random one the first time
+// it is asked for.
+//
+// Insert-if-absent and then read, so two control planes asking at once end up
+// with the same value: the loser's insert changes nothing and it reads the
+// winner's.
+func (s *DB) ClusterSecret(ctx context.Context, name string) (string, error) {
+	b := make([]byte, 32)
+	if _, err := rand.Read(b); err != nil {
+		return "", err
+	}
+	if _, err := s.db.ExecContext(ctx,
+		`INSERT INTO cluster_secrets (name, value, created_at) VALUES (?, ?, ?)
+		 ON CONFLICT (name) DO NOTHING`,
+		name, hex.EncodeToString(b), time.Now().UnixMilli()); err != nil {
+		return "", err
+	}
+	var v string
+	err := s.db.QueryRowContext(ctx,
+		`SELECT value FROM cluster_secrets WHERE name = ?`, name).Scan(&v)
+	return v, err
+}

internal/store/secrets_test.go

@@ -0,0 +1,25 @@
+package store
+
+import (
+	"path/filepath"
+	"testing"
+)
+
+func TestAClusterSecretIsMadeOnceAndKept(t *testing.T) {
+	db, err := Open(t.Context(), filepath.Join(t.TempDir(), "s.db"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer db.Close()
+	a, err := db.ClusterSecret(t.Context(), "engine-key")
+	if err != nil || len(a) != 64 {
+		t.Fatalf("%q %v", a, err)
+	}
+	b, err := db.ClusterSecret(t.Context(), "engine-key")
+	if err != nil || a != b {
+		t.Errorf("a second read made a new secret: %q then %q", a, b)
+	}
+	if c, _ := db.ClusterSecret(t.Context(), "other"); c == a {
+		t.Error("two names share a secret")
+	}
+}

internal/tlsconf/expiry.go

@@ -0,0 +1,58 @@
+package tlsconf
+
+import (
+	"crypto/x509"
+	"encoding/pem"
+	"os"
+
+	"github.com/prometheus/client_golang/prometheus"
+)
+
+// ExpiryCollector publishes when each certificate in a set of files expires.
+//
+// Certificates here last a year and nothing renews them; the first sign of one
+// running out was going to be every agent refusing to connect on the same day.
+// Read at every scrape rather than once, so a certificate replaced on disk is
+// seen without a restart. A file that cannot be read publishes nothing, which
+// is its own thing to notice.
+func ExpiryCollector(files map[string]string) prometheus.Collector {
+	return &expiry{files: files, desc: prometheus.NewDesc("llmgw_certificate_expiry_timestamp_seconds",
+		"When a certificate this control plane uses expires, as a Unix time.",
+		[]string{"cert", "subject"}, nil)}
+}
+
+type expiry struct {
+	files map[string]string
+	desc  *prometheus.Desc
+}
+
+func (e *expiry) Describe(ch chan<- *prometheus.Desc) { ch <- e.desc }
+
+func (e *expiry) Collect(ch chan<- prometheus.Metric) {
+	for name, path := range e.files {
+		if path == "" {
+			continue
+		}
+		b, err := os.ReadFile(path) //nolint:gosec // configured certificate paths
+		if err != nil {
+			continue
+		}
+		// Every certificate in the file: a CA bundle may hold several.
+		for {
+			var blk *pem.Block
+			blk, b = pem.Decode(b)
+			if blk == nil {
+				break
+			}
+			if blk.Type != "CERTIFICATE" {
+				continue
+			}
+			c, err := x509.ParseCertificate(blk.Bytes)
+			if err != nil {
+				continue
+			}
+			ch <- prometheus.MustNewConstMetric(e.desc, prometheus.GaugeValue,
+				float64(c.NotAfter.Unix()), name, c.Subject.CommonName)
+		}
+	}
+}

internal/web/admin_mirror_test.go

@@ -303,7 +303,7 @@ func TestThePageRendersThroughTheRealClient(t *testing.T) {
 
 	log := slog.New(slog.NewTextHandler(io.Discard, nil))
 	srv := httptest.NewServer(
-		mirror.NewAPI(mirror.NewManager(dir, log), "http://mirror:8081").Handler())
+		mirror.NewAPI(privateOK(mirror.NewManager(dir, log)), "http://mirror:8081").Handler())
 	defer srv.Close()
 
 	c, err := mirror.NewClient(srv.URL, tlsconf.Config{}, 0)
@@ -648,3 +648,10 @@ func TestAModelsFamilyIsHowItsNameStarts(t *testing.T) {
 		}
 	}
 }
+
+// privateOK lets a test's manager fetch from the httptest upstreams on
+// loopback, which a real one refuses.
+func privateOK(m *mirror.Manager) *mirror.Manager {
+	m.AllowPrivateSources()
+	return m
+}

scripts/gitea/release_notes.py

@@ -10,6 +10,11 @@ it, it does not replace it.
 Idempotent: a release whose body already matches is left alone, a version with
 no release is skipped, and nothing is created or deleted.
 
+A release candidate — v0.10.4-rc.2 — takes the "## Unreleased" section, which
+is where changes gather until the version they belong to is released, and is
+marked as a pre-release. The final v0.10.4 takes its own section once
+"Unreleased" has been renamed to it.
+
 Usage:
     GITEA_TOKEN=<token> scripts/gitea/release_notes.py [--tag v0.8.1] [--dry-run]
 
@@ -32,7 +37,8 @@ DEFAULT_REPO = "ilker/llmgw"
 
 # "## v0.8.1 — 2026-09-28". The dash is whatever was typed; the version is what
 # matters.
-HEADING = re.compile(r"^## (v\d+\.\d+\.\d+)\b.*$", re.M)
+HEADING = re.compile(r"^## (v\d+\.\d+\.\d+|Unreleased)\b.*$", re.M)
+CANDIDATE = re.compile(r"^v\d+\.\d+\.\d+-rc\.\d+$")
 
 
 def sections(changelog):
@@ -77,11 +83,16 @@ def main():
     with open(args.changelog, encoding="utf-8") as f:
         notes = sections(f.read())
     if args.tag:
-        if args.tag not in notes:
+        section = "Unreleased" if CANDIDATE.match(args.tag) else args.tag
+        if section not in notes:
             # A tag with no section is a release somebody forgot to describe.
             # Failing says so where it is seen; an empty page says nothing.
-            sys.exit(f"{args.changelog} has no section for {args.tag}")
-        notes = {args.tag: notes[args.tag]}
+            sys.exit(f"{args.changelog} has no section for {args.tag} (looked for {section!r})")
+        notes = {args.tag: notes[section]}
+    else:
+        # Syncing everything: "Unreleased" belongs to whichever candidate is
+        # newest, which only a tag says.
+        notes.pop("Unreleased", None)
 
     base = f"{args.instance.rstrip('/')}/api/v1/repos/{args.repo}/releases"
     for tag, body in notes.items():
@@ -92,11 +103,12 @@ def main():
         if rel is None:
             print(f"skip    {tag}  (no release)")
             continue
-        if (rel.get("body") or "").strip() == body.strip():
+        pre = bool(CANDIDATE.match(tag))
+        if (rel.get("body") or "").strip() == body.strip() and bool(rel.get("prerelease")) == pre:
             print(f"same    {tag}")
             continue
-        request("PATCH", f"{base}/{rel['id']}", token, {"body": body})
-        print(f"set     {tag}")
+        request("PATCH", f"{base}/{rel['id']}", token, {"body": body, "prerelease": pre})
+        print(f"set     {tag}" + ("  (pre-release)" if pre else ""))
 
 
 if __name__ == "__main__":