Publish small fixes as release candidates of the next version

Every fix so far became its own patch release, and the natural alternative — v0.10.3-a — sorts before v0.10.3 by SemVer's rules: a hyphen marks a pre-release, so every tool that orders versions would take the fix for the older one. A release candidate of the next version, v0.10.4-rc.1, sorts after v0.10.3 and before v0.10.4, which is what it is. CI already builds any v* tag. Its release notes now come from a "## Unreleased" section in CHANGELOG.md, where changes gather until the version they belong to is released, and the release is marked as a pre-release. Onboarding, in Go and in the shell script, counts a candidate as a release and downloads it rather than copying the operator's binary. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Restart replicas without taking them out of service first
2026-09-30 01:00:11 +02:00 · 2026-09-30 00:59:26 +02:00 · 2026-09-30 00:38:55 +02:00 · 2026-09-30 00:38:29 +02:00 · 2026-09-30 00:29:52 +02:00 · 2026-09-30 00:27:27 +02:00
63 changed files with 1405 additions and 93 deletions
@@ -20,6 +20,82 @@ is only released once its **exit criterion** is met, not when its code lands.
 Until v1.0.0 the minor version is the breaking one: configuration formats and Go
 APIs change between minors without a deprecation period.

+## Unreleased
+
+Changes since v0.10.3, published as release candidates (v0.10.4-rc.N) until
+there are enough for v0.10.4.
+
+### Restarting replicas
+
+`llmgw ctl restart REPLICA... | --model M | --all` replaces replicas with new
+ones. Each replacement is started beside the replica it replaces, which keeps
+serving until the new one is ready; replicas of one model go one at a time.
+Where no second copy fits anywhere, the old replica is stopped after the drain
+grace period and the new one starts in its place. This is how running engines
+pick up the keys introduced in v0.10.3, an edited profile, or a new engine
+version.
+
+### Releases
+
+Small fixes are published as release candidates of the next version —
+v0.10.4-rc.1, rc.2 — which sort after v0.10.3 and before v0.10.4. Their release
+notes come from this section, and they are marked as pre-releases. Onboarding
+treats a release candidate as a release and downloads it.
+
+## v0.10.3 — 2026-09-30
+
+Security fixes from a review of v0.10.2. Three of them were confirmed on the
+live cluster before being fixed.
+
+### Engines demand a key only the router holds
+
+Engines listened on every address of their node with no key. A request from a
+neighbouring node reached the model directly, past the gateway's
+authentication, quotas and accounting. Each replica's engine is now started
+with a key derived from the replica id and a secret in the store. The router,
+its metrics scrape, the agent's readiness check and `ctl bench` send it.
+llama.cpp and vLLM receive it through their environment, not their command
+line. Engines started before this keep running without a key until their
+replica is next started.
+
+### The mirror's management API answers only who it was told to
+
+Without `--api-allow`, the management API answered anyone who could reach it,
+including every node. It now answers this machine only; the control plane is
+let in with `--api-allow` or mutual TLS. **Upgrading a mirror set up by hand:
+add `--api-allow <control plane address>` to its unit.** Downloads started
+through the API go to http and https only, and never to loopback, private or
+link-local addresses (`--allow-private-sources` to permit that). Model,
+version and file names can no longer be `..` or start with a dot.
+
+### The admin listener refuses what a browser page could send
+
+The admin API has no login and trusts whoever reaches it. A page open in an
+operator's browser, reaching it over an ssh tunnel, could post a model
+definition. The listener now requires Host to be an address or localhost,
+refuses cross-site requests, and takes changes only as `application/json`.
+A profile's `env` may no longer set variables that make an engine run other
+code (`LD_*`, `PATH`, `PYTHON*` start-up variables, `BASH_ENV` and similar).
+They are refused where a model is defined and dropped again before an engine
+starts.
+
+### Fixes
+
+- The agent installs only engine versions that look like one, whatever the
+  control plane sent; a version of `../x` could have installed outside
+  `/opt/llmgw/engines`.
+
+## v0.10.2 — 2026-09-29
+
+- After `server init`'s screen closes, only the administrator's password and
+  the next steps are printed, not the whole run again under the screen that
+  just showed it. The whole run is kept in `/var/lib/llmgw/init.log`, root's
+  only, and printed in full when init fails.
+- The screens' spinner and text field are llmgw's own rather than from
+  bubbles, one dependency fewer. Every llmgw command still asks the terminal
+  for its colours as it starts, which bubbletea v1 does on purpose; a terminal
+  that never answers waits for its timeout.
+
 ## v0.10.1 — 2026-09-29

 ### `llmgw ctl fabric-test`: two nodes, rail by rail
@@ -2525,7 +2525,11 @@ type StartShard struct {
 	Fabric string `protobuf:"bytes,14,opt,name=fabric,proto3" json:"fabric,omitempty"`
 	// profile_fp identifies the version of the profile, so the agent can say
 	// which one a running shard was started from. The agent never interprets it.
-	ProfileFp     string `protobuf:"bytes,15,opt,name=profile_fp,json=profileFp,proto3" json:"profile_fp,omitempty"`
+	ProfileFp string `protobuf:"bytes,15,opt,name=profile_fp,json=profileFp,proto3" json:"profile_fp,omitempty"`
+	// api_key is what the engine demands of every request, derived by the
+	// control plane from the replica id. The router sends it; nothing else
+	// that can reach the node's ports has it.
+	ApiKey        string `protobuf:"bytes,16,opt,name=api_key,json=apiKey,proto3" json:"api_key,omitempty"`
 	unknownFields protoimpl.UnknownFields
 	sizeCache     protoimpl.SizeCache
 }
@@ -2665,6 +2669,13 @@ func (x *StartShard) GetProfileFp() string {
 	return ""
 }

+func (x *StartShard) GetApiKey() string {
+	if x != nil {
+		return x.ApiKey
+	}
+	return ""
+}
+
 type EngineProfile struct {
 	state           protoimpl.MessageState `protogen:"open.v1"`
 	Name            string                 `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"`
@@ -3693,7 +3704,7 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
 	"\n" +
 	"bytes_done\x18\x02 \x01(\x03R\tbytesDone\x12\x1f\n" +
 	"\vbytes_total\x18\x03 \x01(\x03R\n" +
-	"bytesTotal\"\xd9\x03\n" +
+	"bytesTotal\"\xf2\x03\n" +
 	"\n" +
 	"StartShard\x12\x1d\n" +
 	"\n" +
@@ -3716,7 +3727,8 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
 	"\x04pool\x18\r \x01(\tR\x04pool\x12\x16\n" +
 	"\x06fabric\x18\x0e \x01(\tR\x06fabric\x12\x1d\n" +
 	"\n" +
-	"profile_fp\x18\x0f \x01(\tR\tprofileFp\"\xfd\x02\n" +
+	"profile_fp\x18\x0f \x01(\tR\tprofileFp\x12\x17\n" +
+	"\aapi_key\x18\x10 \x01(\tR\x06apiKey\"\xfd\x02\n" +
 	"\rEngineProfile\x12\x12\n" +
 	"\x04name\x18\x01 \x01(\tR\x04name\x12\x16\n" +
 	"\x06engine\x18\x02 \x01(\tR\x06engine\x12\x1b\n" +
@@ -524,6 +524,10 @@ message StartShard {
  // profile_fp identifies the version of the profile, so the agent can say
  // which one a running shard was started from. The agent never interprets it.
  string profile_fp = 15;
+  // api_key is what the engine demands of every request, derived by the
+  // control plane from the replica id. The router sends it; nothing else
+  // that can reach the node's ports has it.
+  string api_key = 16;
 }

 message EngineProfile {
@@ -143,7 +143,7 @@ $SSH "set -e
 # that is not a release (a developer's, with commits past the tag) has no such
 # file, and is copied as before.
 TAG="$("$BINARY" --version 2>/dev/null | awk '{print $NF}')"
-if printf '%s' "$TAG" | grep -Eq '^v[0-9]+\.[0-9]+\.[0-9]+$'; then
+if printf '%s' "$TAG" | grep -Eq '^v[0-9]+\.[0-9]+\.[0-9]+(-rc\.[0-9]+)?$'; then
  say "installing llmgw $TAG from the release"
  $SSH "set -e
    arch=\$(dpkg --print-architecture)
@@ -71,6 +71,7 @@ rather than repeated.
 | `lease_ttl` | `time.Duration` | [Control](#control) |
 | `id` | `string` | [Endpoint](#endpoint) |
 | `url` | `string` | [Endpoint](#endpoint) |
+| `api_key` | `string` | [Endpoint](#endpoint) |
 | `models` | `[]string` | [Endpoint](#endpoint) |
 | `engine` | `string` | [Endpoint](#endpoint) |
 | `pool` | `string` | [Endpoint](#endpoint) |
@@ -566,6 +567,13 @@ reading the alert. Zero takes the default.

 `string`

+### `api_key`
+
+`string`
+
+APIKey is sent to the engine as a bearer token, for an engine started
+with one.
+
 ### `models`

 `[]string`
@@ -6,7 +6,6 @@ require (
 	connectrpc.com/connect v1.21.0
 	github.com/NVIDIA/go-nvml v0.13.4-0
 	github.com/cespare/xxhash/v2 v2.3.0
-	github.com/charmbracelet/bubbles v1.0.0
 	github.com/charmbracelet/bubbletea v1.3.10
 	github.com/charmbracelet/lipgloss v1.1.0
 	github.com/jackc/pgx/v5 v5.11.0
@@ -28,7 +27,6 @@ require (
 )

 require (
-	github.com/atotto/clipboard v0.1.4 // indirect
 	github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
 	github.com/beorn7/perks v1.0.1 // indirect
 	github.com/charmbracelet/colorprofile v0.4.1 // indirect
@@ -2,16 +2,12 @@ connectrpc.com/connect v1.21.0 h1:LhqSJt7jHf5NJBo9Jq/t/9FjcYAideif0mg+qe2jCUs=
 connectrpc.com/connect v1.21.0/go.mod h1:A2ygJrukXwWy32vkCAAHNVguZrqZ+jeZ9rGRnGR4dN4=
 github.com/NVIDIA/go-nvml v0.13.4-0 h1:o3jp9u2x1R9ShFE3v+Aesp55XOSIQFMJz/VGNUcJaNE=
 github.com/NVIDIA/go-nvml v0.13.4-0/go.mod h1:id63qwpoDWpFXwnwM6psDCSqW4BmNu6mWpr4YeQtPGo=
-github.com/atotto/clipboard v0.1.4 h1:EH0zSVneZPSuFR11BlR9YppQTVDbh5+16AmcJi4g1z4=
-github.com/atotto/clipboard v0.1.4/go.mod h1:ZY9tmq7sm5xIbd9bOK4onWV4S6X0u6GY7Vn0Yu86PYI=
 github.com/aymanbagabas/go-osc52/v2 v2.0.1 h1:HwpRHbFMcZLEVr42D4p7XBqjyuxQH5SMiErDT4WkJ2k=
 github.com/aymanbagabas/go-osc52/v2 v2.0.1/go.mod h1:uYgXzlJ7ZpABp8OJ+exZzJJhRNQ2ASbcXHWsFqH8hp8=
 github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
 github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
 github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
 github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
-github.com/charmbracelet/bubbles v1.0.0 h1:12J8/ak/uCZEMQ6KU7pcfwceyjLlWsDLAxB5fXonfvc=
-github.com/charmbracelet/bubbles v1.0.0/go.mod h1:9d/Zd5GdnauMI5ivUIVisuEm3ave1XwXtD1ckyV6r3E=
 github.com/charmbracelet/bubbletea v1.3.10 h1:otUDHWMMzQSB0Pkc87rm691KZ3SWa4KUlvF9nRvCICw=
 github.com/charmbracelet/bubbletea v1.3.10/go.mod h1:ORQfo0fk8U+po9VaNvnV95UPWA1BitP1E0N6xJPlHr4=
 github.com/charmbracelet/colorprofile v0.4.1 h1:a1lO03qTrSIRaK8c3JRxJDZOvhvIeSco3ej+ngLk1kk=
@@ -70,6 +70,13 @@ type ShardSpec struct {
 	HTTPPort int
 	Profile  Profile

+	// APIKey is what the engine demands of every request. Engines listen on
+	// all of a node's addresses; without a key anybody who can reach the port
+	// talks to the model directly, past the gateway's authentication, quotas
+	// and accounting. Given to the engine in its environment rather than on
+	// its command line, which every user on the node can read.
+	APIKey string `json:"api_key,omitempty"`
+
 	// Pool and Fabric are control plane concepts the agent cannot derive. They
 	// are carried so the agent can report them back after a restart; Fabric
 	// also says, with an "nvl:" prefix, that the replica is inside a
@@ -249,6 +256,11 @@ func baseEnv(s ShardSpec) map[string]string {
 	// Profile overrides win: they are the escape hatch for the per-cluster
 	// tuning no adapter can guess.
 	for k, v := range s.Profile.Env {
+		// Checked again here, not only where the model was defined: this is
+		// the process about to run as root, whatever sent the profile.
+		if bad, _ := UnsafeEnv(k); bad {
+			continue
+		}
 		env[k] = v
 	}
 	return env
@@ -120,8 +120,12 @@ func (l *LlamaCpp) Commands(s ShardSpec) ([]Command, error) {
 	}
 	args = append(args, p.Args...)

+	env := baseEnv(s)
+	if s.APIKey != "" {
+		env["LLAMA_API_KEY"] = s.APIKey
+	}
 	return []Command{{
-		Name: "llama-server", Path: bin, Args: args, Env: materializeEnv(baseEnv(s)),
+		Name: "llama-server", Path: bin, Args: args, Env: materializeEnv(env),
 	}}, nil
 }

@@ -245,3 +245,19 @@ func TestAMissingPathIsLeftToTheEngine(t *testing.T) {
 		t.Errorf("expected it passed through, got %q %v", got, err)
 	}
 }
+
+func TestLlamaCppIsGivenItsKeyOutOfSightOfTheProcessList(t *testing.T) {
+	// The command line is readable by every user on the node; the
+	// environment of a root process is not.
+	cmds, err := (&LlamaCpp{}).Commands(ShardSpec{HTTPPort: 8080, APIKey: "k1",
+		Profile: Profile{ModelPath: "/m.gguf"}})
+	if err != nil {
+		t.Fatal(err)
+	}
+	if slices.Contains(cmds[0].Args, "k1") || slices.Contains(cmds[0].Args, "--api-key") {
+		t.Errorf("the key is on the command line: %v", cmds[0].Args)
+	}
+	if !slices.Contains(cmds[0].Env, "LLAMA_API_KEY=k1") {
+		t.Error("llama-server was not given its key")
+	}
+}
@@ -96,6 +96,11 @@ func (g *SGLang) serverCommand(s ShardSpec) (Command, error) {
 		)
 	}
 	args = append(args, p.Args...)
+	// SGLang reads its key only from the command line, so on a node with
+	// other users it is visible to them in the process list.
+	if s.APIKey != "" && s.Leader() {
+		args = append(args, "--api-key", s.APIKey)
+	}

 	return Command{Name: "sglang", Path: python, Args: args, Env: materializeEnv(baseEnv(s))}, nil
 }
@@ -0,0 +1,30 @@
+package engines
+
+import "strings"
+
+// UnsafeEnv reports whether a profile may not set an environment variable,
+// and why.
+//
+// A profile's env is the escape hatch for tuning no adapter can guess —
+// CUDA_*, NCCL_*, VLLM_*, HF_* — and it reaches an engine the agent starts as
+// root. Some variables are not tuning but a way to run other code in that
+// process: the dynamic loader's LD_PRELOAD, Python's PYTHONSTARTUP, a shell's
+// BASH_ENV. Set from a model definition, which an admin API or a config file
+// supplies, they would turn "add a model" into "run this as root on every
+// node". They are refused where a model is defined and again where an engine
+// starts.
+func UnsafeEnv(key string) (bool, string) {
+	k := strings.ToUpper(key)
+	switch {
+	case strings.HasPrefix(k, "LD_"), strings.HasPrefix(k, "DYLD_"):
+		return true, "it changes what the dynamic loader loads"
+	case k == "PATH":
+		return true, "it changes which binaries run"
+	case strings.HasPrefix(k, "PYTHON") && k != "PYTHONUNBUFFERED" && k != "PYTHONHASHSEED":
+		return true, "it changes what Python imports or runs at start"
+	case k == "BASH_ENV" || k == "ENV" || k == "PERL5OPT" || k == "PERL5LIB" ||
+		k == "NODE_OPTIONS" || k == "RUBYOPT" || k == "GCONV_PATH" || k == "OPENSSL_CONF":
+		return true, "it makes an interpreter or library load code"
+	}
+	return false, ""
+}
@@ -150,6 +150,9 @@ func (v *VLLM) serverCommand(s ShardSpec) (Command, error) {
 	// Profile arguments last, so an operator can override anything above.
 	args = append(args, p.Args...)

+	if s.APIKey != "" {
+		env["VLLM_API_KEY"] = s.APIKey
+	}
 	return Command{Name: "vllm", Path: bin, Args: args, Env: materializeEnv(env)}, nil
 }

@@ -16,6 +16,7 @@ import (
 	"os"
 	"os/exec"
 	"path/filepath"
+	"regexp"
 	"strings"
 	"time"

@@ -77,6 +78,10 @@ type Installer struct {
 	PyPIBase     string
 }

+// validVersion is what a version may look like: "b10991", "0.29.0",
+// "0.11.0+cu128". No separators, and not starting with a dot.
+var validVersion = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._+-]{0,63}$`)
+
 // DefaultRoot is where engines are installed when nothing says otherwise.
 const DefaultRoot = "/opt/llmgw/engines"

@@ -114,6 +119,15 @@ func (in *Installer) Converge(ctx context.Context, wants []Want) []Status {

 func (in *Installer) one(ctx context.Context, w Want) Status {
 	st := Status{Engine: w.Engine, Version: w.Version}
+	// Checked here, on the node, whatever the control plane checked: the
+	// version names a directory under the install root and is passed to a
+	// package manager, and "../x" would put an install — and the RemoveAll
+	// that replaces one — outside it, as root.
+	if !validVersion.MatchString(w.Version) {
+		st.State = Unsupported
+		st.Detail = fmt.Sprintf("%q is not a version this agent will install", w.Version)
+		return st
+	}
 	recipe, ok := Recipes[w.Engine]
 	if !ok {
 		st.State = Unsupported
@@ -330,3 +330,15 @@ func TestAPackageManagerBusyWithItsOwnUpdatesIsWaitedFor(t *testing.T) {
 		t.Error("the wait was not reported")
 	}
 }
+
+func TestAVersionThatIsAPathIsRefused(t *testing.T) {
+	// The version names a directory under the install root; "../" would put
+	// an install, and the RemoveAll that replaces one, outside it, as root.
+	in := &Installer{Root: t.TempDir(), Host: ubuntu}
+	for _, v := range []string{"../../etc", "..", ".hidden", "b1/../../x", "", "b1 && rm -rf /"} {
+		got := in.Converge(t.Context(), []Want{{Engine: "llamacpp", Version: v}})
+		if got[0].State != Unsupported {
+			t.Errorf("version %q: %+v", v, got[0])
+		}
+	}
+}
@@ -53,8 +53,8 @@ func NewHTTPHealth(timeout time.Duration, opts ...HTTPHealthOption) *HTTPHealth
 }

 // Ready reports whether the engine at baseURL is serving model.
-func (h *HTTPHealth) Ready(ctx context.Context, baseURL, model string) error {
-	if err := h.get(ctx, baseURL+h.healthPath, nil); err != nil {
+func (h *HTTPHealth) Ready(ctx context.Context, baseURL, model, apiKey string) error {
+	if err := h.get(ctx, baseURL+h.healthPath, apiKey, nil); err != nil {
 		return fmt.Errorf("health: %w", err)
 	}
 	if !h.requireName || model == "" {
@@ -66,7 +66,7 @@ func (h *HTTPHealth) Ready(ctx context.Context, baseURL, model string) error {
 			ID string `json:"id"`
 		} `json:"data"`
 	}
-	if err := h.get(ctx, baseURL+h.modelsPath, &list); err != nil {
+	if err := h.get(ctx, baseURL+h.modelsPath, apiKey, &list); err != nil {
 		return fmt.Errorf("models: %w", err)
 	}
 	for _, d := range list.Data {
@@ -77,11 +77,14 @@ func (h *HTTPHealth) Ready(ctx context.Context, baseURL, model string) error {
 	return fmt.Errorf("engine is up but does not serve %q yet", model)
 }

-func (h *HTTPHealth) get(ctx context.Context, url string, out any) error {
+func (h *HTTPHealth) get(ctx context.Context, url, apiKey string, out any) error {
 	req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
 	if err != nil {
 		return err
 	}
+	if apiKey != "" {
+		req.Header.Set("Authorization", "Bearer "+apiKey)
+	}
 	resp, err := h.client.Do(req)
 	if err != nil {
 		return err
@@ -39,7 +39,7 @@ func engineStub(t *testing.T, healthy bool, models ...string) *httptest.Server {

 func TestReadyWhenServingModel(t *testing.T) {
 	srv := engineStub(t, true, "llama-3.3-70b")
-	if err := NewHTTPHealth(time.Second).Ready(t.Context(), srv.URL, "llama-3.3-70b"); err != nil {
+	if err := NewHTTPHealth(time.Second).Ready(t.Context(), srv.URL, "llama-3.3-70b", ""); err != nil {
 		t.Errorf("a serving engine was reported unready: %v", err)
 	}
 }
@@ -48,7 +48,7 @@ func TestNotReadyWhileLoading(t *testing.T) {
 	// The case that matters: the process is up but the weights are not loaded.
 	// Routing on /health alone would send traffic into a loading model.
 	srv := engineStub(t, false, "llama-3.3-70b")
-	err := NewHTTPHealth(time.Second).Ready(t.Context(), srv.URL, "llama-3.3-70b")
+	err := NewHTTPHealth(time.Second).Ready(t.Context(), srv.URL, "llama-3.3-70b", "")
 	if err == nil {
 		t.Fatal("an unhealthy engine was reported ready")
 	}
@@ -61,7 +61,7 @@ func TestNotReadyWhenModelMissing(t *testing.T) {
 	// /health passes but the model is not listed: the engine is up and serving
 	// something else, or still registering.
 	srv := engineStub(t, true, "some-other-model")
-	err := NewHTTPHealth(time.Second).Ready(t.Context(), srv.URL, "llama-3.3-70b")
+	err := NewHTTPHealth(time.Second).Ready(t.Context(), srv.URL, "llama-3.3-70b", "")
 	if err == nil {
 		t.Fatal("a model that is not served was reported ready")
 	}
@@ -73,14 +73,33 @@ func TestNotReadyWhenModelMissing(t *testing.T) {
 func TestModelCheckCanBeSkipped(t *testing.T) {
 	srv := engineStub(t, true)
 	h := NewHTTPHealth(time.Second, WithoutModelCheck())
-	if err := h.Ready(t.Context(), srv.URL, "anything"); err != nil {
+	if err := h.Ready(t.Context(), srv.URL, "anything", ""); err != nil {
 		t.Errorf("the model check was not skipped: %v", err)
 	}
 }

 func TestUnreachableEngine(t *testing.T) {
 	h := NewHTTPHealth(200 * time.Millisecond)
-	if err := h.Ready(t.Context(), "http://127.0.0.1:1", "m"); err == nil {
+	if err := h.Ready(t.Context(), "http://127.0.0.1:1", "m", ""); err == nil {
 		t.Error("an unreachable engine was reported ready")
 	}
 }
+
+func TestTheHealthCheckSendsTheEnginesKey(t *testing.T) {
+	// An engine started with a key refuses /v1/models without it, and a
+	// shard that is serving would read as never ready.
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		if r.Header.Get("Authorization") != "Bearer k1" {
+			w.WriteHeader(http.StatusUnauthorized)
+			return
+		}
+		_, _ = w.Write([]byte(`{"data":[{"id":"m"}]}`))
+	}))
+	defer srv.Close()
+	if err := NewHTTPHealth(time.Second).Ready(t.Context(), srv.URL, "m", "k1"); err != nil {
+		t.Errorf("with the key: %v", err)
+	}
+	if err := NewHTTPHealth(time.Second).Ready(t.Context(), srv.URL, "m", ""); err == nil {
+		t.Error("without the key the engine refused, and that read as ready")
+	}
+}
@@ -79,7 +79,7 @@ func (c *Config) normalize() {
 // supervisor can be tested without an HTTP engine.
 type HealthChecker interface {
 	// Ready reports whether the shard at baseURL serves model.
-	Ready(ctx context.Context, baseURL, model string) error
+	Ready(ctx context.Context, baseURL, model, apiKey string) error
 }

 // Shard is one supervised engine process.
@@ -411,7 +411,7 @@ func (s *Supervisor) WaitReady(ctx context.Context, sh *Shard, baseURL, model st
 			last := strings.Join(sh.Logs(10), "\n")
 			return fmt.Errorf("shard %s not ready within %v; last output:\n%s", sh.ID, timeout, last)
 		case <-t.C:
-			if err := s.health.Ready(ctx, baseURL, model); err != nil {
+			if err := s.health.Ready(ctx, baseURL, model, sh.Spec.APIKey); err != nil {
 				continue
 			}
 			sh.setState(Ready, nil)
@@ -21,7 +21,7 @@ import (
 // scriptedHealth reports ready once the flag is set.
 type scriptedHealth struct{ ready atomic.Bool }

-func (h *scriptedHealth) Ready(context.Context, string, string) error {
+func (h *scriptedHealth) Ready(context.Context, string, string, string) error {
 	if h.ready.Load() {
 		return nil
 	}
@@ -225,6 +225,7 @@ func runStep(ctx context.Context, client *http.Client, cfg Config, concurrency i
 	)

 	sampler := newSampler(cfg.MetricsURL, cfg.Engine)
+	sampler.apiKey = cfg.APIKey
 	sampler.start(ctx)

 	start := time.Now()
@@ -19,6 +19,7 @@ import (
 // Optional, because a run pointed at the gateway has no single engine to read.
 type sampler struct {
 	url    string
+	apiKey string
 	engine string
 	client *http.Client
 	cancel context.CancelFunc
@@ -65,6 +66,9 @@ func (s *sampler) sample(ctx context.Context) {
 	if err != nil {
 		return
 	}
+	if s.apiKey != "" {
+		req.Header.Set("Authorization", "Bearer "+s.apiKey)
+	}
 	resp, err := s.client.Do(req)
 	if err != nil {
 		return
@@ -41,6 +41,11 @@ func newBenchCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 			cfg.MetricsURL = metrics
 			cfg.Engine = engine
 			cfg.APIKey = key
+			// A replica's engine demands its own key. Asked for rather than
+			// typed: the admin API knows which replica answers at the target.
+			if cfg.APIKey == "" {
+				cfg.APIKey = replicaKey(cmd, client, target)
+			}
 			if len(steps) > 0 {
 				cfg.Concurrency = steps
 			}
@@ -227,3 +232,18 @@ func ms(d time.Duration) string {
 		return d.Round(10 * time.Millisecond).String()
 	}
 }
+
+// replicaKey is the engine key of the replica at target, or "" when target is
+// not a replica (the gateway, say) or the admin API cannot be asked.
+func replicaKey(cmd *cobra.Command, client func() *ctlClient, target string) string {
+	var reps []adminapi.ReplicaView
+	if err := client().get(cmd.Context(), "/api/v1/replicas", &reps); err != nil {
+		return ""
+	}
+	for _, rep := range reps {
+		if sameEndpoint(rep.Endpoint, target) {
+			return rep.EngineKey
+		}
+	}
+	return ""
+}
@@ -67,7 +67,9 @@ func (c *ctlClient) do(ctx context.Context, method, path string, body, out any)
 	if err != nil {
 		return err
 	}
-	if body != nil {
+	// On every change, body or not: the admin listener refuses anything else,
+	// because a browser can send a bodiless POST across origins unasked.
+	if method != http.MethodGet {
 		req.Header.Set("Content-Type", "application/json")
 	}
 	// Who made the change, kept with a stored model so a catalog that grew
@@ -133,6 +135,7 @@ func newCtlCmd() *cobra.Command {
 		newStatusCmd(client, &asJSON),
 		newModelCmd(client),
 		newScaleCmd(client),
+		newRestartCmd(client),
 		newEngineCheckCmd(),
 		newToolCheckCmd(),
 		newWeightsCmd(client, &asJSON),
@@ -368,6 +371,9 @@ func newReplicasCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 				if r.Stale {
 					state += " (stale profile)"
 				}
+				if r.Restart != "" {
+					state += " (restart " + r.Restart + ")"
+				}
 				fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n",
 					r.ID, r.Model, state, r.Pool, join(nodes),
 					(time.Duration(r.AgeSec) * time.Second).String(),
@@ -0,0 +1,48 @@
+package cli
+
+import (
+	"encoding/json"
+	"errors"
+	"fmt"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+)
+
+func newRestartCmd(client func() *ctlClient) *cobra.Command {
+	var (
+		models []string
+		all    bool
+	)
+	c := &cobra.Command{
+		Use:   "restart [REPLICA...]",
+		Short: "Replace replicas with new ones, starting each replacement before stopping the old",
+		Long: "How a running engine picks up what only applies at start: an engine key,\n" +
+			"an edited profile, a new engine version.\n\n" +
+			"Each replacement is placed beside the replica it replaces, and the old one\n" +
+			"keeps serving until the new one is ready. Replicas of one model are\n" +
+			"replaced one at a time. Where nothing else has room for a second copy —\n" +
+			"the usual case on a small CPU fleet — the old one is stopped after the\n" +
+			"drain grace period (5 minutes by default) and the new one starts in its\n" +
+			"place; a model with only that replica is unserved while it loads.\n\n" +
+			"  llmgw ctl restart qwen2.5-7b-3f2a91c0\n" +
+			"  llmgw ctl restart --model qwen2.5-7b\n" +
+			"  llmgw ctl restart --all",
+		RunE: func(cmd *cobra.Command, args []string) error {
+			if len(args) == 0 && len(models) == 0 && !all {
+				return errors.New("name replicas, --model, or --all")
+			}
+			body, _ := json.Marshal(adminapi.RestartRequest{Replicas: args, Models: models, All: all})
+			var res adminapi.RestartResult
+			if err := client().post(cmd.Context(), "/api/v1/restart", body, &res); err != nil {
+				return err
+			}
+			fmt.Printf("%d replica(s) queued for restart; watch with: llmgw ctl replicas\n", len(res.Queued))
+			return nil
+		},
+	}
+	c.Flags().StringArrayVar(&models, "model", nil, "every replica of this model; repeatable")
+	c.Flags().BoolVar(&all, "all", false, "every ready replica in the cluster")
+	return c
+}
@@ -233,6 +233,7 @@ func newMirrorServeCmd(dir *string) *cobra.Command {
 	var listen, apiListen, base string
 	var tlsc tlsconf.Config
 	var allow []string
+	var allowPrivate bool

 	c := &cobra.Command{
 		Use:   "serve",
@@ -262,6 +263,9 @@ func newMirrorServeCmd(dir *string) *cobra.Command {
 			out := cmd.OutOrStdout()
 			log := slog.Default()
 			mgr := mirror.NewManager(*dir, log)
+			if allowPrivate {
+				mgr.AllowPrivateSources()
+			}
 			api := mirror.NewAPI(mgr, strings.TrimSuffix(base, "/"))

 			// http.Dir refuses paths that escape the root, which is the whole
@@ -278,7 +282,7 @@ func newMirrorServeCmd(dir *string) *cobra.Command {
 			}
 			admin := &http.Server{
 				Addr:              apiListen,
-				Handler:           onlyFrom(allowed, api.Handler()),
+				Handler:           apiGate(tlsc.Enabled(), allowed, api.Handler()),
 				ReadHeaderTimeout: 10 * time.Second,
 			}

@@ -293,11 +297,11 @@ func newMirrorServeCmd(dir *string) *cobra.Command {
 			case len(allowed) > 0:
 				fmt.Fprintf(out, "management API on %s, for %s and this machine only\n", apiListen, strings.Join(allow, ", "))
 			default:
-				// Said plainly rather than left to be discovered. An API that
-				// can delete models and start downloads, open to anyone who can
-				// reach the port, is a choice the operator should have made
-				// rather than inherited from a default.
-				fmt.Fprintf(out, "management API on %s, UNAUTHENTICATED — keep it on a trusted network\n", apiListen)
+				// This machine only, which was once "anyone": an API that can
+				// delete models and start downloads, open to every node that
+				// could reach the port, is a choice the operator should make
+				// with --api-allow rather than inherit.
+				fmt.Fprintf(out, "management API on %s, for this machine only (--api-allow to add the control plane)\n", apiListen)
 			}
 			fmt.Fprintf(out, "catalog sources should point at %s\n", base)

@@ -312,7 +316,9 @@ func newMirrorServeCmd(dir *string) *cobra.Command {
 	c.Flags().StringVar(&base, "base", "http://localhost:8081",
 		"the address nodes use to reach this mirror, as it should appear in catalog sources")
 	c.Flags().StringArrayVar(&allow, "api-allow", nil,
-		"an address or network the management API answers, besides this machine; repeatable (default: anyone)")
+		"an address or network the management API answers, besides this machine; repeatable (default: this machine only)")
+	c.Flags().BoolVar(&allowPrivate, "allow-private-sources", false,
+		"let downloads reach loopback, private and link-local addresses")
 	c.Flags().StringVar(&tlsc.CA, "tls-ca", "", "CA bundle that management clients must present a certificate from")
 	c.Flags().StringVar(&tlsc.Cert, "tls-cert", "", "this mirror's certificate")
 	c.Flags().StringVar(&tlsc.Key, "tls-key", "", "this mirror's private key")
@@ -435,14 +441,21 @@ func parseAllow(list []string) ([]netip.Prefix, error) {
 	return out, nil
 }

-// onlyFrom answers only this machine and the addresses allowed, when any
-// are: the management API can delete models and start downloads, and a
-// mirror on its own machine is reached over the network by one control plane
-// and no one else. Nothing allowed is the old behaviour, anyone.
-func onlyFrom(allowed []netip.Prefix, next http.Handler) http.Handler {
-	if len(allowed) == 0 {
+// apiGate decides who reaches the management API. With mutual TLS the
+// certificate decides, narrowed by --api-allow when given. Without it, only
+// this machine and the addresses allowed: nothing allowed used to mean anyone,
+// and every node that could reach the port could delete models.
+func apiGate(mtls bool, allowed []netip.Prefix, next http.Handler) http.Handler {
+	if mtls && len(allowed) == 0 {
 		return next
 	}
+	return onlyFrom(allowed, next)
+}
+
+// onlyFrom answers only this machine and the addresses allowed: the management
+// API can delete models and start downloads, and a mirror on its own machine
+// is reached over the network by one control plane and no one else.
+func onlyFrom(allowed []netip.Prefix, next http.Handler) http.Handler {
 	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 		host, _, err := net.SplitHostPort(r.RemoteAddr)
 		a, perr := netip.ParseAddr(host)
@@ -0,0 +1,39 @@
+package cli
+
+import (
+	"net/http"
+	"net/http/httptest"
+	"net/netip"
+	"testing"
+)
+
+func TestTheMirrorAPIAnswersOnlyThisMachineUnlessTold(t *testing.T) {
+	ok := http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {})
+	call := func(h http.Handler, from string) int {
+		r := httptest.NewRequest("DELETE", "/v1/models/m/v", nil)
+		r.RemoteAddr = from + ":5000"
+		w := httptest.NewRecorder()
+		h.ServeHTTP(w, r)
+		return w.Code
+	}
+	cp := []netip.Prefix{netip.MustParsePrefix("10.0.0.140/32")}
+
+	// Nothing allowed and no TLS used to mean anyone: every node could delete
+	// models.
+	if got := call(apiGate(false, nil, ok), "10.0.0.141"); got != http.StatusForbidden {
+		t.Errorf("a node reached an unconfigured mirror API: %d", got)
+	}
+	if got := call(apiGate(false, nil, ok), "127.0.0.1"); got != http.StatusOK {
+		t.Errorf("the mirror's own machine was refused: %d", got)
+	}
+	if got := call(apiGate(false, cp, ok), "10.0.0.140"); got != http.StatusOK {
+		t.Errorf("the allowed control plane was refused: %d", got)
+	}
+	if got := call(apiGate(false, cp, ok), "10.0.0.141"); got != http.StatusForbidden {
+		t.Errorf("a node outside the allowlist got through: %d", got)
+	}
+	// With mutual TLS the certificate decides.
+	if got := call(apiGate(true, nil, ok), "10.0.0.141"); got != http.StatusOK {
+		t.Errorf("mTLS without an allowlist refused a client: %d", got)
+	}
+}
@@ -246,7 +246,7 @@ func newServerCmd() *cobra.Command {
 				// enforces its own per-phase deadlines.
 			}
 			adminSrv := &http.Server{
-				Handler:           adminMux,
+				Handler:           router.AdminGuard(adminMux),
 				ReadHeaderTimeout: 10 * time.Second,
 			}

@@ -385,6 +385,20 @@ func startControlPlane(ctx context.Context, cfg config.Server, cat *catalog.Live
 		ccfg.Reconciler.PortRange = [2]int{cfg.Control.PortRange[0], cfg.Control.PortRange[1]}
 	}

+	// Each replica's engine demands a key only the router has. The secret it
+	// is derived from lives in the store, so every control plane against it —
+	// and this one after a restart — computes the same key for a replica.
+	if h, ok := sink.(interface{ Store() *storepkg.DB }); ok && h.Store() != nil {
+		secret, err := h.Store().ClusterSecret(ctx, "engine-key")
+		if err != nil {
+			return nil, nil, fmt.Errorf("reading the engine key secret: %w", err)
+		}
+		ccfg.Reconciler.EngineKey = control.EngineKeys([]byte(secret))
+	} else {
+		log.Warn("engines start without a key: there is no store to keep the secret " +
+			"they are derived from, so anyone who can reach a node's ports reaches its models")
+	}
+
 	// The service needs the inventory and the status sink, both of which the
 	// controller owns; the controller needs the service as its dispatcher. The
 	// cycle is broken by giving the controller a dispatcher that forwards to
@@ -5,11 +5,11 @@ import (
 	"fmt"
 	"io"
 	"net/mail"
+	"os"
+	"path/filepath"
 	"strings"
 	"sync"

-	"github.com/charmbracelet/bubbles/spinner"
-	"github.com/charmbracelet/bubbles/textinput"
 	tea "github.com/charmbracelet/bubbletea"
 	"github.com/charmbracelet/lipgloss"
 )
@@ -65,8 +65,8 @@ type initModel struct {
 	// The form, until it is submitted.
 	form      bool
 	focus     int
-	admin     textinput.Model
-	hosts     textinput.Model
+	admin     textField
+	hosts     textField
 	toggles   [fCount]bool
 	formError string

@@ -75,7 +75,7 @@ type initModel struct {
 	lines    []string // init's lines, as it wrote them
 	password string
 	next     []string // the closing "Next:" text
-	spin     spinner.Model
+	spin     spinnerModel
 	width    int
 	done     bool
 	err      error
@@ -84,19 +84,17 @@ type initModel struct {
 }

 func newInitModel(in *serverInit, found []string, askForm bool) initModel {
-	admin := textinput.New()
+	admin := newTextField()
 	admin.Placeholder = "you@example.com"
 	admin.SetValue(in.admin)
 	admin.CharLimit = 200
 	admin.Width = 40
-	hosts := textinput.New()
+	hosts := newTextField()
 	hosts.Placeholder = "llm.example.com, 203.0.113.10"
 	hosts.SetValue(strings.Join(in.hosts, ", "))
 	hosts.CharLimit = 400
 	hosts.Width = 40
-	sp := spinner.New()
-	sp.Spinner = spinner.MiniDot
-	sp.Style = lipgloss.NewStyle().Foreground(tuiAccent)
+	sp := newSpinner()
 	m := initModel{in: in, found: found, form: askForm, admin: admin, hosts: hosts, spin: sp, width: 100}
 	m.toggles[fPostgres] = in.postgres
 	m.toggles[fMirror] = !in.noMirror
@@ -110,7 +108,7 @@ func newInitModel(in *serverInit, found []string, askForm bool) initModel {

 func (m initModel) Init() tea.Cmd {
 	if m.form {
-		return textinput.Blink
+		return nil
 	}
 	return tea.Batch(m.spin.Tick, func() tea.Msg { return startInitMsg{} })
 }
@@ -121,7 +119,7 @@ func (m initModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
 	switch msg := msg.(type) {
 	case tea.WindowSizeMsg:
 		m.width = msg.Width
-	case spinner.TickMsg:
+	case spinTickMsg:
 		var cmd tea.Cmd
 		m.spin, cmd = m.spin.Update(msg)
 		return m, cmd
@@ -391,13 +389,33 @@ func (in *serverInit) runInitTUI(ctx context.Context, out io.Writer) error {
 		return fmt.Errorf("stopped; run init again to finish what was started")
 	}

-	// The screen showed the last lines; the whole run goes to the terminal's
-	// history, the password with it.
+	// The whole run is kept, root's only, since it holds the password. On a
+	// failure it is printed too, for the lines the screen had no room for;
+	// otherwise the screen's last frame says it, and only what must outlive
+	// the screen is printed again: the password, and what to do next.
 	logMu.Lock()
 	defer logMu.Unlock()
+	whole := strings.Join(log, "\n") + "\n"
+	saved := ""
+	if in.stateDir != "" {
+		p := filepath.Join(in.stateDir, "init.log")
+		if err := os.WriteFile(p, []byte(whole), 0o600); err == nil {
+			saved = p
+		}
+	}
 	fmt.Fprintln(out)
-	for _, l := range log {
+	if fm.err != nil {
+		fmt.Fprint(out, whole)
+		return fm.err
+	}
+	if fm.password != "" {
+		fmt.Fprintf(out, "Administrator %s, password %s (shown once)\n\n", in.admin, fm.password)
+	}
+	for _, l := range fm.next {
 		fmt.Fprintln(out, l)
 	}
+	if saved != "" {
+		fmt.Fprintf(out, "\nThe whole run is in %s.\n", saved)
+	}
 	return fm.err
 }
@@ -7,8 +7,6 @@ import (
 	"os"
 	"strings"

-	"github.com/charmbracelet/bubbles/spinner"
-	"github.com/charmbracelet/bubbles/textinput"
 	tea "github.com/charmbracelet/bubbletea"
 	"github.com/charmbracelet/lipgloss"
 	"github.com/mattn/go-isatty"
@@ -67,7 +65,7 @@ type nodeModel struct {
 	title   string
 	steps   map[string]stepState
 	notes   []onboard.Event // warnings and failures, with their fixes
-	spin    spinner.Model
+	spin    spinnerModel
 	cancel  context.CancelFunc
 	width   int
 	done    bool
@@ -77,16 +75,14 @@ type nodeModel struct {

 	// The question being asked, if any.
 	fabrics   *askFabricsMsg
-	inputs    []textinput.Model
+	inputs    []textField
 	focus     int
 	confirm   *askContinueMsg
 	quitAsked bool
 }

 func newNodeModel(title string, cancel context.CancelFunc) nodeModel {
-	sp := spinner.New()
-	sp.Spinner = spinner.MiniDot
-	sp.Style = lipgloss.NewStyle().Foreground(tuiAccent)
+	sp := newSpinner()
 	return nodeModel{title: title, steps: map[string]stepState{}, spin: sp, cancel: cancel, width: 100}
 }

@@ -96,7 +92,7 @@ func (m nodeModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
 	switch msg := msg.(type) {
 	case tea.WindowSizeMsg:
 		m.width = msg.Width
-	case spinner.TickMsg:
+	case spinTickMsg:
 		var cmd tea.Cmd
 		m.spin, cmd = m.spin.Update(msg)
 		return m, cmd
@@ -121,7 +117,7 @@ func (m nodeModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
 		m.fabrics = &msg
 		m.inputs = nil
 		for i, f := range msg.suggested {
-			in := textinput.New()
+			in := newTextField()
 			in.Prompt = ""
 			in.SetValue(f.Name)
 			in.CharLimit = 40
@@ -0,0 +1,98 @@
+package cli
+
+import (
+	"strings"
+	"time"
+
+	tea "github.com/charmbracelet/bubbletea"
+	"github.com/charmbracelet/lipgloss"
+)
+
+// Two small widgets, written here rather than taken from bubbles: its
+// packages ask the terminal for its colours as they load, so every llmgw
+// command — llmgw --version included — sent a query and, in a terminal that
+// never answers one, waited five seconds for it. Only the screens need to
+// know the colours, and lipgloss asks when they first draw.
+
+// spinTickMsg advances a spinner.
+type spinTickMsg struct{}
+
+var spinFrames = []string{"⠋", "⠙", "⠹", "⠸", "⠼", "⠴", "⠦", "⠧", "⠇", "⠏"}
+
+type spinnerModel struct {
+	frame int
+	Style lipgloss.Style
+}
+
+func newSpinner() spinnerModel { return spinnerModel{Style: lipgloss.NewStyle().Foreground(tuiAccent)} }
+
+// Tick starts the spinner turning; it is a tea.Cmd.
+func (s spinnerModel) Tick() tea.Msg { return spinTickMsg{} }
+
+func (s spinnerModel) Update(msg tea.Msg) (spinnerModel, tea.Cmd) {
+	if _, ok := msg.(spinTickMsg); !ok {
+		return s, nil
+	}
+	s.frame = (s.frame + 1) % len(spinFrames)
+	return s, tea.Tick(100*time.Millisecond, func(time.Time) tea.Msg { return spinTickMsg{} })
+}
+
+func (s spinnerModel) View() string { return s.Style.Render(spinFrames[s.frame]) }
+
+// textField is one line of text typed at its end: all a form here needs.
+type textField struct {
+	value       []rune
+	focused     bool
+	Prompt      string
+	Placeholder string
+	CharLimit   int
+	Width       int
+}
+
+func newTextField() textField { return textField{Prompt: "> "} }
+
+func (f *textField) SetValue(s string) { f.value = []rune(s) }
+func (f textField) Value() string      { return string(f.value) }
+func (f *textField) Focus()            { f.focused = true }
+func (f *textField) Blur()             { f.focused = false }
+
+func (f textField) Update(msg tea.Msg) (textField, tea.Cmd) {
+	k, ok := msg.(tea.KeyMsg)
+	if !ok || !f.focused {
+		return f, nil
+	}
+	switch k.Type {
+	case tea.KeyRunes, tea.KeySpace:
+		r := k.Runes
+		if k.Type == tea.KeySpace {
+			r = []rune{' '}
+		}
+		for _, c := range r {
+			if f.CharLimit == 0 || len(f.value) < f.CharLimit {
+				f.value = append(f.value, c)
+			}
+		}
+	case tea.KeyBackspace:
+		if len(f.value) > 0 {
+			f.value = f.value[:len(f.value)-1]
+		}
+	case tea.KeyCtrlU:
+		f.value = nil
+	}
+	return f, nil
+}
+
+func (f textField) View() string {
+	text := string(f.value)
+	shown := text
+	if text == "" && f.Placeholder != "" {
+		shown = sDim.Render(f.Placeholder)
+	}
+	if f.focused {
+		shown += "█"
+	}
+	if pad := f.Width - len([]rune(text)); pad > 0 && text != "" {
+		shown += strings.Repeat(" ", pad)
+	}
+	return f.Prompt + shown
+}
@@ -404,8 +404,11 @@ type Scoring struct {

 // Endpoint is one engine replica. Static in phase 1.
 type Endpoint struct {
-	ID     string   `koanf:"id"`
-	URL    string   `koanf:"url"`
+	ID  string `koanf:"id"`
+	URL string `koanf:"url"`
+	// APIKey is sent to the engine as a bearer token, for an engine started
+	// with one.
+	APIKey string   `koanf:"api_key"`
 	Models []string `koanf:"models"`
 	// Engine names the inference engine, which decides what its metrics are
 	// called. Empty means vLLM's names.
@@ -77,6 +77,7 @@ func (a *API) Register(mux *http.ServeMux) {
 	mux.HandleFunc("DELETE /api/v1/nodes/{id}", a.removeNode)
 	mux.HandleFunc("POST /api/v1/fabric-test", a.fabricTest)
 	mux.HandleFunc("GET /api/v1/nodes/{id}/logs", a.nodeLogs)
+	mux.HandleFunc("POST /api/v1/restart", a.restart)
 }

 // --- responses ---
@@ -238,7 +239,14 @@ type ModelView struct {

 // ReplicaView is a replica as `ctl replicas` shows it.
 type ReplicaView struct {
-	ID      string `json:"id"`
+	ID string `json:"id"`
+	// EngineKey is the key this replica's engine demands, for `ctl bench`
+	// and anyone else holding the admin listener who needs to reach an engine
+	// directly. The client-facing pages never render it.
+	EngineKey string `json:"engine_key,omitempty"`
+	// Restart is "queued" or "replacing" while an operator's restart of this
+	// replica is under way.
+	Restart string `json:"restart,omitempty"`
 	Model   string `json:"model"`
 	Profile string `json:"profile"`
 	// Stale is true when the profile has been edited since this replica was
@@ -471,9 +479,11 @@ func (a *API) Replicas(model string) []ReplicaView {
 			ID: rep.ID, Model: rep.Model, Profile: rep.Profile,
 			State: rep.State.String(), Pool: rep.Pool, Fabric: rep.Fabric,
 			Endpoint: rep.Endpoint, Err: rep.Err,
-			Stale:   a.profileChanged(rep.Model, rep.Profile, rep.ProfileFP),
-			AgeSec:  int64(time.Since(rep.CreatedAt).Seconds()),
-			IdleSec: -1,
+			EngineKey: a.ctl.EngineKey(rep.ID),
+			Restart:   a.ctl.Reconciler().RestartState(rep.ID),
+			Stale:     a.profileChanged(rep.Model, rep.Profile, rep.ProfileFP),
+			AgeSec:    int64(time.Since(rep.CreatedAt).Seconds()),
+			IdleSec:   -1,
 		}
 		if rep.LastUsedAt.IsZero() {
 			v.IdleSec = -1
@@ -1039,3 +1049,39 @@ func (a *API) Topology(id string) (TopologyView, bool) {
 func (a *API) NVLink() nvlink.Report {
 	return nvlink.Analyze(a.ctl.Inventory().Nodes())
 }
+
+// RestartRequest names what to restart: replicas by id, every replica of some
+// models, or — All — every ready replica in the cluster.
+type RestartRequest struct {
+	Replicas []string `json:"replicas,omitempty"`
+	Models   []string `json:"models,omitempty"`
+	All      bool     `json:"all,omitempty"`
+}
+
+// RestartResult lists the replicas queued.
+type RestartResult struct {
+	Queued []string `json:"queued"`
+}
+
+func (a *API) restart(w http.ResponseWriter, r *http.Request) {
+	var req RestartRequest
+	if err := json.NewDecoder(io.LimitReader(r.Body, 1<<16)).Decode(&req); err != nil {
+		writeError(w, http.StatusBadRequest, "the restart request is not valid JSON: "+err.Error())
+		return
+	}
+	ids := slices.Clone(req.Replicas)
+	for _, rep := range a.ctl.Reconciler().ReadyReplicas() {
+		if req.All || slices.Contains(req.Models, rep.Model) {
+			ids = append(ids, rep.ID)
+		}
+	}
+	if len(ids) == 0 {
+		writeError(w, http.StatusBadRequest, "nothing ready matches; name replicas, models, or all")
+		return
+	}
+	if err := a.ctl.Reconciler().Restart(ids...); err != nil {
+		writeError(w, http.StatusNotFound, err.Error())
+		return
+	}
+	writeJSON(w, http.StatusAccepted, RestartResult{Queued: ids})
+}
@@ -191,6 +191,7 @@ func StartShardFromCommand(cmd reconciler.ShardCommand) *llmgwv1.StartShard {
 		Pool:             cmd.Pool,
 		Fabric:           cmd.Fabric,
 		ProfileFp:        cmd.ProfileFP,
+		ApiKey:           cmd.Spec.APIKey,
 	}
 }

@@ -211,6 +212,7 @@ func ShardSpecFromStart(m *llmgwv1.StartShard) (engines.ShardSpec, time.Duration
 		Pool:          m.GetPool(),
 		Fabric:        m.GetFabric(),
 		ProfileFP:     m.GetProfileFp(),
+		APIKey:        m.GetApiKey(),
 		Profile: engines.Profile{
 			Name: p.GetName(), Engine: p.GetEngine(), GPUClass: p.GetGpuClass(),
 			TP: int(p.GetTp()), PP: int(p.GetPp()),
@@ -302,6 +302,13 @@ func (m Model) validate() error {
 		}
 		seen[p.Name] = struct{}{}

+		for k := range p.Env {
+			if bad, why := engines.UnsafeEnv(k); bad {
+				return fmt.Errorf("model %q profile %q: env %s is not allowed: %s",
+					m.Name, p.Name, k, why)
+			}
+		}
+
 		if p.GPUClass == "" && !cpuCapable(p.Engine) {
 			return fmt.Errorf("model %q profile %q: gpu_class is required for engine %q",
 				m.Name, p.Name, p.Engine)
@@ -302,3 +302,22 @@ func TestSimulatedNeedsEveryProfileToBeSimulated(t *testing.T) {
 		})
 	}
 }
+
+func TestAProfileCannotSetWhatRunsCodeInTheEngine(t *testing.T) {
+	// A profile's env reaches an engine started as root. LD_PRELOAD from a
+	// model definition would make "add a model" into "run this everywhere".
+	for _, k := range []string{"LD_PRELOAD", "ld_library_path", "PYTHONSTARTUP", "PATH", "BASH_ENV"} {
+		m := Model{Name: "m", Source: "http://x/m.gguf", Profiles: []Profile{{
+			Name: "cpu", Engine: "llamacpp", Env: map[string]string{k: "/tmp/x"},
+		}}}
+		if _, err := New([]Model{m}); err == nil || !strings.Contains(err.Error(), "not allowed") {
+			t.Errorf("%s: %v", k, err)
+		}
+	}
+	ok := Model{Name: "m", Source: "http://x/m.gguf", Profiles: []Profile{{
+		Name: "cpu", Engine: "llamacpp", Env: map[string]string{"OMP_NUM_THREADS": "4", "PYTHONUNBUFFERED": "1"},
+	}}}
+	if _, err := New([]Model{ok}); err != nil {
+		t.Errorf("ordinary tuning refused: %v", err)
+	}
+}
@@ -4,6 +4,9 @@ package control

 import (
 	"context"
+	"crypto/hmac"
+	"crypto/sha256"
+	"encoding/hex"
 	"fmt"
 	"log/slog"
 	"strings"
@@ -304,6 +307,7 @@ func (c *Controller) endpoints() []endpoint.Info {
 			ID:       rep.ID,
 			URL:      strings.TrimRight(rep.Endpoint, "/"),
 			Models:   names,
+			APIKey:   c.engineKey(rep.ID),
 			Engine:   prof.Engine,
 			Pool:     rep.Pool,
 			Weight:   rep.Weight,
@@ -321,3 +325,29 @@ func fingerprint(infos []endpoint.Info) string {
 	}
 	return b.String()
 }
+
+// EngineKey is the key a replica's engine demands, empty when engines run
+// without one.
+func (c *Controller) EngineKey(replicaID string) string { return c.engineKey(replicaID) }
+
+func (c *Controller) engineKey(replicaID string) string {
+	if c.cfg.Reconciler.EngineKey == nil {
+		return ""
+	}
+	return c.cfg.Reconciler.EngineKey(replicaID)
+}
+
+// EngineKeys derives each replica's engine key from one secret.
+//
+// HMAC rather than a stored random key per replica: the router needs the key
+// for every replica it sends to, including ones adopted after a restart that
+// this process never started, and a second control plane has to agree. A
+// function of a shared secret and the replica id gives all three without a
+// table to keep in step. A key taken from one node opens that replica only.
+func EngineKeys(secret []byte) func(replicaID string) string {
+	return func(replicaID string) string {
+		m := hmac.New(sha256.New, secret)
+		m.Write([]byte("llmgw engine key\x00" + replicaID))
+		return "llmgw-" + hex.EncodeToString(m.Sum(nil))[:40]
+	}
+}
@@ -0,0 +1,19 @@
+package control
+
+import "testing"
+
+func TestEngineKeysAreTheSameForTheSameReplicaAndNoOther(t *testing.T) {
+	// A control plane that restarts, or a second one, must compute the key a
+	// running engine was started with; a key taken from one node must open
+	// that replica only.
+	a, b := EngineKeys([]byte("s")), EngineKeys([]byte("s"))
+	if a("r1") != b("r1") {
+		t.Error("the same secret gave two keys for one replica")
+	}
+	if a("r1") == a("r2") {
+		t.Error("two replicas share a key")
+	}
+	if a("r1") == EngineKeys([]byte("other"))("r1") {
+		t.Error("the key does not depend on the secret")
+	}
+}
@@ -3,6 +3,7 @@ package reconciler
 import (
 	"context"
 	crypterand "crypto/rand"
+	"errors"
 	"fmt"
 	"log/slog"
 	"math/rand/v2"
@@ -69,6 +70,11 @@ type Config struct {
 	PortRange [2]int
 	// MgmtInterface is the ethernet interface used for NCCL bootstrap.
 	MgmtInterface string
+	// EngineKey gives each replica the key its engine demands. Derived from
+	// the replica id rather than stored, so a control plane that restarts —
+	// or a second one — computes the same key for a replica it adopts. Nil
+	// starts engines without one.
+	EngineKey func(replicaID string) string
 	// EvictionGrace is how long a replica must have gone unasked-for before it
 	// may be stopped to make room for another model.
 	//
@@ -199,6 +205,10 @@ type Reconciler struct {
 	// because it is state about this moment and not about what the model is.
 	// A model nobody has wanted has no entry and is simply not running.
 	wanted map[string]time.Time
+	// restartQueue holds, per model, the replicas an operator asked to be
+	// restarted and not yet begun. One is replaced at a time: replacing them
+	// all at once would need the memory of every replica twice over.
+	restartQueue map[string][]string

 	wake chan struct{}
 	stop chan struct{}
@@ -227,6 +237,7 @@ func New(cfg Config, cat *catalog.Live, inv *inventory.Store, disp Dispatcher, l
 		attempts:      map[string]int{},
 		reservedPorts: map[int]struct{}{},
 		wanted:        map[string]time.Time{},
+		restartQueue:  map[string][]string{},
 		lastExplain:   map[string]placement.Result{},
 		wake:          make(chan struct{}, 1),
 		stop:          make(chan struct{}),
@@ -703,19 +714,37 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
 	// pointer stops being safe the moment it is in the map: an agent's status
 	// report can mutate it from another goroutine at any time.
 	r.mu.Lock()
-	var liveIDs, drainingIDs, avoid []string
+	var liveIDs, drainingIDs, retiringIDs, avoid []string
+	readyLive := 0
 	for _, rep := range r.replicas {
 		if rep.Model != m.Name {
 			continue
 		}
 		switch {
+		case rep.State.Counts() && !rep.RetiringAt.IsZero():
+			retiringIDs = append(retiringIDs, rep.ID)
+			avoid = append(avoid, rep.NodeIDs()...)
 		case rep.State.Counts():
 			liveIDs = append(liveIDs, rep.ID)
 			avoid = append(avoid, rep.NodeIDs()...)
+			if rep.State == Ready {
+				readyLive++
+			}
 		case rep.State == Draining:
 			drainingIDs = append(drainingIDs, rep.ID)
 		}
 	}
+	// The next restart an operator queued, once the last one is done.
+	if len(retiringIDs) == 0 {
+		if id, ok := r.nextRestartLocked(m.Name); ok {
+			r.replicas[id].RetiringAt = r.now()
+			retiringIDs = append(retiringIDs, id)
+			liveIDs = slices.DeleteFunc(liveIDs, func(x string) bool { return x == id })
+			readyLive--
+			r.log.Info("restarting a replica: placing its replacement first",
+				"replica", id, "model", m.Name)
+		}
+	}
 	blockedUntil := r.nextAttempt[m.Name]
 	r.mu.Unlock()

@@ -758,6 +787,112 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
 			r.stopReplica(ctx, id)
 		}
 	}
+
+	r.retire(ctx, retiringIDs, readyLive, len(liveIDs), desired)
+}
+
+// retire finishes restarting replicas.
+//
+// A retiring replica keeps serving until its replacement is ready, and is then
+// drained like any other. When no replacement could be placed at all —
+// nowhere else has the memory, which on a small CPU fleet is the usual case —
+// it is stopped after DrainGrace so its replacement can take its place: a
+// restart in place, with the model unserved while it loads if it had no other
+// replica. Said in the log, since that is the one case with an outage.
+func (r *Reconciler) retire(ctx context.Context, ids []string, readyLive, live, desired int) {
+	for _, id := range ids {
+		r.mu.Lock()
+		rep, ok := r.replicas[id]
+		var since time.Time
+		if ok {
+			since = rep.RetiringAt
+		}
+		r.mu.Unlock()
+		if !ok {
+			continue
+		}
+		switch {
+		case readyLive >= desired:
+			r.drainForRestart(ctx, id)
+		case live < desired && r.now().Sub(since) > r.cfg.DrainGrace:
+			r.log.Warn("restarting a replica in place: no room for its replacement beside it",
+				"replica", id, "after", r.cfg.DrainGrace)
+			r.drainForRestart(ctx, id)
+			r.stopReplica(ctx, id)
+		}
+	}
+}
+
+func (r *Reconciler) drainForRestart(ctx context.Context, id string) {
+	r.mu.Lock()
+	rep, ok := r.replicas[id]
+	var shards []Shard
+	if ok {
+		rep.State, rep.DrainingAt, rep.RetiringAt = Draining, r.now(), time.Time{}
+		shards = slices.Clone(rep.Shards)
+	}
+	r.mu.Unlock()
+	for _, s := range shards {
+		if err := r.disp.DrainShard(ctx, s.NodeID, s.ID); err != nil {
+			r.log.Warn("draining a shard failed", "shard", s.ID, "err", err)
+		}
+	}
+}
+
+// nextRestartLocked pops the next queued replica of a model that is still
+// ready to be restarted. Called with mu held.
+func (r *Reconciler) nextRestartLocked(model string) (string, bool) {
+	q := r.restartQueue[model]
+	for len(q) > 0 {
+		id := q[0]
+		q = q[1:]
+		if rep, ok := r.replicas[id]; ok && rep.State == Ready && rep.RetiringAt.IsZero() {
+			r.restartQueue[model] = q
+			return id, true
+		}
+	}
+	delete(r.restartQueue, model)
+	return "", false
+}
+
+// ErrNoSuchReplica means no ready replica has that id.
+var ErrNoSuchReplica = errors.New("no ready replica with that id")
+
+// Restart queues replicas to be replaced, each by a new one started beside it
+// before it is stopped — to pick up an edited profile, an engine key, a new
+// engine version. Replicas of one model are replaced one at a time.
+func (r *Reconciler) Restart(ids ...string) error {
+	r.mu.Lock()
+	for _, id := range ids {
+		rep, ok := r.replicas[id]
+		if !ok || rep.State != Ready {
+			r.mu.Unlock()
+			return fmt.Errorf("%w: %s", ErrNoSuchReplica, id)
+		}
+		if !slices.Contains(r.restartQueue[rep.Model], id) && rep.RetiringAt.IsZero() {
+			r.restartQueue[rep.Model] = append(r.restartQueue[rep.Model], id)
+		}
+	}
+	r.mu.Unlock()
+	r.Notify()
+	return nil
+}
+
+// RestartState says where a replica's restart stands: "queued", "replacing",
+// or "" when none was asked for.
+func (r *Reconciler) RestartState(id string) string {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	rep, ok := r.replicas[id]
+	switch {
+	case !ok:
+		return ""
+	case !rep.RetiringAt.IsZero():
+		return "replacing"
+	case slices.Contains(r.restartQueue[rep.Model], id):
+		return "queued"
+	}
+	return ""
 }

 // scaleUp places and starts one replica per pass.
@@ -883,6 +1018,7 @@ func (r *Reconciler) startReplica(ctx context.Context, m catalog.Model, pl *plac
 				MgmtInterface: r.cfg.MgmtInterface,
 				HTTPPort:      s.HTTPPort,
 				Profile:       espec,
+				APIKey:        r.engineKey(id),
 			},
 			StartupTimeout: prof.StartupTimeout,
 			Pool:           pl.Pool,
@@ -1380,3 +1516,10 @@ func (r *Reconciler) profileMemory(model, profile string) int {
 	}
 	return 0
 }
+
+func (r *Reconciler) engineKey(replicaID string) string {
+	if r.cfg.EngineKey == nil {
+		return ""
+	}
+	return r.cfg.EngineKey(replicaID)
+}
@@ -1721,3 +1721,118 @@ func TestAnAgentTooOldToReportTheProfileVersionIsNotGuessedAt(t *testing.T) {
 		t.Errorf("an unreported fingerprint should stay empty, got %q", fp)
 	}
 }
+
+func TestAnEngineIsStartedWithItsReplicasKey(t *testing.T) {
+	// Engines listen on every address a node has. Started without a key, any
+	// machine that could reach the port reached the model past the gateway's
+	// authentication, quotas and accounting — checked from a neighbouring node
+	// on the live cluster.
+	h := newHarness(t, testModel(1), gpuNode("1", 8))
+	h.r.cfg.EngineKey = func(id string) string { return "key-for-" + id }
+
+	h.r.Reconcile(t.Context())
+	started := h.disp.startedShards()
+	if len(started) == 0 {
+		t.Fatal("nothing was started")
+	}
+	for _, c := range started {
+		if c.Spec.APIKey != "key-for-"+c.ReplicaID {
+			t.Errorf("shard %s started with key %q", c.ShardID, c.Spec.APIKey)
+		}
+	}
+}
+
+func replicaByID(h *harness, id string) (Replica, bool) {
+	for _, rep := range h.r.Replicas() {
+		if rep.ID == id {
+			return rep, true
+		}
+	}
+	return Replica{}, false
+}
+
+func TestARestartedReplicaServesUntilItsReplacementIsReady(t *testing.T) {
+	// Restarting is how a running engine picks up a key, an edited profile or
+	// a new engine version. Stopping first would leave a one-replica model
+	// unserved for as long as its replacement takes to load.
+	h := newHarness(t, testModel(1), gpuNode("1", 4), gpuNode("2", 4))
+	h.r.Reconcile(t.Context())
+	h.markReady()
+	old := h.r.ReadyReplicas()[0].ID
+
+	if err := h.r.Restart(old); err != nil {
+		t.Fatal(err)
+	}
+	h.r.Reconcile(t.Context())
+	if rep, _ := replicaByID(h, old); rep.State != Ready || h.r.RestartState(old) != "replacing" {
+		t.Fatalf("the old replica stopped serving before its replacement was up: %+v", rep)
+	}
+	if n := len(h.r.Replicas()); n != 2 {
+		t.Fatalf("no replacement was placed beside it: %d replicas", n)
+	}
+
+	h.markReady()
+	h.r.Reconcile(t.Context())
+	h.r.Reconcile(t.Context())
+	if _, still := replicaByID(h, old); still {
+		if rep, _ := replicaByID(h, old); rep.State != Draining && rep.State != Stopped {
+			t.Errorf("the old replica is %s after its replacement became ready", rep.State)
+		}
+	}
+	if got := len(h.r.ReadyReplicas()); got != 1 {
+		t.Errorf("%d ready replicas at the end; the model asks for 1", got)
+	}
+}
+
+func TestARestartWithNoRoomBesideItHappensInPlace(t *testing.T) {
+	// One node, room for one replica: the replacement can only start once the
+	// old one has gone. Waiting forever would mean never restarting at all.
+	h := newHarness(t, testModel(1), gpuNode("1", 4))
+	h.r.Reconcile(t.Context())
+	h.markReady()
+	old := h.r.ReadyReplicas()[0].ID
+	if err := h.r.Restart(old); err != nil {
+		t.Fatal(err)
+	}
+	h.r.Reconcile(t.Context())
+	if rep, _ := replicaByID(h, old); rep.State != Ready {
+		t.Fatalf("stopped before the grace period: %s", rep.State)
+	}
+
+	h.advance(h.r.cfg.DrainGrace + time.Second)
+	h.r.Reconcile(t.Context())
+	if rep, ok := replicaByID(h, old); ok && rep.State == Ready {
+		t.Fatal("still serving after the grace period with no replacement possible")
+	}
+	for i := 0; i < 3; i++ {
+		h.r.Reconcile(t.Context())
+		h.markReady()
+	}
+	ready := h.r.ReadyReplicas()
+	if len(ready) != 1 || ready[0].ID == old {
+		t.Errorf("expected one new replica in its place, got %+v", ready)
+	}
+}
+
+func TestReplicasOfAModelAreRestartedOneAtATime(t *testing.T) {
+	h := newHarness(t, testModel(2), gpuNode("1", 8), gpuNode("2", 8), gpuNode("3", 8))
+	for i := 0; i < 3; i++ {
+		h.r.Reconcile(t.Context())
+		h.markReady()
+	}
+	var ids []string
+	for _, rep := range h.r.ReadyReplicas() {
+		ids = append(ids, rep.ID)
+	}
+	if err := h.r.Restart(ids...); err != nil {
+		t.Fatal(err)
+	}
+	h.r.Reconcile(t.Context())
+	states := []string{h.r.RestartState(ids[0]), h.r.RestartState(ids[1])}
+	if !slices.Contains(states, "replacing") || !slices.Contains(states, "queued") {
+		t.Errorf("expected one replacing and one queued, got %v", states)
+	}
+	if err := h.r.Restart("no-such"); err == nil {
+		t.Error("an unknown replica was accepted")
+	}
+}
@@ -156,6 +156,10 @@ type Replica struct {
 	// DrainingAt is when the replica stopped taking new work, so the wait for a
 	// replacement can be bounded.
 	DrainingAt time.Time
+	// RetiringAt is set while the replica is being restarted: it keeps serving
+	// but no longer counts toward the model's replicas, so a replacement is
+	// placed beside it, and it is drained only once that replacement is ready.
+	RetiringAt time.Time
 	// Attempt counts how many times this model has failed to come up in a row,
 	// carried across rebuilds so the backoff does not reset on every attempt.
 	Attempt int
@@ -7,7 +7,6 @@ import (
 	"net/http"
 	"os"
 	"path/filepath"
-	"strings"
 	"time"
 )

@@ -29,13 +28,22 @@ type Added struct {
 // the layout — a version that can be overwritten takes away the ability to
 // change the catalog back and still have the old weights.
 func Add(ctx context.Context, root, source string, e Entry, progress func(done, total int64)) (Added, error) {
+	return addWith(ctx, http.DefaultClient, root, source, e, progress)
+}
+
+// addWith is Add through a given client. The management API passes one that
+// will not reach private addresses; `llmgw mirror add`, run by somebody on
+// the mirror, passes the ordinary one.
+func addWith(ctx context.Context, client *http.Client, root, source string, e Entry,
+	progress func(done, total int64),
+) (Added, error) {
 	if e.Model == "" || e.Version == "" || e.File == "" {
 		return Added{}, fmt.Errorf("a model, a version and a filename are all required")
 	}
-	if strings.ContainsAny(e.Model+e.Version+e.File, `/\`) {
-		// Each is one path element. A slash would put the file somewhere the
-		// layout does not describe, and List would not find it again.
-		return Added{}, fmt.Errorf("model, version and filename are single names, not paths")
+	for _, n := range []string{e.Model, e.Version, e.File} {
+		if err := ValidName(n); err != nil {
+			return Added{}, err
+		}
 	}

 	final := e.Path(root)
@@ -50,7 +58,7 @@ func Add(ctx context.Context, root, source string, e Entry, progress func(done,
 	if err != nil {
 		return Added{}, err
 	}
-	resp, err := http.DefaultClient.Do(req)
+	resp, err := client.Do(req)
 	if err != nil {
 		return Added{}, fmt.Errorf("fetching %s: %w", source, err)
 	}
@@ -130,8 +130,8 @@ func (a *API) status(w http.ResponseWriter, r *http.Request) {

 func (a *API) remove(w http.ResponseWriter, r *http.Request) {
 	model, version := r.PathValue("model"), r.PathValue("version")
-	if strings.ContainsAny(model+version, `/\`) {
-		fail(w, http.StatusBadRequest, errors.New("model and version are single names, not paths"))
+	if err := errors.Join(ValidName(model), ValidName(version)); err != nil {
+		fail(w, http.StatusBadRequest, err)
 		return
 	}

@@ -17,7 +17,10 @@ import (
 func testAPI(t *testing.T) (*API, string) {
 	t.Helper()
 	root := t.TempDir()
-	a := NewAPI(NewManager(root, discardLog()), "http://mirror:8081")
+	m := NewManager(root, discardLog())
+	// The upstreams in these tests are httptest servers on loopback.
+	m.AllowPrivateSources()
+	a := NewAPI(m, "http://mirror:8081")

 	// A download outlives the request that started it — that is the point of
 	// the manager. So the temp directory must not be torn down while one is
@@ -434,3 +437,33 @@ func TestRemovingAVersionIsRefusedWhileAnyShardIsArriving(t *testing.T) {
 		t.Errorf("expected 409 while a shard is arriving, got %d", code)
 	}
 }
+
+func TestTheAPIWillNotFetchFromThePrivateNetwork(t *testing.T) {
+	// The file server hands whatever was fetched to anyone, so a fetch that
+	// could reach 169.254.169.254 or a database on the LAN was a way to read
+	// them through the mirror.
+	m := NewManager(t.TempDir(), discardLog())
+	for _, src := range []string{
+		"http://127.0.0.1:1/x.gguf", "http://169.254.169.254/latest/meta-data",
+		"http://10.0.0.140:9090/api/v1/nodes", "http://[::1]:1/x",
+	} {
+		st, err := m.Start(src, Entry{Model: "m", Version: "v", File: "x.gguf"})
+		if err != nil {
+			t.Fatalf("%s: %v", src, err)
+		}
+		deadline := time.Now().Add(5 * time.Second)
+		for st.State == Fetching && time.Now().Before(deadline) {
+			time.Sleep(10 * time.Millisecond)
+			st, _ = m.Status("m", "v")
+		}
+		if st.State != Failed || !strings.Contains(st.Err, "refusing to fetch") {
+			t.Errorf("%s: %+v", src, st)
+		}
+		m.Forget("m", "v")
+	}
+	for _, src := range []string{"file:///etc/passwd", "gopher://x/", "not a url"} {
+		if _, err := m.Start(src, Entry{Model: "m", Version: "v", File: "x"}); err == nil {
+			t.Errorf("%s was accepted as a source", src)
+		}
+	}
+}
@@ -5,6 +5,7 @@ import (
 	"errors"
 	"fmt"
 	"log/slog"
+	"net/http"
 	"os"
 	"sync"
 	"time"
@@ -72,8 +73,18 @@ type Manager struct {
 	// until the first had finished. A fifteen-shard model had to be pulled one
 	// shard at a time, each waiting on the last.
 	running map[string]*Status
+
+	// client fetches sources. It will not reach private addresses unless
+	// AllowPrivateSources was called.
+	client *http.Client
 }

+// AllowPrivateSources lets downloads reach loopback, private and link-local
+// addresses — a model server on the same network, say. Off by default: the
+// file server hands whatever was fetched to anyone, so a fetch that can reach
+// the private network is a way to read it.
+func (m *Manager) AllowPrivateSources() { m.client = fetchClient(true) }
+
 // runKey identifies one download.
 func runKey(e Entry) string { return e.Model + "/" + e.Version + "/" + e.File }

@@ -82,7 +93,7 @@ func NewManager(root string, log *slog.Logger) *Manager {
 	if log == nil {
 		log = slog.Default()
 	}
-	return &Manager{root: root, log: log, running: map[string]*Status{}}
+	return &Manager{root: root, log: log, running: map[string]*Status{}, client: fetchClient(false)}
 }

 // ErrAlreadyFetching means a download for that model and version is running.
@@ -97,6 +108,9 @@ func (m *Manager) Start(source string, e Entry) (Status, error) {
 	if source == "" {
 		return Status{}, fmt.Errorf("a source URL is required")
 	}
+	if err := validSource(source); err != nil {
+		return Status{}, err
+	}
 	key := runKey(e)

 	m.mu.Lock()
@@ -125,7 +139,7 @@ func (m *Manager) run(source string, e Entry, key string) {
 	// minutes to hours and the thing that asked for it is long gone.
 	ctx := context.Background()

-	added, err := Add(ctx, m.root, source, e, func(done, total int64) {
+	added, err := addWith(ctx, m.client, m.root, source, e, func(done, total int64) {
 		m.mu.Lock()
 		if st, ok := m.running[key]; ok {
 			st.BytesDone, st.BytesTotal = done, total
@@ -37,6 +37,21 @@ type Entry struct {
 	Bytes   int64
 }

+// ValidName checks one element of an entry — a model, version or file name.
+//
+// Each is one path element, and a name that is not one writes somewhere the
+// layout does not describe. Rejecting slashes was not enough: ".." is a
+// single element too, and a model called ".." put its file beside the mirror
+// root rather than inside it. A name starting with a dot is refused as well,
+// because that is where the mirror keeps its own bookkeeping.
+func ValidName(n string) error {
+	if n == "" || n[0] == '.' || strings.ContainsAny(n, "/\\\x00") {
+		return fmt.Errorf("%q is not a usable name: it must be a single path element "+
+			"that does not start with a dot", n)
+	}
+	return nil
+}
+
 // Path is where this entry lives under a root.
 func (e Entry) Path(root string) string {
 	return filepath.Join(root, e.Model, e.Version, e.File)
@@ -162,3 +162,18 @@ func TestProgressIsReportedForALongDownload(t *testing.T) {
 		t.Error("a download reported no progress at all")
 	}
 }
+
+func TestANameThatLeavesTheMirrorIsRefused(t *testing.T) {
+	root := t.TempDir()
+	for _, e := range []Entry{
+		{Model: "..", Version: "v", File: "f.gguf"},
+		{Model: "m", Version: "..", File: "f.gguf"},
+		{Model: "m", Version: "v", File: ".f.sha256"},
+		{Model: "a/b", Version: "v", File: "f.gguf"},
+	} {
+		if _, err := Add(t.Context(), root, "http://127.0.0.1:1/x", e, nil); err == nil ||
+			!strings.Contains(err.Error(), "not a usable name") {
+			t.Errorf("%+v: %v", e, err)
+		}
+	}
+}
@@ -0,0 +1,86 @@
+package mirror
+
+import (
+	"errors"
+	"fmt"
+	"net"
+	"net/http"
+	"net/netip"
+	"net/url"
+	"syscall"
+	"time"
+)
+
+// errPrivateSource is a fetch refused because it would reach this machine or
+// the network behind it rather than the internet.
+var errPrivateSource = errors.New("refusing to fetch from a loopback, private or link-local address")
+
+// fetchClient downloads sources for the management API.
+//
+// The API fetches whatever URL it is given and the file server then hands the
+// result to anyone. Without this, "fetch http://169.254.169.254/..." was a way
+// to read a cloud's instance metadata — or anything else on the network the
+// mirror sits in — through the mirror. The check is made on the address
+// actually dialled, after resolution and on every redirect, so a name that
+// resolves somewhere public and then somewhere private is caught too.
+func fetchClient(allowPrivate bool) *http.Client {
+	d := &net.Dialer{Timeout: 30 * time.Second, KeepAlive: 30 * time.Second}
+	if !allowPrivate {
+		d.Control = func(_, address string, _ syscall.RawConn) error {
+			host, _, err := net.SplitHostPort(address)
+			if err != nil {
+				return err
+			}
+			a, err := netip.ParseAddr(host)
+			if err != nil {
+				return err
+			}
+			if blockedAddr(a.Unmap()) {
+				return fmt.Errorf("%w (%s)", errPrivateSource, a)
+			}
+			return nil
+		}
+	}
+	return &http.Client{
+		Transport: &http.Transport{
+			Proxy:                 http.ProxyFromEnvironment,
+			DialContext:           d.DialContext,
+			TLSHandshakeTimeout:   30 * time.Second,
+			ResponseHeaderTimeout: 2 * time.Minute,
+			ForceAttemptHTTP2:     true,
+		},
+		// No overall timeout: a model is tens of gigabytes. The dial, the
+		// handshake and the first byte are bounded above; after that a stall
+		// is the context's to end.
+		CheckRedirect: func(req *http.Request, via []*http.Request) error {
+			if len(via) >= 10 {
+				return errors.New("too many redirects")
+			}
+			return checkScheme(req.URL)
+		},
+	}
+}
+
+func blockedAddr(a netip.Addr) bool {
+	cgnat := netip.MustParsePrefix("100.64.0.0/10")
+	return a.IsLoopback() || a.IsPrivate() || a.IsLinkLocalUnicast() ||
+		a.IsLinkLocalMulticast() || a.IsMulticast() || a.IsUnspecified() || cgnat.Contains(a)
+}
+
+// checkScheme allows the two schemes a model is published over. file:// and
+// the rest would read this machine's own disk.
+func checkScheme(u *url.URL) error {
+	if u.Scheme != "http" && u.Scheme != "https" {
+		return fmt.Errorf("%q is not an http or https source", u.Redacted())
+	}
+	return nil
+}
+
+// validSource checks a source before a download is started for it.
+func validSource(source string) error {
+	u, err := url.Parse(source)
+	if err != nil || u.Host == "" {
+		return fmt.Errorf("%q is not a URL", source)
+	}
+	return checkScheme(u)
+}
@@ -332,7 +332,10 @@ func certNames(path string) ([]string, error) {

 // --- binary ---

-var releaseTag = regexp.MustCompile(`^v\d+\.\d+\.\d+$`)
+// releaseTag is a version CI published: v0.10.4, or a release candidate of
+// it, v0.10.4-rc.2. Anything else — a build with commits past its tag — has no
+// release to download.
+var releaseTag = regexp.MustCompile(`^v\d+\.\d+\.\d+(-rc\.\d+)?$`)

 func (r *Runner) binary(ctx context.Context, res *Result) error {
 	v := r.Opts.Version
@@ -0,0 +1,14 @@
+package onboard
+
+import "testing"
+
+func TestAReleaseCandidateIsARelease(t *testing.T) {
+	for v, want := range map[string]bool{
+		"v0.10.4": true, "v0.10.4-rc.1": true, "v0.10.4-rc.12": true,
+		"v0.10.3-5-gad6d2d3": false, "v0.10.4-rc": false, "v0.10.3-a": false, "dev": false,
+	} {
+		if got := releaseTag.MatchString(v); got != want {
+			t.Errorf("%s: %v", v, got)
+		}
+	}
+}
@@ -0,0 +1,75 @@
+package router
+
+import (
+	"net"
+	"net/http"
+	"net/netip"
+	"net/url"
+	"strings"
+)
+
+// AdminGuard keeps a browser from being used against the admin listener.
+//
+// The listener has no login: reaching it is meant to mean holding the machine,
+// so it binds to loopback and an operator reaches it over an ssh tunnel. But a
+// browser on that operator's machine reaches it too, from any page it has
+// open. A page could POST a model whose profile runs as root on every node,
+// or — by pointing its own domain at 127.0.0.1 — read pprof's heap, which
+// holds API keys in the clear. Three checks close that without touching what
+// ctl and curl do:
+//
+//   - Host must be an address or localhost. A page that rebinds its own name
+//     to 127.0.0.1 still sends that name.
+//   - A request carrying Origin or Sec-Fetch-Site from another site is
+//     refused. Browsers send them; ctl and curl do not.
+//   - Anything but GET and HEAD must be application/json. A page can send a
+//     form or text/plain anywhere without asking; JSON across origins needs a
+//     preflight this listener never grants.
+func AdminGuard(next http.Handler) http.Handler {
+	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		if !addressHost(r.Host) {
+			http.Error(w, "the admin listener answers to an address or localhost, not a name", http.StatusMisdirectedRequest)
+			return
+		}
+		if crossSite(r) {
+			http.Error(w, "cross-site request refused", http.StatusForbidden)
+			return
+		}
+		if r.Method != http.MethodGet && r.Method != http.MethodHead {
+			ct := r.Header.Get("Content-Type")
+			if mt, _, _ := strings.Cut(ct, ";"); strings.TrimSpace(strings.ToLower(mt)) != "application/json" {
+				http.Error(w, "a change to the admin API must be sent as application/json", http.StatusUnsupportedMediaType)
+				return
+			}
+		}
+		next.ServeHTTP(w, r)
+	})
+}
+
+func addressHost(hostport string) bool {
+	host := hostport
+	if h, _, err := net.SplitHostPort(hostport); err == nil {
+		host = h
+	}
+	host = strings.Trim(host, "[]")
+	if strings.EqualFold(host, "localhost") {
+		return true
+	}
+	_, err := netip.ParseAddr(host)
+	return err == nil
+}
+
+func crossSite(r *http.Request) bool {
+	switch r.Header.Get("Sec-Fetch-Site") {
+	case "", "same-origin", "none":
+	default:
+		return true
+	}
+	if o := r.Header.Get("Origin"); o != "" {
+		u, err := url.Parse(o)
+		if err != nil || !strings.EqualFold(u.Host, r.Host) {
+			return true
+		}
+	}
+	return false
+}
@@ -0,0 +1,57 @@
+package router
+
+import (
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"testing"
+)
+
+func TestTheAdminListenerRefusesWhatABrowserPageCouldSend(t *testing.T) {
+	h := AdminGuard(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {}))
+	do := func(method, host, ct string, hdr map[string]string) int {
+		r := httptest.NewRequest(method, "/api/v1/models", strings.NewReader("{}"))
+		r.Host = host
+		if ct != "" {
+			r.Header.Set("Content-Type", ct)
+		}
+		for k, v := range hdr {
+			r.Header.Set(k, v)
+		}
+		w := httptest.NewRecorder()
+		h.ServeHTTP(w, r)
+		return w.Code
+	}
+
+	// What ctl and curl send.
+	if got := do("POST", "127.0.0.1:9090", "application/json", nil); got != 200 {
+		t.Errorf("ctl's request was refused: %d", got)
+	}
+	if got := do("GET", "localhost:9090", "", nil); got != 200 {
+		t.Errorf("a GET through an ssh tunnel was refused: %d", got)
+	}
+	if got := do("DELETE", "[::1]:9090", "application/json", nil); got != 200 {
+		t.Errorf("IPv6 loopback refused: %d", got)
+	}
+
+	// A page posting a form, or text/plain, needs no preflight.
+	if got := do("POST", "127.0.0.1:9090", "text/plain", nil); got != http.StatusUnsupportedMediaType {
+		t.Errorf("text/plain POST accepted: %d", got)
+	}
+	if got := do("POST", "127.0.0.1:9090", "", nil); got != http.StatusUnsupportedMediaType {
+		t.Errorf("a bodiless POST accepted: %d", got)
+	}
+	// DNS rebinding: the page's own name, now resolving to 127.0.0.1.
+	if got := do("GET", "evil.example:9090", "", nil); got != http.StatusMisdirectedRequest {
+		t.Errorf("a rebound name was answered: %d", got)
+	}
+	// And anything a browser marks as coming from elsewhere.
+	if got := do("POST", "127.0.0.1:9090", "application/json",
+		map[string]string{"Sec-Fetch-Site": "cross-site"}); got != http.StatusForbidden {
+		t.Errorf("cross-site fetch accepted: %d", got)
+	}
+	if got := do("POST", "127.0.0.1:9090", "application/json",
+		map[string]string{"Origin": "https://evil.example"}); got != http.StatusForbidden {
+		t.Errorf("foreign Origin accepted: %d", got)
+	}
+}
@@ -22,6 +22,9 @@ type Info struct {
 	ID string
 	// URL is the engine's root address (e.g. http://gpu-a-01:8000), no trailing slash.
 	URL string
+	// APIKey is what the engine demands of every request, sent as a bearer
+	// token. Empty for an engine that asks for none.
+	APIKey string
 	// Models lists the model names this replica serves.
 	Models []string
 	// Engine names the inference engine serving here, which decides what its
@@ -148,6 +148,8 @@ type Request struct {
 	Body []byte
 	// Header carries headers to forward (authorization is deliberately not).
 	Header http.Header
+	// APIKey is the engine's own key, sent in place of the client's.
+	APIKey string
 	// Stream says whether an SSE response is expected.
 	Stream bool
 	// SuppressUsageChunk drops the final usage-only chunk from the client while
@@ -218,6 +220,9 @@ func (p *Proxy) Do(ctx context.Context, w http.ResponseWriter, r Request) (Outco
 	}
 	req.Header = cloneForUpstream(r.Header)
 	req.Header.Set("Content-Type", "application/json")
+	if r.APIKey != "" {
+		req.Header.Set("Authorization", "Bearer "+r.APIKey)
+	}
 	req.ContentLength = int64(len(r.Body))

 	// A non-streaming request waits for the whole completion before a single
@@ -518,8 +523,7 @@ func copyHeaders(dst, src http.Header) {

 // cloneForUpstream forwards only what the engine needs. The client's
 // Authorization header is deliberately dropped: it authenticates the caller to
-// llmgw, not llmgw to the engine, and engines are reached over a trusted
-// network (plan §14).
+// llmgw, not llmgw to the engine. llmgw's own key for the engine is set by Do.
 func cloneForUpstream(h http.Header) http.Header {
 	out := make(http.Header, 4)
 	for _, k := range []string{"Accept", "User-Agent", "Traceparent", "Tracestate"} {
@@ -355,3 +355,22 @@ func TestAStreamingRequestKeepsTheShortLivenessTimeout(t *testing.T) {
 		t.Errorf("took %s; the streaming liveness timeout is not being applied", elapsed)
 	}
 }
+
+func TestTheEngineGetsLlmgwsKeyNotTheClients(t *testing.T) {
+	var got string
+	up := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		got = r.Header.Get("Authorization")
+		io.WriteString(w, `{}`)
+	}))
+	defer up.Close()
+
+	client := http.Header{"Authorization": {"Bearer sk-the-callers-own"}}
+	_, err := testProxy().Do(t.Context(), httptest.NewRecorder(),
+		Request{URL: up.URL, Body: []byte(`{}`), Header: client, APIKey: "llmgw-engine"})
+	if err != nil {
+		t.Fatal(err)
+	}
+	if got != "Bearer llmgw-engine" {
+		t.Errorf("the engine was sent %q", got)
+	}
+}
@@ -373,6 +373,7 @@ func toInfos(eps []config.Endpoint) []endpoint.Info {
 		out = append(out, endpoint.Info{
 			ID:       e.ID,
 			URL:      strings.TrimRight(e.URL, "/"),
+			APIKey:   e.APIKey,
 			Models:   e.Models,
 			Engine:   e.Engine,
 			Pool:     e.Pool,
@@ -673,6 +674,7 @@ func (r *Router) dispatchOne(ctx context.Context, w http.ResponseWriter, req *pa

 		out, perr := r.px.Do(ctx, w, proxy.Request{
 			URL:                dec.Endpoint.URL + req.Path,
+			APIKey:             dec.Endpoint.APIKey,
 			Body:               req.UpstreamBody,
 			Header:             req.Header,
 			Stream:             req.Stream,
@@ -157,7 +157,7 @@ func (p *Poller) loop(ctx context.Context, in endpoint.Info) {
 }

 func (p *Poller) once(ctx context.Context, in endpoint.Info, reported *string) {
-	stats, err := p.fetch(ctx, in.URL, p.mappingFor(in))
+	stats, err := p.fetch(ctx, in.URL, in.APIKey, p.mappingFor(in))
 	if err != nil {
 		if ctx.Err() != nil {
 			return
@@ -201,7 +201,7 @@ func (p *Poller) report(in endpoint.Info, err error, reported *string) {
 	*reported = "unreachable"
 }

-func (p *Poller) fetch(ctx context.Context, base string, mapping Mapping) (endpoint.Stats, error) {
+func (p *Poller) fetch(ctx context.Context, base, apiKey string, mapping Mapping) (endpoint.Stats, error) {
 	ctx, cancel := context.WithTimeout(ctx, p.cfg.Timeout)
 	defer cancel()

@@ -209,6 +209,10 @@ func (p *Poller) fetch(ctx context.Context, base string, mapping Mapping) (endpo
 	if err != nil {
 		return endpoint.Stats{}, err
 	}
+	// llama.cpp guards /metrics with its key; vLLM does not, and ignores one.
+	if apiKey != "" {
+		req.Header.Set("Authorization", "Bearer "+apiKey)
+	}
 	resp, err := p.http.Do(req)
 	if err != nil {
 		return endpoint.Stats{}, err
@@ -38,6 +38,10 @@ var copyOrder = []string{
 	"threads",
 	"thread_messages",
 	"memory_notes",
+	// Copied, not regenerated: every running engine was started with a key
+	// derived from this, and a new secret would lock the router out of all
+	// of them at the moment of the move.
+	"cluster_secrets",
 }

 // notCopied are tables deliberately left behind.
@@ -0,0 +1,12 @@
+-- Secrets the control planes share and nobody types: generated on first use,
+-- read by every control plane against this store afterwards.
+--
+-- The first is the one each replica's engine key is derived from. It has to be
+-- the same for every control plane — the router sends a replica the key it was
+-- started with, whichever control plane started it — and the same after a
+-- restart, when running replicas are adopted rather than restarted.
+CREATE TABLE cluster_secrets (
+  name       TEXT PRIMARY KEY,
+  value      TEXT NOT NULL,
+  created_at BIGINT NOT NULL
+);
@@ -0,0 +1,12 @@
+-- Secrets the control planes share and nobody types: generated on first use,
+-- read by every control plane against this store afterwards.
+--
+-- The first is the one each replica's engine key is derived from. It has to be
+-- the same for every control plane — the router sends a replica the key it was
+-- started with, whichever control plane started it — and the same after a
+-- restart, when running replicas are adopted rather than restarted.
+CREATE TABLE cluster_secrets (
+  name       TEXT PRIMARY KEY,
+  value      TEXT NOT NULL,
+  created_at BIGINT NOT NULL
+);
@@ -0,0 +1,31 @@
+package store
+
+import (
+	"context"
+	"crypto/rand"
+	"encoding/hex"
+	"time"
+)
+
+// ClusterSecret returns the named secret, creating a random one the first time
+// it is asked for.
+//
+// Insert-if-absent and then read, so two control planes asking at once end up
+// with the same value: the loser's insert changes nothing and it reads the
+// winner's.
+func (s *DB) ClusterSecret(ctx context.Context, name string) (string, error) {
+	b := make([]byte, 32)
+	if _, err := rand.Read(b); err != nil {
+		return "", err
+	}
+	if _, err := s.db.ExecContext(ctx,
+		`INSERT INTO cluster_secrets (name, value, created_at) VALUES (?, ?, ?)
+		 ON CONFLICT (name) DO NOTHING`,
+		name, hex.EncodeToString(b), time.Now().UnixMilli()); err != nil {
+		return "", err
+	}
+	var v string
+	err := s.db.QueryRowContext(ctx,
+		`SELECT value FROM cluster_secrets WHERE name = ?`, name).Scan(&v)
+	return v, err
+}
@@ -0,0 +1,25 @@
+package store
+
+import (
+	"path/filepath"
+	"testing"
+)
+
+func TestAClusterSecretIsMadeOnceAndKept(t *testing.T) {
+	db, err := Open(t.Context(), filepath.Join(t.TempDir(), "s.db"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer db.Close()
+	a, err := db.ClusterSecret(t.Context(), "engine-key")
+	if err != nil || len(a) != 64 {
+		t.Fatalf("%q %v", a, err)
+	}
+	b, err := db.ClusterSecret(t.Context(), "engine-key")
+	if err != nil || a != b {
+		t.Errorf("a second read made a new secret: %q then %q", a, b)
+	}
+	if c, _ := db.ClusterSecret(t.Context(), "other"); c == a {
+		t.Error("two names share a secret")
+	}
+}
@@ -303,7 +303,7 @@ func TestThePageRendersThroughTheRealClient(t *testing.T) {

 	log := slog.New(slog.NewTextHandler(io.Discard, nil))
 	srv := httptest.NewServer(
-		mirror.NewAPI(mirror.NewManager(dir, log), "http://mirror:8081").Handler())
+		mirror.NewAPI(privateOK(mirror.NewManager(dir, log)), "http://mirror:8081").Handler())
 	defer srv.Close()

 	c, err := mirror.NewClient(srv.URL, tlsconf.Config{}, 0)
@@ -648,3 +648,10 @@ func TestAModelsFamilyIsHowItsNameStarts(t *testing.T) {
 		}
 	}
 }
+
+// privateOK lets a test's manager fetch from the httptest upstreams on
+// loopback, which a real one refuses.
+func privateOK(m *mirror.Manager) *mirror.Manager {
+	m.AllowPrivateSources()
+	return m
+}
@@ -10,6 +10,11 @@ it, it does not replace it.
 Idempotent: a release whose body already matches is left alone, a version with
 no release is skipped, and nothing is created or deleted.

+A release candidate — v0.10.4-rc.2 — takes the "## Unreleased" section, which
+is where changes gather until the version they belong to is released, and is
+marked as a pre-release. The final v0.10.4 takes its own section once
+"Unreleased" has been renamed to it.
+
 Usage:
    GITEA_TOKEN=<token> scripts/gitea/release_notes.py [--tag v0.8.1] [--dry-run]

@@ -32,7 +37,8 @@ DEFAULT_REPO = "ilker/llmgw"

 # "## v0.8.1 — 2026-09-28". The dash is whatever was typed; the version is what
 # matters.
-HEADING = re.compile(r"^## (v\d+\.\d+\.\d+)\b.*$", re.M)
+HEADING = re.compile(r"^## (v\d+\.\d+\.\d+|Unreleased)\b.*$", re.M)
+CANDIDATE = re.compile(r"^v\d+\.\d+\.\d+-rc\.\d+$")


 def sections(changelog):
@@ -77,11 +83,16 @@ def main():
    with open(args.changelog, encoding="utf-8") as f:
        notes = sections(f.read())
    if args.tag:
-        if args.tag not in notes:
+        section = "Unreleased" if CANDIDATE.match(args.tag) else args.tag
+        if section not in notes:
            # A tag with no section is a release somebody forgot to describe.
            # Failing says so where it is seen; an empty page says nothing.
-            sys.exit(f"{args.changelog} has no section for {args.tag}")
-        notes = {args.tag: notes[args.tag]}
+            sys.exit(f"{args.changelog} has no section for {args.tag} (looked for {section!r})")
+        notes = {args.tag: notes[section]}
+    else:
+        # Syncing everything: "Unreleased" belongs to whichever candidate is
+        # newest, which only a tag says.
+        notes.pop("Unreleased", None)

    base = f"{args.instance.rstrip('/')}/api/v1/repos/{args.repo}/releases"
    for tag, body in notes.items():
@@ -92,11 +103,12 @@ def main():
        if rel is None:
            print(f"skip    {tag}  (no release)")
            continue
-        if (rel.get("body") or "").strip() == body.strip():
+        pre = bool(CANDIDATE.match(tag))
+        if (rel.get("body") or "").strip() == body.strip() and bool(rel.get("prerelease")) == pre:
            print(f"same    {tag}")
            continue
-        request("PATCH", f"{base}/{rel['id']}", token, {"body": body})
-        print(f"set     {tag}")
+        request("PATCH", f"{base}/{rel['id']}", token, {"body": body, "prerelease": pre})
+        print(f"set     {tag}" + ("  (pre-release)" if pre else ""))


 if __name__ == "__main__":
Author	SHA1	Message	Date
Ilker ManapandClaude Opus 5.5	5dac1854c2	Publish small fixes as release candidates of the next version Release / build (push) Successful in 1m1s Details Release / notes (push) Successful in 3s Details CI / test (push) Successful in 3m30s Details Every fix so far became its own patch release, and the natural alternative — v0.10.3-a — sorts before v0.10.3 by SemVer's rules: a hyphen marks a pre-release, so every tool that orders versions would take the fix for the older one. A release candidate of the next version, v0.10.4-rc.1, sorts after v0.10.3 and before v0.10.4, which is what it is. CI already builds any v* tag. Its release notes now come from a "## Unreleased" section in CHANGELOG.md, where changes gather until the version they belong to is released, and the release is marked as a pre-release. Onboarding, in Go and in the shell script, counts a candidate as a release and downloads it rather than copying the operator's binary. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-09-30 01:00:11 +02:00
Ilker ManapandClaude Opus 5.5	ad6d2d348d	Restart replicas without taking them out of service first There was no way to restart a running replica. A running engine picks up an engine key, an edited profile or a new engine version only when it starts, so after v0.10.3 the nine replicas already running stayed without a key — reachable from any node — with nothing to do about it short of draining nodes, which on a small CPU fleet has nowhere to move a model to. `llmgw ctl restart REPLICA... \| --model M \| --all` queues replicas to be replaced. A replica being replaced keeps serving but stops counting toward its model, so the reconciler places a replacement beside it; once that is ready the old one is drained and stopped as a node drain would. Replicas of one model go one at a time, so a model never needs the memory of every replica twice. Where no second copy fits anywhere, the old replica is stopped after the drain grace period and the new one starts in its place, and the log says so, since that is the one case with an outage. `ctl replicas` shows "(restart queued)" and "(restart replacing)" while it happens. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-09-30 00:59:26 +02:00
Ilker ManapandClaude Opus 5.5	611fddd236	Describe v0.10.3 Release / build (push) Successful in 1m4s Details Release / notes (push) Successful in 3s Details CI / test (push) Successful in 3m39s Details Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-09-30 00:38:55 +02:00
Ilker ManapandClaude Opus 5.5	9853936e0b	Give every engine a key only the router holds Engines listen on every address their node has, with no key. Checked on the live cluster: from a neighbouring node, a request straight to an engine's port was answered — past the gateway's authentication, quotas and accounting, by anyone who could reach the node. Each replica's engine is now started with a key and refuses requests without it. The key is derived from the replica id with an HMAC over a secret kept in the store, rather than stored per replica: the router has to send the right key to replicas this process never started — adopted after a restart, or placed by the other control plane — and a function of a shared secret gives that without a table to keep in step. A key taken from one node opens that replica only. The secret is made on first use and copied with the rest of the store when moving to PostgreSQL; a new one would lock the router out of every running engine at the moment of the move. The key reaches llama.cpp and vLLM through their environment (LLAMA_API_KEY, VLLM_API_KEY) rather than their command line, which every user on the node can read; SGLang only takes it as an argument. The agent's readiness check, the router's proxy and its metrics scrape send it, the client's own Authorization still never reaches the engine, and `ctl bench` fetches the key of the replica it is pointed at from the admin API. Engines started before this keep running without a key until their replica is next started. With no store there is nowhere to keep the secret, and engines start without one, with a warning. Also formats two files that went in unformatted. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-09-30 00:38:29 +02:00
Ilker ManapandClaude Opus 5.5	d167686cf1	Stop a browser page, or a model definition, from running code on the nodes The admin listener has no login. Reaching it is meant to mean holding the control plane, so it binds to loopback and an operator reaches it over an ssh tunnel — and so does every page open in that operator's browser. The API decoded JSON whatever the Content-Type said, so a page could POST a text/plain model definition, no preflight needed. A model's profile env reached an engine the agent runs as root, unfiltered: LD_PRELOAD in a definition was code execution on every node. A page that rebound its own name to 127.0.0.1 could also read pprof's heap, which holds API keys. The admin listener now answers only when Host is an address or localhost, refuses requests a browser marks as cross-site (Origin, Sec-Fetch-Site), and takes changes only as application/json, which no page can send across origins without a preflight nobody grants. ctl sends that on every change, bodiless DELETEs included; curl and ctl are otherwise unaffected. Profile env that changes what runs rather than how — the dynamic loader's LD_, PATH, Python's PYTHON start-up variables, BASH_ENV and their like — is refused when a model is defined, and dropped again by the engine adapter before the process starts, whatever sent the profile. None of the live catalog's models sets any. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-09-30 00:29:52 +02:00
Ilker ManapandClaude Opus 5.5	97dc49cbeb	Keep the mirror's management API and its downloads off the network behind it Checked on the live cluster: from any node, the mirror's management API on :8082 answered — list, fetch, delete — with no credentials. Unset --api-allow meant anyone. It now means this machine only; the control plane is let in with --api-allow, which `server init` already wrote for the mirrors it sets up, or by mutual TLS. A fetch went to whatever URL it was given, and the file server on :8081 then handed the result to anyone: "fetch http://169.254.169.254/..." read a cloud's instance metadata through the mirror, and a URL on the LAN read whatever answered there. Downloads started through the API now go to http and https only, and are refused at dial time — after resolution, on every redirect — when the address is loopback, private, link-local, CGNAT or multicast. --allow-private-sources lets them through for a mirror that really does copy from a server on its own network. `llmgw mirror add`, run by someone on the mirror itself, is unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-09-30 00:27:27 +02:00
Ilker ManapandClaude Opus 5.5	3a26277c32	Refuse names that leave their directory, on the mirror and the agent Two places built a path from a name somebody else supplied and checked only for slashes. The mirror refused "a/b" as a model name but took "..", which is a single path element too: a fetch for model ".." wrote beside the mirror root rather than inside it. Model, version and file names now have to be one element that does not start with a dot — the mirror's own checksum files are dotted — on the fetch path and the API alike. The agent took the engine version from the control plane as given and made a directory of it under /opt/llmgw/engines, and a RemoveAll before replacing one. A version of "../x" would have done both outside the install root, as root. The agent now installs only versions that look like one — letters, digits and ._+- — whatever the control plane sent. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-09-30 00:24:48 +02:00
Ilker ManapandClaude Opus 5.5	7062645bcb	Describe v0.10.2 Release / notes (push) Successful in 3s Details CI / test (push) Failing after 4m10s Details Release / build (push) Successful in 3m45s Details Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-09-29 22:37:28 +02:00
Ilker ManapandClaude Opus 5.5	bb90a927aa	Print only what must outlive init's screen, and drop bubbles Once server init's screen closed, the whole run was printed again under the frame that had just shown it. Now the password and the next steps are, the whole run goes to /var/lib/llmgw/init.log (0600: it holds the password), and it is printed in full only when init fails. Seen in a draft of the install video, where it scrolled the screen away. The spinner and the text field are written here instead of taken from bubbles, whose packages query the terminal's colours as they load. bubbletea does the same deliberately, so the query stays; the dependency goes. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-09-29 22:37:28 +02:00