Describe v0.9.1

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

.gitea/workflows/release.yml

@@ -55,6 +55,13 @@ jobs:
             (cd dist && sha256sum "${name}" >> checksums.txt)
           done
 
+          # The reverse proxy configurations, so a release page carries
+          # everything install.md asks for and the repository is not needed.
+          for f in deploy/proxy/Caddyfile deploy/proxy/nginx.conf; do
+            cp "$f" dist/
+            (cd dist && sha256sum "$(basename "$f")" >> checksums.txt)
+          done
+
           # An NVML build for nodes with NVLink, which the portable binary
           # cannot serve.
           #
@@ -150,3 +157,31 @@ jobs:
               || { echo "upload of $(basename "$f") was refused" >&2; exit 1; }
           done
           echo "attached $(ls dist | wc -l) files to $tag"
+
+  # The release page says what changed, copied from CHANGELOG.md. Gitea makes
+  # an empty release for every tag and the job above only attaches files, so
+  # every release until this existed was a list of binaries and nothing else.
+  #
+  # After the build, because that is what guarantees the release exists. Run by
+  # hand from a branch it syncs every version in the changelog, which is how
+  # the releases made before this got their notes.
+  notes:
+    needs: build
+    runs-on: ubuntu-latest
+    permissions:
+      contents: write
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Copy the changelog onto the release
+        env:
+          GITEA_TOKEN: ${{ secrets.GITEA_TOKEN }}
+          GITEA_INSTANCE: ${{ github.server_url }}
+          GITEA_REPO: ${{ github.repository }}
+        run: |
+          set -euo pipefail
+          if [ "${GITHUB_REF_TYPE:-}" = "tag" ]; then
+            python3 scripts/gitea/release_notes.py --tag "${GITHUB_REF_NAME}"
+          else
+            python3 scripts/gitea/release_notes.py
+          fi

CHANGELOG.md

@@ -13,11 +13,219 @@ is only released once its **exit criterion** is met, not when its code lands.
 | v0.6.0 | 4 | Multi-node replicas and fabric awareness | released |
 | v0.7.0 | — | Conversations, agent memory, disk management | released |
 | v0.8.0 | — | PostgreSQL, leader election, agent failover | released |
+| v0.9.0 | — | Managing people and the tenancy tree | released |
 | v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
 
 Until v1.0.0 the minor version is the breaking one: configuration formats and Go
 APIs change between minors without a deprecation period.
 
+## v0.9.1 — 2026-09-28
+
+### The mirror page is one line per model
+
+What the mirror holds used to take two table rows per file — seven columns and
+the address under them — so a sharded model filled the screen. Now each version
+is one line: model and version, the file (or "4 files" for a set), total size,
+and which catalog models use it. The address moved into a "copy address" button;
+the checksum is in the file's tooltip. A version already in use is not offered
+for serving again, and removing one warns that the catalog uses it. The header
+counts versions rather than files.
+
+## v0.9.0 — 2026-09-28
+
+### Managing people and the tenancy tree
+
+Organizations, teams and projects can now be renamed, moved and removed, and an
+approved person can be moved to another project or removed — none of which
+could be done at all before, from the web or the CLI.
+
+- **`/admin/users`** shows the whole tree, each project opening onto its people
+  and keys, with approve, move, suspend, reinstate and remove for people,
+  revoke for keys, and add, rename, move and remove at every level. A search
+  over every account sits below it. Changes update the page in place (htmx,
+  embedded in the binary) and it still works with scripts off.
+- **`llmgw org`** — `tree [--people]`, `add`, `rename`, `move`, `remove`.
+- **`llmgw users move`** and **`llmgw users delete`**.
+- **`llmgw keys list --all`** lists every key with its project and owner.
+- Every command that takes a project, team or organization accepts an id, a
+  path such as `Acme/platform/gateway`, or a unique name, and `--project` may
+  be omitted when there is only one. Users are named by id or email.
+
+Moving a person takes the keys they made with them; keys an operator issued stay
+with the project. Only empty things are removed, and a project that ever had a
+key is never removed, because usage history refers to it.
+
+### Changes
+
+- **`--project` no longer defaults to `default`** in `keys create`, `keys list`,
+  `users approve`, `users create`, `keys threads` and `keys memory`: a tree
+  built from the web has no project of that name. With one project it is
+  chosen; with several the command lists them.
+- **`keys limits --budget` is in whole units**, as the quotas page takes it,
+  rather than micro-units. `--budget 50000000` now means fifty million units;
+  divide existing scripts' figures by 1 000 000.
+
+### Fixes
+
+- `keys init` could silently move an existing team or project to another
+  organization or team by redeclaring it. It now refuses.
+- An account rejected while pending and then reinstated became active with no
+  project, and failed at its first key. Reinstating one is refused; approving
+  gives it a project. Rejecting a signup now removes it.
+
+## v0.8.8 — 2026-09-28
+
+### Fixes
+
+- **`server init --postgres` failed on Rocky Linux** with "postgresql-setup:
+  executable file not found". The PostgreSQL project's own packages (PGDG)
+  provide `postgresql-server` too, so dnf can answer that name with
+  `postgresql16-server`, which names its unit `postgresql-16.service` and keeps
+  its tools in `/usr/pgsql-16/bin`. init now finds the server unit systemd has,
+  reads the data directory from it, and runs `initdb` itself when the directory
+  is empty. A PostgreSQL that was already installed is used as it is.
+
+### Known gaps
+
+Not yet run on a real machine. Three arrangements are not handled: a
+PostgreSQL initialised with `ident` for TCP logins (the connection check fails
+and says so), one listening on a port other than 5432, and several PGDG
+versions where the highest is not the one on 5432.
+
+## v0.8.7 — 2026-09-28
+
+### PostgreSQL in one command
+
+```
+sudo llmgw server init --postgres
+```
+
+Installs PostgreSQL with the machine's own package manager when there is none,
+creates the `llmgw` role and database with a generated password kept in
+`/etc/llmgw/secrets/store-dsn`, and checks that it connects. On a control plane
+already running on SQLite it stops the gateway, copies the store across so
+every account and key survives, points `auth.store` at PostgreSQL and starts
+the gateway again; if the move fails, the gateway is started again on SQLite.
+Given on the first run, there is never a SQLite store.
+
+`server.yaml` is edited in place with its comments intact, and the previous
+file is kept beside it. A PostgreSQL that init did not install is left sized as
+it was, since something else may depend on it.
+
+The PostgreSQL installation itself has not yet run on a real machine; the move,
+the configuration edit and the rerun behaviour are tested.
+
+### Fixes
+
+- **A new control plane with no models did not start.** A configuration with
+  no static endpoints and no models was refused as having nothing to serve, so
+  under systemd the unit restarted every two seconds and nothing listened on
+  8080. It now runs as a control plane with an empty catalog.
+
+## v0.8.6 — 2026-09-28
+
+### Reverse proxy configurations
+
+`deploy/proxy/` has a Caddyfile and an nginx configuration for the client
+port, and both are attached to every release beside the binaries. They do not
+buffer streamed replies, their read timeout outlasts a model's wait before its
+first token, their body limit matches the router's, and they set
+`X-Forwarded-Proto` and `X-Forwarded-Host` themselves — llmgw marks session
+cookies `Secure` from the first and shows the second as the base URL on its
+landing page.
+
+## v0.8.5 — 2026-09-28
+
+### A control plane in one command
+
+```
+sudo llmgw server init --admin you@example.com
+```
+
+Creates the llmgw user and its directories, the certificate authority, the
+control plane's certificate for the machine's hostname and every address it
+has, `server.yaml`, the default project, and an administrator whose password is
+generated and printed once — then installs the systemd unit, starts it and
+waits until it answers. What used to be two steps of install.md is now this.
+
+Each step is skipped when its result exists, so running it again finishes an
+install that stopped and changes nothing on one that did not. `--host` adds an
+address the machine cannot see of itself (a DNS alias, a NAT or floating
+address) and reissues the certificate keeping the old ones. It never overwrites
+`server.yaml` or an authority, and certificates issued by an authority kept on
+another machine are left as they are.
+
+### Fixes
+
+- **The configuration install.md showed could not start.** An empty catalog
+  without `control.enabled: true` is refused as having nothing to serve. The
+  documentation and the file `init` writes both set it.
+
+## v0.8.4 — 2026-09-28
+
+### Node logs on the admin page
+
+Each node on the cluster page links to its logs — the agent's own, and the
+output of every engine running there — with an option to follow them. `llmgw
+ctl logs NODE` reads the same. They come over the agent's existing stream, so
+no ssh, jump host or root is needed to see what a node did. The agent keeps its
+last 2000 lines in memory; each engine its last few hundred.
+
+## v0.8.3 — 2026-09-28
+
+Found by building a node from nothing with v0.8.2.
+
+### Fixes
+
+- **A new node's first engine install failed on apt's lock.** cloud-init and
+  unattended-upgrades run apt in a machine's first minutes, and
+  `DPkg::Lock::Timeout` does not cover the lock `apt-get update` takes. A
+  package command held up by a lock now waits for up to ten minutes, and says
+  so; a failed install is retried after a minute rather than ten.
+- **A new node reported no disk** until its first download created the weights
+  directory. It is created when the agent starts.
+- **Onboarding took one control plane address.** It takes several, comma
+  separated, so a new node fails over like the rest.
+
+## v0.8.2 — 2026-09-28
+
+A node no longer runs whatever engine somebody once put on it. It installs its
+own.
+
+### Engines are installed by the node
+
+`engines:` in server.yaml names the engines every node has and their versions,
+defaulting to the llama.cpp build this release was checked against (b10991).
+Each agent converges on it: it picks the build for its hardware — CUDA with its
+runtime when an NVIDIA driver is loaded, CPU otherwise — installs the OS
+packages that build needs with the machine's own package manager, downloads the
+archive and checks it against the sha256 GitHub publishes, unpacks it refusing
+anything that would land outside its directory, and runs it. `ldd` names a
+missing library or a system too old for the build ("libstdc++.so.6 is too old
+(needs GLIBCXX_3.4.30)"); `--version` proves the rest. Installs are checked
+again every ten minutes.
+
+An engine is reported only once installed and seen to run, or when the agent's
+configuration names it on purpose. A node that has installed nothing yet is not
+a candidate for any model, where before it was a candidate for all of them.
+`ctl nodes`, the cluster page and placement's refusals say where each
+installation stands, and warn about a CPU without AVX2.
+
+Not every system is refused for its name: the build runs on Ubuntu 22.04+,
+Debian 12+ and RHEL-family 10, and `ldd` decides. vLLM and SGLang are not yet
+installed this way and must still be named in the agent's configuration.
+
+### Onboarding
+
+Onboarding installs llmgw from the CI release for the node's architecture,
+checked against the release's checksums, and writes an agent configuration of
+only the control plane's addresses and the TLS files.
+
+### Fixes
+
+- A race on the weights fingerprint, which two heartbeat loops touched during a
+  failover.
+
 ## v0.8.1 — 2026-09-28
 
 Nothing here changes a configuration format: every file that worked with

README.md

@@ -32,6 +32,7 @@ Releases track the roadmap phases; see [CHANGELOG.md](CHANGELOG.md).
 | v0.6.0 | 4 | Multi-node placement and fabric awareness | released |
 | v0.7.0 | — | Conversations, agent memory, disk management | released |
 | v0.8.0 | — | PostgreSQL, leader election, agent failover | released |
+| v0.9.0 | — | Managing people and the tenancy tree | released |
 | v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
 
 As of v0.6.0 a cluster runs unattended: agents report their hardware, the
@@ -66,6 +67,8 @@ on a machine that has them.
   does not exist or miss one that does
 - [deploy/mirror/](deploy/mirror/) — a model mirror, which is the cheapest
   and most useful thing to build around llmgw
+- [deploy/proxy/](deploy/proxy/) — Caddy and nginx in front of the client
+  port, with streaming and the forwarded headers right
 - [deploy/ha/](deploy/ha/) — running two control planes, and the one health
   check that has to be right
 - [docs/node-onboarding.md](docs/node-onboarding.md),

api/proto/llmgw/v1/agent.proto

@@ -23,6 +23,7 @@ message AgentMessage {
     Register register = 1;
     Heartbeat heartbeat = 2;
     ShardStatus shard = 3;
+    LogsReply logs = 4;
   }
 }
 
@@ -34,6 +35,8 @@ message ControlMessage {
     DrainShard drain = 3;
     Ack ack = 4;
     RemoveWeights remove_weights = 5;
+    Provision provision = 6;
+    LogsRequest logs = 7;
   }
 }
 
@@ -101,6 +104,34 @@ message Register {
   // Empty means the node was not asked to look, exactly as for weights, and
   // constrains nothing.
   repeated Storage storage = 12;
+
+  // host is what the machine is, so the control plane can have the mirror
+  // fetch software built for it rather than assuming every node matches the
+  // first.
+  Host host = 13;
+
+  // engines_probed says engine_versions is an answer. Without it an empty map
+  // meant "an agent too old to say" and constrained nothing — so a freshly
+  // built node with no engine installed yet was a candidate for every model.
+  bool engines_probed = 14;
+}
+
+// Host describes the operating system and processor of a node.
+message Host {
+  // From /etc/os-release: "ubuntu", "24.04", "noble".
+  string os_id = 1;
+  string os_version = 2;
+  string os_codename = 3;
+  // Debian's spelling, which is what package indexes are keyed by: "amd64",
+  // "arm64".
+  string arch = 4;
+  // The vector extensions inference cares about, of those the CPU has. A VM
+  // given QEMU's default CPU model has none of them, and runs, and is five
+  // times slower, and nothing says why.
+  repeated string cpu_flags = 5;
+  // nvidia_driver is the driver version, empty when there is none. A card
+  // without a driver is not a GPU anything can use.
+  string nvidia_driver = 6;
 }
 
 // Storage is one filesystem a node keeps weights on.
@@ -234,6 +265,43 @@ message Heartbeat {
   // somebody restarts the agent — which is a step nobody knows to take.
   bool engines_changed = 8;
   map<string, string> engine_versions = 9;
+
+  // provision is where installing the node's software stands, sent when it
+  // changed. The flag makes an empty list mean "nothing to install" rather
+  // than "nothing to say", as for weights.
+  bool provision_changed = 10;
+  repeated ProvisionStatus provision = 11;
+}
+
+// Provision is the software a node should have: which engines, at which
+// versions. Nothing more.
+//
+// Everything else is the node's to work out, because only the node knows it:
+// which build fits its hardware, where that build is published, what its
+// operating system is missing, and whether the result actually runs. The agent
+// converges on this — installs what is absent, checks what is there — and
+// reports each engine through ProvisionStatus. Nothing is taken on trust from
+// what happens to be on the machine: an engine is offered to placement only
+// once the agent has installed it and seen it run.
+message Provision {
+  repeated EngineWant engines = 1;
+}
+
+// EngineWant names one engine at one version.
+message EngineWant {
+  string engine = 1;  // "llamacpp"
+  string version = 2; // "b10991"
+}
+
+// ProvisionStatus is where one engine's installation stands on a node.
+message ProvisionStatus {
+  string engine = 1;
+  string version = 2;
+  string variant = 3;
+  // "installing", "ready", "failed", or "unsupported" when no build fits.
+  string state = 4;
+  // detail says what is happening or what went wrong, in words for a person.
+  string detail = 5;
 }
 
 // ShardStatus reports a shard's lifecycle transition.
@@ -352,3 +420,29 @@ message Ack {
   bool ok = 2;
   string error = 3;
 }
+
+// LogsRequest asks a node for the recent lines of one of its logs.
+//
+// Over the stream the agent already holds, because that is the one way into a
+// node every deployment has: the admin page otherwise means ssh through a jump
+// host to a machine whose address nobody remembers, to read a journal only
+// root may open.
+message LogsRequest {
+  // request_id pairs the reply with its request; several may be in flight.
+  string request_id = 1;
+  // source is "agent" for the agent's own log, or a shard id for that
+  // engine's output.
+  string source = 2;
+  int32 lines = 3;
+}
+
+// LogsReply answers a LogsRequest.
+message LogsReply {
+  string request_id = 1;
+  repeated string lines = 2;
+  // sources are every log this node can show, so a page can offer them
+  // without knowing what runs there.
+  repeated string sources = 3;
+  // error says why the lines are absent: an unknown source, say.
+  string error = 4;
+}

deploy/bootstrap/add-node.sh

@@ -108,6 +108,5 @@ echo "== 2/2  onboarding $NAME"
 echo
 echo "Check it with: llmgw ctl nodes"
 echo
-echo "A node is not finished until the engines its profiles name are installed"
-echo "and the weights are on disk. Onboarding checks that a configured engine"
-echo "runs, but it installs neither — see docs/node-onboarding.md."
+echo "The agent installs its engines itself; the node takes no model until they"
+echo "are installed and seen to run. 'llmgw ctl nodes' shows where that stands."

deploy/bootstrap/onboard-node.sh

@@ -24,7 +24,8 @@ usage: $0 --node ID --addr HOST --server URL [options]
 
   --node ID         node id, hostname and certificate common name
   --addr HOST       how to reach the machine over ssh
-  --server URL      the control plane's agent endpoint, e.g. https://10.0.0.140:7443
+  --server URL      the control plane's agent endpoint, e.g. https://10.0.0.140:7443;
+                    several, comma separated, for a node that fails over
   --fabric NAME     InfiniBand fabric this node is attached to; repeatable as
                     --fabric NAME:hca1,hca2 . Omit for a node with no fabric,
                     which can host single-node replicas and will never be given
@@ -33,8 +34,8 @@ usage: $0 --node ID --addr HOST --server URL [options]
                     address on the route to --server, found by the agent)
   --mgmt-if NAME    interface NCCL bootstraps over (default: the one holding
                     that address)
-  --llamacpp-bin P  path to llama-server (default: PATH, then /opt/llama.cpp)
-  --vllm-bin P      path to the vllm entry point
+  --llamacpp-bin P  use this llama-server instead of the one the agent installs
+  --vllm-bin P      path to the vllm entry point (not yet installed by the agent)
   --sglang-python P python interpreter of the SGLang virtualenv
   --nvidia-smi PATH where nvidia-smi lives, when it is not on PATH. Containerized
                     driver installs routinely put it somewhere else.
@@ -44,6 +45,8 @@ usage: $0 --node ID --addr HOST --server URL [options]
                     Releases carry static linux/amd64 and linux/arm64 builds;
                     downloading one is better than copying whatever happens to
                     be on the machine you are running this from.
+  --releases URL    where release binaries are downloaded from (default the
+                    project's Gitea releases)
   --ssh-user NAME   (default ilker)
 USAGE
   exit 2
@@ -54,6 +57,7 @@ NVIDIA_SMI= IB_ROOT=/sys/class/infiniband
 LLAMACPP_BIN= VLLM_BIN= SGLANG_PYTHON=
 SSH_WAIT=180
 PKI=/etc/llmgw/pki BINARY= SSH_USER=ilker
+RELEASES=https://gitea.mynodes.xyz/ilker/llmgw/releases/download
 FABRIC_NAME= FABRIC_HCAS="mlx5_0,mlx5_1"
 
 while [ $# -gt 0 ]; do
@@ -71,6 +75,7 @@ while [ $# -gt 0 ]; do
     --ib-root) IB_ROOT="$2"; shift 2;;
     --pki) PKI="$2"; shift 2;;
     --binary) BINARY="$2"; shift 2;;
+    --releases) RELEASES="$2"; shift 2;;
     --ssh-user) SSH_USER="$2"; shift 2;;
     *) usage;;
   esac
@@ -131,9 +136,28 @@ $SSH "set -e
   rm -f /tmp/ca.pem /tmp/$NODE.pem /tmp/$NODE-key.pem"
 
 # 4. The binary, the configuration and the unit.
-say "installing the binary"
-scp -q $SSHOPTS "$BINARY" "$SSH_USER@$ADDR:/tmp/llmgw"
-$SSH 'sudo install -m 0755 /tmp/llmgw /usr/local/bin/llmgw && rm -f /tmp/llmgw'
+# The node fetches the release CI built for this version, for its own
+# architecture, and checks it against the release's checksums — rather than
+# being handed whatever binary is on the machine running this script. A build
+# that is not a release (a developer's, with commits past the tag) has no such
+# file, and is copied as before.
+TAG="$("$BINARY" --version 2>/dev/null | awk '{print $NF}')"
+if printf '%s' "$TAG" | grep -Eq '^v[0-9]+\.[0-9]+\.[0-9]+$'; then
+  say "installing llmgw $TAG from the release"
+  $SSH "set -e
+    arch=\$(dpkg --print-architecture)
+    name=llmgw-$TAG-linux-\$arch
+    cd /tmp && rm -f \$name checksums.txt
+    curl -fsSL -o \$name '$RELEASES/$TAG/'\$name
+    curl -fsSL -o checksums.txt '$RELEASES/$TAG/checksums.txt'
+    grep \" \$name\\\$\" checksums.txt | sha256sum -c - >/dev/null
+    sudo install -m 0755 \$name /usr/local/bin/llmgw
+    rm -f \$name checksums.txt"
+else
+  say "installing the binary (${TAG:-unversioned}, not a release)"
+  scp -q $SSHOPTS "$BINARY" "$SSH_USER@$ADDR:/tmp/llmgw"
+  $SSH 'sudo install -m 0755 /tmp/llmgw /usr/local/bin/llmgw && rm -f /tmp/llmgw'
+fi
 
 say "writing the configuration"
 NODE_IS_HOSTNAME=no
@@ -146,7 +170,14 @@ NODE_IS_HOSTNAME=no
 # below is written only when it was given, as an override.
 {
   echo "# $NODE, written by onboard-node.sh."
-  echo "server: \"$SERVER\""
+  # Several addresses, comma separated, are every control plane the node may
+  # fail over to; a standby refuses it and it moves on to the next.
+  if [ "${SERVER#*,}" != "$SERVER" ]; then
+    echo "servers:"
+    printf '%s\n' "$SERVER" | tr ',' '\n' | sed 's/^ *//; s/^/  - "/; s/$/"/'
+  else
+    echo "server: \"$SERVER\""
+  fi
   echo
   echo "tls:"
   echo "  ca: /etc/llmgw/ca.pem"
@@ -242,8 +273,9 @@ $SSH 'set -e
 
 echo
 if [ -z "$LLAMACPP_BIN$VLLM_BIN$SGLANG_PYTHON" ]; then
-  say "no engine paths given; this node can only run the 'sim' engine until"
-  say "one is installed and the configuration points at it"
+  say "the agent now installs the engines the control plane asks for: the"
+  say "build for this machine, checked against its publisher's checksum and run"
+  say "before the node is offered any model. 'llmgw ctl nodes' shows progress."
 fi
 if [ -n "$ENGINE_PROBLEMS" ]; then
   echo "$NODE is onboarded but an engine will not run:$ENGINE_PROBLEMS"

deploy/examples/agent.yaml

@@ -84,6 +84,15 @@ discovery:
 #   - /var/lib/llmgw-models
 #   - /mnt/lustre/models
 
+# The agent installs the engines the control plane names (see `engines:` in
+# server.yaml) under /opt/llmgw/engines: the build for this machine — CUDA when
+# an NVIDIA driver is loaded — checked against its publisher's checksum, with
+# the OS packages it needs, and run before the node takes any model. llama.cpp
+# is installed this way today. Naming a binary below opts that engine out and
+# uses yours instead; vLLM and SGLang still have to be named until the agent
+# installs them too.
+# engines_dir: /opt/llmgw/engines
+
 engines:
   vllm:
     bin: /opt/venvs/vllm/bin/vllm

deploy/examples/server.yaml

@@ -197,6 +197,13 @@ router:
     session_load_factor: 1.25
 
 # How aggressively the control plane reconciles.
+# The engines every node installs, and at which version. Unset keeps the
+# versions this release was checked against (llama.cpp b10991). Each agent
+# picks the build for its own hardware and installs it itself; changing a
+# version here reaches every node on its next connect.
+# engines:
+#   llamacpp: b10991
+
 control:
   interval: 5s
   base_backoff: 10s

deploy/proxy/Caddyfile

@@ -0,0 +1,41 @@
+# llmgw behind Caddy: the client-facing port (8080) on a public name, with a
+# certificate Caddy obtains and renews by itself.
+#
+# Replace llm.example.com, point its DNS at this machine, open 80 and 443, and
+# run: caddy run --config /etc/caddy/Caddyfile
+#
+# Only 8080 goes through here. 7443 (agents) is mutual TLS with the cluster's
+# own authority and must reach llmgw directly; 9090 and 9091 stay private.
+
+llm.example.com {
+	# The router reads at most 32 MiB of body (router.max_body_bytes). Refusing
+	# more here gives the client a 413 before the upload, not after it.
+	request_body {
+		max_size 32MB
+	}
+
+	reverse_proxy 127.0.0.1:8080 {
+		# Tokens are written the moment they are generated. Without this a
+		# streamed reply can arrive in bursts, and a client measuring time to
+		# first token measures the proxy instead.
+		flush_interval -1
+
+		# Caddy sets X-Forwarded-Proto and X-Forwarded-Host itself and, with
+		# no trusted_proxies, discards whatever the client sent. llmgw relies
+		# on that: the proto decides whether session cookies are Secure, and
+		# the host is the base URL the landing page tells people to use.
+		# Behind a load balancer of your own, name it in trusted_proxies.
+
+		transport http {
+			# A long generation can be silent for a while before the first
+			# token. llmgw has its own limits (timeouts.*); this must not be
+			# the shorter one.
+			response_header_timeout 15m
+		}
+	}
+
+	# No `encode` on purpose: compressing a stream buffers it.
+
+	# Access log to stderr, which under systemd is the journal.
+	log
+}

deploy/proxy/README.md

@@ -0,0 +1,36 @@
+# A reverse proxy in front of llmgw
+
+The client-facing port (8080) is plain HTTP. On anything but a trusted network,
+put it behind a proxy that holds a publicly trusted certificate.
+
+| File | For |
+| --- | --- |
+| [Caddyfile](Caddyfile) | Caddy — obtains and renews its own certificate; the least work |
+| [nginx.conf](nginx.conf) | nginx, with a certificate from certbot |
+
+Both are also attached to every release.
+
+Four things both files get right, and any other proxy has to as well:
+
+- **Streaming is not buffered.** Replies are server-sent events; a buffering
+  proxy delivers them in one burst at the end.
+- **The read timeout is longer than a model's silence.** nginx's default of 60
+  seconds turns a slow first token into a 504.
+- **`X-Forwarded-Proto` and `X-Forwarded-Host` are set by the proxy**, replacing
+  whatever the client sent. llmgw uses the first to mark session cookies
+  `Secure` and the second for the base URL on its landing page.
+- **The body limit matches llmgw's** (32 MiB, `router.max_body_bytes`), so an
+  oversized request is refused before it is uploaded.
+
+Only port 8080 is proxied. Agents reach 7443 directly — it is mutual TLS with
+the cluster's own authority, and a proxy terminating it would take away the
+node identity it carries. 9090 (admin, pprof) stays on loopback; 9091
+(metrics) goes to your monitoring host only.
+
+With a proxy on the same machine, bind llmgw to loopback so nothing reaches it
+around the proxy:
+
+```yaml
+# /etc/llmgw/server.yaml
+listen: "127.0.0.1:8080"
+```

deploy/proxy/nginx.conf

@@ -0,0 +1,63 @@
+# llmgw behind nginx: the client-facing port (8080) on a public name.
+#
+# Put this in /etc/nginx/sites-available/llmgw (or conf.d/llmgw.conf), replace
+# llm.example.com, and get a certificate — certbot --nginx -d llm.example.com
+# fills in the ssl_ lines below. Then: nginx -t && systemctl reload nginx
+#
+# Only 8080 goes through here. 7443 (agents) is mutual TLS with the cluster's
+# own authority and must reach llmgw directly; 9090 and 9091 stay private.
+
+upstream llmgw {
+    server 127.0.0.1:8080;
+    keepalive 32;
+}
+
+server {
+    listen 80;
+    listen [::]:80;
+    server_name llm.example.com;
+    return 301 https://$host$request_uri;
+}
+
+server {
+    # `listen ... http2` rather than `http2 on;`: the latter needs nginx
+    # 1.25.1, and Ubuntu 22.04 and Debian 12 ship older ones. Newer nginx
+    # accepts this form with a deprecation warning.
+    listen 443 ssl http2;
+    listen [::]:443 ssl http2;
+    server_name llm.example.com;
+
+    ssl_certificate     /etc/letsencrypt/live/llm.example.com/fullchain.pem;
+    ssl_certificate_key /etc/letsencrypt/live/llm.example.com/privkey.pem;
+
+    # The router reads at most 32 MiB of body (router.max_body_bytes).
+    client_max_body_size 32m;
+
+    location / {
+        proxy_pass http://llmgw;
+        proxy_http_version 1.1;
+        proxy_set_header Connection "";
+
+        # Set, never appended: llmgw decides from X-Forwarded-Proto whether
+        # session cookies are Secure, and builds the base URL the landing page
+        # shows from X-Forwarded-Host. A value passed through from the client
+        # would be the client's choice.
+        proxy_set_header Host              $host;
+        proxy_set_header X-Forwarded-Host  $host;
+        proxy_set_header X-Forwarded-Proto $scheme;
+        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
+
+        # Streamed replies: tokens go out as they are generated. Buffering
+        # turns a stream into one late burst, and gzip buffers too.
+        proxy_buffering off;
+        proxy_cache off;
+        gzip off;
+
+        # nginx's default is 60 seconds between two reads from upstream. A
+        # large model can think longer than that before its first token, and
+        # the client then sees a 504 for a request that was about to succeed.
+        # llmgw has its own limits (timeouts.*); this must not be the shorter.
+        proxy_read_timeout 15m;
+        proxy_send_timeout 15m;
+    }
+}

docs/choosing-models.md

@@ -141,7 +141,7 @@ not count the same free terabyte once per node.
 Set budgets and limits on day one, not later:
 
 ```
-llmgw keys limits --scope project --id <project> --rpm 60 --budget 50000000
+llmgw keys limits --scope project --id <project> --rpm 60 --budget 50
 ```
 
 The reason is not technical. A resource that appears free and unlimited becomes,

docs/install.md

@@ -29,7 +29,7 @@ The reason is boring and unavoidable: drivers and engines are platform decisions
 with kernel modules and version matrices attached.
 
 Weights are the one item on that list llmgw does handle. It includes a mirror —
-step 4 — which holds the files on your own network and hands them to the nodes,
+step 3 — which holds the files on your own network and hands them to the nodes,
 so getting weights onto every machine is not a problem you solve before
 starting. What llmgw will not do is choose the models for you.
 
@@ -58,7 +58,8 @@ them, so nodes need no inbound access except your own SSH.
 | 8082 | the control plane administering the mirror | the control plane only — this port can delete models and start downloads |
 
 Decide now which address nodes will use to reach the control plane. It goes into
-a certificate and changing it later means reissuing one.
+a certificate: the machine's own addresses are found for you, and any other name
+— a DNS alias, a NAT or floating address — has to be given with `--host`.
 
 ## 1. The binary
 
@@ -83,141 +84,110 @@ git clone https://gitea.mynodes.xyz/ilker/llmgw && cd llmgw
 CGO_ENABLED=0 go build -o llmgw ./cmd/llmgw
 ```
 
-## 2. The certificate authority
+## 2. The control plane
 
-Agents authenticate to the control plane with mutual TLS, and a node's identity
-is taken from its certificate's common name. A public authority cannot supply
-these: Let's Encrypt and its peers stopped issuing certificates carrying the
-`clientAuth` usage, so nothing they hand out can authenticate an agent.
-
-Run this wherever you intend to keep the authority. It talks to nothing and
-works on files, so it does not have to be a cluster machine — and there is an
-argument that it should not be.
+On the machine that will be the control plane, as root:
 
 ```
-sudo llmgw ctl cert init-ca --dir /etc/llmgw/pki
+sudo llmgw server init --admin you@example.com
 ```
 
-That writes `ca.pem` and `ca-key.pem`. **Whoever holds `ca-key.pem` can mint any
-node's identity**, so it stays here; `ca.pem` is copied to every machine and is
-not a secret.
+That is the whole step. In order, it:
 
-Now the control plane's own certificate. Every address an agent might dial must
-appear in `--host`, bare IP addresses included — hostname verification reads
-subject alternative names and ignores the common name entirely, so a certificate
-naming the host only in its subject is rejected by every client:
+1. creates the `llmgw` system user, `/etc/llmgw` and `/var/lib/llmgw`
+2. creates the cluster's certificate authority in `/etc/llmgw/pki`
+3. issues the control plane's certificate for this machine's hostname and
+   every address it has, and copies `ca.pem` beside it
+4. writes `/etc/llmgw/server.yaml`: mutual TLS, authentication, default limits
+   and a price, and a control plane with an empty catalog
+5. creates the store, the `default` project, and your account, already approved
+   and promoted to administrator — the password is generated and **printed
+   once**
+6. installs the systemd unit, starts it, and waits until `/health` answers
+
+Add `--postgres` to keep accounts and keys in PostgreSQL rather than SQLite:
+init installs it and sets it up, and on a machine already running on SQLite it
+moves everything across. See [PostgreSQL](#postgresql-when-sqlite-is-no-longer-the-right-shape).
+
+Every step looks before it acts. Running it again finishes an install that
+stopped halfway and changes nothing on one that did not; it never overwrites
+`server.yaml`, the authority, or an existing account's password.
+
+**The addresses in the certificate matter.** Agents check that the address
+they dial is named in it, and the certificate names only what this machine
+could see of itself. If nodes will reach it by another name — a DNS alias, a
+NAT address, a floating IP — give it:
 
 ```
-sudo llmgw ctl cert issue-server llmgw-ctl \
-  --host 10.0.0.140 --host llmgw-ctl.internal \
-  --dir /etc/llmgw/pki --out /etc/llmgw
+sudo llmgw server init --host llmgw-ctl.internal --host 203.0.113.10
 ```
 
+Run that at any time: the certificate is reissued with the new names added to
+the old ones, and the service restarted.
+
+**`ca-key.pem` signs every node's identity.** `init` leaves it in
+`/etc/llmgw/pki`, because onboarding a node needs it and the control plane is
+where most people run onboarding from. Back it up; there is no recovering it.
+If this machine is exposed, move the directory elsewhere once the nodes are in
+and pass `--pki` to onboarding.
+
 This covers the fleet-facing port only. The client-facing API is public and
-belongs behind an ordinary publicly trusted certificate, usually terminated by a
-reverse proxy in front of port 8080.
-
-## 3. The control plane
-
-### The user and directories
-
-```
-sudo useradd --system --home /var/lib/llmgw --shell /usr/sbin/nologin llmgw
-sudo install -d -o llmgw -g llmgw -m 0750 /var/lib/llmgw
-sudo install -d -m 0755 /etc/llmgw
-sudo chown root:llmgw /etc/llmgw/ca.pem /etc/llmgw/server.pem /etc/llmgw/server-key.pem
-sudo chmod 0640 /etc/llmgw/server-key.pem
-```
-
-### The configuration
-
-`/etc/llmgw/server.yaml`. This is the smallest file that is not a toy: mutual
-TLS on, authentication on, one model, and limits that apply to anyone nobody has
-given limits to.
-
-```yaml
-listen: ":8080"
-admin_listen: "127.0.0.1:9090"
-agent_listen: ":7443"
-
-agent_tls:
-  ca: /etc/llmgw/ca.pem
-  cert: /etc/llmgw/server.pem
-  key: /etc/llmgw/server-key.pem
-
-auth:
-  enabled: true
-  store: /var/lib/llmgw/llmgw.db
-  signup_open: true
-  # Applied to any key no scope gives a limit to. Without these, a project
-  # created without limits is metered by nothing at all, and the first sign of
-  # that is the bill or the outage.
-  default_rpm: 60
-  default_tpm: 200000
-  # What a request costs, which is what makes a budget mean anything. Without
-  # a price every request costs zero, so a budget is displayed, counted against
-  # nothing, and never binds — and nobody finds out until they expected it to
-  # stop something. Self-hosted models have no per-token price, so the unit is
-  # yours to choose: GPU-seconds, a notional rate, or 1000 everywhere, which
-  # turns a budget into a plain token quota.
-  default_price:
-    prompt_per_1k: 1000
-    completion_per_1k: 1000
-  # pricing:
-  #   qwen2.5-coder-7b-instruct: { prompt_per_1k: 500, completion_per_1k: 1500 }
-
-landing:
-  contact: "platform@example.com"
-
-# Start with none. Models are added while the cluster runs — see step 8 —
-# and anything listed here is declarative: it is in the catalog on every start
-# and cannot be removed without editing this file.
-models: []
-```
-
-`admin_listen` is on loopback on purpose. It serves pprof, which can dump the
-process heap, and API keys sit in that heap in plaintext. To see the dashboard
-from elsewhere, promote an account and use `/admin` on the client port — do not
-move this listener.
-
-The store file is created on first start; there is no separate migration step.
-
-### The service
-
-```
-sudo cp deploy/systemd/llmgw-server.service /etc/systemd/system/
-sudo systemctl daemon-reload
-sudo systemctl enable --now llmgw-server
-systemctl status llmgw-server
-```
-
-### Your account
-
-Accounts are created and approved from a shell on this machine, deliberately:
-having a shell here is what makes someone an administrator, and there are no
-roles to get subtly wrong.
-
-```
-sudo -u llmgw llmgw users create you@example.com --password 'change me now'
-# prints a user id, usr-xxxxxxxx
-sudo -u llmgw llmgw users promote usr-xxxxxxxx
-```
-
-`promote` is what grants `/admin`. Sign in at `http://<control-plane>:8080/login`,
-change the password, and issue yourself a key from the profile page.
+belongs behind an ordinary publicly trusted certificate, terminated by a
+reverse proxy in front of port 8080. [deploy/proxy/](../deploy/proxy/) has a
+Caddyfile and an nginx configuration that get streaming, timeouts and the
+forwarded headers right; both are attached to every release.
 
 ### Check it
 
 ```
-curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/health   # 200, with an empty body
-sudo -u llmgw llmgw ctl status
+curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/health   # 200
+llmgw ctl status
 ```
 
 `ctl status` will report no nodes and no models, which is correct — nothing has
-been added yet. It is worth running now so its output is familiar before you
-need it.
+been added yet. Sign in at `http://<control-plane>:8080/login` with the printed
+password, change it, and issue yourself a key from the profile page.
 
-## 4. The mirror
+### The configuration
+
+`/etc/llmgw/server.yaml` is yours from here on. Three things in it are worth
+knowing before you edit it:
+
+- `admin_listen` is on loopback on purpose. It serves pprof, which can dump the
+  process heap, and API keys sit in that heap in plaintext. To see the dashboard
+  from elsewhere use `/admin` on the client port — do not move this listener.
+- `default_rpm`, `default_tpm` and `default_price` are what make limits and
+  budgets mean anything. Without a price every request costs zero and a budget
+  never binds; without the defaults, a project created without limits is
+  metered by nothing at all.
+- `control.enabled: true` is what makes this a control plane while its catalog
+  is empty. Without it, a file with no models is refused as having nothing to
+  serve.
+
+Every key is in [reference.md](reference.md).
+
+### Keeping the authority off the control plane
+
+Create it on another machine, issue the control plane's certificate there, and
+bring only the results:
+
+```
+# on the authority's machine
+llmgw ctl cert init-ca --dir /etc/llmgw/pki
+llmgw ctl cert issue-server llmgw-ctl --host 10.0.0.140 --host llmgw-ctl.internal \
+  --dir /etc/llmgw/pki --out ./ctl
+
+# copy ca.pem, ctl/server.pem and ctl/server-key.pem into /etc/llmgw on the
+# control plane, then there:
+sudo llmgw server init --admin you@example.com
+```
+
+`init` finds certificates in `/etc/llmgw` and no authority beside them, and
+keeps them as they are. Adding an address later is then done where the
+authority is; `init --host` says so rather than minting a second authority no
+node trusts.
+
+## 3. The mirror
 
 Do this before the nodes, not after. Every node fetches its own copy of a
 model's weights, and where they fetch from decides whether bringing up a cluster
@@ -313,7 +283,7 @@ llmgw mirror fetch --model qwen2.5-14b-instruct --version 2026-03 \
 or from the admin page, or by copying files into the directory by hand — the
 mirror is a directory, and anything that puts files in it works.
 
-## 5. Preparing a node
+## 4. Preparing a node
 
 **Do this before onboarding.** A node missing its driver or engine registers
 healthy and fails on first placement, and the reason is three logs away.
@@ -418,7 +388,7 @@ llmgw ctl engine-check vllm --bin /opt/venvs/vllm/bin/vllm
 
 ### Weights
 
-Nothing to do here if you built the mirror in step 4 — the agent fetches what a
+Nothing to do here if you built the mirror in step 3 — the agent fetches what a
 model's `source` names, and that source is your mirror. This section is for the
 arrangements where the files are already on the machine.
 
@@ -441,7 +411,7 @@ model's `source` at a file to download and the agent fetches it once, before the
 engine is launched:
 
 ```yaml
-# your mirror, which is what step 4 was for
+# your mirror, which is what step 3 was for
 source: http://models.internal:8081/qwen2.5-coder-3b-instruct/2026-03/q4_k_m.gguf
 # a whole version, for a model of several files — the trailing slash is the signal
 source: http://models.internal:8081/llama-3.1-70b-instruct/2026-03/
@@ -498,7 +468,7 @@ How much disk a node needs follows from the models rather than a rule of thumb.
 `llmgw ctl model plan <repo>` prints the sizes; as a starting point, a 70B at
 bf16 is 141 GB and the same model at q4 is 43 GB.
 
-## 6. Adding the node
+## 5. Adding the node
 
 Two halves, because they need different access: building the machine needs the
 hypervisor, and issuing its identity needs the certificate authority. Where one
@@ -555,7 +525,7 @@ Repeat for each node. Nodes in the same fabric must be given the same `--fabric`
 name; nodes with no InfiniBand take no `--fabric` at all and are each their own
 island, which is fine for single-node models.
 
-## 7. Setting the node's CPU model, if it is a VM
+## 6. Setting the node's CPU model, if it is a VM
 
 Proxmox defaults a VM to `kvm64`, which hides AVX, AVX2, FMA and even SSE4.2
 from the guest. Inference then falls back to scalar code and is not slightly
@@ -572,7 +542,7 @@ only change being `--cpu host`:
 probably did not, and nothing about it looks broken — every number is simply
 small. Check with `grep -qw avx2 /proc/cpuinfo` on the node.
 
-## 8. Adding a model
+## 7. Adding a model
 
 Write the definition. This is the same shape a model takes in `server.yaml`, so
 an entry moves between the two by being copied:
@@ -714,7 +684,7 @@ An engine that publishes no metrics endpoint is measured from the outside only.
 The run says so, and queue depth and KV occupancy are then not part of the
 answer.
 
-## 9. The first request
+## 8. The first request
 
 ```
 curl http://<control-plane>:8080/v1/chat/completions \
@@ -777,6 +747,20 @@ turn, which is the first per-request load this store has had; and SQLite takes
 one writer at a time, so those turns serialise. And a second control plane is
 impossible while the store is a file on one machine's disk.
 
+On the control plane, one command does all of it — installs PostgreSQL if the
+machine has none, creates its role and database with a generated password kept
+in `/etc/llmgw/secrets/store-dsn`, stops the gateway, copies the SQLite store
+into it so every account and key survives, points `server.yaml` at it, and
+starts the gateway again:
+
+```
+sudo llmgw server init --postgres
+```
+
+It can be given on the first run too, and then there is never a SQLite store.
+The rest of this section is what that command does, for a PostgreSQL somewhere
+else.
+
 The setting is the same one, and the address decides:
 
 ```yaml
@@ -957,7 +941,7 @@ the choice is yours rather than silent.
 
 Three things, in descending order of how badly you want them.
 
-**The certificate authority.** `ca-key.pem`, wherever you put it in step 2.
+**The certificate authority.** `ca-key.pem`, `/etc/llmgw/pki` unless you kept it elsewhere (step 2).
 Losing it does not stop the cluster, and it means every node identity has to be
 reissued the next time you touch one. Nothing in llmgw can reconstruct it.
 
@@ -1028,7 +1012,7 @@ output, which is the actual diagnosis.
 node and the reason: wrong GPU class, not enough free cards, too little memory,
 no fabric, a label it does not carry.
 
-**Everything is slow.** On a VM, check AVX2 (step 7). On CPU, check the
+**Everything is slow.** On a VM, check AVX2 (step 6). On CPU, check the
 quantisation — the difference between `q8_0` and `q4_k_m` is roughly a factor of
 two. `llmgw ctl status` shows how full the cluster is.
 
@@ -1065,7 +1049,7 @@ requests. On a slow model, set `timeouts.generation` higher.
   those eight local, which is most of the problem; on a fabric where the nodes
   can see each other, seeding between them is the obvious next thing to build.
 - **A budget without a price never binds.** Set `default_price` or `pricing`
-  (step 3) or every request costs zero, and a budget is something the dashboard
+  (step 2) or every request costs zero, and a budget is something the dashboard
   displays rather than something that stops anything. The quotas page says so
   where it can, which is not the same as being told before you rely on it.
 - **Engines are yours to install.** Onboarding verifies one runs; it installs

docs/onboarding.md

@@ -88,11 +88,36 @@ means anything.
 If you have no structure yet, one of each is fine:
 
 ```
-llmgw keys init --store … --org mynodes --team default --project default
+llmgw org add org mynodes
+llmgw org add team default --in mynodes
+llmgw org add project default --in mynodes/default
+llmgw org tree --people
 ```
 
-It is idempotent: running it again renames rather than duplicating, so it can
-live in whatever provisions the machine.
+Everything that takes an organization, team or project — here, in `users` and
+in `keys` — accepts its id, its path (`mynodes/default/default`) or its name
+when the name is unique, so an id never has to be copied off a page. When there
+is only one project, `--project` can be left out.
+
+The tree is not fixed once made:
+
+```
+llmgw org rename project Acme/platform/gateway api-gateway
+llmgw org move project Acme/platform/gateway --to Scania/ml
+llmgw org remove team Acme/old                 # only when nothing is in it
+llmgw users move ayse@example.com --project Acme/ml/research
+llmgw users delete ayse@example.com --yes      # signs out, revokes their keys
+llmgw keys list --all                          # every key, its project and owner
+```
+
+Moving a person takes the keys they made with them, so they stop spending the
+old project's budget; keys an operator issued in the project stay with it. A
+project that ever had a key cannot be removed, because usage history refers to
+it — rename it instead.
+
+`llmgw keys init` still works for provisioning scripts and is idempotent, but it
+will no longer move a team or project that already exists under another parent;
+it says so and stops.
 
 Split it when you want separate limits or separate reporting, not before.
 
@@ -156,16 +181,18 @@ Allowlists compose by intersection: a key cannot add a model its team withheld.
 A model's canonical name and any of its aliases name the same thing, so either
 form works.
 
-Budgets are counted in micro-units, and what a unit means is yours to decide.
-Self-hosted models have no per-token price, so it can be GPU-seconds, a notional
-rate, or 1 000 000 everywhere — which turns a budget into a plain token quota.
+Budgets are given in whole units, as the quotas page shows them, and what a unit
+means is yours to decide. Self-hosted models have no per-token price, so it can
+be GPU-seconds, a notional rate, or one unit per thousand tokens everywhere —
+a price of 1 000 000 in `auth.pricing`, which is in micro-units — and a budget
+becomes a plain quota of thousands of tokens.
 Models are priced in the server's `auth.pricing` section; a model priced at zero
 is exempt from budget checks, which is what lets a key with nothing left still be
 served a cheap fallback.
 
 ```
-llmgw keys limits --store … --scope project --id default \
-  --budget 50000000 --period month        # 50 units a month
+llmgw keys limits --store … --scope project --id Acme/platform/gateway \
+  --budget 50 --period month              # 50 units a month
 ```
 
 ## The landing page
@@ -286,9 +313,12 @@ page exists.
 
 ## Managing accounts from the browser
 
-`/admin/users` lists who is waiting and who is active. Approving (into a project)
-and suspending are there, because that is the work an operator does constantly
-and it should not need a terminal.
+`/admin/users` is the tree: organizations, teams and projects, each project
+opening onto its people and its keys. From there an admin approves, moves,
+suspends and removes people, revokes keys, and adds, renames, moves and removes
+organizations, teams and projects. A searchable list of every account is at the
+bottom. The page works as plain forms; with scripts on, a change updates it in
+place and the projects you had open stay open.
 
 Three things are deliberately not there:
 

docs/reference.md

@@ -54,6 +54,7 @@ rather than repeated.
 | `adopt_ready_timeout` | `time.Duration` | [Config](#config) |
 | `tls` | `tlsconf.Config` | [Config](#config) |
 | `model_dirs` | `[]string` | [Config](#config) |
+| `engines_dir` | `string` | [Config](#config) |
 | `weights_cache` | `string` | [Config](#config) |
 | `discovery` | `discovery.Options` | [Config](#config) |
 | `engines` | `engines.Config` | [Config](#config) |
@@ -160,6 +161,7 @@ rather than repeated.
 | `agent_tls` | `tlsconf.Config` | [Server](#server) |
 | `monitoring` | `Monitoring` | [Server](#server) |
 | `mirror` | `Mirror` | [Server](#server) |
+| `engines` | `map[string]string` | [Server](#server) |
 | `router` | `Router` | [Server](#server) |
 | `control` | `Control` | [Server](#server) |
 | `models` | `[]catalog.Model` | [Server](#server) |
@@ -432,6 +434,13 @@ exported over the fabric, local NVMe and a copy made by hand all look
 identical from here, and placement only needs to know whether the file
 is present.
 
+### `engines_dir`
+
+`string`
+
+EnginesDir is where the agent installs engines, one directory per
+build. Defaults to /opt/llmgw/engines.
+
 ### `weights_cache`
 
 `string`
@@ -589,8 +598,8 @@ know that this page cannot guess.
 
 Bin is the llama-server binary. It is the server, not the CLI: the
 gateway talks OpenAI over HTTP and needs the one that serves it.
-Empty looks on PATH, then where release archives are unpacked
-(/opt/llama.cpp).
+Empty means the agent installs llama.cpp itself, at the version the
+control plane asks for. Setting it opts out of that.
 
 ### `parallel`
 
@@ -1193,6 +1202,18 @@ Empty means there is none, and /admin/mirror stays unmounted.
 
 Takes a [Mirror](#mirror) block.
 
+### `engines`
+
+`map[string]string`
+
+Engines are the engines every node installs, and at which version. The
+agents install them themselves — the right build for their hardware,
+checked against its publisher's checksum and seen to run — so a node is
+never trusted to already have what somebody once put there.
+
+Unset keeps DefaultEngines. A version replaces the default; an empty
+one leaves that engine to be installed by hand.
+
 ### `router`
 
 `Router`

internal/agent/agent.go

@@ -28,6 +28,7 @@ import (
 	"gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1/llmgwv1connect"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/supervisor"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/weights"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/agentsvc"
@@ -94,6 +95,10 @@ type Config struct {
 	// is present.
 	ModelDirs []string `koanf:"model_dirs"`
 
+	// EnginesDir is where the agent installs engines, one directory per
+	// build. Defaults to /opt/llmgw/engines.
+	EnginesDir string `koanf:"engines_dir"`
+
 	// WeightsCache is where a remote source is downloaded to. Defaults to the
 	// first entry of ModelDirs, because a node that says where its models live
 	// has already answered this question, and otherwise to
@@ -304,16 +309,30 @@ type Agent struct {
 	reg  *engines.Registry
 	http connectHTTPClient
 
-	// engineOnce probes the installed engines once before registering.
+	// engineOnce probes the configured engines once before registering.
 	//
 	// Once, because probing SGLang imports it in a Python interpreter — seconds
 	// on a cold cache — and a reconnect is not a reason to pay that again.
-	// Later changes are found by the slower re-probe the heartbeat starts, not
-	// by registering again: an engine installed while the agent runs used to
-	// stay invisible until somebody thought to restart the agent.
+	// Later changes are found by the slower re-probe the heartbeat starts.
 	engineOnce sync.Once
-	// engineVers is guarded by mu once the first probe has run.
-	engineVers map[string]string
+	// configuredVers are the engines the configuration names and that answer.
+	// Guarded by mu once the first probe has run.
+	configuredVers map[string]string
+	// installedVers are the engines this agent installed and saw run.
+	// Guarded by mu.
+	installedVers map[string]string
+
+	// prov installs the engines the control plane asks for. provWants is what
+	// it asked for, nil until it has; provStatus is where each stands and
+	// provSent the fingerprint of what the control plane last heard. All
+	// guarded by mu. provKick starts a pass without waiting for the timer.
+	logRing *logRing
+
+	prov       *provision.Installer
+	provWants  []provision.Want
+	provStatus map[string]provision.Status
+	provSent   string
+	provKick   chan struct{}
 	// enginesSent is the fingerprint of the engines the control plane last
 	// heard about, from registration or a heartbeat. Guarded by mu.
 	enginesSent string
@@ -329,8 +348,9 @@ type Agent struct {
 	fetcher      *weights.Fetcher
 
 	// lastWeights is the fingerprint of the file list most recently sent, so a
-	// heartbeat carries the list only when it changed. Touched only by the
-	// heartbeat goroutine, which is the only thing that reads or writes it.
+	// heartbeat carries the list only when it changed. Guarded by mu: during a
+	// failover the old stream's heartbeat loop can still be running when the
+	// new one's starts.
 	lastWeights string
 
 	// fetchWatchers are the shards waiting on each source, so progress on one
@@ -354,6 +374,11 @@ func New(cfg Config, httpClient connectHTTPClient, log *slog.Logger) (*Agent, er
 	if log == nil {
 		log = slog.Default()
 	}
+	// Everything the agent logs also goes to a ring the admin page can read,
+	// so seeing what a node did does not mean ssh and a journal only root may
+	// open.
+	ring := newLogRing(agentLogLines)
+	log = slog.New(teeHandler{log.Handler(), slog.NewTextHandler(ring, nil)})
 	if httpClient == nil {
 		httpClient = http.DefaultClient
 	}
@@ -363,12 +388,17 @@ func New(cfg Config, httpClient connectHTTPClient, log *slog.Logger) (*Agent, er
 	if err != nil {
 		return nil, err
 	}
-	return &Agent{
+	a := &Agent{
 		cfg: cfg, log: log, sup: sup,
 		reg:          engines.NewRegistry(cfg.Engines),
 		http:         httpClient,
 		shardReplica: map[string]string{},
-	}, nil
+		provStatus:   map[string]provision.Status{},
+		logRing:      ring,
+		provKick:     make(chan struct{}, 1),
+	}
+	a.prov = &provision.Installer{Root: cfg.EnginesDir, Log: log, OnStatus: a.provisionStatus}
+	return a, nil
 }
 
 // Supervisor exposes the process supervisor, for tests and `ctl logs`.
@@ -387,6 +417,12 @@ func (a *Agent) Run(ctx context.Context) error {
 		a.log.Error("hardware discovery failed", "err", err)
 	}
 	a.recoverOrphans()
+	// Made now rather than by the first download, so a new node reports its
+	// disk from its first heartbeat instead of only once it holds a model.
+	if err := os.MkdirAll(a.cfg.WeightsCache, 0o755); err != nil {
+		a.log.Warn("cannot create the weights directory", "dir", a.cfg.WeightsCache, "err", err)
+	}
+	go a.provisionLoop(ctx)
 
 	attempt := 0
 	next := 0
@@ -645,9 +681,13 @@ func (a *Agent) scan(ctx context.Context) error {
 	opts := a.cfg.Discovery
 	opts.ModelDirs = a.cfg.ModelDirs
 	inv, err := discovery.Discover(ctx, opts, a.log)
+	host := discovery.DiscoverHost()
 	a.mu.Lock()
 	a.inv = inv
+	a.prov.Host = host
 	a.mu.Unlock()
+	a.log.Info("this node", "os", host.OSID+" "+host.OSVersion, "arch", host.Arch,
+		"cpu_flags", host.CPUFlags, "nvidia_driver", host.NVIDIADriver)
 	return err
 }
 
@@ -685,7 +725,18 @@ func (a *Agent) session(ctx context.Context, server string) error {
 	// engine whether or not its binary exists, so without this a node with no
 	// vLLM accepted a vLLM shard and failed it minutes later.
 	reg.EngineVersions = a.engines(parent)
+	reg.EnginesProbed = true
 	reg.RunningShards = a.runningShards()
+	a.mu.Lock()
+	h := a.prov.Host
+	// A new stream is a control plane that knows nothing of where installs
+	// stand, so the next heartbeat tells it again.
+	a.provSent = ""
+	a.mu.Unlock()
+	reg.Host = &llmgwv1.Host{
+		OsId: h.OSID, OsVersion: h.OSVersion, OsCodename: h.OSCodename,
+		Arch: h.Arch, CpuFlags: h.CPUFlags, NvidiaDriver: h.NVIDIADriver,
+	}
 
 	if err := stream.Send(&llmgwv1.AgentMessage{
 		Msg: &llmgwv1.AgentMessage_Register{Register: reg},
@@ -780,10 +831,22 @@ func (a *Agent) heartbeat() *llmgwv1.AgentMessage {
 
 	a.reprobeEngines()
 	a.mu.Lock()
-	if fp := enginesFingerprint(a.engineVers); a.engineVers != nil && fp != a.enginesSent {
-		a.enginesSent = fp
+	if vers := a.engineReportLocked(); a.configuredVers != nil &&
+		enginesFingerprint(vers) != a.enginesSent {
+		a.enginesSent = enginesFingerprint(vers)
 		hb.EnginesChanged = true
-		hb.EngineVersions = maps.Clone(a.engineVers)
+		hb.EngineVersions = vers
+	}
+	if fp := provisionFingerprint(a.provStatus); fp != a.provSent {
+		a.provSent = fp
+		hb.ProvisionChanged = true
+		for _, e := range slices.Sorted(maps.Keys(a.provStatus)) {
+			st := a.provStatus[e]
+			hb.Provision = append(hb.Provision, &llmgwv1.ProvisionStatus{
+				Engine: st.Engine, Version: st.Version, Variant: st.Variant,
+				State: st.State, Detail: st.Detail,
+			})
+		}
 	}
 	a.mu.Unlock()
 
@@ -807,8 +870,12 @@ func (a *Agent) heartbeat() *llmgwv1.AgentMessage {
 	// rather than "nothing to report" — without it, the first heartbeat after
 	// a model was deleted would be indistinguishable from silence.
 	weights := discovery.ScanWeights(a.cfg.ModelDirs)
-	if fp := weightsFingerprint(weights); fp != a.lastWeights {
-		a.lastWeights = fp
+	fp := weightsFingerprint(weights)
+	a.mu.Lock()
+	changed := fp != a.lastWeights
+	a.lastWeights = fp
+	a.mu.Unlock()
+	if changed {
 		hb.WeightsChanged = true
 		for _, w := range weights {
 			hb.Weights = append(hb.Weights, &llmgwv1.LocalWeights{
@@ -899,6 +966,18 @@ func (a *Agent) handle(ctx context.Context, msg *llmgwv1.ControlMessage, out cha
 		}
 	case *llmgwv1.ControlMessage_RemoveWeights:
 		a.removeWeights(m.RemoveWeights.GetPaths())
+	case *llmgwv1.ControlMessage_Provision:
+		a.setProvision(m.Provision)
+	case *llmgwv1.ControlMessage_Logs:
+		reply := a.logsReply(m.Logs)
+		// Off the receive loop: a full send queue must not stop commands being
+		// read.
+		go func() {
+			select {
+			case out <- &llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_Logs{Logs: reply}}:
+			case <-ctx.Done():
+			}
+		}()
 	case *llmgwv1.ControlMessage_Ack:
 		if !m.Ack.GetOk() {
 			a.log.Warn("control plane rejected a message",
@@ -1152,27 +1231,142 @@ func Client(cfg tlsconf.Config, timeout time.Duration) (*http.Client, error) {
 	return &http.Client{Transport: &http2.Transport{TLSClientConfig: tc}}, nil
 }
 
-// engines is what this node can run, probed once.
-//
-// Failures are silence rather than an error: an engine that is not installed is
-// the ordinary case on most nodes, and an agent that refused to register over
-// one would take a working node out of the fleet for a missing engine it was
-// never going to be asked to run.
+// engines is what this node can run: what the agent installed and saw run,
+// and what the configuration names and answers. Nothing merely found on the
+// machine — a llama-server on PATH is somebody's, not necessarily one that
+// works, and placing on it is how a replica fails on a node nobody watches.
 func (a *Agent) engines(ctx context.Context) map[string]string {
 	a.engineOnce.Do(func() {
-		vers := a.reg.Probe(ctx)
+		vers := a.reg.ProbeConfigured(ctx)
 		a.mu.Lock()
-		a.engineVers, a.lastEngineProbe = vers, time.Now()
+		a.configuredVers, a.lastEngineProbe = vers, time.Now()
 		a.mu.Unlock()
-		a.log.Info("engines available on this node",
-			"engines", engines.ProbedNames(vers))
+		if len(vers) > 0 {
+			a.log.Info("engines named in the configuration",
+				"engines", engines.ProbedNames(vers))
+		}
 	})
 	a.mu.Lock()
 	defer a.mu.Unlock()
+	vers := a.engineReportLocked()
 	// Whatever registration sends is what the control plane now knows, so a
 	// heartbeat repeats it only once it differs.
-	a.enginesSent = enginesFingerprint(a.engineVers)
-	return maps.Clone(a.engineVers)
+	a.enginesSent = enginesFingerprint(vers)
+	return vers
+}
+
+// engineReportLocked merges the two sources. Installed wins: it is the one
+// that was seen to run. Called with mu held.
+func (a *Agent) engineReportLocked() map[string]string {
+	out := maps.Clone(a.configuredVers)
+	if out == nil {
+		out = map[string]string{}
+	}
+	maps.Copy(out, a.installedVers)
+	return out
+}
+
+// provisionEvery is how often installed engines are checked again. A library
+// removed by an unrelated upgrade is found here rather than by the next
+// replica to start.
+const provisionEvery = 10 * time.Minute
+
+// setProvision takes the control plane's list of wanted engines.
+func (a *Agent) setProvision(p *llmgwv1.Provision) {
+	wants := []provision.Want{}
+	for _, e := range p.GetEngines() {
+		wants = append(wants, provision.Want{Engine: e.GetEngine(), Version: e.GetVersion()})
+	}
+	a.mu.Lock()
+	a.provWants = wants
+	a.mu.Unlock()
+	select {
+	case a.provKick <- struct{}{}:
+	default:
+	}
+}
+
+// provisionLoop installs and re-checks the wanted engines, one pass at a time.
+func (a *Agent) provisionLoop(ctx context.Context) {
+	// After a failure the next pass comes sooner, and backs off: a lock held
+	// by a new machine's own updates clears in a minute or two, while ten
+	// minutes' wait left a node idle for no reason.
+	next := provisionEvery
+	retry := time.Minute
+	for {
+		select {
+		case <-ctx.Done():
+			return
+		case <-a.provKick:
+		case <-time.After(next):
+		}
+		a.mu.Lock()
+		wants := slices.Clone(a.provWants)
+		managed := a.provWants != nil
+		a.mu.Unlock()
+		if !managed {
+			continue
+		}
+
+		statuses := a.prov.Converge(ctx, wants)
+		next = provisionEvery
+		for _, st := range statuses {
+			if st.State == provision.Failed {
+				next, retry = retry, min(retry*2, provisionEvery)
+				break
+			}
+		}
+		if next == provisionEvery {
+			retry = time.Minute
+		}
+		installed := map[string]string{}
+		keep := map[string]bool{}
+		for _, st := range statuses {
+			keep[st.Engine] = true
+			if st.State == provision.Ready {
+				a.reg.UseInstalled(st.Engine, st.Binary)
+				installed[st.Engine] = st.Version
+			}
+		}
+		a.mu.Lock()
+		if enginesFingerprint(installed) != enginesFingerprint(a.installedVers) {
+			a.log.Info("engines installed and running", "engines", engines.ProbedNames(installed))
+		}
+		a.installedVers = installed
+		// An engine no longer asked for is no longer reported on.
+		for e := range a.provStatus {
+			if !keep[e] {
+				delete(a.provStatus, e)
+			}
+		}
+		a.mu.Unlock()
+	}
+}
+
+// provisionStatus records a change in one engine's installation.
+func (a *Agent) provisionStatus(st provision.Status) {
+	a.mu.Lock()
+	prev, had := a.provStatus[st.Engine]
+	a.provStatus[st.Engine] = st
+	a.mu.Unlock()
+	if !had || prev.State != st.State || prev.Detail != st.Detail {
+		lvl := slog.LevelInfo
+		if st.State == provision.Failed || st.State == provision.Unsupported {
+			lvl = slog.LevelWarn
+		}
+		a.log.Log(context.Background(), lvl, "engine installation",
+			"engine", st.Engine, "version", st.Version, "variant", st.Variant,
+			"state", st.State, "detail", st.Detail)
+	}
+}
+
+func provisionFingerprint(m map[string]provision.Status) string {
+	var b strings.Builder
+	for _, k := range slices.Sorted(maps.Keys(m)) {
+		st := m[k]
+		fmt.Fprintf(&b, "%s|%s|%s|%s|%s\n", st.Engine, st.Version, st.Variant, st.State, st.Detail)
+	}
+	return b.String()
 }
 
 // engineReprobe is how often the installed engines are looked for again.
@@ -1198,14 +1392,14 @@ func (a *Agent) reprobeEngines() {
 	go func() {
 		ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
 		defer cancel()
-		vers := a.reg.Probe(ctx)
+		vers := a.reg.ProbeConfigured(ctx)
 
 		a.mu.Lock()
-		changed := enginesFingerprint(vers) != enginesFingerprint(a.engineVers)
-		a.engineVers, a.lastEngineProbe, a.probingEngines = vers, time.Now(), false
+		changed := enginesFingerprint(vers) != enginesFingerprint(a.configuredVers)
+		a.configuredVers, a.lastEngineProbe, a.probingEngines = vers, time.Now(), false
 		a.mu.Unlock()
 		if changed {
-			a.log.Info("engines available on this node changed",
+			a.log.Info("engines named in the configuration changed",
 				"engines", engines.ProbedNames(vers))
 		}
 	}()
@@ -1218,3 +1412,30 @@ func enginesFingerprint(vers map[string]string) string {
 	}
 	return b.String()
 }
+
+// logsReply answers a request for one of this node's logs.
+func (a *Agent) logsReply(req *llmgwv1.LogsRequest) *llmgwv1.LogsReply {
+	n := int(req.GetLines())
+	if n <= 0 {
+		n = 200
+	}
+	n = min(n, agentLogLines)
+	shards := a.sup.IDs()
+	slices.Sort(shards)
+	out := &llmgwv1.LogsReply{
+		RequestId: req.GetRequestId(),
+		Sources:   append([]string{"agent"}, shards...),
+	}
+	switch src := req.GetSource(); src {
+	case "", "agent":
+		out.Lines = a.logRing.tail(n)
+	default:
+		sh, err := a.sup.Get(src)
+		if err != nil {
+			out.Error = "no engine " + src + " runs on this node"
+			return out
+		}
+		out.Lines = sh.Logs(n)
+	}
+	return out
+}

internal/agent/discovery/host.go

@@ -0,0 +1,154 @@
+package discovery
+
+import (
+	"bufio"
+	"os"
+	"runtime"
+	"strings"
+)
+
+// Host is what a node is, as far as installing software on it is concerned.
+type Host struct {
+	// OSID, OSVersion and OSCodename come from /etc/os-release: "ubuntu",
+	// "24.04", "noble".
+	OSID       string `json:"os_id,omitempty"`
+	OSVersion  string `json:"os_version,omitempty"`
+	OSCodename string `json:"os_codename,omitempty"`
+	// OSLike is ID_LIKE: the families this system says it follows, "debian"
+	// for Ubuntu, "rhel centos fedora" for Rocky. It is what decides the
+	// package manager, which the name alone does not for every derivative.
+	OSLike []string `json:"os_like,omitempty"`
+	// Arch is Debian's spelling, "amd64" or "arm64", because that is what
+	// package names and release archives are keyed by.
+	Arch string `json:"arch,omitempty"`
+	// CPUFlags are the vector extensions inference cares about, of those this
+	// processor has.
+	CPUFlags []string `json:"cpu_flags,omitempty"`
+	// NVIDIADriver is the loaded driver's version, empty when there is none.
+	NVIDIADriver string `json:"nvidia_driver,omitempty"`
+}
+
+// Family is the packaging family: "deb", "rpm", or "" when neither is known.
+func (h Host) Family() string {
+	for _, id := range append([]string{h.OSID}, h.OSLike...) {
+		switch id {
+		case "debian", "ubuntu":
+			return "deb"
+		case "rhel", "fedora", "centos", "rocky", "almalinux", "suse", "opensuse":
+			return "rpm"
+		}
+	}
+	return ""
+}
+
+// HasCPUFlag reports whether the processor has an extension.
+func (h Host) HasCPUFlag(f string) bool {
+	for _, have := range h.CPUFlags {
+		if have == f {
+			return true
+		}
+	}
+	return false
+}
+
+// interestingFlags are the extensions that decide which build of an engine
+// runs well. A VM on QEMU's default CPU model has none of them — and still
+// runs, five times slower, with nothing saying why.
+var interestingFlags = []string{"avx", "avx2", "fma", "f16c", "avx512f", "avx512_bf16", "amx_tile", "asimd", "sve"}
+
+// DiscoverHost reads what the machine is.
+func DiscoverHost() Host {
+	h := hostFromOSRelease("/etc/os-release")
+	h.Arch = debianArch(runtime.GOARCH)
+	h.CPUFlags = cpuFlagsFrom("/proc/cpuinfo")
+	h.NVIDIADriver = nvidiaDriverFrom("/proc/driver/nvidia/version")
+	return h
+}
+
+func hostFromOSRelease(path string) Host {
+	var h Host
+	f, err := os.Open(path) //nolint:gosec // a fixed system path
+	if err != nil {
+		return h
+	}
+	defer f.Close()
+	sc := bufio.NewScanner(f)
+	for sc.Scan() {
+		k, v, ok := strings.Cut(sc.Text(), "=")
+		if !ok {
+			continue
+		}
+		v = strings.Trim(v, `"'`)
+		switch k {
+		case "ID":
+			h.OSID = v
+		case "VERSION_ID":
+			h.OSVersion = v
+		case "VERSION_CODENAME":
+			h.OSCodename = v
+		case "ID_LIKE":
+			h.OSLike = strings.Fields(v)
+		}
+	}
+	return h
+}
+
+func debianArch(goarch string) string {
+	switch goarch {
+	case "amd64", "arm64":
+		return goarch
+	case "386":
+		return "i386"
+	case "arm":
+		return "armhf"
+	}
+	return goarch
+}
+
+func cpuFlagsFrom(path string) []string {
+	b, err := os.ReadFile(path) //nolint:gosec // a fixed system path
+	if err != nil {
+		return nil
+	}
+	have := map[string]bool{}
+	for _, line := range strings.Split(string(b), "\n") {
+		k, v, ok := strings.Cut(line, ":")
+		if !ok {
+			continue
+		}
+		// "flags" on x86, "Features" on ARM. The first processor is enough:
+		// a machine does not mix instruction sets across its cores.
+		if k = strings.TrimSpace(k); k != "flags" && k != "Features" {
+			continue
+		}
+		for _, f := range strings.Fields(v) {
+			have[f] = true
+		}
+		break
+	}
+	var out []string
+	for _, f := range interestingFlags {
+		if have[f] {
+			out = append(out, f)
+		}
+	}
+	return out
+}
+
+// nvidiaDriverFrom reads the loaded kernel module's version, which exists
+// exactly when a driver is loaded — unlike nvidia-smi, which can be installed
+// on a machine whose module failed to load.
+func nvidiaDriverFrom(path string) string {
+	b, err := os.ReadFile(path) //nolint:gosec // a fixed system path
+	if err != nil {
+		return ""
+	}
+	// "NVRM version: NVIDIA UNIX x86_64 Kernel Module  570.86.15  Thu ..."
+	fields := strings.Fields(string(b))
+	for i, f := range fields {
+		if f == "Module" && i+1 < len(fields) {
+			return fields[i+1]
+		}
+	}
+	return ""
+}

internal/agent/engines/engine.go

@@ -151,6 +151,16 @@ func NewRegistry(cfg Config) *Registry {
 	return r
 }
 
+// UseInstalled points an engine at a build the agent installed, and reports
+// whether the engine takes one.
+func (r *Registry) UseInstalled(name, path string) bool {
+	u, ok := r.engines[name].(interface{ UseInstalled(string) })
+	if ok {
+		u.UseInstalled(path)
+	}
+	return ok
+}
+
 // Add registers an engine, replacing any engine of the same name.
 func (r *Registry) Add(e Engine) { r.engines[e.Name()] = e }
 

internal/agent/engines/llamacpp.go

@@ -3,19 +3,19 @@ package engines
 import (
 	"fmt"
 	"os"
-	"os/exec"
 	"path/filepath"
 	"sort"
 	"strconv"
 	"strings"
+	"sync/atomic"
 )
 
 // LlamaCppConfig locates the llama.cpp installation.
 type LlamaCppConfig struct {
 	// Bin is the llama-server binary. It is the server, not the CLI: the
 	// gateway talks OpenAI over HTTP and needs the one that serves it.
-	// Empty looks on PATH, then where release archives are unpacked
-	// (/opt/llama.cpp).
+	// Empty means the agent installs llama.cpp itself, at the version the
+	// control plane asks for. Setting it opts out of that.
 	Bin string `koanf:"bin"`
 	// Parallel is how many requests a server handles at once, when a profile
 	// does not say. llama.cpp defaults to one, which would make every second
@@ -29,7 +29,10 @@ type LlamaCppConfig struct {
 // small model on CPU at a speed that is fine for testing the whole path, and it
 // serves the same OpenAI API as the others, so nothing downstream can tell the
 // difference.
-type LlamaCpp struct{ cfg LlamaCppConfig }
+type LlamaCpp struct {
+	cfg       LlamaCppConfig
+	installed atomic.Pointer[string]
+}
 
 // Name identifies the engine in the catalog.
 func (*LlamaCpp) Name() string { return "llamacpp" }
@@ -170,33 +173,25 @@ func ggufIn(path string) (string, error) {
 	return filepath.Join(path, names[0]), nil
 }
 
-// llamaServerPlaces are where llama.cpp's release archives are unpacked when
-// they are not on PATH. The archive carries its shared libraries beside the
-// binary, so it is usually left in one directory rather than installed.
-var llamaServerPlaces = []string{
-	"/opt/llama.cpp/llama-server",
-	"/opt/llama.cpp/bin/llama-server",
-	"/opt/llama.cpp/build/bin/llama-server",
-	"/usr/local/llama.cpp/llama-server",
-}
+// UseInstalled points the engine at a build the agent installed and checked.
+// It takes precedence over everything else, because it is the only one that
+// was seen to run.
+func (l *LlamaCpp) UseInstalled(path string) { l.installed.Store(&path) }
 
-// bin is the llama-server to run: the configured one, else the one on PATH,
-// else the first of the usual unpack locations that exists.
-//
-// Found rather than required in the configuration, because every node here
-// had the same path written into its file by hand, and a node set up without
-// it reported no engine at all.
+// Configured reports whether the agent's configuration names a binary. A
+// named binary is the operator's deliberate choice; one merely found on PATH
+// is not, and is not reported as installed.
+func (l *LlamaCpp) Configured() bool { return l.cfg.Bin != "" }
+
+// bin is the llama-server to run: the installed build, else the configured
+// one, else whatever PATH has — which is only ever reached by a node not
+// provisioned by the agent and not told where its engine is.
 func (l *LlamaCpp) bin() string {
+	if p := l.installed.Load(); p != nil {
+		return *p
+	}
 	if l.cfg.Bin != "" {
 		return l.cfg.Bin
 	}
-	if p, err := exec.LookPath("llama-server"); err == nil {
-		return p
-	}
-	for _, p := range llamaServerPlaces {
-		if info, err := os.Stat(p); err == nil && !info.IsDir() && info.Mode()&0o111 != 0 {
-			return p
-		}
-	}
 	return "llama-server"
 }

internal/agent/engines/probe.go

@@ -157,11 +157,49 @@ func firstVersion(s string) string {
 	return Unknown
 }
 
+// Configured is an engine whose location the agent's configuration names.
+type Configured interface {
+	Configured() bool
+}
+
+// ProbeConfigured probes only the engines the configuration names.
+//
+// Probe looks everywhere — PATH, the default interpreter — and reports what
+// it finds, which is exactly the trust in whatever happens to be on a machine
+// that installing engines replaces. A node the agent provisions reports what
+// it installed, plus what the operator named on purpose; nothing else.
+func (r *Registry) ProbeConfigured(ctx context.Context) map[string]string {
+	out := map[string]string{}
+	for name, e := range r.engines {
+		c, ok := e.(Configured)
+		if !ok || !c.Configured() {
+			continue
+		}
+		p, ok := e.(Prober)
+		if !ok {
+			continue
+		}
+		if v, err := p.Probe(ctx); err == nil {
+			if v == "" {
+				v = Unknown
+			}
+			out[name] = v
+		}
+	}
+	return out
+}
+
 // Probe reports whether the llama.cpp server is installed.
 func (l *LlamaCpp) Probe(context.Context) (string, error) {
 	return installedBinary(l.bin())
 }
 
+// Configured reports whether the configuration names vLLM.
+func (v *VLLM) Configured() bool { return v.cfg.Bin != "" }
+
+// Configured reports whether the configuration names SGLang's interpreter.
+func (g *SGLang) Configured() bool { return g.cfg.Python != "" }
+
 // Probe reports whether vLLM is installed.
 func (v *VLLM) Probe(context.Context) (string, error) {
 	bin := v.cfg.Bin

internal/agent/logring.go

@@ -0,0 +1,84 @@
+package agent
+
+import (
+	"context"
+	"log/slog"
+	"strings"
+	"sync"
+)
+
+// agentLogLines is how much of its own log an agent keeps for the admin page.
+// Enough to cover an engine install or a failed start; the journal keeps the
+// rest for anyone who can reach it.
+const agentLogLines = 2000
+
+// logRing keeps the most recent lines written to it.
+type logRing struct {
+	mu    sync.Mutex
+	lines []string
+	next  int
+	full  bool
+}
+
+func newLogRing(n int) *logRing { return &logRing{lines: make([]string, n)} }
+
+// Write takes whole lines, as a text handler writes them.
+func (r *logRing) Write(p []byte) (int, error) {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	for _, line := range strings.Split(strings.TrimRight(string(p), "\n"), "\n") {
+		r.lines[r.next] = line
+		r.next = (r.next + 1) % len(r.lines)
+		if r.next == 0 {
+			r.full = true
+		}
+	}
+	return len(p), nil
+}
+
+// tail returns up to n of the most recent lines, oldest first.
+func (r *logRing) tail(n int) []string {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	size := r.next
+	if r.full {
+		size = len(r.lines)
+	}
+	n = min(n, size)
+	out := make([]string, 0, n)
+	for i := size - n; i < size; i++ {
+		idx := i
+		if r.full {
+			idx = (r.next + i) % len(r.lines)
+		}
+		out = append(out, r.lines[idx])
+	}
+	return out
+}
+
+// teeHandler sends every record to two handlers: wherever the agent already
+// logs, and the ring the admin page reads.
+type teeHandler struct{ a, b slog.Handler }
+
+func (t teeHandler) Enabled(ctx context.Context, l slog.Level) bool {
+	return t.a.Enabled(ctx, l) || t.b.Enabled(ctx, l)
+}
+
+func (t teeHandler) Handle(ctx context.Context, r slog.Record) error {
+	var err error
+	if t.a.Enabled(ctx, r.Level) {
+		err = t.a.Handle(ctx, r.Clone())
+	}
+	if t.b.Enabled(ctx, r.Level) {
+		_ = t.b.Handle(ctx, r.Clone())
+	}
+	return err
+}
+
+func (t teeHandler) WithAttrs(as []slog.Attr) slog.Handler {
+	return teeHandler{t.a.WithAttrs(as), t.b.WithAttrs(as)}
+}
+
+func (t teeHandler) WithGroup(name string) slog.Handler {
+	return teeHandler{t.a.WithGroup(name), t.b.WithGroup(name)}
+}

internal/agent/logring_test.go

@@ -0,0 +1,49 @@
+package agent
+
+import (
+	"fmt"
+	"slices"
+	"strings"
+	"testing"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+)
+
+func TestTheLogRingKeepsTheNewestLinesInOrder(t *testing.T) {
+	r := newLogRing(3)
+	if got := r.tail(10); len(got) != 0 {
+		t.Fatalf("empty ring gave %v", got)
+	}
+	for i := 1; i <= 5; i++ {
+		fmt.Fprintf(r, "line %d\n", i)
+	}
+	if got := r.tail(10); !slices.Equal(got, []string{"line 3", "line 4", "line 5"}) {
+		t.Errorf("after wrapping: %v", got)
+	}
+	if got := r.tail(2); !slices.Equal(got, []string{"line 4", "line 5"}) {
+		t.Errorf("tail(2): %v", got)
+	}
+}
+
+func TestANodeAnswersForItsOwnLog(t *testing.T) {
+	cfg := Config{Server: "127.0.0.1:7443", NodeID: "n1", WeightsCache: t.TempDir(), EnginesDir: t.TempDir()}
+	cfg.Superv.WorkDir = t.TempDir()
+	a, err := New(cfg, nil, nil)
+	if err != nil {
+		t.Fatal(err)
+	}
+	a.log.Info("engine installation", "state", "installing")
+
+	r := a.logsReply(&llmgwv1.LogsRequest{RequestId: "x", Source: "agent", Lines: 50})
+	if r.GetRequestId() != "x" || !strings.Contains(strings.Join(r.GetLines(), "\n"), "engine installation") {
+		t.Errorf("the agent's own line is not in its log: %v", r.GetLines())
+	}
+	if !slices.Contains(r.GetSources(), "agent") {
+		t.Errorf("sources %v", r.GetSources())
+	}
+
+	r = a.logsReply(&llmgwv1.LogsRequest{RequestId: "y", Source: "no-such-shard"})
+	if r.GetError() == "" {
+		t.Error("an unknown engine was answered as if it existed")
+	}
+}

internal/agent/provision/install.go

@@ -0,0 +1,666 @@
+package provision
+
+import (
+	"archive/tar"
+	"compress/gzip"
+	"context"
+	"crypto/sha256"
+	"encoding/hex"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"io/fs"
+	"log/slog"
+	"net/http"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"strings"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
+)
+
+// Want is one engine at one version.
+type Want struct {
+	Engine  string
+	Version string
+}
+
+// The states an engine's installation can be in.
+const (
+	Installing  = "installing"
+	Ready       = "ready"
+	Failed      = "failed"
+	Unsupported = "unsupported"
+)
+
+// Status is where one engine's installation stands.
+type Status struct {
+	Engine  string
+	Version string
+	Variant string
+	State   string
+	// Detail is for a person: what is happening, or what went wrong.
+	Detail string
+	// Binary is the executable, once ready.
+	Binary string
+}
+
+// Installer installs engines under one directory.
+type Installer struct {
+	// Root is where engines are installed, one directory per build:
+	// <root>/<engine>/<version>-<variant>.
+	Root string
+	Host discovery.Host
+	Log  *slog.Logger
+	HTTP *http.Client
+	// OnStatus hears every change, so a download that takes minutes is seen
+	// to be under way rather than silent.
+	OnStatus func(Status)
+
+	// Run executes a command; a seam for tests. Nil runs it for real.
+	Run func(ctx context.Context, env []string, name string, args ...string) ([]byte, error)
+	// APIBase and DownloadBase default to GitHub; tests point them at a fake.
+	APIBase      string
+	DownloadBase string
+}
+
+// DefaultRoot is where engines are installed when nothing says otherwise.
+const DefaultRoot = "/opt/llmgw/engines"
+
+// marker is written into a build's directory once it has been installed and
+// seen to run. Its absence means the directory is not to be trusted.
+const marker = ".llmgw-installed"
+
+type markerFile struct {
+	Engine    string            `json:"engine"`
+	Version   string            `json:"version"`
+	Variant   string            `json:"variant"`
+	Binary    string            `json:"binary"`
+	SHA256    map[string]string `json:"sha256"`
+	Installed time.Time         `json:"installed"`
+}
+
+// Converge brings every wanted engine to ready, or says why it cannot.
+//
+// An engine already installed is checked again rather than believed: its
+// directory can be damaged, or a library it needs removed by somebody's
+// upgrade, and that is found here rather than by the next replica to fail.
+func (in *Installer) Converge(ctx context.Context, wants []Want) []Status {
+	out := make([]Status, 0, len(wants))
+	for _, w := range wants {
+		st := in.one(ctx, w)
+		in.report(st)
+		out = append(out, st)
+	}
+	return out
+}
+
+func (in *Installer) one(ctx context.Context, w Want) Status {
+	st := Status{Engine: w.Engine, Version: w.Version}
+	recipe, ok := Recipes[w.Engine]
+	if !ok {
+		st.State = Unsupported
+		st.Detail = "this agent does not know how to install " + w.Engine +
+			"; install it by hand and name it in the agent's configuration"
+		return st
+	}
+	build, err := recipe.Choose(w.Version, in.Host)
+	if err != nil {
+		st.State, st.Detail = Unsupported, err.Error()
+		return st
+	}
+	st.Variant = build.Variant
+	dir := filepath.Join(in.root(), w.Engine, w.Version+"-"+build.Variant)
+
+	// Installed before: check it still runs, and trust nothing else about it.
+	if _, err := os.Stat(filepath.Join(dir, marker)); err == nil {
+		// The packages are checked too. Removing one is exactly the kind of
+		// change an unrelated upgrade makes, and ldd below would catch it only
+		// as a missing library with no hint of which package held it.
+		if err := in.packages(ctx, build.Packages, &st); err != nil {
+			st.State, st.Detail = Failed, err.Error()
+			return st
+		}
+		bin, version, err := in.verify(ctx, dir, build)
+		if err == nil {
+			return in.ready(st, build, bin, version)
+		}
+		in.log().Warn("an installed engine no longer runs; installing it again",
+			"engine", w.Engine, "version", w.Version, "err", err)
+	}
+
+	if err := in.install(ctx, recipe, w, build, dir, &st); err != nil {
+		st.State, st.Detail, st.Binary = Failed, err.Error(), ""
+		return st
+	}
+	bin, version, err := in.verify(ctx, dir, build)
+	if err != nil {
+		st.State, st.Detail = Failed, err.Error()
+		return st
+	}
+	return in.ready(st, build, bin, version)
+}
+
+func (in *Installer) ready(st Status, b Build, bin, version string) Status {
+	st.State, st.Binary = Ready, bin
+	st.Detail = strings.TrimSpace(filepath.Base(bin) + " " + version)
+	var missing []string
+	for _, f := range b.CPUFlags {
+		if !in.Host.HasCPUFlag(f) {
+			missing = append(missing, f)
+		}
+	}
+	if len(missing) > 0 {
+		// Said on every report, not once in a log: it is the difference between
+		// five tokens a second and forty, and a VM given QEMU's default CPU
+		// model looks entirely healthy otherwise.
+		st.Detail += "; this CPU lacks " + strings.Join(missing, ", ") +
+			", so it will run several times slower (a VM needs CPU type \"host\")"
+	}
+	return st
+}
+
+// install fetches, checks and unpacks a build into dir.
+//
+// Into a scratch directory renamed into place at the end, so a failure halfway
+// leaves nothing that looks installed.
+func (in *Installer) install(ctx context.Context, r Recipe, w Want, b Build, dir string, st *Status) error {
+	if err := in.packages(ctx, b.Packages, st); err != nil {
+		return err
+	}
+
+	in.progress(st, "asking GitHub for the checksums of %s %s", w.Engine, w.Version)
+	sums, err := in.checksums(ctx, r.Repo, w.Version)
+	if err != nil {
+		return err
+	}
+
+	tmp := dir + ".partial"
+	_ = os.RemoveAll(tmp)
+	if err := os.MkdirAll(tmp, 0o755); err != nil {
+		return err
+	}
+	defer os.RemoveAll(tmp) //nolint:errcheck // gone either way once renamed
+
+	got := map[string]string{}
+	for _, asset := range b.Assets {
+		want, ok := sums[asset]
+		if !ok {
+			// Refused rather than installed unchecked: this is a binary that
+			// will run as root on every node.
+			return fmt.Errorf("%s %s publishes no checksum for %s, so it cannot be verified",
+				w.Engine, w.Version, asset)
+		}
+		archive := filepath.Join(tmp, asset+".download")
+		if err := in.download(ctx, r.Repo, w.Version, asset, archive, want, st); err != nil {
+			return err
+		}
+		in.progress(st, "unpacking %s", asset)
+		if err := untarGz(archive, tmp); err != nil {
+			return fmt.Errorf("unpacking %s: %w", asset, err)
+		}
+		_ = os.Remove(archive)
+		got[asset] = want
+	}
+
+	in.progress(st, "checking that %s runs", b.Binary)
+	bin, _, err := in.verify(ctx, tmp, b)
+	if err != nil {
+		return err
+	}
+	rel, _ := filepath.Rel(tmp, bin)
+	m, _ := json.MarshalIndent(markerFile{
+		Engine: w.Engine, Version: w.Version, Variant: b.Variant,
+		Binary: rel, SHA256: got, Installed: time.Now().UTC(),
+	}, "", "  ")
+	if err := os.WriteFile(filepath.Join(tmp, marker), m, 0o644); err != nil { //nolint:gosec // not secret
+		return err
+	}
+	_ = os.RemoveAll(dir)
+	if err := os.Rename(tmp, dir); err != nil {
+		return err
+	}
+	in.log().Info("installed an engine", "engine", w.Engine, "version", w.Version,
+		"variant", b.Variant, "dir", dir)
+	return nil
+}
+
+// verify finds the binary and runs it.
+//
+// Two checks, because they fail differently. ldd names a missing library
+// outright, where running the binary would only say it could not start. And
+// running it is the only proof that the rest — the right instruction set, the
+// right driver — is in place.
+func (in *Installer) verify(ctx context.Context, dir string, b Build) (bin, version string, err error) {
+	bin, err = findFile(dir, b.Binary)
+	if err != nil {
+		return "", "", err
+	}
+	if out, err := in.run(ctx, nil, "ldd", bin); err == nil {
+		if missing := lddMissing(string(out)); len(missing) > 0 {
+			return "", "", fmt.Errorf("%s cannot run on this system: %s", b.Binary, strings.Join(missing, "; "))
+		}
+	}
+	tctx, cancel := context.WithTimeout(ctx, 60*time.Second)
+	defer cancel()
+	out, err := in.run(tctx, nil, bin, "--version")
+	if err != nil {
+		return "", "", fmt.Errorf("%s --version failed: %v: %s", b.Binary, err, lastLine(out))
+	}
+	return bin, firstVersionLine(out), nil
+}
+
+// packages installs the operating system packages a build needs.
+//
+// Only those missing, and from the node's own package sources: a package the
+// system already has is not touched, so an engine never upgrades something
+// the rest of the machine depends on.
+func (in *Installer) packages(ctx context.Context, byFamily map[string][]string, st *Status) error {
+	family := in.Host.Family()
+	pkgs := byFamily[family]
+	if len(pkgs) == 0 {
+		// A system of no family this knows is not refused: its packages may
+		// well be there, and if they are not, ldd names the missing library.
+		return nil
+	}
+	pm := packageManagers[family]
+	var missing []string
+	for _, p := range pkgs {
+		if _, err := in.run(ctx, nil, pm.query[0], append(pm.query[1:], p)...); err != nil {
+			missing = append(missing, p)
+		}
+	}
+	if len(missing) == 0 {
+		return nil
+	}
+	tool, cmds := pm.install(in, missing)
+	if tool == "" {
+		return fmt.Errorf("%s is missing and no package manager was found to install it",
+			strings.Join(missing, ", "))
+	}
+	in.progress(st, "installing %s with %s", strings.Join(missing, ", "), tool)
+	for _, c := range cmds {
+		if err := in.runWaitingForLock(ctx, pm.env, c, st); err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+// InstallPackages installs whichever of these packages the machine lacks,
+// from its own package sources, waiting out another package manager's lock.
+// byFamily is keyed by discovery.Host.Family: "deb" or "rpm".
+func (in *Installer) InstallPackages(ctx context.Context, byFamily map[string][]string) error {
+	var st Status
+	return in.packages(ctx, byFamily, &st)
+}
+
+// lockWait bounds how long a package command waits for another to finish.
+const lockWait = 10 * time.Minute
+
+// lockRetry is how often a locked package manager is tried again.
+var lockRetry = 10 * time.Second
+
+// runWaitingForLock runs a package command, waiting while another holds the
+// package manager's lock.
+//
+// A machine a minute old is running cloud-init's and unattended-upgrades' own
+// apt. apt's DPkg::Lock::Timeout covers dpkg's lock and not the one `apt-get
+// update` takes on its lists, so the first install on a new node failed with
+// "Unable to lock directory /var/lib/apt/lists/" — found by building one.
+func (in *Installer) runWaitingForLock(ctx context.Context, env, c []string, st *Status) error {
+	deadline := time.Now().Add(lockWait)
+	for {
+		out, err := in.run(ctx, env, c[0], c[1:]...)
+		if err == nil {
+			return nil
+		}
+		if !lockHeld(string(out)) || time.Now().After(deadline) {
+			return fmt.Errorf("%s: %v: %s", strings.Join(c, " "), err, lastLine(out))
+		}
+		in.progress(st, "waiting for another %s to finish (a new machine runs its own updates first)", c[0])
+		select {
+		case <-ctx.Done():
+			return ctx.Err()
+		case <-time.After(lockRetry):
+		}
+	}
+}
+
+// lockHeld reports whether a package command failed only because another
+// held the lock.
+func lockHeld(out string) bool {
+	for _, s := range []string{"Unable to lock", "Could not get lock", "is locked by another process",
+		"waiting for process with pid", "System management is locked"} {
+		if strings.Contains(out, s) {
+			return true
+		}
+	}
+	return false
+}
+
+// packageManager is how one packaging family asks and installs.
+type packageManager struct {
+	// query exits zero when the package named after it is installed.
+	query []string
+	env   []string
+	// install picks the tool present on this machine and the commands to run.
+	install func(in *Installer, pkgs []string) (tool string, cmds [][]string)
+}
+
+var packageManagers = map[string]packageManager{
+	"deb": {
+		// dpkg -s exits zero for a package merely known about, so the status
+		// is asked for and checked by the command itself: ${db:Status-Abbrev}
+		// is "ii " only when installed.
+		query: []string{"sh", "-c", `test "$(dpkg-query -W -f='${db:Status-Abbrev}' "$0" 2>/dev/null)" = "ii "`},
+		env:   []string{"DEBIAN_FRONTEND=noninteractive"},
+		install: func(_ *Installer, pkgs []string) (string, [][]string) {
+			// A lock timeout rather than failing on the lock: a machine a
+			// minute old is running cloud-init's and unattended-upgrades' own
+			// apt, and the first attempt on every new node would fail.
+			lock := "-o=DPkg::Lock::Timeout=600"
+			return "apt-get", [][]string{
+				{"apt-get", lock, "update"},
+				append([]string{"apt-get", lock, "install", "-y", "--no-install-recommends"}, pkgs...),
+			}
+		},
+	},
+	"rpm": {
+		query: []string{"rpm", "-q"},
+		install: func(in *Installer, pkgs []string) (string, [][]string) {
+			for _, tool := range []string{"dnf", "yum"} {
+				if in.has(tool) {
+					return tool, [][]string{append([]string{tool, "install", "-y"}, pkgs...)}
+				}
+			}
+			if in.has("zypper") {
+				return "zypper", [][]string{append([]string{"zypper", "--non-interactive", "install"}, pkgs...)}
+			}
+			return "", nil
+		},
+	},
+}
+
+// has reports whether a command is on PATH. Asked through the same seam as
+// every other command, so a test can say which tools a machine has.
+func (in *Installer) has(tool string) bool {
+	_, err := in.run(context.Background(), nil, "sh", "-c", `command -v "$0"`, tool)
+	return err == nil
+}
+
+// checksums reads the sha256 GitHub publishes for each asset of a release.
+func (in *Installer) checksums(ctx context.Context, repo, tag string) (map[string]string, error) {
+	url := strings.TrimSuffix(in.apiBase(), "/") + "/repos/" + repo + "/releases/tags/" + tag
+	req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
+	if err != nil {
+		return nil, err
+	}
+	req.Header.Set("Accept", "application/vnd.github+json")
+	resp, err := in.client().Do(req)
+	if err != nil {
+		return nil, fmt.Errorf("reaching GitHub for %s %s: %w", repo, tag, err)
+	}
+	defer resp.Body.Close()
+	switch {
+	case resp.StatusCode == http.StatusNotFound:
+		return nil, fmt.Errorf("%s has no release %s", repo, tag)
+	case resp.StatusCode == http.StatusForbidden || resp.StatusCode == http.StatusTooManyRequests:
+		// Unauthenticated callers get sixty requests an hour per address, and
+		// a whole fleet behind one NAT shares them.
+		return nil, fmt.Errorf("GitHub is rate-limiting this address (%s); it retries later", resp.Status)
+	case resp.StatusCode != http.StatusOK:
+		return nil, fmt.Errorf("GitHub answered %s for %s %s", resp.Status, repo, tag)
+	}
+	var rel struct {
+		Assets []struct {
+			Name   string `json:"name"`
+			Digest string `json:"digest"`
+		} `json:"assets"`
+	}
+	if err := json.NewDecoder(resp.Body).Decode(&rel); err != nil {
+		return nil, fmt.Errorf("reading %s %s from GitHub: %w", repo, tag, err)
+	}
+	out := map[string]string{}
+	for _, a := range rel.Assets {
+		if sum, ok := strings.CutPrefix(a.Digest, "sha256:"); ok && len(sum) == 64 {
+			out[a.Name] = sum
+		}
+	}
+	return out, nil
+}
+
+func (in *Installer) download(ctx context.Context, repo, tag, asset, dst, want string, st *Status) error {
+	url := strings.TrimSuffix(in.downloadBase(), "/") + "/" + repo + "/releases/download/" + tag + "/" + asset
+	req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
+	if err != nil {
+		return err
+	}
+	resp, err := in.client().Do(req)
+	if err != nil {
+		return fmt.Errorf("downloading %s: %w", asset, err)
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode != http.StatusOK {
+		return fmt.Errorf("downloading %s: %s", asset, resp.Status)
+	}
+	if resp.ContentLength > 0 {
+		in.progress(st, "downloading %s (%d MB)", asset, resp.ContentLength>>20)
+	} else {
+		in.progress(st, "downloading %s", asset)
+	}
+	f, err := os.Create(dst) //nolint:gosec // inside the install root
+	if err != nil {
+		return err
+	}
+	h := sha256.New()
+	_, err = io.Copy(io.MultiWriter(f, h), resp.Body)
+	if cerr := f.Close(); err == nil {
+		err = cerr
+	}
+	if err != nil {
+		return fmt.Errorf("downloading %s: %w", asset, err)
+	}
+	if got := hex.EncodeToString(h.Sum(nil)); got != want {
+		return fmt.Errorf("%s does not match the checksum its publisher gave (got %s, expected %s)",
+			asset, got[:12], want[:12])
+	}
+	return nil
+}
+
+// untarGz unpacks an archive under dir, refusing anything that would land
+// outside it. A release archive is somebody else's file, and a path of
+// "../../etc/cron.d/x" in it would otherwise be written as root.
+func untarGz(archive, dir string) error {
+	f, err := os.Open(archive) //nolint:gosec // inside the install root
+	if err != nil {
+		return err
+	}
+	defer f.Close()
+	gz, err := gzip.NewReader(f)
+	if err != nil {
+		return err
+	}
+	tr := tar.NewReader(gz)
+	for {
+		h, err := tr.Next()
+		if errors.Is(err, io.EOF) {
+			return nil
+		}
+		if err != nil {
+			return err
+		}
+		target, err := within(dir, h.Name)
+		if err != nil {
+			return err
+		}
+		switch h.Typeflag {
+		case tar.TypeDir:
+			if err := os.MkdirAll(target, 0o755); err != nil {
+				return err
+			}
+		case tar.TypeReg:
+			if err := os.MkdirAll(filepath.Dir(target), 0o755); err != nil {
+				return err
+			}
+			out, err := os.OpenFile(target, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, fs.FileMode(h.Mode)&0o755) //nolint:gosec // checked above
+			if err != nil {
+				return err
+			}
+			if _, err := io.Copy(out, tr); err != nil { //nolint:gosec // a release archive, size bounded by its checksum
+				out.Close()
+				return err
+			}
+			if err := out.Close(); err != nil {
+				return err
+			}
+		case tar.TypeSymlink:
+			// Libraries are versioned through symlinks, and those are
+			// relative. One pointing anywhere else is refused.
+			if filepath.IsAbs(h.Linkname) {
+				return fmt.Errorf("%s links outside the archive", h.Name)
+			}
+			if _, err := within(dir, filepath.Join(filepath.Dir(h.Name), h.Linkname)); err != nil {
+				return fmt.Errorf("%s links outside the archive", h.Name)
+			}
+			if err := os.MkdirAll(filepath.Dir(target), 0o755); err != nil {
+				return err
+			}
+			_ = os.Remove(target)
+			if err := os.Symlink(h.Linkname, target); err != nil {
+				return err
+			}
+		default:
+			// Devices, FIFOs and hard links have no business in a release.
+		}
+	}
+}
+
+func within(dir, name string) (string, error) {
+	target := filepath.Join(dir, name) //nolint:gosec // checked on the next line
+	rel, err := filepath.Rel(dir, target)
+	if err != nil || rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) {
+		return "", fmt.Errorf("%s would unpack outside the install directory", name)
+	}
+	return target, nil
+}
+
+// lddMissing reads what ldd could not satisfy, in two shapes:
+//
+//	libgomp.so.1 => not found
+//	/lib64/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by ...)
+//
+// The second is a system too old for the build — RHEL 9 against a build that
+// wants GCC 12's libstdc++ — and is said as such, rather than as a library
+// that is plainly there being "not found".
+func lddMissing(out string) []string {
+	seen := map[string]bool{}
+	var missing []string
+	for _, line := range strings.Split(out, "\n") {
+		line = strings.TrimSpace(line)
+		if !strings.Contains(line, "not found") {
+			continue
+		}
+		var what string
+		if lib, rest, ok := strings.Cut(line, ": version `"); ok {
+			sym, _, _ := strings.Cut(rest, "'")
+			what = filepath.Base(lib) + " is too old (needs " + sym + ")"
+		} else {
+			what = strings.TrimSpace(strings.SplitN(line, "=>", 2)[0]) + " is not installed"
+		}
+		if !seen[what] {
+			seen[what] = true
+			missing = append(missing, what)
+		}
+	}
+	return missing
+}
+
+func findFile(dir, name string) (string, error) {
+	var found string
+	_ = filepath.WalkDir(dir, func(p string, d fs.DirEntry, err error) error {
+		if err != nil || found != "" {
+			return nil //nolint:nilerr // an unreadable subtree is not the binary
+		}
+		if !d.IsDir() && d.Name() == name {
+			found = p
+		}
+		return nil
+	})
+	if found == "" {
+		return "", fmt.Errorf("the build has no %s in it", name)
+	}
+	return found, nil
+}
+
+func (in *Installer) progress(st *Status, format string, args ...any) {
+	st.State, st.Detail = Installing, fmt.Sprintf(format, args...)
+	in.report(*st)
+}
+
+func (in *Installer) report(st Status) {
+	if in.OnStatus != nil {
+		in.OnStatus(st)
+	}
+}
+
+func (in *Installer) run(ctx context.Context, env []string, name string, args ...string) ([]byte, error) {
+	if in.Run != nil {
+		return in.Run(ctx, env, name, args...)
+	}
+	cmd := exec.CommandContext(ctx, name, args...) //nolint:gosec // fixed commands and paths we installed
+	cmd.Env = append(os.Environ(), env...)
+	return cmd.CombinedOutput()
+}
+
+func (in *Installer) root() string {
+	if in.Root != "" {
+		return in.Root
+	}
+	return DefaultRoot
+}
+
+func (in *Installer) log() *slog.Logger {
+	if in.Log != nil {
+		return in.Log
+	}
+	return slog.Default()
+}
+
+func (in *Installer) client() *http.Client {
+	if in.HTTP != nil {
+		return in.HTTP
+	}
+	return &http.Client{Timeout: 30 * time.Minute}
+}
+
+func (in *Installer) apiBase() string {
+	if in.APIBase != "" {
+		return in.APIBase
+	}
+	return "https://api.github.com"
+}
+
+func (in *Installer) downloadBase() string {
+	if in.DownloadBase != "" {
+		return in.DownloadBase
+	}
+	return "https://github.com"
+}
+
+func lastLine(b []byte) string {
+	lines := strings.Split(strings.TrimSpace(string(b)), "\n")
+	return strings.TrimSpace(lines[len(lines)-1])
+}
+
+func firstVersionLine(b []byte) string {
+	for _, l := range strings.Split(string(b), "\n") {
+		if strings.Contains(strings.ToLower(l), "version") {
+			return strings.TrimSpace(l)
+		}
+	}
+	return ""
+}

internal/agent/provision/install_test.go

@@ -0,0 +1,332 @@
+package provision
+
+import (
+	"archive/tar"
+	"bytes"
+	"compress/gzip"
+	"context"
+	"crypto/sha256"
+	"encoding/hex"
+	"encoding/json"
+	"errors"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"strings"
+	"sync"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
+)
+
+var ubuntu = discovery.Host{OSID: "ubuntu", OSVersion: "24.04", Arch: "amd64", CPUFlags: []string{"avx2"}}
+
+// release is a fake GitHub: one release, its assets and their digests.
+type release struct {
+	assets  map[string][]byte
+	digests map[string]string
+	gets    int
+	mu      sync.Mutex
+}
+
+func tarGz(t *testing.T, files map[string]string, links map[string]string) []byte {
+	t.Helper()
+	var buf bytes.Buffer
+	gz := gzip.NewWriter(&buf)
+	tw := tar.NewWriter(gz)
+	for name, body := range files {
+		_ = tw.WriteHeader(&tar.Header{Name: name, Mode: 0o755, Size: int64(len(body)), Typeflag: tar.TypeReg})
+		_, _ = tw.Write([]byte(body))
+	}
+	for name, target := range links {
+		_ = tw.WriteHeader(&tar.Header{Name: name, Linkname: target, Typeflag: tar.TypeSymlink})
+	}
+	_ = tw.Close()
+	_ = gz.Close()
+	return buf.Bytes()
+}
+
+func (r *release) serve(t *testing.T) *httptest.Server {
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
+		r.mu.Lock()
+		r.gets++
+		r.mu.Unlock()
+		if strings.HasPrefix(req.URL.Path, "/repos/") {
+			var out struct {
+				Assets []map[string]string `json:"assets"`
+			}
+			for name := range r.assets {
+				out.Assets = append(out.Assets, map[string]string{"name": name, "digest": r.digests[name]})
+			}
+			_ = json.NewEncoder(w).Encode(out)
+			return
+		}
+		body, ok := r.assets[filepath.Base(req.URL.Path)]
+		if !ok {
+			http.NotFound(w, req)
+			return
+		}
+		_, _ = w.Write(body)
+	}))
+	t.Cleanup(srv.Close)
+	return srv
+}
+
+func newRelease(assets map[string][]byte) *release {
+	r := &release{assets: assets, digests: map[string]string{}}
+	for name, b := range assets {
+		sum := sha256.Sum256(b)
+		r.digests[name] = "sha256:" + hex.EncodeToString(sum[:])
+	}
+	return r
+}
+
+// fakeSystem answers the package tools, ldd and the engine itself.
+type fakeSystem struct {
+	mu        sync.Mutex
+	installed map[string]bool
+	tools     map[string]bool
+	lddOut    string
+	calls     []string
+	// locked is how many more package commands find the lock held.
+	locked int
+}
+
+var errExit = errors.New("exit 1")
+
+func (f *fakeSystem) run(_ context.Context, _ []string, name string, args ...string) ([]byte, error) {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	f.calls = append(f.calls, name+" "+strings.Join(args, " "))
+	switch {
+	case name == "sh" && strings.Contains(args[1], "dpkg-query"),
+		name == "rpm" && args[0] == "-q":
+		if f.installed[args[len(args)-1]] {
+			return nil, nil
+		}
+		return nil, errExit
+	case name == "sh" && strings.Contains(args[1], "command -v"):
+		if f.tools[args[2]] {
+			return nil, nil
+		}
+		return nil, errExit
+	case name == "apt-get" || name == "dnf":
+		if f.locked > 0 {
+			f.locked--
+			return []byte("E: Unable to lock directory /var/lib/apt/lists/\n"), errExit
+		}
+		for i, a := range args {
+			if a == "install" {
+				for _, p := range args[i+1:] {
+					if !strings.HasPrefix(p, "-") {
+						f.installed[p] = true
+					}
+				}
+			}
+		}
+		return nil, nil
+	case name == "ldd":
+		return []byte(f.lddOut), nil
+	case strings.HasSuffix(name, "llama-server"):
+		return []byte("version: 10991 (abc)\n"), nil
+	}
+	return nil, errors.New("unexpected command " + name)
+}
+
+func installer(t *testing.T, srv *httptest.Server, sys *fakeSystem, host discovery.Host) (*Installer, *[]Status) {
+	var seen []Status
+	var mu sync.Mutex
+	return &Installer{
+		Root: t.TempDir(), Host: host, Run: sys.run,
+		APIBase: srv.URL, DownloadBase: srv.URL, HTTP: srv.Client(),
+		OnStatus: func(s Status) { mu.Lock(); seen = append(seen, s); mu.Unlock() },
+	}, &seen
+}
+
+func llamaRelease(t *testing.T) *release {
+	return newRelease(map[string][]byte{
+		"llama-b1-bin-ubuntu-x64.tar.gz": tarGz(t,
+			map[string]string{"llama-b1/llama-server": "#!/bin/sh\n", "llama-b1/libllama.so.0": "x"},
+			map[string]string{"llama-b1/libllama.so": "libllama.so.0"}),
+	})
+}
+
+func TestAnEngineIsInstalledFromNothing(t *testing.T) {
+	// A node rebuilt from a bare image: no engine, no OpenMP runtime. What it
+	// ends with has to have come from the publisher, checked, and been run.
+	rel := llamaRelease(t)
+	sys := &fakeSystem{installed: map[string]bool{}}
+	in, seen := installer(t, rel.serve(t), sys, ubuntu)
+
+	got := in.Converge(t.Context(), []Want{{Engine: "llamacpp", Version: "b1"}})
+	if len(got) != 1 || got[0].State != Ready {
+		t.Fatalf("not ready: %+v", got)
+	}
+	if !sys.installed["libgomp1"] {
+		t.Error("the OpenMP runtime the build links against was not installed")
+	}
+	if !strings.HasPrefix(got[0].Binary, in.Root) {
+		t.Errorf("binary %s is not the one installed under %s", got[0].Binary, in.Root)
+	}
+	if _, err := os.Lstat(filepath.Join(filepath.Dir(got[0].Binary), "libllama.so")); err != nil {
+		t.Errorf("the library symlink was not unpacked: %v", err)
+	}
+	var states []string
+	for _, s := range *seen {
+		states = append(states, s.State+": "+s.Detail)
+	}
+	if !strings.Contains(strings.Join(states, "\n"), "downloading") {
+		t.Errorf("a download was not reported while it ran:\n%s", strings.Join(states, "\n"))
+	}
+}
+
+func TestAnInstalledEngineIsCheckedNotFetchedAgain(t *testing.T) {
+	rel := llamaRelease(t)
+	sys := &fakeSystem{installed: map[string]bool{}}
+	in, _ := installer(t, rel.serve(t), sys, ubuntu)
+	w := []Want{{Engine: "llamacpp", Version: "b1"}}
+	in.Converge(t.Context(), w)
+	before := rel.gets
+
+	got := in.Converge(t.Context(), w)
+	if got[0].State != Ready {
+		t.Fatalf("second pass: %+v", got[0])
+	}
+	if rel.gets != before {
+		t.Errorf("a ready engine was downloaded again (%d requests)", rel.gets-before)
+	}
+}
+
+func TestAnEngineThatCannotLoadALibraryIsNotReady(t *testing.T) {
+	// Found on a node before: llama-server present, executable, and unable to
+	// start for want of a library — while the node reported ready.
+	rel := llamaRelease(t)
+	sys := &fakeSystem{installed: map[string]bool{"libgomp1": true},
+		lddOut: "\tlibcuda.so.1 => not found\n"}
+	in, _ := installer(t, rel.serve(t), sys, ubuntu)
+
+	got := in.Converge(t.Context(), []Want{{Engine: "llamacpp", Version: "b1"}})
+	if got[0].State != Failed || !strings.Contains(got[0].Detail, "libcuda.so.1") {
+		t.Fatalf("expected a failure naming the library: %+v", got[0])
+	}
+	if _, err := os.Stat(filepath.Join(in.Root, "llamacpp", "b1-cpu", marker)); err == nil {
+		t.Error("a build that does not run was marked installed")
+	}
+}
+
+func TestADownloadThatDoesNotMatchItsChecksumIsRefused(t *testing.T) {
+	rel := llamaRelease(t)
+	for name := range rel.digests {
+		rel.digests[name] = "sha256:" + strings.Repeat("0", 64)
+	}
+	sys := &fakeSystem{installed: map[string]bool{"libgomp1": true}}
+	in, _ := installer(t, rel.serve(t), sys, ubuntu)
+
+	got := in.Converge(t.Context(), []Want{{Engine: "llamacpp", Version: "b1"}})
+	if got[0].State != Failed || !strings.Contains(got[0].Detail, "checksum") {
+		t.Fatalf("a tampered download was not refused: %+v", got[0])
+	}
+}
+
+func TestAnArchiveCannotWriteOutsideItsDirectory(t *testing.T) {
+	rel := newRelease(map[string][]byte{
+		"llama-b1-bin-ubuntu-x64.tar.gz": tarGz(t,
+			map[string]string{"../../escaped": "x", "llama-server": ""}, nil),
+	})
+	sys := &fakeSystem{installed: map[string]bool{"libgomp1": true}}
+	in, _ := installer(t, rel.serve(t), sys, ubuntu)
+
+	got := in.Converge(t.Context(), []Want{{Engine: "llamacpp", Version: "b1"}})
+	if got[0].State != Failed {
+		t.Fatalf("an archive escaping its directory was installed: %+v", got[0])
+	}
+}
+
+func TestTheBuildFollowsTheHardware(t *testing.T) {
+	gpu := ubuntu
+	gpu.NVIDIADriver = "570.86.15"
+	b, err := chooseLlamaCpp("b1", gpu)
+	if err != nil || !strings.HasPrefix(b.Variant, "cuda") || len(b.Assets) != 2 {
+		t.Errorf("an NVIDIA node got %+v, %v; expected the CUDA build and its runtime", b, err)
+	}
+	if _, err := chooseLlamaCpp("b1", discovery.Host{OSID: "ubuntu", Arch: "riscv64"}); err == nil {
+		t.Error("a build was chosen for an architecture llama.cpp does not publish for")
+	}
+
+	slow := ubuntu
+	slow.CPUFlags = nil
+	st := (&Installer{Host: slow}).ready(Status{}, Build{CPUFlags: []string{"avx2"}}, "/x/llama-server", "")
+	if !strings.Contains(st.Detail, "avx2") {
+		t.Errorf("a CPU without AVX2 was not called out: %q", st.Detail)
+	}
+}
+
+func TestAnEngineTheAgentCannotInstallSaysSo(t *testing.T) {
+	in := &Installer{Root: t.TempDir(), Host: ubuntu}
+	got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.29.0"}})
+	if got[0].State != Unsupported {
+		t.Errorf("got %+v", got[0])
+	}
+}
+
+func TestARedHatFamilyNodeInstallsWithItsOwnPackageManager(t *testing.T) {
+	// The build is not Ubuntu's in any way that matters; the package that
+	// carries OpenMP's runtime is named differently, and installed by dnf.
+	rocky := discovery.Host{OSID: "rocky", OSVersion: "10.0", OSLike: []string{"rhel", "centos", "fedora"},
+		Arch: "amd64", CPUFlags: []string{"avx2"}}
+	rel := llamaRelease(t)
+	sys := &fakeSystem{installed: map[string]bool{}, tools: map[string]bool{"dnf": true}}
+	in, _ := installer(t, rel.serve(t), sys, rocky)
+
+	got := in.Converge(t.Context(), []Want{{Engine: "llamacpp", Version: "b1"}})
+	if got[0].State != Ready {
+		t.Fatalf("not ready: %+v", got[0])
+	}
+	if !sys.installed["libgomp"] || sys.installed["libgomp1"] {
+		t.Errorf("installed %v; expected the rpm name, libgomp", sys.installed)
+	}
+}
+
+func TestASystemTooOldForTheBuildSaysWhatItLacks(t *testing.T) {
+	// RHEL 9: glibc is new enough, libstdc++ is one GCC release short. ldd
+	// finds the library and reports the symbol, and "libstdc++ not found"
+	// would send somebody looking for a package that is installed.
+	rel := llamaRelease(t)
+	sys := &fakeSystem{installed: map[string]bool{"libgomp1": true},
+		lddOut: "\t/lib64/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /x/libllama.so)\n" +
+			"\t/lib64/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /x/llama-server)\n"}
+	in, _ := installer(t, rel.serve(t), sys, ubuntu)
+
+	got := in.Converge(t.Context(), []Want{{Engine: "llamacpp", Version: "b1"}})
+	if got[0].State != Failed || !strings.Contains(got[0].Detail, "libstdc++.so.6 is too old (needs GLIBCXX_3.4.30)") {
+		t.Fatalf("detail %q", got[0].Detail)
+	}
+	if strings.Count(got[0].Detail, "GLIBCXX") != 1 {
+		t.Errorf("the same shortfall was listed more than once: %q", got[0].Detail)
+	}
+}
+
+func TestAPackageManagerBusyWithItsOwnUpdatesIsWaitedFor(t *testing.T) {
+	// A new node's first minutes: cloud-init and unattended-upgrades hold
+	// apt's lists lock, which DPkg::Lock::Timeout does not cover.
+	lockRetry = time.Millisecond
+	t.Cleanup(func() { lockRetry = 10 * time.Second })
+
+	rel := llamaRelease(t)
+	sys := &fakeSystem{installed: map[string]bool{}, locked: 3}
+	in, seen := installer(t, rel.serve(t), sys, ubuntu)
+
+	got := in.Converge(t.Context(), []Want{{Engine: "llamacpp", Version: "b1"}})
+	if got[0].State != Ready {
+		t.Fatalf("gave up on a lock another apt held: %+v", got[0])
+	}
+	waited := false
+	for _, s := range *seen {
+		waited = waited || strings.Contains(s.Detail, "waiting for another apt-get")
+	}
+	if !waited {
+		t.Error("the wait was not reported")
+	}
+}

internal/agent/provision/recipe.go

@@ -0,0 +1,109 @@
+// Package provision installs the engines a node is asked to have.
+//
+// The control plane says which engines at which versions; everything else is
+// worked out here, on the node, because only the node knows it: which build
+// fits its processor and driver, what its operating system is missing, and
+// whether the result actually runs. Nothing already on the machine is taken
+// on trust. An engine counts as installed when this package installed it,
+// checked every file against the checksum its publisher gave, and watched it
+// start.
+package provision
+
+import (
+	"fmt"
+	"strings"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
+)
+
+// Build is one published build of an engine, chosen for one machine.
+type Build struct {
+	// Variant names the build: "cpu", "cuda-12.8".
+	Variant string
+	// Assets are the release files to fetch, unpacked in order into one
+	// directory.
+	Assets []string
+	// Packages are operating system packages the build links against and a
+	// minimal install may not carry, by packaging family ("deb", "rpm"): the
+	// same library is libgomp1 on one and libgomp on the other.
+	Packages map[string][]string
+	// Binary is the executable's file name, found anywhere in the unpacked
+	// tree.
+	Binary string
+	// CPUFlags are extensions the build is slow without. Missing ones are
+	// reported, not refused: slow is still running.
+	CPUFlags []string
+}
+
+// Recipe knows how to install one engine.
+type Recipe struct {
+	// Repo is the GitHub repository whose releases carry the builds.
+	Repo string
+	// Choose picks the build for a machine, or says why none fits.
+	Choose func(version string, h discovery.Host) (Build, error)
+}
+
+// Recipes are the engines this agent can install.
+//
+// vLLM and SGLang are not here yet: they are Python environments of several
+// gigabytes rather than an archive, and are installed by hand and named in the
+// agent's configuration until they are.
+var Recipes = map[string]Recipe{
+	"llamacpp": {Repo: "ggml-org/llama.cpp", Choose: chooseLlamaCpp},
+}
+
+// cudaForLlamaCpp is the CUDA build taken on an NVIDIA node. The release also
+// carries a matching cudart archive, so the node needs a driver and nothing
+// else — no toolkit.
+const cudaForLlamaCpp = "12.8"
+
+func chooseLlamaCpp(version string, h discovery.Host) (Build, error) {
+	// The published Linux builds are built on Ubuntu, but nothing about them
+	// is Ubuntu's: they need glibc 2.34 and the libstdc++ of GCC 12, which
+	// Debian 12 and RHEL 10 have and RHEL 9 does not. So the distribution's
+	// name decides nothing here. The build is tried everywhere, and ldd says
+	// whether this system can load it — in the words of the missing symbol.
+	var arch string
+	switch h.Arch {
+	case "amd64":
+		arch = "x64"
+	case "arm64":
+		arch = "arm64"
+	default:
+		return Build{}, fmt.Errorf("llama.cpp publishes no Linux build for %s", orUnknown(h.Arch))
+	}
+
+	b := Build{
+		Variant: "cpu",
+		Assets:  []string{fmt.Sprintf("llama-%s-bin-ubuntu-%s.tar.gz", version, arch)},
+		// Everything else the build links is in the archive or in any current
+		// glibc system; OpenMP's runtime is the one a cloud image lacks, and
+		// the server does not start without it.
+		Packages: map[string][]string{"deb": {"libgomp1"}, "rpm": {"libgomp"}},
+		Binary:   "llama-server",
+	}
+	if arch == "x64" {
+		// The archive carries a backend per CPU generation and loads the best
+		// one, down to SSE4.2. Below AVX2 it runs, several times slower.
+		b.CPUFlags = []string{"avx2"}
+	}
+	// The CUDA build is published for x64 at this version; on arm64 it is a
+	// newer CUDA that the driver may not support, so an arm64 GPU node runs
+	// the CPU build until someone decides otherwise.
+	if h.NVIDIADriver != "" && arch == "x64" {
+		name := fmt.Sprintf("bin-ubuntu-cuda-%s-x64.tar.gz", cudaForLlamaCpp)
+		b.Variant = "cuda-" + cudaForLlamaCpp
+		b.Assets = []string{
+			fmt.Sprintf("llama-%s-%s", version, name),
+			fmt.Sprintf("cudart-llama-%s-%s", version, name),
+		}
+	}
+	return b, nil
+}
+
+func orUnknown(s string) string {
+	if strings.TrimSpace(s) == "" {
+		return "an unrecognised system"
+	}
+	return s
+}

internal/agent/selfdescribe_test.go

@@ -1,9 +1,11 @@
 package agent
 
 import (
+	"context"
 	"os"
 	"slices"
 	"testing"
+	"time"
 
 	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
 )
@@ -60,15 +62,15 @@ func TestAHeartbeatSaysWhenTheEnginesChanged(t *testing.T) {
 	engines := func(m *llmgwv1.AgentMessage) *llmgwv1.Heartbeat { return m.GetHeartbeat() }
 
 	a.mu.Lock()
-	a.engineVers = map[string]string{"llamacpp": "unknown"}
-	a.enginesSent = enginesFingerprint(a.engineVers)
+	a.configuredVers = map[string]string{"llamacpp": "unknown"}
+	a.enginesSent = enginesFingerprint(a.configuredVers)
 	a.mu.Unlock()
 	if hb := engines(a.heartbeat()); hb.GetEnginesChanged() {
 		t.Errorf("unchanged engines were resent")
 	}
 
 	a.mu.Lock()
-	a.engineVers = map[string]string{"llamacpp": "unknown", "vllm": "unknown"}
+	a.configuredVers = map[string]string{"llamacpp": "unknown", "vllm": "unknown"}
 	a.mu.Unlock()
 	hb := engines(a.heartbeat())
 	if !hb.GetEnginesChanged() || len(hb.GetEngineVersions()) != 2 {
@@ -78,3 +80,40 @@ func TestAHeartbeatSaysWhenTheEnginesChanged(t *testing.T) {
 		t.Errorf("reported again on the next heartbeat")
 	}
 }
+
+func TestWhatTheControlPlaneAsksToInstallIsReportedBack(t *testing.T) {
+	cfg := Config{Server: "127.0.0.1:7443", NodeID: "n1", WeightsCache: t.TempDir(),
+		EnginesDir: t.TempDir()}
+	cfg.Superv.WorkDir = t.TempDir()
+	a, err := New(cfg, nil, nil)
+	if err != nil {
+		t.Fatal(err)
+	}
+	ctx, cancel := context.WithCancel(t.Context())
+	defer cancel()
+	go a.provisionLoop(ctx)
+
+	// An engine this agent cannot install: no network is needed to learn so,
+	// and the answer has to reach the control plane rather than a log.
+	a.setProvision(&llmgwv1.Provision{Engines: []*llmgwv1.EngineWant{{Engine: "vllm", Version: "0.29.0"}}})
+
+	deadline := time.Now().Add(3 * time.Second)
+	for time.Now().Before(deadline) {
+		hb := a.heartbeat().GetHeartbeat()
+		if hb.GetProvisionChanged() {
+			p := hb.GetProvision()
+			if len(p) != 1 || p[0].GetState() != "unsupported" {
+				t.Fatalf("reported %v", p)
+			}
+			a.mu.Lock()
+			_, reported := a.engineReportLocked()["vllm"]
+			a.mu.Unlock()
+			if reported {
+				t.Error("an engine that was not installed is reported as present")
+			}
+			return
+		}
+		time.Sleep(20 * time.Millisecond)
+	}
+	t.Fatal("the installation's outcome never reached a heartbeat")
+}

internal/cli/ctl.go

@@ -116,6 +116,7 @@ func newCtlCmd() *cobra.Command {
 
 	cmd.AddCommand(
 		newNodesCmd(client, &asJSON),
+		newLogsCmd(client, &asJSON),
 		newModelsCmd(client, &asJSON),
 		newReplicasCmd(client, &asJSON),
 		newExplainCmd(client, &asJSON),
@@ -189,13 +190,27 @@ func newNodesCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 			_ = w.Flush()
 
 			// Unhealthy GPUs are usually the answer when a node looks idle but
-			// nothing schedules onto it.
+			// nothing schedules onto it. An engine still installing, or one
+			// that failed to, is the other usual answer.
+			said := false
 			for _, n := range nodes {
 				for _, p := range n.Unhealthy {
 					fmt.Printf("\n%s: %s (kept out of placement)", n.ID, p)
+					said = true
+				}
+				for _, p := range n.Provision {
+					if p.State == "ready" {
+						continue
+					}
+					fmt.Printf("\n%s: %s %s %s: %s", n.ID, p.Engine, p.Version, p.State, p.Detail)
+					said = true
+				}
+				for _, w := range n.Warnings {
+					fmt.Printf("\n%s: %s", n.ID, w)
+					said = true
 				}
 			}
-			if hasUnhealthy(nodes) {
+			if said {
 				fmt.Println()
 			}
 			return nil
@@ -478,12 +493,3 @@ func dash(s string) string {
 	}
 	return s
 }
-
-func hasUnhealthy(nodes []adminapi.NodeView) bool {
-	for _, n := range nodes {
-		if len(n.Unhealthy) > 0 {
-			return true
-		}
-	}
-	return false
-}

internal/cli/ctl_logs.go

@@ -0,0 +1,58 @@
+package cli
+
+import (
+	"encoding/json"
+	"fmt"
+	"net/url"
+	"strconv"
+	"strings"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+)
+
+func newLogsCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
+	var (
+		source string
+		lines  int
+	)
+	c := &cobra.Command{
+		Use:   "logs NODE",
+		Short: "Show the recent lines of a node's agent log, or of an engine on it",
+		Long: "Read through the agent's own stream, so it needs no access to the node:\n" +
+			"no ssh, no jump host, no root for the journal. The agent keeps its last\n" +
+			"2000 lines and each engine's last few hundred.\n\n" +
+			"  llmgw ctl logs llmgw-node06                  the agent\n" +
+			"  llmgw ctl logs llmgw-node06 --source SHARD   an engine's output",
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			q := url.Values{}
+			q.Set("source", source)
+			q.Set("lines", strconv.Itoa(lines))
+			var v adminapi.LogsView
+			err := client().get(cmd.Context(),
+				"/api/v1/nodes/"+url.PathEscape(args[0])+"/logs?"+q.Encode(), &v)
+			if *asJSON {
+				return json.NewEncoder(cmd.OutOrStdout()).Encode(v)
+			}
+			if err != nil {
+				return err
+			}
+			for _, l := range v.Lines {
+				fmt.Fprintln(cmd.OutOrStdout(), l)
+			}
+			if v.Error != "" {
+				return fmt.Errorf("%s", v.Error)
+			}
+			// Where else to look, since the source list is the one thing a
+			// person cannot guess.
+			fmt.Fprintf(cmd.ErrOrStderr(), "\n-- %s on %s; other sources: %s\n",
+				v.Source, v.Node, strings.Join(v.Sources, ", "))
+			return nil
+		},
+	}
+	c.Flags().StringVar(&source, "source", "agent", `"agent", or a shard id for that engine's output`)
+	c.Flags().IntVarP(&lines, "lines", "n", 200, "how many of the most recent lines")
+	return c
+}

internal/cli/keys.go

@@ -6,6 +6,7 @@ import (
 	"encoding/base32"
 	"errors"
 	"fmt"
+	"math"
 	"strings"
 	"text/tabwriter"
 	"time"
@@ -107,7 +108,11 @@ func newKeyCreateCmd(open opener) *cobra.Command {
 			}
 			defer s.Close()
 
-			secret, k, err := store.NewKey(randomID("key"), args[0], project)
+			p, err := resolveProject(cmd.Context(), s, project)
+			if err != nil {
+				return err
+			}
+			secret, k, err := store.NewKey(randomID("key"), args[0], p.ID)
 			if err != nil {
 				return err
 			}
@@ -117,8 +122,7 @@ func newKeyCreateCmd(open opener) *cobra.Command {
 				k.ExpiresAt = k.CreatedAt.Add(expires)
 			}
 			if err := s.CreateKey(cmd.Context(), k); err != nil {
-				return fmt.Errorf("creating the key (does project %q exist? run `llmgw keys init`): %w",
-					project, err)
+				return fmt.Errorf("creating the key in %s: %w", p.Path, err)
 			}
 
 			// The one output that matters goes to stdout and everything else to
@@ -141,13 +145,13 @@ func newKeyCreateCmd(open opener) *cobra.Command {
 				link := strings.TrimRight(baseURL, "/") + "/claim/" + token
 				fmt.Fprintln(out, link)
 				cmd.PrintErrf("\nid       %s\nproject  %s\nexpires  %s\n",
-					k.ID, k.ProjectID, time.Now().Add(orDefault(claimTTL, store.ClaimTTL)).Format(time.RFC3339))
+					k.ID, p.Path, time.Now().Add(orDefault(claimTTL, store.ClaimTTL)).Format(time.RFC3339))
 				cmd.PrintErrln("\nSend the link. It works once, and the key is shown only to whoever opens it.")
 				return nil
 			}
 
 			fmt.Fprintln(out, secret)
-			cmd.PrintErrf("\nid       %s\nproject  %s\n", k.ID, k.ProjectID)
+			cmd.PrintErrf("\nid       %s\nproject  %s\n", k.ID, p.Path)
 			if !k.ExpiresAt.IsZero() {
 				cmd.PrintErrf("expires  %s\n", k.ExpiresAt.Format(time.RFC3339))
 			}
@@ -155,7 +159,8 @@ func newKeyCreateCmd(open opener) *cobra.Command {
 			return nil
 		},
 	}
-	cmd.Flags().StringVar(&project, "project", "default", "project the key belongs to")
+	cmd.Flags().StringVar(&project, "project", "",
+		"project the key belongs to: id, path (Acme/platform/gateway) or unique name")
 	cmd.Flags().StringVar(&band, "band", "", "admission band; empty inherits")
 	cmd.Flags().StringVar(&profile, "profile", "", "routing profile; empty inherits")
 	cmd.Flags().DurationVar(&expires, "expires-in", 0, "lifetime; 0 never expires")
@@ -174,9 +179,10 @@ func orDefault(d, fallback time.Duration) time.Duration {
 
 func newKeyListCmd(open opener) *cobra.Command {
 	var project string
+	var all bool
 	cmd := &cobra.Command{
 		Use:   "list",
-		Short: "List a project's keys",
+		Short: "List a project's keys, or with --all every key and whose it is",
 		Args:  cobra.NoArgs,
 		RunE: func(cmd *cobra.Command, _ []string) error {
 			s, err := open(cmd.Context())
@@ -185,12 +191,31 @@ func newKeyListCmd(open opener) *cobra.Command {
 			}
 			defer s.Close()
 
-			keys, err := s.Keys(cmd.Context(), project)
+			if all {
+				rows, err := s.AllKeys(cmd.Context())
+				if err != nil {
+					return err
+				}
+				w := tabwriter.NewWriter(cmd.OutOrStdout(), 0, 0, 2, ' ', 0)
+				fmt.Fprintln(w, "ID\tNAME\tKEY\tPROJECT\tOWNER\tSTATE\tLAST USED")
+				now := time.Now()
+				for _, r := range rows {
+					fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\t%s\t%s\n", r.ID, r.Name, r.Prefix,
+						r.ProjectPath, dash(r.OwnerEmail), keyState(r.Key, now), ago(r.LastUsedAt, now))
+				}
+				return w.Flush()
+			}
+
+			p, err := resolveProject(cmd.Context(), s, project)
+			if err != nil {
+				return err
+			}
+			keys, err := s.Keys(cmd.Context(), p.ID)
 			if err != nil {
 				return err
 			}
 			if len(keys) == 0 {
-				fmt.Fprintf(cmd.OutOrStdout(), "no keys in project %q\n", project)
+				fmt.Fprintf(cmd.OutOrStdout(), "no keys in %s\n", p.Path)
 				return nil
 			}
 
@@ -204,7 +229,8 @@ func newKeyListCmd(open opener) *cobra.Command {
 			return w.Flush()
 		},
 	}
-	cmd.Flags().StringVar(&project, "project", "default", "project to list")
+	cmd.Flags().StringVar(&project, "project", "", "project to list: id, path or unique name")
+	cmd.Flags().BoolVar(&all, "all", false, "every key in every project, with its owner")
 	return cmd
 }
 
@@ -235,7 +261,7 @@ func newKeyRevokeCmd(open opener) *cobra.Command {
 func newLimitsCmd(open opener) *cobra.Command {
 	var scope, id, period string
 	var rpm, tpm int
-	var budget int64
+	var budget float64
 	var models []string
 
 	cmd := &cobra.Command{
@@ -252,7 +278,34 @@ func newLimitsCmd(open opener) *cobra.Command {
 			}
 			defer s.Close()
 
-			l := store.Limits{Scope: store.Scope(scope), ScopeID: id}
+			scopeID := id
+			switch store.Scope(scope) {
+			case store.ScopeOrg:
+				o, err := resolveOrg(cmd.Context(), s, id)
+				if err != nil {
+					return err
+				}
+				scopeID = o.ID
+			case store.ScopeTeam:
+				t, err := resolveTeam(cmd.Context(), s, id)
+				if err != nil {
+					return err
+				}
+				scopeID = t.ID
+			case store.ScopeProject:
+				p, err := resolveProject(cmd.Context(), s, id)
+				if err != nil {
+					return err
+				}
+				scopeID = p.ID
+			case store.ScopeKey:
+				if id == "" {
+					return errors.New("--scope key needs --id, the key's id")
+				}
+			default:
+				return fmt.Errorf("--scope is org, team, project or key, not %q", scope)
+			}
+			l := store.Limits{Scope: store.Scope(scope), ScopeID: scopeID}
 			if cmd.Flags().Changed("rpm") {
 				l.RPM = &rpm
 			}
@@ -260,7 +313,10 @@ func newLimitsCmd(open opener) *cobra.Command {
 				l.TPM = &tpm
 			}
 			if cmd.Flags().Changed("budget") {
-				l.Budget = &budget
+				// Whole units, as the quotas page takes them; stored as
+				// micro-units so repeated addition does not drift.
+				micros := int64(math.Round(budget * 1e6))
+				l.Budget = &micros
 				l.BudgetPeriod = store.Period(period)
 			}
 			if cmd.Flags().Changed("models") {
@@ -274,15 +330,15 @@ func newLimitsCmd(open opener) *cobra.Command {
 			if err := s.PutLimits(cmd.Context(), l); err != nil {
 				return err
 			}
-			fmt.Fprintf(cmd.OutOrStdout(), "limits set on %s %s\n", scope, id)
+			fmt.Fprintf(cmd.OutOrStdout(), "limits set on %s %s\n", scope, scopeID)
 			return nil
 		},
 	}
 	cmd.Flags().StringVar(&scope, "scope", "project", "org | team | project | key")
-	cmd.Flags().StringVar(&id, "id", "default", "id at that scope")
+	cmd.Flags().StringVar(&id, "id", "", "which one: id, path or unique name (a key's id for --scope key)")
 	cmd.Flags().IntVar(&rpm, "rpm", 0, "requests per minute")
 	cmd.Flags().IntVar(&tpm, "tpm", 0, "tokens per minute")
-	cmd.Flags().Int64Var(&budget, "budget", 0, "budget per period, in micro-units")
+	cmd.Flags().Float64Var(&budget, "budget", 0, "budget per period, in whole units as the quotas page shows them")
 	cmd.Flags().StringVar(&period, "period", "month", "day | month")
 	cmd.Flags().StringSliceVar(&models, "models", nil, "allowed models; empty allows none")
 	return cmd

internal/cli/memory.go

@@ -42,7 +42,12 @@ func newMemoryCmd(open opener) *cobra.Command {
 			if off {
 				keep = 0
 			}
-			if err := s.SetMemoryRetention(cmd.Context(), project, keep); err != nil {
+			p, err := resolveProject(cmd.Context(), s, project)
+			if err != nil {
+				return err
+			}
+			project = p.Path
+			if err := s.SetMemoryRetention(cmd.Context(), p.ID, keep); err != nil {
 				return err
 			}
 			if keep == 0 {
@@ -55,7 +60,7 @@ func newMemoryCmd(open opener) *cobra.Command {
 			return nil
 		},
 	}
-	cmd.Flags().StringVar(&project, "project", "default", "project id")
+	cmd.Flags().StringVar(&project, "project", "", "project: id, path or unique name")
 	cmd.Flags().IntVar(&hours, "retain-hours", 0, "how long a note is kept after it was last written")
 	cmd.Flags().BoolVar(&off, "off", false, "stop keeping notes for this project")
 	return cmd

internal/cli/org.go

@@ -0,0 +1,334 @@
+package cli
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"io"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+func newOrgCmd() *cobra.Command {
+	var sf storeFlags
+	cmd := &cobra.Command{
+		Use:   "org",
+		Short: "Organizations, teams and projects: see, add, rename, move, remove",
+		Long: "The tenancy tree an account lives in. A person is in a project, a project\n" +
+			"in a team, a team in an organization, and limits can be set on any of them.\n\n" +
+			"Anything that takes one of them accepts its id, its path\n" +
+			"(\"Acme/platform/gateway\") or its name when that name is unique.",
+	}
+	sf.bind(cmd)
+	open := func(cmd *cobra.Command) (*store.DB, error) { return sf.open(cmd) }
+	cmd.AddCommand(newOrgTreeCmd(open), newOrgAddCmd(open), newOrgRenameCmd(open),
+		newOrgMoveCmd(open), newOrgRemoveCmd(open))
+	return cmd
+}
+
+type dbOpener func(*cobra.Command) (*store.DB, error)
+
+func newOrgTreeCmd(open dbOpener) *cobra.Command {
+	var people bool
+	cmd := &cobra.Command{
+		Use:   "tree",
+		Short: "Show the whole tree, with how many accounts and keys each project has",
+		Args:  cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			s, err := open(cmd)
+			if err != nil {
+				return err
+			}
+			defer s.Close()
+			tree, err := s.Tree(cmd.Context())
+			if err != nil {
+				return err
+			}
+			var byProject map[string][]store.User
+			if people {
+				users, err := s.Users(cmd.Context(), "")
+				if err != nil {
+					return err
+				}
+				byProject = map[string][]store.User{}
+				for _, u := range users {
+					byProject[u.ProjectID] = append(byProject[u.ProjectID], u)
+				}
+			}
+			printTree(cmd.OutOrStdout(), tree, byProject)
+			if people && len(byProject[""]) > 0 {
+				fmt.Fprintf(cmd.OutOrStdout(), "\nwaiting for approval\n")
+				for _, u := range byProject[""] {
+					fmt.Fprintf(cmd.OutOrStdout(), "  %s  %s  %s\n", u.Email, u.State, u.ID)
+				}
+			}
+			return nil
+		},
+	}
+	cmd.Flags().BoolVar(&people, "people", false, "list the accounts in each project")
+	return cmd
+}
+
+func printTree(w io.Writer, tree []store.TreeOrg, people map[string][]store.User) {
+	if len(tree) == 0 {
+		fmt.Fprintln(w, "no organizations yet; `llmgw org add org <name>` starts one")
+		return
+	}
+	for _, o := range tree {
+		fmt.Fprintf(w, "%s  (%s)\n", o.Name, o.ID)
+		if len(o.Teams) == 0 {
+			fmt.Fprintln(w, "    no teams")
+		}
+		for _, t := range o.Teams {
+			fmt.Fprintf(w, "  %s  (%s)\n", t.Name, t.ID)
+			if len(t.Projects) == 0 {
+				fmt.Fprintln(w, "      no projects")
+			}
+			for _, p := range t.Projects {
+				fmt.Fprintf(w, "    %s  (%s)  %s, %s\n", p.Name, p.ID,
+					countOf(p.Users, "account"), keysOf(p))
+				for _, u := range people[p.ID] {
+					admin := ""
+					if u.IsAdmin {
+						admin = "  admin"
+					}
+					fmt.Fprintf(w, "      %s  %s%s\n", u.Email, u.State, admin)
+				}
+			}
+		}
+	}
+}
+
+func keysOf(p store.TreeProject) string {
+	s := countOf(p.ActiveKeys, "key")
+	if revoked := p.Keys - p.ActiveKeys; revoked > 0 {
+		s += fmt.Sprintf(" (+%d revoked)", revoked)
+	}
+	return s
+}
+
+func countOf(n int, noun string) string {
+	if n == 1 {
+		return "1 " + noun
+	}
+	return fmt.Sprintf("%d %ss", n, noun)
+}
+
+func levelArg(arg string) (string, error) {
+	switch arg {
+	case "org", "organization":
+		return "org", nil
+	case "team", "project":
+		return arg, nil
+	}
+	return "", fmt.Errorf("%q is not org, team or project", arg)
+}
+
+func newOrgAddCmd(open dbOpener) *cobra.Command {
+	var in string
+	cmd := &cobra.Command{
+		Use:   "add <org|team|project> <name>",
+		Short: "Add an organization, a team to one, or a project to a team",
+		Example: "  llmgw org add org Acme\n" +
+			"  llmgw org add team platform --in Acme\n" +
+			"  llmgw org add project gateway --in Acme/platform",
+		Args: cobra.ExactArgs(2),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			level, err := levelArg(args[0])
+			if err != nil {
+				return err
+			}
+			s, err := open(cmd)
+			if err != nil {
+				return err
+			}
+			defer s.Close()
+			ctx := cmd.Context()
+			var id, where string
+			switch level {
+			case "org":
+				id, err = s.CreateOrg(ctx, args[1])
+			case "team":
+				var o store.Org
+				if o, err = resolveOrg(ctx, s, in); err == nil {
+					id, err = s.CreateTeam(ctx, o.ID, args[1])
+					where = " in " + o.Name
+				}
+			case "project":
+				var t store.TeamPath
+				if t, err = resolveTeam(ctx, s, in); err == nil {
+					id, err = s.CreateProject(ctx, t.ID, args[1])
+					where = " in " + t.OrgName + " / " + t.Name
+				}
+			}
+			if errors.Is(err, store.ErrDuplicateName) {
+				return fmt.Errorf("there is already a %s named %q there", level, args[1])
+			}
+			if err != nil {
+				return err
+			}
+			fmt.Fprintf(cmd.OutOrStdout(), "%s %s%s  (%s)\n", level, args[1], where, id)
+			return nil
+		},
+	}
+	cmd.Flags().StringVar(&in, "in", "", "the organization a team goes in, or the team a project goes in")
+	return cmd
+}
+
+func newOrgRenameCmd(open dbOpener) *cobra.Command {
+	return &cobra.Command{
+		Use:     "rename <org|team|project> <which> <new name>",
+		Short:   "Rename an organization, team or project; ids and everything under it stay",
+		Example: "  llmgw org rename project Acme/platform/gateway api-gateway",
+		Args:    cobra.ExactArgs(3),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			level, err := levelArg(args[0])
+			if err != nil {
+				return err
+			}
+			s, err := open(cmd)
+			if err != nil {
+				return err
+			}
+			defer s.Close()
+			id, err := resolveLevel(cmd.Context(), s, level, args[1])
+			if err != nil {
+				return err
+			}
+			switch level {
+			case "org":
+				err = s.RenameOrg(cmd.Context(), id, args[2])
+			case "team":
+				err = s.RenameTeam(cmd.Context(), id, args[2])
+			case "project":
+				err = s.RenameProject(cmd.Context(), id, args[2])
+			}
+			if errors.Is(err, store.ErrDuplicateName) {
+				return fmt.Errorf("there is already a %s named %q beside it", level, args[2])
+			}
+			if err != nil {
+				return err
+			}
+			fmt.Fprintf(cmd.OutOrStdout(), "%s %s is now %q\n", level, id, args[2])
+			return nil
+		},
+	}
+}
+
+func newOrgMoveCmd(open dbOpener) *cobra.Command {
+	var to string
+	cmd := &cobra.Command{
+		Use:   "move <team|project> <which> --to <parent>",
+		Short: "Put a team in another organization, or a project in another team",
+		Long: "Everything under it moves too: projects, accounts and keys. From then on it\n" +
+			"inherits the new parent's limits. Usage already recorded is not rewritten.",
+		Example: "  llmgw org move project Acme/platform/gateway --to Scania/ml\n" +
+			"  llmgw org move team platform --to Scania",
+		Args: cobra.ExactArgs(2),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			s, err := open(cmd)
+			if err != nil {
+				return err
+			}
+			defer s.Close()
+			ctx := cmd.Context()
+			switch args[0] {
+			case "team":
+				t, err := resolveTeam(ctx, s, args[1])
+				if err != nil {
+					return err
+				}
+				o, err := resolveOrg(ctx, s, to)
+				if err != nil {
+					return err
+				}
+				if err := s.MoveTeam(ctx, t.ID, o.ID); err != nil {
+					return dupAs(err, "team", t.Name)
+				}
+				fmt.Fprintf(cmd.OutOrStdout(), "team %s is now in %s\n", t.Name, o.Name)
+			case "project":
+				p, err := resolveProject(ctx, s, args[1])
+				if err != nil {
+					return err
+				}
+				t, err := resolveTeam(ctx, s, to)
+				if err != nil {
+					return err
+				}
+				if err := s.MoveProject(ctx, p.ID, t.ID); err != nil {
+					return dupAs(err, "project", p.Path)
+				}
+				fmt.Fprintf(cmd.OutOrStdout(), "%s is now in %s / %s\n", p.Path, t.OrgName, t.Name)
+			default:
+				return fmt.Errorf("only a team or a project moves; to move an account, `llmgw users move`")
+			}
+			return nil
+		},
+	}
+	cmd.Flags().StringVar(&to, "to", "", "the organization or team it goes to")
+	_ = cmd.MarkFlagRequired("to")
+	return cmd
+}
+
+func dupAs(err error, level, name string) error {
+	if errors.Is(err, store.ErrDuplicateName) {
+		return fmt.Errorf("the destination already has a %s named like %s", level, name)
+	}
+	return err
+}
+
+func newOrgRemoveCmd(open dbOpener) *cobra.Command {
+	return &cobra.Command{
+		Use:   "remove <org|team|project> <which>",
+		Short: "Remove an organization, team or project that nothing is in",
+		Long: "Only something empty is removed: an organization with no teams, a team with\n" +
+			"no projects, a project with no accounts and no keys. A project that ever had\n" +
+			"a key keeps it, revoked, because usage history refers to it; rename such a\n" +
+			"project instead. Its limits go with it.",
+		Args: cobra.ExactArgs(2),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			level, err := levelArg(args[0])
+			if err != nil {
+				return err
+			}
+			s, err := open(cmd)
+			if err != nil {
+				return err
+			}
+			defer s.Close()
+			id, err := resolveLevel(cmd.Context(), s, level, args[1])
+			if err != nil {
+				return err
+			}
+			switch level {
+			case "org":
+				err = s.DeleteOrg(cmd.Context(), id)
+			case "team":
+				err = s.DeleteTeam(cmd.Context(), id)
+			case "project":
+				err = s.DeleteProject(cmd.Context(), id)
+			}
+			if err != nil {
+				return fmt.Errorf("%s %s was not removed: %w", level, args[1], err)
+			}
+			fmt.Fprintf(cmd.OutOrStdout(), "%s %s removed\n", level, id)
+			return nil
+		},
+	}
+}
+
+func resolveLevel(ctx context.Context, s *store.DB, level, arg string) (string, error) {
+	switch level {
+	case "org":
+		o, err := resolveOrg(ctx, s, arg)
+		return o.ID, err
+	case "team":
+		t, err := resolveTeam(ctx, s, arg)
+		return t.ID, err
+	default:
+		p, err := resolveProject(ctx, s, arg)
+		return p.ID, err
+	}
+}

internal/cli/org_test.go

@@ -0,0 +1,124 @@
+package cli
+
+import (
+	"bytes"
+	"context"
+	"path/filepath"
+	"strings"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// run executes the CLI against one store file and returns stdout.
+func run(t *testing.T, db string, args ...string) (string, error) {
+	t.Helper()
+	cmd := Root()
+	var out, errOut bytes.Buffer
+	cmd.SetOut(&out)
+	cmd.SetErr(&errOut)
+	// --config pointed nowhere, so a server.yaml on the test machine is never read.
+	cmd.SetArgs(append(args, "--store", db, "--config", ""))
+	err := cmd.ExecuteContext(context.Background())
+	return out.String(), err
+}
+
+func mustRun(t *testing.T, db string, args ...string) string {
+	t.Helper()
+	out, err := run(t, db, args...)
+	if err != nil {
+		t.Fatalf("%v: %v", args, err)
+	}
+	return out
+}
+
+func TestTheTreeIsManagedByNameNotId(t *testing.T) {
+	db := filepath.Join(t.TempDir(), "s.db")
+	mustRun(t, db, "org", "add", "org", "Acme")
+	mustRun(t, db, "org", "add", "org", "Scania")
+	mustRun(t, db, "org", "add", "team", "platform", "--in", "Acme")
+	mustRun(t, db, "org", "add", "team", "platform", "--in", "Scania")
+
+	// "platform" is in both, so a project needs the path.
+	_, err := run(t, db, "org", "add", "project", "gateway", "--in", "platform")
+	if err == nil || !strings.Contains(err.Error(), "more than one team") {
+		t.Fatalf("an ambiguous name was accepted: %v", err)
+	}
+	mustRun(t, db, "org", "add", "project", "gateway", "--in", "acme/Platform")
+	mustRun(t, db, "org", "add", "project", "research", "--in", "Scania / platform")
+
+	mustRun(t, db, "users", "create", "ayse@example.com", "--password", "correct horse battery", "--project", "Acme/platform/gateway")
+	mustRun(t, db, "keys", "create", "laptop", "--project", "gateway")
+
+	out := mustRun(t, db, "org", "tree", "--people")
+	for _, want := range []string{"Acme", "gateway", "1 account, 1 key", "ayse@example.com"} {
+		if !strings.Contains(out, want) {
+			t.Errorf("tree lacks %q:\n%s", want, out)
+		}
+	}
+
+	out = mustRun(t, db, "users", "move", "ayse@example.com", "--project", "research")
+	if !strings.Contains(out, "Scania / platform / research") {
+		t.Errorf("move said: %s", out)
+	}
+
+	mustRun(t, db, "org", "rename", "project", "research", "lab")
+	mustRun(t, db, "org", "move", "project", "Acme/platform/gateway", "--to", "Scania/platform")
+
+	// gateway still has the operator's key, so it stays.
+	if _, err := run(t, db, "org", "remove", "project", "gateway"); err == nil {
+		t.Error("a project with a key was removed")
+	}
+	mustRun(t, db, "org", "remove", "team", "Acme/platform")
+	mustRun(t, db, "org", "remove", "org", "Acme")
+
+	s, err := store.Open(context.Background(), db)
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer s.Close()
+	paths, _ := s.AllProjects(context.Background())
+	var got []string
+	for _, p := range paths {
+		got = append(got, p.Path)
+	}
+	if strings.Join(got, "|") != "Scania / platform / gateway|Scania / platform / lab" {
+		t.Errorf("projects = %v", got)
+	}
+}
+
+func TestOneProjectNeedsNoFlag(t *testing.T) {
+	db := filepath.Join(t.TempDir(), "s.db")
+	mustRun(t, db, "org", "add", "org", "Acme")
+	mustRun(t, db, "org", "add", "team", "platform")
+	mustRun(t, db, "org", "add", "project", "gateway")
+	mustRun(t, db, "keys", "create", "laptop")
+	if out := mustRun(t, db, "keys", "list"); !strings.Contains(out, "laptop") {
+		t.Errorf("list: %s", out)
+	}
+	mustRun(t, db, "org", "add", "project", "search", "--in", "platform")
+	if _, err := run(t, db, "keys", "create", "other"); err == nil ||
+		!strings.Contains(err.Error(), "which project?") {
+		t.Errorf("two projects and no flag: %v", err)
+	}
+}
+
+func TestKeyBudgetsAreWholeUnits(t *testing.T) {
+	db := filepath.Join(t.TempDir(), "s.db")
+	mustRun(t, db, "org", "add", "org", "Acme")
+	mustRun(t, db, "org", "add", "team", "platform")
+	mustRun(t, db, "org", "add", "project", "gateway")
+	mustRun(t, db, "keys", "limits", "--scope", "project", "--id", "gateway", "--budget", "12.5")
+
+	s, err := store.Open(context.Background(), db)
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer s.Close()
+	all, _ := s.AllLimits(context.Background())
+	for _, l := range all {
+		if l.Budget == nil || *l.Budget != 12_500_000 {
+			t.Errorf("budget stored as %v micro-units", l.Budget)
+		}
+	}
+}

internal/cli/resolve.go

@@ -0,0 +1,131 @@
+package cli
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"strings"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// The tree can be built from the web, which mints ids like proj-3kq7..., or
+// from `keys init`, which makes slugs. A command that wanted the id made the
+// operator copy one off a page, and every --project defaulted to "default",
+// which a tree built from the web does not have. These take what a person
+// would type instead: an id, a path, or a name that is unique.
+
+// normPath lowercases a path and spaces its separators the way the store joins
+// them, so "acme/platform/gateway" matches "Acme / platform / gateway".
+func normPath(p string) string {
+	parts := strings.Split(p, "/")
+	for i := range parts {
+		parts[i] = strings.ToLower(strings.TrimSpace(parts[i]))
+	}
+	return strings.Join(parts, " / ")
+}
+
+type candidate struct{ id, path string }
+
+// pick finds the one candidate an argument names.
+func pick(kind, arg string, all []candidate) (candidate, error) {
+	if arg == "" {
+		if len(all) == 1 {
+			return all[0], nil
+		}
+		if len(all) == 0 {
+			return candidate{}, fmt.Errorf("there is no %s yet; create one with `llmgw org add %s`", kind, kind)
+		}
+		return candidate{}, fmt.Errorf("which %s? give an id, a path or a name:\n%s", kind, listCandidates(all))
+	}
+	for _, c := range all {
+		if c.id == arg {
+			return c, nil
+		}
+	}
+	want := normPath(arg)
+	var matches []candidate
+	for _, c := range all {
+		full := normPath(c.path)
+		segs := strings.Split(full, " / ")
+		if full == want || segs[len(segs)-1] == want {
+			matches = append(matches, c)
+		}
+	}
+	switch len(matches) {
+	case 1:
+		return matches[0], nil
+	case 0:
+		return candidate{}, fmt.Errorf("no %s %q; there is:\n%s", kind, arg, listCandidates(all))
+	default:
+		return candidate{}, fmt.Errorf("%q names more than one %s; give its path:\n%s",
+			arg, kind, listCandidates(matches))
+	}
+}
+
+func listCandidates(cs []candidate) string {
+	var b strings.Builder
+	for _, c := range cs {
+		fmt.Fprintf(&b, "  %-40s %s\n", c.path, c.id)
+	}
+	return strings.TrimRight(b.String(), "\n")
+}
+
+func resolveProject(ctx context.Context, s *store.DB, arg string) (store.ProjectPath, error) {
+	all, err := s.AllProjects(ctx)
+	if err != nil {
+		return store.ProjectPath{}, err
+	}
+	cs := make([]candidate, len(all))
+	for i, p := range all {
+		cs[i] = candidate{p.ID, p.Path}
+	}
+	c, err := pick("project", arg, cs)
+	return store.ProjectPath{ID: c.id, Path: c.path}, err
+}
+
+func resolveTeam(ctx context.Context, s *store.DB, arg string) (store.TeamPath, error) {
+	all, err := s.AllTeams(ctx)
+	if err != nil {
+		return store.TeamPath{}, err
+	}
+	cs := make([]candidate, len(all))
+	for i, t := range all {
+		cs[i] = candidate{t.ID, t.OrgName + " / " + t.Name}
+	}
+	c, err := pick("team", arg, cs)
+	if err != nil {
+		return store.TeamPath{}, err
+	}
+	for _, t := range all {
+		if t.ID == c.id {
+			return t, nil
+		}
+	}
+	return store.TeamPath{}, store.ErrNotFound
+}
+
+func resolveOrg(ctx context.Context, s *store.DB, arg string) (store.Org, error) {
+	all, err := s.AllOrgs(ctx)
+	if err != nil {
+		return store.Org{}, err
+	}
+	cs := make([]candidate, len(all))
+	for i, o := range all {
+		cs[i] = candidate{o.ID, o.Name}
+	}
+	c, err := pick("org", arg, cs)
+	return store.Org{ID: c.id, Name: c.path}, err
+}
+
+// resolveUser takes an account's id or its email.
+func resolveUser(ctx context.Context, s *store.DB, arg string) (store.User, error) {
+	if u, err := s.UserByID(ctx, arg); err == nil {
+		return u, nil
+	}
+	u, err := s.UserByEmail(ctx, arg)
+	if errors.Is(err, store.ErrNotFound) {
+		return u, fmt.Errorf("no account with id or email %q; `llmgw users list` shows them", arg)
+	}
+	return u, err
+}

internal/cli/root.go

@@ -42,6 +42,7 @@ func Root() *cobra.Command {
 		newCtlCmd(),
 		newTenancyCmd(),
 		newUsersCmd(),
+		newOrgCmd(),
 		newMirrorCmd(),
 		newSimEngineCmd(),
 	)

internal/cli/server.go

@@ -203,6 +203,7 @@ func newServerCmd() *cobra.Command {
 					api, usage, version.Version, 5*time.Second, true,
 					web.NavExtras{History: history, Window: cfg.Monitoring.Window}, log))
 				adminMux.Handle("GET /_static/", web.Static())
+				adminMux.HandleFunc("GET /nodes/{id}/logs", web.NodeLogsPage(api, true, version.Version))
 
 				// The same page on the client-facing port at /admin, behind a
 				// session and an admin flag. Handing the router the data rather
@@ -349,6 +350,7 @@ func newServerCmd() *cobra.Command {
 	f.StringVar(&metricsListen, "metrics-listen", "", "serve metrics only on this address, safe to expose")
 	f.StringArrayVar(&endpoints, "endpoint", nil, "static endpoint: id=URL=model[,model] (repeatable)")
 
+	cmd.AddCommand(newServerInitCmd())
 	return cmd
 }
 
@@ -401,6 +403,8 @@ func startControlPlane(ctx context.Context, cfg config.Server, cat *catalog.Live
 		}
 	}
 	svc := agentsvc.New(ctl.Inventory(), ctl.Reconciler(), requireIdentity, log)
+	svc.SetEngines(cfg.WantedEngines())
+	log.Info("nodes install these engines", "engines", cfg.WantedEngines())
 	// A standby must refuse agents as well as refusing to place. Commands
 	// travel down the stream the agent opened, so a standby holding one is a
 	// node the leader cannot command — and the leader's placements would fail
@@ -464,6 +468,21 @@ func (r *dispatcherRef) RemoveWeights(ctx context.Context, nodeID string, paths
 	return rm.RemoveWeights(ctx, nodeID, paths)
 }
 
+// Logs forwards a node log request, the same way RemoveWeights does: the
+// controller finds it by type assertion, so a forwarder without it would make
+// the capability silently absent at runtime.
+func (r *dispatcherRef) Logs(ctx context.Context, nodeID, source string, lines int) ([]string, []string, error) {
+	d, err := r.get()
+	if err != nil {
+		return nil, nil, err
+	}
+	lr, ok := d.(control.LogReader)
+	if !ok {
+		return nil, nil, errors.New("this dispatcher cannot read node logs")
+	}
+	return lr.Logs(ctx, nodeID, source, lines)
+}
+
 func (r *dispatcherRef) StartShard(ctx context.Context, cmd reconciler.ShardCommand) error {
 	d, err := r.get()
 	if err != nil {

internal/cli/serverinit.go

@@ -0,0 +1,623 @@
+package cli
+
+import (
+	"context"
+	"crypto/rand"
+	"encoding/base32"
+	"errors"
+	"fmt"
+	"io"
+	"net"
+	"net/http"
+	"os"
+	"os/exec"
+	"os/user"
+	"path/filepath"
+	"runtime"
+	"slices"
+	"strconv"
+	"strings"
+	"time"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/account"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
+)
+
+// serverUnit is deploy/systemd/llmgw-server.service. A test holds the two
+// together, because a machine set up by init and one set up from the docs
+// should not differ in a way nobody chose.
+const serverUnit = `[Unit]
+Description=llmgw control plane
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+ExecStart=/usr/local/bin/llmgw server --config /etc/llmgw/server.yaml
+User=llmgw
+Group=llmgw
+Restart=on-failure
+RestartSec=2s
+NoNewPrivileges=true
+ProtectSystem=strict
+ProtectHome=true
+PrivateTmp=true
+StateDirectory=llmgw
+
+[Install]
+WantedBy=multi-user.target
+`
+
+// serverInit is everything a control plane needs before it can start, done in
+// the order it has to be done in. Every step looks before it acts, so running
+// it again finishes an install that stopped halfway and changes nothing on
+// one that did not.
+type serverInit struct {
+	etcDir   string // /etc/llmgw
+	stateDir string // /var/lib/llmgw
+	pkiDir   string // where the authority lives
+	unitPath string // "" leaves systemd alone
+
+	// user owns the state and reads the key; "" means no user is created and
+	// nothing is chowned, which is what a test wants.
+	user  string
+	hosts []string
+	admin string
+	// postgres keeps the store in a PostgreSQL on this machine, installed if
+	// need be, moving whatever SQLite already holds.
+	postgres bool
+	// pgSetup prepares PostgreSQL and returns the store address; a seam for
+	// tests. Nil does it for real.
+	pgSetup func(context.Context) (string, error)
+	pg      pgServer
+
+	out io.Writer
+
+	uid, gid int
+	changed  bool
+}
+
+func newServerInitCmd() *cobra.Command {
+	in := serverInit{
+		etcDir:   "/etc/llmgw",
+		stateDir: "/var/lib/llmgw",
+		pkiDir:   defaultPKIDir,
+		unitPath: "/etc/systemd/system/llmgw-server.service",
+		user:     "llmgw",
+	}
+	cmd := &cobra.Command{
+		Use:   "init",
+		Short: "Set this machine up as a control plane and start it",
+		Long: "Creates the llmgw user and its directories, a certificate authority, the\n" +
+			"control plane's certificate for every address this machine has, a\n" +
+			"configuration, an administrator account, and the systemd unit — then\n" +
+			"starts it and waits until it answers.\n\n" +
+			"With --postgres the store is a PostgreSQL database on this machine:\n" +
+			"installed if there is none, with its own role and a generated password\n" +
+			"kept in /etc/llmgw/secrets. An existing SQLite store is copied into it,\n" +
+			"so accounts and keys survive, and server.yaml is pointed at it.\n\n" +
+			"Each step is skipped when its result already exists, so it is safe to run\n" +
+			"again: to finish an install that stopped, or with another --host to\n" +
+			"reissue the certificate. It never overwrites server.yaml or the authority.\n\n" +
+			"The authority's key stays in " + defaultPKIDir + " on this machine. Move it\n" +
+			"somewhere safer once the nodes are onboarded if this machine is exposed.",
+		Args: cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			if runtime.GOOS != "linux" {
+				return fmt.Errorf("init sets up a Linux machine; this is %s", runtime.GOOS)
+			}
+			if os.Geteuid() != 0 {
+				return errors.New("init creates a user, writes /etc and starts a service: run it as root")
+			}
+			in.out = cmd.OutOrStdout()
+			return in.run(cmd.Context())
+		},
+	}
+	f := cmd.Flags()
+	f.StringArrayVar(&in.hosts, "host", nil,
+		"an address agents dial besides the ones found on this machine; repeatable")
+	f.StringVar(&in.admin, "admin", "", "email of the first administrator; a password is generated and printed once")
+	f.StringVar(&in.pkiDir, "pki", in.pkiDir, "directory holding the certificate authority")
+	f.BoolVar(&in.postgres, "postgres", false,
+		"keep the store in PostgreSQL on this machine, installing it and moving an existing SQLite store into it")
+	return cmd
+}
+
+func (in *serverInit) run(ctx context.Context) error {
+	steps := []struct {
+		name string
+		fn   func(context.Context) error
+	}{
+		{"user", in.ensureUser},
+		{"directories", in.ensureDirs},
+		{"certificates", in.ensureCerts},
+		{"configuration", in.ensureConfig},
+		{"database", in.ensureDatabase},
+		{"store", in.ensureStore},
+	}
+	for _, s := range steps {
+		if err := s.fn(ctx); err != nil {
+			return fmt.Errorf("%s: %w", s.name, err)
+		}
+	}
+	if in.unitPath == "" {
+		return nil
+	}
+	if err := in.ensureService(ctx); err != nil {
+		return fmt.Errorf("service: %w", err)
+	}
+	in.nextSteps()
+	return nil
+}
+
+func (in *serverInit) done(format string, a ...any) {
+	fmt.Fprintf(in.out, "  ok    "+format+"\n", a...)
+	in.changed = true
+}
+
+func (in *serverInit) kept(format string, a ...any) {
+	fmt.Fprintf(in.out, "  kept  "+format+"\n", a...)
+}
+
+func (in *serverInit) ensureUser(context.Context) error {
+	if in.user == "" {
+		return nil
+	}
+	u, err := user.Lookup(in.user)
+	if err != nil {
+		var unknown user.UnknownUserError
+		if !errors.As(err, &unknown) {
+			return err
+		}
+		out, err := exec.Command("useradd", "--system", "--home", in.stateDir,
+			"--shell", "/usr/sbin/nologin", in.user).CombinedOutput()
+		if err != nil {
+			return fmt.Errorf("useradd: %v: %s", err, strings.TrimSpace(string(out)))
+		}
+		if u, err = user.Lookup(in.user); err != nil {
+			return err
+		}
+		in.done("system user %s", in.user)
+	} else {
+		in.kept("system user %s", in.user)
+	}
+	in.uid, _ = strconv.Atoi(u.Uid)
+	in.gid, _ = strconv.Atoi(u.Gid)
+	return nil
+}
+
+func (in *serverInit) ensureDirs(context.Context) error {
+	for _, d := range []struct {
+		path  string
+		mode  os.FileMode
+		owned bool
+	}{
+		{in.etcDir, 0o755, false},
+		{in.pkiDir, 0o700, false},
+		{in.stateDir, 0o750, true},
+	} {
+		if err := os.MkdirAll(d.path, d.mode); err != nil {
+			return err
+		}
+		if d.owned {
+			if err := in.chown(d.path, true); err != nil {
+				return err
+			}
+		}
+	}
+	return nil
+}
+
+// chown gives a path to the service user, or only its group when the user
+// should read and not write.
+func (in *serverInit) chown(path string, own bool) error {
+	if in.user == "" {
+		return nil
+	}
+	uid := 0
+	if own {
+		uid = in.uid
+	}
+	return os.Chown(path, uid, in.gid)
+}
+
+func (in *serverInit) ensureCerts(context.Context) error {
+	certPath := filepath.Join(in.etcDir, "server.pem")
+	_, keyErr := os.Stat(filepath.Join(in.pkiDir, "ca-key.pem"))
+	_, caErr := os.Stat(filepath.Join(in.etcDir, "ca.pem"))
+	_, certErr := os.Stat(certPath)
+	if keyErr != nil && caErr == nil && certErr == nil {
+		// Issued by an authority kept elsewhere, which is the arrangement for
+		// anyone who will not have the CA key on the control plane. Making a
+		// new authority here would replace their certificate with one no node
+		// trusts.
+		if len(in.hosts) > 0 {
+			return fmt.Errorf("%s was issued by an authority that is not in %s; "+
+				"reissue it there with the new --host", certPath, in.pkiDir)
+		}
+		in.kept("certificates in %s, issued elsewhere", in.etcDir)
+		return nil
+	}
+
+	var (
+		ca  *tlsconf.CA
+		err error
+	)
+	if keyErr == nil {
+		if ca, err = loadCA(in.pkiDir); err != nil {
+			return err
+		}
+		in.kept("certificate authority in %s", in.pkiDir)
+	} else {
+		if ca, err = tlsconf.NewCA("llmgw cluster CA"); err != nil {
+			return err
+		}
+		if err := ca.Write(in.pkiDir); err != nil {
+			return err
+		}
+		in.done("certificate authority in %s (ca-key.pem signs every node: back it up)", in.pkiDir)
+	}
+	if err := os.WriteFile(filepath.Join(in.etcDir, "ca.pem"), ca.PEM, 0o644); err != nil {
+		return err
+	}
+
+	want := in.certHosts()
+	if cert, err := readCert(certPath); err == nil {
+		have := strings.Split(sanNames(cert), ", ")
+		var missing []string
+		for _, h := range want {
+			if !slices.Contains(have, h) {
+				missing = append(missing, h)
+			}
+		}
+		// A certificate from this authority naming every address is left
+		// alone. One naming fewer is reissued with the union, so an address
+		// that worked yesterday is never dropped by adding one today.
+		if len(missing) == 0 && cert.CheckSignatureFrom(ca.Cert) == nil {
+			in.kept("server certificate for %s", strings.Join(have, ", "))
+			return nil
+		}
+		for _, h := range have {
+			if h != "" && !slices.Contains(want, h) {
+				want = append(want, h)
+			}
+		}
+	}
+
+	name, _ := os.Hostname()
+	if name == "" {
+		name = "llmgw-control"
+	}
+	leaf, err := ca.IssueServer(name, want)
+	if err != nil {
+		return err
+	}
+	if err := leaf.Write(in.etcDir, "server"); err != nil {
+		return err
+	}
+	// The service runs as llmgw and must read its key; nothing else should.
+	keyPath := filepath.Join(in.etcDir, "server-key.pem")
+	if err := in.chown(keyPath, false); err != nil {
+		return err
+	}
+	if in.user != "" {
+		if err := os.Chmod(keyPath, 0o640); err != nil {
+			return err
+		}
+	}
+	in.done("server certificate for %s", strings.Join(want, ", "))
+	return nil
+}
+
+// certHosts is every address an agent could plausibly dial: the hostname and
+// each address on this machine, plus whatever --host added. Asking the
+// operator to list them is how a certificate ends up naming the one address
+// nobody uses, and that is found out by the first node refusing it.
+func (in *serverInit) certHosts() []string {
+	var out []string
+	add := func(h string) {
+		if h != "" && !slices.Contains(out, h) {
+			out = append(out, h)
+		}
+	}
+	for _, h := range in.hosts {
+		add(h)
+	}
+	if name, err := os.Hostname(); err == nil {
+		add(name)
+	}
+	for _, ip := range localAddrs() {
+		add(ip.String())
+	}
+	return out
+}
+
+// localAddrs are this machine's addresses another machine could reach:
+// loopback and link-local are left out.
+func localAddrs() []net.IP {
+	addrs, err := net.InterfaceAddrs()
+	if err != nil {
+		return nil
+	}
+	var out []net.IP
+	for _, a := range addrs {
+		n, ok := a.(*net.IPNet)
+		if !ok || !n.IP.IsGlobalUnicast() {
+			continue
+		}
+		out = append(out, n.IP)
+	}
+	return out
+}
+
+func (in *serverInit) configPath() string { return filepath.Join(in.etcDir, "server.yaml") }
+
+func (in *serverInit) ensureConfig(context.Context) error {
+	path := in.configPath()
+	if _, err := os.Stat(path); err == nil {
+		in.kept("%s (never overwritten)", path)
+	} else {
+		contact := in.admin
+		if contact == "" {
+			contact = "platform@example.com"
+		}
+		body := fmt.Sprintf(serverConfigTemplate, in.etcDir, in.etcDir, in.etcDir,
+			in.storeAddress(), contact)
+		if err := os.WriteFile(path, []byte(body), 0o640); err != nil {
+			return err
+		}
+		if err := in.chown(path, false); err != nil {
+			return err
+		}
+		in.done("%s", path)
+	}
+
+	// Checked whether written now or years ago: a configuration that will not
+	// load is better found here than as a unit restarting every two seconds.
+	cfg, err := config.Load(path)
+	if err != nil {
+		return err
+	}
+	if err := cfg.Validate(); err != nil {
+		return fmt.Errorf("%s: %w", path, err)
+	}
+	return nil
+}
+
+// serverConfigTemplate is the smallest file that is not a toy: mutual TLS,
+// authentication, default limits and a price, and a control plane with an
+// empty catalog.
+const serverConfigTemplate = `# Written by 'llmgw server init'. Every key: docs/reference.md
+listen: ":8080"
+admin_listen: "127.0.0.1:9090"   # serves pprof, which can dump API keys: keep it local
+agent_listen: ":7443"
+
+agent_tls:
+  ca: %s/ca.pem
+  cert: %s/server.pem
+  key: %s/server-key.pem
+
+# A control plane with nothing in its catalog yet. Models are added while it
+# runs, with 'llmgw ctl model add' or from /admin/mirror.
+control:
+  enabled: true
+
+auth:
+  enabled: true
+  store: "%s"   # a SQLite path, or @file holding a PostgreSQL address
+  signup_open: true
+  # Applied to any key no scope gives a limit to. Without them, a project
+  # created without limits is metered by nothing at all.
+  default_rpm: 60
+  default_tpm: 200000
+  # Without a price every request costs zero and no budget ever binds. 1000
+  # everywhere turns a budget into a plain token quota.
+  default_price:
+    prompt_per_1k: 1000
+    completion_per_1k: 1000
+
+landing:
+  contact: "%s"
+
+# The engines every node installs for itself. Omitted, each gets the llama.cpp
+# build this release was checked against.
+# engines:
+#   llamacpp: b10991
+
+# A model mirror, once there is one (docs/install.md, step 3):
+# mirror:
+#   api: https://models.internal:8082
+#   tls: { ca: /etc/llmgw/ca.pem, cert: /etc/llmgw/server.pem, key: /etc/llmgw/server-key.pem }
+`
+
+func (in *serverInit) ensureStore(ctx context.Context) error {
+	cfg, err := config.Load(in.configPath())
+	if err != nil {
+		return err
+	}
+	address := cfg.Auth.Store
+	if address == "" {
+		address = filepath.Join(in.stateDir, "llmgw.db")
+	}
+	s, err := store.Open(ctx, address)
+	if err != nil {
+		return fmt.Errorf("opening %s: %w", address, err)
+	}
+	// Opened as root, so a SQLite file created just now belongs to root and
+	// the service could not write it. Given back before anything else runs.
+	defer in.chownSQLite(address)
+	defer s.Close()
+
+	// The tree an approved account hangs off. Approving into a project that
+	// does not exist succeeds, and the first key that person makes does not.
+	if err := s.PutOrg(ctx, store.Org{ID: "default", Name: "default"}); err != nil {
+		return err
+	}
+	if err := s.PutTeam(ctx, store.Team{ID: "default", OrgID: "default", Name: "default"}); err != nil {
+		return err
+	}
+	if err := s.PutProject(ctx, store.Project{ID: "default", TeamID: "default", Name: "default"}); err != nil {
+		return err
+	}
+
+	if in.admin == "" {
+		in.kept("store at %s (no --admin: create an account with 'llmgw users create')", address)
+		return nil
+	}
+	if u, err := s.UserByEmail(ctx, in.admin); err == nil {
+		if !u.IsAdmin {
+			if err := s.SetAdmin(ctx, u.ID, true); err != nil {
+				return err
+			}
+		}
+		in.kept("administrator %s (%s); its password is unchanged", in.admin, u.ID)
+		return nil
+	} else if !errors.Is(err, store.ErrNotFound) {
+		return err
+	}
+
+	password := randomPassword()
+	hash, err := account.HashPassword(password)
+	if err != nil {
+		return err
+	}
+	id := randomID("usr")
+	now := time.Now()
+	if err := s.CreateUser(ctx, store.User{
+		ID: id, Email: in.admin, Name: in.admin,
+		PasswordHash: hash, State: store.UserPending, CreatedAt: now,
+	}); err != nil {
+		return err
+	}
+	if err := s.ApproveUser(ctx, id, "default", "init", now); err != nil {
+		return err
+	}
+	if err := s.SetAdmin(ctx, id, true); err != nil {
+		return err
+	}
+	in.done("administrator %s (%s)", in.admin, id)
+	fmt.Fprintf(in.out, "\n        password: %s\n        shown once — sign in and change it from /profile\n\n", password)
+	return nil
+}
+
+func (in *serverInit) chownSQLite(address string) {
+	if strings.Contains(address, "://") || strings.HasPrefix(address, "@") {
+		return
+	}
+	for _, suffix := range []string{"", "-wal", "-shm"} {
+		if _, err := os.Stat(address + suffix); err == nil {
+			_ = in.chown(address+suffix, true)
+		}
+	}
+}
+
+func randomPassword() string {
+	var buf [15]byte
+	_, _ = rand.Read(buf[:])
+	return strings.ToLower(base32.StdEncoding.WithPadding(base32.NoPadding).EncodeToString(buf[:]))
+}
+
+func (in *serverInit) ensureService(ctx context.Context) error {
+	if _, err := exec.LookPath("systemctl"); err != nil {
+		fmt.Fprintf(in.out, "  skip  no systemd here; start it with:\n"+
+			"        sudo -u %s llmgw server --config %s\n", in.user, in.configPath())
+		return nil
+	}
+	if b, err := os.ReadFile(in.unitPath); err == nil && string(b) == serverUnit {
+		in.kept("%s", in.unitPath)
+	} else {
+		if err == nil {
+			// Somebody's edits. Kept beside it, because the unit is also where
+			// people put environment overrides and those are not ours to lose.
+			_ = os.WriteFile(in.unitPath+".bak", b, 0o644)
+		}
+		if err := os.WriteFile(in.unitPath, []byte(serverUnit), 0o644); err != nil {
+			return err
+		}
+		in.done("%s", in.unitPath)
+	}
+
+	systemctl := func(args ...string) error {
+		out, err := exec.CommandContext(ctx, "systemctl", args...).CombinedOutput()
+		if err != nil {
+			return fmt.Errorf("systemctl %s: %v: %s", strings.Join(args, " "), err, strings.TrimSpace(string(out)))
+		}
+		return nil
+	}
+	if err := systemctl("daemon-reload"); err != nil {
+		return err
+	}
+	active := exec.CommandContext(ctx, "systemctl", "is-active", "--quiet", "llmgw-server").Run() == nil
+	switch {
+	case !active:
+		if err := systemctl("enable", "--now", "llmgw-server"); err != nil {
+			return err
+		}
+		in.done("llmgw-server started")
+	case in.changed:
+		// A certificate or configuration it has not read is one it is not
+		// using, and the difference shows up as a node that cannot connect.
+		if err := systemctl("restart", "llmgw-server"); err != nil {
+			return err
+		}
+		in.done("llmgw-server restarted to pick up the changes")
+	default:
+		in.kept("llmgw-server running")
+	}
+	return in.waitHealthy(ctx)
+}
+
+func (in *serverInit) waitHealthy(ctx context.Context) error {
+	cfg, err := config.Load(in.configPath())
+	if err != nil {
+		return err
+	}
+	_, port, err := net.SplitHostPort(cfg.Listen)
+	if err != nil {
+		return fmt.Errorf("listen %q: %w", cfg.Listen, err)
+	}
+	url := "http://127.0.0.1:" + port + "/health"
+	deadline := time.Now().Add(30 * time.Second)
+	for {
+		req, _ := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
+		resp, err := http.DefaultClient.Do(req)
+		if err == nil {
+			resp.Body.Close()
+			if resp.StatusCode == http.StatusOK {
+				in.kept("answering on %s", url)
+				return nil
+			}
+		}
+		if time.Now().After(deadline) {
+			return fmt.Errorf("not answering on %s after 30s; 'journalctl -u llmgw-server -n 50' says why", url)
+		}
+		select {
+		case <-ctx.Done():
+			return ctx.Err()
+		case <-time.After(time.Second):
+		}
+	}
+}
+
+func (in *serverInit) nextSteps() {
+	server := "https://<this machine>:7443"
+	if ips := localAddrs(); len(ips) > 0 {
+		server = "https://" + net.JoinHostPort(ips[0].String(), "7443")
+	}
+	fmt.Fprintf(in.out, `
+The control plane is up. Next:
+
+  llmgw ctl status                      # no nodes, no models: correct for now
+
+  # each node, from this machine (it holds the authority):
+  deploy/bootstrap/onboard-node.sh --node <name> --addr <node ip> --server %s
+
+  # then a model:
+  llmgw ctl model add -f model.yaml
+
+The web UI is on port 8080; /admin once you are signed in as the administrator.
+`, server)
+}

internal/cli/serverinit_postgres.go

@@ -0,0 +1,525 @@
+package cli
+
+import (
+	"context"
+	"crypto/rand"
+	"encoding/hex"
+	"fmt"
+	"log/slog"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"strconv"
+	"strings"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// pgRole and pgDatabase are what init creates in a PostgreSQL it manages.
+const (
+	pgRole     = "llmgw"
+	pgDatabase = "llmgw"
+)
+
+// dsnPath is where the PostgreSQL address is kept, out of server.yaml, so the
+// password in it is not in a file that gets copied, diffed and pasted.
+func (in *serverInit) dsnPath() string { return filepath.Join(in.etcDir, "secrets", "store-dsn") }
+
+// storeAddress is what auth.store says for the store init manages.
+func (in *serverInit) storeAddress() string {
+	if in.postgres {
+		return "@" + in.dsnPath()
+	}
+	return filepath.Join(in.stateDir, "llmgw.db")
+}
+
+// ensureDatabase puts the store in PostgreSQL when asked to, moving whatever
+// the SQLite store already holds.
+//
+// The move is the point. Keys are stored hashed, so starting PostgreSQL empty
+// means reissuing every key and being unable to say which was which; and the
+// administrator init created a minute ago would be gone too.
+func (in *serverInit) ensureDatabase(ctx context.Context) (err error) {
+	if !in.postgres {
+		return nil
+	}
+	setup := in.pgSetup
+	if setup == nil {
+		setup = in.setupPostgres
+	}
+	target, err := setup(ctx)
+	if err != nil {
+		return err
+	}
+
+	cfg, err := config.Load(in.configPath())
+	if err != nil {
+		return err
+	}
+	current := cfg.Auth.Store
+	if current == target {
+		in.kept("store in PostgreSQL (%s)", in.dsnPath())
+		return nil
+	}
+	if isPostgresAddress(current) {
+		// Two PostgreSQL stores, and a guess about which is meant would lose
+		// one of them.
+		return fmt.Errorf("server.yaml already names a PostgreSQL store (%s); "+
+			"init moves SQLite into PostgreSQL, not one PostgreSQL into another", current)
+	}
+
+	// Nothing may write the store while it is copied: a row landing after its
+	// table has been passed would be absent, and nothing would say so.
+	stopped, err := in.stopService(ctx)
+	if err != nil {
+		return err
+	}
+	defer func() {
+		// A move that failed leaves the configuration naming SQLite, which
+		// still holds everything: start the service on it again rather than
+		// leave the gateway down over a database that did not work out.
+		if err != nil && stopped {
+			_ = exec.Command("systemctl", "start", "llmgw-server").Run()
+			fmt.Fprintf(in.out, "  ..    llmgw-server started again on its SQLite store\n")
+		}
+	}()
+
+	if current != "" {
+		if _, err := os.Stat(current); err == nil {
+			if err := in.copyStore(ctx, current, target); err != nil {
+				return err
+			}
+		}
+	}
+
+	if err := setAuthStore(in.configPath(), target); err != nil {
+		return err
+	}
+	in.done("server.yaml now keeps the store in PostgreSQL")
+	return nil
+}
+
+func (in *serverInit) copyStore(ctx context.Context, from, to string) error {
+	src, err := store.Open(ctx, from)
+	if err != nil {
+		return fmt.Errorf("opening %s: %w", from, err)
+	}
+	defer src.Close()
+	dst, err := store.Open(ctx, to)
+	if err != nil {
+		return fmt.Errorf("opening PostgreSQL: %w", err)
+	}
+	defer dst.Close()
+
+	empty, err := dst.Empty(ctx)
+	if err != nil {
+		return err
+	}
+	if !empty {
+		// A previous run that copied and then stopped before changing the
+		// configuration. The service was stopped for that copy too, so what
+		// is there is what SQLite held.
+		in.kept("PostgreSQL already holds a store; %s is left as it is", from)
+		return nil
+	}
+	res, err := src.CopyTo(ctx, dst)
+	if err != nil {
+		return fmt.Errorf("copying %s into PostgreSQL: %w", from, err)
+	}
+	var rows int64
+	for _, r := range res {
+		rows += r.Rows
+	}
+	in.done("copied %d rows from %s into PostgreSQL (the SQLite file is kept)", rows, from)
+	return nil
+}
+
+func isPostgresAddress(a string) bool {
+	return strings.HasPrefix(a, "@") || strings.HasPrefix(a, "postgres://") ||
+		strings.HasPrefix(a, "postgresql://")
+}
+
+// setAuthStore changes auth.store in server.yaml and nothing else.
+//
+// Edited as text rather than unmarshalled and written back, because writing
+// it back would drop every comment in a file people annotate. The old file is
+// kept beside it.
+func setAuthStore(path, address string) error {
+	b, err := os.ReadFile(path)
+	if err != nil {
+		return err
+	}
+	lines := strings.Split(string(b), "\n")
+	value := fmt.Sprintf("%q", address)
+
+	authAt, storeAt, indent := -1, -1, "  "
+	for i, l := range lines {
+		trimmed := strings.TrimSpace(l)
+		top := l != "" && l[0] != ' ' && l[0] != '\t' && !strings.HasPrefix(trimmed, "#")
+		if top {
+			if authAt >= 0 && i > authAt {
+				break // past the auth block
+			}
+			if strings.HasPrefix(trimmed, "auth:") {
+				authAt = i
+			}
+			continue
+		}
+		if authAt >= 0 && strings.HasPrefix(trimmed, "store:") {
+			storeAt = i
+			indent = l[:len(l)-len(strings.TrimLeft(l, " \t"))]
+			break
+		}
+	}
+	switch {
+	case storeAt >= 0:
+		lines[storeAt] = indent + "store: " + value
+	case authAt >= 0:
+		lines = append(lines[:authAt+1], append([]string{indent + "store: " + value}, lines[authAt+1:]...)...)
+	default:
+		lines = append(lines, "auth:", "  enabled: true", "  store: "+value)
+	}
+
+	backup := fmt.Sprintf("%s.bak-%s", path, time.Now().Format("20060102-150405"))
+	if err := os.WriteFile(backup, b, 0o640); err != nil {
+		return err
+	}
+	info, err := os.Stat(path)
+	if err != nil {
+		return err
+	}
+	return os.WriteFile(path, []byte(strings.Join(lines, "\n")), info.Mode().Perm())
+}
+
+// pgServer is the PostgreSQL installation on this machine, as systemd and the
+// file system describe it.
+//
+// Found rather than assumed. A distribution's own package calls its unit
+// postgresql.service and puts initdb and psql on PATH; the PostgreSQL
+// project's packages (PGDG) call it postgresql-16.service and keep them in
+// /usr/pgsql-16/bin — and they provide "postgresql-server" too, so asking dnf
+// for that name can install or find one of them. Assuming the first found a
+// Rocky machine with the second unable to initialise the database it had.
+type pgServer struct {
+	unit   string // postgresql.service, postgresql-16.service
+	pgdata string // "" when the unit does not say, which is Debian's case
+	bin    string // directory holding initdb and psql; "" means PATH
+}
+
+func (p pgServer) tool(name string) string {
+	if p.bin != "" {
+		return filepath.Join(p.bin, name)
+	}
+	return name
+}
+
+// pickPostgresUnit chooses the server unit from systemctl's list of
+// postgresql* unit files: the plain one when there is one, else the highest
+// version. Templates (postgresql@.service) are not a server by themselves.
+func pickPostgresUnit(list string) string {
+	best, bestVersion := "", -1
+	for _, line := range strings.Split(list, "\n") {
+		f := strings.Fields(line)
+		if len(f) == 0 {
+			continue
+		}
+		unit := f[0]
+		if !strings.HasSuffix(unit, ".service") || strings.Contains(unit, "@") {
+			continue
+		}
+		name := strings.TrimSuffix(unit, ".service")
+		if name == "postgresql" {
+			return unit
+		}
+		v := strings.TrimPrefix(name, "postgresql-")
+		if n, err := strconv.Atoi(strings.SplitN(v, ".", 2)[0]); err == nil && n > bestVersion {
+			best, bestVersion = unit, n
+		}
+	}
+	return best
+}
+
+// pgdataFrom reads PGDATA out of `systemctl show -p Environment --value`.
+func pgdataFrom(env string) string {
+	for _, kv := range strings.Fields(env) {
+		if v, ok := strings.CutPrefix(kv, "PGDATA="); ok {
+			return v
+		}
+	}
+	return ""
+}
+
+func findPostgres(ctx context.Context) (pgServer, bool) {
+	out, _ := exec.CommandContext(ctx, "systemctl", "list-unit-files", "--type=service",
+		"--no-legend", "--plain", "postgresql*").Output()
+	unit := pickPostgresUnit(string(out))
+	if unit == "" {
+		return pgServer{}, false
+	}
+	p := pgServer{unit: unit}
+	env, _ := exec.CommandContext(ctx, "systemctl", "show", "-p", "Environment", "--value", unit).Output()
+	p.pgdata = pgdataFrom(string(env))
+
+	if v, ok := strings.CutPrefix(strings.TrimSuffix(unit, ".service"), "postgresql-"); ok {
+		if dir := "/usr/pgsql-" + v + "/bin"; isDir(dir) {
+			p.bin = dir
+		}
+		if p.pgdata == "" && isDir("/var/lib/pgsql/"+v) {
+			p.pgdata = "/var/lib/pgsql/" + v + "/data"
+		}
+	}
+	if p.bin == "" {
+		if _, err := exec.LookPath("psql"); err != nil {
+			// Debian keeps the server's tools per version and puts only
+			// wrappers on PATH; without even those, look where it keeps them.
+			if dirs, _ := filepath.Glob("/usr/lib/postgresql/*/bin"); len(dirs) > 0 {
+				p.bin = dirs[len(dirs)-1]
+			}
+		}
+	}
+	return p, true
+}
+
+func isDir(p string) bool {
+	st, err := os.Stat(p)
+	return err == nil && st.IsDir()
+}
+
+// setupPostgres makes sure a PostgreSQL on this machine holds a database for
+// llmgw, installing the server if there is none, and returns the store
+// address.
+func (in *serverInit) setupPostgres(ctx context.Context) (string, error) {
+	installed, err := in.installPostgres(ctx)
+	if err != nil {
+		return "", err
+	}
+	if err := in.waitForPostgres(ctx); err != nil {
+		return "", err
+	}
+
+	exists, err := in.psqlBool(ctx, "SELECT 1 FROM pg_roles WHERE rolname = '"+pgRole+"'")
+	if err != nil {
+		return "", err
+	}
+	_, statErr := os.Stat(in.dsnPath())
+	if exists && statErr == nil {
+		in.kept("PostgreSQL role %s", pgRole)
+	} else {
+		// A role with no address written down is one whose password nobody
+		// knows; giving it a new one is the only way to use it.
+		password := make([]byte, 24)
+		_, _ = rand.Read(password)
+		pass := hex.EncodeToString(password)
+		verb := "CREATE ROLE"
+		if exists {
+			verb = "ALTER ROLE"
+		}
+		if _, err := in.psql(ctx, verb+" "+pgRole+" WITH LOGIN PASSWORD '"+pass+"'"); err != nil {
+			return "", err
+		}
+		dsn := fmt.Sprintf("postgres://%s:%s@127.0.0.1:5432/%s?sslmode=disable", pgRole, pass, pgDatabase)
+		if err := in.writeDSN(dsn); err != nil {
+			return "", err
+		}
+		in.done("PostgreSQL role %s, its address in %s", pgRole, in.dsnPath())
+	}
+
+	exists, err = in.psqlBool(ctx, "SELECT 1 FROM pg_database WHERE datname = '"+pgDatabase+"'")
+	if err != nil {
+		return "", err
+	}
+	if exists {
+		in.kept("PostgreSQL database %s", pgDatabase)
+	} else {
+		if _, err := in.psql(ctx, "CREATE DATABASE "+pgDatabase+" OWNER "+pgRole); err != nil {
+			return "", err
+		}
+		in.done("PostgreSQL database %s", pgDatabase)
+	}
+
+	if installed {
+		// Its defaults assume it owns the machine. A gateway swapped out to
+		// give the database cache is worse than a smaller cache, and a store
+		// whose working set is megabytes needs little. Only a PostgreSQL init
+		// installed itself: one that was already here may serve something
+		// else that needs its connections.
+		for _, q := range []string{
+			"ALTER SYSTEM SET shared_buffers = '96MB'",
+			"ALTER SYSTEM SET max_connections = 40",
+		} {
+			if _, err := in.psql(ctx, q); err != nil {
+				return "", err
+			}
+		}
+		if out, err := exec.CommandContext(ctx, "systemctl", "restart", in.pg.unit).CombinedOutput(); err != nil {
+			return "", fmt.Errorf("restarting PostgreSQL: %v: %s", err, strings.TrimSpace(string(out)))
+		}
+		if err := in.waitForPostgres(ctx); err != nil {
+			return "", err
+		}
+		in.done("PostgreSQL sized for a machine it shares (shared_buffers 96MB, 40 connections)")
+	}
+
+	// Proof the address works, before anything is moved into it.
+	address := "@" + in.dsnPath()
+	s, err := store.Open(ctx, address)
+	if err != nil {
+		return "", fmt.Errorf("connecting with the address in %s: %w (PostgreSQL's pg_hba.conf "+
+			"must allow password logins from 127.0.0.1: scram-sha-256 or md5, not ident)", in.dsnPath(), err)
+	}
+	_ = s.Close()
+	return address, nil
+}
+
+func (in *serverInit) writeDSN(dsn string) error {
+	dir := filepath.Dir(in.dsnPath())
+	if err := os.MkdirAll(dir, 0o750); err != nil {
+		return err
+	}
+	if err := in.chown(dir, false); err != nil {
+		return err
+	}
+	if err := os.WriteFile(in.dsnPath(), []byte(dsn+"\n"), 0o600); err != nil {
+		return err
+	}
+	// Read by the service, which runs as llmgw; root reads it anyway.
+	return in.chown(in.dsnPath(), true)
+}
+
+// installPostgres installs the server when this machine has none, makes sure
+// its database is initialised and running, and says whether it installed it.
+func (in *serverInit) installPostgres(ctx context.Context) (bool, error) {
+	host := discovery.DiscoverHost()
+	pg, found := findPostgres(ctx)
+	installed := false
+	if found {
+		in.kept("PostgreSQL server (%s)", pg.unit)
+	} else {
+		if host.Family() == "" {
+			return false, fmt.Errorf("no PostgreSQL here and no package manager init knows (%s); "+
+				"install PostgreSQL and run init again", host.OSID)
+		}
+		fmt.Fprintf(in.out, "  ..    installing PostgreSQL (a minute or two)\n")
+		pi := &provision.Installer{Host: host, Log: slog.Default()}
+		if err := pi.InstallPackages(ctx, map[string][]string{
+			"deb": {"postgresql"},
+			"rpm": {"postgresql-server", "postgresql"},
+		}); err != nil {
+			return false, err
+		}
+		if pg, found = findPostgres(ctx); !found {
+			q := "rpm -qa 'postgresql*'"
+			if host.Family() == "deb" {
+				q = "dpkg -l 'postgresql*'"
+			}
+			have, _ := exec.CommandContext(ctx, "sh", "-c", q).CombinedOutput()
+			return false, fmt.Errorf("PostgreSQL was installed but no postgresql service unit "+
+				"appeared; what is installed:\n%s", strings.TrimSpace(string(have)))
+		}
+		installed = true
+		in.done("PostgreSQL installed (%s)", pg.unit)
+	}
+	in.pg = pg
+
+	// Debian's package creates and starts a cluster itself. The RHEL family's,
+	// and PGDG's, leave the data directory empty until someone runs initdb.
+	if pg.pgdata != "" {
+		if _, err := os.Stat(filepath.Join(pg.pgdata, "PG_VERSION")); err != nil {
+			// Directly rather than through postgresql-setup, which is named
+			// differently by every packager. Password logins over TCP are
+			// asked for: RHEL's default for them is ident, which no service
+			// account passes.
+			if err := os.MkdirAll(pg.pgdata, 0o700); err != nil {
+				return false, err
+			}
+			if out, err := exec.CommandContext(ctx, "chown", "postgres:postgres", pg.pgdata).CombinedOutput(); err != nil {
+				return false, fmt.Errorf("chown %s: %v: %s", pg.pgdata, err, strings.TrimSpace(string(out)))
+			}
+			cmd := exec.CommandContext(ctx, "runuser", "-u", "postgres", "--", pg.tool("initdb"),
+				"-D", pg.pgdata, "--auth-host=scram-sha-256", "--auth-local=peer")
+			cmd.Dir = "/"
+			if out, err := cmd.CombinedOutput(); err != nil {
+				return false, fmt.Errorf("initdb %s: %v: %s", pg.pgdata, err, lastLines(out, 5))
+			}
+			in.done("PostgreSQL database cluster initialised in %s", pg.pgdata)
+		}
+	}
+	if exec.CommandContext(ctx, "systemctl", "is-active", "--quiet", pg.unit).Run() != nil {
+		if out, err := exec.CommandContext(ctx, "systemctl", "enable", "--now", pg.unit).CombinedOutput(); err != nil {
+			return false, fmt.Errorf("starting %s: %v: %s", pg.unit, err, strings.TrimSpace(string(out)))
+		}
+		in.done("%s started", pg.unit)
+	}
+	return installed, nil
+}
+
+func lastLines(b []byte, n int) string {
+	lines := strings.Split(strings.TrimSpace(string(b)), "\n")
+	if len(lines) > n {
+		lines = lines[len(lines)-n:]
+	}
+	return strings.Join(lines, "\n")
+}
+
+// psql runs one statement as the postgres superuser, over the local socket.
+func (in *serverInit) psql(ctx context.Context, query string) (string, error) {
+	cmd := exec.CommandContext(ctx, "runuser", "-u", "postgres", "--",
+		in.pg.tool("psql"), "-v", "ON_ERROR_STOP=1", "-qAtX", "-c", query)
+	// postgres cannot enter root's home, and psql says so on every call.
+	cmd.Dir = "/"
+	out, err := cmd.CombinedOutput()
+	if err != nil {
+		// The statement can hold a password; the error names only its verb.
+		verb := strings.Join(strings.Fields(query)[:2], " ")
+		return "", fmt.Errorf("psql (%s): %v: %s", verb, err, strings.TrimSpace(string(out)))
+	}
+	return strings.TrimSpace(string(out)), nil
+}
+
+func (in *serverInit) psqlBool(ctx context.Context, query string) (bool, error) {
+	out, err := in.psql(ctx, query)
+	return out == "1", err
+}
+
+// waitForPostgres waits for a server that was just started or restarted to
+// take connections.
+func (in *serverInit) waitForPostgres(ctx context.Context) error {
+	deadline := time.Now().Add(60 * time.Second)
+	for {
+		_, err := in.psql(ctx, "SELECT 1")
+		if err == nil {
+			return nil
+		}
+		if time.Now().After(deadline) {
+			return fmt.Errorf("PostgreSQL is not taking connections after 60s: %w", err)
+		}
+		select {
+		case <-ctx.Done():
+			return ctx.Err()
+		case <-time.After(2 * time.Second):
+		}
+	}
+}
+
+// stopService stops the control plane if it runs, so the store is still.
+// It says whether it stopped it, so a failed move can start it again.
+func (in *serverInit) stopService(ctx context.Context) (bool, error) {
+	if in.unitPath == "" {
+		return false, nil
+	}
+	if _, err := exec.LookPath("systemctl"); err != nil {
+		return false, nil
+	}
+	if exec.CommandContext(ctx, "systemctl", "is-active", "--quiet", "llmgw-server").Run() != nil {
+		return false, nil
+	}
+	if out, err := exec.CommandContext(ctx, "systemctl", "stop", "llmgw-server").CombinedOutput(); err != nil {
+		return false, fmt.Errorf("stopping llmgw-server to move its store: %v: %s", err, strings.TrimSpace(string(out)))
+	}
+	in.done("llmgw-server stopped while its store moves")
+	return true, nil
+}

internal/cli/serverinit_postgres_test.go

@@ -0,0 +1,203 @@
+package cli
+
+import (
+	"context"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+func TestSetAuthStoreChangesOnlyAuthStore(t *testing.T) {
+	p := filepath.Join(t.TempDir(), "server.yaml")
+	body := `# my notes
+threads:
+  store: /var/lib/llmgw/threads.db   # not this one
+  retain_hours: 24
+auth:
+  enabled: true
+  # the store
+  store: /var/lib/llmgw/llmgw.db
+  signup_open: true
+landing:
+  contact: "x"
+`
+	if err := os.WriteFile(p, []byte(body), 0o640); err != nil {
+		t.Fatal(err)
+	}
+	if err := setAuthStore(p, "@/etc/llmgw/secrets/store-dsn"); err != nil {
+		t.Fatal(err)
+	}
+	got, _ := os.ReadFile(p)
+	want := strings.Replace(body, "  store: /var/lib/llmgw/llmgw.db",
+		`  store: "@/etc/llmgw/secrets/store-dsn"`, 1)
+	if string(got) != want {
+		t.Errorf("got:\n%s\nwant:\n%s", got, want)
+	}
+	cfg, err := config.Load(p)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if cfg.Auth.Store != "@/etc/llmgw/secrets/store-dsn" || cfg.Threads.Store != "/var/lib/llmgw/threads.db" {
+		t.Errorf("auth.store %q, threads.store %q", cfg.Auth.Store, cfg.Threads.Store)
+	}
+	backups, _ := filepath.Glob(p + ".bak-*")
+	if len(backups) != 1 {
+		t.Errorf("the old file should be kept beside it, found %v", backups)
+	}
+}
+
+func TestSetAuthStoreAddsTheLineWhenAuthHasNone(t *testing.T) {
+	p := filepath.Join(t.TempDir(), "server.yaml")
+	if err := os.WriteFile(p, []byte("auth:\n  enabled: true\nlanding:\n  contact: x\n"), 0o640); err != nil {
+		t.Fatal(err)
+	}
+	if err := setAuthStore(p, "@/x"); err != nil {
+		t.Fatal(err)
+	}
+	cfg, err := config.Load(p)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if cfg.Auth.Store != "@/x" || !cfg.Auth.Enabled {
+		t.Errorf("auth = %+v", cfg.Auth)
+	}
+}
+
+// fakePostgres stands in for setupPostgres the way it really answers: an
+// address file under /etc/llmgw/secrets, returned as "@file". What the file
+// holds is a SQLite path, so everything after the setup runs for real.
+func fakePostgres(t *testing.T, in *serverInit) (address, db string) {
+	t.Helper()
+	db = filepath.Join(t.TempDir(), "pg-standin.db")
+	in.pgSetup = func(context.Context) (string, error) {
+		if err := os.MkdirAll(filepath.Dir(in.dsnPath()), 0o750); err != nil {
+			return "", err
+		}
+		return "@" + in.dsnPath(), os.WriteFile(in.dsnPath(), []byte(db+"\n"), 0o600)
+	}
+	return "@" + in.dsnPath(), db
+}
+
+// Moving an install that began on SQLite: the administrator it created must
+// be in the new store, the configuration must name it, and the SQLite file
+// must still be there. The new store is SQLite too here — what PostgreSQL
+// adds is the setup, which needs a machine to run on.
+func TestInitMovesAnExistingStore(t *testing.T) {
+	in, out := newTestInit(t)
+	if err := in.run(context.Background()); err != nil {
+		t.Fatalf("%v\n%s", err, out)
+	}
+	old := filepath.Join(in.stateDir, "llmgw.db")
+
+	in.postgres = true
+	target, db := fakePostgres(t, in)
+	out.Reset()
+	if err := in.run(context.Background()); err != nil {
+		t.Fatalf("%v\n%s", err, out)
+	}
+	if !strings.Contains(out.String(), "copied") {
+		t.Errorf("nothing was copied:\n%s", out)
+	}
+
+	cfg, err := config.Load(in.configPath())
+	if err != nil {
+		t.Fatal(err)
+	}
+	if cfg.Auth.Store != target {
+		t.Fatalf("server.yaml still names %q", cfg.Auth.Store)
+	}
+	s, err := store.Open(context.Background(), db)
+	if err != nil {
+		t.Fatal(err)
+	}
+	u, err := s.UserByEmail(context.Background(), "admin@example.com")
+	s.Close()
+	if err != nil || !u.IsAdmin {
+		t.Fatalf("the administrator did not move: %v %+v", err, u)
+	}
+	if _, err := os.Stat(old); err != nil {
+		t.Error("the SQLite store should be kept")
+	}
+
+	// And once moved, again is nothing.
+	in.changed = false
+	out.Reset()
+	if err := in.run(context.Background()); err != nil {
+		t.Fatalf("%v\n%s", err, out)
+	}
+	if in.changed {
+		t.Errorf("a run after the move changed something:\n%s", out)
+	}
+}
+
+func TestInitWritesAPostgresConfigurationFromTheStart(t *testing.T) {
+	in, out := newTestInit(t)
+	in.postgres = true
+	target, _ := fakePostgres(t, in)
+	if err := in.run(context.Background()); err != nil {
+		t.Fatalf("%v\n%s", err, out)
+	}
+	cfg, _ := config.Load(in.configPath())
+	if cfg.Auth.Store != target {
+		t.Errorf("auth.store = %q", cfg.Auth.Store)
+	}
+	if _, err := os.Stat(filepath.Join(in.stateDir, "llmgw.db")); err == nil {
+		t.Error("a PostgreSQL install should not leave a SQLite store behind")
+	}
+}
+
+func TestInitWillNotMoveOnePostgresIntoAnother(t *testing.T) {
+	in, out := newTestInit(t)
+	if err := os.MkdirAll(in.etcDir, 0o755); err != nil {
+		t.Fatal(err)
+	}
+	cfg := "control:\n  enabled: true\nauth:\n  enabled: true\n  store: postgres://elsewhere/llmgw\n"
+	if err := os.WriteFile(in.configPath(), []byte(cfg), 0o640); err != nil {
+		t.Fatal(err)
+	}
+	in.postgres = true
+	in.pgSetup = func(context.Context) (string, error) { return "@/somewhere/else", nil }
+	err := in.run(context.Background())
+	if err == nil || !strings.Contains(err.Error(), "already names a PostgreSQL store") {
+		t.Fatalf("expected a refusal, got %v\n%s", err, out)
+	}
+}
+
+// The lists below are what `systemctl list-unit-files --type=service
+// --no-legend --plain 'postgresql*'` prints on each kind of machine.
+func TestPickPostgresUnit(t *testing.T) {
+	for _, c := range []struct{ name, list, want string }{
+		{"rocky, distribution package",
+			"postgresql.service disabled disabled\npostgresql@.service disabled disabled\n", "postgresql.service"},
+		{"rocky, PGDG package",
+			"postgresql-16.service disabled disabled\n", "postgresql-16.service"},
+		{"two PGDG versions",
+			"postgresql-13.service enabled disabled\npostgresql-16.service disabled disabled\n", "postgresql-16.service"},
+		{"debian",
+			"postgresql.service enabled enabled\npostgresql@.service indirect enabled\n", "postgresql.service"},
+		{"only the template", "postgresql@.service indirect enabled\n", ""},
+		{"none", "", ""},
+	} {
+		if got := pickPostgresUnit(c.list); got != c.want {
+			t.Errorf("%s: got %q, want %q", c.name, got, c.want)
+		}
+	}
+}
+
+// What `systemctl show -p Environment --value` prints for each unit, from
+// the unit files the packages ship.
+func TestPGDATAFromTheUnit(t *testing.T) {
+	for env, want := range map[string]string{
+		"PG_OOM_ADJUST_FILE=/proc/self/oom_score_adj PG_OOM_ADJUST_VALUE=0 PGDATA=/var/lib/pgsql/data": "/var/lib/pgsql/data",
+		"PGDATA=/var/lib/pgsql/16/data/ PG_OOM_ADJUST_FILE=/proc/self/oom_score_adj":                   "/var/lib/pgsql/16/data/",
+		"": "",
+	} {
+		if got := pgdataFrom(env); got != want {
+			t.Errorf("%q: got %q, want %q", env, got, want)
+		}
+	}
+}

internal/cli/serverinit_test.go

@@ -0,0 +1,173 @@
+package cli
+
+import (
+	"bytes"
+	"context"
+	"os"
+	"path/filepath"
+	"slices"
+	"strings"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+func newTestInit(t *testing.T) (*serverInit, *bytes.Buffer) {
+	t.Helper()
+	root := t.TempDir()
+	var out bytes.Buffer
+	return &serverInit{
+		etcDir:   filepath.Join(root, "etc"),
+		stateDir: filepath.Join(root, "state"),
+		pkiDir:   filepath.Join(root, "etc", "pki"),
+		admin:    "admin@example.com",
+		out:      &out,
+	}, &out
+}
+
+func TestInitLeavesAControlPlaneThatStarts(t *testing.T) {
+	in, out := newTestInit(t)
+	if err := in.run(context.Background()); err != nil {
+		t.Fatalf("%v\n%s", err, out)
+	}
+
+	cfg, err := config.Load(in.configPath())
+	if err != nil {
+		t.Fatal(err)
+	}
+	// The configuration install.md used to show had an empty catalog and no
+	// control.enabled, which the server refuses as having nothing to serve.
+	if !cfg.ControlPlaneMode() {
+		t.Error("the written configuration does not run a control plane")
+	}
+	if !cfg.AgentTLS.Enabled() || !cfg.Auth.Enabled {
+		t.Error("mutual TLS and authentication should both be on")
+	}
+	for _, f := range []string{"ca.pem", "server.pem", "server-key.pem"} {
+		if _, err := os.Stat(filepath.Join(in.etcDir, f)); err != nil {
+			t.Error(err)
+		}
+	}
+
+	s, err := store.Open(context.Background(), cfg.Auth.Store)
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer s.Close()
+	u, err := s.UserByEmail(context.Background(), "admin@example.com")
+	if err != nil {
+		t.Fatal(err)
+	}
+	if !u.IsAdmin || u.State != store.UserActive || u.ProjectID != "default" {
+		t.Errorf("administrator not ready to use: %+v", u)
+	}
+	if !strings.Contains(out.String(), "password: ") {
+		t.Error("the generated password was not shown")
+	}
+}
+
+func TestInitAgainChangesNothing(t *testing.T) {
+	in, out := newTestInit(t)
+	if err := in.run(context.Background()); err != nil {
+		t.Fatalf("%v\n%s", err, out)
+	}
+	cert, _ := os.ReadFile(filepath.Join(in.etcDir, "server.pem"))
+
+	again := *in
+	again.changed = false
+	out.Reset()
+	if err := again.run(context.Background()); err != nil {
+		t.Fatalf("%v\n%s", err, out)
+	}
+	if again.changed {
+		t.Errorf("a second run changed something:\n%s", out)
+	}
+	if strings.Contains(out.String(), "password: ") {
+		t.Error("a second run must not reset the administrator's password")
+	}
+	if now, _ := os.ReadFile(filepath.Join(in.etcDir, "server.pem")); !bytes.Equal(now, cert) {
+		t.Error("the certificate was reissued without reason")
+	}
+}
+
+func TestInitWithANewHostKeepsTheOldOnes(t *testing.T) {
+	in, out := newTestInit(t)
+	in.hosts = []string{"old.internal"}
+	if err := in.run(context.Background()); err != nil {
+		t.Fatalf("%v\n%s", err, out)
+	}
+	in.hosts = []string{"new.internal"}
+	if err := in.run(context.Background()); err != nil {
+		t.Fatalf("%v\n%s", err, out)
+	}
+	cert, err := readCert(filepath.Join(in.etcDir, "server.pem"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	for _, h := range []string{"old.internal", "new.internal"} {
+		if !slices.Contains(cert.DNSNames, h) {
+			t.Errorf("certificate lost %s: %v", h, cert.DNSNames)
+		}
+	}
+}
+
+func TestInitNeverOverwritesTheConfiguration(t *testing.T) {
+	in, out := newTestInit(t)
+	if err := os.MkdirAll(in.etcDir, 0o755); err != nil {
+		t.Fatal(err)
+	}
+	mine := "# mine\ncontrol:\n  enabled: true\nauth:\n  enabled: true\n  store: " +
+		filepath.Join(in.stateDir, "mine.db") + "\n"
+	if err := os.WriteFile(in.configPath(), []byte(mine), 0o640); err != nil {
+		t.Fatal(err)
+	}
+	if err := in.run(context.Background()); err != nil {
+		t.Fatalf("%v\n%s", err, out)
+	}
+	if got, _ := os.ReadFile(in.configPath()); string(got) != mine {
+		t.Error("an existing server.yaml was rewritten")
+	}
+	// And the store it names is the one used.
+	if _, err := os.Stat(filepath.Join(in.stateDir, "mine.db")); err != nil {
+		t.Error(err)
+	}
+}
+
+func TestTheInitUnitIsTheShippedUnit(t *testing.T) {
+	b, err := os.ReadFile("../../deploy/systemd/llmgw-server.service")
+	if err != nil {
+		t.Fatal(err)
+	}
+	if string(b) != serverUnit {
+		t.Error("deploy/systemd/llmgw-server.service and the unit 'server init' writes have drifted")
+	}
+}
+
+func TestInitKeepsCertificatesFromAnAuthorityElsewhere(t *testing.T) {
+	in, out := newTestInit(t)
+	// Issued on another machine by an authority that never comes here.
+	elsewhere := &serverInit{etcDir: in.etcDir, pkiDir: filepath.Join(t.TempDir(), "pki"), out: &bytes.Buffer{}}
+	if err := os.MkdirAll(in.etcDir, 0o755); err != nil {
+		t.Fatal(err)
+	}
+	if err := elsewhere.ensureCerts(context.Background()); err != nil {
+		t.Fatal(err)
+	}
+	before, _ := os.ReadFile(filepath.Join(in.etcDir, "server.pem"))
+
+	if err := in.run(context.Background()); err != nil {
+		t.Fatalf("%v\n%s", err, out)
+	}
+	if after, _ := os.ReadFile(filepath.Join(in.etcDir, "server.pem")); !bytes.Equal(before, after) {
+		t.Error("a certificate from another authority was replaced")
+	}
+	if _, err := os.Stat(filepath.Join(in.pkiDir, "ca-key.pem")); err == nil {
+		t.Error("an authority was created beside one that already exists elsewhere")
+	}
+
+	in.hosts = []string{"new.internal"}
+	if err := in.run(context.Background()); err == nil {
+		t.Error("--host cannot be honoured without the authority, and should say so")
+	}
+}

internal/cli/threads.go

@@ -50,7 +50,12 @@ func newThreadsCmd(open opener) *cobra.Command {
 			if off {
 				keep = 0
 			}
-			if err := s.SetThreadRetention(cmd.Context(), project, keep); err != nil {
+			p, err := resolveProject(cmd.Context(), s, project)
+			if err != nil {
+				return err
+			}
+			project = p.Path
+			if err := s.SetThreadRetention(cmd.Context(), p.ID, keep); err != nil {
 				return err
 			}
 			if keep == 0 {
@@ -63,7 +68,7 @@ func newThreadsCmd(open opener) *cobra.Command {
 			return nil
 		},
 	}
-	cmd.Flags().StringVar(&project, "project", "default", "project id")
+	cmd.Flags().StringVar(&project, "project", "", "project: id, path or unique name")
 	cmd.Flags().IntVar(&hours, "retain-hours", 0, "how long a conversation is kept after its last turn")
 	cmd.Flags().BoolVar(&off, "off", false, "stop keeping conversations for this project")
 	return cmd

internal/cli/users.go

@@ -33,6 +33,7 @@ func newUsersCmd() *cobra.Command {
 		newUserStateCmd(open, "reinstate", store.UserActive,
 			"Let a suspended account work again"),
 		newUserCreateCmd(open), newUserResetPasswordCmd(open),
+		newUserMoveCmd(open), newUserDeleteCmd(open),
 		newUserAdminCmd(open, "promote", true, "Let an account see the cluster dashboard at /admin"),
 		newUserAdminCmd(open, "demote", false, "Take away access to the cluster dashboard"))
 	return cmd
@@ -88,7 +89,7 @@ func newUserListCmd(open userOpener) *cobra.Command {
 func newUserApproveCmd(open userOpener) *cobra.Command {
 	var project, by string
 	cmd := &cobra.Command{
-		Use:   "approve <user-id>",
+		Use:   "approve <user>",
 		Short: "Let an account sign in and issue its own keys",
 		Long: "The project is required: it decides which limits this person's keys\n" +
 			"inherit. An account approved into no project would pass every check and\n" +
@@ -101,27 +102,31 @@ func newUserApproveCmd(open userOpener) *cobra.Command {
 			}
 			defer s.Close()
 
-			if err := s.ApproveUser(cmd.Context(), args[0], project, by, time.Now()); err != nil {
-				return fmt.Errorf("approving %s: %w", args[0], err)
-			}
-			u, err := s.UserByID(cmd.Context(), args[0])
+			u, err := resolveUser(cmd.Context(), s, args[0])
 			if err != nil {
 				return err
 			}
-			fmt.Fprintf(cmd.OutOrStdout(), "%s approved into project %s\n", u.Email, project)
+			p, err := resolveProject(cmd.Context(), s, project)
+			if err != nil {
+				return err
+			}
+			if err := s.ApproveUser(cmd.Context(), u.ID, p.ID, by, time.Now()); err != nil {
+				return fmt.Errorf("approving %s: %w", u.Email, err)
+			}
+			fmt.Fprintf(cmd.OutOrStdout(), "%s approved into %s\n", u.Email, p.Path)
 			cmd.PrintErrln("They can sign in and create their own keys from /profile.")
 			return nil
 		},
 	}
-	cmd.Flags().StringVar(&project, "project", "default", "project the person's keys belong to")
+	cmd.Flags().StringVar(&project, "project", "",
+		"project the person's keys belong to: id, path (Acme/platform/gateway) or unique name")
 	cmd.Flags().StringVar(&by, "by", "", "who approved, for the record")
-	_ = cmd.MarkFlagRequired("project")
 	return cmd
 }
 
 func newUserStateCmd(open userOpener, verb string, state store.UserState, short string) *cobra.Command {
 	return &cobra.Command{
-		Use:   verb + " <user-id>",
+		Use:   verb + " <user>",
 		Short: short,
 		Args:  cobra.ExactArgs(1),
 		RunE: func(cmd *cobra.Command, args []string) error {
@@ -131,18 +136,22 @@ func newUserStateCmd(open userOpener, verb string, state store.UserState, short
 			}
 			defer s.Close()
 
-			if err := s.SetUserState(cmd.Context(), args[0], state); err != nil {
+			u, err := resolveUser(cmd.Context(), s, args[0])
+			if err != nil {
+				return err
+			}
+			if err := s.SetUserState(cmd.Context(), u.ID, state); err != nil {
 				return err
 			}
 			if state == store.UserSuspended {
 				// Otherwise the person stays signed in and suspension is
 				// advisory until their session happens to expire.
-				if err := s.DeleteUserSessions(cmd.Context(), args[0]); err != nil {
+				if err := s.DeleteUserSessions(cmd.Context(), u.ID); err != nil {
 					return err
 				}
 				cmd.PrintErrln("Their sessions were closed. Their API keys still work — revoke those separately.")
 			}
-			fmt.Fprintf(cmd.OutOrStdout(), "%s is now %s\n", args[0], state)
+			fmt.Fprintf(cmd.OutOrStdout(), "%s is now %s\n", u.Email, state)
 			return nil
 		},
 	}
@@ -150,7 +159,7 @@ func newUserStateCmd(open userOpener, verb string, state store.UserState, short
 
 func newUserAdminCmd(open userOpener, verb string, admin bool, short string) *cobra.Command {
 	return &cobra.Command{
-		Use:   verb + " <user-id>",
+		Use:   verb + " <user>",
 		Short: short,
 		Long: "The dashboard is read-only, so this grants looking and nothing else.\n" +
 			"Everything that changes the cluster or issues a credential stays here on\n" +
@@ -164,11 +173,11 @@ func newUserAdminCmd(open userOpener, verb string, admin bool, short string) *co
 			}
 			defer s.Close()
 
-			if err := s.SetAdmin(cmd.Context(), args[0], admin); err != nil {
+			u, err := resolveUser(cmd.Context(), s, args[0])
+			if err != nil {
 				return err
 			}
-			u, err := s.UserByID(cmd.Context(), args[0])
-			if err != nil {
+			if err := s.SetAdmin(cmd.Context(), u.ID, admin); err != nil {
 				return err
 			}
 			if admin {
@@ -186,7 +195,7 @@ func newUserAdminCmd(open userOpener, verb string, admin bool, short string) *co
 func newUserResetPasswordCmd(open userOpener) *cobra.Command {
 	var password string
 	cmd := &cobra.Command{
-		Use:   "reset-password <user-id>",
+		Use:   "reset-password <user>",
 		Short: "Set a new password for someone who lost theirs",
 		Long: "There is no self-service reset, because this gateway sends no email and\n" +
 			"a reset link with nowhere to send it is not a reset. Set one here, tell\n" +
@@ -208,13 +217,17 @@ func newUserResetPasswordCmd(open userOpener) *cobra.Command {
 			if err != nil {
 				return err
 			}
-			if err := s.SetPassword(cmd.Context(), args[0], hash); err != nil {
+			u, err := resolveUser(cmd.Context(), s, args[0])
+			if err != nil {
 				return err
 			}
-			if err := s.DeleteUserSessions(cmd.Context(), args[0]); err != nil {
+			if err := s.SetPassword(cmd.Context(), u.ID, hash); err != nil {
 				return err
 			}
-			fmt.Fprintf(cmd.OutOrStdout(), "password set for %s\n", args[0])
+			if err := s.DeleteUserSessions(cmd.Context(), u.ID); err != nil {
+				return err
+			}
+			fmt.Fprintf(cmd.OutOrStdout(), "password set for %s\n", u.Email)
 			cmd.PrintErrln("Their sessions were closed. Their API keys still work.")
 			return nil
 		},
@@ -241,6 +254,10 @@ func newUserCreateCmd(open userOpener) *cobra.Command {
 			}
 			defer s.Close()
 
+			p, err := resolveProject(cmd.Context(), s, project)
+			if err != nil {
+				return err
+			}
 			hash, err := account.HashPassword(password)
 			if err != nil {
 				return err
@@ -253,15 +270,88 @@ func newUserCreateCmd(open userOpener) *cobra.Command {
 			if err := s.CreateUser(cmd.Context(), u); err != nil {
 				return err
 			}
-			if err := s.ApproveUser(cmd.Context(), id, project, "cli", time.Now()); err != nil {
+			if err := s.ApproveUser(cmd.Context(), id, p.ID, "cli", time.Now()); err != nil {
 				return err
 			}
 			fmt.Fprintf(cmd.OutOrStdout(), "%s\n", id)
-			cmd.PrintErrf("%s created and approved into project %s\n", args[0], project)
+			cmd.PrintErrf("%s created and approved into %s\n", args[0], p.Path)
 			return nil
 		},
 	}
-	cmd.Flags().StringVar(&project, "project", "default", "project the person's keys belong to")
+	cmd.Flags().StringVar(&project, "project", "",
+		"project the person's keys belong to: id, path or unique name (optional when there is one)")
 	cmd.Flags().StringVar(&password, "password", "", "initial password")
 	return cmd
 }
+
+func newUserMoveCmd(open userOpener) *cobra.Command {
+	var project string
+	cmd := &cobra.Command{
+		Use:   "move <user> --project <project>",
+		Short: "Put an approved account in another project, with the keys it made",
+		Long: "The person's own keys go with them, so they stop spending the old project's\n" +
+			"budget. Keys an operator issued in the old project stay there. Usage already\n" +
+			"recorded is not rewritten. A running server picks the change up when its key\n" +
+			"cache expires (auth.cache_ttl).",
+		Example: "  llmgw users move ayse@example.com --project Acme/ml/research",
+		Args:    cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			s, err := open(cmd)
+			if err != nil {
+				return err
+			}
+			defer s.Close()
+			u, err := resolveUser(cmd.Context(), s, args[0])
+			if err != nil {
+				return err
+			}
+			p, err := resolveProject(cmd.Context(), s, project)
+			if err != nil {
+				return err
+			}
+			moved, err := s.MoveUser(cmd.Context(), u.ID, p.ID)
+			if err != nil {
+				return err
+			}
+			fmt.Fprintf(cmd.OutOrStdout(), "%s is now in %s (%s moved with them)\n",
+				u.Email, p.Path, countOf(moved, "key"))
+			return nil
+		},
+	}
+	cmd.Flags().StringVar(&project, "project", "", "where they go: id, path or unique name")
+	_ = cmd.MarkFlagRequired("project")
+	return cmd
+}
+
+func newUserDeleteCmd(open userOpener) *cobra.Command {
+	var yes bool
+	cmd := &cobra.Command{
+		Use:   "delete <user>",
+		Short: "Remove an account for good",
+		Long: "Signs them out, revokes every key they made, and removes the account. The\n" +
+			"keys are kept, revoked and ownerless, because usage history refers to them.\n" +
+			"To stop an account working and keep it, use suspend.",
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			s, err := open(cmd)
+			if err != nil {
+				return err
+			}
+			defer s.Close()
+			u, err := resolveUser(cmd.Context(), s, args[0])
+			if err != nil {
+				return err
+			}
+			if !yes {
+				return fmt.Errorf("this removes %s and revokes their keys; run again with --yes", u.Email)
+			}
+			if err := s.DeleteUser(cmd.Context(), u.ID, time.Now()); err != nil {
+				return err
+			}
+			fmt.Fprintf(cmd.OutOrStdout(), "%s removed\n", u.Email)
+			return nil
+		},
+	}
+	cmd.Flags().BoolVar(&yes, "yes", false, "confirm")
+	return cmd
+}

internal/config/config.go

@@ -72,6 +72,15 @@ type Server struct {
 	// Empty means there is none, and /admin/mirror stays unmounted.
 	Mirror Mirror `koanf:"mirror"`
 
+	// Engines are the engines every node installs, and at which version. The
+	// agents install them themselves — the right build for their hardware,
+	// checked against its publisher's checksum and seen to run — so a node is
+	// never trusted to already have what somebody once put there.
+	//
+	// Unset keeps DefaultEngines. A version replaces the default; an empty
+	// one leaves that engine to be installed by hand.
+	Engines map[string]string `koanf:"engines"`
+
 	Router  Router          `koanf:"router"`
 	Control Control         `koanf:"control"`
 	Models  []catalog.Model `koanf:"models"`
@@ -531,8 +540,14 @@ func (s Server) Catalog() (*catalog.Catalog, error) {
 // A model with no profiles cannot be placed, so a configuration of bare names
 // plus a static endpoint list is the router-only mode used for testing against
 // sim-engines.
+//
+// With no static endpoints there is nothing a router-only process could serve,
+// so a configuration naming neither endpoints nor models is a control plane
+// with an empty catalog — which is what every new cluster is. It used to be
+// refused as having nothing to serve, and under systemd that is a unit
+// restarting every two seconds with nothing on 8080.
 func (s Server) ControlPlaneMode() bool {
-	if s.Control.Enabled {
+	if s.Control.Enabled || len(s.Endpoints) == 0 {
 		return true
 	}
 	for _, m := range s.Models {
@@ -542,3 +557,25 @@ func (s Server) ControlPlaneMode() bool {
 	}
 	return false
 }
+
+// DefaultEngines are the engine versions this release was checked against:
+// `llmgw ctl engine-check` accepted every flag llmgw builds for them.
+var DefaultEngines = map[string]string{
+	"llamacpp": "b10991",
+}
+
+// WantedEngines is DefaultEngines with the configuration laid over it.
+func (s Server) WantedEngines() map[string]string {
+	out := map[string]string{}
+	for k, v := range DefaultEngines {
+		out[k] = v
+	}
+	for k, v := range s.Engines {
+		if v == "" {
+			delete(out, k)
+			continue
+		}
+		out[k] = v
+	}
+	return out
+}

internal/config/config_test.go

@@ -155,19 +155,26 @@ func TestValidateAcceptsDistinctAliases(t *testing.T) {
 	}
 }
 
-func TestAControlPlaneCanBeAskedForWithNoModelsInTheFile(t *testing.T) {
-	// Without this, a deployment whose catalog lives entirely in the database
-	// cannot start: the mode is inferred from the file, an empty models block
-	// reads as "router only", and the process refuses to start for want of
-	// anything to serve — so the stored models it would have loaded are never
-	// read at all.
+func TestAnEmptyConfigurationIsAControlPlane(t *testing.T) {
+	// A new cluster has no models and no endpoints. Refusing it as having
+	// nothing to serve left a systemd unit restarting every two seconds and
+	// nothing listening on 8080 — on the first start anyone ever made.
 	var s Server
+	if !s.ControlPlaneMode() {
+		t.Fatal("a configuration with no endpoints and no models should run a control plane")
+	}
+}
+
+func TestStaticEndpointsAloneAreRouterOnly(t *testing.T) {
+	var s Server
+	s.Endpoints = []Endpoint{{ID: "e", URL: "http://127.0.0.1:1"}}
+	s.Models = []catalog.Model{{Name: "m"}}
 	if s.ControlPlaneMode() {
-		t.Fatal("an empty configuration is not a control plane")
+		t.Error("static endpoints and bare model names are the router-only mode")
 	}
 	s.Control.Enabled = true
 	if !s.ControlPlaneMode() {
-		t.Error("control.enabled should run the control plane with no models declared")
+		t.Error("control.enabled should run the control plane alongside static endpoints")
 	}
 }
 
@@ -182,3 +189,20 @@ func TestAModelWithProfilesStillMakesAControlPlane(t *testing.T) {
 		t.Error("a declared model with profiles is still a control plane")
 	}
 }
+
+func TestEnginesDefaultToTheCheckedVersions(t *testing.T) {
+	var s Server
+	if got := s.WantedEngines()["llamacpp"]; got != DefaultEngines["llamacpp"] {
+		t.Errorf("unset config wants llamacpp %q", got)
+	}
+	s.Engines = map[string]string{"llamacpp": "b11000", "vllm": "0.29.0"}
+	got := s.WantedEngines()
+	if got["llamacpp"] != "b11000" || got["vllm"] != "0.29.0" {
+		t.Errorf("configured versions not taken: %v", got)
+	}
+	// Empty leaves an engine to be installed by hand.
+	s.Engines = map[string]string{"llamacpp": ""}
+	if _, ok := s.WantedEngines()["llamacpp"]; ok {
+		t.Error("an emptied engine is still wanted")
+	}
+}

internal/control/adminapi/api.go

@@ -13,6 +13,7 @@ import (
 	"io"
 	"net/http"
 	"slices"
+	"strings"
 	"time"
 
 	"gitea.mynodes.xyz/ilker/llmgw/internal/bench"
@@ -55,6 +56,7 @@ func (a *API) Register(mux *http.ServeMux) {
 	mux.HandleFunc("POST /api/v1/weights/collect", a.collectWeights)
 	mux.HandleFunc("GET /api/v1/explain-placement", a.explainPlacement)
 	mux.HandleFunc("POST /api/v1/nodes/{id}/state", a.setNodeState)
+	mux.HandleFunc("GET /api/v1/nodes/{id}/logs", a.nodeLogs)
 }
 
 // --- responses ---
@@ -78,6 +80,15 @@ type NodeView struct {
 	// which is not the same as having none — placement treats it that way too,
 	// or a control plane upgrade would empty a fleet.
 	Engines []string `json:"engines,omitempty"`
+	// Provision is where installing each wanted engine stands. An engine is in
+	// Engines only once its installation here is ready.
+	Provision []inventory.EngineInstall `json:"provision,omitempty"`
+	// OS is the operating system and architecture, as the agent read them.
+	OS string `json:"os,omitempty"`
+	// Warnings are facts about the machine that make it slow or unusable
+	// without making it unhealthy: a VM with no vector instructions runs every
+	// model, at a fifth of the speed.
+	Warnings []string `json:"warnings,omitempty"`
 
 	// Models are the weights this node can see. Absent means it was not asked
 	// to look, which is not the same as holding none.
@@ -230,6 +241,14 @@ func (a *API) Nodes() []NodeView {
 			v.Engines = append(v.Engines, name)
 		}
 		slices.Sort(v.Engines)
+		v.Provision = n.Provision
+		if h := n.Host; h.OSID != "" {
+			v.OS = strings.TrimSpace(h.OSID + " " + h.OSVersion + " " + h.Arch)
+			if h.Arch == "amd64" && !h.HasCPUFlag("avx2") {
+				v.Warnings = append(v.Warnings, "no AVX2: inference runs several times slower "+
+					"(a VM needs CPU type \"host\")")
+			}
+		}
 		for _, w := range n.Weights {
 			v.Models = append(v.Models, w.Path)
 		}

internal/control/adminapi/logs.go

@@ -0,0 +1,60 @@
+package adminapi
+
+import (
+	"context"
+	"net/http"
+	"strconv"
+)
+
+// LogsView is one of a node's logs, as the node sent it.
+type LogsView struct {
+	Node   string   `json:"node"`
+	Source string   `json:"source"`
+	Lines  []string `json:"lines"`
+	// Sources are every log the node can show: "agent", and one per engine
+	// running there.
+	Sources []string `json:"sources"`
+	Error   string   `json:"error,omitempty"`
+}
+
+// maxLogLines bounds a request. The agent keeps two thousand lines of its own
+// and a few hundred per engine, so more would only be refused further down.
+const maxLogLines = 2000
+
+// NodeLogs asks a node for the recent lines of one of its logs.
+//
+// Exported because the cluster page renders it server-side, the same as
+// Nodes.
+func (a *API) NodeLogs(ctx context.Context, node, source string, lines int) LogsView {
+	if source == "" {
+		source = "agent"
+	}
+	v := LogsView{Node: node, Source: source}
+	lr := a.ctl.LogReader()
+	if lr == nil {
+		v.Error = "this control plane cannot read node logs"
+		return v
+	}
+	if lines <= 0 {
+		lines = 300
+	}
+	out, sources, err := lr.Logs(ctx, node, source, min(lines, maxLogLines))
+	v.Lines, v.Sources = out, sources
+	if err != nil {
+		v.Error = err.Error()
+	}
+	if len(v.Sources) == 0 {
+		v.Sources = []string{"agent"}
+	}
+	return v
+}
+
+func (a *API) nodeLogs(w http.ResponseWriter, r *http.Request) {
+	lines, _ := strconv.Atoi(r.URL.Query().Get("lines"))
+	v := a.NodeLogs(r.Context(), r.PathValue("id"), r.URL.Query().Get("source"), lines)
+	code := http.StatusOK
+	if v.Error != "" && len(v.Lines) == 0 {
+		code = http.StatusBadGateway
+	}
+	writeJSON(w, code, v)
+}

internal/control/agentsvc/convert.go

@@ -23,8 +23,15 @@ func NodeFromRegister(r *llmgwv1.Register) inventory.Node {
 		State:          inventory.Ready,
 		AgentVersion:   r.GetAgentVersion(),
 		EngineVersions: r.GetEngineVersions(),
+		EnginesProbed:  r.GetEnginesProbed(),
 		MemoryTotalMB:  int(r.GetMemoryTotalMb()),
 	}
+	if h := r.GetHost(); h != nil {
+		n.Host = discovery.Host{
+			OSID: h.GetOsId(), OSVersion: h.GetOsVersion(), OSCodename: h.GetOsCodename(),
+			Arch: h.GetArch(), CPUFlags: h.GetCpuFlags(), NVIDIADriver: h.GetNvidiaDriver(),
+		}
+	}
 	for _, w := range r.GetWeights() {
 		n.Weights = append(n.Weights, discovery.LocalWeights{
 			Path: w.GetPath(), SizeBytes: w.GetSizeBytes(), Shared: w.GetShared(),

internal/control/agentsvc/heartbeat_test.go

@@ -155,3 +155,36 @@ func TestAHeartbeatCarriesAnEngineInstalledAfterRegistration(t *testing.T) {
 		t.Errorf("the newly installed engine is not known: %v", n.EngineVersions)
 	}
 }
+
+func TestALogRequestIsAnsweredByTheNodeItWasSentTo(t *testing.T) {
+	s := newTestService()
+	c := &conn{nodeID: "n1", send: make(chan *llmgwv1.ControlMessage, 4), closed: make(chan struct{})}
+	s.conns["n1"] = c
+
+	type result struct {
+		lines, sources []string
+		err            error
+	}
+	done := make(chan result, 1)
+	go func() {
+		l, src, err := s.Logs(t.Context(), "n1", "agent", 10)
+		done <- result{l, src, err}
+	}()
+
+	req := (<-c.send).GetLogs()
+	if req.GetSource() != "agent" || req.GetLines() != 10 {
+		t.Fatalf("sent %v", req)
+	}
+	s.handle("n1", &llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_Logs{Logs: &llmgwv1.LogsReply{
+		RequestId: req.GetRequestId(), Lines: []string{"hello"}, Sources: []string{"agent", "shard-1"},
+	}}})
+
+	r := <-done
+	if r.err != nil || len(r.lines) != 1 || r.lines[0] != "hello" || len(r.sources) != 2 {
+		t.Errorf("got %+v", r)
+	}
+
+	if _, _, err := s.Logs(t.Context(), "nobody", "agent", 10); err == nil {
+		t.Error("a node with no stream answered")
+	}
+}

internal/control/agentsvc/service.go

@@ -10,6 +10,7 @@ import (
 	"net/http"
 	"slices"
 	"sync"
+	"time"
 
 	"connectrpc.com/connect"
 
@@ -49,8 +50,19 @@ type Service struct {
 
 	mu    sync.RWMutex
 	conns map[string]*conn
+
+	// engines is what every node is told to install, set once at start.
+	engines map[string]string
+
+	// logWaits are the log requests awaiting a reply, by request id.
+	logMu    sync.Mutex
+	logWaits map[string]chan *llmgwv1.LogsReply
+	logSeq   uint64
 }
 
+// SetEngines sets the engines every node installs, by name and version.
+func (s *Service) SetEngines(e map[string]string) { s.engines = maps.Clone(e) }
+
 // conn is one agent's open stream.
 type conn struct {
 	nodeID string
@@ -157,6 +169,21 @@ func (s *Service) Connect(
 		return err
 	}
 
+	// What this node should have installed. Sent on every registration, so a
+	// node rebuilt from nothing is told the same as one that has been here a
+	// month, and a changed version reaches every node on its next connect.
+	if s.engines != nil {
+		p := &llmgwv1.Provision{}
+		for _, e := range slices.Sorted(maps.Keys(s.engines)) {
+			p.Engines = append(p.Engines, &llmgwv1.EngineWant{Engine: e, Version: s.engines[e]})
+		}
+		if err := stream.Send(&llmgwv1.ControlMessage{
+			Msg: &llmgwv1.ControlMessage_Provision{Provision: p},
+		}); err != nil {
+			return err
+		}
+	}
+
 	// Recover replica state from what this node is already running, before any
 	// reconciliation pass can conclude the cluster is empty and place
 	// duplicates (plan §6.5).
@@ -255,6 +282,15 @@ func (s *Service) handle(nodeID string, msg *llmgwv1.AgentMessage) {
 				"last_output", m.Shard.GetLastOutput())
 		}
 
+	case *llmgwv1.AgentMessage_Logs:
+		s.logMu.Lock()
+		ch, ok := s.logWaits[m.Logs.GetRequestId()]
+		delete(s.logWaits, m.Logs.GetRequestId())
+		s.logMu.Unlock()
+		if ok {
+			ch <- m.Logs // buffered; the asker may have given up
+		}
+
 	case *llmgwv1.AgentMessage_Register:
 		// A re-registration on a live stream means the agent's view of its own
 		// hardware changed — a GPU fell out, say. Take it.
@@ -272,7 +308,8 @@ func (s *Service) applyHeartbeat(nodeID string, hb *llmgwv1.Heartbeat) {
 	storage := hb.GetStorage()
 	memMB := int(hb.GetMemoryTotalMb())
 	if len(used) == 0 && len(problems) == 0 && len(storage) == 0 &&
-		!hb.GetWeightsChanged() && memMB == 0 && !hb.GetEnginesChanged() {
+		!hb.GetWeightsChanged() && memMB == 0 && !hb.GetEnginesChanged() &&
+		!hb.GetProvisionChanged() {
 		return
 	}
 
@@ -339,6 +376,30 @@ func (s *Service) applyHeartbeat(nodeID string, hb *llmgwv1.Heartbeat) {
 		}
 		n.EngineVersions = vers
 	}
+	if hb.GetProvisionChanged() {
+		prev := map[string]inventory.EngineInstall{}
+		for _, p := range n.Provision {
+			prev[p.Engine] = p
+		}
+		n.Provision = n.Provision[:0]
+		for _, p := range hb.GetProvision() {
+			e := inventory.EngineInstall{
+				Engine: p.GetEngine(), Version: p.GetVersion(), Variant: p.GetVariant(),
+				State: p.GetState(), Detail: p.GetDetail(),
+			}
+			// A state change is worth a line; a download's progress is not.
+			if old, ok := prev[e.Engine]; !ok || old.State != e.State {
+				lvl := slog.LevelInfo
+				if e.State == "failed" || e.State == "unsupported" {
+					lvl = slog.LevelWarn
+				}
+				s.log.Log(context.Background(), lvl, "engine installation on a node",
+					"node", nodeID, "engine", e.Engine, "version", e.Version,
+					"variant", e.Variant, "state", e.State, "detail", e.Detail)
+			}
+			n.Provision = append(n.Provision, e)
+		}
+	}
 	s.inv.Upsert(n)
 }
 
@@ -465,3 +526,46 @@ func (s *Service) send(nodeID string, msg *llmgwv1.ControlMessage) error {
 		return fmt.Errorf("node %s is not keeping up with commands", nodeID)
 	}
 }
+
+// Logs asks a node for the recent lines of one of its logs: "agent", or a
+// shard id for that engine's output.
+//
+// It returns the lines, and every source the node can show.
+func (s *Service) Logs(ctx context.Context, nodeID, source string, lines int) ([]string, []string, error) {
+	s.logMu.Lock()
+	if s.logWaits == nil {
+		s.logWaits = map[string]chan *llmgwv1.LogsReply{}
+	}
+	s.logSeq++
+	id := fmt.Sprintf("logs-%d", s.logSeq)
+	ch := make(chan *llmgwv1.LogsReply, 1)
+	s.logWaits[id] = ch
+	s.logMu.Unlock()
+	defer func() {
+		s.logMu.Lock()
+		delete(s.logWaits, id)
+		s.logMu.Unlock()
+	}()
+
+	if source == "" {
+		source = "agent"
+	}
+	if err := s.send(nodeID, &llmgwv1.ControlMessage{Msg: &llmgwv1.ControlMessage_Logs{
+		Logs: &llmgwv1.LogsRequest{RequestId: id, Source: source, Lines: int32(lines)}, //nolint:gosec // bounded by the caller
+	}}); err != nil {
+		return nil, nil, err
+	}
+	// Bounded, because a page is waiting: an agent older than this request
+	// never answers it at all.
+	ctx, cancel := context.WithTimeout(ctx, 10*time.Second)
+	defer cancel()
+	select {
+	case r := <-ch:
+		if r.GetError() != "" {
+			return nil, r.GetSources(), errors.New(r.GetError())
+		}
+		return r.GetLines(), r.GetSources(), nil
+	case <-ctx.Done():
+		return nil, nil, fmt.Errorf("node %s did not answer; an agent older than v0.8.4 cannot", nodeID)
+	}
+}

internal/control/control.go

@@ -204,6 +204,17 @@ func (c *Controller) WeightRemover() WeightRemover {
 	return r
 }
 
+// LogReader can ask a node for the recent lines of one of its logs.
+type LogReader interface {
+	Logs(ctx context.Context, nodeID, source string, lines int) (out, sources []string, err error)
+}
+
+// LogReader returns the dispatcher's log capability, or nil.
+func (c *Controller) LogReader() LogReader {
+	r, _ := c.disp.(LogReader)
+	return r
+}
+
 // Start runs the reconciler and begins publishing endpoints.
 func (c *Controller) Start(ctx context.Context) {
 	if c.lead != nil {

internal/control/inventory/inventory.go

@@ -102,6 +102,24 @@ type Node struct {
 	// version decides which metric names and CLI flags apply (plan §14).
 	AgentVersion   string            `json:"agent_version,omitempty"`
 	EngineVersions map[string]string `json:"engine_versions,omitempty"`
+	// EnginesProbed says EngineVersions is an answer, so an empty map means
+	// "has none" rather than "an agent too old to say".
+	EnginesProbed bool `json:"engines_probed,omitempty"`
+
+	// Host is the operating system and processor, as the agent reads them.
+	Host discovery.Host `json:"host"`
+	// Provision is where installing each wanted engine stands on this node.
+	Provision []EngineInstall `json:"provision,omitempty"`
+}
+
+// EngineInstall is where one engine's installation stands on a node.
+type EngineInstall struct {
+	Engine  string `json:"engine"`
+	Version string `json:"version"`
+	Variant string `json:"variant,omitempty"`
+	// State is "installing", "ready", "failed" or "unsupported".
+	State  string `json:"state"`
+	Detail string `json:"detail,omitempty"`
 }
 
 // HasFabric reports whether the node is attached to a fabric.

internal/control/modelsvc/modelsvc.go

@@ -70,6 +70,22 @@ func (s *Service) Declared(name string) bool {
 	return false
 }
 
+// Sources maps each weights source in the catalog to the models that name it,
+// so a page listing files can say which of them are in use.
+func (s *Service) Sources(ctx context.Context) (map[string][]string, error) {
+	c, err := s.Load(ctx)
+	if err != nil {
+		return nil, err
+	}
+	out := map[string][]string{}
+	for _, m := range c.Models() {
+		if m.Source != "" {
+			out[m.Source] = append(out[m.Source], m.Name)
+		}
+	}
+	return out, nil
+}
+
 // Add puts a model in the catalog and keeps it across restarts.
 //
 // Validated by building the whole catalog before anything is written. A model

internal/control/placement/placement.go

@@ -403,9 +403,14 @@ func (s *Scheduler) eligibleNodes(p Profile, fabric string, avoid []string, reje
 		// A node that reports no engines at all is not excluded. That is what
 		// every agent older than this reports, and emptying a fleet on a
 		// control plane upgrade is a worse failure than the one being fixed.
-		if p.Engine != "" && len(n.EngineVersions) > 0 {
+		//
+		// An agent that says it probed is believed when it reports nothing:
+		// that is a node still installing, or one where installing failed,
+		// and either way not a place to start an engine. sim is built into
+		// the agent and never installed.
+		if p.Engine != "" && p.Engine != "sim" && (n.EnginesProbed || len(n.EngineVersions) > 0) {
 			if _, ok := n.EngineVersions[p.Engine]; !ok {
-				reject("does not have " + p.Engine + " installed")
+				reject("does not have " + p.Engine + " installed" + installing(n, p.Engine))
 				continue
 			}
 		}
@@ -526,3 +531,13 @@ func (s *Scheduler) claimedMemory(nodeID string) int {
 	}
 	return total
 }
+
+// installing says why an engine is missing, when the node has said.
+func installing(n inventory.Node, engine string) string {
+	for _, p := range n.Provision {
+		if p.Engine == engine && p.Detail != "" {
+			return " (" + p.State + ": " + p.Detail + ")"
+		}
+	}
+	return ""
+}

internal/control/placement/placement_test.go

@@ -797,3 +797,33 @@ func TestAProfileThatDeclaresNothingClaimsNothing(t *testing.T) {
 		t.Errorf("an undeclared neighbour should not fill the node:\n%s", res.Explain())
 	}
 }
+
+func TestANodeStillInstallingItsEngineIsNotACandidate(t *testing.T) {
+	// A node rebuilt from nothing registers before its agent has installed
+	// anything, and reports no engines. Treated as "too old to say", it was a
+	// candidate for every model at the moment it could run none of them.
+	fresh := node("fresh", "h100-80g", 8)
+	fresh.EnginesProbed = true
+	fresh.Provision = []inventory.EngineInstall{{
+		Engine: "vllm", Version: "0.29.0", State: "installing", Detail: "downloading",
+	}}
+
+	res := New(store(fresh)).Place([]Profile{{
+		Name: "tp1", Engine: "vllm", GPUClass: "h100-80g", GPUsPerNode: 1, Nodes: 1,
+	}}, nil)
+	if res.Placement != nil {
+		t.Fatal("placed on a node whose engine is not installed yet")
+	}
+	if !strings.Contains(res.Explain(), "installing: downloading") {
+		t.Errorf("the rejection should say the engine is on its way:\n%s", res.Explain())
+	}
+
+	// The simulated engine is part of the agent, so it is never installed and
+	// never missing.
+	res = New(store(fresh)).Place([]Profile{{
+		Name: "sim", Engine: "sim", GPUClass: "h100-80g", GPUsPerNode: 1, Nodes: 1,
+	}}, nil)
+	if res.Placement == nil {
+		t.Errorf("sim was refused:\n%s", res.Explain())
+	}
+}

internal/router/handlers.go

@@ -157,6 +157,8 @@ func (r *Router) Handler() http.Handler {
 				web.NavExtras{
 					Mirror: r.mirror != nil, History: r.history, Window: r.histWindow,
 				}, r.log)))
+			mux.HandleFunc("GET /admin/nodes/{id}/logs", pages.AdminPage(
+				web.NodeLogsPage(r.clusterView, false, version.Version)))
 			pages.RegisterAdminUsers(mux, r.store)
 			// Quotas are written on the same tree the accounts page builds, so
 			// they are mounted together.

internal/store/copy.go

@@ -96,6 +96,13 @@ func (s *DB) CopyTo(ctx context.Context, dst *DB) ([]CopyResult, error) {
 }
 
 // isEmpty reports whether any table has rows, and which one first does.
+// Empty reports whether the store holds no rows at all, which is what CopyTo
+// needs of its destination.
+func (s *DB) Empty(ctx context.Context) (bool, error) {
+	empty, _, err := s.isEmpty(ctx)
+	return empty, err
+}
+
 func (s *DB) isEmpty(ctx context.Context) (bool, string, error) {
 	for _, table := range copyOrder {
 		var n int64

internal/store/hierarchy.go

@@ -0,0 +1,336 @@
+package store
+
+import (
+	"context"
+	"database/sql"
+	"errors"
+	"fmt"
+	"strings"
+	"time"
+)
+
+// ErrNotEmpty means something still hangs off what was to be deleted.
+var ErrNotEmpty = errors.New("not empty")
+
+// TreeOrg, TreeTeam and TreeProject are the tenancy tree as one value, with
+// what hangs off each project counted, for a page or a command that shows it
+// whole.
+type TreeOrg struct {
+	ID, Name string
+	Teams    []TreeTeam
+}
+
+// TreeTeam is a team and its projects.
+type TreeTeam struct {
+	ID, Name string
+	Projects []TreeProject
+}
+
+// TreeProject is a project and what uses it.
+type TreeProject struct {
+	ID, Name string
+	// Users counts accounts in the project, whatever their state.
+	Users int
+	// Keys counts every key ever issued in it; ActiveKeys those not revoked.
+	Keys, ActiveKeys int
+}
+
+// Tree reads the whole tenancy tree, organizations with no teams and teams
+// with no projects included.
+func (s *DB) Tree(ctx context.Context) ([]TreeOrg, error) {
+	orgs, err := s.AllOrgs(ctx)
+	if err != nil {
+		return nil, err
+	}
+	teams, err := s.AllTeams(ctx)
+	if err != nil {
+		return nil, err
+	}
+	type proj struct {
+		TreeProject
+		teamID string
+	}
+	projects, err := queryAll(ctx, s.db, `
+		SELECT p.id, p.name, p.team_id,
+		  (SELECT COUNT(*) FROM users u WHERE u.project_id = p.id),
+		  (SELECT COUNT(*) FROM api_keys k WHERE k.project_id = p.id),
+		  (SELECT COUNT(*) FROM api_keys k WHERE k.project_id = p.id AND k.revoked_at IS NULL)
+		FROM projects p ORDER BY p.name`, nil, func(rows *sql.Rows) (proj, error) {
+		var p proj
+		return p, rows.Scan(&p.ID, &p.Name, &p.teamID, &p.Users, &p.Keys, &p.ActiveKeys)
+	})
+	if err != nil {
+		return nil, err
+	}
+
+	byTeam := map[string][]TreeProject{}
+	for _, p := range projects {
+		byTeam[p.teamID] = append(byTeam[p.teamID], p.TreeProject)
+	}
+	byOrg := map[string][]TreeTeam{}
+	for _, t := range teams {
+		byOrg[t.OrgID] = append(byOrg[t.OrgID], TreeTeam{ID: t.ID, Name: t.Name, Projects: byTeam[t.ID]})
+	}
+	out := make([]TreeOrg, 0, len(orgs))
+	for _, o := range orgs {
+		out = append(out, TreeOrg{ID: o.ID, Name: o.Name, Teams: byOrg[o.ID]})
+	}
+	return out, nil
+}
+
+// RenameOrg renames an organization. Names are unique among organizations.
+func (s *DB) RenameOrg(ctx context.Context, id, name string) error {
+	name = strings.TrimSpace(name)
+	if name == "" {
+		return errors.New("a name cannot be empty")
+	}
+	if err := s.nameIsFree(ctx,
+		`SELECT COUNT(*) FROM organizations WHERE name = ? AND id <> ?`, name, id); err != nil {
+		return err
+	}
+	return s.updateOne(ctx, `UPDATE organizations SET name = ? WHERE id = ?`, name, id)
+}
+
+// RenameTeam renames a team. Names are unique within an organization.
+func (s *DB) RenameTeam(ctx context.Context, id, name string) error {
+	name = strings.TrimSpace(name)
+	if name == "" {
+		return errors.New("a name cannot be empty")
+	}
+	if err := s.nameIsFree(ctx, `SELECT COUNT(*) FROM teams
+		WHERE name = ? AND id <> ? AND org_id = (SELECT org_id FROM teams WHERE id = ?)`,
+		name, id, id); err != nil {
+		return err
+	}
+	return s.updateOne(ctx, `UPDATE teams SET name = ? WHERE id = ?`, name, id)
+}
+
+// RenameProject renames a project. Names are unique within a team.
+func (s *DB) RenameProject(ctx context.Context, id, name string) error {
+	name = strings.TrimSpace(name)
+	if name == "" {
+		return errors.New("a name cannot be empty")
+	}
+	if err := s.nameIsFree(ctx, `SELECT COUNT(*) FROM projects
+		WHERE name = ? AND id <> ? AND team_id = (SELECT team_id FROM projects WHERE id = ?)`,
+		name, id, id); err != nil {
+		return err
+	}
+	return s.updateOne(ctx, `UPDATE projects SET name = ? WHERE id = ?`, name, id)
+}
+
+// MoveTeam puts a team, and everything under it, in another organization.
+//
+// Its limits go with it, and from now on it inherits the new organization's.
+func (s *DB) MoveTeam(ctx context.Context, teamID, orgID string) error {
+	if err := s.exists(ctx, "organization", `SELECT COUNT(*) FROM organizations WHERE id = ?`, orgID); err != nil {
+		return err
+	}
+	if err := s.nameIsFree(ctx, `SELECT COUNT(*) FROM teams
+		WHERE org_id = ? AND id <> ? AND name = (SELECT name FROM teams WHERE id = ?)`,
+		orgID, teamID, teamID); err != nil {
+		return err
+	}
+	return s.updateOne(ctx, `UPDATE teams SET org_id = ? WHERE id = ?`, orgID, teamID)
+}
+
+// MoveProject puts a project, with its people and keys, in another team.
+func (s *DB) MoveProject(ctx context.Context, projectID, teamID string) error {
+	if err := s.exists(ctx, "team", `SELECT COUNT(*) FROM teams WHERE id = ?`, teamID); err != nil {
+		return err
+	}
+	if err := s.nameIsFree(ctx, `SELECT COUNT(*) FROM projects
+		WHERE team_id = ? AND id <> ? AND name = (SELECT name FROM projects WHERE id = ?)`,
+		teamID, projectID, projectID); err != nil {
+		return err
+	}
+	return s.updateOne(ctx, `UPDATE projects SET team_id = ? WHERE id = ?`, teamID, projectID)
+}
+
+// DeleteOrg removes an organization with no teams, and its limits.
+func (s *DB) DeleteOrg(ctx context.Context, id string) error {
+	return s.deleteEmpty(ctx, ScopeOrg, id, `DELETE FROM organizations WHERE id = ?`,
+		emptyCheck{`SELECT COUNT(*) FROM teams WHERE org_id = ?`, "team"})
+}
+
+// DeleteTeam removes a team with no projects, and its limits.
+func (s *DB) DeleteTeam(ctx context.Context, id string) error {
+	return s.deleteEmpty(ctx, ScopeTeam, id, `DELETE FROM teams WHERE id = ?`,
+		emptyCheck{`SELECT COUNT(*) FROM projects WHERE team_id = ?`, "project"})
+}
+
+// DeleteProject removes a project nobody is in and no key was ever issued in.
+//
+// A revoked key still counts. Usage history names its key and project, and a
+// project deleted from under that history leaves rows that belong to nothing;
+// such a project can be renamed or left, not removed.
+func (s *DB) DeleteProject(ctx context.Context, id string) error {
+	return s.deleteEmpty(ctx, ScopeProject, id, `DELETE FROM projects WHERE id = ?`,
+		emptyCheck{`SELECT COUNT(*) FROM users WHERE project_id = ?`, "account"},
+		emptyCheck{`SELECT COUNT(*) FROM api_keys WHERE project_id = ?`, "key"})
+}
+
+type emptyCheck struct {
+	query, noun string
+}
+
+func (s *DB) deleteEmpty(ctx context.Context, scope Scope, id, del string, checks ...emptyCheck) error {
+	return s.inTx(ctx, func(c conn) error {
+		var busy []string
+		for _, ch := range checks {
+			var n int
+			if err := c.QueryRowContext(ctx, ch.query, id).Scan(&n); err != nil {
+				return err
+			}
+			if n > 0 {
+				busy = append(busy, plural(n, ch.noun))
+			}
+		}
+		if len(busy) > 0 {
+			return fmt.Errorf("%w: it still has %s", ErrNotEmpty, strings.Join(busy, " and "))
+		}
+		if _, err := c.ExecContext(ctx,
+			`DELETE FROM limits WHERE scope_type = ? AND scope_id = ?`, string(scope), id); err != nil {
+			return err
+		}
+		res, err := c.ExecContext(ctx, del, id)
+		if err != nil {
+			return err
+		}
+		if n, _ := res.RowsAffected(); n == 0 {
+			return ErrNotFound
+		}
+		return nil
+	})
+}
+
+// MoveUser puts an approved account in another project, with the keys it
+// made for itself, and returns how many keys moved.
+//
+// The keys move because they are the person's: left behind they would keep
+// spending the old project's budget under the new project's name. Keys an
+// operator issued in the project belong to the project and stay. Usage already
+// recorded stays where it was spent.
+func (s *DB) MoveUser(ctx context.Context, userID, projectID string) (int, error) {
+	if err := s.exists(ctx, "project", `SELECT COUNT(*) FROM projects WHERE id = ?`, projectID); err != nil {
+		return 0, err
+	}
+	var moved int
+	err := s.inTx(ctx, func(c conn) error {
+		var state string
+		if err := c.QueryRowContext(ctx, `SELECT state FROM users WHERE id = ?`, userID).Scan(&state); err != nil {
+			if errors.Is(err, sql.ErrNoRows) {
+				return ErrNotFound
+			}
+			return err
+		}
+		if UserState(state) == UserPending {
+			return errors.New("this account is waiting for approval; approve it into a project instead")
+		}
+		if _, err := c.ExecContext(ctx, `UPDATE users SET project_id = ? WHERE id = ?`, projectID, userID); err != nil {
+			return err
+		}
+		res, err := c.ExecContext(ctx, `UPDATE api_keys SET project_id = ? WHERE user_id = ?`, projectID, userID)
+		if err != nil {
+			return err
+		}
+		n, _ := res.RowsAffected()
+		moved = int(n)
+		return nil
+	})
+	return moved, err
+}
+
+// DeleteUser removes an account: its sessions end, its keys are revoked, and
+// the row goes.
+//
+// The keys stay, revoked and without an owner, because usage history refers
+// to them; the account itself is what the person asked to have removed.
+func (s *DB) DeleteUser(ctx context.Context, userID string, at time.Time) error {
+	return s.inTx(ctx, func(c conn) error {
+		if _, err := c.ExecContext(ctx, `DELETE FROM sessions WHERE user_id = ?`, userID); err != nil {
+			return err
+		}
+		if _, err := c.ExecContext(ctx, `UPDATE api_keys
+			SET revoked_at = COALESCE(revoked_at, ?), user_id = NULL WHERE user_id = ?`,
+			millis(at), userID); err != nil {
+			return err
+		}
+		res, err := c.ExecContext(ctx, `DELETE FROM users WHERE id = ?`, userID)
+		if err != nil {
+			return err
+		}
+		if n, _ := res.RowsAffected(); n == 0 {
+			return ErrNotFound
+		}
+		return nil
+	})
+}
+
+// KeyRow is a key with where it sits and who made it, for an administrator's
+// list of every key.
+type KeyRow struct {
+	Key
+	ProjectPath string
+	// OwnerEmail is empty for a key an operator issued in the project.
+	OwnerEmail string
+}
+
+// AllKeys lists every key, newest first, with its project's path and owner.
+func (s *DB) AllKeys(ctx context.Context) ([]KeyRow, error) {
+	cols := make([]string, 0, 11)
+	for _, c := range strings.Split(keyColumns, ",") {
+		cols = append(cols, "k."+strings.TrimSpace(c))
+	}
+	return queryAll(ctx, s.db, `
+		SELECT `+strings.Join(cols, ", ")+`,
+		  o.name || ' / ' || t.name || ' / ' || p.name, COALESCE(u.email, '')
+		FROM api_keys k
+		JOIN projects p ON p.id = k.project_id
+		JOIN teams t ON t.id = p.team_id
+		JOIN organizations o ON o.id = t.org_id
+		LEFT JOIN users u ON u.id = k.user_id
+		ORDER BY k.created_at DESC`, nil, func(rows *sql.Rows) (KeyRow, error) {
+		var r KeyRow
+		k, err := scanKey(keyAndMore{rows, []any{&r.ProjectPath, &r.OwnerEmail}})
+		r.Key = k
+		return r, err
+	})
+}
+
+// keyAndMore lets scanKey read a row that carries columns after the key's.
+type keyAndMore struct {
+	sc   scanner
+	more []any
+}
+
+func (k keyAndMore) Scan(dest ...any) error { return k.sc.Scan(append(dest, k.more...)...) }
+
+func (s *DB) updateOne(ctx context.Context, q string, args ...any) error {
+	res, err := s.db.ExecContext(ctx, q, args...)
+	if err != nil {
+		return err
+	}
+	if n, _ := res.RowsAffected(); n == 0 {
+		return ErrNotFound
+	}
+	return nil
+}
+
+func (s *DB) exists(ctx context.Context, noun, q string, id string) error {
+	var n int
+	if err := s.db.QueryRowContext(ctx, q, id).Scan(&n); err != nil {
+		return err
+	}
+	if n == 0 {
+		return fmt.Errorf("no %s %q", noun, id)
+	}
+	return nil
+}
+
+func plural(n int, noun string) string {
+	if n == 1 {
+		return "1 " + noun
+	}
+	return fmt.Sprintf("%d %ss", n, noun)
+}

internal/store/hierarchy_test.go

@@ -0,0 +1,253 @@
+package store
+
+import (
+	"context"
+	"errors"
+	"strings"
+	"testing"
+	"time"
+)
+
+// tree builds Acme/platform/{gateway,search} and Scania/ml, and an active
+// user in gateway with one key of their own and one operator key there.
+type fixture struct {
+	s                          *DB
+	acme, scania               string
+	platform, ml               string
+	gateway, search, research  string
+	user, userKey, operatorKey string
+}
+
+func newFixture(t *testing.T) fixture {
+	t.Helper()
+	ctx := t.Context()
+	f := fixture{s: tenancyStore(t)}
+	must := func(id string, err error) string {
+		t.Helper()
+		if err != nil {
+			t.Fatal(err)
+		}
+		return id
+	}
+	f.acme = must(f.s.CreateOrg(ctx, "Acme"))
+	f.scania = must(f.s.CreateOrg(ctx, "Scania"))
+	f.platform = must(f.s.CreateTeam(ctx, f.acme, "platform"))
+	f.ml = must(f.s.CreateTeam(ctx, f.scania, "ml"))
+	f.gateway = must(f.s.CreateProject(ctx, f.platform, "gateway"))
+	f.search = must(f.s.CreateProject(ctx, f.platform, "search"))
+	f.research = must(f.s.CreateProject(ctx, f.ml, "research"))
+
+	f.user = "usr-1"
+	if err := f.s.CreateUser(ctx, User{ID: f.user, Email: "a@x", Name: "a", PasswordHash: "h",
+		State: UserPending, CreatedAt: time.Now()}); err != nil {
+		t.Fatal(err)
+	}
+	if err := f.s.ApproveUser(ctx, f.user, f.gateway, "test", time.Now()); err != nil {
+		t.Fatal(err)
+	}
+	_, k, _ := NewKey("key-u", "mine", f.gateway)
+	if err := f.s.CreateUserKey(ctx, f.user, k); err != nil {
+		t.Fatal(err)
+	}
+	f.userKey = k.ID
+	_, k, _ = NewKey("key-o", "ci", f.gateway)
+	if err := f.s.CreateKey(ctx, k); err != nil {
+		t.Fatal(err)
+	}
+	f.operatorKey = k.ID
+	return f
+}
+
+func TestTreeCountsWhatUsesEachProject(t *testing.T) {
+	f := newFixture(t)
+	tree, err := f.s.Tree(t.Context())
+	if err != nil {
+		t.Fatal(err)
+	}
+	if len(tree) != 2 || tree[0].Name != "Acme" || len(tree[0].Teams) != 1 {
+		t.Fatalf("tree = %+v", tree)
+	}
+	gw := tree[0].Teams[0].Projects[0]
+	if gw.Name != "gateway" || gw.Users != 1 || gw.Keys != 2 || gw.ActiveKeys != 2 {
+		t.Errorf("gateway = %+v", gw)
+	}
+}
+
+func TestMovingAUserTakesTheirKeysAndLeavesTheProjects(t *testing.T) {
+	f := newFixture(t)
+	ctx := t.Context()
+	moved, err := f.s.MoveUser(ctx, f.user, f.research)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if moved != 1 {
+		t.Errorf("moved %d keys, want the one the person made", moved)
+	}
+	u, _ := f.s.UserByID(ctx, f.user)
+	if u.ProjectID != f.research {
+		t.Errorf("user is in %s", u.ProjectID)
+	}
+	keys, _ := f.s.Keys(ctx, f.gateway)
+	if len(keys) != 1 || keys[0].ID != f.operatorKey {
+		t.Errorf("the operator's key should stay in the project: %+v", keys)
+	}
+}
+
+func TestAPendingAccountIsApprovedNotMoved(t *testing.T) {
+	f := newFixture(t)
+	ctx := t.Context()
+	_ = f.s.CreateUser(ctx, User{ID: "usr-2", Email: "b@x", Name: "b", PasswordHash: "h",
+		State: UserPending, CreatedAt: time.Now()})
+	if _, err := f.s.MoveUser(ctx, "usr-2", f.gateway); err == nil {
+		t.Error("moving a pending account should be refused")
+	}
+}
+
+func TestAnAccountWithNoProjectCannotBeMadeActive(t *testing.T) {
+	f := newFixture(t)
+	ctx := t.Context()
+	_ = f.s.CreateUser(ctx, User{ID: "usr-2", Email: "b@x", Name: "b", PasswordHash: "h",
+		State: UserPending, CreatedAt: time.Now()})
+	// Rejected, then reinstated: the path that used to produce an active
+	// account whose first key failed.
+	if err := f.s.SetUserState(ctx, "usr-2", UserSuspended); err != nil {
+		t.Fatal(err)
+	}
+	if err := f.s.SetUserState(ctx, "usr-2", UserActive); err == nil {
+		t.Error("an account with no project was made active")
+	}
+}
+
+func TestRenamingKeepsNamesUniqueAmongSiblingsOnly(t *testing.T) {
+	f := newFixture(t)
+	ctx := t.Context()
+	if err := f.s.RenameProject(ctx, f.search, "gateway"); !errors.Is(err, ErrDuplicateName) {
+		t.Errorf("a sibling's name was accepted: %v", err)
+	}
+	// research is in another team, so "gateway" is free there.
+	if err := f.s.RenameProject(ctx, f.research, "gateway"); err != nil {
+		t.Error(err)
+	}
+	if err := f.s.RenameOrg(ctx, f.acme, "Acme AB"); err != nil {
+		t.Error(err)
+	}
+	if err := f.s.RenameTeam(ctx, f.platform, "  "); err == nil {
+		t.Error("an empty name was accepted")
+	}
+}
+
+func TestMovingATeamOrProjectChecksTheNewParent(t *testing.T) {
+	f := newFixture(t)
+	ctx := t.Context()
+	if err := f.s.MoveProject(ctx, f.search, f.ml); err != nil {
+		t.Fatal(err)
+	}
+	projects, _ := f.s.AllProjects(ctx)
+	var found bool
+	for _, p := range projects {
+		if p.ID == f.search && p.Path == "Scania / ml / search" {
+			found = true
+		}
+	}
+	if !found {
+		t.Errorf("search did not move: %+v", projects)
+	}
+	if err := f.s.MoveTeam(ctx, f.platform, "org-nope"); err == nil {
+		t.Error("moved into an organization that does not exist")
+	}
+	// A team named ml already exists in Scania.
+	if _, err := f.s.CreateTeam(ctx, f.acme, "ml"); err != nil {
+		t.Fatal(err)
+	}
+	teams, _ := f.s.AllTeams(ctx)
+	for _, tm := range teams {
+		if tm.OrgID == f.acme && tm.Name == "ml" {
+			if err := f.s.MoveTeam(ctx, tm.ID, f.scania); !errors.Is(err, ErrDuplicateName) {
+				t.Errorf("two teams named ml in one organization: %v", err)
+			}
+		}
+	}
+}
+
+func TestOnlyEmptyThingsAreDeleted(t *testing.T) {
+	f := newFixture(t)
+	ctx := t.Context()
+	err := f.s.DeleteProject(ctx, f.gateway)
+	if !errors.Is(err, ErrNotEmpty) || !strings.Contains(err.Error(), "1 account and 2 keys") {
+		t.Errorf("deleting a used project: %v", err)
+	}
+	if err := f.s.DeleteTeam(ctx, f.platform); !errors.Is(err, ErrNotEmpty) {
+		t.Errorf("deleting a team with projects: %v", err)
+	}
+	if err := f.s.DeleteProject(ctx, f.search); err != nil {
+		t.Errorf("deleting an unused project: %v", err)
+	}
+	if err := f.s.DeleteProject(ctx, f.research); err != nil {
+		t.Fatal(err)
+	}
+	if err := f.s.DeleteTeam(ctx, f.ml); err != nil {
+		t.Fatal(err)
+	}
+	if err := f.s.DeleteOrg(ctx, f.scania); err != nil {
+		t.Fatal(err)
+	}
+	if err := f.s.DeleteOrg(ctx, f.scania); !errors.Is(err, ErrNotFound) {
+		t.Errorf("deleting twice: %v", err)
+	}
+}
+
+func TestDeletingAUserRevokesTheirKeysAndKeepsThem(t *testing.T) {
+	f := newFixture(t)
+	ctx := t.Context()
+	if err := f.s.DeleteUser(ctx, f.user, time.Now()); err != nil {
+		t.Fatal(err)
+	}
+	if _, err := f.s.UserByID(ctx, f.user); !errors.Is(err, ErrNotFound) {
+		t.Errorf("user still there: %v", err)
+	}
+	keys, _ := f.s.Keys(ctx, f.gateway)
+	for _, k := range keys {
+		if k.ID == f.userKey && k.RevokedAt.IsZero() {
+			t.Error("the deleted person's key still works")
+		}
+		if k.ID == f.operatorKey && !k.RevokedAt.IsZero() {
+			t.Error("an operator key was revoked with the person")
+		}
+	}
+	if len(keys) != 2 {
+		t.Errorf("keys are kept for usage history; have %d", len(keys))
+	}
+}
+
+func TestAllKeysNamesProjectAndOwner(t *testing.T) {
+	f := newFixture(t)
+	rows, err := f.s.AllKeys(t.Context())
+	if err != nil {
+		t.Fatal(err)
+	}
+	if len(rows) != 2 {
+		t.Fatalf("rows = %+v", rows)
+	}
+	for _, r := range rows {
+		if r.ProjectPath != "Acme / platform / gateway" {
+			t.Errorf("path %q", r.ProjectPath)
+		}
+		if (r.ID == f.userKey) != (r.OwnerEmail == "a@x") {
+			t.Errorf("owner of %s is %q", r.ID, r.OwnerEmail)
+		}
+	}
+}
+
+func TestRedeclaringATeamDoesNotMoveIt(t *testing.T) {
+	f := newFixture(t)
+	err := f.s.PutTeam(context.Background(), Team{ID: f.platform, OrgID: f.scania, Name: "platform"})
+	if err == nil {
+		t.Fatal("PutTeam moved a team to another organization")
+	}
+	teams, _ := f.s.AllTeams(context.Background())
+	for _, tm := range teams {
+		if tm.ID == f.platform && tm.OrgID != f.acme {
+			t.Error("the team changed owner")
+		}
+	}
+}

internal/store/sqlite.go

@@ -96,17 +96,31 @@ func (s *DB) PutOrg(ctx context.Context, o Org) error {
 }
 
 // PutTeam upserts a team.
+//
+// It will rename a team but not move it: an upsert that rewrote the parent
+// let `keys init --org B --team platform` take organization A's team without
+// a word. Moving is MoveTeam, asked for by name.
 func (s *DB) PutTeam(ctx context.Context, t Team) error {
-	_, err := s.db.ExecContext(ctx,
+	var org string
+	err := s.db.QueryRowContext(ctx, `SELECT org_id FROM teams WHERE id = ?`, t.ID).Scan(&org)
+	if err == nil && org != t.OrgID {
+		return fmt.Errorf("team %s belongs to organization %s; move it rather than redeclare it", t.ID, org)
+	}
+	_, err = s.db.ExecContext(ctx,
 		`INSERT INTO teams (id, org_id, name) VALUES (?, ?, ?)
 		 ON CONFLICT (id) DO UPDATE SET org_id = excluded.org_id, name = excluded.name`,
 		t.ID, t.OrgID, t.Name)
 	return err
 }
 
-// PutProject upserts a project.
+// PutProject upserts a project. Like PutTeam, it renames but never moves.
 func (s *DB) PutProject(ctx context.Context, p Project) error {
-	_, err := s.db.ExecContext(ctx,
+	var team string
+	err := s.db.QueryRowContext(ctx, `SELECT team_id FROM projects WHERE id = ?`, p.ID).Scan(&team)
+	if err == nil && team != p.TeamID {
+		return fmt.Errorf("project %s belongs to team %s; move it rather than redeclare it", p.ID, team)
+	}
+	_, err = s.db.ExecContext(ctx,
 		`INSERT INTO projects (id, team_id, name) VALUES (?, ?, ?)
 		 ON CONFLICT (id) DO UPDATE SET team_id = excluded.team_id, name = excluded.name`,
 		p.ID, p.TeamID, p.Name)

internal/store/users.go

@@ -120,7 +120,19 @@ func (s *DB) ApproveUser(ctx context.Context, id, projectID, by string, at time.
 }
 
 // SetUserState suspends or reinstates an account.
+//
+// An account with no project cannot be made active. A pending account that
+// was rejected and then reinstated used to become exactly that: able to sign
+// in, and failing with "something went wrong" at the first key it made.
+// Approving is what gives it a project.
 func (s *DB) SetUserState(ctx context.Context, id string, state UserState) error {
+	if state == UserActive {
+		var project sql.NullString
+		if err := s.db.QueryRowContext(ctx, `SELECT project_id FROM users WHERE id = ?`, id).
+			Scan(&project); err == nil && !project.Valid {
+			return errors.New("this account has no project; approve it into one instead")
+		}
+	}
 	res, err := s.db.ExecContext(ctx, `UPDATE users SET state = ? WHERE id = ?`, string(state), id)
 	if err != nil {
 		return err

internal/web/admin_mirror.go

@@ -70,6 +70,14 @@ type mirrorRow struct {
 	// then not drawn at all: a progress bar that has to invent its own total is
 	// worse than a byte count that only goes up.
 	HasPercent bool
+
+	// Files is how many files a held version has. A version of several is one
+	// row — a fifteen-shard model is one thing to serve or remove, and fifteen
+	// rows of it pushed everything else off the page.
+	Files int
+	// UsedBy names the catalog models whose source is this row's address, so
+	// a file that is in use does not look like one waiting to be served.
+	UsedBy []string
 }
 
 type adminMirrorData struct {
@@ -276,25 +284,46 @@ func (p *AccountPages) renderMirror(w http.ResponseWriter, r *http.Request,
 	}
 
 	d.Base = got.Base
+	var sources map[string][]string
+	if p.catalog != nil {
+		if sources, err = p.catalog.Sources(r.Context()); err != nil {
+			p.log.Warn("could not read which models use the mirror", "err", err)
+		}
+	}
+	held := map[string]int{} // model/version -> index of its row
 	for _, st := range got.Models {
-		row := mirrorRow{
-			Status: st,
-			URL: mirror.Entry{Model: st.Model, Version: st.Version, File: st.File}.
-				URL(got.Base),
-		}
-		if pc, ok := st.Percent(); ok {
-			row.Percent, row.HasPercent = pc, true
-		}
 		switch st.State {
 		case mirror.Fetching:
 			d.Fetching++
 		case mirror.Failed:
 			d.Failed++
 		default:
-			d.Present++
+			key := st.Model + "/" + st.Version
+			if i, ok := held[key]; ok {
+				// Another file of a version already shown: counted into it.
+				d.Rows[i].Files++
+				d.Rows[i].BytesDone += st.BytesDone
+				d.Rows[i].URL = strings.TrimSuffix(
+					mirror.Entry{Model: st.Model, Version: st.Version}.URL(got.Base), "/") + "/"
+				continue
+			}
+			held[key] = len(d.Rows)
+			d.Present++ // versions, not files: what the rows below count
+		}
+		row := mirrorRow{
+			Status: st,
+			Files:  1,
+			URL: mirror.Entry{Model: st.Model, Version: st.Version, File: st.File}.
+				URL(got.Base),
+		}
+		if pc, ok := st.Percent(); ok {
+			row.Percent, row.HasPercent = pc, true
 		}
 		d.Rows = append(d.Rows, row)
 	}
+	for i := range d.Rows {
+		d.Rows[i].UsedBy = sources[d.Rows[i].URL]
+	}
 
 	// The one row whose serve form is open, and only when the catalog can be
 	// changed at all.

internal/web/admin_mirror_test.go

@@ -331,7 +331,7 @@ func TestAFailedRowOffersRetryRatherThanRemove(t *testing.T) {
 	if strings.Contains(body, "/admin/mirror/m/v/remove") {
 		t.Error("a failed row should not offer a remove that can only 404")
 	}
-	if !strings.Contains(body, "Retry") {
+	if !strings.Contains(body, ">retry<") {
 		t.Error("a failed row should offer to start the download again")
 	}
 	if !strings.Contains(body, `value="http://upstream/model.gguf"`) {
@@ -535,3 +535,35 @@ func TestAlternativeQuantisationsStaySeparateRows(t *testing.T) {
 		t.Errorf("expected a form each plus the manual one, found %d", n)
 	}
 }
+
+func TestAHeldVersionIsOneLineAndSaysWhoUsesIt(t *testing.T) {
+	m := &fakeMirror{models: []mirror.Status{
+		{Model: "big", Version: "v1", File: "a-00001-of-00002.gguf", State: mirror.Present, BytesDone: 1 << 30},
+		{Model: "big", Version: "v1", File: "a-00002-of-00002.gguf", State: mirror.Present, BytesDone: 1 << 30},
+		{Model: "small", Version: "v1", File: "s.gguf", State: mirror.Present, BytesDone: 1 << 20},
+	}}
+	cat := &fakeCatalog{sources: map[string][]string{"http://mirror:8081/small/v1/s.gguf": {"small-chat"}}}
+	body := mirrorGet(t, mirrorPageDeps(t, MirrorDeps{Mirror: m, Catalog: cat}, true),
+		"/admin/mirror").Body.String()
+
+	if n := strings.Count(body, "/admin/mirror/big/v1/remove"); n != 1 {
+		t.Errorf("a two-file version should be one row, found %d", n)
+	}
+	if !strings.Contains(body, "2 files") || !strings.Contains(body, "2.0 GiB") {
+		t.Error("the row should count its files and add up their size")
+	}
+	if !strings.Contains(body, `data-url="http://mirror:8081/big/v1/"`) {
+		t.Error("a set's address is its directory, trailing slash included")
+	}
+	if !strings.Contains(body, ">small-chat<") {
+		t.Error("the model serving a file should be named on its row")
+	}
+	// Already served: offering to serve it again invites a second catalog
+	// entry for the same weights.
+	if strings.Contains(body, "serve=small:v1") {
+		t.Error("a file in use should not be offered for serving")
+	}
+	if !strings.Contains(body, "serve=big:v1") {
+		t.Error("a file nobody uses should be offered for serving")
+	}
+}

internal/web/admin_serve.go

@@ -15,6 +15,8 @@ import (
 type CatalogAdmin interface {
 	Add(ctx context.Context, m catalog.Model, by string) error
 	Declared(name string) bool
+	// Sources maps a weights source to the catalog models that use it.
+	Sources(ctx context.Context) (map[string][]string, error)
 }
 
 // serveForm is the offer to put one mirrored version into service.

internal/web/admin_serve_test.go

@@ -17,6 +17,7 @@ import (
 type fakeCatalog struct {
 	added    []catalog.Model
 	declared string
+	sources  map[string][]string
 	err      error
 }
 
@@ -28,6 +29,9 @@ func (f *fakeCatalog) Add(_ context.Context, m catalog.Model, _ string) error {
 	return nil
 }
 func (f *fakeCatalog) Declared(name string) bool { return name == f.declared }
+func (f *fakeCatalog) Sources(context.Context) (map[string][]string, error) {
+	return f.sources, nil
+}
 
 // fleet reports a cluster with a given number of cards and engines.
 type fakeFleet struct {

internal/web/admin_tenancy_test.go

@@ -60,6 +60,26 @@ func (f *fakeAdmin) CreateProject(_ context.Context, teamID, name string) (strin
 	return "proj-1", f.err
 }
 
+func (f *fakeAdmin) Tree(context.Context) ([]store.TreeOrg, error) {
+	var out []store.TreeOrg
+	for _, o := range f.orgs {
+		out = append(out, store.TreeOrg{ID: o.ID, Name: o.Name})
+	}
+	return out, nil
+}
+func (f *fakeAdmin) RenameOrg(context.Context, string, string) error       { return f.err }
+func (f *fakeAdmin) RenameTeam(context.Context, string, string) error      { return f.err }
+func (f *fakeAdmin) RenameProject(context.Context, string, string) error   { return f.err }
+func (f *fakeAdmin) MoveTeam(context.Context, string, string) error        { return f.err }
+func (f *fakeAdmin) MoveProject(context.Context, string, string) error     { return f.err }
+func (f *fakeAdmin) DeleteOrg(context.Context, string) error               { return f.err }
+func (f *fakeAdmin) DeleteTeam(context.Context, string) error              { return f.err }
+func (f *fakeAdmin) DeleteProject(context.Context, string) error           { return f.err }
+func (f *fakeAdmin) MoveUser(context.Context, string, string) (int, error) { return 0, f.err }
+func (f *fakeAdmin) DeleteUser(context.Context, string, time.Time) error   { return f.err }
+func (f *fakeAdmin) AllKeys(context.Context) ([]store.KeyRow, error)       { return nil, nil }
+func (f *fakeAdmin) RevokeKey(context.Context, string, time.Time) error    { return f.err }
+
 func tenancyPage(t *testing.T, s *fakeAdmin, admin bool) *http.ServeMux {
 	t.Helper()
 	a := fakeAccounts{user: store.User{ID: "u1", Email: "a@example.com", IsAdmin: admin}}
@@ -156,16 +176,17 @@ func TestThePageSaysWhatToDoWhenTheTreeIsEmpty(t *testing.T) {
 	}
 }
 
-func TestQuotasAreStillNotSettableFromTheBrowser(t *testing.T) {
+func TestThisPageSetsNoQuotas(t *testing.T) {
 	body := mirrorGet(t, tenancyPage(t, &fakeAdmin{}, true), "/admin/users").Body.String()
 
-	// Making somewhere to put people is structural; raising what they may spend
-	// is financial. A stolen admin session should be able to do the first and
-	// not the second, and the page says so.
-	if !strings.Contains(body, "llmgw keys limits") {
-		t.Error("the page should still send quota changes to the shell")
-	}
+	// Quotas have their own page; this one arranges people. What stays off
+	// the web entirely is what would hand a stolen session more access.
 	if strings.Contains(body, `name="budget"`) {
-		t.Error("a budget field has appeared on the accounts page")
+		t.Error("a budget field has appeared on the people page")
+	}
+	for _, shell := range []string{"llmgw users promote", "reset-password", "llmgw keys create"} {
+		if !strings.Contains(body, shell) {
+			t.Errorf("the page should send %q to the shell", shell)
+		}
 	}
 }

internal/web/admin_users.go

@@ -3,7 +3,9 @@ package web
 import (
 	"context"
 	"errors"
+	"fmt"
 	"net/http"
+	"strings"
 	"time"
 
 	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
@@ -26,36 +28,77 @@ type AccountAdmin interface {
 	CreateOrg(ctx context.Context, name string) (string, error)
 	CreateTeam(ctx context.Context, orgID, name string) (string, error)
 	CreateProject(ctx context.Context, teamID, name string) (string, error)
+
+	// And keeping it in shape afterwards, which used to have no interface at
+	// all: nothing could be renamed, moved or removed, and an approved person
+	// stayed in the project they were first put in.
+	Tree(ctx context.Context) ([]store.TreeOrg, error)
+	RenameOrg(ctx context.Context, id, name string) error
+	RenameTeam(ctx context.Context, id, name string) error
+	RenameProject(ctx context.Context, id, name string) error
+	MoveTeam(ctx context.Context, teamID, orgID string) error
+	MoveProject(ctx context.Context, projectID, teamID string) error
+	DeleteOrg(ctx context.Context, id string) error
+	DeleteTeam(ctx context.Context, id string) error
+	DeleteProject(ctx context.Context, id string) error
+	MoveUser(ctx context.Context, userID, projectID string) (int, error)
+	DeleteUser(ctx context.Context, userID string, at time.Time) error
+	AllKeys(ctx context.Context) ([]store.KeyRow, error)
+	RevokeKey(ctx context.Context, id string, at time.Time) error
+}
+
+// userRow is an account as the page shows it: with where it is, not an id.
+type userRow struct {
+	store.User
+	ProjectPath string
+}
+
+// keyRow is a key as the page shows it.
+type keyRow struct {
+	store.KeyRow
+	State string
 }
 
 type adminUsersData struct {
 	Pending  []store.User
-	Active   []store.User
+	Accounts []userRow
+	Tree     []store.TreeOrg
+	// People and Keys are keyed by project id.
+	People   map[string][]userRow
+	Keys     map[string][]keyRow
 	Projects []store.ProjectPath
 	Orgs     []store.Org
 	Teams    []store.TeamPath
-	Message  string
-	Error    string
-	CSRF     string
-	Version  string
-	Now      string
+	// Open is the project whose section is shown expanded: the one just acted
+	// on, so the result of a change is in view without the reader hunting for
+	// it. Scripted pages keep every open section open; this is for the rest.
+	Open    string
+	AdminID string
+	Message string
+	Error   string
+	CSRF    string
+	Version string
+	Now     string
 	// HasMirror and HasQuotas keep the nav from offering a page that was never
 	// mounted.
 	HasMirror bool
 	HasQuotas bool
 }
 
-// RegisterAdminUsers mounts account administration under /admin/users.
+// RegisterAdminUsers mounts account and tenancy administration under
+// /admin/users.
 //
-// What is here and what is not is the whole design. Approving and suspending
-// are the work an operator does constantly and should not need a terminal for.
-// Making someone an admin, resetting a password and issuing keys on another
-// person's behalf are not here: each of them turns a stolen admin session into
-// more than it was — more access, or somebody else's account — and each is rare
-// enough that a shell is no hardship.
+// What is here and what is not is the whole design. Approving, suspending,
+// moving people and keeping the tree in shape are the work an operator does
+// constantly and should not need a terminal for. Making someone an admin,
+// resetting a password and issuing keys on another person's behalf are not
+// here: each of them turns a stolen admin session into more than it was —
+// more access, or somebody else's account — and each is rare enough that a
+// shell is no hardship. Revoking a key is here, because it only ever takes
+// access away.
 func (p *AccountPages) RegisterAdminUsers(mux *http.ServeMux, s AccountAdmin) {
 	mux.HandleFunc("GET /admin/users", p.adminOnly(func(w http.ResponseWriter, r *http.Request, admin store.User) {
-		p.renderUsers(w, r, s, "", "")
+		p.renderUsers(w, r, s, admin, "", "", r.URL.Query().Get("open"))
 	}))
 
 	mux.HandleFunc("POST /admin/users/{id}/approve", p.adminAction(s,
@@ -67,7 +110,7 @@ func (p *AccountPages) RegisterAdminUsers(mux *http.ServeMux, s AccountAdmin) {
 			if err := s.ApproveUser(ctx, id, project, admin.Email, time.Now()); err != nil {
 				return "", err
 			}
-			return "approved into " + project, nil
+			return "approved into " + projectPath(ctx, s, project), nil
 		}))
 
 	mux.HandleFunc("POST /admin/users/{id}/suspend", p.adminAction(s,
@@ -88,17 +131,47 @@ func (p *AccountPages) RegisterAdminUsers(mux *http.ServeMux, s AccountAdmin) {
 			return "suspended, and signed out everywhere", nil
 		}))
 
-	// The tenancy tree. Creating a container is not granting access: nobody can
-	// use one until an account is approved into it, which is the step above and
-	// already lives here. What is still not here is limits — raising a budget
-	// is a financial act, and a stolen session must not be able to perform one.
-	mux.HandleFunc("POST /admin/tenancy/org", p.tenancyAction(s,
-		func(ctx context.Context, r *http.Request) (string, error) {
-			id, err := s.CreateOrg(ctx, r.FormValue("name"))
+	mux.HandleFunc("POST /admin/users/{id}/reinstate", p.adminAction(s,
+		func(ctx context.Context, _ store.User, id string, _ *http.Request) (string, error) {
+			if err := s.SetUserState(ctx, id, store.UserActive); err != nil {
+				return "", err
+			}
+			return "reinstated", nil
+		}))
+
+	mux.HandleFunc("POST /admin/users/{id}/move", p.adminAction(s,
+		func(ctx context.Context, _ store.User, id string, r *http.Request) (string, error) {
+			project := r.FormValue("project")
+			if project == "" {
+				return "", errors.New("choose the project they go to")
+			}
+			moved, err := s.MoveUser(ctx, id, project)
 			if err != nil {
 				return "", err
 			}
-			return "organization " + r.FormValue("name") + " created as " + id, nil
+			return fmt.Sprintf("moved to %s, with %s", projectPath(ctx, s, project), plural(moved, "key", "keys")), nil
+		}))
+
+	mux.HandleFunc("POST /admin/users/{id}/delete", p.adminAction(s,
+		func(ctx context.Context, admin store.User, id string, _ *http.Request) (string, error) {
+			if id == admin.ID {
+				return "", errors.New("you cannot remove your own account")
+			}
+			if err := s.DeleteUser(ctx, id, time.Now()); err != nil {
+				return "", err
+			}
+			return "removed; their keys are revoked", nil
+		}))
+
+	// The tenancy tree. Creating a container is not granting access: nobody can
+	// use one until an account is approved into it.
+	mux.HandleFunc("POST /admin/tenancy/org", p.tenancyAction(s,
+		func(ctx context.Context, r *http.Request) (string, error) {
+			_, err := s.CreateOrg(ctx, r.FormValue("name"))
+			if err != nil {
+				return "", err
+			}
+			return "organization " + r.FormValue("name") + " created", nil
 		}))
 
 	mux.HandleFunc("POST /admin/tenancy/team", p.tenancyAction(s,
@@ -107,11 +180,11 @@ func (p *AccountPages) RegisterAdminUsers(mux *http.ServeMux, s AccountAdmin) {
 			if org == "" {
 				return "", errors.New("choose the organization this team belongs to")
 			}
-			id, err := s.CreateTeam(ctx, org, r.FormValue("name"))
+			_, err := s.CreateTeam(ctx, org, r.FormValue("name"))
 			if err != nil {
 				return "", err
 			}
-			return "team " + r.FormValue("name") + " created as " + id, nil
+			return "team " + r.FormValue("name") + " created", nil
 		}))
 
 	mux.HandleFunc("POST /admin/tenancy/project", p.tenancyAction(s,
@@ -124,16 +197,104 @@ func (p *AccountPages) RegisterAdminUsers(mux *http.ServeMux, s AccountAdmin) {
 			if err != nil {
 				return "", err
 			}
-			return "project " + r.FormValue("name") + " created as " + id, nil
+			r.Form.Set("open", id)
+			return "project " + r.FormValue("name") + " created", nil
 		}))
 
-	mux.HandleFunc("POST /admin/users/{id}/reinstate", p.adminAction(s,
-		func(ctx context.Context, _ store.User, id string, _ *http.Request) (string, error) {
-			if err := s.SetUserState(ctx, id, store.UserActive); err != nil {
+	mux.HandleFunc("POST /admin/tenancy/{kind}/{id}/rename", p.tenancyAction(s,
+		func(ctx context.Context, r *http.Request) (string, error) {
+			id, name := r.PathValue("id"), strings.TrimSpace(r.FormValue("name"))
+			var err error
+			switch r.PathValue("kind") {
+			case "org":
+				err = s.RenameOrg(ctx, id, name)
+			case "team":
+				err = s.RenameTeam(ctx, id, name)
+			case "project":
+				err = s.RenameProject(ctx, id, name)
+			default:
+				return "", errors.New("nothing of that kind")
+			}
+			if err != nil {
 				return "", err
 			}
-			return "reinstated", nil
+			return "renamed to " + name, nil
 		}))
+
+	mux.HandleFunc("POST /admin/tenancy/{kind}/{id}/move", p.tenancyAction(s,
+		func(ctx context.Context, r *http.Request) (string, error) {
+			id, to := r.PathValue("id"), r.FormValue("to")
+			if to == "" {
+				return "", errors.New("choose where it goes")
+			}
+			switch r.PathValue("kind") {
+			case "team":
+				if err := s.MoveTeam(ctx, id, to); err != nil {
+					return "", err
+				}
+				return "team moved", nil
+			case "project":
+				if err := s.MoveProject(ctx, id, to); err != nil {
+					return "", err
+				}
+				return "project moved to " + projectPath(ctx, s, id), nil
+			}
+			return "", errors.New("only a team or a project moves")
+		}))
+
+	mux.HandleFunc("POST /admin/tenancy/{kind}/{id}/delete", p.tenancyAction(s,
+		func(ctx context.Context, r *http.Request) (string, error) {
+			id := r.PathValue("id")
+			var err error
+			switch r.PathValue("kind") {
+			case "org":
+				err = s.DeleteOrg(ctx, id)
+			case "team":
+				err = s.DeleteTeam(ctx, id)
+			case "project":
+				err = s.DeleteProject(ctx, id)
+			default:
+				return "", errors.New("nothing of that kind")
+			}
+			if err != nil {
+				return "", err
+			}
+			r.Form.Del("open")
+			return "removed", nil
+		}))
+
+	mux.HandleFunc("POST /admin/keys/{id}/revoke", p.tenancyAction(s,
+		func(ctx context.Context, r *http.Request) (string, error) {
+			if err := s.RevokeKey(ctx, r.PathValue("id"), time.Now()); err != nil {
+				return "", err
+			}
+			return "key revoked; a running server stops accepting it when its cache expires", nil
+		}))
+}
+
+func projectPath(ctx context.Context, s AccountAdmin, id string) string {
+	all, err := s.AllProjects(ctx)
+	if err == nil {
+		for _, p := range all {
+			if p.ID == id {
+				return p.Path
+			}
+		}
+	}
+	return id
+}
+
+// plainError says what a store error means to the person who caused it.
+func plainError(err error) error {
+	switch {
+	case errors.Is(err, store.ErrDuplicateName):
+		return errors.New("there is already one of those here")
+	case errors.Is(err, store.ErrNotEmpty):
+		return fmt.Errorf("not removed: %s", strings.TrimPrefix(err.Error(), store.ErrNotEmpty.Error()+": "))
+	case errors.Is(err, store.ErrNotFound):
+		return errors.New("that is no longer there; the page was out of date")
+	}
+	return err
 }
 
 // tenancyAction wraps a change to the tree: admin session, CSRF, audit, render.
@@ -153,15 +314,15 @@ func (p *AccountPages) tenancyAction(s AccountAdmin,
 		}
 		msg, err := do(r.Context(), r)
 		if err != nil {
-			if errors.Is(err, store.ErrDuplicateName) {
-				err = errors.New("there is already one of those here")
-			}
-			p.log.Warn("a tenancy change was refused", "admin", admin.Email, "err", err)
-			p.renderUsers(w, r, s, "", err.Error())
+			err = plainError(err)
+			p.log.Warn("a tenancy change was refused", "admin", admin.Email,
+				"path", r.URL.Path, "err", err)
+			p.renderUsers(w, r, s, admin, "", err.Error(), r.FormValue("open"))
 			return
 		}
-		p.log.Info("the tenancy tree was changed", "admin", admin.Email, "what", msg)
-		p.renderUsers(w, r, s, msg, "")
+		p.log.Info("the tenancy tree was changed", "admin", admin.Email,
+			"path", r.URL.Path, "what", msg)
+		p.renderUsers(w, r, s, admin, msg, "", r.FormValue("open"))
 	})
 }
 
@@ -185,7 +346,8 @@ func (p *AccountPages) adminOnly(
 	}
 }
 
-// adminAction wraps a state change: admin session, CSRF, audit line, redirect.
+// adminAction wraps a change to one account: admin session, CSRF, audit line,
+// render.
 func (p *AccountPages) adminAction(s AccountAdmin,
 	do func(ctx context.Context, admin store.User, id string, r *http.Request) (string, error),
 ) http.HandlerFunc {
@@ -204,51 +366,86 @@ func (p *AccountPages) adminAction(s AccountAdmin,
 
 		msg, err := do(r.Context(), admin, id, r)
 		if err != nil {
+			err = plainError(err)
 			// Logged at warn whether it worked or not: a refused administrative
 			// action is as worth knowing about as one that went through.
 			p.log.Warn("an administrative action was refused",
 				"admin", admin.Email, "subject", subject, "err", err)
-			p.renderUsers(w, r, s, "", err.Error())
+			p.renderUsers(w, r, s, admin, "", subject+": "+err.Error(), r.FormValue("open"))
 			return
 		}
 		p.log.Info("an account was changed",
 			"admin", admin.Email, "subject", subject, "what", msg)
-		p.renderUsers(w, r, s, subject+" "+msg, "")
+		p.renderUsers(w, r, s, admin, subject+" "+msg, "", r.FormValue("open"))
 	})
 }
 
 func (p *AccountPages) renderUsers(w http.ResponseWriter, r *http.Request,
-	s AccountAdmin, msg, errMsg string,
+	s AccountAdmin, admin store.User, msg, errMsg, open string,
 ) {
+	ctx := r.Context()
 	d := adminUsersData{
-		Message: msg, Error: errMsg,
+		Message: msg, Error: errMsg, Open: open, AdminID: admin.ID,
 		CSRF:      csrfFor(r),
 		Version:   p.version,
 		HasMirror: p.hasMirror,
 		HasQuotas: p.hasQuotas,
 		Now:       time.Now().UTC().Format(time.RFC3339),
+		People:    map[string][]userRow{},
+		Keys:      map[string][]keyRow{},
 	}
-
-	all, err := s.Users(r.Context(), "")
-	if err != nil {
-		p.log.Error("could not list accounts", "err", err)
-		d.Error = "could not read the accounts"
-	}
-	for _, u := range all {
-		if u.State == store.UserPending {
-			d.Pending = append(d.Pending, u)
-		} else {
-			d.Active = append(d.Active, u)
+	var err error
+	logErr := func(what string, err error) {
+		if err != nil {
+			p.log.Error("could not read "+what, "err", err)
+			if d.Error == "" {
+				d.Error = "could not read " + what
+			}
 		}
 	}
-	if d.Projects, err = s.AllProjects(r.Context()); err != nil {
-		p.log.Error("could not list projects", "err", err)
+
+	d.Projects, err = s.AllProjects(ctx)
+	logErr("the projects", err)
+	paths := make(map[string]string, len(d.Projects))
+	for _, pp := range d.Projects {
+		paths[pp.ID] = pp.Path
 	}
-	if d.Orgs, err = s.AllOrgs(r.Context()); err != nil {
-		p.log.Error("could not list organizations", "err", err)
+
+	all, err := s.Users(ctx, "")
+	logErr("the accounts", err)
+	for _, u := range all {
+		if u.State == store.UserPending && u.ProjectID == "" {
+			d.Pending = append(d.Pending, u)
+			continue
+		}
+		row := userRow{User: u, ProjectPath: paths[u.ProjectID]}
+		d.Accounts = append(d.Accounts, row)
+		d.People[u.ProjectID] = append(d.People[u.ProjectID], row)
 	}
-	if d.Teams, err = s.AllTeams(r.Context()); err != nil {
-		p.log.Error("could not list teams", "err", err)
+
+	keys, err := s.AllKeys(ctx)
+	logErr("the keys", err)
+	now := time.Now()
+	for _, k := range keys {
+		d.Keys[k.ProjectID] = append(d.Keys[k.ProjectID], keyRow{KeyRow: k, State: keyState(k.Key, now)})
 	}
+
+	d.Tree, err = s.Tree(ctx)
+	logErr("the tree", err)
+	d.Orgs, err = s.AllOrgs(ctx)
+	logErr("the organizations", err)
+	d.Teams, err = s.AllTeams(ctx)
+	logErr("the teams", err)
 	render(w, "admin_users", d)
 }
+
+func keyState(k store.Key, now time.Time) string {
+	switch {
+	case !k.RevokedAt.IsZero():
+		return "revoked"
+	case !k.ExpiresAt.IsZero() && !now.Before(k.ExpiresAt):
+		return "expired"
+	default:
+		return "active"
+	}
+}

internal/web/node_logs.go

@@ -0,0 +1,68 @@
+package web
+
+import (
+	"context"
+	"net/http"
+	"strconv"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+)
+
+// LogSource reads a node's logs. Asked for by type assertion on the cluster
+// source, so a source that cannot — a test's, an older one — leaves the page
+// saying so rather than every Source having to grow the method.
+type LogSource interface {
+	NodeLogs(ctx context.Context, node, source string, lines int) adminapi.LogsView
+}
+
+type nodeLogsData struct {
+	adminapi.LogsView
+	OnAdminListener bool
+	// Base is where this page lives, which differs between the admin
+	// listener and the client-facing port.
+	Base    string
+	Back    string
+	Count   int
+	Counts  []int
+	Follow  bool
+	Version string
+	Now     string
+}
+
+// NodeLogsPage shows the recent lines of one of a node's logs: the agent's
+// own, or an engine's output.
+//
+// What a node did used to be learned by ssh through a jump host and a journal
+// only root may read — which, when an engine install fails at two in the
+// morning, is several steps between the page that says it failed and the line
+// that says why.
+func NodeLogsPage(src Source, onAdminListener bool, version string) http.HandlerFunc {
+	return func(w http.ResponseWriter, r *http.Request) {
+		node := r.PathValue("id")
+		d := nodeLogsData{
+			OnAdminListener: onAdminListener,
+			Base:            "/admin/nodes/" + node + "/logs",
+			Back:            "/admin",
+			Counts:          []int{100, 300, 1000, 2000},
+			Follow:          r.URL.Query().Get("follow") != "",
+			Version:         version,
+			Now:             time.Now().UTC().Format(time.RFC3339),
+		}
+		if onAdminListener {
+			d.Base, d.Back = "/nodes/"+node+"/logs", "/"
+		}
+		d.Count, _ = strconv.Atoi(r.URL.Query().Get("lines"))
+		if d.Count <= 0 {
+			d.Count = 300
+		}
+		ls, ok := src.(LogSource)
+		if !ok {
+			d.LogsView = adminapi.LogsView{Node: node, Error: "this control plane cannot read node logs"}
+		} else {
+			d.LogsView = ls.NodeLogs(r.Context(), node, r.URL.Query().Get("source"), d.Count)
+		}
+		noStore(w)
+		render(w, "node_logs", d)
+	}
+}

internal/web/node_logs_test.go

@@ -0,0 +1,31 @@
+package web
+
+import (
+	"context"
+	"net/http/httptest"
+	"strings"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+)
+
+type logsFleet struct{ fakeSource }
+
+func (logsFleet) NodeLogs(_ context.Context, node, source string, _ int) adminapi.LogsView {
+	return adminapi.LogsView{Node: node, Source: source, Lines: []string{"installed an engine <b>"},
+		Sources: []string{"agent", "shard-1"}}
+}
+
+func TestANodesLogIsShownWithItsOtherSources(t *testing.T) {
+	req := httptest.NewRequest("GET", "/admin/nodes/n6/logs?source=agent", nil)
+	req.SetPathValue("id", "n6")
+	rec := httptest.NewRecorder()
+	NodeLogsPage(logsFleet{}, false, "test")(rec, req)
+
+	body := rec.Body.String()
+	for _, want := range []string{"installed an engine &lt;b&gt;", "source=shard-1", "/admin/nodes/n6/logs"} {
+		if !strings.Contains(body, want) {
+			t.Errorf("page lacks %q", want)
+		}
+	}
+}

internal/web/static/base.css

@@ -317,3 +317,48 @@ tr.editing form { margin: .4rem 0; }
    difference has to survive a glance. */
 .chip.on { background: var(--accent); border-color: var(--accent); color: var(--bg);
            font-weight: 600; }
+
+/* People: the tenancy tree with its people and keys. Indented by level and
+   ruled on the left, so which team a project is in reads at a glance without
+   a table per level. */
+.org { background: var(--card); border: 1px solid var(--line); border-radius: 8px;
+       padding: .9rem 1rem; margin: 0 0 1rem; }
+.team { margin: .8rem 0 0 .4rem; padding-left: .9rem; border-left: 2px solid var(--line); }
+.node-head { display: flex; align-items: center; gap: .4rem .9rem; flex-wrap: wrap; }
+.node-name { font-weight: 600; }
+.org > .node-head .node-name { font-size: 1.05rem; }
+.node-meta { color: var(--muted); font-size: .82rem; font-weight: 400; }
+details.edit { display: inline-block; font-size: .85rem; }
+details.edit > summary { cursor: pointer; color: var(--accent); list-style: none;
+                         text-decoration: underline; }
+details.edit > summary::-webkit-details-marker { display: none; }
+details.edit[open] > summary { color: var(--muted); }
+details.edit form { display: flex; gap: .4rem; align-items: center; margin: .35rem 0; }
+details.edit input[type=text] { width: 14rem; }
+details.project { margin: .45rem 0; border: 1px solid var(--line); border-radius: 6px;
+                  background: var(--bg); }
+details.project > summary { cursor: pointer; padding: .45rem .7rem; display: flex;
+                            gap: .8rem; align-items: baseline; }
+details.project[open] > summary { border-bottom: 1px solid var(--line); }
+.project-body { padding: .6rem .7rem .8rem; display: grid; gap: .7rem; }
+table.compact { font-size: .83rem; }
+table.compact th, table.compact td { padding: .35rem .55rem; }
+form.add { display: flex; gap: .4rem; margin: .5rem 0 0; }
+form.add input[type=text], details.edit input[type=text] { margin: 0; padding: .35rem .55rem; }
+form.add input[type=text] { max-width: 18rem; }
+form.add button, button.small { padding: .3rem .7rem; font-size: .82rem; }
+form.org-add { margin: 0 0 1rem; }
+td.actions { white-space: nowrap; }
+td.actions form + form { margin-left: .6rem; }
+.tag { font-size: .7rem; font-weight: 600; text-transform: uppercase; letter-spacing: .04em;
+       color: var(--accent); border: 1px solid var(--accent); border-radius: 4px;
+       padding: 0 .3rem; margin-left: .3rem; }
+input.search { width: 100%; max-width: 26rem; margin: 0 0 .7rem; padding: .45rem .6rem;
+               font: inherit; border: 1px solid var(--line); border-radius: 6px;
+               background: var(--card); color: var(--fg); }
+tr[hidden] { display: none; }
+
+/* The mirror's list of what it holds: one line per version. */
+table.held td.file { max-width: 22rem; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
+table.held td.actions > * + * { margin-left: .7rem; }
+table.held .tag { margin: 0 .3rem 0 0; text-transform: none; letter-spacing: 0; }

internal/web/static/htmx-2.0.11.min.js

@@ -0,0 +1 @@
+var htmx=function(){"use strict";const Y={onLoad:null,process:null,on:null,off:null,trigger:null,ajax:null,find:null,findAll:null,closest:null,values:function(e,t){const n=pn(e,t||"post");return n.values},remove:null,addClass:null,removeClass:null,toggleClass:null,takeClass:null,swap:null,defineExtension:null,removeExtension:null,logAll:null,logNone:null,logger:null,config:{historyEnabled:true,historyCacheSize:10,refreshOnHistoryMiss:false,defaultSwapStyle:"innerHTML",defaultSwapDelay:0,defaultSettleDelay:20,includeIndicatorStyles:true,indicatorClass:"htmx-indicator",requestClass:"htmx-request",addedClass:"htmx-added",settlingClass:"htmx-settling",swappingClass:"htmx-swapping",allowEval:true,allowScriptTags:true,inlineScriptNonce:"",inlineStyleNonce:"",attributesToSettle:["class","style","width","height"],withCredentials:false,timeout:0,wsReconnectDelay:"full-jitter",wsBinaryType:"blob",disableSelector:"[hx-disable], [data-hx-disable]",scrollBehavior:"instant",defaultFocusScroll:false,getCacheBusterParam:false,globalViewTransitions:false,methodsThatUseUrlParams:["get","delete"],selfRequestsOnly:true,ignoreTitle:false,scrollIntoViewOnBoost:true,triggerSpecsCache:null,disableInheritance:false,responseHandling:[{code:"204",swap:false},{code:"[23]..",swap:true},{code:"[45]..",swap:false,error:true}],allowNestedOobSwaps:true,historyRestoreAsHxRequest:true,reportValidityOfForms:false},parseInterval:null,location:location,_:null,version:"2.0.11"};Y.onLoad=V;Y.process=Bt;Y.on=ye;Y.off=xe;Y.trigger=ae;Y.ajax=In;Y.find=a;Y.findAll=x;Y.closest=g;Y.remove=z;Y.addClass=w;Y.removeClass=S;Y.toggleClass=G;Y.takeClass=W;Y.swap=C;Y.defineExtension=zn;Y.removeExtension=Jn;Y.logAll=$;Y.logNone=_;Y.parseInterval=d;Y._=e;const n={addTriggerHandler:Et,bodyContains:ie,canAccessLocalStorage:U,findThisElement:we,filterValues:xn,swap:C,hasAttribute:s,getAttributeValue:f,getClosestAttributeValue:te,getClosestMatch:A,getExpressionVars:qn,getHeaders:yn,getInputValues:pn,getInternalData:re,getSwapSpecification:vn,getTriggerSpecs:lt,getTarget:Se,makeFragment:D,mergeObjects:se,makeSettleInfo:En,oobSwap:Te,querySelectorExt:ce,settleImmediately:Qt,shouldCancel:dt,triggerEvent:ae,triggerErrorEvent:ue,withExtensions:Vt};const he=["get","post","put","delete","patch"];const R=he.map(function(e){return"[hx-"+e+"], [data-hx-"+e+"]"}).join(", ");function d(e){if(e==undefined){return undefined}let t=NaN;if(e.slice(-2)=="ms"){t=parseFloat(e.slice(0,-2))}else if(e.slice(-1)=="s"){t=parseFloat(e.slice(0,-1))*1e3}else if(e.slice(-1)=="m"){t=parseFloat(e.slice(0,-1))*1e3*60}else{t=parseFloat(e)}return isNaN(t)?undefined:t}function Q(e,t){return e instanceof Element&&e.getAttribute(t)}function s(e,t){return!!e.hasAttribute&&(e.hasAttribute(t)||e.hasAttribute("data-"+t))}function f(e,t){return Q(e,t)||Q(e,"data-"+t)}function u(e){const t=e.parentElement;if(!t&&e.parentNode instanceof ShadowRoot)return e.parentNode;return t}function ee(){return document}function q(e,t){return e.getRootNode?e.getRootNode({composed:t}):ee()}function A(e,t){while(e&&!t(e)){e=u(e)}return e||null}function o(e,t,n){const r=f(t,n);const o=f(t,"hx-disinherit");var i=f(t,"hx-inherit");if(e!==t){if(Y.config.disableInheritance){if(i&&(i==="*"||i.split(" ").indexOf(n)>=0)){return r}else{return null}}if(o&&(o==="*"||o.split(" ").indexOf(n)>=0)){return"unset"}}return r}function te(t,n){let r=null;A(t,function(e){return!!(r=o(t,le(e),n))});if(r!=="unset"){return r}}function h(e,t){return e instanceof Element&&e.matches(t)}function N(e){const t=/<([a-z][^\/\0>\x20\t\r\n\f]*)/i;const n=t.exec(e);if(n){return n[1].toLowerCase()}else{return""}}function I(e){if("parseHTMLUnsafe"in Document){return Document.parseHTMLUnsafe(e)}const t=new DOMParser;return t.parseFromString(e,"text/html")}function L(e,t){while(t.childNodes.length>0){e.append(t.childNodes[0])}}function r(e){const t=ee().createElement("script");oe(e.attributes,function(e){t.setAttribute(e.name,e.value)});t.textContent=e.textContent;t.async=false;if(Y.config.inlineScriptNonce){t.nonce=Y.config.inlineScriptNonce}return t}function i(e){return e.matches("script")&&(e.type==="text/javascript"||e.type==="module"||e.type==="")}function k(e){if(!Y.config.allowScriptTags){e.querySelectorAll("script").forEach(e=>e.remove());return}Array.from(e.querySelectorAll("script")).forEach(e=>{if(i(e)){const t=r(e);const n=e.parentNode;try{n.insertBefore(t,e)}catch(e){T(e)}finally{e.remove()}}})}function D(e){e=e.replace(/<hx-([a-z]+)((?:\s[^>]*)?)>/gi,'<template hx type="$1"$2>').replace(/<\/hx-[a-z]+>/gi,"</template>");const t=e.replace(/<head(\s[^>]*)?>[\s\S]*?<\/head>/i,"");const n=N(t);let r;if(n==="html"){r=new DocumentFragment;const i=I(e);L(r,i.body);r.title=i.title}else if(n==="body"){r=new DocumentFragment;const i=I(t);L(r,i.body);r.title=i.title}else{const i=I('<body><template class="internal-htmx-wrapper">'+t+"</template></body>");r=i.querySelector("template").content;r.title=i.title;var o=r.querySelector("title");if(o&&o.parentNode===r){oLine truncated

internal/web/templates/admin_mirror.html

@@ -12,7 +12,7 @@
     {{if .Unreachable}}{{.Addr}} did not answer{{else}}{{.Present}} held ·
     {{.Fetching}} downloading · {{.Failed}} failed · {{.Addr}}{{end}}
   </p>
-  <nav><a href="/admin">cluster</a> <a href="/admin/users">accounts</a>
+  <nav><a href="/admin">cluster</a> <a href="/admin/users">people</a>
        <a href="/profile">your keys</a></nav>
 </header>
 
@@ -31,17 +31,15 @@ reachable from this machine.</p>
 
 <h2>What the mirror holds</h2>
 {{if .Rows}}
-<table>
-<tr><th>Model</th><th>Version</th><th>File</th><th>State</th><th>Size</th>
-    <th>Checksum</th><th></th></tr>
+<table class="compact held">
+<tr><th>Model</th><th>Files</th><th class="num">Size</th><th>In use</th><th></th></tr>
 {{range .Rows}}<tr>
-  <td>{{.Model}}</td>
-  <td>{{.Version}}</td>
-  <td><code>{{.File}}</code></td>
-  <td class="state state-{{.State}}">{{.State}}</td>
+  <td><strong>{{.Model}}</strong> <span class="opt">{{.Version}}</span></td>
+  <td class="file" title="{{.File}}{{if .SHA256}} · sha256 {{.SHA256}}{{end}}">
+    {{if gt .Files 1}}{{.Files}} files{{else}}<code>{{.File}}</code>{{end}}</td>
   <td class="num">
     {{if eq (printf "%s" .State) "fetching"}}
-      {{if .HasPercent}}{{bytes .BytesDone}} / {{bytes .BytesTotal}} ({{.Percent}}%)
+      {{if .HasPercent}}{{.Percent}}% of {{bytes .BytesTotal}}
       {{else}}{{/* No total came from upstream, so no percentage is invented —
                    a bar that guesses its own end and then stops moving is worse
                    than a number that only goes up. */}}
@@ -49,11 +47,14 @@ reachable from this machine.</p>
     {{else if eq (printf "%s" .State) "failed"}}—
     {{else}}{{bytes .BytesDone}}{{end}}
   </td>
-  <td>{{if .SHA256}}<code title="{{.SHA256}}">{{slice .SHA256 0 12}}</code>{{else}}—{{end}}</td>
   <td>
-    {{if eq (printf "%s" .State) "fetching"}}
-      <span class="note">downloading</span>
-    {{else if eq (printf "%s" .State) "failed"}}
+    {{if eq (printf "%s" .State) "fetching"}}<span class="state state-starting">downloading</span>
+    {{else if eq (printf "%s" .State) "failed"}}<span class="state state-failed">failed</span>
+    {{else if .UsedBy}}{{range .UsedBy}}<span class="tag">{{.}}</span>{{end}}
+    {{else}}<span class="opt">not served</span>{{end}}
+  </td>
+  <td class="actions">
+    {{if eq (printf "%s" .State) "failed"}}
     {{/* Retry, not remove. A fetch that failed left nothing on disk, so the
          remove would come back "the mirror holds nothing for this" — a button
          whose only outcome is a confusing error. Starting the same download
@@ -65,25 +66,24 @@ reachable from this machine.</p>
       <input type="hidden" name="model" value="{{.Model}}">
       <input type="hidden" name="version" value="{{.Version}}">
       <input type="hidden" name="file" value="{{.File}}">
-      <button type="submit" class="link">Retry</button>
+      <button type="submit" class="link">retry</button>
     </form>
-    {{else}}
-    {{if $.HasCatalog}}{{if .Serve}}<a href="/admin/mirror">close</a>
-    {{else}}<a href="/admin/mirror?serve={{.Model}}:{{.Version}}">serve</a>{{end}}
-    {{end}}
+    {{else if eq (printf "%s" .State) "present"}}
+    <button type="button" class="link copy-url" data-url="{{.URL}}" title="{{.URL}}">copy address</button>
+    {{if and $.HasCatalog (not .UsedBy)}}{{if .Serve}}<a href="/admin/mirror">close</a>
+    {{else}}<a href="/admin/mirror?serve={{.Model}}:{{.Version}}">serve</a>{{end}}{{end}}
     <form method="post" action="/admin/mirror/{{.Model}}/{{.Version}}/remove"
-          class="inline" onsubmit="return confirm('Remove {{.Model}}/{{.Version}} and every file in it?')">
+          class="inline" onsubmit="return confirm('Remove {{.Model}}/{{.Version}} and every file in it?{{if .UsedBy}} It is in use by the catalog.{{end}}')">
       <input type="hidden" name="csrf" value="{{$.CSRF}}">
-      <button type="submit" class="link danger">Remove</button>
+      <button type="submit" class="link danger">remove</button>
     </form>
     {{end}}
   </td>
 </tr>
-{{if .Err}}<tr><td colspan="7" class="error-row">{{.Err}}</td></tr>{{end}}
-{{if not .Err}}<tr><td colspan="7" class="url-row"><code>{{.URL}}</code></td></tr>{{end}}
+{{if .Err}}<tr><td colspan="5" class="error-row">{{.Err}}</td></tr>{{end}}
 
 {{with .Serve}}
-<tr class="editing"><td colspan="7">
+<tr class="editing"><td colspan="5">
 {{if .Refusal}}
 {{/* Offered and then failing at placement is worse than not offered at all. */}}
 <p class="error">Cannot be served as it stands: {{.Refusal}}.</p>
@@ -140,10 +140,22 @@ job by name if you would rather pin it.</p>
 {{end}}
 {{end}}
 </table>
-<p class="note">The address under each row is what a catalog entry should say.
-It is rendered here rather than assembled by hand because the mirror and the
-catalog have to spell the same file the same way — two spellings is what makes a
-file that is in use look unreferenced.</p>
+<p class="note">“copy address” gives what a catalog entry's <code>source</code>
+should say — the file, or for a version of several files the directory, with its
+trailing slash. It is rendered rather than assembled by hand because the mirror
+and the catalog have to spell a file the same way, which is also how this page
+knows which models use it.</p>
+<script>
+document.addEventListener('click', function (e) {
+  var b = e.target.closest('.copy-url');
+  if (!b) return;
+  var done = function (t) { var o = b.textContent; b.textContent = t; setTimeout(function () { b.textContent = o; }, 1500); };
+  if (navigator.clipboard) {
+    navigator.clipboard.writeText(b.dataset.url).then(function () { done('copied'); },
+      function () { window.prompt('Address', b.dataset.url); });
+  } else { window.prompt('Address', b.dataset.url); }
+});
+</script>
 {{else}}<p class="empty">The mirror is reachable and holds nothing yet.</p>{{end}}
 
 {{if .HasRegistry}}

internal/web/templates/admin_quotas.html

@@ -6,7 +6,7 @@
 <header>
   <h1>Quotas</h1>
   <p class="sub">What each organization, team and project may use · spend over the last {{.Since}}</p>
-  <nav><a href="/admin">cluster</a> <a href="/admin/users">accounts</a>
+  <nav><a href="/admin">cluster</a> <a href="/admin/users">people</a>
        <a href="/admin/usage">usage</a> <a href="/profile">your keys</a></nav>
 </header>
 
@@ -89,7 +89,7 @@ unaffected — they count requests and tokens, not money. Set prices in the
 </table>
 {{else}}
 <p class="empty">Nothing to give a quota to yet. Make an organization on the
-<a href="/admin/users">accounts page</a> first.</p>
+<a href="/admin/users">people page</a> first.</p>
 {{end}}
 
 <p class="note">Limits compose downward by taking the most restrictive value, so

internal/web/templates/admin_usage.html

@@ -6,7 +6,7 @@
 <header>
   <h1>Usage over time</h1>
   <p class="sub">Who is using the cluster, and when</p>
-  <nav><a href="/admin">cluster</a> <a href="/admin/users">accounts</a>
+  <nav><a href="/admin">cluster</a> <a href="/admin/users">people</a>
        <a href="/admin/quotas">quotas</a> <a href="/profile">your keys</a></nav>
 </header>
 

internal/web/templates/admin_users.html

@@ -1,11 +1,16 @@
 {{define "admin_users"}}<!doctype html>
 <html lang="en"><head>
 <meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
-<title>Accounts · llmgw</title><link rel="stylesheet" href="/_static/base.css">
+<title>People · llmgw</title><link rel="stylesheet" href="/_static/base.css">
+{{/* Forms here post and get the whole page back, which works with no script at
+     all. htmx, when it loads, swaps only #people from that page instead of
+     reloading it, so an open project stays open and the page does not jump. */}}
+<meta name="htmx-config" content='{"scrollIntoViewOnBoost":false,"defaultFocusScroll":false}'>
+<script src="/_static/htmx-2.0.11.min.js" defer></script>
 </head><body><div class="wrap">
 <header>
-  <h1>Accounts</h1>
-  <p class="sub">{{len .Pending}} waiting for approval · {{len .Active}} active</p>
+  <h1>People</h1>
+  <p class="sub">{{len .Pending}} waiting for approval · {{len .Accounts}} accounts · {{len .Projects}} projects</p>
   <nav><a href="/admin">cluster</a>
 {{if .HasMirror}}       <a href="/admin/mirror">mirror</a>
 {{end}}{{if .HasQuotas}}       <a href="/admin/quotas">quotas</a>
@@ -13,111 +18,276 @@
 {{end}}       <a href="/profile">your keys</a></nav>
 </header>
 
-{{if .Error}}<p class="error">{{.Error}}</p>{{end}}
-{{if .Message}}<p class="ok">{{.Message}}</p>{{end}}
+<div id="people" hx-boost="true" hx-target="#people" hx-select="#people"
+     hx-swap="outerHTML" hx-push-url="false">
+
+{{if .Error}}<p class="error" role="alert">{{.Error}}</p>{{end}}
+{{if .Message}}<p class="ok" role="status">{{.Message}}</p>{{end}}
 
-<h2>Waiting for approval</h2>
 {{if .Pending}}
-<table><tr><th>Email</th><th>Name</th><th>Signed up</th><th>Project</th><th></th></tr>
+<h2>Waiting for approval</h2>
+<table><tr><th>Email</th><th>Name</th><th>Signed up</th><th>Put them in</th><th></th></tr>
 {{range .Pending}}<tr>
   <td>{{.Email}}</td><td>{{.Name}}</td><td>{{ago .CreatedAt}}</td>
-  <td colspan="2">
+  <td>
+    {{if $.Projects}}
     <form method="post" action="/admin/users/{{.ID}}/approve" class="inline">
       <input type="hidden" name="csrf" value="{{$.CSRF}}">
-      <select name="project">
+      <select name="project" aria-label="Project for {{.Email}}">
         {{range $.Projects}}<option value="{{.ID}}">{{.Path}}</option>{{end}}
       </select>
       <button type="submit">Approve</button>
     </form>
-    <form method="post" action="/admin/users/{{.ID}}/suspend" class="inline">
+    {{else}}<span class="note">Create a project first, below.</span>{{end}}
+  </td>
+  <td>
+    <form method="post" action="/admin/users/{{.ID}}/delete" class="inline"
+          hx-confirm="Reject and remove {{.Email}}?">
       <input type="hidden" name="csrf" value="{{$.CSRF}}">
       <button type="submit" class="link danger">Reject</button>
     </form>
   </td>
 </tr>{{end}}</table>
-{{else}}<p class="empty">Nobody is waiting.</p>{{end}}
-
-<h2>Accounts</h2>
-{{if .Active}}
-<table><tr><th>Email</th><th>State</th><th>Admin</th><th>Project</th><th>Last login</th><th></th></tr>
-{{range .Active}}<tr>
-  <td>{{.Email}}</td>
-  <td class="state state-{{.State}}">{{.State}}</td>
-  <td>{{if .IsAdmin}}yes{{else}}—{{end}}</td>
-  <td>{{or .ProjectID "—"}}</td>
-  <td>{{ago .LastLoginAt}}</td>
-  <td>
-    {{if eq (printf "%s" .State) "active"}}
-    <form method="post" action="/admin/users/{{.ID}}/suspend" class="inline">
-      <input type="hidden" name="csrf" value="{{$.CSRF}}">
-      <button type="submit" class="link danger">Suspend</button>
-    </form>
-    {{else}}
-    <form method="post" action="/admin/users/{{.ID}}/reinstate" class="inline">
-      <input type="hidden" name="csrf" value="{{$.CSRF}}">
-      <button type="submit" class="link">Reinstate</button>
-    </form>
-    {{end}}
-  </td>
-</tr>{{end}}</table>
-{{else}}<p class="empty">No accounts yet.</p>{{end}}
-
-<h2>Organizations, teams and projects</h2>
-<p class="note">Where an account lives. A person is approved into a project, a
-project belongs to a team, and a team to an organization — and a quota can be
-written on any of the three, so the shape here is the shape the bill takes.</p>
-
-<div class="field-row">
-  <form method="post" action="/admin/tenancy/org">
-    <input type="hidden" name="csrf" value="{{.CSRF}}">
-    <label>New organization
-      <input type="text" name="name" required placeholder="Scania"></label>
-    <button type="submit">Create</button>
-  </form>
-
-  <form method="post" action="/admin/tenancy/team">
-    <input type="hidden" name="csrf" value="{{.CSRF}}">
-    <label>New team
-      <input type="text" name="name" required placeholder="platform"></label>
-    <label>in
-      <select name="org" required>
-        {{range .Orgs}}<option value="{{.ID}}">{{.Name}}</option>{{end}}
-      </select></label>
-    <button type="submit"{{if not .Orgs}} disabled{{end}}>Create</button>
-  </form>
-
-  <form method="post" action="/admin/tenancy/project">
-    <input type="hidden" name="csrf" value="{{.CSRF}}">
-    <label>New project
-      <input type="text" name="name" required placeholder="gateway"></label>
-    <label>in
-      <select name="team" required>
-        {{range .Teams}}<option value="{{.ID}}">{{.OrgName}} / {{.Name}}</option>{{end}}
-      </select></label>
-    <button type="submit"{{if not .Teams}} disabled{{end}}>Create</button>
-  </form>
-</div>
-
-{{if not .Orgs}}
-<p class="empty">No organizations yet. Create one, then a team in it, then a
-project in the team — an account can only be approved into a project.</p>
-{{else}}
-{{/* One path column, not three: the store returns the chain already joined,
-     which is the string the approval chooser shows, and three headers over one
-     value line up with nothing. */}}
-<table><tr><th>Organization / team / project</th><th>ID</th></tr>
-{{range .Projects}}<tr><td>{{.Path}}</td><td><code>{{.ID}}</code></td></tr>{{end}}
-{{if not .Projects}}<tr><td colspan="2" class="empty">No projects yet, so nobody can be approved.</td></tr>{{end}}
-</table>
 {{end}}
 
-<p class="note">Four things are deliberately not here. Making someone an admin,
-resetting a password, issuing keys on another
-person's behalf and setting any quota all need a shell on the control plane — <code>llmgw users promote</code>,
-<code>reset-password</code>, <code>llmgw keys</code>, <code>llmgw keys limits</code>.
-A stolen admin session can approve, suspend and make somewhere to put people; it
-cannot hand itself more access, take over an account, or raise what anybody is
-allowed to spend.</p>
+<h2>Organizations, teams and projects</h2>
+<p class="note">A person is in a project, a project in a team, a team in an
+organization{{if .HasQuotas}}, and <a href="/admin/quotas">quotas</a> can be set
+on any of them{{end}}. Open a project to see its people and keys.</p>
+
+{{if not .Tree}}
+<p class="empty">No organizations yet. Create one, then a team in it, then a
+project in the team — an account can only be approved into a project.</p>
+{{end}}
+
+{{range $org := .Tree}}
+<section class="org">
+  <div class="node-head">
+    <strong class="node-name">{{$org.Name}}</strong>
+    <span class="node-meta">{{len $org.Teams}} team{{if ne (len $org.Teams) 1}}s{{end}}</span>
+    <details class="edit"><summary>rename</summary>
+      <form method="post" action="/admin/tenancy/org/{{$org.ID}}/rename">
+        <input type="hidden" name="csrf" value="{{$.CSRF}}">
+        <input type="text" name="name" value="{{$org.Name}}" required aria-label="New name">
+        <button type="submit">Save</button>
+      </form>
+    </details>
+    {{if not $org.Teams}}
+    <form method="post" action="/admin/tenancy/org/{{$org.ID}}/delete" class="inline"
+          hx-confirm="Remove the organization {{$org.Name}}?">
+      <input type="hidden" name="csrf" value="{{$.CSRF}}">
+      <button type="submit" class="link danger">remove</button>
+    </form>
+    {{end}}
+  </div>
+
+  {{range $team := $org.Teams}}
+  <div class="team">
+    <div class="node-head">
+      <span class="node-name">{{$team.Name}}</span>
+      <span class="node-meta">{{len $team.Projects}} project{{if ne (len $team.Projects) 1}}s{{end}}</span>
+      <details class="edit"><summary>rename</summary>
+        <form method="post" action="/admin/tenancy/team/{{$team.ID}}/rename">
+          <input type="hidden" name="csrf" value="{{$.CSRF}}">
+          <input type="text" name="name" value="{{$team.Name}}" required aria-label="New name">
+          <button type="submit">Save</button>
+        </form>
+      </details>
+      {{if gt (len $.Orgs) 1}}
+      <details class="edit"><summary>move</summary>
+        <form method="post" action="/admin/tenancy/team/{{$team.ID}}/move">
+          <input type="hidden" name="csrf" value="{{$.CSRF}}">
+          <select name="to" aria-label="Organization to move {{$team.Name}} to">
+            {{range $.Orgs}}{{if ne .ID $org.ID}}<option value="{{.ID}}">{{.Name}}</option>{{end}}{{end}}
+          </select>
+          <button type="submit">Move</button>
+        </form>
+      </details>
+      {{end}}
+      {{if not $team.Projects}}
+      <form method="post" action="/admin/tenancy/team/{{$team.ID}}/delete" class="inline"
+            hx-confirm="Remove the team {{$team.Name}}?">
+        <input type="hidden" name="csrf" value="{{$.CSRF}}">
+        <button type="submit" class="link danger">remove</button>
+      </form>
+      {{end}}
+    </div>
+
+    {{range $p := $team.Projects}}
+    <details class="project" id="p-{{$p.ID}}"{{if eq $.Open $p.ID}} open{{end}}>
+      <summary>
+        <span class="node-name">{{$p.Name}}</span>
+        <span class="node-meta">{{$p.Users}} {{if eq $p.Users 1}}person{{else}}people{{end}}
+          · {{$p.ActiveKeys}} key{{if ne $p.ActiveKeys 1}}s{{end}}{{if gt $p.Keys $p.ActiveKeys}} ({{sub $p.Keys $p.ActiveKeys}} revoked){{end}}</span>
+      </summary>
+      <div class="project-body">
+        <div class="node-head">
+          <details class="edit"><summary>rename</summary>
+            <form method="post" action="/admin/tenancy/project/{{$p.ID}}/rename">
+              <input type="hidden" name="csrf" value="{{$.CSRF}}">
+              <input type="hidden" name="open" value="{{$p.ID}}">
+              <input type="text" name="name" value="{{$p.Name}}" required aria-label="New name">
+              <button type="submit">Save</button>
+            </form>
+          </details>
+          {{if gt (len $.Teams) 1}}
+          <details class="edit"><summary>move to another team</summary>
+            <form method="post" action="/admin/tenancy/project/{{$p.ID}}/move">
+              <input type="hidden" name="csrf" value="{{$.CSRF}}">
+              <input type="hidden" name="open" value="{{$p.ID}}">
+              <select name="to" aria-label="Team to move {{$p.Name}} to">
+                {{range $.Teams}}{{if ne .ID $team.ID}}<option value="{{.ID}}">{{.OrgName}} / {{.Name}}</option>{{end}}{{end}}
+              </select>
+              <button type="submit">Move</button>
+            </form>
+          </details>
+          {{end}}
+          {{if and (eq $p.Users 0) (eq $p.Keys 0)}}
+          <form method="post" action="/admin/tenancy/project/{{$p.ID}}/delete" class="inline"
+                hx-confirm="Remove the project {{$p.Name}}?">
+            <input type="hidden" name="csrf" value="{{$.CSRF}}">
+            <button type="submit" class="link danger">remove</button>
+          </form>
+          {{else}}<span class="note">Removable once it has no people and never had a key.</span>{{end}}
+        </div>
+
+        {{with index $.People $p.ID}}
+        <table class="compact"><tr><th>Person</th><th>State</th><th>Last sign-in</th><th>Move to</th><th></th></tr>
+        {{range .}}{{template "user-row-actions" (dict "U" . "D" $ "Open" $p.ID)}}{{end}}
+        </table>
+        {{else}}<p class="empty">Nobody is in this project.</p>{{end}}
+
+        {{with index $.Keys $p.ID}}
+        <table class="compact"><tr><th>Key</th><th>Owner</th><th>State</th><th>Last used</th><th></th></tr>
+        {{range .}}<tr>
+          <td>{{.Name}} <code>{{.Prefix}}</code></td>
+          <td>{{or .OwnerEmail "issued by an operator"}}</td>
+          <td class="state state-{{.State}}">{{.State}}</td>
+          <td>{{ago .LastUsedAt}}</td>
+          <td>{{if eq .State "active"}}
+            <form method="post" action="/admin/keys/{{.ID}}/revoke" class="inline"
+                  hx-confirm="Revoke the key {{.Name}}? Whatever uses it stops working.">
+              <input type="hidden" name="csrf" value="{{$.CSRF}}">
+              <input type="hidden" name="open" value="{{$p.ID}}">
+              <button type="submit" class="link danger">revoke</button>
+            </form>{{end}}</td>
+        </tr>{{end}}
+        </table>
+        {{end}}
+      </div>
+    </details>
+    {{end}}
+
+    <form method="post" action="/admin/tenancy/project" class="add">
+      <input type="hidden" name="csrf" value="{{$.CSRF}}">
+      <input type="hidden" name="team" value="{{$team.ID}}">
+      <input type="text" name="name" required placeholder="new project in {{$team.Name}}" aria-label="New project in {{$team.Name}}">
+      <button type="submit">Add</button>
+    </form>
+  </div>
+  {{end}}
+
+  <form method="post" action="/admin/tenancy/team" class="add team-add">
+    <input type="hidden" name="csrf" value="{{$.CSRF}}">
+    <input type="hidden" name="org" value="{{$org.ID}}">
+    <input type="text" name="name" required placeholder="new team in {{$org.Name}}" aria-label="New team in {{$org.Name}}">
+    <button type="submit">Add</button>
+  </form>
+</section>
+{{end}}
+
+<form method="post" action="/admin/tenancy/org" class="add org-add">
+  <input type="hidden" name="csrf" value="{{.CSRF}}">
+  <input type="text" name="name" required placeholder="new organization" aria-label="New organization">
+  <button type="submit">Add organization</button>
+</form>
+
+<h2 id="accounts">All accounts</h2>
+{{if .Accounts}}
+<table id="account-list"><tr><th>Person</th><th>State</th><th>Project</th><th>Last sign-in</th><th>Move to</th><th></th></tr>
+{{range .Accounts}}{{template "user-row-actions" (dict "U" . "D" $ "Open" "" "Path" true)}}{{end}}
+</table>
+{{else}}<p class="empty">No accounts yet.</p>{{end}}
+
+<p class="note">Not here on purpose: making someone an admin, resetting a password
+and issuing a key for someone else. Each turns a stolen admin session into more
+access than it had, so they need a shell on the control plane —
+<code>llmgw users promote</code>, <code>llmgw users reset-password</code>,
+<code>llmgw keys create</code>.</p>
+</div>
 
 <footer><span>llmgw {{.Version}}</span><span>{{.Now}}</span></footer>
-</div></body></html>{{end}}
+</div>
+<script>
+// Search the account list. Outside #people so what was typed survives a swap,
+// and applied again after every swap.
+(function () {
+  var box = document.createElement('input');
+  box.type = 'search'; box.placeholder = 'Find a person by email, name or project';
+  box.className = 'search'; box.setAttribute('aria-label', 'Find a person');
+  function apply() {
+    var q = box.value.trim().toLowerCase();
+    document.querySelectorAll('#account-list tr[data-q]').forEach(function (tr) {
+      tr.hidden = q !== '' && tr.dataset.q.indexOf(q) < 0;
+    });
+  }
+  box.addEventListener('input', apply);
+  function place() {
+    var h = document.getElementById('accounts');
+    if (h && box.parentNode !== h.parentNode) h.after(box);
+    apply();
+  }
+  place();
+  // Projects the reader had open stay open across a change.
+  var open = [];
+  document.addEventListener('htmx:beforeSwap', function () {
+    open = Array.prototype.map.call(document.querySelectorAll('details.project[open]'), function (d) { return d.id; });
+  });
+  document.addEventListener('htmx:afterSettle', function () {
+    open.forEach(function (id) { var d = document.getElementById(id); if (d) d.open = true; });
+    place();
+  });
+})();
+</script>
+</body></html>{{end}}
+
+{{define "user-row-actions"}}{{$u := .U}}{{$d := .D}}
+<tr data-q="{{lower $u.Email}} {{lower $u.Name}} {{lower $u.ProjectPath}}">
+  <td>{{$u.Email}}{{if $u.IsAdmin}} <span class="tag">admin</span>{{end}}</td>
+  <td class="state state-{{$u.State}}">{{$u.State}}</td>
+  {{if .Path}}<td>{{or $u.ProjectPath "—"}}</td>{{end}}
+  <td>{{ago $u.LastLoginAt}}</td>
+  <td>
+    <form method="post" action="/admin/users/{{$u.ID}}/move" class="inline">
+      <input type="hidden" name="csrf" value="{{$d.CSRF}}">
+      <input type="hidden" name="open" value="{{.Open}}">
+      <select name="project" aria-label="Project for {{$u.Email}}">
+        {{range $d.Projects}}<option value="{{.ID}}"{{if eq .ID $u.ProjectID}} selected{{end}}>{{.Path}}</option>{{end}}
+      </select>
+      <button type="submit" class="small">Move</button>
+    </form>
+  </td>
+  <td class="actions">
+    {{if ne $u.ID $d.AdminID}}
+    {{if eq (printf "%s" $u.State) "active"}}
+    <form method="post" action="/admin/users/{{$u.ID}}/suspend" class="inline">
+      <input type="hidden" name="csrf" value="{{$d.CSRF}}">
+      <input type="hidden" name="open" value="{{.Open}}">
+      <button type="submit" class="link danger">suspend</button>
+    </form>
+    {{else}}
+    <form method="post" action="/admin/users/{{$u.ID}}/reinstate" class="inline">
+      <input type="hidden" name="csrf" value="{{$d.CSRF}}">
+      <input type="hidden" name="open" value="{{.Open}}">
+      <button type="submit" class="link">reinstate</button>
+    </form>
+    {{end}}
+    <form method="post" action="/admin/users/{{$u.ID}}/delete" class="inline"
+          hx-confirm="Remove {{$u.Email}} for good? Their keys are revoked.">
+      <input type="hidden" name="csrf" value="{{$d.CSRF}}">
+      <input type="hidden" name="open" value="{{.Open}}">
+      <button type="submit" class="link danger">remove</button>
+    </form>
+    {{else}}<span class="note">you</span>{{end}}
+  </td>
+</tr>{{end}}

internal/web/templates/dashboard.html

@@ -13,7 +13,7 @@
     <a href="/debug/pprof/">pprof</a>
     <a href="/api/v1/nodes">api</a>
 {{else}}
-    <a href="/admin/users">accounts</a>
+    <a href="/admin/users">people</a>
     <a href="/admin/usage">usage</a>
 {{if .Nav.Mirror}}    <a href="/admin/mirror">mirror</a>
 {{end}}
@@ -101,9 +101,10 @@ requested whenever a fallback served it.</p>
 
 <h2>Nodes</h2>
 {{if .Nodes}}
-<table><tr><th>Node</th><th>State</th><th class="num">GPUs</th><th class="num">RAM</th><th class="num">Disk</th><th>Classes</th><th>Fabrics</th><th class="num">Models</th><th class="num">Replicas</th><th>Last seen</th></tr>
+<table><tr><th>Node</th><th>State</th><th class="num">GPUs</th><th class="num">RAM</th><th class="num">Disk</th><th>Engines</th><th>Classes</th><th>Fabrics</th><th class="num">Models</th><th class="num">Replicas</th><th>Last seen</th></tr>
 {{range .Nodes}}<tr>
-  <td><code>{{.ID}}</code>{{if .MgmtAddr}} <span class="note">{{.MgmtAddr}}</span>{{end}}</td>
+  <td><code>{{.ID}}</code>{{if .MgmtAddr}} <span class="note">{{.MgmtAddr}}</span>{{end}}
+    <a class="note" href="{{if $.OnAdminListener}}/nodes/{{.ID}}/logs{{else}}/admin/nodes/{{.ID}}/logs{{end}}">logs</a></td>
   <td class="state state-{{.State}}">{{.State}}</td>
   <td class="num">{{.GPUFree}}/{{.GPUTotal}}</td>
   <td class="num">{{if .MemoryGB}}{{.MemoryGB}}G{{else}}—{{end}}</td>
@@ -111,12 +112,15 @@ requested whenever a fallback served it.</p>
        second, and an em dash where nothing was reported: an agent that
        was never asked about its disk must not read as a full one. */}}
   <td class="num">{{if .DiskTotalGB}}{{.DiskFreeGB}}G/{{.DiskTotalGB}}G{{else}}—{{end}}</td>
+  <td>{{if .Engines}}{{join .Engines}}{{else}}<span class="note">none yet</span>{{end}}</td>
   <td>{{join .Classes}}</td>
   <td>{{join .Fabrics}}</td>
   <td class="num">{{if .Models}}{{len .Models}}{{else}}—{{end}}</td>
   <td class="num">{{.Replicas}}</td>
   <td>{{or .LastSeenAgo "—"}}</td>
-</tr>{{if .Unhealthy}}<tr><td colspan="9" class="note bad">{{join .Unhealthy}}</td></tr>{{end}}
+</tr>{{if .Unhealthy}}<tr><td colspan="11" class="note bad">{{join .Unhealthy}}</td></tr>{{end}}
+{{range .Provision}}{{if ne .State "ready"}}<tr><td colspan="11" class="note{{if or (eq .State "failed") (eq .State "unsupported")}} bad{{end}}">{{.Engine}} {{.Version}}{{if .Variant}} ({{.Variant}}){{end}} {{.State}}: {{.Detail}}</td></tr>{{end}}{{end}}
+{{range .Warnings}}<tr><td colspan="11" class="note bad">{{.}}</td></tr>{{end}}
 {{end}}</table>
 {{else}}<p class="empty">No agents have registered.</p>{{end}}
 

internal/web/templates/node_logs.html

@@ -0,0 +1,31 @@
+{{define "node_logs"}}<!doctype html>
+<html lang="en"><head>
+<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
+{{if .Follow}}<meta http-equiv="refresh" content="5">{{end}}
+<title>llmgw — {{.Node}} logs</title><link rel="stylesheet" href="/_static/base.css">
+</head><body><div class="wrap">
+<header>
+  <h1>{{.Node}}</h1>
+  <p class="sub">{{if eq .Source "agent"}}the agent's log{{else}}output of engine <code>{{.Source}}</code>{{end}} · last {{.Count}} lines</p>
+  <nav><a href="{{.Back}}">cluster</a></nav>
+</header>
+
+<p>
+{{- $d := . -}}
+{{range .Sources}}<a href="{{$d.Base}}?source={{.}}&lines={{$d.Count}}{{if $d.Follow}}&follow=1{{end}}"{{if eq . $d.Source}} aria-current="page"><strong>{{.}}</strong>{{else}}>{{.}}{{end}}</a> &nbsp; {{end}}
+</p>
+<p class="note">
+{{range .Counts}}<a href="{{$d.Base}}?source={{$d.Source}}&lines={{.}}{{if $d.Follow}}&follow=1{{end}}">{{.}}</a> {{end}}lines ·
+{{if .Follow}}<a href="{{.Base}}?source={{.Source}}&lines={{.Count}}">stop following</a>{{else}}<a href="{{.Base}}?source={{.Source}}&lines={{.Count}}&follow=1">follow</a> (refresh every 5s){{end}}
+</p>
+
+{{if .Error}}<p class="bad">{{.Error}}</p>{{end}}
+{{if .Lines}}<pre id="log">{{range .Lines}}{{.}}
+{{end}}</pre>
+{{/* Newest at the bottom, like a terminal, and scrolled there: the line
+     worth reading is almost always the last one. */}}
+<script>window.scrollTo(0, document.body.scrollHeight);</script>
+{{else if not .Error}}<p class="empty">Nothing logged yet.</p>{{end}}
+
+<footer class="note">llmgw {{.Version}} · {{.Now}}</footer>
+</div></body></html>{{end}}

internal/web/web.go

@@ -36,6 +36,8 @@ var files embed.FS
 var staticFiles embed.FS
 
 var funcs = template.FuncMap{
+	"lower": strings.ToLower,
+	"sub":   func(a, b int) int { return a - b },
 	"join": func(v []string) string {
 		if len(v) == 0 {
 			return "—"

scripts/gitea/release_notes.py

@@ -0,0 +1,103 @@
+#!/usr/bin/env python3
+"""Put each version's CHANGELOG.md section on its Gitea release.
+
+Gitea materialises a release for every pushed tag and leaves its body empty,
+and the release workflow only ever attached binaries — so every release page
+was a list of files with nothing saying what had changed. The changelog is
+where that is written, and it stays the one place it is written: this copies
+it, it does not replace it.
+
+Idempotent: a release whose body already matches is left alone, a version with
+no release is skipped, and nothing is created or deleted.
+
+Usage:
+    GITEA_TOKEN=<token> scripts/gitea/release_notes.py [--tag v0.8.1] [--dry-run]
+
+Without --tag, every version in the changelog is synced. The release workflow
+runs it with the tag it just built; running the workflow by hand runs it for
+all of them, which is how releases made before this existed get their notes.
+"""
+
+import argparse
+import json
+import os
+import re
+import sys
+import urllib.error
+import urllib.parse
+import urllib.request
+
+DEFAULT_INSTANCE = "https://gitea.mynodes.xyz"
+DEFAULT_REPO = "ilker/llmgw"
+
+# "## v0.8.1 — 2026-09-28". The dash is whatever was typed; the version is what
+# matters.
+HEADING = re.compile(r"^## (v\d+\.\d+\.\d+)\b.*$", re.M)
+
+
+def sections(changelog):
+    """Map each version to the text under its heading, up to the next one."""
+    out = {}
+    marks = list(HEADING.finditer(changelog))
+    for i, m in enumerate(marks):
+        end = marks[i + 1].start() if i + 1 < len(marks) else len(changelog)
+        out[m.group(1)] = changelog[m.end():end].strip() + "\n"
+    return out
+
+
+def request(method, url, token, payload=None):
+    data = json.dumps(payload).encode() if payload is not None else None
+    req = urllib.request.Request(url, data=data, method=method)
+    req.add_header("Authorization", f"token {token}")
+    req.add_header("Content-Type", "application/json")
+    try:
+        with urllib.request.urlopen(req, timeout=30) as resp:
+            body = resp.read()
+            return json.loads(body) if body else None
+    except urllib.error.HTTPError as e:
+        if e.code == 404:
+            return None
+        detail = e.read().decode(errors="replace")[:300]
+        raise SystemExit(f"{method} {url} failed: {e.code} {detail}") from e
+
+
+def main():
+    ap = argparse.ArgumentParser(description=__doc__)
+    ap.add_argument("--instance", default=os.environ.get("GITEA_INSTANCE", DEFAULT_INSTANCE))
+    ap.add_argument("--repo", default=os.environ.get("GITEA_REPO", DEFAULT_REPO))
+    ap.add_argument("--changelog", default="CHANGELOG.md")
+    ap.add_argument("--tag", help="sync only this version")
+    ap.add_argument("--dry-run", action="store_true", help="show what would change")
+    args = ap.parse_args()
+
+    token = os.environ.get("GITEA_TOKEN", "")
+    if not token and not args.dry_run:
+        sys.exit("GITEA_TOKEN is not set (use --dry-run to preview)")
+
+    with open(args.changelog, encoding="utf-8") as f:
+        notes = sections(f.read())
+    if args.tag:
+        if args.tag not in notes:
+            # A tag with no section is a release somebody forgot to describe.
+            # Failing says so where it is seen; an empty page says nothing.
+            sys.exit(f"{args.changelog} has no section for {args.tag}")
+        notes = {args.tag: notes[args.tag]}
+
+    base = f"{args.instance.rstrip('/')}/api/v1/repos/{args.repo}/releases"
+    for tag, body in notes.items():
+        if args.dry_run:
+            print(f"would set {tag} ({len(body.splitlines())} lines)")
+            continue
+        rel = request("GET", f"{base}/tags/{urllib.parse.quote(tag)}", token)
+        if rel is None:
+            print(f"skip    {tag}  (no release)")
+            continue
+        if (rel.get("body") or "").strip() == body.strip():
+            print(f"same    {tag}")
+            continue
+        request("PATCH", f"{base}/{rel['id']}", token, {"body": body})
+        print(f"set     {tag}")
+
+
+if __name__ == "__main__":
+    main()