llmgw/BACKLOG.md

# Backlog

What a review of the code on 2026-10-04 (commit c2238ab) found should be done
next: security fixes first, then guardrails, then features. Written for
whoever implements it — a person or a model — without the review in hand.
Every item names the files it touches and what "done" is. Paths are relative
to the repository root. Sizes: S about a week, M two to four weeks, L more.

Conventions this codebase keeps, which every item here must keep too:
- A change is refused before anything is written (validate, then persist).
- Nothing on the serving path logs prompt text, keys or passwords.
- Each item gets unit tests beside the code (`*_test.go`), and the TUI
  (`internal/cli/tui_*.go`), `docs/console.md` and `CHANGELOG.md` are
  updated when behaviour an operator sees changes.
- `go test ./...`, `go vet ./...` and `govulncheck ./...` pass. No release is
  cut and nothing is deployed to the PoC cluster without being told to.

---

## A. Security — do first

**Status (2026-10-04): done and merged.** A1–A9, A6b included — etcd and
Patroni's API on TLS, the PoC moved in maintenance mode without a failover.
Deliberately narrower than written: the admin API is a token on loopback
rather than a unix socket (A5); thread ids are bounded and documented as
project-wide rather than scoped per user (A8); Python engines resolve with an
explicit PyTorch index and a PEP 751 lock rather than locks shipped per engine
version (A8). All of A is live on the PoC.

### A1. Toolchain (S, hours)
`govulncheck` finds 11 standard-library vulnerabilities reachable from this
code at Go 1.26.3 (net/http, crypto/x509, os.Root, net/textproto). Raise the
`go` directive in `go.mod` to 1.26.6 or later, rebuild, run `govulncheck
./...` until it reports none reachable. Add `govulncheck` to the Makefile's
check target.

### A2. Release signing is effectively off (S)
- `internal/release/trusted.pub` is empty, so `TrustedKey()` returns "" and
  `release.go:103` skips signature verification. Ship a real public key;
  make a missing or invalid signature fatal. Drop the `LLMGW_RELEASE_KEY`
  environment override, or honour it only in `-tags dev` builds.
- `internal/onboard/onboard.go:365`: the node-side download checks only a
  same-origin `checksums.txt`. Verify the signature there too (simplest:
  have the node run `llmgw` itself to fetch and verify once it has any
  binary), and `shellQuote` the `Releases` URL, which is interpolated into
  single quotes unescaped.
- `internal/agent/upgrade.go:44`: accept only `https://` release bases.
- Done when: a tampered binary or checksums file is refused by self-upgrade,
  node onboarding and agent upgrade, with a test for each.

### A3. Agent port without TLS is anonymous and plaintext (S)
`internal/cli/server.go:423` serves h2c on `:7443` when `agent_tls` is not
configured; `agentsvc` then accepts any node id. Refuse to start control
mode without TLS unless `agent_tls.insecure: true` is set explicitly; when
insecure, bind `agent_listen` to 127.0.0.1 and log a warning at startup and
in `llmgw doctor`. Keep `server init` writing TLS as it does.

### A4. Model profiles reach root on every node (M)
Profile `args` and `env` from a model definition are appended to the engine
command unchecked (`internal/agent/engines/llamacpp.go:170`, `vllm.go:151`,
`sglang.go:98`); `internal/agent/engines/unsafeenv.go` is a denylist that
misses `NCCL_NET_PLUGIN`, `NCCL_TUNER_PLUGIN`, `NCCL_PROFILER_PLUGIN`,
`CUDA_INJECTION64_PATH`, `NVTX_INJECTION64_PATH`, `TRITON_*` paths; engines
run as root.
- Replace the env denylist with an allowlist (CUDA_/NCCL_ tuning keys minus
  anything ending in `_PLUGIN` or containing `INJECTION`, `VLLM_*`,
  `HF_*` minus path keys, `OMP_*`, `GGML_*`). Apply at catalog validation
  (`internal/control/catalog/catalog.go`, where `UnsafeEnv` is called) and
  again in the agent.
- Deny engine flags that write files or run code: `--trust-remote-code`,
  `--log-file`, `--slot-save-path`, `--chat-template` with a path,
  `--host`, `--api-key`, `--port`; vLLM/SGLang equivalents. Allowlist is
  better than denylist if the flag set is enumerable per engine.
- Run engines as an unprivileged user with the video/render groups
  (`internal/agent/supervisor`, systemd unit in
  `internal/cli/serverinit_cluster.go:52`). Weights cache must be readable
  by it.
- Done when: a profile with `--trust-remote-code` or `NCCL_NET_PLUGIN` is
  refused at `ctl model add` and at the agent; `ps` shows engines not as
  root.

### A5. Admin API trusts any local process (M)
`internal/cli/server.go:357` serves the admin API and pprof on
`127.0.0.1:9090` with no authentication; `adminapi.actorOf`
(`internal/control/adminapi/api.go:1613`) takes the actor from the
client-sent `X-Llmgw-Actor` header.
- Serve the admin API on a unix socket (`/run/llmgw/admin.sock`, 0660,
  group `llmgw-admin`) by default, keeping TCP as an option that requires
  a bearer token read from a root-only file (`/etc/llmgw/secrets/admin-token`).
- Take the actor from `SO_PEERCRED` (uid → username) on the socket, or from
  the token's name; ignore the header.
- Refuse `admin_listen` on a non-loopback address unless `admin_insecure:
  true`.
- Put pprof behind its own `debug.pprof: true` setting, off by default.
- `ctlClient` (`internal/cli/ctl.go`), the console and `node add` must work
  over the socket; `sudo llmgw` already runs as root.

### A6. Patroni REST and etcd are unauthenticated and plaintext (M)
`internal/pgha/config.go:99` writes a `restapi:` with no `authentication`
and `:54` etcd over `http://` with no client auth; `hba.go:56` PATCHes
Patroni's config unauthenticated. Anyone reaching :8008 or :2379 owns the
store.
- Generate a Patroni REST username/password at setup, store it in
  `/etc/llmgw/secrets/`, write `restapi.authentication` and
  `restapi.certfile/cafile` from the cluster CA; send it from
  `internal/pgha/hba.go` and `status.go`.
- etcd: client and peer TLS from the cluster CA with `--client-cert-auth`.
- Document, and have `server init` write, nftables rules limiting
  2379/2380/8008/5432 to member addresses (see A9).
- Done when: an unauthenticated `curl :8008/config` is refused and the
  store status in the console still works.

### A7. PostgreSQL traffic is plaintext (S)
`internal/pgha/config.go:168` and `internal/cli/serverinit_postgres.go:330`
use `sslmode=disable`; HBA lines are `host`. Enable server TLS in Patroni's
postgresql parameters with a certificate from the cluster CA, write
`hostssl` lines, use `sslmode=verify-full&sslrootcert=/etc/llmgw/ca.pem`.
Existing clusters: `llmgw server` must keep working against a store that
has not been migrated, with a warning.

### A8. Smaller fixes, each S or less
- **Capacity fallback skips the allowlist.** `internal/router/router.go:700`
  builds `[canonical]+entry.Fallbacks` without `permits`. Filter the chain
  with `r.permits(p, model)`; price against the model that served.
- **Suspended users keep working API keys.** `store.(*DB).Resolve`
  (`internal/store/memberships.go`) never joins `users.state`;
  `Authenticator.Invalidate()` is never called. Reject keys whose owner is
  not active; call `Invalidate` on revoke, suspend, limit change.
- **Join codes never expire server-side.** `internal/join/join.go:103`
  checks expiry only in `Parse`; `admit` (`:212`) does not. Check it there;
  bind the listener to the chosen address, not `:7444` on all interfaces;
  use the peer's address rather than the claimed `hello.Addr`, and show the
  joiner's address for the operator to confirm before secrets are sent.
- **Leftover sudoers grant.** `internal/onboard/gossh.go:248` writes
  `NOPASSWD:ALL` and removes it only in `Close()` (`:271`). Write the grant
  with a self-removal (`at`/systemd-run timer, 30 min) so a crash leaves
  nothing; wipe password buffers after use. In `ssh.go:39` pass `--` before
  the target.
- **Auth cache and pre-auth DoS.** `internal/auth/auth.go:127`: unbounded
  map, one DB query per bogus well-formed key. Bounded LRU, negative cache
  with short TTL, per-IP token bucket before the DB.
- **HTTP server hardening.** `server.go:351-360`, the h2c agent server,
  both mirror servers: set `ReadHeaderTimeout` (exists), `IdleTimeout`,
  `MaxHeaderBytes`, a body-read deadline, `netutil.LimitListener`; connect
  `WithReadMaxBytes` on the agent stream.
- **Login/signup abuse.** Default `signup_open: false`; per-IP token buckets
  on `/login` and `/signup`; bounded wait for argon2 slots; CSRF token on
  the login form; the per-address throttle must not lock the real owner
  out (use IP+address pairs).
- **Trusted proxies.** `internal/web/account.go:557`, `landing.go:60` honour
  `X-Forwarded-Proto/Host` from anyone; add `router.trusted_proxies` and
  only then trust them; `Cache-Control: no-store` on `/setup/*`.
- **Certificate revocation.** `DELETE /api/v1/nodes/{id}` only drops
  inventory; add a revoked-CN/serial list in the store checked in
  `agentsvc` `Connect` and mirror `onlyClients`; `serverinit_cluster`
  always passes `--api-client`.
- **Engine exposure.** Engines bind `0.0.0.0` (`llamacpp.go:102`,
  `vllm.go:109`): bind the management address. SGLang receives its key on
  the command line (`sglang.go:102`): pass it by env or file.
- **Backups.** `internal/cli/backup.go`: encrypt archives to an `age`
  recipient from `backup.recipient` in server.yaml; keep the directory 0700.
- **Python dependency confusion.** `internal/agent/provision/python.go:121`
  uses `--extra-index-url` with `unsafe-best-match`. Pin torch to the
  PyTorch index explicitly (uv `[[index]] explicit = true`), `first-index`,
  hash-pinned lockfiles per engine version.
- **Control-plane SSRF.** `internal/sourcecheck/sourcecheck.go:39` and
  `gguf.FromURL` use the default client; reuse the mirror's
  `fetchClient(false)` (private-address filter) and refuse a proxy unless
  configured; block NAT64/6to4-embedded private addresses in
  `internal/mirror/safefetch.go`.
- **Low:** `agentsvc/service.go:430` indexes `n.GPUs[0]` without a length
  check (CPU-only node panics the handler); thread ids are client-chosen
  with no length cap and shared per project (`router/threads.go`) — cap
  length, document, offer per-user scoping; `/v1/models` lists models the
  key may not use (`handlers.go:799`); sessions have no idle timeout and
  `PurgeSessions`/`PurgeClaims` are never called — add a janitor; add a CSP
  header and set `htmx.config.allowEval=false`; cap GGUF array nesting
  depth in `internal/gguf/gguf.go`; mirror `.source` sidecars are served by
  the public file server — exclude dotfiles from `:8081`; prune
  `modelserve.jobs` and the mirror `running` map; KaTeX → 0.16.21+.

### A9. Firewall (S)
No machine in a default install has a firewall. Have `server init`, `node
add` and `server add` write an nftables ruleset: ssh open; `:8080` open;
`:7443` and `:9091` open to the cluster's networks; `:9100` to the
monitoring host; `:2379/2380/8008/5432` to store members and control
planes; mirror `:8081` to the cluster, `:8082` to control planes; `:9090`
Prometheus and `:3000` Grafana to operators. `llmgw doctor` reports a
missing ruleset. Make it skippable with `--no-firewall`.

---

## B. Guardrails (M–L, build in this order)

**Status (2026-10-04, branch `guardrails-b`):** B1–B4 done, and tried live on
the PoC. That found four bugs, all fixed with tests:
- a shadow default merged with an attached policy of unwritten mode left it
  in shadow;
- a streamed word held back at one release was dropped at the next;
- llama.cpp continues a conversation ending on the answer, so Llama Guard's
  verdict on answers is now asked with its own prompt as a completion;
- a judge model was named in a client's "no endpoint" error.

The PoC runs `observe` (shadow) for everyone, and Llama Guard 3 1B
(`llama-guard-3-1b`, CPU, on cuda01) as a judge, which blocks a harmful
request in ~0.7s and a harmful answer in ~1.4s there. Still left: see each
item.

Today nothing on the request path reads, classifies, redacts or filters
message text in either direction (confirmed; the only "moderation" hit is a
comment in `internal/router/budget_fallback.go:20`). Existing controls are
about who, how much and how fast: allowlists (`router/handlers.go:450`),
RPM/TPM/budget (`internal/accounting`), context precheck
(`router/context_limit.go`), admission bands (`router/admission`),
timeouts (`router/proxy/proxy.go:26-57`). Gaps to close on the way: no
default output cap when the client omits `max_tokens`; `catalog.Capabilities`
is advertised but never enforced; tool results are trusted input.

### B1. Hooks in the proxy and router (S)
**Done (branch `guardrails-b`):** `internal/guard` holds the interface;
proxy `Filter`/`StreamFilter`; the input hook runs after `loadThread` (the
simpler choice); error codes, headers, Responses `incomplete`, usage columns
`guard_*`. Moved to B2: holding `delta.tool_calls` until they are whole and
reading their arguments — that is the tool policy's filter, not the proxy's.

- `internal/router/proxy/proxy.go`: add optional `InputRewrite func(body
  []byte) ([]byte, Verdict)` on the request and two output hooks: `Filter`
  for `relayBuffered` (between `ReadAll` and `WriteHeader`) and
  `StreamFilter{OnEvent(data) (emit []string, stop bool); OnEnd()}` for
  `relayStream`, called where each whole SSE event is assembled
  (`eventBuffer`, before `io.WriteString` at `:343`). Nil hooks cost
  nothing.
- `internal/router/handlers.go`: call the input policy after the allowlist
  and limits and before admission (around `:395`); the principal, scopes
  and the model that will serve are known there. Thread history is spliced
  later (`:416`): scan stored turns when written, or move the hook after
  `loadThread` (decide; the second is simpler).
- Blocking conventions: input → 400 `invalid_request_error` with
  `code: "content_policy_violation"` (today `writeError` always sends
  `code: null`); output non-stream → 200 with `finish_reason:
  "content_filter"` and empty/redacted content; output stream → a final
  chunk `{"choices":[{"index":0,"delta":{},"finish_reason":"content_filter"}]}`,
  then `data: [DONE]`, then cancel upstream. Headers `x-llmgw-guard:
  blocked|redacted|flagged` and `x-llmgw-guard-policy` set before the proxy
  writes headers.
- `internal/router/responses.go`: map `content_filter` to
  `response.incomplete` with `incomplete_details.reason: "content_filter"`
  (today it always emits `response.completed`, `:525`).
- Tool calls: hold `delta.tool_calls` until `finish_reason: "tool_calls"`,
  then validate; `replyDelta` (`reply.go:33`) must also read
  `tool_calls[].function.arguments`.
- Accounting: a blocked input is recorded like `RecordRefused` (nothing
  charged); a blocked output is charged (the GPU was used). `store.Usage`
  gains `GuardAction`, `GuardPolicy`, `GuardCategory`, `GuardMs`.

### B2. Package `internal/guard` (M)
**Mostly done (branch `guardrails-b`):** policies in server.yaml with
attachments in the store (`ctl guard attach`), narrowed by the limits' rule;
`guard/detect` (validated PII, secrets, injection through normalisation and
decoding); redaction with consistent placeholders before prefix routing;
tool allow/deny/strip, call count and argument schemas; capabilities and
structured output (opt-in); default `max_tokens`; pinned system prompt and
dropping the client's; `no_store` (threads, notes, prefix tree, external
models); shadow mode. **Left:** a `guard_events` table — the usage row's
`guard_*` columns carry action/policy/category/time, not rule and stage;
language identification; the re-identification map that restores
redacted values in answers.

- `Policy`, `Effective` resolution over the tenancy tree with the same
  narrowing rule as limits (`internal/store/sqlite.go:383-416` `tighten`):
  deny sets union, allow sets intersect, `mode` takes the strictest of
  `enforce > shadow > off`, `fail` takes `closed > open`, `no_store` at any
  level wins, thresholds take the minimum.
- Store: `guard_policies(scope, scope_id, policy json)` and
  `guard_events(request_id, scopes, policy, rule, category, action, stage,
  score, judge_ms, shadow)` — ids, categories, offsets and lengths only,
  never text. Migrations for sqlite and postgres. Pruned with
  `usage_retention`.
- `guard/detect`: regex/keyword sets; PII with validators — email, phone,
  IBAN (mod-97), card (Luhn), TCKN (checksum), IP, secrets (`sk-`, AWS,
  JWT, PEM); normaliser (NFKC, strip zero-width/bidi, homoglyph fold,
  base64/hex/URL decode to depth 2); injection heuristics (role-switch
  phrases, fake chat-template tokens, long base64, invisible Unicode) —
  default action `flag`; language id.
- Redaction rewrites `messages[].content` in the upstream body with
  placeholders (`[EMAIL_1]`) via the same `map[string]json.RawMessage`
  round-trip `withModel`/`spliceThread` use; an optional re-identification
  map lives only in request memory and restores originals in the output.
  Redact before prefix routing so `PromptPrefix` hashes the redacted text.
- `guard/toolpolicy`: tool name allow/deny (strip or 400), JSON-Schema
  validation of assembled arguments against the declared `parameters`,
  argument regex rules, `max_calls_per_turn`. Enforce
  `Capabilities.Tools`/`Vision` here: a `tools` array or image parts to a
  model without the capability → 400.
- `guard/structured`: validate `response_format.json_schema` on the final
  text → `content_filter`, or flag in shadow.
- Output constraints: inject a default `max_tokens` when absent (from the
  policy or `MaxOutput`), language check, structured check.
- System-prompt pinning per project: prepend before `checkContextFits` and
  before `spliceThread`; optionally drop client `system`/`developer`
  roles; exclude the pin from stored thread turns; apply to Responses
  `instructions` in `toChat`.
- `no_store`: `loadThread` and `PutNote` refuse; `captureBytes()` returns 0;
  skip the prefix-tree observe (`router.go:840`); disable External
  fallbacks for the scope (filter in `dispatch` and `budgetFallback`).

### B3. Judge model on the cluster (M)
**Done (branch `guardrails-b`):** `role: guard` + `guard.format`; hidden and
refused to clients and fallbacks; critical by default, never on demand;
`Router.Judge` past admission/accounting/guardrails; llamaguard3,
shieldgemma and json parsers; 5-minute cache; latency budget then
fail open/closed; streamed answers read every 600 characters, or held with
buffer_all. Simpler than written: the input judge runs after admission, not
alongside it, and shadow mode judges synchronously within the budget rather
than after the response. Not enforced: two replicas on different nodes —
documented, left to `scaling.min`.

- `catalog.Model.Role: "guard"` plus `Guard{Format: llamaguard3 |
  shieldgemma | json, Categories}`. Guard models are hidden from
  `/v1/models`, the chat picker and allowlists; placed like any model,
  class `critical`, `scaling.min ≥ 1`, at least 2 replicas on different
  nodes.
- `guard/judge`: calls the guard model through the router's own `dispatch`
  with an internal principal and its own admission band (never the
  tenant's queue); parsers per format (Llama Guard `safe`/`unsafe\nS1,S10`;
  ShieldGemma yes/no probability → per-tenant thresholds; generic JSON
  verdict); verdict cache keyed by hash of normalised new turns (5 min);
  `latency_budget` (default 250 ms) then `fail` open/closed; input judge
  runs in parallel with admission queueing.
- Streaming strategy: default sentence-buffered scanning (regex/PII per
  sentence or ~200 chars before release; judge every 400–800 chars on the
  cumulative tail; final judge at end); `buffer_all` for `fail: closed`
  scopes. Document that released text cannot be retracted.
- Shadow mode runs everything asynchronously after the response
  (`context.WithoutCancel`, like `saveThread`) and only records.

### B4. Config, metrics, UI (S)
**Done (branch `guardrails-b`):** library in server.yaml, attachments via
`ctl guard`, the console's **Guardrails…** with "applies from" marks, and
the portal's `/admin/guard`; the metrics and alerts below (judge latency
labelled by model only at the judge; the check latency by stage); `ctl
request` shows the verdict; the chat page says "stopped by policy"; the
profile lists effective policies; docs/guardrails.md and data.md written;
the access log carries action, stage and policy only. Attaching from the
portal is not done — the page shows; the console and ctl change.

- Gateway-level policy library in `server.yaml` under `guard:` (see the
  YAML in the review, reproduced in `docs/guardrails.md` to be written);
  attachments to scopes in the store via `llmgw keys guard`, the console
  ("Guardrails…" beside "Limits…" in `internal/cli/tui_people.go`, with
  "inherited from team:X" badges) and the portal.
- Metrics in `router/metrics.go` style:
  `llmgw_guard_checks_total{stage,policy,action,category,mode}`,
  `llmgw_guard_latency_seconds{stage,judge_model}`,
  `llmgw_guard_judge_errors_total{reason}`, `llmgw_guard_fail_open_total`,
  `llmgw_guard_stream_stops_total`, `llmgw_guard_redactions_total{kind}`,
  `llmgw_tenant_guard_blocks_total{tenant}`. Alerts: fail-open > 1% for 5m,
  judge p95 over budget, judge has no ready replicas, per-tenant block rate
  > 3× its 7-day baseline, any `secret` detected in output.
- Portal `/admin/guard`: violations by tenant/category/policy over time,
  shadow vs enforce, judge health, recent events by request id; `ctl
  request <id>` shows the verdict; chat page renders `content_filter` as
  "stopped by policy X"; `/me` lists effective policies.
- Update `docs/data.md`: the judge sees all text but in-cluster; External
  fallbacks send prompts outside the cluster (the current "nothing leaves
  the cluster" is wrong); `no_store` semantics.
- Never log: prompt/completion text, PII values, the re-identification
  map, judge inputs or rationale that quotes input. Check `accessLog`'s
  `err` field and span errors.

---

## C. Features, ranked by value × feasibility

1. **Done (branch `features-c`):** `audit_events`, written by the admin
   API middleware and the /admin pages (with each change's result), `ctl
   audit`, the console's Audit view; staff roles viewer / operator /
   billing / admin. Left: before/after values (the request's fields are
   recorded, not the state they replaced); a tenant-admin role that changes
   its own organization (managers still only view it); hiding the forms a
   viewer cannot submit. **Audit log and roles (M).** Today "audit" is slog lines
   (`internal/web/admin_users.go:403-480`) and roles are the `IsAdmin` flag
   plus an org "manager" who can view. Add `audit_events` (who, what,
   before/after, source) written by the `adminAction`/`tenancyAction`
   wrappers and by every admin-API mutation; roles viewer / operator /
   tenant-admin / billing; `ctl audit`. Depends on A5 for a real actor.
2. **Done (branch `features-c`):** OIDC sign-in (`auth.oidc`, PKCE,
   verified email, roles and memberships from groups, `no_passwords`) and
   SCIM 2.0 (`/scim/v2` Users and Groups). The sign-in cookie's key comes
   from the store (2026-10-10), so a callback may reach either control
   plane behind a balancer. **OIDC/SSO login (M), then SCIM (M).** `internal/account/service.go`,
   `internal/web/account.go`; map IdP groups to org/team/admin.
3. **Done (branch `features-c`):** key lifetime limits (gateway, org, team,
   project), `keys rotate --overlap`, kinds and expiry in `keys list`.
   **Service accounts and key lifecycle (S).** Per-org maximum key TTL,
   `keys rotate` with overlap, last-used shown, project-owned service keys
   listed apart. `internal/cli/keys.go`, `internal/store/keys.go`,
   `internal/auth/auth.go`.
4. **Done (branch `features-c`):** CSV export from the rollups (API, ctl,
   console). Left: Parquet, and a scheduled push to S3/SFTP. **Usage export (S).** CSV/Parquet by org/team/project/model/period from
   the rollups (`internal/store/usageseries.go`), `GET
   /api/v1/usage/export`, optional scheduled push to S3/SFTP.
5. **Done (branch `features-c`):** `notify.webhooks` (model parked, node
   lost, budget spent, update available; signed JSON or Slack; retries,
   hourly de-duplication), `ctl notify test`, Alertmanager receiver
   templates. Left: `server init` and the console do not yet ask for a
   receiver — the templates are filled in by hand. Events are claimed in
   the store before they are sent (2026-10-10), so two control planes send
   each once.
   **Alerting out of the box (S).** `deploy/monitoring/alertmanager.yml`
   ships `receiver: none`; add Slack/email/webhook/PagerDuty receiver
   templates driven by `server init` or the console; llmgw-native webhooks
   for model parked, budget spent, update available (from the reconciler
   and `modelserve`).
6. **Done (branch `features-c2`):** `metadata`, `x-llmgw-tags` and `user`
   on the usage record (migration 0023), `ctl request` shows them,
   `router.tag_labels` metrics with 100 values per tag. Left: export and
   `ctl usage` by tag or end user (the hourly and daily sums do not carry
   tags; it would read the request records, within their retention).
   **Request metadata and tags (S).** Accept `metadata`/`user` from the
   body or `x-llmgw-tags`; store on the usage row; tenant metric labels
   behind an allowlist. `router/handlers.go` parse, `accounting/recorder.go`.
7. **Done (branch `features-c2`):** `ctl model canary/promote/canaries`,
   session-sticky split with `x-llmgw-variant` pins, rollback on error rate
   or TTFT, `canary_rolled_back` notification, console menu. Left: A/B
   between two different models (the split is between a model and its
   newer weights); the comparison is per control plane and lives in memory,
   so a restart begins it again; a key-level pin.
   **Canary and weighted A/B between model revisions (M).**
   `control/modelserve/updates.go` `replace` swaps every replica; add
   weighted routing across two revisions/aliases (scheduler filter, alias
   resolution in `router/router.go:569`), a header/key pin, automatic
   rollback on error-rate or TTFT regression from per-replica metrics.
8. **Done (branch `features-c2`):** `/v1/files` and `/v1/batches`
   (OpenAI's shape), worker under a store lease, lines kept as served,
   lowest band, usage tagged `batch=<id>`, output and error files with
   retention, `no_store` refused. Left: the Responses API (`/v1/responses`)
   as a batch endpoint; the console and `ctl` do not list batches (an
   operator sees them through usage tags); a file is one row in the store,
   so a store backup carries batch files with it.
   **Batch API and Files (M–L).** `/v1/files`, `/v1/batches`, JSONL
   storage, a worker feeding the bronze band, accounting. New
   `internal/router/batch`.
9. **Done (branch `features-c2`):** `keys requests` (retention, sample),
   kept after redaction, `ctl request --body/--replay`, console, pruning.
   Left: a streamed answer is kept as its text, not its chunks; replay is
   one request at a time (a set of them against a model is item 21's eval).
   **Request/response logging and replay (M).** Opt-in per project with
   retention like threads; sampled or full; redaction hooks from B;
   `ctl request <id> --body`; replay against another model.
10. **Done (branch `features-c2`):** Responses `text.format`,
    `input_image`, `previous_response_id` (thread in the id); rerank and
    score passthrough; transcription and translation as multipart;
    `/v1/moderations` by the judge. Left: audio is charged no tokens (an
    engine's duration usage is not read); `input_image` by `file_id`;
    moderation scores are 0 or 1, as a judge says yes or no.
    **OpenAI compatibility gaps (S–M each).** Responses: `text.format` →
    `response_format`, `input_image`, `previous_response_id` via threads
    (`router/responses.go`). Rerank/score passthrough for vLLM,
    `/v1/audio/transcriptions` for a whisper engine, `/v1/moderations`
    mapped to the guard model.
11. **Done (branch `features-c2`):** `/v1/messages` and `count_tokens`,
    streamed and not, tools, images, `x-api-key`, Anthropic error shape.
    Left: extended thinking (a reasoning model's thoughts are not mapped to
    thinking blocks); prompt caching; the account page's "Connect your
    editor" does not offer Claude Code yet.
    **Anthropic `/v1/messages` (M).** Translate like `responses.go`.
12. **Done (branch `features-c2`):** `server restore` (dry run, version
    check, decryption, modes and owners, SQLite and PostgreSQL),
    `backup_copy_to` (directory or ssh), the drill in docs/install.md.
    Left: restoring a replicated store's members (restore into the primary;
    Patroni replicates it); the console's Back up now does not offer
    restore; an ssh copy holds the archive in memory while it sends it.
    **`server restore` (S).** Restore is manual shell today
    (`docs/install.md:1187`); add the command with version check, dry run,
    off-host copy target; a documented DR drill.
13. **Done (branch `features-c2`):** TPOT, inter-token and tokens/s
    histograms by model, pool and tenant; `slo` pressure in admission's
    saturation; `llmgw_model_slo_seconds` and `llmgw-slo` alerts. Left: the
    p99 is per control plane, from its own requests; an SLO per band or per
    tenant; placement does not yet add replicas for an SLO (item 14).
    **SLO-based admission and token-level metrics (S–M).** TPOT and
    inter-token histograms, output tokens/s per model and tenant in
    `router/metrics.go`; use `catalog.SLO` in `admission/saturation.go`.
14. **Done (branch `features-c2`):** `scaling.schedule` (days, from, to,
    min, max; past midnight), `scaling.prewarm`, `control.timezone`;
    queue-depth scaling was there already. Left: `ctl scale` sets min and
    max only (a schedule is set in the definition); scaling for an SLO
    rather than for queue depth.
    **Scheduled and warm-pool autoscaling (S–M).** Time-of-day min/max,
    pre-warm, queue-depth scaling in `catalog.Scaling`,
    `control/reconciler/demand.go`, `headroom.go`.
15. **Done (branch `features-c2`):** `env:` and `vault:` references
    wherever a secret file is named (store address, api_key_file, OIDC,
    SCIM, webhooks), Vault KV with token or AppRole, `secrets.master_key`
    sealing cluster_secrets (Vault transit or a local key). Left: the
    certificate authority's key is still a file in /etc/llmgw (signing in
    Vault's PKI engine instead is its own change); rotating the master key;
    `backup_copy_key` and `admin_token_file` are files only.
    **Secrets management (S–M).** Vault/KMS for `api_key_file`, the DSN, the
    CA key, `cluster_secrets` (`store/secrets.go`, `store/seal.go`);
    envelope-encrypt them in backups (with A8 backups).
16. **Done (branch `features-c2`):** `users forget` (pseudonymised usage
    and sums, kept requests, files, batches, limits and audit names), `keys
    residency --in-cluster` (asked-for, fallback and canary). Left: an
    export of everything held about a person (article 15); residency by
    region between several clusters; backups keep what was forgotten until
    they age out.
    **GDPR and residency (S).** "Forget user" anonymises `user_id` in usage
    and rollups and erases threads/memory (`store/hierarchy.go:241` leaves
    it today); per-project "never route to an External provider".
17. **Done (branch `features-c2`):** exact-match answer cache per project (`keys
    cache --ttl/--any/--off`, console **Cache answers…**, `projects`
    columns, 30-second setting cache), reproducible answers only unless
    `--any`, LRU in memory bounded by `router.cache.max_bytes` and
    `max_entry_bytes`, streamed answers kept and replayed as chunks,
    `x-llmgw-cache` and `Cache-Control: no-cache`, guardrails before a hit
    and the output policy on it, `no_store` and threads bypass, a hit
    recorded as `cache_hit` with no tokens and no cost (migration 0027),
    metrics, docs. Left: semantic (embedding-similarity) caching through the
    embeddings models; a cache shared across control planes (each process
    keeps its own, deliberately: a shared one would write every answer to
    the store); a hit still waits its turn in admission, because the
    guardrails judge the request after admission and the key is the request
    they let through; logprobs in a streamed answer leave it unkept; `users
    forget` does not reach into the cache (answers age out by the TTL); no
    listing or flushing of what is cached (`--off` stops it being served at
    once, the memory is freed by TTL and LRU).
    **Response/semantic cache (M).** Exact-match first, per project,
    opt-in, TTL; semantic via the embeddings models later. Before
    `dispatch` in `router/handlers.go`.
18. **Done (branch `features-c2`):** offline bundles — `llmgw bundle create`
    (releases with signed checksums, every llama.cpp build per arch/variant
    with GitHub's checksums, starter/judge and named Hugging Face model
    files pinned and checked; directory or tar; manifest sha256), `bundle
    import` (checked, all-or-nothing, into the mirror under `_llmgw/`),
    `bundle trust`, `bundle show --verify`;
    `offline.mirror`/`offline.checksums` in server.yaml, with nodes
    installing llama.cpp from the mirror against checksums the control plane
    sends over the agent stream (proto `EngineWant.offline_*`), releases
    defaulting to the mirror for `node add`, `ctl upgrade` and agent
    upgrades, update checks off; `server init --bundle` and the setup form's
    bundle fields; docs/data.md's offline claim corrected; install.md
    "Offline sites". A Helm chart for the control plane
    (`deploy/helm/llmgw`, one replica, SQLite PVC or external PostgreSQL,
    Secrets, Services, Ingress, ServiceMonitor) with
    `deploy/container/Dockerfile` and docs/kubernetes.md; GPU nodes stay
    outside Kubernetes by design. Left: vLLM/SGLang offline (a wheelhouse of
    several GB per CUDA build and a uv `--find-links` install path); the
    NVML release binary and uv in bundles; the setup wizard's preflight
    still fails a machine without internet; `join`'s printed `curl` from the
    mirror checks nothing; models in a bundle are single files (no whole-
    repository safetensors sets yet); the chart runs one replica (a standby
    is never ready, so HA in Kubernetes needs a different readiness design),
    has not been installed on a real cluster, and ships no agent DaemonSet;
    no published container image.
    **Kubernetes/Helm and an air-gapped bundle (M).** `deploy/`,
    `agent/provision`; `docs/data.md` claims offline operation but install
    assumes the internet.
19. **Done (branch `features-c2`):** `adminapi.Routes` and
    `internal/docgen/openapi` generating `docs/admin-api.openapi.json` (3.1,
    bearer security, every route, query parameters, schemas from the Go
    types with their comments), served embedded at `GET
    /api/v1/openapi.json`, kept true by a route-table test and a staleness
    test; validated with openapi-spec-validator and openapi-generator-cli.
    `GET /api/v1/models/{name}`. `docs/admin-api.md`. `terraform-provider-
    llmgw/` (terraform-plugin-framework): `llmgw_model`, `llmgw_quota`,
    `llmgw_guard_attachment`, data sources `llmgw_models` and `llmgw_nodes`;
    client tests against a fake admin API and the OpenAPI document;
    acceptance tests with Terraform and OpenTofu against the fake; one
    apply, empty plan, in-place change and destroy against a real `llmgw
    server` run by hand. Organizations, teams, projects, keys and
    memberships are in the admin API since 2026-10-08 (`/api/v1/orgs`,
    `teams`, `projects`, `keys`, `users/{id}/memberships`) and in the
    provider (`llmgw_org`, `llmgw_team`, `llmgw_project`, `llmgw_key`,
    `llmgw_membership`); accounts are still made by sign-up, invitation or
    OIDC/SCIM. Left: a model's definition beyond
    scaling, class, owner and adapters is changed by replacing it (the API
    has no edit, and `resources` is not used yet); `canary`, `slo`,
    `scaling.schedule` and `prewarm`, `guard` role and `origin` are not in
    `llmgw_model`; models cannot be imported; the provider is not published
    to a registry and has no release build; the acceptance tests are not run
    in CI (they need Terraform); the document's schemas mark nothing
    required and give no enums; `catalog.SLO`'s fields have no JSON tags, so
    they are `TTFTp99` and `TPOTp99` on the wire.
    **OpenAPI for the admin API, Terraform provider (M).** Generate from
    `adminapi/api.go` routes via `docgen`.
20. **Done (branch `features-c2`):** spans for the reconcile pass (begun only
    when a pass does something), placement with its answer or rejections,
    scale-up/down, drain, stop, eviction, preemption, reaping, restarts and
    lost nodes; a `replica start` span from placement to serving whose
    context travels to the agent in the start command
    (`ControlMessage.trace`), where weights fetch, engine launch, rendezvous
    and readiness wait continue the trace; stops, drains, adoption and
    engine installs; modelserve jobs as a trace with a span per step linked
    to the replica's start; an agent `tracing` block; docs in install.md's
    "Tracing". Left: the router's span for a request does not link to the
    replica it was sent to's start; heartbeats, fabric scans, logs and agent
    upgrades are not traced; the supervisor has no restart span because it
    never restarts an engine itself (a restart is the reconciler's `restart
    replica` and reaches a node as a stop and a start); canary and promote
    jobs, which landed on `features-c2` after this was branched, are traced
    only through the shared step recording (as `<kind> model`), unchecked by
    a test.
    **Tracing beyond the router (S).** Spans for reconciler, placement,
    agent start (`obs/trace.go`, `control/reconciler`, `agent/supervisor`).
21. **Done (branch `features-c2`):** `internal/bench/eval` (suites in YAML or
    JSON lines; exact, contains, regex, number, a JSON Schema subset, and a
    judge model with a rubric; weights and a threshold), run on the control
    plane through `Router.Replay`; `llmgw ctl eval MODEL --suite FILE|NAME`,
    `--history`, `--run`, `--suites`; runs and their cases kept (migration
    0028, `eval_runs` and `eval_cases`); `model update`/`promote --eval
    SUITE --min-score N`, a model's standing gate (`ctl model gate`, `eval:`
    in the catalog) — a promote asks the canary first, an update tries the
    new weights as a 0% canary and promotes or removes it; the console's
    model menu; `arithmetic` and `json-format` in `deploy/eval/`, carried in
    the binary. Left: the suite of a model's standing gate is read by name
    on the control plane at the time of the update, so two control planes
    need the same `control.eval_dir`; a gated update needs room for one more
    replica while it runs (there is no in-place gate, and a declared model
    in server.yaml cannot be updated at all); no comparison against the old
    weights' score (the gate is an absolute threshold, not "no worse than
    before"); a run is not stopped once started (Esc in the console only
    stops watching it), and the in-memory progress of a run lives on the
    control plane that started it; runs are never pruned; the JSON Schema
    support is a subset (no `$ref`, `oneOf`, `not`, `uniqueItems`); judge
    cases go through the same concurrency as the model's, and a judge's
    answers are not kept, only its score and reason; a gateway without a
    control plane cannot run suites (there is no mode that asks through the
    client API with a key).
    **Eval harness (M).** `llmgw eval` extending `internal/bench` with
    quality suites gating `model update`.
22. **Done (branch `features-c2`):** MIG instances discovered on both paths
    (nvidia-smi `-L` + `-q -x`; NVML through an interface with a faithful
    fake) and reported instead of their card, each with its MIG UUID, own
    memory and class `<card class>-mig-<profile>`; device numbers 1000 + 100
    × card + instance, shown as `card/migN`; placement on MIG classes;
    tp/pp/gpus/nodes above 1 on a MIG class refused in catalog validation
    and again in placement; `CUDA_VISIBLE_DEVICES=MIG-<uuid>` for vLLM and
    llama.cpp (no Vulkan device variable on MIG); per-instance memory and
    processes in heartbeats and the GPU check, card sensors under each
    instance, utilisation absent rather than zero; doctor MIG findings (`mig
    -lgi`/`-lci` parsed); `ctl nodes`, console, `ctl status`, `ctl model
    plan`/`serve` by class; docs/install.md MIG section. Written and tested
    against fakes only — untested on real MIG hardware. Left: run it on an
    A100/H100 in MIG mode (check `-L`/`-q -x`/`mig -lgi`/`-lci` and NVML MIG
    device names against the parsers, and that `CUDA_VISIBLE_DEVICES=MIG-…`
    reaches vLLM's workers through Ray); instances changed while the agent
    runs are only seen when it restarts (no rescan); llmgw does not create
    or reshape instances (no mig-parted integration); quotas count an
    instance as one GPU, the same as a whole card; SGLang on MIG not
    checked; device numbers assume fewer than 100 instances per card and
    that instance numbers stay put while shards run on them.
    **MIG (M–L).** Treat a MIG slice as a GPU class in
    `agent/discovery/gpu_nvml.go` and `control/placement`.
23. **Done (branch `features-c2`):** engine `trtllm`
    (`internal/agent/engines/trtllm.go`): `trtllm-serve` through a launcher
    of llmgw's that demands the replica's key (trtllm-serve has none) and
    writes the options file that turns on `/prometheus/metrics` and the
    iteration statistics; the scrape contract for it
    (`scrape.TRTLLMMapping`, with a per-engine metrics path); found, not
    installed — `engines.trtllm.python` in `agent.yaml`, probed from the
    package's metadata; catalog validation (GPU class, one node, no GGUF
    source, the flags a profile may not give); `ctl engine-check trtllm`,
    which asks `trtllm-serve serve --help` through its module. Ollama/GGUF
    import: `mirror import-ollama` (manifest parsing, blob digest check,
    projector beside the model, all or nothing, a model definition printed),
    `mirror import` for any local file, and the console's **I** on the
    Mirror view. Left: TensorRT-LLM has not run on real hardware — the first
    node with one should run `engine-check` and serve a model before anyone
    relies on it; no installer for it; no multi-node TensorRT-LLM replica
    (it needs an MPI or Slurm launch from one node); Triton Inference Server
    as an engine is not done; trtllm's queue gauges are set per iteration,
    so an idle engine shows its last batch until the next request;
    `--num_serve_frontends` is refused because extra front ends would not
    demand the key. The Ollama import runs on the mirror machine only — the
    mirror's API takes no files, so an Ollama elsewhere means copying the
    blob or running the import there; Ollama models kept as separate tensors
    or as several weight layers, and its LoRA adapters, are refused or left
    with a sentence; files an import run with `sudo` writes are root's, as
    `mirror add`'s are.
    **More engines (M each).** TensorRT-LLM/Triton adapter; Ollama/GGUF
    import. `agent/engines/engine.go` interface and the metric contract in
    `router/scrape`.
24. **Done (branch `features-c2`), all six parts.** Lower: precise KV-index routing (plan phase 6), LoRA affinity for
    vLLM, multi-cluster federation via External providers, facility-aware
    placement (`docs/facility-aware-placement.md`), portal i18n, SDK/docs
    portal.
    What was built, and what is left:
    - 24 (portal i18n): built — English/Turkish catalogue with fallback, per-
      language templates, cookie/Accept-Language/`landing.language` choice
      and a footer switcher, Turkish numbers and dates, every template and
      handler message translated, tests that no template text stays English;
      left — errors built with fmt.Errorf in other packages (store, catalog,
      mirror, adminapi) still English on Turkish pages, percent sign
      placement is mixed (%50 on limits, 93% on cluster pages), RFC3339
      footer timestamps and plain-text http.Error bodies untranslated, the
      /docs documents themselves English, a per-person language column
      (cookie only today).
    - 24 (SDK/docs portal): built — /docs renders the embedded manual (goldmark,
      GFM, heading anchors, links rewritten, no script; public,
      `landing.docs_sign_in` to require a session), and docs/clients.md
      shows the official OpenAI/Anthropic SDKs against llmgw. Left: the
      console's generated "Commands" page is not in the portal (it is built
      from the cobra tree, which internal/web cannot see); no search across
      documents.
    - [~] 24. Precise KV index (tier 2) — built: vLLM KV events via the agent
      (loopback ZMQ, pure-Go zmq4, msgpack in both of vLLM's encodings)
      forwarded as net block-hash changes over the agent stream; router
      index, tokenize via the replica's `/tokenize`, sha256_cbor block-hash
      chain checked against cbor2/hashlib golden values; scorer by leading-
      block share with tier 1 fallback; opt-in `kv_events` per profile;
      metrics; docs. Left: validation against a real vLLM (event shape, hash
      agreement, `/tokenize` matching the served prompt, cost of
      tokenizing); vLLM's replay endpoint instead of resetting on a gap;
      data-parallel ranks (one publisher each, ports offset); hybrid models'
      cache groups and CPU-offloaded blocks; LoRA/multimodal/cache_salt
      requests (extra keys); a router-only deployment (no agents)
      subscribing directly; standby control planes have no index until they
      lead; a per-replica fallback when only some candidates have an index;
      caching tokenizations of repeated prompts.
    - **24 LoRA affinity (branch `features-c2`):** built `max_loaded_adapters`
      (catalog), vLLM adapter support at start and on demand
      (`engines/lora.go`, `vllm.go`), the agent linking fetched adapters as
      `llmgw-lora/<name>` in the engines' working directory, the router's
      adapter tracker (`router/lora.go`: loads/unloads with the engine key,
      LRU eviction never of an adapter in use, one load per adapter per
      replica, `/v1/models` sync, 404 reload) and the `adapter-affinity`
      scheduler filter in the chat, batch and long-context profiles, with
      metrics and docs; tested against a fake of vLLM's HTTP interface.
      **Left:** never run against a real vLLM — in particular that vLLM
      resolves the relative `llmgw-lora/<name>` path against its working
      directory on every node (multi-node too) and that unloading an adapter
      mid-request is harmless; the agent still fetches every adapter at
      replica start, so a new adapter needs a restart (fetching on first
      load would need an agent command and a proto change); load timeout and
      resync interval are constants, not config; SGLang's runtime LoRA API
      is not driven; no per-replica "adapters held" view in the console or
      admin API; the debug profile has no affinity.
    - 24. Multi-cluster federation — built: peers via `ctl federation` kept in
      the store, their models as PEER/MODEL synced from /v1/models
      (withdrawn-while-referenced), /ready health with latency and routing
      withdrawal, `ctl model fallbacks` for overflow, hop header loop
      refusal, residency/no_store hold, optional attribution tags, console
      view, metrics, docs. Left: server.yaml-declared models cannot name a
      peer model as a fallback (config catalog is validated before peers are
      read); per-peer CA / mTLS for peers behind a private CA; transitive
      federation (A→B→C models) deliberately not offered; latency is /ready
      RTT only (no per-request peer TTFT used in routing or weighting); the
      peer's adapters are not imported (listed as aliases); no console form
      to add/remove peers (CLI only); key file must exist on every control
      plane.
    - 24 (facility) — built: location labels (`ctl label`, agent.yaml),
      spreading across racks/PDUs, `keep_within`, held domains with drain
      (`ctl facility hold/release`, kept in the store), power caps from
      `facility.budgets` against declared/derived/measured draw, shown in
      explain-placement, `ctl nodes`, `ctl facility`, the console and the
      API. Left: temperature conditions with hysteresis and their
      enforcement (steps 3–4), a Prometheus/DCIM source, power headroom as a
      rack-levelling score, spreading multi-node replicas across racks,
      `llmgw_domain_*` metrics and alerts (near cap, over cap, stale data),
      shedding and tokens per watt (step 5), and a drained node's state
      surviving a restart.

---

## E. Every setting takes effect when it is changed (2026-10-05)

Asked for after a site visit: an upgrade left server.yaml without guardrails,
a node on another subnet stayed lost until firewall_networks was edited by
hand, and nothing in server.yaml, agent.yaml or a generated unit applies until
somebody restarts the right service on every machine. The rule from here on:
**a file holds only what one machine needs to start; everything else is the
cluster's, kept in the store, changed through ctl, the console or the web,
audited, and in force on every control plane within seconds.**

### E1. Cluster settings in the store (L) — done 2026-10-05
Live on the PoC (140, 150): imported as version 1; a guard change applied on
both without a restart; a router change restarted the standby at once and the
leader 30s later with /health answering throughout; rollback did the same.
- Table `cluster_settings` (versioned YAML document, who and why). The
  cluster-wide part of server.yaml lives there; server.yaml keeps the
  per-machine keys only (listeners, TLS paths, store DSN, secrets, tracing,
  monitoring, mirror.api/tls, backup_dir/key, endpoints).
- First start against a store with no document imports the file's cluster
  keys (and the baseline guard policy when there is none); after that those
  keys in a file are ignored, with a warning.
- Each control plane polls the document's version and its own server.yaml's
  hash. A change is validated, then applied: `guard` live, the rest by a
  graceful restart of the process (standbys first, the leader last), which
  systemd brings back. No "restart now?" anywhere.
- `llmgw ctl config show|history|get|set|unset|edit|rollback`; the console's
  settings screen edits the store for cluster keys; admin API
  `GET/PUT /api/v1/settings`, `/history`, `/rollback`.
- A standby that becomes leader re-reads the catalog, engine choices and
  facility state from the store (it kept what it read at start).
- Retires `appendGuardIfMissing` (server init): the import does it.

### E2. Generated files from settings (L) — firewall on/off done 2026-10-06, every machine 2026-10-09
`firewall.enabled` in the cluster's settings reaches nodes over the agent
stream and control planes through llmgw-firewall-sync.path. Since
2026-10-09 the mirror machine too (the leader PUTs /v1/firewall, the mirror
writes `.firewall.json`, the same path unit applies it) and store members
that run no llmgw (the leader keeps it in `cluster_state`, a root timer
reads the member's own copy with psql); `firewall_networks` reach all of
them, and the mirror lets in the network of every node and control plane
the leader knows. Tried on the PoC: off and on, and a network added, on
control planes, mirror, store member and nodes. Left: the allowed set
replaced by membership rather than added to (a network taken out of the
settings stays let in), the other files (the mirror's unit flags,
keepalived, etcd and Patroni configuration).

Every machine runs the agent (root) with roles — node, control plane, mirror,
store member — and the agent renders and applies what was written once by
init: the llmgw nft table, the mirror's unit flags, keepalived, etcd and
Patroni configuration. The firewall's allowed set is the cluster's membership
(every node, control plane, mirror and store member) plus
`firewall.extra` from the store; 7443 stays open (mTLS). A node added or
removed changes every machine's rules. Retires `firewall.Admit` and
`firewall_networks` in server.yaml (imported into the store).

### E3. agent.yaml down to bootstrap (M) — done 2026-10-08 (branch `settings-e`)
agent.yaml keeps the control planes' addresses, its certificate and its id.
Labels, fabrics, engines, heartbeat and supervisor settings move to the
store per node and reach the agent over its stream; a change to the file is
noticed and the agent restarts itself.
Built as a section of the cluster's settings rather than a store of its own,
so history, audit, `ctl config`, the console and the portal page cover it:
`agents.default` and `agents.nodes.<id>` (config.AgentSettings; bootstrap
keys node_id/server(s)/hostname/mgmt_addr/tls refused; each node's merge
loaded as an agent would, so a bad fabric is refused on save). A live key:
the control plane sends ControlMessage.AgentSettings on register and, a
second apart per node, on change; the agent compares with what it started on,
checks it loads beside its agent.yaml (else logs and keeps running), keeps it
in /var/lib/llmgw-agent/agent-settings.yaml and restarts (engines adopted).
At start: agent.yaml < kept settings < LLMGW_ env; kept settings that no
longer load are set aside, never a reason not to start. agent.yaml is
watched every 10 s and restarted on when it changes and loads. Left: file
keys are not imported into the settings (they keep working under them);
`node add` still writes a full agent.yaml; labels have two homes now
(`ctl label`, which wins, and agents.*.labels).

### E4. Web settings page (S) — done 2026-10-08 (branch `settings-e`)
The same document, edited on /admin/settings, for whoever has no terminal.
Built: administrators only (staff roles get 404: webhook secrets), checked by
the same function as PUT /api/v1/settings, made from a base version or
refused on a newer one (the edit kept in the editor), history with view and
roll back as a new version, audit with the size not the body, English and
Turkish, a Settings link in the menu.

---

## F. Company documents: knowledge bases (RAG), then adapters (2026-10-08)

A company wants a model to answer from its own documents. Retrieval is the
tool for *knowledge* (sources shown, re-indexed in minutes, access per
tenant); LoRA training is for *behaviour* (vocabulary, format) and has no
access control — whatever goes into weights reaches everyone who may use the
adapter. RAG first. Decisions taken: access per collection (no per-document
ACL yet); a personal key or session names a project or team and gets that
chain only, never the union of its person's memberships; PostgreSQL store
only, off on SQLite with a sentence saying why.

### F1. Store: collections, grants, chunks (M) — done 2026-10-08 (branch `rag`)
Built (`internal/store/knowledge.go`, migration 0035 PostgreSQL only):
- `kb_collections` (owner scope/id, org, embed_model, dims), `kb_grants`
  (read|write, any scope, never across organizations; a key needs a
  project; a person needs a membership in the organization),
  `kb_documents` (digest: the same again is no change, a new one replaces
  in one transaction).
- `kb_chunks` is created when the store opens, not in a migration (a
  migration needing an extension would stop the control plane; one that
  skipped it would never run again): partitioned by organization (hashed
  partition name), `tsv` generated (`simple`), GIN on the parent, one
  partial HNSW index per collection on its organization's partition
  (`embedding::vector(dims)`, `WHERE collection = '<id>'`). EXPLAIN on
  20k rows: partition pruned, that index used, policy applied.
  Later (scale, 2026-10-08): a partial GIN per collection too
  (`kb_tsv_<id>`, backfilled at open); `vec_type` per collection, `halfvec`
  on pgvector ≥ 0.7 (up to 4000 dims); `SET LOCAL hnsw.ef_search` to
  candidates(k) clamped 40..1000 (EXPLAIN: 40 rows without it, all with
  it); the text arm takes candidates by `ts_rank` in a subquery and scores
  only those.
- RLS FORCEd on `kb_chunks` against `llmgw.kb_org`/`llmgw.kb_collections`
  (set with set_config per transaction), one role — the table owner — not
  a separate one. Chunks are deleted by the foreign-key cascade from their
  document, which RLS does not block.
- `CollectionsFor(Effective, write)`: owner or grant anywhere on the chain;
  no organization (personal key without a project) reads nothing.
- `SearchChunks`: per collection nearest-neighbour (literal id and cast so
  the partial index is used) plus a full-text arm with any word
  (`a | b`, lowercased by the database, not Go), fused by reciprocal rank.
- Deleting a project/team/org that owns collections is refused like one
  with members (changed from "deleted with them": every other delete in the
  tree refuses); DeleteUser/ForgetUser delete the person's collections;
  deleting an organization drops its partition; grants to a deleted scope go.
- `server init` installs `postgresql-N-pgvector` / `pgvector_N` and enables
  it; pgha installs it on every member at setup and enables it on the
  leader on every run. Neither stops on failure.
Left: a cluster set up before (the PoC) gets no package from a rerun of
pgha on a running cluster — install by hand or add an upgrade step;
`server restore`/`CopyTo` do not carry kb tables (SQLite → PostgreSQL only,
and SQLite has none); tests need `LLMGW_TEST_POSTGRES` pointing at a
pgvector database as a non-superuser (a superuser bypasses RLS).

### F2. Ingest on the user's machine (L) — done 2026-10-08 (branch `rag`)
Uploading whole files to the control plane was dropped: a PDF is 20–100×
its text, and the work belongs where the files are. Instead a small client,
`llmgw-kb`, run by the person on their own machine with their own key:
- **Its own binary** (`cmd/llmgw-kb`), for Windows, macOS and Linux on amd64
  and arm64. `llmgw` itself does not build on Windows (agent syscalls), and
  employees' laptops should not carry the server, agent and setup screens. A
  test keeps server packages out of its imports.
- `llmgw-kb sync <collection> <dir|file…> [--include/--exclude globs]
  [--dry-run] [--delete]`: scan → local catalog (path, size, mtime, sha256;
  cache under os.UserCacheDir, digests not recomputed for unchanged files)
  → the collection's manifest from the gateway (name → digest only) → diff
  → for new/changed files only: extract, chunk, embed through the gateway's
  `/v1/embeddings` with the person's key (its accounting, limits, guard and
  band apply), and send name, digest, chunk texts and vectors; `--delete`
  removes documents whose files are gone. `--dry-run` lists what would leave
  the machine and how much. A file that cannot be read (locked, unsupported,
  no text) is skipped with the reason; the run goes on.
- **Formats, as many as possible.** Built in, pure Go: PDF (text layer),
  DOCX/XLSX/PPTX, ODT/ODS/ODP, RTF, EPUB, HTML/XHTML, EML (text and HTML
  parts), Jupyter notebooks, subtitles (SRT/VTT), CSV/TSV, JSON/YAML/XML,
  Markdown/RST/AsciiDoc/Org, plain text and source code; ZIP archives
  recursed into. With a helper found on the machine: legacy DOC/XLS/PPT, RTF
  and anything else LibreOffice opens (`soffice --headless --convert-to`),
  images and scanned PDFs through Tesseract (`tur+eng`). Text in UTF-8,
  UTF-16 (BOM) or Windows-1254/1252 is converted to UTF-8.
- **Across systems:** document names relative to the synced root, `/`
  separated, Unicode NFC (macOS hands out NFD), a warning for two names that
  differ only in case; CRLF → LF; key from `LLMGW_API_KEY` or `llmgw-kb
  login` (os.UserConfigDir, 0600 where that means something); the system's
  CA store, `--ca` for another.
- **Gateway endpoints** (client listener, person's key):
  `GET /v1/kb/collections` (what this key may read/write),
  `GET /v1/kb/collections/{id}/manifest`, `PUT
  /v1/kb/collections/{id}/documents` (one document per request: name,
  digest, bytes, chunks with text and embedding; dims checked, chunk count
  and size bounded), `DELETE …/documents?name=`. Write needs a write grant
  or ownership on the key's chain (`CollectionsFor(…, write)`); otherwise
  404.
- **Administration** stays with operators: `llmgw ctl kb
  create|list|delete|grant|revoke|docs`, admin API `/api/v1/kb`, audit,
  console screen; `docs/knowledge.md`.
- Trade-off accepted: vectors come from the client, so a writer could store
  vectors that do not match their text — only in collections they may
  already write.
Built: `internal/kb/extract` (formats above; helpers found on PATH and in
each system's install places), `internal/kb/chunk` (≈1800 characters under
the heading path, a paragraph of overlap, sections kept apart once a chunk
is a third full), `internal/kb/kbsync` (ledger in SQLite at
os.UserConfigDir/llmgw-kb/ledger.db: documents keyed by gateway, collection
and name with file, size, mtime, digest, state, reason; runs), `cmd/llmgw-kb`
(login, collections, sync, status, history, formats), `internal/router/kb.go`
(the four gateway endpoints, `X-Llmgw-Project` for personal keys, audit),
`internal/control/adminapi/kb.go` + `llmgw ctl kb` + console Knowledge
screen, `docs/knowledge.md`; CI builds the client for all six targets and
releases attach it. Tested: every format from generated fixtures and a
Turkish PDF (OCR path with Tesseract here), sync against a fake gateway and
end to end against the router on pgvector.
Tried on the PoC 2026-10-08 (control planes 140 and 150 on a dev build of
this branch; their previous binary is /usr/local/bin/llmgw.pre-rag): Ubuntu's
pgvector 0.6.0 installed on 149/150 and enabled; bge-m3's weights were never
in the mirror (fetched from CompendiumLabs/bge-m3-gguf@1864f47f) and it is
held at 1 replica; tree Demo/ik/belgeler and Demo/satis/crm with a key each;
collection Demo/politikalar filled from a Mac through an SSH tunnel with ten
Turkish files in nine formats (9 sent in 5 s, the .doc skipped for want of
LibreOffice; a rerun sent nothing in 0 s; a changed file alone was sent).
Nine Turkish questions each found their page first once search scored by
similarity plus word coverage (rank fusion had got two wrong). The other
team's key saw no collection and got 404 by id; with a read grant it read
and could not write; revoked, nothing again. On the live store the llmgw
role (not superuser, no BYPASSRLS) sees 0 chunks with no scope set and 0
with the organization alone.
Left: the gateway's collection list shows the owner as team:ID, not its
path; no LibreOffice on the dev machine, so the legacy
Office path is untested beyond its error; the client is unsigned (Gatekeeper
and SmartScreen warn); Outlook `.msg` is not read; one document over ~60 MiB
of JSON (a few thousand pages) is refused rather than split.
PostgreSQL tests of store, router and adminapi share LLMGW_TEST_POSTGRES and
the store's TRUNCATE each test; run them with `go test -p 1` or one package
at a time, or a router test loses its rows mid-run.

### F3. Retrieval on the request path (M) — done 2026-10-08 (branch `rag`)
- `llmgw.knowledge: [collections]` in the body (stripped before the
  engine), or default collections per project/key.
- Allowed set from the caller's chain (§7.9.2), cached like limits; grants
  add up down the tree. Personal keys and sessions need `x-llmgw-project`
  (or the playground's picker) naming a scope the person is a member of.
- A collection not allowed answers 404, exactly as a missing one.
- Hybrid search with the filter inside the query (per collection, its own
  partial index; pgvector 0.6 is enough —
  scored by similarity plus the share of the question's words held (built
  in F1/F2 as SearchChunks), `/v1/rerank` when a reranker is in the catalog. Top
  chunks into a system message with document and position;
  `x-llmgw-sources` lists them.
- Leaks closed: answer-cache key includes the allowed set; per-organization
  `cache_salt` on RAG requests; injected passages under `no_store` and the
  guard's prompt-injection check; returned doc ids in the audit.
- Done: router tests that tenant B never receives tenant A's chunks
  (including via cache), that a personal key without a project is refused,
  metrics (retrieval latency, chunks returned), docs.

Built (`internal/router/rag.go`): `X-Llmgw-Knowledge` (header, as the
other llmgw extensions are, not a body field), `X-Llmgw-Knowledge-K`;
collections from the key's chain (kbEffective, shared with the kb
endpoints), 404 for one not readable; the question embedded through the
ordinary path on the caller's principal; SearchChunks; passages trimmed to
the model's context; added to the one leading system message (or a new
one); `x-llmgw-sources`; before guardInput; never cached; metrics. Default collections built (kb_defaults on org/team/project/key, narrowest
wins, 30 s cache, `ctl kb default`, console action; never refuse); the
portal chat's Documents picker (collection, default or none; sources listed
under the answer; a header the page sends itself is dropped). `cache_salt` per organization on requests given passages, sent to vLLM
replicas only (chosen replica's engine). Checked on the PoC 2026-10-08:
vLLM 0.30.0 (cuda01) with TinyLlama-1.1B AWQ run by hand beside the 35B in
the card's last 2.3 GB — a 668-token prompt reused 656 tokens under the same
salt and 0 under another's, with identical text (llmgw cannot place it
there: placement takes whole GPUs). Live on the PoC: a team default answers
a request without the header, `none` turns it off, an explicit header with
k=2 gives two sources and no more, llmgw-kb lists owners by path. Left: returned passages
are not audited per request (the header and the kept request record them).

### F4. Adapter training (L, after F1–F3) — done 2026-10-08 (branch `adapters-f4`)
A run-to-completion job placed by the control plane like a replica and
supervised by the agent like an engine, in a Python environment the agent
installs (TRL + PEFT, QLoRA). Data: Q/A pairs generated from a collection by
a cluster model through `/v1/batches`, reviewed by people. Output to the
mirror, into the model's `adapters`, through canary and a quality suite.
Runs when the scaling schedule frees a GPU; 8–14B dense bases on one card.
Never train on a collection some users of the adapter may not read.
Tried 2026-10-08 on the PoC, by hand: Qwen2.5-0.5B-Instruct (fp16) with a
LoRA (r=16, every linear layer, 8.8M trainable), trained beside the 35B in
cuda01's last 2.3 GB (peak 2158 MiB on the card, 64 s for 138 steps, PEFT in a
venv over the vLLM engine's torch). Data: 90 Turkish Q/A pairs a cluster
model wrote from the nine Demo/politikalar documents, through the gateway.
On nine questions it had not seen, phrased otherwise: base 0/9, 6 epochs
4/9, 4 epochs at a lower rate 2/9. Wrong answers mixed facts across
documents (the meal card's amount, the office day). The same nine through
retrieval: 9/9 with sources. So F4 is for format, vocabulary and tone, and
facts come from F1–F3; a bigger base on a free card would do better but not
change that. The training itself fits a small slot on a shared card, which
the job design can rely on for small bases.

Built: `ctl kb dataset` (POST /api/v1/kb/collections/{id}/dataset, NDJSON
with each pair's collection); train_jobs/train_artifacts (migrations 0037,
0038 owner); internal/control/train (queue, one job per node, artifacts in
1 MiB pieces, a node that goes fails its job); internal/agent/train (the
newest vLLM's python3 + `uv pip install --no-deps --target` peft 0.21.2 and
accelerate 1.15.0; llama.cpp's conversion code for the installed release,
untarred from its source; the card with the most free memory, if enough;
runs as llmgw-engine); `ctl train start|list|show --follow|cancel|rm|
download|publish`; the mirror's PUT /v1/models/{model}/{version}/{file};
publish uploads the GGUF (llama.cpp) or PEFT files (vLLM) and sets the
adapter after checkAdapters. Owners: catalog.Adapter.Owner, enforced in the
router (404, hidden from /v1/models); a job's owner defaults to its single
collection's owner, must be able to read every collection the pairs came
from, and pairs from a knowledge base are never public. Tried on the PoC:
dataset of 90 pairs in 89 s; a job trained beside the 35B (peak 2214 MiB,
138 steps in about a minute) and sent back PEFT and GGUF.
Left: a job dies with its agent's stream (no resume); no console screen for
training yet; quality gating of a published adapter is by hand (ctl eval).

---

### F5. Large imports (M) — measured 2026-10-09, open
The metallurgy corpus (2,314 files, 108,779 chunks, bge-m3 on the one
GPU) went in at about 25 chunks/s, the same over a phone hotspot and the
LAN. The engine's slots were the limit: llama.cpp `--parallel 4` embeds
four texts per pass, while llmgw-kb sends 32 a request from 6 workers. The
usage rows show 4.6 s a request on average, 1.8 s of it in the gateway's
queue and 0.04 s in the guard. The GPU sat at 18–50% and the store machine
at a third of its CPUs.
- An embedding model's profile with many slots by default (`--parallel`
  32–64, `--ctx-size` to match, `--batch-size`/`--ubatch-size` at least the
  longest chunk), or `ctl model serve` choosing that for a model with
  `capabilities.embeddings`. With the default 512, one 516-token chunk failed.
- Embedding on the gateway's side of a document PUT, so the client sends
  text only: today an embeddings answer is ~21 KB of JSON a chunk, and
  the text and vector go up again (~11 KB).
- The store next: HNSW inserts took half of one of pg01's two cores; at a
  million chunks the vectors alone outgrow its 4 GB. halfvec (pgvector ≥
  0.7) and a larger store machine.
- Done: a batch of embeddings checked text by text against the context
  (branch `embeddings-batch`); it was refused as one over-long prompt.
- Done: `query_prefix` on an embedding model, before questions only.
- Qwen3-Embedding-4B on the RTX 6000 (2026-10-09): llama.cpp Q8 1,575
  tok/s as first set up (32k cache, 16 slots), 2,700 with a 16k cache,
  FlashAttention and ubatch 1–2k; f16 no better. vLLM fp16 (`--runner
  pooling`) 6,200 direct, 4,700–4,900 through the gateway in the import,
  and its vectors match llama.cpp's (cosine 0.9995). The 103,579 chunks
  took 3 h. Done (2026-10-10, branch `admission-bulk`): a source per
  profile, so an embedding model's GPU profile runs vLLM and its CPU
  profile llama.cpp; a band's own `queue_ttl` for bulk imports, and
  `Retry-After` on a busy refusal, which is what had turned passages away
  (llmgw-kb retries only what says when to).
- Retrieval on 40 questions written from the corpus, document in the top
  5: bge-m3 31/40 (Turkish questions 13/21), Qwen3-Embedding-4B 37/40
  (18/21), with its query prefix 36/40 and first place 31/40 instead of
  28 (MRR 0.738 → 0.789 → 0.838). Next: translating the question with the
  chat model, and a reranker.

## G. Jobs: 3D generation through ComfyUI (2026-10-10)

A request for a 3D model takes minutes on the one 24 GB card, and the card has to be
emptied of the chat model first. That is not a request to hold a socket for
— `internal/router/activate.go:46` already says so — it is a job: queued,
answered with an id, asked after, its result downloaded. The job machinery
is built first and tested without a GPU; the ComfyUI generalisation and the
custom-node install come after. Decided 2026-10-10: a `jobs` table of its
own rather than a one-line batch (the JSONL shape would be artificial and
has nowhere for progress), and progress is shown, not just a status.

Which models (decided 2026-10-10): the assets go into a game sold on
Steam, worldwide, the EU included, so the licence of the output decides
before the quality does. Hunyuan3D (every version) is out: Tencent's
licence excludes the EU, UK and South Korea, and whether a game shipped
there with its outputs is "use in the Territory" is not a question to
leave to a court. In, in this order: **TRELLIS.2** shape (MIT; the best
geometry; its own texture baking goes through nvdiffrec and nvdiffrast,
both under NVIDIA's non-commercial source licence, so that stage is not
run), **Step1X-3D** (Apache 2.0, StepFun; shape and texture in one, the
EU-safe textured path — its ComfyUI node and VRAM need are to be
checked), **Direct3D-S2** (MIT, DreamTech; high-detail shape, has a
ComfyUI node) if TRELLIS.2's detail falls short. Not in: TripoSR and
InstantMesh (permissive but superseded), Stable Fast 3D (Stability's
community licence, revenue-capped; served all the same since 2026-10-10
at the user's request, see G3), SAM 3D Objects (Meta's own licence,
to be read before it is considered). Steam's disclosure form (changed
2026-01-16) lists pre-generated 3D models as content that must be
declared on the store page: the licence notice G3 writes into the GLB is
what the studio answers that form from.

### G1. Jobs: store, worker, endpoints (M) — done 2026-10-10 (branch `jobs-g1`)
Built as written, with two choices: a job found running after its gateway
died is run again from the start (the prompt id column is there for G2 to
follow the same work instead), and the request is validated at the door
only as far as costs nothing — key, model, allowlist, the request's shape
— with limits, guardrails and admission applied when the worker replays it,
said as the job's error. The result file's `content_type` is the job's; the
files endpoint still serves every file as an octet stream.
- Store: `jobs` (id, owner key/principal, kind, model, status, input
  (JSON), progress (JSON: queue position, percent, step), result file id,
  error, created/started/finished, lease owner/until), migrations for
  sqlite and postgres beside `internal/store/batches.go`, whose
  claim/renew/release lease pattern is copied, not shared: a job is one
  unit of work, a batch is many lines.
- Worker: `internal/router/jobworker.go` after `batchworker.go`. Any
  gateway on the store claims a queued job, asks for the model with
  `activator.Want` and waits as long as it takes (no `ActivationWait`
  bound: nobody is holding a connection), runs it, stores the output
  through the files store, writes usage and price, marks it done. A
  gateway that dies mid-job loses its lease and another one picks it up;
  the ComfyUI prompt id is in the job row so the second gateway asks
  after the same work rather than queueing it again.
- Endpoints: `POST /v1/3d/generations` validates like any inference
  request (key, allowlist, limits, admission, guardrails on the prompt),
  writes the job, answers 202 with `{id, status: "queued"}`;
  `GET /v1/3d/generations/{id}` returns status, progress and, when done,
  `output_file_id` and the usage; `POST .../{id}/cancel`; the file comes
  from `GET /v1/files/{id}/content`. Owner-scoped like batches. Results
  are kept for the batch files' retention and deleted with them: a
  departure from `images.go`'s "kept nowhere", because a mesh is a
  download, not a JSON field; the retention is the bound.
- Done when: a job survives the gateway that took it being killed, a
  cancelled job leaves ComfyUI's queue (`comfy.Client.cancel`), two
  gateways never both run one job (test with the lease), and the usage
  row carries the job's price.

### G2. Progress from ComfyUI (S, after G1) — done 2026-10-10 (branch `jobs-3d`)
Built as written: the gate's websocket under its own client id, the
client_id of every /prompt rewritten, /progress/{prompt_id} on the gate,
the router reading it and /queue into the job (only into a job of the same
key). Left: a node that reports no steps (TripoSR's) shows its name and no
percent; a vanished replica is said by the job failing, not by progress.
The gate (`internal/comfy/gate.go`) passes seven paths and no websocket.
Rather than open `/ws`, the gate subscribes to ComfyUI's own `/ws` once,
with `golang.org/x/net/websocket` (already a dependency), under a client
id of its own, and rewrites the `client_id` of every `/prompt` it
forwards to that one. It keeps the last `executing`, `progress`
(value/max/node) and `execution_error` message per prompt id and serves
them at `GET /progress/{prompt_id}` on the gate; the router polls that
beside `/history` and `/queue` and writes queue position, percent and the
current node's class into the job's progress. The gate's buffer is
bounded (the last N prompts) and the websocket reconnects. Done when a
3D job shows "queued, 2 ahead" then "running 40%, step 12/30" in
`ctl jobs` and on the account page, and a job whose replica vanished
says so within one poll.

### G3. ComfyUI beyond pictures: 3D models and custom nodes (M, after G1) — done 2026-10-10 (branch `jobs-3d`), with TripoSR
Built: output kinds, image upload through the gate, every output list
fetched, OBJ to GLB, licence in /v1/models, the job and the GLB, per_mesh
and usage.meshes (migration 0045), custom nodes cloned at their commit by
the gate with packages beside the replica (a node's named packages at its
range, the rest at ComfyUI's versions). Not as written: the model. On the
PoC's Turing card TRELLIS.2 cannot run — it needs Meta's gated DINOv3,
its published background remover (RMBG-2.0) is non-commercial, and its
sparse attention and Direct3D-S2's need flash-attention (Ampere and
newer). TripoSR (MIT) proves the chain on cuda01: a .glb from a picture
in about 30 s. Left: background removal before TripoSR (BiRefNet, MIT,
as a node), TRELLIS.2 or Direct3D-S2 on an Ampere card with an ungated
encoder, a missing node refusing the replica by name.
Added 2026-10-10 at the user's request, beside the order above: **Stable
Fast 3D** as `sf3d` (Stability AI Community License: free under US$1 M
annual revenue, an enterprise licence above, "Powered by Stability AI"),
the textured mesh that runs on Turing: about 1 s a mesh warm in fp16,
UV-mapped GLB with its texture, 18 GB of the card at peak. Its weights are
gated, so they were fetched once with the user's token into
/srv/llmgw-comfy/stable-fast-3d and linked into the engine user's
Hugging Face cache, which its loader node reads by repository name; no
token is on the machine at runtime. Its own save node returns the GLB as
base64, which the gate cannot fetch, so `LLMGWSaveMesh` writes the file;
`LLMGWPrepareForMesh` now also gives the object's mask. Left: its two
CUDA extensions (texture_baker, uv_unwrapper) are not on PyPI, so they
were built on cuda01 and installed into the engine's venv by hand; a
custom node's requirements need a way to name a local build, or the
engine install must carry them, before a customer site gets this. And
rembg's default session now fetches BRIA RMBG-2.0 (non-commercial):
every workflow must name its rembg model (ours say u2net).
- Catalog: a model makes `pictures` or `meshes` (`MakesImages` becomes an
  output kind; `ServesComfyUI` stays). 3D weight folders in the gate's
  `modelFolders`; the workflow's placeholders gain `image` (the input
  picture, base64 in the request, written to the replica's `input/` by
  the gate through a new `POST /upload/image` the gate allows — or a
  base64-loading node, whichever TRELLIS.2's wrapper wants).
- `comfy.Client.fetch` reads every output key, not `images` only, and
  fetches each file named by `/view`; `Image` becomes `Artifact` with its
  content type (model/gltf-binary, application/octet-stream for .obj).
- Custom nodes: a model (or the engine config) names git repositories
  and a pip requirements list; the agent's provision step clones them
  into the replica's `custom_nodes/` (the gate's `--base-directory` is
  where ComfyUI looks, not the install directory) and installs the
  requirements into the engine's venv with the same PyTorch index and
  lock discipline as A8. A replica with a missing node refuses to start
  with the node's name in the error rather than ComfyUI's traceback.
- Licence of the output (decided 2026-10-10: it matters). A catalog model
  carries `license: {name, url, notice}`; a job's answer, `/v1/models`
  and the portal page show it beside the result, and the mesh's GLB
  `asset.extras` carries the notice too, since the file travels without
  the API. What was checked on 2026-10-10: TRELLIS.2 weights are MIT and
  its outputs free, but its texture baking depends on nvdiffrast, whose
  NVIDIA Source Code License is non-commercial — a commercial service
  must render textures another way or skip that stage. Hunyuan3D 2.0 and
  2.1 claim no rights in outputs, but the licence excludes the EU, UK and
  South Korea, needs Tencent's permission above 1 M monthly active users,
  forbids using outputs to train other models, and asks for "Powered by
  Tencent Hunyuan"; the notice ships with any distribution. TripoSR is
  MIT, InstantMesh Apache 2.0. Every one of them trained on Objaverse,
  some of whose objects are CC-BY-NC: a provenance question no licence
  text settles, to be said in the notice rather than hidden.
- Pricing: `per_image` stays; `per_mesh` beside it in `accounting.Price`,
  `Outcome.Meshes`, the usage sums and exports (as eaea671 did for
  pictures).
- Done when: TRELLIS.2 on cuda01 with `scaling: {min: 0,
  idle_timeout: 10m}` makes a .glb from a picture through G1's endpoint
  with nvdiffrast and nvdiffrec not installed in its venv (the licence
  is kept by the install, not by a note), the GLB carries the MIT
  notice,
  the card is free again after the idle timeout, and the same request
  against flux-schnell is refused as "makes pictures, not meshes".

### G4. Portal and console (S, after G2) — done 2026-10-10 (branch `jobs-3d`)
Built: /3d with model-viewer served from the gateway, progress, download
and licence; `ctl 3d list|get|cancel` over /api/v1/client-jobs (ctl jobs
was taken by the control plane's own work). Left: the console's Jobs view.
A 3D page beside `/images` (upload a picture, watch progress, download
the .glb, a viewer with `<model-viewer>` from the allowed CDNs), `ctl
jobs list|get|cancel`, the console's Jobs view. The page sends down the
same path as the API, like `imagesSend` does.

## H. ComfyUI as a platform (written 2026-10-11)

G made one model at a time work: flux for pictures, TripoSR and SF3D for
meshes, each with its own endpoint, its own request shape and its own
replica. The user's direction (2026-10-11): llmgw is a generic platform,
and should serve whatever ComfyUI can run — pictures, edits, video,
audio, meshes, upscaling, segmentation — as data in the catalog rather
than code per model. Two decisions shape everything below:

- **The licence is metadata, not a filter.** llmgw does not leave a
  model out because of its licence. The licence travels with the model
  and with every file it makes; what a site will not serve is that
  site's policy (H6). The game studio's EU constraint is one such policy,
  not a decision the code makes for everyone.
- **The install fits the hardware.** The PoC has one Turing card; other
  sites will have Ampere, Ada and Hopper. A catalog entry must be
  writable once and placed sensibly anywhere: fp16 on Turing, bf16 or
  fp8 where the card has them, flash-attention models only where it
  runs, CUDA extensions built for the card that runs them (H2, H4).

The first set of models proves the structure at its corners rather than
by licence: Qwen-Image-Edit (several weight files, two pictures in),
Wan 2.2 small (video out, a long job), ACE-Step (audio out), SF3D (a
package not on PyPI, gated weights), Hunyuan3D (a licence that excludes
the EU, which H6 must be able to say no to).

### H1. A workflow says what it takes and makes (M) — built 2026-10-11, not yet tried on the PoC
Built as written, with three choices: a run answers with one file, the
first of its output kind (several outputs as one zip is left); the
workflow's placeholders are checked against the inputs when a run is
served, since the catalog never reads the workflow file; and video,
audio and files count as nothing in usage until a per-run price exists
(per_image and per_mesh still count pictures and meshes).
- Catalog: `comfyui.inputs` — a list of `{name, type, required, default,
  min, max, choices, max_mb, doc}`, type one of text, int, float, bool,
  choice, image, mask, audio, video, file. Each is a placeholder of the
  same name in the workflow. The template's own placeholders are checked
  against the list when the catalog is built: one the workflow uses that
  no input or built-in names is refused there, not at the first request.
- `comfyui.output` gains `video`, `audio` and `files` beside `pictures`
  and `meshes`; `files` takes whatever the workflow saves. `comfy.Image`
  becomes `comfy.File` and `typeByName` knows mp4, webm, gif, wav, flac,
  mp3, ogg.
- Endpoint: `POST /v1/workflows/runs` with `{model, inputs: {...}}`, a
  job like G1's (`GET /v1/workflows/runs[/{id}]`, `.../cancel`). A file
  input is base64, a data URL, or a `file-…` id from `/v1/files` for
  what is too big to send inline (video, long audio). Files go into the
  replica's input directory through the gate's `/upload/image`, which
  ComfyUI uses for any file. `/v1/images/generations` and
  `/v1/3d/generations` stay as they are, as fixed shapes over the same
  machinery.
- The models list (`/v1/models`) gives each ComfyUI model its inputs and
  output kind, so a client can build a form without reading the catalog.
- Done when: the existing flux, triposr and sf3d entries work unchanged,
  and one model with a picture and a text input (Qwen-Image-Edit) runs
  through `/v1/workflows/runs` with a 400 for a missing required input
  before any job is written.

### H2. Profiles that fit the hardware (M) — built 2026-10-11
Built as written. A profile by need is left out of placement until the
cluster's cards are known: with no class, placement would have read it as
a CPU profile. A profile's own `source` is loaded under its own file name
by the workflow's `{{checkpoint}}`.
- A profile may say `gpu: {min_compute_capability, min_memory_gb}` in
  place of `gpu_class`. When the reconciler places a model, such a
  profile stands for every GPU class in the cluster that meets it, at
  the profile's priority: the catalog function that turns profiles into
  placement's view takes the cluster's classes (class, compute
  capability, memory) and returns one concrete profile per class that
  fits, named `<profile>@<class>`. Placement itself is unchanged: it
  still sees exact classes, and a tensor-parallel group still never
  mixes them.
- A ComfyUI profile may carry its own `comfyui` block (workflow, nodes,
  checkpoint) over the model's: the fp8 workflow on compute 8.9 and up,
  the fp16 one on 7.5. A Turing card is not an error, it is the profile
  that fits it.
- Done when: one catalog entry with an "ampere+" profile (min 8.0) and a
  "turing" fallback places the fallback on cuda01, and the same entry,
  in a test with an A100 in the inventory, places the other.

### H3. Environments: which models can share a replica (S, the catalog side)
- `comfyui.environment: <name>` puts a model in a named environment.
  Models in one environment must agree on every custom node they both
  name (same repository, same commit) and their requirements must
  resolve together; the catalog refuses an environment that cannot, with
  both models' names. Models with no environment are each their own, as
  today.
- `Catalog.Environments()` returns each environment with its models and
  the union of their nodes: what one replica of it must install.
- Placement and eviction of a replica that serves every model of its
  environment (so flux, SF3D and Qwen-Image-Edit stop handing the card to
  each other with a 30 s cold start each time) is the other session's
  work (llmgw-b7), built on this.

### H4. Custom nodes with packages that are built, not downloaded (S) — built 2026-10-11
Built as written, with `uv build --no-build-isolation` against ComfyUI's
Python, the card's capability asked of torch in the replica (so
CUDA_VISIBLE_DEVICES is the card it will run on), and wheels kept under
the engines' cache by commit, `sm<arch>` and directory. Also
`comfyui.huggingface_repo`: a node that loads its weights by repository
name (SF3D's) finds them in the engines' Hugging Face cache, linked by the
gate, so no token is on the node.
- `nodes[].build: [subdir, ...]`: directories of the node's repository
  that are Python packages with C++/CUDA extensions (SF3D's texture_baker
  and uv_unwrapper). The gate builds each as a wheel with the engine's
  Python, `TORCH_CUDA_ARCH_LIST` set from the compute capability of the
  card it runs on, and installs it into the replica's own `site` like
  any other requirement. Built wheels are kept under the engine's home by
  node commit, subdirectory and architecture, so the second replica on
  the same kind of card does not build again.
- A node needing nvcc on a node without it refuses the replica with that
  said, not with a compiler's traceback.
- Done when: SF3D runs on cuda01 with nothing installed into the engine's
  venv by hand (the hand install of 2026-10-10 removed first).

### H5. Licence on every output (S)
- `License` gains `id` (SPDX where there is one, else a lowercase name:
  stability-community, tencent-hunyuan-community), `commercial` (unset is
  not said), `excluded_regions` (ISO 3166 codes, `EU` for the union),
  `revenue_cap_usd` and `attribution` ("Powered by Stability AI"). All
  optional but the name. Done 2026-10-11 in the catalog, so H6 can match
  on `id`.
- Written into the file where the format has a place for it: glTF
  `asset.extras.license` (as now), PNG an `iTXt` chunk named `license`
  (the workflow's own text chunks are still stripped), MP4/WebM and the
  audio formats in their metadata where ffmpeg is at hand on the router,
  otherwise beside it in the job's answer only — said in the docs.

### H6. Licence choices (S) — the other session's (llmgw-b7), at the user's request
The user asked (2026-10-11) for the choice to be a person's: on their
profile they pick the licences they will not use, and their keys and
portal session neither see those models in `/v1/models` nor may call
them (403, saying which licence). Matched on `license.id`, and on
`commercial` / `excluded_regions` where a choice is put that way. Its
store table is migration 0046. A site-wide or team-wide default can sit
on top of it later.

### H7. A job worker per model, not one for all (S)
The worker runs one job at a time: a job waiting for the card holds up a
job for another model that could run now (seen on the PoC, 2026-10-10).
Jobs are claimed and run concurrently up to a limit per model (or per
environment, after H3) and a cluster-wide cap, with the lease and
leader-only rules of G1 kept.

## D. Facts worth knowing before starting

- Admin API: `127.0.0.1:9090`, no auth, `ctlClient` in `internal/cli/ctl.go`.
  Metrics on `:9091` with `/prometheus/targets` beside them.
- Store: `internal/store`, migrations under `internal/store/migrations/{sqlite,postgres}`;
  the PoC runs PostgreSQL under Patroni.
- The console (`sudo llmgw`) is `internal/cli/tui_console.go` with actions
  in `tui_actions.go`; every operator-visible change needs a row action or
  a form there, and a line in `docs/console.md`.
- Tests of the console use a fake admin API (`internal/cli/tui_actions_test.go`,
  `adminStub`); tests of serving use fakes in
  `internal/control/modelserve/modelserve_test.go`.
- `go generate ./...` regenerates `docs/reference.md` and
  `internal/docgen/reference_gen.go` whenever a config struct changes; the
  docgen test fails otherwise.