llmgw/BACKLOG.md

# Backlog

What a review of the code on 2026-10-04 (commit c2238ab) found should be done
next: security fixes first, then guardrails, then features. Written for
whoever implements it — a person or a model — without the review in hand.
Every item names the files it touches and what "done" is. Paths are relative
to the repository root. Sizes: S about a week, M two to four weeks, L more.

Conventions this codebase keeps, which every item here must keep too:
- A change is refused before anything is written (validate, then persist).
- Nothing on the serving path logs prompt text, keys or passwords.
- Each item gets unit tests beside the code (`*_test.go`), and the TUI
  (`internal/cli/tui_*.go`), `docs/console.md` and `CHANGELOG.md` are
  updated when behaviour an operator sees changes.
- `go test ./...`, `go vet ./...` and `govulncheck ./...` pass. No release is
  cut and nothing is deployed to the PoC cluster without being told to.

---

## A. Security — do first

**Status (2026-10-04): done and merged.** A1–A9, A6b included — etcd and
Patroni's API on TLS, the PoC moved in maintenance mode without a failover.
Deliberately narrower than written: the admin API is a token on loopback
rather than a unix socket (A5); thread ids are bounded and documented as
project-wide rather than scoped per user (A8); Python engines resolve with an
explicit PyTorch index and a PEP 751 lock rather than locks shipped per engine
version (A8). All of A is live on the PoC.

### A1. Toolchain (S, hours)
`govulncheck` finds 11 standard-library vulnerabilities reachable from this
code at Go 1.26.3 (net/http, crypto/x509, os.Root, net/textproto). Raise the
`go` directive in `go.mod` to 1.26.6 or later, rebuild, run `govulncheck
./...` until it reports none reachable. Add `govulncheck` to the Makefile's
check target.

### A2. Release signing is effectively off (S)
- `internal/release/trusted.pub` is empty, so `TrustedKey()` returns "" and
  `release.go:103` skips signature verification. Ship a real public key;
  make a missing or invalid signature fatal. Drop the `LLMGW_RELEASE_KEY`
  environment override, or honour it only in `-tags dev` builds.
- `internal/onboard/onboard.go:365`: the node-side download checks only a
  same-origin `checksums.txt`. Verify the signature there too (simplest:
  have the node run `llmgw` itself to fetch and verify once it has any
  binary), and `shellQuote` the `Releases` URL, which is interpolated into
  single quotes unescaped.
- `internal/agent/upgrade.go:44`: accept only `https://` release bases.
- Done when: a tampered binary or checksums file is refused by self-upgrade,
  node onboarding and agent upgrade, with a test for each.

### A3. Agent port without TLS is anonymous and plaintext (S)
`internal/cli/server.go:423` serves h2c on `:7443` when `agent_tls` is not
configured; `agentsvc` then accepts any node id. Refuse to start control
mode without TLS unless `agent_tls.insecure: true` is set explicitly; when
insecure, bind `agent_listen` to 127.0.0.1 and log a warning at startup and
in `llmgw doctor`. Keep `server init` writing TLS as it does.

### A4. Model profiles reach root on every node (M)
Profile `args` and `env` from a model definition are appended to the engine
command unchecked (`internal/agent/engines/llamacpp.go:170`, `vllm.go:151`,
`sglang.go:98`); `internal/agent/engines/unsafeenv.go` is a denylist that
misses `NCCL_NET_PLUGIN`, `NCCL_TUNER_PLUGIN`, `NCCL_PROFILER_PLUGIN`,
`CUDA_INJECTION64_PATH`, `NVTX_INJECTION64_PATH`, `TRITON_*` paths; engines
run as root.
- Replace the env denylist with an allowlist (CUDA_/NCCL_ tuning keys minus
  anything ending in `_PLUGIN` or containing `INJECTION`, `VLLM_*`,
  `HF_*` minus path keys, `OMP_*`, `GGML_*`). Apply at catalog validation
  (`internal/control/catalog/catalog.go`, where `UnsafeEnv` is called) and
  again in the agent.
- Deny engine flags that write files or run code: `--trust-remote-code`,
  `--log-file`, `--slot-save-path`, `--chat-template` with a path,
  `--host`, `--api-key`, `--port`; vLLM/SGLang equivalents. Allowlist is
  better than denylist if the flag set is enumerable per engine.
- Run engines as an unprivileged user with the video/render groups
  (`internal/agent/supervisor`, systemd unit in
  `internal/cli/serverinit_cluster.go:52`). Weights cache must be readable
  by it.
- Done when: a profile with `--trust-remote-code` or `NCCL_NET_PLUGIN` is
  refused at `ctl model add` and at the agent; `ps` shows engines not as
  root.

### A5. Admin API trusts any local process (M)
`internal/cli/server.go:357` serves the admin API and pprof on
`127.0.0.1:9090` with no authentication; `adminapi.actorOf`
(`internal/control/adminapi/api.go:1613`) takes the actor from the
client-sent `X-Llmgw-Actor` header.
- Serve the admin API on a unix socket (`/run/llmgw/admin.sock`, 0660,
  group `llmgw-admin`) by default, keeping TCP as an option that requires
  a bearer token read from a root-only file (`/etc/llmgw/secrets/admin-token`).
- Take the actor from `SO_PEERCRED` (uid → username) on the socket, or from
  the token's name; ignore the header.
- Refuse `admin_listen` on a non-loopback address unless `admin_insecure:
  true`.
- Put pprof behind its own `debug.pprof: true` setting, off by default.
- `ctlClient` (`internal/cli/ctl.go`), the console and `node add` must work
  over the socket; `sudo llmgw` already runs as root.

### A6. Patroni REST and etcd are unauthenticated and plaintext (M)
`internal/pgha/config.go:99` writes a `restapi:` with no `authentication`
and `:54` etcd over `http://` with no client auth; `hba.go:56` PATCHes
Patroni's config unauthenticated. Anyone reaching :8008 or :2379 owns the
store.
- Generate a Patroni REST username/password at setup, store it in
  `/etc/llmgw/secrets/`, write `restapi.authentication` and
  `restapi.certfile/cafile` from the cluster CA; send it from
  `internal/pgha/hba.go` and `status.go`.
- etcd: client and peer TLS from the cluster CA with `--client-cert-auth`.
- Document, and have `server init` write, nftables rules limiting
  2379/2380/8008/5432 to member addresses (see A9).
- Done when: an unauthenticated `curl :8008/config` is refused and the
  store status in the console still works.

### A7. PostgreSQL traffic is plaintext (S)
`internal/pgha/config.go:168` and `internal/cli/serverinit_postgres.go:330`
use `sslmode=disable`; HBA lines are `host`. Enable server TLS in Patroni's
postgresql parameters with a certificate from the cluster CA, write
`hostssl` lines, use `sslmode=verify-full&sslrootcert=/etc/llmgw/ca.pem`.
Existing clusters: `llmgw server` must keep working against a store that
has not been migrated, with a warning.

### A8. Smaller fixes, each S or less
- **Capacity fallback skips the allowlist.** `internal/router/router.go:700`
  builds `[canonical]+entry.Fallbacks` without `permits`. Filter the chain
  with `r.permits(p, model)`; price against the model that served.
- **Suspended users keep working API keys.** `store.(*DB).Resolve`
  (`internal/store/memberships.go`) never joins `users.state`;
  `Authenticator.Invalidate()` is never called. Reject keys whose owner is
  not active; call `Invalidate` on revoke, suspend, limit change.
- **Join codes never expire server-side.** `internal/join/join.go:103`
  checks expiry only in `Parse`; `admit` (`:212`) does not. Check it there;
  bind the listener to the chosen address, not `:7444` on all interfaces;
  use the peer's address rather than the claimed `hello.Addr`, and show the
  joiner's address for the operator to confirm before secrets are sent.
- **Leftover sudoers grant.** `internal/onboard/gossh.go:248` writes
  `NOPASSWD:ALL` and removes it only in `Close()` (`:271`). Write the grant
  with a self-removal (`at`/systemd-run timer, 30 min) so a crash leaves
  nothing; wipe password buffers after use. In `ssh.go:39` pass `--` before
  the target.
- **Auth cache and pre-auth DoS.** `internal/auth/auth.go:127`: unbounded
  map, one DB query per bogus well-formed key. Bounded LRU, negative cache
  with short TTL, per-IP token bucket before the DB.
- **HTTP server hardening.** `server.go:351-360`, the h2c agent server,
  both mirror servers: set `ReadHeaderTimeout` (exists), `IdleTimeout`,
  `MaxHeaderBytes`, a body-read deadline, `netutil.LimitListener`; connect
  `WithReadMaxBytes` on the agent stream.
- **Login/signup abuse.** Default `signup_open: false`; per-IP token buckets
  on `/login` and `/signup`; bounded wait for argon2 slots; CSRF token on
  the login form; the per-address throttle must not lock the real owner
  out (use IP+address pairs).
- **Trusted proxies.** `internal/web/account.go:557`, `landing.go:60` honour
  `X-Forwarded-Proto/Host` from anyone; add `router.trusted_proxies` and
  only then trust them; `Cache-Control: no-store` on `/setup/*`.
- **Certificate revocation.** `DELETE /api/v1/nodes/{id}` only drops
  inventory; add a revoked-CN/serial list in the store checked in
  `agentsvc` `Connect` and mirror `onlyClients`; `serverinit_cluster`
  always passes `--api-client`.
- **Engine exposure.** Engines bind `0.0.0.0` (`llamacpp.go:102`,
  `vllm.go:109`): bind the management address. SGLang receives its key on
  the command line (`sglang.go:102`): pass it by env or file.
- **Backups.** `internal/cli/backup.go`: encrypt archives to an `age`
  recipient from `backup.recipient` in server.yaml; keep the directory 0700.
- **Python dependency confusion.** `internal/agent/provision/python.go:121`
  uses `--extra-index-url` with `unsafe-best-match`. Pin torch to the
  PyTorch index explicitly (uv `[[index]] explicit = true`), `first-index`,
  hash-pinned lockfiles per engine version.
- **Control-plane SSRF.** `internal/sourcecheck/sourcecheck.go:39` and
  `gguf.FromURL` use the default client; reuse the mirror's
  `fetchClient(false)` (private-address filter) and refuse a proxy unless
  configured; block NAT64/6to4-embedded private addresses in
  `internal/mirror/safefetch.go`.
- **Low:** `agentsvc/service.go:430` indexes `n.GPUs[0]` without a length
  check (CPU-only node panics the handler); thread ids are client-chosen
  with no length cap and shared per project (`router/threads.go`) — cap
  length, document, offer per-user scoping; `/v1/models` lists models the
  key may not use (`handlers.go:799`); sessions have no idle timeout and
  `PurgeSessions`/`PurgeClaims` are never called — add a janitor; add a CSP
  header and set `htmx.config.allowEval=false`; cap GGUF array nesting
  depth in `internal/gguf/gguf.go`; mirror `.source` sidecars are served by
  the public file server — exclude dotfiles from `:8081`; prune
  `modelserve.jobs` and the mirror `running` map; KaTeX → 0.16.21+.

### A9. Firewall (S)
No machine in a default install has a firewall. Have `server init`, `node
add` and `server add` write an nftables ruleset: ssh open; `:8080` open;
`:7443` and `:9091` open to the cluster's networks; `:9100` to the
monitoring host; `:2379/2380/8008/5432` to store members and control
planes; mirror `:8081` to the cluster, `:8082` to control planes; `:9090`
Prometheus and `:3000` Grafana to operators. `llmgw doctor` reports a
missing ruleset. Make it skippable with `--no-firewall`.

---

## B. Guardrails (M–L, build in this order)

**Status (2026-10-04, branch `guardrails-b`):** B1–B4 done, and tried live on
the PoC. That found four bugs, all fixed with tests:
- a shadow default merged with an attached policy of unwritten mode left it
  in shadow;
- a streamed word held back at one release was dropped at the next;
- llama.cpp continues a conversation ending on the answer, so Llama Guard's
  verdict on answers is now asked with its own prompt as a completion;
- a judge model was named in a client's "no endpoint" error.

The PoC runs `observe` (shadow) for everyone, and Llama Guard 3 1B
(`llama-guard-3-1b`, CPU, on cuda01) as a judge, which blocks a harmful
request in ~0.7s and a harmful answer in ~1.4s there. Still left: see each
item.

Today nothing on the request path reads, classifies, redacts or filters
message text in either direction (confirmed; the only "moderation" hit is a
comment in `internal/router/budget_fallback.go:20`). Existing controls are
about who, how much and how fast: allowlists (`router/handlers.go:450`),
RPM/TPM/budget (`internal/accounting`), context precheck
(`router/context_limit.go`), admission bands (`router/admission`),
timeouts (`router/proxy/proxy.go:26-57`). Gaps to close on the way: no
default output cap when the client omits `max_tokens`; `catalog.Capabilities`
is advertised but never enforced; tool results are trusted input.

### B1. Hooks in the proxy and router (S)
**Done (branch `guardrails-b`):** `internal/guard` holds the interface;
proxy `Filter`/`StreamFilter`; the input hook runs after `loadThread` (the
simpler choice); error codes, headers, Responses `incomplete`, usage columns
`guard_*`. Moved to B2: holding `delta.tool_calls` until they are whole and
reading their arguments — that is the tool policy's filter, not the proxy's.

- `internal/router/proxy/proxy.go`: add optional `InputRewrite func(body
  []byte) ([]byte, Verdict)` on the request and two output hooks: `Filter`
  for `relayBuffered` (between `ReadAll` and `WriteHeader`) and
  `StreamFilter{OnEvent(data) (emit []string, stop bool); OnEnd()}` for
  `relayStream`, called where each whole SSE event is assembled
  (`eventBuffer`, before `io.WriteString` at `:343`). Nil hooks cost
  nothing.
- `internal/router/handlers.go`: call the input policy after the allowlist
  and limits and before admission (around `:395`); the principal, scopes
  and the model that will serve are known there. Thread history is spliced
  later (`:416`): scan stored turns when written, or move the hook after
  `loadThread` (decide; the second is simpler).
- Blocking conventions: input → 400 `invalid_request_error` with
  `code: "content_policy_violation"` (today `writeError` always sends
  `code: null`); output non-stream → 200 with `finish_reason:
  "content_filter"` and empty/redacted content; output stream → a final
  chunk `{"choices":[{"index":0,"delta":{},"finish_reason":"content_filter"}]}`,
  then `data: [DONE]`, then cancel upstream. Headers `x-llmgw-guard:
  blocked|redacted|flagged` and `x-llmgw-guard-policy` set before the proxy
  writes headers.
- `internal/router/responses.go`: map `content_filter` to
  `response.incomplete` with `incomplete_details.reason: "content_filter"`
  (today it always emits `response.completed`, `:525`).
- Tool calls: hold `delta.tool_calls` until `finish_reason: "tool_calls"`,
  then validate; `replyDelta` (`reply.go:33`) must also read
  `tool_calls[].function.arguments`.
- Accounting: a blocked input is recorded like `RecordRefused` (nothing
  charged); a blocked output is charged (the GPU was used). `store.Usage`
  gains `GuardAction`, `GuardPolicy`, `GuardCategory`, `GuardMs`.

### B2. Package `internal/guard` (M)
**Mostly done (branch `guardrails-b`):** policies in server.yaml with
attachments in the store (`ctl guard attach`), narrowed by the limits' rule;
`guard/detect` (validated PII, secrets, injection through normalisation and
decoding); redaction with consistent placeholders before prefix routing;
tool allow/deny/strip, call count and argument schemas; capabilities and
structured output (opt-in); default `max_tokens`; pinned system prompt and
dropping the client's; `no_store` (threads, notes, prefix tree, external
models); shadow mode. **Left:** a `guard_events` table — the usage row's
`guard_*` columns carry action/policy/category/time, not rule and stage;
language identification; the re-identification map that restores
redacted values in answers.

- `Policy`, `Effective` resolution over the tenancy tree with the same
  narrowing rule as limits (`internal/store/sqlite.go:383-416` `tighten`):
  deny sets union, allow sets intersect, `mode` takes the strictest of
  `enforce > shadow > off`, `fail` takes `closed > open`, `no_store` at any
  level wins, thresholds take the minimum.
- Store: `guard_policies(scope, scope_id, policy json)` and
  `guard_events(request_id, scopes, policy, rule, category, action, stage,
  score, judge_ms, shadow)` — ids, categories, offsets and lengths only,
  never text. Migrations for sqlite and postgres. Pruned with
  `usage_retention`.
- `guard/detect`: regex/keyword sets; PII with validators — email, phone,
  IBAN (mod-97), card (Luhn), TCKN (checksum), IP, secrets (`sk-`, AWS,
  JWT, PEM); normaliser (NFKC, strip zero-width/bidi, homoglyph fold,
  base64/hex/URL decode to depth 2); injection heuristics (role-switch
  phrases, fake chat-template tokens, long base64, invisible Unicode) —
  default action `flag`; language id.
- Redaction rewrites `messages[].content` in the upstream body with
  placeholders (`[EMAIL_1]`) via the same `map[string]json.RawMessage`
  round-trip `withModel`/`spliceThread` use; an optional re-identification
  map lives only in request memory and restores originals in the output.
  Redact before prefix routing so `PromptPrefix` hashes the redacted text.
- `guard/toolpolicy`: tool name allow/deny (strip or 400), JSON-Schema
  validation of assembled arguments against the declared `parameters`,
  argument regex rules, `max_calls_per_turn`. Enforce
  `Capabilities.Tools`/`Vision` here: a `tools` array or image parts to a
  model without the capability → 400.
- `guard/structured`: validate `response_format.json_schema` on the final
  text → `content_filter`, or flag in shadow.
- Output constraints: inject a default `max_tokens` when absent (from the
  policy or `MaxOutput`), language check, structured check.
- System-prompt pinning per project: prepend before `checkContextFits` and
  before `spliceThread`; optionally drop client `system`/`developer`
  roles; exclude the pin from stored thread turns; apply to Responses
  `instructions` in `toChat`.
- `no_store`: `loadThread` and `PutNote` refuse; `captureBytes()` returns 0;
  skip the prefix-tree observe (`router.go:840`); disable External
  fallbacks for the scope (filter in `dispatch` and `budgetFallback`).

### B3. Judge model on the cluster (M)
**Done (branch `guardrails-b`):** `role: guard` + `guard.format`; hidden and
refused to clients and fallbacks; critical by default, never on demand;
`Router.Judge` past admission/accounting/guardrails; llamaguard3,
shieldgemma and json parsers; 5-minute cache; latency budget then
fail open/closed; streamed answers read every 600 characters, or held with
buffer_all. Simpler than written: the input judge runs after admission, not
alongside it, and shadow mode judges synchronously within the budget rather
than after the response. Not enforced: two replicas on different nodes —
documented, left to `scaling.min`.

- `catalog.Model.Role: "guard"` plus `Guard{Format: llamaguard3 |
  shieldgemma | json, Categories}`. Guard models are hidden from
  `/v1/models`, the chat picker and allowlists; placed like any model,
  class `critical`, `scaling.min ≥ 1`, at least 2 replicas on different
  nodes.
- `guard/judge`: calls the guard model through the router's own `dispatch`
  with an internal principal and its own admission band (never the
  tenant's queue); parsers per format (Llama Guard `safe`/`unsafe\nS1,S10`;
  ShieldGemma yes/no probability → per-tenant thresholds; generic JSON
  verdict); verdict cache keyed by hash of normalised new turns (5 min);
  `latency_budget` (default 250 ms) then `fail` open/closed; input judge
  runs in parallel with admission queueing.
- Streaming strategy: default sentence-buffered scanning (regex/PII per
  sentence or ~200 chars before release; judge every 400–800 chars on the
  cumulative tail; final judge at end); `buffer_all` for `fail: closed`
  scopes. Document that released text cannot be retracted.
- Shadow mode runs everything asynchronously after the response
  (`context.WithoutCancel`, like `saveThread`) and only records.

### B4. Config, metrics, UI (S)
**Done (branch `guardrails-b`):** library in server.yaml, attachments via
`ctl guard`, the console's **Guardrails…** with "applies from" marks, and
the portal's `/admin/guard`; the metrics and alerts below (judge latency
labelled by model only at the judge; the check latency by stage); `ctl
request` shows the verdict; the chat page says "stopped by policy"; the
profile lists effective policies; docs/guardrails.md and data.md written;
the access log carries action, stage and policy only. Attaching from the
portal is not done — the page shows; the console and ctl change.

- Gateway-level policy library in `server.yaml` under `guard:` (see the
  YAML in the review, reproduced in `docs/guardrails.md` to be written);
  attachments to scopes in the store via `llmgw keys guard`, the console
  ("Guardrails…" beside "Limits…" in `internal/cli/tui_people.go`, with
  "inherited from team:X" badges) and the portal.
- Metrics in `router/metrics.go` style:
  `llmgw_guard_checks_total{stage,policy,action,category,mode}`,
  `llmgw_guard_latency_seconds{stage,judge_model}`,
  `llmgw_guard_judge_errors_total{reason}`, `llmgw_guard_fail_open_total`,
  `llmgw_guard_stream_stops_total`, `llmgw_guard_redactions_total{kind}`,
  `llmgw_tenant_guard_blocks_total{tenant}`. Alerts: fail-open > 1% for 5m,
  judge p95 over budget, judge has no ready replicas, per-tenant block rate
  > 3× its 7-day baseline, any `secret` detected in output.
- Portal `/admin/guard`: violations by tenant/category/policy over time,
  shadow vs enforce, judge health, recent events by request id; `ctl
  request <id>` shows the verdict; chat page renders `content_filter` as
  "stopped by policy X"; `/me` lists effective policies.
- Update `docs/data.md`: the judge sees all text but in-cluster; External
  fallbacks send prompts outside the cluster (the current "nothing leaves
  the cluster" is wrong); `no_store` semantics.
- Never log: prompt/completion text, PII values, the re-identification
  map, judge inputs or rationale that quotes input. Check `accessLog`'s
  `err` field and span errors.

---

## C. Features, ranked by value × feasibility

1. **Done (branch `features-c`):** `audit_events`, written by the admin
   API middleware and the /admin pages (with each change's result), `ctl
   audit`, the console's Audit view; staff roles viewer / operator /
   billing / admin. Left: before/after values (the request's fields are
   recorded, not the state they replaced); a tenant-admin role that changes
   its own organization (managers still only view it); hiding the forms a
   viewer cannot submit. **Audit log and roles (M).** Today "audit" is slog lines
   (`internal/web/admin_users.go:403-480`) and roles are the `IsAdmin` flag
   plus an org "manager" who can view. Add `audit_events` (who, what,
   before/after, source) written by the `adminAction`/`tenancyAction`
   wrappers and by every admin-API mutation; roles viewer / operator /
   tenant-admin / billing; `ctl audit`. Depends on A5 for a real actor.
2. **Done (branch `features-c`):** OIDC sign-in (`auth.oidc`, PKCE,
   verified email, roles and memberships from groups, `no_passwords`) and
   SCIM 2.0 (`/scim/v2` Users and Groups). Left: the sign-in cookie is
   signed with a per-process key, so with two control planes behind a
   balancer the callback must reach the one that started it (sticky
   sessions) until the key comes from the store. **OIDC/SSO login (M), then SCIM (M).** `internal/account/service.go`,
   `internal/web/account.go`; map IdP groups to org/team/admin.
3. **Done (branch `features-c`):** key lifetime limits (gateway, org, team,
   project), `keys rotate --overlap`, kinds and expiry in `keys list`.
   **Service accounts and key lifecycle (S).** Per-org maximum key TTL,
   `keys rotate` with overlap, last-used shown, project-owned service keys
   listed apart. `internal/cli/keys.go`, `internal/store/keys.go`,
   `internal/auth/auth.go`.
4. **Done (branch `features-c`):** CSV export from the rollups (API, ctl,
   console). Left: Parquet, and a scheduled push to S3/SFTP. **Usage export (S).** CSV/Parquet by org/team/project/model/period from
   the rollups (`internal/store/usageseries.go`), `GET
   /api/v1/usage/export`, optional scheduled push to S3/SFTP.
5. **Done (branch `features-c`):** `notify.webhooks` (model parked, node
   lost, budget spent, update available; signed JSON or Slack; retries,
   hourly de-duplication), `ctl notify test`, Alertmanager receiver
   templates. Left: `server init` and the console do not yet ask for a
   receiver — the templates are filled in by hand; each control plane sends
   budget events on its own, so two behind a balancer may both send one.
   **Alerting out of the box (S).** `deploy/monitoring/alertmanager.yml`
   ships `receiver: none`; add Slack/email/webhook/PagerDuty receiver
   templates driven by `server init` or the console; llmgw-native webhooks
   for model parked, budget spent, update available (from the reconciler
   and `modelserve`).
6. **Done (branch `features-c2`):** `metadata`, `x-llmgw-tags` and `user`
   on the usage record (migration 0023), `ctl request` shows them,
   `router.tag_labels` metrics with 100 values per tag. Left: export and
   `ctl usage` by tag or end user (the hourly and daily sums do not carry
   tags; it would read the request records, within their retention).
   **Request metadata and tags (S).** Accept `metadata`/`user` from the
   body or `x-llmgw-tags`; store on the usage row; tenant metric labels
   behind an allowlist. `router/handlers.go` parse, `accounting/recorder.go`.
7. **Done (branch `features-c2`):** `ctl model canary/promote/canaries`,
   session-sticky split with `x-llmgw-variant` pins, rollback on error rate
   or TTFT, `canary_rolled_back` notification, console menu. Left: A/B
   between two different models (the split is between a model and its
   newer weights); the comparison is per control plane and lives in memory,
   so a restart begins it again; a key-level pin.
   **Canary and weighted A/B between model revisions (M).**
   `control/modelserve/updates.go` `replace` swaps every replica; add
   weighted routing across two revisions/aliases (scheduler filter, alias
   resolution in `router/router.go:569`), a header/key pin, automatic
   rollback on error-rate or TTFT regression from per-replica metrics.
8. **Done (branch `features-c2`):** `/v1/files` and `/v1/batches`
   (OpenAI's shape), worker under a store lease, lines kept as served,
   lowest band, usage tagged `batch=<id>`, output and error files with
   retention, `no_store` refused. Left: the Responses API (`/v1/responses`)
   as a batch endpoint; the console and `ctl` do not list batches (an
   operator sees them through usage tags); a file is one row in the store,
   so a store backup carries batch files with it.
   **Batch API and Files (M–L).** `/v1/files`, `/v1/batches`, JSONL
   storage, a worker feeding the bronze band, accounting. New
   `internal/router/batch`.
9. **Done (branch `features-c2`):** `keys requests` (retention, sample),
   kept after redaction, `ctl request --body/--replay`, console, pruning.
   Left: a streamed answer is kept as its text, not its chunks; replay is
   one request at a time (a set of them against a model is item 21's eval).
   **Request/response logging and replay (M).** Opt-in per project with
   retention like threads; sampled or full; redaction hooks from B;
   `ctl request <id> --body`; replay against another model.
10. **Done (branch `features-c2`):** Responses `text.format`,
    `input_image`, `previous_response_id` (thread in the id); rerank and
    score passthrough; transcription and translation as multipart;
    `/v1/moderations` by the judge. Left: audio is charged no tokens (an
    engine's duration usage is not read); `input_image` by `file_id`;
    moderation scores are 0 or 1, as a judge says yes or no.
    **OpenAI compatibility gaps (S–M each).** Responses: `text.format` →
    `response_format`, `input_image`, `previous_response_id` via threads
    (`router/responses.go`). Rerank/score passthrough for vLLM,
    `/v1/audio/transcriptions` for a whisper engine, `/v1/moderations`
    mapped to the guard model.
11. **Done (branch `features-c2`):** `/v1/messages` and `count_tokens`,
    streamed and not, tools, images, `x-api-key`, Anthropic error shape.
    Left: extended thinking (a reasoning model's thoughts are not mapped to
    thinking blocks); prompt caching; the account page's "Connect your
    editor" does not offer Claude Code yet.
    **Anthropic `/v1/messages` (M).** Translate like `responses.go`.
12. **Done (branch `features-c2`):** `server restore` (dry run, version
    check, decryption, modes and owners, SQLite and PostgreSQL),
    `backup_copy_to` (directory or ssh), the drill in docs/install.md.
    Left: restoring a replicated store's members (restore into the primary;
    Patroni replicates it); the console's Back up now does not offer
    restore; an ssh copy holds the archive in memory while it sends it.
    **`server restore` (S).** Restore is manual shell today
    (`docs/install.md:1187`); add the command with version check, dry run,
    off-host copy target; a documented DR drill.
13. **Done (branch `features-c2`):** TPOT, inter-token and tokens/s
    histograms by model, pool and tenant; `slo` pressure in admission's
    saturation; `llmgw_model_slo_seconds` and `llmgw-slo` alerts. Left: the
    p99 is per control plane, from its own requests; an SLO per band or per
    tenant; placement does not yet add replicas for an SLO (item 14).
    **SLO-based admission and token-level metrics (S–M).** TPOT and
    inter-token histograms, output tokens/s per model and tenant in
    `router/metrics.go`; use `catalog.SLO` in `admission/saturation.go`.
14. **Done (branch `features-c2`):** `scaling.schedule` (days, from, to,
    min, max; past midnight), `scaling.prewarm`, `control.timezone`;
    queue-depth scaling was there already. Left: `ctl scale` sets min and
    max only (a schedule is set in the definition); scaling for an SLO
    rather than for queue depth.
    **Scheduled and warm-pool autoscaling (S–M).** Time-of-day min/max,
    pre-warm, queue-depth scaling in `catalog.Scaling`,
    `control/reconciler/demand.go`, `headroom.go`.
15. **Done (branch `features-c2`):** `env:` and `vault:` references
    wherever a secret file is named (store address, api_key_file, OIDC,
    SCIM, webhooks), Vault KV with token or AppRole, `secrets.master_key`
    sealing cluster_secrets (Vault transit or a local key). Left: the
    certificate authority's key is still a file in /etc/llmgw (signing in
    Vault's PKI engine instead is its own change); rotating the master key;
    `backup_copy_key` and `admin_token_file` are files only.
    **Secrets management (S–M).** Vault/KMS for `api_key_file`, the DSN, the
    CA key, `cluster_secrets` (`store/secrets.go`, `store/seal.go`);
    envelope-encrypt them in backups (with A8 backups).
16. **Done (branch `features-c2`):** `users forget` (pseudonymised usage
    and sums, kept requests, files, batches, limits and audit names), `keys
    residency --in-cluster` (asked-for, fallback and canary). Left: an
    export of everything held about a person (article 15); residency by
    region between several clusters; backups keep what was forgotten until
    they age out.
    **GDPR and residency (S).** "Forget user" anonymises `user_id` in usage
    and rollups and erases threads/memory (`store/hierarchy.go:241` leaves
    it today); per-project "never route to an External provider".
17. **Done (branch `features-c2`):** exact-match answer cache per project (`keys
    cache --ttl/--any/--off`, console **Cache answers…**, `projects`
    columns, 30-second setting cache), reproducible answers only unless
    `--any`, LRU in memory bounded by `router.cache.max_bytes` and
    `max_entry_bytes`, streamed answers kept and replayed as chunks,
    `x-llmgw-cache` and `Cache-Control: no-cache`, guardrails before a hit
    and the output policy on it, `no_store` and threads bypass, a hit
    recorded as `cache_hit` with no tokens and no cost (migration 0027),
    metrics, docs. Left: semantic (embedding-similarity) caching through the
    embeddings models; a cache shared across control planes (each process
    keeps its own, deliberately: a shared one would write every answer to
    the store); a hit still waits its turn in admission, because the
    guardrails judge the request after admission and the key is the request
    they let through; logprobs in a streamed answer leave it unkept; `users
    forget` does not reach into the cache (answers age out by the TTL); no
    listing or flushing of what is cached (`--off` stops it being served at
    once, the memory is freed by TTL and LRU).
    **Response/semantic cache (M).** Exact-match first, per project,
    opt-in, TTL; semantic via the embeddings models later. Before
    `dispatch` in `router/handlers.go`.
18. **Done (branch `features-c2`):** offline bundles — `llmgw bundle create`
    (releases with signed checksums, every llama.cpp build per arch/variant
    with GitHub's checksums, starter/judge and named Hugging Face model
    files pinned and checked; directory or tar; manifest sha256), `bundle
    import` (checked, all-or-nothing, into the mirror under `_llmgw/`),
    `bundle trust`, `bundle show --verify`;
    `offline.mirror`/`offline.checksums` in server.yaml, with nodes
    installing llama.cpp from the mirror against checksums the control plane
    sends over the agent stream (proto `EngineWant.offline_*`), releases
    defaulting to the mirror for `node add`, `ctl upgrade` and agent
    upgrades, update checks off; `server init --bundle` and the setup form's
    bundle fields; docs/data.md's offline claim corrected; install.md
    "Offline sites". A Helm chart for the control plane
    (`deploy/helm/llmgw`, one replica, SQLite PVC or external PostgreSQL,
    Secrets, Services, Ingress, ServiceMonitor) with
    `deploy/container/Dockerfile` and docs/kubernetes.md; GPU nodes stay
    outside Kubernetes by design. Left: vLLM/SGLang offline (a wheelhouse of
    several GB per CUDA build and a uv `--find-links` install path); the
    NVML release binary and uv in bundles; the setup wizard's preflight
    still fails a machine without internet; `join`'s printed `curl` from the
    mirror checks nothing; models in a bundle are single files (no whole-
    repository safetensors sets yet); the chart runs one replica (a standby
    is never ready, so HA in Kubernetes needs a different readiness design),
    has not been installed on a real cluster, and ships no agent DaemonSet;
    no published container image.
    **Kubernetes/Helm and an air-gapped bundle (M).** `deploy/`,
    `agent/provision`; `docs/data.md` claims offline operation but install
    assumes the internet.
19. **Done (branch `features-c2`):** `adminapi.Routes` and
    `internal/docgen/openapi` generating `docs/admin-api.openapi.json` (3.1,
    bearer security, every route, query parameters, schemas from the Go
    types with their comments), served embedded at `GET
    /api/v1/openapi.json`, kept true by a route-table test and a staleness
    test; validated with openapi-spec-validator and openapi-generator-cli.
    `GET /api/v1/models/{name}`. `docs/admin-api.md`. `terraform-provider-
    llmgw/` (terraform-plugin-framework): `llmgw_model`, `llmgw_quota`,
    `llmgw_guard_attachment`, data sources `llmgw_models` and `llmgw_nodes`;
    client tests against a fake admin API and the OpenAPI document;
    acceptance tests with Terraform and OpenTofu against the fake; one
    apply, empty plan, in-place change and destroy against a real `llmgw
    server` run by hand. Left: organizations, teams, projects, people and
    keys are not in the admin API (they are `llmgw org`/`users`/`keys`
    against the store), so not in the provider; a model's definition beyond
    scaling, class, owner and adapters is changed by replacing it (the API
    has no edit, and `resources` is not used yet); `canary`, `slo`,
    `scaling.schedule` and `prewarm`, `guard` role and `origin` are not in
    `llmgw_model`; models cannot be imported; the provider is not published
    to a registry and has no release build; the acceptance tests are not run
    in CI (they need Terraform); the document's schemas mark nothing
    required and give no enums; `catalog.SLO`'s fields have no JSON tags, so
    they are `TTFTp99` and `TPOTp99` on the wire.
    **OpenAPI for the admin API, Terraform provider (M).** Generate from
    `adminapi/api.go` routes via `docgen`.
20. **Done (branch `features-c2`):** spans for the reconcile pass (begun only
    when a pass does something), placement with its answer or rejections,
    scale-up/down, drain, stop, eviction, preemption, reaping, restarts and
    lost nodes; a `replica start` span from placement to serving whose
    context travels to the agent in the start command
    (`ControlMessage.trace`), where weights fetch, engine launch, rendezvous
    and readiness wait continue the trace; stops, drains, adoption and
    engine installs; modelserve jobs as a trace with a span per step linked
    to the replica's start; an agent `tracing` block; docs in install.md's
    "Tracing". Left: the router's span for a request does not link to the
    replica it was sent to's start; heartbeats, fabric scans, logs and agent
    upgrades are not traced; the supervisor has no restart span because it
    never restarts an engine itself (a restart is the reconciler's `restart
    replica` and reaches a node as a stop and a start); canary and promote
    jobs, which landed on `features-c2` after this was branched, are traced
    only through the shared step recording (as `<kind> model`), unchecked by
    a test.
    **Tracing beyond the router (S).** Spans for reconciler, placement,
    agent start (`obs/trace.go`, `control/reconciler`, `agent/supervisor`).
21. **Done (branch `features-c2`):** `internal/bench/eval` (suites in YAML or
    JSON lines; exact, contains, regex, number, a JSON Schema subset, and a
    judge model with a rubric; weights and a threshold), run on the control
    plane through `Router.Replay`; `llmgw ctl eval MODEL --suite FILE|NAME`,
    `--history`, `--run`, `--suites`; runs and their cases kept (migration
    0028, `eval_runs` and `eval_cases`); `model update`/`promote --eval
    SUITE --min-score N`, a model's standing gate (`ctl model gate`, `eval:`
    in the catalog) — a promote asks the canary first, an update tries the
    new weights as a 0% canary and promotes or removes it; the console's
    model menu; `arithmetic` and `json-format` in `deploy/eval/`, carried in
    the binary. Left: the suite of a model's standing gate is read by name
    on the control plane at the time of the update, so two control planes
    need the same `control.eval_dir`; a gated update needs room for one more
    replica while it runs (there is no in-place gate, and a declared model
    in server.yaml cannot be updated at all); no comparison against the old
    weights' score (the gate is an absolute threshold, not "no worse than
    before"); a run is not stopped once started (Esc in the console only
    stops watching it), and the in-memory progress of a run lives on the
    control plane that started it; runs are never pruned; the JSON Schema
    support is a subset (no `$ref`, `oneOf`, `not`, `uniqueItems`); judge
    cases go through the same concurrency as the model's, and a judge's
    answers are not kept, only its score and reason; a gateway without a
    control plane cannot run suites (there is no mode that asks through the
    client API with a key).
    **Eval harness (M).** `llmgw eval` extending `internal/bench` with
    quality suites gating `model update`.
22. **Done (branch `features-c2`):** MIG instances discovered on both paths
    (nvidia-smi `-L` + `-q -x`; NVML through an interface with a faithful
    fake) and reported instead of their card, each with its MIG UUID, own
    memory and class `<card class>-mig-<profile>`; device numbers 1000 + 100
    × card + instance, shown as `card/migN`; placement on MIG classes;
    tp/pp/gpus/nodes above 1 on a MIG class refused in catalog validation
    and again in placement; `CUDA_VISIBLE_DEVICES=MIG-<uuid>` for vLLM and
    llama.cpp (no Vulkan device variable on MIG); per-instance memory and
    processes in heartbeats and the GPU check, card sensors under each
    instance, utilisation absent rather than zero; doctor MIG findings (`mig
    -lgi`/`-lci` parsed); `ctl nodes`, console, `ctl status`, `ctl model
    plan`/`serve` by class; docs/install.md MIG section. Written and tested
    against fakes only — untested on real MIG hardware. Left: run it on an
    A100/H100 in MIG mode (check `-L`/`-q -x`/`mig -lgi`/`-lci` and NVML MIG
    device names against the parsers, and that `CUDA_VISIBLE_DEVICES=MIG-…`
    reaches vLLM's workers through Ray); instances changed while the agent
    runs are only seen when it restarts (no rescan); llmgw does not create
    or reshape instances (no mig-parted integration); quotas count an
    instance as one GPU, the same as a whole card; SGLang on MIG not
    checked; device numbers assume fewer than 100 instances per card and
    that instance numbers stay put while shards run on them.
    **MIG (M–L).** Treat a MIG slice as a GPU class in
    `agent/discovery/gpu_nvml.go` and `control/placement`.
23. **Done (branch `features-c2`):** engine `trtllm`
    (`internal/agent/engines/trtllm.go`): `trtllm-serve` through a launcher
    of llmgw's that demands the replica's key (trtllm-serve has none) and
    writes the options file that turns on `/prometheus/metrics` and the
    iteration statistics; the scrape contract for it
    (`scrape.TRTLLMMapping`, with a per-engine metrics path); found, not
    installed — `engines.trtllm.python` in `agent.yaml`, probed from the
    package's metadata; catalog validation (GPU class, one node, no GGUF
    source, the flags a profile may not give); `ctl engine-check trtllm`,
    which asks `trtllm-serve serve --help` through its module. Ollama/GGUF
    import: `mirror import-ollama` (manifest parsing, blob digest check,
    projector beside the model, all or nothing, a model definition printed),
    `mirror import` for any local file, and the console's **I** on the
    Mirror view. Left: TensorRT-LLM has not run on real hardware — the first
    node with one should run `engine-check` and serve a model before anyone
    relies on it; no installer for it; no multi-node TensorRT-LLM replica
    (it needs an MPI or Slurm launch from one node); Triton Inference Server
    as an engine is not done; trtllm's queue gauges are set per iteration,
    so an idle engine shows its last batch until the next request;
    `--num_serve_frontends` is refused because extra front ends would not
    demand the key. The Ollama import runs on the mirror machine only — the
    mirror's API takes no files, so an Ollama elsewhere means copying the
    blob or running the import there; Ollama models kept as separate tensors
    or as several weight layers, and its LoRA adapters, are refused or left
    with a sentence; files an import run with `sudo` writes are root's, as
    `mirror add`'s are.
    **More engines (M each).** TensorRT-LLM/Triton adapter; Ollama/GGUF
    import. `agent/engines/engine.go` interface and the metric contract in
    `router/scrape`.
24. **Done (branch `features-c2`), all six parts.** Lower: precise KV-index routing (plan phase 6), LoRA affinity for
    vLLM, multi-cluster federation via External providers, facility-aware
    placement (`docs/facility-aware-placement.md`), portal i18n, SDK/docs
    portal.
    What was built, and what is left:
    - 24 (portal i18n): built — English/Turkish catalogue with fallback, per-
      language templates, cookie/Accept-Language/`landing.language` choice
      and a footer switcher, Turkish numbers and dates, every template and
      handler message translated, tests that no template text stays English;
      left — errors built with fmt.Errorf in other packages (store, catalog,
      mirror, adminapi) still English on Turkish pages, percent sign
      placement is mixed (%50 on limits, 93% on cluster pages), RFC3339
      footer timestamps and plain-text http.Error bodies untranslated, the
      /docs documents themselves English, a per-person language column
      (cookie only today).
    - 24 (SDK/docs portal): built — /docs renders the embedded manual (goldmark,
      GFM, heading anchors, links rewritten, no script; public,
      `landing.docs_sign_in` to require a session), and docs/clients.md
      shows the official OpenAI/Anthropic SDKs against llmgw. Left: the
      console's generated "Commands" page is not in the portal (it is built
      from the cobra tree, which internal/web cannot see); no search across
      documents.
    - [~] 24. Precise KV index (tier 2) — built: vLLM KV events via the agent
      (loopback ZMQ, pure-Go zmq4, msgpack in both of vLLM's encodings)
      forwarded as net block-hash changes over the agent stream; router
      index, tokenize via the replica's `/tokenize`, sha256_cbor block-hash
      chain checked against cbor2/hashlib golden values; scorer by leading-
      block share with tier 1 fallback; opt-in `kv_events` per profile;
      metrics; docs. Left: validation against a real vLLM (event shape, hash
      agreement, `/tokenize` matching the served prompt, cost of
      tokenizing); vLLM's replay endpoint instead of resetting on a gap;
      data-parallel ranks (one publisher each, ports offset); hybrid models'
      cache groups and CPU-offloaded blocks; LoRA/multimodal/cache_salt
      requests (extra keys); a router-only deployment (no agents)
      subscribing directly; standby control planes have no index until they
      lead; a per-replica fallback when only some candidates have an index;
      caching tokenizations of repeated prompts.
    - **24 LoRA affinity (branch `features-c2`):** built `max_loaded_adapters`
      (catalog), vLLM adapter support at start and on demand
      (`engines/lora.go`, `vllm.go`), the agent linking fetched adapters as
      `llmgw-lora/<name>` in the engines' working directory, the router's
      adapter tracker (`router/lora.go`: loads/unloads with the engine key,
      LRU eviction never of an adapter in use, one load per adapter per
      replica, `/v1/models` sync, 404 reload) and the `adapter-affinity`
      scheduler filter in the chat, batch and long-context profiles, with
      metrics and docs; tested against a fake of vLLM's HTTP interface.
      **Left:** never run against a real vLLM — in particular that vLLM
      resolves the relative `llmgw-lora/<name>` path against its working
      directory on every node (multi-node too) and that unloading an adapter
      mid-request is harmless; the agent still fetches every adapter at
      replica start, so a new adapter needs a restart (fetching on first
      load would need an agent command and a proto change); load timeout and
      resync interval are constants, not config; SGLang's runtime LoRA API
      is not driven; no per-replica "adapters held" view in the console or
      admin API; the debug profile has no affinity.
    - 24. Multi-cluster federation — built: peers via `ctl federation` kept in
      the store, their models as PEER/MODEL synced from /v1/models
      (withdrawn-while-referenced), /ready health with latency and routing
      withdrawal, `ctl model fallbacks` for overflow, hop header loop
      refusal, residency/no_store hold, optional attribution tags, console
      view, metrics, docs. Left: server.yaml-declared models cannot name a
      peer model as a fallback (config catalog is validated before peers are
      read); per-peer CA / mTLS for peers behind a private CA; transitive
      federation (A→B→C models) deliberately not offered; latency is /ready
      RTT only (no per-request peer TTFT used in routing or weighting); the
      peer's adapters are not imported (listed as aliases); no console form
      to add/remove peers (CLI only); key file must exist on every control
      plane.
    - 24 (facility) — built: location labels (`ctl label`, agent.yaml),
      spreading across racks/PDUs, `keep_within`, held domains with drain
      (`ctl facility hold/release`, kept in the store), power caps from
      `facility.budgets` against declared/derived/measured draw, shown in
      explain-placement, `ctl nodes`, `ctl facility`, the console and the
      API. Left: temperature conditions with hysteresis and their
      enforcement (steps 3–4), a Prometheus/DCIM source, power headroom as a
      rack-levelling score, spreading multi-node replicas across racks,
      `llmgw_domain_*` metrics and alerts (near cap, over cap, stale data),
      shedding and tokens per watt (step 5), and a drained node's state
      surviving a restart.

---

## E. Every setting takes effect when it is changed (2026-10-05)

Asked for after a site visit: an upgrade left server.yaml without guardrails,
a node on another subnet stayed lost until firewall_networks was edited by
hand, and nothing in server.yaml, agent.yaml or a generated unit applies until
somebody restarts the right service on every machine. The rule from here on:
**a file holds only what one machine needs to start; everything else is the
cluster's, kept in the store, changed through ctl, the console or the web,
audited, and in force on every control plane within seconds.**

### E1. Cluster settings in the store (L) — done 2026-10-05
Live on the PoC (140, 150): imported as version 1; a guard change applied on
both without a restart; a router change restarted the standby at once and the
leader 30s later with /health answering throughout; rollback did the same.
- Table `cluster_settings` (versioned YAML document, who and why). The
  cluster-wide part of server.yaml lives there; server.yaml keeps the
  per-machine keys only (listeners, TLS paths, store DSN, secrets, tracing,
  monitoring, mirror.api/tls, backup_dir/key, endpoints).
- First start against a store with no document imports the file's cluster
  keys (and the baseline guard policy when there is none); after that those
  keys in a file are ignored, with a warning.
- Each control plane polls the document's version and its own server.yaml's
  hash. A change is validated, then applied: `guard` live, the rest by a
  graceful restart of the process (standbys first, the leader last), which
  systemd brings back. No "restart now?" anywhere.
- `llmgw ctl config show|history|get|set|unset|edit|rollback`; the console's
  settings screen edits the store for cluster keys; admin API
  `GET/PUT /api/v1/settings`, `/history`, `/rollback`.
- A standby that becomes leader re-reads the catalog, engine choices and
  facility state from the store (it kept what it read at start).
- Retires `appendGuardIfMissing` (server init): the import does it.

### E2. Generated files from settings (L) — firewall on/off done 2026-10-06
`firewall.enabled` in the cluster's settings reaches nodes over the agent
stream and control planes through llmgw-firewall-sync.path. Left: the mirror
machine and store members, the allowed set from membership, the other files.

Every machine runs the agent (root) with roles — node, control plane, mirror,
store member — and the agent renders and applies what was written once by
init: the llmgw nft table, the mirror's unit flags, keepalived, etcd and
Patroni configuration. The firewall's allowed set is the cluster's membership
(every node, control plane, mirror and store member) plus
`firewall.extra` from the store; 7443 stays open (mTLS). A node added or
removed changes every machine's rules. Retires `firewall.Admit` and
`firewall_networks` in server.yaml (imported into the store).

### E3. agent.yaml down to bootstrap (M)
agent.yaml keeps the control planes' addresses, its certificate and its id.
Labels, fabrics, engines, heartbeat and supervisor settings move to the
store per node and reach the agent over its stream; a change to the file is
noticed and the agent restarts itself.

### E4. Web settings page (S)
The same document, edited on /admin/settings, for whoever has no terminal.

---

## F. Company documents: knowledge bases (RAG), then adapters (2026-10-08)

A company wants a model to answer from its own documents. Retrieval is the
tool for *knowledge* (sources shown, re-indexed in minutes, access per
tenant); LoRA training is for *behaviour* (vocabulary, format) and has no
access control — whatever goes into weights reaches everyone who may use the
adapter. RAG first. Decisions taken: access per collection (no per-document
ACL yet); a personal key or session names a project or team and gets that
chain only, never the union of its person's memberships; PostgreSQL store
only, off on SQLite with a sentence saying why.

### F1. Store: collections, grants, chunks (M) — done 2026-10-08 (branch `rag`)
Built (`internal/store/knowledge.go`, migration 0035 PostgreSQL only):
- `kb_collections` (owner scope/id, org, embed_model, dims), `kb_grants`
  (read|write, any scope, never across organizations; a key needs a
  project; a person needs a membership in the organization),
  `kb_documents` (digest: the same again is no change, a new one replaces
  in one transaction).
- `kb_chunks` is created when the store opens, not in a migration (a
  migration needing an extension would stop the control plane; one that
  skipped it would never run again): partitioned by organization (hashed
  partition name), `tsv` generated (`simple`), GIN on the parent, one
  partial HNSW index per collection on its organization's partition
  (`embedding::vector(dims)`, `WHERE collection = '<id>'`). EXPLAIN on
  20k rows: partition pruned, that index used, policy applied.
- RLS FORCEd on `kb_chunks` against `llmgw.kb_org`/`llmgw.kb_collections`
  (set with set_config per transaction), one role — the table owner — not
  a separate one. Chunks are deleted by the foreign-key cascade from their
  document, which RLS does not block.
- `CollectionsFor(Effective, write)`: owner or grant anywhere on the chain;
  no organization (personal key without a project) reads nothing.
- `SearchChunks`: per collection nearest-neighbour (literal id and cast so
  the partial index is used) plus a full-text arm with any word
  (`a | b`, lowercased by the database, not Go), fused by reciprocal rank.
- Deleting a project/team/org that owns collections is refused like one
  with members (changed from "deleted with them": every other delete in the
  tree refuses); DeleteUser/ForgetUser delete the person's collections;
  deleting an organization drops its partition; grants to a deleted scope go.
- `server init` installs `postgresql-N-pgvector` / `pgvector_N` and enables
  it; pgha installs it on every member at setup and enables it on the
  leader on every run. Neither stops on failure.
Left: a cluster set up before (the PoC) gets no package from a rerun of
pgha on a running cluster — install by hand or add an upgrade step;
`server restore`/`CopyTo` do not carry kb tables (SQLite → PostgreSQL only,
and SQLite has none); tests need `LLMGW_TEST_POSTGRES` pointing at a
pgvector database as a non-superuser (a superuser bypasses RLS).

### F2. Ingest on the user's machine (L) — done 2026-10-08 (branch `rag`)
Uploading whole files to the control plane was dropped: a PDF is 20–100×
its text, and the work belongs where the files are. Instead a small client,
`llmgw-kb`, run by the person on their own machine with their own key:
- **Its own binary** (`cmd/llmgw-kb`), for Windows, macOS and Linux on amd64
  and arm64. `llmgw` itself does not build on Windows (agent syscalls), and
  employees' laptops should not carry the server, agent and setup screens. A
  test keeps server packages out of its imports.
- `llmgw-kb sync <collection> <dir|file…> [--include/--exclude globs]
  [--dry-run] [--delete]`: scan → local catalog (path, size, mtime, sha256;
  cache under os.UserCacheDir, digests not recomputed for unchanged files)
  → the collection's manifest from the gateway (name → digest only) → diff
  → for new/changed files only: extract, chunk, embed through the gateway's
  `/v1/embeddings` with the person's key (its accounting, limits, guard and
  band apply), and send name, digest, chunk texts and vectors; `--delete`
  removes documents whose files are gone. `--dry-run` lists what would leave
  the machine and how much. A file that cannot be read (locked, unsupported,
  no text) is skipped with the reason; the run goes on.
- **Formats, as many as possible.** Built in, pure Go: PDF (text layer),
  DOCX/XLSX/PPTX, ODT/ODS/ODP, RTF, EPUB, HTML/XHTML, EML (text and HTML
  parts), Jupyter notebooks, subtitles (SRT/VTT), CSV/TSV, JSON/YAML/XML,
  Markdown/RST/AsciiDoc/Org, plain text and source code; ZIP archives
  recursed into. With a helper found on the machine: legacy DOC/XLS/PPT, RTF
  and anything else LibreOffice opens (`soffice --headless --convert-to`),
  images and scanned PDFs through Tesseract (`tur+eng`). Text in UTF-8,
  UTF-16 (BOM) or Windows-1254/1252 is converted to UTF-8.
- **Across systems:** document names relative to the synced root, `/`
  separated, Unicode NFC (macOS hands out NFD), a warning for two names that
  differ only in case; CRLF → LF; key from `LLMGW_API_KEY` or `llmgw-kb
  login` (os.UserConfigDir, 0600 where that means something); the system's
  CA store, `--ca` for another.
- **Gateway endpoints** (client listener, person's key):
  `GET /v1/kb/collections` (what this key may read/write),
  `GET /v1/kb/collections/{id}/manifest`, `PUT
  /v1/kb/collections/{id}/documents` (one document per request: name,
  digest, bytes, chunks with text and embedding; dims checked, chunk count
  and size bounded), `DELETE …/documents?name=`. Write needs a write grant
  or ownership on the key's chain (`CollectionsFor(…, write)`); otherwise
  404.
- **Administration** stays with operators: `llmgw ctl kb
  create|list|delete|grant|revoke|docs`, admin API `/api/v1/kb`, audit,
  console screen; `docs/knowledge.md`.
- Trade-off accepted: vectors come from the client, so a writer could store
  vectors that do not match their text — only in collections they may
  already write.
Built: `internal/kb/extract` (formats above; helpers found on PATH and in
each system's install places), `internal/kb/chunk` (≈1800 characters under
the heading path, a paragraph of overlap, sections kept apart once a chunk
is a third full), `internal/kb/kbsync` (ledger in SQLite at
os.UserConfigDir/llmgw-kb/ledger.db: documents keyed by gateway, collection
and name with file, size, mtime, digest, state, reason; runs), `cmd/llmgw-kb`
(login, collections, sync, status, history, formats), `internal/router/kb.go`
(the four gateway endpoints, `X-Llmgw-Project` for personal keys, audit),
`internal/control/adminapi/kb.go` + `llmgw ctl kb` + console Knowledge
screen, `docs/knowledge.md`; CI builds the client for all six targets and
releases attach it. Tested: every format from generated fixtures and a
Turkish PDF (OCR path with Tesseract here), sync against a fake gateway and
end to end against the router on pgvector.
Tried on the PoC 2026-10-08 (control planes 140 and 150 on a dev build of
this branch; their previous binary is /usr/local/bin/llmgw.pre-rag): Ubuntu's
pgvector 0.6.0 installed on 149/150 and enabled; bge-m3's weights were never
in the mirror (fetched from CompendiumLabs/bge-m3-gguf@1864f47f) and it is
held at 1 replica; tree Demo/ik/belgeler and Demo/satis/crm with a key each;
collection Demo/politikalar filled from a Mac through an SSH tunnel with ten
Turkish files in nine formats (9 sent in 5 s, the .doc skipped for want of
LibreOffice; a rerun sent nothing in 0 s; a changed file alone was sent).
Nine Turkish questions each found their page first once search scored by
similarity plus word coverage (rank fusion had got two wrong). The other
team's key saw no collection and got 404 by id; with a read grant it read
and could not write; revoked, nothing again. On the live store the llmgw
role (not superuser, no BYPASSRLS) sees 0 chunks with no scope set and 0
with the organization alone.
Left: the gateway's collection list shows the owner as team:ID, not its
path; no LibreOffice on the dev machine, so the legacy
Office path is untested beyond its error; the client is unsigned (Gatekeeper
and SmartScreen warn); Outlook `.msg` is not read; one document over ~60 MiB
of JSON (a few thousand pages) is refused rather than split.
PostgreSQL tests of store, router and adminapi share LLMGW_TEST_POSTGRES and
the store's TRUNCATE each test; run them with `go test -p 1` or one package
at a time, or a router test loses its rows mid-run.

### F3. Retrieval on the request path (M) — done 2026-10-08 (branch `rag`)
- `llmgw.knowledge: [collections]` in the body (stripped before the
  engine), or default collections per project/key.
- Allowed set from the caller's chain (§7.9.2), cached like limits; grants
  add up down the tree. Personal keys and sessions need `x-llmgw-project`
  (or the playground's picker) naming a scope the person is a member of.
- A collection not allowed answers 404, exactly as a missing one.
- Hybrid search with the filter inside the query (per collection, its own
  partial index; pgvector 0.6 is enough —
  scored by similarity plus the share of the question's words held (built
  in F1/F2 as SearchChunks), `/v1/rerank` when a reranker is in the catalog. Top
  chunks into a system message with document and position;
  `x-llmgw-sources` lists them.
- Leaks closed: answer-cache key includes the allowed set; per-organization
  `cache_salt` on RAG requests; injected passages under `no_store` and the
  guard's prompt-injection check; returned doc ids in the audit.
- Done: router tests that tenant B never receives tenant A's chunks
  (including via cache), that a personal key without a project is refused,
  metrics (retrieval latency, chunks returned), docs.

Built (`internal/router/rag.go`): `X-Llmgw-Knowledge` (header, as the
other llmgw extensions are, not a body field), `X-Llmgw-Knowledge-K`;
collections from the key's chain (kbEffective, shared with the kb
endpoints), 404 for one not readable; the question embedded through the
ordinary path on the caller's principal; SearchChunks; passages trimmed to
the model's context; added to the one leading system message (or a new
one); `x-llmgw-sources`; before guardInput; never cached; metrics. Default collections built (kb_defaults on org/team/project/key, narrowest
wins, 30 s cache, `ctl kb default`, console action; never refuse); the
portal chat's Documents picker (collection, default or none; sources listed
under the answer; a header the page sends itself is dropped). `cache_salt` per organization on requests given passages, sent to vLLM
replicas only (chosen replica's engine). Left: returned passages
are not audited per request (the header and the kept request record them).

### F4. Adapter training (L, after F1–F3)
A run-to-completion job placed by the control plane like a replica and
supervised by the agent like an engine, in a Python environment the agent
installs (TRL + PEFT, QLoRA). Data: Q/A pairs generated from a collection by
a cluster model through `/v1/batches`, reviewed by people. Output to the
mirror, into the model's `adapters`, through canary and a quality suite.
Runs when the scaling schedule frees a GPU; 8–14B dense bases on one card.
Never train on a collection some users of the adapter may not read.

---

## D. Facts worth knowing before starting

- Admin API: `127.0.0.1:9090`, no auth, `ctlClient` in `internal/cli/ctl.go`.
  Metrics on `:9091` with `/prometheus/targets` beside them.
- Store: `internal/store`, migrations under `internal/store/migrations/{sqlite,postgres}`;
  the PoC runs PostgreSQL under Patroni.
- The console (`sudo llmgw`) is `internal/cli/tui_console.go` with actions
  in `tui_actions.go`; every operator-visible change needs a row action or
  a form there, and a line in `docs/console.md`.
- Tests of the console use a fake admin API (`internal/cli/tui_actions_test.go`,
  `adminStub`); tests of serving use fakes in
  `internal/control/modelserve/modelserve_test.go`.
- `go generate ./...` regenerates `docs/reference.md` and
  `internal/docgen/reference_gen.go` whenever a config struct changes; the
  docgen test fails otherwise.