# Backlog What a review of the code on 2026-10-04 (commit c2238ab) found should be done next: security fixes first, then guardrails, then features. Written for whoever implements it — a person or a model — without the review in hand. Every item names the files it touches and what "done" is. Paths are relative to the repository root. Sizes: S about a week, M two to four weeks, L more. Conventions this codebase keeps, which every item here must keep too: - A change is refused before anything is written (validate, then persist). - Nothing on the serving path logs prompt text, keys or passwords. - Each item gets unit tests beside the code (`*_test.go`), and the TUI (`internal/cli/tui_*.go`), `docs/console.md` and `CHANGELOG.md` are updated when behaviour an operator sees changes. - `go test ./...`, `go vet ./...` and `govulncheck ./...` pass. No release is cut and nothing is deployed to the PoC cluster without being told to. --- ## A. Security — do first **Status (2026-10-04): done and merged.** A1–A9, A6b included — etcd and Patroni's API on TLS, the PoC moved in maintenance mode without a failover. Deliberately narrower than written: the admin API is a token on loopback rather than a unix socket (A5); thread ids are bounded and documented as project-wide rather than scoped per user (A8); Python engines resolve with an explicit PyTorch index and a PEP 751 lock rather than locks shipped per engine version (A8). All of A is live on the PoC. ### A1. Toolchain (S, hours) `govulncheck` finds 11 standard-library vulnerabilities reachable from this code at Go 1.26.3 (net/http, crypto/x509, os.Root, net/textproto). Raise the `go` directive in `go.mod` to 1.26.6 or later, rebuild, run `govulncheck ./...` until it reports none reachable. Add `govulncheck` to the Makefile's check target. ### A2. Release signing is effectively off (S) - `internal/release/trusted.pub` is empty, so `TrustedKey()` returns "" and `release.go:103` skips signature verification. Ship a real public key; make a missing or invalid signature fatal. Drop the `LLMGW_RELEASE_KEY` environment override, or honour it only in `-tags dev` builds. - `internal/onboard/onboard.go:365`: the node-side download checks only a same-origin `checksums.txt`. Verify the signature there too (simplest: have the node run `llmgw` itself to fetch and verify once it has any binary), and `shellQuote` the `Releases` URL, which is interpolated into single quotes unescaped. - `internal/agent/upgrade.go:44`: accept only `https://` release bases. - Done when: a tampered binary or checksums file is refused by self-upgrade, node onboarding and agent upgrade, with a test for each. ### A3. Agent port without TLS is anonymous and plaintext (S) `internal/cli/server.go:423` serves h2c on `:7443` when `agent_tls` is not configured; `agentsvc` then accepts any node id. Refuse to start control mode without TLS unless `agent_tls.insecure: true` is set explicitly; when insecure, bind `agent_listen` to 127.0.0.1 and log a warning at startup and in `llmgw doctor`. Keep `server init` writing TLS as it does. ### A4. Model profiles reach root on every node (M) Profile `args` and `env` from a model definition are appended to the engine command unchecked (`internal/agent/engines/llamacpp.go:170`, `vllm.go:151`, `sglang.go:98`); `internal/agent/engines/unsafeenv.go` is a denylist that misses `NCCL_NET_PLUGIN`, `NCCL_TUNER_PLUGIN`, `NCCL_PROFILER_PLUGIN`, `CUDA_INJECTION64_PATH`, `NVTX_INJECTION64_PATH`, `TRITON_*` paths; engines run as root. - Replace the env denylist with an allowlist (CUDA_/NCCL_ tuning keys minus anything ending in `_PLUGIN` or containing `INJECTION`, `VLLM_*`, `HF_*` minus path keys, `OMP_*`, `GGML_*`). Apply at catalog validation (`internal/control/catalog/catalog.go`, where `UnsafeEnv` is called) and again in the agent. - Deny engine flags that write files or run code: `--trust-remote-code`, `--log-file`, `--slot-save-path`, `--chat-template` with a path, `--host`, `--api-key`, `--port`; vLLM/SGLang equivalents. Allowlist is better than denylist if the flag set is enumerable per engine. - Run engines as an unprivileged user with the video/render groups (`internal/agent/supervisor`, systemd unit in `internal/cli/serverinit_cluster.go:52`). Weights cache must be readable by it. - Done when: a profile with `--trust-remote-code` or `NCCL_NET_PLUGIN` is refused at `ctl model add` and at the agent; `ps` shows engines not as root. ### A5. Admin API trusts any local process (M) `internal/cli/server.go:357` serves the admin API and pprof on `127.0.0.1:9090` with no authentication; `adminapi.actorOf` (`internal/control/adminapi/api.go:1613`) takes the actor from the client-sent `X-Llmgw-Actor` header. - Serve the admin API on a unix socket (`/run/llmgw/admin.sock`, 0660, group `llmgw-admin`) by default, keeping TCP as an option that requires a bearer token read from a root-only file (`/etc/llmgw/secrets/admin-token`). - Take the actor from `SO_PEERCRED` (uid → username) on the socket, or from the token's name; ignore the header. - Refuse `admin_listen` on a non-loopback address unless `admin_insecure: true`. - Put pprof behind its own `debug.pprof: true` setting, off by default. - `ctlClient` (`internal/cli/ctl.go`), the console and `node add` must work over the socket; `sudo llmgw` already runs as root. ### A6. Patroni REST and etcd are unauthenticated and plaintext (M) `internal/pgha/config.go:99` writes a `restapi:` with no `authentication` and `:54` etcd over `http://` with no client auth; `hba.go:56` PATCHes Patroni's config unauthenticated. Anyone reaching :8008 or :2379 owns the store. - Generate a Patroni REST username/password at setup, store it in `/etc/llmgw/secrets/`, write `restapi.authentication` and `restapi.certfile/cafile` from the cluster CA; send it from `internal/pgha/hba.go` and `status.go`. - etcd: client and peer TLS from the cluster CA with `--client-cert-auth`. - Document, and have `server init` write, nftables rules limiting 2379/2380/8008/5432 to member addresses (see A9). - Done when: an unauthenticated `curl :8008/config` is refused and the store status in the console still works. ### A7. PostgreSQL traffic is plaintext (S) `internal/pgha/config.go:168` and `internal/cli/serverinit_postgres.go:330` use `sslmode=disable`; HBA lines are `host`. Enable server TLS in Patroni's postgresql parameters with a certificate from the cluster CA, write `hostssl` lines, use `sslmode=verify-full&sslrootcert=/etc/llmgw/ca.pem`. Existing clusters: `llmgw server` must keep working against a store that has not been migrated, with a warning. ### A8. Smaller fixes, each S or less - **Capacity fallback skips the allowlist.** `internal/router/router.go:700` builds `[canonical]+entry.Fallbacks` without `permits`. Filter the chain with `r.permits(p, model)`; price against the model that served. - **Suspended users keep working API keys.** `store.(*DB).Resolve` (`internal/store/memberships.go`) never joins `users.state`; `Authenticator.Invalidate()` is never called. Reject keys whose owner is not active; call `Invalidate` on revoke, suspend, limit change. - **Join codes never expire server-side.** `internal/join/join.go:103` checks expiry only in `Parse`; `admit` (`:212`) does not. Check it there; bind the listener to the chosen address, not `:7444` on all interfaces; use the peer's address rather than the claimed `hello.Addr`, and show the joiner's address for the operator to confirm before secrets are sent. - **Leftover sudoers grant.** `internal/onboard/gossh.go:248` writes `NOPASSWD:ALL` and removes it only in `Close()` (`:271`). Write the grant with a self-removal (`at`/systemd-run timer, 30 min) so a crash leaves nothing; wipe password buffers after use. In `ssh.go:39` pass `--` before the target. - **Auth cache and pre-auth DoS.** `internal/auth/auth.go:127`: unbounded map, one DB query per bogus well-formed key. Bounded LRU, negative cache with short TTL, per-IP token bucket before the DB. - **HTTP server hardening.** `server.go:351-360`, the h2c agent server, both mirror servers: set `ReadHeaderTimeout` (exists), `IdleTimeout`, `MaxHeaderBytes`, a body-read deadline, `netutil.LimitListener`; connect `WithReadMaxBytes` on the agent stream. - **Login/signup abuse.** Default `signup_open: false`; per-IP token buckets on `/login` and `/signup`; bounded wait for argon2 slots; CSRF token on the login form; the per-address throttle must not lock the real owner out (use IP+address pairs). - **Trusted proxies.** `internal/web/account.go:557`, `landing.go:60` honour `X-Forwarded-Proto/Host` from anyone; add `router.trusted_proxies` and only then trust them; `Cache-Control: no-store` on `/setup/*`. - **Certificate revocation.** `DELETE /api/v1/nodes/{id}` only drops inventory; add a revoked-CN/serial list in the store checked in `agentsvc` `Connect` and mirror `onlyClients`; `serverinit_cluster` always passes `--api-client`. - **Engine exposure.** Engines bind `0.0.0.0` (`llamacpp.go:102`, `vllm.go:109`): bind the management address. SGLang receives its key on the command line (`sglang.go:102`): pass it by env or file. - **Backups.** `internal/cli/backup.go`: encrypt archives to an `age` recipient from `backup.recipient` in server.yaml; keep the directory 0700. - **Python dependency confusion.** `internal/agent/provision/python.go:121` uses `--extra-index-url` with `unsafe-best-match`. Pin torch to the PyTorch index explicitly (uv `[[index]] explicit = true`), `first-index`, hash-pinned lockfiles per engine version. - **Control-plane SSRF.** `internal/sourcecheck/sourcecheck.go:39` and `gguf.FromURL` use the default client; reuse the mirror's `fetchClient(false)` (private-address filter) and refuse a proxy unless configured; block NAT64/6to4-embedded private addresses in `internal/mirror/safefetch.go`. - **Low:** `agentsvc/service.go:430` indexes `n.GPUs[0]` without a length check (CPU-only node panics the handler); thread ids are client-chosen with no length cap and shared per project (`router/threads.go`) — cap length, document, offer per-user scoping; `/v1/models` lists models the key may not use (`handlers.go:799`); sessions have no idle timeout and `PurgeSessions`/`PurgeClaims` are never called — add a janitor; add a CSP header and set `htmx.config.allowEval=false`; cap GGUF array nesting depth in `internal/gguf/gguf.go`; mirror `.source` sidecars are served by the public file server — exclude dotfiles from `:8081`; prune `modelserve.jobs` and the mirror `running` map; KaTeX → 0.16.21+. ### A9. Firewall (S) No machine in a default install has a firewall. Have `server init`, `node add` and `server add` write an nftables ruleset: ssh open; `:8080` open; `:7443` and `:9091` open to the cluster's networks; `:9100` to the monitoring host; `:2379/2380/8008/5432` to store members and control planes; mirror `:8081` to the cluster, `:8082` to control planes; `:9090` Prometheus and `:3000` Grafana to operators. `llmgw doctor` reports a missing ruleset. Make it skippable with `--no-firewall`. --- ## B. Guardrails (M–L, build in this order) **Status (2026-10-04, branch `guardrails-b`):** B1–B4 done, and tried live on the PoC. That found four bugs, all fixed with tests: - a shadow default merged with an attached policy of unwritten mode left it in shadow; - a streamed word held back at one release was dropped at the next; - llama.cpp continues a conversation ending on the answer, so Llama Guard's verdict on answers is now asked with its own prompt as a completion; - a judge model was named in a client's "no endpoint" error. The PoC runs `observe` (shadow) for everyone, and Llama Guard 3 1B (`llama-guard-3-1b`, CPU, on cuda01) as a judge, which blocks a harmful request in ~0.7s and a harmful answer in ~1.4s there. Still left: see each item. Today nothing on the request path reads, classifies, redacts or filters message text in either direction (confirmed; the only "moderation" hit is a comment in `internal/router/budget_fallback.go:20`). Existing controls are about who, how much and how fast: allowlists (`router/handlers.go:450`), RPM/TPM/budget (`internal/accounting`), context precheck (`router/context_limit.go`), admission bands (`router/admission`), timeouts (`router/proxy/proxy.go:26-57`). Gaps to close on the way: no default output cap when the client omits `max_tokens`; `catalog.Capabilities` is advertised but never enforced; tool results are trusted input. ### B1. Hooks in the proxy and router (S) **Done (branch `guardrails-b`):** `internal/guard` holds the interface; proxy `Filter`/`StreamFilter`; the input hook runs after `loadThread` (the simpler choice); error codes, headers, Responses `incomplete`, usage columns `guard_*`. Moved to B2: holding `delta.tool_calls` until they are whole and reading their arguments — that is the tool policy's filter, not the proxy's. - `internal/router/proxy/proxy.go`: add optional `InputRewrite func(body []byte) ([]byte, Verdict)` on the request and two output hooks: `Filter` for `relayBuffered` (between `ReadAll` and `WriteHeader`) and `StreamFilter{OnEvent(data) (emit []string, stop bool); OnEnd()}` for `relayStream`, called where each whole SSE event is assembled (`eventBuffer`, before `io.WriteString` at `:343`). Nil hooks cost nothing. - `internal/router/handlers.go`: call the input policy after the allowlist and limits and before admission (around `:395`); the principal, scopes and the model that will serve are known there. Thread history is spliced later (`:416`): scan stored turns when written, or move the hook after `loadThread` (decide; the second is simpler). - Blocking conventions: input → 400 `invalid_request_error` with `code: "content_policy_violation"` (today `writeError` always sends `code: null`); output non-stream → 200 with `finish_reason: "content_filter"` and empty/redacted content; output stream → a final chunk `{"choices":[{"index":0,"delta":{},"finish_reason":"content_filter"}]}`, then `data: [DONE]`, then cancel upstream. Headers `x-llmgw-guard: blocked|redacted|flagged` and `x-llmgw-guard-policy` set before the proxy writes headers. - `internal/router/responses.go`: map `content_filter` to `response.incomplete` with `incomplete_details.reason: "content_filter"` (today it always emits `response.completed`, `:525`). - Tool calls: hold `delta.tool_calls` until `finish_reason: "tool_calls"`, then validate; `replyDelta` (`reply.go:33`) must also read `tool_calls[].function.arguments`. - Accounting: a blocked input is recorded like `RecordRefused` (nothing charged); a blocked output is charged (the GPU was used). `store.Usage` gains `GuardAction`, `GuardPolicy`, `GuardCategory`, `GuardMs`. ### B2. Package `internal/guard` (M) **Mostly done (branch `guardrails-b`):** policies in server.yaml with attachments in the store (`ctl guard attach`), narrowed by the limits' rule; `guard/detect` (validated PII, secrets, injection through normalisation and decoding); redaction with consistent placeholders before prefix routing; tool allow/deny/strip, call count and argument schemas; capabilities and structured output (opt-in); default `max_tokens`; pinned system prompt and dropping the client's; `no_store` (threads, notes, prefix tree, external models); shadow mode. **Left:** a `guard_events` table — the usage row's `guard_*` columns carry action/policy/category/time, not rule and stage; language identification; the re-identification map that restores redacted values in answers. - `Policy`, `Effective` resolution over the tenancy tree with the same narrowing rule as limits (`internal/store/sqlite.go:383-416` `tighten`): deny sets union, allow sets intersect, `mode` takes the strictest of `enforce > shadow > off`, `fail` takes `closed > open`, `no_store` at any level wins, thresholds take the minimum. - Store: `guard_policies(scope, scope_id, policy json)` and `guard_events(request_id, scopes, policy, rule, category, action, stage, score, judge_ms, shadow)` — ids, categories, offsets and lengths only, never text. Migrations for sqlite and postgres. Pruned with `usage_retention`. - `guard/detect`: regex/keyword sets; PII with validators — email, phone, IBAN (mod-97), card (Luhn), TCKN (checksum), IP, secrets (`sk-`, AWS, JWT, PEM); normaliser (NFKC, strip zero-width/bidi, homoglyph fold, base64/hex/URL decode to depth 2); injection heuristics (role-switch phrases, fake chat-template tokens, long base64, invisible Unicode) — default action `flag`; language id. - Redaction rewrites `messages[].content` in the upstream body with placeholders (`[EMAIL_1]`) via the same `map[string]json.RawMessage` round-trip `withModel`/`spliceThread` use; an optional re-identification map lives only in request memory and restores originals in the output. Redact before prefix routing so `PromptPrefix` hashes the redacted text. - `guard/toolpolicy`: tool name allow/deny (strip or 400), JSON-Schema validation of assembled arguments against the declared `parameters`, argument regex rules, `max_calls_per_turn`. Enforce `Capabilities.Tools`/`Vision` here: a `tools` array or image parts to a model without the capability → 400. - `guard/structured`: validate `response_format.json_schema` on the final text → `content_filter`, or flag in shadow. - Output constraints: inject a default `max_tokens` when absent (from the policy or `MaxOutput`), language check, structured check. - System-prompt pinning per project: prepend before `checkContextFits` and before `spliceThread`; optionally drop client `system`/`developer` roles; exclude the pin from stored thread turns; apply to Responses `instructions` in `toChat`. - `no_store`: `loadThread` and `PutNote` refuse; `captureBytes()` returns 0; skip the prefix-tree observe (`router.go:840`); disable External fallbacks for the scope (filter in `dispatch` and `budgetFallback`). ### B3. Judge model on the cluster (M) **Done (branch `guardrails-b`):** `role: guard` + `guard.format`; hidden and refused to clients and fallbacks; critical by default, never on demand; `Router.Judge` past admission/accounting/guardrails; llamaguard3, shieldgemma and json parsers; 5-minute cache; latency budget then fail open/closed; streamed answers read every 600 characters, or held with buffer_all. Simpler than written: the input judge runs after admission, not alongside it, and shadow mode judges synchronously within the budget rather than after the response. Not enforced: two replicas on different nodes — documented, left to `scaling.min`. - `catalog.Model.Role: "guard"` plus `Guard{Format: llamaguard3 | shieldgemma | json, Categories}`. Guard models are hidden from `/v1/models`, the chat picker and allowlists; placed like any model, class `critical`, `scaling.min ≥ 1`, at least 2 replicas on different nodes. - `guard/judge`: calls the guard model through the router's own `dispatch` with an internal principal and its own admission band (never the tenant's queue); parsers per format (Llama Guard `safe`/`unsafe\nS1,S10`; ShieldGemma yes/no probability → per-tenant thresholds; generic JSON verdict); verdict cache keyed by hash of normalised new turns (5 min); `latency_budget` (default 250 ms) then `fail` open/closed; input judge runs in parallel with admission queueing. - Streaming strategy: default sentence-buffered scanning (regex/PII per sentence or ~200 chars before release; judge every 400–800 chars on the cumulative tail; final judge at end); `buffer_all` for `fail: closed` scopes. Document that released text cannot be retracted. - Shadow mode runs everything asynchronously after the response (`context.WithoutCancel`, like `saveThread`) and only records. ### B4. Config, metrics, UI (S) **Done (branch `guardrails-b`):** library in server.yaml, attachments via `ctl guard`, the console's **Guardrails…** with "applies from" marks, and the portal's `/admin/guard`; the metrics and alerts below (judge latency labelled by model only at the judge; the check latency by stage); `ctl request` shows the verdict; the chat page says "stopped by policy"; the profile lists effective policies; docs/guardrails.md and data.md written; the access log carries action, stage and policy only. Attaching from the portal is not done — the page shows; the console and ctl change. - Gateway-level policy library in `server.yaml` under `guard:` (see the YAML in the review, reproduced in `docs/guardrails.md` to be written); attachments to scopes in the store via `llmgw keys guard`, the console ("Guardrails…" beside "Limits…" in `internal/cli/tui_people.go`, with "inherited from team:X" badges) and the portal. - Metrics in `router/metrics.go` style: `llmgw_guard_checks_total{stage,policy,action,category,mode}`, `llmgw_guard_latency_seconds{stage,judge_model}`, `llmgw_guard_judge_errors_total{reason}`, `llmgw_guard_fail_open_total`, `llmgw_guard_stream_stops_total`, `llmgw_guard_redactions_total{kind}`, `llmgw_tenant_guard_blocks_total{tenant}`. Alerts: fail-open > 1% for 5m, judge p95 over budget, judge has no ready replicas, per-tenant block rate > 3× its 7-day baseline, any `secret` detected in output. - Portal `/admin/guard`: violations by tenant/category/policy over time, shadow vs enforce, judge health, recent events by request id; `ctl request ` shows the verdict; chat page renders `content_filter` as "stopped by policy X"; `/me` lists effective policies. - Update `docs/data.md`: the judge sees all text but in-cluster; External fallbacks send prompts outside the cluster (the current "nothing leaves the cluster" is wrong); `no_store` semantics. - Never log: prompt/completion text, PII values, the re-identification map, judge inputs or rationale that quotes input. Check `accessLog`'s `err` field and span errors. --- ## C. Features, ranked by value × feasibility 1. **Done (branch `features-c`):** `audit_events`, written by the admin API middleware and the /admin pages (with each change's result), `ctl audit`, the console's Audit view; staff roles viewer / operator / billing / admin. Left: before/after values (the request's fields are recorded, not the state they replaced); a tenant-admin role that changes its own organization (managers still only view it); hiding the forms a viewer cannot submit. **Audit log and roles (M).** Today "audit" is slog lines (`internal/web/admin_users.go:403-480`) and roles are the `IsAdmin` flag plus an org "manager" who can view. Add `audit_events` (who, what, before/after, source) written by the `adminAction`/`tenancyAction` wrappers and by every admin-API mutation; roles viewer / operator / tenant-admin / billing; `ctl audit`. Depends on A5 for a real actor. 2. **Done (branch `features-c`):** OIDC sign-in (`auth.oidc`, PKCE, verified email, roles and memberships from groups, `no_passwords`) and SCIM 2.0 (`/scim/v2` Users and Groups). Left: the sign-in cookie is signed with a per-process key, so with two control planes behind a balancer the callback must reach the one that started it (sticky sessions) until the key comes from the store. **OIDC/SSO login (M), then SCIM (M).** `internal/account/service.go`, `internal/web/account.go`; map IdP groups to org/team/admin. 3. **Done (branch `features-c`):** key lifetime limits (gateway, org, team, project), `keys rotate --overlap`, kinds and expiry in `keys list`. **Service accounts and key lifecycle (S).** Per-org maximum key TTL, `keys rotate` with overlap, last-used shown, project-owned service keys listed apart. `internal/cli/keys.go`, `internal/store/keys.go`, `internal/auth/auth.go`. 4. **Done (branch `features-c`):** CSV export from the rollups (API, ctl, console). Left: Parquet, and a scheduled push to S3/SFTP. **Usage export (S).** CSV/Parquet by org/team/project/model/period from the rollups (`internal/store/usageseries.go`), `GET /api/v1/usage/export`, optional scheduled push to S3/SFTP. 5. **Done (branch `features-c`):** `notify.webhooks` (model parked, node lost, budget spent, update available; signed JSON or Slack; retries, hourly de-duplication), `ctl notify test`, Alertmanager receiver templates. Left: `server init` and the console do not yet ask for a receiver — the templates are filled in by hand; each control plane sends budget events on its own, so two behind a balancer may both send one. **Alerting out of the box (S).** `deploy/monitoring/alertmanager.yml` ships `receiver: none`; add Slack/email/webhook/PagerDuty receiver templates driven by `server init` or the console; llmgw-native webhooks for model parked, budget spent, update available (from the reconciler and `modelserve`). 6. **Done (branch `features-c2`):** `metadata`, `x-llmgw-tags` and `user` on the usage record (migration 0023), `ctl request` shows them, `router.tag_labels` metrics with 100 values per tag. Left: export and `ctl usage` by tag or end user (the hourly and daily sums do not carry tags; it would read the request records, within their retention). **Request metadata and tags (S).** Accept `metadata`/`user` from the body or `x-llmgw-tags`; store on the usage row; tenant metric labels behind an allowlist. `router/handlers.go` parse, `accounting/recorder.go`. 7. **Done (branch `features-c2`):** `ctl model canary/promote/canaries`, session-sticky split with `x-llmgw-variant` pins, rollback on error rate or TTFT, `canary_rolled_back` notification, console menu. Left: A/B between two different models (the split is between a model and its newer weights); the comparison is per control plane and lives in memory, so a restart begins it again; a key-level pin. **Canary and weighted A/B between model revisions (M).** `control/modelserve/updates.go` `replace` swaps every replica; add weighted routing across two revisions/aliases (scheduler filter, alias resolution in `router/router.go:569`), a header/key pin, automatic rollback on error-rate or TTFT regression from per-replica metrics. 8. **Done (branch `features-c2`):** `/v1/files` and `/v1/batches` (OpenAI's shape), worker under a store lease, lines kept as served, lowest band, usage tagged `batch=`, output and error files with retention, `no_store` refused. Left: the Responses API (`/v1/responses`) as a batch endpoint; the console and `ctl` do not list batches (an operator sees them through usage tags); a file is one row in the store, so a store backup carries batch files with it. **Batch API and Files (M–L).** `/v1/files`, `/v1/batches`, JSONL storage, a worker feeding the bronze band, accounting. New `internal/router/batch`. 9. **Done (branch `features-c2`):** `keys requests` (retention, sample), kept after redaction, `ctl request --body/--replay`, console, pruning. Left: a streamed answer is kept as its text, not its chunks; replay is one request at a time (a set of them against a model is item 21's eval). **Request/response logging and replay (M).** Opt-in per project with retention like threads; sampled or full; redaction hooks from B; `ctl request --body`; replay against another model. 10. **Done (branch `features-c2`):** Responses `text.format`, `input_image`, `previous_response_id` (thread in the id); rerank and score passthrough; transcription and translation as multipart; `/v1/moderations` by the judge. Left: audio is charged no tokens (an engine's duration usage is not read); `input_image` by `file_id`; moderation scores are 0 or 1, as a judge says yes or no. **OpenAI compatibility gaps (S–M each).** Responses: `text.format` → `response_format`, `input_image`, `previous_response_id` via threads (`router/responses.go`). Rerank/score passthrough for vLLM, `/v1/audio/transcriptions` for a whisper engine, `/v1/moderations` mapped to the guard model. 11. **Done (branch `features-c2`):** `/v1/messages` and `count_tokens`, streamed and not, tools, images, `x-api-key`, Anthropic error shape. Left: extended thinking (a reasoning model's thoughts are not mapped to thinking blocks); prompt caching; the account page's "Connect your editor" does not offer Claude Code yet. **Anthropic `/v1/messages` (M).** Translate like `responses.go`. 12. **Done (branch `features-c2`):** `server restore` (dry run, version check, decryption, modes and owners, SQLite and PostgreSQL), `backup_copy_to` (directory or ssh), the drill in docs/install.md. Left: restoring a replicated store's members (restore into the primary; Patroni replicates it); the console's Back up now does not offer restore; an ssh copy holds the archive in memory while it sends it. **`server restore` (S).** Restore is manual shell today (`docs/install.md:1187`); add the command with version check, dry run, off-host copy target; a documented DR drill. 13. **Done (branch `features-c2`):** TPOT, inter-token and tokens/s histograms by model, pool and tenant; `slo` pressure in admission's saturation; `llmgw_model_slo_seconds` and `llmgw-slo` alerts. Left: the p99 is per control plane, from its own requests; an SLO per band or per tenant; placement does not yet add replicas for an SLO (item 14). **SLO-based admission and token-level metrics (S–M).** TPOT and inter-token histograms, output tokens/s per model and tenant in `router/metrics.go`; use `catalog.SLO` in `admission/saturation.go`. 14. **Done (branch `features-c2`):** `scaling.schedule` (days, from, to, min, max; past midnight), `scaling.prewarm`, `control.timezone`; queue-depth scaling was there already. Left: `ctl scale` sets min and max only (a schedule is set in the definition); scaling for an SLO rather than for queue depth. **Scheduled and warm-pool autoscaling (S–M).** Time-of-day min/max, pre-warm, queue-depth scaling in `catalog.Scaling`, `control/reconciler/demand.go`, `headroom.go`. 15. **Done (branch `features-c2`):** `env:` and `vault:` references wherever a secret file is named (store address, api_key_file, OIDC, SCIM, webhooks), Vault KV with token or AppRole, `secrets.master_key` sealing cluster_secrets (Vault transit or a local key). Left: the certificate authority's key is still a file in /etc/llmgw (signing in Vault's PKI engine instead is its own change); rotating the master key; `backup_copy_key` and `admin_token_file` are files only. **Secrets management (S–M).** Vault/KMS for `api_key_file`, the DSN, the CA key, `cluster_secrets` (`store/secrets.go`, `store/seal.go`); envelope-encrypt them in backups (with A8 backups). 16. **Done (branch `features-c2`):** `users forget` (pseudonymised usage and sums, kept requests, files, batches, limits and audit names), `keys residency --in-cluster` (asked-for, fallback and canary). Left: an export of everything held about a person (article 15); residency by region between several clusters; backups keep what was forgotten until they age out. **GDPR and residency (S).** "Forget user" anonymises `user_id` in usage and rollups and erases threads/memory (`store/hierarchy.go:241` leaves it today); per-project "never route to an External provider". 17. **Done (branch `features-c2`):** exact-match answer cache per project (`keys cache --ttl/--any/--off`, console **Cache answers…**, `projects` columns, 30-second setting cache), reproducible answers only unless `--any`, LRU in memory bounded by `router.cache.max_bytes` and `max_entry_bytes`, streamed answers kept and replayed as chunks, `x-llmgw-cache` and `Cache-Control: no-cache`, guardrails before a hit and the output policy on it, `no_store` and threads bypass, a hit recorded as `cache_hit` with no tokens and no cost (migration 0027), metrics, docs. Left: semantic (embedding-similarity) caching through the embeddings models; a cache shared across control planes (each process keeps its own, deliberately: a shared one would write every answer to the store); a hit still waits its turn in admission, because the guardrails judge the request after admission and the key is the request they let through; logprobs in a streamed answer leave it unkept; `users forget` does not reach into the cache (answers age out by the TTL); no listing or flushing of what is cached (`--off` stops it being served at once, the memory is freed by TTL and LRU). **Response/semantic cache (M).** Exact-match first, per project, opt-in, TTL; semantic via the embeddings models later. Before `dispatch` in `router/handlers.go`. 18. **Done (branch `features-c2`):** offline bundles — `llmgw bundle create` (releases with signed checksums, every llama.cpp build per arch/variant with GitHub's checksums, starter/judge and named Hugging Face model files pinned and checked; directory or tar; manifest sha256), `bundle import` (checked, all-or-nothing, into the mirror under `_llmgw/`), `bundle trust`, `bundle show --verify`; `offline.mirror`/`offline.checksums` in server.yaml, with nodes installing llama.cpp from the mirror against checksums the control plane sends over the agent stream (proto `EngineWant.offline_*`), releases defaulting to the mirror for `node add`, `ctl upgrade` and agent upgrades, update checks off; `server init --bundle` and the setup form's bundle fields; docs/data.md's offline claim corrected; install.md "Offline sites". A Helm chart for the control plane (`deploy/helm/llmgw`, one replica, SQLite PVC or external PostgreSQL, Secrets, Services, Ingress, ServiceMonitor) with `deploy/container/Dockerfile` and docs/kubernetes.md; GPU nodes stay outside Kubernetes by design. Left: vLLM/SGLang offline (a wheelhouse of several GB per CUDA build and a uv `--find-links` install path); the NVML release binary and uv in bundles; the setup wizard's preflight still fails a machine without internet; `join`'s printed `curl` from the mirror checks nothing; models in a bundle are single files (no whole- repository safetensors sets yet); the chart runs one replica (a standby is never ready, so HA in Kubernetes needs a different readiness design), has not been installed on a real cluster, and ships no agent DaemonSet; no published container image. **Kubernetes/Helm and an air-gapped bundle (M).** `deploy/`, `agent/provision`; `docs/data.md` claims offline operation but install assumes the internet. 19. **Done (branch `features-c2`):** `adminapi.Routes` and `internal/docgen/openapi` generating `docs/admin-api.openapi.json` (3.1, bearer security, every route, query parameters, schemas from the Go types with their comments), served embedded at `GET /api/v1/openapi.json`, kept true by a route-table test and a staleness test; validated with openapi-spec-validator and openapi-generator-cli. `GET /api/v1/models/{name}`. `docs/admin-api.md`. `terraform-provider- llmgw/` (terraform-plugin-framework): `llmgw_model`, `llmgw_quota`, `llmgw_guard_attachment`, data sources `llmgw_models` and `llmgw_nodes`; client tests against a fake admin API and the OpenAPI document; acceptance tests with Terraform and OpenTofu against the fake; one apply, empty plan, in-place change and destroy against a real `llmgw server` run by hand. Organizations, teams, projects, keys and memberships are in the admin API since 2026-10-08 (`/api/v1/orgs`, `teams`, `projects`, `keys`, `users/{id}/memberships`) and in the provider (`llmgw_org`, `llmgw_team`, `llmgw_project`, `llmgw_key`, `llmgw_membership`); accounts are still made by sign-up, invitation or OIDC/SCIM. Left: a model's definition beyond scaling, class, owner and adapters is changed by replacing it (the API has no edit, and `resources` is not used yet); `canary`, `slo`, `scaling.schedule` and `prewarm`, `guard` role and `origin` are not in `llmgw_model`; models cannot be imported; the provider is not published to a registry and has no release build; the acceptance tests are not run in CI (they need Terraform); the document's schemas mark nothing required and give no enums; `catalog.SLO`'s fields have no JSON tags, so they are `TTFTp99` and `TPOTp99` on the wire. **OpenAPI for the admin API, Terraform provider (M).** Generate from `adminapi/api.go` routes via `docgen`. 20. **Done (branch `features-c2`):** spans for the reconcile pass (begun only when a pass does something), placement with its answer or rejections, scale-up/down, drain, stop, eviction, preemption, reaping, restarts and lost nodes; a `replica start` span from placement to serving whose context travels to the agent in the start command (`ControlMessage.trace`), where weights fetch, engine launch, rendezvous and readiness wait continue the trace; stops, drains, adoption and engine installs; modelserve jobs as a trace with a span per step linked to the replica's start; an agent `tracing` block; docs in install.md's "Tracing". Left: the router's span for a request does not link to the replica it was sent to's start; heartbeats, fabric scans, logs and agent upgrades are not traced; the supervisor has no restart span because it never restarts an engine itself (a restart is the reconciler's `restart replica` and reaches a node as a stop and a start); canary and promote jobs, which landed on `features-c2` after this was branched, are traced only through the shared step recording (as ` model`), unchecked by a test. **Tracing beyond the router (S).** Spans for reconciler, placement, agent start (`obs/trace.go`, `control/reconciler`, `agent/supervisor`). 21. **Done (branch `features-c2`):** `internal/bench/eval` (suites in YAML or JSON lines; exact, contains, regex, number, a JSON Schema subset, and a judge model with a rubric; weights and a threshold), run on the control plane through `Router.Replay`; `llmgw ctl eval MODEL --suite FILE|NAME`, `--history`, `--run`, `--suites`; runs and their cases kept (migration 0028, `eval_runs` and `eval_cases`); `model update`/`promote --eval SUITE --min-score N`, a model's standing gate (`ctl model gate`, `eval:` in the catalog) — a promote asks the canary first, an update tries the new weights as a 0% canary and promotes or removes it; the console's model menu; `arithmetic` and `json-format` in `deploy/eval/`, carried in the binary. Left: the suite of a model's standing gate is read by name on the control plane at the time of the update, so two control planes need the same `control.eval_dir`; a gated update needs room for one more replica while it runs (there is no in-place gate, and a declared model in server.yaml cannot be updated at all); no comparison against the old weights' score (the gate is an absolute threshold, not "no worse than before"); a run is not stopped once started (Esc in the console only stops watching it), and the in-memory progress of a run lives on the control plane that started it; runs are never pruned; the JSON Schema support is a subset (no `$ref`, `oneOf`, `not`, `uniqueItems`); judge cases go through the same concurrency as the model's, and a judge's answers are not kept, only its score and reason; a gateway without a control plane cannot run suites (there is no mode that asks through the client API with a key). **Eval harness (M).** `llmgw eval` extending `internal/bench` with quality suites gating `model update`. 22. **Done (branch `features-c2`):** MIG instances discovered on both paths (nvidia-smi `-L` + `-q -x`; NVML through an interface with a faithful fake) and reported instead of their card, each with its MIG UUID, own memory and class `-mig-`; device numbers 1000 + 100 × card + instance, shown as `card/migN`; placement on MIG classes; tp/pp/gpus/nodes above 1 on a MIG class refused in catalog validation and again in placement; `CUDA_VISIBLE_DEVICES=MIG-` for vLLM and llama.cpp (no Vulkan device variable on MIG); per-instance memory and processes in heartbeats and the GPU check, card sensors under each instance, utilisation absent rather than zero; doctor MIG findings (`mig -lgi`/`-lci` parsed); `ctl nodes`, console, `ctl status`, `ctl model plan`/`serve` by class; docs/install.md MIG section. Written and tested against fakes only — untested on real MIG hardware. Left: run it on an A100/H100 in MIG mode (check `-L`/`-q -x`/`mig -lgi`/`-lci` and NVML MIG device names against the parsers, and that `CUDA_VISIBLE_DEVICES=MIG-…` reaches vLLM's workers through Ray); instances changed while the agent runs are only seen when it restarts (no rescan); llmgw does not create or reshape instances (no mig-parted integration); quotas count an instance as one GPU, the same as a whole card; SGLang on MIG not checked; device numbers assume fewer than 100 instances per card and that instance numbers stay put while shards run on them. **MIG (M–L).** Treat a MIG slice as a GPU class in `agent/discovery/gpu_nvml.go` and `control/placement`. 23. **Done (branch `features-c2`):** engine `trtllm` (`internal/agent/engines/trtllm.go`): `trtllm-serve` through a launcher of llmgw's that demands the replica's key (trtllm-serve has none) and writes the options file that turns on `/prometheus/metrics` and the iteration statistics; the scrape contract for it (`scrape.TRTLLMMapping`, with a per-engine metrics path); found, not installed — `engines.trtllm.python` in `agent.yaml`, probed from the package's metadata; catalog validation (GPU class, one node, no GGUF source, the flags a profile may not give); `ctl engine-check trtllm`, which asks `trtllm-serve serve --help` through its module. Ollama/GGUF import: `mirror import-ollama` (manifest parsing, blob digest check, projector beside the model, all or nothing, a model definition printed), `mirror import` for any local file, and the console's **I** on the Mirror view. Left: TensorRT-LLM has not run on real hardware — the first node with one should run `engine-check` and serve a model before anyone relies on it; no installer for it; no multi-node TensorRT-LLM replica (it needs an MPI or Slurm launch from one node); Triton Inference Server as an engine is not done; trtllm's queue gauges are set per iteration, so an idle engine shows its last batch until the next request; `--num_serve_frontends` is refused because extra front ends would not demand the key. The Ollama import runs on the mirror machine only — the mirror's API takes no files, so an Ollama elsewhere means copying the blob or running the import there; Ollama models kept as separate tensors or as several weight layers, and its LoRA adapters, are refused or left with a sentence; files an import run with `sudo` writes are root's, as `mirror add`'s are. **More engines (M each).** TensorRT-LLM/Triton adapter; Ollama/GGUF import. `agent/engines/engine.go` interface and the metric contract in `router/scrape`. 24. **Done (branch `features-c2`), all six parts.** Lower: precise KV-index routing (plan phase 6), LoRA affinity for vLLM, multi-cluster federation via External providers, facility-aware placement (`docs/facility-aware-placement.md`), portal i18n, SDK/docs portal. What was built, and what is left: - 24 (portal i18n): built — English/Turkish catalogue with fallback, per- language templates, cookie/Accept-Language/`landing.language` choice and a footer switcher, Turkish numbers and dates, every template and handler message translated, tests that no template text stays English; left — errors built with fmt.Errorf in other packages (store, catalog, mirror, adminapi) still English on Turkish pages, percent sign placement is mixed (%50 on limits, 93% on cluster pages), RFC3339 footer timestamps and plain-text http.Error bodies untranslated, the /docs documents themselves English, a per-person language column (cookie only today). - 24 (SDK/docs portal): built — /docs renders the embedded manual (goldmark, GFM, heading anchors, links rewritten, no script; public, `landing.docs_sign_in` to require a session), and docs/clients.md shows the official OpenAI/Anthropic SDKs against llmgw. Left: the console's generated "Commands" page is not in the portal (it is built from the cobra tree, which internal/web cannot see); no search across documents. - [~] 24. Precise KV index (tier 2) — built: vLLM KV events via the agent (loopback ZMQ, pure-Go zmq4, msgpack in both of vLLM's encodings) forwarded as net block-hash changes over the agent stream; router index, tokenize via the replica's `/tokenize`, sha256_cbor block-hash chain checked against cbor2/hashlib golden values; scorer by leading- block share with tier 1 fallback; opt-in `kv_events` per profile; metrics; docs. Left: validation against a real vLLM (event shape, hash agreement, `/tokenize` matching the served prompt, cost of tokenizing); vLLM's replay endpoint instead of resetting on a gap; data-parallel ranks (one publisher each, ports offset); hybrid models' cache groups and CPU-offloaded blocks; LoRA/multimodal/cache_salt requests (extra keys); a router-only deployment (no agents) subscribing directly; standby control planes have no index until they lead; a per-replica fallback when only some candidates have an index; caching tokenizations of repeated prompts. - **24 LoRA affinity (branch `features-c2`):** built `max_loaded_adapters` (catalog), vLLM adapter support at start and on demand (`engines/lora.go`, `vllm.go`), the agent linking fetched adapters as `llmgw-lora/` in the engines' working directory, the router's adapter tracker (`router/lora.go`: loads/unloads with the engine key, LRU eviction never of an adapter in use, one load per adapter per replica, `/v1/models` sync, 404 reload) and the `adapter-affinity` scheduler filter in the chat, batch and long-context profiles, with metrics and docs; tested against a fake of vLLM's HTTP interface. **Left:** never run against a real vLLM — in particular that vLLM resolves the relative `llmgw-lora/` path against its working directory on every node (multi-node too) and that unloading an adapter mid-request is harmless; the agent still fetches every adapter at replica start, so a new adapter needs a restart (fetching on first load would need an agent command and a proto change); load timeout and resync interval are constants, not config; SGLang's runtime LoRA API is not driven; no per-replica "adapters held" view in the console or admin API; the debug profile has no affinity. - 24. Multi-cluster federation — built: peers via `ctl federation` kept in the store, their models as PEER/MODEL synced from /v1/models (withdrawn-while-referenced), /ready health with latency and routing withdrawal, `ctl model fallbacks` for overflow, hop header loop refusal, residency/no_store hold, optional attribution tags, console view, metrics, docs. Left: server.yaml-declared models cannot name a peer model as a fallback (config catalog is validated before peers are read); per-peer CA / mTLS for peers behind a private CA; transitive federation (A→B→C models) deliberately not offered; latency is /ready RTT only (no per-request peer TTFT used in routing or weighting); the peer's adapters are not imported (listed as aliases); no console form to add/remove peers (CLI only); key file must exist on every control plane. - 24 (facility) — built: location labels (`ctl label`, agent.yaml), spreading across racks/PDUs, `keep_within`, held domains with drain (`ctl facility hold/release`, kept in the store), power caps from `facility.budgets` against declared/derived/measured draw, shown in explain-placement, `ctl nodes`, `ctl facility`, the console and the API. Left: temperature conditions with hysteresis and their enforcement (steps 3–4), a Prometheus/DCIM source, power headroom as a rack-levelling score, spreading multi-node replicas across racks, `llmgw_domain_*` metrics and alerts (near cap, over cap, stale data), shedding and tokens per watt (step 5), and a drained node's state surviving a restart. --- ## E. Every setting takes effect when it is changed (2026-10-05) Asked for after a site visit: an upgrade left server.yaml without guardrails, a node on another subnet stayed lost until firewall_networks was edited by hand, and nothing in server.yaml, agent.yaml or a generated unit applies until somebody restarts the right service on every machine. The rule from here on: **a file holds only what one machine needs to start; everything else is the cluster's, kept in the store, changed through ctl, the console or the web, audited, and in force on every control plane within seconds.** ### E1. Cluster settings in the store (L) — done 2026-10-05 Live on the PoC (140, 150): imported as version 1; a guard change applied on both without a restart; a router change restarted the standby at once and the leader 30s later with /health answering throughout; rollback did the same. - Table `cluster_settings` (versioned YAML document, who and why). The cluster-wide part of server.yaml lives there; server.yaml keeps the per-machine keys only (listeners, TLS paths, store DSN, secrets, tracing, monitoring, mirror.api/tls, backup_dir/key, endpoints). - First start against a store with no document imports the file's cluster keys (and the baseline guard policy when there is none); after that those keys in a file are ignored, with a warning. - Each control plane polls the document's version and its own server.yaml's hash. A change is validated, then applied: `guard` live, the rest by a graceful restart of the process (standbys first, the leader last), which systemd brings back. No "restart now?" anywhere. - `llmgw ctl config show|history|get|set|unset|edit|rollback`; the console's settings screen edits the store for cluster keys; admin API `GET/PUT /api/v1/settings`, `/history`, `/rollback`. - A standby that becomes leader re-reads the catalog, engine choices and facility state from the store (it kept what it read at start). - Retires `appendGuardIfMissing` (server init): the import does it. ### E2. Generated files from settings (L) — firewall on/off done 2026-10-06, every machine 2026-10-09 `firewall.enabled` in the cluster's settings reaches nodes over the agent stream and control planes through llmgw-firewall-sync.path. Since 2026-10-09 the mirror machine too (the leader PUTs /v1/firewall, the mirror writes `.firewall.json`, the same path unit applies it) and store members that run no llmgw (the leader keeps it in `cluster_state`, a root timer reads the member's own copy with psql); `firewall_networks` reach all of them, and the mirror lets in the network of every node and control plane the leader knows. Tried on the PoC: off and on, and a network added, on control planes, mirror, store member and nodes. Left: the allowed set replaced by membership rather than added to (a network taken out of the settings stays let in), the other files (the mirror's unit flags, keepalived, etcd and Patroni configuration). Every machine runs the agent (root) with roles — node, control plane, mirror, store member — and the agent renders and applies what was written once by init: the llmgw nft table, the mirror's unit flags, keepalived, etcd and Patroni configuration. The firewall's allowed set is the cluster's membership (every node, control plane, mirror and store member) plus `firewall.extra` from the store; 7443 stays open (mTLS). A node added or removed changes every machine's rules. Retires `firewall.Admit` and `firewall_networks` in server.yaml (imported into the store). ### E3. agent.yaml down to bootstrap (M) — done 2026-10-08 (branch `settings-e`) agent.yaml keeps the control planes' addresses, its certificate and its id. Labels, fabrics, engines, heartbeat and supervisor settings move to the store per node and reach the agent over its stream; a change to the file is noticed and the agent restarts itself. Built as a section of the cluster's settings rather than a store of its own, so history, audit, `ctl config`, the console and the portal page cover it: `agents.default` and `agents.nodes.` (config.AgentSettings; bootstrap keys node_id/server(s)/hostname/mgmt_addr/tls refused; each node's merge loaded as an agent would, so a bad fabric is refused on save). A live key: the control plane sends ControlMessage.AgentSettings on register and, a second apart per node, on change; the agent compares with what it started on, checks it loads beside its agent.yaml (else logs and keeps running), keeps it in /var/lib/llmgw-agent/agent-settings.yaml and restarts (engines adopted). At start: agent.yaml < kept settings < LLMGW_ env; kept settings that no longer load are set aside, never a reason not to start. agent.yaml is watched every 10 s and restarted on when it changes and loads. Left: file keys are not imported into the settings (they keep working under them); `node add` still writes a full agent.yaml; labels have two homes now (`ctl label`, which wins, and agents.*.labels). ### E4. Web settings page (S) — done 2026-10-08 (branch `settings-e`) The same document, edited on /admin/settings, for whoever has no terminal. Built: administrators only (staff roles get 404: webhook secrets), checked by the same function as PUT /api/v1/settings, made from a base version or refused on a newer one (the edit kept in the editor), history with view and roll back as a new version, audit with the size not the body, English and Turkish, a Settings link in the menu. --- ## F. Company documents: knowledge bases (RAG), then adapters (2026-10-08) A company wants a model to answer from its own documents. Retrieval is the tool for *knowledge* (sources shown, re-indexed in minutes, access per tenant); LoRA training is for *behaviour* (vocabulary, format) and has no access control — whatever goes into weights reaches everyone who may use the adapter. RAG first. Decisions taken: access per collection (no per-document ACL yet); a personal key or session names a project or team and gets that chain only, never the union of its person's memberships; PostgreSQL store only, off on SQLite with a sentence saying why. ### F1. Store: collections, grants, chunks (M) — done 2026-10-08 (branch `rag`) Built (`internal/store/knowledge.go`, migration 0035 PostgreSQL only): - `kb_collections` (owner scope/id, org, embed_model, dims), `kb_grants` (read|write, any scope, never across organizations; a key needs a project; a person needs a membership in the organization), `kb_documents` (digest: the same again is no change, a new one replaces in one transaction). - `kb_chunks` is created when the store opens, not in a migration (a migration needing an extension would stop the control plane; one that skipped it would never run again): partitioned by organization (hashed partition name), `tsv` generated (`simple`), GIN on the parent, one partial HNSW index per collection on its organization's partition (`embedding::vector(dims)`, `WHERE collection = ''`). EXPLAIN on 20k rows: partition pruned, that index used, policy applied. Later (scale, 2026-10-08): a partial GIN per collection too (`kb_tsv_`, backfilled at open); `vec_type` per collection, `halfvec` on pgvector ≥ 0.7 (up to 4000 dims); `SET LOCAL hnsw.ef_search` to candidates(k) clamped 40..1000 (EXPLAIN: 40 rows without it, all with it); the text arm takes candidates by `ts_rank` in a subquery and scores only those. - RLS FORCEd on `kb_chunks` against `llmgw.kb_org`/`llmgw.kb_collections` (set with set_config per transaction), one role — the table owner — not a separate one. Chunks are deleted by the foreign-key cascade from their document, which RLS does not block. - `CollectionsFor(Effective, write)`: owner or grant anywhere on the chain; no organization (personal key without a project) reads nothing. - `SearchChunks`: per collection nearest-neighbour (literal id and cast so the partial index is used) plus a full-text arm with any word (`a | b`, lowercased by the database, not Go), fused by reciprocal rank. - Deleting a project/team/org that owns collections is refused like one with members (changed from "deleted with them": every other delete in the tree refuses); DeleteUser/ForgetUser delete the person's collections; deleting an organization drops its partition; grants to a deleted scope go. - `server init` installs `postgresql-N-pgvector` / `pgvector_N` and enables it; pgha installs it on every member at setup and enables it on the leader on every run. Neither stops on failure. Left: a cluster set up before (the PoC) gets no package from a rerun of pgha on a running cluster — install by hand or add an upgrade step; `server restore`/`CopyTo` do not carry kb tables (SQLite → PostgreSQL only, and SQLite has none); tests need `LLMGW_TEST_POSTGRES` pointing at a pgvector database as a non-superuser (a superuser bypasses RLS). ### F2. Ingest on the user's machine (L) — done 2026-10-08 (branch `rag`) Uploading whole files to the control plane was dropped: a PDF is 20–100× its text, and the work belongs where the files are. Instead a small client, `llmgw-kb`, run by the person on their own machine with their own key: - **Its own binary** (`cmd/llmgw-kb`), for Windows, macOS and Linux on amd64 and arm64. `llmgw` itself does not build on Windows (agent syscalls), and employees' laptops should not carry the server, agent and setup screens. A test keeps server packages out of its imports. - `llmgw-kb sync [--include/--exclude globs] [--dry-run] [--delete]`: scan → local catalog (path, size, mtime, sha256; cache under os.UserCacheDir, digests not recomputed for unchanged files) → the collection's manifest from the gateway (name → digest only) → diff → for new/changed files only: extract, chunk, embed through the gateway's `/v1/embeddings` with the person's key (its accounting, limits, guard and band apply), and send name, digest, chunk texts and vectors; `--delete` removes documents whose files are gone. `--dry-run` lists what would leave the machine and how much. A file that cannot be read (locked, unsupported, no text) is skipped with the reason; the run goes on. - **Formats, as many as possible.** Built in, pure Go: PDF (text layer), DOCX/XLSX/PPTX, ODT/ODS/ODP, RTF, EPUB, HTML/XHTML, EML (text and HTML parts), Jupyter notebooks, subtitles (SRT/VTT), CSV/TSV, JSON/YAML/XML, Markdown/RST/AsciiDoc/Org, plain text and source code; ZIP archives recursed into. With a helper found on the machine: legacy DOC/XLS/PPT, RTF and anything else LibreOffice opens (`soffice --headless --convert-to`), images and scanned PDFs through Tesseract (`tur+eng`). Text in UTF-8, UTF-16 (BOM) or Windows-1254/1252 is converted to UTF-8. - **Across systems:** document names relative to the synced root, `/` separated, Unicode NFC (macOS hands out NFD), a warning for two names that differ only in case; CRLF → LF; key from `LLMGW_API_KEY` or `llmgw-kb login` (os.UserConfigDir, 0600 where that means something); the system's CA store, `--ca` for another. - **Gateway endpoints** (client listener, person's key): `GET /v1/kb/collections` (what this key may read/write), `GET /v1/kb/collections/{id}/manifest`, `PUT /v1/kb/collections/{id}/documents` (one document per request: name, digest, bytes, chunks with text and embedding; dims checked, chunk count and size bounded), `DELETE …/documents?name=`. Write needs a write grant or ownership on the key's chain (`CollectionsFor(…, write)`); otherwise 404. - **Administration** stays with operators: `llmgw ctl kb create|list|delete|grant|revoke|docs`, admin API `/api/v1/kb`, audit, console screen; `docs/knowledge.md`. - Trade-off accepted: vectors come from the client, so a writer could store vectors that do not match their text — only in collections they may already write. Built: `internal/kb/extract` (formats above; helpers found on PATH and in each system's install places), `internal/kb/chunk` (≈1800 characters under the heading path, a paragraph of overlap, sections kept apart once a chunk is a third full), `internal/kb/kbsync` (ledger in SQLite at os.UserConfigDir/llmgw-kb/ledger.db: documents keyed by gateway, collection and name with file, size, mtime, digest, state, reason; runs), `cmd/llmgw-kb` (login, collections, sync, status, history, formats), `internal/router/kb.go` (the four gateway endpoints, `X-Llmgw-Project` for personal keys, audit), `internal/control/adminapi/kb.go` + `llmgw ctl kb` + console Knowledge screen, `docs/knowledge.md`; CI builds the client for all six targets and releases attach it. Tested: every format from generated fixtures and a Turkish PDF (OCR path with Tesseract here), sync against a fake gateway and end to end against the router on pgvector. Tried on the PoC 2026-10-08 (control planes 140 and 150 on a dev build of this branch; their previous binary is /usr/local/bin/llmgw.pre-rag): Ubuntu's pgvector 0.6.0 installed on 149/150 and enabled; bge-m3's weights were never in the mirror (fetched from CompendiumLabs/bge-m3-gguf@1864f47f) and it is held at 1 replica; tree Demo/ik/belgeler and Demo/satis/crm with a key each; collection Demo/politikalar filled from a Mac through an SSH tunnel with ten Turkish files in nine formats (9 sent in 5 s, the .doc skipped for want of LibreOffice; a rerun sent nothing in 0 s; a changed file alone was sent). Nine Turkish questions each found their page first once search scored by similarity plus word coverage (rank fusion had got two wrong). The other team's key saw no collection and got 404 by id; with a read grant it read and could not write; revoked, nothing again. On the live store the llmgw role (not superuser, no BYPASSRLS) sees 0 chunks with no scope set and 0 with the organization alone. Left: the gateway's collection list shows the owner as team:ID, not its path; no LibreOffice on the dev machine, so the legacy Office path is untested beyond its error; the client is unsigned (Gatekeeper and SmartScreen warn); Outlook `.msg` is not read; one document over ~60 MiB of JSON (a few thousand pages) is refused rather than split. PostgreSQL tests of store, router and adminapi share LLMGW_TEST_POSTGRES and the store's TRUNCATE each test; run them with `go test -p 1` or one package at a time, or a router test loses its rows mid-run. ### F3. Retrieval on the request path (M) — done 2026-10-08 (branch `rag`) - `llmgw.knowledge: [collections]` in the body (stripped before the engine), or default collections per project/key. - Allowed set from the caller's chain (§7.9.2), cached like limits; grants add up down the tree. Personal keys and sessions need `x-llmgw-project` (or the playground's picker) naming a scope the person is a member of. - A collection not allowed answers 404, exactly as a missing one. - Hybrid search with the filter inside the query (per collection, its own partial index; pgvector 0.6 is enough — scored by similarity plus the share of the question's words held (built in F1/F2 as SearchChunks), `/v1/rerank` when a reranker is in the catalog. Top chunks into a system message with document and position; `x-llmgw-sources` lists them. - Leaks closed: answer-cache key includes the allowed set; per-organization `cache_salt` on RAG requests; injected passages under `no_store` and the guard's prompt-injection check; returned doc ids in the audit. - Done: router tests that tenant B never receives tenant A's chunks (including via cache), that a personal key without a project is refused, metrics (retrieval latency, chunks returned), docs. Built (`internal/router/rag.go`): `X-Llmgw-Knowledge` (header, as the other llmgw extensions are, not a body field), `X-Llmgw-Knowledge-K`; collections from the key's chain (kbEffective, shared with the kb endpoints), 404 for one not readable; the question embedded through the ordinary path on the caller's principal; SearchChunks; passages trimmed to the model's context; added to the one leading system message (or a new one); `x-llmgw-sources`; before guardInput; never cached; metrics. Default collections built (kb_defaults on org/team/project/key, narrowest wins, 30 s cache, `ctl kb default`, console action; never refuse); the portal chat's Documents picker (collection, default or none; sources listed under the answer; a header the page sends itself is dropped). `cache_salt` per organization on requests given passages, sent to vLLM replicas only (chosen replica's engine). Checked on the PoC 2026-10-08: vLLM 0.30.0 (cuda01) with TinyLlama-1.1B AWQ run by hand beside the 35B in the card's last 2.3 GB — a 668-token prompt reused 656 tokens under the same salt and 0 under another's, with identical text (llmgw cannot place it there: placement takes whole GPUs). Live on the PoC: a team default answers a request without the header, `none` turns it off, an explicit header with k=2 gives two sources and no more, llmgw-kb lists owners by path. Left: returned passages are not audited per request (the header and the kept request record them). ### F4. Adapter training (L, after F1–F3) — done 2026-10-08 (branch `adapters-f4`) A run-to-completion job placed by the control plane like a replica and supervised by the agent like an engine, in a Python environment the agent installs (TRL + PEFT, QLoRA). Data: Q/A pairs generated from a collection by a cluster model through `/v1/batches`, reviewed by people. Output to the mirror, into the model's `adapters`, through canary and a quality suite. Runs when the scaling schedule frees a GPU; 8–14B dense bases on one card. Never train on a collection some users of the adapter may not read. Tried 2026-10-08 on the PoC, by hand: Qwen2.5-0.5B-Instruct (fp16) with a LoRA (r=16, every linear layer, 8.8M trainable), trained beside the 35B in cuda01's last 2.3 GB (peak 2158 MiB on the card, 64 s for 138 steps, PEFT in a venv over the vLLM engine's torch). Data: 90 Turkish Q/A pairs a cluster model wrote from the nine Demo/politikalar documents, through the gateway. On nine questions it had not seen, phrased otherwise: base 0/9, 6 epochs 4/9, 4 epochs at a lower rate 2/9. Wrong answers mixed facts across documents (the meal card's amount, the office day). The same nine through retrieval: 9/9 with sources. So F4 is for format, vocabulary and tone, and facts come from F1–F3; a bigger base on a free card would do better but not change that. The training itself fits a small slot on a shared card, which the job design can rely on for small bases. Built: `ctl kb dataset` (POST /api/v1/kb/collections/{id}/dataset, NDJSON with each pair's collection); train_jobs/train_artifacts (migrations 0037, 0038 owner); internal/control/train (queue, one job per node, artifacts in 1 MiB pieces, a node that goes fails its job); internal/agent/train (the newest vLLM's python3 + `uv pip install --no-deps --target` peft 0.21.2 and accelerate 1.15.0; llama.cpp's conversion code for the installed release, untarred from its source; the card with the most free memory, if enough; runs as llmgw-engine); `ctl train start|list|show --follow|cancel|rm| download|publish`; the mirror's PUT /v1/models/{model}/{version}/{file}; publish uploads the GGUF (llama.cpp) or PEFT files (vLLM) and sets the adapter after checkAdapters. Owners: catalog.Adapter.Owner, enforced in the router (404, hidden from /v1/models); a job's owner defaults to its single collection's owner, must be able to read every collection the pairs came from, and pairs from a knowledge base are never public. Tried on the PoC: dataset of 90 pairs in 89 s; a job trained beside the 35B (peak 2214 MiB, 138 steps in about a minute) and sent back PEFT and GGUF. Left: a job dies with its agent's stream (no resume); no console screen for training yet; quality gating of a published adapter is by hand (ctl eval). --- ### F5. Large imports (M) — measured 2026-10-09, open The metallurgy corpus (2,314 files, 108,779 chunks, bge-m3 on the one GPU) went in at about 25 chunks/s, the same over a phone hotspot and the LAN. The engine's slots were the limit: llama.cpp `--parallel 4` embeds four texts per pass, while llmgw-kb sends 32 a request from 6 workers. The usage rows show 4.6 s a request on average, 1.8 s of it in the gateway's queue and 0.04 s in the guard. The GPU sat at 18–50% and the store machine at a third of its CPUs. - An embedding model's profile with many slots by default (`--parallel` 32–64, `--ctx-size` to match, `--batch-size`/`--ubatch-size` at least the longest chunk), or `ctl model serve` choosing that for a model with `capabilities.embeddings`. With the default 512, one 516-token chunk failed. - Embedding on the gateway's side of a document PUT, so the client sends text only: today an embeddings answer is ~21 KB of JSON a chunk, and the text and vector go up again (~11 KB). - The store next: HNSW inserts took half of one of pg01's two cores; at a million chunks the vectors alone outgrow its 4 GB. halfvec (pgvector ≥ 0.7) and a larger store machine. - Done: a batch of embeddings checked text by text against the context (branch `embeddings-batch`); it was refused as one over-long prompt. - Done: `query_prefix` on an embedding model, before questions only. - Qwen3-Embedding-4B on the RTX 6000 (2026-10-09): llama.cpp Q8 1,575 tok/s as first set up (32k cache, 16 slots), 2,700 with a 16k cache, FlashAttention and ubatch 1–2k; f16 no better. vLLM fp16 (`--runner pooling`) 6,200 direct, 4,700–4,900 through the gateway in the import, and its vectors match llama.cpp's (cosine 0.9995). The 103,579 chunks took 3 h. Wanted: an embedding model's GPU profile on vLLM and CPU profile on llama.cpp, which needs a source per profile (the catalog has one per model; the import swapped the entry by hand), and an admission queue limit that suits bulk imports (30 s turned some away). - Retrieval on 40 questions written from the corpus, document in the top 5: bge-m3 31/40 (Turkish questions 13/21), Qwen3-Embedding-4B 37/40 (18/21), with its query prefix 36/40 and first place 31/40 instead of 28 (MRR 0.738 → 0.789 → 0.838). Next: translating the question with the chat model, and a reranker. ## D. Facts worth knowing before starting - Admin API: `127.0.0.1:9090`, no auth, `ctlClient` in `internal/cli/ctl.go`. Metrics on `:9091` with `/prometheus/targets` beside them. - Store: `internal/store`, migrations under `internal/store/migrations/{sqlite,postgres}`; the PoC runs PostgreSQL under Patroni. - The console (`sudo llmgw`) is `internal/cli/tui_console.go` with actions in `tui_actions.go`; every operator-visible change needs a row action or a form there, and a line in `docs/console.md`. - Tests of the console use a fake admin API (`internal/cli/tui_actions_test.go`, `adminStub`); tests of serving use fakes in `internal/control/modelserve/modelserve_test.go`. - `go generate ./...` regenerates `docs/reference.md` and `internal/docgen/reference_gen.go` whenever a config struct changes; the docgen test fails otherwise.