llmgw/BACKLOG.md

Backlog

What a review of the code on 2026-10-04 (commit c2238ab) found should be done next: security fixes first, then guardrails, then features. Written for whoever implements it — a person or a model — without the review in hand. Every item names the files it touches and what "done" is. Paths are relative to the repository root. Sizes: S about a week, M two to four weeks, L more.

Conventions this codebase keeps, which every item here must keep too:

• A change is refused before anything is written (validate, then persist).
• Nothing on the serving path logs prompt text, keys or passwords.
• Each item gets unit tests beside the code (*_test.go), and the TUI (internal/cli/tui_*.go), docs/console.md and CHANGELOG.md are updated when behaviour an operator sees changes.
• go test ./..., go vet ./... and govulncheck ./... pass. No release is cut and nothing is deployed to the PoC cluster without being told to.

---

A. Security — do first

Status (2026-10-04): done and merged. A1–A9, A6b included — etcd and Patroni's API on TLS, the PoC moved in maintenance mode without a failover. Deliberately narrower than written: the admin API is a token on loopback rather than a unix socket (A5); thread ids are bounded and documented as project-wide rather than scoped per user (A8); Python engines resolve with an explicit PyTorch index and a PEP 751 lock rather than locks shipped per engine version (A8). All of A is live on the PoC.

A1. Toolchain (S, hours)

govulncheck finds 11 standard-library vulnerabilities reachable from this code at Go 1.26.3 (net/http, crypto/x509, os.Root, net/textproto). Raise the go directive in go.mod to 1.26.6 or later, rebuild, run govulncheck ./... until it reports none reachable. Add govulncheck to the Makefile's check target.

A2. Release signing is effectively off (S)

• internal/release/trusted.pub is empty, so TrustedKey() returns "" and release.go:103 skips signature verification. Ship a real public key; make a missing or invalid signature fatal. Drop the LLMGW_RELEASE_KEY environment override, or honour it only in -tags dev builds.
• internal/onboard/onboard.go:365: the node-side download checks only a same-origin checksums.txt. Verify the signature there too (simplest: have the node run llmgw itself to fetch and verify once it has any binary), and shellQuote the Releases URL, which is interpolated into single quotes unescaped.
• internal/agent/upgrade.go:44: accept only https:// release bases.
• Done when: a tampered binary or checksums file is refused by self-upgrade, node onboarding and agent upgrade, with a test for each.

A3. Agent port without TLS is anonymous and plaintext (S)

internal/cli/server.go:423 serves h2c on :7443 when agent_tls is not configured; agentsvc then accepts any node id. Refuse to start control mode without TLS unless agent_tls.insecure: true is set explicitly; when insecure, bind agent_listen to 127.0.0.1 and log a warning at startup and in llmgw doctor. Keep server init writing TLS as it does.

A4. Model profiles reach root on every node (M)

Profile args and env from a model definition are appended to the engine command unchecked (internal/agent/engines/llamacpp.go:170, vllm.go:151, sglang.go:98); internal/agent/engines/unsafeenv.go is a denylist that misses NCCL_NET_PLUGIN, NCCL_TUNER_PLUGIN, NCCL_PROFILER_PLUGIN, CUDA_INJECTION64_PATH, NVTX_INJECTION64_PATH, TRITON_* paths; engines run as root.

• Replace the env denylist with an allowlist (CUDA_/NCCL_ tuning keys minus anything ending in _PLUGIN or containing INJECTION, VLLM_*, HF_* minus path keys, OMP_*, GGML_*). Apply at catalog validation (internal/control/catalog/catalog.go, where UnsafeEnv is called) and again in the agent.
• Deny engine flags that write files or run code: --trust-remote-code, --log-file, --slot-save-path, --chat-template with a path, --host, --api-key, --port; vLLM/SGLang equivalents. Allowlist is better than denylist if the flag set is enumerable per engine.
• Run engines as an unprivileged user with the video/render groups (internal/agent/supervisor, systemd unit in internal/cli/serverinit_cluster.go:52). Weights cache must be readable by it.
• Done when: a profile with --trust-remote-code or NCCL_NET_PLUGIN is refused at ctl model add and at the agent; ps shows engines not as root.

A5. Admin API trusts any local process (M)

internal/cli/server.go:357 serves the admin API and pprof on 127.0.0.1:9090 with no authentication; adminapi.actorOf (internal/control/adminapi/api.go:1613) takes the actor from the client-sent X-Llmgw-Actor header.

• Serve the admin API on a unix socket (/run/llmgw/admin.sock, 0660, group llmgw-admin) by default, keeping TCP as an option that requires a bearer token read from a root-only file (/etc/llmgw/secrets/admin-token).
• Take the actor from SO_PEERCRED (uid → username) on the socket, or from the token's name; ignore the header.
• Refuse admin_listen on a non-loopback address unless admin_insecure: true.
• Put pprof behind its own debug.pprof: true setting, off by default.
• ctlClient (internal/cli/ctl.go), the console and node add must work over the socket; sudo llmgw already runs as root.

A6. Patroni REST and etcd are unauthenticated and plaintext (M)

internal/pgha/config.go:99 writes a restapi: with no authentication and :54 etcd over http:// with no client auth; hba.go:56 PATCHes Patroni's config unauthenticated. Anyone reaching :8008 or :2379 owns the store.

• Generate a Patroni REST username/password at setup, store it in /etc/llmgw/secrets/, write restapi.authentication and restapi.certfile/cafile from the cluster CA; send it from internal/pgha/hba.go and status.go.
• etcd: client and peer TLS from the cluster CA with --client-cert-auth.
• Document, and have server init write, nftables rules limiting 2379/2380/8008/5432 to member addresses (see A9).
• Done when: an unauthenticated curl :8008/config is refused and the store status in the console still works.

A7. PostgreSQL traffic is plaintext (S)

internal/pgha/config.go:168 and internal/cli/serverinit_postgres.go:330 use sslmode=disable; HBA lines are host. Enable server TLS in Patroni's postgresql parameters with a certificate from the cluster CA, write hostssl lines, use sslmode=verify-full&sslrootcert=/etc/llmgw/ca.pem. Existing clusters: llmgw server must keep working against a store that has not been migrated, with a warning.

A8. Smaller fixes, each S or less

• Capacity fallback skips the allowlist. internal/router/router.go:700 builds [canonical]+entry.Fallbacks without permits. Filter the chain with r.permits(p, model); price against the model that served.
• Suspended users keep working API keys. store.(*DB).Resolve (internal/store/memberships.go) never joins users.state; Authenticator.Invalidate() is never called. Reject keys whose owner is not active; call Invalidate on revoke, suspend, limit change.
• Join codes never expire server-side. internal/join/join.go:103 checks expiry only in Parse; admit (:212) does not. Check it there; bind the listener to the chosen address, not :7444 on all interfaces; use the peer's address rather than the claimed hello.Addr, and show the joiner's address for the operator to confirm before secrets are sent.
• Leftover sudoers grant. internal/onboard/gossh.go:248 writes NOPASSWD:ALL and removes it only in Close() (:271). Write the grant with a self-removal (at/systemd-run timer, 30 min) so a crash leaves nothing; wipe password buffers after use. In ssh.go:39 pass -- before the target.
• Auth cache and pre-auth DoS. internal/auth/auth.go:127: unbounded map, one DB query per bogus well-formed key. Bounded LRU, negative cache with short TTL, per-IP token bucket before the DB.
• HTTP server hardening. server.go:351-360, the h2c agent server, both mirror servers: set ReadHeaderTimeout (exists), IdleTimeout, MaxHeaderBytes, a body-read deadline, netutil.LimitListener; connect WithReadMaxBytes on the agent stream.
• Login/signup abuse. Default signup_open: false; per-IP token buckets on /login and /signup; bounded wait for argon2 slots; CSRF token on the login form; the per-address throttle must not lock the real owner out (use IP+address pairs).
• Trusted proxies. internal/web/account.go:557, landing.go:60 honour X-Forwarded-Proto/Host from anyone; add router.trusted_proxies and only then trust them; Cache-Control: no-store on /setup/*.
• Certificate revocation. DELETE /api/v1/nodes/{id} only drops inventory; add a revoked-CN/serial list in the store checked in agentsvc Connect and mirror onlyClients; serverinit_cluster always passes --api-client.
• Engine exposure. Engines bind 0.0.0.0 (llamacpp.go:102, vllm.go:109): bind the management address. SGLang receives its key on the command line (sglang.go:102): pass it by env or file.
• Backups. internal/cli/backup.go: encrypt archives to an age recipient from backup.recipient in server.yaml; keep the directory 0700.
• Python dependency confusion. internal/agent/provision/python.go:121 uses --extra-index-url with unsafe-best-match. Pin torch to the PyTorch index explicitly (uv [[index]] explicit = true), first-index, hash-pinned lockfiles per engine version.
• Control-plane SSRF. internal/sourcecheck/sourcecheck.go:39 and gguf.FromURL use the default client; reuse the mirror's fetchClient(false) (private-address filter) and refuse a proxy unless configured; block NAT64/6to4-embedded private addresses in internal/mirror/safefetch.go.
• Low: agentsvc/service.go:430 indexes n.GPUs[0] without a length check (CPU-only node panics the handler); thread ids are client-chosen with no length cap and shared per project (router/threads.go) — cap length, document, offer per-user scoping; /v1/models lists models the key may not use (handlers.go:799); sessions have no idle timeout and PurgeSessions/PurgeClaims are never called — add a janitor; add a CSP header and set htmx.config.allowEval=false; cap GGUF array nesting depth in internal/gguf/gguf.go; mirror .source sidecars are served by the public file server — exclude dotfiles from :8081; prune modelserve.jobs and the mirror running map; KaTeX → 0.16.21+.

A9. Firewall (S)

No machine in a default install has a firewall. Have server init, node add and server add write an nftables ruleset: ssh open; :8080 open; :7443 and :9091 open to the cluster's networks; :9100 to the monitoring host; :2379/2380/8008/5432 to store members and control planes; mirror :8081 to the cluster, :8082 to control planes; :9090 Prometheus and :3000 Grafana to operators. llmgw doctor reports a missing ruleset. Make it skippable with --no-firewall.

---

B. Guardrails (M–L, build in this order)

Status (2026-10-04, branch guardrails-b): B1–B4 done, and tried live on the PoC. That found four bugs, all fixed with tests:

• a shadow default merged with an attached policy of unwritten mode left it in shadow;
• a streamed word held back at one release was dropped at the next;
• llama.cpp continues a conversation ending on the answer, so Llama Guard's verdict on answers is now asked with its own prompt as a completion;
• a judge model was named in a client's "no endpoint" error.

The PoC runs observe (shadow) for everyone, and Llama Guard 3 1B (llama-guard-3-1b, CPU, on cuda01) as a judge, which blocks a harmful request in ~0.7s and a harmful answer in ~1.4s there. Still left: see each item.

Today nothing on the request path reads, classifies, redacts or filters message text in either direction (confirmed; the only "moderation" hit is a comment in internal/router/budget_fallback.go:20). Existing controls are about who, how much and how fast: allowlists (router/handlers.go:450), RPM/TPM/budget (internal/accounting), context precheck (router/context_limit.go), admission bands (router/admission), timeouts (router/proxy/proxy.go:26-57). Gaps to close on the way: no default output cap when the client omits max_tokens; catalog.Capabilities is advertised but never enforced; tool results are trusted input.

B1. Hooks in the proxy and router (S)

Done (branch guardrails-b): internal/guard holds the interface; proxy Filter/StreamFilter; the input hook runs after loadThread (the simpler choice); error codes, headers, Responses incomplete, usage columns guard_*. Moved to B2: holding delta.tool_calls until they are whole and reading their arguments — that is the tool policy's filter, not the proxy's.

• internal/router/proxy/proxy.go: add optional InputRewrite func(body []byte) ([]byte, Verdict) on the request and two output hooks: Filter for relayBuffered (between ReadAll and WriteHeader) and StreamFilter{OnEvent(data) (emit []string, stop bool); OnEnd()} for relayStream, called where each whole SSE event is assembled (eventBuffer, before io.WriteString at :343). Nil hooks cost nothing.
• internal/router/handlers.go: call the input policy after the allowlist and limits and before admission (around :395); the principal, scopes and the model that will serve are known there. Thread history is spliced later (:416): scan stored turns when written, or move the hook after loadThread (decide; the second is simpler).
• Blocking conventions: input → 400 invalid_request_error with code: "content_policy_violation" (today writeError always sends code: null); output non-stream → 200 with finish_reason: "content_filter" and empty/redacted content; output stream → a final chunk {"choices":[{"index":0,"delta":{},"finish_reason":"content_filter"}]}, then data: [DONE], then cancel upstream. Headers x-llmgw-guard: blocked|redacted|flagged and x-llmgw-guard-policy set before the proxy writes headers.
• internal/router/responses.go: map content_filter to response.incomplete with incomplete_details.reason: "content_filter" (today it always emits response.completed, :525).
• Tool calls: hold delta.tool_calls until finish_reason: "tool_calls", then validate; replyDelta (reply.go:33) must also read tool_calls[].function.arguments.
• Accounting: a blocked input is recorded like RecordRefused (nothing charged); a blocked output is charged (the GPU was used). store.Usage gains GuardAction, GuardPolicy, GuardCategory, GuardMs.

B2. Package internal/guard (M)

Mostly done (branch guardrails-b): policies in server.yaml with attachments in the store (ctl guard attach), narrowed by the limits' rule; guard/detect (validated PII, secrets, injection through normalisation and decoding); redaction with consistent placeholders before prefix routing; tool allow/deny/strip, call count and argument schemas; capabilities and structured output (opt-in); default max_tokens; pinned system prompt and dropping the client's; no_store (threads, notes, prefix tree, external models); shadow mode. Left: a guard_events table — the usage row's guard_* columns carry action/policy/category/time, not rule and stage; language identification; the re-identification map that restores redacted values in answers.

• Policy, Effective resolution over the tenancy tree with the same narrowing rule as limits (internal/store/sqlite.go:383-416 tighten): deny sets union, allow sets intersect, mode takes the strictest of enforce > shadow > off, fail takes closed > open, no_store at any level wins, thresholds take the minimum.
• Store: guard_policies(scope, scope_id, policy json) and guard_events(request_id, scopes, policy, rule, category, action, stage, score, judge_ms, shadow) — ids, categories, offsets and lengths only, never text. Migrations for sqlite and postgres. Pruned with usage_retention.
• guard/detect: regex/keyword sets; PII with validators — email, phone, IBAN (mod-97), card (Luhn), TCKN (checksum), IP, secrets (sk-, AWS, JWT, PEM); normaliser (NFKC, strip zero-width/bidi, homoglyph fold, base64/hex/URL decode to depth 2); injection heuristics (role-switch phrases, fake chat-template tokens, long base64, invisible Unicode) — default action flag; language id.
• Redaction rewrites messages[].content in the upstream body with placeholders ([EMAIL_1]) via the same map[string]json.RawMessage round-trip withModel/spliceThread use; an optional re-identification map lives only in request memory and restores originals in the output. Redact before prefix routing so PromptPrefix hashes the redacted text.
• guard/toolpolicy: tool name allow/deny (strip or 400), JSON-Schema validation of assembled arguments against the declared parameters, argument regex rules, max_calls_per_turn. Enforce Capabilities.Tools/Vision here: a tools array or image parts to a model without the capability → 400.
• guard/structured: validate response_format.json_schema on the final text → content_filter, or flag in shadow.
• Output constraints: inject a default max_tokens when absent (from the policy or MaxOutput), language check, structured check.
• System-prompt pinning per project: prepend before checkContextFits and before spliceThread; optionally drop client system/developer roles; exclude the pin from stored thread turns; apply to Responses instructions in toChat.
• no_store: loadThread and PutNote refuse; captureBytes() returns 0; skip the prefix-tree observe (router.go:840); disable External fallbacks for the scope (filter in dispatch and budgetFallback).

B3. Judge model on the cluster (M)

Done (branch guardrails-b): role: guard + guard.format; hidden and refused to clients and fallbacks; critical by default, never on demand; Router.Judge past admission/accounting/guardrails; llamaguard3, shieldgemma and json parsers; 5-minute cache; latency budget then fail open/closed; streamed answers read every 600 characters, or held with buffer_all. Simpler than written: the input judge runs after admission, not alongside it, and shadow mode judges synchronously within the budget rather than after the response. Not enforced: two replicas on different nodes — documented, left to scaling.min.

• catalog.Model.Role: "guard" plus Guard{Format: llamaguard3 | shieldgemma | json, Categories}. Guard models are hidden from /v1/models, the chat picker and allowlists; placed like any model, class critical, scaling.min ≥ 1, at least 2 replicas on different nodes.
• guard/judge: calls the guard model through the router's own dispatch with an internal principal and its own admission band (never the tenant's queue); parsers per format (Llama Guard safe/unsafe\nS1,S10; ShieldGemma yes/no probability → per-tenant thresholds; generic JSON verdict); verdict cache keyed by hash of normalised new turns (5 min); latency_budget (default 250 ms) then fail open/closed; input judge runs in parallel with admission queueing.
• Streaming strategy: default sentence-buffered scanning (regex/PII per sentence or ~200 chars before release; judge every 400–800 chars on the cumulative tail; final judge at end); buffer_all for fail: closed scopes. Document that released text cannot be retracted.
• Shadow mode runs everything asynchronously after the response (context.WithoutCancel, like saveThread) and only records.

B4. Config, metrics, UI (S)

Done (branch guardrails-b): library in server.yaml, attachments via ctl guard, the console's Guardrails… with "applies from" marks, and the portal's /admin/guard; the metrics and alerts below (judge latency labelled by model only at the judge; the check latency by stage); ctl request shows the verdict; the chat page says "stopped by policy"; the profile lists effective policies; docs/guardrails.md and data.md written; the access log carries action, stage and policy only. Attaching from the portal is not done — the page shows; the console and ctl change.

• Gateway-level policy library in server.yaml under guard: (see the YAML in the review, reproduced in docs/guardrails.md to be written); attachments to scopes in the store via llmgw keys guard, the console ("Guardrails…" beside "Limits…" in internal/cli/tui_people.go, with "inherited from team:X" badges) and the portal.
• Metrics in router/metrics.go style: llmgw_guard_checks_total{stage,policy,action,category,mode}, llmgw_guard_latency_seconds{stage,judge_model}, llmgw_guard_judge_errors_total{reason}, llmgw_guard_fail_open_total, llmgw_guard_stream_stops_total, llmgw_guard_redactions_total{kind}, llmgw_tenant_guard_blocks_total{tenant}. Alerts: fail-open > 1% for 5m, judge p95 over budget, judge has no ready replicas, per-tenant block rate

  3× its 7-day baseline, any secret detected in output.

• Portal /admin/guard: violations by tenant/category/policy over time, shadow vs enforce, judge health, recent events by request id; ctl request <id> shows the verdict; chat page renders content_filter as "stopped by policy X"; /me lists effective policies.
• Update docs/data.md: the judge sees all text but in-cluster; External fallbacks send prompts outside the cluster (the current "nothing leaves the cluster" is wrong); no_store semantics.
• Never log: prompt/completion text, PII values, the re-identification map, judge inputs or rationale that quotes input. Check accessLog's err field and span errors.

---

C. Features, ranked by value × feasibility

1. Done (branch features-c): audit_events, written by the admin API middleware and the /admin pages (with each change's result), ctl audit, the console's Audit view; staff roles viewer / operator / billing / admin. Left: before/after values (the request's fields are recorded, not the state they replaced); a tenant-admin role that changes its own organization (managers still only view it); hiding the forms a viewer cannot submit. Audit log and roles (M). Today "audit" is slog lines (internal/web/admin_users.go:403-480) and roles are the IsAdmin flag plus an org "manager" who can view. Add audit_events (who, what, before/after, source) written by the adminAction/tenancyAction wrappers and by every admin-API mutation; roles viewer / operator / tenant-admin / billing; ctl audit. Depends on A5 for a real actor.
2. Done (branch features-c): OIDC sign-in (auth.oidc, PKCE, verified email, roles and memberships from groups, no_passwords) and SCIM 2.0 (/scim/v2 Users and Groups). The sign-in cookie's key comes from the store (2026-10-10), so a callback may reach either control plane behind a balancer. OIDC/SSO login (M), then SCIM (M). internal/account/service.go, internal/web/account.go; map IdP groups to org/team/admin.
3. Done (branch features-c): key lifetime limits (gateway, org, team, project), keys rotate --overlap, kinds and expiry in keys list. Service accounts and key lifecycle (S). Per-org maximum key TTL, keys rotate with overlap, last-used shown, project-owned service keys listed apart. internal/cli/keys.go, internal/store/keys.go, internal/auth/auth.go.
4. Done (branch features-c): CSV export from the rollups (API, ctl, console). Left: Parquet, and a scheduled push to S3/SFTP. Usage export (S). CSV/Parquet by org/team/project/model/period from the rollups (internal/store/usageseries.go), GET /api/v1/usage/export, optional scheduled push to S3/SFTP.
5. Done (branch features-c): notify.webhooks (model parked, node lost, budget spent, update available; signed JSON or Slack; retries, hourly de-duplication), ctl notify test, Alertmanager receiver templates. Left: server init and the console do not yet ask for a receiver — the templates are filled in by hand. Events are claimed in the store before they are sent (2026-10-10), so two control planes send each once. Alerting out of the box (S). deploy/monitoring/alertmanager.yml ships receiver: none; add Slack/email/webhook/PagerDuty receiver templates driven by server init or the console; llmgw-native webhooks for model parked, budget spent, update available (from the reconciler and modelserve).
6. Done (branch features-c2): metadata, x-llmgw-tags and user on the usage record (migration 0023), ctl request shows them, router.tag_labels metrics with 100 values per tag. Left: export and ctl usage by tag or end user (the hourly and daily sums do not carry tags; it would read the request records, within their retention). Request metadata and tags (S). Accept metadata/user from the body or x-llmgw-tags; store on the usage row; tenant metric labels behind an allowlist. router/handlers.go parse, accounting/recorder.go.
7. Done (branch features-c2): ctl model canary/promote/canaries, session-sticky split with x-llmgw-variant pins, rollback on error rate or TTFT, canary_rolled_back notification, console menu. Left: A/B between two different models (the split is between a model and its newer weights); the comparison is per control plane and lives in memory, so a restart begins it again; a key-level pin. Canary and weighted A/B between model revisions (M). control/modelserve/updates.go replace swaps every replica; add weighted routing across two revisions/aliases (scheduler filter, alias resolution in router/router.go:569), a header/key pin, automatic rollback on error-rate or TTFT regression from per-replica metrics.
8. Done (branch features-c2): /v1/files and /v1/batches (OpenAI's shape), worker under a store lease, lines kept as served, lowest band, usage tagged batch=<id>, output and error files with retention, no_store refused. Left: the Responses API (/v1/responses) as a batch endpoint; the console and ctl do not list batches (an operator sees them through usage tags); a file is one row in the store, so a store backup carries batch files with it. Batch API and Files (M–L). /v1/files, /v1/batches, JSONL storage, a worker feeding the bronze band, accounting. New internal/router/batch.
9. Done (branch features-c2): keys requests (retention, sample), kept after redaction, ctl request --body/--replay, console, pruning. Left: a streamed answer is kept as its text, not its chunks; replay is one request at a time (a set of them against a model is item 21's eval). Request/response logging and replay (M). Opt-in per project with retention like threads; sampled or full; redaction hooks from B; ctl request <id> --body; replay against another model.
10. Done (branch features-c2): Responses text.format, input_image, previous_response_id (thread in the id); rerank and score passthrough; transcription and translation as multipart; /v1/moderations by the judge. Left: audio is charged no tokens (an engine's duration usage is not read); input_image by file_id; moderation scores are 0 or 1, as a judge says yes or no. OpenAI compatibility gaps (S–M each). Responses: text.format → response_format, input_image, previous_response_id via threads (router/responses.go). Rerank/score passthrough for vLLM, /v1/audio/transcriptions for a whisper engine, /v1/moderations mapped to the guard model.
11. Done (branch features-c2): /v1/messages and count_tokens, streamed and not, tools, images, x-api-key, Anthropic error shape. Left: extended thinking (a reasoning model's thoughts are not mapped to thinking blocks); prompt caching; the account page's "Connect your editor" does not offer Claude Code yet. Anthropic /v1/messages (M). Translate like responses.go.
12. Done (branch features-c2): server restore (dry run, version check, decryption, modes and owners, SQLite and PostgreSQL), backup_copy_to (directory or ssh), the drill in docs/install.md. Left: restoring a replicated store's members (restore into the primary; Patroni replicates it); the console's Back up now does not offer restore; an ssh copy holds the archive in memory while it sends it. server restore (S). Restore is manual shell today (docs/install.md:1187); add the command with version check, dry run, off-host copy target; a documented DR drill.
13. Done (branch features-c2): TPOT, inter-token and tokens/s histograms by model, pool and tenant; slo pressure in admission's saturation; llmgw_model_slo_seconds and llmgw-slo alerts. Left: the p99 is per control plane, from its own requests; an SLO per band or per tenant; placement does not yet add replicas for an SLO (item 14). SLO-based admission and token-level metrics (S–M). TPOT and inter-token histograms, output tokens/s per model and tenant in router/metrics.go; use catalog.SLO in admission/saturation.go.
14. Done (branch features-c2): scaling.schedule (days, from, to, min, max; past midnight), scaling.prewarm, control.timezone; queue-depth scaling was there already. Left: ctl scale sets min and max only (a schedule is set in the definition); scaling for an SLO rather than for queue depth. Scheduled and warm-pool autoscaling (S–M). Time-of-day min/max, pre-warm, queue-depth scaling in catalog.Scaling, control/reconciler/demand.go, headroom.go.
15. Done (branch features-c2): env: and vault: references wherever a secret file is named (store address, api_key_file, OIDC, SCIM, webhooks), Vault KV with token or AppRole, secrets.master_key sealing cluster_secrets (Vault transit or a local key). Left: the certificate authority's key is still a file in /etc/llmgw (signing in Vault's PKI engine instead is its own change); rotating the master key; backup_copy_key and admin_token_file are files only. Secrets management (S–M). Vault/KMS for api_key_file, the DSN, the CA key, cluster_secrets (store/secrets.go, store/seal.go); envelope-encrypt them in backups (with A8 backups).
16. Done (branch features-c2): users forget (pseudonymised usage and sums, kept requests, files, batches, limits and audit names), keys residency --in-cluster (asked-for, fallback and canary). Left: an export of everything held about a person (article 15); residency by region between several clusters; backups keep what was forgotten until they age out. GDPR and residency (S). "Forget user" anonymises user_id in usage and rollups and erases threads/memory (store/hierarchy.go:241 leaves it today); per-project "never route to an External provider".
17. Done (branch features-c2): exact-match answer cache per project (keys cache --ttl/--any/--off, console Cache answers…, projects columns, 30-second setting cache), reproducible answers only unless --any, LRU in memory bounded by router.cache.max_bytes and max_entry_bytes, streamed answers kept and replayed as chunks, x-llmgw-cache and Cache-Control: no-cache, guardrails before a hit and the output policy on it, no_store and threads bypass, a hit recorded as cache_hit with no tokens and no cost (migration 0027), metrics, docs. Left: semantic (embedding-similarity) caching through the embeddings models; a cache shared across control planes (each process keeps its own, deliberately: a shared one would write every answer to the store); a hit still waits its turn in admission, because the guardrails judge the request after admission and the key is the request they let through; logprobs in a streamed answer leave it unkept; users forget does not reach into the cache (answers age out by the TTL); no listing or flushing of what is cached (--off stops it being served at once, the memory is freed by TTL and LRU). Response/semantic cache (M). Exact-match first, per project, opt-in, TTL; semantic via the embeddings models later. Before dispatch in router/handlers.go.
18. Done (branch features-c2): offline bundles — llmgw bundle create (releases with signed checksums, every llama.cpp build per arch/variant with GitHub's checksums, starter/judge and named Hugging Face model files pinned and checked; directory or tar; manifest sha256), bundle import (checked, all-or-nothing, into the mirror under _llmgw/), bundle trust, bundle show --verify; offline.mirror/offline.checksums in server.yaml, with nodes installing llama.cpp from the mirror against checksums the control plane sends over the agent stream (proto EngineWant.offline_*), releases defaulting to the mirror for node add, ctl upgrade and agent upgrades, update checks off; server init --bundle and the setup form's bundle fields; docs/data.md's offline claim corrected; install.md "Offline sites". A Helm chart for the control plane (deploy/helm/llmgw, one replica, SQLite PVC or external PostgreSQL, Secrets, Services, Ingress, ServiceMonitor) with deploy/container/Dockerfile and docs/kubernetes.md; GPU nodes stay outside Kubernetes by design. Left: vLLM/SGLang offline (a wheelhouse of several GB per CUDA build and a uv --find-links install path); the NVML release binary and uv in bundles; the setup wizard's preflight still fails a machine without internet; join's printed curl from the mirror checks nothing; models in a bundle are single files (no whole- repository safetensors sets yet); the chart runs one replica (a standby is never ready, so HA in Kubernetes needs a different readiness design), has not been installed on a real cluster, and ships no agent DaemonSet; no published container image. Kubernetes/Helm and an air-gapped bundle (M). deploy/, agent/provision; docs/data.md claims offline operation but install assumes the internet.
19. Done (branch features-c2): adminapi.Routes and internal/docgen/openapi generating docs/admin-api.openapi.json (3.1, bearer security, every route, query parameters, schemas from the Go types with their comments), served embedded at GET /api/v1/openapi.json, kept true by a route-table test and a staleness test; validated with openapi-spec-validator and openapi-generator-cli. GET /api/v1/models/{name}. docs/admin-api.md. terraform-provider- llmgw/ (terraform-plugin-framework): llmgw_model, llmgw_quota, llmgw_guard_attachment, data sources llmgw_models and llmgw_nodes; client tests against a fake admin API and the OpenAPI document; acceptance tests with Terraform and OpenTofu against the fake; one apply, empty plan, in-place change and destroy against a real llmgw server run by hand. Organizations, teams, projects, keys and memberships are in the admin API since 2026-10-08 (/api/v1/orgs, teams, projects, keys, users/{id}/memberships) and in the provider (llmgw_org, llmgw_team, llmgw_project, llmgw_key, llmgw_membership); accounts are still made by sign-up, invitation or OIDC/SCIM. Left: a model's definition beyond scaling, class, owner and adapters is changed by replacing it (the API has no edit, and resources is not used yet); canary, slo, scaling.schedule and prewarm, guard role and origin are not in llmgw_model; models cannot be imported; the provider is not published to a registry and has no release build; the acceptance tests are not run in CI (they need Terraform); the document's schemas mark nothing required and give no enums; catalog.SLO's fields have no JSON tags, so they are TTFTp99 and TPOTp99 on the wire. OpenAPI for the admin API, Terraform provider (M). Generate from adminapi/api.go routes via docgen.
20. Done (branch features-c2): spans for the reconcile pass (begun only when a pass does something), placement with its answer or rejections, scale-up/down, drain, stop, eviction, preemption, reaping, restarts and lost nodes; a replica start span from placement to serving whose context travels to the agent in the start command (ControlMessage.trace), where weights fetch, engine launch, rendezvous and readiness wait continue the trace; stops, drains, adoption and engine installs; modelserve jobs as a trace with a span per step linked to the replica's start; an agent tracing block; docs in install.md's "Tracing". Left: the router's span for a request does not link to the replica it was sent to's start; heartbeats, fabric scans, logs and agent upgrades are not traced; the supervisor has no restart span because it never restarts an engine itself (a restart is the reconciler's restart replica and reaches a node as a stop and a start); canary and promote jobs, which landed on features-c2 after this was branched, are traced only through the shared step recording (as <kind> model), unchecked by a test. Tracing beyond the router (S). Spans for reconciler, placement, agent start (obs/trace.go, control/reconciler, agent/supervisor).
21. Done (branch features-c2): internal/bench/eval (suites in YAML or JSON lines; exact, contains, regex, number, a JSON Schema subset, and a judge model with a rubric; weights and a threshold), run on the control plane through Router.Replay; llmgw ctl eval MODEL --suite FILE|NAME, --history, --run, --suites; runs and their cases kept (migration 0028, eval_runs and eval_cases); model update/promote --eval SUITE --min-score N, a model's standing gate (ctl model gate, eval: in the catalog) — a promote asks the canary first, an update tries the new weights as a 0% canary and promotes or removes it; the console's model menu; arithmetic and json-format in deploy/eval/, carried in the binary. Left: the suite of a model's standing gate is read by name on the control plane at the time of the update, so two control planes need the same control.eval_dir; a gated update needs room for one more replica while it runs (there is no in-place gate, and a declared model in server.yaml cannot be updated at all); no comparison against the old weights' score (the gate is an absolute threshold, not "no worse than before"); a run is not stopped once started (Esc in the console only stops watching it), and the in-memory progress of a run lives on the control plane that started it; runs are never pruned; the JSON Schema support is a subset (no $ref, oneOf, not, uniqueItems); judge cases go through the same concurrency as the model's, and a judge's answers are not kept, only its score and reason; a gateway without a control plane cannot run suites (there is no mode that asks through the client API with a key). Eval harness (M). llmgw eval extending internal/bench with quality suites gating model update.
22. Done (branch features-c2): MIG instances discovered on both paths (nvidia-smi -L + -q -x; NVML through an interface with a faithful fake) and reported instead of their card, each with its MIG UUID, own memory and class <card class>-mig-<profile>; device numbers 1000 + 100 × card + instance, shown as card/migN; placement on MIG classes; tp/pp/gpus/nodes above 1 on a MIG class refused in catalog validation and again in placement; CUDA_VISIBLE_DEVICES=MIG-<uuid> for vLLM and llama.cpp (no Vulkan device variable on MIG); per-instance memory and processes in heartbeats and the GPU check, card sensors under each instance, utilisation absent rather than zero; doctor MIG findings (mig -lgi/-lci parsed); ctl nodes, console, ctl status, ctl model plan/serve by class; docs/install.md MIG section. Written and tested against fakes only — untested on real MIG hardware. Left: run it on an A100/H100 in MIG mode (check -L/-q -x/mig -lgi/-lci and NVML MIG device names against the parsers, and that CUDA_VISIBLE_DEVICES=MIG-… reaches vLLM's workers through Ray); instances changed while the agent runs are only seen when it restarts (no rescan); llmgw does not create or reshape instances (no mig-parted integration); quotas count an instance as one GPU, the same as a whole card; SGLang on MIG not checked; device numbers assume fewer than 100 instances per card and that instance numbers stay put while shards run on them. MIG (M–L). Treat a MIG slice as a GPU class in agent/discovery/gpu_nvml.go and control/placement.
23. Done (branch features-c2): engine trtllm (internal/agent/engines/trtllm.go): trtllm-serve through a launcher of llmgw's that demands the replica's key (trtllm-serve has none) and writes the options file that turns on /prometheus/metrics and the iteration statistics; the scrape contract for it (scrape.TRTLLMMapping, with a per-engine metrics path); found, not installed — engines.trtllm.python in agent.yaml, probed from the package's metadata; catalog validation (GPU class, one node, no GGUF source, the flags a profile may not give); ctl engine-check trtllm, which asks trtllm-serve serve --help through its module. Ollama/GGUF import: mirror import-ollama (manifest parsing, blob digest check, projector beside the model, all or nothing, a model definition printed), mirror import for any local file, and the console's I on the Mirror view. Left: TensorRT-LLM has not run on real hardware — the first node with one should run engine-check and serve a model before anyone relies on it; no installer for it; no multi-node TensorRT-LLM replica (it needs an MPI or Slurm launch from one node); Triton Inference Server as an engine is not done; trtllm's queue gauges are set per iteration, so an idle engine shows its last batch until the next request; --num_serve_frontends is refused because extra front ends would not demand the key. The Ollama import runs on the mirror machine only — the mirror's API takes no files, so an Ollama elsewhere means copying the blob or running the import there; Ollama models kept as separate tensors or as several weight layers, and its LoRA adapters, are refused or left with a sentence; files an import run with sudo writes are root's, as mirror add's are. More engines (M each). TensorRT-LLM/Triton adapter; Ollama/GGUF import. agent/engines/engine.go interface and the metric contract in router/scrape.
24. Done (branch features-c2), all six parts. Lower: precise KV-index routing (plan phase 6), LoRA affinity for vLLM, multi-cluster federation via External providers, facility-aware placement (docs/facility-aware-placement.md), portal i18n, SDK/docs portal. What was built, and what is left:
    • 24 (portal i18n): built — English/Turkish catalogue with fallback, per- language templates, cookie/Accept-Language/landing.language choice and a footer switcher, Turkish numbers and dates, every template and handler message translated, tests that no template text stays English; left — errors built with fmt.Errorf in other packages (store, catalog, mirror, adminapi) still English on Turkish pages, percent sign placement is mixed (%50 on limits, 93% on cluster pages), RFC3339 footer timestamps and plain-text http.Error bodies untranslated, the /docs documents themselves English, a per-person language column (cookie only today).
    • 24 (SDK/docs portal): built — /docs renders the embedded manual (goldmark, GFM, heading anchors, links rewritten, no script; public, landing.docs_sign_in to require a session), and docs/clients.md shows the official OpenAI/Anthropic SDKs against llmgw. Left: the console's generated "Commands" page is not in the portal (it is built from the cobra tree, which internal/web cannot see); no search across documents.
    • [~] 24. Precise KV index (tier 2) — built: vLLM KV events via the agent (loopback ZMQ, pure-Go zmq4, msgpack in both of vLLM's encodings) forwarded as net block-hash changes over the agent stream; router index, tokenize via the replica's /tokenize, sha256_cbor block-hash chain checked against cbor2/hashlib golden values; scorer by leading- block share with tier 1 fallback; opt-in kv_events per profile; metrics; docs. Left: validation against a real vLLM (event shape, hash agreement, /tokenize matching the served prompt, cost of tokenizing); vLLM's replay endpoint instead of resetting on a gap; data-parallel ranks (one publisher each, ports offset); hybrid models' cache groups and CPU-offloaded blocks; LoRA/multimodal/cache_salt requests (extra keys); a router-only deployment (no agents) subscribing directly; standby control planes have no index until they lead; a per-replica fallback when only some candidates have an index; caching tokenizations of repeated prompts.
    • 24 LoRA affinity (branch features-c2): built max_loaded_adapters (catalog), vLLM adapter support at start and on demand (engines/lora.go, vllm.go), the agent linking fetched adapters as llmgw-lora/<name> in the engines' working directory, the router's adapter tracker (router/lora.go: loads/unloads with the engine key, LRU eviction never of an adapter in use, one load per adapter per replica, /v1/models sync, 404 reload) and the adapter-affinity scheduler filter in the chat, batch and long-context profiles, with metrics and docs; tested against a fake of vLLM's HTTP interface. Left: never run against a real vLLM — in particular that vLLM resolves the relative llmgw-lora/<name> path against its working directory on every node (multi-node too) and that unloading an adapter mid-request is harmless; the agent still fetches every adapter at replica start, so a new adapter needs a restart (fetching on first load would need an agent command and a proto change); load timeout and resync interval are constants, not config; SGLang's runtime LoRA API is not driven; no per-replica "adapters held" view in the console or admin API; the debug profile has no affinity.
    • 24. Multi-cluster federation — built: peers via ctl federation kept in the store, their models as PEER/MODEL synced from /v1/models (withdrawn-while-referenced), /ready health with latency and routing withdrawal, ctl model fallbacks for overflow, hop header loop refusal, residency/no_store hold, optional attribution tags, console view, metrics, docs. Left: server.yaml-declared models cannot name a peer model as a fallback (config catalog is validated before peers are read); per-peer CA / mTLS for peers behind a private CA; transitive federation (A→B→C models) deliberately not offered; latency is /ready RTT only (no per-request peer TTFT used in routing or weighting); the peer's adapters are not imported (listed as aliases); no console form to add/remove peers (CLI only); key file must exist on every control plane.
    • 24 (facility) — built: location labels (ctl label, agent.yaml), spreading across racks/PDUs, keep_within, held domains with drain (ctl facility hold/release, kept in the store), power caps from facility.budgets against declared/derived/measured draw, shown in explain-placement, ctl nodes, ctl facility, the console and the API. Left: temperature conditions with hysteresis and their enforcement (steps 3–4), a Prometheus/DCIM source, power headroom as a rack-levelling score, spreading multi-node replicas across racks, llmgw_domain_* metrics and alerts (near cap, over cap, stale data), shedding and tokens per watt (step 5), and a drained node's state surviving a restart.

---

E. Every setting takes effect when it is changed (2026-10-05)

Asked for after a site visit: an upgrade left server.yaml without guardrails, a node on another subnet stayed lost until firewall_networks was edited by hand, and nothing in server.yaml, agent.yaml or a generated unit applies until somebody restarts the right service on every machine. The rule from here on: a file holds only what one machine needs to start; everything else is the cluster's, kept in the store, changed through ctl, the console or the web, audited, and in force on every control plane within seconds.

E1. Cluster settings in the store (L) — done 2026-10-05

Live on the PoC (140, 150): imported as version 1; a guard change applied on both without a restart; a router change restarted the standby at once and the leader 30s later with /health answering throughout; rollback did the same.

• Table cluster_settings (versioned YAML document, who and why). The cluster-wide part of server.yaml lives there; server.yaml keeps the per-machine keys only (listeners, TLS paths, store DSN, secrets, tracing, monitoring, mirror.api/tls, backup_dir/key, endpoints).
• First start against a store with no document imports the file's cluster keys (and the baseline guard policy when there is none); after that those keys in a file are ignored, with a warning.
• Each control plane polls the document's version and its own server.yaml's hash. A change is validated, then applied: guard live, the rest by a graceful restart of the process (standbys first, the leader last), which systemd brings back. No "restart now?" anywhere.
• llmgw ctl config show|history|get|set|unset|edit|rollback; the console's settings screen edits the store for cluster keys; admin API GET/PUT /api/v1/settings, /history, /rollback.
• A standby that becomes leader re-reads the catalog, engine choices and facility state from the store (it kept what it read at start).
• Retires appendGuardIfMissing (server init): the import does it.

E2. Generated files from settings (L) — firewall on/off done 2026-10-06, every machine 2026-10-09

firewall.enabled in the cluster's settings reaches nodes over the agent stream and control planes through llmgw-firewall-sync.path. Since 2026-10-09 the mirror machine too (the leader PUTs /v1/firewall, the mirror writes .firewall.json, the same path unit applies it) and store members that run no llmgw (the leader keeps it in cluster_state, a root timer reads the member's own copy with psql); firewall_networks reach all of them, and the mirror lets in the network of every node and control plane the leader knows. Tried on the PoC: off and on, and a network added, on control planes, mirror, store member and nodes. Left: the allowed set replaced by membership rather than added to (a network taken out of the settings stays let in), the other files (the mirror's unit flags, keepalived, etcd and Patroni configuration).

Every machine runs the agent (root) with roles — node, control plane, mirror, store member — and the agent renders and applies what was written once by init: the llmgw nft table, the mirror's unit flags, keepalived, etcd and Patroni configuration. The firewall's allowed set is the cluster's membership (every node, control plane, mirror and store member) plus firewall.extra from the store; 7443 stays open (mTLS). A node added or removed changes every machine's rules. Retires firewall.Admit and firewall_networks in server.yaml (imported into the store).

E3. agent.yaml down to bootstrap (M) — done 2026-10-08 (branch settings-e)

agent.yaml keeps the control planes' addresses, its certificate and its id. Labels, fabrics, engines, heartbeat and supervisor settings move to the store per node and reach the agent over its stream; a change to the file is noticed and the agent restarts itself. Built as a section of the cluster's settings rather than a store of its own, so history, audit, ctl config, the console and the portal page cover it: agents.default and agents.nodes.<id> (config.AgentSettings; bootstrap keys node_id/server(s)/hostname/mgmt_addr/tls refused; each node's merge loaded as an agent would, so a bad fabric is refused on save). A live key: the control plane sends ControlMessage.AgentSettings on register and, a second apart per node, on change; the agent compares with what it started on, checks it loads beside its agent.yaml (else logs and keeps running), keeps it in /var/lib/llmgw-agent/agent-settings.yaml and restarts (engines adopted). At start: agent.yaml < kept settings < LLMGW_ env; kept settings that no longer load are set aside, never a reason not to start. agent.yaml is watched every 10 s and restarted on when it changes and loads. Left: file keys are not imported into the settings (they keep working under them); node add still writes a full agent.yaml; labels have two homes now (ctl label, which wins, and agents.*.labels).

E4. Web settings page (S) — done 2026-10-08 (branch settings-e)

The same document, edited on /admin/settings, for whoever has no terminal. Built: administrators only (staff roles get 404: webhook secrets), checked by the same function as PUT /api/v1/settings, made from a base version or refused on a newer one (the edit kept in the editor), history with view and roll back as a new version, audit with the size not the body, English and Turkish, a Settings link in the menu.

---

F. Company documents: knowledge bases (RAG), then adapters (2026-10-08)

A company wants a model to answer from its own documents. Retrieval is the tool for knowledge (sources shown, re-indexed in minutes, access per tenant); LoRA training is for behaviour (vocabulary, format) and has no access control — whatever goes into weights reaches everyone who may use the adapter. RAG first. Decisions taken: access per collection (no per-document ACL yet); a personal key or session names a project or team and gets that chain only, never the union of its person's memberships; PostgreSQL store only, off on SQLite with a sentence saying why.

F1. Store: collections, grants, chunks (M) — done 2026-10-08 (branch rag)

Built (internal/store/knowledge.go, migration 0035 PostgreSQL only):

• kb_collections (owner scope/id, org, embed_model, dims), kb_grants (read|write, any scope, never across organizations; a key needs a project; a person needs a membership in the organization), kb_documents (digest: the same again is no change, a new one replaces in one transaction).
• kb_chunks is created when the store opens, not in a migration (a migration needing an extension would stop the control plane; one that skipped it would never run again): partitioned by organization (hashed partition name), tsv generated (simple), GIN on the parent, one partial HNSW index per collection on its organization's partition (embedding::vector(dims), WHERE collection = '<id>'). EXPLAIN on 20k rows: partition pruned, that index used, policy applied. Later (scale, 2026-10-08): a partial GIN per collection too (kb_tsv_<id>, backfilled at open); vec_type per collection, halfvec on pgvector ≥ 0.7 (up to 4000 dims); SET LOCAL hnsw.ef_search to candidates(k) clamped 40..1000 (EXPLAIN: 40 rows without it, all with it); the text arm takes candidates by ts_rank in a subquery and scores only those.
• RLS FORCEd on kb_chunks against llmgw.kb_org/llmgw.kb_collections (set with set_config per transaction), one role — the table owner — not a separate one. Chunks are deleted by the foreign-key cascade from their document, which RLS does not block.
• CollectionsFor(Effective, write): owner or grant anywhere on the chain; no organization (personal key without a project) reads nothing.
• SearchChunks: per collection nearest-neighbour (literal id and cast so the partial index is used) plus a full-text arm with any word (a | b, lowercased by the database, not Go), fused by reciprocal rank.
• Deleting a project/team/org that owns collections is refused like one with members (changed from "deleted with them": every other delete in the tree refuses); DeleteUser/ForgetUser delete the person's collections; deleting an organization drops its partition; grants to a deleted scope go.
• server init installs postgresql-N-pgvector / pgvector_N and enables it; pgha installs it on every member at setup and enables it on the leader on every run. Neither stops on failure. Left: a cluster set up before (the PoC) gets no package from a rerun of pgha on a running cluster — install by hand or add an upgrade step; server restore/CopyTo do not carry kb tables (SQLite → PostgreSQL only, and SQLite has none); tests need LLMGW_TEST_POSTGRES pointing at a pgvector database as a non-superuser (a superuser bypasses RLS).

F2. Ingest on the user's machine (L) — done 2026-10-08 (branch rag)

Uploading whole files to the control plane was dropped: a PDF is 20–100× its text, and the work belongs where the files are. Instead a small client, llmgw-kb, run by the person on their own machine with their own key:

• Its own binary (cmd/llmgw-kb), for Windows, macOS and Linux on amd64 and arm64. llmgw itself does not build on Windows (agent syscalls), and employees' laptops should not carry the server, agent and setup screens. A test keeps server packages out of its imports.
• llmgw-kb sync <collection> <dir|file…> [--include/--exclude globs] [--dry-run] [--delete]: scan → local catalog (path, size, mtime, sha256; cache under os.UserCacheDir, digests not recomputed for unchanged files) → the collection's manifest from the gateway (name → digest only) → diff → for new/changed files only: extract, chunk, embed through the gateway's /v1/embeddings with the person's key (its accounting, limits, guard and band apply), and send name, digest, chunk texts and vectors; --delete removes documents whose files are gone. --dry-run lists what would leave the machine and how much. A file that cannot be read (locked, unsupported, no text) is skipped with the reason; the run goes on.
• Formats, as many as possible. Built in, pure Go: PDF (text layer), DOCX/XLSX/PPTX, ODT/ODS/ODP, RTF, EPUB, HTML/XHTML, EML (text and HTML parts), Jupyter notebooks, subtitles (SRT/VTT), CSV/TSV, JSON/YAML/XML, Markdown/RST/AsciiDoc/Org, plain text and source code; ZIP archives recursed into. With a helper found on the machine: legacy DOC/XLS/PPT, RTF and anything else LibreOffice opens (soffice --headless --convert-to), images and scanned PDFs through Tesseract (tur+eng). Text in UTF-8, UTF-16 (BOM) or Windows-1254/1252 is converted to UTF-8.
• Across systems: document names relative to the synced root, / separated, Unicode NFC (macOS hands out NFD), a warning for two names that differ only in case; CRLF → LF; key from LLMGW_API_KEY or llmgw-kb login (os.UserConfigDir, 0600 where that means something); the system's CA store, --ca for another.
• Gateway endpoints (client listener, person's key): GET /v1/kb/collections (what this key may read/write), GET /v1/kb/collections/{id}/manifest, PUT /v1/kb/collections/{id}/documents (one document per request: name, digest, bytes, chunks with text and embedding; dims checked, chunk count and size bounded), DELETE …/documents?name=. Write needs a write grant or ownership on the key's chain (CollectionsFor(…, write)); otherwise 404.
• Administration stays with operators: llmgw ctl kb create|list|delete|grant|revoke|docs, admin API /api/v1/kb, audit, console screen; docs/knowledge.md.
• Trade-off accepted: vectors come from the client, so a writer could store vectors that do not match their text — only in collections they may already write. Built: internal/kb/extract (formats above; helpers found on PATH and in each system's install places), internal/kb/chunk (≈1800 characters under the heading path, a paragraph of overlap, sections kept apart once a chunk is a third full), internal/kb/kbsync (ledger in SQLite at os.UserConfigDir/llmgw-kb/ledger.db: documents keyed by gateway, collection and name with file, size, mtime, digest, state, reason; runs), cmd/llmgw-kb (login, collections, sync, status, history, formats), internal/router/kb.go (the four gateway endpoints, X-Llmgw-Project for personal keys, audit), internal/control/adminapi/kb.go + llmgw ctl kb + console Knowledge screen, docs/knowledge.md; CI builds the client for all six targets and releases attach it. Tested: every format from generated fixtures and a Turkish PDF (OCR path with Tesseract here), sync against a fake gateway and end to end against the router on pgvector. Tried on the PoC 2026-10-08 (control planes 140 and 150 on a dev build of this branch; their previous binary is /usr/local/bin/llmgw.pre-rag): Ubuntu's pgvector 0.6.0 installed on 149/150 and enabled; bge-m3's weights were never in the mirror (fetched from CompendiumLabs/bge-m3-gguf@1864f47f) and it is held at 1 replica; tree Demo/ik/belgeler and Demo/satis/crm with a key each; collection Demo/politikalar filled from a Mac through an SSH tunnel with ten Turkish files in nine formats (9 sent in 5 s, the .doc skipped for want of LibreOffice; a rerun sent nothing in 0 s; a changed file alone was sent). Nine Turkish questions each found their page first once search scored by similarity plus word coverage (rank fusion had got two wrong). The other team's key saw no collection and got 404 by id; with a read grant it read and could not write; revoked, nothing again. On the live store the llmgw role (not superuser, no BYPASSRLS) sees 0 chunks with no scope set and 0 with the organization alone. Left: the gateway's collection list shows the owner as team:ID, not its path; no LibreOffice on the dev machine, so the legacy Office path is untested beyond its error; the client is unsigned (Gatekeeper and SmartScreen warn); Outlook .msg is not read; one document over ~60 MiB of JSON (a few thousand pages) is refused rather than split. PostgreSQL tests of store, router and adminapi share LLMGW_TEST_POSTGRES and the store's TRUNCATE each test; run them with go test -p 1 or one package at a time, or a router test loses its rows mid-run.

F3. Retrieval on the request path (M) — done 2026-10-08 (branch rag)

• llmgw.knowledge: [collections] in the body (stripped before the engine), or default collections per project/key.
• Allowed set from the caller's chain (§7.9.2), cached like limits; grants add up down the tree. Personal keys and sessions need x-llmgw-project (or the playground's picker) naming a scope the person is a member of.
• A collection not allowed answers 404, exactly as a missing one.
• Hybrid search with the filter inside the query (per collection, its own partial index; pgvector 0.6 is enough — scored by similarity plus the share of the question's words held (built in F1/F2 as SearchChunks), /v1/rerank when a reranker is in the catalog. Top chunks into a system message with document and position; x-llmgw-sources lists them.
• Leaks closed: answer-cache key includes the allowed set; per-organization cache_salt on RAG requests; injected passages under no_store and the guard's prompt-injection check; returned doc ids in the audit.
• Done: router tests that tenant B never receives tenant A's chunks (including via cache), that a personal key without a project is refused, metrics (retrieval latency, chunks returned), docs.

Built (internal/router/rag.go): X-Llmgw-Knowledge (header, as the other llmgw extensions are, not a body field), X-Llmgw-Knowledge-K; collections from the key's chain (kbEffective, shared with the kb endpoints), 404 for one not readable; the question embedded through the ordinary path on the caller's principal; SearchChunks; passages trimmed to the model's context; added to the one leading system message (or a new one); x-llmgw-sources; before guardInput; never cached; metrics. Default collections built (kb_defaults on org/team/project/key, narrowest wins, 30 s cache, ctl kb default, console action; never refuse); the portal chat's Documents picker (collection, default or none; sources listed under the answer; a header the page sends itself is dropped). cache_salt per organization on requests given passages, sent to vLLM replicas only (chosen replica's engine). Checked on the PoC 2026-10-08: vLLM 0.30.0 (cuda01) with TinyLlama-1.1B AWQ run by hand beside the 35B in the card's last 2.3 GB — a 668-token prompt reused 656 tokens under the same salt and 0 under another's, with identical text (llmgw cannot place it there: placement takes whole GPUs). Live on the PoC: a team default answers a request without the header, none turns it off, an explicit header with k=2 gives two sources and no more, llmgw-kb lists owners by path. Left: returned passages are not audited per request (the header and the kept request record them).

F4. Adapter training (L, after F1–F3) — done 2026-10-08 (branch adapters-f4)

A run-to-completion job placed by the control plane like a replica and supervised by the agent like an engine, in a Python environment the agent installs (TRL + PEFT, QLoRA). Data: Q/A pairs generated from a collection by a cluster model through /v1/batches, reviewed by people. Output to the mirror, into the model's adapters, through canary and a quality suite. Runs when the scaling schedule frees a GPU; 8–14B dense bases on one card. Never train on a collection some users of the adapter may not read. Tried 2026-10-08 on the PoC, by hand: Qwen2.5-0.5B-Instruct (fp16) with a LoRA (r=16, every linear layer, 8.8M trainable), trained beside the 35B in cuda01's last 2.3 GB (peak 2158 MiB on the card, 64 s for 138 steps, PEFT in a venv over the vLLM engine's torch). Data: 90 Turkish Q/A pairs a cluster model wrote from the nine Demo/politikalar documents, through the gateway. On nine questions it had not seen, phrased otherwise: base 0/9, 6 epochs 4/9, 4 epochs at a lower rate 2/9. Wrong answers mixed facts across documents (the meal card's amount, the office day). The same nine through retrieval: 9/9 with sources. So F4 is for format, vocabulary and tone, and facts come from F1–F3; a bigger base on a free card would do better but not change that. The training itself fits a small slot on a shared card, which the job design can rely on for small bases.

Built: ctl kb dataset (POST /api/v1/kb/collections/{id}/dataset, NDJSON with each pair's collection); train_jobs/train_artifacts (migrations 0037, 0038 owner); internal/control/train (queue, one job per node, artifacts in 1 MiB pieces, a node that goes fails its job); internal/agent/train (the newest vLLM's python3 + uv pip install --no-deps --target peft 0.21.2 and accelerate 1.15.0; llama.cpp's conversion code for the installed release, untarred from its source; the card with the most free memory, if enough; runs as llmgw-engine); ctl train start|list|show --follow|cancel|rm| download|publish; the mirror's PUT /v1/models/{model}/{version}/{file}; publish uploads the GGUF (llama.cpp) or PEFT files (vLLM) and sets the adapter after checkAdapters. Owners: catalog.Adapter.Owner, enforced in the router (404, hidden from /v1/models); a job's owner defaults to its single collection's owner, must be able to read every collection the pairs came from, and pairs from a knowledge base are never public. Tried on the PoC: dataset of 90 pairs in 89 s; a job trained beside the 35B (peak 2214 MiB, 138 steps in about a minute) and sent back PEFT and GGUF. Left: a job dies with its agent's stream (no resume); no console screen for training yet; quality gating of a published adapter is by hand (ctl eval).

---

F5. Large imports (M) — measured 2026-10-09, open

The metallurgy corpus (2,314 files, 108,779 chunks, bge-m3 on the one GPU) went in at about 25 chunks/s, the same over a phone hotspot and the LAN. The engine's slots were the limit: llama.cpp --parallel 4 embeds four texts per pass, while llmgw-kb sends 32 a request from 6 workers. The usage rows show 4.6 s a request on average, 1.8 s of it in the gateway's queue and 0.04 s in the guard. The GPU sat at 18–50% and the store machine at a third of its CPUs.

• An embedding model's profile with many slots by default (--parallel 32–64, --ctx-size to match, --batch-size/--ubatch-size at least the longest chunk), or ctl model serve choosing that for a model with capabilities.embeddings. With the default 512, one 516-token chunk failed.
• Embedding on the gateway's side of a document PUT, so the client sends text only: today an embeddings answer is ~21 KB of JSON a chunk, and the text and vector go up again (~11 KB).
• The store next: HNSW inserts took half of one of pg01's two cores; at a million chunks the vectors alone outgrow its 4 GB. halfvec (pgvector ≥ 0.7) and a larger store machine.
• Done: a batch of embeddings checked text by text against the context (branch embeddings-batch); it was refused as one over-long prompt.
• Done: query_prefix on an embedding model, before questions only.
• Qwen3-Embedding-4B on the RTX 6000 (2026-10-09): llama.cpp Q8 1,575 tok/s as first set up (32k cache, 16 slots), 2,700 with a 16k cache, FlashAttention and ubatch 1–2k; f16 no better. vLLM fp16 (--runner pooling) 6,200 direct, 4,700–4,900 through the gateway in the import, and its vectors match llama.cpp's (cosine 0.9995). The 103,579 chunks took 3 h. Done (2026-10-10, branch admission-bulk): a source per profile, so an embedding model's GPU profile runs vLLM and its CPU profile llama.cpp; a band's own queue_ttl for bulk imports, and Retry-After on a busy refusal, which is what had turned passages away (llmgw-kb retries only what says when to).
• Retrieval on 40 questions written from the corpus, document in the top 5: bge-m3 31/40 (Turkish questions 13/21), Qwen3-Embedding-4B 37/40 (18/21), with its query prefix 36/40 and first place 31/40 instead of 28 (MRR 0.738 → 0.789 → 0.838). Next: translating the question with the chat model, and a reranker.

G. Jobs: 3D generation through ComfyUI (2026-10-10)

A request for a 3D model takes minutes on the one 24 GB card, and the card has to be emptied of the chat model first. That is not a request to hold a socket for — internal/router/activate.go:46 already says so — it is a job: queued, answered with an id, asked after, its result downloaded. The job machinery is built first and tested without a GPU; the ComfyUI generalisation and the custom-node install come after. Decided 2026-10-10: a jobs table of its own rather than a one-line batch (the JSONL shape would be artificial and has nowhere for progress), and progress is shown, not just a status.

Which models (decided 2026-10-10): the assets go into a game sold on Steam, worldwide, the EU included, so the licence of the output decides before the quality does. Hunyuan3D (every version) is out: Tencent's licence excludes the EU, UK and South Korea, and whether a game shipped there with its outputs is "use in the Territory" is not a question to leave to a court. In, in this order: TRELLIS.2 shape (MIT; the best geometry; its own texture baking goes through nvdiffrec and nvdiffrast, both under NVIDIA's non-commercial source licence, so that stage is not run), Step1X-3D (Apache 2.0, StepFun; shape and texture in one, the EU-safe textured path — its ComfyUI node and VRAM need are to be checked), Direct3D-S2 (MIT, DreamTech; high-detail shape, has a ComfyUI node) if TRELLIS.2's detail falls short. Not in: TripoSR and InstantMesh (permissive but superseded), Stable Fast 3D (Stability's community licence, revenue-capped; served all the same since 2026-10-10 at the user's request, see G3), SAM 3D Objects (Meta's own licence, to be read before it is considered). Steam's disclosure form (changed 2026-01-16) lists pre-generated 3D models as content that must be declared on the store page: the licence notice G3 writes into the GLB is what the studio answers that form from.

G1. Jobs: store, worker, endpoints (M) — done 2026-10-10 (branch jobs-g1)

Built as written, with two choices: a job found running after its gateway died is run again from the start (the prompt id column is there for G2 to follow the same work instead), and the request is validated at the door only as far as costs nothing — key, model, allowlist, the request's shape — with limits, guardrails and admission applied when the worker replays it, said as the job's error. The result file's content_type is the job's; the files endpoint still serves every file as an octet stream.

• Store: jobs (id, owner key/principal, kind, model, status, input (JSON), progress (JSON: queue position, percent, step), result file id, error, created/started/finished, lease owner/until), migrations for sqlite and postgres beside internal/store/batches.go, whose claim/renew/release lease pattern is copied, not shared: a job is one unit of work, a batch is many lines.
• Worker: internal/router/jobworker.go after batchworker.go. Any gateway on the store claims a queued job, asks for the model with activator.Want and waits as long as it takes (no ActivationWait bound: nobody is holding a connection), runs it, stores the output through the files store, writes usage and price, marks it done. A gateway that dies mid-job loses its lease and another one picks it up; the ComfyUI prompt id is in the job row so the second gateway asks after the same work rather than queueing it again.
• Endpoints: POST /v1/3d/generations validates like any inference request (key, allowlist, limits, admission, guardrails on the prompt), writes the job, answers 202 with {id, status: "queued"}; GET /v1/3d/generations/{id} returns status, progress and, when done, output_file_id and the usage; POST .../{id}/cancel; the file comes from GET /v1/files/{id}/content. Owner-scoped like batches. Results are kept for the batch files' retention and deleted with them: a departure from images.go's "kept nowhere", because a mesh is a download, not a JSON field; the retention is the bound.
• Done when: a job survives the gateway that took it being killed, a cancelled job leaves ComfyUI's queue (comfy.Client.cancel), two gateways never both run one job (test with the lease), and the usage row carries the job's price.

G2. Progress from ComfyUI (S, after G1) — done 2026-10-10 (branch jobs-3d)

Built as written: the gate's websocket under its own client id, the client_id of every /prompt rewritten, /progress/{prompt_id} on the gate, the router reading it and /queue into the job (only into a job of the same key). Left: a node that reports no steps (TripoSR's) shows its name and no percent; a vanished replica is said by the job failing, not by progress. The gate (internal/comfy/gate.go) passes seven paths and no websocket. Rather than open /ws, the gate subscribes to ComfyUI's own /ws once, with golang.org/x/net/websocket (already a dependency), under a client id of its own, and rewrites the client_id of every /prompt it forwards to that one. It keeps the last executing, progress (value/max/node) and execution_error message per prompt id and serves them at GET /progress/{prompt_id} on the gate; the router polls that beside /history and /queue and writes queue position, percent and the current node's class into the job's progress. The gate's buffer is bounded (the last N prompts) and the websocket reconnects. Done when a 3D job shows "queued, 2 ahead" then "running 40%, step 12/30" in ctl jobs and on the account page, and a job whose replica vanished says so within one poll.

G3. ComfyUI beyond pictures: 3D models and custom nodes (M, after G1) — done 2026-10-10 (branch jobs-3d), with TripoSR

Built: output kinds, image upload through the gate, every output list fetched, OBJ to GLB, licence in /v1/models, the job and the GLB, per_mesh and usage.meshes (migration 0045), custom nodes cloned at their commit by the gate with packages beside the replica (a node's named packages at its range, the rest at ComfyUI's versions). Not as written: the model. On the PoC's Turing card TRELLIS.2 cannot run — it needs Meta's gated DINOv3, its published background remover (RMBG-2.0) is non-commercial, and its sparse attention and Direct3D-S2's need flash-attention (Ampere and newer). TripoSR (MIT) proves the chain on cuda01: a .glb from a picture in about 30 s. Left: background removal before TripoSR (BiRefNet, MIT, as a node), TRELLIS.2 or Direct3D-S2 on an Ampere card with an ungated encoder, a missing node refusing the replica by name. Added 2026-10-10 at the user's request, beside the order above: Stable Fast 3D as sf3d (Stability AI Community License: free under US$1 M annual revenue, an enterprise licence above, "Powered by Stability AI"), the textured mesh that runs on Turing: about 1 s a mesh warm in fp16, UV-mapped GLB with its texture, 18 GB of the card at peak. Its weights are gated, so they were fetched once with the user's token into /srv/llmgw-comfy/stable-fast-3d and linked into the engine user's Hugging Face cache, which its loader node reads by repository name; no token is on the machine at runtime. Its own save node returns the GLB as base64, which the gate cannot fetch, so LLMGWSaveMesh writes the file; LLMGWPrepareForMesh now also gives the object's mask. Left: its two CUDA extensions (texture_baker, uv_unwrapper) are not on PyPI, so they were built on cuda01 and installed into the engine's venv by hand; a custom node's requirements need a way to name a local build, or the engine install must carry them, before a customer site gets this. And rembg's default session now fetches BRIA RMBG-2.0 (non-commercial): every workflow must name its rembg model (ours say u2net).

• Catalog: a model makes pictures or meshes (MakesImages becomes an output kind; ServesComfyUI stays). 3D weight folders in the gate's modelFolders; the workflow's placeholders gain image (the input picture, base64 in the request, written to the replica's input/ by the gate through a new POST /upload/image the gate allows — or a base64-loading node, whichever TRELLIS.2's wrapper wants).
• comfy.Client.fetch reads every output key, not images only, and fetches each file named by /view; Image becomes Artifact with its content type (model/gltf-binary, application/octet-stream for .obj).
• Custom nodes: a model (or the engine config) names git repositories and a pip requirements list; the agent's provision step clones them into the replica's custom_nodes/ (the gate's --base-directory is where ComfyUI looks, not the install directory) and installs the requirements into the engine's venv with the same PyTorch index and lock discipline as A8. A replica with a missing node refuses to start with the node's name in the error rather than ComfyUI's traceback.
• Licence of the output (decided 2026-10-10: it matters). A catalog model carries license: {name, url, notice}; a job's answer, /v1/models and the portal page show it beside the result, and the mesh's GLB asset.extras carries the notice too, since the file travels without the API. What was checked on 2026-10-10: TRELLIS.2 weights are MIT and its outputs free, but its texture baking depends on nvdiffrast, whose NVIDIA Source Code License is non-commercial — a commercial service must render textures another way or skip that stage. Hunyuan3D 2.0 and 2.1 claim no rights in outputs, but the licence excludes the EU, UK and South Korea, needs Tencent's permission above 1 M monthly active users, forbids using outputs to train other models, and asks for "Powered by Tencent Hunyuan"; the notice ships with any distribution. TripoSR is MIT, InstantMesh Apache 2.0. Every one of them trained on Objaverse, some of whose objects are CC-BY-NC: a provenance question no licence text settles, to be said in the notice rather than hidden.
• Pricing: per_image stays; per_mesh beside it in accounting.Price, Outcome.Meshes, the usage sums and exports (as eaea671 did for pictures).
• Done when: TRELLIS.2 on cuda01 with scaling: {min: 0, idle_timeout: 10m} makes a .glb from a picture through G1's endpoint with nvdiffrast and nvdiffrec not installed in its venv (the licence is kept by the install, not by a note), the GLB carries the MIT notice, the card is free again after the idle timeout, and the same request against flux-schnell is refused as "makes pictures, not meshes".

G4. Portal and console (S, after G2) — done 2026-10-10 (branch jobs-3d)

Built: /3d with model-viewer served from the gateway, progress, download and licence; ctl 3d list|get|cancel over /api/v1/client-jobs (ctl jobs was taken by the control plane's own work). Left: the console's Jobs view. A 3D page beside /images (upload a picture, watch progress, download the .glb, a viewer with <model-viewer> from the allowed CDNs), ctl jobs list|get|cancel, the console's Jobs view. The page sends down the same path as the API, like imagesSend does.

D. Facts worth knowing before starting

• Admin API: 127.0.0.1:9090, no auth, ctlClient in internal/cli/ctl.go. Metrics on :9091 with /prometheus/targets beside them.
• Store: internal/store, migrations under internal/store/migrations/{sqlite,postgres}; the PoC runs PostgreSQL under Patroni.
• The console (sudo llmgw) is internal/cli/tui_console.go with actions in tui_actions.go; every operator-visible change needs a row action or a form there, and a line in docs/console.md.
• Tests of the console use a fake admin API (internal/cli/tui_actions_test.go, adminStub); tests of serving use fakes in internal/control/modelserve/modelserve_test.go.
• go generate ./... regenerates docs/reference.md and internal/docgen/reference_gen.go whenever a config struct changes; the docgen test fails otherwise.