llmgw/BACKLOG.md

Backlog

What a review of the code on 2026-10-04 (commit c2238ab) found should be done next: security fixes first, then guardrails, then features. Written for whoever implements it — a person or a model — without the review in hand. Every item names the files it touches and what "done" is. Paths are relative to the repository root. Sizes: S about a week, M two to four weeks, L more.

Conventions this codebase keeps, which every item here must keep too:

• A change is refused before anything is written (validate, then persist).
• Nothing on the serving path logs prompt text, keys or passwords.
• Each item gets unit tests beside the code (*_test.go), and the TUI (internal/cli/tui_*.go), docs/console.md and CHANGELOG.md are updated when behaviour an operator sees changes.
• go test ./..., go vet ./... and govulncheck ./... pass. No release is cut and nothing is deployed to the PoC cluster without being told to.

---

A. Security — do first

Status (2026-10-04): done and merged. A1–A9, A6b included — etcd and Patroni's API on TLS, the PoC moved in maintenance mode without a failover. Deliberately narrower than written: the admin API is a token on loopback rather than a unix socket (A5); thread ids are bounded and documented as project-wide rather than scoped per user (A8); Python engines resolve with an explicit PyTorch index and a PEP 751 lock rather than locks shipped per engine version (A8). All of A is live on the PoC.

A1. Toolchain (S, hours)

govulncheck finds 11 standard-library vulnerabilities reachable from this code at Go 1.26.3 (net/http, crypto/x509, os.Root, net/textproto). Raise the go directive in go.mod to 1.26.6 or later, rebuild, run govulncheck ./... until it reports none reachable. Add govulncheck to the Makefile's check target.

A2. Release signing is effectively off (S)

• internal/release/trusted.pub is empty, so TrustedKey() returns "" and release.go:103 skips signature verification. Ship a real public key; make a missing or invalid signature fatal. Drop the LLMGW_RELEASE_KEY environment override, or honour it only in -tags dev builds.
• internal/onboard/onboard.go:365: the node-side download checks only a same-origin checksums.txt. Verify the signature there too (simplest: have the node run llmgw itself to fetch and verify once it has any binary), and shellQuote the Releases URL, which is interpolated into single quotes unescaped.
• internal/agent/upgrade.go:44: accept only https:// release bases.
• Done when: a tampered binary or checksums file is refused by self-upgrade, node onboarding and agent upgrade, with a test for each.

A3. Agent port without TLS is anonymous and plaintext (S)

internal/cli/server.go:423 serves h2c on :7443 when agent_tls is not configured; agentsvc then accepts any node id. Refuse to start control mode without TLS unless agent_tls.insecure: true is set explicitly; when insecure, bind agent_listen to 127.0.0.1 and log a warning at startup and in llmgw doctor. Keep server init writing TLS as it does.

A4. Model profiles reach root on every node (M)

Profile args and env from a model definition are appended to the engine command unchecked (internal/agent/engines/llamacpp.go:170, vllm.go:151, sglang.go:98); internal/agent/engines/unsafeenv.go is a denylist that misses NCCL_NET_PLUGIN, NCCL_TUNER_PLUGIN, NCCL_PROFILER_PLUGIN, CUDA_INJECTION64_PATH, NVTX_INJECTION64_PATH, TRITON_* paths; engines run as root.

• Replace the env denylist with an allowlist (CUDA_/NCCL_ tuning keys minus anything ending in _PLUGIN or containing INJECTION, VLLM_*, HF_* minus path keys, OMP_*, GGML_*). Apply at catalog validation (internal/control/catalog/catalog.go, where UnsafeEnv is called) and again in the agent.
• Deny engine flags that write files or run code: --trust-remote-code, --log-file, --slot-save-path, --chat-template with a path, --host, --api-key, --port; vLLM/SGLang equivalents. Allowlist is better than denylist if the flag set is enumerable per engine.
• Run engines as an unprivileged user with the video/render groups (internal/agent/supervisor, systemd unit in internal/cli/serverinit_cluster.go:52). Weights cache must be readable by it.
• Done when: a profile with --trust-remote-code or NCCL_NET_PLUGIN is refused at ctl model add and at the agent; ps shows engines not as root.

A5. Admin API trusts any local process (M)

internal/cli/server.go:357 serves the admin API and pprof on 127.0.0.1:9090 with no authentication; adminapi.actorOf (internal/control/adminapi/api.go:1613) takes the actor from the client-sent X-Llmgw-Actor header.

• Serve the admin API on a unix socket (/run/llmgw/admin.sock, 0660, group llmgw-admin) by default, keeping TCP as an option that requires a bearer token read from a root-only file (/etc/llmgw/secrets/admin-token).
• Take the actor from SO_PEERCRED (uid → username) on the socket, or from the token's name; ignore the header.
• Refuse admin_listen on a non-loopback address unless admin_insecure: true.
• Put pprof behind its own debug.pprof: true setting, off by default.
• ctlClient (internal/cli/ctl.go), the console and node add must work over the socket; sudo llmgw already runs as root.

A6. Patroni REST and etcd are unauthenticated and plaintext (M)

internal/pgha/config.go:99 writes a restapi: with no authentication and :54 etcd over http:// with no client auth; hba.go:56 PATCHes Patroni's config unauthenticated. Anyone reaching :8008 or :2379 owns the store.

• Generate a Patroni REST username/password at setup, store it in /etc/llmgw/secrets/, write restapi.authentication and restapi.certfile/cafile from the cluster CA; send it from internal/pgha/hba.go and status.go.
• etcd: client and peer TLS from the cluster CA with --client-cert-auth.
• Document, and have server init write, nftables rules limiting 2379/2380/8008/5432 to member addresses (see A9).
• Done when: an unauthenticated curl :8008/config is refused and the store status in the console still works.

A7. PostgreSQL traffic is plaintext (S)

internal/pgha/config.go:168 and internal/cli/serverinit_postgres.go:330 use sslmode=disable; HBA lines are host. Enable server TLS in Patroni's postgresql parameters with a certificate from the cluster CA, write hostssl lines, use sslmode=verify-full&sslrootcert=/etc/llmgw/ca.pem. Existing clusters: llmgw server must keep working against a store that has not been migrated, with a warning.

A8. Smaller fixes, each S or less

• Capacity fallback skips the allowlist. internal/router/router.go:700 builds [canonical]+entry.Fallbacks without permits. Filter the chain with r.permits(p, model); price against the model that served.
• Suspended users keep working API keys. store.(*DB).Resolve (internal/store/memberships.go) never joins users.state; Authenticator.Invalidate() is never called. Reject keys whose owner is not active; call Invalidate on revoke, suspend, limit change.
• Join codes never expire server-side. internal/join/join.go:103 checks expiry only in Parse; admit (:212) does not. Check it there; bind the listener to the chosen address, not :7444 on all interfaces; use the peer's address rather than the claimed hello.Addr, and show the joiner's address for the operator to confirm before secrets are sent.
• Leftover sudoers grant. internal/onboard/gossh.go:248 writes NOPASSWD:ALL and removes it only in Close() (:271). Write the grant with a self-removal (at/systemd-run timer, 30 min) so a crash leaves nothing; wipe password buffers after use. In ssh.go:39 pass -- before the target.
• Auth cache and pre-auth DoS. internal/auth/auth.go:127: unbounded map, one DB query per bogus well-formed key. Bounded LRU, negative cache with short TTL, per-IP token bucket before the DB.
• HTTP server hardening. server.go:351-360, the h2c agent server, both mirror servers: set ReadHeaderTimeout (exists), IdleTimeout, MaxHeaderBytes, a body-read deadline, netutil.LimitListener; connect WithReadMaxBytes on the agent stream.
• Login/signup abuse. Default signup_open: false; per-IP token buckets on /login and /signup; bounded wait for argon2 slots; CSRF token on the login form; the per-address throttle must not lock the real owner out (use IP+address pairs).
• Trusted proxies. internal/web/account.go:557, landing.go:60 honour X-Forwarded-Proto/Host from anyone; add router.trusted_proxies and only then trust them; Cache-Control: no-store on /setup/*.
• Certificate revocation. DELETE /api/v1/nodes/{id} only drops inventory; add a revoked-CN/serial list in the store checked in agentsvc Connect and mirror onlyClients; serverinit_cluster always passes --api-client.
• Engine exposure. Engines bind 0.0.0.0 (llamacpp.go:102, vllm.go:109): bind the management address. SGLang receives its key on the command line (sglang.go:102): pass it by env or file.
• Backups. internal/cli/backup.go: encrypt archives to an age recipient from backup.recipient in server.yaml; keep the directory 0700.
• Python dependency confusion. internal/agent/provision/python.go:121 uses --extra-index-url with unsafe-best-match. Pin torch to the PyTorch index explicitly (uv [[index]] explicit = true), first-index, hash-pinned lockfiles per engine version.
• Control-plane SSRF. internal/sourcecheck/sourcecheck.go:39 and gguf.FromURL use the default client; reuse the mirror's fetchClient(false) (private-address filter) and refuse a proxy unless configured; block NAT64/6to4-embedded private addresses in internal/mirror/safefetch.go.
• Low: agentsvc/service.go:430 indexes n.GPUs[0] without a length check (CPU-only node panics the handler); thread ids are client-chosen with no length cap and shared per project (router/threads.go) — cap length, document, offer per-user scoping; /v1/models lists models the key may not use (handlers.go:799); sessions have no idle timeout and PurgeSessions/PurgeClaims are never called — add a janitor; add a CSP header and set htmx.config.allowEval=false; cap GGUF array nesting depth in internal/gguf/gguf.go; mirror .source sidecars are served by the public file server — exclude dotfiles from :8081; prune modelserve.jobs and the mirror running map; KaTeX → 0.16.21+.

A9. Firewall (S)

No machine in a default install has a firewall. Have server init, node add and server add write an nftables ruleset: ssh open; :8080 open; :7443 and :9091 open to the cluster's networks; :9100 to the monitoring host; :2379/2380/8008/5432 to store members and control planes; mirror :8081 to the cluster, :8082 to control planes; :9090 Prometheus and :3000 Grafana to operators. llmgw doctor reports a missing ruleset. Make it skippable with --no-firewall.

---

B. Guardrails (M–L, build in this order)

Status (2026-10-04, branch guardrails-b): B1–B4 done, and tried live on the PoC. That found four bugs, all fixed with tests:

• a shadow default merged with an attached policy of unwritten mode left it in shadow;
• a streamed word held back at one release was dropped at the next;
• llama.cpp continues a conversation ending on the answer, so Llama Guard's verdict on answers is now asked with its own prompt as a completion;
• a judge model was named in a client's "no endpoint" error.

The PoC runs observe (shadow) for everyone, and Llama Guard 3 1B (llama-guard-3-1b, CPU, on cuda01) as a judge, which blocks a harmful request in ~0.7s and a harmful answer in ~1.4s there. Still left: see each item.

Today nothing on the request path reads, classifies, redacts or filters message text in either direction (confirmed; the only "moderation" hit is a comment in internal/router/budget_fallback.go:20). Existing controls are about who, how much and how fast: allowlists (router/handlers.go:450), RPM/TPM/budget (internal/accounting), context precheck (router/context_limit.go), admission bands (router/admission), timeouts (router/proxy/proxy.go:26-57). Gaps to close on the way: no default output cap when the client omits max_tokens; catalog.Capabilities is advertised but never enforced; tool results are trusted input.

B1. Hooks in the proxy and router (S)

Done (branch guardrails-b): internal/guard holds the interface; proxy Filter/StreamFilter; the input hook runs after loadThread (the simpler choice); error codes, headers, Responses incomplete, usage columns guard_*. Moved to B2: holding delta.tool_calls until they are whole and reading their arguments — that is the tool policy's filter, not the proxy's.

• internal/router/proxy/proxy.go: add optional InputRewrite func(body []byte) ([]byte, Verdict) on the request and two output hooks: Filter for relayBuffered (between ReadAll and WriteHeader) and StreamFilter{OnEvent(data) (emit []string, stop bool); OnEnd()} for relayStream, called where each whole SSE event is assembled (eventBuffer, before io.WriteString at :343). Nil hooks cost nothing.
• internal/router/handlers.go: call the input policy after the allowlist and limits and before admission (around :395); the principal, scopes and the model that will serve are known there. Thread history is spliced later (:416): scan stored turns when written, or move the hook after loadThread (decide; the second is simpler).
• Blocking conventions: input → 400 invalid_request_error with code: "content_policy_violation" (today writeError always sends code: null); output non-stream → 200 with finish_reason: "content_filter" and empty/redacted content; output stream → a final chunk {"choices":[{"index":0,"delta":{},"finish_reason":"content_filter"}]}, then data: [DONE], then cancel upstream. Headers x-llmgw-guard: blocked|redacted|flagged and x-llmgw-guard-policy set before the proxy writes headers.
• internal/router/responses.go: map content_filter to response.incomplete with incomplete_details.reason: "content_filter" (today it always emits response.completed, :525).
• Tool calls: hold delta.tool_calls until finish_reason: "tool_calls", then validate; replyDelta (reply.go:33) must also read tool_calls[].function.arguments.
• Accounting: a blocked input is recorded like RecordRefused (nothing charged); a blocked output is charged (the GPU was used). store.Usage gains GuardAction, GuardPolicy, GuardCategory, GuardMs.

B2. Package internal/guard (M)

Mostly done (branch guardrails-b): policies in server.yaml with attachments in the store (ctl guard attach), narrowed by the limits' rule; guard/detect (validated PII, secrets, injection through normalisation and decoding); redaction with consistent placeholders before prefix routing; tool allow/deny/strip, call count and argument schemas; capabilities and structured output (opt-in); default max_tokens; pinned system prompt and dropping the client's; no_store (threads, notes, prefix tree, external models); shadow mode. Left: a guard_events table — the usage row's guard_* columns carry action/policy/category/time, not rule and stage; language identification; the re-identification map that restores redacted values in answers.

• Policy, Effective resolution over the tenancy tree with the same narrowing rule as limits (internal/store/sqlite.go:383-416 tighten): deny sets union, allow sets intersect, mode takes the strictest of enforce > shadow > off, fail takes closed > open, no_store at any level wins, thresholds take the minimum.
• Store: guard_policies(scope, scope_id, policy json) and guard_events(request_id, scopes, policy, rule, category, action, stage, score, judge_ms, shadow) — ids, categories, offsets and lengths only, never text. Migrations for sqlite and postgres. Pruned with usage_retention.
• guard/detect: regex/keyword sets; PII with validators — email, phone, IBAN (mod-97), card (Luhn), TCKN (checksum), IP, secrets (sk-, AWS, JWT, PEM); normaliser (NFKC, strip zero-width/bidi, homoglyph fold, base64/hex/URL decode to depth 2); injection heuristics (role-switch phrases, fake chat-template tokens, long base64, invisible Unicode) — default action flag; language id.
• Redaction rewrites messages[].content in the upstream body with placeholders ([EMAIL_1]) via the same map[string]json.RawMessage round-trip withModel/spliceThread use; an optional re-identification map lives only in request memory and restores originals in the output. Redact before prefix routing so PromptPrefix hashes the redacted text.
• guard/toolpolicy: tool name allow/deny (strip or 400), JSON-Schema validation of assembled arguments against the declared parameters, argument regex rules, max_calls_per_turn. Enforce Capabilities.Tools/Vision here: a tools array or image parts to a model without the capability → 400.
• guard/structured: validate response_format.json_schema on the final text → content_filter, or flag in shadow.
• Output constraints: inject a default max_tokens when absent (from the policy or MaxOutput), language check, structured check.
• System-prompt pinning per project: prepend before checkContextFits and before spliceThread; optionally drop client system/developer roles; exclude the pin from stored thread turns; apply to Responses instructions in toChat.
• no_store: loadThread and PutNote refuse; captureBytes() returns 0; skip the prefix-tree observe (router.go:840); disable External fallbacks for the scope (filter in dispatch and budgetFallback).

B3. Judge model on the cluster (M)

Done (branch guardrails-b): role: guard + guard.format; hidden and refused to clients and fallbacks; critical by default, never on demand; Router.Judge past admission/accounting/guardrails; llamaguard3, shieldgemma and json parsers; 5-minute cache; latency budget then fail open/closed; streamed answers read every 600 characters, or held with buffer_all. Simpler than written: the input judge runs after admission, not alongside it, and shadow mode judges synchronously within the budget rather than after the response. Not enforced: two replicas on different nodes — documented, left to scaling.min.

• catalog.Model.Role: "guard" plus Guard{Format: llamaguard3 | shieldgemma | json, Categories}. Guard models are hidden from /v1/models, the chat picker and allowlists; placed like any model, class critical, scaling.min ≥ 1, at least 2 replicas on different nodes.
• guard/judge: calls the guard model through the router's own dispatch with an internal principal and its own admission band (never the tenant's queue); parsers per format (Llama Guard safe/unsafe\nS1,S10; ShieldGemma yes/no probability → per-tenant thresholds; generic JSON verdict); verdict cache keyed by hash of normalised new turns (5 min); latency_budget (default 250 ms) then fail open/closed; input judge runs in parallel with admission queueing.
• Streaming strategy: default sentence-buffered scanning (regex/PII per sentence or ~200 chars before release; judge every 400–800 chars on the cumulative tail; final judge at end); buffer_all for fail: closed scopes. Document that released text cannot be retracted.
• Shadow mode runs everything asynchronously after the response (context.WithoutCancel, like saveThread) and only records.

B4. Config, metrics, UI (S)

Done (branch guardrails-b): library in server.yaml, attachments via ctl guard, the console's Guardrails… with "applies from" marks, and the portal's /admin/guard; the metrics and alerts below (judge latency labelled by model only at the judge; the check latency by stage); ctl request shows the verdict; the chat page says "stopped by policy"; the profile lists effective policies; docs/guardrails.md and data.md written; the access log carries action, stage and policy only. Attaching from the portal is not done — the page shows; the console and ctl change.

• Gateway-level policy library in server.yaml under guard: (see the YAML in the review, reproduced in docs/guardrails.md to be written); attachments to scopes in the store via llmgw keys guard, the console ("Guardrails…" beside "Limits…" in internal/cli/tui_people.go, with "inherited from team:X" badges) and the portal.
• Metrics in router/metrics.go style: llmgw_guard_checks_total{stage,policy,action,category,mode}, llmgw_guard_latency_seconds{stage,judge_model}, llmgw_guard_judge_errors_total{reason}, llmgw_guard_fail_open_total, llmgw_guard_stream_stops_total, llmgw_guard_redactions_total{kind}, llmgw_tenant_guard_blocks_total{tenant}. Alerts: fail-open > 1% for 5m, judge p95 over budget, judge has no ready replicas, per-tenant block rate

  3× its 7-day baseline, any secret detected in output.

• Portal /admin/guard: violations by tenant/category/policy over time, shadow vs enforce, judge health, recent events by request id; ctl request <id> shows the verdict; chat page renders content_filter as "stopped by policy X"; /me lists effective policies.
• Update docs/data.md: the judge sees all text but in-cluster; External fallbacks send prompts outside the cluster (the current "nothing leaves the cluster" is wrong); no_store semantics.
• Never log: prompt/completion text, PII values, the re-identification map, judge inputs or rationale that quotes input. Check accessLog's err field and span errors.

---

C. Features, ranked by value × feasibility

1. Done (branch features-c): audit_events, written by the admin API middleware and the /admin pages (with each change's result), ctl audit, the console's Audit view; staff roles viewer / operator / billing / admin. Left: before/after values (the request's fields are recorded, not the state they replaced); a tenant-admin role that changes its own organization (managers still only view it); hiding the forms a viewer cannot submit. Audit log and roles (M). Today "audit" is slog lines (internal/web/admin_users.go:403-480) and roles are the IsAdmin flag plus an org "manager" who can view. Add audit_events (who, what, before/after, source) written by the adminAction/tenancyAction wrappers and by every admin-API mutation; roles viewer / operator / tenant-admin / billing; ctl audit. Depends on A5 for a real actor.
2. Done (branch features-c): OIDC sign-in (auth.oidc, PKCE, verified email, roles and memberships from groups, no_passwords) and SCIM 2.0 (/scim/v2 Users and Groups). Left: the sign-in cookie is signed with a per-process key, so with two control planes behind a balancer the callback must reach the one that started it (sticky sessions) until the key comes from the store. OIDC/SSO login (M), then SCIM (M). internal/account/service.go, internal/web/account.go; map IdP groups to org/team/admin.
3. Done (branch features-c): key lifetime limits (gateway, org, team, project), keys rotate --overlap, kinds and expiry in keys list. Service accounts and key lifecycle (S). Per-org maximum key TTL, keys rotate with overlap, last-used shown, project-owned service keys listed apart. internal/cli/keys.go, internal/store/keys.go, internal/auth/auth.go.
4. Done (branch features-c): CSV export from the rollups (API, ctl, console). Left: Parquet, and a scheduled push to S3/SFTP. Usage export (S). CSV/Parquet by org/team/project/model/period from the rollups (internal/store/usageseries.go), GET /api/v1/usage/export, optional scheduled push to S3/SFTP.
5. Done (branch features-c): notify.webhooks (model parked, node lost, budget spent, update available; signed JSON or Slack; retries, hourly de-duplication), ctl notify test, Alertmanager receiver templates. Left: server init and the console do not yet ask for a receiver — the templates are filled in by hand; each control plane sends budget events on its own, so two behind a balancer may both send one. Alerting out of the box (S). deploy/monitoring/alertmanager.yml ships receiver: none; add Slack/email/webhook/PagerDuty receiver templates driven by server init or the console; llmgw-native webhooks for model parked, budget spent, update available (from the reconciler and modelserve).
6. Done (branch features-c2): metadata, x-llmgw-tags and user on the usage record (migration 0023), ctl request shows them, router.tag_labels metrics with 100 values per tag. Left: export and ctl usage by tag or end user (the hourly and daily sums do not carry tags; it would read the request records, within their retention). Request metadata and tags (S). Accept metadata/user from the body or x-llmgw-tags; store on the usage row; tenant metric labels behind an allowlist. router/handlers.go parse, accounting/recorder.go.
7. Done (branch features-c2): ctl model canary/promote/canaries, session-sticky split with x-llmgw-variant pins, rollback on error rate or TTFT, canary_rolled_back notification, console menu. Left: A/B between two different models (the split is between a model and its newer weights); the comparison is per control plane and lives in memory, so a restart begins it again; a key-level pin. Canary and weighted A/B between model revisions (M). control/modelserve/updates.go replace swaps every replica; add weighted routing across two revisions/aliases (scheduler filter, alias resolution in router/router.go:569), a header/key pin, automatic rollback on error-rate or TTFT regression from per-replica metrics.
8. Done (branch features-c2): /v1/files and /v1/batches (OpenAI's shape), worker under a store lease, lines kept as served, lowest band, usage tagged batch=<id>, output and error files with retention, no_store refused. Left: the Responses API (/v1/responses) as a batch endpoint; the console and ctl do not list batches (an operator sees them through usage tags); a file is one row in the store, so a store backup carries batch files with it. Batch API and Files (M–L). /v1/files, /v1/batches, JSONL storage, a worker feeding the bronze band, accounting. New internal/router/batch.
9. Done (branch features-c2): keys requests (retention, sample), kept after redaction, ctl request --body/--replay, console, pruning. Left: a streamed answer is kept as its text, not its chunks; replay is one request at a time (a set of them against a model is item 21's eval). Request/response logging and replay (M). Opt-in per project with retention like threads; sampled or full; redaction hooks from B; ctl request <id> --body; replay against another model.
10. Done (branch features-c2): Responses text.format, input_image, previous_response_id (thread in the id); rerank and score passthrough; transcription and translation as multipart; /v1/moderations by the judge. Left: audio is charged no tokens (an engine's duration usage is not read); input_image by file_id; moderation scores are 0 or 1, as a judge says yes or no. OpenAI compatibility gaps (S–M each). Responses: text.format → response_format, input_image, previous_response_id via threads (router/responses.go). Rerank/score passthrough for vLLM, /v1/audio/transcriptions for a whisper engine, /v1/moderations mapped to the guard model.
11. Done (branch features-c2): /v1/messages and count_tokens, streamed and not, tools, images, x-api-key, Anthropic error shape. Left: extended thinking (a reasoning model's thoughts are not mapped to thinking blocks); prompt caching; the account page's "Connect your editor" does not offer Claude Code yet. Anthropic /v1/messages (M). Translate like responses.go.
12. Done (branch features-c2): server restore (dry run, version check, decryption, modes and owners, SQLite and PostgreSQL), backup_copy_to (directory or ssh), the drill in docs/install.md. Left: restoring a replicated store's members (restore into the primary; Patroni replicates it); the console's Back up now does not offer restore; an ssh copy holds the archive in memory while it sends it. server restore (S). Restore is manual shell today (docs/install.md:1187); add the command with version check, dry run, off-host copy target; a documented DR drill.
13. Done (branch features-c2): TPOT, inter-token and tokens/s histograms by model, pool and tenant; slo pressure in admission's saturation; llmgw_model_slo_seconds and llmgw-slo alerts. Left: the p99 is per control plane, from its own requests; an SLO per band or per tenant; placement does not yet add replicas for an SLO (item 14). SLO-based admission and token-level metrics (S–M). TPOT and inter-token histograms, output tokens/s per model and tenant in router/metrics.go; use catalog.SLO in admission/saturation.go.
14. Done (branch features-c2): scaling.schedule (days, from, to, min, max; past midnight), scaling.prewarm, control.timezone; queue-depth scaling was there already. Left: ctl scale sets min and max only (a schedule is set in the definition); scaling for an SLO rather than for queue depth. Scheduled and warm-pool autoscaling (S–M). Time-of-day min/max, pre-warm, queue-depth scaling in catalog.Scaling, control/reconciler/demand.go, headroom.go.
15. Done (branch features-c2): env: and vault: references wherever a secret file is named (store address, api_key_file, OIDC, SCIM, webhooks), Vault KV with token or AppRole, secrets.master_key sealing cluster_secrets (Vault transit or a local key). Left: the certificate authority's key is still a file in /etc/llmgw (signing in Vault's PKI engine instead is its own change); rotating the master key; backup_copy_key and admin_token_file are files only. Secrets management (S–M). Vault/KMS for api_key_file, the DSN, the CA key, cluster_secrets (store/secrets.go, store/seal.go); envelope-encrypt them in backups (with A8 backups).
16. Done (branch features-c2): users forget (pseudonymised usage and sums, kept requests, files, batches, limits and audit names), keys residency --in-cluster (asked-for, fallback and canary). Left: an export of everything held about a person (article 15); residency by region between several clusters; backups keep what was forgotten until they age out. GDPR and residency (S). "Forget user" anonymises user_id in usage and rollups and erases threads/memory (store/hierarchy.go:241 leaves it today); per-project "never route to an External provider".
17. Done (branch features-c2): exact-match answer cache per project (keys cache --ttl/--any/--off, console Cache answers…, projects columns, 30-second setting cache), reproducible answers only unless --any, LRU in memory bounded by router.cache.max_bytes and max_entry_bytes, streamed answers kept and replayed as chunks, x-llmgw-cache and Cache-Control: no-cache, guardrails before a hit and the output policy on it, no_store and threads bypass, a hit recorded as cache_hit with no tokens and no cost (migration 0027), metrics, docs. Left: semantic (embedding-similarity) caching through the embeddings models; a cache shared across control planes (each process keeps its own, deliberately: a shared one would write every answer to the store); a hit still waits its turn in admission, because the guardrails judge the request after admission and the key is the request they let through; logprobs in a streamed answer leave it unkept; users forget does not reach into the cache (answers age out by the TTL); no listing or flushing of what is cached (--off stops it being served at once, the memory is freed by TTL and LRU). Response/semantic cache (M). Exact-match first, per project, opt-in, TTL; semantic via the embeddings models later. Before dispatch in router/handlers.go.
18. Done (branch features-c2): offline bundles — llmgw bundle create (releases with signed checksums, every llama.cpp build per arch/variant with GitHub's checksums, starter/judge and named Hugging Face model files pinned and checked; directory or tar; manifest sha256), bundle import (checked, all-or-nothing, into the mirror under _llmgw/), bundle trust, bundle show --verify; offline.mirror/offline.checksums in server.yaml, with nodes installing llama.cpp from the mirror against checksums the control plane sends over the agent stream (proto EngineWant.offline_*), releases defaulting to the mirror for node add, ctl upgrade and agent upgrades, update checks off; server init --bundle and the setup form's bundle fields; docs/data.md's offline claim corrected; install.md "Offline sites". A Helm chart for the control plane (deploy/helm/llmgw, one replica, SQLite PVC or external PostgreSQL, Secrets, Services, Ingress, ServiceMonitor) with deploy/container/Dockerfile and docs/kubernetes.md; GPU nodes stay outside Kubernetes by design. Left: vLLM/SGLang offline (a wheelhouse of several GB per CUDA build and a uv --find-links install path); the NVML release binary and uv in bundles; the setup wizard's preflight still fails a machine without internet; join's printed curl from the mirror checks nothing; models in a bundle are single files (no whole- repository safetensors sets yet); the chart runs one replica (a standby is never ready, so HA in Kubernetes needs a different readiness design), has not been installed on a real cluster, and ships no agent DaemonSet; no published container image. Kubernetes/Helm and an air-gapped bundle (M). deploy/, agent/provision; docs/data.md claims offline operation but install assumes the internet.
19. Done (branch features-c2): adminapi.Routes and internal/docgen/openapi generating docs/admin-api.openapi.json (3.1, bearer security, every route, query parameters, schemas from the Go types with their comments), served embedded at GET /api/v1/openapi.json, kept true by a route-table test and a staleness test; validated with openapi-spec-validator and openapi-generator-cli. GET /api/v1/models/{name}. docs/admin-api.md. terraform-provider- llmgw/ (terraform-plugin-framework): llmgw_model, llmgw_quota, llmgw_guard_attachment, data sources llmgw_models and llmgw_nodes; client tests against a fake admin API and the OpenAPI document; acceptance tests with Terraform and OpenTofu against the fake; one apply, empty plan, in-place change and destroy against a real llmgw server run by hand. Left: organizations, teams, projects, people and keys are not in the admin API (they are llmgw org/users/keys against the store), so not in the provider; a model's definition beyond scaling, class, owner and adapters is changed by replacing it (the API has no edit, and resources is not used yet); canary, slo, scaling.schedule and prewarm, guard role and origin are not in llmgw_model; models cannot be imported; the provider is not published to a registry and has no release build; the acceptance tests are not run in CI (they need Terraform); the document's schemas mark nothing required and give no enums; catalog.SLO's fields have no JSON tags, so they are TTFTp99 and TPOTp99 on the wire. OpenAPI for the admin API, Terraform provider (M). Generate from adminapi/api.go routes via docgen.
20. Done (branch features-c2): spans for the reconcile pass (begun only when a pass does something), placement with its answer or rejections, scale-up/down, drain, stop, eviction, preemption, reaping, restarts and lost nodes; a replica start span from placement to serving whose context travels to the agent in the start command (ControlMessage.trace), where weights fetch, engine launch, rendezvous and readiness wait continue the trace; stops, drains, adoption and engine installs; modelserve jobs as a trace with a span per step linked to the replica's start; an agent tracing block; docs in install.md's "Tracing". Left: the router's span for a request does not link to the replica it was sent to's start; heartbeats, fabric scans, logs and agent upgrades are not traced; the supervisor has no restart span because it never restarts an engine itself (a restart is the reconciler's restart replica and reaches a node as a stop and a start); canary and promote jobs, which landed on features-c2 after this was branched, are traced only through the shared step recording (as <kind> model), unchecked by a test. Tracing beyond the router (S). Spans for reconciler, placement, agent start (obs/trace.go, control/reconciler, agent/supervisor).
21. Done (branch features-c2): internal/bench/eval (suites in YAML or JSON lines; exact, contains, regex, number, a JSON Schema subset, and a judge model with a rubric; weights and a threshold), run on the control plane through Router.Replay; llmgw ctl eval MODEL --suite FILE|NAME, --history, --run, --suites; runs and their cases kept (migration 0028, eval_runs and eval_cases); model update/promote --eval SUITE --min-score N, a model's standing gate (ctl model gate, eval: in the catalog) — a promote asks the canary first, an update tries the new weights as a 0% canary and promotes or removes it; the console's model menu; arithmetic and json-format in deploy/eval/, carried in the binary. Left: the suite of a model's standing gate is read by name on the control plane at the time of the update, so two control planes need the same control.eval_dir; a gated update needs room for one more replica while it runs (there is no in-place gate, and a declared model in server.yaml cannot be updated at all); no comparison against the old weights' score (the gate is an absolute threshold, not "no worse than before"); a run is not stopped once started (Esc in the console only stops watching it), and the in-memory progress of a run lives on the control plane that started it; runs are never pruned; the JSON Schema support is a subset (no $ref, oneOf, not, uniqueItems); judge cases go through the same concurrency as the model's, and a judge's answers are not kept, only its score and reason; a gateway without a control plane cannot run suites (there is no mode that asks through the client API with a key). Eval harness (M). llmgw eval extending internal/bench with quality suites gating model update.
22. Done (branch features-c2): MIG instances discovered on both paths (nvidia-smi -L + -q -x; NVML through an interface with a faithful fake) and reported instead of their card, each with its MIG UUID, own memory and class <card class>-mig-<profile>; device numbers 1000 + 100 × card + instance, shown as card/migN; placement on MIG classes; tp/pp/gpus/nodes above 1 on a MIG class refused in catalog validation and again in placement; CUDA_VISIBLE_DEVICES=MIG-<uuid> for vLLM and llama.cpp (no Vulkan device variable on MIG); per-instance memory and processes in heartbeats and the GPU check, card sensors under each instance, utilisation absent rather than zero; doctor MIG findings (mig -lgi/-lci parsed); ctl nodes, console, ctl status, ctl model plan/serve by class; docs/install.md MIG section. Written and tested against fakes only — untested on real MIG hardware. Left: run it on an A100/H100 in MIG mode (check -L/-q -x/mig -lgi/-lci and NVML MIG device names against the parsers, and that CUDA_VISIBLE_DEVICES=MIG-… reaches vLLM's workers through Ray); instances changed while the agent runs are only seen when it restarts (no rescan); llmgw does not create or reshape instances (no mig-parted integration); quotas count an instance as one GPU, the same as a whole card; SGLang on MIG not checked; device numbers assume fewer than 100 instances per card and that instance numbers stay put while shards run on them. MIG (M–L). Treat a MIG slice as a GPU class in agent/discovery/gpu_nvml.go and control/placement.
23. Done (branch features-c2): engine trtllm (internal/agent/engines/trtllm.go): trtllm-serve through a launcher of llmgw's that demands the replica's key (trtllm-serve has none) and writes the options file that turns on /prometheus/metrics and the iteration statistics; the scrape contract for it (scrape.TRTLLMMapping, with a per-engine metrics path); found, not installed — engines.trtllm.python in agent.yaml, probed from the package's metadata; catalog validation (GPU class, one node, no GGUF source, the flags a profile may not give); ctl engine-check trtllm, which asks trtllm-serve serve --help through its module. Ollama/GGUF import: mirror import-ollama (manifest parsing, blob digest check, projector beside the model, all or nothing, a model definition printed), mirror import for any local file, and the console's I on the Mirror view. Left: TensorRT-LLM has not run on real hardware — the first node with one should run engine-check and serve a model before anyone relies on it; no installer for it; no multi-node TensorRT-LLM replica (it needs an MPI or Slurm launch from one node); Triton Inference Server as an engine is not done; trtllm's queue gauges are set per iteration, so an idle engine shows its last batch until the next request; --num_serve_frontends is refused because extra front ends would not demand the key. The Ollama import runs on the mirror machine only — the mirror's API takes no files, so an Ollama elsewhere means copying the blob or running the import there; Ollama models kept as separate tensors or as several weight layers, and its LoRA adapters, are refused or left with a sentence; files an import run with sudo writes are root's, as mirror add's are. More engines (M each). TensorRT-LLM/Triton adapter; Ollama/GGUF import. agent/engines/engine.go interface and the metric contract in router/scrape.
24. Lower: precise KV-index routing (plan phase 6), LoRA affinity for vLLM, multi-cluster federation via External providers, facility-aware placement (docs/facility-aware-placement.md), portal i18n, SDK/docs portal. Done so far (branch features-c2):
    • 24 (portal i18n): built — English/Turkish catalogue with fallback, per- language templates, cookie/Accept-Language/landing.language choice and a footer switcher, Turkish numbers and dates, every template and handler message translated, tests that no template text stays English; left — errors built with fmt.Errorf in other packages (store, catalog, mirror, adminapi) still English on Turkish pages, percent sign placement is mixed (%50 on limits, 93% on cluster pages), RFC3339 footer timestamps and plain-text http.Error bodies untranslated, the /docs documents themselves English, a per-person language column (cookie only today).
    • 24 (SDK/docs portal): built — /docs renders the embedded manual (goldmark, GFM, heading anchors, links rewritten, no script; public, landing.docs_sign_in to require a session), and docs/clients.md shows the official OpenAI/Anthropic SDKs against llmgw. Left: the console's generated "Commands" page is not in the portal (it is built from the cobra tree, which internal/web cannot see); no search across documents.

---

D. Facts worth knowing before starting

• Admin API: 127.0.0.1:9090, no auth, ctlClient in internal/cli/ctl.go. Metrics on :9091 with /prometheus/targets beside them.
• Store: internal/store, migrations under internal/store/migrations/{sqlite,postgres}; the PoC runs PostgreSQL under Patroni.
• The console (sudo llmgw) is internal/cli/tui_console.go with actions in tui_actions.go; every operator-visible change needs a row action or a form there, and a line in docs/console.md.
• Tests of the console use a fake admin API (internal/cli/tui_actions_test.go, adminStub); tests of serving use fakes in internal/control/modelserve/modelserve_test.go.
• go generate ./... regenerates docs/reference.md and internal/docgen/reference_gen.go whenever a config struct changes; the docgen test fails otherwise.