llmgw/BACKLOG.md

# Backlog

What a review of the code on 2026-10-04 (commit c2238ab) found should be done
next: security fixes first, then guardrails, then features. Written for
whoever implements it — a person or a model — without the review in hand.
Every item names the files it touches and what "done" is. Paths are relative
to the repository root. Sizes: S about a week, M two to four weeks, L more.

Conventions this codebase keeps, which every item here must keep too:
- A change is refused before anything is written (validate, then persist).
- Nothing on the serving path logs prompt text, keys or passwords.
- Each item gets unit tests beside the code (`*_test.go`), and the TUI
  (`internal/cli/tui_*.go`), `docs/console.md` and `CHANGELOG.md` are
  updated when behaviour an operator sees changes.
- `go test ./...`, `go vet ./...` and `govulncheck ./...` pass. No release is
  cut and nothing is deployed to the PoC cluster without being told to.

---

## A. Security — do first

**Status (2026-10-04): done and merged.** A1–A9, A6b included — etcd and
Patroni's API on TLS, the PoC moved in maintenance mode without a failover.
Deliberately narrower than written: the admin API is a token on loopback
rather than a unix socket (A5); thread ids are bounded and documented as
project-wide rather than scoped per user (A8); Python engines resolve with an
explicit PyTorch index and a PEP 751 lock rather than locks shipped per engine
version (A8). All of A is live on the PoC.

### A1. Toolchain (S, hours)
`govulncheck` finds 11 standard-library vulnerabilities reachable from this
code at Go 1.26.3 (net/http, crypto/x509, os.Root, net/textproto). Raise the
`go` directive in `go.mod` to 1.26.6 or later, rebuild, run `govulncheck
./...` until it reports none reachable. Add `govulncheck` to the Makefile's
check target.

### A2. Release signing is effectively off (S)
- `internal/release/trusted.pub` is empty, so `TrustedKey()` returns "" and
  `release.go:103` skips signature verification. Ship a real public key;
  make a missing or invalid signature fatal. Drop the `LLMGW_RELEASE_KEY`
  environment override, or honour it only in `-tags dev` builds.
- `internal/onboard/onboard.go:365`: the node-side download checks only a
  same-origin `checksums.txt`. Verify the signature there too (simplest:
  have the node run `llmgw` itself to fetch and verify once it has any
  binary), and `shellQuote` the `Releases` URL, which is interpolated into
  single quotes unescaped.
- `internal/agent/upgrade.go:44`: accept only `https://` release bases.
- Done when: a tampered binary or checksums file is refused by self-upgrade,
  node onboarding and agent upgrade, with a test for each.

### A3. Agent port without TLS is anonymous and plaintext (S)
`internal/cli/server.go:423` serves h2c on `:7443` when `agent_tls` is not
configured; `agentsvc` then accepts any node id. Refuse to start control
mode without TLS unless `agent_tls.insecure: true` is set explicitly; when
insecure, bind `agent_listen` to 127.0.0.1 and log a warning at startup and
in `llmgw doctor`. Keep `server init` writing TLS as it does.

### A4. Model profiles reach root on every node (M)
Profile `args` and `env` from a model definition are appended to the engine
command unchecked (`internal/agent/engines/llamacpp.go:170`, `vllm.go:151`,
`sglang.go:98`); `internal/agent/engines/unsafeenv.go` is a denylist that
misses `NCCL_NET_PLUGIN`, `NCCL_TUNER_PLUGIN`, `NCCL_PROFILER_PLUGIN`,
`CUDA_INJECTION64_PATH`, `NVTX_INJECTION64_PATH`, `TRITON_*` paths; engines
run as root.
- Replace the env denylist with an allowlist (CUDA_/NCCL_ tuning keys minus
  anything ending in `_PLUGIN` or containing `INJECTION`, `VLLM_*`,
  `HF_*` minus path keys, `OMP_*`, `GGML_*`). Apply at catalog validation
  (`internal/control/catalog/catalog.go`, where `UnsafeEnv` is called) and
  again in the agent.
- Deny engine flags that write files or run code: `--trust-remote-code`,
  `--log-file`, `--slot-save-path`, `--chat-template` with a path,
  `--host`, `--api-key`, `--port`; vLLM/SGLang equivalents. Allowlist is
  better than denylist if the flag set is enumerable per engine.
- Run engines as an unprivileged user with the video/render groups
  (`internal/agent/supervisor`, systemd unit in
  `internal/cli/serverinit_cluster.go:52`). Weights cache must be readable
  by it.
- Done when: a profile with `--trust-remote-code` or `NCCL_NET_PLUGIN` is
  refused at `ctl model add` and at the agent; `ps` shows engines not as
  root.

### A5. Admin API trusts any local process (M)
`internal/cli/server.go:357` serves the admin API and pprof on
`127.0.0.1:9090` with no authentication; `adminapi.actorOf`
(`internal/control/adminapi/api.go:1613`) takes the actor from the
client-sent `X-Llmgw-Actor` header.
- Serve the admin API on a unix socket (`/run/llmgw/admin.sock`, 0660,
  group `llmgw-admin`) by default, keeping TCP as an option that requires
  a bearer token read from a root-only file (`/etc/llmgw/secrets/admin-token`).
- Take the actor from `SO_PEERCRED` (uid → username) on the socket, or from
  the token's name; ignore the header.
- Refuse `admin_listen` on a non-loopback address unless `admin_insecure:
  true`.
- Put pprof behind its own `debug.pprof: true` setting, off by default.
- `ctlClient` (`internal/cli/ctl.go`), the console and `node add` must work
  over the socket; `sudo llmgw` already runs as root.

### A6. Patroni REST and etcd are unauthenticated and plaintext (M)
`internal/pgha/config.go:99` writes a `restapi:` with no `authentication`
and `:54` etcd over `http://` with no client auth; `hba.go:56` PATCHes
Patroni's config unauthenticated. Anyone reaching :8008 or :2379 owns the
store.
- Generate a Patroni REST username/password at setup, store it in
  `/etc/llmgw/secrets/`, write `restapi.authentication` and
  `restapi.certfile/cafile` from the cluster CA; send it from
  `internal/pgha/hba.go` and `status.go`.
- etcd: client and peer TLS from the cluster CA with `--client-cert-auth`.
- Document, and have `server init` write, nftables rules limiting
  2379/2380/8008/5432 to member addresses (see A9).
- Done when: an unauthenticated `curl :8008/config` is refused and the
  store status in the console still works.

### A7. PostgreSQL traffic is plaintext (S)
`internal/pgha/config.go:168` and `internal/cli/serverinit_postgres.go:330`
use `sslmode=disable`; HBA lines are `host`. Enable server TLS in Patroni's
postgresql parameters with a certificate from the cluster CA, write
`hostssl` lines, use `sslmode=verify-full&sslrootcert=/etc/llmgw/ca.pem`.
Existing clusters: `llmgw server` must keep working against a store that
has not been migrated, with a warning.

### A8. Smaller fixes, each S or less
- **Capacity fallback skips the allowlist.** `internal/router/router.go:700`
  builds `[canonical]+entry.Fallbacks` without `permits`. Filter the chain
  with `r.permits(p, model)`; price against the model that served.
- **Suspended users keep working API keys.** `store.(*DB).Resolve`
  (`internal/store/memberships.go`) never joins `users.state`;
  `Authenticator.Invalidate()` is never called. Reject keys whose owner is
  not active; call `Invalidate` on revoke, suspend, limit change.
- **Join codes never expire server-side.** `internal/join/join.go:103`
  checks expiry only in `Parse`; `admit` (`:212`) does not. Check it there;
  bind the listener to the chosen address, not `:7444` on all interfaces;
  use the peer's address rather than the claimed `hello.Addr`, and show the
  joiner's address for the operator to confirm before secrets are sent.
- **Leftover sudoers grant.** `internal/onboard/gossh.go:248` writes
  `NOPASSWD:ALL` and removes it only in `Close()` (`:271`). Write the grant
  with a self-removal (`at`/systemd-run timer, 30 min) so a crash leaves
  nothing; wipe password buffers after use. In `ssh.go:39` pass `--` before
  the target.
- **Auth cache and pre-auth DoS.** `internal/auth/auth.go:127`: unbounded
  map, one DB query per bogus well-formed key. Bounded LRU, negative cache
  with short TTL, per-IP token bucket before the DB.
- **HTTP server hardening.** `server.go:351-360`, the h2c agent server,
  both mirror servers: set `ReadHeaderTimeout` (exists), `IdleTimeout`,
  `MaxHeaderBytes`, a body-read deadline, `netutil.LimitListener`; connect
  `WithReadMaxBytes` on the agent stream.
- **Login/signup abuse.** Default `signup_open: false`; per-IP token buckets
  on `/login` and `/signup`; bounded wait for argon2 slots; CSRF token on
  the login form; the per-address throttle must not lock the real owner
  out (use IP+address pairs).
- **Trusted proxies.** `internal/web/account.go:557`, `landing.go:60` honour
  `X-Forwarded-Proto/Host` from anyone; add `router.trusted_proxies` and
  only then trust them; `Cache-Control: no-store` on `/setup/*`.
- **Certificate revocation.** `DELETE /api/v1/nodes/{id}` only drops
  inventory; add a revoked-CN/serial list in the store checked in
  `agentsvc` `Connect` and mirror `onlyClients`; `serverinit_cluster`
  always passes `--api-client`.
- **Engine exposure.** Engines bind `0.0.0.0` (`llamacpp.go:102`,
  `vllm.go:109`): bind the management address. SGLang receives its key on
  the command line (`sglang.go:102`): pass it by env or file.
- **Backups.** `internal/cli/backup.go`: encrypt archives to an `age`
  recipient from `backup.recipient` in server.yaml; keep the directory 0700.
- **Python dependency confusion.** `internal/agent/provision/python.go:121`
  uses `--extra-index-url` with `unsafe-best-match`. Pin torch to the
  PyTorch index explicitly (uv `[[index]] explicit = true`), `first-index`,
  hash-pinned lockfiles per engine version.
- **Control-plane SSRF.** `internal/sourcecheck/sourcecheck.go:39` and
  `gguf.FromURL` use the default client; reuse the mirror's
  `fetchClient(false)` (private-address filter) and refuse a proxy unless
  configured; block NAT64/6to4-embedded private addresses in
  `internal/mirror/safefetch.go`.
- **Low:** `agentsvc/service.go:430` indexes `n.GPUs[0]` without a length
  check (CPU-only node panics the handler); thread ids are client-chosen
  with no length cap and shared per project (`router/threads.go`) — cap
  length, document, offer per-user scoping; `/v1/models` lists models the
  key may not use (`handlers.go:799`); sessions have no idle timeout and
  `PurgeSessions`/`PurgeClaims` are never called — add a janitor; add a CSP
  header and set `htmx.config.allowEval=false`; cap GGUF array nesting
  depth in `internal/gguf/gguf.go`; mirror `.source` sidecars are served by
  the public file server — exclude dotfiles from `:8081`; prune
  `modelserve.jobs` and the mirror `running` map; KaTeX → 0.16.21+.

### A9. Firewall (S)
No machine in a default install has a firewall. Have `server init`, `node
add` and `server add` write an nftables ruleset: ssh open; `:8080` open;
`:7443` and `:9091` open to the cluster's networks; `:9100` to the
monitoring host; `:2379/2380/8008/5432` to store members and control
planes; mirror `:8081` to the cluster, `:8082` to control planes; `:9090`
Prometheus and `:3000` Grafana to operators. `llmgw doctor` reports a
missing ruleset. Make it skippable with `--no-firewall`.

---

## B. Guardrails (M–L, build in this order)

**Status (2026-10-04, branch `guardrails-b`):** B1–B4 done, and tried live on
the PoC. That found four bugs, all fixed with tests:
- a shadow default merged with an attached policy of unwritten mode left it
  in shadow;
- a streamed word held back at one release was dropped at the next;
- llama.cpp continues a conversation ending on the answer, so Llama Guard's
  verdict on answers is now asked with its own prompt as a completion;
- a judge model was named in a client's "no endpoint" error.

The PoC runs `observe` (shadow) for everyone, and Llama Guard 3 1B
(`llama-guard-3-1b`, CPU, on cuda01) as a judge, which blocks a harmful
request in ~0.7s and a harmful answer in ~1.4s there. Still left: see each
item.

Today nothing on the request path reads, classifies, redacts or filters
message text in either direction (confirmed; the only "moderation" hit is a
comment in `internal/router/budget_fallback.go:20`). Existing controls are
about who, how much and how fast: allowlists (`router/handlers.go:450`),
RPM/TPM/budget (`internal/accounting`), context precheck
(`router/context_limit.go`), admission bands (`router/admission`),
timeouts (`router/proxy/proxy.go:26-57`). Gaps to close on the way: no
default output cap when the client omits `max_tokens`; `catalog.Capabilities`
is advertised but never enforced; tool results are trusted input.

### B1. Hooks in the proxy and router (S)
**Done (branch `guardrails-b`):** `internal/guard` holds the interface;
proxy `Filter`/`StreamFilter`; the input hook runs after `loadThread` (the
simpler choice); error codes, headers, Responses `incomplete`, usage columns
`guard_*`. Moved to B2: holding `delta.tool_calls` until they are whole and
reading their arguments — that is the tool policy's filter, not the proxy's.

- `internal/router/proxy/proxy.go`: add optional `InputRewrite func(body
  []byte) ([]byte, Verdict)` on the request and two output hooks: `Filter`
  for `relayBuffered` (between `ReadAll` and `WriteHeader`) and
  `StreamFilter{OnEvent(data) (emit []string, stop bool); OnEnd()}` for
  `relayStream`, called where each whole SSE event is assembled
  (`eventBuffer`, before `io.WriteString` at `:343`). Nil hooks cost
  nothing.
- `internal/router/handlers.go`: call the input policy after the allowlist
  and limits and before admission (around `:395`); the principal, scopes
  and the model that will serve are known there. Thread history is spliced
  later (`:416`): scan stored turns when written, or move the hook after
  `loadThread` (decide; the second is simpler).
- Blocking conventions: input → 400 `invalid_request_error` with
  `code: "content_policy_violation"` (today `writeError` always sends
  `code: null`); output non-stream → 200 with `finish_reason:
  "content_filter"` and empty/redacted content; output stream → a final
  chunk `{"choices":[{"index":0,"delta":{},"finish_reason":"content_filter"}]}`,
  then `data: [DONE]`, then cancel upstream. Headers `x-llmgw-guard:
  blocked|redacted|flagged` and `x-llmgw-guard-policy` set before the proxy
  writes headers.
- `internal/router/responses.go`: map `content_filter` to
  `response.incomplete` with `incomplete_details.reason: "content_filter"`
  (today it always emits `response.completed`, `:525`).
- Tool calls: hold `delta.tool_calls` until `finish_reason: "tool_calls"`,
  then validate; `replyDelta` (`reply.go:33`) must also read
  `tool_calls[].function.arguments`.
- Accounting: a blocked input is recorded like `RecordRefused` (nothing
  charged); a blocked output is charged (the GPU was used). `store.Usage`
  gains `GuardAction`, `GuardPolicy`, `GuardCategory`, `GuardMs`.

### B2. Package `internal/guard` (M)
**Mostly done (branch `guardrails-b`):** policies in server.yaml with
attachments in the store (`ctl guard attach`), narrowed by the limits' rule;
`guard/detect` (validated PII, secrets, injection through normalisation and
decoding); redaction with consistent placeholders before prefix routing;
tool allow/deny/strip, call count and argument schemas; capabilities and
structured output (opt-in); default `max_tokens`; pinned system prompt and
dropping the client's; `no_store` (threads, notes, prefix tree, external
models); shadow mode. **Left:** a `guard_events` table — the usage row's
`guard_*` columns carry action/policy/category/time, not rule and stage;
language identification; the re-identification map that restores
redacted values in answers.

- `Policy`, `Effective` resolution over the tenancy tree with the same
  narrowing rule as limits (`internal/store/sqlite.go:383-416` `tighten`):
  deny sets union, allow sets intersect, `mode` takes the strictest of
  `enforce > shadow > off`, `fail` takes `closed > open`, `no_store` at any
  level wins, thresholds take the minimum.
- Store: `guard_policies(scope, scope_id, policy json)` and
  `guard_events(request_id, scopes, policy, rule, category, action, stage,
  score, judge_ms, shadow)` — ids, categories, offsets and lengths only,
  never text. Migrations for sqlite and postgres. Pruned with
  `usage_retention`.
- `guard/detect`: regex/keyword sets; PII with validators — email, phone,
  IBAN (mod-97), card (Luhn), TCKN (checksum), IP, secrets (`sk-`, AWS,
  JWT, PEM); normaliser (NFKC, strip zero-width/bidi, homoglyph fold,
  base64/hex/URL decode to depth 2); injection heuristics (role-switch
  phrases, fake chat-template tokens, long base64, invisible Unicode) —
  default action `flag`; language id.
- Redaction rewrites `messages[].content` in the upstream body with
  placeholders (`[EMAIL_1]`) via the same `map[string]json.RawMessage`
  round-trip `withModel`/`spliceThread` use; an optional re-identification
  map lives only in request memory and restores originals in the output.
  Redact before prefix routing so `PromptPrefix` hashes the redacted text.
- `guard/toolpolicy`: tool name allow/deny (strip or 400), JSON-Schema
  validation of assembled arguments against the declared `parameters`,
  argument regex rules, `max_calls_per_turn`. Enforce
  `Capabilities.Tools`/`Vision` here: a `tools` array or image parts to a
  model without the capability → 400.
- `guard/structured`: validate `response_format.json_schema` on the final
  text → `content_filter`, or flag in shadow.
- Output constraints: inject a default `max_tokens` when absent (from the
  policy or `MaxOutput`), language check, structured check.
- System-prompt pinning per project: prepend before `checkContextFits` and
  before `spliceThread`; optionally drop client `system`/`developer`
  roles; exclude the pin from stored thread turns; apply to Responses
  `instructions` in `toChat`.
- `no_store`: `loadThread` and `PutNote` refuse; `captureBytes()` returns 0;
  skip the prefix-tree observe (`router.go:840`); disable External
  fallbacks for the scope (filter in `dispatch` and `budgetFallback`).

### B3. Judge model on the cluster (M)
**Done (branch `guardrails-b`):** `role: guard` + `guard.format`; hidden and
refused to clients and fallbacks; critical by default, never on demand;
`Router.Judge` past admission/accounting/guardrails; llamaguard3,
shieldgemma and json parsers; 5-minute cache; latency budget then
fail open/closed; streamed answers read every 600 characters, or held with
buffer_all. Simpler than written: the input judge runs after admission, not
alongside it, and shadow mode judges synchronously within the budget rather
than after the response. Not enforced: two replicas on different nodes —
documented, left to `scaling.min`.

- `catalog.Model.Role: "guard"` plus `Guard{Format: llamaguard3 |
  shieldgemma | json, Categories}`. Guard models are hidden from
  `/v1/models`, the chat picker and allowlists; placed like any model,
  class `critical`, `scaling.min ≥ 1`, at least 2 replicas on different
  nodes.
- `guard/judge`: calls the guard model through the router's own `dispatch`
  with an internal principal and its own admission band (never the
  tenant's queue); parsers per format (Llama Guard `safe`/`unsafe\nS1,S10`;
  ShieldGemma yes/no probability → per-tenant thresholds; generic JSON
  verdict); verdict cache keyed by hash of normalised new turns (5 min);
  `latency_budget` (default 250 ms) then `fail` open/closed; input judge
  runs in parallel with admission queueing.
- Streaming strategy: default sentence-buffered scanning (regex/PII per
  sentence or ~200 chars before release; judge every 400–800 chars on the
  cumulative tail; final judge at end); `buffer_all` for `fail: closed`
  scopes. Document that released text cannot be retracted.
- Shadow mode runs everything asynchronously after the response
  (`context.WithoutCancel`, like `saveThread`) and only records.

### B4. Config, metrics, UI (S)
**Done (branch `guardrails-b`):** library in server.yaml, attachments via
`ctl guard`, the console's **Guardrails…** with "applies from" marks, and
the portal's `/admin/guard`; the metrics and alerts below (judge latency
labelled by model only at the judge; the check latency by stage); `ctl
request` shows the verdict; the chat page says "stopped by policy"; the
profile lists effective policies; docs/guardrails.md and data.md written;
the access log carries action, stage and policy only. Attaching from the
portal is not done — the page shows; the console and ctl change.

- Gateway-level policy library in `server.yaml` under `guard:` (see the
  YAML in the review, reproduced in `docs/guardrails.md` to be written);
  attachments to scopes in the store via `llmgw keys guard`, the console
  ("Guardrails…" beside "Limits…" in `internal/cli/tui_people.go`, with
  "inherited from team:X" badges) and the portal.
- Metrics in `router/metrics.go` style:
  `llmgw_guard_checks_total{stage,policy,action,category,mode}`,
  `llmgw_guard_latency_seconds{stage,judge_model}`,
  `llmgw_guard_judge_errors_total{reason}`, `llmgw_guard_fail_open_total`,
  `llmgw_guard_stream_stops_total`, `llmgw_guard_redactions_total{kind}`,
  `llmgw_tenant_guard_blocks_total{tenant}`. Alerts: fail-open > 1% for 5m,
  judge p95 over budget, judge has no ready replicas, per-tenant block rate
  > 3× its 7-day baseline, any `secret` detected in output.
- Portal `/admin/guard`: violations by tenant/category/policy over time,
  shadow vs enforce, judge health, recent events by request id; `ctl
  request <id>` shows the verdict; chat page renders `content_filter` as
  "stopped by policy X"; `/me` lists effective policies.
- Update `docs/data.md`: the judge sees all text but in-cluster; External
  fallbacks send prompts outside the cluster (the current "nothing leaves
  the cluster" is wrong); `no_store` semantics.
- Never log: prompt/completion text, PII values, the re-identification
  map, judge inputs or rationale that quotes input. Check `accessLog`'s
  `err` field and span errors.

---

## C. Features, ranked by value × feasibility

1. **Done (branch `features-c`):** `audit_events`, written by the admin
   API middleware and the /admin pages (with each change's result), `ctl
   audit`, the console's Audit view; staff roles viewer / operator /
   billing / admin. Left: before/after values (the request's fields are
   recorded, not the state they replaced); a tenant-admin role that changes
   its own organization (managers still only view it); hiding the forms a
   viewer cannot submit. **Audit log and roles (M).** Today "audit" is slog lines
   (`internal/web/admin_users.go:403-480`) and roles are the `IsAdmin` flag
   plus an org "manager" who can view. Add `audit_events` (who, what,
   before/after, source) written by the `adminAction`/`tenancyAction`
   wrappers and by every admin-API mutation; roles viewer / operator /
   tenant-admin / billing; `ctl audit`. Depends on A5 for a real actor.
2. **Done (branch `features-c`):** OIDC sign-in (`auth.oidc`, PKCE,
   verified email, roles and memberships from groups, `no_passwords`) and
   SCIM 2.0 (`/scim/v2` Users and Groups). Left: the sign-in cookie is
   signed with a per-process key, so with two control planes behind a
   balancer the callback must reach the one that started it (sticky
   sessions) until the key comes from the store. **OIDC/SSO login (M), then SCIM (M).** `internal/account/service.go`,
   `internal/web/account.go`; map IdP groups to org/team/admin.
3. **Done (branch `features-c`):** key lifetime limits (gateway, org, team,
   project), `keys rotate --overlap`, kinds and expiry in `keys list`.
   **Service accounts and key lifecycle (S).** Per-org maximum key TTL,
   `keys rotate` with overlap, last-used shown, project-owned service keys
   listed apart. `internal/cli/keys.go`, `internal/store/keys.go`,
   `internal/auth/auth.go`.
4. **Done (branch `features-c`):** CSV export from the rollups (API, ctl,
   console). Left: Parquet, and a scheduled push to S3/SFTP. **Usage export (S).** CSV/Parquet by org/team/project/model/period from
   the rollups (`internal/store/usageseries.go`), `GET
   /api/v1/usage/export`, optional scheduled push to S3/SFTP.
5. **Done (branch `features-c`):** `notify.webhooks` (model parked, node
   lost, budget spent, update available; signed JSON or Slack; retries,
   hourly de-duplication), `ctl notify test`, Alertmanager receiver
   templates. Left: `server init` and the console do not yet ask for a
   receiver — the templates are filled in by hand; each control plane sends
   budget events on its own, so two behind a balancer may both send one.
   **Alerting out of the box (S).** `deploy/monitoring/alertmanager.yml`
   ships `receiver: none`; add Slack/email/webhook/PagerDuty receiver
   templates driven by `server init` or the console; llmgw-native webhooks
   for model parked, budget spent, update available (from the reconciler
   and `modelserve`).
6. **Done (branch `features-c2`):** `metadata`, `x-llmgw-tags` and `user`
   on the usage record (migration 0023), `ctl request` shows them,
   `router.tag_labels` metrics with 100 values per tag. Left: export and
   `ctl usage` by tag or end user (the hourly and daily sums do not carry
   tags; it would read the request records, within their retention).
   **Request metadata and tags (S).** Accept `metadata`/`user` from the
   body or `x-llmgw-tags`; store on the usage row; tenant metric labels
   behind an allowlist. `router/handlers.go` parse, `accounting/recorder.go`.
7. **Done (branch `features-c2`):** `ctl model canary/promote/canaries`,
   session-sticky split with `x-llmgw-variant` pins, rollback on error rate
   or TTFT, `canary_rolled_back` notification, console menu. Left: A/B
   between two different models (the split is between a model and its
   newer weights); the comparison is per control plane and lives in memory,
   so a restart begins it again; a key-level pin.
   **Canary and weighted A/B between model revisions (M).**
   `control/modelserve/updates.go` `replace` swaps every replica; add
   weighted routing across two revisions/aliases (scheduler filter, alias
   resolution in `router/router.go:569`), a header/key pin, automatic
   rollback on error-rate or TTFT regression from per-replica metrics.
8. **Done (branch `features-c2`):** `/v1/files` and `/v1/batches`
   (OpenAI's shape), worker under a store lease, lines kept as served,
   lowest band, usage tagged `batch=<id>`, output and error files with
   retention, `no_store` refused. Left: the Responses API (`/v1/responses`)
   as a batch endpoint; the console and `ctl` do not list batches (an
   operator sees them through usage tags); a file is one row in the store,
   so a store backup carries batch files with it.
   **Batch API and Files (M–L).** `/v1/files`, `/v1/batches`, JSONL
   storage, a worker feeding the bronze band, accounting. New
   `internal/router/batch`.
9. **Done (branch `features-c2`):** `keys requests` (retention, sample),
   kept after redaction, `ctl request --body/--replay`, console, pruning.
   Left: a streamed answer is kept as its text, not its chunks; replay is
   one request at a time (a set of them against a model is item 21's eval).
   **Request/response logging and replay (M).** Opt-in per project with
   retention like threads; sampled or full; redaction hooks from B;
   `ctl request <id> --body`; replay against another model.
10. **Done (branch `features-c2`):** Responses `text.format`,
    `input_image`, `previous_response_id` (thread in the id); rerank and
    score passthrough; transcription and translation as multipart;
    `/v1/moderations` by the judge. Left: audio is charged no tokens (an
    engine's duration usage is not read); `input_image` by `file_id`;
    moderation scores are 0 or 1, as a judge says yes or no.
    **OpenAI compatibility gaps (S–M each).** Responses: `text.format` →
    `response_format`, `input_image`, `previous_response_id` via threads
    (`router/responses.go`). Rerank/score passthrough for vLLM,
    `/v1/audio/transcriptions` for a whisper engine, `/v1/moderations`
    mapped to the guard model.
11. **Done (branch `features-c2`):** `/v1/messages` and `count_tokens`,
    streamed and not, tools, images, `x-api-key`, Anthropic error shape.
    Left: extended thinking (a reasoning model's thoughts are not mapped to
    thinking blocks); prompt caching; the account page's "Connect your
    editor" does not offer Claude Code yet.
    **Anthropic `/v1/messages` (M).** Translate like `responses.go`.
12. **Done (branch `features-c2`):** `server restore` (dry run, version
    check, decryption, modes and owners, SQLite and PostgreSQL),
    `backup_copy_to` (directory or ssh), the drill in docs/install.md.
    Left: restoring a replicated store's members (restore into the primary;
    Patroni replicates it); the console's Back up now does not offer
    restore; an ssh copy holds the archive in memory while it sends it.
    **`server restore` (S).** Restore is manual shell today
    (`docs/install.md:1187`); add the command with version check, dry run,
    off-host copy target; a documented DR drill.
13. **Done (branch `features-c2`):** TPOT, inter-token and tokens/s
    histograms by model, pool and tenant; `slo` pressure in admission's
    saturation; `llmgw_model_slo_seconds` and `llmgw-slo` alerts. Left: the
    p99 is per control plane, from its own requests; an SLO per band or per
    tenant; placement does not yet add replicas for an SLO (item 14).
    **SLO-based admission and token-level metrics (S–M).** TPOT and
    inter-token histograms, output tokens/s per model and tenant in
    `router/metrics.go`; use `catalog.SLO` in `admission/saturation.go`.
14. **Done (branch `features-c2`):** `scaling.schedule` (days, from, to,
    min, max; past midnight), `scaling.prewarm`, `control.timezone`;
    queue-depth scaling was there already. Left: `ctl scale` sets min and
    max only (a schedule is set in the definition); scaling for an SLO
    rather than for queue depth.
    **Scheduled and warm-pool autoscaling (S–M).** Time-of-day min/max,
    pre-warm, queue-depth scaling in `catalog.Scaling`,
    `control/reconciler/demand.go`, `headroom.go`.
15. **Done (branch `features-c2`):** `env:` and `vault:` references
    wherever a secret file is named (store address, api_key_file, OIDC,
    SCIM, webhooks), Vault KV with token or AppRole, `secrets.master_key`
    sealing cluster_secrets (Vault transit or a local key). Left: the
    certificate authority's key is still a file in /etc/llmgw (signing in
    Vault's PKI engine instead is its own change); rotating the master key;
    `backup_copy_key` and `admin_token_file` are files only.
    **Secrets management (S–M).** Vault/KMS for `api_key_file`, the DSN, the
    CA key, `cluster_secrets` (`store/secrets.go`, `store/seal.go`);
    envelope-encrypt them in backups (with A8 backups).
16. **Done (branch `features-c2`):** `users forget` (pseudonymised usage
    and sums, kept requests, files, batches, limits and audit names), `keys
    residency --in-cluster` (asked-for, fallback and canary). Left: an
    export of everything held about a person (article 15); residency by
    region between several clusters; backups keep what was forgotten until
    they age out.
    **GDPR and residency (S).** "Forget user" anonymises `user_id` in usage
    and rollups and erases threads/memory (`store/hierarchy.go:241` leaves
    it today); per-project "never route to an External provider".
17. **Done (branch `features-c2`):** exact-match answer cache per project (`keys
    cache --ttl/--any/--off`, console **Cache answers…**, `projects`
    columns, 30-second setting cache), reproducible answers only unless
    `--any`, LRU in memory bounded by `router.cache.max_bytes` and
    `max_entry_bytes`, streamed answers kept and replayed as chunks,
    `x-llmgw-cache` and `Cache-Control: no-cache`, guardrails before a hit
    and the output policy on it, `no_store` and threads bypass, a hit
    recorded as `cache_hit` with no tokens and no cost (migration 0027),
    metrics, docs. Left: semantic (embedding-similarity) caching through the
    embeddings models; a cache shared across control planes (each process
    keeps its own, deliberately: a shared one would write every answer to
    the store); a hit still waits its turn in admission, because the
    guardrails judge the request after admission and the key is the request
    they let through; logprobs in a streamed answer leave it unkept; `users
    forget` does not reach into the cache (answers age out by the TTL); no
    listing or flushing of what is cached (`--off` stops it being served at
    once, the memory is freed by TTL and LRU).
    **Response/semantic cache (M).** Exact-match first, per project,
    opt-in, TTL; semantic via the embeddings models later. Before
    `dispatch` in `router/handlers.go`.
18. **Done (branch `features-c2`):** offline bundles — `llmgw bundle create`
    (releases with signed checksums, every llama.cpp build per arch/variant
    with GitHub's checksums, starter/judge and named Hugging Face model
    files pinned and checked; directory or tar; manifest sha256), `bundle
    import` (checked, all-or-nothing, into the mirror under `_llmgw/`),
    `bundle trust`, `bundle show --verify`;
    `offline.mirror`/`offline.checksums` in server.yaml, with nodes
    installing llama.cpp from the mirror against checksums the control plane
    sends over the agent stream (proto `EngineWant.offline_*`), releases
    defaulting to the mirror for `node add`, `ctl upgrade` and agent
    upgrades, update checks off; `server init --bundle` and the setup form's
    bundle fields; docs/data.md's offline claim corrected; install.md
    "Offline sites". A Helm chart for the control plane
    (`deploy/helm/llmgw`, one replica, SQLite PVC or external PostgreSQL,
    Secrets, Services, Ingress, ServiceMonitor) with
    `deploy/container/Dockerfile` and docs/kubernetes.md; GPU nodes stay
    outside Kubernetes by design. Left: vLLM/SGLang offline (a wheelhouse of
    several GB per CUDA build and a uv `--find-links` install path); the
    NVML release binary and uv in bundles; the setup wizard's preflight
    still fails a machine without internet; `join`'s printed `curl` from the
    mirror checks nothing; models in a bundle are single files (no whole-
    repository safetensors sets yet); the chart runs one replica (a standby
    is never ready, so HA in Kubernetes needs a different readiness design),
    has not been installed on a real cluster, and ships no agent DaemonSet;
    no published container image.
    **Kubernetes/Helm and an air-gapped bundle (M).** `deploy/`,
    `agent/provision`; `docs/data.md` claims offline operation but install
    assumes the internet.
19. **Done (branch `features-c2`):** `adminapi.Routes` and
    `internal/docgen/openapi` generating `docs/admin-api.openapi.json` (3.1,
    bearer security, every route, query parameters, schemas from the Go
    types with their comments), served embedded at `GET
    /api/v1/openapi.json`, kept true by a route-table test and a staleness
    test; validated with openapi-spec-validator and openapi-generator-cli.
    `GET /api/v1/models/{name}`. `docs/admin-api.md`. `terraform-provider-
    llmgw/` (terraform-plugin-framework): `llmgw_model`, `llmgw_quota`,
    `llmgw_guard_attachment`, data sources `llmgw_models` and `llmgw_nodes`;
    client tests against a fake admin API and the OpenAPI document;
    acceptance tests with Terraform and OpenTofu against the fake; one
    apply, empty plan, in-place change and destroy against a real `llmgw
    server` run by hand. Left: organizations, teams, projects, people and
    keys are not in the admin API (they are `llmgw org`/`users`/`keys`
    against the store), so not in the provider; a model's definition beyond
    scaling, class, owner and adapters is changed by replacing it (the API
    has no edit, and `resources` is not used yet); `canary`, `slo`,
    `scaling.schedule` and `prewarm`, `guard` role and `origin` are not in
    `llmgw_model`; models cannot be imported; the provider is not published
    to a registry and has no release build; the acceptance tests are not run
    in CI (they need Terraform); the document's schemas mark nothing
    required and give no enums; `catalog.SLO`'s fields have no JSON tags, so
    they are `TTFTp99` and `TPOTp99` on the wire.
    **OpenAPI for the admin API, Terraform provider (M).** Generate from
    `adminapi/api.go` routes via `docgen`.
20. **Done (branch `features-c2`):** spans for the reconcile pass (begun only
    when a pass does something), placement with its answer or rejections,
    scale-up/down, drain, stop, eviction, preemption, reaping, restarts and
    lost nodes; a `replica start` span from placement to serving whose
    context travels to the agent in the start command
    (`ControlMessage.trace`), where weights fetch, engine launch, rendezvous
    and readiness wait continue the trace; stops, drains, adoption and
    engine installs; modelserve jobs as a trace with a span per step linked
    to the replica's start; an agent `tracing` block; docs in install.md's
    "Tracing". Left: the router's span for a request does not link to the
    replica it was sent to's start; heartbeats, fabric scans, logs and agent
    upgrades are not traced; the supervisor has no restart span because it
    never restarts an engine itself (a restart is the reconciler's `restart
    replica` and reaches a node as a stop and a start); canary and promote
    jobs, which landed on `features-c2` after this was branched, are traced
    only through the shared step recording (as `<kind> model`), unchecked by
    a test.
    **Tracing beyond the router (S).** Spans for reconciler, placement,
    agent start (`obs/trace.go`, `control/reconciler`, `agent/supervisor`).
21. **Done (branch `features-c2`):** `internal/bench/eval` (suites in YAML or
    JSON lines; exact, contains, regex, number, a JSON Schema subset, and a
    judge model with a rubric; weights and a threshold), run on the control
    plane through `Router.Replay`; `llmgw ctl eval MODEL --suite FILE|NAME`,
    `--history`, `--run`, `--suites`; runs and their cases kept (migration
    0028, `eval_runs` and `eval_cases`); `model update`/`promote --eval
    SUITE --min-score N`, a model's standing gate (`ctl model gate`, `eval:`
    in the catalog) — a promote asks the canary first, an update tries the
    new weights as a 0% canary and promotes or removes it; the console's
    model menu; `arithmetic` and `json-format` in `deploy/eval/`, carried in
    the binary. Left: the suite of a model's standing gate is read by name
    on the control plane at the time of the update, so two control planes
    need the same `control.eval_dir`; a gated update needs room for one more
    replica while it runs (there is no in-place gate, and a declared model
    in server.yaml cannot be updated at all); no comparison against the old
    weights' score (the gate is an absolute threshold, not "no worse than
    before"); a run is not stopped once started (Esc in the console only
    stops watching it), and the in-memory progress of a run lives on the
    control plane that started it; runs are never pruned; the JSON Schema
    support is a subset (no `$ref`, `oneOf`, `not`, `uniqueItems`); judge
    cases go through the same concurrency as the model's, and a judge's
    answers are not kept, only its score and reason; a gateway without a
    control plane cannot run suites (there is no mode that asks through the
    client API with a key).
    **Eval harness (M).** `llmgw eval` extending `internal/bench` with
    quality suites gating `model update`.
22. **Done (branch `features-c2`):** MIG instances discovered on both paths
    (nvidia-smi `-L` + `-q -x`; NVML through an interface with a faithful
    fake) and reported instead of their card, each with its MIG UUID, own
    memory and class `<card class>-mig-<profile>`; device numbers 1000 + 100
    × card + instance, shown as `card/migN`; placement on MIG classes;
    tp/pp/gpus/nodes above 1 on a MIG class refused in catalog validation
    and again in placement; `CUDA_VISIBLE_DEVICES=MIG-<uuid>` for vLLM and
    llama.cpp (no Vulkan device variable on MIG); per-instance memory and
    processes in heartbeats and the GPU check, card sensors under each
    instance, utilisation absent rather than zero; doctor MIG findings (`mig
    -lgi`/`-lci` parsed); `ctl nodes`, console, `ctl status`, `ctl model
    plan`/`serve` by class; docs/install.md MIG section. Written and tested
    against fakes only — untested on real MIG hardware. Left: run it on an
    A100/H100 in MIG mode (check `-L`/`-q -x`/`mig -lgi`/`-lci` and NVML MIG
    device names against the parsers, and that `CUDA_VISIBLE_DEVICES=MIG-…`
    reaches vLLM's workers through Ray); instances changed while the agent
    runs are only seen when it restarts (no rescan); llmgw does not create
    or reshape instances (no mig-parted integration); quotas count an
    instance as one GPU, the same as a whole card; SGLang on MIG not
    checked; device numbers assume fewer than 100 instances per card and
    that instance numbers stay put while shards run on them.
    **MIG (M–L).** Treat a MIG slice as a GPU class in
    `agent/discovery/gpu_nvml.go` and `control/placement`.
23. **Done (branch `features-c2`):** engine `trtllm`
    (`internal/agent/engines/trtllm.go`): `trtllm-serve` through a launcher
    of llmgw's that demands the replica's key (trtllm-serve has none) and
    writes the options file that turns on `/prometheus/metrics` and the
    iteration statistics; the scrape contract for it
    (`scrape.TRTLLMMapping`, with a per-engine metrics path); found, not
    installed — `engines.trtllm.python` in `agent.yaml`, probed from the
    package's metadata; catalog validation (GPU class, one node, no GGUF
    source, the flags a profile may not give); `ctl engine-check trtllm`,
    which asks `trtllm-serve serve --help` through its module. Ollama/GGUF
    import: `mirror import-ollama` (manifest parsing, blob digest check,
    projector beside the model, all or nothing, a model definition printed),
    `mirror import` for any local file, and the console's **I** on the
    Mirror view. Left: TensorRT-LLM has not run on real hardware — the first
    node with one should run `engine-check` and serve a model before anyone
    relies on it; no installer for it; no multi-node TensorRT-LLM replica
    (it needs an MPI or Slurm launch from one node); Triton Inference Server
    as an engine is not done; trtllm's queue gauges are set per iteration,
    so an idle engine shows its last batch until the next request;
    `--num_serve_frontends` is refused because extra front ends would not
    demand the key. The Ollama import runs on the mirror machine only — the
    mirror's API takes no files, so an Ollama elsewhere means copying the
    blob or running the import there; Ollama models kept as separate tensors
    or as several weight layers, and its LoRA adapters, are refused or left
    with a sentence; files an import run with `sudo` writes are root's, as
    `mirror add`'s are.
    **More engines (M each).** TensorRT-LLM/Triton adapter; Ollama/GGUF
    import. `agent/engines/engine.go` interface and the metric contract in
    `router/scrape`.
24. Lower: precise KV-index routing (plan phase 6), LoRA affinity for
    vLLM, multi-cluster federation via External providers, facility-aware
    placement (`docs/facility-aware-placement.md`), portal i18n, SDK/docs
    portal.
    Done so far (branch `features-c2`):
    - 24 (portal i18n): built — English/Turkish catalogue with fallback, per-
      language templates, cookie/Accept-Language/`landing.language` choice
      and a footer switcher, Turkish numbers and dates, every template and
      handler message translated, tests that no template text stays English;
      left — errors built with fmt.Errorf in other packages (store, catalog,
      mirror, adminapi) still English on Turkish pages, percent sign
      placement is mixed (%50 on limits, 93% on cluster pages), RFC3339
      footer timestamps and plain-text http.Error bodies untranslated, the
      /docs documents themselves English, a per-person language column
      (cookie only today).
    - 24 (SDK/docs portal): built — /docs renders the embedded manual (goldmark,
      GFM, heading anchors, links rewritten, no script; public,
      `landing.docs_sign_in` to require a session), and docs/clients.md
      shows the official OpenAI/Anthropic SDKs against llmgw. Left: the
      console's generated "Commands" page is not in the portal (it is built
      from the cobra tree, which internal/web cannot see); no search across
      documents.

---

## D. Facts worth knowing before starting

- Admin API: `127.0.0.1:9090`, no auth, `ctlClient` in `internal/cli/ctl.go`.
  Metrics on `:9091` with `/prometheus/targets` beside them.
- Store: `internal/store`, migrations under `internal/store/migrations/{sqlite,postgres}`;
  the PoC runs PostgreSQL under Patroni.
- The console (`sudo llmgw`) is `internal/cli/tui_console.go` with actions
  in `tui_actions.go`; every operator-visible change needs a row action or
  a form there, and a line in `docs/console.md`.
- Tests of the console use a fake admin API (`internal/cli/tui_actions_test.go`,
  `adminStub`); tests of serving use fakes in
  `internal/control/modelserve/modelserve_test.go`.
- `go generate ./...` regenerates `docs/reference.md` and
  `internal/docgen/reference_gen.go` whenever a config struct changes; the
  docgen test fails otherwise.