llmgw/BACKLOG.md

# Backlog

What a review of the code on 2026-10-04 (commit c2238ab) found should be done
next: security fixes first, then guardrails, then features. Written for
whoever implements it — a person or a model — without the review in hand.
Every item names the files it touches and what "done" is. Paths are relative
to the repository root. Sizes: S about a week, M two to four weeks, L more.

Conventions this codebase keeps, which every item here must keep too:
- A change is refused before anything is written (validate, then persist).
- Nothing on the serving path logs prompt text, keys or passwords.
- Each item gets unit tests beside the code (`*_test.go`), and the TUI
  (`internal/cli/tui_*.go`), `docs/console.md` and `CHANGELOG.md` are
  updated when behaviour an operator sees changes.
- `go test ./...`, `go vet ./...` and `govulncheck ./...` pass. No release is
  cut and nothing is deployed to the PoC cluster without being told to.

---

## A. Security — do first

**Status (2026-10-04): done and merged.** A1–A9, A6b included — etcd and
Patroni's API on TLS, the PoC moved in maintenance mode without a failover.
Deliberately narrower than written: the admin API is a token on loopback
rather than a unix socket (A5); thread ids are bounded and documented as
project-wide rather than scoped per user (A8); Python engines resolve with an
explicit PyTorch index and a PEP 751 lock rather than locks shipped per engine
version (A8). All of A is live on the PoC.

### A1. Toolchain (S, hours)
`govulncheck` finds 11 standard-library vulnerabilities reachable from this
code at Go 1.26.3 (net/http, crypto/x509, os.Root, net/textproto). Raise the
`go` directive in `go.mod` to 1.26.6 or later, rebuild, run `govulncheck
./...` until it reports none reachable. Add `govulncheck` to the Makefile's
check target.

### A2. Release signing is effectively off (S)
- `internal/release/trusted.pub` is empty, so `TrustedKey()` returns "" and
  `release.go:103` skips signature verification. Ship a real public key;
  make a missing or invalid signature fatal. Drop the `LLMGW_RELEASE_KEY`
  environment override, or honour it only in `-tags dev` builds.
- `internal/onboard/onboard.go:365`: the node-side download checks only a
  same-origin `checksums.txt`. Verify the signature there too (simplest:
  have the node run `llmgw` itself to fetch and verify once it has any
  binary), and `shellQuote` the `Releases` URL, which is interpolated into
  single quotes unescaped.
- `internal/agent/upgrade.go:44`: accept only `https://` release bases.
- Done when: a tampered binary or checksums file is refused by self-upgrade,
  node onboarding and agent upgrade, with a test for each.

### A3. Agent port without TLS is anonymous and plaintext (S)
`internal/cli/server.go:423` serves h2c on `:7443` when `agent_tls` is not
configured; `agentsvc` then accepts any node id. Refuse to start control
mode without TLS unless `agent_tls.insecure: true` is set explicitly; when
insecure, bind `agent_listen` to 127.0.0.1 and log a warning at startup and
in `llmgw doctor`. Keep `server init` writing TLS as it does.

### A4. Model profiles reach root on every node (M)
Profile `args` and `env` from a model definition are appended to the engine
command unchecked (`internal/agent/engines/llamacpp.go:170`, `vllm.go:151`,
`sglang.go:98`); `internal/agent/engines/unsafeenv.go` is a denylist that
misses `NCCL_NET_PLUGIN`, `NCCL_TUNER_PLUGIN`, `NCCL_PROFILER_PLUGIN`,
`CUDA_INJECTION64_PATH`, `NVTX_INJECTION64_PATH`, `TRITON_*` paths; engines
run as root.
- Replace the env denylist with an allowlist (CUDA_/NCCL_ tuning keys minus
  anything ending in `_PLUGIN` or containing `INJECTION`, `VLLM_*`,
  `HF_*` minus path keys, `OMP_*`, `GGML_*`). Apply at catalog validation
  (`internal/control/catalog/catalog.go`, where `UnsafeEnv` is called) and
  again in the agent.
- Deny engine flags that write files or run code: `--trust-remote-code`,
  `--log-file`, `--slot-save-path`, `--chat-template` with a path,
  `--host`, `--api-key`, `--port`; vLLM/SGLang equivalents. Allowlist is
  better than denylist if the flag set is enumerable per engine.
- Run engines as an unprivileged user with the video/render groups
  (`internal/agent/supervisor`, systemd unit in
  `internal/cli/serverinit_cluster.go:52`). Weights cache must be readable
  by it.
- Done when: a profile with `--trust-remote-code` or `NCCL_NET_PLUGIN` is
  refused at `ctl model add` and at the agent; `ps` shows engines not as
  root.

### A5. Admin API trusts any local process (M)
`internal/cli/server.go:357` serves the admin API and pprof on
`127.0.0.1:9090` with no authentication; `adminapi.actorOf`
(`internal/control/adminapi/api.go:1613`) takes the actor from the
client-sent `X-Llmgw-Actor` header.
- Serve the admin API on a unix socket (`/run/llmgw/admin.sock`, 0660,
  group `llmgw-admin`) by default, keeping TCP as an option that requires
  a bearer token read from a root-only file (`/etc/llmgw/secrets/admin-token`).
- Take the actor from `SO_PEERCRED` (uid → username) on the socket, or from
  the token's name; ignore the header.
- Refuse `admin_listen` on a non-loopback address unless `admin_insecure:
  true`.
- Put pprof behind its own `debug.pprof: true` setting, off by default.
- `ctlClient` (`internal/cli/ctl.go`), the console and `node add` must work
  over the socket; `sudo llmgw` already runs as root.

### A6. Patroni REST and etcd are unauthenticated and plaintext (M)
`internal/pgha/config.go:99` writes a `restapi:` with no `authentication`
and `:54` etcd over `http://` with no client auth; `hba.go:56` PATCHes
Patroni's config unauthenticated. Anyone reaching :8008 or :2379 owns the
store.
- Generate a Patroni REST username/password at setup, store it in
  `/etc/llmgw/secrets/`, write `restapi.authentication` and
  `restapi.certfile/cafile` from the cluster CA; send it from
  `internal/pgha/hba.go` and `status.go`.
- etcd: client and peer TLS from the cluster CA with `--client-cert-auth`.
- Document, and have `server init` write, nftables rules limiting
  2379/2380/8008/5432 to member addresses (see A9).
- Done when: an unauthenticated `curl :8008/config` is refused and the
  store status in the console still works.

### A7. PostgreSQL traffic is plaintext (S)
`internal/pgha/config.go:168` and `internal/cli/serverinit_postgres.go:330`
use `sslmode=disable`; HBA lines are `host`. Enable server TLS in Patroni's
postgresql parameters with a certificate from the cluster CA, write
`hostssl` lines, use `sslmode=verify-full&sslrootcert=/etc/llmgw/ca.pem`.
Existing clusters: `llmgw server` must keep working against a store that
has not been migrated, with a warning.

### A8. Smaller fixes, each S or less
- **Capacity fallback skips the allowlist.** `internal/router/router.go:700`
  builds `[canonical]+entry.Fallbacks` without `permits`. Filter the chain
  with `r.permits(p, model)`; price against the model that served.
- **Suspended users keep working API keys.** `store.(*DB).Resolve`
  (`internal/store/memberships.go`) never joins `users.state`;
  `Authenticator.Invalidate()` is never called. Reject keys whose owner is
  not active; call `Invalidate` on revoke, suspend, limit change.
- **Join codes never expire server-side.** `internal/join/join.go:103`
  checks expiry only in `Parse`; `admit` (`:212`) does not. Check it there;
  bind the listener to the chosen address, not `:7444` on all interfaces;
  use the peer's address rather than the claimed `hello.Addr`, and show the
  joiner's address for the operator to confirm before secrets are sent.
- **Leftover sudoers grant.** `internal/onboard/gossh.go:248` writes
  `NOPASSWD:ALL` and removes it only in `Close()` (`:271`). Write the grant
  with a self-removal (`at`/systemd-run timer, 30 min) so a crash leaves
  nothing; wipe password buffers after use. In `ssh.go:39` pass `--` before
  the target.
- **Auth cache and pre-auth DoS.** `internal/auth/auth.go:127`: unbounded
  map, one DB query per bogus well-formed key. Bounded LRU, negative cache
  with short TTL, per-IP token bucket before the DB.
- **HTTP server hardening.** `server.go:351-360`, the h2c agent server,
  both mirror servers: set `ReadHeaderTimeout` (exists), `IdleTimeout`,
  `MaxHeaderBytes`, a body-read deadline, `netutil.LimitListener`; connect
  `WithReadMaxBytes` on the agent stream.
- **Login/signup abuse.** Default `signup_open: false`; per-IP token buckets
  on `/login` and `/signup`; bounded wait for argon2 slots; CSRF token on
  the login form; the per-address throttle must not lock the real owner
  out (use IP+address pairs).
- **Trusted proxies.** `internal/web/account.go:557`, `landing.go:60` honour
  `X-Forwarded-Proto/Host` from anyone; add `router.trusted_proxies` and
  only then trust them; `Cache-Control: no-store` on `/setup/*`.
- **Certificate revocation.** `DELETE /api/v1/nodes/{id}` only drops
  inventory; add a revoked-CN/serial list in the store checked in
  `agentsvc` `Connect` and mirror `onlyClients`; `serverinit_cluster`
  always passes `--api-client`.
- **Engine exposure.** Engines bind `0.0.0.0` (`llamacpp.go:102`,
  `vllm.go:109`): bind the management address. SGLang receives its key on
  the command line (`sglang.go:102`): pass it by env or file.
- **Backups.** `internal/cli/backup.go`: encrypt archives to an `age`
  recipient from `backup.recipient` in server.yaml; keep the directory 0700.
- **Python dependency confusion.** `internal/agent/provision/python.go:121`
  uses `--extra-index-url` with `unsafe-best-match`. Pin torch to the
  PyTorch index explicitly (uv `[[index]] explicit = true`), `first-index`,
  hash-pinned lockfiles per engine version.
- **Control-plane SSRF.** `internal/sourcecheck/sourcecheck.go:39` and
  `gguf.FromURL` use the default client; reuse the mirror's
  `fetchClient(false)` (private-address filter) and refuse a proxy unless
  configured; block NAT64/6to4-embedded private addresses in
  `internal/mirror/safefetch.go`.
- **Low:** `agentsvc/service.go:430` indexes `n.GPUs[0]` without a length
  check (CPU-only node panics the handler); thread ids are client-chosen
  with no length cap and shared per project (`router/threads.go`) — cap
  length, document, offer per-user scoping; `/v1/models` lists models the
  key may not use (`handlers.go:799`); sessions have no idle timeout and
  `PurgeSessions`/`PurgeClaims` are never called — add a janitor; add a CSP
  header and set `htmx.config.allowEval=false`; cap GGUF array nesting
  depth in `internal/gguf/gguf.go`; mirror `.source` sidecars are served by
  the public file server — exclude dotfiles from `:8081`; prune
  `modelserve.jobs` and the mirror `running` map; KaTeX → 0.16.21+.

### A9. Firewall (S)
No machine in a default install has a firewall. Have `server init`, `node
add` and `server add` write an nftables ruleset: ssh open; `:8080` open;
`:7443` and `:9091` open to the cluster's networks; `:9100` to the
monitoring host; `:2379/2380/8008/5432` to store members and control
planes; mirror `:8081` to the cluster, `:8082` to control planes; `:9090`
Prometheus and `:3000` Grafana to operators. `llmgw doctor` reports a
missing ruleset. Make it skippable with `--no-firewall`.

---

## B. Guardrails (M–L, build in this order)

**Status (2026-10-04, branch `guardrails-b`):** B1–B4 done, and tried live on
the PoC. That found four bugs, all fixed with tests:
- a shadow default merged with an attached policy of unwritten mode left it
  in shadow;
- a streamed word held back at one release was dropped at the next;
- llama.cpp continues a conversation ending on the answer, so Llama Guard's
  verdict on answers is now asked with its own prompt as a completion;
- a judge model was named in a client's "no endpoint" error.

The PoC runs `observe` (shadow) for everyone, and Llama Guard 3 1B
(`llama-guard-3-1b`, CPU, on cuda01) as a judge, which blocks a harmful
request in ~0.7s and a harmful answer in ~1.4s there. Still left: see each
item.

Today nothing on the request path reads, classifies, redacts or filters
message text in either direction (confirmed; the only "moderation" hit is a
comment in `internal/router/budget_fallback.go:20`). Existing controls are
about who, how much and how fast: allowlists (`router/handlers.go:450`),
RPM/TPM/budget (`internal/accounting`), context precheck
(`router/context_limit.go`), admission bands (`router/admission`),
timeouts (`router/proxy/proxy.go:26-57`). Gaps to close on the way: no
default output cap when the client omits `max_tokens`; `catalog.Capabilities`
is advertised but never enforced; tool results are trusted input.

### B1. Hooks in the proxy and router (S)
**Done (branch `guardrails-b`):** `internal/guard` holds the interface;
proxy `Filter`/`StreamFilter`; the input hook runs after `loadThread` (the
simpler choice); error codes, headers, Responses `incomplete`, usage columns
`guard_*`. Moved to B2: holding `delta.tool_calls` until they are whole and
reading their arguments — that is the tool policy's filter, not the proxy's.

- `internal/router/proxy/proxy.go`: add optional `InputRewrite func(body
  []byte) ([]byte, Verdict)` on the request and two output hooks: `Filter`
  for `relayBuffered` (between `ReadAll` and `WriteHeader`) and
  `StreamFilter{OnEvent(data) (emit []string, stop bool); OnEnd()}` for
  `relayStream`, called where each whole SSE event is assembled
  (`eventBuffer`, before `io.WriteString` at `:343`). Nil hooks cost
  nothing.
- `internal/router/handlers.go`: call the input policy after the allowlist
  and limits and before admission (around `:395`); the principal, scopes
  and the model that will serve are known there. Thread history is spliced
  later (`:416`): scan stored turns when written, or move the hook after
  `loadThread` (decide; the second is simpler).
- Blocking conventions: input → 400 `invalid_request_error` with
  `code: "content_policy_violation"` (today `writeError` always sends
  `code: null`); output non-stream → 200 with `finish_reason:
  "content_filter"` and empty/redacted content; output stream → a final
  chunk `{"choices":[{"index":0,"delta":{},"finish_reason":"content_filter"}]}`,
  then `data: [DONE]`, then cancel upstream. Headers `x-llmgw-guard:
  blocked|redacted|flagged` and `x-llmgw-guard-policy` set before the proxy
  writes headers.
- `internal/router/responses.go`: map `content_filter` to
  `response.incomplete` with `incomplete_details.reason: "content_filter"`
  (today it always emits `response.completed`, `:525`).
- Tool calls: hold `delta.tool_calls` until `finish_reason: "tool_calls"`,
  then validate; `replyDelta` (`reply.go:33`) must also read
  `tool_calls[].function.arguments`.
- Accounting: a blocked input is recorded like `RecordRefused` (nothing
  charged); a blocked output is charged (the GPU was used). `store.Usage`
  gains `GuardAction`, `GuardPolicy`, `GuardCategory`, `GuardMs`.

### B2. Package `internal/guard` (M)
**Mostly done (branch `guardrails-b`):** policies in server.yaml with
attachments in the store (`ctl guard attach`), narrowed by the limits' rule;
`guard/detect` (validated PII, secrets, injection through normalisation and
decoding); redaction with consistent placeholders before prefix routing;
tool allow/deny/strip, call count and argument schemas; capabilities and
structured output (opt-in); default `max_tokens`; pinned system prompt and
dropping the client's; `no_store` (threads, notes, prefix tree, external
models); shadow mode. **Left:** a `guard_events` table — the usage row's
`guard_*` columns carry action/policy/category/time, not rule and stage;
language identification; the re-identification map that restores
redacted values in answers.

- `Policy`, `Effective` resolution over the tenancy tree with the same
  narrowing rule as limits (`internal/store/sqlite.go:383-416` `tighten`):
  deny sets union, allow sets intersect, `mode` takes the strictest of
  `enforce > shadow > off`, `fail` takes `closed > open`, `no_store` at any
  level wins, thresholds take the minimum.
- Store: `guard_policies(scope, scope_id, policy json)` and
  `guard_events(request_id, scopes, policy, rule, category, action, stage,
  score, judge_ms, shadow)` — ids, categories, offsets and lengths only,
  never text. Migrations for sqlite and postgres. Pruned with
  `usage_retention`.
- `guard/detect`: regex/keyword sets; PII with validators — email, phone,
  IBAN (mod-97), card (Luhn), TCKN (checksum), IP, secrets (`sk-`, AWS,
  JWT, PEM); normaliser (NFKC, strip zero-width/bidi, homoglyph fold,
  base64/hex/URL decode to depth 2); injection heuristics (role-switch
  phrases, fake chat-template tokens, long base64, invisible Unicode) —
  default action `flag`; language id.
- Redaction rewrites `messages[].content` in the upstream body with
  placeholders (`[EMAIL_1]`) via the same `map[string]json.RawMessage`
  round-trip `withModel`/`spliceThread` use; an optional re-identification
  map lives only in request memory and restores originals in the output.
  Redact before prefix routing so `PromptPrefix` hashes the redacted text.
- `guard/toolpolicy`: tool name allow/deny (strip or 400), JSON-Schema
  validation of assembled arguments against the declared `parameters`,
  argument regex rules, `max_calls_per_turn`. Enforce
  `Capabilities.Tools`/`Vision` here: a `tools` array or image parts to a
  model without the capability → 400.
- `guard/structured`: validate `response_format.json_schema` on the final
  text → `content_filter`, or flag in shadow.
- Output constraints: inject a default `max_tokens` when absent (from the
  policy or `MaxOutput`), language check, structured check.
- System-prompt pinning per project: prepend before `checkContextFits` and
  before `spliceThread`; optionally drop client `system`/`developer`
  roles; exclude the pin from stored thread turns; apply to Responses
  `instructions` in `toChat`.
- `no_store`: `loadThread` and `PutNote` refuse; `captureBytes()` returns 0;
  skip the prefix-tree observe (`router.go:840`); disable External
  fallbacks for the scope (filter in `dispatch` and `budgetFallback`).

### B3. Judge model on the cluster (M)
**Done (branch `guardrails-b`):** `role: guard` + `guard.format`; hidden and
refused to clients and fallbacks; critical by default, never on demand;
`Router.Judge` past admission/accounting/guardrails; llamaguard3,
shieldgemma and json parsers; 5-minute cache; latency budget then
fail open/closed; streamed answers read every 600 characters, or held with
buffer_all. Simpler than written: the input judge runs after admission, not
alongside it, and shadow mode judges synchronously within the budget rather
than after the response. Not enforced: two replicas on different nodes —
documented, left to `scaling.min`.

- `catalog.Model.Role: "guard"` plus `Guard{Format: llamaguard3 |
  shieldgemma | json, Categories}`. Guard models are hidden from
  `/v1/models`, the chat picker and allowlists; placed like any model,
  class `critical`, `scaling.min ≥ 1`, at least 2 replicas on different
  nodes.
- `guard/judge`: calls the guard model through the router's own `dispatch`
  with an internal principal and its own admission band (never the
  tenant's queue); parsers per format (Llama Guard `safe`/`unsafe\nS1,S10`;
  ShieldGemma yes/no probability → per-tenant thresholds; generic JSON
  verdict); verdict cache keyed by hash of normalised new turns (5 min);
  `latency_budget` (default 250 ms) then `fail` open/closed; input judge
  runs in parallel with admission queueing.
- Streaming strategy: default sentence-buffered scanning (regex/PII per
  sentence or ~200 chars before release; judge every 400–800 chars on the
  cumulative tail; final judge at end); `buffer_all` for `fail: closed`
  scopes. Document that released text cannot be retracted.
- Shadow mode runs everything asynchronously after the response
  (`context.WithoutCancel`, like `saveThread`) and only records.

### B4. Config, metrics, UI (S)
**Done (branch `guardrails-b`):** library in server.yaml, attachments via
`ctl guard`, the console's **Guardrails…** with "applies from" marks, and
the portal's `/admin/guard`; the metrics and alerts below (judge latency
labelled by model only at the judge; the check latency by stage); `ctl
request` shows the verdict; the chat page says "stopped by policy"; the
profile lists effective policies; docs/guardrails.md and data.md written;
the access log carries action, stage and policy only. Attaching from the
portal is not done — the page shows; the console and ctl change.

- Gateway-level policy library in `server.yaml` under `guard:` (see the
  YAML in the review, reproduced in `docs/guardrails.md` to be written);
  attachments to scopes in the store via `llmgw keys guard`, the console
  ("Guardrails…" beside "Limits…" in `internal/cli/tui_people.go`, with
  "inherited from team:X" badges) and the portal.
- Metrics in `router/metrics.go` style:
  `llmgw_guard_checks_total{stage,policy,action,category,mode}`,
  `llmgw_guard_latency_seconds{stage,judge_model}`,
  `llmgw_guard_judge_errors_total{reason}`, `llmgw_guard_fail_open_total`,
  `llmgw_guard_stream_stops_total`, `llmgw_guard_redactions_total{kind}`,
  `llmgw_tenant_guard_blocks_total{tenant}`. Alerts: fail-open > 1% for 5m,
  judge p95 over budget, judge has no ready replicas, per-tenant block rate
  > 3× its 7-day baseline, any `secret` detected in output.
- Portal `/admin/guard`: violations by tenant/category/policy over time,
  shadow vs enforce, judge health, recent events by request id; `ctl
  request <id>` shows the verdict; chat page renders `content_filter` as
  "stopped by policy X"; `/me` lists effective policies.
- Update `docs/data.md`: the judge sees all text but in-cluster; External
  fallbacks send prompts outside the cluster (the current "nothing leaves
  the cluster" is wrong); `no_store` semantics.
- Never log: prompt/completion text, PII values, the re-identification
  map, judge inputs or rationale that quotes input. Check `accessLog`'s
  `err` field and span errors.

---

## C. Features, ranked by value × feasibility

1. **Done (branch `features-c`):** `audit_events`, written by the admin
   API middleware and the /admin pages (with each change's result), `ctl
   audit`, the console's Audit view; staff roles viewer / operator /
   billing / admin. Left: before/after values (the request's fields are
   recorded, not the state they replaced); a tenant-admin role that changes
   its own organization (managers still only view it); hiding the forms a
   viewer cannot submit. **Audit log and roles (M).** Today "audit" is slog lines
   (`internal/web/admin_users.go:403-480`) and roles are the `IsAdmin` flag
   plus an org "manager" who can view. Add `audit_events` (who, what,
   before/after, source) written by the `adminAction`/`tenancyAction`
   wrappers and by every admin-API mutation; roles viewer / operator /
   tenant-admin / billing; `ctl audit`. Depends on A5 for a real actor.
2. **Done (branch `features-c`):** OIDC sign-in (`auth.oidc`, PKCE,
   verified email, roles and memberships from groups, `no_passwords`) and
   SCIM 2.0 (`/scim/v2` Users and Groups). Left: the sign-in cookie is
   signed with a per-process key, so with two control planes behind a
   balancer the callback must reach the one that started it (sticky
   sessions) until the key comes from the store. **OIDC/SSO login (M), then SCIM (M).** `internal/account/service.go`,
   `internal/web/account.go`; map IdP groups to org/team/admin.
3. **Done (branch `features-c`):** key lifetime limits (gateway, org, team,
   project), `keys rotate --overlap`, kinds and expiry in `keys list`.
   **Service accounts and key lifecycle (S).** Per-org maximum key TTL,
   `keys rotate` with overlap, last-used shown, project-owned service keys
   listed apart. `internal/cli/keys.go`, `internal/store/keys.go`,
   `internal/auth/auth.go`.
4. **Done (branch `features-c`):** CSV export from the rollups (API, ctl,
   console). Left: Parquet, and a scheduled push to S3/SFTP. **Usage export (S).** CSV/Parquet by org/team/project/model/period from
   the rollups (`internal/store/usageseries.go`), `GET
   /api/v1/usage/export`, optional scheduled push to S3/SFTP.
5. **Done (branch `features-c`):** `notify.webhooks` (model parked, node
   lost, budget spent, update available; signed JSON or Slack; retries,
   hourly de-duplication), `ctl notify test`, Alertmanager receiver
   templates. Left: `server init` and the console do not yet ask for a
   receiver — the templates are filled in by hand; each control plane sends
   budget events on its own, so two behind a balancer may both send one.
   **Alerting out of the box (S).** `deploy/monitoring/alertmanager.yml`
   ships `receiver: none`; add Slack/email/webhook/PagerDuty receiver
   templates driven by `server init` or the console; llmgw-native webhooks
   for model parked, budget spent, update available (from the reconciler
   and `modelserve`).
6. **Done (branch `features-c2`):** `metadata`, `x-llmgw-tags` and `user`
   on the usage record (migration 0023), `ctl request` shows them,
   `router.tag_labels` metrics with 100 values per tag. Left: export and
   `ctl usage` by tag or end user (the hourly and daily sums do not carry
   tags; it would read the request records, within their retention).
   **Request metadata and tags (S).** Accept `metadata`/`user` from the
   body or `x-llmgw-tags`; store on the usage row; tenant metric labels
   behind an allowlist. `router/handlers.go` parse, `accounting/recorder.go`.
7. **Done (branch `features-c2`):** `ctl model canary/promote/canaries`,
   session-sticky split with `x-llmgw-variant` pins, rollback on error rate
   or TTFT, `canary_rolled_back` notification, console menu. Left: A/B
   between two different models (the split is between a model and its
   newer weights); the comparison is per control plane and lives in memory,
   so a restart begins it again; a key-level pin.
   **Canary and weighted A/B between model revisions (M).**
   `control/modelserve/updates.go` `replace` swaps every replica; add
   weighted routing across two revisions/aliases (scheduler filter, alias
   resolution in `router/router.go:569`), a header/key pin, automatic
   rollback on error-rate or TTFT regression from per-replica metrics.
8. **Done (branch `features-c2`):** `/v1/files` and `/v1/batches`
   (OpenAI's shape), worker under a store lease, lines kept as served,
   lowest band, usage tagged `batch=<id>`, output and error files with
   retention, `no_store` refused. Left: the Responses API (`/v1/responses`)
   as a batch endpoint; the console and `ctl` do not list batches (an
   operator sees them through usage tags); a file is one row in the store,
   so a store backup carries batch files with it.
   **Batch API and Files (M–L).** `/v1/files`, `/v1/batches`, JSONL
   storage, a worker feeding the bronze band, accounting. New
   `internal/router/batch`.
9. **Done (branch `features-c2`):** `keys requests` (retention, sample),
   kept after redaction, `ctl request --body/--replay`, console, pruning.
   Left: a streamed answer is kept as its text, not its chunks; replay is
   one request at a time (a set of them against a model is item 21's eval).
   **Request/response logging and replay (M).** Opt-in per project with
   retention like threads; sampled or full; redaction hooks from B;
   `ctl request <id> --body`; replay against another model.
10. **OpenAI compatibility gaps (S–M each).** Responses: `text.format` →
    `response_format`, `input_image`, `previous_response_id` via threads
    (`router/responses.go`). Rerank/score passthrough for vLLM,
    `/v1/audio/transcriptions` for a whisper engine, `/v1/moderations`
    mapped to the guard model.
11. **Anthropic `/v1/messages` (M).** Translate like `responses.go`.
12. **`server restore` (S).** Restore is manual shell today
    (`docs/install.md:1187`); add the command with version check, dry run,
    off-host copy target; a documented DR drill.
13. **SLO-based admission and token-level metrics (S–M).** TPOT and
    inter-token histograms, output tokens/s per model and tenant in
    `router/metrics.go`; use `catalog.SLO` in `admission/saturation.go`.
14. **Scheduled and warm-pool autoscaling (S–M).** Time-of-day min/max,
    pre-warm, queue-depth scaling in `catalog.Scaling`,
    `control/reconciler/demand.go`, `headroom.go`.
15. **Secrets management (S–M).** Vault/KMS for `api_key_file`, the DSN, the
    CA key, `cluster_secrets` (`store/secrets.go`, `store/seal.go`);
    envelope-encrypt them in backups (with A8 backups).
16. **GDPR and residency (S).** "Forget user" anonymises `user_id` in usage
    and rollups and erases threads/memory (`store/hierarchy.go:241` leaves
    it today); per-project "never route to an External provider".
17. **Response/semantic cache (M).** Exact-match first, per project,
    opt-in, TTL; semantic via the embeddings models later. Before
    `dispatch` in `router/handlers.go`.
18. **Kubernetes/Helm and an air-gapped bundle (M).** `deploy/`,
    `agent/provision`; `docs/data.md` claims offline operation but install
    assumes the internet.
19. **OpenAPI for the admin API, Terraform provider (M).** Generate from
    `adminapi/api.go` routes via `docgen`.
20. **Tracing beyond the router (S).** Spans for reconciler, placement,
    agent start (`obs/trace.go`, `control/reconciler`, `agent/supervisor`).
21. **Eval harness (M).** `llmgw eval` extending `internal/bench` with
    quality suites gating `model update`.
22. **MIG (M–L).** Treat a MIG slice as a GPU class in
    `agent/discovery/gpu_nvml.go` and `control/placement`.
23. **More engines (M each).** TensorRT-LLM/Triton adapter; Ollama/GGUF
    import. `agent/engines/engine.go` interface and the metric contract in
    `router/scrape`.
24. Lower: precise KV-index routing (plan phase 6), LoRA affinity for
    vLLM, multi-cluster federation via External providers, facility-aware
    placement (`docs/facility-aware-placement.md`), portal i18n, SDK/docs
    portal.

---

## D. Facts worth knowing before starting

- Admin API: `127.0.0.1:9090`, no auth, `ctlClient` in `internal/cli/ctl.go`.
  Metrics on `:9091` with `/prometheus/targets` beside them.
- Store: `internal/store`, migrations under `internal/store/migrations/{sqlite,postgres}`;
  the PoC runs PostgreSQL under Patroni.
- The console (`sudo llmgw`) is `internal/cli/tui_console.go` with actions
  in `tui_actions.go`; every operator-visible change needs a row action or
  a form there, and a line in `docs/console.md`.
- Tests of the console use a fake admin API (`internal/cli/tui_actions_test.go`,
  `adminStub`); tests of serving use fakes in
  `internal/control/modelserve/modelserve_test.go`.
- `go generate ./...` regenerates `docs/reference.md` and
  `internal/docgen/reference_gen.go` whenever a config struct changes; the
  docgen test fails otherwise.