From e01107d525c430159e95415ff76be03e808d9f06 Mon Sep 17 00:00:00 2001 From: Ilker Manap Date: Sun, 4 Oct 2026 09:38:01 +0200 Subject: [PATCH] Write down what the review of 2026-10-04 found to do: security first, then guardrails, then features Co-Authored-By: Claude Fable 5.1 --- BACKLOG.md | 409 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 409 insertions(+) create mode 100644 BACKLOG.md diff --git a/BACKLOG.md b/BACKLOG.md new file mode 100644 index 00000000..f76f6333 --- /dev/null +++ b/BACKLOG.md @@ -0,0 +1,409 @@ +# Backlog + +What a review of the code on 2026-10-04 (commit c2238ab) found should be done +next: security fixes first, then guardrails, then features. Written for +whoever implements it — a person or a model — without the review in hand. +Every item names the files it touches and what "done" is. Paths are relative +to the repository root. Sizes: S about a week, M two to four weeks, L more. + +Conventions this codebase keeps, which every item here must keep too: +- A change is refused before anything is written (validate, then persist). +- Nothing on the serving path logs prompt text, keys or passwords. +- Each item gets unit tests beside the code (`*_test.go`), and the TUI + (`internal/cli/tui_*.go`), `docs/console.md` and `CHANGELOG.md` are + updated when behaviour an operator sees changes. +- `go test ./...`, `go vet ./...` and `govulncheck ./...` pass. No release is + cut and nothing is deployed to the PoC cluster without being told to. + +--- + +## A. Security — do first + +### A1. Toolchain (S, hours) +`govulncheck` finds 11 standard-library vulnerabilities reachable from this +code at Go 1.26.3 (net/http, crypto/x509, os.Root, net/textproto). Raise the +`go` directive in `go.mod` to 1.26.6 or later, rebuild, run `govulncheck +./...` until it reports none reachable. Add `govulncheck` to the Makefile's +check target. + +### A2. Release signing is effectively off (S) +- `internal/release/trusted.pub` is empty, so `TrustedKey()` returns "" and + `release.go:103` skips signature verification. Ship a real public key; + make a missing or invalid signature fatal. Drop the `LLMGW_RELEASE_KEY` + environment override, or honour it only in `-tags dev` builds. +- `internal/onboard/onboard.go:365`: the node-side download checks only a + same-origin `checksums.txt`. Verify the signature there too (simplest: + have the node run `llmgw` itself to fetch and verify once it has any + binary), and `shellQuote` the `Releases` URL, which is interpolated into + single quotes unescaped. +- `internal/agent/upgrade.go:44`: accept only `https://` release bases. +- Done when: a tampered binary or checksums file is refused by self-upgrade, + node onboarding and agent upgrade, with a test for each. + +### A3. Agent port without TLS is anonymous and plaintext (S) +`internal/cli/server.go:423` serves h2c on `:7443` when `agent_tls` is not +configured; `agentsvc` then accepts any node id. Refuse to start control +mode without TLS unless `agent_tls.insecure: true` is set explicitly; when +insecure, bind `agent_listen` to 127.0.0.1 and log a warning at startup and +in `llmgw doctor`. Keep `server init` writing TLS as it does. + +### A4. Model profiles reach root on every node (M) +Profile `args` and `env` from a model definition are appended to the engine +command unchecked (`internal/agent/engines/llamacpp.go:170`, `vllm.go:151`, +`sglang.go:98`); `internal/agent/engines/unsafeenv.go` is a denylist that +misses `NCCL_NET_PLUGIN`, `NCCL_TUNER_PLUGIN`, `NCCL_PROFILER_PLUGIN`, +`CUDA_INJECTION64_PATH`, `NVTX_INJECTION64_PATH`, `TRITON_*` paths; engines +run as root. +- Replace the env denylist with an allowlist (CUDA_/NCCL_ tuning keys minus + anything ending in `_PLUGIN` or containing `INJECTION`, `VLLM_*`, + `HF_*` minus path keys, `OMP_*`, `GGML_*`). Apply at catalog validation + (`internal/control/catalog/catalog.go`, where `UnsafeEnv` is called) and + again in the agent. +- Deny engine flags that write files or run code: `--trust-remote-code`, + `--log-file`, `--slot-save-path`, `--chat-template` with a path, + `--host`, `--api-key`, `--port`; vLLM/SGLang equivalents. Allowlist is + better than denylist if the flag set is enumerable per engine. +- Run engines as an unprivileged user with the video/render groups + (`internal/agent/supervisor`, systemd unit in + `internal/cli/serverinit_cluster.go:52`). Weights cache must be readable + by it. +- Done when: a profile with `--trust-remote-code` or `NCCL_NET_PLUGIN` is + refused at `ctl model add` and at the agent; `ps` shows engines not as + root. + +### A5. Admin API trusts any local process (M) +`internal/cli/server.go:357` serves the admin API and pprof on +`127.0.0.1:9090` with no authentication; `adminapi.actorOf` +(`internal/control/adminapi/api.go:1613`) takes the actor from the +client-sent `X-Llmgw-Actor` header. +- Serve the admin API on a unix socket (`/run/llmgw/admin.sock`, 0660, + group `llmgw-admin`) by default, keeping TCP as an option that requires + a bearer token read from a root-only file (`/etc/llmgw/secrets/admin-token`). +- Take the actor from `SO_PEERCRED` (uid → username) on the socket, or from + the token's name; ignore the header. +- Refuse `admin_listen` on a non-loopback address unless `admin_insecure: + true`. +- Put pprof behind its own `debug.pprof: true` setting, off by default. +- `ctlClient` (`internal/cli/ctl.go`), the console and `node add` must work + over the socket; `sudo llmgw` already runs as root. + +### A6. Patroni REST and etcd are unauthenticated and plaintext (M) +`internal/pgha/config.go:99` writes a `restapi:` with no `authentication` +and `:54` etcd over `http://` with no client auth; `hba.go:56` PATCHes +Patroni's config unauthenticated. Anyone reaching :8008 or :2379 owns the +store. +- Generate a Patroni REST username/password at setup, store it in + `/etc/llmgw/secrets/`, write `restapi.authentication` and + `restapi.certfile/cafile` from the cluster CA; send it from + `internal/pgha/hba.go` and `status.go`. +- etcd: client and peer TLS from the cluster CA with `--client-cert-auth`. +- Document, and have `server init` write, nftables rules limiting + 2379/2380/8008/5432 to member addresses (see A9). +- Done when: an unauthenticated `curl :8008/config` is refused and the + store status in the console still works. + +### A7. PostgreSQL traffic is plaintext (S) +`internal/pgha/config.go:168` and `internal/cli/serverinit_postgres.go:330` +use `sslmode=disable`; HBA lines are `host`. Enable server TLS in Patroni's +postgresql parameters with a certificate from the cluster CA, write +`hostssl` lines, use `sslmode=verify-full&sslrootcert=/etc/llmgw/ca.pem`. +Existing clusters: `llmgw server` must keep working against a store that +has not been migrated, with a warning. + +### A8. Smaller fixes, each S or less +- **Capacity fallback skips the allowlist.** `internal/router/router.go:700` + builds `[canonical]+entry.Fallbacks` without `permits`. Filter the chain + with `r.permits(p, model)`; price against the model that served. +- **Suspended users keep working API keys.** `store.(*DB).Resolve` + (`internal/store/memberships.go`) never joins `users.state`; + `Authenticator.Invalidate()` is never called. Reject keys whose owner is + not active; call `Invalidate` on revoke, suspend, limit change. +- **Join codes never expire server-side.** `internal/join/join.go:103` + checks expiry only in `Parse`; `admit` (`:212`) does not. Check it there; + bind the listener to the chosen address, not `:7444` on all interfaces; + use the peer's address rather than the claimed `hello.Addr`, and show the + joiner's address for the operator to confirm before secrets are sent. +- **Leftover sudoers grant.** `internal/onboard/gossh.go:248` writes + `NOPASSWD:ALL` and removes it only in `Close()` (`:271`). Write the grant + with a self-removal (`at`/systemd-run timer, 30 min) so a crash leaves + nothing; wipe password buffers after use. In `ssh.go:39` pass `--` before + the target. +- **Auth cache and pre-auth DoS.** `internal/auth/auth.go:127`: unbounded + map, one DB query per bogus well-formed key. Bounded LRU, negative cache + with short TTL, per-IP token bucket before the DB. +- **HTTP server hardening.** `server.go:351-360`, the h2c agent server, + both mirror servers: set `ReadHeaderTimeout` (exists), `IdleTimeout`, + `MaxHeaderBytes`, a body-read deadline, `netutil.LimitListener`; connect + `WithReadMaxBytes` on the agent stream. +- **Login/signup abuse.** Default `signup_open: false`; per-IP token buckets + on `/login` and `/signup`; bounded wait for argon2 slots; CSRF token on + the login form; the per-address throttle must not lock the real owner + out (use IP+address pairs). +- **Trusted proxies.** `internal/web/account.go:557`, `landing.go:60` honour + `X-Forwarded-Proto/Host` from anyone; add `router.trusted_proxies` and + only then trust them; `Cache-Control: no-store` on `/setup/*`. +- **Certificate revocation.** `DELETE /api/v1/nodes/{id}` only drops + inventory; add a revoked-CN/serial list in the store checked in + `agentsvc` `Connect` and mirror `onlyClients`; `serverinit_cluster` + always passes `--api-client`. +- **Engine exposure.** Engines bind `0.0.0.0` (`llamacpp.go:102`, + `vllm.go:109`): bind the management address. SGLang receives its key on + the command line (`sglang.go:102`): pass it by env or file. +- **Backups.** `internal/cli/backup.go`: encrypt archives to an `age` + recipient from `backup.recipient` in server.yaml; keep the directory 0700. +- **Python dependency confusion.** `internal/agent/provision/python.go:121` + uses `--extra-index-url` with `unsafe-best-match`. Pin torch to the + PyTorch index explicitly (uv `[[index]] explicit = true`), `first-index`, + hash-pinned lockfiles per engine version. +- **Control-plane SSRF.** `internal/sourcecheck/sourcecheck.go:39` and + `gguf.FromURL` use the default client; reuse the mirror's + `fetchClient(false)` (private-address filter) and refuse a proxy unless + configured; block NAT64/6to4-embedded private addresses in + `internal/mirror/safefetch.go`. +- **Low:** `agentsvc/service.go:430` indexes `n.GPUs[0]` without a length + check (CPU-only node panics the handler); thread ids are client-chosen + with no length cap and shared per project (`router/threads.go`) — cap + length, document, offer per-user scoping; `/v1/models` lists models the + key may not use (`handlers.go:799`); sessions have no idle timeout and + `PurgeSessions`/`PurgeClaims` are never called — add a janitor; add a CSP + header and set `htmx.config.allowEval=false`; cap GGUF array nesting + depth in `internal/gguf/gguf.go`; mirror `.source` sidecars are served by + the public file server — exclude dotfiles from `:8081`; prune + `modelserve.jobs` and the mirror `running` map; KaTeX → 0.16.21+. + +### A9. Firewall (S) +No machine in a default install has a firewall. Have `server init`, `node +add` and `server add` write an nftables ruleset: ssh open; `:8080` open; +`:7443` and `:9091` open to the cluster's networks; `:9100` to the +monitoring host; `:2379/2380/8008/5432` to store members and control +planes; mirror `:8081` to the cluster, `:8082` to control planes; `:9090` +Prometheus and `:3000` Grafana to operators. `llmgw doctor` reports a +missing ruleset. Make it skippable with `--no-firewall`. + +--- + +## B. Guardrails (M–L, build in this order) + +Today nothing on the request path reads, classifies, redacts or filters +message text in either direction (confirmed; the only "moderation" hit is a +comment in `internal/router/budget_fallback.go:20`). Existing controls are +about who, how much and how fast: allowlists (`router/handlers.go:450`), +RPM/TPM/budget (`internal/accounting`), context precheck +(`router/context_limit.go`), admission bands (`router/admission`), +timeouts (`router/proxy/proxy.go:26-57`). Gaps to close on the way: no +default output cap when the client omits `max_tokens`; `catalog.Capabilities` +is advertised but never enforced; tool results are trusted input. + +### B1. Hooks in the proxy and router (S) +- `internal/router/proxy/proxy.go`: add optional `InputRewrite func(body + []byte) ([]byte, Verdict)` on the request and two output hooks: `Filter` + for `relayBuffered` (between `ReadAll` and `WriteHeader`) and + `StreamFilter{OnEvent(data) (emit []string, stop bool); OnEnd()}` for + `relayStream`, called where each whole SSE event is assembled + (`eventBuffer`, before `io.WriteString` at `:343`). Nil hooks cost + nothing. +- `internal/router/handlers.go`: call the input policy after the allowlist + and limits and before admission (around `:395`); the principal, scopes + and the model that will serve are known there. Thread history is spliced + later (`:416`): scan stored turns when written, or move the hook after + `loadThread` (decide; the second is simpler). +- Blocking conventions: input → 400 `invalid_request_error` with + `code: "content_policy_violation"` (today `writeError` always sends + `code: null`); output non-stream → 200 with `finish_reason: + "content_filter"` and empty/redacted content; output stream → a final + chunk `{"choices":[{"index":0,"delta":{},"finish_reason":"content_filter"}]}`, + then `data: [DONE]`, then cancel upstream. Headers `x-llmgw-guard: + blocked|redacted|flagged` and `x-llmgw-guard-policy` set before the proxy + writes headers. +- `internal/router/responses.go`: map `content_filter` to + `response.incomplete` with `incomplete_details.reason: "content_filter"` + (today it always emits `response.completed`, `:525`). +- Tool calls: hold `delta.tool_calls` until `finish_reason: "tool_calls"`, + then validate; `replyDelta` (`reply.go:33`) must also read + `tool_calls[].function.arguments`. +- Accounting: a blocked input is recorded like `RecordRefused` (nothing + charged); a blocked output is charged (the GPU was used). `store.Usage` + gains `GuardAction`, `GuardPolicy`, `GuardCategory`, `GuardMs`. + +### B2. Package `internal/guard` (M) +- `Policy`, `Effective` resolution over the tenancy tree with the same + narrowing rule as limits (`internal/store/sqlite.go:383-416` `tighten`): + deny sets union, allow sets intersect, `mode` takes the strictest of + `enforce > shadow > off`, `fail` takes `closed > open`, `no_store` at any + level wins, thresholds take the minimum. +- Store: `guard_policies(scope, scope_id, policy json)` and + `guard_events(request_id, scopes, policy, rule, category, action, stage, + score, judge_ms, shadow)` — ids, categories, offsets and lengths only, + never text. Migrations for sqlite and postgres. Pruned with + `usage_retention`. +- `guard/detect`: regex/keyword sets; PII with validators — email, phone, + IBAN (mod-97), card (Luhn), TCKN (checksum), IP, secrets (`sk-`, AWS, + JWT, PEM); normaliser (NFKC, strip zero-width/bidi, homoglyph fold, + base64/hex/URL decode to depth 2); injection heuristics (role-switch + phrases, fake chat-template tokens, long base64, invisible Unicode) — + default action `flag`; language id. +- Redaction rewrites `messages[].content` in the upstream body with + placeholders (`[EMAIL_1]`) via the same `map[string]json.RawMessage` + round-trip `withModel`/`spliceThread` use; an optional re-identification + map lives only in request memory and restores originals in the output. + Redact before prefix routing so `PromptPrefix` hashes the redacted text. +- `guard/toolpolicy`: tool name allow/deny (strip or 400), JSON-Schema + validation of assembled arguments against the declared `parameters`, + argument regex rules, `max_calls_per_turn`. Enforce + `Capabilities.Tools`/`Vision` here: a `tools` array or image parts to a + model without the capability → 400. +- `guard/structured`: validate `response_format.json_schema` on the final + text → `content_filter`, or flag in shadow. +- Output constraints: inject a default `max_tokens` when absent (from the + policy or `MaxOutput`), language check, structured check. +- System-prompt pinning per project: prepend before `checkContextFits` and + before `spliceThread`; optionally drop client `system`/`developer` + roles; exclude the pin from stored thread turns; apply to Responses + `instructions` in `toChat`. +- `no_store`: `loadThread` and `PutNote` refuse; `captureBytes()` returns 0; + skip the prefix-tree observe (`router.go:840`); disable External + fallbacks for the scope (filter in `dispatch` and `budgetFallback`). + +### B3. Judge model on the cluster (M) +- `catalog.Model.Role: "guard"` plus `Guard{Format: llamaguard3 | + shieldgemma | json, Categories}`. Guard models are hidden from + `/v1/models`, the chat picker and allowlists; placed like any model, + class `critical`, `scaling.min ≥ 1`, at least 2 replicas on different + nodes. +- `guard/judge`: calls the guard model through the router's own `dispatch` + with an internal principal and its own admission band (never the + tenant's queue); parsers per format (Llama Guard `safe`/`unsafe\nS1,S10`; + ShieldGemma yes/no probability → per-tenant thresholds; generic JSON + verdict); verdict cache keyed by hash of normalised new turns (5 min); + `latency_budget` (default 250 ms) then `fail` open/closed; input judge + runs in parallel with admission queueing. +- Streaming strategy: default sentence-buffered scanning (regex/PII per + sentence or ~200 chars before release; judge every 400–800 chars on the + cumulative tail; final judge at end); `buffer_all` for `fail: closed` + scopes. Document that released text cannot be retracted. +- Shadow mode runs everything asynchronously after the response + (`context.WithoutCancel`, like `saveThread`) and only records. + +### B4. Config, metrics, UI (S) +- Gateway-level policy library in `server.yaml` under `guard:` (see the + YAML in the review, reproduced in `docs/guardrails.md` to be written); + attachments to scopes in the store via `llmgw keys guard`, the console + ("Guardrails…" beside "Limits…" in `internal/cli/tui_people.go`, with + "inherited from team:X" badges) and the portal. +- Metrics in `router/metrics.go` style: + `llmgw_guard_checks_total{stage,policy,action,category,mode}`, + `llmgw_guard_latency_seconds{stage,judge_model}`, + `llmgw_guard_judge_errors_total{reason}`, `llmgw_guard_fail_open_total`, + `llmgw_guard_stream_stops_total`, `llmgw_guard_redactions_total{kind}`, + `llmgw_tenant_guard_blocks_total{tenant}`. Alerts: fail-open > 1% for 5m, + judge p95 over budget, judge has no ready replicas, per-tenant block rate + > 3× its 7-day baseline, any `secret` detected in output. +- Portal `/admin/guard`: violations by tenant/category/policy over time, + shadow vs enforce, judge health, recent events by request id; `ctl + request ` shows the verdict; chat page renders `content_filter` as + "stopped by policy X"; `/me` lists effective policies. +- Update `docs/data.md`: the judge sees all text but in-cluster; External + fallbacks send prompts outside the cluster (the current "nothing leaves + the cluster" is wrong); `no_store` semantics. +- Never log: prompt/completion text, PII values, the re-identification + map, judge inputs or rationale that quotes input. Check `accessLog`'s + `err` field and span errors. + +--- + +## C. Features, ranked by value × feasibility + +1. **Audit log and roles (M).** Today "audit" is slog lines + (`internal/web/admin_users.go:403-480`) and roles are the `IsAdmin` flag + plus an org "manager" who can view. Add `audit_events` (who, what, + before/after, source) written by the `adminAction`/`tenancyAction` + wrappers and by every admin-API mutation; roles viewer / operator / + tenant-admin / billing; `ctl audit`. Depends on A5 for a real actor. +2. **OIDC/SSO login (M), then SCIM (M).** `internal/account/service.go`, + `internal/web/account.go`; map IdP groups to org/team/admin. +3. **Service accounts and key lifecycle (S).** Per-org maximum key TTL, + `keys rotate` with overlap, last-used shown, project-owned service keys + listed apart. `internal/cli/keys.go`, `internal/store/keys.go`, + `internal/auth/auth.go`. +4. **Usage export (S).** CSV/Parquet by org/team/project/model/period from + the rollups (`internal/store/usageseries.go`), `GET + /api/v1/usage/export`, optional scheduled push to S3/SFTP. +5. **Alerting out of the box (S).** `deploy/monitoring/alertmanager.yml` + ships `receiver: none`; add Slack/email/webhook/PagerDuty receiver + templates driven by `server init` or the console; llmgw-native webhooks + for model parked, budget spent, update available (from the reconciler + and `modelserve`). +6. **Request metadata and tags (S).** Accept `metadata`/`user` from the + body or `x-llmgw-tags`; store on the usage row; tenant metric labels + behind an allowlist. `router/handlers.go` parse, `accounting/recorder.go`. +7. **Canary and weighted A/B between model revisions (M).** + `control/modelserve/updates.go` `replace` swaps every replica; add + weighted routing across two revisions/aliases (scheduler filter, alias + resolution in `router/router.go:569`), a header/key pin, automatic + rollback on error-rate or TTFT regression from per-replica metrics. +8. **Batch API and Files (M–L).** `/v1/files`, `/v1/batches`, JSONL + storage, a worker feeding the bronze band, accounting. New + `internal/router/batch`. +9. **Request/response logging and replay (M).** Opt-in per project with + retention like threads; sampled or full; redaction hooks from B; + `ctl request --body`; replay against another model. +10. **OpenAI compatibility gaps (S–M each).** Responses: `text.format` → + `response_format`, `input_image`, `previous_response_id` via threads + (`router/responses.go`). Rerank/score passthrough for vLLM, + `/v1/audio/transcriptions` for a whisper engine, `/v1/moderations` + mapped to the guard model. +11. **Anthropic `/v1/messages` (M).** Translate like `responses.go`. +12. **`server restore` (S).** Restore is manual shell today + (`docs/install.md:1187`); add the command with version check, dry run, + off-host copy target; a documented DR drill. +13. **SLO-based admission and token-level metrics (S–M).** TPOT and + inter-token histograms, output tokens/s per model and tenant in + `router/metrics.go`; use `catalog.SLO` in `admission/saturation.go`. +14. **Scheduled and warm-pool autoscaling (S–M).** Time-of-day min/max, + pre-warm, queue-depth scaling in `catalog.Scaling`, + `control/reconciler/demand.go`, `headroom.go`. +15. **Secrets management (S–M).** Vault/KMS for `api_key_file`, the DSN, the + CA key, `cluster_secrets` (`store/secrets.go`, `store/seal.go`); + envelope-encrypt them in backups (with A8 backups). +16. **GDPR and residency (S).** "Forget user" anonymises `user_id` in usage + and rollups and erases threads/memory (`store/hierarchy.go:241` leaves + it today); per-project "never route to an External provider". +17. **Response/semantic cache (M).** Exact-match first, per project, + opt-in, TTL; semantic via the embeddings models later. Before + `dispatch` in `router/handlers.go`. +18. **Kubernetes/Helm and an air-gapped bundle (M).** `deploy/`, + `agent/provision`; `docs/data.md` claims offline operation but install + assumes the internet. +19. **OpenAPI for the admin API, Terraform provider (M).** Generate from + `adminapi/api.go` routes via `docgen`. +20. **Tracing beyond the router (S).** Spans for reconciler, placement, + agent start (`obs/trace.go`, `control/reconciler`, `agent/supervisor`). +21. **Eval harness (M).** `llmgw eval` extending `internal/bench` with + quality suites gating `model update`. +22. **MIG (M–L).** Treat a MIG slice as a GPU class in + `agent/discovery/gpu_nvml.go` and `control/placement`. +23. **More engines (M each).** TensorRT-LLM/Triton adapter; Ollama/GGUF + import. `agent/engines/engine.go` interface and the metric contract in + `router/scrape`. +24. Lower: precise KV-index routing (plan phase 6), LoRA affinity for + vLLM, multi-cluster federation via External providers, facility-aware + placement (`docs/facility-aware-placement.md`), portal i18n, SDK/docs + portal. + +--- + +## D. Facts worth knowing before starting + +- Admin API: `127.0.0.1:9090`, no auth, `ctlClient` in `internal/cli/ctl.go`. + Metrics on `:9091` with `/prometheus/targets` beside them. +- Store: `internal/store`, migrations under `internal/store/migrations/{sqlite,postgres}`; + the PoC runs PostgreSQL under Patroni. +- The console (`sudo llmgw`) is `internal/cli/tui_console.go` with actions + in `tui_actions.go`; every operator-visible change needs a row action or + a form there, and a line in `docs/console.md`. +- Tests of the console use a fake admin API (`internal/cli/tui_actions_test.go`, + `adminStub`); tests of serving use fakes in + `internal/control/modelserve/modelserve_test.go`. +- `go generate ./...` regenerates `docs/reference.md` and + `internal/docgen/reference_gen.go` whenever a config struct changes; the + docgen test fails otherwise.