Write down what the review of 2026-10-04 found to do: security first, then guardrails, then features
CI / test (push) Successful in 4m1s

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Fable 5.1 committed 2026-10-04 09:38:01 +02:00
1 parent c2238abf4e
commit e01107d525
1 file changed
+409
+409
View File
@@ -0,0 +1,409 @@
# Backlog
What a review of the code on 2026-10-04 (commit c2238ab) found should be done
next: security fixes first, then guardrails, then features. Written for
whoever implements it — a person or a model — without the review in hand.
Every item names the files it touches and what "done" is. Paths are relative
to the repository root. Sizes: S about a week, M two to four weeks, L more.
Conventions this codebase keeps, which every item here must keep too:
- A change is refused before anything is written (validate, then persist).
- Nothing on the serving path logs prompt text, keys or passwords.
- Each item gets unit tests beside the code (`*_test.go`), and the TUI
(`internal/cli/tui_*.go`), `docs/console.md` and `CHANGELOG.md` are
updated when behaviour an operator sees changes.
- `go test ./...`, `go vet ./...` and `govulncheck ./...` pass. No release is
cut and nothing is deployed to the PoC cluster without being told to.
---
## A. Security — do first
### A1. Toolchain (S, hours)
`govulncheck` finds 11 standard-library vulnerabilities reachable from this
code at Go 1.26.3 (net/http, crypto/x509, os.Root, net/textproto). Raise the
`go` directive in `go.mod` to 1.26.6 or later, rebuild, run `govulncheck
./...` until it reports none reachable. Add `govulncheck` to the Makefile's
check target.
### A2. Release signing is effectively off (S)
- `internal/release/trusted.pub` is empty, so `TrustedKey()` returns "" and
`release.go:103` skips signature verification. Ship a real public key;
make a missing or invalid signature fatal. Drop the `LLMGW_RELEASE_KEY`
environment override, or honour it only in `-tags dev` builds.
- `internal/onboard/onboard.go:365`: the node-side download checks only a
same-origin `checksums.txt`. Verify the signature there too (simplest:
have the node run `llmgw` itself to fetch and verify once it has any
binary), and `shellQuote` the `Releases` URL, which is interpolated into
single quotes unescaped.
- `internal/agent/upgrade.go:44`: accept only `https://` release bases.
- Done when: a tampered binary or checksums file is refused by self-upgrade,
node onboarding and agent upgrade, with a test for each.
### A3. Agent port without TLS is anonymous and plaintext (S)
`internal/cli/server.go:423` serves h2c on `:7443` when `agent_tls` is not
configured; `agentsvc` then accepts any node id. Refuse to start control
mode without TLS unless `agent_tls.insecure: true` is set explicitly; when
insecure, bind `agent_listen` to 127.0.0.1 and log a warning at startup and
in `llmgw doctor`. Keep `server init` writing TLS as it does.
### A4. Model profiles reach root on every node (M)
Profile `args` and `env` from a model definition are appended to the engine
command unchecked (`internal/agent/engines/llamacpp.go:170`, `vllm.go:151`,
`sglang.go:98`); `internal/agent/engines/unsafeenv.go` is a denylist that
misses `NCCL_NET_PLUGIN`, `NCCL_TUNER_PLUGIN`, `NCCL_PROFILER_PLUGIN`,
`CUDA_INJECTION64_PATH`, `NVTX_INJECTION64_PATH`, `TRITON_*` paths; engines
run as root.
- Replace the env denylist with an allowlist (CUDA_/NCCL_ tuning keys minus
anything ending in `_PLUGIN` or containing `INJECTION`, `VLLM_*`,
`HF_*` minus path keys, `OMP_*`, `GGML_*`). Apply at catalog validation
(`internal/control/catalog/catalog.go`, where `UnsafeEnv` is called) and
again in the agent.
- Deny engine flags that write files or run code: `--trust-remote-code`,
`--log-file`, `--slot-save-path`, `--chat-template` with a path,
`--host`, `--api-key`, `--port`; vLLM/SGLang equivalents. Allowlist is
better than denylist if the flag set is enumerable per engine.
- Run engines as an unprivileged user with the video/render groups
(`internal/agent/supervisor`, systemd unit in
`internal/cli/serverinit_cluster.go:52`). Weights cache must be readable
by it.
- Done when: a profile with `--trust-remote-code` or `NCCL_NET_PLUGIN` is
refused at `ctl model add` and at the agent; `ps` shows engines not as
root.
### A5. Admin API trusts any local process (M)
`internal/cli/server.go:357` serves the admin API and pprof on
`127.0.0.1:9090` with no authentication; `adminapi.actorOf`
(`internal/control/adminapi/api.go:1613`) takes the actor from the
client-sent `X-Llmgw-Actor` header.
- Serve the admin API on a unix socket (`/run/llmgw/admin.sock`, 0660,
group `llmgw-admin`) by default, keeping TCP as an option that requires
a bearer token read from a root-only file (`/etc/llmgw/secrets/admin-token`).
- Take the actor from `SO_PEERCRED` (uid → username) on the socket, or from
the token's name; ignore the header.
- Refuse `admin_listen` on a non-loopback address unless `admin_insecure:
true`.
- Put pprof behind its own `debug.pprof: true` setting, off by default.
- `ctlClient` (`internal/cli/ctl.go`), the console and `node add` must work
over the socket; `sudo llmgw` already runs as root.
### A6. Patroni REST and etcd are unauthenticated and plaintext (M)
`internal/pgha/config.go:99` writes a `restapi:` with no `authentication`
and `:54` etcd over `http://` with no client auth; `hba.go:56` PATCHes
Patroni's config unauthenticated. Anyone reaching :8008 or :2379 owns the
store.
- Generate a Patroni REST username/password at setup, store it in
`/etc/llmgw/secrets/`, write `restapi.authentication` and
`restapi.certfile/cafile` from the cluster CA; send it from
`internal/pgha/hba.go` and `status.go`.
- etcd: client and peer TLS from the cluster CA with `--client-cert-auth`.
- Document, and have `server init` write, nftables rules limiting
2379/2380/8008/5432 to member addresses (see A9).
- Done when: an unauthenticated `curl :8008/config` is refused and the
store status in the console still works.
### A7. PostgreSQL traffic is plaintext (S)
`internal/pgha/config.go:168` and `internal/cli/serverinit_postgres.go:330`
use `sslmode=disable`; HBA lines are `host`. Enable server TLS in Patroni's
postgresql parameters with a certificate from the cluster CA, write
`hostssl` lines, use `sslmode=verify-full&sslrootcert=/etc/llmgw/ca.pem`.
Existing clusters: `llmgw server` must keep working against a store that
has not been migrated, with a warning.
### A8. Smaller fixes, each S or less
- **Capacity fallback skips the allowlist.** `internal/router/router.go:700`
builds `[canonical]+entry.Fallbacks` without `permits`. Filter the chain
with `r.permits(p, model)`; price against the model that served.
- **Suspended users keep working API keys.** `store.(*DB).Resolve`
(`internal/store/memberships.go`) never joins `users.state`;
`Authenticator.Invalidate()` is never called. Reject keys whose owner is
not active; call `Invalidate` on revoke, suspend, limit change.
- **Join codes never expire server-side.** `internal/join/join.go:103`
checks expiry only in `Parse`; `admit` (`:212`) does not. Check it there;
bind the listener to the chosen address, not `:7444` on all interfaces;
use the peer's address rather than the claimed `hello.Addr`, and show the
joiner's address for the operator to confirm before secrets are sent.
- **Leftover sudoers grant.** `internal/onboard/gossh.go:248` writes
`NOPASSWD:ALL` and removes it only in `Close()` (`:271`). Write the grant
with a self-removal (`at`/systemd-run timer, 30 min) so a crash leaves
nothing; wipe password buffers after use. In `ssh.go:39` pass `--` before
the target.
- **Auth cache and pre-auth DoS.** `internal/auth/auth.go:127`: unbounded
map, one DB query per bogus well-formed key. Bounded LRU, negative cache
with short TTL, per-IP token bucket before the DB.
- **HTTP server hardening.** `server.go:351-360`, the h2c agent server,
both mirror servers: set `ReadHeaderTimeout` (exists), `IdleTimeout`,
`MaxHeaderBytes`, a body-read deadline, `netutil.LimitListener`; connect
`WithReadMaxBytes` on the agent stream.
- **Login/signup abuse.** Default `signup_open: false`; per-IP token buckets
on `/login` and `/signup`; bounded wait for argon2 slots; CSRF token on
the login form; the per-address throttle must not lock the real owner
out (use IP+address pairs).
- **Trusted proxies.** `internal/web/account.go:557`, `landing.go:60` honour
`X-Forwarded-Proto/Host` from anyone; add `router.trusted_proxies` and
only then trust them; `Cache-Control: no-store` on `/setup/*`.
- **Certificate revocation.** `DELETE /api/v1/nodes/{id}` only drops
inventory; add a revoked-CN/serial list in the store checked in
`agentsvc` `Connect` and mirror `onlyClients`; `serverinit_cluster`
always passes `--api-client`.
- **Engine exposure.** Engines bind `0.0.0.0` (`llamacpp.go:102`,
`vllm.go:109`): bind the management address. SGLang receives its key on
the command line (`sglang.go:102`): pass it by env or file.
- **Backups.** `internal/cli/backup.go`: encrypt archives to an `age`
recipient from `backup.recipient` in server.yaml; keep the directory 0700.
- **Python dependency confusion.** `internal/agent/provision/python.go:121`
uses `--extra-index-url` with `unsafe-best-match`. Pin torch to the
PyTorch index explicitly (uv `[[index]] explicit = true`), `first-index`,
hash-pinned lockfiles per engine version.
- **Control-plane SSRF.** `internal/sourcecheck/sourcecheck.go:39` and
`gguf.FromURL` use the default client; reuse the mirror's
`fetchClient(false)` (private-address filter) and refuse a proxy unless
configured; block NAT64/6to4-embedded private addresses in
`internal/mirror/safefetch.go`.
- **Low:** `agentsvc/service.go:430` indexes `n.GPUs[0]` without a length
check (CPU-only node panics the handler); thread ids are client-chosen
with no length cap and shared per project (`router/threads.go`) — cap
length, document, offer per-user scoping; `/v1/models` lists models the
key may not use (`handlers.go:799`); sessions have no idle timeout and
`PurgeSessions`/`PurgeClaims` are never called — add a janitor; add a CSP
header and set `htmx.config.allowEval=false`; cap GGUF array nesting
depth in `internal/gguf/gguf.go`; mirror `.source` sidecars are served by
the public file server — exclude dotfiles from `:8081`; prune
`modelserve.jobs` and the mirror `running` map; KaTeX → 0.16.21+.
### A9. Firewall (S)
No machine in a default install has a firewall. Have `server init`, `node
add` and `server add` write an nftables ruleset: ssh open; `:8080` open;
`:7443` and `:9091` open to the cluster's networks; `:9100` to the
monitoring host; `:2379/2380/8008/5432` to store members and control
planes; mirror `:8081` to the cluster, `:8082` to control planes; `:9090`
Prometheus and `:3000` Grafana to operators. `llmgw doctor` reports a
missing ruleset. Make it skippable with `--no-firewall`.
---
## B. Guardrails (M–L, build in this order)
Today nothing on the request path reads, classifies, redacts or filters
message text in either direction (confirmed; the only "moderation" hit is a
comment in `internal/router/budget_fallback.go:20`). Existing controls are
about who, how much and how fast: allowlists (`router/handlers.go:450`),
RPM/TPM/budget (`internal/accounting`), context precheck
(`router/context_limit.go`), admission bands (`router/admission`),
timeouts (`router/proxy/proxy.go:26-57`). Gaps to close on the way: no
default output cap when the client omits `max_tokens`; `catalog.Capabilities`
is advertised but never enforced; tool results are trusted input.
### B1. Hooks in the proxy and router (S)
- `internal/router/proxy/proxy.go`: add optional `InputRewrite func(body
[]byte) ([]byte, Verdict)` on the request and two output hooks: `Filter`
for `relayBuffered` (between `ReadAll` and `WriteHeader`) and
`StreamFilter{OnEvent(data) (emit []string, stop bool); OnEnd()}` for
`relayStream`, called where each whole SSE event is assembled
(`eventBuffer`, before `io.WriteString` at `:343`). Nil hooks cost
nothing.
- `internal/router/handlers.go`: call the input policy after the allowlist
and limits and before admission (around `:395`); the principal, scopes
and the model that will serve are known there. Thread history is spliced
later (`:416`): scan stored turns when written, or move the hook after
`loadThread` (decide; the second is simpler).
- Blocking conventions: input → 400 `invalid_request_error` with
`code: "content_policy_violation"` (today `writeError` always sends
`code: null`); output non-stream → 200 with `finish_reason:
"content_filter"` and empty/redacted content; output stream → a final
chunk `{"choices":[{"index":0,"delta":{},"finish_reason":"content_filter"}]}`,
then `data: [DONE]`, then cancel upstream. Headers `x-llmgw-guard:
blocked|redacted|flagged` and `x-llmgw-guard-policy` set before the proxy
writes headers.
- `internal/router/responses.go`: map `content_filter` to
`response.incomplete` with `incomplete_details.reason: "content_filter"`
(today it always emits `response.completed`, `:525`).
- Tool calls: hold `delta.tool_calls` until `finish_reason: "tool_calls"`,
then validate; `replyDelta` (`reply.go:33`) must also read
`tool_calls[].function.arguments`.
- Accounting: a blocked input is recorded like `RecordRefused` (nothing
charged); a blocked output is charged (the GPU was used). `store.Usage`
gains `GuardAction`, `GuardPolicy`, `GuardCategory`, `GuardMs`.
### B2. Package `internal/guard` (M)
- `Policy`, `Effective` resolution over the tenancy tree with the same
narrowing rule as limits (`internal/store/sqlite.go:383-416` `tighten`):
deny sets union, allow sets intersect, `mode` takes the strictest of
`enforce > shadow > off`, `fail` takes `closed > open`, `no_store` at any
level wins, thresholds take the minimum.
- Store: `guard_policies(scope, scope_id, policy json)` and
`guard_events(request_id, scopes, policy, rule, category, action, stage,
score, judge_ms, shadow)` — ids, categories, offsets and lengths only,
never text. Migrations for sqlite and postgres. Pruned with
`usage_retention`.
- `guard/detect`: regex/keyword sets; PII with validators — email, phone,
IBAN (mod-97), card (Luhn), TCKN (checksum), IP, secrets (`sk-`, AWS,
JWT, PEM); normaliser (NFKC, strip zero-width/bidi, homoglyph fold,
base64/hex/URL decode to depth 2); injection heuristics (role-switch
phrases, fake chat-template tokens, long base64, invisible Unicode) —
default action `flag`; language id.
- Redaction rewrites `messages[].content` in the upstream body with
placeholders (`[EMAIL_1]`) via the same `map[string]json.RawMessage`
round-trip `withModel`/`spliceThread` use; an optional re-identification
map lives only in request memory and restores originals in the output.
Redact before prefix routing so `PromptPrefix` hashes the redacted text.
- `guard/toolpolicy`: tool name allow/deny (strip or 400), JSON-Schema
validation of assembled arguments against the declared `parameters`,
argument regex rules, `max_calls_per_turn`. Enforce
`Capabilities.Tools`/`Vision` here: a `tools` array or image parts to a
model without the capability → 400.
- `guard/structured`: validate `response_format.json_schema` on the final
text → `content_filter`, or flag in shadow.
- Output constraints: inject a default `max_tokens` when absent (from the
policy or `MaxOutput`), language check, structured check.
- System-prompt pinning per project: prepend before `checkContextFits` and
before `spliceThread`; optionally drop client `system`/`developer`
roles; exclude the pin from stored thread turns; apply to Responses
`instructions` in `toChat`.
- `no_store`: `loadThread` and `PutNote` refuse; `captureBytes()` returns 0;
skip the prefix-tree observe (`router.go:840`); disable External
fallbacks for the scope (filter in `dispatch` and `budgetFallback`).
### B3. Judge model on the cluster (M)
- `catalog.Model.Role: "guard"` plus `Guard{Format: llamaguard3 |
shieldgemma | json, Categories}`. Guard models are hidden from
`/v1/models`, the chat picker and allowlists; placed like any model,
class `critical`, `scaling.min ≥ 1`, at least 2 replicas on different
nodes.
- `guard/judge`: calls the guard model through the router's own `dispatch`
with an internal principal and its own admission band (never the
tenant's queue); parsers per format (Llama Guard `safe`/`unsafe\nS1,S10`;
ShieldGemma yes/no probability → per-tenant thresholds; generic JSON
verdict); verdict cache keyed by hash of normalised new turns (5 min);
`latency_budget` (default 250 ms) then `fail` open/closed; input judge
runs in parallel with admission queueing.
- Streaming strategy: default sentence-buffered scanning (regex/PII per
sentence or ~200 chars before release; judge every 400–800 chars on the
cumulative tail; final judge at end); `buffer_all` for `fail: closed`
scopes. Document that released text cannot be retracted.
- Shadow mode runs everything asynchronously after the response
(`context.WithoutCancel`, like `saveThread`) and only records.
### B4. Config, metrics, UI (S)
- Gateway-level policy library in `server.yaml` under `guard:` (see the
YAML in the review, reproduced in `docs/guardrails.md` to be written);
attachments to scopes in the store via `llmgw keys guard`, the console
("Guardrails…" beside "Limits…" in `internal/cli/tui_people.go`, with
"inherited from team:X" badges) and the portal.
- Metrics in `router/metrics.go` style:
`llmgw_guard_checks_total{stage,policy,action,category,mode}`,
`llmgw_guard_latency_seconds{stage,judge_model}`,
`llmgw_guard_judge_errors_total{reason}`, `llmgw_guard_fail_open_total`,
`llmgw_guard_stream_stops_total`, `llmgw_guard_redactions_total{kind}`,
`llmgw_tenant_guard_blocks_total{tenant}`. Alerts: fail-open > 1% for 5m,
judge p95 over budget, judge has no ready replicas, per-tenant block rate
> 3× its 7-day baseline, any `secret` detected in output.
- Portal `/admin/guard`: violations by tenant/category/policy over time,
shadow vs enforce, judge health, recent events by request id; `ctl
request <id>` shows the verdict; chat page renders `content_filter` as
"stopped by policy X"; `/me` lists effective policies.
- Update `docs/data.md`: the judge sees all text but in-cluster; External
fallbacks send prompts outside the cluster (the current "nothing leaves
the cluster" is wrong); `no_store` semantics.
- Never log: prompt/completion text, PII values, the re-identification
map, judge inputs or rationale that quotes input. Check `accessLog`'s
`err` field and span errors.
---
## C. Features, ranked by value × feasibility
1. **Audit log and roles (M).** Today "audit" is slog lines
(`internal/web/admin_users.go:403-480`) and roles are the `IsAdmin` flag
plus an org "manager" who can view. Add `audit_events` (who, what,
before/after, source) written by the `adminAction`/`tenancyAction`
wrappers and by every admin-API mutation; roles viewer / operator /
tenant-admin / billing; `ctl audit`. Depends on A5 for a real actor.
2. **OIDC/SSO login (M), then SCIM (M).** `internal/account/service.go`,
`internal/web/account.go`; map IdP groups to org/team/admin.
3. **Service accounts and key lifecycle (S).** Per-org maximum key TTL,
`keys rotate` with overlap, last-used shown, project-owned service keys
listed apart. `internal/cli/keys.go`, `internal/store/keys.go`,
`internal/auth/auth.go`.
4. **Usage export (S).** CSV/Parquet by org/team/project/model/period from
the rollups (`internal/store/usageseries.go`), `GET
/api/v1/usage/export`, optional scheduled push to S3/SFTP.
5. **Alerting out of the box (S).** `deploy/monitoring/alertmanager.yml`
ships `receiver: none`; add Slack/email/webhook/PagerDuty receiver
templates driven by `server init` or the console; llmgw-native webhooks
for model parked, budget spent, update available (from the reconciler
and `modelserve`).
6. **Request metadata and tags (S).** Accept `metadata`/`user` from the
body or `x-llmgw-tags`; store on the usage row; tenant metric labels
behind an allowlist. `router/handlers.go` parse, `accounting/recorder.go`.
7. **Canary and weighted A/B between model revisions (M).**
`control/modelserve/updates.go` `replace` swaps every replica; add
weighted routing across two revisions/aliases (scheduler filter, alias
resolution in `router/router.go:569`), a header/key pin, automatic
rollback on error-rate or TTFT regression from per-replica metrics.
8. **Batch API and Files (M–L).** `/v1/files`, `/v1/batches`, JSONL
storage, a worker feeding the bronze band, accounting. New
`internal/router/batch`.
9. **Request/response logging and replay (M).** Opt-in per project with
retention like threads; sampled or full; redaction hooks from B;
`ctl request <id> --body`; replay against another model.
10. **OpenAI compatibility gaps (S–M each).** Responses: `text.format` →
`response_format`, `input_image`, `previous_response_id` via threads
(`router/responses.go`). Rerank/score passthrough for vLLM,
`/v1/audio/transcriptions` for a whisper engine, `/v1/moderations`
mapped to the guard model.
11. **Anthropic `/v1/messages` (M).** Translate like `responses.go`.
12. **`server restore` (S).** Restore is manual shell today
(`docs/install.md:1187`); add the command with version check, dry run,
off-host copy target; a documented DR drill.
13. **SLO-based admission and token-level metrics (S–M).** TPOT and
inter-token histograms, output tokens/s per model and tenant in
`router/metrics.go`; use `catalog.SLO` in `admission/saturation.go`.
14. **Scheduled and warm-pool autoscaling (S–M).** Time-of-day min/max,
pre-warm, queue-depth scaling in `catalog.Scaling`,
`control/reconciler/demand.go`, `headroom.go`.
15. **Secrets management (S–M).** Vault/KMS for `api_key_file`, the DSN, the
CA key, `cluster_secrets` (`store/secrets.go`, `store/seal.go`);
envelope-encrypt them in backups (with A8 backups).
16. **GDPR and residency (S).** "Forget user" anonymises `user_id` in usage
and rollups and erases threads/memory (`store/hierarchy.go:241` leaves
it today); per-project "never route to an External provider".
17. **Response/semantic cache (M).** Exact-match first, per project,
opt-in, TTL; semantic via the embeddings models later. Before
`dispatch` in `router/handlers.go`.
18. **Kubernetes/Helm and an air-gapped bundle (M).** `deploy/`,
`agent/provision`; `docs/data.md` claims offline operation but install
assumes the internet.
19. **OpenAPI for the admin API, Terraform provider (M).** Generate from
`adminapi/api.go` routes via `docgen`.
20. **Tracing beyond the router (S).** Spans for reconciler, placement,
agent start (`obs/trace.go`, `control/reconciler`, `agent/supervisor`).
21. **Eval harness (M).** `llmgw eval` extending `internal/bench` with
quality suites gating `model update`.
22. **MIG (M–L).** Treat a MIG slice as a GPU class in
`agent/discovery/gpu_nvml.go` and `control/placement`.
23. **More engines (M each).** TensorRT-LLM/Triton adapter; Ollama/GGUF
import. `agent/engines/engine.go` interface and the metric contract in
`router/scrape`.
24. Lower: precise KV-index routing (plan phase 6), LoRA affinity for
vLLM, multi-cluster federation via External providers, facility-aware
placement (`docs/facility-aware-placement.md`), portal i18n, SDK/docs
portal.
---
## D. Facts worth knowing before starting
- Admin API: `127.0.0.1:9090`, no auth, `ctlClient` in `internal/cli/ctl.go`.
Metrics on `:9091` with `/prometheus/targets` beside them.
- Store: `internal/store`, migrations under `internal/store/migrations/{sqlite,postgres}`;
the PoC runs PostgreSQL under Patroni.
- The console (`sudo llmgw`) is `internal/cli/tui_console.go` with actions
in `tui_actions.go`; every operator-visible change needs a row action or
a form there, and a line in `docs/console.md`.
- Tests of the console use a fake admin API (`internal/cli/tui_actions_test.go`,
`adminStub`); tests of serving use fakes in
`internal/control/modelserve/modelserve_test.go`.
- `go generate ./...` regenerates `docs/reference.md` and
`internal/docgen/reference_gen.go` whenever a config struct changes; the
docgen test fails otherwise.