Ilker ManapandClaude Opus 5.5
7f320c5f65
A finished job kept the last word heard from ComfyUI, the node running; it is now said done before the job is marked completed. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
1270 lines
80 KiB
Markdown
1270 lines
80 KiB
Markdown
# Backlog
|
||
|
||
What a review of the code on 2026-10-04 (commit c2238ab) found should be done
|
||
next: security fixes first, then guardrails, then features. Written for
|
||
whoever implements it — a person or a model — without the review in hand.
|
||
Every item names the files it touches and what "done" is. Paths are relative
|
||
to the repository root. Sizes: S about a week, M two to four weeks, L more.
|
||
|
||
Conventions this codebase keeps, which every item here must keep too:
|
||
- A change is refused before anything is written (validate, then persist).
|
||
- Nothing on the serving path logs prompt text, keys or passwords.
|
||
- Each item gets unit tests beside the code (`*_test.go`), and the TUI
|
||
(`internal/cli/tui_*.go`), `docs/console.md` and `CHANGELOG.md` are
|
||
updated when behaviour an operator sees changes.
|
||
- `go test ./...`, `go vet ./...` and `govulncheck ./...` pass. No release is
|
||
cut and nothing is deployed to the PoC cluster without being told to.
|
||
|
||
---
|
||
|
||
## A. Security — do first
|
||
|
||
**Status (2026-10-04): done and merged.** A1–A9, A6b included — etcd and
|
||
Patroni's API on TLS, the PoC moved in maintenance mode without a failover.
|
||
Deliberately narrower than written: the admin API is a token on loopback
|
||
rather than a unix socket (A5); thread ids are bounded and documented as
|
||
project-wide rather than scoped per user (A8); Python engines resolve with an
|
||
explicit PyTorch index and a PEP 751 lock rather than locks shipped per engine
|
||
version (A8). All of A is live on the PoC.
|
||
|
||
### A1. Toolchain (S, hours)
|
||
`govulncheck` finds 11 standard-library vulnerabilities reachable from this
|
||
code at Go 1.26.3 (net/http, crypto/x509, os.Root, net/textproto). Raise the
|
||
`go` directive in `go.mod` to 1.26.6 or later, rebuild, run `govulncheck
|
||
./...` until it reports none reachable. Add `govulncheck` to the Makefile's
|
||
check target.
|
||
|
||
### A2. Release signing is effectively off (S)
|
||
- `internal/release/trusted.pub` is empty, so `TrustedKey()` returns "" and
|
||
`release.go:103` skips signature verification. Ship a real public key;
|
||
make a missing or invalid signature fatal. Drop the `LLMGW_RELEASE_KEY`
|
||
environment override, or honour it only in `-tags dev` builds.
|
||
- `internal/onboard/onboard.go:365`: the node-side download checks only a
|
||
same-origin `checksums.txt`. Verify the signature there too (simplest:
|
||
have the node run `llmgw` itself to fetch and verify once it has any
|
||
binary), and `shellQuote` the `Releases` URL, which is interpolated into
|
||
single quotes unescaped.
|
||
- `internal/agent/upgrade.go:44`: accept only `https://` release bases.
|
||
- Done when: a tampered binary or checksums file is refused by self-upgrade,
|
||
node onboarding and agent upgrade, with a test for each.
|
||
|
||
### A3. Agent port without TLS is anonymous and plaintext (S)
|
||
`internal/cli/server.go:423` serves h2c on `:7443` when `agent_tls` is not
|
||
configured; `agentsvc` then accepts any node id. Refuse to start control
|
||
mode without TLS unless `agent_tls.insecure: true` is set explicitly; when
|
||
insecure, bind `agent_listen` to 127.0.0.1 and log a warning at startup and
|
||
in `llmgw doctor`. Keep `server init` writing TLS as it does.
|
||
|
||
### A4. Model profiles reach root on every node (M)
|
||
Profile `args` and `env` from a model definition are appended to the engine
|
||
command unchecked (`internal/agent/engines/llamacpp.go:170`, `vllm.go:151`,
|
||
`sglang.go:98`); `internal/agent/engines/unsafeenv.go` is a denylist that
|
||
misses `NCCL_NET_PLUGIN`, `NCCL_TUNER_PLUGIN`, `NCCL_PROFILER_PLUGIN`,
|
||
`CUDA_INJECTION64_PATH`, `NVTX_INJECTION64_PATH`, `TRITON_*` paths; engines
|
||
run as root.
|
||
- Replace the env denylist with an allowlist (CUDA_/NCCL_ tuning keys minus
|
||
anything ending in `_PLUGIN` or containing `INJECTION`, `VLLM_*`,
|
||
`HF_*` minus path keys, `OMP_*`, `GGML_*`). Apply at catalog validation
|
||
(`internal/control/catalog/catalog.go`, where `UnsafeEnv` is called) and
|
||
again in the agent.
|
||
- Deny engine flags that write files or run code: `--trust-remote-code`,
|
||
`--log-file`, `--slot-save-path`, `--chat-template` with a path,
|
||
`--host`, `--api-key`, `--port`; vLLM/SGLang equivalents. Allowlist is
|
||
better than denylist if the flag set is enumerable per engine.
|
||
- Run engines as an unprivileged user with the video/render groups
|
||
(`internal/agent/supervisor`, systemd unit in
|
||
`internal/cli/serverinit_cluster.go:52`). Weights cache must be readable
|
||
by it.
|
||
- Done when: a profile with `--trust-remote-code` or `NCCL_NET_PLUGIN` is
|
||
refused at `ctl model add` and at the agent; `ps` shows engines not as
|
||
root.
|
||
|
||
### A5. Admin API trusts any local process (M)
|
||
`internal/cli/server.go:357` serves the admin API and pprof on
|
||
`127.0.0.1:9090` with no authentication; `adminapi.actorOf`
|
||
(`internal/control/adminapi/api.go:1613`) takes the actor from the
|
||
client-sent `X-Llmgw-Actor` header.
|
||
- Serve the admin API on a unix socket (`/run/llmgw/admin.sock`, 0660,
|
||
group `llmgw-admin`) by default, keeping TCP as an option that requires
|
||
a bearer token read from a root-only file (`/etc/llmgw/secrets/admin-token`).
|
||
- Take the actor from `SO_PEERCRED` (uid → username) on the socket, or from
|
||
the token's name; ignore the header.
|
||
- Refuse `admin_listen` on a non-loopback address unless `admin_insecure:
|
||
true`.
|
||
- Put pprof behind its own `debug.pprof: true` setting, off by default.
|
||
- `ctlClient` (`internal/cli/ctl.go`), the console and `node add` must work
|
||
over the socket; `sudo llmgw` already runs as root.
|
||
|
||
### A6. Patroni REST and etcd are unauthenticated and plaintext (M)
|
||
`internal/pgha/config.go:99` writes a `restapi:` with no `authentication`
|
||
and `:54` etcd over `http://` with no client auth; `hba.go:56` PATCHes
|
||
Patroni's config unauthenticated. Anyone reaching :8008 or :2379 owns the
|
||
store.
|
||
- Generate a Patroni REST username/password at setup, store it in
|
||
`/etc/llmgw/secrets/`, write `restapi.authentication` and
|
||
`restapi.certfile/cafile` from the cluster CA; send it from
|
||
`internal/pgha/hba.go` and `status.go`.
|
||
- etcd: client and peer TLS from the cluster CA with `--client-cert-auth`.
|
||
- Document, and have `server init` write, nftables rules limiting
|
||
2379/2380/8008/5432 to member addresses (see A9).
|
||
- Done when: an unauthenticated `curl :8008/config` is refused and the
|
||
store status in the console still works.
|
||
|
||
### A7. PostgreSQL traffic is plaintext (S)
|
||
`internal/pgha/config.go:168` and `internal/cli/serverinit_postgres.go:330`
|
||
use `sslmode=disable`; HBA lines are `host`. Enable server TLS in Patroni's
|
||
postgresql parameters with a certificate from the cluster CA, write
|
||
`hostssl` lines, use `sslmode=verify-full&sslrootcert=/etc/llmgw/ca.pem`.
|
||
Existing clusters: `llmgw server` must keep working against a store that
|
||
has not been migrated, with a warning.
|
||
|
||
### A8. Smaller fixes, each S or less
|
||
- **Capacity fallback skips the allowlist.** `internal/router/router.go:700`
|
||
builds `[canonical]+entry.Fallbacks` without `permits`. Filter the chain
|
||
with `r.permits(p, model)`; price against the model that served.
|
||
- **Suspended users keep working API keys.** `store.(*DB).Resolve`
|
||
(`internal/store/memberships.go`) never joins `users.state`;
|
||
`Authenticator.Invalidate()` is never called. Reject keys whose owner is
|
||
not active; call `Invalidate` on revoke, suspend, limit change.
|
||
- **Join codes never expire server-side.** `internal/join/join.go:103`
|
||
checks expiry only in `Parse`; `admit` (`:212`) does not. Check it there;
|
||
bind the listener to the chosen address, not `:7444` on all interfaces;
|
||
use the peer's address rather than the claimed `hello.Addr`, and show the
|
||
joiner's address for the operator to confirm before secrets are sent.
|
||
- **Leftover sudoers grant.** `internal/onboard/gossh.go:248` writes
|
||
`NOPASSWD:ALL` and removes it only in `Close()` (`:271`). Write the grant
|
||
with a self-removal (`at`/systemd-run timer, 30 min) so a crash leaves
|
||
nothing; wipe password buffers after use. In `ssh.go:39` pass `--` before
|
||
the target.
|
||
- **Auth cache and pre-auth DoS.** `internal/auth/auth.go:127`: unbounded
|
||
map, one DB query per bogus well-formed key. Bounded LRU, negative cache
|
||
with short TTL, per-IP token bucket before the DB.
|
||
- **HTTP server hardening.** `server.go:351-360`, the h2c agent server,
|
||
both mirror servers: set `ReadHeaderTimeout` (exists), `IdleTimeout`,
|
||
`MaxHeaderBytes`, a body-read deadline, `netutil.LimitListener`; connect
|
||
`WithReadMaxBytes` on the agent stream.
|
||
- **Login/signup abuse.** Default `signup_open: false`; per-IP token buckets
|
||
on `/login` and `/signup`; bounded wait for argon2 slots; CSRF token on
|
||
the login form; the per-address throttle must not lock the real owner
|
||
out (use IP+address pairs).
|
||
- **Trusted proxies.** `internal/web/account.go:557`, `landing.go:60` honour
|
||
`X-Forwarded-Proto/Host` from anyone; add `router.trusted_proxies` and
|
||
only then trust them; `Cache-Control: no-store` on `/setup/*`.
|
||
- **Certificate revocation.** `DELETE /api/v1/nodes/{id}` only drops
|
||
inventory; add a revoked-CN/serial list in the store checked in
|
||
`agentsvc` `Connect` and mirror `onlyClients`; `serverinit_cluster`
|
||
always passes `--api-client`.
|
||
- **Engine exposure.** Engines bind `0.0.0.0` (`llamacpp.go:102`,
|
||
`vllm.go:109`): bind the management address. SGLang receives its key on
|
||
the command line (`sglang.go:102`): pass it by env or file.
|
||
- **Backups.** `internal/cli/backup.go`: encrypt archives to an `age`
|
||
recipient from `backup.recipient` in server.yaml; keep the directory 0700.
|
||
- **Python dependency confusion.** `internal/agent/provision/python.go:121`
|
||
uses `--extra-index-url` with `unsafe-best-match`. Pin torch to the
|
||
PyTorch index explicitly (uv `[[index]] explicit = true`), `first-index`,
|
||
hash-pinned lockfiles per engine version.
|
||
- **Control-plane SSRF.** `internal/sourcecheck/sourcecheck.go:39` and
|
||
`gguf.FromURL` use the default client; reuse the mirror's
|
||
`fetchClient(false)` (private-address filter) and refuse a proxy unless
|
||
configured; block NAT64/6to4-embedded private addresses in
|
||
`internal/mirror/safefetch.go`.
|
||
- **Low:** `agentsvc/service.go:430` indexes `n.GPUs[0]` without a length
|
||
check (CPU-only node panics the handler); thread ids are client-chosen
|
||
with no length cap and shared per project (`router/threads.go`) — cap
|
||
length, document, offer per-user scoping; `/v1/models` lists models the
|
||
key may not use (`handlers.go:799`); sessions have no idle timeout and
|
||
`PurgeSessions`/`PurgeClaims` are never called — add a janitor; add a CSP
|
||
header and set `htmx.config.allowEval=false`; cap GGUF array nesting
|
||
depth in `internal/gguf/gguf.go`; mirror `.source` sidecars are served by
|
||
the public file server — exclude dotfiles from `:8081`; prune
|
||
`modelserve.jobs` and the mirror `running` map; KaTeX → 0.16.21+.
|
||
|
||
### A9. Firewall (S)
|
||
No machine in a default install has a firewall. Have `server init`, `node
|
||
add` and `server add` write an nftables ruleset: ssh open; `:8080` open;
|
||
`:7443` and `:9091` open to the cluster's networks; `:9100` to the
|
||
monitoring host; `:2379/2380/8008/5432` to store members and control
|
||
planes; mirror `:8081` to the cluster, `:8082` to control planes; `:9090`
|
||
Prometheus and `:3000` Grafana to operators. `llmgw doctor` reports a
|
||
missing ruleset. Make it skippable with `--no-firewall`.
|
||
|
||
---
|
||
|
||
## B. Guardrails (M–L, build in this order)
|
||
|
||
**Status (2026-10-04, branch `guardrails-b`):** B1–B4 done, and tried live on
|
||
the PoC. That found four bugs, all fixed with tests:
|
||
- a shadow default merged with an attached policy of unwritten mode left it
|
||
in shadow;
|
||
- a streamed word held back at one release was dropped at the next;
|
||
- llama.cpp continues a conversation ending on the answer, so Llama Guard's
|
||
verdict on answers is now asked with its own prompt as a completion;
|
||
- a judge model was named in a client's "no endpoint" error.
|
||
|
||
The PoC runs `observe` (shadow) for everyone, and Llama Guard 3 1B
|
||
(`llama-guard-3-1b`, CPU, on cuda01) as a judge, which blocks a harmful
|
||
request in ~0.7s and a harmful answer in ~1.4s there. Still left: see each
|
||
item.
|
||
|
||
Today nothing on the request path reads, classifies, redacts or filters
|
||
message text in either direction (confirmed; the only "moderation" hit is a
|
||
comment in `internal/router/budget_fallback.go:20`). Existing controls are
|
||
about who, how much and how fast: allowlists (`router/handlers.go:450`),
|
||
RPM/TPM/budget (`internal/accounting`), context precheck
|
||
(`router/context_limit.go`), admission bands (`router/admission`),
|
||
timeouts (`router/proxy/proxy.go:26-57`). Gaps to close on the way: no
|
||
default output cap when the client omits `max_tokens`; `catalog.Capabilities`
|
||
is advertised but never enforced; tool results are trusted input.
|
||
|
||
### B1. Hooks in the proxy and router (S)
|
||
**Done (branch `guardrails-b`):** `internal/guard` holds the interface;
|
||
proxy `Filter`/`StreamFilter`; the input hook runs after `loadThread` (the
|
||
simpler choice); error codes, headers, Responses `incomplete`, usage columns
|
||
`guard_*`. Moved to B2: holding `delta.tool_calls` until they are whole and
|
||
reading their arguments — that is the tool policy's filter, not the proxy's.
|
||
|
||
- `internal/router/proxy/proxy.go`: add optional `InputRewrite func(body
|
||
[]byte) ([]byte, Verdict)` on the request and two output hooks: `Filter`
|
||
for `relayBuffered` (between `ReadAll` and `WriteHeader`) and
|
||
`StreamFilter{OnEvent(data) (emit []string, stop bool); OnEnd()}` for
|
||
`relayStream`, called where each whole SSE event is assembled
|
||
(`eventBuffer`, before `io.WriteString` at `:343`). Nil hooks cost
|
||
nothing.
|
||
- `internal/router/handlers.go`: call the input policy after the allowlist
|
||
and limits and before admission (around `:395`); the principal, scopes
|
||
and the model that will serve are known there. Thread history is spliced
|
||
later (`:416`): scan stored turns when written, or move the hook after
|
||
`loadThread` (decide; the second is simpler).
|
||
- Blocking conventions: input → 400 `invalid_request_error` with
|
||
`code: "content_policy_violation"` (today `writeError` always sends
|
||
`code: null`); output non-stream → 200 with `finish_reason:
|
||
"content_filter"` and empty/redacted content; output stream → a final
|
||
chunk `{"choices":[{"index":0,"delta":{},"finish_reason":"content_filter"}]}`,
|
||
then `data: [DONE]`, then cancel upstream. Headers `x-llmgw-guard:
|
||
blocked|redacted|flagged` and `x-llmgw-guard-policy` set before the proxy
|
||
writes headers.
|
||
- `internal/router/responses.go`: map `content_filter` to
|
||
`response.incomplete` with `incomplete_details.reason: "content_filter"`
|
||
(today it always emits `response.completed`, `:525`).
|
||
- Tool calls: hold `delta.tool_calls` until `finish_reason: "tool_calls"`,
|
||
then validate; `replyDelta` (`reply.go:33`) must also read
|
||
`tool_calls[].function.arguments`.
|
||
- Accounting: a blocked input is recorded like `RecordRefused` (nothing
|
||
charged); a blocked output is charged (the GPU was used). `store.Usage`
|
||
gains `GuardAction`, `GuardPolicy`, `GuardCategory`, `GuardMs`.
|
||
|
||
### B2. Package `internal/guard` (M)
|
||
**Mostly done (branch `guardrails-b`):** policies in server.yaml with
|
||
attachments in the store (`ctl guard attach`), narrowed by the limits' rule;
|
||
`guard/detect` (validated PII, secrets, injection through normalisation and
|
||
decoding); redaction with consistent placeholders before prefix routing;
|
||
tool allow/deny/strip, call count and argument schemas; capabilities and
|
||
structured output (opt-in); default `max_tokens`; pinned system prompt and
|
||
dropping the client's; `no_store` (threads, notes, prefix tree, external
|
||
models); shadow mode. **Left:** a `guard_events` table — the usage row's
|
||
`guard_*` columns carry action/policy/category/time, not rule and stage;
|
||
language identification; the re-identification map that restores
|
||
redacted values in answers.
|
||
|
||
- `Policy`, `Effective` resolution over the tenancy tree with the same
|
||
narrowing rule as limits (`internal/store/sqlite.go:383-416` `tighten`):
|
||
deny sets union, allow sets intersect, `mode` takes the strictest of
|
||
`enforce > shadow > off`, `fail` takes `closed > open`, `no_store` at any
|
||
level wins, thresholds take the minimum.
|
||
- Store: `guard_policies(scope, scope_id, policy json)` and
|
||
`guard_events(request_id, scopes, policy, rule, category, action, stage,
|
||
score, judge_ms, shadow)` — ids, categories, offsets and lengths only,
|
||
never text. Migrations for sqlite and postgres. Pruned with
|
||
`usage_retention`.
|
||
- `guard/detect`: regex/keyword sets; PII with validators — email, phone,
|
||
IBAN (mod-97), card (Luhn), TCKN (checksum), IP, secrets (`sk-`, AWS,
|
||
JWT, PEM); normaliser (NFKC, strip zero-width/bidi, homoglyph fold,
|
||
base64/hex/URL decode to depth 2); injection heuristics (role-switch
|
||
phrases, fake chat-template tokens, long base64, invisible Unicode) —
|
||
default action `flag`; language id.
|
||
- Redaction rewrites `messages[].content` in the upstream body with
|
||
placeholders (`[EMAIL_1]`) via the same `map[string]json.RawMessage`
|
||
round-trip `withModel`/`spliceThread` use; an optional re-identification
|
||
map lives only in request memory and restores originals in the output.
|
||
Redact before prefix routing so `PromptPrefix` hashes the redacted text.
|
||
- `guard/toolpolicy`: tool name allow/deny (strip or 400), JSON-Schema
|
||
validation of assembled arguments against the declared `parameters`,
|
||
argument regex rules, `max_calls_per_turn`. Enforce
|
||
`Capabilities.Tools`/`Vision` here: a `tools` array or image parts to a
|
||
model without the capability → 400.
|
||
- `guard/structured`: validate `response_format.json_schema` on the final
|
||
text → `content_filter`, or flag in shadow.
|
||
- Output constraints: inject a default `max_tokens` when absent (from the
|
||
policy or `MaxOutput`), language check, structured check.
|
||
- System-prompt pinning per project: prepend before `checkContextFits` and
|
||
before `spliceThread`; optionally drop client `system`/`developer`
|
||
roles; exclude the pin from stored thread turns; apply to Responses
|
||
`instructions` in `toChat`.
|
||
- `no_store`: `loadThread` and `PutNote` refuse; `captureBytes()` returns 0;
|
||
skip the prefix-tree observe (`router.go:840`); disable External
|
||
fallbacks for the scope (filter in `dispatch` and `budgetFallback`).
|
||
|
||
### B3. Judge model on the cluster (M)
|
||
**Done (branch `guardrails-b`):** `role: guard` + `guard.format`; hidden and
|
||
refused to clients and fallbacks; critical by default, never on demand;
|
||
`Router.Judge` past admission/accounting/guardrails; llamaguard3,
|
||
shieldgemma and json parsers; 5-minute cache; latency budget then
|
||
fail open/closed; streamed answers read every 600 characters, or held with
|
||
buffer_all. Simpler than written: the input judge runs after admission, not
|
||
alongside it, and shadow mode judges synchronously within the budget rather
|
||
than after the response. Not enforced: two replicas on different nodes —
|
||
documented, left to `scaling.min`.
|
||
|
||
- `catalog.Model.Role: "guard"` plus `Guard{Format: llamaguard3 |
|
||
shieldgemma | json, Categories}`. Guard models are hidden from
|
||
`/v1/models`, the chat picker and allowlists; placed like any model,
|
||
class `critical`, `scaling.min ≥ 1`, at least 2 replicas on different
|
||
nodes.
|
||
- `guard/judge`: calls the guard model through the router's own `dispatch`
|
||
with an internal principal and its own admission band (never the
|
||
tenant's queue); parsers per format (Llama Guard `safe`/`unsafe\nS1,S10`;
|
||
ShieldGemma yes/no probability → per-tenant thresholds; generic JSON
|
||
verdict); verdict cache keyed by hash of normalised new turns (5 min);
|
||
`latency_budget` (default 250 ms) then `fail` open/closed; input judge
|
||
runs in parallel with admission queueing.
|
||
- Streaming strategy: default sentence-buffered scanning (regex/PII per
|
||
sentence or ~200 chars before release; judge every 400–800 chars on the
|
||
cumulative tail; final judge at end); `buffer_all` for `fail: closed`
|
||
scopes. Document that released text cannot be retracted.
|
||
- Shadow mode runs everything asynchronously after the response
|
||
(`context.WithoutCancel`, like `saveThread`) and only records.
|
||
|
||
### B4. Config, metrics, UI (S)
|
||
**Done (branch `guardrails-b`):** library in server.yaml, attachments via
|
||
`ctl guard`, the console's **Guardrails…** with "applies from" marks, and
|
||
the portal's `/admin/guard`; the metrics and alerts below (judge latency
|
||
labelled by model only at the judge; the check latency by stage); `ctl
|
||
request` shows the verdict; the chat page says "stopped by policy"; the
|
||
profile lists effective policies; docs/guardrails.md and data.md written;
|
||
the access log carries action, stage and policy only. Attaching from the
|
||
portal is not done — the page shows; the console and ctl change.
|
||
|
||
- Gateway-level policy library in `server.yaml` under `guard:` (see the
|
||
YAML in the review, reproduced in `docs/guardrails.md` to be written);
|
||
attachments to scopes in the store via `llmgw keys guard`, the console
|
||
("Guardrails…" beside "Limits…" in `internal/cli/tui_people.go`, with
|
||
"inherited from team:X" badges) and the portal.
|
||
- Metrics in `router/metrics.go` style:
|
||
`llmgw_guard_checks_total{stage,policy,action,category,mode}`,
|
||
`llmgw_guard_latency_seconds{stage,judge_model}`,
|
||
`llmgw_guard_judge_errors_total{reason}`, `llmgw_guard_fail_open_total`,
|
||
`llmgw_guard_stream_stops_total`, `llmgw_guard_redactions_total{kind}`,
|
||
`llmgw_tenant_guard_blocks_total{tenant}`. Alerts: fail-open > 1% for 5m,
|
||
judge p95 over budget, judge has no ready replicas, per-tenant block rate
|
||
> 3× its 7-day baseline, any `secret` detected in output.
|
||
- Portal `/admin/guard`: violations by tenant/category/policy over time,
|
||
shadow vs enforce, judge health, recent events by request id; `ctl
|
||
request <id>` shows the verdict; chat page renders `content_filter` as
|
||
"stopped by policy X"; `/me` lists effective policies.
|
||
- Update `docs/data.md`: the judge sees all text but in-cluster; External
|
||
fallbacks send prompts outside the cluster (the current "nothing leaves
|
||
the cluster" is wrong); `no_store` semantics.
|
||
- Never log: prompt/completion text, PII values, the re-identification
|
||
map, judge inputs or rationale that quotes input. Check `accessLog`'s
|
||
`err` field and span errors.
|
||
|
||
---
|
||
|
||
## C. Features, ranked by value × feasibility
|
||
|
||
1. **Done (branch `features-c`):** `audit_events`, written by the admin
|
||
API middleware and the /admin pages (with each change's result), `ctl
|
||
audit`, the console's Audit view; staff roles viewer / operator /
|
||
billing / admin. Left: before/after values (the request's fields are
|
||
recorded, not the state they replaced); a tenant-admin role that changes
|
||
its own organization (managers still only view it); hiding the forms a
|
||
viewer cannot submit. **Audit log and roles (M).** Today "audit" is slog lines
|
||
(`internal/web/admin_users.go:403-480`) and roles are the `IsAdmin` flag
|
||
plus an org "manager" who can view. Add `audit_events` (who, what,
|
||
before/after, source) written by the `adminAction`/`tenancyAction`
|
||
wrappers and by every admin-API mutation; roles viewer / operator /
|
||
tenant-admin / billing; `ctl audit`. Depends on A5 for a real actor.
|
||
2. **Done (branch `features-c`):** OIDC sign-in (`auth.oidc`, PKCE,
|
||
verified email, roles and memberships from groups, `no_passwords`) and
|
||
SCIM 2.0 (`/scim/v2` Users and Groups). The sign-in cookie's key comes
|
||
from the store (2026-10-10), so a callback may reach either control
|
||
plane behind a balancer. **OIDC/SSO login (M), then SCIM (M).** `internal/account/service.go`,
|
||
`internal/web/account.go`; map IdP groups to org/team/admin.
|
||
3. **Done (branch `features-c`):** key lifetime limits (gateway, org, team,
|
||
project), `keys rotate --overlap`, kinds and expiry in `keys list`.
|
||
**Service accounts and key lifecycle (S).** Per-org maximum key TTL,
|
||
`keys rotate` with overlap, last-used shown, project-owned service keys
|
||
listed apart. `internal/cli/keys.go`, `internal/store/keys.go`,
|
||
`internal/auth/auth.go`.
|
||
4. **Done (branch `features-c`):** CSV export from the rollups (API, ctl,
|
||
console). Left: Parquet, and a scheduled push to S3/SFTP. **Usage export (S).** CSV/Parquet by org/team/project/model/period from
|
||
the rollups (`internal/store/usageseries.go`), `GET
|
||
/api/v1/usage/export`, optional scheduled push to S3/SFTP.
|
||
5. **Done (branch `features-c`):** `notify.webhooks` (model parked, node
|
||
lost, budget spent, update available; signed JSON or Slack; retries,
|
||
hourly de-duplication), `ctl notify test`, Alertmanager receiver
|
||
templates. Left: `server init` and the console do not yet ask for a
|
||
receiver — the templates are filled in by hand. Events are claimed in
|
||
the store before they are sent (2026-10-10), so two control planes send
|
||
each once.
|
||
**Alerting out of the box (S).** `deploy/monitoring/alertmanager.yml`
|
||
ships `receiver: none`; add Slack/email/webhook/PagerDuty receiver
|
||
templates driven by `server init` or the console; llmgw-native webhooks
|
||
for model parked, budget spent, update available (from the reconciler
|
||
and `modelserve`).
|
||
6. **Done (branch `features-c2`):** `metadata`, `x-llmgw-tags` and `user`
|
||
on the usage record (migration 0023), `ctl request` shows them,
|
||
`router.tag_labels` metrics with 100 values per tag. Left: export and
|
||
`ctl usage` by tag or end user (the hourly and daily sums do not carry
|
||
tags; it would read the request records, within their retention).
|
||
**Request metadata and tags (S).** Accept `metadata`/`user` from the
|
||
body or `x-llmgw-tags`; store on the usage row; tenant metric labels
|
||
behind an allowlist. `router/handlers.go` parse, `accounting/recorder.go`.
|
||
7. **Done (branch `features-c2`):** `ctl model canary/promote/canaries`,
|
||
session-sticky split with `x-llmgw-variant` pins, rollback on error rate
|
||
or TTFT, `canary_rolled_back` notification, console menu. Left: A/B
|
||
between two different models (the split is between a model and its
|
||
newer weights); the comparison is per control plane and lives in memory,
|
||
so a restart begins it again; a key-level pin.
|
||
**Canary and weighted A/B between model revisions (M).**
|
||
`control/modelserve/updates.go` `replace` swaps every replica; add
|
||
weighted routing across two revisions/aliases (scheduler filter, alias
|
||
resolution in `router/router.go:569`), a header/key pin, automatic
|
||
rollback on error-rate or TTFT regression from per-replica metrics.
|
||
8. **Done (branch `features-c2`):** `/v1/files` and `/v1/batches`
|
||
(OpenAI's shape), worker under a store lease, lines kept as served,
|
||
lowest band, usage tagged `batch=<id>`, output and error files with
|
||
retention, `no_store` refused. Left: the Responses API (`/v1/responses`)
|
||
as a batch endpoint; the console and `ctl` do not list batches (an
|
||
operator sees them through usage tags); a file is one row in the store,
|
||
so a store backup carries batch files with it.
|
||
**Batch API and Files (M–L).** `/v1/files`, `/v1/batches`, JSONL
|
||
storage, a worker feeding the bronze band, accounting. New
|
||
`internal/router/batch`.
|
||
9. **Done (branch `features-c2`):** `keys requests` (retention, sample),
|
||
kept after redaction, `ctl request --body/--replay`, console, pruning.
|
||
Left: a streamed answer is kept as its text, not its chunks; replay is
|
||
one request at a time (a set of them against a model is item 21's eval).
|
||
**Request/response logging and replay (M).** Opt-in per project with
|
||
retention like threads; sampled or full; redaction hooks from B;
|
||
`ctl request <id> --body`; replay against another model.
|
||
10. **Done (branch `features-c2`):** Responses `text.format`,
|
||
`input_image`, `previous_response_id` (thread in the id); rerank and
|
||
score passthrough; transcription and translation as multipart;
|
||
`/v1/moderations` by the judge. Left: audio is charged no tokens (an
|
||
engine's duration usage is not read); `input_image` by `file_id`;
|
||
moderation scores are 0 or 1, as a judge says yes or no.
|
||
**OpenAI compatibility gaps (S–M each).** Responses: `text.format` →
|
||
`response_format`, `input_image`, `previous_response_id` via threads
|
||
(`router/responses.go`). Rerank/score passthrough for vLLM,
|
||
`/v1/audio/transcriptions` for a whisper engine, `/v1/moderations`
|
||
mapped to the guard model.
|
||
11. **Done (branch `features-c2`):** `/v1/messages` and `count_tokens`,
|
||
streamed and not, tools, images, `x-api-key`, Anthropic error shape.
|
||
Left: extended thinking (a reasoning model's thoughts are not mapped to
|
||
thinking blocks); prompt caching; the account page's "Connect your
|
||
editor" does not offer Claude Code yet.
|
||
**Anthropic `/v1/messages` (M).** Translate like `responses.go`.
|
||
12. **Done (branch `features-c2`):** `server restore` (dry run, version
|
||
check, decryption, modes and owners, SQLite and PostgreSQL),
|
||
`backup_copy_to` (directory or ssh), the drill in docs/install.md.
|
||
Left: restoring a replicated store's members (restore into the primary;
|
||
Patroni replicates it); the console's Back up now does not offer
|
||
restore; an ssh copy holds the archive in memory while it sends it.
|
||
**`server restore` (S).** Restore is manual shell today
|
||
(`docs/install.md:1187`); add the command with version check, dry run,
|
||
off-host copy target; a documented DR drill.
|
||
13. **Done (branch `features-c2`):** TPOT, inter-token and tokens/s
|
||
histograms by model, pool and tenant; `slo` pressure in admission's
|
||
saturation; `llmgw_model_slo_seconds` and `llmgw-slo` alerts. Left: the
|
||
p99 is per control plane, from its own requests; an SLO per band or per
|
||
tenant; placement does not yet add replicas for an SLO (item 14).
|
||
**SLO-based admission and token-level metrics (S–M).** TPOT and
|
||
inter-token histograms, output tokens/s per model and tenant in
|
||
`router/metrics.go`; use `catalog.SLO` in `admission/saturation.go`.
|
||
14. **Done (branch `features-c2`):** `scaling.schedule` (days, from, to,
|
||
min, max; past midnight), `scaling.prewarm`, `control.timezone`;
|
||
queue-depth scaling was there already. Left: `ctl scale` sets min and
|
||
max only (a schedule is set in the definition); scaling for an SLO
|
||
rather than for queue depth.
|
||
**Scheduled and warm-pool autoscaling (S–M).** Time-of-day min/max,
|
||
pre-warm, queue-depth scaling in `catalog.Scaling`,
|
||
`control/reconciler/demand.go`, `headroom.go`.
|
||
15. **Done (branch `features-c2`):** `env:` and `vault:` references
|
||
wherever a secret file is named (store address, api_key_file, OIDC,
|
||
SCIM, webhooks), Vault KV with token or AppRole, `secrets.master_key`
|
||
sealing cluster_secrets (Vault transit or a local key). Left: the
|
||
certificate authority's key is still a file in /etc/llmgw (signing in
|
||
Vault's PKI engine instead is its own change); rotating the master key;
|
||
`backup_copy_key` and `admin_token_file` are files only.
|
||
**Secrets management (S–M).** Vault/KMS for `api_key_file`, the DSN, the
|
||
CA key, `cluster_secrets` (`store/secrets.go`, `store/seal.go`);
|
||
envelope-encrypt them in backups (with A8 backups).
|
||
16. **Done (branch `features-c2`):** `users forget` (pseudonymised usage
|
||
and sums, kept requests, files, batches, limits and audit names), `keys
|
||
residency --in-cluster` (asked-for, fallback and canary). Left: an
|
||
export of everything held about a person (article 15); residency by
|
||
region between several clusters; backups keep what was forgotten until
|
||
they age out.
|
||
**GDPR and residency (S).** "Forget user" anonymises `user_id` in usage
|
||
and rollups and erases threads/memory (`store/hierarchy.go:241` leaves
|
||
it today); per-project "never route to an External provider".
|
||
17. **Done (branch `features-c2`):** exact-match answer cache per project (`keys
|
||
cache --ttl/--any/--off`, console **Cache answers…**, `projects`
|
||
columns, 30-second setting cache), reproducible answers only unless
|
||
`--any`, LRU in memory bounded by `router.cache.max_bytes` and
|
||
`max_entry_bytes`, streamed answers kept and replayed as chunks,
|
||
`x-llmgw-cache` and `Cache-Control: no-cache`, guardrails before a hit
|
||
and the output policy on it, `no_store` and threads bypass, a hit
|
||
recorded as `cache_hit` with no tokens and no cost (migration 0027),
|
||
metrics, docs. Left: semantic (embedding-similarity) caching through the
|
||
embeddings models; a cache shared across control planes (each process
|
||
keeps its own, deliberately: a shared one would write every answer to
|
||
the store); a hit still waits its turn in admission, because the
|
||
guardrails judge the request after admission and the key is the request
|
||
they let through; logprobs in a streamed answer leave it unkept; `users
|
||
forget` does not reach into the cache (answers age out by the TTL); no
|
||
listing or flushing of what is cached (`--off` stops it being served at
|
||
once, the memory is freed by TTL and LRU).
|
||
**Response/semantic cache (M).** Exact-match first, per project,
|
||
opt-in, TTL; semantic via the embeddings models later. Before
|
||
`dispatch` in `router/handlers.go`.
|
||
18. **Done (branch `features-c2`):** offline bundles — `llmgw bundle create`
|
||
(releases with signed checksums, every llama.cpp build per arch/variant
|
||
with GitHub's checksums, starter/judge and named Hugging Face model
|
||
files pinned and checked; directory or tar; manifest sha256), `bundle
|
||
import` (checked, all-or-nothing, into the mirror under `_llmgw/`),
|
||
`bundle trust`, `bundle show --verify`;
|
||
`offline.mirror`/`offline.checksums` in server.yaml, with nodes
|
||
installing llama.cpp from the mirror against checksums the control plane
|
||
sends over the agent stream (proto `EngineWant.offline_*`), releases
|
||
defaulting to the mirror for `node add`, `ctl upgrade` and agent
|
||
upgrades, update checks off; `server init --bundle` and the setup form's
|
||
bundle fields; docs/data.md's offline claim corrected; install.md
|
||
"Offline sites". A Helm chart for the control plane
|
||
(`deploy/helm/llmgw`, one replica, SQLite PVC or external PostgreSQL,
|
||
Secrets, Services, Ingress, ServiceMonitor) with
|
||
`deploy/container/Dockerfile` and docs/kubernetes.md; GPU nodes stay
|
||
outside Kubernetes by design. Left: vLLM/SGLang offline (a wheelhouse of
|
||
several GB per CUDA build and a uv `--find-links` install path); the
|
||
NVML release binary and uv in bundles; the setup wizard's preflight
|
||
still fails a machine without internet; `join`'s printed `curl` from the
|
||
mirror checks nothing; models in a bundle are single files (no whole-
|
||
repository safetensors sets yet); the chart runs one replica (a standby
|
||
is never ready, so HA in Kubernetes needs a different readiness design),
|
||
has not been installed on a real cluster, and ships no agent DaemonSet;
|
||
no published container image.
|
||
**Kubernetes/Helm and an air-gapped bundle (M).** `deploy/`,
|
||
`agent/provision`; `docs/data.md` claims offline operation but install
|
||
assumes the internet.
|
||
19. **Done (branch `features-c2`):** `adminapi.Routes` and
|
||
`internal/docgen/openapi` generating `docs/admin-api.openapi.json` (3.1,
|
||
bearer security, every route, query parameters, schemas from the Go
|
||
types with their comments), served embedded at `GET
|
||
/api/v1/openapi.json`, kept true by a route-table test and a staleness
|
||
test; validated with openapi-spec-validator and openapi-generator-cli.
|
||
`GET /api/v1/models/{name}`. `docs/admin-api.md`. `terraform-provider-
|
||
llmgw/` (terraform-plugin-framework): `llmgw_model`, `llmgw_quota`,
|
||
`llmgw_guard_attachment`, data sources `llmgw_models` and `llmgw_nodes`;
|
||
client tests against a fake admin API and the OpenAPI document;
|
||
acceptance tests with Terraform and OpenTofu against the fake; one
|
||
apply, empty plan, in-place change and destroy against a real `llmgw
|
||
server` run by hand. Organizations, teams, projects, keys and
|
||
memberships are in the admin API since 2026-10-08 (`/api/v1/orgs`,
|
||
`teams`, `projects`, `keys`, `users/{id}/memberships`) and in the
|
||
provider (`llmgw_org`, `llmgw_team`, `llmgw_project`, `llmgw_key`,
|
||
`llmgw_membership`); accounts are still made by sign-up, invitation or
|
||
OIDC/SCIM. Left: a model's definition beyond
|
||
scaling, class, owner and adapters is changed by replacing it (the API
|
||
has no edit, and `resources` is not used yet); `canary`, `slo`,
|
||
`scaling.schedule` and `prewarm`, `guard` role and `origin` are not in
|
||
`llmgw_model`; models cannot be imported; the provider is not published
|
||
to a registry and has no release build; the acceptance tests are not run
|
||
in CI (they need Terraform); the document's schemas mark nothing
|
||
required and give no enums; `catalog.SLO`'s fields have no JSON tags, so
|
||
they are `TTFTp99` and `TPOTp99` on the wire.
|
||
**OpenAPI for the admin API, Terraform provider (M).** Generate from
|
||
`adminapi/api.go` routes via `docgen`.
|
||
20. **Done (branch `features-c2`):** spans for the reconcile pass (begun only
|
||
when a pass does something), placement with its answer or rejections,
|
||
scale-up/down, drain, stop, eviction, preemption, reaping, restarts and
|
||
lost nodes; a `replica start` span from placement to serving whose
|
||
context travels to the agent in the start command
|
||
(`ControlMessage.trace`), where weights fetch, engine launch, rendezvous
|
||
and readiness wait continue the trace; stops, drains, adoption and
|
||
engine installs; modelserve jobs as a trace with a span per step linked
|
||
to the replica's start; an agent `tracing` block; docs in install.md's
|
||
"Tracing". Left: the router's span for a request does not link to the
|
||
replica it was sent to's start; heartbeats, fabric scans, logs and agent
|
||
upgrades are not traced; the supervisor has no restart span because it
|
||
never restarts an engine itself (a restart is the reconciler's `restart
|
||
replica` and reaches a node as a stop and a start); canary and promote
|
||
jobs, which landed on `features-c2` after this was branched, are traced
|
||
only through the shared step recording (as `<kind> model`), unchecked by
|
||
a test.
|
||
**Tracing beyond the router (S).** Spans for reconciler, placement,
|
||
agent start (`obs/trace.go`, `control/reconciler`, `agent/supervisor`).
|
||
21. **Done (branch `features-c2`):** `internal/bench/eval` (suites in YAML or
|
||
JSON lines; exact, contains, regex, number, a JSON Schema subset, and a
|
||
judge model with a rubric; weights and a threshold), run on the control
|
||
plane through `Router.Replay`; `llmgw ctl eval MODEL --suite FILE|NAME`,
|
||
`--history`, `--run`, `--suites`; runs and their cases kept (migration
|
||
0028, `eval_runs` and `eval_cases`); `model update`/`promote --eval
|
||
SUITE --min-score N`, a model's standing gate (`ctl model gate`, `eval:`
|
||
in the catalog) — a promote asks the canary first, an update tries the
|
||
new weights as a 0% canary and promotes or removes it; the console's
|
||
model menu; `arithmetic` and `json-format` in `deploy/eval/`, carried in
|
||
the binary. Left: the suite of a model's standing gate is read by name
|
||
on the control plane at the time of the update, so two control planes
|
||
need the same `control.eval_dir`; a gated update needs room for one more
|
||
replica while it runs (there is no in-place gate, and a declared model
|
||
in server.yaml cannot be updated at all); no comparison against the old
|
||
weights' score (the gate is an absolute threshold, not "no worse than
|
||
before"); a run is not stopped once started (Esc in the console only
|
||
stops watching it), and the in-memory progress of a run lives on the
|
||
control plane that started it; runs are never pruned; the JSON Schema
|
||
support is a subset (no `$ref`, `oneOf`, `not`, `uniqueItems`); judge
|
||
cases go through the same concurrency as the model's, and a judge's
|
||
answers are not kept, only its score and reason; a gateway without a
|
||
control plane cannot run suites (there is no mode that asks through the
|
||
client API with a key).
|
||
**Eval harness (M).** `llmgw eval` extending `internal/bench` with
|
||
quality suites gating `model update`.
|
||
22. **Done (branch `features-c2`):** MIG instances discovered on both paths
|
||
(nvidia-smi `-L` + `-q -x`; NVML through an interface with a faithful
|
||
fake) and reported instead of their card, each with its MIG UUID, own
|
||
memory and class `<card class>-mig-<profile>`; device numbers 1000 + 100
|
||
× card + instance, shown as `card/migN`; placement on MIG classes;
|
||
tp/pp/gpus/nodes above 1 on a MIG class refused in catalog validation
|
||
and again in placement; `CUDA_VISIBLE_DEVICES=MIG-<uuid>` for vLLM and
|
||
llama.cpp (no Vulkan device variable on MIG); per-instance memory and
|
||
processes in heartbeats and the GPU check, card sensors under each
|
||
instance, utilisation absent rather than zero; doctor MIG findings (`mig
|
||
-lgi`/`-lci` parsed); `ctl nodes`, console, `ctl status`, `ctl model
|
||
plan`/`serve` by class; docs/install.md MIG section. Written and tested
|
||
against fakes only — untested on real MIG hardware. Left: run it on an
|
||
A100/H100 in MIG mode (check `-L`/`-q -x`/`mig -lgi`/`-lci` and NVML MIG
|
||
device names against the parsers, and that `CUDA_VISIBLE_DEVICES=MIG-…`
|
||
reaches vLLM's workers through Ray); instances changed while the agent
|
||
runs are only seen when it restarts (no rescan); llmgw does not create
|
||
or reshape instances (no mig-parted integration); quotas count an
|
||
instance as one GPU, the same as a whole card; SGLang on MIG not
|
||
checked; device numbers assume fewer than 100 instances per card and
|
||
that instance numbers stay put while shards run on them.
|
||
**MIG (M–L).** Treat a MIG slice as a GPU class in
|
||
`agent/discovery/gpu_nvml.go` and `control/placement`.
|
||
23. **Done (branch `features-c2`):** engine `trtllm`
|
||
(`internal/agent/engines/trtllm.go`): `trtllm-serve` through a launcher
|
||
of llmgw's that demands the replica's key (trtllm-serve has none) and
|
||
writes the options file that turns on `/prometheus/metrics` and the
|
||
iteration statistics; the scrape contract for it
|
||
(`scrape.TRTLLMMapping`, with a per-engine metrics path); found, not
|
||
installed — `engines.trtllm.python` in `agent.yaml`, probed from the
|
||
package's metadata; catalog validation (GPU class, one node, no GGUF
|
||
source, the flags a profile may not give); `ctl engine-check trtllm`,
|
||
which asks `trtllm-serve serve --help` through its module. Ollama/GGUF
|
||
import: `mirror import-ollama` (manifest parsing, blob digest check,
|
||
projector beside the model, all or nothing, a model definition printed),
|
||
`mirror import` for any local file, and the console's **I** on the
|
||
Mirror view. Left: TensorRT-LLM has not run on real hardware — the first
|
||
node with one should run `engine-check` and serve a model before anyone
|
||
relies on it; no installer for it; no multi-node TensorRT-LLM replica
|
||
(it needs an MPI or Slurm launch from one node); Triton Inference Server
|
||
as an engine is not done; trtllm's queue gauges are set per iteration,
|
||
so an idle engine shows its last batch until the next request;
|
||
`--num_serve_frontends` is refused because extra front ends would not
|
||
demand the key. The Ollama import runs on the mirror machine only — the
|
||
mirror's API takes no files, so an Ollama elsewhere means copying the
|
||
blob or running the import there; Ollama models kept as separate tensors
|
||
or as several weight layers, and its LoRA adapters, are refused or left
|
||
with a sentence; files an import run with `sudo` writes are root's, as
|
||
`mirror add`'s are.
|
||
**More engines (M each).** TensorRT-LLM/Triton adapter; Ollama/GGUF
|
||
import. `agent/engines/engine.go` interface and the metric contract in
|
||
`router/scrape`.
|
||
24. **Done (branch `features-c2`), all six parts.** Lower: precise KV-index routing (plan phase 6), LoRA affinity for
|
||
vLLM, multi-cluster federation via External providers, facility-aware
|
||
placement (`docs/facility-aware-placement.md`), portal i18n, SDK/docs
|
||
portal.
|
||
What was built, and what is left:
|
||
- 24 (portal i18n): built — English/Turkish catalogue with fallback, per-
|
||
language templates, cookie/Accept-Language/`landing.language` choice
|
||
and a footer switcher, Turkish numbers and dates, every template and
|
||
handler message translated, tests that no template text stays English;
|
||
left — errors built with fmt.Errorf in other packages (store, catalog,
|
||
mirror, adminapi) still English on Turkish pages, percent sign
|
||
placement is mixed (%50 on limits, 93% on cluster pages), RFC3339
|
||
footer timestamps and plain-text http.Error bodies untranslated, the
|
||
/docs documents themselves English, a per-person language column
|
||
(cookie only today).
|
||
- 24 (SDK/docs portal): built — /docs renders the embedded manual (goldmark,
|
||
GFM, heading anchors, links rewritten, no script; public,
|
||
`landing.docs_sign_in` to require a session), and docs/clients.md
|
||
shows the official OpenAI/Anthropic SDKs against llmgw. Left: the
|
||
console's generated "Commands" page is not in the portal (it is built
|
||
from the cobra tree, which internal/web cannot see); no search across
|
||
documents.
|
||
- [~] 24. Precise KV index (tier 2) — built: vLLM KV events via the agent
|
||
(loopback ZMQ, pure-Go zmq4, msgpack in both of vLLM's encodings)
|
||
forwarded as net block-hash changes over the agent stream; router
|
||
index, tokenize via the replica's `/tokenize`, sha256_cbor block-hash
|
||
chain checked against cbor2/hashlib golden values; scorer by leading-
|
||
block share with tier 1 fallback; opt-in `kv_events` per profile;
|
||
metrics; docs. Left: validation against a real vLLM (event shape, hash
|
||
agreement, `/tokenize` matching the served prompt, cost of
|
||
tokenizing); vLLM's replay endpoint instead of resetting on a gap;
|
||
data-parallel ranks (one publisher each, ports offset); hybrid models'
|
||
cache groups and CPU-offloaded blocks; LoRA/multimodal/cache_salt
|
||
requests (extra keys); a router-only deployment (no agents)
|
||
subscribing directly; standby control planes have no index until they
|
||
lead; a per-replica fallback when only some candidates have an index;
|
||
caching tokenizations of repeated prompts.
|
||
- **24 LoRA affinity (branch `features-c2`):** built `max_loaded_adapters`
|
||
(catalog), vLLM adapter support at start and on demand
|
||
(`engines/lora.go`, `vllm.go`), the agent linking fetched adapters as
|
||
`llmgw-lora/<name>` in the engines' working directory, the router's
|
||
adapter tracker (`router/lora.go`: loads/unloads with the engine key,
|
||
LRU eviction never of an adapter in use, one load per adapter per
|
||
replica, `/v1/models` sync, 404 reload) and the `adapter-affinity`
|
||
scheduler filter in the chat, batch and long-context profiles, with
|
||
metrics and docs; tested against a fake of vLLM's HTTP interface.
|
||
**Left:** never run against a real vLLM — in particular that vLLM
|
||
resolves the relative `llmgw-lora/<name>` path against its working
|
||
directory on every node (multi-node too) and that unloading an adapter
|
||
mid-request is harmless; the agent still fetches every adapter at
|
||
replica start, so a new adapter needs a restart (fetching on first
|
||
load would need an agent command and a proto change); load timeout and
|
||
resync interval are constants, not config; SGLang's runtime LoRA API
|
||
is not driven; no per-replica "adapters held" view in the console or
|
||
admin API; the debug profile has no affinity.
|
||
- 24. Multi-cluster federation — built: peers via `ctl federation` kept in
|
||
the store, their models as PEER/MODEL synced from /v1/models
|
||
(withdrawn-while-referenced), /ready health with latency and routing
|
||
withdrawal, `ctl model fallbacks` for overflow, hop header loop
|
||
refusal, residency/no_store hold, optional attribution tags, console
|
||
view, metrics, docs. Left: server.yaml-declared models cannot name a
|
||
peer model as a fallback (config catalog is validated before peers are
|
||
read); per-peer CA / mTLS for peers behind a private CA; transitive
|
||
federation (A→B→C models) deliberately not offered; latency is /ready
|
||
RTT only (no per-request peer TTFT used in routing or weighting); the
|
||
peer's adapters are not imported (listed as aliases); no console form
|
||
to add/remove peers (CLI only); key file must exist on every control
|
||
plane.
|
||
- 24 (facility) — built: location labels (`ctl label`, agent.yaml),
|
||
spreading across racks/PDUs, `keep_within`, held domains with drain
|
||
(`ctl facility hold/release`, kept in the store), power caps from
|
||
`facility.budgets` against declared/derived/measured draw, shown in
|
||
explain-placement, `ctl nodes`, `ctl facility`, the console and the
|
||
API. Left: temperature conditions with hysteresis and their
|
||
enforcement (steps 3–4), a Prometheus/DCIM source, power headroom as a
|
||
rack-levelling score, spreading multi-node replicas across racks,
|
||
`llmgw_domain_*` metrics and alerts (near cap, over cap, stale data),
|
||
shedding and tokens per watt (step 5), and a drained node's state
|
||
surviving a restart.
|
||
|
||
---
|
||
|
||
## E. Every setting takes effect when it is changed (2026-10-05)
|
||
|
||
Asked for after a site visit: an upgrade left server.yaml without guardrails,
|
||
a node on another subnet stayed lost until firewall_networks was edited by
|
||
hand, and nothing in server.yaml, agent.yaml or a generated unit applies until
|
||
somebody restarts the right service on every machine. The rule from here on:
|
||
**a file holds only what one machine needs to start; everything else is the
|
||
cluster's, kept in the store, changed through ctl, the console or the web,
|
||
audited, and in force on every control plane within seconds.**
|
||
|
||
### E1. Cluster settings in the store (L) — done 2026-10-05
|
||
Live on the PoC (140, 150): imported as version 1; a guard change applied on
|
||
both without a restart; a router change restarted the standby at once and the
|
||
leader 30s later with /health answering throughout; rollback did the same.
|
||
- Table `cluster_settings` (versioned YAML document, who and why). The
|
||
cluster-wide part of server.yaml lives there; server.yaml keeps the
|
||
per-machine keys only (listeners, TLS paths, store DSN, secrets, tracing,
|
||
monitoring, mirror.api/tls, backup_dir/key, endpoints).
|
||
- First start against a store with no document imports the file's cluster
|
||
keys (and the baseline guard policy when there is none); after that those
|
||
keys in a file are ignored, with a warning.
|
||
- Each control plane polls the document's version and its own server.yaml's
|
||
hash. A change is validated, then applied: `guard` live, the rest by a
|
||
graceful restart of the process (standbys first, the leader last), which
|
||
systemd brings back. No "restart now?" anywhere.
|
||
- `llmgw ctl config show|history|get|set|unset|edit|rollback`; the console's
|
||
settings screen edits the store for cluster keys; admin API
|
||
`GET/PUT /api/v1/settings`, `/history`, `/rollback`.
|
||
- A standby that becomes leader re-reads the catalog, engine choices and
|
||
facility state from the store (it kept what it read at start).
|
||
- Retires `appendGuardIfMissing` (server init): the import does it.
|
||
|
||
### E2. Generated files from settings (L) — firewall on/off done 2026-10-06, every machine 2026-10-09
|
||
`firewall.enabled` in the cluster's settings reaches nodes over the agent
|
||
stream and control planes through llmgw-firewall-sync.path. Since
|
||
2026-10-09 the mirror machine too (the leader PUTs /v1/firewall, the mirror
|
||
writes `.firewall.json`, the same path unit applies it) and store members
|
||
that run no llmgw (the leader keeps it in `cluster_state`, a root timer
|
||
reads the member's own copy with psql); `firewall_networks` reach all of
|
||
them, and the mirror lets in the network of every node and control plane
|
||
the leader knows. Tried on the PoC: off and on, and a network added, on
|
||
control planes, mirror, store member and nodes. Left: the allowed set
|
||
replaced by membership rather than added to (a network taken out of the
|
||
settings stays let in), the other files (the mirror's unit flags,
|
||
keepalived, etcd and Patroni configuration).
|
||
|
||
Every machine runs the agent (root) with roles — node, control plane, mirror,
|
||
store member — and the agent renders and applies what was written once by
|
||
init: the llmgw nft table, the mirror's unit flags, keepalived, etcd and
|
||
Patroni configuration. The firewall's allowed set is the cluster's membership
|
||
(every node, control plane, mirror and store member) plus
|
||
`firewall.extra` from the store; 7443 stays open (mTLS). A node added or
|
||
removed changes every machine's rules. Retires `firewall.Admit` and
|
||
`firewall_networks` in server.yaml (imported into the store).
|
||
|
||
### E3. agent.yaml down to bootstrap (M) — done 2026-10-08 (branch `settings-e`)
|
||
agent.yaml keeps the control planes' addresses, its certificate and its id.
|
||
Labels, fabrics, engines, heartbeat and supervisor settings move to the
|
||
store per node and reach the agent over its stream; a change to the file is
|
||
noticed and the agent restarts itself.
|
||
Built as a section of the cluster's settings rather than a store of its own,
|
||
so history, audit, `ctl config`, the console and the portal page cover it:
|
||
`agents.default` and `agents.nodes.<id>` (config.AgentSettings; bootstrap
|
||
keys node_id/server(s)/hostname/mgmt_addr/tls refused; each node's merge
|
||
loaded as an agent would, so a bad fabric is refused on save). A live key:
|
||
the control plane sends ControlMessage.AgentSettings on register and, a
|
||
second apart per node, on change; the agent compares with what it started on,
|
||
checks it loads beside its agent.yaml (else logs and keeps running), keeps it
|
||
in /var/lib/llmgw-agent/agent-settings.yaml and restarts (engines adopted).
|
||
At start: agent.yaml < kept settings < LLMGW_ env; kept settings that no
|
||
longer load are set aside, never a reason not to start. agent.yaml is
|
||
watched every 10 s and restarted on when it changes and loads. Left: file
|
||
keys are not imported into the settings (they keep working under them);
|
||
`node add` still writes a full agent.yaml; labels have two homes now
|
||
(`ctl label`, which wins, and agents.*.labels).
|
||
|
||
### E4. Web settings page (S) — done 2026-10-08 (branch `settings-e`)
|
||
The same document, edited on /admin/settings, for whoever has no terminal.
|
||
Built: administrators only (staff roles get 404: webhook secrets), checked by
|
||
the same function as PUT /api/v1/settings, made from a base version or
|
||
refused on a newer one (the edit kept in the editor), history with view and
|
||
roll back as a new version, audit with the size not the body, English and
|
||
Turkish, a Settings link in the menu.
|
||
|
||
---
|
||
|
||
## F. Company documents: knowledge bases (RAG), then adapters (2026-10-08)
|
||
|
||
A company wants a model to answer from its own documents. Retrieval is the
|
||
tool for *knowledge* (sources shown, re-indexed in minutes, access per
|
||
tenant); LoRA training is for *behaviour* (vocabulary, format) and has no
|
||
access control — whatever goes into weights reaches everyone who may use the
|
||
adapter. RAG first. Decisions taken: access per collection (no per-document
|
||
ACL yet); a personal key or session names a project or team and gets that
|
||
chain only, never the union of its person's memberships; PostgreSQL store
|
||
only, off on SQLite with a sentence saying why.
|
||
|
||
### F1. Store: collections, grants, chunks (M) — done 2026-10-08 (branch `rag`)
|
||
Built (`internal/store/knowledge.go`, migration 0035 PostgreSQL only):
|
||
- `kb_collections` (owner scope/id, org, embed_model, dims), `kb_grants`
|
||
(read|write, any scope, never across organizations; a key needs a
|
||
project; a person needs a membership in the organization),
|
||
`kb_documents` (digest: the same again is no change, a new one replaces
|
||
in one transaction).
|
||
- `kb_chunks` is created when the store opens, not in a migration (a
|
||
migration needing an extension would stop the control plane; one that
|
||
skipped it would never run again): partitioned by organization (hashed
|
||
partition name), `tsv` generated (`simple`), GIN on the parent, one
|
||
partial HNSW index per collection on its organization's partition
|
||
(`embedding::vector(dims)`, `WHERE collection = '<id>'`). EXPLAIN on
|
||
20k rows: partition pruned, that index used, policy applied.
|
||
Later (scale, 2026-10-08): a partial GIN per collection too
|
||
(`kb_tsv_<id>`, backfilled at open); `vec_type` per collection, `halfvec`
|
||
on pgvector ≥ 0.7 (up to 4000 dims); `SET LOCAL hnsw.ef_search` to
|
||
candidates(k) clamped 40..1000 (EXPLAIN: 40 rows without it, all with
|
||
it); the text arm takes candidates by `ts_rank` in a subquery and scores
|
||
only those.
|
||
- RLS FORCEd on `kb_chunks` against `llmgw.kb_org`/`llmgw.kb_collections`
|
||
(set with set_config per transaction), one role — the table owner — not
|
||
a separate one. Chunks are deleted by the foreign-key cascade from their
|
||
document, which RLS does not block.
|
||
- `CollectionsFor(Effective, write)`: owner or grant anywhere on the chain;
|
||
no organization (personal key without a project) reads nothing.
|
||
- `SearchChunks`: per collection nearest-neighbour (literal id and cast so
|
||
the partial index is used) plus a full-text arm with any word
|
||
(`a | b`, lowercased by the database, not Go), fused by reciprocal rank.
|
||
- Deleting a project/team/org that owns collections is refused like one
|
||
with members (changed from "deleted with them": every other delete in the
|
||
tree refuses); DeleteUser/ForgetUser delete the person's collections;
|
||
deleting an organization drops its partition; grants to a deleted scope go.
|
||
- `server init` installs `postgresql-N-pgvector` / `pgvector_N` and enables
|
||
it; pgha installs it on every member at setup and enables it on the
|
||
leader on every run. Neither stops on failure.
|
||
Left: a cluster set up before (the PoC) gets no package from a rerun of
|
||
pgha on a running cluster — install by hand or add an upgrade step;
|
||
`server restore`/`CopyTo` do not carry kb tables (SQLite → PostgreSQL only,
|
||
and SQLite has none); tests need `LLMGW_TEST_POSTGRES` pointing at a
|
||
pgvector database as a non-superuser (a superuser bypasses RLS).
|
||
|
||
### F2. Ingest on the user's machine (L) — done 2026-10-08 (branch `rag`)
|
||
Uploading whole files to the control plane was dropped: a PDF is 20–100×
|
||
its text, and the work belongs where the files are. Instead a small client,
|
||
`llmgw-kb`, run by the person on their own machine with their own key:
|
||
- **Its own binary** (`cmd/llmgw-kb`), for Windows, macOS and Linux on amd64
|
||
and arm64. `llmgw` itself does not build on Windows (agent syscalls), and
|
||
employees' laptops should not carry the server, agent and setup screens. A
|
||
test keeps server packages out of its imports.
|
||
- `llmgw-kb sync <collection> <dir|file…> [--include/--exclude globs]
|
||
[--dry-run] [--delete]`: scan → local catalog (path, size, mtime, sha256;
|
||
cache under os.UserCacheDir, digests not recomputed for unchanged files)
|
||
→ the collection's manifest from the gateway (name → digest only) → diff
|
||
→ for new/changed files only: extract, chunk, embed through the gateway's
|
||
`/v1/embeddings` with the person's key (its accounting, limits, guard and
|
||
band apply), and send name, digest, chunk texts and vectors; `--delete`
|
||
removes documents whose files are gone. `--dry-run` lists what would leave
|
||
the machine and how much. A file that cannot be read (locked, unsupported,
|
||
no text) is skipped with the reason; the run goes on.
|
||
- **Formats, as many as possible.** Built in, pure Go: PDF (text layer),
|
||
DOCX/XLSX/PPTX, ODT/ODS/ODP, RTF, EPUB, HTML/XHTML, EML (text and HTML
|
||
parts), Jupyter notebooks, subtitles (SRT/VTT), CSV/TSV, JSON/YAML/XML,
|
||
Markdown/RST/AsciiDoc/Org, plain text and source code; ZIP archives
|
||
recursed into. With a helper found on the machine: legacy DOC/XLS/PPT, RTF
|
||
and anything else LibreOffice opens (`soffice --headless --convert-to`),
|
||
images and scanned PDFs through Tesseract (`tur+eng`). Text in UTF-8,
|
||
UTF-16 (BOM) or Windows-1254/1252 is converted to UTF-8.
|
||
- **Across systems:** document names relative to the synced root, `/`
|
||
separated, Unicode NFC (macOS hands out NFD), a warning for two names that
|
||
differ only in case; CRLF → LF; key from `LLMGW_API_KEY` or `llmgw-kb
|
||
login` (os.UserConfigDir, 0600 where that means something); the system's
|
||
CA store, `--ca` for another.
|
||
- **Gateway endpoints** (client listener, person's key):
|
||
`GET /v1/kb/collections` (what this key may read/write),
|
||
`GET /v1/kb/collections/{id}/manifest`, `PUT
|
||
/v1/kb/collections/{id}/documents` (one document per request: name,
|
||
digest, bytes, chunks with text and embedding; dims checked, chunk count
|
||
and size bounded), `DELETE …/documents?name=`. Write needs a write grant
|
||
or ownership on the key's chain (`CollectionsFor(…, write)`); otherwise
|
||
404.
|
||
- **Administration** stays with operators: `llmgw ctl kb
|
||
create|list|delete|grant|revoke|docs`, admin API `/api/v1/kb`, audit,
|
||
console screen; `docs/knowledge.md`.
|
||
- Trade-off accepted: vectors come from the client, so a writer could store
|
||
vectors that do not match their text — only in collections they may
|
||
already write.
|
||
Built: `internal/kb/extract` (formats above; helpers found on PATH and in
|
||
each system's install places), `internal/kb/chunk` (≈1800 characters under
|
||
the heading path, a paragraph of overlap, sections kept apart once a chunk
|
||
is a third full), `internal/kb/kbsync` (ledger in SQLite at
|
||
os.UserConfigDir/llmgw-kb/ledger.db: documents keyed by gateway, collection
|
||
and name with file, size, mtime, digest, state, reason; runs), `cmd/llmgw-kb`
|
||
(login, collections, sync, status, history, formats), `internal/router/kb.go`
|
||
(the four gateway endpoints, `X-Llmgw-Project` for personal keys, audit),
|
||
`internal/control/adminapi/kb.go` + `llmgw ctl kb` + console Knowledge
|
||
screen, `docs/knowledge.md`; CI builds the client for all six targets and
|
||
releases attach it. Tested: every format from generated fixtures and a
|
||
Turkish PDF (OCR path with Tesseract here), sync against a fake gateway and
|
||
end to end against the router on pgvector.
|
||
Tried on the PoC 2026-10-08 (control planes 140 and 150 on a dev build of
|
||
this branch; their previous binary is /usr/local/bin/llmgw.pre-rag): Ubuntu's
|
||
pgvector 0.6.0 installed on 149/150 and enabled; bge-m3's weights were never
|
||
in the mirror (fetched from CompendiumLabs/bge-m3-gguf@1864f47f) and it is
|
||
held at 1 replica; tree Demo/ik/belgeler and Demo/satis/crm with a key each;
|
||
collection Demo/politikalar filled from a Mac through an SSH tunnel with ten
|
||
Turkish files in nine formats (9 sent in 5 s, the .doc skipped for want of
|
||
LibreOffice; a rerun sent nothing in 0 s; a changed file alone was sent).
|
||
Nine Turkish questions each found their page first once search scored by
|
||
similarity plus word coverage (rank fusion had got two wrong). The other
|
||
team's key saw no collection and got 404 by id; with a read grant it read
|
||
and could not write; revoked, nothing again. On the live store the llmgw
|
||
role (not superuser, no BYPASSRLS) sees 0 chunks with no scope set and 0
|
||
with the organization alone.
|
||
Left: the gateway's collection list shows the owner as team:ID, not its
|
||
path; no LibreOffice on the dev machine, so the legacy
|
||
Office path is untested beyond its error; the client is unsigned (Gatekeeper
|
||
and SmartScreen warn); Outlook `.msg` is not read; one document over ~60 MiB
|
||
of JSON (a few thousand pages) is refused rather than split.
|
||
PostgreSQL tests of store, router and adminapi share LLMGW_TEST_POSTGRES and
|
||
the store's TRUNCATE each test; run them with `go test -p 1` or one package
|
||
at a time, or a router test loses its rows mid-run.
|
||
|
||
### F3. Retrieval on the request path (M) — done 2026-10-08 (branch `rag`)
|
||
- `llmgw.knowledge: [collections]` in the body (stripped before the
|
||
engine), or default collections per project/key.
|
||
- Allowed set from the caller's chain (§7.9.2), cached like limits; grants
|
||
add up down the tree. Personal keys and sessions need `x-llmgw-project`
|
||
(or the playground's picker) naming a scope the person is a member of.
|
||
- A collection not allowed answers 404, exactly as a missing one.
|
||
- Hybrid search with the filter inside the query (per collection, its own
|
||
partial index; pgvector 0.6 is enough —
|
||
scored by similarity plus the share of the question's words held (built
|
||
in F1/F2 as SearchChunks), `/v1/rerank` when a reranker is in the catalog. Top
|
||
chunks into a system message with document and position;
|
||
`x-llmgw-sources` lists them.
|
||
- Leaks closed: answer-cache key includes the allowed set; per-organization
|
||
`cache_salt` on RAG requests; injected passages under `no_store` and the
|
||
guard's prompt-injection check; returned doc ids in the audit.
|
||
- Done: router tests that tenant B never receives tenant A's chunks
|
||
(including via cache), that a personal key without a project is refused,
|
||
metrics (retrieval latency, chunks returned), docs.
|
||
|
||
Built (`internal/router/rag.go`): `X-Llmgw-Knowledge` (header, as the
|
||
other llmgw extensions are, not a body field), `X-Llmgw-Knowledge-K`;
|
||
collections from the key's chain (kbEffective, shared with the kb
|
||
endpoints), 404 for one not readable; the question embedded through the
|
||
ordinary path on the caller's principal; SearchChunks; passages trimmed to
|
||
the model's context; added to the one leading system message (or a new
|
||
one); `x-llmgw-sources`; before guardInput; never cached; metrics. Default collections built (kb_defaults on org/team/project/key, narrowest
|
||
wins, 30 s cache, `ctl kb default`, console action; never refuse); the
|
||
portal chat's Documents picker (collection, default or none; sources listed
|
||
under the answer; a header the page sends itself is dropped). `cache_salt` per organization on requests given passages, sent to vLLM
|
||
replicas only (chosen replica's engine). Checked on the PoC 2026-10-08:
|
||
vLLM 0.30.0 (cuda01) with TinyLlama-1.1B AWQ run by hand beside the 35B in
|
||
the card's last 2.3 GB — a 668-token prompt reused 656 tokens under the same
|
||
salt and 0 under another's, with identical text (llmgw cannot place it
|
||
there: placement takes whole GPUs). Live on the PoC: a team default answers
|
||
a request without the header, `none` turns it off, an explicit header with
|
||
k=2 gives two sources and no more, llmgw-kb lists owners by path. Left: returned passages
|
||
are not audited per request (the header and the kept request record them).
|
||
|
||
### F4. Adapter training (L, after F1–F3) — done 2026-10-08 (branch `adapters-f4`)
|
||
A run-to-completion job placed by the control plane like a replica and
|
||
supervised by the agent like an engine, in a Python environment the agent
|
||
installs (TRL + PEFT, QLoRA). Data: Q/A pairs generated from a collection by
|
||
a cluster model through `/v1/batches`, reviewed by people. Output to the
|
||
mirror, into the model's `adapters`, through canary and a quality suite.
|
||
Runs when the scaling schedule frees a GPU; 8–14B dense bases on one card.
|
||
Never train on a collection some users of the adapter may not read.
|
||
Tried 2026-10-08 on the PoC, by hand: Qwen2.5-0.5B-Instruct (fp16) with a
|
||
LoRA (r=16, every linear layer, 8.8M trainable), trained beside the 35B in
|
||
cuda01's last 2.3 GB (peak 2158 MiB on the card, 64 s for 138 steps, PEFT in a
|
||
venv over the vLLM engine's torch). Data: 90 Turkish Q/A pairs a cluster
|
||
model wrote from the nine Demo/politikalar documents, through the gateway.
|
||
On nine questions it had not seen, phrased otherwise: base 0/9, 6 epochs
|
||
4/9, 4 epochs at a lower rate 2/9. Wrong answers mixed facts across
|
||
documents (the meal card's amount, the office day). The same nine through
|
||
retrieval: 9/9 with sources. So F4 is for format, vocabulary and tone, and
|
||
facts come from F1–F3; a bigger base on a free card would do better but not
|
||
change that. The training itself fits a small slot on a shared card, which
|
||
the job design can rely on for small bases.
|
||
|
||
Built: `ctl kb dataset` (POST /api/v1/kb/collections/{id}/dataset, NDJSON
|
||
with each pair's collection); train_jobs/train_artifacts (migrations 0037,
|
||
0038 owner); internal/control/train (queue, one job per node, artifacts in
|
||
1 MiB pieces, a node that goes fails its job); internal/agent/train (the
|
||
newest vLLM's python3 + `uv pip install --no-deps --target` peft 0.21.2 and
|
||
accelerate 1.15.0; llama.cpp's conversion code for the installed release,
|
||
untarred from its source; the card with the most free memory, if enough;
|
||
runs as llmgw-engine); `ctl train start|list|show --follow|cancel|rm|
|
||
download|publish`; the mirror's PUT /v1/models/{model}/{version}/{file};
|
||
publish uploads the GGUF (llama.cpp) or PEFT files (vLLM) and sets the
|
||
adapter after checkAdapters. Owners: catalog.Adapter.Owner, enforced in the
|
||
router (404, hidden from /v1/models); a job's owner defaults to its single
|
||
collection's owner, must be able to read every collection the pairs came
|
||
from, and pairs from a knowledge base are never public. Tried on the PoC:
|
||
dataset of 90 pairs in 89 s; a job trained beside the 35B (peak 2214 MiB,
|
||
138 steps in about a minute) and sent back PEFT and GGUF.
|
||
Left: a job dies with its agent's stream (no resume); no console screen for
|
||
training yet; quality gating of a published adapter is by hand (ctl eval).
|
||
|
||
---
|
||
|
||
### F5. Large imports (M) — measured 2026-10-09, open
|
||
The metallurgy corpus (2,314 files, 108,779 chunks, bge-m3 on the one
|
||
GPU) went in at about 25 chunks/s, the same over a phone hotspot and the
|
||
LAN. The engine's slots were the limit: llama.cpp `--parallel 4` embeds
|
||
four texts per pass, while llmgw-kb sends 32 a request from 6 workers. The
|
||
usage rows show 4.6 s a request on average, 1.8 s of it in the gateway's
|
||
queue and 0.04 s in the guard. The GPU sat at 18–50% and the store machine
|
||
at a third of its CPUs.
|
||
- An embedding model's profile with many slots by default (`--parallel`
|
||
32–64, `--ctx-size` to match, `--batch-size`/`--ubatch-size` at least the
|
||
longest chunk), or `ctl model serve` choosing that for a model with
|
||
`capabilities.embeddings`. With the default 512, one 516-token chunk failed.
|
||
- Embedding on the gateway's side of a document PUT, so the client sends
|
||
text only: today an embeddings answer is ~21 KB of JSON a chunk, and
|
||
the text and vector go up again (~11 KB).
|
||
- The store next: HNSW inserts took half of one of pg01's two cores; at a
|
||
million chunks the vectors alone outgrow its 4 GB. halfvec (pgvector ≥
|
||
0.7) and a larger store machine.
|
||
- Done: a batch of embeddings checked text by text against the context
|
||
(branch `embeddings-batch`); it was refused as one over-long prompt.
|
||
- Done: `query_prefix` on an embedding model, before questions only.
|
||
- Qwen3-Embedding-4B on the RTX 6000 (2026-10-09): llama.cpp Q8 1,575
|
||
tok/s as first set up (32k cache, 16 slots), 2,700 with a 16k cache,
|
||
FlashAttention and ubatch 1–2k; f16 no better. vLLM fp16 (`--runner
|
||
pooling`) 6,200 direct, 4,700–4,900 through the gateway in the import,
|
||
and its vectors match llama.cpp's (cosine 0.9995). The 103,579 chunks
|
||
took 3 h. Done (2026-10-10, branch `admission-bulk`): a source per
|
||
profile, so an embedding model's GPU profile runs vLLM and its CPU
|
||
profile llama.cpp; a band's own `queue_ttl` for bulk imports, and
|
||
`Retry-After` on a busy refusal, which is what had turned passages away
|
||
(llmgw-kb retries only what says when to).
|
||
- Retrieval on 40 questions written from the corpus, document in the top
|
||
5: bge-m3 31/40 (Turkish questions 13/21), Qwen3-Embedding-4B 37/40
|
||
(18/21), with its query prefix 36/40 and first place 31/40 instead of
|
||
28 (MRR 0.738 → 0.789 → 0.838). Next: translating the question with the
|
||
chat model, and a reranker.
|
||
|
||
## G. Jobs: 3D generation through ComfyUI (2026-10-10)
|
||
|
||
A request for a 3D model takes minutes on the one 24 GB card, and the card has to be
|
||
emptied of the chat model first. That is not a request to hold a socket for
|
||
— `internal/router/activate.go:46` already says so — it is a job: queued,
|
||
answered with an id, asked after, its result downloaded. The job machinery
|
||
is built first and tested without a GPU; the ComfyUI generalisation and the
|
||
custom-node install come after. Decided 2026-10-10: a `jobs` table of its
|
||
own rather than a one-line batch (the JSONL shape would be artificial and
|
||
has nowhere for progress), and progress is shown, not just a status.
|
||
|
||
Which models (decided 2026-10-10): the assets go into a game sold on
|
||
Steam, worldwide, the EU included, so the licence of the output decides
|
||
before the quality does. Hunyuan3D (every version) is out: Tencent's
|
||
licence excludes the EU, UK and South Korea, and whether a game shipped
|
||
there with its outputs is "use in the Territory" is not a question to
|
||
leave to a court. In, in this order: **TRELLIS.2** shape (MIT; the best
|
||
geometry; its own texture baking goes through nvdiffrec and nvdiffrast,
|
||
both under NVIDIA's non-commercial source licence, so that stage is not
|
||
run), **Step1X-3D** (Apache 2.0, StepFun; shape and texture in one, the
|
||
EU-safe textured path — its ComfyUI node and VRAM need are to be
|
||
checked), **Direct3D-S2** (MIT, DreamTech; high-detail shape, has a
|
||
ComfyUI node) if TRELLIS.2's detail falls short. Not in: TripoSR and
|
||
InstantMesh (permissive but superseded), Stable Fast 3D (Stability's
|
||
community licence, revenue-capped), SAM 3D Objects (Meta's own licence,
|
||
to be read before it is considered). Steam's disclosure form (changed
|
||
2026-01-16) lists pre-generated 3D models as content that must be
|
||
declared on the store page: the licence notice G3 writes into the GLB is
|
||
what the studio answers that form from.
|
||
|
||
### G1. Jobs: store, worker, endpoints (M) — done 2026-10-10 (branch `jobs-g1`)
|
||
Built as written, with two choices: a job found running after its gateway
|
||
died is run again from the start (the prompt id column is there for G2 to
|
||
follow the same work instead), and the request is validated at the door
|
||
only as far as costs nothing — key, model, allowlist, the request's shape
|
||
— with limits, guardrails and admission applied when the worker replays it,
|
||
said as the job's error. The result file's `content_type` is the job's; the
|
||
files endpoint still serves every file as an octet stream.
|
||
- Store: `jobs` (id, owner key/principal, kind, model, status, input
|
||
(JSON), progress (JSON: queue position, percent, step), result file id,
|
||
error, created/started/finished, lease owner/until), migrations for
|
||
sqlite and postgres beside `internal/store/batches.go`, whose
|
||
claim/renew/release lease pattern is copied, not shared: a job is one
|
||
unit of work, a batch is many lines.
|
||
- Worker: `internal/router/jobworker.go` after `batchworker.go`. Any
|
||
gateway on the store claims a queued job, asks for the model with
|
||
`activator.Want` and waits as long as it takes (no `ActivationWait`
|
||
bound: nobody is holding a connection), runs it, stores the output
|
||
through the files store, writes usage and price, marks it done. A
|
||
gateway that dies mid-job loses its lease and another one picks it up;
|
||
the ComfyUI prompt id is in the job row so the second gateway asks
|
||
after the same work rather than queueing it again.
|
||
- Endpoints: `POST /v1/3d/generations` validates like any inference
|
||
request (key, allowlist, limits, admission, guardrails on the prompt),
|
||
writes the job, answers 202 with `{id, status: "queued"}`;
|
||
`GET /v1/3d/generations/{id}` returns status, progress and, when done,
|
||
`output_file_id` and the usage; `POST .../{id}/cancel`; the file comes
|
||
from `GET /v1/files/{id}/content`. Owner-scoped like batches. Results
|
||
are kept for the batch files' retention and deleted with them: a
|
||
departure from `images.go`'s "kept nowhere", because a mesh is a
|
||
download, not a JSON field; the retention is the bound.
|
||
- Done when: a job survives the gateway that took it being killed, a
|
||
cancelled job leaves ComfyUI's queue (`comfy.Client.cancel`), two
|
||
gateways never both run one job (test with the lease), and the usage
|
||
row carries the job's price.
|
||
|
||
### G2. Progress from ComfyUI (S, after G1) — done 2026-10-10 (branch `jobs-3d`)
|
||
Built as written: the gate's websocket under its own client id, the
|
||
client_id of every /prompt rewritten, /progress/{prompt_id} on the gate,
|
||
the router reading it and /queue into the job (only into a job of the same
|
||
key). Left: a node that reports no steps (TripoSR's) shows its name and no
|
||
percent; a vanished replica is said by the job failing, not by progress.
|
||
The gate (`internal/comfy/gate.go`) passes seven paths and no websocket.
|
||
Rather than open `/ws`, the gate subscribes to ComfyUI's own `/ws` once,
|
||
with `golang.org/x/net/websocket` (already a dependency), under a client
|
||
id of its own, and rewrites the `client_id` of every `/prompt` it
|
||
forwards to that one. It keeps the last `executing`, `progress`
|
||
(value/max/node) and `execution_error` message per prompt id and serves
|
||
them at `GET /progress/{prompt_id}` on the gate; the router polls that
|
||
beside `/history` and `/queue` and writes queue position, percent and the
|
||
current node's class into the job's progress. The gate's buffer is
|
||
bounded (the last N prompts) and the websocket reconnects. Done when a
|
||
3D job shows "queued, 2 ahead" then "running 40%, step 12/30" in
|
||
`ctl jobs` and on the account page, and a job whose replica vanished
|
||
says so within one poll.
|
||
|
||
### G3. ComfyUI beyond pictures: 3D models and custom nodes (M, after G1) — done 2026-10-10 (branch `jobs-3d`), with TripoSR
|
||
Built: output kinds, image upload through the gate, every output list
|
||
fetched, OBJ to GLB, licence in /v1/models, the job and the GLB, per_mesh
|
||
and usage.meshes (migration 0045), custom nodes cloned at their commit by
|
||
the gate with packages beside the replica (a node's named packages at its
|
||
range, the rest at ComfyUI's versions). Not as written: the model. On the
|
||
PoC's Turing card TRELLIS.2 cannot run — it needs Meta's gated DINOv3,
|
||
its published background remover (RMBG-2.0) is non-commercial, and its
|
||
sparse attention and Direct3D-S2's need flash-attention (Ampere and
|
||
newer). TripoSR (MIT) proves the chain on cuda01: a .glb from a picture
|
||
in about 30 s. Left: background removal before TripoSR (BiRefNet, MIT,
|
||
as a node), TRELLIS.2 or Direct3D-S2 on an Ampere card with an ungated
|
||
encoder, a missing node refusing the replica by name.
|
||
- Catalog: a model makes `pictures` or `meshes` (`MakesImages` becomes an
|
||
output kind; `ServesComfyUI` stays). 3D weight folders in the gate's
|
||
`modelFolders`; the workflow's placeholders gain `image` (the input
|
||
picture, base64 in the request, written to the replica's `input/` by
|
||
the gate through a new `POST /upload/image` the gate allows — or a
|
||
base64-loading node, whichever TRELLIS.2's wrapper wants).
|
||
- `comfy.Client.fetch` reads every output key, not `images` only, and
|
||
fetches each file named by `/view`; `Image` becomes `Artifact` with its
|
||
content type (model/gltf-binary, application/octet-stream for .obj).
|
||
- Custom nodes: a model (or the engine config) names git repositories
|
||
and a pip requirements list; the agent's provision step clones them
|
||
into the replica's `custom_nodes/` (the gate's `--base-directory` is
|
||
where ComfyUI looks, not the install directory) and installs the
|
||
requirements into the engine's venv with the same PyTorch index and
|
||
lock discipline as A8. A replica with a missing node refuses to start
|
||
with the node's name in the error rather than ComfyUI's traceback.
|
||
- Licence of the output (decided 2026-10-10: it matters). A catalog model
|
||
carries `license: {name, url, notice}`; a job's answer, `/v1/models`
|
||
and the portal page show it beside the result, and the mesh's GLB
|
||
`asset.extras` carries the notice too, since the file travels without
|
||
the API. What was checked on 2026-10-10: TRELLIS.2 weights are MIT and
|
||
its outputs free, but its texture baking depends on nvdiffrast, whose
|
||
NVIDIA Source Code License is non-commercial — a commercial service
|
||
must render textures another way or skip that stage. Hunyuan3D 2.0 and
|
||
2.1 claim no rights in outputs, but the licence excludes the EU, UK and
|
||
South Korea, needs Tencent's permission above 1 M monthly active users,
|
||
forbids using outputs to train other models, and asks for "Powered by
|
||
Tencent Hunyuan"; the notice ships with any distribution. TripoSR is
|
||
MIT, InstantMesh Apache 2.0. Every one of them trained on Objaverse,
|
||
some of whose objects are CC-BY-NC: a provenance question no licence
|
||
text settles, to be said in the notice rather than hidden.
|
||
- Pricing: `per_image` stays; `per_mesh` beside it in `accounting.Price`,
|
||
`Outcome.Meshes`, the usage sums and exports (as eaea671 did for
|
||
pictures).
|
||
- Done when: TRELLIS.2 on cuda01 with `scaling: {min: 0,
|
||
idle_timeout: 10m}` makes a .glb from a picture through G1's endpoint
|
||
with nvdiffrast and nvdiffrec not installed in its venv (the licence
|
||
is kept by the install, not by a note), the GLB carries the MIT
|
||
notice,
|
||
the card is free again after the idle timeout, and the same request
|
||
against flux-schnell is refused as "makes pictures, not meshes".
|
||
|
||
### G4. Portal and console (S, after G2) — done 2026-10-10 (branch `jobs-3d`)
|
||
Built: /3d with model-viewer served from the gateway, progress, download
|
||
and licence; `ctl 3d list|get|cancel` over /api/v1/client-jobs (ctl jobs
|
||
was taken by the control plane's own work). Left: the console's Jobs view.
|
||
A 3D page beside `/images` (upload a picture, watch progress, download
|
||
the .glb, a viewer with `<model-viewer>` from the allowed CDNs), `ctl
|
||
jobs list|get|cancel`, the console's Jobs view. The page sends down the
|
||
same path as the API, like `imagesSend` does.
|
||
|
||
## D. Facts worth knowing before starting
|
||
|
||
- Admin API: `127.0.0.1:9090`, no auth, `ctlClient` in `internal/cli/ctl.go`.
|
||
Metrics on `:9091` with `/prometheus/targets` beside them.
|
||
- Store: `internal/store`, migrations under `internal/store/migrations/{sqlite,postgres}`;
|
||
the PoC runs PostgreSQL under Patroni.
|
||
- The console (`sudo llmgw`) is `internal/cli/tui_console.go` with actions
|
||
in `tui_actions.go`; every operator-visible change needs a row action or
|
||
a form there, and a line in `docs/console.md`.
|
||
- Tests of the console use a fake admin API (`internal/cli/tui_actions_test.go`,
|
||
`adminStub`); tests of serving use fakes in
|
||
`internal/control/modelserve/modelserve_test.go`.
|
||
- `go generate ./...` regenerates `docs/reference.md` and
|
||
`internal/docgen/reference_gen.go` whenever a config struct changes; the
|
||
docgen test fails otherwise.
|