ilker/llmgw
2
0 Fork 1
Code Issues Pull Requests Actions Packages Projects Releases 95 Wiki Activity
Files
3a3db8cd201a940c6e25e296d1c6436b452b048f
llmgw/BACKLOG.md
T
Ilker ManapandClaude Opus 5.5 3a3db8cd20
CI / test (push) Successful in 8m52s
Details
Answers to similar questions, not only the same one 
llmgw keys cache --similar 0.95 also gives a chat question the answer to
one like it: the last user message embedded by router.cache.embedding_model
(asked on the gateway's account, as a judge is, within 2 s), compared by
cosine with the questions answered in the same conversation — system
prompt, earlier turns, tools and sampling the same. The exact cache is
asked first; an exact hit embeds nothing. Said as x-llmgw-cache: similar
with x-llmgw-cache-similarity, charged as a hit.

The vectors live with their answers in the gateway's memory, counted in
max_bytes and evicted with them, not in a pgvector table as BACKLOG I1
planned: the cache keeps answer text out of the store on purpose. A
question over 2,000 characters is reused only when the same. Migration
0051 holds the setting. BACKLOG I1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 17:47:11 +01:00

1803 lines
112 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Backlog
What a review of the code on 2026-10-04 (commit c2238ab) found should be done
next: security fixes first, then guardrails, then features. Written for
whoever implements it — a person or a model — without the review in hand.
Every item names the files it touches and what "done" is. Paths are relative
to the repository root. Sizes: S about a week, M two to four weeks, L more.
Conventions this codebase keeps, which every item here must keep too:
- A change is refused before anything is written (validate, then persist).
- Nothing on the serving path logs prompt text, keys or passwords.
- Each item gets unit tests beside the code (`*_test.go`), and the TUI
(`internal/cli/tui_*.go`), `docs/console.md` and `CHANGELOG.md` are
updated when behaviour an operator sees changes.
- `go test ./...`, `go vet ./...` and `govulncheck ./...` pass. No release is
cut and nothing is deployed to the PoC cluster without being told to.
## Open, in order (2026-10-20)
What is left, most needed first. The detail is in the item each names.
**Now: what goes wrong where it is installed.**
1. ~~**J8**~~: done 2026-10-20. `ctl guard judge check`, and init times the judge. Left of **J3**: its load test
against judges on the control plane, which was stopped half way. Judges on a 4-core node of their own refused
0/60 at 2 at once, 13/60 at 4 and 42/60 at 10.
2. ~~**I2 P2**~~: done 2026-10-20: the guardrails both ways, a rate per caller and server, a breaker. A portal page
for the servers is left, with P3 (stdio, per-person OAuth).
3. **K**: fixes from the fabric test the operator runs with docs/large-models.md.
**Soon: the customer's experience, and HA.**
4. ~~**C24**~~: done 2026-10-21. The store's, catalog's and mirror's errors, percent signs, footer times and stale
forms are in the page's language. Left: the /docs documents, and a language kept per person (cookie only today).
5. **J4**: the model mirror on more than one machine, the last single point of an HA install. (M)
6. **F5**: large imports limited by the embedding engine's slots. (M)
**From Kong** (section L): L1 done; L9, then L2 and L3, then L4/L5. L6 and L7 are llmgw-80's.
**Governance.**
7. **C1**: a tenant-admin role, and forms hidden from viewers.
8. **Audit before and after**: what a change changed, and changes made with the CLI recorded too.
**When there is room.**
9. **I1**: answers to similar questions from the cache. (M)
10. **H1**: tried on the PoC.
11. **C4**: Parquet, and a scheduled push for usage export.
**When needed.**
12. **H5**: licences in video and audio. 13. **H3**: replicas shared, the catalog side. 14. **J7**: nginx in front
of several control planes. 15. **J6**: rate limits per process (a note).
**Done since 2026-10-16:**
- J1, J5, I3, I4, J2.
- Active Directory and Entra ID, tried against a Samba DC, with the console's Sign-in screen.
- J3's code.
- K's code and docs/large-models.md.
- I2 P1 and the cluster's own read-only MCP server (llmgw:admin).
- The node console's Overview; a node's page showing its version and the models on each GPU.
**Not code, the operator's to decide or do:**
- Where Alertmanager sends (its receiver is "none"): with a judge down, nobody hears.
- The fabric test for K.
- The PoC's control planes run a dev build of main (v0.14.9-11-g4eb49d0); its nodes run v0.14.8.
---
## A. Security — do first
**Status (2026-10-04): done and merged.** A1–A9, A6b included — etcd and
Patroni's API on TLS, the PoC moved in maintenance mode without a failover.
Deliberately narrower than written: the admin API is a token on loopback
rather than a unix socket (A5); thread ids are bounded and documented as
project-wide rather than scoped per user (A8); Python engines resolve with an
explicit PyTorch index and a PEP 751 lock rather than locks shipped per engine
version (A8). All of A is live on the PoC.
### A1. Toolchain (S, hours)
`govulncheck` finds 11 standard-library vulnerabilities reachable from this
code at Go 1.26.3 (net/http, crypto/x509, os.Root, net/textproto). Raise the
`go` directive in `go.mod` to 1.26.6 or later, rebuild, run `govulncheck
./...` until it reports none reachable. Add `govulncheck` to the Makefile's
check target.
### A2. Release signing is effectively off (S)
- `internal/release/trusted.pub` is empty, so `TrustedKey()` returns "" and
`release.go:103` skips signature verification. Ship a real public key;
make a missing or invalid signature fatal. Drop the `LLMGW_RELEASE_KEY`
environment override, or honour it only in `-tags dev` builds.
- `internal/onboard/onboard.go:365`: the node-side download checks only a
same-origin `checksums.txt`. Verify the signature there too (simplest:
have the node run `llmgw` itself to fetch and verify once it has any
binary), and `shellQuote` the `Releases` URL, which is interpolated into
single quotes unescaped.
- `internal/agent/upgrade.go:44`: accept only `https://` release bases.
- Done when: a tampered binary or checksums file is refused by self-upgrade,
node onboarding and agent upgrade, with a test for each.
### A3. Agent port without TLS is anonymous and plaintext (S)
`internal/cli/server.go:423` serves h2c on `:7443` when `agent_tls` is not
configured; `agentsvc` then accepts any node id. Refuse to start control
mode without TLS unless `agent_tls.insecure: true` is set explicitly; when
insecure, bind `agent_listen` to 127.0.0.1 and log a warning at startup and
in `llmgw doctor`. Keep `server init` writing TLS as it does.
### A4. Model profiles reach root on every node (M)
Profile `args` and `env` from a model definition are appended to the engine
command unchecked (`internal/agent/engines/llamacpp.go:170`, `vllm.go:151`,
`sglang.go:98`); `internal/agent/engines/unsafeenv.go` is a denylist that
misses `NCCL_NET_PLUGIN`, `NCCL_TUNER_PLUGIN`, `NCCL_PROFILER_PLUGIN`,
`CUDA_INJECTION64_PATH`, `NVTX_INJECTION64_PATH`, `TRITON_*` paths; engines
run as root.
- Replace the env denylist with an allowlist (CUDA_/NCCL_ tuning keys minus
anything ending in `_PLUGIN` or containing `INJECTION`, `VLLM_*`,
`HF_*` minus path keys, `OMP_*`, `GGML_*`). Apply at catalog validation
(`internal/control/catalog/catalog.go`, where `UnsafeEnv` is called) and
again in the agent.
- Deny engine flags that write files or run code: `--trust-remote-code`,
`--log-file`, `--slot-save-path`, `--chat-template` with a path,
`--host`, `--api-key`, `--port`; vLLM/SGLang equivalents. Allowlist is
better than denylist if the flag set is enumerable per engine.
- Run engines as an unprivileged user with the video/render groups
(`internal/agent/supervisor`, systemd unit in
`internal/cli/serverinit_cluster.go:52`). Weights cache must be readable
by it.
- Done when: a profile with `--trust-remote-code` or `NCCL_NET_PLUGIN` is
refused at `ctl model add` and at the agent; `ps` shows engines not as
root.
### A5. Admin API trusts any local process (M)
`internal/cli/server.go:357` serves the admin API and pprof on
`127.0.0.1:9090` with no authentication; `adminapi.actorOf`
(`internal/control/adminapi/api.go:1613`) takes the actor from the
client-sent `X-Llmgw-Actor` header.
- Serve the admin API on a unix socket (`/run/llmgw/admin.sock`, 0660,
group `llmgw-admin`) by default, keeping TCP as an option that requires
a bearer token read from a root-only file (`/etc/llmgw/secrets/admin-token`).
- Take the actor from `SO_PEERCRED` (uid → username) on the socket, or from
the token's name; ignore the header.
- Refuse `admin_listen` on a non-loopback address unless `admin_insecure:
true`.
- Put pprof behind its own `debug.pprof: true` setting, off by default.
- `ctlClient` (`internal/cli/ctl.go`), the console and `node add` must work
over the socket; `sudo llmgw` already runs as root.
### A6. Patroni REST and etcd are unauthenticated and plaintext (M)
`internal/pgha/config.go:99` writes a `restapi:` with no `authentication`
and `:54` etcd over `http://` with no client auth; `hba.go:56` PATCHes
Patroni's config unauthenticated. Anyone reaching :8008 or :2379 owns the
store.
- Generate a Patroni REST username/password at setup, store it in
`/etc/llmgw/secrets/`, write `restapi.authentication` and
`restapi.certfile/cafile` from the cluster CA; send it from
`internal/pgha/hba.go` and `status.go`.
- etcd: client and peer TLS from the cluster CA with `--client-cert-auth`.
- Document, and have `server init` write, nftables rules limiting
2379/2380/8008/5432 to member addresses (see A9).
- Done when: an unauthenticated `curl :8008/config` is refused and the
store status in the console still works.
### A7. PostgreSQL traffic is plaintext (S)
`internal/pgha/config.go:168` and `internal/cli/serverinit_postgres.go:330`
use `sslmode=disable`; HBA lines are `host`. Enable server TLS in Patroni's
postgresql parameters with a certificate from the cluster CA, write
`hostssl` lines, use `sslmode=verify-full&sslrootcert=/etc/llmgw/ca.pem`.
Existing clusters: `llmgw server` must keep working against a store that
has not been migrated, with a warning.
### A8. Smaller fixes, each S or less
- **Capacity fallback skips the allowlist.** `internal/router/router.go:700`
builds `[canonical]+entry.Fallbacks` without `permits`. Filter the chain
with `r.permits(p, model)`; price against the model that served.
- **Suspended users keep working API keys.** `store.(*DB).Resolve`
(`internal/store/memberships.go`) never joins `users.state`;
`Authenticator.Invalidate()` is never called. Reject keys whose owner is
not active; call `Invalidate` on revoke, suspend, limit change.
- **Join codes never expire server-side.** `internal/join/join.go:103`
checks expiry only in `Parse`; `admit` (`:212`) does not. Check it there;
bind the listener to the chosen address, not `:7444` on all interfaces;
use the peer's address rather than the claimed `hello.Addr`, and show the
joiner's address for the operator to confirm before secrets are sent.
- **Leftover sudoers grant.** `internal/onboard/gossh.go:248` writes
`NOPASSWD:ALL` and removes it only in `Close()` (`:271`). Write the grant
with a self-removal (`at`/systemd-run timer, 30 min) so a crash leaves
nothing; wipe password buffers after use. In `ssh.go:39` pass `--` before
the target.
- **Auth cache and pre-auth DoS.** `internal/auth/auth.go:127`: unbounded
map, one DB query per bogus well-formed key. Bounded LRU, negative cache
with short TTL, per-IP token bucket before the DB.
- **HTTP server hardening.** `server.go:351-360`, the h2c agent server,
both mirror servers: set `ReadHeaderTimeout` (exists), `IdleTimeout`,
`MaxHeaderBytes`, a body-read deadline, `netutil.LimitListener`; connect
`WithReadMaxBytes` on the agent stream.
- **Login/signup abuse.** Default `signup_open: false`; per-IP token buckets
on `/login` and `/signup`; bounded wait for argon2 slots; CSRF token on
the login form; the per-address throttle must not lock the real owner
out (use IP+address pairs).
- **Trusted proxies.** `internal/web/account.go:557`, `landing.go:60` honour
`X-Forwarded-Proto/Host` from anyone; add `router.trusted_proxies` and
only then trust them; `Cache-Control: no-store` on `/setup/*`.
- **Certificate revocation.** `DELETE /api/v1/nodes/{id}` only drops
inventory; add a revoked-CN/serial list in the store checked in
`agentsvc` `Connect` and mirror `onlyClients`; `serverinit_cluster`
always passes `--api-client`.
- **Engine exposure.** Engines bind `0.0.0.0` (`llamacpp.go:102`,
`vllm.go:109`): bind the management address. SGLang receives its key on
the command line (`sglang.go:102`): pass it by env or file.
- **Backups.** `internal/cli/backup.go`: encrypt archives to an `age`
recipient from `backup.recipient` in server.yaml; keep the directory 0700.
- **Python dependency confusion.** `internal/agent/provision/python.go:121`
uses `--extra-index-url` with `unsafe-best-match`. Pin torch to the
PyTorch index explicitly (uv `[[index]] explicit = true`), `first-index`,
hash-pinned lockfiles per engine version.
- **Control-plane SSRF.** `internal/sourcecheck/sourcecheck.go:39` and
`gguf.FromURL` use the default client; reuse the mirror's
`fetchClient(false)` (private-address filter) and refuse a proxy unless
configured; block NAT64/6to4-embedded private addresses in
`internal/mirror/safefetch.go`.
- **Low:** `agentsvc/service.go:430` indexes `n.GPUs[0]` without a length
check (CPU-only node panics the handler); thread ids are client-chosen
with no length cap and shared per project (`router/threads.go`) — cap
length, document, offer per-user scoping; `/v1/models` lists models the
key may not use (`handlers.go:799`); sessions have no idle timeout and
`PurgeSessions`/`PurgeClaims` are never called — add a janitor; add a CSP
header and set `htmx.config.allowEval=false`; cap GGUF array nesting
depth in `internal/gguf/gguf.go`; mirror `.source` sidecars are served by
the public file server — exclude dotfiles from `:8081`; prune
`modelserve.jobs` and the mirror `running` map; KaTeX → 0.16.21+.
### A9. Firewall (S)
No machine in a default install has a firewall. Have `server init`, `node
add` and `server add` write an nftables ruleset: ssh open; `:8080` open;
`:7443` and `:9091` open to the cluster's networks; `:9100` to the
monitoring host; `:2379/2380/8008/5432` to store members and control
planes; mirror `:8081` to the cluster, `:8082` to control planes; `:9090`
Prometheus and `:3000` Grafana to operators. `llmgw doctor` reports a
missing ruleset. Make it skippable with `--no-firewall`.
---
## B. Guardrails (M–L, build in this order)
**Status (2026-10-04, branch `guardrails-b`):** B1–B4 done, and tried live on
the PoC. That found four bugs, all fixed with tests:
- a shadow default merged with an attached policy of unwritten mode left it
in shadow;
- a streamed word held back at one release was dropped at the next;
- llama.cpp continues a conversation ending on the answer, so Llama Guard's
verdict on answers is now asked with its own prompt as a completion;
- a judge model was named in a client's "no endpoint" error.
The PoC runs `observe` (shadow) for everyone, and Llama Guard 3 1B
(`llama-guard-3-1b`, CPU, on cuda01) as a judge, which blocks a harmful
request in ~0.7s and a harmful answer in ~1.4s there. Still left: see each
item.
Today nothing on the request path reads, classifies, redacts or filters
message text in either direction (confirmed; the only "moderation" hit is a
comment in `internal/router/budget_fallback.go:20`). Existing controls are
about who, how much and how fast: allowlists (`router/handlers.go:450`),
RPM/TPM/budget (`internal/accounting`), context precheck
(`router/context_limit.go`), admission bands (`router/admission`),
timeouts (`router/proxy/proxy.go:26-57`). Gaps to close on the way: no
default output cap when the client omits `max_tokens`; `catalog.Capabilities`
is advertised but never enforced; tool results are trusted input.
### B1. Hooks in the proxy and router (S)
**Done (branch `guardrails-b`):** `internal/guard` holds the interface;
proxy `Filter`/`StreamFilter`; the input hook runs after `loadThread` (the
simpler choice); error codes, headers, Responses `incomplete`, usage columns
`guard_*`. Moved to B2: holding `delta.tool_calls` until they are whole and
reading their arguments — that is the tool policy's filter, not the proxy's.
- `internal/router/proxy/proxy.go`: add optional `InputRewrite func(body
[]byte) ([]byte, Verdict)` on the request and two output hooks: `Filter`
for `relayBuffered` (between `ReadAll` and `WriteHeader`) and
`StreamFilter{OnEvent(data) (emit []string, stop bool); OnEnd()}` for
`relayStream`, called where each whole SSE event is assembled
(`eventBuffer`, before `io.WriteString` at `:343`). Nil hooks cost
nothing.
- `internal/router/handlers.go`: call the input policy after the allowlist
and limits and before admission (around `:395`); the principal, scopes
and the model that will serve are known there. Thread history is spliced
later (`:416`): scan stored turns when written, or move the hook after
`loadThread` (decide; the second is simpler).
- Blocking conventions: input → 400 `invalid_request_error` with
`code: "content_policy_violation"` (today `writeError` always sends
`code: null`); output non-stream → 200 with `finish_reason:
"content_filter"` and empty/redacted content; output stream → a final
chunk `{"choices":[{"index":0,"delta":{},"finish_reason":"content_filter"}]}`,
then `data: [DONE]`, then cancel upstream. Headers `x-llmgw-guard:
blocked|redacted|flagged` and `x-llmgw-guard-policy` set before the proxy
writes headers.
- `internal/router/responses.go`: map `content_filter` to
`response.incomplete` with `incomplete_details.reason: "content_filter"`
(today it always emits `response.completed`, `:525`).
- Tool calls: hold `delta.tool_calls` until `finish_reason: "tool_calls"`,
then validate; `replyDelta` (`reply.go:33`) must also read
`tool_calls[].function.arguments`.
- Accounting: a blocked input is recorded like `RecordRefused` (nothing
charged); a blocked output is charged (the GPU was used). `store.Usage`
gains `GuardAction`, `GuardPolicy`, `GuardCategory`, `GuardMs`.
### B2. Package `internal/guard` (M)
**Mostly done (branch `guardrails-b`):** policies in server.yaml with
attachments in the store (`ctl guard attach`), narrowed by the limits' rule;
`guard/detect` (validated PII, secrets, injection through normalisation and
decoding); redaction with consistent placeholders before prefix routing;
tool allow/deny/strip, call count and argument schemas; capabilities and
structured output (opt-in); default `max_tokens`; pinned system prompt and
dropping the client's; `no_store` (threads, notes, prefix tree, external
models); shadow mode. **Left:** a `guard_events` table — the usage row's
`guard_*` columns carry action/policy/category/time, not rule and stage;
language identification; the re-identification map that restores
redacted values in answers.
- `Policy`, `Effective` resolution over the tenancy tree with the same
narrowing rule as limits (`internal/store/sqlite.go:383-416` `tighten`):
deny sets union, allow sets intersect, `mode` takes the strictest of
`enforce > shadow > off`, `fail` takes `closed > open`, `no_store` at any
level wins, thresholds take the minimum.
- Store: `guard_policies(scope, scope_id, policy json)` and
`guard_events(request_id, scopes, policy, rule, category, action, stage,
score, judge_ms, shadow)` — ids, categories, offsets and lengths only,
never text. Migrations for sqlite and postgres. Pruned with
`usage_retention`.
- `guard/detect`: regex/keyword sets; PII with validators — email, phone,
IBAN (mod-97), card (Luhn), TCKN (checksum), IP, secrets (`sk-`, AWS,
JWT, PEM); normaliser (NFKC, strip zero-width/bidi, homoglyph fold,
base64/hex/URL decode to depth 2); injection heuristics (role-switch
phrases, fake chat-template tokens, long base64, invisible Unicode) —
default action `flag`; language id.
- Redaction rewrites `messages[].content` in the upstream body with
placeholders (`[EMAIL_1]`) via the same `map[string]json.RawMessage`
round-trip `withModel`/`spliceThread` use; an optional re-identification
map lives only in request memory and restores originals in the output.
Redact before prefix routing so `PromptPrefix` hashes the redacted text.
- `guard/toolpolicy`: tool name allow/deny (strip or 400), JSON-Schema
validation of assembled arguments against the declared `parameters`,
argument regex rules, `max_calls_per_turn`. Enforce
`Capabilities.Tools`/`Vision` here: a `tools` array or image parts to a
model without the capability → 400.
- `guard/structured`: validate `response_format.json_schema` on the final
text → `content_filter`, or flag in shadow.
- Output constraints: inject a default `max_tokens` when absent (from the
policy or `MaxOutput`), language check, structured check.
- System-prompt pinning per project: prepend before `checkContextFits` and
before `spliceThread`; optionally drop client `system`/`developer`
roles; exclude the pin from stored thread turns; apply to Responses
`instructions` in `toChat`.
- `no_store`: `loadThread` and `PutNote` refuse; `captureBytes()` returns 0;
skip the prefix-tree observe (`router.go:840`); disable External
fallbacks for the scope (filter in `dispatch` and `budgetFallback`).
### B3. Judge model on the cluster (M)
**Done (branch `guardrails-b`):** `role: guard` + `guard.format`; hidden and
refused to clients and fallbacks; critical by default, never on demand;
`Router.Judge` past admission/accounting/guardrails; llamaguard3,
shieldgemma and json parsers; 5-minute cache; latency budget then
fail open/closed; streamed answers read every 600 characters, or held with
buffer_all. Simpler than written: the input judge runs after admission, not
alongside it, and shadow mode judges synchronously within the budget rather
than after the response. Not enforced: two replicas on different nodes —
documented, left to `scaling.min`.
- `catalog.Model.Role: "guard"` plus `Guard{Format: llamaguard3 |
shieldgemma | json, Categories}`. Guard models are hidden from
`/v1/models`, the chat picker and allowlists; placed like any model,
class `critical`, `scaling.min ≥ 1`, at least 2 replicas on different
nodes.
- `guard/judge`: calls the guard model through the router's own `dispatch`
with an internal principal and its own admission band (never the
tenant's queue); parsers per format (Llama Guard `safe`/`unsafe\nS1,S10`;
ShieldGemma yes/no probability → per-tenant thresholds; generic JSON
verdict); verdict cache keyed by hash of normalised new turns (5 min);
`latency_budget` (default 250 ms) then `fail` open/closed; input judge
runs in parallel with admission queueing.
- Streaming strategy: default sentence-buffered scanning (regex/PII per
sentence or ~200 chars before release; judge every 400–800 chars on the
cumulative tail; final judge at end); `buffer_all` for `fail: closed`
scopes. Document that released text cannot be retracted.
- Shadow mode runs everything asynchronously after the response
(`context.WithoutCancel`, like `saveThread`) and only records.
### B4. Config, metrics, UI (S)
**Done (branch `guardrails-b`):** library in server.yaml, attachments via
`ctl guard`, the console's **Guardrails…** with "applies from" marks, and
the portal's `/admin/guard`; the metrics and alerts below (judge latency
labelled by model only at the judge; the check latency by stage); `ctl
request` shows the verdict; the chat page says "stopped by policy"; the
profile lists effective policies; docs/guardrails.md and data.md written;
the access log carries action, stage and policy only. Attaching from the
portal is not done — the page shows; the console and ctl change.
- Gateway-level policy library in `server.yaml` under `guard:` (see the
YAML in the review, reproduced in `docs/guardrails.md` to be written);
attachments to scopes in the store via `llmgw keys guard`, the console
("Guardrails…" beside "Limits…" in `internal/cli/tui_people.go`, with
"inherited from team:X" badges) and the portal.
- Metrics in `router/metrics.go` style:
`llmgw_guard_checks_total{stage,policy,action,category,mode}`,
`llmgw_guard_latency_seconds{stage,judge_model}`,
`llmgw_guard_judge_errors_total{reason}`, `llmgw_guard_fail_open_total`,
`llmgw_guard_stream_stops_total`, `llmgw_guard_redactions_total{kind}`,
`llmgw_tenant_guard_blocks_total{tenant}`. Alerts: fail-open > 1% for 5m,
judge p95 over budget, judge has no ready replicas, per-tenant block rate
> 3× its 7-day baseline, any `secret` detected in output.
- Portal `/admin/guard`: violations by tenant/category/policy over time,
shadow vs enforce, judge health, recent events by request id; `ctl
request <id>` shows the verdict; chat page renders `content_filter` as
"stopped by policy X"; `/me` lists effective policies.
- Update `docs/data.md`: the judge sees all text but in-cluster; External
fallbacks send prompts outside the cluster (the current "nothing leaves
the cluster" is wrong); `no_store` semantics.
- Never log: prompt/completion text, PII values, the re-identification
map, judge inputs or rationale that quotes input. Check `accessLog`'s
`err` field and span errors.
---
## C. Features, ranked by value × feasibility
1. **Done (branch `features-c`):** `audit_events`, written by the admin
API middleware and the /admin pages (with each change's result), `ctl
audit`, the console's Audit view; staff roles viewer / operator /
billing / admin. Left: before/after values (the request's fields are
recorded, not the state they replaced); a tenant-admin role that changes
its own organization (managers still only view it); hiding the forms a
viewer cannot submit. **Audit log and roles (M).** Today "audit" is slog lines
(`internal/web/admin_users.go:403-480`) and roles are the `IsAdmin` flag
plus an org "manager" who can view. Add `audit_events` (who, what,
before/after, source) written by the `adminAction`/`tenancyAction`
wrappers and by every admin-API mutation; roles viewer / operator /
tenant-admin / billing; `ctl audit`. Depends on A5 for a real actor.
2. **Done (branch `features-c`):** OIDC sign-in (`auth.oidc`, PKCE,
verified email, roles and memberships from groups, `no_passwords`) and
SCIM 2.0 (`/scim/v2` Users and Groups). The sign-in cookie's key comes
from the store (2026-10-10), so a callback may reach either control
plane behind a balancer. **OIDC/SSO login (M), then SCIM (M).** `internal/account/service.go`,
`internal/web/account.go`; map IdP groups to org/team/admin.
3. **Done (branch `features-c`):** key lifetime limits (gateway, org, team,
project), `keys rotate --overlap`, kinds and expiry in `keys list`.
**Service accounts and key lifecycle (S).** Per-org maximum key TTL,
`keys rotate` with overlap, last-used shown, project-owned service keys
listed apart. `internal/cli/keys.go`, `internal/store/keys.go`,
`internal/auth/auth.go`.
4. **Done (branch `features-c`):** CSV export from the rollups (API, ctl,
console). Left: Parquet, and a scheduled push to S3/SFTP. **Usage export (S).** CSV/Parquet by org/team/project/model/period from
the rollups (`internal/store/usageseries.go`), `GET
/api/v1/usage/export`, optional scheduled push to S3/SFTP.
5. **Done (branch `features-c`):** `notify.webhooks` (model parked, node
lost, budget spent, update available; signed JSON or Slack; retries,
hourly de-duplication), `ctl notify test`, Alertmanager receiver
templates. Left: `server init` and the console do not yet ask for a
receiver — the templates are filled in by hand. Events are claimed in
the store before they are sent (2026-10-10), so two control planes send
each once.
**Alerting out of the box (S).** `deploy/monitoring/alertmanager.yml`
ships `receiver: none`; add Slack/email/webhook/PagerDuty receiver
templates driven by `server init` or the console; llmgw-native webhooks
for model parked, budget spent, update available (from the reconciler
and `modelserve`).
6. **Done (branch `features-c2`):** `metadata`, `x-llmgw-tags` and `user`
on the usage record (migration 0023), `ctl request` shows them,
`router.tag_labels` metrics with 100 values per tag. Left: export and
`ctl usage` by tag or end user (the hourly and daily sums do not carry
tags; it would read the request records, within their retention).
**Request metadata and tags (S).** Accept `metadata`/`user` from the
body or `x-llmgw-tags`; store on the usage row; tenant metric labels
behind an allowlist. `router/handlers.go` parse, `accounting/recorder.go`.
7. **Done (branch `features-c2`):** `ctl model canary/promote/canaries`,
session-sticky split with `x-llmgw-variant` pins, rollback on error rate
or TTFT, `canary_rolled_back` notification, console menu. Left: A/B
between two different models (the split is between a model and its
newer weights); the comparison is per control plane and lives in memory,
so a restart begins it again; a key-level pin.
**Canary and weighted A/B between model revisions (M).**
`control/modelserve/updates.go` `replace` swaps every replica; add
weighted routing across two revisions/aliases (scheduler filter, alias
resolution in `router/router.go:569`), a header/key pin, automatic
rollback on error-rate or TTFT regression from per-replica metrics.
8. **Done (branch `features-c2`):** `/v1/files` and `/v1/batches`
(OpenAI's shape), worker under a store lease, lines kept as served,
lowest band, usage tagged `batch=<id>`, output and error files with
retention, `no_store` refused. Left: the Responses API (`/v1/responses`)
as a batch endpoint; the console and `ctl` do not list batches (an
operator sees them through usage tags); a file is one row in the store,
so a store backup carries batch files with it.
**Batch API and Files (M–L).** `/v1/files`, `/v1/batches`, JSONL
storage, a worker feeding the bronze band, accounting. New
`internal/router/batch`.
9. **Done (branch `features-c2`):** `keys requests` (retention, sample),
kept after redaction, `ctl request --body/--replay`, console, pruning.
Left: a streamed answer is kept as its text, not its chunks; replay is
one request at a time (a set of them against a model is item 21's eval).
**Request/response logging and replay (M).** Opt-in per project with
retention like threads; sampled or full; redaction hooks from B;
`ctl request <id> --body`; replay against another model.
10. **Done (branch `features-c2`):** Responses `text.format`,
`input_image`, `previous_response_id` (thread in the id); rerank and
score passthrough; transcription and translation as multipart;
`/v1/moderations` by the judge. Left: audio is charged no tokens (an
engine's duration usage is not read); `input_image` by `file_id`;
moderation scores are 0 or 1, as a judge says yes or no.
**OpenAI compatibility gaps (S–M each).** Responses: `text.format` →
`response_format`, `input_image`, `previous_response_id` via threads
(`router/responses.go`). Rerank/score passthrough for vLLM,
`/v1/audio/transcriptions` for a whisper engine, `/v1/moderations`
mapped to the guard model.
11. **Done (branch `features-c2`):** `/v1/messages` and `count_tokens`,
streamed and not, tools, images, `x-api-key`, Anthropic error shape.
Left: extended thinking (a reasoning model's thoughts are not mapped to
thinking blocks); prompt caching; the account page's "Connect your
editor" does not offer Claude Code yet.
**Anthropic `/v1/messages` (M).** Translate like `responses.go`.
12. **Done (branch `features-c2`):** `server restore` (dry run, version
check, decryption, modes and owners, SQLite and PostgreSQL),
`backup_copy_to` (directory or ssh), the drill in docs/install.md.
Left: restoring a replicated store's members (restore into the primary;
Patroni replicates it); the console's Back up now does not offer
restore; an ssh copy holds the archive in memory while it sends it.
**`server restore` (S).** Restore is manual shell today
(`docs/install.md:1187`); add the command with version check, dry run,
off-host copy target; a documented DR drill.
13. **Done (branch `features-c2`):** TPOT, inter-token and tokens/s
histograms by model, pool and tenant; `slo` pressure in admission's
saturation; `llmgw_model_slo_seconds` and `llmgw-slo` alerts. Left: the
p99 is per control plane, from its own requests; an SLO per band or per
tenant; placement does not yet add replicas for an SLO (item 14).
**SLO-based admission and token-level metrics (S–M).** TPOT and
inter-token histograms, output tokens/s per model and tenant in
`router/metrics.go`; use `catalog.SLO` in `admission/saturation.go`.
14. **Done (branch `features-c2`):** `scaling.schedule` (days, from, to,
min, max; past midnight), `scaling.prewarm`, `control.timezone`;
queue-depth scaling was there already. Left: `ctl scale` sets min and
max only (a schedule is set in the definition); scaling for an SLO
rather than for queue depth.
**Scheduled and warm-pool autoscaling (S–M).** Time-of-day min/max,
pre-warm, queue-depth scaling in `catalog.Scaling`,
`control/reconciler/demand.go`, `headroom.go`.
15. **Done (branch `features-c2`):** `env:` and `vault:` references
wherever a secret file is named (store address, api_key_file, OIDC,
SCIM, webhooks), Vault KV with token or AppRole, `secrets.master_key`
sealing cluster_secrets (Vault transit or a local key). Left: the
certificate authority's key is still a file in /etc/llmgw (signing in
Vault's PKI engine instead is its own change); rotating the master key;
`backup_copy_key` and `admin_token_file` are files only.
**Secrets management (S–M).** Vault/KMS for `api_key_file`, the DSN, the
CA key, `cluster_secrets` (`store/secrets.go`, `store/seal.go`);
envelope-encrypt them in backups (with A8 backups).
16. **Done (branch `features-c2`):** `users forget` (pseudonymised usage
and sums, kept requests, files, batches, limits and audit names), `keys
residency --in-cluster` (asked-for, fallback and canary). Left: an
export of everything held about a person (article 15); residency by
region between several clusters; backups keep what was forgotten until
they age out.
**GDPR and residency (S).** "Forget user" anonymises `user_id` in usage
and rollups and erases threads/memory (`store/hierarchy.go:241` leaves
it today); per-project "never route to an External provider".
17. **Done (branch `features-c2`):** exact-match answer cache per project (`keys
cache --ttl/--any/--off`, console **Cache answers…**, `projects`
columns, 30-second setting cache), reproducible answers only unless
`--any`, LRU in memory bounded by `router.cache.max_bytes` and
`max_entry_bytes`, streamed answers kept and replayed as chunks,
`x-llmgw-cache` and `Cache-Control: no-cache`, guardrails before a hit
and the output policy on it, `no_store` and threads bypass, a hit
recorded as `cache_hit` with no tokens and no cost (migration 0027),
metrics, docs. Left: semantic (embedding-similarity) caching through the
embeddings models; a cache shared across control planes (each process
keeps its own, deliberately: a shared one would write every answer to
the store); a hit still waits its turn in admission, because the
guardrails judge the request after admission and the key is the request
they let through; logprobs in a streamed answer leave it unkept; `users
forget` does not reach into the cache (answers age out by the TTL); no
listing or flushing of what is cached (`--off` stops it being served at
once, the memory is freed by TTL and LRU).
**Response/semantic cache (M).** Exact-match first, per project,
opt-in, TTL; semantic via the embeddings models later. Before
`dispatch` in `router/handlers.go`.
18. **Done (branch `features-c2`):** offline bundles — `llmgw bundle create`
(releases with signed checksums, every llama.cpp build per arch/variant
with GitHub's checksums, starter/judge and named Hugging Face model
files pinned and checked; directory or tar; manifest sha256), `bundle
import` (checked, all-or-nothing, into the mirror under `_llmgw/`),
`bundle trust`, `bundle show --verify`;
`offline.mirror`/`offline.checksums` in server.yaml, with nodes
installing llama.cpp from the mirror against checksums the control plane
sends over the agent stream (proto `EngineWant.offline_*`), releases
defaulting to the mirror for `node add`, `ctl upgrade` and agent
upgrades, update checks off; `server init --bundle` and the setup form's
bundle fields; docs/data.md's offline claim corrected; install.md
"Offline sites". A Helm chart for the control plane
(`deploy/helm/llmgw`, one replica, SQLite PVC or external PostgreSQL,
Secrets, Services, Ingress, ServiceMonitor) with
`deploy/container/Dockerfile` and docs/kubernetes.md; GPU nodes stay
outside Kubernetes by design. Left: vLLM/SGLang offline (a wheelhouse of
several GB per CUDA build and a uv `--find-links` install path); the
NVML release binary and uv in bundles; the setup wizard's preflight
still fails a machine without internet; `join`'s printed `curl` from the
mirror checks nothing; models in a bundle are single files (no whole-
repository safetensors sets yet); the chart runs one replica (a standby
is never ready, so HA in Kubernetes needs a different readiness design),
has not been installed on a real cluster, and ships no agent DaemonSet;
no published container image.
**Kubernetes/Helm and an air-gapped bundle (M).** `deploy/`,
`agent/provision`; `docs/data.md` claims offline operation but install
assumes the internet.
19. **Done (branch `features-c2`):** `adminapi.Routes` and
`internal/docgen/openapi` generating `docs/admin-api.openapi.json` (3.1,
bearer security, every route, query parameters, schemas from the Go
types with their comments), served embedded at `GET
/api/v1/openapi.json`, kept true by a route-table test and a staleness
test; validated with openapi-spec-validator and openapi-generator-cli.
`GET /api/v1/models/{name}`. `docs/admin-api.md`. `terraform-provider-
llmgw/` (terraform-plugin-framework): `llmgw_model`, `llmgw_quota`,
`llmgw_guard_attachment`, data sources `llmgw_models` and `llmgw_nodes`;
client tests against a fake admin API and the OpenAPI document;
acceptance tests with Terraform and OpenTofu against the fake; one
apply, empty plan, in-place change and destroy against a real `llmgw
server` run by hand. Organizations, teams, projects, keys and
memberships are in the admin API since 2026-10-08 (`/api/v1/orgs`,
`teams`, `projects`, `keys`, `users/{id}/memberships`) and in the
provider (`llmgw_org`, `llmgw_team`, `llmgw_project`, `llmgw_key`,
`llmgw_membership`); accounts are still made by sign-up, invitation or
OIDC/SCIM. Left: a model's definition beyond
scaling, class, owner and adapters is changed by replacing it (the API
has no edit, and `resources` is not used yet); `canary`, `slo`,
`scaling.schedule` and `prewarm`, `guard` role and `origin` are not in
`llmgw_model`; models cannot be imported; the provider is not published
to a registry and has no release build; the acceptance tests are not run
in CI (they need Terraform); the document's schemas mark nothing
required and give no enums; `catalog.SLO`'s fields have no JSON tags, so
they are `TTFTp99` and `TPOTp99` on the wire.
**OpenAPI for the admin API, Terraform provider (M).** Generate from
`adminapi/api.go` routes via `docgen`.
20. **Done (branch `features-c2`):** spans for the reconcile pass (begun only
when a pass does something), placement with its answer or rejections,
scale-up/down, drain, stop, eviction, preemption, reaping, restarts and
lost nodes; a `replica start` span from placement to serving whose
context travels to the agent in the start command
(`ControlMessage.trace`), where weights fetch, engine launch, rendezvous
and readiness wait continue the trace; stops, drains, adoption and
engine installs; modelserve jobs as a trace with a span per step linked
to the replica's start; an agent `tracing` block; docs in install.md's
"Tracing". Left: the router's span for a request does not link to the
replica it was sent to's start; heartbeats, fabric scans, logs and agent
upgrades are not traced; the supervisor has no restart span because it
never restarts an engine itself (a restart is the reconciler's `restart
replica` and reaches a node as a stop and a start); canary and promote
jobs, which landed on `features-c2` after this was branched, are traced
only through the shared step recording (as `<kind> model`), unchecked by
a test.
**Tracing beyond the router (S).** Spans for reconciler, placement,
agent start (`obs/trace.go`, `control/reconciler`, `agent/supervisor`).
21. **Done (branch `features-c2`):** `internal/bench/eval` (suites in YAML or
JSON lines; exact, contains, regex, number, a JSON Schema subset, and a
judge model with a rubric; weights and a threshold), run on the control
plane through `Router.Replay`; `llmgw ctl eval MODEL --suite FILE|NAME`,
`--history`, `--run`, `--suites`; runs and their cases kept (migration
0028, `eval_runs` and `eval_cases`); `model update`/`promote --eval
SUITE --min-score N`, a model's standing gate (`ctl model gate`, `eval:`
in the catalog) — a promote asks the canary first, an update tries the
new weights as a 0% canary and promotes or removes it; the console's
model menu; `arithmetic` and `json-format` in `deploy/eval/`, carried in
the binary. Left: the suite of a model's standing gate is read by name
on the control plane at the time of the update, so two control planes
need the same `control.eval_dir`; a gated update needs room for one more
replica while it runs (there is no in-place gate, and a declared model
in server.yaml cannot be updated at all); no comparison against the old
weights' score (the gate is an absolute threshold, not "no worse than
before"); a run is not stopped once started (Esc in the console only
stops watching it), and the in-memory progress of a run lives on the
control plane that started it; runs are never pruned; the JSON Schema
support is a subset (no `$ref`, `oneOf`, `not`, `uniqueItems`); judge
cases go through the same concurrency as the model's, and a judge's
answers are not kept, only its score and reason; a gateway without a
control plane cannot run suites (there is no mode that asks through the
client API with a key).
**Eval harness (M).** `llmgw eval` extending `internal/bench` with
quality suites gating `model update`.
22. **Done (branch `features-c2`):** MIG instances discovered on both paths
(nvidia-smi `-L` + `-q -x`; NVML through an interface with a faithful
fake) and reported instead of their card, each with its MIG UUID, own
memory and class `<card class>-mig-<profile>`; device numbers 1000 + 100
× card + instance, shown as `card/migN`; placement on MIG classes;
tp/pp/gpus/nodes above 1 on a MIG class refused in catalog validation
and again in placement; `CUDA_VISIBLE_DEVICES=MIG-<uuid>` for vLLM and
llama.cpp (no Vulkan device variable on MIG); per-instance memory and
processes in heartbeats and the GPU check, card sensors under each
instance, utilisation absent rather than zero; doctor MIG findings (`mig
-lgi`/`-lci` parsed); `ctl nodes`, console, `ctl status`, `ctl model
plan`/`serve` by class; docs/install.md MIG section. Written and tested
against fakes only — untested on real MIG hardware. Left: run it on an
A100/H100 in MIG mode (check `-L`/`-q -x`/`mig -lgi`/`-lci` and NVML MIG
device names against the parsers, and that `CUDA_VISIBLE_DEVICES=MIG-…`
reaches vLLM's workers through Ray); instances changed while the agent
runs are only seen when it restarts (no rescan); llmgw does not create
or reshape instances (no mig-parted integration); quotas count an
instance as one GPU, the same as a whole card; SGLang on MIG not
checked; device numbers assume fewer than 100 instances per card and
that instance numbers stay put while shards run on them.
**MIG (M–L).** Treat a MIG slice as a GPU class in
`agent/discovery/gpu_nvml.go` and `control/placement`.
23. **Done (branch `features-c2`):** engine `trtllm`
(`internal/agent/engines/trtllm.go`): `trtllm-serve` through a launcher
of llmgw's that demands the replica's key (trtllm-serve has none) and
writes the options file that turns on `/prometheus/metrics` and the
iteration statistics; the scrape contract for it
(`scrape.TRTLLMMapping`, with a per-engine metrics path); found, not
installed — `engines.trtllm.python` in `agent.yaml`, probed from the
package's metadata; catalog validation (GPU class, one node, no GGUF
source, the flags a profile may not give); `ctl engine-check trtllm`,
which asks `trtllm-serve serve --help` through its module. Ollama/GGUF
import: `mirror import-ollama` (manifest parsing, blob digest check,
projector beside the model, all or nothing, a model definition printed),
`mirror import` for any local file, and the console's **I** on the
Mirror view. Left: TensorRT-LLM has not run on real hardware — the first
node with one should run `engine-check` and serve a model before anyone
relies on it; no installer for it; no multi-node TensorRT-LLM replica
(it needs an MPI or Slurm launch from one node); Triton Inference Server
as an engine is not done; trtllm's queue gauges are set per iteration,
so an idle engine shows its last batch until the next request;
`--num_serve_frontends` is refused because extra front ends would not
demand the key. The Ollama import runs on the mirror machine only — the
mirror's API takes no files, so an Ollama elsewhere means copying the
blob or running the import there; Ollama models kept as separate tensors
or as several weight layers, and its LoRA adapters, are refused or left
with a sentence; files an import run with `sudo` writes are root's, as
`mirror add`'s are.
**More engines (M each).** TensorRT-LLM/Triton adapter; Ollama/GGUF
import. `agent/engines/engine.go` interface and the metric contract in
`router/scrape`.
24. **Done (branch `features-c2`), all six parts.** Lower: precise KV-index routing (plan phase 6), LoRA affinity for
vLLM, multi-cluster federation via External providers, facility-aware
placement (`docs/facility-aware-placement.md`), portal i18n, SDK/docs
portal.
What was built, and what is left:
- 24 (portal i18n): built — English/Turkish catalogue with fallback, per-
language templates, cookie/Accept-Language/`landing.language` choice
and a footer switcher, Turkish numbers and dates, every template and
handler message translated, tests that no template text stays English;
left — errors built with fmt.Errorf in other packages (store, catalog,
mirror, adminapi) still English on Turkish pages, percent sign
placement is mixed (%50 on limits, 93% on cluster pages), RFC3339
footer timestamps and plain-text http.Error bodies untranslated, the
/docs documents themselves English, a per-person language column
(cookie only today).
- 24 (SDK/docs portal): built — /docs renders the embedded manual (goldmark,
GFM, heading anchors, links rewritten, no script; public,
`landing.docs_sign_in` to require a session), and docs/clients.md
shows the official OpenAI/Anthropic SDKs against llmgw. Left: the
console's generated "Commands" page is not in the portal (it is built
from the cobra tree, which internal/web cannot see); no search across
documents.
- [~] 24. Precise KV index (tier 2) — built: vLLM KV events via the agent
(loopback ZMQ, pure-Go zmq4, msgpack in both of vLLM's encodings)
forwarded as net block-hash changes over the agent stream; router
index, tokenize via the replica's `/tokenize`, sha256_cbor block-hash
chain checked against cbor2/hashlib golden values; scorer by leading-
block share with tier 1 fallback; opt-in `kv_events` per profile;
metrics; docs. Left: validation against a real vLLM (event shape, hash
agreement, `/tokenize` matching the served prompt, cost of
tokenizing); vLLM's replay endpoint instead of resetting on a gap;
data-parallel ranks (one publisher each, ports offset); hybrid models'
cache groups and CPU-offloaded blocks; LoRA/multimodal/cache_salt
requests (extra keys); a router-only deployment (no agents)
subscribing directly; standby control planes have no index until they
lead; a per-replica fallback when only some candidates have an index;
caching tokenizations of repeated prompts.
- **24 LoRA affinity (branch `features-c2`):** built `max_loaded_adapters`
(catalog), vLLM adapter support at start and on demand
(`engines/lora.go`, `vllm.go`), the agent linking fetched adapters as
`llmgw-lora/<name>` in the engines' working directory, the router's
adapter tracker (`router/lora.go`: loads/unloads with the engine key,
LRU eviction never of an adapter in use, one load per adapter per
replica, `/v1/models` sync, 404 reload) and the `adapter-affinity`
scheduler filter in the chat, batch and long-context profiles, with
metrics and docs; tested against a fake of vLLM's HTTP interface.
**Left:** never run against a real vLLM — in particular that vLLM
resolves the relative `llmgw-lora/<name>` path against its working
directory on every node (multi-node too) and that unloading an adapter
mid-request is harmless; the agent still fetches every adapter at
replica start, so a new adapter needs a restart (fetching on first
load would need an agent command and a proto change); load timeout and
resync interval are constants, not config; SGLang's runtime LoRA API
is not driven; no per-replica "adapters held" view in the console or
admin API; the debug profile has no affinity.
- 24. Multi-cluster federation — built: peers via `ctl federation` kept in
the store, their models as PEER/MODEL synced from /v1/models
(withdrawn-while-referenced), /ready health with latency and routing
withdrawal, `ctl model fallbacks` for overflow, hop header loop
refusal, residency/no_store hold, optional attribution tags, console
view, metrics, docs. Left: server.yaml-declared models cannot name a
peer model as a fallback (config catalog is validated before peers are
read); per-peer CA / mTLS for peers behind a private CA; transitive
federation (A→B→C models) deliberately not offered; latency is /ready
RTT only (no per-request peer TTFT used in routing or weighting); the
peer's adapters are not imported (listed as aliases); no console form
to add/remove peers (CLI only); key file must exist on every control
plane.
- 24 (facility) — built: location labels (`ctl label`, agent.yaml),
spreading across racks/PDUs, `keep_within`, held domains with drain
(`ctl facility hold/release`, kept in the store), power caps from
`facility.budgets` against declared/derived/measured draw, shown in
explain-placement, `ctl nodes`, `ctl facility`, the console and the
API. Left: temperature conditions with hysteresis and their
enforcement (steps 3–4), a Prometheus/DCIM source, power headroom as a
rack-levelling score, spreading multi-node replicas across racks,
`llmgw_domain_*` metrics and alerts (near cap, over cap, stale data),
shedding and tokens per watt (step 5), and a drained node's state
surviving a restart.
---
## E. Every setting takes effect when it is changed (2026-10-05)
Asked for after a site visit: an upgrade left server.yaml without guardrails,
a node on another subnet stayed lost until firewall_networks was edited by
hand, and nothing in server.yaml, agent.yaml or a generated unit applies until
somebody restarts the right service on every machine. The rule from here on:
**a file holds only what one machine needs to start; everything else is the
cluster's, kept in the store, changed through ctl, the console or the web,
audited, and in force on every control plane within seconds.**
### E1. Cluster settings in the store (L) — done 2026-10-05
Live on the PoC (140, 150): imported as version 1; a guard change applied on
both without a restart; a router change restarted the standby at once and the
leader 30s later with /health answering throughout; rollback did the same.
- Table `cluster_settings` (versioned YAML document, who and why). The
cluster-wide part of server.yaml lives there; server.yaml keeps the
per-machine keys only (listeners, TLS paths, store DSN, secrets, tracing,
monitoring, mirror.api/tls, backup_dir/key, endpoints).
- First start against a store with no document imports the file's cluster
keys (and the baseline guard policy when there is none); after that those
keys in a file are ignored, with a warning.
- Each control plane polls the document's version and its own server.yaml's
hash. A change is validated, then applied: `guard` live, the rest by a
graceful restart of the process (standbys first, the leader last), which
systemd brings back. No "restart now?" anywhere.
- `llmgw ctl config show|history|get|set|unset|edit|rollback`; the console's
settings screen edits the store for cluster keys; admin API
`GET/PUT /api/v1/settings`, `/history`, `/rollback`.
- A standby that becomes leader re-reads the catalog, engine choices and
facility state from the store (it kept what it read at start).
- Retires `appendGuardIfMissing` (server init): the import does it.
### E2. Generated files from settings (L) — firewall on/off done 2026-10-06, every machine 2026-10-09
`firewall.enabled` in the cluster's settings reaches nodes over the agent
stream and control planes through llmgw-firewall-sync.path. Since
2026-10-09 the mirror machine too (the leader PUTs /v1/firewall, the mirror
writes `.firewall.json`, the same path unit applies it) and store members
that run no llmgw (the leader keeps it in `cluster_state`, a root timer
reads the member's own copy with psql); `firewall_networks` reach all of
them, and the mirror lets in the network of every node and control plane
the leader knows. Tried on the PoC: off and on, and a network added, on
control planes, mirror, store member and nodes. Left: the allowed set
replaced by membership rather than added to (a network taken out of the
settings stays let in), the other files (the mirror's unit flags,
keepalived, etcd and Patroni configuration).
Every machine runs the agent (root) with roles — node, control plane, mirror,
store member — and the agent renders and applies what was written once by
init: the llmgw nft table, the mirror's unit flags, keepalived, etcd and
Patroni configuration. The firewall's allowed set is the cluster's membership
(every node, control plane, mirror and store member) plus
`firewall.extra` from the store; 7443 stays open (mTLS). A node added or
removed changes every machine's rules. Retires `firewall.Admit` and
`firewall_networks` in server.yaml (imported into the store).
### E3. agent.yaml down to bootstrap (M) — done 2026-10-08 (branch `settings-e`)
agent.yaml keeps the control planes' addresses, its certificate and its id.
Labels, fabrics, engines, heartbeat and supervisor settings move to the
store per node and reach the agent over its stream; a change to the file is
noticed and the agent restarts itself.
Built as a section of the cluster's settings rather than a store of its own,
so history, audit, `ctl config`, the console and the portal page cover it:
`agents.default` and `agents.nodes.<id>` (config.AgentSettings; bootstrap
keys node_id/server(s)/hostname/mgmt_addr/tls refused; each node's merge
loaded as an agent would, so a bad fabric is refused on save). A live key:
the control plane sends ControlMessage.AgentSettings on register and, a
second apart per node, on change; the agent compares with what it started on,
checks it loads beside its agent.yaml (else logs and keeps running), keeps it
in /var/lib/llmgw-agent/agent-settings.yaml and restarts (engines adopted).
At start: agent.yaml < kept settings < LLMGW_ env; kept settings that no
longer load are set aside, never a reason not to start. agent.yaml is
watched every 10 s and restarted on when it changes and loads. Left: file
keys are not imported into the settings (they keep working under them);
`node add` still writes a full agent.yaml; labels have two homes now
(`ctl label`, which wins, and agents.*.labels).
### E4. Web settings page (S) — done 2026-10-08 (branch `settings-e`)
The same document, edited on /admin/settings, for whoever has no terminal.
Built: administrators only (staff roles get 404: webhook secrets), checked by
the same function as PUT /api/v1/settings, made from a base version or
refused on a newer one (the edit kept in the editor), history with view and
roll back as a new version, audit with the size not the body, English and
Turkish, a Settings link in the menu.
---
## F. Company documents: knowledge bases (RAG), then adapters (2026-10-08)
A company wants a model to answer from its own documents. Retrieval is the
tool for *knowledge* (sources shown, re-indexed in minutes, access per
tenant); LoRA training is for *behaviour* (vocabulary, format) and has no
access control — whatever goes into weights reaches everyone who may use the
adapter. RAG first. Decisions taken: access per collection (no per-document
ACL yet); a personal key or session names a project or team and gets that
chain only, never the union of its person's memberships; PostgreSQL store
only, off on SQLite with a sentence saying why.
### F1. Store: collections, grants, chunks (M) — done 2026-10-08 (branch `rag`)
Built (`internal/store/knowledge.go`, migration 0035 PostgreSQL only):
- `kb_collections` (owner scope/id, org, embed_model, dims), `kb_grants`
(read|write, any scope, never across organizations; a key needs a
project; a person needs a membership in the organization),
`kb_documents` (digest: the same again is no change, a new one replaces
in one transaction).
- `kb_chunks` is created when the store opens, not in a migration (a
migration needing an extension would stop the control plane; one that
skipped it would never run again): partitioned by organization (hashed
partition name), `tsv` generated (`simple`), GIN on the parent, one
partial HNSW index per collection on its organization's partition
(`embedding::vector(dims)`, `WHERE collection = '<id>'`). EXPLAIN on
20k rows: partition pruned, that index used, policy applied.
Later (scale, 2026-10-08): a partial GIN per collection too
(`kb_tsv_<id>`, backfilled at open); `vec_type` per collection, `halfvec`
on pgvector ≥ 0.7 (up to 4000 dims); `SET LOCAL hnsw.ef_search` to
candidates(k) clamped 40..1000 (EXPLAIN: 40 rows without it, all with
it); the text arm takes candidates by `ts_rank` in a subquery and scores
only those.
- RLS FORCEd on `kb_chunks` against `llmgw.kb_org`/`llmgw.kb_collections`
(set with set_config per transaction), one role — the table owner — not
a separate one. Chunks are deleted by the foreign-key cascade from their
document, which RLS does not block.
- `CollectionsFor(Effective, write)`: owner or grant anywhere on the chain;
no organization (personal key without a project) reads nothing.
- `SearchChunks`: per collection nearest-neighbour (literal id and cast so
the partial index is used) plus a full-text arm with any word
(`a | b`, lowercased by the database, not Go), fused by reciprocal rank.
- Deleting a project/team/org that owns collections is refused like one
with members (changed from "deleted with them": every other delete in the
tree refuses); DeleteUser/ForgetUser delete the person's collections;
deleting an organization drops its partition; grants to a deleted scope go.
- `server init` installs `postgresql-N-pgvector` / `pgvector_N` and enables
it; pgha installs it on every member at setup and enables it on the
leader on every run. Neither stops on failure.
Left: a cluster set up before (the PoC) gets no package from a rerun of
pgha on a running cluster — install by hand or add an upgrade step;
`server restore`/`CopyTo` do not carry kb tables (SQLite → PostgreSQL only,
and SQLite has none); tests need `LLMGW_TEST_POSTGRES` pointing at a
pgvector database as a non-superuser (a superuser bypasses RLS).
### F2. Ingest on the user's machine (L) — done 2026-10-08 (branch `rag`)
Uploading whole files to the control plane was dropped: a PDF is 20–100×
its text, and the work belongs where the files are. Instead a small client,
`llmgw-kb`, run by the person on their own machine with their own key:
- **Its own binary** (`cmd/llmgw-kb`), for Windows, macOS and Linux on amd64
and arm64. `llmgw` itself does not build on Windows (agent syscalls), and
employees' laptops should not carry the server, agent and setup screens. A
test keeps server packages out of its imports.
- `llmgw-kb sync <collection> <dir|file…> [--include/--exclude globs]
[--dry-run] [--delete]`: scan → local catalog (path, size, mtime, sha256;
cache under os.UserCacheDir, digests not recomputed for unchanged files)
→ the collection's manifest from the gateway (name → digest only) → diff
→ for new/changed files only: extract, chunk, embed through the gateway's
`/v1/embeddings` with the person's key (its accounting, limits, guard and
band apply), and send name, digest, chunk texts and vectors; `--delete`
removes documents whose files are gone. `--dry-run` lists what would leave
the machine and how much. A file that cannot be read (locked, unsupported,
no text) is skipped with the reason; the run goes on.
- **Formats, as many as possible.** Built in, pure Go: PDF (text layer),
DOCX/XLSX/PPTX, ODT/ODS/ODP, RTF, EPUB, HTML/XHTML, EML (text and HTML
parts), Jupyter notebooks, subtitles (SRT/VTT), CSV/TSV, JSON/YAML/XML,
Markdown/RST/AsciiDoc/Org, plain text and source code; ZIP archives
recursed into. With a helper found on the machine: legacy DOC/XLS/PPT, RTF
and anything else LibreOffice opens (`soffice --headless --convert-to`),
images and scanned PDFs through Tesseract (`tur+eng`). Text in UTF-8,
UTF-16 (BOM) or Windows-1254/1252 is converted to UTF-8.
- **Across systems:** document names relative to the synced root, `/`
separated, Unicode NFC (macOS hands out NFD), a warning for two names that
differ only in case; CRLF → LF; key from `LLMGW_API_KEY` or `llmgw-kb
login` (os.UserConfigDir, 0600 where that means something); the system's
CA store, `--ca` for another.
- **Gateway endpoints** (client listener, person's key):
`GET /v1/kb/collections` (what this key may read/write),
`GET /v1/kb/collections/{id}/manifest`, `PUT
/v1/kb/collections/{id}/documents` (one document per request: name,
digest, bytes, chunks with text and embedding; dims checked, chunk count
and size bounded), `DELETE …/documents?name=`. Write needs a write grant
or ownership on the key's chain (`CollectionsFor(…, write)`); otherwise
404.
- **Administration** stays with operators: `llmgw ctl kb
create|list|delete|grant|revoke|docs`, admin API `/api/v1/kb`, audit,
console screen; `docs/knowledge.md`.
- Trade-off accepted: vectors come from the client, so a writer could store
vectors that do not match their text — only in collections they may
already write.
Built: `internal/kb/extract` (formats above; helpers found on PATH and in
each system's install places), `internal/kb/chunk` (≈1800 characters under
the heading path, a paragraph of overlap, sections kept apart once a chunk
is a third full), `internal/kb/kbsync` (ledger in SQLite at
os.UserConfigDir/llmgw-kb/ledger.db: documents keyed by gateway, collection
and name with file, size, mtime, digest, state, reason; runs), `cmd/llmgw-kb`
(login, collections, sync, status, history, formats), `internal/router/kb.go`
(the four gateway endpoints, `X-Llmgw-Project` for personal keys, audit),
`internal/control/adminapi/kb.go` + `llmgw ctl kb` + console Knowledge
screen, `docs/knowledge.md`; CI builds the client for all six targets and
releases attach it. Tested: every format from generated fixtures and a
Turkish PDF (OCR path with Tesseract here), sync against a fake gateway and
end to end against the router on pgvector.
Tried on the PoC 2026-10-08 (control planes 140 and 150 on a dev build of
this branch; their previous binary is /usr/local/bin/llmgw.pre-rag): Ubuntu's
pgvector 0.6.0 installed on 149/150 and enabled; bge-m3's weights were never
in the mirror (fetched from CompendiumLabs/bge-m3-gguf@1864f47f) and it is
held at 1 replica; tree Demo/ik/belgeler and Demo/satis/crm with a key each;
collection Demo/politikalar filled from a Mac through an SSH tunnel with ten
Turkish files in nine formats (9 sent in 5 s, the .doc skipped for want of
LibreOffice; a rerun sent nothing in 0 s; a changed file alone was sent).
Nine Turkish questions each found their page first once search scored by
similarity plus word coverage (rank fusion had got two wrong). The other
team's key saw no collection and got 404 by id; with a read grant it read
and could not write; revoked, nothing again. On the live store the llmgw
role (not superuser, no BYPASSRLS) sees 0 chunks with no scope set and 0
with the organization alone.
Left: the gateway's collection list shows the owner as team:ID, not its
path; no LibreOffice on the dev machine, so the legacy
Office path is untested beyond its error; the client is unsigned (Gatekeeper
and SmartScreen warn); Outlook `.msg` is not read; one document over ~60 MiB
of JSON (a few thousand pages) is refused rather than split.
PostgreSQL tests of store, router and adminapi share LLMGW_TEST_POSTGRES and
the store's TRUNCATE each test; run them with `go test -p 1` or one package
at a time, or a router test loses its rows mid-run.
### F3. Retrieval on the request path (M) — done 2026-10-08 (branch `rag`)
- `llmgw.knowledge: [collections]` in the body (stripped before the
engine), or default collections per project/key.
- Allowed set from the caller's chain (§7.9.2), cached like limits; grants
add up down the tree. Personal keys and sessions need `x-llmgw-project`
(or the playground's picker) naming a scope the person is a member of.
- A collection not allowed answers 404, exactly as a missing one.
- Hybrid search with the filter inside the query (per collection, its own
partial index; pgvector 0.6 is enough —
scored by similarity plus the share of the question's words held (built
in F1/F2 as SearchChunks), `/v1/rerank` when a reranker is in the catalog. Top
chunks into a system message with document and position;
`x-llmgw-sources` lists them.
- Leaks closed: answer-cache key includes the allowed set; per-organization
`cache_salt` on RAG requests; injected passages under `no_store` and the
guard's prompt-injection check; returned doc ids in the audit.
- Done: router tests that tenant B never receives tenant A's chunks
(including via cache), that a personal key without a project is refused,
metrics (retrieval latency, chunks returned), docs.
Built (`internal/router/rag.go`): `X-Llmgw-Knowledge` (header, as the
other llmgw extensions are, not a body field), `X-Llmgw-Knowledge-K`;
collections from the key's chain (kbEffective, shared with the kb
endpoints), 404 for one not readable; the question embedded through the
ordinary path on the caller's principal; SearchChunks; passages trimmed to
the model's context; added to the one leading system message (or a new
one); `x-llmgw-sources`; before guardInput; never cached; metrics. Default collections built (kb_defaults on org/team/project/key, narrowest
wins, 30 s cache, `ctl kb default`, console action; never refuse); the
portal chat's Documents picker (collection, default or none; sources listed
under the answer; a header the page sends itself is dropped). `cache_salt` per organization on requests given passages, sent to vLLM
replicas only (chosen replica's engine). Checked on the PoC 2026-10-08:
vLLM 0.30.0 (cuda01) with TinyLlama-1.1B AWQ run by hand beside the 35B in
the card's last 2.3 GB — a 668-token prompt reused 656 tokens under the same
salt and 0 under another's, with identical text (llmgw cannot place it
there: placement takes whole GPUs). Live on the PoC: a team default answers
a request without the header, `none` turns it off, an explicit header with
k=2 gives two sources and no more, llmgw-kb lists owners by path. Left: returned passages
are not audited per request (the header and the kept request record them).
### F4. Adapter training (L, after F1–F3) — done 2026-10-08 (branch `adapters-f4`)
A run-to-completion job placed by the control plane like a replica and
supervised by the agent like an engine, in a Python environment the agent
installs (TRL + PEFT, QLoRA). Data: Q/A pairs generated from a collection by
a cluster model through `/v1/batches`, reviewed by people. Output to the
mirror, into the model's `adapters`, through canary and a quality suite.
Runs when the scaling schedule frees a GPU; 8–14B dense bases on one card.
Never train on a collection some users of the adapter may not read.
Tried 2026-10-08 on the PoC, by hand: Qwen2.5-0.5B-Instruct (fp16) with a
LoRA (r=16, every linear layer, 8.8M trainable), trained beside the 35B in
cuda01's last 2.3 GB (peak 2158 MiB on the card, 64 s for 138 steps, PEFT in a
venv over the vLLM engine's torch). Data: 90 Turkish Q/A pairs a cluster
model wrote from the nine Demo/politikalar documents, through the gateway.
On nine questions it had not seen, phrased otherwise: base 0/9, 6 epochs
4/9, 4 epochs at a lower rate 2/9. Wrong answers mixed facts across
documents (the meal card's amount, the office day). The same nine through
retrieval: 9/9 with sources. So F4 is for format, vocabulary and tone, and
facts come from F1–F3; a bigger base on a free card would do better but not
change that. The training itself fits a small slot on a shared card, which
the job design can rely on for small bases.
Built: `ctl kb dataset` (POST /api/v1/kb/collections/{id}/dataset, NDJSON
with each pair's collection); train_jobs/train_artifacts (migrations 0037,
0038 owner); internal/control/train (queue, one job per node, artifacts in
1 MiB pieces, a node that goes fails its job); internal/agent/train (the
newest vLLM's python3 + `uv pip install --no-deps --target` peft 0.21.2 and
accelerate 1.15.0; llama.cpp's conversion code for the installed release,
untarred from its source; the card with the most free memory, if enough;
runs as llmgw-engine); `ctl train start|list|show --follow|cancel|rm|
download|publish`; the mirror's PUT /v1/models/{model}/{version}/{file};
publish uploads the GGUF (llama.cpp) or PEFT files (vLLM) and sets the
adapter after checkAdapters. Owners: catalog.Adapter.Owner, enforced in the
router (404, hidden from /v1/models); a job's owner defaults to its single
collection's owner, must be able to read every collection the pairs came
from, and pairs from a knowledge base are never public. Tried on the PoC:
dataset of 90 pairs in 89 s; a job trained beside the 35B (peak 2214 MiB,
138 steps in about a minute) and sent back PEFT and GGUF.
Left: a job dies with its agent's stream (no resume); no console screen for
training yet; quality gating of a published adapter is by hand (ctl eval).
---
### F5. Large imports (M) — measured 2026-10-09, open
The metallurgy corpus (2,314 files, 108,779 chunks, bge-m3 on the one
GPU) went in at about 25 chunks/s, the same over a phone hotspot and the
LAN. The engine's slots were the limit: llama.cpp `--parallel 4` embeds
four texts per pass, while llmgw-kb sends 32 a request from 6 workers. The
usage rows show 4.6 s a request on average, 1.8 s of it in the gateway's
queue and 0.04 s in the guard. The GPU sat at 18–50% and the store machine
at a third of its CPUs.
- An embedding model's profile with many slots by default (`--parallel`
32–64, `--ctx-size` to match, `--batch-size`/`--ubatch-size` at least the
longest chunk), or `ctl model serve` choosing that for a model with
`capabilities.embeddings`. With the default 512, one 516-token chunk failed.
- Embedding on the gateway's side of a document PUT, so the client sends
text only: today an embeddings answer is ~21 KB of JSON a chunk, and
the text and vector go up again (~11 KB).
- The store next: HNSW inserts took half of one of pg01's two cores; at a
million chunks the vectors alone outgrow its 4 GB. halfvec (pgvector ≥
0.7) and a larger store machine.
- Done: a batch of embeddings checked text by text against the context
(branch `embeddings-batch`); it was refused as one over-long prompt.
- Done: `query_prefix` on an embedding model, before questions only.
- Qwen3-Embedding-4B on the RTX 6000 (2026-10-09): llama.cpp Q8 1,575
tok/s as first set up (32k cache, 16 slots), 2,700 with a 16k cache,
FlashAttention and ubatch 1–2k; f16 no better. vLLM fp16 (`--runner
pooling`) 6,200 direct, 4,700–4,900 through the gateway in the import,
and its vectors match llama.cpp's (cosine 0.9995). The 103,579 chunks
took 3 h. Done (2026-10-10, branch `admission-bulk`): a source per
profile, so an embedding model's GPU profile runs vLLM and its CPU
profile llama.cpp; a band's own `queue_ttl` for bulk imports, and
`Retry-After` on a busy refusal, which is what had turned passages away
(llmgw-kb retries only what says when to).
- Retrieval on 40 questions written from the corpus, document in the top
5: bge-m3 31/40 (Turkish questions 13/21), Qwen3-Embedding-4B 37/40
(18/21), with its query prefix 36/40 and first place 31/40 instead of
28 (MRR 0.738 → 0.789 → 0.838). Next: translating the question with the
chat model, and a reranker.
## G. Jobs: 3D generation through ComfyUI (2026-10-10)
A request for a 3D model takes minutes on the one 24 GB card, and the card has to be
emptied of the chat model first. That is not a request to hold a socket for
— `internal/router/activate.go:46` already says so — it is a job: queued,
answered with an id, asked after, its result downloaded. The job machinery
is built first and tested without a GPU; the ComfyUI generalisation and the
custom-node install come after. Decided 2026-10-10: a `jobs` table of its
own rather than a one-line batch (the JSONL shape would be artificial and
has nowhere for progress), and progress is shown, not just a status.
Which models (decided 2026-10-10): the assets go into a game sold on
Steam, worldwide, the EU included, so the licence of the output decides
before the quality does. Hunyuan3D (every version) is out: Tencent's
licence excludes the EU, UK and South Korea, and whether a game shipped
there with its outputs is "use in the Territory" is not a question to
leave to a court. In, in this order: **TRELLIS.2** shape (MIT; the best
geometry; its own texture baking goes through nvdiffrec and nvdiffrast,
both under NVIDIA's non-commercial source licence, so that stage is not
run), **Step1X-3D** (Apache 2.0, StepFun; shape and texture in one, the
EU-safe textured path — its ComfyUI node and VRAM need are to be
checked), **Direct3D-S2** (MIT, DreamTech; high-detail shape, has a
ComfyUI node) if TRELLIS.2's detail falls short. Not in: TripoSR and
InstantMesh (permissive but superseded), Stable Fast 3D (Stability's
community licence, revenue-capped; served all the same since 2026-10-10
at the user's request, see G3), SAM 3D Objects (Meta's own licence,
to be read before it is considered). Steam's disclosure form (changed
2026-01-16) lists pre-generated 3D models as content that must be
declared on the store page: the licence notice G3 writes into the GLB is
what the studio answers that form from.
### G1. Jobs: store, worker, endpoints (M) — done 2026-10-10 (branch `jobs-g1`)
Built as written, with two choices: a job found running after its gateway
died is run again from the start (the prompt id column is there for G2 to
follow the same work instead), and the request is validated at the door
only as far as costs nothing — key, model, allowlist, the request's shape
— with limits, guardrails and admission applied when the worker replays it,
said as the job's error. The result file's `content_type` is the job's; the
files endpoint still serves every file as an octet stream.
- Store: `jobs` (id, owner key/principal, kind, model, status, input
(JSON), progress (JSON: queue position, percent, step), result file id,
error, created/started/finished, lease owner/until), migrations for
sqlite and postgres beside `internal/store/batches.go`, whose
claim/renew/release lease pattern is copied, not shared: a job is one
unit of work, a batch is many lines.
- Worker: `internal/router/jobworker.go` after `batchworker.go`. Any
gateway on the store claims a queued job, asks for the model with
`activator.Want` and waits as long as it takes (no `ActivationWait`
bound: nobody is holding a connection), runs it, stores the output
through the files store, writes usage and price, marks it done. A
gateway that dies mid-job loses its lease and another one picks it up;
the ComfyUI prompt id is in the job row so the second gateway asks
after the same work rather than queueing it again.
- Endpoints: `POST /v1/3d/generations` validates like any inference
request (key, allowlist, limits, admission, guardrails on the prompt),
writes the job, answers 202 with `{id, status: "queued"}`;
`GET /v1/3d/generations/{id}` returns status, progress and, when done,
`output_file_id` and the usage; `POST .../{id}/cancel`; the file comes
from `GET /v1/files/{id}/content`. Owner-scoped like batches. Results
are kept for the batch files' retention and deleted with them: a
departure from `images.go`'s "kept nowhere", because a mesh is a
download, not a JSON field; the retention is the bound.
- Done when: a job survives the gateway that took it being killed, a
cancelled job leaves ComfyUI's queue (`comfy.Client.cancel`), two
gateways never both run one job (test with the lease), and the usage
row carries the job's price.
### G2. Progress from ComfyUI (S, after G1) — done 2026-10-10 (branch `jobs-3d`)
Built as written: the gate's websocket under its own client id, the
client_id of every /prompt rewritten, /progress/{prompt_id} on the gate,
the router reading it and /queue into the job (only into a job of the same
key). Left: a node that reports no steps (TripoSR's) shows its name and no
percent; a vanished replica is said by the job failing, not by progress.
The gate (`internal/comfy/gate.go`) passes seven paths and no websocket.
Rather than open `/ws`, the gate subscribes to ComfyUI's own `/ws` once,
with `golang.org/x/net/websocket` (already a dependency), under a client
id of its own, and rewrites the `client_id` of every `/prompt` it
forwards to that one. It keeps the last `executing`, `progress`
(value/max/node) and `execution_error` message per prompt id and serves
them at `GET /progress/{prompt_id}` on the gate; the router polls that
beside `/history` and `/queue` and writes queue position, percent and the
current node's class into the job's progress. The gate's buffer is
bounded (the last N prompts) and the websocket reconnects. Done when a
3D job shows "queued, 2 ahead" then "running 40%, step 12/30" in
`ctl jobs` and on the account page, and a job whose replica vanished
says so within one poll.
### G3. ComfyUI beyond pictures: 3D models and custom nodes (M, after G1) — done 2026-10-10 (branch `jobs-3d`), with TripoSR
Built: output kinds, image upload through the gate, every output list
fetched, OBJ to GLB, licence in /v1/models, the job and the GLB, per_mesh
and usage.meshes (migration 0045), custom nodes cloned at their commit by
the gate with packages beside the replica (a node's named packages at its
range, the rest at ComfyUI's versions). Not as written: the model. On the
PoC's Turing card TRELLIS.2 cannot run — it needs Meta's gated DINOv3,
its published background remover (RMBG-2.0) is non-commercial, and its
sparse attention and Direct3D-S2's need flash-attention (Ampere and
newer). TripoSR (MIT) proves the chain on cuda01: a .glb from a picture
in about 30 s. Left: background removal before TripoSR (BiRefNet, MIT,
as a node), TRELLIS.2 or Direct3D-S2 on an Ampere card with an ungated
encoder, a missing node refusing the replica by name.
Added 2026-10-10 at the user's request, beside the order above: **Stable
Fast 3D** as `sf3d` (Stability AI Community License: free under US$1 M
annual revenue, an enterprise licence above, "Powered by Stability AI"),
the textured mesh that runs on Turing: about 1 s a mesh warm in fp16,
UV-mapped GLB with its texture, 18 GB of the card at peak. Its weights are
gated, so they were fetched once with the user's token into
/srv/llmgw-comfy/stable-fast-3d and linked into the engine user's
Hugging Face cache, which its loader node reads by repository name; no
token is on the machine at runtime. Its own save node returns the GLB as
base64, which the gate cannot fetch, so `LLMGWSaveMesh` writes the file;
`LLMGWPrepareForMesh` now also gives the object's mask. Left: its two
CUDA extensions (texture_baker, uv_unwrapper) are not on PyPI, so they
were built on cuda01 and installed into the engine's venv by hand; a
custom node's requirements need a way to name a local build, or the
engine install must carry them, before a customer site gets this. And
rembg's default session now fetches BRIA RMBG-2.0 (non-commercial):
every workflow must name its rembg model (ours say u2net).
- Catalog: a model makes `pictures` or `meshes` (`MakesImages` becomes an
output kind; `ServesComfyUI` stays). 3D weight folders in the gate's
`modelFolders`; the workflow's placeholders gain `image` (the input
picture, base64 in the request, written to the replica's `input/` by
the gate through a new `POST /upload/image` the gate allows — or a
base64-loading node, whichever TRELLIS.2's wrapper wants).
- `comfy.Client.fetch` reads every output key, not `images` only, and
fetches each file named by `/view`; `Image` becomes `Artifact` with its
content type (model/gltf-binary, application/octet-stream for .obj).
- Custom nodes: a model (or the engine config) names git repositories
and a pip requirements list; the agent's provision step clones them
into the replica's `custom_nodes/` (the gate's `--base-directory` is
where ComfyUI looks, not the install directory) and installs the
requirements into the engine's venv with the same PyTorch index and
lock discipline as A8. A replica with a missing node refuses to start
with the node's name in the error rather than ComfyUI's traceback.
- Licence of the output (decided 2026-10-10: it matters). A catalog model
carries `license: {name, url, notice}`; a job's answer, `/v1/models`
and the portal page show it beside the result, and the mesh's GLB
`asset.extras` carries the notice too, since the file travels without
the API. What was checked on 2026-10-10: TRELLIS.2 weights are MIT and
its outputs free, but its texture baking depends on nvdiffrast, whose
NVIDIA Source Code License is non-commercial — a commercial service
must render textures another way or skip that stage. Hunyuan3D 2.0 and
2.1 claim no rights in outputs, but the licence excludes the EU, UK and
South Korea, needs Tencent's permission above 1 M monthly active users,
forbids using outputs to train other models, and asks for "Powered by
Tencent Hunyuan"; the notice ships with any distribution. TripoSR is
MIT, InstantMesh Apache 2.0. Every one of them trained on Objaverse,
some of whose objects are CC-BY-NC: a provenance question no licence
text settles, to be said in the notice rather than hidden.
- Pricing: `per_image` stays; `per_mesh` beside it in `accounting.Price`,
`Outcome.Meshes`, the usage sums and exports (as eaea671 did for
pictures).
- Done when: TRELLIS.2 on cuda01 with `scaling: {min: 0,
idle_timeout: 10m}` makes a .glb from a picture through G1's endpoint
with nvdiffrast and nvdiffrec not installed in its venv (the licence
is kept by the install, not by a note), the GLB carries the MIT
notice,
the card is free again after the idle timeout, and the same request
against flux-schnell is refused as "makes pictures, not meshes".
### G4. Portal and console (S, after G2) — done 2026-10-10 (branch `jobs-3d`)
Built: /3d with model-viewer served from the gateway, progress, download
and licence; `ctl 3d list|get|cancel` over /api/v1/client-jobs (ctl jobs
was taken by the control plane's own work). Left: the console's Jobs view.
A 3D page beside `/images` (upload a picture, watch progress, download
the .glb, a viewer with `<model-viewer>` from the allowed CDNs), `ctl
jobs list|get|cancel`, the console's Jobs view. The page sends down the
same path as the API, like `imagesSend` does.
## H. ComfyUI as a platform (written 2026-10-11)
G made one model at a time work: flux for pictures, TripoSR and SF3D for
meshes, each with its own endpoint, its own request shape and its own
replica. The user's direction (2026-10-11): llmgw is a generic platform,
and should serve whatever ComfyUI can run — pictures, edits, video,
audio, meshes, upscaling, segmentation — as data in the catalog rather
than code per model. Two decisions shape everything below:
- **The licence is metadata, not a filter.** llmgw does not leave a
model out because of its licence. The licence travels with the model
and with every file it makes; what a site will not serve is that
site's policy (H6). The game studio's EU constraint is one such policy,
not a decision the code makes for everyone.
- **The install fits the hardware.** The PoC has one Turing card; other
sites will have Ampere, Ada and Hopper. A catalog entry must be
writable once and placed sensibly anywhere: fp16 on Turing, bf16 or
fp8 where the card has them, flash-attention models only where it
runs, CUDA extensions built for the card that runs them (H2, H4).
The first set of models proves the structure at its corners rather than
by licence: Qwen-Image-Edit (several weight files, two pictures in),
Wan 2.2 small (video out, a long job), ACE-Step (audio out), SF3D (a
package not on PyPI, gated weights), Hunyuan3D (a licence that excludes
the EU, which H6 must be able to say no to).
### H1. A workflow says what it takes and makes (M) — built 2026-10-11, not yet tried on the PoC
Built as written, with three choices: a run answers with one file, the
first of its output kind (several outputs as one zip is left); the
workflow's placeholders are checked against the inputs when a run is
served, since the catalog never reads the workflow file; and video,
audio and files count as nothing in usage until a per-run price exists
(per_image and per_mesh still count pictures and meshes).
- Catalog: `comfyui.inputs` — a list of `{name, type, required, default,
min, max, choices, max_mb, doc}`, type one of text, int, float, bool,
choice, image, mask, audio, video, file. Each is a placeholder of the
same name in the workflow. The template's own placeholders are checked
against the list when the catalog is built: one the workflow uses that
no input or built-in names is refused there, not at the first request.
- `comfyui.output` gains `video`, `audio` and `files` beside `pictures`
and `meshes`; `files` takes whatever the workflow saves. `comfy.Image`
becomes `comfy.File` and `typeByName` knows mp4, webm, gif, wav, flac,
mp3, ogg.
- Endpoint: `POST /v1/workflows/runs` with `{model, inputs: {...}}`, a
job like G1's (`GET /v1/workflows/runs[/{id}]`, `.../cancel`). A file
input is base64, a data URL, or a `file-…` id from `/v1/files` for
what is too big to send inline (video, long audio). Files go into the
replica's input directory through the gate's `/upload/image`, which
ComfyUI uses for any file. `/v1/images/generations` and
`/v1/3d/generations` stay as they are, as fixed shapes over the same
machinery.
- The models list (`/v1/models`) gives each ComfyUI model its inputs and
output kind, so a client can build a form without reading the catalog.
- Done when: the existing flux, triposr and sf3d entries work unchanged,
and one model with a picture and a text input (Qwen-Image-Edit) runs
through `/v1/workflows/runs` with a 400 for a missing required input
before any job is written.
### H2. Profiles that fit the hardware (M) — built 2026-10-11
Built as written. A profile by need is left out of placement until the
cluster's cards are known: with no class, placement would have read it as
a CPU profile. A profile's own `source` is loaded under its own file name
by the workflow's `{{checkpoint}}`.
- A profile may say `gpu: {min_compute_capability, min_memory_gb}` in
place of `gpu_class`. When the reconciler places a model, such a
profile stands for every GPU class in the cluster that meets it, at
the profile's priority: the catalog function that turns profiles into
placement's view takes the cluster's classes (class, compute
capability, memory) and returns one concrete profile per class that
fits, named `<profile>@<class>`. Placement itself is unchanged: it
still sees exact classes, and a tensor-parallel group still never
mixes them.
- A ComfyUI profile may carry its own `comfyui` block (workflow, nodes,
checkpoint) over the model's: the fp8 workflow on compute 8.9 and up,
the fp16 one on 7.5. A Turing card is not an error, it is the profile
that fits it.
- Done when: one catalog entry with an "ampere+" profile (min 8.0) and a
"turing" fallback places the fallback on cuda01, and the same entry,
in a test with an A100 in the inventory, places the other.
### H3. Environments: which models can share a replica (S, the catalog side)
- `comfyui.environment: <name>` puts a model in a named environment.
Models in one environment must agree on every custom node they both
name (same repository, same commit) and their requirements must
resolve together; the catalog refuses an environment that cannot, with
both models' names. Models with no environment are each their own, as
today.
- `Catalog.Environments()` returns each environment with its models and
the union of their nodes: what one replica of it must install.
- Placement and eviction of a replica that serves every model of its
environment (so flux, SF3D and Qwen-Image-Edit stop handing the card to
each other with a 30 s cold start each time) is the other session's
work (llmgw-b7), built on this.
### H4. Custom nodes with packages that are built, not downloaded (S) — built 2026-10-11
Built as written, with `uv build --no-build-isolation` against ComfyUI's
Python, the card's capability asked of torch in the replica (so
CUDA_VISIBLE_DEVICES is the card it will run on), and wheels kept under
the engines' cache by commit, `sm<arch>` and directory. Also
`comfyui.huggingface_repo`: a node that loads its weights by repository
name (SF3D's) finds them in the engines' Hugging Face cache, linked by the
gate, so no token is on the node.
- `nodes[].build: [subdir, ...]`: directories of the node's repository
that are Python packages with C++/CUDA extensions (SF3D's texture_baker
and uv_unwrapper). The gate builds each as a wheel with the engine's
Python, `TORCH_CUDA_ARCH_LIST` set from the compute capability of the
card it runs on, and installs it into the replica's own `site` like
any other requirement. Built wheels are kept under the engine's home by
node commit, subdirectory and architecture, so the second replica on
the same kind of card does not build again.
- A node needing nvcc on a node without it refuses the replica with that
said, not with a compiler's traceback.
- Done when: SF3D runs on cuda01 with nothing installed into the engine's
venv by hand (the hand install of 2026-10-10 removed first).
### H5. Licence on every output (S) — done 2026-10-14: the whole licence in GLB asset.extras and PNG iTXt; video and audio not yet
- `License` gains `id` (SPDX where there is one, else a lowercase name:
stability-community, tencent-hunyuan-community), `commercial` (unset is
not said), `excluded_regions` (ISO 3166 codes, `EU` for the union),
`revenue_cap_usd` and `attribution` ("Powered by Stability AI"). All
optional but the name. Done 2026-10-11 in the catalog, so H6 can match
on `id`.
- Written into the file where the format has a place for it: glTF
`asset.extras.license` (as now), PNG an `iTXt` chunk named `license`
(the workflow's own text chunks are still stripped), MP4/WebM and the
audio formats in their metadata where ffmpeg is at hand on the router,
otherwise beside it in the job's answer only — said in the docs.
### H6. Licence choices (S) — done 2026-10-11 (4bbac08), live on the PoC
The user asked (2026-10-11) for the choice to be a person's: on their
profile they pick the licences they will not use, and their keys and
portal session neither see those models in `/v1/models` nor may call
them (403, saying which licence). Matched on `license.id`, and on
`commercial` / `excluded_regions` where a choice is put that way. Its
store table is migration 0046. A site-wide or team-wide default can sit
on top of it later.
### H7. A job worker per model, not one for all (S) — done 2026-10-14: one job per model at a time, eight in all
The worker runs one job at a time: a job waiting for the card holds up a
job for another model that could run now (seen on the PoC, 2026-10-10).
Jobs are claimed and run concurrently up to a limit per model (or per
environment, after H3) and a cluster-wide cap, with the lease and
leader-only rules of G1 kept.
### H8. Presets that add themselves (S) — done 2026-10-13 (v0.14.5)
flux-schnell and triposr come with llmgw (`ctl model add <preset>`), and
the leader adds them on demand where a node fits one, the cluster installs
ComfyUI and the catalog makes nothing of that kind (`presets.auto`); a
preset removed is declined and not added back. The cluster page and the
console show and add them.
### H9. Small things learned on the PoC (2026-10-14)
- Done: `albedo_match` on sf3d (its delighting made dark objects light);
`on_demand.evict_after_idle` / `min_lifetime` as live settings; an
upgrade that leaves nodes behind says which; `ctl model serve` serves an
embedding model for embedding; refreshes wait while a form is filled in.
- Done 2026-10-15: llmgw's `/mcp` with picture, 3D, VLM and workflow tools,
so a client needs no server of its own (asked by the game team); the
shared ComfyUI replica (H3's placement side).
- Done 2026-10-16 (v0.14.6, v0.14.7): setup lists every part with what it
is and its settings; three machines each run the control plane, with a
floating address (VRRP id from the address); a lost store primary costs no
request (3 s connect per member, keepalive and TCP_USER_TIMEOUT, Patroni
ttl 20, a standby leaves the lease alone a round after the store returns);
a new leader waits for its judge within the budget; a judge is asked again
on another machine at half its budget; init sizes the judge (2 GB, no
prompt cache, two cores, one judge on up to four cores on one machine).
## I. What other gateways do that llmgw does not (written 2026-10-16)
From a comparison with Bifrost (maximhq/bifrost, Apache 2.0, Go), with
LiteLLM and Portkey in mind. Bifrost routes to 23+ cloud providers and runs
no model; llmgw runs the models, so most of Bifrost is either in llmgw
already (keys, budgets, fallbacks, guardrails, tracing, SSO, HA — the last
three paid in Bifrost) or does not apply. These four are worth taking for
a site that runs its own models. Left until a customer asks: native
Azure/Anthropic/Bedrock/Vertex adapters, request hooks a customer plugs in,
routing by what a request holds. Not taken: a realtime (WebSocket audio)
API, prompt management, an `npx` install.
### I1. Answers to similar questions, not only the same one (M) — done 2026-10-25
`keys cache --similar`, `router.cache.embedding_model`, `internal/router/similar.go`. Changed
from the plan: the vectors are in the gateway's memory beside the exact answers, not a pgvector
table. The exact cache keeps answer text out of the store on purpose (data.md), and a table of
answers would undo that; it also works on SQLite. The usage row marks a similar hit as a hit
(no column of its own). Open: the 40-rewordings and eval-suite runs on the PoC.
The original item follows.
The answer cache (`internal/router/cache.go`, C17) answers a request whose
body is the same byte for byte. A help desk asked "how do I reset my
password" in forty wordings asks the model forty times.
- Per project, off by default, beside the exact cache: `llmgw keys cache
--project P --similar 0.95` (the cosine similarity a question must reach;
higher is stricter). The exact cache is asked first.
- The question embedded by a model the operator names
(`router.cache.embedding_model`, an embedding model in the catalog), into a
pgvector table per project with the answer, the model and the TTL; the
store's halfvec and text index work from F applies. SQLite stores keep the
exact cache only.
- What counts as the question: the last user turn, with the system prompt
and the earlier turns hashed and required to match exactly — a similar
question in another conversation is not the same question. Tools,
`response_format` and the sampling settings also exact.
- Said: `x-llmgw-cache: similar` with the similarity, and the usage row
says so; guardrails judge the request before a hit, as for the exact
cache. A project with `no_store` gets neither.
- Done when: forty rewordings of one question on the PoC cost one model call
and 39 lookups, and an eval suite run with the cache on scores within a
point of the run with it off.
### I2. MCP gateway: the customer's tool servers behind llmgw (L)
P1 done 2026-10-20: HTTP servers registered (`ctl mcp`, `/api/v1/mcp/servers`,
store `mcp_servers`), tools as `server.tool` in `/mcp`, `allowed_tools`
grants in the tree (deny by default), credentials in the store or by
reference, the caller named in `X-Llmgw-*` headers, time and size bounds,
progress passed on, audit (Source `mcp`) and metrics. Left: P2, the
guardrails both ways, a per-tool rate limit, a breaker, and a portal page;
P3, stdio servers and per-person OAuth. The "done when" with Claude Code
against a real server is still to try.
`/mcp` serves llmgw's own tools. An agent inside a company also wants Jira,
the file server, a database — each an MCP server, each today configured
in every client with its own credentials and nobody seeing what was called.
- Servers registered once, by an administrator: `llmgw ctl mcp add jira
--url https://… --header-file /etc/llmgw/secrets/jira` (HTTP transport;
stdio servers run on the control plane under their own user, like an
engine). Their credentials stay on the control plane, never in a client.
- `/mcp` lists every server's tools, prefixed by its name (`jira.search`),
filtered by what the key may use: an allowlist per org, team, project or
key, by server or by tool, in the tenancy tree as model allowlists are.
- Each call goes to the audit log — who, which tool, the arguments' size,
the outcome and how long — and through the guardrails both ways (a tool's
answer is prompt input). A per-tool rate limit.
- Later, if it earns it: Bifrost's "code mode" (the tools as one typed API
the model writes code against), which cuts the tokens of a long tool list.
- Done when: Claude Code given only llmgw's `/mcp` reaches a test MCP server
through it, a key without the grant does not see the server's tools, and
every call is in `ctl audit`.
### I3. What llmgw adds to a request, measured (S) — done 2026-10-16
`llmgw ctl bench overhead` (`internal/cli/bench_overhead.go`): an instant
engine and the gateway in one process, four modes (direct, bare, auth, auth
and guard patterns) at rising rates. On an Apple M5 the gateway added
0.05–0.42 ms at p50 and at most 0.49 ms at p99, in the README.
`TestTheGatewaysOverheadIsMeasured` runs a second of it in CI with no bound
asserted. Left: the CPU figure is rough with everything in one process; a
gateway in a process of its own would measure it properly. The original
item follows.
Bifrost's headline is "11 µs at 5k RPS" (its own figure, and its own pages
disagree with it). llmgw has none, so any question about its overhead is
answered with a guess.
- `llmgw ctl bench --overhead`: the same load through the gateway and straight
at a `sim-engine` (`internal/cli/simengine.go`) that answers at once, at
rising request rates, with auth, accounting and the guardrails' pattern
checks on and off. Reports the added latency at p50/p99 and the gateway's
CPU per thousand requests.
- On the PoC the CPU is oversubscribed (pve1, 28 threads for ~125 vCPUs):
report it as a ratio and record the host's load beside it, and repeat on
an idle machine before a figure goes into the README.
- Done when: the README states the overhead, with how it was measured, and
CI runs the short version so a change that doubles it is seen.
### I4. Several keys for one outside provider (S) — done 2026-10-17
`external.api_key_files` (`catalog.ExternalKey`: file, weight), one endpoint
each in `control.externalEndpoints`, 401 retried on another key and
`external_key_refused` notified (`router.keyRefused`). With one key a 401 is
still the client's. `TestAKeyAtItsLimitOrRefusedIsPassedOverForAnother`. The
original item follows.
An external model has one `api_key_file` (`internal/control/control.go`,
where external endpoints are made). A provider's limit is per key; one key
answering 429 makes the whole fallback answer 429.
- `api_key_files: [a, b, c]`, each with an optional `weight`: one endpoint
per key (`external:<model>#<n>`), so the router's scoring, retries and
breakers work per key as they already do per replica. A key answering 429
or 401 is passed over while its breaker is open; a 401 also notifies
(`external_key_refused`).
- The usage row says which key answered (by its index, never the key).
- Done when: with two keys and one returning 429, every request is answered
by the other, and the console shows the one breaker open.
## J. Found testing HA and single-machine installs (written 2026-10-16)
Three VMs installed as HA from the setup screens, a fourth as a judge node,
a fifth as a single 4-core, 8 GB machine; machines powered off under load.
### J1. A restart refused for room another restart was giving back (S) — done 2026-10-16
Not a wait for ever, as first written: on the 4-core VM the judge's restart
was *refused* ("fits on no node even with this replica stopped") because the
starter, restarted at the same time to reserve fewer cores, still held its
old ones. Once refused, nothing tried again, so the judges kept their old
figures. Two fixes in `internal/control/reconciler`:
- a restart whose replacement fits once other restarts under way have gone
(queued, retiring or draining) waits for them, up to DrainGrace, instead
of being refused (`fitsAfterRestarts`, restart.go);
- a drained replica counts a retiring one that still answers as serving, so
one restart after another no longer leaves the first's drained replica,
and the room it holds, waiting out its grace (reconciler.go).
`TestARestartWaitsForRoomAnotherRestartWillGiveBack` is the 4-core case.
### J2. A judge the leader adds is sized as init's is (S)
When a policy names a judge the catalog lacks, the leader serves it itself
(`internal/cli/judgekeeper.go` via `knownJudge.serveRequest` in
`judges_known.go`, through modelserve). That judge reserves 3 GB (the
weights, an 8192-token KV cache and a fixed gigabyte), no cores, and runs
with llama.cpp's prompt cache, which grew one to 3.2 GB. init's judge
(`guardModel` in `internal/cli/serverinit_cluster.go`) reserves 2 GB and
its cores and runs with `--cache-ram 0`.
- `modelserve` gives a guard model `--cache-ram 0`, and the keeper's
request the same memory and cores rule as `cpuPlan`.
- Done when: a judge removed from the catalog comes back with the same
figures init would have given it.
### J3. HA setup puts the judges on nodes of their own (S)
Under load on a 4-core control plane that was also a node, 38 of 60
requests were refused for want of a judgment. With the judges on a node of
their own (4 cores, 32 GB, two cores each) the same test refused 2.
- The HA setup offers a judge node or two: machines given like the store's,
added as nodes with a label (`llmgw.role: judge`), and the judge's
profile selects that label.
- Done when: the setup installs the judges on the machines given for them,
and the load test above refuses no more than it did by hand.
### J4. The model mirror on more than one machine (M)
In an HA install the mirror stays on the first machine. While it is down,
models the nodes already have keep serving and new ones wait for it.
- A second mirror kept in step (each machine of the store, or one more), the
control planes and nodes given both, the first that answers used.
- Done when: with the first machine powered off, a model added then is
fetched and served.
### J5. An upgrade corrects an earlier install's judge (S) — done 2026-10-16
Built in the judge keeper (`internal/cli/judgekeeper.go`, `refit`), on the
leader: a judge whose entry is exactly what init before v0.14.7 wrote is
given 2 GB, `--cache-ram 0` and cores by init's rule (`refitCores`: up to
four for one judge, two each for more), through `modelsvc.SetProfiles`,
and its ready replicas are restarted (after J1). Its replica count is left
as it is. One that differs is left alone and said once in the log. The
original item follows.
v0.14.6 and earlier installed two judges reserving no cores or one, with
2 or 4 GB and the prompt cache on. On fewer than eight cores every judgment
missed its 5 s budget and every request was refused. v0.14.7's notes say
how to fix it by hand (`ctl model resources`, then `ctl restart`).
- On upgrade, or on `server init` run again, a judge whose profile is still
exactly what an earlier init wrote is moved to today's figures and
restarted (after J1). One changed by hand is left alone and named.
- Done when: a v0.14.6 single-machine install upgraded to it judges within
budget without anything typed.
### J6. Rate limits per process (note)
Rate limits are counted in each control plane's memory. A failover starts
every key's minute afresh. Documented in install.md; worth a shared count in
the store only if a customer's limits must hold across a failover.
### J7. nginx in front of several control planes (S)
`deploy/proxy/Caddyfile` sends to whichever control plane answers `/leader`.
nginx's open-source build has no active health checks, so its `upstream`
block would send to standbys and to a lost leader alike.
- nginx.conf: the floating address as its one upstream, and why; or the
`max_fails` passive approach with what it costs.
### J8. A judge budget that fits the machine (S)
A judgment takes 2.3 s on four cores and 4.2 on two, against the 5 s
default: two cores are close to it, and a busy machine passes it.
- At install, after the judge is up, time three judgments. If the slowest is
more than half the budget, say so in the plan's closing text with the
choices (more cores, a GPU judge, a longer `latency_budget`, `fail:
open`). The budget itself is not changed without being asked.
- Done when: an install on a machine too slow for its judge says so before
its first refused request.
## L. Taken from Kong's AI Gateway (written 2026-10-25)
From a look at Kong AI Gateway 3.9–3.16 and 2.x. Kong's open source stops at
3.9, and only six of its AI plugins are open (proxy, prompt template,
decorator, regex prompt guard, request and response transformer); the rest
is Enterprise or Konnect. So nothing of Kong is embedded: the two meet over
plain OpenAI/Anthropic HTTP, and these are the ideas worth taking. Not taken:
prompt compression (Kong ships it as a private LLMLingua sidecar, its benefit
unshown), prompt templates and decorators (no one has asked), Konnect's
entity model. MCP items are llmgw-80's (I2); they are listed for it.
### L1. Traces in the OpenTelemetry GenAI conventions (S) — done 2026-10-25
`gen_ai.*` on the `inference` and `upstream` spans (`internal/router/trace.go`):
operation, request and response model, max_tokens, temperature, top_p, the
session as conversation id, input/output/cached tokens, the engine as
provider. Never prompt or answer text (`TestTracesSpeakTheGenAIConventionsAndHoldNoText`).
Prometheus already counted tokens by direction, cached apart. Left: the MCP
calls' spans as `gen_ai.tool.*`, once /mcp is traced at all.
### L2. Redacted personal data put back in the answer (S) — done 2026-10-25
`restore: true` (`guard.Rules.Restore`), `Verdict.Restore`, `internal/router/restore.go`
ahead of the output filter. The original item follows.
The guardrails rewrite personal data as `[KIND_n]` (`internal/guard/gate.go`,
`placeholders`), the same value always the same placeholder. A model that
answers "Dear [PERSON_1]" leaves it so. Kong's sanitizer restores.
- Per policy, `restore: true`: the placeholders of one request kept in memory
for its life, and swapped back in the answer, streamed too (a placeholder
split across two chunks held until it is whole).
- Only into the answer to the request that held them; never stored, never in
usage rows or traces.
- Done when: "write to Ayşe Yılmaz at ayse@example.com" through an enforce
policy reaches the model as placeholders and the client reads the real name
and address back.
### L3. A guardrail that is someone else's service (S) — done 2026-10-25
`guard.Service` (`internal/guard/service.go`), policy `services`, asked
before the judge on input and once on the whole answer. The original item follows.
Customers may be required to use one: Azure Content Safety, Lakera, NeMo
Guardrails, an in-house classifier. Kong's AI Custom Guardrail is a
templated HTTP call with an expression for "block".
- A detector kind `http`: URL, headers from secret files, a body template
with the text, a JSON path that says blocked and one that says why;
timeout and `fail` as the judge's. Checked on input, output or both.
- Done when: a policy with an `http` detector against a test service blocks
what it says to, and a service that does not answer is treated as `fail`
says.
### L4. Routing by latency and by cost, and failover by priority (M) — latency done 2026-10-25
`endpoint.Registry.ObserveTTFT`, `scheduler.Latency()` in `chat`. Failover in tiers is
what `fallbacks` already does (local, then an outside model when no room). Left:
routing by cost between several outside providers. The original item follows.
The scorers (`internal/router/scheduler/scorers.go`) know sessions, queues
and KV cache. Kong also balances by measured latency, by tokens or cost
used, and in priority tiers.
- A `latency` scorer: a moving average of time to first token per endpoint.
- Fallbacks in tiers: local replicas first, then an outside provider, with
the tier said in the usage row.
- Done when: with two replicas of one model, one on a slower card, `latency`
sends most of the traffic to the faster and the slower still gets some.
### L5. Prices by kind and by cache (S–M) — done 2026-10-25
`cached_per_1k`, charged on the answer's `cached_tokens`. A budget in money was already
there (`keys limits --budget`). Not done: cache-write and audio/video token prices —
no engine llmgw runs reports them, and no provider it reaches is billed by them yet.
The original item follows.
`catalog` prices are per million input/output tokens and per picture/mesh.
Kong prices cached reads and cache writes apart, and audio, image and video
tokens apart, and limits by cost.
- Price fields for cached input and by modality; the usage row's cost from
them; a budget limit in money as well as tokens (`keys limits --cost`).
- Done when: a request with 90% cached prompt costs what the cached rate says.
### L6. REST APIs as MCP tools (M) — for llmgw-80, beside I2
Kong turns OpenAPI operations into MCP tools. A company with an internal
REST API and no MCP server could offer it to agents through `/mcp`:
`llmgw ctl mcp add hr --openapi https://hr.internal/openapi.json`, each
operation a tool, under the same grants, audit, guardrails and rate as I2.
### L7. MCP sign-in by the MCP spec (M) — for llmgw-80, part of I2 P3
Kong's MCP OAuth2: llmgw's `/mcp` as an OAuth 2.1 resource server (protected
resource metadata, tokens checked by JWKS against the company's identity
provider), and a token exchanged for a short-lived one per upstream server
rather than one long-lived secret.
### L8. A2A (S, when asked)
Agent-to-agent traffic through the gateway: agent cards' URLs rewritten to
llmgw's, calls audited and traced. The standard is young; wait for a
customer.
### L9. llmgw behind Kong, written down (S) — done 2026-10-25
docs/kong.md, tried with kong:3.9: plain route and ai-proxy, keys passed on;
Kong 3.9 counts no completion tokens of a stream. The original item follows.
A customer that already runs Kong puts llmgw behind it: `ai-proxy` with
`provider: openai` and `upstream_url` at llmgw (3.9, open source), or a plain
route. A page in docs/ with the configuration, tried against Kong 3.9 in a
container, and what Kong's token accounting sees.
## D. Facts worth knowing before starting
- Admin API: `127.0.0.1:9090`, behind the token in
`/var/lib/llmgw/admin-token` (A5), `ctlClient` in `internal/cli/ctl.go`.
Metrics on `:9091` with `/prometheus/targets` beside them.
- Store: `internal/store`, migrations under `internal/store/migrations/{sqlite,postgres}`;
the PoC runs PostgreSQL under Patroni.
- The console (`sudo llmgw`) is `internal/cli/tui_console.go` with actions
in `tui_actions.go`; every operator-visible change needs a row action or
a form there, and a line in `docs/console.md`.
- Tests of the console use a fake admin API (`internal/cli/tui_actions_test.go`,
`adminStub`); tests of serving use fakes in
`internal/control/modelserve/modelserve_test.go`.
- `go generate ./...` regenerates `docs/reference.md` and
`internal/docgen/reference_gen.go` whenever a config struct changes; the
docgen test fails otherwise.
Reference in new issue View Git Blame Copy Permalink