diff --git a/BACKLOG.md b/BACKLOG.md index 2d061db..19a78b9 100644 --- a/BACKLOG.md +++ b/BACKLOG.md @@ -15,6 +15,35 @@ Conventions this codebase keeps, which every item here must keep too: - `go test ./...`, `go vet ./...` and `govulncheck ./...` pass. No release is cut and nothing is deployed to the PoC cluster without being told to. +## Open, in order (2026-10-16, after v0.14.7) + +What is left, most needed first. The detail is in the item each names. + +1. **J1** — `ctl restart --model` waits for ever on a node whose cores are + all reserved. A bug, met while testing v0.14.7. (S) +2. **J5** — an upgrade leaves an earlier install's judge as it was. A + v0.14.6 install on fewer than eight cores refuses every request until + fixed by hand. (S) +3. **I3** — what llmgw adds to a request, measured, in the README. (S) +4. **I4** — several keys for one outside provider. (S) +5. **J2** — a judge the leader adds by itself is sized as init's is. (S) +6. **J3** — HA setup puts the judges on nodes of their own. (S) +7. **J8** — a judge budget that fits the machine, measured at install. (S) +8. **F5** — large imports limited by the embedding engine's slots. (M) +9. **I1** — answers to similar questions from the cache. (M) +10. **J4** — the model mirror on more than one machine. (M) +11. **I2** — the customer's MCP servers behind llmgw. (L) +12. Smaller leftovers: audit before/after values, a tenant-admin role and + forms hidden from viewers (C1); Parquet and a scheduled push for usage + export (C4); English errors on Turkish pages (C24); H1 tried on the PoC; + licences in video and audio (H5); nginx in front of several control + planes (J7). + +Not code, the operator's to decide or do: the PoC on v0.14.7 (it runs a +v0.14.5 dev build); Apertus 8B on demand or held (it holds cuda01's card); +where Alertmanager sends (its receiver is "none"); the test VMs 157–159, 165 +and 166 on pve1 kept or removed; whose the e2e-* VMs 151 and 153–156 are. + --- ## A. Security — do first @@ -1429,6 +1458,14 @@ console show and add them. - Done 2026-10-15: llmgw's `/mcp` with picture, 3D, VLM and workflow tools, so a client needs no server of its own (asked by the game team); the shared ComfyUI replica (H3's placement side). +- Done 2026-10-16 (v0.14.6, v0.14.7): setup lists every part with what it + is and its settings; three machines each run the control plane, with a + floating address (VRRP id from the address); a lost store primary costs no + request (3 s connect per member, keepalive and TCP_USER_TIMEOUT, Patroni + ttl 20, a standby leaves the lease alone a round after the store returns); + a new leader waits for its judge within the budget; a judge is asked again + on another machine at half its budget; init sizes the judge (2 GB, no + prompt cache, two cores, one judge on up to four cores on one machine). ## I. What other gateways do that llmgw does not (written 2026-10-16) @@ -1513,9 +1550,95 @@ answering 429 makes the whole fallback answer 429. - Done when: with two keys and one returning 429, every request is answered by the other, and the console shows the one breaker open. +## J. Found testing HA and single-machine installs (written 2026-10-16) + +Three VMs installed as HA from the setup screens, a fourth as a judge node, +a fifth as a single 4-core, 8 GB machine; machines powered off under load. + +### J1. A restart that has no room waits for ever (S) +`ctl restart --model M` starts each new replica beside the old one before +stopping it. On a node whose cores (or memory) are all reserved, as a +single machine's are once init divides them, the new one fits nowhere and +the restart waits with the old still running. Met on the 4-core VM: the +judge and the starter kept their old figures until their engines were +killed by hand. +- When the new replica fits nowhere beside the old, replace in place: stop + one old replica, then start its successor, saying so (the model has a + replica fewer meanwhile; with one replica it is down for the load time). +- `ctl restart` says which it is doing before it does it. +- Files: the reconciler's rolling restart (`internal/control/reconciler`), + `internal/cli/ctl_restart.go`. +- Done when: on a node with every core reserved, `ctl restart --model` + brings every replica to its new figures without anything killed by hand. + +### J2. A judge the leader adds is sized as init's is (S) +When a policy names a judge the catalog lacks, the leader serves it itself +(`internal/cli/judgekeeper.go` via `knownJudge.serveRequest` in +`judges_known.go`, through modelserve). That judge reserves 3 GB (the +weights, an 8192-token KV cache and a fixed gigabyte), no cores, and runs +with llama.cpp's prompt cache, which grew one to 3.2 GB. init's judge +(`guardModel` in `internal/cli/serverinit_cluster.go`) reserves 2 GB and +its cores and runs with `--cache-ram 0`. +- `modelserve` gives a guard model `--cache-ram 0`, and the keeper's + request the same memory and cores rule as `cpuPlan`. +- Done when: a judge removed from the catalog comes back with the same + figures init would have given it. + +### J3. HA setup puts the judges on nodes of their own (S) +Under load on a 4-core control plane that was also a node, 38 of 60 +requests were refused for want of a judgment. With the judges on a node of +their own (4 cores, 32 GB, two cores each) the same test refused 2. +- The HA setup offers a judge node or two: machines given like the store's, + added as nodes with a label (`llmgw.role: judge`), and the judge's + profile selects that label. +- Done when: the setup installs the judges on the machines given for them, + and the load test above refuses no more than it did by hand. + +### J4. The model mirror on more than one machine (M) +In an HA install the mirror stays on the first machine. While it is down, +models the nodes already have keep serving and new ones wait for it. +- A second mirror kept in step (each machine of the store, or one more), the + control planes and nodes given both, the first that answers used. +- Done when: with the first machine powered off, a model added then is + fetched and served. + +### J5. An upgrade corrects an earlier install's judge (S) +v0.14.6 and earlier installed two judges reserving no cores or one, with +2 or 4 GB and the prompt cache on. On fewer than eight cores every judgment +missed its 5 s budget and every request was refused. v0.14.7's notes say +how to fix it by hand (`ctl model resources`, then `ctl restart`). +- On upgrade, or on `server init` run again, a judge whose profile is still + exactly what an earlier init wrote is moved to today's figures and + restarted (after J1). One changed by hand is left alone and named. +- Done when: a v0.14.6 single-machine install upgraded to it judges within + budget without anything typed. + +### J6. Rate limits per process (note) +Rate limits are counted in each control plane's memory. A failover starts +every key's minute afresh. Documented in install.md; worth a shared count in +the store only if a customer's limits must hold across a failover. + +### J7. nginx in front of several control planes (S) +`deploy/proxy/Caddyfile` sends to whichever control plane answers `/leader`. +nginx's open-source build has no active health checks, so its `upstream` +block would send to standbys and to a lost leader alike. +- nginx.conf: the floating address as its one upstream, and why; or the + `max_fails` passive approach with what it costs. + +### J8. A judge budget that fits the machine (S) +A judgment takes 2.3 s on four cores and 4.2 on two, against the 5 s +default: two cores are close to it, and a busy machine passes it. +- At install, after the judge is up, time three judgments. If the slowest is + more than half the budget, say so in the plan's closing text with the + choices (more cores, a GPU judge, a longer `latency_budget`, `fail: + open`). The budget itself is not changed without being asked. +- Done when: an install on a machine too slow for its judge says so before + its first refused request. + ## D. Facts worth knowing before starting -- Admin API: `127.0.0.1:9090`, no auth, `ctlClient` in `internal/cli/ctl.go`. +- Admin API: `127.0.0.1:9090`, behind the token in + `/var/lib/llmgw/admin-token` (A5), `ctlClient` in `internal/cli/ctl.go`. Metrics on `:9091` with `/prometheus/targets` beside them. - Store: `internal/store`, migrations under `internal/store/migrations/{sqlite,postgres}`; the PoC runs PostgreSQL under Patroni.