Backlog: what is open, in order; J, what the HA and single-machine tests found

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-16 21:07:57 +02:00
1 parent af942aca27
commit fbb5df05b8
1 file changed
+124 -1
+124 -1
View File
@@ -15,6 +15,35 @@ Conventions this codebase keeps, which every item here must keep too:
- `go test ./...`, `go vet ./...` and `govulncheck ./...` pass. No release is
cut and nothing is deployed to the PoC cluster without being told to.
## Open, in order (2026-10-16, after v0.14.7)
What is left, most needed first. The detail is in the item each names.
1. **J1** — `ctl restart --model` waits for ever on a node whose cores are
all reserved. A bug, met while testing v0.14.7. (S)
2. **J5** — an upgrade leaves an earlier install's judge as it was. A
v0.14.6 install on fewer than eight cores refuses every request until
fixed by hand. (S)
3. **I3** — what llmgw adds to a request, measured, in the README. (S)
4. **I4** — several keys for one outside provider. (S)
5. **J2** — a judge the leader adds by itself is sized as init's is. (S)
6. **J3** — HA setup puts the judges on nodes of their own. (S)
7. **J8** — a judge budget that fits the machine, measured at install. (S)
8. **F5** — large imports limited by the embedding engine's slots. (M)
9. **I1** — answers to similar questions from the cache. (M)
10. **J4** — the model mirror on more than one machine. (M)
11. **I2** — the customer's MCP servers behind llmgw. (L)
12. Smaller leftovers: audit before/after values, a tenant-admin role and
forms hidden from viewers (C1); Parquet and a scheduled push for usage
export (C4); English errors on Turkish pages (C24); H1 tried on the PoC;
licences in video and audio (H5); nginx in front of several control
planes (J7).
Not code, the operator's to decide or do: the PoC on v0.14.7 (it runs a
v0.14.5 dev build); Apertus 8B on demand or held (it holds cuda01's card);
where Alertmanager sends (its receiver is "none"); the test VMs 157–159, 165
and 166 on pve1 kept or removed; whose the e2e-* VMs 151 and 153–156 are.
---
## A. Security — do first
@@ -1429,6 +1458,14 @@ console show and add them.
- Done 2026-10-15: llmgw's `/mcp` with picture, 3D, VLM and workflow tools,
so a client needs no server of its own (asked by the game team); the
shared ComfyUI replica (H3's placement side).
- Done 2026-10-16 (v0.14.6, v0.14.7): setup lists every part with what it
is and its settings; three machines each run the control plane, with a
floating address (VRRP id from the address); a lost store primary costs no
request (3 s connect per member, keepalive and TCP_USER_TIMEOUT, Patroni
ttl 20, a standby leaves the lease alone a round after the store returns);
a new leader waits for its judge within the budget; a judge is asked again
on another machine at half its budget; init sizes the judge (2 GB, no
prompt cache, two cores, one judge on up to four cores on one machine).
## I. What other gateways do that llmgw does not (written 2026-10-16)
@@ -1513,9 +1550,95 @@ answering 429 makes the whole fallback answer 429.
- Done when: with two keys and one returning 429, every request is answered
by the other, and the console shows the one breaker open.
## J. Found testing HA and single-machine installs (written 2026-10-16)
Three VMs installed as HA from the setup screens, a fourth as a judge node,
a fifth as a single 4-core, 8 GB machine; machines powered off under load.
### J1. A restart that has no room waits for ever (S)
`ctl restart --model M` starts each new replica beside the old one before
stopping it. On a node whose cores (or memory) are all reserved, as a
single machine's are once init divides them, the new one fits nowhere and
the restart waits with the old still running. Met on the 4-core VM: the
judge and the starter kept their old figures until their engines were
killed by hand.
- When the new replica fits nowhere beside the old, replace in place: stop
one old replica, then start its successor, saying so (the model has a
replica fewer meanwhile; with one replica it is down for the load time).
- `ctl restart` says which it is doing before it does it.
- Files: the reconciler's rolling restart (`internal/control/reconciler`),
`internal/cli/ctl_restart.go`.
- Done when: on a node with every core reserved, `ctl restart --model`
brings every replica to its new figures without anything killed by hand.
### J2. A judge the leader adds is sized as init's is (S)
When a policy names a judge the catalog lacks, the leader serves it itself
(`internal/cli/judgekeeper.go` via `knownJudge.serveRequest` in
`judges_known.go`, through modelserve). That judge reserves 3 GB (the
weights, an 8192-token KV cache and a fixed gigabyte), no cores, and runs
with llama.cpp's prompt cache, which grew one to 3.2 GB. init's judge
(`guardModel` in `internal/cli/serverinit_cluster.go`) reserves 2 GB and
its cores and runs with `--cache-ram 0`.
- `modelserve` gives a guard model `--cache-ram 0`, and the keeper's
request the same memory and cores rule as `cpuPlan`.
- Done when: a judge removed from the catalog comes back with the same
figures init would have given it.
### J3. HA setup puts the judges on nodes of their own (S)
Under load on a 4-core control plane that was also a node, 38 of 60
requests were refused for want of a judgment. With the judges on a node of
their own (4 cores, 32 GB, two cores each) the same test refused 2.
- The HA setup offers a judge node or two: machines given like the store's,
added as nodes with a label (`llmgw.role: judge`), and the judge's
profile selects that label.
- Done when: the setup installs the judges on the machines given for them,
and the load test above refuses no more than it did by hand.
### J4. The model mirror on more than one machine (M)
In an HA install the mirror stays on the first machine. While it is down,
models the nodes already have keep serving and new ones wait for it.
- A second mirror kept in step (each machine of the store, or one more), the
control planes and nodes given both, the first that answers used.
- Done when: with the first machine powered off, a model added then is
fetched and served.
### J5. An upgrade corrects an earlier install's judge (S)
v0.14.6 and earlier installed two judges reserving no cores or one, with
2 or 4 GB and the prompt cache on. On fewer than eight cores every judgment
missed its 5 s budget and every request was refused. v0.14.7's notes say
how to fix it by hand (`ctl model resources`, then `ctl restart`).
- On upgrade, or on `server init` run again, a judge whose profile is still
exactly what an earlier init wrote is moved to today's figures and
restarted (after J1). One changed by hand is left alone and named.
- Done when: a v0.14.6 single-machine install upgraded to it judges within
budget without anything typed.
### J6. Rate limits per process (note)
Rate limits are counted in each control plane's memory. A failover starts
every key's minute afresh. Documented in install.md; worth a shared count in
the store only if a customer's limits must hold across a failover.
### J7. nginx in front of several control planes (S)
`deploy/proxy/Caddyfile` sends to whichever control plane answers `/leader`.
nginx's open-source build has no active health checks, so its `upstream`
block would send to standbys and to a lost leader alike.
- nginx.conf: the floating address as its one upstream, and why; or the
`max_fails` passive approach with what it costs.
### J8. A judge budget that fits the machine (S)
A judgment takes 2.3 s on four cores and 4.2 on two, against the 5 s
default: two cores are close to it, and a busy machine passes it.
- At install, after the judge is up, time three judgments. If the slowest is
more than half the budget, say so in the plan's closing text with the
choices (more cores, a GPU judge, a longer `latency_budget`, `fail:
open`). The budget itself is not changed without being asked.
- Done when: an install on a machine too slow for its judge says so before
its first refused request.
## D. Facts worth knowing before starting
- Admin API: `127.0.0.1:9090`, no auth, `ctlClient` in `internal/cli/ctl.go`.
- Admin API: `127.0.0.1:9090`, behind the token in
`/var/lib/llmgw/admin-token` (A5), `ctlClient` in `internal/cli/ctl.go`.
Metrics on `:9091` with `/prometheus/targets` beside them.
- Store: `internal/store`, migrations under `internal/store/migrations/{sqlite,postgres}`;
the PoC runs PostgreSQL under Patroni.