Backlog: what is open, in order; J, what the HA and single-machine tests found
CI / test (push) Successful in 8m59s
CI / test (push) Successful in 8m59s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
af942aca27
commit
fbb5df05b8
1 file changed
+124
-1
+124
-1
@@ -15,6 +15,35 @@ Conventions this codebase keeps, which every item here must keep too:
|
||||
- `go test ./...`, `go vet ./...` and `govulncheck ./...` pass. No release is
|
||||
cut and nothing is deployed to the PoC cluster without being told to.
|
||||
|
||||
## Open, in order (2026-10-16, after v0.14.7)
|
||||
|
||||
What is left, most needed first. The detail is in the item each names.
|
||||
|
||||
1. **J1** — `ctl restart --model` waits for ever on a node whose cores are
|
||||
all reserved. A bug, met while testing v0.14.7. (S)
|
||||
2. **J5** — an upgrade leaves an earlier install's judge as it was. A
|
||||
v0.14.6 install on fewer than eight cores refuses every request until
|
||||
fixed by hand. (S)
|
||||
3. **I3** — what llmgw adds to a request, measured, in the README. (S)
|
||||
4. **I4** — several keys for one outside provider. (S)
|
||||
5. **J2** — a judge the leader adds by itself is sized as init's is. (S)
|
||||
6. **J3** — HA setup puts the judges on nodes of their own. (S)
|
||||
7. **J8** — a judge budget that fits the machine, measured at install. (S)
|
||||
8. **F5** — large imports limited by the embedding engine's slots. (M)
|
||||
9. **I1** — answers to similar questions from the cache. (M)
|
||||
10. **J4** — the model mirror on more than one machine. (M)
|
||||
11. **I2** — the customer's MCP servers behind llmgw. (L)
|
||||
12. Smaller leftovers: audit before/after values, a tenant-admin role and
|
||||
forms hidden from viewers (C1); Parquet and a scheduled push for usage
|
||||
export (C4); English errors on Turkish pages (C24); H1 tried on the PoC;
|
||||
licences in video and audio (H5); nginx in front of several control
|
||||
planes (J7).
|
||||
|
||||
Not code, the operator's to decide or do: the PoC on v0.14.7 (it runs a
|
||||
v0.14.5 dev build); Apertus 8B on demand or held (it holds cuda01's card);
|
||||
where Alertmanager sends (its receiver is "none"); the test VMs 157–159, 165
|
||||
and 166 on pve1 kept or removed; whose the e2e-* VMs 151 and 153–156 are.
|
||||
|
||||
---
|
||||
|
||||
## A. Security — do first
|
||||
@@ -1429,6 +1458,14 @@ console show and add them.
|
||||
- Done 2026-10-15: llmgw's `/mcp` with picture, 3D, VLM and workflow tools,
|
||||
so a client needs no server of its own (asked by the game team); the
|
||||
shared ComfyUI replica (H3's placement side).
|
||||
- Done 2026-10-16 (v0.14.6, v0.14.7): setup lists every part with what it
|
||||
is and its settings; three machines each run the control plane, with a
|
||||
floating address (VRRP id from the address); a lost store primary costs no
|
||||
request (3 s connect per member, keepalive and TCP_USER_TIMEOUT, Patroni
|
||||
ttl 20, a standby leaves the lease alone a round after the store returns);
|
||||
a new leader waits for its judge within the budget; a judge is asked again
|
||||
on another machine at half its budget; init sizes the judge (2 GB, no
|
||||
prompt cache, two cores, one judge on up to four cores on one machine).
|
||||
|
||||
## I. What other gateways do that llmgw does not (written 2026-10-16)
|
||||
|
||||
@@ -1513,9 +1550,95 @@ answering 429 makes the whole fallback answer 429.
|
||||
- Done when: with two keys and one returning 429, every request is answered
|
||||
by the other, and the console shows the one breaker open.
|
||||
|
||||
## J. Found testing HA and single-machine installs (written 2026-10-16)
|
||||
|
||||
Three VMs installed as HA from the setup screens, a fourth as a judge node,
|
||||
a fifth as a single 4-core, 8 GB machine; machines powered off under load.
|
||||
|
||||
### J1. A restart that has no room waits for ever (S)
|
||||
`ctl restart --model M` starts each new replica beside the old one before
|
||||
stopping it. On a node whose cores (or memory) are all reserved, as a
|
||||
single machine's are once init divides them, the new one fits nowhere and
|
||||
the restart waits with the old still running. Met on the 4-core VM: the
|
||||
judge and the starter kept their old figures until their engines were
|
||||
killed by hand.
|
||||
- When the new replica fits nowhere beside the old, replace in place: stop
|
||||
one old replica, then start its successor, saying so (the model has a
|
||||
replica fewer meanwhile; with one replica it is down for the load time).
|
||||
- `ctl restart` says which it is doing before it does it.
|
||||
- Files: the reconciler's rolling restart (`internal/control/reconciler`),
|
||||
`internal/cli/ctl_restart.go`.
|
||||
- Done when: on a node with every core reserved, `ctl restart --model`
|
||||
brings every replica to its new figures without anything killed by hand.
|
||||
|
||||
### J2. A judge the leader adds is sized as init's is (S)
|
||||
When a policy names a judge the catalog lacks, the leader serves it itself
|
||||
(`internal/cli/judgekeeper.go` via `knownJudge.serveRequest` in
|
||||
`judges_known.go`, through modelserve). That judge reserves 3 GB (the
|
||||
weights, an 8192-token KV cache and a fixed gigabyte), no cores, and runs
|
||||
with llama.cpp's prompt cache, which grew one to 3.2 GB. init's judge
|
||||
(`guardModel` in `internal/cli/serverinit_cluster.go`) reserves 2 GB and
|
||||
its cores and runs with `--cache-ram 0`.
|
||||
- `modelserve` gives a guard model `--cache-ram 0`, and the keeper's
|
||||
request the same memory and cores rule as `cpuPlan`.
|
||||
- Done when: a judge removed from the catalog comes back with the same
|
||||
figures init would have given it.
|
||||
|
||||
### J3. HA setup puts the judges on nodes of their own (S)
|
||||
Under load on a 4-core control plane that was also a node, 38 of 60
|
||||
requests were refused for want of a judgment. With the judges on a node of
|
||||
their own (4 cores, 32 GB, two cores each) the same test refused 2.
|
||||
- The HA setup offers a judge node or two: machines given like the store's,
|
||||
added as nodes with a label (`llmgw.role: judge`), and the judge's
|
||||
profile selects that label.
|
||||
- Done when: the setup installs the judges on the machines given for them,
|
||||
and the load test above refuses no more than it did by hand.
|
||||
|
||||
### J4. The model mirror on more than one machine (M)
|
||||
In an HA install the mirror stays on the first machine. While it is down,
|
||||
models the nodes already have keep serving and new ones wait for it.
|
||||
- A second mirror kept in step (each machine of the store, or one more), the
|
||||
control planes and nodes given both, the first that answers used.
|
||||
- Done when: with the first machine powered off, a model added then is
|
||||
fetched and served.
|
||||
|
||||
### J5. An upgrade corrects an earlier install's judge (S)
|
||||
v0.14.6 and earlier installed two judges reserving no cores or one, with
|
||||
2 or 4 GB and the prompt cache on. On fewer than eight cores every judgment
|
||||
missed its 5 s budget and every request was refused. v0.14.7's notes say
|
||||
how to fix it by hand (`ctl model resources`, then `ctl restart`).
|
||||
- On upgrade, or on `server init` run again, a judge whose profile is still
|
||||
exactly what an earlier init wrote is moved to today's figures and
|
||||
restarted (after J1). One changed by hand is left alone and named.
|
||||
- Done when: a v0.14.6 single-machine install upgraded to it judges within
|
||||
budget without anything typed.
|
||||
|
||||
### J6. Rate limits per process (note)
|
||||
Rate limits are counted in each control plane's memory. A failover starts
|
||||
every key's minute afresh. Documented in install.md; worth a shared count in
|
||||
the store only if a customer's limits must hold across a failover.
|
||||
|
||||
### J7. nginx in front of several control planes (S)
|
||||
`deploy/proxy/Caddyfile` sends to whichever control plane answers `/leader`.
|
||||
nginx's open-source build has no active health checks, so its `upstream`
|
||||
block would send to standbys and to a lost leader alike.
|
||||
- nginx.conf: the floating address as its one upstream, and why; or the
|
||||
`max_fails` passive approach with what it costs.
|
||||
|
||||
### J8. A judge budget that fits the machine (S)
|
||||
A judgment takes 2.3 s on four cores and 4.2 on two, against the 5 s
|
||||
default: two cores are close to it, and a busy machine passes it.
|
||||
- At install, after the judge is up, time three judgments. If the slowest is
|
||||
more than half the budget, say so in the plan's closing text with the
|
||||
choices (more cores, a GPU judge, a longer `latency_budget`, `fail:
|
||||
open`). The budget itself is not changed without being asked.
|
||||
- Done when: an install on a machine too slow for its judge says so before
|
||||
its first refused request.
|
||||
|
||||
## D. Facts worth knowing before starting
|
||||
|
||||
- Admin API: `127.0.0.1:9090`, no auth, `ctlClient` in `internal/cli/ctl.go`.
|
||||
- Admin API: `127.0.0.1:9090`, behind the token in
|
||||
`/var/lib/llmgw/admin-token` (A5), `ctlClient` in `internal/cli/ctl.go`.
|
||||
Metrics on `:9091` with `/prometheus/targets` beside them.
|
||||
- Store: `internal/store`, migrations under `internal/store/migrations/{sqlite,postgres}`;
|
||||
the PoC runs PostgreSQL under Patroni.
|
||||
|
||||
Reference in new issue
Block a user