Setup lets every part be chosen, each with its settings; three machines each run the gateway
CI / test (push) Failing after 30m4s
CI / test (push) Failing after 30m4s
The setup screens list the parts of an install: control plane, store, model mirror, this machine as a node, starter model, pictures and 3D, the judge, firewall, notifications, limits, on-demand turns and an offline bundle. Each says what it is or what is missing without it. Space turns a part on or off; Enter opens its settings in a dialog that explains it. A plan page lists what is installed and what is left out, and the same install as a command. Every setting is a server init flag too, written into server.yaml when init writes it. With three machines, the two database machines become control planes too, standing by (server add's steps), with an optional floating address. The mirror's management API answers all three control planes, so a standby that comes to lead can still fetch models. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
733e20924c
commit
94401e21e6
16 files changed
+2255
-483
No files matched your search
+67
-8
@@ -89,13 +89,36 @@ CGO_ENABLED=0 go build -o llmgw ./cmd/llmgw
|
||||
|
||||
**The short way is the screens.** On the machine that will be the control
|
||||
plane, run `sudo llmgw` with nothing after it. With nothing installed it asks
|
||||
whether this is one machine or three for high availability; for three, it asks
|
||||
for the other two — address, user, and an ssh key, the ssh agent, or a
|
||||
password, and the sudo password if sudo asks for one; or, for a machine
|
||||
nothing may log into, a join code to run there with `sudo llmgw join <code>`,
|
||||
which dials out to this one on port 7444 — checks all three before
|
||||
installing anything (system, room, clock, ports, internet, whether they reach
|
||||
one another), shows the plan, and installs. Passwords are held in memory for
|
||||
whether this is one machine or three for high availability, then lists what
|
||||
an install is made of, each part with what it is for and what is missing
|
||||
without it:
|
||||
|
||||
| Part | | Its settings (Enter) |
|
||||
|---|---|---|
|
||||
| Control plane | always | the administrator, other names for the certificate, the portal's language, the sign-up form |
|
||||
| Store | always | SQLite or PostgreSQL on this machine; with three machines, PostgreSQL under Patroni on the other two |
|
||||
| Control plane on all three | three machines, on | a floating address for whichever leads |
|
||||
| Model mirror | on | here, and in which directory, or on another machine |
|
||||
| This machine as a node | on for one machine | |
|
||||
| Starter model | on for one machine | which: Qwen2.5 0.5B, 1.5B or 3B, Qwen3 4B, 8B or 14B, or any GGUF on Hugging Face; its context |
|
||||
| Pictures and 3D | off | |
|
||||
| Guardrails' judge | on | refuse or serve flagged when it does not answer; copies; how long it is waited for; what it refuses |
|
||||
| Firewall | on | |
|
||||
| Notifications | off | the webhooks |
|
||||
| Limits and keys | always | requests and tokens a minute; the longest a key lives |
|
||||
| Taking turns on a card | always | how long an on-demand model goes unused, and is safe after loading |
|
||||
| Offline bundle | off | the bundle and its sha256 |
|
||||
|
||||
Space turns a part on or off; Enter opens its settings, which say more. What
|
||||
cannot go together is said before anything is done — a starter model with no
|
||||
mirror to fetch it into — and closing the message opens the setting that is
|
||||
wrong. For three machines it then asks for the other two — address, user, and
|
||||
an ssh key, the ssh agent, or a password, and the sudo password if sudo asks
|
||||
for one; or, for a machine nothing may log into, a join code to run there
|
||||
with `sudo llmgw join <code>`, which dials out to this one on port 7444 —
|
||||
checks all three before installing anything (system, room, clock, ports,
|
||||
internet, whether they reach one another). Last it shows the plan, part by
|
||||
part, what is left out and what that costs, and installs. Passwords are held in memory for
|
||||
the run only; a sudo password lets the user sudo without one until the install
|
||||
finishes, and is then taken back. At the end it shows the administrator's
|
||||
password and the command that does the same from a script.
|
||||
@@ -137,7 +160,16 @@ That is the whole step. In order, it:
|
||||
machine without room for it is told so within two minutes, and why.
|
||||
|
||||
`--no-mirror` and `--no-node` leave out 7 and 8, `--no-firewall` 9,
|
||||
`--no-guard-model` 10 and `--no-model` 11. An NVIDIA card with no working
|
||||
`--no-guard-model` 10 and `--no-model` 11. Every setting the screens ask for
|
||||
is a flag too, and the plan shows the command with them: `--starter qwen3-8b`
|
||||
(or `hf:<owner>/<repo>/<file>.gguf`) and `--starter-context`, `--mirror-dir`,
|
||||
`--judge-fail open`, `--judge-categories`, `--judge-replicas`,
|
||||
`--judge-budget`, `--language tr`, `--signup-open`, `--default-rpm`,
|
||||
`--default-tpm`, `--max-key-ttl`, `--media` (pictures and 3D: ComfyUI on the
|
||||
nodes, and `presets.auto`), `--evict-after-idle` and `--min-lifetime`. They
|
||||
go into `server.yaml` as init writes it, and become the cluster's first
|
||||
settings when it starts; a `server.yaml` already there is never overwritten,
|
||||
so run again they change nothing — the settings page does. An NVIDIA card with no working
|
||||
driver is reported rather than silently ignored; install the driver and run
|
||||
init again to serve on it.
|
||||
|
||||
@@ -2319,6 +2351,33 @@ Without authentication, `threads.store` and `threads.retain_hours` in
|
||||
`server.yaml` keep both conversations and notes in one shared pool. That is for
|
||||
one trusted machine only; see [development.md](development.md).
|
||||
|
||||
## Three machines, each a control plane
|
||||
|
||||
The screens' three machines are high availability for the gateway as well as
|
||||
the store: the two database machines are made control planes too, standing
|
||||
by while the first leads — what `server add`, below, does for one machine.
|
||||
By command, `--standbys` with `--patroni`, and `--vip` for the floating
|
||||
address:
|
||||
|
||||
```
|
||||
sudo --preserve-env=SSH_AUTH_SOCK llmgw server init --admin you@example.com \
|
||||
--patroni ubuntu@10.0.0.6 --patroni ubuntu@10.0.0.7 \
|
||||
--standbys --vip 10.0.0.160/24 --no-node --no-model
|
||||
```
|
||||
|
||||
| Machine | Runs |
|
||||
|---|---|
|
||||
| the first | control plane, leading at first; model mirror; etcd |
|
||||
| the other two | control plane, standing by; PostgreSQL under Patroni; etcd |
|
||||
|
||||
etcd votes on all three and PostgreSQL is on two, so any one machine can be
|
||||
lost: a standby leads within seconds, Patroni promotes the standby database
|
||||
if it was the primary, and the floating address follows whichever leads. The
|
||||
mirror's management API answers the three control planes' addresses, not
|
||||
only its own machine's, so a standby that comes to lead can still fetch and
|
||||
add models. The mirror itself stays on the first machine: while it is down,
|
||||
models the nodes already have keep serving, and new ones wait for it.
|
||||
|
||||
## Two control planes
|
||||
|
||||
Once the store is replicated (`server init --patroni`, above), one command on
|
||||
|
||||
Reference in new issue
Block a user