Set up a control plane with one command
`llmgw server init` does what install.md's steps 2 and 3 asked of the operator: the llmgw user and directories, a certificate authority, the control plane's certificate for every address the machine has, a configuration, the default project, an administrator with a generated password, and the systemd unit — then starts it and waits for /health. Every step looks before it acts, so a second run changes nothing, and --host later reissues the certificate with the old names kept. It never overwrites server.yaml or an authority, and leaves alone certificates issued by an authority kept on another machine. The configuration install.md showed could not start: an empty catalog without control.enabled is refused as having nothing to serve. The one init writes sets it, and a test holds that. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
746d4852af
commit
2a381ad669
4 files changed
+873
-126
No files matched your search
+90
-126
@@ -29,7 +29,7 @@ The reason is boring and unavoidable: drivers and engines are platform decisions
|
||||
with kernel modules and version matrices attached.
|
||||
|
||||
Weights are the one item on that list llmgw does handle. It includes a mirror —
|
||||
step 4 — which holds the files on your own network and hands them to the nodes,
|
||||
step 3 — which holds the files on your own network and hands them to the nodes,
|
||||
so getting weights onto every machine is not a problem you solve before
|
||||
starting. What llmgw will not do is choose the models for you.
|
||||
|
||||
@@ -58,7 +58,8 @@ them, so nodes need no inbound access except your own SSH.
|
||||
| 8082 | the control plane administering the mirror | the control plane only — this port can delete models and start downloads |
|
||||
|
||||
Decide now which address nodes will use to reach the control plane. It goes into
|
||||
a certificate and changing it later means reissuing one.
|
||||
a certificate: the machine's own addresses are found for you, and any other name
|
||||
— a DNS alias, a NAT or floating address — has to be given with `--host`.
|
||||
|
||||
## 1. The binary
|
||||
|
||||
@@ -83,141 +84,104 @@ git clone https://gitea.mynodes.xyz/ilker/llmgw && cd llmgw
|
||||
CGO_ENABLED=0 go build -o llmgw ./cmd/llmgw
|
||||
```
|
||||
|
||||
## 2. The certificate authority
|
||||
## 2. The control plane
|
||||
|
||||
Agents authenticate to the control plane with mutual TLS, and a node's identity
|
||||
is taken from its certificate's common name. A public authority cannot supply
|
||||
these: Let's Encrypt and its peers stopped issuing certificates carrying the
|
||||
`clientAuth` usage, so nothing they hand out can authenticate an agent.
|
||||
|
||||
Run this wherever you intend to keep the authority. It talks to nothing and
|
||||
works on files, so it does not have to be a cluster machine — and there is an
|
||||
argument that it should not be.
|
||||
On the machine that will be the control plane, as root:
|
||||
|
||||
```
|
||||
sudo llmgw ctl cert init-ca --dir /etc/llmgw/pki
|
||||
sudo llmgw server init --admin you@example.com
|
||||
```
|
||||
|
||||
That writes `ca.pem` and `ca-key.pem`. **Whoever holds `ca-key.pem` can mint any
|
||||
node's identity**, so it stays here; `ca.pem` is copied to every machine and is
|
||||
not a secret.
|
||||
That is the whole step. In order, it:
|
||||
|
||||
Now the control plane's own certificate. Every address an agent might dial must
|
||||
appear in `--host`, bare IP addresses included — hostname verification reads
|
||||
subject alternative names and ignores the common name entirely, so a certificate
|
||||
naming the host only in its subject is rejected by every client:
|
||||
1. creates the `llmgw` system user, `/etc/llmgw` and `/var/lib/llmgw`
|
||||
2. creates the cluster's certificate authority in `/etc/llmgw/pki`
|
||||
3. issues the control plane's certificate for this machine's hostname and
|
||||
every address it has, and copies `ca.pem` beside it
|
||||
4. writes `/etc/llmgw/server.yaml`: mutual TLS, authentication, default limits
|
||||
and a price, and a control plane with an empty catalog
|
||||
5. creates the store, the `default` project, and your account, already approved
|
||||
and promoted to administrator — the password is generated and **printed
|
||||
once**
|
||||
6. installs the systemd unit, starts it, and waits until `/health` answers
|
||||
|
||||
Every step looks before it acts. Running it again finishes an install that
|
||||
stopped halfway and changes nothing on one that did not; it never overwrites
|
||||
`server.yaml`, the authority, or an existing account's password.
|
||||
|
||||
**The addresses in the certificate matter.** Agents check that the address
|
||||
they dial is named in it, and the certificate names only what this machine
|
||||
could see of itself. If nodes will reach it by another name — a DNS alias, a
|
||||
NAT address, a floating IP — give it:
|
||||
|
||||
```
|
||||
sudo llmgw ctl cert issue-server llmgw-ctl \
|
||||
--host 10.0.0.140 --host llmgw-ctl.internal \
|
||||
--dir /etc/llmgw/pki --out /etc/llmgw
|
||||
sudo llmgw server init --host llmgw-ctl.internal --host 203.0.113.10
|
||||
```
|
||||
|
||||
Run that at any time: the certificate is reissued with the new names added to
|
||||
the old ones, and the service restarted.
|
||||
|
||||
**`ca-key.pem` signs every node's identity.** `init` leaves it in
|
||||
`/etc/llmgw/pki`, because onboarding a node needs it and the control plane is
|
||||
where most people run onboarding from. Back it up; there is no recovering it.
|
||||
If this machine is exposed, move the directory elsewhere once the nodes are in
|
||||
and pass `--pki` to onboarding.
|
||||
|
||||
This covers the fleet-facing port only. The client-facing API is public and
|
||||
belongs behind an ordinary publicly trusted certificate, usually terminated by a
|
||||
reverse proxy in front of port 8080.
|
||||
|
||||
## 3. The control plane
|
||||
|
||||
### The user and directories
|
||||
|
||||
```
|
||||
sudo useradd --system --home /var/lib/llmgw --shell /usr/sbin/nologin llmgw
|
||||
sudo install -d -o llmgw -g llmgw -m 0750 /var/lib/llmgw
|
||||
sudo install -d -m 0755 /etc/llmgw
|
||||
sudo chown root:llmgw /etc/llmgw/ca.pem /etc/llmgw/server.pem /etc/llmgw/server-key.pem
|
||||
sudo chmod 0640 /etc/llmgw/server-key.pem
|
||||
```
|
||||
|
||||
### The configuration
|
||||
|
||||
`/etc/llmgw/server.yaml`. This is the smallest file that is not a toy: mutual
|
||||
TLS on, authentication on, one model, and limits that apply to anyone nobody has
|
||||
given limits to.
|
||||
|
||||
```yaml
|
||||
listen: ":8080"
|
||||
admin_listen: "127.0.0.1:9090"
|
||||
agent_listen: ":7443"
|
||||
|
||||
agent_tls:
|
||||
ca: /etc/llmgw/ca.pem
|
||||
cert: /etc/llmgw/server.pem
|
||||
key: /etc/llmgw/server-key.pem
|
||||
|
||||
auth:
|
||||
enabled: true
|
||||
store: /var/lib/llmgw/llmgw.db
|
||||
signup_open: true
|
||||
# Applied to any key no scope gives a limit to. Without these, a project
|
||||
# created without limits is metered by nothing at all, and the first sign of
|
||||
# that is the bill or the outage.
|
||||
default_rpm: 60
|
||||
default_tpm: 200000
|
||||
# What a request costs, which is what makes a budget mean anything. Without
|
||||
# a price every request costs zero, so a budget is displayed, counted against
|
||||
# nothing, and never binds — and nobody finds out until they expected it to
|
||||
# stop something. Self-hosted models have no per-token price, so the unit is
|
||||
# yours to choose: GPU-seconds, a notional rate, or 1000 everywhere, which
|
||||
# turns a budget into a plain token quota.
|
||||
default_price:
|
||||
prompt_per_1k: 1000
|
||||
completion_per_1k: 1000
|
||||
# pricing:
|
||||
# qwen2.5-coder-7b-instruct: { prompt_per_1k: 500, completion_per_1k: 1500 }
|
||||
|
||||
landing:
|
||||
contact: "platform@example.com"
|
||||
|
||||
# Start with none. Models are added while the cluster runs — see step 8 —
|
||||
# and anything listed here is declarative: it is in the catalog on every start
|
||||
# and cannot be removed without editing this file.
|
||||
models: []
|
||||
```
|
||||
|
||||
`admin_listen` is on loopback on purpose. It serves pprof, which can dump the
|
||||
process heap, and API keys sit in that heap in plaintext. To see the dashboard
|
||||
from elsewhere, promote an account and use `/admin` on the client port — do not
|
||||
move this listener.
|
||||
|
||||
The store file is created on first start; there is no separate migration step.
|
||||
|
||||
### The service
|
||||
|
||||
```
|
||||
sudo cp deploy/systemd/llmgw-server.service /etc/systemd/system/
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable --now llmgw-server
|
||||
systemctl status llmgw-server
|
||||
```
|
||||
|
||||
### Your account
|
||||
|
||||
Accounts are created and approved from a shell on this machine, deliberately:
|
||||
having a shell here is what makes someone an administrator, and there are no
|
||||
roles to get subtly wrong.
|
||||
|
||||
```
|
||||
sudo -u llmgw llmgw users create you@example.com --password 'change me now'
|
||||
# prints a user id, usr-xxxxxxxx
|
||||
sudo -u llmgw llmgw users promote usr-xxxxxxxx
|
||||
```
|
||||
|
||||
`promote` is what grants `/admin`. Sign in at `http://<control-plane>:8080/login`,
|
||||
change the password, and issue yourself a key from the profile page.
|
||||
|
||||
### Check it
|
||||
|
||||
```
|
||||
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/health # 200, with an empty body
|
||||
sudo -u llmgw llmgw ctl status
|
||||
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/health # 200
|
||||
llmgw ctl status
|
||||
```
|
||||
|
||||
`ctl status` will report no nodes and no models, which is correct — nothing has
|
||||
been added yet. It is worth running now so its output is familiar before you
|
||||
need it.
|
||||
been added yet. Sign in at `http://<control-plane>:8080/login` with the printed
|
||||
password, change it, and issue yourself a key from the profile page.
|
||||
|
||||
## 4. The mirror
|
||||
### The configuration
|
||||
|
||||
`/etc/llmgw/server.yaml` is yours from here on. Three things in it are worth
|
||||
knowing before you edit it:
|
||||
|
||||
- `admin_listen` is on loopback on purpose. It serves pprof, which can dump the
|
||||
process heap, and API keys sit in that heap in plaintext. To see the dashboard
|
||||
from elsewhere use `/admin` on the client port — do not move this listener.
|
||||
- `default_rpm`, `default_tpm` and `default_price` are what make limits and
|
||||
budgets mean anything. Without a price every request costs zero and a budget
|
||||
never binds; without the defaults, a project created without limits is
|
||||
metered by nothing at all.
|
||||
- `control.enabled: true` is what makes this a control plane while its catalog
|
||||
is empty. Without it, a file with no models is refused as having nothing to
|
||||
serve.
|
||||
|
||||
Every key is in [reference.md](reference.md).
|
||||
|
||||
### Keeping the authority off the control plane
|
||||
|
||||
Create it on another machine, issue the control plane's certificate there, and
|
||||
bring only the results:
|
||||
|
||||
```
|
||||
# on the authority's machine
|
||||
llmgw ctl cert init-ca --dir /etc/llmgw/pki
|
||||
llmgw ctl cert issue-server llmgw-ctl --host 10.0.0.140 --host llmgw-ctl.internal \
|
||||
--dir /etc/llmgw/pki --out ./ctl
|
||||
|
||||
# copy ca.pem, ctl/server.pem and ctl/server-key.pem into /etc/llmgw on the
|
||||
# control plane, then there:
|
||||
sudo llmgw server init --admin you@example.com
|
||||
```
|
||||
|
||||
`init` finds certificates in `/etc/llmgw` and no authority beside them, and
|
||||
keeps them as they are. Adding an address later is then done where the
|
||||
authority is; `init --host` says so rather than minting a second authority no
|
||||
node trusts.
|
||||
|
||||
## 3. The mirror
|
||||
|
||||
Do this before the nodes, not after. Every node fetches its own copy of a
|
||||
model's weights, and where they fetch from decides whether bringing up a cluster
|
||||
@@ -313,7 +277,7 @@ llmgw mirror fetch --model qwen2.5-14b-instruct --version 2026-03 \
|
||||
or from the admin page, or by copying files into the directory by hand — the
|
||||
mirror is a directory, and anything that puts files in it works.
|
||||
|
||||
## 5. Preparing a node
|
||||
## 4. Preparing a node
|
||||
|
||||
**Do this before onboarding.** A node missing its driver or engine registers
|
||||
healthy and fails on first placement, and the reason is three logs away.
|
||||
@@ -418,7 +382,7 @@ llmgw ctl engine-check vllm --bin /opt/venvs/vllm/bin/vllm
|
||||
|
||||
### Weights
|
||||
|
||||
Nothing to do here if you built the mirror in step 4 — the agent fetches what a
|
||||
Nothing to do here if you built the mirror in step 3 — the agent fetches what a
|
||||
model's `source` names, and that source is your mirror. This section is for the
|
||||
arrangements where the files are already on the machine.
|
||||
|
||||
@@ -441,7 +405,7 @@ model's `source` at a file to download and the agent fetches it once, before the
|
||||
engine is launched:
|
||||
|
||||
```yaml
|
||||
# your mirror, which is what step 4 was for
|
||||
# your mirror, which is what step 3 was for
|
||||
source: http://models.internal:8081/qwen2.5-coder-3b-instruct/2026-03/q4_k_m.gguf
|
||||
# a whole version, for a model of several files — the trailing slash is the signal
|
||||
source: http://models.internal:8081/llama-3.1-70b-instruct/2026-03/
|
||||
@@ -498,7 +462,7 @@ How much disk a node needs follows from the models rather than a rule of thumb.
|
||||
`llmgw ctl model plan <repo>` prints the sizes; as a starting point, a 70B at
|
||||
bf16 is 141 GB and the same model at q4 is 43 GB.
|
||||
|
||||
## 6. Adding the node
|
||||
## 5. Adding the node
|
||||
|
||||
Two halves, because they need different access: building the machine needs the
|
||||
hypervisor, and issuing its identity needs the certificate authority. Where one
|
||||
@@ -555,7 +519,7 @@ Repeat for each node. Nodes in the same fabric must be given the same `--fabric`
|
||||
name; nodes with no InfiniBand take no `--fabric` at all and are each their own
|
||||
island, which is fine for single-node models.
|
||||
|
||||
## 7. Setting the node's CPU model, if it is a VM
|
||||
## 6. Setting the node's CPU model, if it is a VM
|
||||
|
||||
Proxmox defaults a VM to `kvm64`, which hides AVX, AVX2, FMA and even SSE4.2
|
||||
from the guest. Inference then falls back to scalar code and is not slightly
|
||||
@@ -572,7 +536,7 @@ only change being `--cpu host`:
|
||||
probably did not, and nothing about it looks broken — every number is simply
|
||||
small. Check with `grep -qw avx2 /proc/cpuinfo` on the node.
|
||||
|
||||
## 8. Adding a model
|
||||
## 7. Adding a model
|
||||
|
||||
Write the definition. This is the same shape a model takes in `server.yaml`, so
|
||||
an entry moves between the two by being copied:
|
||||
@@ -714,7 +678,7 @@ An engine that publishes no metrics endpoint is measured from the outside only.
|
||||
The run says so, and queue depth and KV occupancy are then not part of the
|
||||
answer.
|
||||
|
||||
## 9. The first request
|
||||
## 8. The first request
|
||||
|
||||
```
|
||||
curl http://<control-plane>:8080/v1/chat/completions \
|
||||
@@ -957,7 +921,7 @@ the choice is yours rather than silent.
|
||||
|
||||
Three things, in descending order of how badly you want them.
|
||||
|
||||
**The certificate authority.** `ca-key.pem`, wherever you put it in step 2.
|
||||
**The certificate authority.** `ca-key.pem`, `/etc/llmgw/pki` unless you kept it elsewhere (step 2).
|
||||
Losing it does not stop the cluster, and it means every node identity has to be
|
||||
reissued the next time you touch one. Nothing in llmgw can reconstruct it.
|
||||
|
||||
@@ -1028,7 +992,7 @@ output, which is the actual diagnosis.
|
||||
node and the reason: wrong GPU class, not enough free cards, too little memory,
|
||||
no fabric, a label it does not carry.
|
||||
|
||||
**Everything is slow.** On a VM, check AVX2 (step 7). On CPU, check the
|
||||
**Everything is slow.** On a VM, check AVX2 (step 6). On CPU, check the
|
||||
quantisation — the difference between `q8_0` and `q4_k_m` is roughly a factor of
|
||||
two. `llmgw ctl status` shows how full the cluster is.
|
||||
|
||||
@@ -1065,7 +1029,7 @@ requests. On a slow model, set `timeouts.generation` higher.
|
||||
those eight local, which is most of the problem; on a fabric where the nodes
|
||||
can see each other, seeding between them is the obvious next thing to build.
|
||||
- **A budget without a price never binds.** Set `default_price` or `pricing`
|
||||
(step 3) or every request costs zero, and a budget is something the dashboard
|
||||
(step 2) or every request costs zero, and a budget is something the dashboard
|
||||
displays rather than something that stops anything. The quotas page says so
|
||||
where it can, which is not the same as being told before you rely on it.
|
||||
- **Engines are yours to install.** Onboarding verifies one runs; it installs
|
||||
|
||||
Reference in new issue
Block a user